The goal of this project is to supplement or replace the precautionary principle with decision guidance that better handles both normative and empirical uncertainty in contexts of speculative but potentially catastrophic consequences. It has been claimed that emerging technologies such as biotechnology or machine intelligence could have catastrophic impacts on human civilization or the biosphere, indicating the need for precaution until scientific uncertainty has been resolved. Yet it is unclear how to apply the precautionary principle to cases where the deeper investigations of scientific uncertainties that it calls for can themselves be a source of catastrophic risk (in e.g. gain-of-function research). Furthermore, the precautionary principle fails to account for moral uncertainty—even though many decisions depend more sensitively on ethical parameters (e.g. our obligations to future generations, discount rates, intrinsic value of nature) than on remaining scientific uncertainties.
Building on recent advances in decision theory, computational modelling, and domain-specific risk assessment techniques, and using tools of analytic and moral philosophy, this project will: (1) develop methods that account for normative uncertainty by combining voting theory with ongoing work on moral uncertainty, identifying parameters that make the largest practical difference; (2) combine these with methods for dealing with sources of empirical ignorance (such as information hazards, anthropic shadows, model uncertainties) into a theoretically well-motivated framework that comprises normative and empirical uncertainty; (3) derive mid-level principles and procedures from this theoretical framework by working through three case studies (information hazards, dual-use biotechnology, and automation and machine intelligence) offering better practical guidance on speculative but potentially catastrophic risks than does the precautionary principle.
