Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes

1 Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes Toby Ord, Rafaela Hillerbrand, Anders Sandberg* Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calculations often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation. When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons such as a flaw in the underlying theory, a flaw in the modeling of the problem, or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect. We develop this idea formally, explaining how it differs from the related distinctions of model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it. 1. Introduction Large asteroid impacts are highly unlikely events.1 Nonetheless, governments spend large sums on assessing the associated risks. It is the high stakes that make these otherwise rare events worth examining. Assessing a risk involves consideration of both the stakes involved and the likelihood of the hazard occurring. If a risk threatens the lives of a great many people it is not only rational but morally imperative to examine the risk in some detail and to see what we can do to reduce it. This paper focuses on low-probability high-stakes risks. In section 2, we show that the probability estimates in scientific analysis cannot be equated with the likelihood of these events occurring. Instead of the probability of the event occurring, scientific analysis gives the event’s probability conditioned on the given argument being sound. Though this is the case in all probability estimates, we show how it becomes crucial when the estimated probabilities are smaller than a certain threshold. To proceed, we need to know something about the reliability of the argument. To do so, risk analysis commonly falls back on the distinction between model and parameter uncertainty. We argue that this dichotomy is not well suited for

• Future of Humanity Institute, University of Oxford. 1 Experts estimate the annual probability as approximately one in a billion (Near-Earth Object Science Definition Team 2003). 2 incorporating information about the reliability of the theories involved in the risk assessment. Furthermore the distinction does not account for mistakes made unknowingly. In section 3, we therefore propose a three-fold distinction between an argument’s theory, its model, and its calculations. While explaining this distinction in more detail, we illustrate it with historic examples of errors in each of the three areas. We indicate how specific risk assessment can make use of the proposed theory-model-calculation distinction in order to evaluate the reliability of the given argument and thus improve the reliability of their probability estimate for rare events. Recently concerns have been raised that high-energy experiments in particle physics, such as the RHIC (Relativistic Heavy Ion Collider) at Brookhaven National Laboratory or the LHC (Large Hadron Collider) at CERN, Geneva, may threaten humanity. If these fears are justified, these experiments pose a risk to humanity that can be avoided by simply not turning on the experiment. In section 4, we use the methods of this paper to address the current debate on the safety of experiments within particle physics. We evaluate current reports in the light of our findings and give suggestions for future research. The final section brings the debate back to the general issue of assessing low- probability risk. We stress that the findings in this paper are not to be interpreted as an argument for anti-intellectualism, but rather as arguments for making the noisy and fallible nature of scientific and technical research subject to intellectual reasoning, especially in situations where the probabilities are very low and the stakes very high.

2. Probability Estimates Suppose you read a report which examines a potentially catastrophic risk and concludes that the probability of catastrophe is one in a billion. What probability should you assign to the catastrophe occurring? We argue that direct use of the report’s estimate of one in a billion is naïve. This is because the report’s authors are not infallible and their argument might have a hidden flaw. What the

…(Full text truncated)…

This content is AI-processed based on ArXiv data.