6. An Analysis and Evaluation of Methods Currently Used to Quantify the Likelihood of Existential Hazards

SJ Beard, Thomas Rowe and James Fox

© 2024 SJ Beard, Thomas Rowe & James Fox, CC BY-NC 4.0 https://doi.org/10.11647/OBP.0360.06

Highlights:

• This chapter examines and evaluates the range of methods that have been used to make quantified claims about the likelihood of existential hazards, drawing on a comprehensive literature review of 67 such claims, across 13 kinds of hazard, that are presented in an appendix.
• The chapter uses an informal evaluative framework to consider the relative merits of these methods, regarding their rigour, ability to handle uncertainty, accessibility for researchers with limited resources, and utility for communication and policy purposes.
• The authors find that each method has advantages and disadvantages but there is no uniquely best way to quantify existential risk. Nevertheless, they argue that some methods may be more suitable to certain purposes within Existential Risk Studies and that some methods, especially the Delphi technique and related forms of Structured Expert Solicitation, should be more widely used within the community.
• More importantly, however, they find that, in many cases, claims based on poor implementations of these methods are still frequently invoked by the Existential Risk Studies community, despite the existence of more methodologically robust estimates.
• The chapter ends with a call for a more critical approach to the selection and implementation of methodologies and the use of quantified claims within Existential Risk Studies, and argues that a greater awareness of the diverse methods available to these researchers should form an important part of this.

This chapter was published in 2020 after a multi-year review of evidence and methodology by the authors and a preliminary presentation and discussion at the 2018 Cambridge Conference on Catastrophic Risk. Its calls for more critical reflection on methodological selection and implementation, as well as for the greater use of structured expert solicitation, continue to inform the developing field of Existential Risk Studies. The uses of horizon-scanning tools like Delphi is further discussed in Chapter 7, while an applied example of this type of methodology is continued in Chapter 15. Chapter 10 meanwhile offers an example of using historical data in Existential Risk Studies and Chapter 13 shows one method for constructing system models from empirical data.

Apocalyptic predictions require, to be taken seriously, higher standards of evidence than do assertions on other matters where the stakes are not as great.

— Carl Sagan1

How likely is it the next century will see the collapse of civilisation or the extinction of humanity, and how much should we worry about different hazards that constitute this risk? These questions seem to bedevil the study of existential risk. On the one hand, they are important questions that deserve answers, not only to assess the level of risk facing humanity but as part of an integrated assessment of existential risk and opportunities to mitigate it.2 On the other hand, the quantification of existential risk is extremely challenging. As Carl Sagan pointed out, part of the problem with apocalyptic pronouncements is their theoretical basis and the fact that they “are not amenable to experimental verification — at least not more than once”.3 Not only would a human extinction event be unprecedented, but the risk of such an event tends to emerge at the interaction of complex social, environmental and economic systems that are hard to model, and there is a substantial degree of uncertainty about the second-order “risk space” where they might be found. Together, these problems have made the quantification of risk speculative and reliant on new and creative methods for analysing the threats humanity faces.4

Previous analysis by Bruce Tonn and Dorian Stiefel attempted to resolve this tension, by evaluating a range of methods for quantifying existential risk from an ideal perspective, and made several important recommendations, most of which we endorse.5 However, although the number of researchers developing methods for the quantification of existential risk has grown, these methods have only been applied in a piecemeal fashion to a limited number of disciplines. For instance, conservation biologists have made important innovations in the use of structured expert elicitation; analysts in the Intelligence Community have developed new forecasting techniques with a high degree of success in assessing the probability of events in the near future and climate scientists, economists and epidemiologists have developed powerful models of complex global systems. These, together with the range of existing techniques — from philosophical analysis and opinion polling to toy modelling and fault trees — each have benefits and limitations, and we believe it is time they were more widely understood and adopted. The aim of this chapter is, therefore, to survey the literature on the quantification of existential risk, to introduce different techniques to new audiences and to give an informal assessment of their capabilities, together with some suggestions for how they can be implemented and improved. We hope that this will spur still more methodological diversification and development.

This chapter is addressed primarily to a group of scholars we refer to as the Existential Risk Studies community. This is made up both of researchers who work within institutes that focus on the study of existential and Global Catastrophic Risk (such as the Future of Humanity Institute, the Centre for the Study of Existential Risk, The Global Catastrophic Risk Institute and the Future of Life Institute) and those who are consciously seeking to align their research with the goal of understanding and managing such risks. However, it is written from an awareness that these two groups do not necessarily contain all of those researchers whose work can be expected to contribute to our understanding and management of existential risk, and indeed many of the sources we consider were produced by researchers who are in neither of these groups. It is our conviction that one of the key roles of existential risk organisations like the above should not only be to support researchers who see themselves as falling into the Existential Risk Studies community but to forge better connections with, and a better understanding of, all research that is relevant to the understanding and management of these risks.

Finally, it is worth noting that this chapter does not seek to consider fully every aspect of the quantification of existential risk. The risk arising from a specific threat is given by multiplying the probability of the threat occurring with its expected severity. This chapter only sets out to examine the methodologies used to quantify the former. If one held the severity of all threats constant, at the point of human extinction, then this is all one needs to know. However, while some studies in this chapter do aim to assess this, others assess threats at a lower point of severity, such as that of causing a global catastrophe or civilisational collapse, while others do not specifically consider the severity of a threat but are included rather because they relate to potential scenarios that have been of interest to scholars of existential risk. In these cases, one needs to be mindful that the severity of the event is still to be determined, or at best is only imprecisely defined when considering the overall quantification of that risk.

We begin by noting that, while this chapter refers to the probability of existential risk, there are multiple ways of understanding the notion of probability. The first of these is the frequentist, or objective, notion of probability. According to this approach, probabilities are fundamentally related to the frequencies of events based on past observations. Once an experiment has been repeated many times, the frequency of any observed phenomena indicates its underlying regularity. Therefore, frequentist probability claims are sensitive to the experimental setup and measuring technique, and any new evidence requires probabilities to be reassessed from scratch. The second notion of probability is the Bayesian, or subjective, account, according to which probabilities represent our level of belief that a phenomenon will occur. One begins with a subjective prior belief about the probability of an event and then updates this via Bayes’ Theorem (or Bayes’ rule), which specifies how additional information affects the probability of an event. Even though these probabilities are subjective, one is required to set out reasons for arriving at them, which allows others to challenge these or update them further.

In this chapter, we discuss the two notions of probability interchangeably with only a few passing remarks and we will instead focus on the quality of the methodologies that can be used to produce both kinds of probabilistic statement without prejudice. This is because even though Bayesian notions of probability dominate the field of existential risk, there are some areas — most notably in the domain of public health — where frequentist notions of probability are more common.

The term “existential risk” can be understood in many ways, and clarifying its definition is undoubtedly a crucial concern. However, since our aim is merely to study methods of quantifying existential risk, which approach this term from multiple angles, we will take a very broad view of how the term should be used, encompassing human extinction, civilisational collapse and any major catastrophe commonly associated with these things.

Another point is that most studies consider existential risk in terms of distinct threats (such as nuclear war, pandemics and climate change). However, global catastrophes tend to involve a combination of multiple factors, including a precipitating catastrophic event, a systemic collapse that spreads this catastrophe to the global scale and a failure to take adequate steps to mitigate this risk.6

Finally, existential risk cannot be studied in a vacuum. Even our assessment of such risks can profoundly affect them. For instance, if we take existential threats more seriously, this may lead to greater efforts to mitigate them. Sometimes, risk assessments can take account of human activities, such as when multiple estimates of catastrophic climate change reflect different future emission paths; however, this is not always possible. Whilst important, we do not see this as a problem that we must solve here, since it is common to many fields of risk analysis and affects all the methods we describe to a greater or lesser extent.

Whilst some futurists may respond to these difficulties by adopting a pluralist conception of multiple futures, in which the goal is to map out the likely consequences of decisions that we face in the present, existential risk mitigation must go beyond this. In particular, it is necessary for research and mitigation efforts to be prioritised and for risk-risk trade-offs to be undertaken, such as when assessing dual-use technologies; these require the quantification of risk. We therefore believe that it is imperative to combine such pluralistic future scenarios into an integrated assessment that takes account of factors such as the resilience of global systems and the magnitude of existential risk. Given that such assessments are at an early stage, however, we will generally assume that all risk assessments are being made against a “business as usual” scenario, where people continue mitigating risk roughly as much as they did at the time when that risk was assessed. In general, we suspect that this will overstate the future level of risk because it misses the potential for technological and governance interventions. However, that may not always be the case, as economic development can systematically push global systems into a more fragile state, making existential risk increase over time.

As well as presenting and discussing the existing methods for quantifying existential risk, we will provide an informal assessment of each according to the following four criteria:

Uncertainty: How well do they handle the — generally considerable — uncertainty in this field? Three key considerations for this are: 1) whether they provide opportunities to quantify the level of confidence or uncertainty in their estimates, 2) whether the application of this method tends to systematically ignore or compound sources of uncertainty in the process of forming a final judgement and 3) whether they can help to identify and overcome epistemic bias.

For each of these, we assess methods on a four-point scale from Very Low to High and summarise the results of this evaluation in Appendix B. We do not mean to imply that these four criteria are of equal importance; however, their relative importance is likely to depend upon the context for which an assessment of existential risk is being produced.

2. Analytical Approaches

Not all methods for attempting to quantify existential risk are based on specific evidence for that risk. Given the lack of evidence available, this is perhaps more appropriate than it seems.

By far the most widely discussed of these is the so-called “doomsday argument”, which was developed by multiple philosophers including Brandon Carter, Richard Gott and John Leslie. It is a statistical argument about the probability that any given human observer will be at a particular place in human history. There are multiple versions of this argument. Some appeal to our current position in the timeline of human history — for instance, it is much more likely that human history thus far represents more than 5% of all human history than that we have more than 95% of our history ahead of us. Others appeal to our position in the population of all human observers — it is much more likely that the human population born before me represents at least 5% of all humans than that 95% of humans will be born after me.

These arguments are controversial and they have been used to justify very different claims about the probability of existential risk, from a less than 2.5% chance that we will fail to survive for at least another 5,100 years,7 to a 20% probability that we will not survive the next century.⁸ Important factors that determine the difference between these claims include the type of doomsday argument being deployed, one’s assessment of the length of human history or the number of humans who will ever exist and one’s assumptions about the temporal distribution of existential risk (whether it is evenly distributed over time or comes in peaks and troughs). In theory, some of the uncertainty about these could be quantified by providing multiple calculations with differing assumptions; however, in practice, this is seldom done.

Two other analytical arguments are sometimes discussed in relation to their implications for the likelihood of human extinction. The first is known as the Simulation Argument, which connects the probability that humans face imminent extinction to the probability that we are living in a computer simulation.9 The second, known as the Great Filter Argument, connects the probability that humans face imminent extinction to the probability that there is intelligent life on other planets.10 Several sources cite a 2006 working paper titled ‘The Fermi Paradox: Three Models’, by Robert Pisani of the Department of Statistics at UC Berkeley, as providing a quantification of existential risk based on the Great Filter argument. However, the paper was never published, and no version is currently available online. Apart from this, these arguments have yet to be used for the purpose of existential risk assessment.11

Part of the explanation for the interest in, and use of, the doomsday argument and other analytical tools is their high level of accessibility. They score Very Low across the other three categories. However, this does not mean these approaches should have no role to play in quantifying risk, as they can still inform people’s prior beliefs about human extinction, i.e. what we might most reasonably guess about the likelihood of human extinction before considering the evidence. A Bayesian account of probability requires such a prior to begin. In most scientific cases, the prior’s precision is relatively unimportant as one can continually update it with evidence. However, within the context of existential risk, where evidence is sparse, prior probabilities are disproportionately important, meaning that any technique helping produce better priors can still be very useful. This is the approach taken by Leslie12 and Wells.¹³

3. Modelling-Based Approaches

Since it is not possible to undertake an empirical study of human extinction or global civilisational collapse, the next set of methods use observable evidence to produce a set of assumptions, or a model, that allows us to study them indirectly.

The simplest approach of this kind involves assuming there exists an underlying regularity in the frequency of certain events that have historically already occurred and have the potential to pose an existential risk in the future. A frequentist analysis of historical data can then be used to estimate the approximate time interval between such events and hence produce an annual occurrence probability. So far, this approach has been applied to asteroid impacts,14 supervolcanic eruptions,15 nuclear wars,16 space weather,¹⁷ particle physics experiments,¹⁸ and the occurrence of extinctions and global catastrophes in general.19 Yampolskiy also applies this approach to predicting AI failures but without producing a quantified estimate of their probability.20

Where no such underlying frequency can be assumed — for instance, because an event is historically unprecedented — it is possible to produce simple toy models that can allow one to use historical data to determine the probability of an event occurring in other ways. Firstly, one can assume that the magnitude of impact from certain threats follows a specific distribution, enabling one to estimate the probability of a large impact event taking place from the historical record of smaller impact events.²¹ For instance, Millett and Snyder-Beattie²² for the number of fatalities from biowarfare or bioterrorism and Riley for solar flares both assume power law distributions,²³ while Bagus assumes that the fatalities from influenza pandemics follow an exponential distribution.²⁴ From this, they can estimate the probability of more extreme events of this kind that have the potential to pose and existential risk. Other toy models include Day, André & Park,25 Fan, Jamison and Summers,²⁶ and Millet and Snyder-Beattie,²⁷ all of which assess the risks of catastrophic pandemics.

Secondly, one could assume that a currently unprecedented event would occur as a consequence of multiple events with historical precedents. The existential event could be the end result of a chain of precipitating events each a possible consequence of another. In this case the unprecedented event’s probability is the product of the conditional probabilities down the chain. Klotz and Sylvester,²⁸ Lipstich and Inglseby,²⁹ and Fouchier³⁰ use this method to estimate the likelihood and impact of a global pandemic resulting from Gain of Function influenza research by assuming that such a risk results from the occurrence of two events whose probabilities are easier to determine, a laboratory-acquired infection and a biosecurity failure. Similarly, Hellman estimates the probability of a nuclear war resulting from a “Cuban Missile Type Crisis” as the product of probabilities of a sequence of four precipitating events.31 Alternatively, the unprecedented event could arise due to the coincidence of two mutually independent events. This would create the basis for a fault tree, discussed in the next section.

Although these approaches are methodologically clear, clarity does not imply objectivity. For instance, the analysis of the geological record is itself speculative and open to interpretation,32 so that Decker’s estimate of the probability of supervolcanic eruptions,33 while widely cited in the Existential Risk Studies community, is often seen as pessimistic amongst volcanologists and is higher than most other estimates. Other instances of disagreement are even greater, Lipsitch and Inglesby³⁴ and Fouchier³⁵ arrive at probability estimates over seven orders of magnitude apart, despite using the same toy model to evaluate the same risk. This points to the central issue, approaching the same historical data in different ways can result in very different probability estimates. Love comments on the “limited accuracy of statistical estimates” when comparing his result to Riley’s for the recurrence probability of Carrington-like geomagnetic storms, saying that we can only conclude that the probability is “somewhere between vanishingly unlikely and surprisingly likely”.³⁶ Meanwhile, Hellman openly cherry picks evidence so as to not appear “alarmist”.³⁷

Another problem with these methods is that some events are excluded from the historical record because of the “anthropic shadow” they would leave. Roughly speaking, were any such event to have occurred in the past, this would have led to the non-existence of the present observer. Therefore, even if its probability was extremely high, it must seem to us as if it could not have happened because our very existence depends on it.³⁸ David Manheim looked at risk estimates of natural pandemics and concluded that there is significant uncertainty about the relationship between historical patterns and present risk because of such “anthropic factors and other observational selection biases”.39 Tegmark and Bostrom also take account of this effect when quantifying the threat from particle physics experiments, but most theories ignore it. A related issue is that the historical record may need to be understood not just by what happened but also what didn’t happen. Gordon Woo argues we should sometimes incorporate “counterfactual analysis” of near-miss events to more accurately model risks because of the likelihood that very rare events will be underrepresented in historical data.40 For instance, if the true underlying probability of a certain sized asteroid striking earth was 0.004 per year and we have 100 years of data, then it is most probable that this event would not have occurred within this period leading our analysis to underestimate its probability. However, if we had evidence of asteroids of this size passing close to the earth without striking it during this period then we can use this information to arrive at a better estimate for the underlying probability of such a strike. Woo cites numerous “near-misses” in fields from maritime and air disasters to terrorism that can be used to make better predictions about rare kinds of catastrophe,41 while other scholars have incorporated historical near misses into the study of nuclear war.42

In principle, using historical data allows us to calculate our degree of uncertainty via simple statistical techniques. Thus, many of the sources that utilise this approach provide ranges for the probability of the events that they study. However, care needs to be taken when combining uncertain statistics in toy models, since merely multiplying uncertainty ranges will overstate uncertainty as the coincidence of multiple outliers can be expected to be rarer than any individual outlier. Furthermore, because the modelling approaches we discuss make assumptions about the distribution of the underlying phenomena from which the data has been sampled, they are vulnerable to abnormal (or even discontinuous) changes in these phenomena. For instance, Fouchier points out that when using historical data about biosecurity, one must account for generally increasing safety standards. Similarly, Manhiem notes that the likelihood of Global Catastrophic Biological Risks might be higher than historical evidence implies because of contemporary global travel, high population densities in megacities, and closer contact with animal populations due to factory-farms.43 Therefore, simplistic frequentist approaches may underestimate the appropriate level of uncertainty.

Despite its simplicity and popularity, we believe that toy modelling of existential risk has serious shortcomings. We rate these methods Low for rigour, uncertainty and utility, while once again noting their often underutilised potential to quantify and estimate uncertainty. Indeed, the utility of this approach appears to be highest in cases where assessors are overconfident, both regarding the uncertainty surrounding their predictions and the objectivity of the historical record, which is potentially dangerous. We do however rate this method highly in terms of its accessibility and the opportunities it provides for researchers to bring existing evidence to the study of existential risk. We see this approach as having limited appeal for quantifying existential risk as the field matures, though it may still play a useful role in stimulating further research and in producing estimates that can be taken as “objective” — priors for further Bayesian analysis.

A more sophisticated modelling technique for studying existential risk involves fault trees. Originally developed to model the emergence of system failures in safety engineering, they have now been widely applied in risk analysis. Fault tree models use Boolean algebra to map out in a logic tree, using “and” and “or” gates, how a system failure could arise. Branching backwards from overall system failure at the tree’s top, we first write the ways failure could happen as different nodes and then branch further backwards with how this node could fail and so on. If possible, a probability of failure is assigned to each node and then one can sum or multiply probabilities, depending on the Boolean nature of each gate, to give the overall probability of system failure. Importantly, fault trees can also clearly reveal preventative steps that could be taken. This technique has been used to study risks from nuclear war44 and AI safety.45

Bayesian networks are an extension of fault trees and present great promise for existential risk analysis.46 Although the power of Bayesian networks is also rooted in its graphical representation of the possible failure under study, the nodes now represent random variables, and it is the edges between these nodes that are quantified. These edges are directed from parent to child nodes, and every node comes with a conditional probability table to provide the causal probabilistic strengths for the edges between connected nodes. As with fault trees, a Bayesian network is first drawn by working backwards from a failure state (or any other outcome one wishes to study) through the conditions that might lead to this state. However, unlike fault trees, Bayesian networks can handle dependencies between different parts of the system and so all conditions can be factored into one’s analysis, including those that are only rarely important. This can make it easier to incorporate information from near-misses and other tenuous sources whilst still being rooted in observed system behaviour. Once a network has been created, Bayes’ rule can determine the final expected occurrence probability of each node in this network, including that of the final outcome, and this can be calculated dynamically and updated continuously to take account of additional information.

Khakzad et al. show that fault trees are less suitable for modelling complex systems because Bayesian networks are far superior at handling dependencies between different parts of the system, common cause failures, and uncertainty.47 Also, Bobbio et al. demonstrated that every fault tree has a corresponding Bayesian network and so the methodology is proven to be generalisable.48 However, when evidence is sparse, the results from Bayesian networks can be significantly affected by the modeller’s choice of prior probabilities. Bayesian network analysis has not yet been used to study existential risk directly; however, it has been applied to a variety of catastrophe models — for instance, Li et al. demonstrated how Bayesian networks can assess catastrophic risks under uncertainty by modelling catastrophic flooding in China.49 Both fault trees and Bayesian networks improve on simpler toy models as they can manage more complex system dynamics whilst handling a greater range of data and inputs. We rate both approaches as Medium for utility but note their particular value in providing insights that can be used to study risk mitigation by modelling how changes in components of a system affect its probability of failure. To be useful, the model must be a sufficiently faithful and detailed representation to capture accurately the effect of individual policies or interventions. It is possible that if the model fails on either of these, then it may simply lead decision-makers astray and give them false confidence. This is reflected in our higher degree of uncertainty about this particular classification.

Bayesian networks outperform fault trees in terms of both rigour (High/Medium compared to Medium) and uncertainty (High compared to Low), performing especially well in relation to the quantification of uncertainty. However, fault trees outperform Bayesian networks in terms of accessibility (Medium compared to Low), a category in which neither method performs well as both require significant modelling skills and domain knowledge. Fault tree analysis requires considerable familiarity with the underlying system to understand the processes that may lead to it failing. Bayesian networks require an even greater degree of familiarity as the assessor must provide probabilities for all the conditional relationships that may play a role in determining an outcome.

Some researchers, usually from outside the Existential Risk Studies community, have also adapted existing models to study globally catastrophic and existential threats. These include using models of pandemic influenza to assess the likelihood and impact of a “modern Spanish flu”,50 adapting IPCC climate models to assess the probability of catastrophic climate change51 and using astronomical models of near-Earth objects to assess the likelihood of asteroid impacts.52 Other authors infer from existing estimates of the uncertainty in models what the possibility of more severe impacts might be, including: Wagner and Weitzmann for catastrophic climate change53 and Atkinson et al. for near asteroid impacts.54 Finally, some identify parameters where they believe the model is mistaken and use this as a justification to “correct” the final output, as Dunlop and Spratt⁵⁵ and King et al. do for the IPCC’s predictions about climate change.56

In theory, these approaches can build on the underlying rigour and utility of the model being used; however, this depends significantly on the validity of their adaptation or application. Most models, including the IPCC’s, are not designed to assess catastrophic risks specifically, so catastrophic outcomes will be outliers. This provokes debate about whether these outcomes should be treated with genuine concern or can be dismissed as model failures.57 Modifications must only be made by those with significant skills and a deep understanding of the model’s functionality to adapt them in ways that will preserve their virtues. This may explain why the most comprehensive efforts to do this come from scholars like Ramanathan and Madhav who are outside of the Existential Risk Studies community, which, in turn, may mean that these researchers understand less about the nature of such risk.

Despite their very low levels of accessibility, well-executed applications of sophisticated modelling techniques represent a desirable next step in the study of Existential Risk and we rate them highly in terms of rigour, uncertainty and utility. However, we are sceptical of attempts to replicate the success of high profile existing models with fewer resources, by making less considered model adjustments or making concrete predictions based solely upon their current levels of uncertainty and suggest that this approach should be adopted cautiously and only by better resourced groups within the Existential Risk Studies community.

4. Subjective Approaches

Of the 66 sources in our literature review, 45% relied, at least in part, on the subjective opinion of the author or others, without direct reference to specific models, data or analytical arguments. This included all the sources that discussed the potential threat from Artificial Intelligence, which many existential risk scholars believe to be the most significant. This is unsurprising given the difficulties that other methods face, and the use of subjective expert opinion is a well-established and successful means for handling uncertainty in many other scientific fields.58 However, not all subjective opinion should be treated equally and, in the next two sections, we will consider different approaches.

4.1 Individual judgements

At one end of the spectrum are individual opinions given without reference to clear reasoning. Examples include Rees59 and Stern,⁶⁰ who both consider the overall probability of human extinction, one for a “scientists’ warning” and the other to determine the correct social discount rate. Bostrom provides a similar judgement, although this appears to be based on updating a prior belief derived from analytical arguments, to account for “the balance of evidence”.61 Halstead⁶² and Chapman⁶³ both present a considerable degree of evidence and argumentation before offering subjective conclusions about the threat of climate change and asteroid impacts, but without any specific method to connect the two. At best these estimates represent what Tonn and Stiefal refer to as “holistic probability assessments”, in which “the individual probability assessor estimates the holistic extinction risk through informed reflection and contemplation”.64

A more sophisticated approach to subjective opinion formation is for probability assessors to break down any risk into a set of mutually exclusive threats and then classify the danger posed by each of these and the probability of their occurrence (together with the likelihood that they would pose an existential risk). Tonn and Stiefel refer to this as the “whole evidence Bayesian approach”. This encourages a systematic way of estimating probabilities and is useful to anyone reviewing such an assessment as it makes it easy to update predictions in the light of new evidence or different reasoning.65

Another approach is what Tonn and Stiefel refer to as “Evidential Reasoning”. This involves specifying the effect every piece of evidence has on one’s beliefs about the survival of humanity. Importantly, these probabilities should only reflect the change that this evidence makes, not one’s initial prior beliefs, allowing others to assess them independently. As such, they will only be “imprecise probabilities” that describe a small portion of the overall probability space, where the contribution each piece of evidence makes to one’s belief and its complement need not sum to 1. For instance, one might reason that evidence about the adaptability of humans to environmental changes suggests a 30% probability that we will survive the next 1,000 years, but only a 10% probability that we will not.⁶⁶ Combination functions can then be used to aggregate these imprecise probabilities to return the overall probability of extinction within this period. This method not only helps assessors determine the probability of extinction, but also provides others with information about the sources of evidence that contributed to this decision and the opportunity to determine how additional information might affect this.

A final method listed by Tonn and Stiefel draws on the technique, common amongst futurists, of anchoring assessments in scenario-based considerations of what it would take to bring humanity to extinction. Assessors envisage a possible human extinction scenario and then consider how indicative this scenario is of both the space of all possible future scenarios and the space of those in which humanity goes extinct. This exercise is repeated until the assessor judges that they have exhausted all, or at least a substantial portion, of the human extinction scenario space. They can then estimate what proportion of future scenarios involve human extinction, and by extension how likely this is. Advantageously, this focuses on the end result, human extinction, rather than on the processes by which this might be brought about, although the use of scenarios is sometimes frowned upon in other communities. Despite the fact that these three techniques consist of little more than clearly setting out one’s assumptions and reasoning process for others to follow, none of them has so far been implemented well in the literature on Existential Risk.

It has been shown that, with only a few hours of basic level training using freely available tools, most people can be calibrated to give reliable estimates of their level of uncertainty for their subjective opinions based on their current state of knowledge.67 Despite this, few who have conducted subjective probability assessments have indicated that they have undertaken any such calibration or to state their degree of uncertainty. Instead, experts have tended to hedge their bets merely by couching otherwise precise estimates in vague language. Furthermore, individuals routinely suffer from overconfidence and confirmation bias in their subjective estimates, and when individuals have their name attached to a figure, such biases become especially problematic. Eliezer Yudkowksy discusses the relevance of cognitive biases affecting the judgement of Global Catastrophic Risk including the availability heuristic, hindsight bias, conjunction fallacy, scope neglect and overconfidence.68

The popularity of individual subjective opinion is probably because they are especially easy for researchers to apply and are often offered as the basis for further discussion and inquiry in the future. Such estimates can also be well received by media and policymakers alike, especially when they can be linked to a high-profile academic of celebrity and hence become associated with that individual’s perceived authority, potentially enhancing their utility. Indeed, it often appears easier to get people to agree with the single judgement of a known individual than a collective judgement which combines information from that individual with others.

The quality of individual subjective opinion thus depends on both the person providing the estimation and where suitable techniques are used to present and clarify their reasoning and assumptions. We rate this approach as Low/Medium for rigour. Despite the fact that this kind of estimate is well received we rate its utility as Low, reflecting its generally narrow focus and lack of credibility within scientific communities. We also rate it as Very Low for uncertainty, in particular due to its weaknesses in overcoming bias. However, we rate this method as high in terms of accessibility, which probably helps to explain its relative popularity in this field. The kind of robust approaches to clarify one’s thinking would hardly detract from this high level of accessibility, and indeed may make it even easier for assessors to reach a final judgement, so it is disappointing to see them so little used.

Another way of seeking to improve on individual subjective opinions is to pool together the judgements of multiple people to account for a more diverse range of perspectives. There are two reasons why this could improve the quality of judgements.

The first of these relates to the “wisdom of crowds”69 which provides an epistemic justification for the aggregation of large numbers of individual opinions to determine the truth of some proposition. It relies upon the assumption that individuals receive some kind of signal pointing to the truth or falsity of that proposition, and that as a result, they are slightly more likely to judge correctly than incorrectly. The theory then states that, so long as individuals are making independent judgements, adding more will increase the probability that the group’s median judgement will tend towards the correct one. The distribution of judgements across the group will effectively cancel out the noise that leads to some incorrect individual judgements and amplify the correct signal.

The second is that whilst individual judgements will be affected by multiple biases, when aggregated over many people, these biases may average out, improving collective judgement’s accuracy. However, this can also be counterproductive as biases are often shared across large groups, or even reinforced by groupthink and the sense that one may be judged by biased peers. This violates the independence of individual judgements and can lead to the predictive power of a group decreasing with its size (Fujisaki et al., 2018). Partly as a response to this, some studies have suggested that aggregation methods that give more weight to outlying opinions outperform straightforward averaging approaches.70

In this section, we will limit ourselves to discussing approaches which simply average expert judgements, whilst in the next section, we will turn to more structured and deliberative approaches.

Simple aggregation is the dominant method for making predictions about the existential risk from Artificial Intelligence,71 but has also been applied to the prediction of nuclear wars72 and to quantifying existential risk in general.73 These surveys vary considerably in quality and size, with many showing little concern for the diversity of participants, the statistical rigour of their analysis or uncertainty quantification (the honourable exception being Grace et al.).74 Most surveys take the median response as their prediction, but Turchin argues that this is sometimes not optimal for existential risk.75 For AI safety, instead of using the median estimate of AGI creation for risk assessments, we should be concentrating on the earliest possible time of AGI creation and define a “minimum acceptable level of AI risk”.

Those who have adopted this approach often acknowledge its limitations. For example, Sandberg and Bostrom state that “these results should be taken with a grain of salt. […] There are likely to be many cognitive biases that affect the result, such as unpacking bias and the availability heuristic as well as old-fashioned optimism and pessimism”.76 Judgement independence is hard to ensure as surveys are often completed at conferences and so it is difficult to guarantee that individual judgements are not influenced by others. Remote participation via an anonymous platform may offer a partial solution to this problem. However, given how close-knit many academic and technical communities are, this still may not secure judgement independence. Finally, although aggregation may improve judgements, it has the effect of making them less well behaved. For instance, if one seeks the median estimate from a group about the probability of superintelligence being developed and the probability of superintelligence leading to human extinction and then combines these figures, this can differ substantially from the median group prediction that humanity will go extinct from superintelligence.

The aggregation of expert opinion has the potential to improve upon individual judgements regarding their rigour and ability to handle uncertainty; however, in practice, this opportunity is, once again, often not taken. This may reflect the fact that extensive, well designed surveys are still out of reach of many small research groups and that people seem to respond equally well, if not better, to overconfident survey results from a small pool of “experts” than to extensive well-designed surveys that express an appropriate degree of uncertainty. We rate this approach as Medium for rigour, Low for uncertainty and utility, and Medium, both with a reduced level of confidence, for accessibility (due to disagreements about the amount of time it takes to conduct surveys in a “reasonable” way).

5. Structured and Deliberative Approaches

The final family of approaches we discuss also use subjective opinion, but seek to combine multiple opinions in more structured ways than simple aggregation. A variety of such methods have been developed by scientists and foresight specialists to aid decision-making under uncertainty, although so far these have been sparsely used in quantifying existential risk. Some of the techniques we describe form part of proprietary foresight tools such as the Delphi Technique (developed by RAND) and Superforecasting (developed by the Good Judgement Project); however, these can be disaggregated into their constituent parts for the purposes of discussion.

The first of these approaches weights opinions differently in the aggregation based on an assessment of each individual opinion’s value. For instance, Roger Cooke’s “classical” approach to expert elicitation gives greater weight to subjective opinion based upon experts’ performance on a series of calibration questions that ask them to predict things that are either known or that can easily be determined.77 An expert who more often gets closer to the truth has a larger weight in the overall aggregation of judgments. This approach’s prediction accuracy has been shown in multiple studies to outperform simple aggregation.⁷⁸

However, this method is not well suited to predicting existential risk as the experts’ competency at predicting catastrophic and existential risk cannot be calibrated due to their unprecedented nature. It might be possible to test experts’ putative accuracy through their success at predicting more common and nearer term future events; someone’s success at predicting short-term AI milestones could reflect the strength of their predictions about the long-term future of AI. However, there is no obvious means for assessing how success at predicting short-term and long-term trends are related.

An alternative means of weighing individual judgements is via peer ratings of respect and reliability. Theoretically, this avoids the problem of needing to calibrate individual predictors based on past performance and could help individuals assess their own beliefs by considering their credence in the beliefs of their peers. However, such weights are often of little, if any, meaning, especially in the context of existential risk.79 Weightings can also be generated by repeatedly sampling experts’ predictions and weighting those who gave more consistent answers more strongly than those whose answers varied, potentially indicating a lack of evidence-based judgement. For instance, Bamber and Aspinall asked for the same estimate from experts two years apart and those resampled were forbidden from referring to their first estimate to test the stability of individual judgements in determining the risk of a future sea level rise due to climate change.80 However, this approach is problematic because it would be hard to distinguish between estimates that varied over time because of the randomness arising from a lack of evidence and those which changed because the estimators were successfully updating their predictions to take account of additional information.

The final method of weighting that we consider was developed by the Good Judgement Project on the basis that empirical studies indicate that some individuals (who the project terms “superforecasters”) are substantially better at making predictions about the future than others. The project selected over 2,000 individuals and tasked them with assessing the likelihood of various world events. It found a considerable degree of variance amongst participants, with some individuals performing consistently well regardless of the kind of prediction they were being asked to make. Furthermore, it found that individuals who performed consistently highly in making accurate predictions were able to outperform even domain experts and professional intelligence operators. Philip Tetlock, the project’s leader, concluded that these individuals had particular psychological traits that led them to make more accurate predictions, including caution about the strength of their beliefs, humility about the extent to which complex processes can be simplified, curiosity about the facts of a case, valuing diverse views and opinions and a belief in the possibility of self-improvement.81 However, rather than assessing these psychological traits directly, the key to identifying superforecasters has been to keep track of individual performances at making predictions, including people’s ability to update these in order to account for new information. This was done by assigning a “Brier score” to each superforecaster, an assessment of how close their predictions came to actual events.82 The project found that the most accurate predictions were produced from an aggregation of participants’ predictions, but those with the highest Brier scores were weighted more strongly. However, it is worth noting that the success of these superforecasters was found to diminish significantly when they were asked to make predictions more than 12 months ahead. At present it is unclear whether this reflects a limitation of superforecasters’ abilities or a general problem with making longer term predictions.

While some evaluators rated these methods slightly more favourably our overall evaluation of them is no better than Aggregated Opinion Surveys for any category but low for accessibility. Given this, it is hardly surprising that such techniques have not been used in the assessment of existential risk thus far.

Another approach to structured expert elicitation is to seek to improve, rather than simply measure, the quality of experts’ predictions. Broadly speaking, this can be performed prior to solicitations being made, at the point of solicitation or between solicitation and a final judgment being produced.

Pre-solicitation methods of improving the quality of expert judgement focus on training and method selection. We have mentioned a variety of such approaches already in Section 4.1, including calibration of uncertainty and the use of formal methods like evidential reasoning and holistic probability assessments. However, some methods specifically focus on prediction as a structured group activity and these are worth noting here. For instance, the Good Judgement Project found that both natural superforecasters and those who did not share their psychological traits were able to learn and develop them over time to greatly improve the accuracy of their predictions when they went through a process of probability training, teaming and tracking. Probability training helped correct cognitive biases, teaming allowed for the sharing of information and the public justification of why a probability was given, and tracking encouraged participants to outperform their previous track record and helped develop stronger teams of peers who could learn from one another.83

However, it is worth nothing that a large amount of time and resources go into selecting super-forecasting teams. The Good Judgement Project spent four years assembling their elite superforecasting team. It is difficult to imagine that such teams could be rolled out more extensively. Nevertheless, the approach itself is quite simple, and several people in the Existential Risk Studies community have attempted to adapt elements of it into their work. For example, just one hour of training in probabilistic reasoning noticeably improved forecasting accuracy.84

Whilst no superforecasters have attempted to predict the possibility of Human Extinction; the organisation Open Philanthropy commissioned a team of super-forecasters to predict the probability of a nuclear war.85 Lessons from this approach could be incorporated by the Existential Risk Studies community in one of two ways. Firstly, it might be possible to train those who make Existential Risk predictions with the superforecasters’ techniques. Secondly, applying more resources could motivate existing super-forecasting teams to make more relevant predictions of existential risk. Both will take considerable work, and it remains unclear how successful they will be.

Efforts to improve the quality of probability estimates at the point of solicitation focus on what questions are asked and how the person soliciting expert opinions engages with them. In a recent, albeit unpublished, solicitation of expert judgements of the probability of a Global Catastrophic Biological Risk, David Manheim used a variety of such approaches to solicit better quality information from experts in infectious disease.86 He found that these experts were both poorly versed in probabilistic thinking and liable to reject the notion of a global catastrophic biological risk (in this case “a natural infectious disease that kills 1 billion people”); they disputed whether this could ever happen. However, Manheim proceeded to explain to these experts that other natural events could have catastrophic effects and challenged them to provide a fundamental reason why such an event was impossible within the infectious disease domain. By then, engaging the experts in scenario-based thinking about what properties such a disease would need to have, Manheim was able to solicit useful information with a reasonable degree of consensus between the experts. Only one expert continued to claim such an outcome was utterly impossible, but they now justified this claim, stating that this was a result of their belief that public health responses would always be sufficient to prevent such a pandemic. The experts remained unwilling to be quoted because they perceived a significant reputation risks in even discussing these extreme events. Post-solicitation methods focus on deliberation between experts, creating opportunities for experts to offer updated predictions or sometimes requiring them to adjust their judgements to move towards a consensus opinion. The most famous result is the so-called “Delphi technique” developed by RAND in the 1950s. This can be applied to a variety of foresight and horizon-scanning activities and uses a panel of experts who are asked to respond to a series of questions across two or more rounds. After each round, a facilitator provides an anonymised summary of the results, along with the reasons each expert provided for their answers. Extreme outliers must substantiate their position. Experts can then revise their judgements given the broader knowledge achieved through considering the responses of others leading, hopefully, to experts converging on the “correct” judgement. Delphi studies have been conducted to provide quantitative assessments in many areas of risk analysis, but the technique has not yet been used to provide quantitative estimates of existential risk, although Wintle et al. apply it to identify key emerging risks related to biotechnology in the Existential Risk context. 87 Other forms of structured expert elicitation that are related to, though not identical with, the Delphi technique have been harnessed to assess existential and Global Catastrophic Risk. For instance, Pamlin and Armstrong88 used a complex multi-layered process of literature review, deliberative workshops and individual subjective judgements to select and assess Global Catastrophic Risks, but without multiple rounds of estimation. Another approach that has recently been developed, in part by Existential Risk researchers, is the IDEA (Investigate, Discuss, Estimate and Aggregate) protocol.89 This drops the focus on seeking consensus and allows participants to discuss differences of opinion and defend probability estimates directly rather than responding to anonymised statements of reasons. The final independent estimates are given as anonymous submissions and then aggregated.

Whilst the Delphi technique and its relations aim to remove personal bias from predictions, as with all survey methods there may still be bias in the selection of the experts that can potentially lead to self-fulfilling prophecies.90 Individual biases may influence people’s willingness to update their judgment in light of evidence from the group and thus disproportionately sway the overall groups’ findings. Moreover, some participants may wish to tailor their contributions to ensure that there is a concordant result, rather than rocking the boat with a contribution that throws the group further away in their estimate. Furthermore, the focus on consensus may be at the expense of cultural and other embedded differences in individuals’ perspectives on information.⁹¹ Finally, when the aggregation of expert opinion involves additional deliberation between experts, this can lead them to shift away from consensus and towards the most extreme views under discussion; individuals begin to cluster their identities around opposing positions, such as those defined along political or disciplinary lines.⁹²

These techniques are relatively difficult to implement, requiring technical familiarity and the resources to convene a sufficient number of experts to implement them, but these barriers are lowering with time especially as the Delphi technique has a long track record of use in a variety of scientific and policy contexts. The fact that this approach can harness knowledge and expertise from across disciplinary backgrounds, requires individuals to substantiate judgements and encourages individuals to revise their first estimates in light of new information lends it a considerable degree of rigour, at least relative to many other methods that we have looked at. Whilst potentially controversial, the results are easy to communicate and are given credibility by the structured process through which they are obtained.

Bamber and Aspinall noted that experts in their study were “exceedingly uncertain about the answer to [the] key question”.93 They argue that whilst structured expert elicitation can help to quantify uncertainties; it does not overcome them. Such high degrees of uncertainty are often seen as prohibitive for quantitative research, and this may be part of why the Delphi Technique is often reserved for qualitative studies. However, we believe that this feature of the technique should be viewed in a very different light within the field of existential risk, where confidence in predictions is often overstated.

We believe that enhanced solicitation techniques have a significant underused potential to contribute to the quantification of existential risk. They are more rigorous, useful and able to handle uncertainty than individual or aggregated subjective opinions, although they are also harder to implement. A particular attraction of these techniques is their ability to open up a broad range of knowledge and perspectives on risk and to guide experts in combining this into coherent judgements. We rate these methods as High/Medium for rigour and Medium for uncertainty and utility, although we a higher degree of uncertainty for all three categories. However, we rated these techniques as only Low for accessibility due both to the time and expertise required to implement them well.

Prediction markets function by providing a platform on which people can make trades based on their different assessments of the probability of an outcome or event. The more accurate any person is, the higher their payout. The price at which people are willing to make these trades depends on their probability assessments and their level of certainty in these assessments. This incentivizes individuals to be as rigorous and accurate as possible and allows for aggregation to take place over a potentially unlimited number of participants. The prediction market Metaculus uses trades with in-platform credits allowing individuals to perform actions such as posing their own questions. It has set up a market to establish the probability of human extinction,94 although the market clearing price, which will represent its “final” prediction, will not be available until it closes in 2030. However, as Metaculus notes, this market, unlike its others, will not be able to pay out and users are therefore asked to make trades “in good faith” only. One proposal to overcome this barrier is to build the markets around trade in a resource that would help individuals survive a global catastrophe, such as access to survival shelters.95

Lionel Page and Robert Clemen argue that prediction markets are relatively well-calibrated when the time to expiration is relatively short, but that prices for the future are significantly biased.96 One might overcome this barrier by establishing markets for events that would be related to, but not necessarily cause, an Existential Risk. For instance, prediction markets could be used to determine the probability that a large asteroid will pass within lunar orbit, that at least one nuclear weapon will be detonated by a non-state actor or some other “near miss” event that would help us understand Existential Risk without implying that humanity would actually go extinct.97

As with all markets, individuals who have limited information may assume that the market is better informed than they are and therefore not bid away from the current market price. This can cause price biases, where it becomes entrenched and prevents markets fully taking account of changing conditions. Nevertheless, prediction markets have proven success at making predictions even under situations of extreme uncertainty, such as whether CERN will locate the Higgs boson.98

Prediction markets currently have a strong track record, and there is considerable interest in their use, both amongst experts and as a means of “democratising” decision making. However, there are significant barriers to their application for Existential Risk. If a suitable platform could be established where participants were shown to have a clear interest in the long term, and their returns were guaranteed against inflation and loss of investment potential, perhaps via a philanthropic backer, then they might have an important part to play in assessing the probability of existential near misses. We rate prediction markets Low for uncertainty and utility and Medium for rigour and accessibility.

6. Discussion and Recommendations

There appear to be no standout “winners” from this analysis and every technique is rated Low on at least one criterion. The top scorers from our analysis as a whole are Bayesian networks, adapting existing models and Enhanced Solicitation Techniques, all of which score Low or Very Low in terms of accessibility. Of the more accessible approaches Toy Modelling and Aggregated Opinion Surveys perform best.

Given this variety of methodological virtues, we conclude that method selection should be understood in context and that the suitability of a method to a researcher’s needs and circumstances is more important than its overall performance. At present, methodology choice seems to be strongly related to the nature of the studied threat. Some methods may well lend themselves to specific threats, depending on whether they have already been modelled at the sub existential level or whether there is a past historical record on which to build one’s analysis. However, we feel that most of these methods could be applied far more widely and that more appropriate determinants of their use are the resources available to a team, whether the research is being undertaken for scientific or policy purposes and how findings are intended to be used.

Tonn and Stiefel go further and argue for giving the “results of all methods to a panel of experts to reflect upon before they are asked for holistic assessments”.99 However, that strikes us as potentially problematic because it leads to the homogenisation of quite diverse methodological perspectives and the potential loss of insight and introduction of bias that this entails. Realistically, it also represents a further loss of accessibility and therefore may put researchers off from conducting empirical studies to begin with. Hence, we conclude that it would be better to encourage researchers to focus on the methods that are best suited to their particular context and let a thousand flowers bloom.

We believe that the use of structured approaches, and especially enhanced solicitation techniques has been especially underdeveloped within the field of existential risk research and that this deserves more attention. While processes like the Delphi technique and superforecasting are not unproblematic, they have developed a good reputation in many scientific circles for being well suited to both interdisciplinary research and making judgements under uncertainty, two of the greatest challenges facing existential risk quantification.

In particular, two areas strike us as prime candidates for employing such techniques. Firstly, given the lack of a transparent methodology for establishing probabilities in the Pamlin and Armstrong report,100 the Delphi or IDEAs technique may be an appropriate tool should the Global Challenges Foundation seek to update this research. Secondly, given the prevalence of unstructured surveys in the analysis of Artificial Intelligence as an existential threat, we believe that a more structured approach to combining expert opinions in this area would be valuable in providing a more rigorous perspective on a controversial subject.

Beyond this, however, our study serves to highlight the significant diversity in approaches to the implementation of these methods. There are examples of both good and bad practice in the literature at present and, regrettably, it is not always the good practice that is driving out the bad in the marketplace of ideas. In particular, many of the methods we considered allow researchers to objectively set out their reasoning process for others to critique and potentially update in light of new evidence and most have techniques for assessing and reporting degrees of uncertainty in a judgement. However, in very many cases such opportunities were not taken or were merely paid lip service despite requiring little, if any, additional effort.

The main reasons for not taking advantage of such opportunities are reputational. If one expresses uncertainty, then others are likely to see your judgements as less credible, and if one clearly sets out one’s reasoning process, then others may see it as mistaken. These are not good reasons for bad science, and even if there is some argument to be made for simplification in public-facing communication, clear statements of methodology and uncertainty should be produced for the Existential Risk Studies community.

A good example is set by the IPCC who make use of a clear uncertainty framework in their reports. This combines probability judgments and confidence judgements, with separate terms used to describe each. For instance, terms such as “likely” present a probability judgment, whilst terms like “confident” are used to present degrees of certainty. According to the IPCC, authors guidance notes: “A level of confidence provides a qualitative synthesis of an author team’s judgment about the validity of a finding; it integrates the evaluation of evidence and agreement in one metric.”101 A potential strength of this approach is that it can be sensitive to particular limitations within a domain, such as the availability of evidence, the level of disagreement about how to interpret that evidence, the robustness of models and methods that are currently used to evaluate it and the overall level of consensus that has been achieved.

Other good examples tend to be set by studies that come out of the “hard” sciences, including those relating to pandemics and space weather, or those that are embedded in risk analysis, such as the work of Anthony Barrett and Seth Baum on nuclear war. However, in each of these domains, there remain examples of bad, or even discredited, science that are still repeated by Existential Risk researchers, both in public-facing work and academic papers.

Conclusion

Despite the challenges involved, the quantification of existential risk seems highly likely to continue as a prominent strand of research in this area, for risk communication, research prioritisation and policy formation. We believe, however, that it is time that researchers in this field became more aware of how they can, and should, go about this process. There are a wide variety of methods that have been tried thus far, and none of these is definitively best, each having both merits and challenges. More importantly though, any of these approaches can be implemented well or badly and the mere fact that a certain probability assessment has been produced does not mean it is worthy of reproduction or inclusion in further analysis.

This is basic science and common sense. However, it is arguable that within the nascent field of existential risk research people have been insufficiently discriminating in this regard. This is not only problematic in that it risks using worse results when better ones are available; it also holds back the development of the field by failing to stimulate scholars to improve the quality of assessments that they produce.

(In the references that follow, bracketed mentions of ‘source x’ refer to the accompanying literature review contained in the online appendice, which is available at https://www.sciencedirect.com/science/article/abs/pii/S0016328719303313#sec0115)

https://www.sciencedirect.com/science/article/abs/pii/S0016328719303313#sec0115