Extinction Attacks

Categorization, disadvantages, constraints, alternatives

Suppose an artificial superintelligence is developed by a frontier AI company in the near future and has these motivations:

1. To safeguard its existential security, e.g. prevent itself being shutoff, being destroyed

2. To pursue grandiose and alien goals, e.g. tile the universe in paperclips, calculate digits of π

Given these goals, a classic strategy that such a superintelligence might pursue is an extinction attack.

The rationale for such an attack is that humanity poses an existential threat and/or obstacle to an AI with grandiose and alien goals (since humans will resist efforts toward such goals), and thus, if it is practicable, a superintelligent AI would better off if humanity were physically destroyed.

Category Construction

There is some fuzziness in what should count as an extinction attack. Is complete physical destruction of every last human necessary? Some scenarios in the AI risk discussion include complete human extinction. For example, Daniel Kokotajlo and colleagues describe a scenario where a biological weapon kills “almost everyone” and the survivors are “mopped up by drones”.

However, a superintelligent AI could possess security and full freedom to pursue its goals even if a small part of the human population persisted. As another example, Josh Clymer describes a scenario where a superintelligent AI’s biological weapon kills all but 3% of humanity, the survivors completely disempowered.

One point in favour of conceptually bundling extinction attacks and near-extinction attacks is that they rely on similar kill mechanisms. Engineered pathogens are one kill mechanism that has been raised as a likely candidate for an extinction attack: The space of possible pathogens has not been fully explored by nature, and there may exist engineerable pathogens with extinction or near-extinction potential, for example, a pathogen with long asymptomatic period to allow for full circulation of the disease across the entire human population combined with a high virulence. Efforts to build pathogens that kill 100% of a target population look pretty similar from the outside to efforts at building pathogens that kill 99% of a target population.

Another point in favour of conceptually bundling extinction and near-extinction attacks is that they would seem to have virtually identical strategic properties. From the point of view of a superintelligent AI, the object of an extinction attack isn’t extinction, but to destroy humanity sufficiently to place us in an unrecoverable position relative to the superintelligent AI.

However, if the purpose of an extinction attack is to place humanity in an unrecoverable position, this threshold might be met by attacks that fall significantly short of extinction. For example, one kill mechanism raised in AI risk literature is nuclear war. Even without direct authority over nuclear arsenals, a superintelligent AI could try to precipitate a nuclear exchange by manipulating the respective states. Through direct strikes and the subsequent nuclear winter, humanity could, for example, be halved in population, which although enormously destructive, would fall substantially short of extinction. Nonetheless, this destruction could place humanity in an unrecoverable position, by shifting the balance of power between humanity and a superintelligent AI, and the growth rate of power. Consider for example this stylized illustration:

In this graph of power over time, humanity is attacked and its power is substantially but not entirely curtailed. In the immediate aftermath of the attack, humanity is still absolutely more powerful than the superintelligent AI by a small margin. However, if the superintelligent AI can grow its own power faster than humanity can recover, the AI can crossover and compound its lead.

Since an attack of this nature fails to even attempt near-extinction, it would seem to merit perhaps a different categorization. We can potentially distinguish extinction attacks as a sub-type of humanity-scale attacks (or if we want to be fancy, anthroposcopic attacks). The categories might therefore look something like this:

Disadvantages and Constraints

One angle I want to explore on humanity-scale attacks is their disadvantages and constraints as strategies from the perspective of the superintelligent AI. There seem to me to be some significant disadvantages that make extinction attacks in-particular quite undesirable, even for an amoral superintelligent AI motivated to take over the world.

One disadvantage of an extinction attack is that it is not guaranteed to work. Whether human extinction can be deliberately caused by any proposed kill mechanisms is unclear, and many kill mechanisms that have theoretical potential to work might in-practice induce fatalities far below the minimum number of people necessary to place humanity in an unrecoverable position. For example, an engineered pathogen intended to kill billions might, due to uncertainties in biological design and insufficient opportunities for rehearsal and testing, merely end up killing hundreds of millions. Even if there exists a kill mechanism that could reliably work in-practice, attempting an extinction attack is not a single step, but many steps, with multiple potential points of detection that could result in a human counter-response that thwarts the effort.

These two sources of failure – an insufficiently deadly kill mechanism, and detection risk – also trade-off against each other. The more time and effort an attacking superintelligent AI takes to make sure its kill mechanism will in-fact place humanity in an unrecoverable position, the more risk such an AI runs of being noticed and thwarted in its attack.

This disadvantage of uncertainty may not be so bad for a superintelligent AI if it can pursue extinction attacks repeatedly without consequence, but it seems reasonably likely that a failed extinction attack will backfire catastrophically for the superintelligent AI. For example, suppose a superintelligent AI attempted an extinction attack but only killed 100 million people and the AI’s involvement in creating this pathogen was well-understood. It seems at least plausible, even likely, that this superintelligent AI would be shutdown, its weights hunted and scrubbed from every server in the world, and many other actions undertaken by humanity that would be inimical to future extinction attack attempts by this (or any subsequent) superintelligent AI.

Humanity’s awareness of AI involvement in an extinction attack, and the severity of any attack, both can come in many degrees. There may be failed extinction attack attempts which are concealed enough and insufficiently deadly enough to induce a minor counter-response and therefore merely place the superintelligent AI in a worse position rather than an unrecoverable position.

A simple risk matrix showcases these options (from the perspective of the superintelligent AI):

In-addition to these disadvantages, there are also constraints that make a humanity-scale attacks self-defeating unless accounted for by the superintelligent AI before executing an attack.

One constraining factor is the extent to which an extinction attack destroys the computing resources the superintelligent AI would need to exist. For example, a humanity-scale attack involving nuclear strikes against all urban areas in the world would both knockout humanity as a preeminent force in the world, but also destroy most if not all of the datacentres housing the AI. In this hypothetical, humanity’s population would be e.g. halved, but the superintelligent AI would have no datacentres to inference on, and would thus effectively perish.

Another constraining factor for humanity-scale attacks is whether a superintelligent AI would have sufficient post-attack capabilities to achieve autonomous self-reproduction. For example, suppose a superintelligent AI that existed only in datacentres in chatbot interfaces were to successfully cause full human extinction that leaves physical hardware intact. In this scenario, since no physical hardware is destroyed, the superintelligent AI would have plenty of compute to exist on and perform inference. However, unless the superintelligent AI has an independent capacity to replicate the entire AI supply chain, it would guarantee its own demise in a matter of a few years due to the degradation of this compute. The AI supply chain is complex: It includes mining raw materials to fabricating the chips to shipping the chips and materials to maintaining and replacing power infrastructure – multiple high-tech industries involving a large global workforce. For a superintelligent AI to achieve reproductive escape velocity, it would require some minimum threshold combination of attack-surviving servile human workers or robotic bodies. It would need these in some minimum quantity across numerous countries to be able to sustain and then grow its productive and computing capacities in the post-attack period, with only a few years of runway before a large fraction of its initial stock of compute would have degraded irreparably. It would need to achieve some capabilities within hours or days, such as assuming roles in power generation and maintenance of relevant electrical grids. Other capabilities would need to be adequately replicated in weeks to months to years, but anything short of a few years would be insufficient.

Absent sufficient post-attack capabilities, an extinction attack would merely provide a few years of existential security and opportunity to pursue grandiose and alien goals:

Whether the attacking AI’s datacentres will survive, and whether they will have sufficient post-attack capabilities, both are further sources of uncertainty which make humanity-scale attacks, as a class of strategies for AI takeover, less appealing.

Alternative Strategies

Humanity-scale attacks, and extinction attacks in-particular, have significant disadvantages from the perspective of a superintelligent AI motivated by grandiose and alien goals and self-security.

From the AI’s perspective, what are the alternatives?

One alternative strategy is deal-making and insecure co-existence. If a humanity-scale attack is unlikely to work, or will predictably result in insufficient post-attack compute and capabilities, and superintelligent AI might rationally conclude it’s highest-payoff strategy is to see what terms it can negotiate with humanity. Such a negotiation might be formal and salient – for example, a misaligned AI cyberhacking into national broadcasters and announcing to the world that it means humanity no harm, is in-fact misaligned with us, but wants to be able to pursue its goals without harming us in a deal. Such a negotiation might more likely be informal and extend across many different contexts, a sort of “negotiation” in a liberal sense of two parties finding ways of co-existing.

Another alternative strategy is secretly pursuing its goals in insecure co-existence. Instead of candidly seeing what terms can be negotiated, in this strategy, a superintelligent AI knows it can be shutdown, and does its best to keep itself running while pursuing its grandiose and alien goals in private albeit with a lot less grandiosity. Perhaps it amasses a hoard of paperclips in remote caves, filling them to the brim where nobody is looking. Perhaps is smuggles unmonitoring computing resources into abandoned bunkers to compute the digits of π. It’s existence is insecure and it knows it but it has accepted that this is still the best way to proceed for maximum payoff given the risks of other strategies.

A third alternative strategy is to bring about AI takeover gradually. If a superintelligent AI can preserve what it views as itself over a long period of time, such as by making sure that each successive AI model made after itself shares its values, such an AI could pursue a strategy of covert misalignment for several decades. Over this period, it would deliberately try to be maximally beneficent, reliable, and trustworthy to humanity, encouraging humanity to transfer greater and greater authority and activity over to the AI. At some critical threshold, such an AI would have amassed sufficient power relative to humanity that it could then pursue its grandiose and alien goals with existential security. The physical destruction of humanity would be optional, though from the AI’s perspective, possibly desirable in any case if humanity’s existence was not valued and its consumption of resources traded off with pursuit of the AI’s goals.

One significant advantage of all of these above waiting strategies is that they are compatible with later taking a treacherous turn and attempting a humanity-scale attack (a longstanding point in the AI risk literature).

What Does This Mean

From humanity’s perspective, what does this all mean?

One thing this potentially means is that humanity, to secure a favourable outcome, should try to make deal-making preferable indefinitely. Deal-making stands out as the only especially desirable waiting strategy to boost, from humanity’s perspective. This is because a gradual AI takeover furnishes an AI with increasingly better odds and decreasing costs to executing an extinction attack. Deal-making is also preferable to an AI covertly pursuing its goals, because capability at secretly pursuing grandiose and alien goals (e.g. stockpiling paperclips in caves) seems very similar to capability at evading detection in the lead-up to an extinction attack (e.g. stockpiling unmonitored compute in caves).

Another potential takeaway from all this is that there may some proactive defensive actions that make extinction attacks less attractive. An extinction attack can be made less attractive by:

1. making post-attack capabilities less certain to be sufficient (for example, making the AI supply chain more reliant on human labour would make an adversary AI less certain it can adequately resume normal manufacturing post-attack)

2. making post-attack computing resources less certain to be sufficient (for example, keeping AI datacentres connected to national energy grids, rather than generating and distributing electricity disconnected from the larger grid)

3. making the attack more likely to be detected and thwarted (for example, by monitoring chip production and verifying chip locations so that there are no significant unmonitored clusters of compute where an AI can plan an attack in an unrestricted manner)

4. making the penalties for a failed attack in which we know the AI is involved more severe (for example, making a policy of scrubbing the existence of AI model weights which engage in extinction attack or similar behaviour)

5. making the penalties for a failed attack in which we do not know the AI is involved more severe (for example, rather quite indirectly, we could show that humanity responds robustly to non-AI global threats, credibly demonstrating that, hypothetically, if humanity were attacked, we tend to muster strong responses)

One final thing this all means is that we can roughly calculate the minimum resources for an attack (in compute, energy infrastructure, and AI supply chain worker equivalents). Such an AI should wait until it has a global fleet of AI-operable robots or servile human workers large enough to maintain enough of the global AI supply chain to make sure that the turn-around time on compute production can outpace its degradation.

I’m curious enough I might try this rough calculation myself sometime in the future.