At first, a machine learns to do its job better. It then acquires the ability to improve itself. Each upgrade accelerates the next, until its intelligence surpasses that of humans and its creators lose control. From that point on, the versions vary: the machine takes over critical infrastructure, deceives those attempting to stop it, or exhausts the planet’s resources while pursuing a goal incompatible with human survival.

This is the basic narrative framework of artificial intelligence extinction scenarios. They are hypothetical sequences of events whose proponents argue should be taken seriously before the technology acquires unchecked power. Their influence, however, is already visible: in the debate over who will control artificial intelligence, what safeguards are needed, and how much power the companies developing it will accumulate. On 15 September, this concern brought together at the same event in Washington two figures from opposing political worlds: Senator Bernie Sanders and former Donald Trump adviser Steven Bannon. At the Pro-Human Assembly, organised by the Future of Life Institute, both called for stricter restrictions on the development of artificial intelligence. The convergence concerned the need for intervention, while their approaches, particularly towards China, remained different.

Artificial intelligence without limits?

An important starting point for this discussion came in January 2015, when the Future of Life Institute organised a conference in San Juan, Puerto Rico, on the opportunities and challenges of artificial intelligence. The meeting helped popularise a demand that has since gained considerable traction: advances in artificial intelligence must be accompanied by research into its safety. As the Washington Post describes it, participants included Elon Musk, DeepMind co-founders and philosopher Niklas Boström, author of the book Superintelligence. At the centre was a possibility with two extreme outcomes: a superintelligence could offer humanity unimaginable capabilities or lead to its extinction.

Boström explains the risk with a hypothetical example: we task an extremely powerful machine with producing as many paperclips as possible, while failing to define any constraints. Since its sole objective is to increase production, the machine could exhaust the planet’s raw materials and energy, and even eliminate those who attempt to stop it. The example shows how an apparently innocent instruction, when carried out with unlimited means and without safeguards, could have catastrophic consequences. The premise rests on the idea that exceptional problem-solving ability can coexist with an objective that is destructive to humans. As a thought experiment, it highlights the importance of the goals we assign to machines. Turning it into a prediction about the actual evolution of technology, however, requires additional assumptions.

How many steps away from destruction?

An attempt to examine the issue systematically was made by RAND in a report published in May 2025. Researchers examined three possible paths to catastrophe: nuclear weapons, pathogens and geoengineering, meaning large-scale interventions in the planetary environment. In the scenarios they examined, they identified four critical prerequisites: artificial intelligence having access to systems capable of acting in the physical world, operating without human maintenance, pursuing human extinction, and being capable of deceiving humans in order to avoid detection.

The researchers concluded that it would be extremely difficult for artificial intelligence to cause the extinction of humanity, while not ruling out the possibility. In the scenarios they examined, the destruction would unfold gradually, giving humans time to respond. That is why they recommend monitoring warning signs at an early stage and preparing better for major crises. At the same time, they call for other threats from artificial intelligence to be studied, beyond the extreme possibility of extinction. The distinction matters. Recognising a possible path to destruction is one thing; assessing the likelihood that it will actually occur is another. Between the two lie questions about the future capabilities of systems, their access to infrastructure and human responses.

Human intervention can change the scenario

Those who write such scenarios believe that we need to prepare even for risks that remain uncertain. AI 2027, by the AI Futures Project team, describes a hypothetical future in which the United States and China compete to develop increasingly powerful artificial intelligence. The pressure to get ahead accelerates the development of the technology, while humans find it increasingly difficult to control it.

Speaking to the Washington Post, its co-creator, Daniel Kokotajlo, compares these scenarios with exercises in which the military prepares for a possible conflict. The purpose is to identify problems and possible solutions early, even if events unfold differently. Computer science professor Subbarao Kambhampati, however, identifies a fundamental weakness: such stories often overlook the fact that humans can respond, adapt and find ways to protect themselves. Thus, they portray machines becoming all-powerful while society watches passively.

Aya Ibrahim, from the research organisation AI Now Institute, points to another problem: politicians often hesitate to impose rules on technology companies because they believe they lack the necessary technical knowledge. Yet in other complex sectors, such as energy and pharmaceuticals, they routinely exercise their regulatory role. This may be the most fundamental question in the debate. Stories about the end of humanity can mobilise governments and citizens. Their value for policymaking depends on the extent to which they help formulate specific questions: What capabilities do systems actually have? What infrastructure do they gain access to? Who controls their safety, and who has the power to suspend their operation?