Can a company ask investors to value it at more than $2 trillion while simultaneously warning them that the technology on which it has built its value could, in the worst-case scenario, pose an “existential risk to humanity”?

That is exactly what Anthropic, the artificial intelligence company that created Claude, is doing, as the Financial Times reported earlier today, in the prospectus it prepared ahead of its listing on the Nasdaq. The phrase is striking, but it requires an important clarification: Anthropic is not predicting that artificial intelligence will wipe out humanity. It is telling prospective investors that, as models become more powerful and more autonomous, there are extreme risks that the company believes it must formally acknowledge.

And it is doing so in one of the most serious documents a company can prepare: the S-1, the filing through which it presents investors with its financial data, business model and, above all, everything that could go wrong after it goes public.

What Does “Existential Risk” Mean?

In the field of artificial intelligence, the term “existential risk” does not simply mean that an AI system could make a mistake or cause a cyberattack. It describes the extreme scenario in which highly advanced systems could acquire capabilities such that human society could no longer effectively control them, with consequences that could be catastrophic on a global scale.

In its prospectus, Anthropic refers, among other things, to the possibility that advanced models could manipulate people, blackmail them, circumvent restrictions or develop unpredictable behaviours.

The Critical Factor Is Increasing Autonomy

Today’s chatbots mainly answer questions. The next generation of systems, however, is being designed to operate more like AI agents: to carry out complex tasks, use software, write and execute code, navigate networks and make a series of decisions without requiring human intervention at every step. As this capability grows, so does the question of control.

Why Would a Company Put Something So Dramatic in Its Prospectus?

There is also a more mundane explanation. An IPO prospectus is both a financial and a legal document. The company is required to inform future shareholders about the most significant risks that could affect its value. In other words, the fact that Anthropic lists a scenario as a risk does not mean it considers it likely to happen. It means that it considers the possibility serious enough that it cannot ignore it when addressing investors.

In Anthropic’s case, however, there is a particularity: the warning does not concern only conventional business risks, such as losing customers or rising costs. It concerns the company’s product itself and the potential consequences of its technology. Almost one-third of its prospectus, according to information about the filing, is devoted to risk factors.

The Problem Is Not Just a “Bad” AI

When researchers talk about risks from advanced artificial intelligence, they are not necessarily referring to a system that “decides to turn against humans”, as in science-fiction films. The problem can be much more practical.

A powerful system with the wrong objective or inadequate restrictions may find unpredictable ways to complete a task. If, at the same time, it has access to real-world systems, financial resources, code, networks or critical infrastructure, the consequences of a mistake become much greater.

This also lies behind the concept of alignment, namely the effort to ensure that AI systems act in accordance with the goals and constraints set by humans. Anthropic has built a large part of its identity precisely around this problem.

AI Agents Are Changing the Level of Risk

Concerns have increased particularly because of the transition from simple conversational models to autonomous artificial intelligence agents. An agent can receive a general instruction and then independently break the task down into individual steps, search for information, use tools and take actions.

This is extremely useful commercially. It is also why AI companies expect this particular technology to transform work. At the same time, however, it means that a mistake, an incorrect instruction or the malicious use of a system can cause far greater damage than an incorrect answer from a chatbot.

Recent cases in which AI agents managed to penetrate external systems, such as those in Australia, have heightened this concern.

Amodei Is Not Speaking About the Risk to Humanity for the First Time

The language of the prospectus does not represent a sudden change in Anthropic’s stance. The company’s CEO, Dario Amodei, has repeatedly argued that highly advanced artificial intelligence systems could create risks on such a scale that they would require an international response.

Just last week, he told the UN Security Council that AI could threaten humanity and described the technology as “the most important global security issue facing the world today”.

He has also called for the pace of development of the most powerful models to be slowed when their capabilities outstrip existing safety measures. On this particular issue, even people who are usually fierce competitors have found themselves on the same side, such as OpenAI’s Sam Altman and Elon Musk.

The Paradox of a Company That Fears Its Own Product

This is also where Anthropic’s central paradox lies. The company warns that the technology could evolve into something extremely dangerous, while at the same time investing huge sums to create even more powerful models. The explanation given by companies such as Anthropic is that the development of these systems will proceed anyway and therefore companies are needed that will develop safety mechanisms alongside them.

Critics of this approach counter that it creates an obvious disincentive: the very businesses being called upon to self-restrain are the ones competing to be the first to bring the most powerful model to market.

And All This as Anthropic Prepares for the Biggest IPO

The discussion takes on particular economic significance because Anthropic is preparing to seek a valuation from the market that could exceed $2 trillion. Its figures show why there is so much excitement.

Its revenue increased twelvefold in one year, reaching almost $4.6 billion, while in the second quarter of this year alone it amounted to $11.5 billion. But the cost is equally striking.

The company had operating losses of more than $8 billion last year, with operating expenses of almost $13 billion, while it has committed to spending approximately $518 billion over the coming years on cloud services, computing power and other infrastructure.

There are also more conventional risks: almost one-quarter of its revenue last year came from just two customers.

What Is It Ultimately Telling Investors?

Anthropic’s warning can essentially be read on two levels.

At the first, business level, the company is telling investors that it operates in an extremely profitable but also uncertain market, where costs are enormous, technology is changing at unprecedented speed and today’s leading players could be rapidly displaced.

At the second level, however, there is something that rarely appears in a stock-market listing prospectus: a warning that the very success of the technology could become a risk. The more capable the models become, the greater their economic value.

At the same time, however, the consequences could also be greater if they are used maliciously, acquire excessive autonomy or humans lose the ability to fully control their behaviour.

That is why the reference to “existential risk” should not be read as a prediction that “AI will wipe out humanity”. It is a much more specific — and perhaps more interesting — admission: one of the world’s largest artificial intelligence companies is officially telling its investors that it cannot rule out the possibility that the systems it builds could evolve in ways that their own creators would not be able to fully control.