Why we need to curb the reckless race toward superintelligence

An essay I wrote for the Swedish magazine Opulens

What follows below is an English translation of an essay that I published today in the Swedish magazine Opulens. The original title was AI-utvecklingen måste stävjas snarast möjligt (AI development must be curbed as soon as possible),¹ and the intended audience is broader and somewhat less in-the-know regarding AI than what I expect from my typical Substack reader, who is likely already familiar with most ideas in this piece. Anyway, here comes:

The great global problems – rising geopolitical tensions, the advance of right-wing populism, climate change, and so on – have, in this year of the Lord 2026, begun to pile up in a deeply worrying way. For those who follow developments in AI, however, this technological trajectory stands out as the single greatest risk factor for the remainder of the 2020s, and quite possibly for the following decade as well (assuming we even make it that far before wiping ourselves out).

As early as 1951, Alan Turing – the founder of modern computer science – speculated in a forward-looking essay about how the technology he himself had helped set in motion might lead to the creation of cognitively superhuman machines, what we today call superintelligent AI. He concluded, ominously, that “at some stage […] we should have to expect the machines to take control”. The very term artificial intelligence was not coined until 1955, the year after Turing’s untimely and tragic death, but one might have expected his fateful warning to leave a lasting imprint on AI research in the decades that followed. That did not happen. Instead, the field – unlike Hollywood – largely ignored these ideas for more than half a century, advancing in a spirit of carefree technological optimism.

This lack of risk awareness can partly be explained by the fact that, for most of the intervening period, the major AI breakthrough Turing had in mind seemed so distant as to feel more like a theoretical abstraction than a plausible reality. That situation has changed dramatically over the past decade, as a result of the sudden acceleration of AI progress driven by advances in so-called deep learning, most visibly exemplified by today’s large language models such as OpenAI’s ChatGPT and Anthropic’s Claude. These two American AI companies, together with Google DeepMind and a handful of others, are now engaged in a furious race that has caused the curves to suddenly point straight upward. This, in turn, has led informed AI experts such as Leopold Aschenbrenner and Daniel Kokotajlo (both of whom are former OpenAI employees) to conclude that we may be only a few short years away from the point at which the leading AI developers are no longer flesh-and-blood humans, but AIs themselves. At that stage, AI companies may be in a position to deploy an army of hundreds of thousands or even millions of digital AI researchers, triggering a kind of turbocharged feedback loop in which progress becomes so explosive that superintelligence could emerge within mere months.

If this scenario materializes, Homo sapiens will, in effect, have abdicated its position as the planet’s ruling species, and everything will then hinge on what these new superintelligent machines are motivated to do. What makes this especially troubling is that there is currently no convincing plan for solving what is known as the AI alignment problem. AI alignment concerns ensuring that the first truly powerful AI systems have goals and motivations that are aligned with what we want, and that they prioritize, in a sufficiently robust way, human well-being and a flourishing human society. If we succeed in solving AI alignment in time for the emergence of superintelligence, there is the potential to create a world of abundance without human labor. Whether such a world could be organized in a good way remains an open question, but for us to even have a chance of doing so would require major redistributive policy measures to be put in place well in advance.

If, on the other hand, we fail to solve AI alignment in time, a conflict arises between our values and those of the superintelligent machines. The default outcome in such a situation was memorably and chillingly described in 2008 by AI alignment pioneer Eliezer Yudkowsky: “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” A conflict with a superior opponent of this kind is one we would be unlikely to survive, and the risk that – due to the ongoing AI race – we may soon find ourselves in precisely this situation by creating superintelligent AI without first solving the alignment problem can no longer be ignored. An increasing number of leading AI researchers have begun to warn about this risk, including Nobel Prize–winning physicist Geoffrey Hinton, who shocked the assembled guests at the 2024 Nobel Banquet by pointing out that the very AI technology for which the Nobel Committee had chosen to honor him could lead to exactly this kind of catastrophe.

In a column published in the Swedish web magazine Opulens on January 7, Nette Wermeld Enström writes about whistleblowers within the AI elite, focusing in particular on Yoshua Bengio, who is one of only a small handful of people who can seriously rival Hinton as the world’s foremost AI researcher. Since 2023, Bengio has become increasingly engaged in efforts to curb catastrophic AI risk, and has expressed regret, saying that he “should have seen this coming earlier.” Further names can be added, including Yudkowsky himself, who recently published – together with his colleague Nate Soares – the well-argued and highly readable book If Anyone Builds It, Everyone Dies, where “It” refers to superintelligent AI.

Yet despite leading experts shouting at the top of their voices about how potentially disastrous AI development could become, concern about this kind of AI risk has so far received very little attention among mainstream media, the general public, and policymakers. To remedy this, it is important to try to understand why that is. Wermeld Enström touches on this question in a follow-up column on January 14, where she takes as her starting point Jean-Paul Sartre’s existentialist philosophy of human agency and our responsibility to create meaning in our own lives. We are free beings, and with that freedom comes a burden of responsibility so heavy that many find it unbearable, leading them into resignation and various forms of self-deception in order to deny their own agency. In the AI debate, Wermeld Enström sees this phenomenon manifested in ideas around technological determinism and the idea that the path ahead with ever more capable AI is inevitable.

I am convinced that she is right in identifying this psychology, and I have often witnessed the same kind of flight impulse in AI discussions first-hand – most recently in a podcast conversation with Adam Cwejman, who is political editor at Göteborgs-Posten, one of Sweden’s major newspapers. It was a good conversation, in which Cwejman demonstrated a level of familiarity with the AI risk landscape that is uncommon among journalists. But a little over halfway through the discussion, he presents a long list of political and market mechanisms that stand in the way of attempts to steer development away from the insane race toward superintelligence in which we now find ourselves, and he sums it all up (around 40:28 into the audio) by saying, “the die is cast; now we must wait for the outcome.” I had no objection to the claim that these obstacles exist, but argued that it is premature and unacceptable to elevate them to the status of insurmountable barriers without even having tried to address them. Since civilization-threatening superintelligent AI does not yet exist, it remains possible to choose not to create it. Had I read Wermeld Enström’s text before speaking with Cwejman, I might have invoked her admonition that “even passivity is a choice,” and that the same applies to “every failure to regulate” and “every decision to prioritize speed over safety.” We can choose better than that.

The AI debate, however, is multifaceted, and people enter it with different starting points and different conceptions of the technology. These differences give rise to a number of distinct mechanisms that can help explain the behavior of those who drag their feet instead of climbing the barricades and insisting that it is unacceptable that all eight billion inhabitants of this planet risk annihilation because a few thousand AI developers in Northern California are unwilling to restrain themselves in their pursuit of superintelligence. The kind of resignation that Wermeld Enström describes certainly explains the passivity of some, but in other cases different explanations are needed. Most often, in my view, the right explanation is simply denial that the risk of AI catastrophe is real.

This denial itself takes several forms, the two most common being the denial that superhumanly capable AI is possible at all, and the denial that such an AI would ever want to harm humans. Elsewhere, such as in my book Tänkande maskiner and in my essay Our AI Future and the Need to Stop the Bear, I have written at length about both of these lines of thought and why I consider them misguided. Here, I will just conclude by briefly touching on each of them.

The idea that superintelligent AI is not possible – or at least not something we should expect as a continuation of current technological trends – usually rests on unfounded beliefs that human intelligence involves something so special and mysterious that it cannot be replicated by machines. With respect to language models like ChatGPT and Claude, it is often claimed that they are built on such fundamentally simple principles – that they merely make statistical predictions of the next word in internet text, or that the dynamics of the neural networks they consist of are driven by the soulless mathematical operation of matrix multiplication – that they obviously cannot be genuinely intelligent. These arguments rest, implicitly, on the principle that interesting and complex properties like intelligence cannot possibly emerge from systems made of simple components. Since this principle is false, the entire argument collapses. To see why it is false, one need only look at the human brain, which gives rise to our intelligence despite being nothing more than a jelly-like piece of matter made up of atoms and elementary particles that mechanically and mindlessly move around and collide with one another.

The second line of thought holds that we have nothing to worry about because a sufficiently intelligent AI will automatically also be moral in a way that benefits humans. But intelligence, in this context, is nothing more than the capacity to flexibly and efficiently predict and plan in order to optimally achieve given goals – whatever those goals may be. As I noted in a blog post last fall, the statement that sufficiently advanced AI will have deep knowledge of human norms and values (which is already true in some respects) is an entirely different matter from whether it will endorse those values and act accordingly.

The available body of theory within the field of AI safety strongly suggests that intelligence and goals are separable properties, in the sense that almost any goal is compatible with arbitrarily high intelligence. It is sobering to reflect on how the most capable and intelligent species on the planet today – humans – treat other species, through ecosystem destruction and a barbaric meat industry. If we create superintelligent AI before we have found a way to ensure that its goals are the ones we want – what I have referred to above as the AI alignment problem – we cannot expect it to treat us with more care and moral concern than we currently extend to pigs and chickens in factory farming. Knowledge of how to solve AI alignment does not exist today, neither within the leading AI companies nor anywhere else. That is why we urgently need strong interventions from legislators and other societal actors to curb their reckless race toward superintelligence.

1. The title was set not by me but by the editors, so I cannot be blamed for the word “must” slightly contradicting the existentialist philosophy discussed in the essay.