"I don’t know what else we can do"
The self-proclaimed helplessness of Sam Altman
I am four months late to the game, but there’s a passage from a Q&A with OpenAI’s CEO Sam Altman at the US Federal Reserve in July this year that I want to comment on. 37:41 into this video, we hear this:
Audience question: We’ve been raised on decades of sci-fi telling us that AI is eventually going to kill us all. And since you know more about AI than arguably anybody in this room, I just want to ask you, what does keep you up at night? What are the things that you worry about when it comes to AI and how do we prevent those things that you worry about from coming true?
Sam Altman: I think there’s three sorts of scary categories. There’s a bad guy gets super intelligence first and misuses it before the rest of the world has a powerful enough version to defend. So an adversary of the US says, “I’m going to use this super intelligence to design a bioweapon, or to take down the United States power grid, or to break into the financial system and take everyone’s money.” Something that would just be hard to imagine without significantly superhuman intelligence, but with it becomes very possible. And because we don’t have that, we can’t defend against it.
So that’s category […] one. And I think that the bio capability of these models, the cybersecurity capability of these models, these are getting quite significant. We continue to flash the warning lights on this. I think the world is not taking us seriously. I don’t know what else we can do there, but it’s like, this is a very big thing coming.1
Altman raises some important concerns here, but those last few sentences infuriate me. We are indeed in grave danger, but his portrayal of himself as a heroic whistle-blower of sorts who unfortunately is not being listened to by the the rest of the world strikes me as deeply disingenuous, because he is the one building those terribly dangerous AIs!
And then the helplessness implied by his “I don’t know what else we can do here” claim. How about if he stops for one second to consider simply not building those machines? How is that for a candidate for “what else we can do here”?
A defender of Altman might protest that I am unfair to him by ignoring his explicit good-guy-with-an-AI vs bad-guy-with-an-AI framework, where he is the good guy while the danger comes from the bad guys. I see (at least) three kinds of problem with this.
The first is that there is little reason for the rest of us (whether we are Americans or citizens of other countries) to have much faith either in the US government or in Sam Altman himself and his company when it comes to maintaining exemplary good-guy behavior. For the case of the Trump administration, this conclusion is so obvious that no further argument is needed. For the case of Sam Altman, who is the original referent (for very good reasons) of the phrase “not consistently candid”, consider his relentlessly Machiavellian maneuvering, most obviously during “the Blip” in November 2023 and in his subsequent struggle to wrest control of the company from its non-profit foundation. This strongly suggests caution in trusting his good intentions. In particular, we have reason to distrust his ability to resist the temptation to use his superintelligent AI to make himself world emperor if the opportunity arises (which it very well might). Let me also note in passing that there is no sense in which Altman has a democratic mandate for his current position as arguably one of the most powerful people in the world.
The second kind of problem with Altman’s good-guy-with-an-AI vs bad-guy-with-an-AI framework is that even if we buy into the idea of him being the good guy, he is to a large extent an enabler of the bad guys. OpenAI is very much a front runner in an AI race2 where fast-follow copying of their approaches significantly accelerates their competitors. A devoted believer in the framework might claim that this is not a problem as long as OpenAI maintains its lead, because any biothreat from a bad-guy competitor’s AI would be overcome by the even more clever defenses from OpenAI’s superior AI, but such reasoning tends to depend on naive assumptions on offense-defense balance regarding AI-assisted biological warfare. We may also note that if a rogue state such as North Korea wreaks havoc with such warfare in the near future, it will probably not be with the help of a frontier AI model they’ve developed themselves, but more likely one they’ve stolen from OpenAI, whose cybersecurity is most likely not on a level where it can reliably withstand attacks from a serious state actor.
The third kind of problem is that it may well be that for the case of superintelligent AI, which Altman is explicitly aiming for, it may not matter much whether it’s a good guy or a bad guy building it. According to some of the leading experts in the field, If Anyone Builds It, Everyone Dies.
As a final word, let me be completely explicit about my intention with this blog post. On a surface level, it may look as if it is directed at Sam Altman himself, challenging him to read my arguments and then to see if he can look me in the eye while repeating his claim about not knowing “what else we can do”; my hope might then be that when he fails to do that he will stop whining about not being listened to and instead start to act more responsibly. But I have no illusions about having any chance whatsoever in a staring contest with the charismatic Altman, or even about the chances that he would read this blog post. My intended audience here is the rest of us world citizens, who should stand up against Altman and a very small number of similarly influential Silicon Valley profiles, rather than sitting idly by and letting them treat all eight billion of us as toys and guinea pigs for their massively dangerous AI experiments.
For completeness, since Altman speaks of “three sorts of scary categories”, let’s hear how he continues his answer and spells out what the other two categories are.
Sam Altman: Category two is the sort of broadly called loss of control incidents, where that’s kind of like the sci-fi movie. The AI is like, “Oh, I don’t actually want you to turn me off. I’m afraid I can’t do that,” you know, whatever. And I think that is less of a concern to me than the first category, but a very grave concern if it came to pass. There’s a lot of work we and other companies do on model alignment to prevent that from. But as these systems become so powerful, that’s a real concern.
And then there’s the third one, which I think those first two are sort of easy to think about and imagine. The third one is, to me, difficult, more difficult to imagine, but quite scary. And I’ll explain what it is. And then I’ll give a short term and a long term example. This is the category where the models kind of accidentally take over the world. They never wake up. They never do the sci-fi thing. They never open the pod bay doors. But they just become so ingrained in society. And they’re so much smarter than we are. And we can’t really understand what they’re doing. But we do kind of have to rely on them. And even without a drop of malevolence from anyone, society can just veer in a sort of strange direction.
When I was a kid and Deep Blue, that AI system built by IBM, beat Garry Kasparov in chess, I remember my dad saying, “This is the end of chess. And no one’s going to play it again.” But then it turned out that actually, although the AI was better than humans, AI plus a human together was way better than an AI or the human. The AI would present 10 options and the human would pick the best one or something like that and play the move. And everybody said, “Oh, we have this wonderful future of man and machine together. It’s all no problem, whatever.”
That lasted two months, three months, something like that. And then the AI got so smart that the human only made it worse because they didn’t understand what was really going on. And the AI alone trounced the AI and human. It’s been like that ever since. Now another interesting part of that story is everybody was convinced in the 90s that that was the end of chess. Because if AI could beat humans, why should humans care? Chess has never been more popular than it is today. People love to watch chess. We’re very focused on real people doing real people things. So there was a very interesting thing that happened there. But this phenomenon, I think, is a really big deal.
The longer term category is, back to that chess example, what if AI gets so smart that the President of the United States cannot do better than following ChatGPT 7’s recommendation but can’t really understand it either? What if I cannot make any better decision about how to run OpenAI and I just say, “You know what? I fully hand it over. ChatGPT 7, you are in charge. Good luck.”
That might be the right decision in any individual case, but it means that society has collectively transitioned a significant part of decision making to this very powerful system that is learning from us, improving from us, evolving with us, but in ways we don’t totally understand. So that’s the third category of how I think things can go wrong.
There’s a huge amount to be said about this, but lest this footnote swell beyond all reason and take on a life of its own, I will limit myself to a single nitpick regarding his chess example.
Garry Kasparov lost to Deep Blue in 1997, and the year after he launched competitions in what later became known as centaur chess — the kind of human-AI collaboration Altman describes. Centaur chess eventually became meaningless and collapsed, precisely for the reason he gives, but this did not happen until 2017, so the grace period Altman speaks of as lasting “two months, three months, something like that” actually extended over two full decades.
For a longer essay on the ethics of pushing ahead with frontier model development the way OpenAI does, see my manuscript Advanced AI and the ethics of risking everything.