I don't think that focusing on those who are currently alive is particularly selfish/bad. Those who would never be born would not be bothered by this fact either while the sicknesses and suffering of those currently alive are very real. Having a chance to cure these people is, in my opinion, is of very high importance. (Additionally, when talking about an astronomical number of not yet born people, one also assumes that no other catastrophy/process would otherwise wipe out humankind which seems overly optimistic to me.)
Accepting a 95% conditional probability sounds quite extreme indeed, but I would absolutely accept 1% and would seriously think about 20% as well. I am not sure why most people would, as you write, dismiss even 1% risk for such a potential gain.
I find this unconvincing, mainly for the following reasons.
1. The idea that the typical normie would accept the kind of calculus you defend (rather than dismissing it as batshit crazy) clashes strongly with my experience of normies. Admittedly this is anecdotal evidence, but at the very least they should be asked for permission before we set the world on fire.
2. Assuming that creating superintelligent AI under anything like the present circumstances would produce an extinction risk as small as 1% (or even 20%) requires huge amounts of wishful thinking.
1. Well, it would be interesting to conduct a study to figure this out. Out of curiosity, did Trump being elected twice clash strongly with your experience of normies? To me, having 99% chance for all current and future generations of living good lives of 1000 years is absolutely worth the 1% risk of extinction. I agree that it would be nice to ask people about this, but if I were you, I would not take the result of such a referendum for granted. (Besides, most decisions in history were taken without consent or voting - while I agree that it would be nice to do it differently this time, it seems highly unlikely that this will happen. But that dies not mean that one should not fight for it.)
2. What makes you think that even 20% requires huge amounts of wishful thinking? Generally, where do you get your double-digit risk estimates from? I have read a lot of your blog posts and publications and do not remember such a calculation, but I would be very curious to see it!
3. Thanks for this reference! These are interesting thoughts and I may read more about them in the future.
Regarding item 2, the problem is obviously too complicated to admit an exact calculation of extinction probability, so it is more a matter of noticing that the situation as whole points strongly towards extinction being the default outcome if we build superintelligence in the near future. Part of this is the theory of orthogonality and instrumental convergence, and another part is taht no remotely convincing plan for AI alignment is on the table, but three is more; see, e.g., my "stop the bear" paper (https://www.math.chalmers.se/~olleh/AIandHumanCivilization.pdf) for an attempt at a digestible overview.
Thanks! I have read your "stop the bear" paper before and can recommend it to others. But I still think it would be appropriate to attempt providing some calculation (based on assumed probabilities, of course) that could justify a double-digit lower bound for the risk. Subjective feelings and intuition regarding probabilities is known to be error-prone. Anyway, thanks for spending time on these replies, and have a nice day!
I agree it would be good to have such a calculation! It would not entirely remove the subjectivity element, because there is always an amount of subjectivity in the choice of mathematical model and input parameters for the calculation, but it might nevertheless improve our epistemics and understanding of the problem at hand.
And I could easily improvise one. Yet I won't. Why is that?
Mathematical calculations in scientific arguments typically serve one of two purposes (or sometimes both), namely (a) to provide insight that is at least partly reliable and that would not have been readily available without the calculation, or (b) to serve as a kind of decoration, giving the scientific argument quasi-precision, superficial credence and unwarranted prestige.
In the present case, I judge that the uncertainties are so large, and the issue so complex, and that the question concerns a world (one with superintelligent AI) that is so far outside the limits of the familiar where most of science takes place, that a mathematical model precise enough to support a calculation achieving (a) seems infeasible. This leaves (b), but my professional ethics prohibits me from offering calculations that serve only purpose (b).
So no calculation, at least not at this time. Sorry for that. And I urge you NOT to commit the tempting and all-to-common the-entire-burden-of-proof-alwys-falls-on-the-other-guy fallacy so as to conclude that you can safely assume that the sought-after probability is low.
Thanks for the detailed reply! I am still somewhat bothered by the statements implying >10% risks without a calculation which could be checked and iteratively updated based on new evidence, but I understand your point of view. Additionally, at other places, you state that providing such a calculation would tempt reviewers to disprove that specific calculation and thereby conclude that the risks are low. That also is a reasonable standpoint, although having the possibility to disprove statements is the foundation of scientific research.
Anyway, I am not assuming that the risks are low. I currently believe they are moderate-to-high, and the only difference is that my "high" is mainly due to humans in the loop (i.e. human-initiated catastrophe performed by AI) which you seem to disagree with. Also, it is completely reasonable to rather place the burden of proof on the companies developing a new and potentially dangerous company.
At the same time, I see a very high risk that without superintelligence, humanity will not survive the next 1000 years anyway or end up in a dictatorship. This belief may be one motivation for my preference for risk-taking in a situation where you imply that doing so may be batshit crazy. This is also why I symphatize with Bostrom's perspective of comparing the development of AGI to a risky but necessary operation.
However, one should generally postpone an operation until it is safe to perform it, and similarly, I agree that the development of AGI should also be preceded by proper alignment efforts. So, I am not against your views at all, and I am very interested in reading, discussing and sometimes questioning the outlined arguments.
This stood out for me, as I've written almost the exact same thing (in old and new articles):
***
I wrote about an older consideration of Bostrom’s that “I feel extremely uneasy about the prospect that it might become recognized among politicians and decision-makers as a guide to policy worth taking literally”. At that time, the discussion still seemed abstract and hypothetical, but now that we have arrived at crunch time for humanity, the same sentence applies with even greater force to his Optimal Timing for Superintelligence paper, and the prospect that that paper might fall into the hands of policymakers gives me nightmares.
***
"Abstract and hypothetical" -- I've said that, too, in talking about Yudkowsky's claim that nearly everyone on Earth should be "allowed" to die to preserve the glorious transhumanist future.
A primary focus of mine these days is the pro-extinctionist elements of TESCREALism. It's not just Faggella and the rest, but Bostrom, Ord, MacAskill, too. I wrote about it starting in section 4 of this paper: https://link.springer.com/article/10.1007/s10790-025-10072-7. For whatever that's worth.
It's all pretty shocking. I think that, with a few qualifications, the options before us re: ASI are extinction or, alternatively, extinction. There's no scenario being pursued by the most powerful and influential tech folks in which our species makes it through. The future is posthumanity, not humanity.
I haven't read it yet, but it's at the top of my list.
Re: footnote #2, the claim isn't that Altman is a pro-extinctionist because he's signed up with Nectome. There's a difference between digital eugenics and digital eschatology -- the former says that AI should replace humanity, whereas the latter just says that, in important ways, the future will be dominated by digital rather than biological beings. Altman, like virtually everyone else in Silicon Valley, is an advocate of digital eschatology.
This leads to a question, though: if the future will be dominated by digital beings, what exactly should we expect to happen to biological beings? Obviously, biological beings will be sidelined, marginalized, disempowered, and ultimately eliminated. Does anyone seriously think that, in a world ruled and run by digital posthumans, biological humans like myself, my niece and nephew, etc. will have any say or control over their lives or future? Of course not! That's why I argue that, in practice, digital eschatology (a largely descriptive view) is more or less indistinguishable from pro-extinctionism (a normative view).
So, I think your characterization of my position isn't totally accurate -- and I think you'd probably agree with my analysis! Advocating for a world ruled and run by digital posthumans -- as Altman very much is, consistent with him claiming that his brain will be digitized within his lifetime -- is *obviously* going to pose a direct threat to the continued survival and flourishing of Homo sapiens.
(Underlying all this is a metaphysical point of debate, too: I don't think that everything that matters -- life, intelligence, consciousness, our "values," etc. -- is wholly reducible to patterns of information flow. I'm more of a downwinger in the table just below the sentence "As with the case of left versus right, Up/Down distinctions ..." in this article by my friend Dan Zimmer: https://www.noemamag.com/a-new-political-compass/. I suspect we might disagree about this, as I see you more of an upwinger? Am I right?)
Sorry for another long message! Looking forward to reading your article on the successionists!
I don't think that focusing on those who are currently alive is particularly selfish/bad. Those who would never be born would not be bothered by this fact either while the sicknesses and suffering of those currently alive are very real. Having a chance to cure these people is, in my opinion, is of very high importance. (Additionally, when talking about an astronomical number of not yet born people, one also assumes that no other catastrophy/process would otherwise wipe out humankind which seems overly optimistic to me.)
Accepting a 95% conditional probability sounds quite extreme indeed, but I would absolutely accept 1% and would seriously think about 20% as well. I am not sure why most people would, as you write, dismiss even 1% risk for such a potential gain.
I find this unconvincing, mainly for the following reasons.
1. The idea that the typical normie would accept the kind of calculus you defend (rather than dismissing it as batshit crazy) clashes strongly with my experience of normies. Admittedly this is anecdotal evidence, but at the very least they should be asked for permission before we set the world on fire.
2. Assuming that creating superintelligent AI under anything like the present circumstances would produce an extinction risk as small as 1% (or even 20%) requires huge amounts of wishful thinking.
3. Insisting on the person-affecting view is known to lead to some pretty bizarre places; see, e.g., Bentham's Bulldog, https://benthams.substack.com/p/the-worlds-most-dangerous-population
1. Well, it would be interesting to conduct a study to figure this out. Out of curiosity, did Trump being elected twice clash strongly with your experience of normies? To me, having 99% chance for all current and future generations of living good lives of 1000 years is absolutely worth the 1% risk of extinction. I agree that it would be nice to ask people about this, but if I were you, I would not take the result of such a referendum for granted. (Besides, most decisions in history were taken without consent or voting - while I agree that it would be nice to do it differently this time, it seems highly unlikely that this will happen. But that dies not mean that one should not fight for it.)
2. What makes you think that even 20% requires huge amounts of wishful thinking? Generally, where do you get your double-digit risk estimates from? I have read a lot of your blog posts and publications and do not remember such a calculation, but I would be very curious to see it!
3. Thanks for this reference! These are interesting thoughts and I may read more about them in the future.
Regarding item 2, the problem is obviously too complicated to admit an exact calculation of extinction probability, so it is more a matter of noticing that the situation as whole points strongly towards extinction being the default outcome if we build superintelligence in the near future. Part of this is the theory of orthogonality and instrumental convergence, and another part is taht no remotely convincing plan for AI alignment is on the table, but three is more; see, e.g., my "stop the bear" paper (https://www.math.chalmers.se/~olleh/AIandHumanCivilization.pdf) for an attempt at a digestible overview.
Thanks! I have read your "stop the bear" paper before and can recommend it to others. But I still think it would be appropriate to attempt providing some calculation (based on assumed probabilities, of course) that could justify a double-digit lower bound for the risk. Subjective feelings and intuition regarding probabilities is known to be error-prone. Anyway, thanks for spending time on these replies, and have a nice day!
I agree it would be good to have such a calculation! It would not entirely remove the subjectivity element, because there is always an amount of subjectivity in the choice of mathematical model and input parameters for the calculation, but it might nevertheless improve our epistemics and understanding of the problem at hand.
And I could easily improvise one. Yet I won't. Why is that?
Mathematical calculations in scientific arguments typically serve one of two purposes (or sometimes both), namely (a) to provide insight that is at least partly reliable and that would not have been readily available without the calculation, or (b) to serve as a kind of decoration, giving the scientific argument quasi-precision, superficial credence and unwarranted prestige.
In the present case, I judge that the uncertainties are so large, and the issue so complex, and that the question concerns a world (one with superintelligent AI) that is so far outside the limits of the familiar where most of science takes place, that a mathematical model precise enough to support a calculation achieving (a) seems infeasible. This leaves (b), but my professional ethics prohibits me from offering calculations that serve only purpose (b).
So no calculation, at least not at this time. Sorry for that. And I urge you NOT to commit the tempting and all-to-common the-entire-burden-of-proof-alwys-falls-on-the-other-guy fallacy so as to conclude that you can safely assume that the sought-after probability is low.
Thanks for the detailed reply! I am still somewhat bothered by the statements implying >10% risks without a calculation which could be checked and iteratively updated based on new evidence, but I understand your point of view. Additionally, at other places, you state that providing such a calculation would tempt reviewers to disprove that specific calculation and thereby conclude that the risks are low. That also is a reasonable standpoint, although having the possibility to disprove statements is the foundation of scientific research.
Anyway, I am not assuming that the risks are low. I currently believe they are moderate-to-high, and the only difference is that my "high" is mainly due to humans in the loop (i.e. human-initiated catastrophe performed by AI) which you seem to disagree with. Also, it is completely reasonable to rather place the burden of proof on the companies developing a new and potentially dangerous company.
At the same time, I see a very high risk that without superintelligence, humanity will not survive the next 1000 years anyway or end up in a dictatorship. This belief may be one motivation for my preference for risk-taking in a situation where you imply that doing so may be batshit crazy. This is also why I symphatize with Bostrom's perspective of comparing the development of AGI to a risky but necessary operation.
However, one should generally postpone an operation until it is safe to perform it, and similarly, I agree that the development of AGI should also be preceded by proper alignment efforts. So, I am not against your views at all, and I am very interested in reading, discussing and sometimes questioning the outlined arguments.
Very interesting. We seem to be writing about many of the same things, making many of the same arguments these days: https://haggstrom.substack.com/p/those-who-welcome-the-end-of-the
This stood out for me, as I've written almost the exact same thing (in old and new articles):
***
I wrote about an older consideration of Bostrom’s that “I feel extremely uneasy about the prospect that it might become recognized among politicians and decision-makers as a guide to policy worth taking literally”. At that time, the discussion still seemed abstract and hypothetical, but now that we have arrived at crunch time for humanity, the same sentence applies with even greater force to his Optimal Timing for Superintelligence paper, and the prospect that that paper might fall into the hands of policymakers gives me nightmares.
***
"Abstract and hypothetical" -- I've said that, too, in talking about Yudkowsky's claim that nearly everyone on Earth should be "allowed" to die to preserve the glorious transhumanist future.
A primary focus of mine these days is the pro-extinctionist elements of TESCREALism. It's not just Faggella and the rest, but Bostrom, Ord, MacAskill, too. I wrote about it starting in section 4 of this paper: https://link.springer.com/article/10.1007/s10790-025-10072-7. For whatever that's worth.
It's all pretty shocking. I think that, with a few qualifications, the options before us re: ASI are extinction or, alternatively, extinction. There's no scenario being pursued by the most powerful and influential tech folks in which our species makes it through. The future is posthumanity, not humanity.
Thanks for writing this. Will subscribe. :-)
I am happy, of course, if some of what I write happens to resonate with you.
Regarding pro-extinctionism, did you see my essay in December that year on precisely that topic? https://haggstrom.substack.com/p/those-who-welcome-the-end-of-the (You are referenced in its Footnote 2.)
I haven't read it yet, but it's at the top of my list.
Re: footnote #2, the claim isn't that Altman is a pro-extinctionist because he's signed up with Nectome. There's a difference between digital eugenics and digital eschatology -- the former says that AI should replace humanity, whereas the latter just says that, in important ways, the future will be dominated by digital rather than biological beings. Altman, like virtually everyone else in Silicon Valley, is an advocate of digital eschatology.
This leads to a question, though: if the future will be dominated by digital beings, what exactly should we expect to happen to biological beings? Obviously, biological beings will be sidelined, marginalized, disempowered, and ultimately eliminated. Does anyone seriously think that, in a world ruled and run by digital posthumans, biological humans like myself, my niece and nephew, etc. will have any say or control over their lives or future? Of course not! That's why I argue that, in practice, digital eschatology (a largely descriptive view) is more or less indistinguishable from pro-extinctionism (a normative view).
Does that make sense? I discuss the idea in section 4 of my just-published Journal of Value Theory article, which argues that Toby Ord et al. are pro-extinctionists in practice (https://link.springer.com/article/10.1007/s10790-025-10072-7), as well as the "Pro-Extinctionism" section toward the end of this also-just-published Oxford Research Encylopedia entry (https://c8df8822-f112-4676-8332-ad89713358e3.filesusr.com/ugd/d9aaad_7121f8e57ecd424388e338cd0d3016d8.pdf).
So, I think your characterization of my position isn't totally accurate -- and I think you'd probably agree with my analysis! Advocating for a world ruled and run by digital posthumans -- as Altman very much is, consistent with him claiming that his brain will be digitized within his lifetime -- is *obviously* going to pose a direct threat to the continued survival and flourishing of Homo sapiens.
(Underlying all this is a metaphysical point of debate, too: I don't think that everything that matters -- life, intelligence, consciousness, our "values," etc. -- is wholly reducible to patterns of information flow. I'm more of a downwinger in the table just below the sentence "As with the case of left versus right, Up/Down distinctions ..." in this article by my friend Dan Zimmer: https://www.noemamag.com/a-new-political-compass/. I suspect we might disagree about this, as I see you more of an upwinger? Am I right?)
Sorry for another long message! Looking forward to reading your article on the successionists!
PS. All my "best" popular media articles on pro-extinctionism are here: https://www.xriskology.com/siliconvalleyproextinctionism.