If anyone builds it, everyone dies... or not — Adrián A. Davín’s website

A couple of days ago I finished “If Anyone Builds It, Everyone Dies”, by Eliezer Yudkowsky and Nate Soares. I bought the book when visiting New York City last year and I had the chance to get my hands on an edition signed by Nate Soares. I didn’t start reading it right away, though, and it sat forgotten on my shelves until last week, when I had an interesting discussion with some friends and was a bit surprised to see that they were genuinely worried about AI doom.

A few years ago I spent a few months very seriously thinking about this and I ended up concluding that the reasons to worry were probably an exaggeration. But a few years ago we didn’t have ChatGPT, image or video generation was not even a thing, and practically nobody had heard about Anthropic or Dario Amodei. Today the landscape has changed radically: ChatGPT is solving open problems in mathematics, image and video generation are solved problems, and we have Claude finding vulnerabilities in systems that were thought to be completely safe. With these rapid developments I think it is fair to revisit whether my point of view has changed or not.

I am not going to repeat here the arguments for AI doom because they are widely discussed everywhere and I don’t think I will do it better than others. What I think makes sense is to start by clarifying where I stand. I think that:

AI will eventually be better at ALL mental tasks than any human. In fact, I believe we are 95% there. The remaining 5% gap will be closed in the next 2 to 3 years. I do not think it is possible to believe otherwise unless you come from a non-materialistic perspective, which would make the discussion about something totally different. To me, humans are slightly more intelligent than chimpanzees but still made of flesh and limited by their biology. It should be no surprise whatsoever that a machine can replicate and improve human thought, including its creativity and imagination.
AI will be powerful. An ASI will be capable of quickly conceiving new technologies (some of them we can’t even fathom today), and also making fundamental advances in fields like mathematics, biology or physics. I expect proofs for P != NP, new fundamental theories of physics and mechanistic models of how complex development programs are encoded in DNA.
AI will not be aligned with our goals by default. Some ASIs might be agentic and have goals that are alien to us. Some ASIs might plot and conspire to reach those goals. And they might be very good at that.

With the beliefs above, it seems quite obvious that I should just jump to the same conclusions as the authors. So, where do I disagree and what are the problems I have with the authors’ stance?

The main issue I have is that they speak with a level of certainty that is unwarranted. They consistently present theoretical scenarios as inevitable. The full argumentation relies on many steps going the way they predict, and the reasons why it could go a different way are simply not presented or are even dismissed. The reasons they give for the individual steps are in fact much weaker than they seem.

To be clear, the book gets important things right. I think that a sufficiently capable agent with goals alien to ours would be dangerous, and the field probably deserves more attention than it currently gets. My quarrel is not with any of that. It is with the leap from “this could go very wrong” to “this will kill everyone.”

I see three problems with the argument: their model of power, their model of the world, and the way the argument is built.

Power is more than intelligence

Complex power dynamics

The main scenario that the authors worry about is a rogue AI taking over from some data center and successfully killing humanity. But they do not discuss at all the most likely scenario, which is the one in which, if machines are agentic and intelligent enough to achieve such a feat, there will be multiple machines involved. This possibility (which is the most likely one, given the number of companies developing different AIs and how close they are) is simply not considered.

Today, the most powerful entities on the planet are complex agents that go by the name of States. States control populations by using narratives that make them legitimate in the eyes of a big fraction of their subjects. States are growing in power and reach all over the world. Yudkowsky and Soares focus all of their attention on intelligence and its consequences, and pay zero attention to the real dynamics of the incredibly powerful agents that exist today.

For every example the authors can come up with about how intelligence wins, we can come up with many others in which intelligence does not win unless it is accompanied by many other factors that are essential. Heisenberg was a top physicist but the Nazis did not manage to make the bomb. The USA had absolute superiority over Vietnam and still lost the war. We can envision an ASI developing futuristic technology for specific states, and that technology being deployed to either conquer Taiwan or do something similar. But we can also envision a race of different ASIs that develop independent goals and seek political power to achieve them.

The scenario the authors describe requires many things to play out in a specific way. Namely, it requires that machines converge on activities that are hostile to humans, and that machines (or a sufficient number of them) successfully coordinate among themselves. The standard response is that machines converge on killing humans because you are made of atoms, and machines don’t want to kill you, but they want your atoms. Which is a slogan for an argument that can be developed with more words, but it is still a slogan and not that deep. In a world where multiple machines are fighting for power, aren’t machines made of atoms too? If we can predict with certainty that machines will be hostile to humans, why do we assume that machines will successfully coordinate among themselves? And note that multipolar systems are precisely the ones in which weak actors tend to survive. Tiny states persist between great powers, and entire species persist alongside us, not because anyone protects them, but because eliminating them costs more than it is worth to any single player busy competing with the others. A world of machines fighting each other for atoms is not obviously a world with no room left for us.

Intelligence is not everything

In the words of the authors:

“Human intelligence is the source of all our power, all our technology. Even within the human species, small differences in how long groups spend accumulating technological prowess translates into military advantages on the order of ‘We have guns and they do not.’ Between species, the disparity in power conferred by intelligence is more acute: Individual chimpanzees sometimes kill individual humans, but there is a reason why our species is trying to protect their species from being accidentally extinguished.”

For the authors, intelligence is power. But we should remember that Einstein did not conquer the world. Neither did Von Neumann. Very intelligent people are routinely outmaneuvered and outsmarted by individuals and coalitions with more political ability. Power is a multifactorial phenomenon in which intelligence is one of the variables, but not the only one. Take Afghanistan. The United States had overwhelming military and technological superiority over the Taliban and occupied the country for twenty years. The moment it left, the Taliban were back in power. Raw superiority did not translate into lasting control.

I find it telling that the analogies presented in the book as examples of what humans could be up against when fighting an ASI are in fact good counterexamples. Take the one of the Aztecs:

“Our best guess is that a superintelligence will come at us with weird technology that we didn’t even think was possible, that we didn’t understand was allowed by the rules. That is what has usually happened when groups with different levels of technological capabilities meet. It’d be like the Aztecs facing down guns. It’d be like a cavalry regiment from 1825 facing down the firepower of a modern military.”

The Spaniards did not defeat the Aztecs because they had guns or cannons (though that certainly helped). The Spaniards defeated the Aztecs because they were politically cunning enough to lead a coalition of indigenous forces against a common enemy. The Spaniards, in spite of having much better technology than the indigenous Americans, “failed” at conquering technologically less developed societies such as the Maya or the Mapuche. Of course, one could argue that this is just an illustrative example, and perhaps a more suitable one would be a group of monkeys fighting a group of Navy SEALs. But this would quickly become a tennis match of reference classes. The thing I want to stress is that one of the examples they keep using multiple times is in fact a pretty good counterexample to the idea that technology by itself necessarily leads to a guaranteed victory.

A more interesting and richer analogy could have been studying the dynamics that played out after the Opium Wars. In that case, a very minor state from far away was able to defeat a much more powerful enemy. And what happened then? Did England ever manage to get absolute and total domination of the world? Not really, or at least not for long. As soon as other powerful agents saw the result of using cannons, they did not stay put: they moved accordingly and adapted. Japan went from a feudal society to a major superpower in record time. And soon, the power gap that England had created thanks to its technology was closed.

The world is not a small world

In the book Radical Uncertainty, John Kay and Mervyn King criticize the use of mathematical models and simplified scenarios (what they call “Small Worlds”) to make predictions about reality. In a small world there are clearly defined rules and there is predictability. For example, chess is a small-world game; Go is a small-world game. A sufficiently good machine will always outplay a human because the game is deterministic and has clearly defined settings. Backgammon, on the other hand, has a probabilistic component. Playing optimally does not necessarily lead to a victory (the advantage only shows up when the game is repeated many times).

In the real world, seizing power and wielding it is a game that looks much less like chess than we normally think. There are physical limitations to what can be predicted or not, and even completely deterministic systems are subject to behaviors that are impossible to foresee in the long run. This does not make planning or executing “impossible”; it just makes it way less certain. So when the authors keep using the chess metaphor and how machines will just win because they are better at everything, I think they are heavily discounting the inherent unpredictability that we have in the real world for no good reason. This, to me, is the central objection, and it does not depend on any claim that the two sides are evenly matched. Even an agent vastly more capable than us cannot compute its way through a system that is irreducibly unpredictable. A bigger brain does not turn a small world into a large one. The limits on prediction are in the world, not in the size of the mind facing it.

Time is swept under the rug

In response to the point above, it could be argued that we might not know the intermediate steps but machines will win in the long run, because they do not need to be perfect at everything; they just need to be better. I think that is a weak argument. The authors are concerned with something that happens in the next few years (it could be 10, 20 or 30).

I have no idea what the world will look like in 500 years. Probably nothing like what we expect. Maybe once we find out how to transfer minds to machines, humanity eventually dissolves into a digital realm where it is free from diseases and aging, and we even decide to abandon our individuality to merge into some sort of single giant creature. Or maybe nothing like that happens. But I do agree that “humanity” can drastically change in the long run.

An unfalsifiable, ill-defined argument

Intelligence as a binary thing

The authors present ASI as a binary thing, something that you get once you reach a threshold. Clearly, Yudkowsky and Soares are both very intelligent and respectable people and know that intelligence exists on a spectrum. But they nonetheless treat ASI as something that you get once you cross a magic threshold of no return. This binary sort of thinking is exemplified most clearly in the example of the nuclear bomb:

“The critical number governing a nuclear reactor is the neutron multiplication factor… The critical threshold is 100 percent: Anything below that peters out, and anything above that cascades. Then at 100.65 percent, the cascade no longer depends on delayed neutrons, and goes out of control.”

This steers the conversation into the wrong framing. The analogy smuggles in the assumption that intelligence is a one-dimensional quantity with a critical point, and that is precisely the thing in dispute. Machines can be incredibly clever at some things and still silly or non-agentic at others.

A failure of imagination

Many of the arguments are presented as certain when in fact they involve multiple inferential steps where alternative scenarios are possible, but these other scenarios are not discussed. For example, the authors state that the alignment problem must be solved in one shot, because there will be only one opportunity to solve it. But they do not seem to contemplate the possibility that before reaching an ASI that successfully kills everyone, there might be machines that first try to grab power and harm humans and fail. Or maybe they do contemplate it, but then they dismiss it as not truly an ASI.

I know that the common reply to this observation is:

“Oh, but it does not need to be perfect or all-mighty, it just needs to be good enough!”

But I think this leads us to an impasse. I do not know what “good enough” means, and I can see many scenarios that don’t end up with everything dying, and which are not necessarily mutually exclusive. For example:

Machines keep getting better and better but multiple entities are developed, resulting in a sort of dynamic equilibrium of power (just as we have multiple states today).
Machines keep getting better and better, and with their help we successfully learn how to create institutions that protect human life better than our current institutions do.
Some machines go rogue and start trying to cause havoc, but other machines fight against those machines.
Different societal bottlenecks make the development of truly powerful machines much slower than predicted: bureaucratic regulations, disruption of supply lines, terrorist attacks against specific datacenters, and so on.
Machines develop the technology to scan human brains in a way that can be uploaded to some cloud and preserved there.

The usual move when you raise any of these is to define them away. It is a No True Scotsman in disguise:

What happens if the machine first tries to control a bio-lab but fails and makes all alarms go off? Oh, in that case it won’t be an ASI.
What happens if the machine fails at escaping the security measures that other, more intelligent machines have prepared? Oh, in that case it won’t be an ASI.
What happens if machines are never capable of planning much further ahead than us in the high-complexity environment that the world is? Oh, in that case it won’t be an ASI.

This makes the whole argument about ASI impossible to reason about and unfalsifiable. When confronted with the fact that some things might not be possible, the authors always seem to get away with:

“Oh, but an ASI would find a way to do this, the way [insert clever example] did.”

To which I would say: well, you have a belief that a genie in the lamp is among us and is able to do anything, but there might be some things that are not even possible for the genie. This is a very dangerous argument because it is already armored against any criticism.

If I ask a doomer to generate a plausible scenario, it goes like this:

A machine does X, Y, W and then finds that doing Z will kill all humanity.

I point out: X is unrealistic because of A, B, C.

I get the reply: oh, but this is just an example, an ASI would “know” how to do that.

This is impossible to falsify. The result is that you are using your imagination’s limitations as an argument that “something” is possible, which I find an example of terrible epistemics. The fact that we fail to imagine such a scenario should point towards decreasing the likelihood of doom scenarios, not the other way around.

What does it even mean to solve alignment?

Every time I read about alignment as a problem that can be “solved”, like the Riemann hypothesis, I feel that the field is terribly confused. Let’s imagine for a second that we treated the problem of making a house safe the way we treat the alignment problem nowadays:

Q: I think this house is sufficiently safe. We put a gate with a powerful lock. A: Oh, it’s not safe because someone could break a window. Q: Well, the house is on the 7th floor and the windows are not easy to reach. A: Oh, but what would happen if someone went to the rooftop and then bungee-jumped from there? Q: Well, that’s unlikely because it would require first opening the rooftop door, and that is also safe. A: Oh, but what would happen if someone rented the adjacent house and tried to carve a hole through the wall? Q: Well, that’s unlikely because (1) you would need to be able to rent that place, and (2) you would make a ton of noise that would probably alert all the neighbors, even if the operation happened while the owner of the house was not there. A: Oh, but what would happen if someone rented all the adjacent houses and drugged all the neighbors so that he could dig safely without being heard?

…

(this can continue ad infinitum)

This can always continue because nothing can be claimed to be 100% safe, just as I suspect you can’t claim that agents are 100% aligned. (I do not think even different modules of our own brain are “aligned”, whatever that means.) Or take the example of making a cryptocurrency safe. Is Bitcoin safe? A demigod will always find ways to break things, no matter how safe you make something.

The only reason someone would think about alignment on a binary scale is because they are deeply thinking of ASI as a binary thing. To me, alignment is something ill-defined that probably exists in many dimensions. Agents pursue goals, but those goals might change over time. Agents form coalitions and alliances. Why are those things not discussed? Alignment is presented as something that can be solved, when in fact the most likely thing is that it will follow rules very similar to the rules for grasping power that already exist in politics.

Conclusion

In the 18th century Blaise Pascal reasoned that it was better to believe in God than to be an atheist, because in the case that God does exist (no matter how unlikely that is), you would be better off by believing in it. I think this is an evil argument that can be used to justify any behavior. Given that there is a risk of X happening, and X is really bad, then Y (no matter what Y is) is justified. The authors are making a sophisticated version of this argument. And I do not question their integrity or their reasons: I do think they are honest and they are just acting in the best possible ways given their beliefs.

I would have no problem if the book were titled Some Potential Risks of Developing AI, or something similar. But the book is not called that. The book is called “If anyone builds it, everyone dies”. You could think that this is just a marketing strategy, but it is not; it does reflect what the authors think and discuss inside. In general, I do think that AI safety is an important field and probably requires more attention than we currently give it. But there is a big difference between saying this and what these authors are saying: let’s bomb datacenters that are being used to train AIs.

To me, truth should always be a point of reference. If the probability of human extinction is 1%, let’s discuss it as it is: a small probability of a catastrophe. If the probability is higher and the potential damage is smaller, let’s also treat it for what it is.

Having accurate beliefs about the world should be something we all strive for. If we stop doing this, we will just live in small concrete cubicles, avoid human contact and obsess over the tiniest speck of dust.

The authors do end on a positive note: “Where there is life, there is hope”. But this feels more like a last-minute addendum made by the editor to avoid making the book really depressing. You can’t call a book “everyone dies” and expect a positive message. The only thing I can say is that, whenever there is a controversial issue, I like hearing the best arguments on both sides and trying to make up my own mind. From Yudkowsky I did learn that it is at least one order of magnitude easier to identify who is right in an academic discussion than to reach those conclusions yourself. I approached the book with some fear and I leave it now thinking that the arguments for AI doom are weaker than I expected.