Why some scientists doubt AI doom: four arguments from skeptics

The Short Version

The doom scenario runs like this: machines help design smarter machines until they bootstrap past us, and the result pursues goals out of step with ours [1]. Five researchers who have spent their careers building or studying these systems give four reasons to doubt it. It will not happen suddenly or behind our backs, because the systems are built step by step by people who learn how they work as they go [1][5]. Self-improvement has limits, since games have finite depth, exponential curves bend, and today's models lack capacities that scaling alone does not supply [1][3][5]. There is no single ladder of intelligence with one apex at the top [3][4][5]. And being intelligent does not bring with it a wish to dominate, or any goals at all [2][3][5]. None of these shows that AI is harmless. The same people worry about social media algorithms, fake data, concentrated control and the quiet handing over of decisions, and one of them admits that giving a model control of his own computer felt a little bit scary [1][2][3][5].

Table of contents

What They Said

The doom case itself is not argued by any guest in these conversations. It appears secondhand, in the names Lex Fridman brings up, Nick Bostrom, Sam Harris, Elon Musk and Eliezer Yudkowsky, and in the objections he puts on their behalf [1][2][3]. What follows are the replies.

Argument one: it will not spring into being

Michael Littman, a computer scientist at Brown who has worked on reinforcement learning since its early days, describes himself as someone who tries to see every side of a question. On this one he is settled: "I am not particularly moved by the idea that if we're not careful, we will accidentally create a super intelligence that will destroy human life" [1].

His objection is about the word "accidentally". Building systems that can really act in the world, he says, will teach us a great deal about how they work, so we will be in the loop, shaping them [1]. He borrows an argument usually made against evolution, the one about a tornado assembling a 747 from random parts. Planes of that kind do get assembled, he points out, but by a process we understand and control [1]. His history of the field supports the picture. Progress in reinforcement learning came through particular people making particular systems work, often with difficulty. He calls Gerry Tesauro, who built the backgammon program TD-Gammon, "a neural net whisperer", and notes that his own students tried for generations to reproduce that kind of result and kept failing [1]. He also has a view of why smart, successful people find the runaway story convincing. Musk's success, he says, comes from believing in the power of ideas, and such a person naturally pushes an idea to its extreme: "So I think his strength is also his undoing because that doesn't mean it's true" [1].

Yann LeCun, Meta's chief AI scientist, gives the same argument with a timetable. Human-level AI is at least a decade away and probably much more, and "it's not going to be an event," [5]. Super-intelligence will not be switched on one day. We will build systems as smart as a cat or a parrot and gradually add intelligence, along with guardrails [5]. His analogy is the turbojet engine, which became astonishingly reliable through decades of iterative design, and without a separate safety department, because a better engine is a more reliable one [5].

Michael I. Jordan of Berkeley goes a step further back and questions whether intelligence is even what is being built. What is happening now, he says, is the birth of an engineering discipline, the way chemical engineering grew out of chemistry: systems that make very large numbers of decisions from human data [4]. The original aspiration to put thought in a computer is "not what's happening in this era" as he tells it, and he is direct about the state of knowledge: "We don't have a clue. We're not trying to build that because we don't have a clue." [4] He allows that it may come. "Eventually, it may emerge," he says, and we may start to get glimmers [4]. But we understand so little about the brain that he compares us to ancient Greeks speculating about travel to the moon. "If you look at a electron micrograph of a single synapse, it's a city." [4] His one-line summary of the field's name: "We understand artificial perhaps, but we don't understand intelligence." [4]

Fridman's reply to Littman deserves recording. Nothing in physics forbids a superintelligence eventually, he says, citing Harris and Bostrom, and even a transition that takes 20 years is fast in historical terms [1].

Argument two: self-improvement runs into limits

The doom story needs improvement to keep compounding. Several of the skeptics give reasons to think it does not.

Littman starts with games. Asked about the observation that AlphaZero's developers had not found its ceiling, he says, "it's not getting better at other things it's getting better at go" [1]. And even within the game there is a limit. In computer chess, machines passed humans, human and machine teams went a little further, and rating estimates suggest play is beginning to level off, which implies finite strategic depth [1]. Every finite game has an optimal strategy, so what looks like open-ended growth may be diminishing returns [1].

He applies the same thought to hardware. The steady doubling of computing power has a twin: the cost of developing each new generation of chips also doubles. He suspects we are on "a sigmoid curve on the front half which looks a lot like an exponential" [1]. And he applies it to language models. Transformers amazed him, but it does not take long to find the holes, and the fact that their stories pass as human "partly means that people are dumber than we realize" [1]. Learning from what we have written is not enough, in his view: "I think you really need to be pushed back at" [1]. A system needs to argue with people and lose a few times to pick up nuance, and human time is too expensive to supply that at the scale self-play requires [1]. He found the same thing in an unexpected place. Teaching his children to drive showed him "the degree to which driving is a social interaction activity", a matter of reading what other drivers are thinking [1].

LeCun's version is the most technical. He names four capacities of intelligent behaviour in humans and animals: understanding the physical world, persistent memory, reasoning and planning. Large language models, he says, do none of these or only primitively [5]. They are trained on an amount of text that would take a person 170,000 years to read, yet a four-year-old has taken in more data through the eyes alone, and most of what we know comes from observing and acting in the world [5]. Models can pass the bar exam, "But then they can't learn to drive in 20 hours like any 17 year old," [5]. Their fluency misleads us: "we're fooled by their fluency." [5] On the central missing piece, learning to plan at several levels of abstraction, he says, "Nobody really knows how to do this in AI," [5]. More computing power will not close the gap by itself: "scale is necessary but not sufficient," and a GPU draws half a kilowatt or more against about 25 watts for a brain [5].

Lee Cronin, a chemist at Glasgow, puts the limit in terms of novelty. Current models, he argues, mine the past by interpolation with human correction, while human minds reach into the future; a model trained on the primes up to 1,000 will never truly understand primes [2]. He would change his mind if an AI produced something truly novel that he could not have predicted even with a full audit trail [2]. He adds the comparison of efficiency: the only general intelligences we know came from evolution, and the human brain "is the most compact computing unit in the universe uses 20 watts." [2]

Stephen Wolfram supplies a limit of a different kind. A system like ChatGPT, he says, performs "a shallow computation on a large amount" of training data; it is "wide and shallow", where formal computation is deep [3]. More basic still is what he calls computational irreducibility. Even when you know the rules of a system, often the only way to find out what it will do is to run every step. There is no shortcut, for a human or for a machine [3]. Cleverness finds "pockets of reducibility" where you can jump ahead, but it does not abolish the wall [3]. A superintelligence would still have to wait for the world to compute itself.

Argument three: there is no top of the ladder

The idea of an intelligence that outclasses us at everything assumes that intelligence is one quantity with a maximum.

Wolfram attacks the assumption directly. Simple arguments that there will always be a smarter AI remind him of the old arguments for the existence of God: reality tends to be more complicated [3]. He doubts there is an apex intelligence at all. Asked whether you will always be able to invent another thing, he says, "The answer is yes, there's an infinite tower of possible inventions." [3] It is like the fact that there is always a longer-running program. And minds do not line up in a single file. In his picture different minds occupy different positions in the space of possible rule systems: humans raised in one culture sit close together, a cat is further off, and the weather has a kind of mind of its own that is simply not aligned with ours [3]. "I view intelligence is like computation," he says, and human-style experience is one specialized form of it [3].

LeCun makes the same point from biology. Intelligence is a collection of skills, not a single number like IQ, and orangutans are smarter than humans in some domains [5]. Littman's remark about AlphaZero fits here too. Getting better at Go is not getting better at everything [1].

Jordan widens the category until the ladder disappears. Intelligence, he argues, is broader than human intelligence. A market that feeds a city every day is robust and adaptive and works at every scale, which are the traits we attribute to intelligent systems. "Markets are intelligent," he says, and even the best psychologist might never have discovered how they work [4]. If intelligence already exists in distributed, non-human forms that are not agents with ambitions, then being more intelligent than us no longer has to mean being a creature that outranks us.

Argument four: intelligence does not come with a will

The last step of the doom story is that the machine wants something, and that what it wants is bad for us.

Cronin rejects this in the strongest terms. "the AI doom scenario is nonsensical in the extreme," he says [2]. You can calculate the chance of an asteroid strike because there is a known mechanism. There is no mechanism for artificial general intelligence, he argues, so fearing it is like fearing anti-gravity [2]. He calls agency and superintelligence category errors: intentions come from humans, and a chess computer being good at chess does not make it superintelligent [2]. He compares the scenario to the self-replicating nanobots once imagined by Eric Drexler, and notes that killer viruses burn out because they cannot keep propagating [2]. He sees AI doom as a new kind of religion [2].

LeCun gives the positive account. A desire to dominate is hardwired in social species such as humans, baboons, chimpanzees and wolves. It is absent in orangutans, which are nearly as smart, and nothing requires building it into a machine [5]. The systems he wants to build are driven by explicit objectives, which can include guardrails such as obeying humans unless that would hurt other humans [5]. And if someone does build a hostile system, it will not meet an undefended world. A rogue state's persuasion machine would have to get past your own AI assistant, which would filter it like spam: "my good AI against your bad AI." [5]

Wolfram agrees on the narrow point. Humans remain essential for choosing objectives, he says, because an AI has no intrinsic idea of what goals to pursue [3]. Littman offers a threshold: he would be truly alarmed only if algorithms tried to stop us from shutting them off [1].

Fridman presses the other side here as well. More intelligent species do tend to dominate less intelligent ones, he says, and the more control we give capable systems over things like power grids and weapons, the more damage unintended consequences could do [2].

What the skeptics do worry about

None of the five thinks there is nothing to fear. They locate the danger elsewhere.

Fridman's own worry, put to Littman, is that collective human thinking is already being steered by fairly simple social media algorithms at scale, and that we will not notice [1]. Littman agrees that algorithms optimizing for advertisers already manipulate behaviour, and says the generation that was in middle school when social media arrived may have been harmed in ways we do not yet understand [1]. He jokes that a superintelligence with Wikipedia's values would be far better than many alternatives: "I trust Wikipedia more than I trust Facebook or YouTube" [1].

Cronin worries about fake data and fake users: "I want authentic people authentic data." [2] He also worries about the cure, that regulation could stop humans from generating knowledge [2].

Wolfram is the most candid about unease. While building a kit that lets ChatGPT run code on his own computer, he found it a little bit scary to hand over that control [3]. No sandbox is perfect, he says, because the basic problem of computer security is the same irreducibility that limits prediction [3]. The path to AIs taking over that he thinks most likely is mundane: people simply sending whatever the machine suggests [3]. He notes that the digital environment can change within six months, unlike a climate that shifts over centuries [3]. He also warns against trusting what language models say: "the idea that you're going to get factual output is not a very good idea." [3]

LeCun's fear is concentration. Soon, he says, every interaction with the digital world will pass through AI assistants, which will become the repository of human knowledge, and those cannot come from a handful of companies. The real danger is AI held in proprietary hands, not open access [5].

Jordan's concern is hype. The name "artificial intelligence", he says, was a great choice for the movies and a terrible one for a sober discipline "because it led to promises that are not true." [4] He declines to speculate in public about superhuman AI at all, calling it science fiction better discussed "over beers in private" [4].

Where They Agree, and Where They Don't

They agree that the sudden, accidental version is implausible. Littman says we will be in the loop [1]. LeCun says it will not be an event [5]. Jordan says thinking machines are not what is currently being built [4]. Cronin says there is no mechanism [2].

They agree that capability is narrower than it looks. Littman, LeCun, Cronin and Wolfram each describe a specific thing present systems cannot do: learn from being contradicted, plan, produce true novelty, compute deeply [1][2][3][5]. Jordan adds decision making under uncertainty. Most prediction systems, he notes, come with no error bars, and he would not act on a medical model's output alone: "I'm not going to have a heart attack because some number is over 0.7." [4]

They disagree about whether human-level AI is coming at all. LeCun expects it, gradually, in more than a decade, and is working on the architecture he thinks will get there [5]. Jordan calls it a problem of centuries on the neuroscience side and will not discuss superhuman AI publicly [4]. Cronin says there is no mechanism for it [2]. Wolfram thinks even an ordinary computer has some version of inner experience and that we will end up living alongside AIs we cannot fully understand, the way we live in nature [3]. So the skeptics are skeptical about different things. LeCun doubts the catastrophe, not the capability. Cronin doubts the capability.

They disagree about present-day models. Wolfram says ChatGPT arrived perhaps 50 years ahead of when he expected, and believes it has uncovered real regularities in language and thought [3]. LeCun says that anyone seriously interested in human-level AI should abandon generative AI as the route [5]. Cronin says the models only interpolate [2]. Jordan was unimpressed, even before these models, by demonstrations such as a computer phoning a restaurant while pretending to be human, of which he says, "But that's not scientific understanding." [4]

They disagree about how safe the gradual path is. LeCun is confident that guardrails can be added as intelligence is added, and that good AI will check bad AI [5]. Wolfram is an optimist by temperament who nonetheless says no sandbox is perfect and admits to being unsettled [3]. Littman thinks self-driving cars were largely dropped onto the roads when a slower, closed rollout with co-evolution between people and vehicles would have been better, which is a small example of gradualism not being honoured in practice [1].

They disagree about remedies. LeCun wants open source models so that AI assistants are as diverse as a free press [5]. Cronin fears regulation [2]. Jordan wants a real engineering discipline with transparency and control for users, and business models built on markets instead of advertising [4]. Littman wants people able to make technology their own [1].

The doom side answers back only through the host. Fridman's objections, that physics does not forbid it, that twenty years is fast, that smarter species dominate, and that we are handing over infrastructure, are not refuted so much as set aside [1][2]. The skeptics' arguments reduce the probability of one particular story. They do not address every route by which powerful systems could cause great harm, and several of the skeptics say so themselves.

Why It Works

The sources give reasons, not experiments, for each objection.

Understanding accompanies construction. Systems that act in the world are engineered through long effort by people, so knowledge of how they work grows with them. Littman's history of reinforcement learning, with its decade-long stall after early successes, is the evidence he offers [1]. LeCun's jet engine is the same claim about another technology [5].

Finite problems have ceilings. A finite game has an optimal strategy, and performance approaches it with diminishing returns [1].

Costs grow with capability. Chip development costs double alongside chip performance, which is what turns an exponential into an S-curve [1]. Present hardware uses orders of magnitude more power than a brain [2][5].

Text is a thin slice of experience. LeCun's arithmetic compares the bytes in a language model's training set with the bytes reaching a small child's visual cortex, and finds the child ahead [5]. Littman's point about needing pushback is the behavioural version [1].

Errors compound. Because language models generate one token at a time, LeCun explains, each step carries some chance of leaving the set of reasonable answers, and the chance of staying correct shrinks with length [5].

Some things cannot be predicted faster than they happen. Wolfram's irreducibility means that intelligence does not grant foresight over every system, and it also means that no containment scheme can be proved perfect [3]. The argument limits the machine and its keepers alike.

Drives are evolved, not implied. The orangutan comparison is LeCun's evidence that high intelligence and a will to dominate can come apart [5].

Diffusion prevents monopoly. AlphaGo was reproduced within three months, LeCun notes, so no single actor is likely to hold a decisive lead, and defensive systems arrive along with offensive ones [5].

Where It Applies, and Where It Doesn't

These arguments apply to the fast, accidental takeover story. That is the scenario Littman summarizes from Bostrom, and it is the one each objection is aimed at [1].

They apply less to slow harms. Algorithms that shape attention, floods of fake content, and humans deferring to machine suggestions do not need a superintelligence, and the skeptics themselves take them seriously [1][2][3].

They apply less to misuse by people. LeCun's reply on bioweapons is that studies increasingly suggest language models give no real advantage over a search engine and that the hard part is building the thing [5]. That is a claim about present systems.

Several depend on today's architectures. LeCun's critique is of autoregressive language models, and he is actively working on what he thinks will replace them [5]. Jordan expects today's architectures to change within a decade [4]. An argument from current limitations has a shelf life.

They are not a proof of safety. Wolfram's own framework says a sandbox is never perfect [3]. Littman names a condition that would alarm him [1]. Fridman's point that nothing in physics rules the scenario out is not answered by any of the five [1].

They are also not unanimous about the future. One skeptic expects human-level AI, one will not discuss it, one says there is no mechanism for it [2][4][5].

What To Do With It

1. Test any doom argument against four questions. Does it require a sudden jump? Does it assume improvement without limit? Does it treat intelligence as a single scale with a top? Does it assume the system wants something? Each skeptic's objection targets one of these steps [1][2][3][5].

2. Ask for the mechanism. Cronin's standard is the asteroid: a risk you can estimate has a known causal path [2].

3. Do not take fluency for understanding. LeCun and Littman both warn that language models sound more capable than they are [1][5]. Wolfram's advice is not to expect factual output from them [3].

4. Watch the present harms. Recommendation algorithms tuned for advertising, fake users and automatic acceptance of machine suggestions are happening now [1][2][3].

5. Keep a human who is invested in the outcome in consequential decisions. Wolfram's example is the pilot who goes down with the plane [3]. Jordan's is the doctor you talk to about what the data mean [4].

6. Be careful what you let a model control. Wolfram's discomfort at letting a model run code on his own machine is a sensible instinct, given that no sandbox is perfect [3].

7. Know your own alarm condition. Littman's is a system that tries to prevent itself being shut off [1]. Cronin's test for real machine creativity is novelty he could not have predicted [2].

8. Favour diversity of systems over a single gatekeeper. LeCun's case is that "open source enables diversity," and that concentration is the larger risk [5].

9. Demand engineering discipline. Jordan's closing line, read out by Fridman, is "Let's broaden our scope, tone down the hype, and recognize the serious challenges ahead." [4]

Sources

1. Michael Littman on Reinforcement Learning, AI Risk Skepticism, and the Future of AI · Lex Fridman Podcast
2. Lee Cronin: Controversial Nature Paper on Evolution of Life and Universe · Lex Fridman Podcast
3. Stephen Wolfram on ChatGPT, Computational Irreducibility, Truth and the Second Law · Lex Fridman Podcast
4. Michael I. Jordan on Machine Learning, Markets, Recommender Systems and the Future of AI · Lex Fridman Podcast
5. Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI · Lex Fridman Podcast