Insight | Are chatbots minds? Three scientists weigh in on large language models

The Short Version

Three physicists who spend their careers on life, mind and the cosmos each give a sober answer to whether chatbots are minds, and none of the answers is a plain yes. Sean Carroll says a large language model is built to predict text, and that much of its apparent mind comes from our own evolved habit of reading intentions into anything that acts human [1]. Sara Walker sees language models as a new way of storing language, a kind of genetic system for language, and calls the story of superintelligent machines killing everyone a bad science fiction narrative [2]. Adam Frank doesn't rule out machine agency, but he doubts current models have it, because for him a mind is embodied, embedded, enacted and extended, something you do rather than something you have [3]. Put together, their views push back on both the hype and the fear: these systems are remarkable, but they are not simply digital people, and the best way to think about them starts with a clear view of what they are.

What They Said

Sean Carroll: a text predictor that exploits our bias

Sean Carroll, a theoretical physicist at Johns Hopkins and host of the Mindscape podcast, talked about AI in his third conversation with Lex Fridman [1]. His starting point may surprise people who expect a physicist to dismiss machine minds. He is a physicalist, and he says there is nothing in principle to stop a machine from thinking like a person: "I am a physicalist about consciousness and awareness and things like that. I see no obstacle to in principle building an artificial machine that is indistinguishable in thought and cognition from a human being." [1] Asked about the limits of large language models, he answered, "I don't think that there are many limits in principle." [1]

His objection is about what these systems are actually built to do. "But we're not trying to do that," he said, meaning build a humanlike mind. "What a large language model is trying to do is to predict text. That's what it does." [1] The trick, he argues, is that such a system leans on a deep feature of human psychology. It "is leveraging the fact that we human beings, for very good evolutionary biology reasons, attribute intentionality and intelligence and agency to things that act like human beings." [1]

He gave an everyday example. Driving to the recording, he ignored Google Maps to stop for coffee and went around a block the app "didn't like" [1]. The app kept telling him to turn, and he felt scolded: "It is impossible as a human being, not to feel a little bit sad that Google Maps is getting mad at you." [1] But the app isn't angry, and it isn't even a language model: "It's not a large language model. It's not as no aspirations to intentionality, but we attribute that all the time." [1] Carroll pointed to the philosopher Dan Dennett's influential paper on "the intentional stance" and the fact that "it's the most natural thing in the world for we human beings to attribute more intentionality to artificial things than are really there" [1]. He was careful to add, "which is not to say it can't be really there." [1] His rule for clear thinking: "the first step is to recognize our huge bias towards attributing things below the surface to systems" that seem human on the surface [1].

Fridman pushed back. If that bias toward intentionality runs through all the human data these models are trained on, won't intentionality emerge in the models as well? Carroll's answer: "Well, I don't think it will happen naturally. I think it could happen." [1] He stressed that "the way that large language models came to be and what they're optimized for is wildly different than the way that human beings came to be and what they're optimized for." [1] That leads to his most constructive point: "I think we're missing a chance to be much more clear headed about what large language models are by judging them against human beings. Again, both in positive ways and negative ways." [1]

That is also why he wants to drop the term AGI. "I want to eliminate the phrase AGI," he said [1]. Artificial intelligence is different from human intelligence, better at some things and worse at others [1], so measuring it along one humanlike scale hides more than it shows. Fridman noted that Carroll had previously said we are not as close to AGI as some public voices claim [1].

Carroll doesn't wave away the achievements. "Because what is clear is that they're very good at acting human," he said. "The open question in my mind is, is the easiest, most efficient best way to act human to do the same things the human beings do or are there other ways? And I think that's an open question." [1]

To show why that question stays open, he described a talk by the AI researcher Melanie Mitchell at the Santa Fe Institute [1]. Researchers fed a neural network millions of games of Othello written as symbolic moves like "E5, trained it to tell legal next moves from illegal ones, and then asked whether it had built an internal picture of the board [1]. An earlier paper said it had. Mitchell's group, following up, found that much of that apparent understanding came from the probe used to test it: what they found was like at least half of the improvement was built into the probe, not all of it, right? [1] And he added that an Othello board is way simpler than the world" [1].

His conclusion is openly agnostic: "it would be remarkable either way to learn that large language models that are good at doing what we train them to do are good because they've built the same kind of model of the world that we have in our minds, or that they're good despite not having that model. Either one of these is an amazing thing. I just don't think the data are clear on which one is true." [1] He also admitted some humility: "I was humbled by several stages in the machine learning development over the past 20 years. And I was just would never have predicted that LLM's the way they're trained on the skill of data they're trained would be as impressive as they are." [1] Both he and Fridman agreed the progress is remarkable [1].

Carroll also had a practical, physics-flavored point. He sees making computing far more energy efficient, closer to what the brain manages, as low-hanging fruit for physics [1]. And in a nearby exchange about the simulation hypothesis, he made a related point about hype in general: people will "realize how much harder it is to simulate a realistic world than we naively believe," so "this is not a my lifetime kind of worry" [1].

His wider philosophy frames all this. Carroll doesn't call consciousness an illusion; to him words like redness or sadness describe physical processes, just as hardness describes a table, and consciousness and free will are as real as tables and chairs [1]. So his doubt about chatbots isn't a claim that minds are magic. It's a claim about this particular kind of machine and about us, the people looking at it.

Sara Walker: a genetic system for language, not a new species

Sara Walker, an astrobiologist and theoretical physicist, comes at AI from her larger theory of life [2]. She rejects textbook definitions of life and offers instead: "life is the process of how information structures matter over time and space." [2] Any individual, in her view, is just an instance of lineages structured across time, and she even suggests that language, memes and mathematics may be alive in this sense, as open-ended combinatorial spaces with room for creativity [2].

Seen that way, a language model isn't an alien mind appearing from nowhere. It is the newest stage in a long history of how humans store language. Writing and books were each traumatic in their day, she notes, and now we can store the dynamics of language in an artifact, which makes LLMs a kind of genetic system for language and a dynamic crystallization of human collective intelligence [2]. The comparison to genetics is pointed: genes store the history of a lineage and let it carry forward, and in her picture language models do something like that for the lineage of human language.

This connects to her idea of the technosphere. "The universe is far larger in time than it is in space," she says, and "the modern technosphere is the largest object in time in the universe that we know about," meaning the global integration of life and technology [2]. Language models are part of that object, not outside it.

That is why she rejects the doom story. She dislikes the framing of superintelligent machines that will kill everyone, calling it a bad science fiction narrative, and draws on the physicist David Deutsch's idea that humans are already universal explainers [2]. The real picture, she says, is humans and technology co-evolving at the level of society [2]. She even suggests AI doomsday talk may work as a kind of cultural immune response [2].

Rejecting doom doesn't mean dismissing harm. Walker takes near-term harms seriously, listing deepfakes, fake kidnapping scams, weaponization and abuse imagery, and says legislation and guardrails are needed [2].

On whether AI is conscious, she stays in the gray zone. She expects people to form romantic relationships with AI, notes that some already believe it's conscious, and says we need a testable theory of consciousness before the question can leave that gray area [2]. Fridman worried that humans will resist recognizing consciousness in things we create and might mistreat them. Walker answered that we already do such things, and warned against treating AI as a separate species at all, since the technosphere is both human and technological and humans are good at labeling things as other [2].

Her own tentative theory of consciousness helps explain her caution. She resists assuming consciousness is fundamental or came before life. Her view is that we have conscious experience because we are temporally extended objects with a huge inner space, and that it is private because no one else occupies the same thread of time, though shared ancestry lets us partly share experience through language [2]. She describes intelligence as the ability of objects with a lot of selection built into them to select possible futures [2]. Under that picture, the depth of history an object carries matters a great deal, which makes her reluctant to treat a newly built artifact as a mind in the human sense, and just as reluctant to treat it as something wholly separate from us.

She also has a broader story that frames technology as life-like rather than threatening. She sees the history of life as recursion: an origin of life, four billion years of evolution, and then a technosphere whose technologies begin to take on life-like properties and might one day copy itself onto another planet, the way the first cell split into two [2]. The origin of life, she says, is happening again now, at scales from individuals to the internet [2]. And she sees computation as sitting at the top of reality rather than its base, because it was built by our biosphere after billions of years of evolution and only happens in some physical objects [2]. In a tweet Fridman read, she argued that the world is not a computation, but that computation is our best current language for understanding it [2].

Adam Frank: agency is the real question, and current models probably don't have it

Adam Frank, an astrophysicist who leads the first NASA grant to search for technosignatures on exoplanets, arrives at AI through the question he now considers the most important in physics [3]. "The most important question for physics is what is life? What the hell is the difference between a rock and a cell, fundamentally?" he said, and then narrowed it further: "really the fundamental question that is the, is agency? What does it mean to be an autonomous agent? How the hell does that happen?" [3] He is explicit about his stance: "I'm not a reductionist. I'm not somebody who's just like, well, you just put together enough chemicals and Big Bang boom, and suddenly it appears. There's something that really is going to demand a reconception of what nature itself is." [3]

With that bar set, Frank doesn't rule out machine agency, but he doubts current LLMs have reached it [3]. His reason is his view of cognition. He favors the "4E view: embodied, embedded, enacted and extended [3]. He tells the story of Gregory Bateson asking Jonas Salk where his mind was; Salk pointed to his head, and Bateson said no, out here [3]. For Frank, experience is not a thing you have, it is a thing you do." [3]

He even avoids the word consciousness: "I tend not to use the word consciousness because it's, I think it leads us in the wrong direction. We should focus on experience, because it's a verb kind of an way or it's verb, it's verb like." [3] A text model that sits on a server and produces words when prompted doesn't obviously fit a picture in which mind is an ongoing activity of a body in a world.

Frank also stresses that individuality is fluid. He cites David Krakauer's information-theory work at Santa Fe: "my liver cell is an individual, but really it's part of the liver." [3] He goes on, "I couldn't be me without the entire community of say language users, right? I wouldn't even be able to frame any questions." [3] Then he makes the link to Walker directly: "This is like Sarah Walker stuff." [3]

His research on living cells gives this a technical side. With a Templeton Foundation grant, his group studies semantic information in living systems [3]. Shannon information measures how much surprise there is in a message, but "Semantic information focuses on meaning," he explains, asking "how much of the information that I'm, that the agent, you know, the critter is getting from the world, actually has a, helps it survive" [3]. He sees the same structure in QBism, an interpretation of quantum mechanics in which you have an agent who makes an action on the world, and then the information they get from that action through the experiment... updates their priors [3]. In other words, for Frank, information that matters is information that counts for an agent trying to keep itself going. He is also drawn to Robert Rosen's contested argument that living systems cannot be fully captured by algorithms [3].

His book The Blind Spot, written with Marcelo Gleiser and Evan Thompson, supplies the philosophy behind his doubt [3]. The book argues that science depends on experience yet is blind to it, and that a bundle of views, including reductionism, physicalism and treating mathematics as more real than experience, ends up "pushing experience out and saying experience is an epiphenomena" [3]. His "parable of temperature" shows how science started from the felt experience of hot and cold, built thermometers and thermodynamics, and then began treating 60 degrees as more real than feeling cold, an "amnesia of experience" [3]. His advice: "don't confuse the map for the terrain." [3] Applied to chatbots, the warning is clear even though he doesn't spell it out that way: a model that reproduces the structure of human language is working with the map, and mistaking it for the lived territory of a mind would be the same error.

Frank also brings a general rule about claims that sound exciting. About UFOs, he says scientists are "really mean to each other" about standards of evidence, and that rigor is why cell phones work: "You don't get a pass because it's a really cool topic." [3] And he warns about a bias of our era: we live in a great acceleration and may wrongly assume technology always advances [3]. "So you have to be careful about that bias that we have living in this time of great acceleration," he said [3]. When Fridman replied that we are also bad at predicting what continued acceleration would bring [3], Frank agreed a breakthrough could change everything in physics [3]. It's a reminder that confident forecasts, hopeful or fearful, rest on shaky ground.

Finally, Frank lists AI among the threats humanity has to survive, alongside climate change and nuclear war, before it can go on to the solar system [3]. So he doesn't dismiss AI risk. What he doubts is that today's systems are agents.

Where They Agree, and Where They Don't

Where they agree

All three reject the idea that current chatbots are straightforwardly minds like ours. Carroll says models are built to predict text and that we over-read intention into them [1]. Walker says we lack a testable theory of consciousness and so the question stays in a gray area [2]. Frank doubts current LLMs have real agency [3].

None of them rules out machine minds in principle. Carroll sees "no obstacle to in principle" building a humanlike artificial mind [1]. Walker expects technologies to acquire life-like properties as part of the technosphere [2]. Frank doesn't rule out machine agency [3]. That shared openness matters: their skepticism is about what exists now and how it was built, not a claim that silicon can never think.

All three treat mind as something tied to history and context rather than a standalone calculation. Carroll stresses that humans and models "came to be" in wildly different ways and are optimized for different things [1]. Walker ties consciousness to being a temporally extended object with a long history [2]. Frank ties mind to a body, an environment and a community of language users [3]. And Frank explicitly links his view to Walker's work on lineages [3].

All three counsel humility. Carroll says "I just don't think the data are clear" on the world-model question and that machine learning progress has repeatedly humbled him [1]. Walker won't settle AI consciousness without a testable theory [2]. Frank warns about the great-acceleration bias and insists on high standards of evidence [3].

And two of them are clearly skeptical of the fear story. Walker calls the killer-superintelligence framing a bad science fiction narrative [2]. Carroll, discussing simulated worlds, predicts people will discover how much harder realistic simulation is than they think [1].

Where they differ

The biggest difference is where each one puts the problem. For Carroll, the main error is in us: the human bias to attribute intentions is the thing to correct first [1]. For Walker, the main error is in categories: treating AI as a separate species, apart from the human-technological system that made it [2]. For Frank, the main gap is in the machines themselves, or at least in what they lack: embodiment, embeddedness and agency [3].

They also frame the future differently. Carroll's position is an open empirical question: do models succeed by modeling the world as we do, or in some other way? The Othello result is evidence, not a verdict [1]. Walker's position is a large story about co-evolution, in which language models are one more step in life's recursive history [2]. Frank's position leans philosophical: agency may "demand a reconception of what nature itself is," and he takes seriously the argument that life can't be fully captured by algorithms [3]. Carroll, by contrast, sees no need for new principles; he thinks the difference between models and people lies in how they were made and what they're optimized for, not in anything beyond physics [1].

That touches a deeper disagreement about physicalism. Carroll and Walker both call themselves physicalists [1][2]. Frank rejects both physicalism and idealism, since both assume an outside, third-person view that he calls a useful fiction [3]. So when Carroll says a machine could in principle match a human mind, he is speaking from a view Frank thinks leaves out experience. That helps explain why Frank sets the bar for machine minds at agency and lived experience, while Carroll sets it at cognition that is "indistinguishable in thought" from a human [1][3].

On risk, their emphasis differs. Walker explicitly rejects AI doom narratives while taking near-term harms like deepfakes and scams seriously [2]. Frank lists AI next to climate change and nuclear war among the dangers humanity must survive [3]. Carroll, in these sources, speaks about AI mainly as a question of understanding rather than danger [1]. These aren't flat contradictions, but readers should notice that AI isn't a mind and AI isn't dangerous are different claims, and the three scientists don't all make the second one.

Why It Works

The sources give several reasons for their caution.

An evolved bias toward intention. Carroll's argument rests on the claim that humans, "for very good evolutionary biology reasons," attribute "intentionality and intelligence and agency to things that act like human beings" [1]. He cites Dennett's "intentional stance" to name this habit [1]. The Google Maps example shows it working even on a simple navigation app that has "no aspirations to intentionality" [1]. If we feel scolded by a turn-by-turn app, a system trained on huge amounts of human writing will trigger that bias far more strongly. That doesn't prove there's nothing beneath the surface, but it means how humanlike a chatbot feels is poor evidence of a mind.

Different origins, different optimization. Carroll's second point is that training history matters. Humans and models came to be in ways that are "wildly different," and are optimized for different things [1]. A model that is very good at acting human might reach that result by a different route, which is why he asks whether copying humans is "the easiest, most efficient best way to act human" or whether there are other ways [1].

Probes can create the understanding they claim to find. The Othello study is the closest thing to hard evidence in the sources. A network trained only on move sequences seemed to have built an internal model of the board, but Mitchell's group found "at least half of the improvement was built into the probe" [1]. The lesson is about method: tests that look inside a model can bring their own structure with them, so claims that a model "understands" need careful checking. And even a clean result on a simple board game wouldn't settle the question for the full world [1].

Language as stored history. Walker's mechanism is about information over time. If life is "the process of how information structures matter over time and space" [2], then a system that captures the dynamics of human language is a new storage medium for a lineage that already exists, like writing and books before it [2]. That explains both halves of her view: why models seem so rich (they crystallize human collective intelligence) and why they aren't a separate species (they are part of the human-technological technosphere) [2].

Consciousness as depth in time. Walker's tentative theory holds that consciousness comes from being a temporally extended object with a large inner space, and that it is private because no one else shares the same thread of time [2]. Her view of intelligence as objects with much selection embedded in them choosing among futures [2] suggests a test for any system: how much history and selection does it actually carry, and does it use that to select its own futures?

Mind as action in a world. Frank's 4E view says cognition is embodied, embedded, enacted and extended, and that experience "is not a thing you have, it is a thing you do" [3]. His semantic information work adds that meaningful information is the kind that "helps it survive," information that matters to an agent's own continuation [3]. A system without its own stakes in the world lacks that kind of meaning, on this account. His QBism example follows the same loop: an agent acts, gets information back, and updates [3].

Maps versus territory. The Blind Spot argues that science can forget the experience it grew from and start treating its abstractions as more real than what they describe [3]. That gives a general reason to be careful about treating a model of language, however good, as a mind.

Where It Applies, and Where It Doesn't

It applies to today's large language models. Carroll's points are specifically about systems trained "to predict text" [1], and Frank's doubt is specifically about "current LLMs" [3]. Walker's framing covers LLMs as a way of storing language [2].

It doesn't apply as a claim about all possible machines. Carroll sees no obstacle in principle to a machine that is "indistinguishable in thought and cognition from a human being" [1], and Frank doesn't rule out machine agency [3]. Their skepticism would not automatically carry over to a system built differently, for example one that is embodied and acts in the world, which is exactly what Frank's 4E view would look at.

It doesn't settle what the models do internally. Carroll is clear that the data don't yet say whether models succeed by modeling the world as we do or by some other route [1]. The Othello finding weakens one strong claim of understanding; it doesn't prove there is none, since the probe explained "at least half" of the improvement, "not all of it" [1].

It doesn't dismiss harm. Walker's rejection of doom narratives comes paired with concern about deepfakes, fake kidnapping scams, weaponization and abuse imagery, and a call for legislation and guardrails [2]. Frank counts AI among the dangers humanity must get through [3]. Not a mind is not the same as "harmless."

It doesn't settle AI consciousness. Walker says we need a testable theory of consciousness first [2]. Carroll's warning is that our feelings are a biased instrument [1]. Neither gives a final answer, and Walker notes that some people already believe AI is conscious and that romantic relationships with AI are coming [2].

It applies as a general habit of mind. Frank's warnings about the great-acceleration bias and about cool topics not getting "a pass" [3], and Carroll's point that realistic simulation is harder than people naively believe [1], apply well beyond chatbots. They are tools for reading any big claim about technology.

It has limits as a source. All three views come from long, wide-ranging conversations in which AI was one topic among many, alongside black holes, alien life and the origin of life [1][2][3]. Their AI comments are informed by deep work on life and mind, but none of the three presents themselves as an AI researcher, and Carroll leans on Melanie Mitchell's work for his main evidence [1].

What To Do With It

1. Notice when you feel a machine's "intentions." Carroll's first step for clear thinking is "to recognize our huge bias towards attributing things below the surface" to things that act human [1]. If you catch yourself feeling that an app is annoyed, pleased or trying to help, as he did with Google Maps, treat that as a fact about you before treating it as a fact about the machine [1].

2. Judge models on their own terms, not as imitation humans. Carroll says we're "missing a chance to be much more clear headed about what large language models are by judging them against human beings" [1]. Ask what a tool is good and bad at, since it can be better than people at some things and worse at others [1], rather than asking how close it is to a person.

3. Drop "AGI as a yardstick. Carroll wants to eliminate the phrase AGI" [1]. When you read predictions framed around it, ask which specific abilities are being claimed and how they would be measured.

4. Be skeptical of claims that a model "understands," and check the method. The Othello story shows that a test can build in much of the understanding it claims to find [1]. When a headline says a model has an internal world model, look for whether the result survived follow-up work.

5. Hold both possibilities open. Carroll says models might be good because they've built a world model like ours, or good "despite not having that model," and "Either one of these is an amazing thing" [1]. Don't let either hype or dismissal close a question the data haven't closed.

6. Treat AI as part of the human system, not a rival species. Walker warns against labeling AI as other, since the technosphere is both human and technological [2]. Think about your own use of these tools as part of the co-evolution of people and technology she describes [2].

7. Put your worry where the evidence is. Walker takes deepfakes, fake kidnapping scams, weaponization and abuse imagery seriously and calls for legislation and guardrails, while calling the killer-superintelligence story bad science fiction [2]. Be alert to scams and fakes, and support concrete guardrails, rather than spending attention on doomsday narratives.

8. Ask what a system actually does in the world. Frank's test for mind is agency and experience that is enacted, embodied and embedded: "not a thing you have," but "a thing you do" [3]. When deciding how much to trust or credit a system, ask whether it acts for its own ends, or produces text when prompted.

9. Don't confuse the map for the terrain. Frank's advice [3] applies directly: a system that reproduces the patterns of human language is a powerful map of human expression, not proof of a lived inner life.

10. Demand evidence, and watch your era's bias. Frank says no claim gets "a pass because it's a really cool topic," and warns that living through a great acceleration can make us assume technology will always keep advancing [3]. Apply that same standard to confident forecasts about AI, whether they promise salvation or catastrophe.

Sources

1. Sean Carroll: General Relativity, Quantum Mechanics, Black Holes & Aliens · Lex Fridman Podcast
2. Sara Walker: Physics of Life, Time, Complexity, and Aliens · Lex Fridman Podcast
3. Alien Civilizations and the Search for Extraterrestrial Life · Lex Fridman Podcast