What brains do that AI still does not: neuroscientists' list

The Short Version

Ask researchers who study brains, vision, language and learning what a brain does that current AI does not, and they give a consistent list. A brain physically rewires itself as it learns, perceives by projecting a model of the world onto what it senses, grounds its concepts in a body's experience, thinks without needing language, learns from a handful of examples, keeps two kinds of memory, generates its own thoughts when nothing is coming in, and cares about outcomes [1][2][3][4][5][7]. Language models, by their accounts, have mastered the form of language and learned the statistics of an enormous record of human output, which is a real achievement and a partial one [2][4][5]. The experts differ on how big the gap is: Terry Sejnowski thinks the models generalize and build internal models of meaning, while Dileep George says text does not contain the missing information at all [2][3]. And Michael I. Jordan adds a caution that sits over the whole list: "We understand artificial perhaps, but we don't understand intelligence" [6].

Table of contents

What They Said

The seven people here are not all neuroscientists. David Eagleman, Terry Sejnowski and Charan Ranganath study the brain. Dileep George and Fei-Fei Li build AI systems inspired by it. Ted Gibson is a psycholinguist, and Michael I. Jordan is a statistician and machine learning researcher [1][2][3][4][5][6][7]. Their conversations were also recorded years apart, and two of them were talking about GPT-3 [1][2]. What follows is organized by ability, since several of them arrive at the same items from different directions.

It rewires itself

Eagleman's starting point is that the hardware-and-software picture fails for the brain. He calls it livewired: "a system that's constantly reconfiguring itself physically as it learns and adapts to the world around it" [1]. In a computer there is a fixed hardware layer with software on top. In the brain, activity passing through the system changes the system [1].

His proof is what brains survive. "I can't tear half the circuitry out of your cell phone and expect it to still function. But you can do that with the brain," he says. Children under about seven can have an entire hemisphere removed and come out with a slight limp, and children born without one hemisphere have their visual system rewire onto the side that remains [1]. What thousands of such cases teach, in his words, is that the brain is "simply trying to accomplish the tasks in front of it by rewiring itself with the available real estate" [1].

The change never stops. He uses the word plasticity sparingly, because plastic is molded once and holds its shape, and with the brain "The whole point is for it to keep changing" [1]. It also happens at many levels at once. The synaptic weight, the quantity engineers focus on, is only one of them. The distribution of receptors, the structure of the neuron, the biochemical cascades inside it and the epigenome all change too, and synapses get the attention mainly because they are what can be measured [1]. Different regions harden on different schedules, depending on how stable their input is: the visual cortex settles within a few years because the visual world keeps offering the same angles and colours, while the areas for touch and movement stay malleable because the body keeps changing [1].

The same flexibility lets the brain accept inputs it was never built for. Cochlear and retinal implants do not use the natural biological code. "They speak the dialect of Silicon Valley," Eagleman says, and the brain works out how to use them anyway by finding correlations with its other senses [1]. His company's wristband turns sound into vibration patterns on the skin, and deaf users go from decoding buzzes to simply perceiving that a dog is barking [1].

His contrast with machines comes from the next-to-last chapter of his book. A wolf caught in a trap chews off its leg and learns to walk on three. The Curiosity rover, a multibillion-dollar machine pre-programmed for its wheels, was finished by one of them getting stuck [1].

Sejnowski supplies a related observation about language models. Once trained, a model's weights are fixed, and all it does afterwards is inference. Yet over a long dialogue it seems to get better, a phenomenon called in-context learning, "even though there's no plasticity" [3]. He calls that a mystery, and adds that nobody knows how humans hold an hour-long lecture in mind either [3].

It wants something

Eagleman's second item is the one he thinks matters most. GPT-3 is "extremely impressive," he says, but it is "a good impersonator," and the passages people share have already been filtered by humans throwing out the nonsense [1]. He does not think adding parameters will close the gap [1].

What is missing is relevance and drive. Humans encode only what matters to them, grounded ultimately in survival and built up into countless personal goals, such as how a tweet will look to a department chairman or to your mother [1]. Everything he says in the conversation, he points out, is shaped for the particular person listening [1]. Learning in a brain follows from this. When electrodes in the motor cortex drive a robotic arm for a paralyzed woman, her brain learns to use it because "it's all about relevance": if there is food and she is hungry, she will work out how to get it to her mouth [1]. His current writing carries the title "if you want to build a robot, start with the stomach" [1].

Fei-Fei Li makes the same point about today's chatbots from the engineering side. Asked whether machines have anything like urgency or motivation, she says a mode such as thinking longer or answering quickly is "just a different kind of objective for the algorithm" [4]. A chatbot says it is sorry you are sick because of patterns in its data, not because it has known pain. "We operate fundamentally different from today's AI," she says, and she does not want the public confused about that [4].

Sejnowski adds a qualification. The brain's basic motivational learning rule is not foreign to machines. The basal ganglia learn by a rule he describes as "simply to predict the next reward you're going to get," comparing what was expected with what arrived and updating, and this builds a value function, an accumulated sense of what is good or bad for you [3]. That same temporal difference learning is what DeepMind used in AlphaGo [3]. What differs in a person is what the function is built from: years of rewards and punishments, including a single terrible experience that is never forgotten, in a creature that arrives not knowing its language or culture and has to learn both [3].

It perceives by explaining

George's contribution is about the direction information flows. The visual cortex is arranged in roughly four hierarchical levels, and the fact that surprises machine learning people is that it has more feedback connections than feed-forward ones [2].

He gives two experiments. Shown a textured triangle on a textured background, neurons settle on the edges first and then fill in the surface from the edges inward [2]. Shown the Kanizsa triangle, an illusion made of three Pac-Man shapes, neurons signal edges that are not in the image, and they do so later than they signal real edges, which points to feedback from context as the cause [2]. His summary: "our brain is building a model of the world," and "We are constantly projecting our expectations onto the world" [2]. Perception, on this account, is a search for the best explanation of the input given the model you hold [2].

That requires inference of a particular kind. His example is "explaining away." A home alarm can be triggered by a burglar or an earthquake. You hear it and suspect a burglar, then the radio reports an earthquake and your belief in the burglar drops, because competing hypotheses fight to explain the same evidence [2]. A neural network can produce something like this reasoning if it was shown every combination in training, which he calls amortized inference. His own system, the Recursive Cortical Network, works out the explanation when it sees the image, and it can run backward, generating as well as recognizing [2].

His test case was the text CAPTCHA, where crowded letters make an r beside an n look like an m, so local evidence misleads and only the global view makes sense [2]. He is careful about what he claims. The problem is not solved the way humans solve it: a five-year-old who has just learned letters can read a CAPTCHA style never seen before, and no current deep learning system can do that without training on the style [2]. He quotes Douglas Hofstadter that the "central problem in AI is what is the letter A?", since recognizing every possible A, made of leaves or of two people leaning together, would require understanding nearly everything [2].

He also notes that the transformer, the architecture behind language models, is still feed-forward [2]. Sejnowski's picture of the brain points the same way: recordings show even simple tasks engaging many brain areas at once, and in mice the visual cortex receives more input from the motor system than from the eyes [3].

It knows a world, not a text

For George, perception is the base of thought. Concepts are grounded in sensory and motor experience, and he says he disagrees with Chomsky "completely" on this [2]. His example is a sentence: Sally pounded a nail on the ceiling. Is the nail horizontal or vertical? People answer vertical by picturing the scene, drawing on knowledge built from experience and mostly in place by age five. His summary is that "language is simulation control": words steer a simulation that lives in the visual and motor systems [2].

From there his criticism of GPT-3 follows directly. "It is not really carrying a world state around," he says, which is why earlier models wrote of going up a hill to reach a valley. Such a system does have a model, but it is a "model of the text world" [2]. When Fridman asks whether compressing enough text could force real abstractions to form, George answers, "No I don't think it is possible because the information is not there" [2]. Most of what we know about how the world works, such as that walking forward is easier than walking backward, is never written down [2]. Causality, counterfactual reasoning and intervention need other kinds of models, and he jokes that we hand simple text puzzles to a system that has read all of quantum mechanics, when a real general intelligence would run experiments and revise hypotheses [2].

Li describes the same limit in video. Models can now generate clips of a cat moving plausibly. Does the algorithm know the muscle structure of a cat's legs? "I would say the algorithm doesn't," she says. It has seen a great many cat videos: "It's the statistics" [4]. She is candid that this resembles people more than we might like, since most of us could not describe feline anatomy either and have simply seen many cats move [4].

Her conclusion is about where to go. "The next frontier of AI, as I have been saying, is beyond language," she says, because humans develop pre-verbally and "Evolution took, you know, 500 million years without verbal communication" [4]. Vision, in her telling, is the root: an estimated half of cortical activity involves it, and children are visual before they are verbal [4]. Her company World Labs builds models for what she calls spatial intelligence [4]. George states the order of difficulty as a paradox: "I think of perception as kind of the first thing that you have to build, but the last thing that you will be actually solved" [2].

It thinks without language

Gibson brings the most direct evidence that a language system is not a thinking system. His wife, the MIT neuroscientist Ev Fedorenko, can locate a person's language network with brain imaging in about 15 minutes. It sits mainly in the left hemisphere and responds the same way to spoken and written language, to understanding and producing it, and to every language a person speaks fluently [5]. It is stable over years: "my language network is like my face," Gibson says [5].

The key finding is what does not activate it. Spatial memory, music, mathematics and reading computer code leave it silent and use other areas, including a network for hard tasks generally [5]. A second line of evidence comes from Rosemary Varley's patients with global aphasia, whose strokes destroyed their language networks. They cannot speak, understand, read or write, and they can play chess, drive, do arithmetic and perform as well as healthy people of the same age on such tasks [5]. Gibson's conclusion: "language is a communication system for the meaning. It's not the meaning" [5].

He applies it to language models with care, because he also admires them. He has written that they are arguably the best current theories of human language, since they predict what is and is not good English better than any other theory, and Stanford researchers found evidence that models build something like the dependency structure of sentences internally [5]. They even struggle with deeply nested sentences in the way people do, without being trained to [5]. "They're doing the form, they're doing it really, really well," he says [5].

Meaning is another matter. His example is a modified Monty Hall puzzle. Tell a model you already know the car is behind door one, and when the host opens door three it still advises switching, because it is following the familiar form of the puzzle. People rarely make that mistake [5]. He notes that Fedorenko's group has argued that the models lack the thinking and reasoning the brain handles separately from language [5].

It learns from very little

Li came at this from the opposite side. In 2006 she noticed how little data AI algorithms had to learn from, compared with children. "Humans by age six can learn tens of thousands of different object categories," she says, having been flooded with visual experience since birth [4]. Her bet that data was the bottleneck produced ImageNet, 15 million images, and the combination of that data, neural networks and graphics chips started the modern era in 2012 [4]. The recipe has not changed since, she says. About video generation she remarks that "the bottom line is it's still data" [4].

But the comparison still runs against the machine. A model recognizes a cat from a glimpse of a tail because it has learned from enormous amounts of data. A child who has seen perhaps three to ten cats manages the same thing through a learning pathway that, she says, science has not fully explained [4]. The practical consequence shows up wherever examples are scarce. Her father's liver surgery was done by a surgeon operating a robot, and she asked whether AI might someday do it alone. The liver is so complex and variable that even all the world's liver operations might not supply enough data [4].

George's results make the same point in miniature. He designed his benchmarks so the training set was small, and recalls about 95 percent accuracy on handwritten digits with training examples numbering in the tens or hundreds [2]. Sejnowski says "The brain is really, really good at generalizing," often from one example, such as learning how restaurants work after a single visit [3].

It keeps two kinds of memory

Ranganath explains part of how a brain manages that economy. Adding memory to AI is easy superficially and hard deeply, he says, because of a dilemma between stability and plasticity: a system that keeps its rules has trouble learning the exception, and one that learns quickly overwrites what it knew [7].

Humans solve it with two systems. One is what he calls "a fairly dumb system which just says this happened once at this point in time," episodic memory. The other is semantic memory, knowledge accumulated slowly from experience [7]. When something surprising violates expectations, the first system records it, and next time the brain can supplement its general knowledge with that one event. This lets a person "stop on a dime and change without having to erase everything we've already learned," and "it gives you the ability to learn from so much less information" [7]. In his words, "one of the things I think that makes humans great is having both episodic and semantic memory" [7].

The hard part to copy is the judgment built into it. When do you record a moment, how much of it, what do you prioritize and when do you discard it? "These are the hard questions we're still trying to figure out in people," he says [7]. His own lab's work suggests the brain does not record constantly. It records at moments of surprise or change called event boundaries [7]. And the purpose is forward-looking: the building blocks of past experience can be reassembled into models of events that have not happened, which is why imagining the future and remembering the past look almost identical in brain maps [7].

George offers a compatible hypothesis, that the hippocampus stores a sequence of pointers into the cortex's model of the world and the cortex replays the event on recall [2]. Sejnowski describes the transfer between the two systems: during sleep the hippocampus replays the day's experiences, and brief traveling waves across the cortex called sleep spindles weave them in without overwriting existing knowledge [3].

It keeps running

Sejnowski names what he considers the clearest difference. Sit in a quiet room with no stimulation and "your brain keeps thinking." You plan, or go back over the day. "Your brain is always generating internally" [3]. A language model, after you stop talking to it, "just goes blank. There is no self-continuous, self-generated thoughts" [3].

He treats that internal activity as productive. His own best ideas arrive while walking or jogging without anything in his ears, and the course he co-created teaches students who are stuck on a problem to walk away from it [3].

It decides under uncertainty

Jordan's item is the difference between predicting and deciding. Much of machine learning, he says, has emphasized pattern recognition and prediction, and he finds that far too limited [6]. "We cannot do perfect prediction," he says, and life constantly presents situations nobody anticipated, in which a person has to weigh what they want, what each action costs, the risks, and what other people are doing [6]. Most current prediction systems do no risk evaluation: "They have no error bars" [6].

His example is medical. Even with all the heart-attack data ever collected, he would not simply act on a network's output: "I'm not going to have a heart attack because some number is over 0.7" [6]. He would ask what-if questions, look at other data, think about causes, and talk to a doctor about how the data was gathered [6].

It has an inside nobody recorded

Li points to a limit of the training material itself. The "Internet is the biggest collection of human behavior in multimodal forms," she says, decades of text, photographs, video and music [4]. But a private insight, such as what Picasso was thinking behind a portrait or the childhood memory a grey cup calls up, was never written down. "That thought is not captured. Therefore, it's not on the internet. Therefore, AI has not seen it" [4].

Jordan makes a similar objection to systems that model people from their clicks. A sleepy click on a story about the Queen of England would be read as interest. People are too rich and quirky to be reconstructed that way, he says: "What's unknown about me is the most interesting" [6]. Eagleman describes the general condition. Each brain is shut in silence and darkness building a model from a few sensors, "essentially on your own planet in there," connected to others by a very thin channel [1].

Where They Agree, and Where They Don't

Language models learned something real. Nobody here dismisses them. Eagleman calls GPT-3 extremely impressive, Gibson calls the models the best theory of language form we have, and Sejnowski and Li both describe uses in science and medicine [1][3][4][5].

What they learned is the form or the statistics. Gibson says form, George says a model of the text world, Li says statistics and patterns, Eagleman says impersonation [1][2][4][5].

The core of intelligence is older than language. George puts perception first, Li puts vision first, and Gibson shows that thinking and language are separate systems in the brain [2][4][5].

How much do the models understand? Here they split. Sejnowski says they are not parrots that memorize: they generalize to new cases, and although trained only to predict the next word, they build internal semantic models of meaning. He even reads hallucination as "a sign of a real generative internal model" [3]. George holds that the needed information is absent from text, and Gibson's puzzle shows a model following form over meaning [2][5]. Sejnowski's reply to the whole dispute is that the terms are "weasel words because you can't pin them down," and that the argument reveals that "we don't understand what understanding is" [3]. The disagreement partly reflects what each is looking at. Sejnowski sees generalization in behaviour, while George and Gibson are asking whether a model of the world sits behind it.

Will scale close the gap? Eagleman does not think more parameters will do it, and George says no amount of text contains the missing knowledge [1][2]. Fridman, partly as devil's advocate, argues that compression of huge amounts of text might force abstractions to form, that self-play and neural networks have repeatedly surprised him, and that form alone gets surprisingly close to apparent understanding [2][5]. Li's history cuts both ways: more data is exactly what worked, and it is also what runs out [4].

Should AI copy the brain? George says biology is a serious input, insists "I don't treat brain inspired as a marketing term," and warns against "slavishly mimicking all the things of the brain." Convolution, for example, is an engineering trick the cortex does not use [2]. To those who say "airplanes don't flap wings," he answers that the Wright brothers learned flapping was not the point by watching buzzards soar, which showed them control was the real problem [2]. Jordan is much more sceptical. We have no clue how the brain computes, he says: "If you look at a electron micrograph of a single synapse, it's a city." The metaphors are fun and are not science, and layered arithmetic circuits did not come from neuroscience [6]. Eagleman, a neuroscientist, shares the humility if not the pessimism. "Our science is still quite young," he says, though he expects the complexity to shrink once general principles are found [1]. Sejnowski sees the two fields converging on a middle level of algorithms, and his current hypothesis is that the brain does something like a transformer's self-attention through the basal ganglia [3].

Is human intelligence even the right yardstick? Jordan says intelligence is broader than the human kind. "Markets are intelligent," he argues, being robust and adaptive at every scale, and he sees the present era as the birth of an engineering discipline for decisions at scale, not the creation of thinking machines [6]. Li keeps the human at the centre, and Huberman objects in the Sejnowski conversation that models are unfairly compared with the best humans instead of average ones [3][4].

Why It Works

The sources explain why brains have these abilities more than they prove that machines lack them.

A brain is built to be shaped. Eagleman says humans "drop into a world with a half-baked brain" and absorb language, culture and customs, while an alligator arrives essentially finished. Genetics builds just enough structure for the organ to become a sponge [1].

Stakes select what is learned. Relevance decides what a brain encodes, and Sejnowski's reward prediction rule is the mechanism that turns outcomes into a lasting value function [1][3]. Ranganath adds that the brain runs on roughly 12 to 20 watts and so must prioritize, with surprise, fear and stress releasing chemicals that promote lasting memories [7].

Feedback lets a model correct its senses. The delayed neural response to illusory edges is George's evidence that higher levels send expectations down, and inference among competing explanations is what makes sense of ambiguous input [2].

Experience supplies what text leaves out. George's nail on the ceiling is answered from simulation, using knowledge nobody writes down [2].

Separate systems do separate jobs. The imaging and aphasia evidence shows language handled apart from reasoning, and the pairing of episodic and semantic memory shows fast and slow learning handled apart [5][7].

The limits of the evidence. Much of this is observation and argument. Gibson's Monty Hall case is one example, and Fridman points out that anecdotes are not systematic evidence [5]. Li says the child's learning pathway is not fully explained, Ranganath says the memory questions are still open in people, and Sejnowski says in-context learning is unexplained in machines [3][4][7].

Where It Applies, and Where It Doesn't

The list describes systems at the time of each conversation. Eagleman and George were discussing GPT-3, and Li says video training arrived around 2023 [1][2][4]. Li also says her company is building spatial models now, so grounding is a research direction and not a permanent wall [4].

Some items are already partly present in machines. Reward-prediction learning runs in AlphaGo, George's network uses feedback and generation, and language models show in-context learning [2][3].

Machines beat people where patterns are plentiful. Image-labelling algorithms passed the human error rate of about 4 percent by around 2016. A model trained on past hurricanes predicts landfall on a laptop in ten minutes better than supercomputer simulations that take days [3][4].

The gaps are complementary in practice. In a dermatology study Sejnowski cites, doctors and an AI each classified skin lesions with about 90 percent accuracy, and doctors using the AI reached 98 percent, because the AI knew rare lesions and the doctors knew common ones in depth. "It's not going to be adversarial. It's going to be a partnership" [3]. Li expects the same for hard mathematics: "My current conjecture is hybrid" [4].

Human abilities have their own failure modes. Ranganath stresses that memory is a construction prone to distortion, and Eagleman that an ageing brain's model may stop changing because it decides it already understands the world [1][7]. Ranganath would doubt claims that AI replicates human intelligence until it makes the same characteristic mistakes people do [7].

Where data is thin, keep a person involved. Li's lesson from the liver is that human and machine together beat an under-trained robot [4]. For anything involving your own health, a model's answer is a prompt for a conversation with a doctor, as Jordan's example suggests [6].

What To Do With It

1. Read fluent output as form, then check the meaning. Change a familiar problem slightly and see whether the answer changes with it, as Gibson's puzzle does [5].

2. Ask what the system could have seen. Knowledge that is rarely written down, private thoughts and rare cases are where it is weakest [2][4].

3. Do not act on a bare number. Ask how the data was gathered, what the uncertainty is and what would happen under different conditions, and take medical predictions to a doctor [6].

4. Use the model where it complements you. Let it supply breadth and rare cases while you supply depth, as in the dermatology result, and use it as what Rusty Gage's lab calls an "idea pump" by giving it your results and asking for new experiments [3].

5. Talk to it as you would to a person. Sejnowski relays the account of a writer who got better answers and felt less drained once she did: "You treat it like a machine. It will treat you like a machine" [3].

6. Learn how the technology works. Li's advice applies to everyone, coder or not, and she counts asking good questions as a skill to be taught [4].

7. Exercise the abilities on the list. Seek novelty and challenge to keep the brain rewiring, wrestle with problems yourself because "it has to be active learning," and follow Ranganath's mantra to "diversify your training data" [1][3][7].

8. Leave room for self-generated thought. Walk without input when you are stuck, and think about the problem before sleep [3].

9. If you build systems, test what the brain does well. Use benchmarks that demand generalization from a few examples, give models a world state to carry, and take Eagleman's suggestion to start with the stomach [1][2].

10. Keep claims modest. Jordan and Sejnowski both note that words like intelligence and understanding lack agreed definitions, so say what a system does on a given task [3][6].

Sources

1. David Eagleman on the Livewired Brain: Neuroplasticity, New Senses and What AI Is Missing · Lex Fridman Podcast
2. Dileep George on Brain-Inspired AI: Feedback, Inference, CAPTCHAs and the Limits of GPT-3 · Lex Fridman Podcast
3. Dr Terry Sejnowski How To Improve At Learning Using Neuroscience Ai · Huberman Lab
4. Dr. Fei-Fei Li on How AI Learns, What It Can't Do, and Keeping Human Agency at the Center · Huberman Lab
5. Human Language: Psycholinguistics, Syntax, Grammar & LLMs · Lex Fridman Podcast
6. Michael I. Jordan on Machine Learning, Markets, Recommender Systems and the Future of AI · Lex Fridman Podcast
7. Charan Ranganath on How Memory Works: Imagination, Deja Vu and False Memories · Lex Fridman Podcast