Dileep George on Brain-Inspired AI: Feedback, Inference, CAPTCHAs and the Limits of GPT-3

High-Level Overview

Lex Fridman talks with Dileep George, a researcher who works where neuroscience meets artificial intelligence. George co-founded Vicarious and, before that, Numenta with Jeff Hawkins. The whole conversation turns on one question: what can the biology of the brain, especially the visual cortex, teach us about building intelligent machines? George explains why simulating neurons in bulk won't produce a working brain, and why the visual system is full of feedback loops rather than being a one-way pipeline. He describes how his Recursive Cortical Network (RCN) used those insights to crack text-based CAPTCHAs with very little training data. The two also cover hype and skepticism around "brain-inspired" AI, how the brain might learn, the link between perception and cognition, GPT-3's limits, episodic memory, Neuralink, consciousness, books, and the meaning of life.

The tone is technical but curious and friendly. George is patient and precise, and he often reaches for concrete examples such as a burglar alarm and an earthquake, a nail pounded into a ceiling, or the Wright brothers watching buzzards. Fridman pushes back in places and plays devil's advocate, especially on large language models. The conversation starts with detailed neuroscience and computer vision, then widens into cognition, language, memory and philosophy.

The Discussion Explores:

• Why you have to understand the brain to build one, and why bottom-up simulation projects like Blue Brain fall short
• The visual cortex as a system full of feedback that projects its model of the world onto incoming signals
• Probabilistic graphical models, inference and "explaining away," and how cortical columns might implement them
• The Recursive Cortical Network, and how it solved CAPTCHAs and MNIST with very little data
• Hype, the press and skepticism around brain-inspired AI, and when to deviate from biology
• Perception as the base for cognition: grounded concepts, simulation, and language as a way to steer simulation
• GPT-3 and the limits of learning from text alone
• Episodic memory, brain-computer interfaces, consciousness, book picks and career advice

Deep Dive into Main Ideas

You Can't Build a Brain Without a Theory of It

Fridman opens by asking whether we need to understand the brain in order to build it. George's answer is a firm yes. He criticizes the approach of Henry Markram's Blue Brain project: pack very detailed biophysical neuron models, based on Hodgkin-Huxley equations of signals moving along a cable, into a huge simulation, wire them up according to measured connection statistics, and see what happens. According to George, such simulations reliably produce one thing, oscillations, which can be tuned to match brain rhythms. But when the system doesn't do what you hoped, you have no theory to debug it with, and all you can do is add more detail.

His analogy is the microprocessor. You could model one transistor almost perfectly, but without understanding Boolean logic and gates, wiring transistors together from guesses would almost never give you a working chip. Fridman brings up the well-known paper that asked whether a neuroscientist could understand a microprocessor. The paper concluded that neuroscience's tools couldn't even recover basic logic. George knows the paper but thinks the conclusion goes too far. Even an electrical engineer would fail using that method, he says, and neuroscientists do find valuable insights. Those insights just have to be assembled into a computational, information-processing framework.

His approach is a loop. Take insights from neuroscience, build a functional model that actually does the task the brain does, let the model fill in the pieces biology leaves out, then go back to biology with new hypotheses about what a given layer or circuit is doing. On timelines, George is humble. He recalls that a New York Times article in 1900 said humans might not fly for another hundred years, and the first flight came in 1903.

The Visual Cortex Is Built on Feedback

George says the visual cortex is huge and is organized into roughly four hierarchical levels for object recognition: V1, V2, V4 and IT. Each level has its own detailed microcircuit of about six layers arranged in columns. The surprise for machine learning people is that the brain has more feedback connections than feed-forward ones. He praises experimental neuroscientists for being highly systematic and hypothesis-driven. Whenever his team asks whether anyone has studied how lateral connections work, he says, someone usually has, and carefully.

He gives two examples. When you show a textured triangle on a textured background, neurons settle on the edges first and then fill in the surface from the edges inward. Researchers can even tell which layer of V1 encodes edges, surfaces, feedback and feed-forward signals. The Kanizsa triangle, the illusion made of three Pac-Man shapes, makes the brain "see" edges that aren't in the input. Neurons do signal those illusory edges, but later than they signal real ones, which points to feedback from context as the cause.

George calls this the beautiful picture of how the brain works: "our brain is building a model of the world," and "We are constantly projecting our expectations onto the world." Perception is not just subtracting the prediction from the input. It is a search for the best explanation of what you're seeing, based on the model of the world I have, in his words. Illusions show that process at work.

Graphical Models, Explaining Away, and Cortical Columns

To explain inference, George contrasts neural networks, which approximate functions through stacked dot products, with probabilistic graphical models. In a graphical model, each node is a variable, such as whether an edge or an object is present, and the connections encode how those variables relate. His classic example is "explaining away." Your home alarm can be set off by a burglar or by an earthquake. When you hear the alarm, you suspect a burglar. When the radio then reports an earthquake nearby, your belief in the burglar drops, because the alarm is now explained by another cause. Competing hypotheses fight to explain the same evidence.

He then describes a hypothesis from an upcoming paper on cortical microcircuits. Each cortical column might represent one concept, basically a binary random variable like "edge present" or "object present." The connections between columns would store the knowledge of how concepts relate. The neurons inside a column, working with the thalamus, would carry out the inference, including explaining away and competition between hypotheses. He notes that neuroscientists have already shown effects like a higher-level neuron inhibiting another column through a loop that runs through the thalamus. These "concepts" don't need to be human-interpretable, he adds. They are useful as entities connected to other entities. Fridman likes the circuit vocabulary and says machine learning people should use it more.

Inside the Recursive Cortical Network

The RCN is a vision system built from these neuroscience insights. Given an image, it classifies it, segments it, and answers questions like where an object's edges and interior are. A key design constraint was that perception shouldn't be treated as just a pre-processing step for cognition. The generative model has to be controllable from the top down, the way you can close your eyes, zoom in on an object, imagine its texture, or picture what happens if something hits it. That model doesn't need to be photorealistic. George points out that people can't recall the color of the letter E in the Google logo despite seeing it countless times, yet they can easily imagine recoloring or stretching it.

The RCN models foreground separately from background, objects as made of parts, and shape separately from texture. George says these splits come from basic properties of natural signals, such as continuity, not from hand-coded rules. Fridman compares them to the priors in François Chollet's ARC challenge. Compared with a convolutional network, the RCN has a similar feed-forward path of feature detectors and pooling, but it is also generative, so it can run backward. It also has lateral connections that keep transformations consistent across the visual field, which prevents jagged results. It is not trained with backpropagation.

Its core strength is dynamic inference. A neural network can get something like reasoning by being shown every combination during training, which George calls amortized inference. The RCN works out explanations at the time it sees the image. It also returns a full explanation of the scene: which pixels made it think a letter was an A, which parts it filled in, where the contours are, and which object is in front of which. George says the errors it makes look a lot like the errors humans make.

Why CAPTCHAs, and What Solving Them Really Means

CAPTCHAs are human interaction proofs, designed to be easy for people and hard for computers. Text CAPTCHAs were common around 2014. Their main trick is crowding letters so that, for example, an r next to an n looks like an m. Locally the evidence points to the wrong letter, and only the global view makes sense. That is exactly the kind of reasoning from local to global that dynamic inference handles. George explains that standard benchmarks have training sets that can be far larger than the test set, and his team wanted to flip that. CAPTCHAs require strong generalization beyond the training data. On MNIST, he recalls reaching about 95% accuracy with training examples on the order of 10 or 100, which was better than other systems at the time.

He insists the CAPTCHA problem is not solved in the way humans solve it. A five-year-old who has just learned letters can read a CAPTCHA style they have never seen. No current deep learning system can do that without training on that style. He cites Douglas Hofstadter: the "central problem in AI is what is the letter A?" A system that can reliably recognize every possible A, whether made of leaves or of two people leaning together, would need to understand nearly everything. Fridman observes that CAPTCHAs appear in papers far less often than they should, and George blames the momentum of established benchmarks and graduate-student incentives.

Hype, Skepticism, and Knowing When to Deviate From Biology

Fridman raises the criticism that Vicarious, like other brain-inspired efforts, attracts, including from the machine learning subreddit. George stands by the RCN paper, which was published in Science. He says its lasting contribution is combining feedback, recursive inference, lateral connections, and joint recognition and segmentation into a working system that fits with higher-level cognition. He then tells a funny story: the paper argued explicitly that its model was a graphical model and not deep learning, yet the journal's press office headlined it as a new deep learning model that solves CAPTCHAs. To George, that shows the bandwagon effect in the field.

He insists, "I don't treat brain inspired as a marketing term." He says he really does dig into the biology. But he also says you have to know when to depart from it. Convolution, for example, is not something the visual cortex does. The cortex has local receptive fields but no shared weights, and a fovea makes the center very different from the periphery. Convolution is an efficient engineering trick that he expects to fade once robots have moving eyes and heads. The goal is to understand the brain's principles, not to be "slavishly mimicking all the things of the brain."

Looking back on his time at Numenta about ten years ago, George says Jeff Hawkins's big-picture ideas in On Intelligence still hold if you blur out the details, but the details differ and they matter a lot. Their main disagreement was about learning algorithms. George didn't want to discard an algorithm based on a biological-plausibility filter that was current then, since new biophysical mechanisms keep being discovered. On how the brain learns, he says some kind of credit assignment is unavoidable, but it needn't be differentiable. Backpropagation and expectation maximization are two versions of it. He is much stricter about biological constraints on the inference side, where experiments are easier to run.

Perception, Concepts, and Language as Simulation Control

Next George describes a Science Robotics paper on "cognitive programs." It combines a perceptual system, a model of world dynamics (schema networks), and a system that learns concepts as programs. The motivating scenario is showing a person who doesn't speak your language a before picture and an after picture, and having them work out the task, like gathering all the red cups. This is pre-verbal, grounded concept learning. George contrasts it with the associations you can pull from text alone, like cows tending to be on grass. He says he disagrees with Chomsky "completely." In his view, grounded sensorimotor concepts are a prerequisite for real language understanding.

His favorite example: Sally pounded a nail on the ceiling. Is the nail horizontal or vertical? Fridman answers vertical by picturing the scene. George's point is that the answer came from simulation, drawing on knowledge built from experience and mostly in place by age five, stored in the visual and motor systems but reachable through language. His summary is that "language is simulation control." He also puts it as a paradox: "I think of perception as kind of the first thing that you have to build, but the last thing that you will be actually solved." Vicarious's next step is connecting language to grounded simulation in simple worlds.

GPT-3 and the Limits of Text

George hasn't tested the 175-billion-parameter GPT-3 directly but calls it intriguing. Its weakness, he says, is that "it is not really carrying a world state around." That's why earlier models produced lines like going up a hill to reach a valley. Using Shannon's Markov chains of English text as a reference, he grants that higher-order models do have a model, but it is a "model of the text world," not of the world itself. When Fridman asks whether compressing huge amounts of text could force real abstractions to form, George says: "No I don't think it is possible because the information is not there." Most of what we know about how the world works, such as that walking forward is easier than walking backward, never gets written down.

He adds that transformers are still feed-forward, while causality, counterfactual reasoning and intervention need other kinds of models. He jokes that we give a system that has read all of quantum mechanics simple text puzzles, when a real AGI would run experiments and revise hypotheses. Fridman admits he is partly playing devil's advocate. He says reinforcement learning, AlphaZero's self-play, and neural networks in general have surprised him, and he wonders whether humans just don't want intelligence to turn out to be "dumb" at its core.

Memory, Neuralink, Consciousness, and the Meaning of Life

On memory, George says we clearly need two kinds: statistical knowledge of the world, and a one-time timeline of personal experience. His hypothesis is that the hippocampus stores a sequence of pointers into the cortex's causal model. To recall an event, the cortex replays it, and a sense of déjà vu in a new situation can bring back similar timelines that then shape what you see next.

He says his interest in brains started with a brain-computer interface talk by Krishna Shenoy at Stanford. He calls Neuralink's intermediate goals reasonable and valuable, such as controlling prosthetics and helping people with paralysis, and he also mentions Paradromics. He thinks the most promising route is to let the brain adapt to an electrode array in one direction, rather than trying to decode the whole brain. Fridman adds that "if you want to learn something do the most difficult version of it." On consciousness, George explains the name Vicarious: after learning a model of the world, you can run it in your head vicariously, and you can model other people. Turning that same modeling apparatus on yourself, he suggests, is what gives rise to consciousness. He isn't worried about AI being motivated to kill humans, and he notes that an AI system can be copied in a way a person can't.

For books, he recommends Judea Pearl's Probabilistic Reasoning in Intelligent Systems for its philosophical asides. He calls Pearl's Causality too hard to recommend and The Book of Why more enjoyable. He also recommends Hofstadter and Dennett's The Mind's I and The Bishop's Boys, a history of the Wright brothers. Here he makes a sharp point: people say "airplanes don't flap wings" to argue AI needn't learn from biology, but the Wright brothers learned that flapping wasn't the key by watching buzzards soar on thermals, which showed them that control was the real problem. His career advice, which he says should be "taken with a pint of salt," is that anyone who wants to understand the brain in order to build one may do better in computer science, electrical engineering or machine learning, studying neuroscience on the side. He adds that a panel at a Redwood Neuroscience Institute workshop once favored electrical engineering. On the meaning of life, he calls it completely open. What matters is understanding the machinery of the world well enough to pursue whatever goals you choose.

Conclusion

Dileep George makes a steady case that biology should be a serious input to AI without becoming a straitjacket. The brain's heavy use of feedback, its drive to explain what it sees, its separation of shape, texture, foreground and background, and its grounding of concepts in simulation all point toward models that do real inference rather than just recognizing patterns. His Recursive Cortical Network, with its data-efficient CAPTCHA and MNIST results, is his proof of concept. His criticism of GPT-3, that it models text rather than the world, follows from the same philosophy. Fridman's pushback on scale and surprise keeps the discussion honest, and the Wright brothers anecdote captures the lesson: study nature carefully enough to learn which parts actually matter.

This is a summary of "#115 – Dileep George: Brain-Inspired AI" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.