Yann LeCun on Self-Supervised Learning, World Models and the Dark Matter of Intelligence

High-Level Overview

Yann LeCun, chief AI scientist at Meta, NYU professor, Turing Award winner and one of the founding figures of deep learning, returns to the Lex Fridman Podcast for a wide-ranging conversation centered on the idea he calls the dark matter of intelligence: the kind of learning humans and animals do by simply observing the world, which today's machines still cannot reproduce. He explains why supervised and reinforcement learning are too inefficient, why self-supervised learning of predictive world models is the central challenge for AI, and why he is newly excited about non-contrastive joint embedding methods such as VICReg. The conversation then broadens to consciousness, what is learned versus hardwired, fear of death, emotions and rights for future machines, the history and role of Facebook AI Research, criticism of Facebook, the flaws of academic peer review, complexity and emergence, his homemade electronic wind instrument, and advice for young people.

The tone is technical yet accessible and often playful. LeCun is confident and opinionated, pushing back on common critiques of machine learning and on popular narratives about social media, while Lex plays devil's advocate on consciousness and mortality. The conversation moves from deep technical explanation to philosophy, institutional critique and personal passions.

The Discussion Explores:

• Why supervised and reinforcement learning are too sample-inefficient, and what self-supervised learning offers
• Filling in the blanks: why predicting missing text works but predicting video remains unsolved
• The three big challenges of machine learning: world models, reasoning compatible with gradient learning, and hierarchical planning
• Contrastive versus non-contrastive joint embedding methods, data augmentation and VICReg
• Grounded intelligence, why text alone is not enough, and the house-cat benchmark
• Consciousness as a consequence of having a single world model, and what is learned versus hardwired
• Emotions and rights for autonomous machines, and the future of human-level AI
• FAIR and Meta AI, defending Facebook, peer review reform, complexity, music and advice for young people

Deep Dive into Main Ideas

The Dark Matter of Intelligence

LeCun begins with a puzzle: most teenagers learn to drive in about 20 hours, while self-driving systems fail even with millions of hours of simulated practice. Supervised learning needs enormous numbers of labeled examples, and reinforcement learning needs a ridiculously large number of trials. Humans succeed because of background knowledge, and the big question is how that knowledge is acquired. Babies in their first months learn mostly by observation, since they can barely act, building world models that may be the basis of common sense. Self-supervised learning is an attempt to reproduce that kind of learning in machines.

He uses his well-known cake analogy to compare how much information each learning paradigm gets. Reinforcement learning receives a single scalar reward occasionally; supervised learning gets a few bits per sample, such as fewer than 10 bits for an ImageNet label. Self-supervised learning, ideally, would show a machine a video clip, stop it, ask it to predict what comes next, then show the actual future, providing vastly more signal. He gives a vivid example of why world models matter: a driver near a cliff knows that turning the wheel will send the car over the edge, while a reinforcement learning system without a world model would have to fall off the cliff thousands of times before learning not to.

Filling in the Blanks and the Problem of Uncertainty

LeCun describes intelligence as largely the ability to fill in the blanks: predicting the future, inferring the past and filling in missing information using a model of what is possible. This approach has been unbelievably successful in natural language processing, where modern systems are pretrained by removing about 10 percent of the words in text and training a giant neural net to predict them. It has made progress in images but has not yet succeeded for learning from video.

The core obstacle is uncertainty. The world is not entirely predictable, so a video has many plausible continuations. For missing words, a system can output a probability for each of roughly 100,000 words in a vocabulary, but for video there is an effectively infinite, continuous space of plausible futures, and we do not know how to represent that distribution. Even in text, current methods treat multiple missing words as independent, when in reality they are not, as in a sentence where a lion or a cheetah chases a zebra or a gazelle. Asked about the critique that this is just statistics, LeCun says he avoids those debates but that "at some level of description intelligence is just statistics," which does not rule out learning deep, causal, mechanistic models; the question is how to learn them.

Three Challenges and Model Predictive Control

LeCun frames three main challenges for machine learning. First, getting machines to learn to represent the world, which he proposes to address with self-supervised learning. Second, getting machines to reason in ways compatible with gradient-based learning, which is what deep learning is about. Third, which he admits he has no idea how to solve, getting machines to learn hierarchical representations of action plans. He explains model predictive control, a method from 1960s optimal control used to compute rocket trajectories: given a differentiable model of how a system evolves, you can backpropagate through time to find the optimal sequence of actions. That is a form of reasoning and planning, and the big challenge of the next decade, he argues, is learning such predictive models of the real world with all its uncertainty rather than building them by hand.

He sketches an architecture for an intelligent agent: a world model, which in humans is roughly the front half of the brain; an intrinsic objective function, driven by the basal ganglia, measuring things like pain and hunger; and a learned critic that predicts future outcomes. If all of these are differentiable, gradient-based methods can be used for planning, reasoning and learning. He notes that humans are terrible at chess and Go because of limited working memory, as AlphaGo revealed, but excellent at learning models of the world, and that current AI does not have as much common sense as a house cat, which manages with about 800 million neurons.

Contrastive and Non-Contrastive Learning

LeCun explains Siamese networks, an idea he worked on in the early 1990s: two identical copies of a network are shown two views of the same thing and trained to produce the same representation. The danger is collapse, where the system ignores its input and outputs a constant. One remedy is contrastive learning, using negative pairs of different images and pushing their representations apart. He and colleagues at Bell Labs originally developed this for signature verification on credit cards, a system that worked but was never deployed. He is not enthusiastic about contrastive methods now, because in high dimensions there are too many ways for things to differ.

What excites him instead are non-contrastive methods, rooted in an early 1990s idea from Geoffrey Hinton and Sue Becker of maximizing mutual information between the outputs of two networks shown only positive pairs. His group revived this with Barlow Twins and then VICReg, which stands for variance, invariance, covariance regularization, and which he calls "the thing I'm the most excited about in machine learning in the last 15 years." He explains that standard data augmentations like cropping, scaling, rotation, color shifts and blur teach invariance useful for object recognition but erase position information needed for localization, and calls augmentation "a temporary necessary evil." He describes promising masking approaches, like hiding patches of images and training transformers to reconstruct them, and "data augmentation by waiting," predicting the representation of a video's future from its past. In the second half of the conversation, he explains that he has changed his mind in the last year and a half: rather than generative latent variable models that predict pixels, he now favors joint embedding architectures that predict abstract representations, dropping unpredictable details, and calls this collection of methods "the best thing since sliced bread."

Grounded Intelligence and the House Cat

LeCun says plainly: "I believe in grounded intelligence." He does not think a machine can become intelligent purely from text, because text contains only a tiny fraction of what we need to know. His example: if you put an object on a table and push the table, it is obvious the object moves too, yet no text explains this, so even a hypothetical "GPT 5000" would never learn it. He considers efforts to write down all common sense facts, like the Cyc project, essentially hopeless. He also argues we give language too much importance as a substrate of intelligence; orangutans and octopuses are highly intelligent while largely solitary, and octopuses never meet their parents.

On what makes human intelligence distinct, he argues it cannot be that complicated, since it emerged in roughly the last million years and involves less than 1 percent of the genome difference from other primates. Most of the hard work already exists in cats, dogs and other primates, which is why basic physics of the world is problem number one. He believes YouTube contains enough raw data to learn to be a cat, though some selection would be needed. On active learning, he agrees that taking actions is necessary to learn causal models, and that curiosity makes learning more efficient, but says the fundamental learning process must be understood first.

Consciousness, Learning and Fear of Death

LeCun admits he does not know what consciousness is, and compares some discussions of it to 17th century puzzlement over why we do not see the world upside down despite the inverted image on the retina. His speculative hypothesis is that we have essentially one world model engine in the prefrontal cortex, configurable to the situation at hand, which is why we can attend to only one deliberate task at a time. Repeated tasks become subconscious, like a chess grandmaster reacting in seconds or an experienced driver chatting. Consciousness may be the executive that configures this single world model, meaning "consciousness is not a consequence of the power of our mind, but of the limitation of our brains."

He believes much of what nativists think is hardwired, such as object permanence or oriented edge detectors in the visual cortex, is actually easy to learn, citing algorithms that learn edge detectors in the equivalent of minutes and experiments rewiring the optic nerve of baby ferrets. What is hardwired are intrinsic drives, like the desire to walk or seek company, while the critic that predicts outcomes is learned, as when you recoil after being pinched once. Lex raises Ernest Becker's The Denial of Death and the idea that fear of death drives human civilization. LeCun argues humans simply have a better long-term planning engine than cats, making awareness of mortality a side effect of intelligence, and says religion may increase worry about death, while believing you simply stop existing resolves it for him. He says building intelligent artifacts, like building airplanes to understand bird flight, is how we will understand intelligence.

Emotions, Rights and Human-Level AI

LeCun argues that "emotions are an integral part of autonomous intelligence." Any autonomous system with intrinsic objectives and a critic that predicts good or bad outcomes will have something like fear and elation, and if it is driven to relate to humans it will have social emotions. He calls the idea of an emotion chip you can switch off ridiculous. On robot rights, he suggests that if minds could be backed up and restored, our ideas about rights, murder and risk would change. He imagines a domestic robot or AR assistant that learns from you for years, raising questions of attachment, intellectual property and privacy, such as whether you could erase what it knows about you if it chose to leave.

He is certain machines will eventually surpass human intelligence in every domain where humans are intelligent, but says it will take a long time and is much harder than many people, including some at DeepMind, once thought. He sees AI as an amplifier for human intelligence rather than a threat.

FAIR, Meta and the Case for Facebook

LeCun recounts founding Facebook AI Research about eight years earlier from a tiny group, running it for three and a half years, and establishing a culture of open fundamental research. He calls it an unqualified success, producing top research and open source tools like PyTorch, and says Meta would crumble without deep learning. FAIR is now part of Meta AI within Reality Labs, split into FAIR Labs and FAIR Accel, and he suggests the F could stand for "fundamental." He sees the metaverse as the next step of the internet, noting augmented reality is easier to grasp than full virtual reality.

He defends Facebook against claims that it is evil or drives polarization, citing independent studies, including one where people paid to stop using Facebook before a sensitive anniversary in the former Yugoslavia became more polarized, and research showing US polarization has risen steadily for 40 years while countries using Facebook just as much are becoming less polarized. You can find a scapegoat, he says, but not a cause. He compares social media to the printing press, which also brought division but which no one calls a bad idea, and praises Mark Zuckerberg and Mike Schroepfer as far more human than their public images.

Peer Review, Complexity and Advice

LeCun describes his VICReg paper being "proudly" rejected from NeurIPS for not being different enough from Barlow Twins, even as others were already citing it. Because the field grows exponentially, most reviewers are junior, and the easy way to review is to find flaws rather than recognize new ideas, which favors incremental benchmark-chasing papers. Years ago, while helping create ICLR, he proposed a system where papers are posted openly and any reviewing entity can publicly evaluate them, with reviewer reputations rising when their evaluations predict future impact.

On complexity, he says we do not understand how simple interacting elements produce complex systems, and we lack a good measure of complexity. His example: randomly permuting the pixels of MNIST digits makes them look complex, but with glasses that undo the permutation they look simple again, so "complexity is in the eye of the beholder." He shares his love of building things, from model airplanes and drones to a custom electronic wind instrument meant to be as expressive as an acoustic one. His advice to young people: be interested in big questions, learn fundamentals with a long shelf life, choosing quantum mechanics over mobile programming, study statistical physics and signal processing, and consider applying AI to science, such as Meta's Open Catalyst project to find materials for efficient hydrogen production, or controlling fusion plasma. And read some literature and history for wisdom.

Conclusion

Yann LeCun's central message is that the key to more capable AI lies not in more labels or more trial and error, but in machines that learn how the world works by observing it, the dark matter of intelligence that lets a teenager learn to drive in 20 hours and a house cat navigate its world with a modest brain. He argues that joint embedding methods like VICReg, which learn predictive abstract representations while discarding unpredictable detail, are the most promising path toward such world models, and that grounded, perceptual learning matters more than language alone. Along the way, he offers provocative views: consciousness may reflect our brains' limits, emotions are inseparable from autonomous intelligence, social media is a scapegoat rather than a cause, and scientific publishing rewards fault-finding over new ideas. For anyone interested in where AI is heading, the episode is a rich look at how one of the field's founders thinks about the road to human-level intelligence.

This is a summary of "#258 – Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.