Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI

High-Level Overview

In his third appearance on the podcast, Yann LeCun, Meta's Chief AI Scientist, NYU professor and Turing Award winner, lays out why he believes today's autoregressive large language models, for all their usefulness, will not get us to human-level intelligence. He explains what he thinks is missing (an understanding of the physical world, persistent memory, reasoning and planning) and describes the alternative he is betting on: joint embedding predictive architectures (JEPA) that learn abstract world models from video, paired with objective-driven systems that plan their answers through optimization. The second half turns to open source, bias and the Gemini controversy, his dismissal of AI "doomer" scenarios, the future of humanoid robots, advice for PhD students, and his case that AI will make humanity smarter, much as the printing press did.

The tone is technical, spirited and often playful. Lex Fridman repeatedly pushes back, especially on whether language alone might contain enough knowledge to build a world model, and LeCun answers with patient, detailed arguments and a few of his trademark sardonic jabs. The conversation moves from deep technical architecture to research history, then to politics, safety and society, and finishes on optimism about humanity.

The Discussion Explores:

• Why LLMs fall short: the four essential capacities of intelligence and the data gap between text and a child's sensory experience.
• World models and the limits of generative prediction: why predicting pixels in video failed for a decade.
• JEPA and joint embedding: learning abstract representations by predicting in representation space instead of reconstructing inputs.
• Planning, hierarchy and objective-driven AI: model predictive control, energy-based models and thinking before speaking.
• Hallucinations and the long tail: why errors compound and why fine-tuning cannot cover every prompt.
• Open source as the answer to bias: why no AI can be unbiased and why AI assistants must be as diverse as a free press.
• Pushing back on AI doom: gradual progress, guardrails, the jet-engine analogy and "good AI" against bad AI.
• Robots, research directions and hope: humanoid robots, open problems for students, and AI as a new printing press.

Deep Dive into Main Ideas

What LLMs Can't Do, and the Data They Never See

LeCun names four essential characteristics of intelligent behavior in humans and animals: understanding the physical world, persistent memory, the ability to reason, and the ability to plan. LLMs, he says, can do none of these or only in a very primitive way. That doesn't make them useless; a whole ecosystem of applications can be built on them. But as a path to human-level intelligence, they are missing essential components.

He backs this with a striking comparison. LLMs train on roughly 10 to the 13 tokens, about 2 times 10 to the 13 bytes, which would take a person 170,000 years to read at eight hours a day. Yet a four-year-old child has been awake about 16,000 hours, and with the optic nerve carrying about 20 megabytes per second, roughly 10 to the 15 bytes have reached the child's visual cortex. His conclusion is that most of what we learn comes from observing and interacting with the world, not from language. Fridman pushes back that language is highly compressed and full of wisdom. LeCun agrees it is compressed, but argues that makes it less redundant, and self-supervised learning captures more structure from redundant data like vision. He adds that language is only an approximate representation of our percepts and mental models, and much of our thinking, like imagining a bottle rotated 90 degrees or planning how to build something, has nothing to do with language.

He invokes the Moravec paradox: things that seem hard for humans, like chess or passing the bar exam, are easy for computers, while things we take for granted are hard. LLMs can pass the bar exam, "But then they can't learn to drive in 20 hours like any 17 year old," or clear a dinner table and load a dishwasher, which any 10-year-old learns in one shot.

Why Predicting Pixels Doesn't Work

An LLM works by predicting the next token: it outputs a probability distribution over a finite vocabulary, samples a token, feeds it back in and repeats, which is why it's called autoregressive. Humans, LeCun argues, plan what they are going to say at an abstract level that is largely independent of language; he says his thinking is pretty much the same in French as in English, although he makes better puns in French. LLMs don't plan their answers; they produce one word after another, a bit like answering a question automatically while distracted.

The same trick doesn't transfer to video. Predicting a distribution over all possible next frames means representing distributions over high-dimensional continuous spaces, which nobody knows how to do usefully. If a camera pans around a room, no system could predict the texture of the carpet or the painting on the wall. LeCun says he and colleagues at FAIR tried for about ten years with latent variable models, GANs, VAEs and regularized autoencoders, including masked autoencoders that reconstruct images with patches removed. Training a system to reconstruct corrupted images or video, he says, "has been essentially a complete failure" at producing good general representations. The proof is that the same architecture trained with labeled data gives much better features.

JEPA: Predicting in Representation Space

The alternative is joint embedding. Both the full input and a corrupted or transformed version pass through encoders, and a predictor learns to predict the representation of the full input from the representation of the corrupted one. LeCun calls this a joint embedding predictive architecture, or JEPA. The trick is avoiding collapse, where the system ignores its input and outputs constant representations. Contrastive methods, which LeCun worked on as early as a 1993 paper, push apart the representations of different images; newer non-contrastive methods such as distillation-based techniques (BYOL from DeepMind, DINO and I-JEPA from FAIR) and VICReg avoid collapse without needing negative examples.

The key advantage is that a JEPA doesn't try to predict every pixel. It extracts as much information as possible while keeping only what is predictable, discarding things like the chaotic motion of leaves on a windy day. This lifts the level of abstraction, much as science describes the world at different levels rather than always in terms of quantum field theory. With language, we could skip this step because text is already abstract. V-JEPA, which masks a "tube" across 16 frames of video, is the first system LeCun's team has built that learns good video representations, good enough to recognize actions with a simple classifier, and early results suggest it can tell whether a video is physically possible or impossible, for example when an object disappears or jumps.

Could this be combined with language? Eventually yes, but LeCun warns that doing it too early is "cheating," using language as a crutch to cover weaknesses in vision systems. Cats and dogs have no language yet understand the world far better than any LLM, and he wants machines to learn that kind of understanding first.

World Models, Planning and Hierarchy

A world model, in LeCun's framing, takes the state of the world at time t and a possible action and predicts the state at t plus 1. Feed a JEPA an action, such as a steering wheel turned 10 degrees, and it can predict at an abstract level what a dash camera will see next. With such a model, a system can plan: imagine a sequence of actions, predict the outcome and choose actions that best satisfy an objective. This is model predictive control, the classical method used to plan rocket trajectories since the early 1960s, and it is something LLMs cannot do.

Complex actions need hierarchical planning. LeCun's favorite example is traveling from his NYU office to Paris: the goal breaks into getting to the airport and catching a plane, then into hailing a taxi, taking the elevator, standing up from the chair, all the way down to millisecond muscle control, which would be impossible to plan in full in advance. "Nobody really knows how to do this in AI," he says, meaning nobody knows how to have a system learn the right levels of representation on its own. Fridman asks whether an LLM could simply break the trip into steps. LeCun agrees it could, down to the level things can be described in words and only if it was trained on similar scenarios, but not how to stand up from a chair. Most human plans are learned from others, he notes, so LLMs will handle high-level plans while JEPA-like systems handle the link to low-level action.

Why LLMs Work So Well, and Where They Fool Us

Fridman asks why, if LeCun is right, LLMs are so impressive. LeCun credits self-supervised learning, which he has championed for years: corrupt a text, train a giant network to fill in what's missing, and you get systems that translate hundreds of languages, summarize and answer questions. The surprise, around GPT-2, was that decoder-only models kept understanding more as they scaled. Meta's multilingual content moderation, the wav2vec speech system that needs only minutes of labeled data, and real-time speech-to-speech translation even for languages without a written form all grew from this. LeCun recounts reviving unsupervised learning with Yoshua Bengio and Geoffrey Hinton in the early 2000s and pushing self-supervised learning again when FAIR started in 2014.

But on the Turing test, he says, "we're fooled by their fluency." We assume fluent language means human intelligence, and that impression is false. Alan Turing would decide the Turing test is a really bad test, which the AI community concluded long ago, and Hans Moravec would say his paradox still applies. His bottom line from decades of this research: "If you're really interested in human-level AI, abandon the idea of generative AI."

Hallucinations and the Long Tail

Because LLMs generate token by token, LeCun explains, each token carries some chance of leaving the set of reasonable answers. If those errors are independent, a strong assumption, the probability of staying correct shrinks exponentially with the length of the answer. Fine-tuning can cover perhaps 80% of the questions people commonly ask, but the space of possible prompts is gigantic and has a very long tail. Random character strings or swapping a few words into another language can derail a model, which he says means the system is essentially a giant lookup table.

He also notes that LLMs spend the same computation per token whether a question is trivial or impossible, unlike humans who spend more time on harder problems. That is the difference between System 1, fast and automatic like an expert driver or chess player, and System 2, deliberate planning with an internal world model, which LLMs cannot yet do.

Objective-Driven AI and Energy-Based Models

LeCun's blueprint for future dialogue systems is to have them "think about their answer, plan their answer by optimization before turning it into text." Imagine a network with a single scalar output that is zero when an answer fits a prompt and large when it doesn't, an energy-based model. The system would search for an answer that minimizes that energy, not over strings of text but in an abstract representation space, using gradient descent, then pass the result to a simple decoder that turns it into words in any language. Current approaches that generate many candidates and pick the best are, he says, horribly inefficient by comparison.

Training such a model requires making sure the energy is low for compatible pairs and high elsewhere. Contrastive methods push up the energy of bad answers, which is what reward models in RLHF effectively do, but they need huge numbers of negative samples. LeCun prefers regularized methods that limit how much of the space can have low energy. Fridman sums up his recommendations: abandon generative models for joint embedding, autoregressive generation, probabilistic models for energy-based models, contrastive methods for regularized ones, and minimize reinforcement learning. LeCun doesn't hate RL, but says it is incredibly sample-inefficient; it should be used only to adjust a world model or objective when they prove inaccurate, as in curiosity and play. RLHF's real power, he says, comes from the human feedback, not the RL.

Open Source Is the Answer to Bias

Asked about criticism of Google's Gemini 1.5 for rewriting history in image generation and refusing to depict Tiananmen Square, LeCun argues that an unbiased AI system is impossible, not mainly for technical reasons but because bias is in the eye of the beholder. The answer is what liberal democracies found for the press: it must be free and diverse. Soon every interaction with the digital world will be mediated by AI assistants, through things like Meta's Ray-Ban smart glasses, which he says can answer questions about a monument you are looking at or translate a menu. Those systems will become the repository of all human knowledge, and he argues they cannot come from a handful of companies on the US West Coast.

Open-source base models let anyone fine-tune their own. He cites the French government's refusal to let three companies control its citizens' digital diet, an Infosys founder funding a Llama 2 fine-tune for all 22 official languages of India, and former FAIR scientist Moustapha Cissé's startup building models that speak local languages in Senegal to give people access to medical information. Meta can afford this, he says, because it already has a huge user base and earns money through ads and business customers, like a pizza shop taking orders through WhatsApp; releasing the base model doesn't hurt, and Llama 2 has already been downloaded millions of times with thousands of people contributing ideas. Big companies over-sanitize, he says, not because of their engineers' politics but because they cannot afford to offend their customers. As he and Fridman agree, "open source enables diversity," and basic guardrails that everyone agrees on can be built into objective-driven systems, with communities adding their own. On bioweapons, he says studies increasingly suggest LLMs give no real advantage over a search engine, and the hard part is actually building such things, just as detailed rocket engine instructions won't stop a team from blowing up a dozen engines first.

Against AI Doom

LeCun expects Llama to keep improving and become multimodal, with future systems trained from video that understand the world and can plan, and he says he hasn't been this excited about AI's direction since FAIR started, or since his early work on convolutional nets. Still, compute matters: "scale is necessary but not sufficient," and a GPU draws half a kilowatt to a kilowatt versus about 25 watts for the human brain, so new hardware principles may be needed. He believes human-level AI is at least a decade away and probably much more, and "it's not going to be an event," but gradual progress. He argues intelligence is a multidimensional set of skills, not a single number like IQ; orangutans are smarter than humans in some domains.

Doomer scenarios, he says, rest on false assumptions. First, super-intelligence won't be switched on one day; we'll build systems as smart as a cat or parrot and gradually add intelligence along with guardrails. Second, intelligence doesn't imply a desire to dominate. That desire is hardwired in social species like humans, baboons, chimpanzees and wolves, but not in the nearly as smart orangutans, and nothing requires building it into AI. Objective-driven systems can include guardrails like obeying humans unless it would hurt other humans. He compares AI safety to turbojet engines, which became astonishingly reliable through decades of iterative design, with no separate safety department, because a better engine is a more reliable one. If a rogue state builds a persuasion AI, it will have to get past your own AI assistant, which will filter it like spam: "my good AI against your bad AI." Unlike nuclear weapons, AI breakthroughs spread quickly; AlphaGo was reproduced within three months. He points to the Pessimist Archive, full of old newspaper panic over trains, electricity, jazz and comic books, and says the real danger is concentrating AI in proprietary hands, not open access.

Robots, Research Advice and a New Printing Press

LeCun thinks the next decade will be very interesting for robotics, but real progress depends on world models. Boston Dynamics relies on handcrafted dynamical models, and even FAIR's embodied AI demos, where a robot dog fetches a can from the fridge, fall far short of clearing a dinner table. For students, he sees plenty of room for innovative work without massive compute: training world models from observation, planning with learned world models including actions like using tools, search engines or calculators (mentioning FAIR's Toolformer), and hierarchical planning, where he says there has been zero demonstration of learned multi-level action representations.

What gives him hope is that AI will amplify human intelligence. Everyone will manage a staff of smart AI assistants, some smarter than they are, which he says should feel no more threatening than managing talented people. He compares it to the printing press, which enabled the Enlightenment, science and democracy, despite also fueling 200 years of religious conflict in Europe. The Ottoman Empire banned printing in Arabic for about 200 years, and he says the UAE's AI minister, Omar Al Olama, told him it was partly to protect the powerful guild of calligraphers, which raises the question of whose jobs AI regulation is really protecting. Economists tell him AI will shift professions rather than cause mass unemployment. He believes people are fundamentally good, and many doomers are doomers because they don't trust people or institutions.

Conclusion

LeCun's message is both a critique and a roadmap. LLMs are impressive products of self-supervised learning, but they fool us with fluency while lacking a grasp of the physical world, memory, reasoning and planning. His proposed path runs through JEPA-style models that learn abstract world models from video, energy-based objectives that let systems plan answers before speaking, and hierarchical planning that no one has yet cracked. Around that technical vision he builds a social one: AI assistants will mediate everything we see, so they must be open source and as diverse as a free press, and progress will be gradual and correctable rather than a sudden catastrophe. The result is a spirited, deeply informed case for optimism about AI as a tool that makes humanity smarter.

This is a summary of "Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.