Oriol Vinyals on Gato, Flamingo, Meta-Learning and the Road to AGI

High-Level Overview

Oriol Vinyals, research director and deep learning lead at DeepMind, returns to the Lex Fridman Podcast for a deep, technical conversation about where large AI models are heading. The centerpiece is Gato, DeepMind's general agent that uses a single neural network to handle text, images, games and robotics, along with Flamingo, a vision-and-language model built on top of the Chinchilla language model. From there they discuss how data from different modalities is turned into tokens, why models are still trained from scratch, how prompting is changing meta-learning, why transformers and attention have proven so durable, how abilities seem to emerge suddenly at scale, the Google engineer who claimed a language model was sentient, and whether AGI will arrive in Vinyals' lifetime.

The tone is curious, careful and good-humored. Vinyals is precise about what current systems can and cannot do, repeatedly emphasizing that Gato is a beginning rather than an end, while Lex pushes toward philosophical questions about agency, sentience and the future of humanity. The conversation moves from AI interviewers to concrete architecture details to reflections on teams, engineering, benchmarks and consciousness.

The Discussion Explores:

• Whether an AI could replace Lex as an interviewer, and what current models lack
• Why models learn offline, have short working memory, and are trained from scratch
• How Gato turns text, images, actions and robotics data into one sequence of tokens
• Flamingo and the case for modularity by freezing and extending existing models
• How prompting has redefined meta-learning, and what interactive teaching could look like
• Transformers, attention and the surprising emergence of abilities at scale
• The human side of breakthroughs: teams, engineering, benchmarks and constraints
• Sentience claims, consciousness, and hopes for human-level AI

Deep Dive into Main Ideas

Could an AI Replace the Interviewer?

Lex opens by asking whether an AI could one day replace him as an interviewer. Vinyals questions whether we would even want that, noting that with StarCraft agents, what people really cared about was how they behaved against humans. He expects AI to empower interviewers, for example by suggesting questions that a human then cherry-picks, and allows that self-play interviews could be interesting or educational. Lex suggests that excitement could become part of the objective function, and that a compelling conversational AI might need a strong identity, a backstory, even flaws and stakes. Vinyals agrees that rewards beyond simple win-or-lose, such as excitement or truthfulness, are an exciting research direction, though the labels would have to come from humans.

He identifies two big gaps. First, models learn offline as passive observers of data and do not keep updating their weights when deployed; their only dynamic memory is a working context of about 2,000 words, after which they forget. Second, every new model is trained from scratch with random weights, which feels wasteful compared with how species and knowledge build on previous iterations.

The Deep Learning Recipe and the Rise of Prompting

At the core of deep learning, Vinyals explains, is the beautiful idea that a single algorithm can solve any task: take a blank network, feed it examples, and train. The field is searching for a universal recipe that needs fewer task-specific tricks, though high-value problems like protein folding still justify specialized work, as with AlphaFold.

The big shift of recent years, starting with GPT-3, is that a single trained model can be taught new tasks through prompting, by showing a few examples in natural language, without retraining. This has since expanded beyond language to vision and actions. The open challenge is how to grow these models' capacity and whether they should change their weights as they are taught.

How Gato Works

Vinyals explains that DeepMind named its sequence models after animals: Gopher and Chinchilla for language, Flamingo for vision and language, and Gato, the Spanish word for cat, echoing the letters "G" and "A" of "general agent. His tweet about it read, in part, Gato's not the end. Gato's the beginning." He also confirms, with a laugh, that meow is universal.

Gato is a transformer trained to predict the next token in any sequence, whether words, images, Atari moves or robot sensor readings, so it can chat, caption and act. What makes it an agent is its ability to take actions in an environment and respond to new observations. It was trained as a massive imitation-learning system on web-scale text plus trajectories from many DeepMind reinforcement learning agents playing 3D games, Atari and robotics tasks, all mixed together in one brain. At about one billion parameters, it is small compared with models of 100 billion or more, and it is not yet as proficient as the specialist agents it learned from. Vinyals believes scale, and better data preparation such as telling the model in words that it is about to play an Atari game, may unlock synergies between modalities.

Tokenization: Turning Everything into Puzzle Pieces

Tokenization, he explains, turns all data into little puzzle pieces laid out in a sequence. Text is split into common substrings, typically two to five tokens per English word. Images are compressed into patches, roughly 16 by 16 pixels, and quantized into tokens based on data statistics, somewhat like JPEG. Actions, such as Atari's discrete moves or robot torques, are also discretized.

Each modality occupies its own range of integers, so they start out completely separate, and what connects them is the data, such as images paired with captions. Each token maps to a learned vector, and as training proceeds, the vector for the word "cat and patches showing a cat may start to align. Lex asks whether the network is schizophrenic or of one mind; Vinyals says it is one mind, trained with essentially the most boring algorithm" of gradient descent, with shared weights across all tasks. He suggests a future direction could describe actions in words, so models could leverage language more heavily.

Flamingo and the Case for Modularity

Retraining a larger version of Gato is straightforward, but reusing its learned weights to build a bigger brain is extraordinarily hard. Flamingo shows another path: DeepMind froze the 70-billion-parameter Chinchilla language model and inserted about 10 billion new parameters, some pre-trained to understand vision, then trained the system on image-and-text data. The result, about 80 billion parameters, is essentially a chatbot you can talk to about images, with few-shot abilities that emerged beyond what the training data directly taught. Vinyals mentions it could engage with a famous joke photo involving Barack Obama and a scale, which Andrej Karpathy once said no vision system could understand, while stressing this is anecdotal.

He sees modularity as akin to software engineering, where we reuse libraries instead of rebuilding them, and says the big question for the community is whether to train from scratch or embrace modularity. Lex imagines plugging in hundreds of networks, from weather models to game players, without instabilities growing.

Meta-Learning Redefined

In 2019, meta-learning mostly meant learning new object categories from a few examples, a benchmark largely tied to vision. GPT-3's paper, Language Models are Few-Shot Learners, which Vinyals praises for its great title, showed that tasks could be defined on the fly in natural language. Flamingo extended this to vision, even learning arithmetic from pictures of numbers after a few examples.

Looking ahead five to ten years, Vinyals hopes for a single set of weights that can be taught through interaction to play a complex game like StarCraft, perhaps by showing examples, uploading its Wikipedia page, and letting it ask for feedback after it plays. He compares prompting to a sophisticated version of nearest-neighbor classification, where knowledge is uploaded rather than trained in, and frames a possible path to AGI as going from 100 percent performance on individual games to perhaps 20 percent across nearly all tasks. Whether a model that has only seen land images but read about water could handle the ocean remains an open research question.

Teams, Engineering, Benchmarks and Attention

Asked what surprised him most, Vinyals points to the people. Behind every breakthrough are teams, and critical ingredients include data engineering, deploying models on large compute clusters, and benchmarks that let researchers measure progress over months or years of uncertain work. AlphaFold, he says, is the clearest example: the metrics existed, and a team aimed very high and iterated for years, partly during a pandemic.

On architecture, he says "attention might indeed be all you need," noting the transformer has changed remarkably little in five years. Unlike LSTMs, it has a built-in inductive bias for attention: when deciding whether the next word is "cat" or "dog," it queries the past by content, not just position, and pulls out what is relevant. Its main limitation is context length; teaching StarCraft would require videos, articles and interaction far beyond a thousand-word prompt. A friend once told him some ideas want to work, and the transformer was clearly one.

Vinyals says individuals shape the timing and order of discoveries, and that diversity of research styles matters. Industry labs have "the particle accelerator" of compute and should answer questions that need it, but attention itself was discovered in Montreal partly because of limited compute, when his own group at Google Brain was using about eight GPUs. Lex adds his belief that much genius lies in engineering details, and Vinyals notes the deep learning revolution rode on GPUs built for video games, plus data centers, better software and carefully built benchmarks like Atari and ImageNet.

Emergence and the Science of Scale

In their survey paper on emergent abilities of large language models, Vinyals and colleagues describe tasks where performance stays random until a model reaches a certain scale, then jumps. Simple one-step tasks like image classification show smooth curves, but tasks requiring more reasoning seem to need several "bits of thought" to fall into place before any progress appears. The downside is that some architectural questions can only be answered at large scale. The upside is a growing science of scale, such as the Chinchilla scaling laws, derived from models of hundreds of millions to a billion parameters, which helped save compute. He calls this empirical theory, which he says is way ahead of actual theory of machine learning.

Sentience, Consciousness and the Future

On the Google engineer who claimed a language model was sentient, Vinyals says he has never thought any current model, even AlphaStar, is sentient. Working near AlphaFold gave him a sense of how vastly more complex biology is than trained weights, and he notes that with the right software, a model could be created with about ten lines of code and a dump of the internet. Still, he says no one should tell others how to feel, and experts should help demystify the math so people use the technology safely. He does not think consciousness is necessary for intelligence, though lessons from biology and evolution can inspire algorithms. Lex predicts a coming "civil rights movement for robots" and possible laws on when systems may display sentience; Vinyals credits science fiction for preparing society and welcomes the growth of safety and ethics research.

He praises his friend Ilya Sutskever, recalling a famous talk from the sequence-to-sequence paper, rehearsed in a hotel room, whose unfiltered version was even more controversial. Discussing Rich Sutton's "bitter lesson, he agrees general methods and scale are essential, pointing to AlphaCode, but is less sure about search outside domains with clear rewards. He is convinced human-level AI will arrive in his lifetime through imitation, while going far beyond will require reinforcement learning and well-defined rewards. He worries about resource limits and hopes AI broadens access to knowledge. On a multi-planetary future, he says if the ratio is not at most one to one I would not be happy." Lex closes with Alan Turing: "those who can imagine anything can create the impossible."

Conclusion

Oriol Vinyals offers an insider's map of where deep learning stood in 2022: a single simple recipe, the transformer trained on huge and diverse data, can now handle language, vision and action in one model, and prompting lets trained models learn new tasks without retraining. Gato and Flamingo are, in his words, beginnings, with key open problems around memory, growing and reusing models, longer context and interactive teaching. He credits teams, engineering and benchmarks as much as big ideas, doubts current systems are sentient, and remains confident that human-level AI is coming, ideally as a tool that empowers people rather than replaces them.

This is a summary of "#306 – Oriol Vinyals: Deep Learning and Artificial General Intelligence" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.