Mark Zuckerberg: First Interview in the Metaverse

High-Level Overview

This episode is Lex Fridman's first interview conducted inside the metaverse. Lex in Austin and Mark Zuckerberg in a conference room hundreds of miles away appear to each other as photorealistic Codec Avatars, 3D models built from detailed face scans, with spatial audio, in a shared dark room that mirrors Lex's usual set. The conversation begins with Lex's near-speechless reaction to how real it feels, then moves through how Codec Avatars work and when ordinary people might get them, the Quest 3 headset and mixed reality, Zuckerberg's view that the "real world" is a blend of physical and digital, questions of identity and grief, Meta's new AI personas and creator AIs, the open-sourcing of Llama 2, and what all of this might mean for human connection.

The tone is wonder-struck and warm on Lex's side (he says it's almost getting emotional and that his heart is beating fast) and measured, product-focused and occasionally philosophical on Zuckerberg's. The conversation flows from a live demonstration of the technology, to its engineering challenges, to broader questions about reality, ethics, AI and the future of how people connect.

The Discussion Explores:

• How Codec Avatars capture facial expressions and why they're more bandwidth-efficient than video
• The path from hours-long scans to a two-to-three-minute phone scan, and tuning how expressive avatars should be
• Quest 3 as the first mainstream mixed reality headset, and the engineering behind passthrough and hand tracking
• Zuckerberg's idea that the real world is the combination of the physical and the digital, with holograms everywhere
• Identity, idealized avatars, and whether we should talk to digital versions of loved ones who have died
• Meta's AI personas, from the neutral Meta AI to Snoop Dogg as a dungeon master, and future creator AIs
• Why Meta open-sourced Llama 2, and plans for future models
• What is and isn't acceptable in virtual worlds, anonymity, and the promise for remote work

Deep Dive into Main Ideas

What Codec Avatars Are and How They Work

Lex opens by marveling that it doesn't even feel awkward to be close to Mark, who jokes that he moved Lex back a few feet before the session began. Zuckerberg explains that both of them were scanned for a Meta research project called Codec Avatars. Instead of cartoony avatars or streamed video, the system builds a computer model of each person's face and body across many expressions and compresses it into a codec. The headset watches your face and sends an encoded version of what you should look like over the wire, which is photorealistic and also far more bandwidth-efficient than sending full video, let alone immersive 3D video of a whole scene.

Lex notes that the realism comes from imperfections: slight asymmetry, the corners of the eyes, what eyes do when you smile. Zuckerberg agrees that eyes are a huge part of it, pointing out that much of communication is expression rather than the words spoken. That, he says, is the core vision of virtual and augmented reality: delivering a sense of presence as if you were together, no matter where you are in the world.

Making Avatars Accessible and Expressive

Lex went to Pittsburgh for a lengthy scanning session involving extensive hardware and software. Zuckerberg says the team over-collects expressions today because they haven't yet worked out how far the process can be reduced. The goal, with a project already underway, is a quick scan in which you wave your phone in front of your face for a couple of minutes, say a few sentences, make some expressions, and get something of today's quality in two to three minutes. The headsets needed to drive the avatars already exist; efficient scan production is one of the last pieces to nail. He expects a progressive rollout over the next few years, starting with productivity and meetings, where Workrooms users have said cartoon avatars feel too unrealistic.

He raises a subtle design question: different people emote to different extents, so how wide should your avatar's smile be? Joking that he gets critiqued for a relatively stiff expression, he says he might prefer an avatar that expresses how he feels better than his face physically does. Lex quips that the meme will be the two most monotone people meeting in the metaverse, and says his own emotion is subtle and mostly in the eyes. Zuckerberg also explains that full-body codecs are computationally intensive, so a likely approach is a lower-fidelity body stitched to a high-resolution face, since moving an eyebrow a millimeter changes an expression more than moving an arm an inch.

Quest 3 and Mixed Reality

Zuckerberg calls Quest 3 the first mainstream mixed reality device. Quest Pro cost $1,500; Quest 3 is $500, and in some ways its mixed reality is better. The passthrough cameras and sensors have multiple times the resolution, and a custom version of Qualcomm's next-generation VR and MR chipset means the cheaper headset has a stronger chip than the pro line at a third of the cost. It also has a better display and is 40% thinner. Unlike smartphones, where yearly upgrades are barely distinguishable, he says each new headset brings a major new capability, and this time it's mixed reality.

He describes practical benefits: in fitness or shooter games, people worry about hitting a lamp, and seeing the real room makes the experience feel safer. Lex describes zombies coming through walls, ducking behind a real couch, and adding virtual windows to a windowless room. Zuckerberg explains the engineering: multiple cameras capture the room in 3D, and a complex compute and AI problem remaps it to your eyes' perspective in real time before overlaying digital objects. Hand tracking has progressed faster than he expected, allowing demos like learning piano on a tabletop without a controller. He emphasizes that getting a device into millions of hands is what brings developers to build for it.

The Real World Is Physical Plus Digital

Zuckerberg predicts glasses that look normal but show holograms, and suggests that perhaps by the end of the decade there may be as many holograms in a room as physical objects. He pushes back on the habit of calling the physical world the real world: "the real world is the combination of the physical world and the digital world's coming together." Today the digital world is accessed through screens, small ones you carry or big ones you strap in to for long sessions, and the two are disconnected. Screens, media, books, art, and games like cards or ping pong can often be better as holograms that appear and disappear with a snap of the fingers.

The most important element, he says, is people: meetings and hangouts where some attendees are physically present and others appear as holograms, and even embodied AIs, such as an AI engineer on your team joining the meeting as a realistic avatar. He suspects photorealistic avatars may be a bridge that helps people take these experiences seriously, but people could eventually prefer expressive, less literal self-representations. Internal experiments show people are comfortable mixing photorealistic people with cartoony backgrounds or cartoon avatars. Lex says that for sitting on a beach with a friend and a beer while apart, he wants maximum realism.

Identity, Idealized Selves and Grief

Zuckerberg notes that his avatar's hair is shorter than his current hair and that an avatar could appear clean-shaven even if he were several days unshaven, so avatars reflect you at the time of capture rather than the present moment. Lex asks whether people will try to look their best for scans, and whether we are the entity driving the avatar or the avatar itself. Zuckerberg replies that physical bodies also fluctuate, so the question of which version is really you already exists, and says psychologists and social scientists will have a lot to study.

Lex asks the harder question: could we someday talk to loved ones who have died? Zuckerberg says there may be ways reliving memories could help someone grieving, but also ways it could become unhealthy, and admits he isn't an expert and it would need study. He points to existing practice on Meta's platforms, where people who use the services die every day: accounts can be memorialized so families keep access, but private material such as messages isn't handed over.

AI Personas and Creator AIs

Zuckerberg says Meta doesn't want a world with a single AI superintelligence but one in which many people are empowered by AI tools, and just as you don't work with one person on everything, you'll want a variety of AIs. These include the plain, neutral Meta AI assistant usable in any chat thread; business AIs that let you resolve product issues, make reservations or buy things through natural conversation rather than waiting on hold; and fun AIs that tell jokes or liven up group chats around fashion, trivia or video games. Internal testers' favorite was a text-based adventure dungeon master played by Snoop Dogg, which he calls hilarious, and Meta also built personas of Jane Austen and Marcus Aurelius.

For creators like Lex, who have more demand from their communities than hours in the day, he envisions an AI version of yourself that people know is an AI but that is the next best thing. That is harder than inventing new characters, because the AI must not say things the creator wouldn't want and must know and represent things as the creator would. Since these are trained models rather than programmed ones, the challenge is making them predictable enough to stay within set bounds. He expects creator AIs sometime next year, with new characters, games and small-business experiments rolling out right after Connect, and an AI Studio platform that eventually lets anyone create AIs like any other user-generated content. Lex jokes about sitting down with an AI of himself to tell it to stop the dad jokes.

Open-Sourcing Llama 2

Recalling their earlier conversation about whether to open-source Llama 2, Zuckerberg says he's glad Meta did, and that he now believes the value of open-sourcing a foundation model like it significantly outweighs the risks, after rigorous assessment and red-teaming. Downloads and usage exceeded even his optimistic expectations. Llama 2 itself is infrastructure rather than a consumer product, so the current priority is fine-tuning it and its branches into consumer products he hopes hundreds of millions, and one day billions, will use.

On a future model, he says there is always another model in training, but he has no news or timeline. He expects a similar debate and red-teaming process and hopes to open-source the next version too, though it's not close. Lex praises the transparency and argues that putting AI personas in people's hands lets society learn together from failure cases, weirdness and fun on the path toward AGI. Zuckerberg agrees that launching a diverse set of use cases is partly about discovering what people actually want.

Ethics, Anonymity and Remote Presence

Asked where to draw the line between video-game freedom and real-world rules, Zuckerberg says society should be as free as possible except where you damage others or infringe on their rights, and that harm differs in digital spaces: shooting friends in a game lobby hurts no one. The key questions are which physical harms we're freed from, allowing fewer restrictions, and which new harms appear. Anonymity is a major one: at a restaurant you pay because you have one identity and life is a repeat game, but anonymity online weakens the incentive to be a good citizen and fuels toxic behavior.

He also highlights what becomes possible when you bypass the laws of physics: rather than flying for hours, you put on a headset and are effectively teleported together. His dream is that people could work wherever they want with the same opportunities, because they'd feel physically together, building the culture, relationships and trust that suffer when colleagues never meet in person. He admits it's hard to foresee all the implications from first principles and that Meta will build it one year at a time, trying to accentuate the good and mitigate the bad.

Presence as the Next Frontier

Lex says he keeps forgetting for long stretches that they aren't in the same room and suggests that what is real may be the conscious, subjective experience rather than physical co-location. Zuckerberg responds that he is definitely there; what isn't physically real is only their being in the same place. Presence carries intimacy, trust and nonverbal communication that previously required getting on a plane, and delivering those social and psychological benefits to anyone is, in his view, the promise of the metaverse. He adds that humans aren't "just like a brain and a tank": the human experience is physical, which is why he wants computing to be something you move with rather than sit still in front of.

He places this in the arc of his career: Facebook began as text with a single profile photo, moved to photos, and is now mostly video, with expression becoming ever richer and more vivid. Immersion and presence are clearly the next frontier, and though technologically very hard, he says building it is what Meta is there to do, calling it potentially "by far the most human and social computing platform that has ever existed." Lex closes by calling it one of the most incredible experiences of his life, joking that he might be at a beach with no pants on.

Conclusion

This conversation is as much a demonstration as an interview: two people hundreds of miles apart, rendered as photorealistic avatars, discussing what that very experience could mean. Zuckerberg lays out a coherent vision in which Codec Avatars, affordable mixed reality like Quest 3, varied AI personas and open models like Llama 2 converge into a world where the real world is the blend of physical and digital. Lex's visible emotion makes the case for presence more vividly than any spec sheet, while their discussion of identity, grief, anonymity and AI reliability is a reminder that much still needs to be worked out. For anyone curious where immersive computing and consumer AI were heading in 2023, it's a rare first-hand look.

This is a summary of "Mark Zuckerberg: First Interview in the Metaverse" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.