Podcast Summary | Chris Lattner on the Future of Programming Languages, Compilers and Chip Design

Lex Fridman Podcastwith Chris Lattner· · Episode aired October 19, 2020· 18 min read
Listen to this summary

High-Level Overview

In his second appearance on the Lex Fridman Podcast, Chris Lattner, the engineer behind the LLVM compiler infrastructure, the Clang compiler and the Swift programming language, who has also worked on TensorFlow and TPUs at Google, led Autopilot software at Tesla and is now senior vice president of platform engineering at SiFive, talks with Lex about why programming languages matter, how good language design actually happens, and where computing is heading as Moore's Law slows down. The conversation ranges from the fine-grained details of value semantics and syntactic sugar to chip design, RISC-V, parallel programming, machine learning compilers, GPT-3 and Andrej Karpathy's Software 2.0 idea. It closes with reflections on the pandemic, online toxicity, advice for young people and the meaning of life.

The tone is warm, curious and playful. Lex openly admires Chris, jokes about wanting to print a walrus operator shirt, and cheerfully admits to asking basic questions, which Chris encourages. Chris is precise and patient, comes back often to trade-offs, and keeps returning to one theme: tools, languages and leaders exist to make people more productive. The conversation moves from leadership to deep technical territory (languages, compilers, hardware) and then widens into society, history, optimism and personal meaning.

Table of contents

The Discussion Explores:

  • Leadership lessons from working with Steve Jobs, Elon Musk and Jeff Dean, and why good leaders focus on getting the right answer rather than having it
  • Programming languages as user interfaces, and the idea of a language as a "bicycle for the mind"
  • Swift's design choices: value semantics, copy-on-write, progressive disclosure of complexity, and libraries that feel native
  • How language communities decide things, from Swift's core team and evolution process to the walrus-operator fight that led Guido van Rossum to step down from Python
  • LLVM, MLIR and SiFive, and why chip design is really "a whole bunch of compilers" in need of better tooling
  • Moore's Law, parallelism and concurrency, including the Swift concurrency manifesto and the problem of unpredictable compiler magic
  • Machine learning as a new programming paradigm: GPT-3, sparse mixture-of-experts models and a friendly disagreement with Software 2.0
  • The pandemic, remote work, online pile-ons and long-term optimism, ending with advice on hard work, luck, imposter syndrome and meaning

Deep Dive into Main Ideas

Leadership: Jobs, Musk, Dean and the Art of Getting the Right Answer

Lex opens by asking what Chris learned from three very different people he has worked with. Chris says he worked much more with Elon Musk than with Steve Jobs, but sees a lot in common: both are visionary and demanding in their own way. His sense is that Jobs was more focused on human factors, building things that feel good and that people love, while Musk focuses more on understanding how exponentials work and predicting where technology will go. Jobs, he believes, pushed people hard but let go once he got an explanation that made sense, and he knew when someone was holding back. Jeff Dean is different again: a "really, really, really nice guy", a classic Googler who still programs, leads through building technology himself, and pulls people in and inspires them.

Asked what he took from them for his own leadership, Chris says you really have to know what you're talking about: be grounded in the product, the technology, the business and the mission, and understand why people are there. At Tesla, for example, people join because they believe in clean energy and electrification. Just as important is being comfortable not knowing. In his words, "it's not about having the right answer. It's about getting the right answer." That means building trust, letting people be vulnerable with ideas that aren't fully formed yet, and creating a safe space where people will tell you when you're wrong.

He also admits that "dumb questions are my specialty." Now working on hardware at SiFive, a field he says he knows little about from the inside, he has found that "if you ask a lot of dumb questions, you get smarter really quick," especially when surrounded by people who like to teach.

Why Programming Languages Matter: Design as User Interface

Chris frames programming languages as a trade-off along a ladder of abstraction, from assembly code up to designing neural networks that do the work for you. Most languages give you basics like portability across Intel, ARM and RISC-V hardware, but beyond that they differ in how they balance their trade-offs. He divides a language into two levels: the "nuts and bolts" (type systems, compiler machinery, efficiency) and the user interface. Many people treat language syntax as purely subjective, a matter of curly braces versus tabs, but Chris argues that "good design is something that you can feel", and that there really is better and worse.

What people care about, he says, is not go-to statements or braces but getting a goal done, such as setting up a web server. Two things matter there: what libraries exist, and how quickly you can put them together, including the time spent debugging and fixing. Python is great at assembling things, but performance-critical libraries such as machine learning frameworks end up written in C or C++ and wrapped from Python, so developers live in "two worlds", where tools like the debugger don't work well across the boundary.

He borrows an old Steve Jobs image: "a programming language is a bicycle for the mind." By raising the level of abstraction, a language lets you fit more in your head and reuse other people's libraries. He points to SwiftUI, Apple's declarative UI framework, as an example that rules out whole classes of bugs and lets developers do much more with much less code. Lex adds his own experience: he spent years in C++, and felt a "stupid smile" of productivity when he first used Python, which, unlike Perl, did not bog him down in scalars, arrays, hashes and type globs.

Value Semantics and Copy-on-Write: Making Math Behave Like Math

The most technical stretch of the conversation is Chris's explanation of value semantics. In PyTorch, Java and many other languages, objects have reference semantics: you hold a pointer to the thing rather than the thing itself. That is why PyTorch users sometimes need to clone a tensor. If you pass an array to a function and either side later changes it, the other side sees the change, which leads to defensive copying and hard-to-debug errors. As Lex puts it, a tensor that should behave like math "quietly doesn't behave like math."

Swift defaults to value semantics, including for arrays, dictionaries and other collections. If you copy a value and change one copy, the other is unaffected, so this class of bugs disappears. Swift avoids the usual cost through copy-on-write: data is only copied when someone actually changes a shared value. Chris compares this with Java's immutable strings, where appending two strings always allocates a new third string because someone might still hold a pointer to the originals. In Swift, if you have the only reference to a string, appending to it can happen in place, while values passed to several readers can still be shared. He calls this "a really nice, sweet spot" he hasn't seen in other languages: you get most of the safety of purely immutable designs without paying for constant copying. There are still trade-offs at very low levels, he concedes, but the question is what the right default should be.

Progressive Disclosure and Libraries That Feel Native

Chris names a UI principle that he thinks people love about Swift: progressive disclosure of complexity. Like Python, Swift lets a beginner start with a simple "print hello world", instead of the "public static void main" ceremony of other languages that forces a new learner to face access control, classes and packages on day one. Yet a power user can still drop down to C-level performance, raw pointers and even calling malloc directly. The design problem is how to factor that complexity so the normal case just works.

When Lex asks about the most beautiful feature in a programming language (his own early favorite was Python's list comprehensions), Chris answers that what matters most is what enables high-quality libraries. In Swift, even basic types like Int and Float are not hard-coded in the compiler: Int is simply a struct in the standard library with a plus operator defined on it. Anything the standard library can do, your own types can do too, so a community that cares about quaternions can build a library that feels as native as integers. He contrasts this with Java, where built-in numbers are passed by value but a user-defined complex number becomes a class passed by reference, so "You lost math." He cites Lisp as a classic example of a language where libraries become part of the language itself.

He also argues that languages should model at least the most important "inherent complexity" of real problems. Lisp and JavaScript are simple languages without built-in packages, so ecosystems like npm grow up around them. Sometimes that flexibility is great, but it often leads to fragmentation, and too much in the language is also expensive, so the job is finding the balance.

Types, Syntactic Sugar and the Walrus Operator

On typing, Chris repeats a line Lex has heard from him before: Python doesn't lack types, it has one type. That gives duck typing for free and makes it easy to pound out code, but it also means passing a string where an integer was expected may not fail until deep in some stack trace. Types, he says, can be thought of as asserts checked at compile time. Design means weighing trade-offs like these along many dimensions: static versus JIT compilation, garbage collection versus reference counting versus manual memory management, and concurrency.

The pair also discuss breaking conventions, such as using the equals sign for assignment. Chris calls it a return-on-investment question: what class of bugs does a change define away? In C, assignment returns a value, which is "very rarely used for good" and often causes bugs. Swift's open evolution process removed several early mistakes that had been carried over from C without much thought, including the ++ and -- operators, because they caused confusion and offered little over writing x += 1.

Then Lex turns to "drama": Python 3.8's walrus operator, the controversy that pushed Guido van Rossum to step down as Python's leader. Chris explains that the feature is polarizing because it is syntactic sugar, a more concise way to write something already possible, and judging whether sugar is worth it becomes almost completely subjective. People also overestimate the burden of learning something new: if it had been there from the start, nobody would question it. On Guido himself, Chris, who says he has met him only a couple of times, suspects the decision-making weight had become so tied to Guido's personal identity that the criticism (even from 1% of a community of millions) wore him down. In a 25-year-old language with a huge user base, he adds, the community is right that adding something carries a heavy burden, because once people build on it, it can't be removed. His summary: "Python isn't about Guido anymore. It's about the users."

How Swift Makes Decisions: Core Team, Community and "Case Law"

Chris describes Swift's governance as layers. A core team of six or seven people who have worked on Swift since early on (it became public in 2014) provides continuity. They remember which "boulders are still missing and keep other proposals from invading that space until it is filled. Around them is the Swift Evolution community, hundreds of people on a discussion forum who act as a rock tumbler for ideas". A formal proposal process lets the core team seek consensus, set guardrails and keep the long-term direction.

Because the decision rests with a group and every decision comes with a published rationale, it doesn't become personal the way it did for Guido. Lex compares this to the Supreme Court and case law, and Chris agrees: people griping later criticize the decision, not a person, and future discussions have a record to build on. On leadership more broadly, Chris explains why successors to great founders often can't make bold pivots: they weren't there when earlier decisions were made, so they don't know the principles behind them and those decisions start to look like gospel. Even a leader in charge has to explain a change and get people to believe in it, because "there's only so much power you have as a leader."

Get new Lex Fridman Podcast summaries by email

The GIST of every new Lex Fridman Podcast summary in your inbox, daily or weekly. Free, and you can unsubscribe any time.

LLVM at 20, MLIR and Reinventing Chip Design at SiFive

LLVM turns 20 that December, and Chris candidly admits it is at risk of becoming calcified. Many contributors have worked on it for a decade and can't see its faults, and while he sees many of them himself, they grate on him like sand under his fingernails because he no longer has time to fix them. That is part of why MLIR was built. As he jokes, the "ML stands for multi-level, not machine learning, because compiler people can't name things very well. LLVM is excellent for CPUs and only the tip of the iceberg on GPUs, while domain-specific compilers, such as the JVM's HotSpot or Swift's own high-level layer, have historically been built from scratch with lots of duplicated work. MLIR provides general infrastructure to build such compilers quickly, for machine learning accelerators and even for hardware synthesis, where you are compiling for a domain of transistors." MLIR has now joined the LLVM family, and Chris finds it fascinating to watch the new ideas slowly win over people used to the old way.

At SiFive, which was founded by three of the creators of RISC-V at Berkeley working with Dave Patterson, Chris leads engineering and product. RISC-V is an open instruction set, unlike x86 (Intel and AMD) or ARM, so anyone can build chips for it. He explains the value with a history lesson: HP moved its PA-RISC server business to Intel's Itanium, which "didn't work out so well," and customers who sink billions into proprietary instruction sets can be stranded. With RISC-V, if a vendor disappears in 20 years, you can buy cores from someone else. He also describes the chip design flow, from hardware languages such as Verilog down to the masks handed to fabs such as TSMC and Samsung, as a well-studied but fragmented space dominated by EDA tools from three big companies, with lots of "duct tape and lost productivity. To him it is really a whole bunch of compilers written without thinking about it through that lens." Lowering the cost of designing chips would mean more custom silicon, better battery life and better-fit designs for devices like smart toasters, instead of companies buying off-the-shelf chips with silicon they never use.

Moore's Law, Parallelism and Predictable Performance

Asked whether Moore's Law is dead, Chris says both sides are right. Jim Keller is correct that physics still has lots of room left, and Dave Patterson is correct that Moore's Law, technically, was an economic observation about cost. Chris adds a third view: single-threaded performance no longer doubles the way it did in the Pentium days, when you "just wrote a check" and waited a year for the same program to run twice as fast. Instead, programmers are pushed toward CUDA, threads and domain-specific accelerators, which changes the programming model, and that brings the question back to languages.

He contrasts implicitly parallel hardware languages like Verilog and VHDL, CUDA's approach (write what looks like a scalar program that runs on a thousand things at once), and his Swift concurrency manifesto. The manifesto lays out a path from async/await to memory safety that avoids race conditions (value semantics help a great deal here), then to actors, which he describes as "islands of single threadedness" communicating asynchronously, and on to distributed programming across processes and machines. A recurring principle is a simple model you can explain with an escape hatch, rather than a system like Logo that is easy to start but hits a ceiling, or Python, where the global interpreter lock leaves you with bad trade-offs once you need concurrency.

He is openly critical of "magic" auto-parallelizing compilers for ordinary imperative code. They can deliver a 10x speedup through clever pattern matching, but a small refactor can silently defeat the heuristic and cost you 10x, and you won't know why. If you care about performance, he argues, predictability matters. The exception is machine learning: by composing operations into graphs, the ML community produced a programming model abstract enough that you can drive a TPU pod of 1,024 nodes from Python. He lists where future ML speedups will come from: process technology, architecture (such as the TPU's systolic array), system-level interconnects, programming models and algorithms, noting OpenAI's analysis of exponential algorithmic gains. He also cites a law that "Compilers double the performance of any given code every 18 years," adding that the real power of compilers is unlocking new hardware and even producing better error messages.

Quantum Computing, GPT-3 and Software 2.0

On whether the universe is a simulation, and what hardware it would run on, Chris notes that as far as we know quantum systems are the bottom of the pile of turtles. He says he knows almost nothing about quantum computing, which means he will eventually have to find an excuse to get involved. He sees it as "literally a science project" still looking for the useful problem that justifies its cost, much as machine learning went through winters before AlexNet brought in talent and money. He guesses it is perhaps ten years away. And the simulation's programming language? "Oh probably Lisp."

On GPT-3, Chris says it wasn't surprising given the known rule that bigger models trained on more data do better, but it was a tremendous engineering achievement with serious implications for misuse. He is more interested in architectural leaps than raw scaling: transformers are simple, dense and not recurrent, while the brain is sparse. He points to Jeff Dean's idea of sparsely gated mixture-of-experts models, where specialized networks spread across a cluster and queries are routed to the right parts. On code generation, he thinks a GPT-style model trained on the right corpus could write syntactically correct Swift, but the unsolved problem is expressing intent, for example writing scaffolding plus assertions and having the system hill-climb toward code that passes. He observes that these models do a lot of pattern matching and not much equational reasoning, though both men joke that humans may also be pattern matching and then justifying it afterwards.

He respectfully disagrees with Andrej Karpathy's Software 2.0 thesis, which he summarizes as replacing more and more imperative code with learned models. Chris sees machine learning as a new programming paradigm alongside structured, functional, object-oriented, generic and differentiable programming, not as a replacement for Software 1.0. Learned models are the best tool for the human world (perception, language translation, cat detectors), but you won't write a bootloader with a deep learning model, which is also far more energy-hungry. He agrees with Karpathy that ML systems need Software 1.0 discipline in testing, continuous integration and validation, and warns that focusing on a single paradigm, as object-oriented programming did in the 1980s, "puts you in a box."

The Pandemic, Online Toxicity and Long-Term Optimism

Turning to COVID-19, Chris calls remote work "a great normalizer": it takes physical appearance out of some conversations, puts already-remote employees on equal footing (no one misses the whiteboard discussion), and forces asynchronous thinking. At the same time, he says, it "just sucks" to be apart, because most people are wired to work together in person. He expects remote work to become more normal and ease the pressure behind Bay Area housing prices. He sees forced change, such as losing a business, as leading to reflection and a "fresh slate", and believes the world had been over-optimized for one set of priorities. His core view is that progress needs a catalyst: people are rightly conservative, especially when they carry responsibility for thousands of others, so real change often comes from crises.

Lex raises the rise in online pile-ons, citing the backlash against Yann LeCun over a comment about bias in datasets. Chris traces it to an "us versus them" instinct magnified as the internet shrinks the world and clickbait rewards outrage. His antidote is empathy: "sometimes people just have a bad day," and he admits to firing off badly phrased emails between meetings himself. Lex mentions reading The Rise and Fall of the Third Reich and worrying about economic pain combined with a sense of being disrespected. Chris answers that most people assume the world won't change, when it changes constantly, and that by studying the catalysts behind past change you can deliberately drive change in ways that hurt fewer people. "I'm a long-term optimist on pretty much everything," he says, pointing to how far humanity has come in a century.

Advice for Young People and the Meaning of It All

Chris's advice to students is that profound leaps happen because people believe change is possible and are willing to do the hard work. That work is hard, and after two years on a project, self-doubt creeps in. The secret, he says, is loving what you do, which is why it's worth experimenting widely early in your career until something resonates. He is candid that luck plays a huge role, and says "real value comes from doing things that are hard." He deliberately puts himself in rooms where others know far more than he does, because smart people usually love to teach, and because a lot of progress comes from people who hop between fields and bring a new perspective.

On meaning, Chris says he is prepared for life not to mean anything, but doesn't find that a productive way to live. He finds happiness in other people, including having kids and building relationships, and asks how he can take what he's good at and add value. He sometimes steps out of familiar roles, both to go back into learning mode and to give a younger generation their chance. On AGI, he pushes back on being species-centric: we are also machines optimizing an objective, and it would be premature to say future AGIs have no soul. His real worry is that humanity needs a challenge, and a world without personal contribution could bring out the worst in people. He ends by urging listeners to spend less time consuming outrage in the news and more time creating, learning and growing. Lex jokes that this means everyone should go into compiler design, and Chris playfully agrees that it is less crowded than machine learning, especially at the end of Moore's Law.


Conclusion

This conversation shows the philosophy that links Chris Lattner's work on LLVM, Swift, MLIR and SiFive: every tool, language and organization should be judged by how productive it makes the people using it. Good design is about trade-offs made openly, with sensible defaults and escape hatches for power users, whether that means value semantics in Swift, predictable performance instead of compiler magic, open instruction sets instead of proprietary lock-in, or group decisions with published reasons instead of a single overburdened leader. Chris treats machine learning as a powerful new paradigm rather than a replacement for everything that came before, and sees hardware's future in cheaper, more customized chips. Underneath the technical depth is a humane message: ask dumb questions, assume good intent, love the hard things you choose to do, and stay a long-term optimist. As the episode's closing quote puts it, "So much of language design is about tradeoffs", and that applies just as well to leadership, careers and life.


This is a summary of "#131 – Chris Lattner: The Future of Computing and Programming Languages" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.

Books and products from this episode

  • Deep Work · Cal Newport(our summary)
    Lex highly recommends it while describing his deep work sessions.
  • The Rise and Fall of the Third Reich · William L. Shirer(our summary)
    Lex is reading it and draws parallels to 2020's turmoil.
Get new Lex Fridman Podcast summaries by email

The GIST of every new Lex Fridman Podcast summary in your inbox, daily or weekly. Free, and you can unsubscribe any time.

Want the REST?

This is an independent summary. Listen to the full conversation from Lex Fridman Podcast.

Full episode

More from Lex Fridman Podcast

Report a problem

Something wrong on this page? A mistake, a broken link, audio that won't play? Tell us and we'll fix it.