Where software speed really comes from, according to people who deliver it

The Short Version

Software speed comes from several different layers, and the people who deliver it each work at a different one. Guido van Rossum's team made Python 3.11 10 to 60 percent faster without touching the language, by trimming the interpreter's internal data structures and teaching it to specialize its instructions for the types it actually sees [1]. Chris Lattner's Mojo goes after gains of a different order, with speedups cited as up to 35,000 times over standard Python, by compiling the code, taking values out of heap objects, and using vectors, threads and the memory hierarchy [2]. The FFmpeg developers Kieran Kunhya and Jean-Baptiste Kempf write assembly by hand because, in their measurements, it runs 10 to 50 times faster than C, and they reject the idea that compilers can do it for them [3]. The three accounts fit together: each layer removes a different kind of waste, each has a cost in complexity or compatibility, and the work at one layer does not make the others unnecessary [1][2][3].

Table of contents

What They Said

Guido van Rossum: a faster interpreter, and nothing else changed

Van Rossum, Python's creator, starts his explanation of the 3.11 speedup with a principle: "It has to do with simplicity of software versus performance." [1] When he began Python he did not expect great success, and "I tried to get something working and useful in about three months" [1]. In his words, "I sort of cut corners" [1]. He wrote much of the code as simply as it could be written, knowing that "there are many things that you can code more efficiently by adding more code" [1].

He illustrates the trade with prime numbers. A naive check tries every divisor. Smarter versions stop at the square root, skip even numbers, and check only previous primes, and each refinement buys speed at the price of complexity [1]. Thirty years of choosing the simple version left low-hanging fruit in the implementation [1].

To explain where the fruit was, he describes how Python runs. Source code is compiled into bytecode, a set of instructions for a machine that does not physically exist: "it's an imaginary machine" [1]. The interpreter is the program that carries those instructions out [1]. For 3.11, almost all the work went into the interpreter and not the compiler, and the team did not build a just-in-time compiler [1]. Part of the gain came from "removing the fat from some internal data structures used by the interpreter" [1]. Then he names the main one: "But the key idea is an adaptive specializing interpreter." [1]

The problem it solves is that Python's operations are generic. As van Rossum puts it, "actually in Python, the language, there's no such thing as adding two numbers" [1]. An add might be adding integers, floats, strings, lists or user-defined objects, so every time the interpreter reaches one it follows pointers to find the right type's add function and does error checking [1]. The specializing interpreter watches what types actually turn up at a particular line. If it keeps seeing integers, it swaps in a specialized add-integer instruction that does a quick check and adds directly, and it falls back to the generic path if the guess turns out wrong [1]. The reasoning is a bet on repetition: "we assume that the weather tomorrow is going to be the same as the weather today." [1]

He is open about the limits. Someone could construct a pathological program that runs slower, though it would be unrealistic [1]. The trick is well known from other dynamic languages, including Facebook's HHVM for PHP and the JVM [1]. What recommends it for Python is that it preserves the dynamic typing that makes the language friendly to beginners: users change nothing and their programs get faster [1].

Van Rossum also explains two things Python has chosen not to do for speed. The first concerns type hints. Programmers can annotate the types of variables and arguments, and a separate static type checker reads those annotations without running the code [1]. One might expect the interpreter to use them, but "the type annotations currently are not used for speeding up the interpreter" [1]. One reason is that annotations sometimes contain "lies": the checker cannot prove an integer is ever anything else, yet at runtime someone violates the assumption and the interpreter copes [1]. "If we started enforcing type annotations in Python, many Python programs would no longer work." [1] Checking them at runtime is no shortcut either, since "all that extra runtime type checking is going to slow your code down instead of speed it up" [1]. A future version might use annotations as hints with a fallback [1].

The second concerns the global interpreter lock, the GIL. It exists because Python was written before threads were common; when operating systems added threading in the early 1990s, the team wanted to give Python programmers a safe version without rewriting the whole interpreter to be thread safe [1]. That was fine when computers had one CPU, and became infamous when chips went multi-core and people found their Python threads still ran on a single core [1]. A developer at Facebook has built a no-GIL fork of CPython that uses all cores with limited single-thread slowdown [1]. Van Rossum is unconvinced it is worth adopting: "I'm personally not convinced that there are enough people needing the speed of multiple threads" to justify the added complexity, the maintenance and the overhead for single-threaded programs [1]. He calls the GIL "a pretty nice goldilocks point between no threads and all threads all the time", and adds, "But not everybody agrees on that." [1] The nearer-term route he expects is multiple sub-interpreters, each running an independent Python program with faster communication between them [1].

Chris Lattner: start from the hardware and work backward

Lattner, the creator of LLVM, Clang and Swift, built Mojo as a superset of Python with the performance of C and C++ [2]. He likes Python and calls it "the universal connector" that lets people compose large systems; its main weakness is that it is slow [2]. His view is that Python is not broken, just "not done yet." [2]

He walks through where Mojo's gains over CPython, the standard implementation, come from, and it is a stack of separate effects [2].

The first is compilation. Moving from an interpreter to a compiler alone gives 2x to 10x [2].

The second is how values are stored. CPython represents every value, even an integer, as a heap-allocated object with a header, reached through a pointer and reference counted [2]. Getting values out of the heap and into machine registers is roughly another 10x [2]. He notes that reference counting is also tightly intertwined with the global interpreter lock that hurts parallelism [2].

Then come the hardware's own capabilities: vectors, which do 4, 8, 16 or 32 operations at once; threads across many cores; and control over the memory hierarchy [2]. He describes a sequence of techniques from a demonstration by Jeremy Howard: vectorization for a single core, parallelizing to spread work across cores, and tiling to keep data close to the compute [2].

The last piece is autotuning. Hardware has fiddly magic numbers, such as cache sizes, vector lengths, GPU warp sizes and the 128 tile size of a TPU, and Lattner believes most programmers do not want to know them [2]. In Mojo a programmer can list candidate options and let the compiler test them empirically on the target machine, cache the best result and stitch it in [2]. The stakes are high: in high-performance computing, he says, getting these numbers wrong costs not 1 percent but 2x or 10x, because data spills out of cache [2]. The same mechanism lets old code adapt to new hardware without the tangle of preprocessor conditionals that makes C and C++ code unmaintainable [2].

Together these produce the figure Lex Fridman cites, up to 35,000x over CPython [2]. More modestly, people have moved small pieces of Python code into Mojo unchanged and seen 12x speedups [2].

Lattner makes the comparison with van Rossum's layer directly. On the work to speed up CPython 3.11, which he puts at 15 to 20 percent, he is appreciative but says it does not run GPUs or vectors, and adds, "20% good 35 000 times is better." [2] Mojo, he stresses, worked backward from the limits of physics, where the CPython effort works forward from making Python a little faster [2].

Types are part of his answer, and here he departs from Python. He notes that Python's type hints are ignored by the interpreter, interpreted differently by competing checker tools, and cannot be used for performance [2]. In Mojo, if you declare a type, the compiler enforces it [2]. Adoption is progressive: prototypers are not pressured to add types, while large teams may require them [2]. Mojo even implements integers and floats in the library instead of hard-coding them into the compiler, following his principle of taking magic out of the compiler and putting it in libraries [2].

Several other design choices serve speed. Errors are implemented as returning either a result or an error, so throwing is as fast as returning; he says C++-style zero-cost exceptions make throwing extremely expensive, bloat binaries and would never run on accelerators [2]. Value semantics with lazy copying, backed by an ownership system, mean data is copied only when actually modified and values are transferred instead of duplicated [2]. Mojo destroys values as soon as possible after their last use, which improves memory use [2]. And compile-time metaprogramming runs Python's expressive dynamic features at compile time so that they can execute efficiently on GPUs and embedded processors [2]. He admits the compiler work is "super hard" and depends on a completely new compiler design [2].

For Lattner the reason to care is practical. He jokes that, measured this way, Python is bad for the environment; better performance means fewer servers, lower costs and better products, because faster models can be made bigger [2]. He also points to a structural waste: many Python libraries are a layer of Python over C and C++ underneath, and large models must be rewritten from Python into C++ for deployment [2]. One language that is fast enough would remove that split [2].

Kieran Kunhya and Jean-Baptiste Kempf: write the instructions yourself

Kunhya, a codec engineer and FFmpeg contributor, and Kempf, president of VideoLAN and longtime lead of VLC, defend the oldest layer of all. FFmpeg, the set of libraries behind nearly all video on the internet, contains a great deal of handwritten assembly [3].

Kunhya explains what the assembly does. SIMD instructions operate on many values at once. Given 16 elements, "I can run one instruction, and that one instruction sums all 16 elements" [3]. This suits the material: "Video is a pixel grid, so I can perform operations on multiple pixels" at the same time [3].

There is a middle path that FFmpeg declines. Part of the industry uses intrinsics, C functions that behave very similarly to assembly but leave one decision to the compiler: which CPU registers hold the data. With intrinsics, "the compiler allocates those for you" [3]. FFmpeg's position is that "we don't use any abstractions or any major abstractions on top of that" [3].

The reason is the size of the result. Kunhya says that when they write SIMD, "we have a 10x, not percentage, 10x to 50X speed improvement", and of one function he shows, "That function is 62X." [3] Both men reject the claim that compiler auto-vectorization can match this, saying the gap is multiples, not a few percent [3].

Kempf's example is dav1d, VideoLAN's decoder for the AV1 video format, whose motto was "every cycle matters." [3] It contains roughly 30,000 lines of C and 240,000 lines of handwritten assembly [3]. It even uses its own internal calling convention to avoid saving registers the standard way [3].

The need is growing, in their view. Each codec generation gives about 30 percent better compression, but Kunhya notes that this costs far more compute, and he estimates that up to 45 percent of files cannot be decoded on the GPU and need a software fallback [3]. Kempf argues that as hardware gains slow and compute demands climb, low-level optimization for hardware becomes more valuable, and that it is something that cannot simply be vibe-coded [3]. Kunhya notes that current LLMs struggle with this kind of assembly because there is not enough training data [3].

The craft is rare but teachable. Kunhya wrote FFmpeg's asm-lessons, whose prerequisites are high school math and C, especially pointers [3]. They single out Henrik Gramner, who knows x86 so well that Intel reportedly defers to him, and Martin Storsjö, who writes ARM assembly on his phone at the playground [3]. An FFmpeg tweet boasted that "teenagers have written more assembly in FFMPEG than Google engineers." [3]

Handwritten code needs heavy checking. FFmpeg enforces bit-exactness, meaning every decoder must produce identical output, through FATE, a volunteer-hosted testing system covering a huge matrix of operating systems, compilers and CPU architectures [3]. On Rust, Kunhya values its memory-safety goals and would welcome it if it met the same testing and stability bar; Kempf, whose startup is written mostly in Rust, says pairing Rust with handwritten assembly breaks the safety model, and suggests compile-time checked assembly as an answer [3].

Where They Agree, and Where They Don't

All three locate the waste in generality. Van Rossum's generic add has to discover its types every time [1]. Lattner's boxed integer has to be fetched from the heap through a pointer [2]. The FFmpeg developers' C code leaves register allocation and vectorization to a compiler that does not know what they know [3]. Each speedup comes from replacing a general mechanism with one fitted to the actual case.

All three accept that speed costs complexity. Van Rossum says outright that efficiency is bought by adding code [1]. Lattner calls his compiler work "super hard" and names complexity the "bitter enemy" of his whole field [2]. dav1d carries eight lines of assembly for every line of C [3].

Lattner and the FFmpeg developers agree that the hardware is where the big numbers are. Vectors are central to both accounts [2][3]. Kempf's argument that slowing hardware gains make low-level optimization more valuable matches Lattner's starting point, that new chips are useless if software people cannot program them [2][3].

Van Rossum and Lattner disagree about what counts as enough. Van Rossum presents 10 to 60 percent as a real improvement that costs users nothing [1]. Lattner answers that 20 percent is good and 35,000 times is better [2]. Each is right about his own constraint. Van Rossum is improving the implementation that the whole Python world runs, so he cannot break dynamic programs, and he will not take on complexity he doubts enough people need [1]. Lattner is building a new language whose compiler can enforce types and target accelerators, and he keeps existing code working by importing Python packages through CPython itself [2].

They disagree about using types for speed. Van Rossum explains why CPython does not: annotations can be wrong, and enforcing them would break programs [1]. Lattner makes enforcement the rule whenever a type is declared [2]. The disagreement is narrower than it looks, since van Rossum allows that a future Python might treat annotations as hints with a fallback, and Lattner does not force anyone to write types [1][2].

They differ on parallelism. Van Rossum regards the GIL as a reasonable middle point and doubts the demand for multithreaded speed [1]. Lattner lists threads across many cores as one of the major multipliers and notes that Python's reference counting is tied to the lock that blocks it [2]. Van Rossum himself says not everybody agrees with him [1].

Lattner and the FFmpeg developers disagree, at least in emphasis, about trusting the compiler. Lattner's programme is to let the compiler find the best tile sizes and vector widths by testing on the target machine, because most programmers do not want to know those details [2]. Kunhya and Kempf say compilers fall short of hand-written SIMD by multiples and refuse even the abstraction of intrinsics [3]. The two positions come from different jobs. Lattner is trying to make good performance available to people who are not specialists, across thousands of chips [2]. The FFmpeg developers maintain a small number of extremely hot functions that the whole internet's video passes through, where a specialist's time is repaid many times over [3]. Their criticism is aimed at auto-vectorization of C, and the sources contain no verdict from them on Mojo.

Why It Works

An interpreter repeats decisions a program has already made. Every generic add in CPython follows pointers and checks for errors, even on the millionth pass through the same loop with the same types [1]. Specialization works because programs are predictable: the types seen at a line today are almost always the types seen there tomorrow [1]. The quick check and the fallback keep it correct when they are not [1].

Memory layout dominates. Lattner's 10x for unboxing is about where a number lives. A value in a register can be operated on immediately; a value in a heap object with a header has to be located and reference counted first [2]. His warning about cache makes the same point at a larger scale: data that spills out of cache costs 2x or 10x [2]. Tiling exists to keep data close to the computation [2].

Modern processors do many things per instruction and per clock tick. One SIMD instruction handles 16 elements in Kunhya's example, and Lattner cites vector widths of 4 to 32 [2][3]. Code that processes one value at a time leaves that capacity unused, which is why pixel grids and matrices reward vector code so heavily [2][3].

A human can know things a compiler cannot assume. FFmpeg's developers choose which registers hold what, and dav1d's private calling convention skips work that a standard convention requires [3]. A compiler bound to general rules has to be conservative; the authors of one function do not.

The gains multiply. Lattner's figure is a product of stages: compile, unbox, vectorize, thread, tile, tune [2]. That is why the layers do not replace one another. A faster interpreter still interprets [1][2]. A compiler still needs someone, or some search, to choose the tile size [2]. And inside a compiled program, the inner loop written in hand-tuned SIMD is still many times faster than the same loop in C [3].

Where It Applies, and Where It Doesn't

Interpreter work helps everyone a little. The 3.11 gains arrive with an upgrade and require no changes to code [1]. They vary by program, from 10 to 60 percent, and they do not reach GPUs or vectors, as Lattner points out [1][2].

A compiled superset helps most where Python was weakest. Numeric and AI workloads on varied hardware are Mojo's target [2]. At the time of the conversation it was a 0.1 release, about seven months old and not for production, with traits, classes, keyword arguments and list comprehensions still to come, so the headline number should be read as a demonstration and the 12x on unchanged code as the more typical early experience [2].

Hand assembly pays only on hot paths. The FFmpeg case is pixel-level routines run on enormous volumes of data [3]. The cost is real: separate code for each CPU architecture, a scarce skill, and a testing system to match [3]. Nothing in their account suggests writing ordinary application code this way.

Simplicity is also a legitimate choice. Van Rossum's three-month Python and his view that code is written for other people as much as for machines are a reminder that the fastest version is not always the right one [1]. His prime-number example shows the simple version is easier to understand [1].

Sometimes the speed is already there. Van Rossum describes how scientists adopted Python as a way to tie together Fortran and C++ math libraries, and how its extensibility let others write efficient array packages [1]. Lattner observes the same structure, a layer of Python over C and C++ [2]. For many users the slow language is only steering fast code.

Not all speed questions are about the CPU. Van Rossum's fisherman tending four rods is his picture of concurrency: when the work is mostly waiting, as with network connections, the answer is switching between tasks, which is what asyncio provides [1].

What To Do With It

1. Upgrade the interpreter before changing your code. The 3.11 gains come from the runtime itself and cost nothing to adopt [1].

2. Find out which layer your slowness lives in. Interpretive overhead, boxed values, single-core execution, poor cache use and unvectorized inner loops are different problems with different cures [1][2][3].

3. Write the simple version first, and add complexity where measurement demands it. That is how Python itself was built, and van Rossum's prime example shows what each refinement costs [1].

4. Lean on libraries that already do the hard part. Python's array packages and FFmpeg's hand-tuned routines put specialist work behind an ordinary interface [1][3].

5. Keep hot loops type-stable. The specializing interpreter bets that a line will keep seeing the same types, so code that honors the bet benefits most [1].

6. Use type hints for correctness, and do not expect speed from them in CPython. Run a static checker in development; the interpreter ignores the annotations [1].

7. For numeric work, think in Lattner's order. Compile, get values out of the heap, vectorize, parallelize, then tile for cache, and let a tool search for the tuning constants where one exists [2].

8. Reserve hand-written SIMD for the few functions that dominate runtime, and test them hard. FFmpeg pairs its assembly with bit-exact tests across many platforms [3].

9. Be skeptical that a compiler has vectorized your code as well as it could. The FFmpeg developers measure the gap in multiples [3].

10. If you want to learn the lowest layer, start with FFmpeg's asm-lessons. The prerequisites are high school math and C with pointers [3].

Sources

1. Guido van Rossum on Python 3.11's Speedup, Type Hints, the GIL and the Future of Programming · Lex Fridman Podcast
2. Chris Lattner on Mojo, Modular and the Future of Programming and AI · Lex Fridman Podcast
3. FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya: Codecs, Assembly, and the Volunteers Behind Internet Video · Lex Fridman Podcast