Speed is a feature: why top engineers obsess over latency

The Short Version

Three very different teams, the volunteers behind FFmpeg and VLC, the founders of the Cursor code editor, and Perplexity's CEO Aravind Srinivas, describe speed as part of the product rather than a finishing touch. FFmpeg writes handwritten assembly that runs 10x to 50x faster than C, and its AV1 decoder dav1d was built under the motto "every cycle matters" [1]. Cursor reuses the KV cache across keystrokes, warms the cache while you are still typing, and invented "speculative edits" because a slow suggestion is a useless one, and because, as the team puts it, much of what makes an editor fun comes from speed [2]. Perplexity tracks P90 and P99 latency at every component and tests its product over in-flight Wi-Fi, a habit Srinivas took straight from Larry Page's obsession with latency [3]. The shared pattern: people who build software that others love treat milliseconds and CPU cycles as a measure of quality, and they build the whole architecture around that from the start instead of bolting it on at the end.

Table of contents

What They Said

FFmpeg and VLC: "every cycle matters"

Jean-Baptiste Kempf (JB), president of VideoLAN and longtime lead behind VLC, and Kieran Kunhya, a codec engineer and FFmpeg contributor, give the most extreme version of the speed-first mindset [1]. Their world is video, where the numbers are unforgiving. MP3 compresses audio about 10x, but video needs 100x to 200x, with 1,000x as the target [1]. Each new codec generation gives about 30% better compression, but Kieran notes this costs far more compute [1]. So every step forward in quality makes the performance problem harder, not easier.

Their answer is handwritten SIMD assembly, which processes many pixels with a single instruction [1]. Kieran says it typically runs 10x to 50x faster than C, and shows one function running 62x faster [1]. JB describes dav1d, VideoLAN's AV1 decoder, which contains roughly 30,000 lines of C and 240,000 lines of handwritten assembly [1]. That ratio is the point: eight lines of hand-tuned assembly for every line of portable C. The team went so far as to use its own internal calling convention to avoid saving registers the standard way [1], shaving cost off the most basic act of calling a function.

Both men reject the idea that a compiler can do this for you. They dismiss the claim that compiler auto-vectorization can match handwritten assembly, saying the gap is multiples, not a few percent [1]. This matters because the usual argument against low-level work is that modern tools make it unnecessary. Kempf and Kunhya say the tools are not close.

JB also gives a forward-looking reason. As hardware gains slow and compute demands climb, he argues, low-level optimization for hardware becomes more valuable, and it is something that can't simply be vibe-coded [1]. Kieran adds that current LLMs struggle with this kind of assembly because there isn't enough training data [1]. In their view, performance craft is becoming scarcer and more valuable at the same time.

The speed obsession is not only about raw throughput. It shapes how they talk about the people involved. They single out Henrik Gramner, who knows x86 so well that Intel reportedly defers to him, and Martin Storsjö, who writes ARM assembly on his phone at the playground [1]. One FFmpeg tweet, which they celebrate, read that "teenagers have written more assembly in FFMPEG than Google engineers." [1] Kieran wrote FFmpeg's asm-lessons to teach the craft, and says the prerequisites are only high school math and C, especially pointers [1].

JB frames this as a matter of pride and excellence rather than a chore. In his words, people come to VLC and FFmpeg for two reasons, the first being love of the topic: "Work on something you love." [1] The second is technical: "Because we search for excellent. This is the best school ever, right? This is the best school ever of programming." [1] He goes further: "If you're good in C, in FFMPEG, if you know how to write assembly, I assure you you're going to be one of the best programmers ever, even if you're working on writing TypeScript, because this is the most amazing thing to do." [1] And part of that school is review: "you will have to get reviews by some of the most seasoned programmer ever who are going to look at every part. of your code and tell you why it's not great." [1]

Speed as a constraint that sets the whole design

The FFmpeg and VLC story also shows why speed has to be designed in. Kieran explains that the player first checks whether the GPU can decode the codec, and estimates that up to 45% of files cannot be decoded on the GPU and need a software fallback [1]. That software path has to remove entropy coding, read intra-prediction data, apply residuals stored in the frequency domain and run inverse transforms back to pixels [1]. Nearly half of the world's files land on that CPU path, which is why cycles in it matter so much.

JB's newest venture shows the same thinking applied to latency rather than throughput. His company Kyber is an open source SDK for real-time remote control of robots, drones and vehicles over a single QUIC connection, using forward error correction and correcting for clock drift across multiple sensors [1]. Its goal is 4 milliseconds glass-to-glass latency; they have reached about 7 ms so far, and at CES they demonstrated a 3D-printed rover controlled from France [1]. Here the product literally is the latency number: a remote-control system that responds late is not a remote-control system.

Speed paired with correctness

What stops this from being reckless micro-tuning is an equally obsessive testing culture. Bit-exactness means every decoder must produce identical output [1]. Kieran explains: "every implementation must produce exactly the same bits bit for bits in exactly the same data that comes out of a decoder," and "lex's implementation jb's implementation and my implementation of h264 must match bit exactly" [1]. He calls its absence in MPEG-2 "one of the biggest mistakes the video industry made" [1]. This is enforced through FATE, FFmpeg's volunteer-hosted testing system covering a huge matrix of operating systems, compilers and CPU architectures [1]. Hand-tuned assembly for dozens of targets is only safe because every version must produce identical bits.

Where quality meets efficiency: x264

Kieran's account of x264 shows a second layer to speed as a feature: spending bits and compute where humans will notice them. He credits x264's dominance to its focus on what looks good to human eyes rather than PSNR, the mathematical metric the industry and academia had chased for 20 years [1]. As he explains, minimizing mean squared error "leads to blurring because you actually want to you want to minimize you want to add a little bit of error to everything to reduce the mean squared error as opposed to having a big error" [1]. x264 instead introduced psychovisual rate distortion and adaptive quantization [1]. Lex summed it up as "compressing in a way that loses the least amount of information for the stuff that matters for us humans," and Kieran answered "yes exactly" [1]. Timing mattered too: "a lot of development took place when hd video was coming out," and CPUs "were getting fast you could do real-time video" [1]. The broader lesson is that efficiency work is not about hitting a number; it is about serving what the user actually perceives.

The results have large practical stakes. Kieran's "finger in the air" estimate is that 80%+ of FFmpeg pipelines rely on x264 [1]. JB notes that AV1 saves 40-60% bandwidth over H.264 and that YouTube re-encodes popular videos in AV1, and that AV2 is about 30% better again, with VideoLAN planning a decoder called dav2d [1]. Every one of those gains depends on decoders fast enough to run on ordinary hardware.

Cursor: speed is where the fun comes from

The four Cursor founders, Michael Truell, Sualeh Asif, Arvid Lunnemark and Aman Sanger, make the product-level argument most directly. "I think also a code editor should just be fun," one of them says, and they explain that they often throw out experiments that work but aren't fun, and that much of fun comes from speed [2]. That is a striking design rule: a feature that functions correctly can still be cut if it feels slow.

Tab and the "zero-entropy" keystroke

Their core feature, Cursor Tab, extends autocomplete from predicting characters to predicting your next edit, the next place you'll jump, even a change in another file or a terminal command [2]. Arvid describes the goal as making all the zero-entropy actions, where your intent is already clear but you'd still have to type, happen with a press of Tab [2]. Internally, the competition was how many tabs they could make you press in a row [2]. Aman adds that code has lower loss per byte than natural language, meaning many characters are highly predictable [2].

That goal only works if the prediction appears instantly. Michael describes the ideal as a fast colleague looking over your shoulder who predicts what you'll do next [2]. A colleague who takes several seconds to finish your sentence is not helping. So, the team says, making this work requires tiny, low-latency models that read long prompts but produce few tokens, which led them to a sparse mixture-of-experts model and the technique they call speculative edits [2].

The KV cache: not optional

The team is explicit that caching is essential: without reusing the KV cache across keystrokes, they would both destroy latency and overload their GPUs [2]. Aman explains the KV cache as storing the keys and values of previous tokens so the model only needs a forward pass for the newest token [2]. Here speed and cost are the same problem: the cache that makes the product feel instant is also the cache that keeps the GPU bill survivable.

They layer several tricks on top. Cache warming loads your current file into the KV cache while you're still typing, so there is little to compute when you press enter, reducing time to first token [2]. Cursor Tab also speculatively computes the next suggestion as if you had accepted the current one, so it is waiting when you press Tab [2]. Aman then covers attention variants that shrink the KV cache, from multi-head to grouped-query and multi-query attention, and DeepSeek's multi-head latent attention, which compresses keys and values into a latent vector [2]. Smaller caches mean more cache hits, bigger prompts and larger batches without slowing token generation [2].

Speculative edits

Aman's explanation of speculative edits shows how a latency goal can produce genuine invention. Normal speculative decoding uses a small model to draft tokens that a large model verifies, exploiting the fact that processing many tokens at once is faster than generating one at a time [2]. For edits, the team realized the best draft is the original code itself, so Cursor feeds chunks of existing code back and the model agrees with them in parallel until it reaches the part it wants to change [2]. The result streams much faster and lets people start reviewing before the edit finishes [2]. The speed gain changes the workflow: the user is reading while the model is still writing.

Splitting work across models for speed and cost

Cursor runs an ensemble of custom models alongside frontier models [2]. Frontier models are good at sketching a change but surprisingly bad at producing exact diffs, so Cursor lets the big model sketch and a separate trained model apply it, which also saves the expensive model's tokens [2]. Simple deterministic matching, they say, fails at least 40% of the time [2]. They imagine a future where smarter models plan and cheaper ones implement [2]. Latency-conscious architecture here means putting the fast, cheap model on the hot path.

Fast infrastructure beyond the model

The same thinking reaches their indexing. To keep local and server state in sync without hammering users' Wi-Fi or reading a roughly 20-terabyte database every second, Cursor uses a Merkle tree: compare a single root hash and only descend into folders whose hashes differ [2]. Embeddings are cached by chunk hash so teammates on the same codebase don't pay to embed it again [2]. Context itself is treated as a latency budget: more context makes requests slower and more expensive and can confuse models, so the bar for relevance should be high [2]. Their prompt system, Preempt, renders prompts with priorities to fit the available context window, giving the line where your cursor sits the highest priority [2].

Speed also explains some of what they don't ship. o1 isn't part of the default Cursor experience yet, in part because it doesn't stream [2]. And while they think agents are really cool, they believe most programming is iterative, so you mostly want instant results to react to [2]. For the Cursor team, a capability that arrives slowly is a different, lesser product.

Speed as the future of programming

The founders tie all of this to a larger vision. Michael says the team is excited about a future that emphasizes the programmer's speed, agency and control, not one where you chat with a bot like an engineering department over Slack [2]. Arvid describes a Node.js migration that took him and Sualeh five days and could someday take ten minutes, which would let programmers try things first instead of planning everything at the blackboard [2]. He adds that pressing Tab is really injecting intent, and that programming will become higher- and higher-bandwidth communication with the computer [2]. Fridman reads from Cursor's manifesto about a hybrid human-AI engineer who will "iterate at the speed of their judgment." [2] In this framing, latency is the gap between a thought and its result, and shrinking it changes how people think.

They also apply speed to their own company. On competing with Microsoft, they argue that being even a few months ahead matters enormously: "I think the cursor a year from now will need to make the cursor of today look obsolete." [2] Michael warns that "if you stop innovating on the product you will you will lose." [2]

Perplexity: latency as a founder's habit

Aravind Srinivas, co-founder and CEO of Perplexity, frames speed as a lesson inherited from great founders, and as something measured relentlessly. He describes his approach as an "ensemble algorithm" over many founders [3]. From Larry Page he took several things, including Page's obsession with latency, including testing Chrome on old laptops [3]. Srinivas applies the same habit: he tests Perplexity on in-flight Wi-Fi and benchmarks it against ChatGPT and Gemini, sweating details like the cursor being ready in the search bar and how quickly the mobile keyboard appears [3].

Those details are tiny, which is exactly the point. Whether the cursor is already blinking in the search bar, or how fast the phone keyboard pops up, is measured in fractions of a second. Srinivas treats them as product decisions, not polish.

Tail latency at every component

On the engineering side, he borrows Google's idea of tail latency: track P90 and P99 latency at every component, plus time to first token and throughput for the LLM [3]. He works with Nvidia, an investor, on TensorRT-LLM kernels [3]. Measuring the slowest 10% and 1% of requests, rather than the average, reflects the view that users remember the slow moments.

Indexing: fresh, fast and good enough

Perplexity's latency work sits inside a larger system of crawling, indexing and ranking. The crawler, PerplexityBot, must decide which URLs and domains to crawl and how often, render JavaScript-heavy pages, and respect robots.txt and crawl delays [3]. Srinivas explains that pure vector embeddings are not enough; old-school BM25, a refined version of TF-IDF, still beats many embeddings on ranking benchmarks [3]. Real search needs a hybrid of term-based retrieval, n-grams, embeddings and signals like domain authority and recency [3].

He also explains why this pipeline has to scale: much of the work is spotting failure cases that represent broad classes of mistakes, since per-query fixes don't scale from 10,000 to 10 million queries a day [3]. And one useful slack in the system: LLMs tolerate imperfect ranking, because they can find the right link even if it's ninth [3]. Freshness of the index is also tied to quality, since stale or thin snippets are one of four named causes of hallucination [3].

Speed and "the user is never wrong"

Srinivas pairs the latency obsession with another Page principle: "the user is never wrong." [3] He tells how his mother, who isn't fluent in English, complained that answers weren't relevant; his first instinct was to blame her query, until he realized the product should understand her [3]. He thinks prompt engineering won't last: "a better product should be one that allows you to be more lazy, not less." [3] Latency and laziness are two faces of the same idea: the product should absorb effort, whether that effort is waiting or phrasing.

He is candid about where his product is not the fastest. A live answer Perplexity gave during the conversation conceded that Google is still faster and better for navigational queries and real-time information such as sports scores [3].

Speed from not running your own servers

Srinivas also describes a kind of organizational speed. Perplexity runs on AWS rather than its own data centers, which he says frees the team from unnecessary problems and makes hiring easier; GPUs and models still require discrete capacity decisions [3]. The Cursor team makes the same choice: Sualeh says "AWS is just really really good," and that if something goes wrong, "it's probably your problem." [2]

Where They Agree, and Where They Don't

Agreement: speed is designed in, not added later

All three sources describe performance as something that shapes architecture from the beginning. dav1d's motto "every cycle matters" defined a decoder that is mostly assembly [1]. Cursor's need for instant suggestions drove its choice of tiny mixture-of-experts models, speculative edits and cache reuse [2]. Perplexity measures P90 and P99 at every component, not just end to end [3]. None of them treats speed as a final tuning pass.

Agreement: small details are real product quality

Srinivas sweats how quickly the mobile keyboard appears [3]. Cursor held internal competitions on how many tabs in a row a user would press [2] and throws out experiments that work but aren't fun [2]. The FFmpeg community built its own calling convention to avoid saving registers the standard way [1]. Each is a detail invisible to most users individually, but together they are what users feel.

Agreement: speed and cost are the same problem

Cursor says skipping the KV cache would both destroy latency and overload their GPUs [2], and its apply model saves the expensive model's tokens [2]. Perplexity's per-component tracking of throughput sits next to its latency tracking [3]. In video, each codec generation costs far more compute [1], so efficient decoding is what lets better compression reach ordinary machines. Across all three, efficient code is also cheaper code.

Agreement: fast must also be correct

Speed never excuses wrong answers. FFmpeg enforces bit-exactness through FATE across a huge matrix of platforms [1]. Cursor's speculative edits are verified by the model in parallel, and the team leans heavily on verification and review as models propose bigger changes [2]. Perplexity insists "you're not supposed to say anything that you don't retrieve," [3] even as it optimizes latency.

Where they differ: human craft versus machine assistance

The sharpest difference is about who does the optimizing. Kempf and Kunhya say compiler auto-vectorization can't match handwritten assembly, that the gap is multiples [1], that current LLMs struggle with this assembly for lack of training data [1], and that low-level optimization can't simply be vibe-coded [1]. They see speed as a human craft that grows more valuable as hardware gains slow.

The Cursor team, by contrast, builds its speed with machine-learning techniques and hopes AI will speed up engineering itself. Michael argues engineering talent is the real bottleneck in research, and that making it 10 times easier to turn a new architecture idea into efficient GPU code would speed research enormously [2]. Both sides actually agree that turning ideas into efficient low-level code is hard and valuable; they differ on whether AI will soon help. FFmpeg speaks from what current LLMs can do with assembly today [1]; Cursor speaks from a bet on where models are going [2].

Where they differ: what "speed" means

The sources use speed in different senses. For FFmpeg it is mostly throughput per CPU cycle [1], plus, in Kyber's case, end-to-end latency in milliseconds [1]. For Cursor it is time to first token and the feel of instant suggestions [2], and also company speed: staying months ahead [2]. For Perplexity it is tail latency at every component and the responsiveness of small UI details [3]. This is less a disagreement than a reminder that each team picks the kind of speed its users notice.

Where they differ: when slower is acceptable

The sources also show that speed is not the only value. Cursor thinks agents are really cool but not yet super useful for most tasks, precisely because users mostly want instant results to react to [2]; yet Arvid would still love to hand off a well-specified bug and come back to a verified fix [2], a case where waiting is fine. Srinivas imagines paying $10 million or $100 million for a single AI research job, such as a drug design breakthrough [3], where a slow, costly answer could be worth a great deal. And on long context windows he warns of a trade-off with instruction following [3], while Cursor notes more context makes requests slower and more expensive [2]. Speed is central, but these teams weigh it against depth.

Why It Works

Latency changes behavior, not just satisfaction

The Cursor team gives the clearest mechanism. Their target is zero-entropy keystrokes, where intent is already clear [2]. If the suggestion appears instantly, pressing Tab is cheaper than typing; if it lags, the user has already typed. Speed decides whether the feature is used at all. Speculative edits add a second effect: because output streams faster, people start reviewing before the edit finishes [2], so human and machine work overlap. And they describe the long-term payoff as higher-bandwidth communication with the computer [2], with programmers who can "iterate at the speed of their judgment." [2]

Reusing work is the fastest computation

Most of the techniques in the sources avoid doing work twice. The KV cache stores previous tokens' keys and values so only the newest token needs a forward pass [2]. Cache warming does the work before the user presses enter [2]. Cursor Tab computes the next suggestion in advance as if you had accepted the current one [2]. The Merkle tree compares one root hash and only descends into folders whose hashes differ [2]. Embeddings are cached by chunk hash [2]. Speculative edits use the original code as the draft [2]. FFmpeg's custom calling convention avoids saving registers the standard way [1]. Each is a way of not repeating effort.

Doing many things at once

SIMD assembly processes many pixels with a single instruction [1], which is how it reaches 10x to 50x over C [1]. Speculative decoding exploits the fact that processing many tokens at once is faster than generating one at a time [2]. Srinivas's history of AI makes a similar point at the research level: masked convolutions allowed fully parallel training, and the transformer combined attention with that parallelism [3]. He also notes that soft attention beat brute-force RNN translation results with far less compute [3]. Parallelism is a recurring source of speed.

Measuring the right thing

Perplexity tracks P90 and P99 rather than averages [3], so the worst experiences are visible. x264's lesson is the same in a different field: optimizing PSNR produced blurry images, while psychovisual metrics served human perception [1]. Srinivas benchmarks against ChatGPT and Gemini on in-flight Wi-Fi [3], and Page tested Chrome on old laptops [3], so they measure under conditions real users face. Cursor notes that more than 80% of its users are on Windows machines, many not powerful [2], which is part of why it does heavy work on servers.

Correctness makes aggressive speed safe

Bit-exact output and FATE's huge test matrix [1] let FFmpeg run 240,000-line assembly codebases across many architectures [1] without users seeing different results. Cursor's model agrees with existing code in parallel [2], so speculative edits speed up output without changing what the model would write. Perplexity's hybrid retrieval [3] and the LLM's tolerance for imperfect ranking [3] give it room to be fast without being wrong.

Why the value is rising

JB argues that as hardware gains slow and compute demands climb, low-level optimization becomes more valuable [1]. Each codec generation gives about 30% better compression at far more compute cost [1]. Srinivas describes the next frontier of AI as reasoning powered by inference compute, and worries about who can afford it [3]. Aman calls test-time compute a way around the looming data wall, running a smaller model longer on hard problems [2]. In each case, demand for compute is outrunning supply, which makes every saved cycle worth more.

Where It Applies, and Where It Doesn't

Where it applies

• Interactive tools where speed decides whether a feature is used. Cursor's Tab, which only makes sense if it beats typing [2], and Perplexity's search box and keyboard timing [3].
• Software running at huge scale. Home videos and trillion-dollar companies run on the same FFmpeg stack [1], and Perplexity's fixes must hold from 10,000 to 10 million queries a day [3]. Small gains multiply.
• Weak or varied hardware. Up to 45% of files need software decoding [1]; VLC still runs from Windows XP to Windows 11, from iOS 9 onward, and even on OS/2 [1]; many Cursor users are on modest Windows machines [2].
• Real-time control. Kyber's 4 ms glass-to-glass target for robots, drones and vehicles [1] is a case where latency is the whole product.
• Expensive compute. Where GPUs are the cost, as with Cursor's KV cache [2] and Perplexity's LLM throughput [3], speed work pays twice.

Where it doesn't, or where the sources add caveats

• Tasks where users are happy to wait. A well-specified bug handed off to an agent [2], or a very expensive AI research job [3], trades speed for depth.
• When speed would cost truth or quality. Perplexity won't say anything it didn't retrieve [3]; FFmpeg won't accept output that isn't bit-exact [1]; x264 chose visual quality over the easy metric [1].
• Rewrites for their own sake. JB, whose startup Kyber is written mostly in Rust, warns against rewrites, since reading code is far harder than writing it [1]. He also notes that pairing Rust with handwritten assembly breaks the safety model [1]. Kieran would welcome Rust in FFmpeg only if it meets the same testing and stability bar [1]. Performance and safety have to be weighed together.
• Infrastructure you don't need to own. Both Perplexity and Cursor run on AWS [3][2]; Srinivas says this frees the team from unnecessary problems [3]. Not every layer needs custom speed work.
• Where the sources are silent. None of the three sources discuss the classic warning about "premature optimization" directly, and none give data on how much latency changes user retention. The claim that speed drives love rests on the teams' own experience and design choices, such as Cursor's much of fun comes from speed [2], not on studies cited in the conversations.

What To Do With It

1. Write speed into the design principles. Adopt a short, explicit rule like dav1d's "every cycle matters" [1] or Cursor's practice of cutting experiments that work but aren't fun [2], so performance is judged alongside features from day one.

2. Measure tail latency at every component. Follow Perplexity: track P90 and P99 for each part of the system, plus time to first token and throughput for any LLM step [3], instead of watching averages.

3. Test on bad conditions and rivals. Try your product on in-flight Wi-Fi or old laptops, as Srinivas and Page did [3], benchmark it against competitors [3], and remember that many users run modest hardware [2][1].

4. Sweat the micro-interactions. Check whether the cursor is ready in the input box and how fast the keyboard appears [3]; aim to remove every "zero-entropy" action where the user's intent is already clear [2].

5. Stop recomputing. Cache aggressively: reuse state across user actions like Cursor's KV cache [2], warm caches while the user is still typing [2], precompute the likely next result [2], and use hashes (as with Cursor's Merkle tree and chunk-hash embedding cache) to skip unchanged work [2].

6. Put the cheap, fast component on the hot path. Let an expensive model plan and a small, trained model execute, as Cursor does with apply [2]; use tiny low-latency models for frequent predictions [2]; and keep context lean because more context is slower and costlier [2].

7. Find the draft that's already there. Look for structure you can reuse, as speculative edits reuse the original code as a draft and let users review while output streams [2].

8. Go low-level where the numbers justify it. For hot loops over large data, study SIMD; FFmpeg's asm-lessons need only high school math and C, especially pointers [1]. Don't assume the compiler will close the gap: FFmpeg's engineers say it is multiples, not a few percent [1].

9. Guard speed with exact tests. Before optimizing aggressively, build tests that demand identical output across platforms, as FFmpeg's bit-exactness and FATE matrix do [1], so faster never means different.

10. Optimize for what users perceive. Choose the metric that matches human experience, as x264 chose psychovisual quality over PSNR [1] and Perplexity chose "the user is never wrong" over blaming the query [3].

Sources

1. FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya: Codecs, Assembly, and the Volunteers Behind Internet Video · Lex Fridman Podcast
2. Cursor Team: Future of Programming with AI · Lex Fridman Podcast
3. Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet · Lex Fridman Podcast