Test obsessively: asserts, fuzzing and verification from FFmpeg to Cursor

Code only becomes trustworthy when checks are built in from the start and never switched off. Three conversations point to a layered protocol: exact reference outputs, asserts that stay on, randomized and simulated input, clearly labeled dangerous code, and suspicion of anyone, human or machine, who claims to have found or fixed a bug without proof.

Kieran Kunhya and Jean-Baptiste Kempf maintain FFmpeg and VLC, software that most internet video depends on. Their standard is bit-exactness: every implementation of a modern codec must produce exactly the same bits. That turns correctness from an opinion into a mechanical check. It is enforced by FATE, a volunteer-run test matrix across operating systems, compilers and chip architectures, and fed by real sample files gathered from the public. Kieran calls the lack of an exact definition in an older standard one of the video industry's biggest mistakes. Both men value Rust's safety goals but say a safer language does not replace testing.

ThePrimeagen, a former Netflix engineer, says he learns the most when a project reaches about 5,000 to 10,000 lines, where it stops fitting in your head. He adopted asserts that crash the program on an impossible state, in the style of the TigerBeetle database, which keeps at least two asserts per function switched on in production and runs simulation testing equal to hundreds of years of random input. Trying this on his own projects, he found bugs he says he would never have caught otherwise. With an AI coding tool he ended up introducing more bugs: "I wasn't co-piloting well. I was autopilotting much better."

The Cursor team builds AI coding tools and starts from humility: "it's hard to overstate how hard bugs are to avoid." They say frontier models are strikingly bad at finding bugs when simply asked, and lack a sense of which bugs matter. Their practices include commenting dangerous code on every line and testing AI changes in a hidden workspace before they touch a user's files.

They agree that correctness must be mechanical, that scale changes everything, and that AI finds bugs badly and reports them carelessly. Kieran describes AI-generated security reports landing on volunteers with deadlines, including one for an obscure 1990s game codec, though he notes Google later began sending patches.

They disagree on whether tests will disappear. Arvid of Cursor imagines models proving that code matches a written spec. His colleagues push back that intent is hard to specify formally, and that side effects and outside services resist proof. There is also a difference of emphasis: asserts crash on a program's own impossible states, while VLC was built to survive broken input from outside.

The approach applies most to code that handles untrusted input, lives a long time or has grown past a few thousand lines. It applies less to solo, fast-moving prototypes, and not every bug is a crisis.

What to do with it: define reference outputs and demand exact matches. Build an automated test matrix early. Collect real-world samples on purpose. Add asserts and leave them on. Run large-scale random input. Test against malicious requests. Label dangerous code, check AI changes in isolation, and rank bug reports by real severity.