Test obsessively: asserts, fuzzing and verification from FFmpeg to Cursor
FFmpeg and VLC maintainers describe obsessive testing and reverse engineering, ThePrimeagen swears by asserts past the 5,000-line mark, and the Cursor team explores AI bug finding and the limits of formal verification. Together they offer a practical case for building correctness checks into code from the start.
What does bit-exactness mean in FFmpeg and modern video codecs? Show answer
Every implementation of a codec must produce exactly the same output, bit for bit, as the reference. Correctness becomes a pass or fail comparison.
What is FATE? Show answer
FFmpeg's volunteer-hosted testing system, which runs tests across a large matrix of operating systems, compilers and CPU architectures.
Why do the FFmpeg maintainers collect real-world sample files? Show answer
Different files exercise different parts of a codec. A large, varied collection lets them test exact output across the full range of coding tools.
At what size does ThePrimeagen say projects teach him the most? Show answer
Around 5,000 to 10,000 lines of code, where early design choices either weigh you down or free you.
What kind of asserts did ThePrimeagen adopt, and what is special about them? Show answer
Asserts that crash the program when an impossible state occurs. Following TigerBeetle's practice, they stay on in production instead of being stripped out.
What is deterministic simulation testing, as described here? Show answer
Feeding a system reproducible random input on a huge scale. TigerBeetle runs the equivalent of hundreds of years of random input.
What happened when ThePrimeagen leaned on an AI coding tool? Show answer
Over time he introduced more bugs, because he was letting it run on autopilot. He found that reading code uses a different skill than writing it.
Why does the Cursor team say AI models are poor at finding bugs? Show answer
Real bug detection is rare in their training data, and they lack calibration about which bugs actually matter.
What is Cursor's shadow workspace? Show answer
A hidden instance of the editor where AI can edit code and get feedback such as type errors without touching the user's files.
What is Arvid Lunnemark's vision for replacing tests? Show answer
A model would suggest a spec for each function and a reasoning model would prove the implementation matches it.
What objection do his colleagues raise to formal verification? Show answer
Intent is often hard to specify formally, and side effects and external dependencies such as a payment service are hard to cover.
What complaint do the FFmpeg maintainers have about AI-generated security reports? Show answer
Reports arrived with deadlines and publicity before fixes existed, including for obscure code, pushing the cost onto volunteer maintainers.