Software 2.0: when learned models replace hand-written rules
October 5, 2026 · 14 min read
Andrej Karpathy describes Software 2.0 and Tesla's data engine, George Hotz explains comma's learned driving policy, and the DeepSeek analysts describe reinforcement learning's aha moment. Each shows systems improving when engineers stop writing rules and let data and training do the work.
The rules, numbers and steps worth keeping in front of you.
- Software 2.0: behavior is learned from data and an objective instead of written by hand.
- Karpathy: training is the compiler; the dataset is the source code.
- Data must be very large, accurate and diverse.
- Data engine: deploy, find hard cases, establish the truth, add to training, retrain.
- Karpathy: every extra sensor or map adds code and noise; "best part is no part".
- Keep the architecture fixed and scale the data and evaluation.
- Hotz: learn the human driving policy from data, then practice in a learned simulator.
- Hotz's reward: whether a human would take over, not how human the driving looks.
- Patel and Lambert: reinforcement learning on checkable problems made reasoning emerge unprompted.
- Lambert: data filtering and quality are the top determinant of model quality.
- Start from a pretrained model; trial and error from scratch failed for Karpathy.
- Patel: reliability per step compounds into failure over many chained steps.
- Supervise AI-written code; it makes plausible, subtle mistakes.
- Limits: no way to check answers, costly errors, biased data, high cost.
This is the short form. The full insight has every expert's reasoning, their own words and the sources.