Software 2.0: when learned models replace hand-written rules

The Short Version

In three separate fields the same thing happened: a system got better when engineers stopped writing its behavior by hand and started supplying data, an objective and training instead [1][2][3]. Andrej Karpathy calls this Software 2.0 and watched it happen at Tesla, where neural networks took over more and more of the Autopilot code because, in his words, "Humans are just not very good at writing software basically" [1]. George Hotz describes comma's bet as learning a human driving policy from data instead of hand-coding one [2]. Dylan Patel and Nathan Lambert describe DeepSeek's reasoning model, where reinforcement learning on problems with checkable answers made reasoning behavior emerge without anyone writing it in [3]. The engineering does not disappear; it moves to data quality, reward design, verification and the infrastructure that makes training possible, and the sources are clear about where learned systems still fall short [1][2][3].

Table of contents

What They Said

Andrej Karpathy: code written in weights, and the data engine

Karpathy, the former director of AI at Tesla, wrote the essay that named the idea. Its argument was that more and more code would be written not in a language like C++ but in the weights of a neural network [1]. Programming then becomes curating datasets and objectives, with training acting as the compiler [1]. He says the post was not well received at first [1].

He traces the same shift through the history of computer vision in three steps: hand-designed algorithms, then hand-designed features with a learned classifier on top, then fully learned convolutional networks [1]. Each step handed more of the work to training.

At Tesla he saw it inside one product. Autopilot began as mostly C++ with small networks that detected things in individual images [1]. Over time networks took over the harder jobs: fusing all eight cameras, predicting directly in three dimensions and tracking objects over time [1].

If the code is in the weights, the programmer's lever is the data, and Karpathy is specific about what it needs to be: very large, accurate and diverse [1]. Tesla gets three-dimensional ground truth from three sources: human labeling, simulation and an offline tracker that reconstructs a scene from all cameras using massive networks that could never run in the car [1]. He grew the annotation team from nothing to about a thousand people, giving humans the tasks they do well, such as labeling pixels in flat images, and leaving three-dimensional reconstruction to machines [1].

The data engine is the loop that ties this together: deploy the model, find the rare cases where it struggles, reconstruct the truth for those cases, add them to the training set and retrain [1]. He describes the result as a staircase of improvement [1]. He stresses that the hard part is execution rather than ideas, and that driving the system yourself gives an intuition that numbers and plots hide [1].

The same logic shaped the hardware. Tesla dropped radar and ultrasonic sensors, and Karpathy reframes sensors as liabilities: each brings supply chains, firmware, calibration and fusion code, and different sensor versions add noise to the data [1]. He credits the motto "best part is no part" [1]. Centimeter-accurate maps are, in his view, a massive dependency that will not scale globally [1]. On lidar he argues the real question is whether you have a fleet feeding a data engine [1].

He sees the pattern beyond driving. The transformer has become one architecture for video, images, speech and text, and he sums up the current mood as "the zeitgeist is do not touch the transformer, touch everything else," meaning keep the architecture fixed and scale the data and evaluation [1]. At that scale, he says, predicting the next word becomes massive multitasking: to predict well across the internet a model has to learn chemistry, physics and human nature [1]. When Tesla began its humanoid robot, the prototype came together in months because so much was copied from Autopilot, including the data engine, and he jokes that "this robot currently thinks it's a car" [1].

Karpathy also explains why the shift happens in jumps. Deep learning did not arrive by incrementally tuning hand-built vision features; it replaced them [1]. That is his example for the claim that "10x problems are not 10x hard," because a far more ambitious goal forces a fundamentally new approach [1].

George Hotz: learn the human policy, then let it practice

Hotz, the founder of comma.ai, defines the self-driving problem in one question: "how do you build a model that outputs a human policy for driving?" [2]. He divides the industry by its answer. Some companies, he says, are hand-coding what he calls quasi-human policies, and he names Waymo and Cruise; that is his description of competitors who do not speak in these conversations [2]. Then there is Tesla, and to an even greater extent comma, asking "how do we just learn the human policy from data?" [2]. He says comma solved that core problem years ago [2].

The next step goes further than imitation. Comma is building what amounts to a Drive GPT: a learned simulator, trained on driving data to predict the next frame given the driver's actions, inside which the company can run reinforcement learning [2]. Comma published a first version of the idea, a paper on learning a driving simulator, at its founding, with the machine learning tools of that time, and is returning to it with newer ones [2].

His choice of reward is the instructive part. The system is not rewarded for how human-like its driving is but for whether a human would take over, a disengagement [2]. He regards that as asking exactly the right question, which is what makes customers happy [2].

Hotz is modest about who leads. He says Tesla has always been about two years ahead and will probably stay there "because they're not doing anything wrong" [2]. Everyone, he suggests, knows roughly what needs to be done: "It's just a question who can actually build and deploy the system" [2]. Comma has about 5,000 daily active users and trains on its own compute cluster rather than the cloud [2].

Dylan Patel and Nathan Lambert: reinforcement learning's aha moment

Lambert, a research scientist, lays out how a modern language model is made [3]. Pre-training predicts the next token over trillions of tokens, mostly scraped from the web [3]. Post-training then adds instruction tuning, preference tuning such as reinforcement learning from human feedback, and the newest stage: reinforcement learning on verifiable problems, such as math with known answers or code with unit tests [3].

The aha moment is what DeepSeek did with that last stage. In the version called R1-Zero, the lab applied large-scale reinforcement learning on verifiable questions directly to the base model, with no human preference data, and reasoning behaviors such as "wait, let me check this" emerged on their own [3]. Nobody wrote a rule telling the model to check its work.

Lex Fridman reads a reflection from Karpathy on the two ways systems learn: imitation, which covers pre-training and fine-tuning, and trial and error, which is reinforcement learning [3]. Of the two, Karpathy wrote that "two is significantly more powerful" and "the source of all magic is always two," as when AlphaGo defeated the champion Lee Sedol [3]. Lambert adds that AlphaZero became far stronger by removing human data altogether [3].

Lambert connects this to Richard Sutton's bitter lesson, which favors simple, scalable methods over clever human priors [3]. He notes that even a one-billion-parameter model's grade-school math improves quickly with this training, and his own institute's open recipe uses reinforcement learning with verifiable rewards [3].

Patel looks ahead. He expects the real breakthrough moment for language models to come from computer use and robotics, because models are sample-inefficient and learn best through self-play in verifiable environments, such as sandboxes where an agent tries to create accounts or make money [3]. He expects software engineering to be the first domain where agents succeed, because code is verifiable; scores on one coding benchmark rose from about 4% to 60% in a year [3].

They also keep the picture honest about what DeepSeek's engineers did by hand. The lab's efficiency came from a mixture-of-experts design with a new routing mechanism, a memory-saving form of attention and programming its restricted chips at a very low level; Patel's comment is "necessity is the mother of innovation" [3]. Lambert suggests, in the spirit of the bitter lesson, that some of these tricks may matter less over time [3]. And Lambert states the Software 2.0 rule in his own words: "the data processing, data filtering, data quality is the number one determinant of the model quality" [3].

Where They Agree, and Where They Don't

All three agree on the direction. Hand-written behavior gives way to learned behavior, whether the behavior is seeing a road, steering a car or reasoning through a math problem [1][2][3]. Hotz's account of Tesla matches Karpathy's from the inside [1][2].

They agree that data is where the work goes. Karpathy built a thousand-person labeling operation and a loop for finding hard cases [1]. Lambert puts data quality first among everything that determines a model [3]. Hotz's simulator is trained on driving data [2].

They agree that ideas are not the bottleneck. Karpathy says the hard part is execution [1]. Hotz says it is a question of who can build and deploy [2]. Patel and Lambert describe the many failed runs, compute and research behind one successful training run, and say research and ablations typically take two to four times the compute of the final run [3].

The differences are about how far to take it.

Imitate humans, or go beyond them? Tesla's data engine, as Karpathy describes it, teaches networks from labeled ground truth, much of it produced or checked by people [1]. Hotz starts from imitation of human driving and then adds reinforcement learning in a learned simulator [2]. The DeepSeek and AlphaZero examples go furthest, improving by removing human data [3]. The reason for the difference is the reward. Math, code and board games have answers a machine can check, so trial and error can run without people [3]. Driving has no such automatic check, which is why Hotz has to choose a stand-in, the disengagement, and why Tesla needs reconstructed ground truth [1][2].

Can you skip the imitation step? Karpathy's own experience says no. An early project to let networks use a keyboard and mouse failed because it trained from scratch with reinforcement learning, and a randomly initialized agent never stumbled onto a correct booking by chance [1]. The idea is back now that a system can start from a pretrained language model that already knows what a submit button is [1]. DeepSeek's result fits this: its reinforcement learning was applied to a pretrained base model, not to a blank one [3]. So the claim that trial and error is the more powerful kind of learning comes with a condition that the same sources supply [1][3].

How much human cleverness should remain? Karpathy removes parts: sensors, maps, architectural variety [1]. The bitter lesson says the same about human priors [3]. Yet Patel credits DeepSeek's success to intricate hand engineering below the level of the model [3]. These fit together if the cleverness has moved: away from writing the behavior and toward making training cheaper and bigger.

How close is the payoff? Hotz says the core problem was solved years ago [2]. Karpathy declines to give a date for full self-driving, since no one has built it before, though he calls the problem tractable [1]. On broader AI, Lambert says general intelligence is probably after 2030 and discounts shorter forecasts from company leaders, while Patel expects the capabilities around 2027 or 2028 with cost slowing deployment [3].

Is AI-written code itself ready? This is Software 2.0 applied to programming. Karpathy loves his coding assistant but warns it can introduce subtle off-by-one bugs and must be supervised [1]. Hotz, at the time he spoke, used none and said he wanted a tool that spots bugs instead of one that writes nearly correct code [2]. Patel expects coding agents to succeed first because code can be verified [3].

Why It Works

The problem is too large to write down. Karpathy's explanation for Tesla's shift is blunt: people are not good at writing software for something like perception, and driving is hard mainly because of the rare tail of cases [1]. A hand-written rule covers the cases its author imagined. A data engine goes and finds the cases nobody imagined and adds them to the training set [1].

A hard objective forces a general solution. A neural network, Karpathy says, is a fairly simple mathematical expression with many adjustable knobs, and when it is large and trained on a hard enough problem it is forced to find interesting solutions [1]. He calls the result "a very complicated alien artifact" [1].

One general, trainable architecture. He describes the transformer as a general-purpose computer that is expressive, optimizable by gradient descent and efficient on parallel hardware, which is why effort can go into data instead of redesigning the model for each task [1].

Fewer parts, cleaner data. Every extra sensor or map adds code to maintain and inconsistency in the data; removing it concentrates effort on one input and speeds progress [1].

Verifiable rewards allow practice at scale. Where an answer can be checked automatically, a model can try, fail and improve without a human in the loop, and useful behaviors appear that no one specified [3]. Patel's point that models learn best through self-play in verifiable environments explains why math and code moved first [3].

The right reward keeps the system honest. Hotz's disengagement signal ties training to what the product is for instead of to a proxy like resemblance to human driving [2].

A loop, not a release. Karpathy's staircase and Hotz's emphasis on deployment are the same mechanism: a system in real use produces the failures that become its next training data [1][2].

The evidence in these conversations is mostly first-hand engineering experience: Tesla's progress over five years as Karpathy saw it, comma's product in the hands of its users, and DeepSeek's published results as read by two analysts [1][2][3]. None of the three offers a controlled comparison of hand-coded and learned systems on the same task.

Where It Applies, and Where It Doesn't

It applies where three things are available: a lot of data, an objective that can be stated, and a way to check results. Vision, driving, next-word prediction, math and code all have them in some form [1][2][3].

It applies less well in the following cases.

• When nothing verifies the answer. Reinforcement learning's recent gains are on verifiable problems, and Patel expects progress to come first in narrow domains [3].
• When a task has many steps. Patel compares chained tasks to manufacturing yields: high reliability per step still compounds into failure over many steps, and Lambert calls the talk around agents overblown [3].
• When the starting point is blank. Trial and error from scratch failed in Karpathy's early project [1].
• When subtle errors are costly. Learned systems produce plausible mistakes; Karpathy expects the hard problems ahead to be how to steer, audit and verify what AI writes [1].
• When the data carries what you do not want. Lambert and Patel describe bias that comes in with pre-training text and is hard to remove, and research showing that planted phrases in training data can trigger hidden behaviors [3].
• When you cannot afford it. The approach needs fleets, labelers, clusters and many failed runs; DeepSeek's famous cost figure covered only one final training run [1][3].

Hand-written rules also persist around learned models. Lambert separates behavior that comes from training from system-level interventions, and gives the example of an image-generation controversy that came from a prompt-rewriting layer, not from the model's weights [3]. Hotz, for all his commitment to a learned policy, argues that ordinary code should be as simple as possible and protected by tests he can trust [2]. Software 2.0 replaces some hand-written code; it does not remove the need for the rest to be good.

Two cautions on the sources. Karpathy's conversation is available here only in summary [1]. And each speaker describes a field at one moment: Karpathy shortly after leaving Tesla, Hotz while comma's simulator work was still under way, the DeepSeek analysts at the start of 2025 [1][2][3].

What To Do With It

1. Ask whether the behavior can be learned before writing more rules. If you are adding special cases to cover a long tail, that is the sign Karpathy saw in vision and driving [1].
2. Treat the dataset as the source code. Invest in data that is large, accurate and diverse, and put filtering and quality ahead of model tweaks [1][3].
3. Build the loop, not just the model. Deploy, find the cases where it struggles, establish the truth for them, add them to training and retrain [1].
4. Give people and machines the labeling jobs each does well. Humans for simple judgments on flat images, heavy offline models for reconstruction [1].
5. Choose a reward that matches what users care about. Measure whether a person would have to take over, not how closely the system resembles a person [2].
6. Start from a pretrained model, then add trial and error where answers can be checked. Use reinforcement learning on problems with known answers or tests, and do not begin from scratch [1][3].
7. Remove parts and dependencies that do not earn their place. Each extra input adds code and noise to the data [1].
8. Keep the architecture stable and scale the data and evaluation. Change everything else first [1].
9. Supervise and verify the output. Keep tests you trust around learned components and review AI-written code for subtle errors [1][2].
10. Use the system yourself. Direct use shows failures that metrics and plots hide [1].

Sources

1. Andrej Karpathy on Neural Nets, Tesla Autopilot, Optimus, Aliens and the Road to AGI · Lex Fridman Podcast
2. George Hotz on tiny corp, tinygrad, Open-Source AI Safety, Self-Driving, and Twitter · Lex Fridman Podcast
3. DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters · Lex Fridman Podcast