Learned or engineered? The split over how robots should get smart

The Short Version

The split is real, and narrower than the slogans suggest. Andrej Karpathy argues that more and more of a robot's software should be learned from data, because "Humans are just not very good at writing software basically." [3] Russ Tedrake uses deep learning heavily and still warns that it "makes it easy to get things to work without thinking as critically about it," and that some problems, contact above all, need models people understand [1]. Marc Raibert reports from the hardware side that the most impressive robot performances, including most of Atlas's, still come from traditional control, and expects the best results from combining the two [4]. Michael Littman, who lived through reinforcement learning's long stall, endorses the lesson that scale beats cleverness while doubting that the curve runs forever and insisting on how much human skill hides behind every learned system [2]. All four use learning; they disagree about how much a person still has to understand, and about how far lessons from driving and games carry over to machines that must touch things.

Table of contents

What They Said

Andrej Karpathy: let the data write the program

Karpathy's position has a name. His Software 2.0 essay argued that more and more code would be written not in C++ but in the weights of neural networks [3]. On that view programming becomes the work of curating datasets and objectives, and training plays the part of the compiler [3].

He supports it with the history of his own field. Computer vision went from hand-designed algorithms, to hand-designed features with a learned classifier on top, to convolutional networks that learn everything [3]. Each step removed something a person had written, and each step worked better [3].

Tesla's Autopilot followed the same path under his direction. It began as mostly C++ with small networks detecting things in individual images [3]. Over time the networks took over the job of fusing all eight cameras, predicting directly in three dimensions and tracking objects through time [3]. His explanation for why that was the right direction is the line "Humans are just not very good at writing software basically." [3]

What replaces the hand-written code is a process he calls the data engine. The key ingredient is data that is very large, accurate and diverse [3]. Tesla gets three-dimensional ground truth through human labeling, through simulation, and through an offline tracker that reconstructs a scene from all the cameras using networks far too large to run in a car [3]. The loop is: deploy the model, find the rare cases where it struggles, reconstruct what really happened, add those cases to the training set, retrain [3]. He describes the result as a staircase of improvement, and says the hard part is execution, not ideas [3].

Two details show how much engineering sits inside this learned approach. Karpathy grew the annotation team from nothing to about a thousand people, and he was careful to give humans the tasks they do well, like labeling pixels in flat images, while leaving three-dimensional reconstruction to machines [3]. And he treats the choice of sensors as a design decision with a strong prior toward fewer. Every sensor brings supply chains, firmware, calibration and fusion code, and adds noise to the data; he credits Elon Musk's motto that "best part is no part." [3] Vision, he argues, is necessary because the world is designed for human eyes and sufficient because humans drive with it [3].

His account of an early failure explains what he thinks learning needs. An OpenAI project called World of Bits tried to train agents to use a keyboard and mouse from scratch with reinforcement learning, and it came too early: a randomly initialized agent never stumbled onto a correct booking by chance [3]. The idea returned once agents could start from a pretrained language model that already knows what a booking or a submit button is [3]. Learning worked when it began from a large store of prior knowledge, not from nothing [3].

On robots proper, Karpathy thinks humanoids are the right bet because the physical world is designed for the human form, and building one general machine is more practical than a special one for each task [3]. Tesla's Optimus prototype came together in months because so much was copied from Autopilot: the computer vision, the data engine and the offline trackers [3]. He jokes that "this robot currently thinks it's a car." [3]

He also sees Optimus as a hedge. If internet data is not enough and intelligence requires acting in the physical world, millions of humanoid robots interacting with people might be the path [3]. He is unsure that text alone suffices, since people rarely write down obvious physical facts [3]. And he supplies the reason hard problems reward a change of method: "10x problems are not 10x hard," because they force a fundamentally new approach, as deep learning did when it replaced incremental tuning of hand-built vision features [3].

Russ Tedrake: keep the clear thinking

Tedrake, an MIT professor who leads robotics research at the Toyota Research Institute, is not an opponent of learning. "deep learning is amazing," he says, he uses it heavily, and "I think it's changed the world unquestionable." [1] His concern is what its ease does to the people using it. "It makes it easy to get things to work without thinking as critically about it." [1] Students, he says, are not immediately rewarded for going through more rigorous approaches [1]. He thinks people underestimate "maybe the power of clear thinking," [1].

He knows the strongest form of the opposing argument and states it himself: there is a case that you must use learning even for the parts we think we already understand, because anything a human has touched carries the bias and the bottleneck of that human's mental model [1]. And he allows that for a particular problem "it might be the case that data should tell us the story" [1].

His counterexample is deliberately plain. Inverting a matrix is something we know how to do well, even for a large one. You could train a deep network to do it, he says, "but you shouldn't probably" [1]. Rigor, in his sense, includes knowing the scope and limits of each tool [1].

His deeper argument is about understanding. He asks "what if Newton and Galileo had deep learning" and, after their experiments, had simply handed the world the weights of a neural network [1]. He does not think we would be as far along [1]. A network can be trained to predict what a physical law predicts, but he still wants "another Newton to come along," because there is more to do in finding simple models for complicated tasks [1]. He is optimistic that learning is a route to that. The condition is that "the representation matters" and an arbitrarily complex function from inputs to outputs may not be the end goal [1]. He adds a lesson the field has learned: "simple doesn't mean low dimensional" [1].

Contact is his main case for why this matters in robotics. Classical control grew up stabilizing fighter jets and power grids, systems in which small changes produce small effects [1]. Touching the world is not like that. Making or breaking contact, or switching from sticking to sliding, changes the system abruptly [1]. Rigid-body assumptions even produce paradoxes: a three-legged stool at rest has uniquely determined forces in its legs, a perfectly made four-legged table does not, and because friction depends on those forces, the laws cannot say which way the table will slide when pushed [1]. Every configuration of contacts looks like a separate system, and the number of them explodes. A flat foot alone has four; listing every order in which a hand might touch a water bottle is hopeless [1].

The result, he says, is a field that avoids touching. "we mostly in robotics are afraid of contact on the rest of our body, which is crazy," he says, pointing to the whole discipline of collision-free motion planning, while people use incidental contact all the time [1]. Robots that are to work near people "have to be not afraid of touching the world" [1].

A second strand of his view is that intelligence in movement is not all computation. His favorite demonstrations are passive dynamic walkers, machines with no controllers and no batteries that walk down a ramp powered by gravity, and a dead trout that, placed behind a cylinder in flowing water, "basically starts swimming upstream." [1] "Completely dead. No brain, no motors, no control." [1] Conventional control, he says, tries to "erase the dynamics of the world" and impose simple ones with motors; the alternative is "letting the world push us around and we go with the flow." [1] His course on underactuated robotics rests on the idea that "the philosophy there is let physics do more of the work," [1].

None of this makes him a defender of hand-written plans. His own team's famous fall at the DARPA Robotics Challenge came when one unticked box on a checklist left a controller running, and the robot ended up in a situation no one had anticipated: "nothing in our plans had ever told us what to do if you're butts on the seat and your feet are in the air." [1] The lesson he took was about testing. With one robot running constantly they still saw new failures days before the competition, and "We just didn't have enough hours in the day to test that robot." [1] The winning team had two robots and tested with more discipline [1].

So the problem he now emphasizes is validation: how to take a system with deep-learning perception, complex dynamics, controllers and planners and gain enough confidence to put it in anybody's kitchen [1]. His group builds algorithms that hunt for the rare situations where a controller fails; "We joke and we call it the black swan generator." [1] They also pursue fleet learning, on the reasoning that one robot cannot visit all its states in a lifetime but many similar robots together might [1]. And he wants to "bring some of the clear thinking of controls into these complex systems we're building for robots" [1].

Even perception, for him, leads back to representation. "what's the state of my shirt?" he asks [1]. Chopping a carrot turns one object into a hundred pieces, and a controller does not need the position of each [1]. The right state is a task-relevant latent representation, not necessarily low dimensional but easy to reason about [1].

Michael Littman: the bitter lesson, with reservations

Littman, a reinforcement learning researcher at Brown, takes up Rich Sutton's Bitter Lesson, which Lex Fridman summarizes as simple algorithms that leverage computation winning over decades of clever tricks [2]. Littman's first response is a joke with a point: machine learning is really a subfield of computer architecture, waiting for the hardware people to do their job [2]. He recalls a line attributed to Fred Jelinek that every time they fired a linguist, performance went up [2].

So he grants the lesson. His reservations are about where it leads. He doubts the exponential lasts: there is a counter-Moore's law in which the cost of developing chips also doubles each generation, and he suspects we are in "a sigmoid curve on the front half which looks a lot like an exponential" [2].

His history of the field is a caution against reading too much into a single success. Gerry Tesauro's TD-Gammon learned backgammon by playing itself, and people extrapolated that many other hard problems would fall [2]. Few did. Littman's students tried generation after generation to make neural networks approximate value functions and kept failing [2]. His conclusion: "I now believe that Jerry is a neural net whisperer" and we discount the human skill behind such results [2]. A couple of promising real-world demonstrations appeared, neural networks controlling a bank of elevators and managing cell phone handoffs between towers, and none reached production as far as he knows [2]. Neural networks then fell out of favor for more than a decade [2].

He applies the same reading to the modern triumphs. He says "alpha go knock my socks off" mainly for the engineering, the integration of many ideas into one giant system, with David Silver as another neural net whisperer [2]. When Fridman raises the remark that AlphaZero's ceiling has not been found, Littman answers "it's not getting better at other things it's getting better at go" [2].

Littman also defends a kind of rigor that pure empiricism can skip. Worst-case guarantees matter, he argues, because they give modularity: something that only works on a particular distribution of data can break when it is plugged into a larger system that shifts the distribution [2]. That is a concern for any machine assembled from learned parts [2].

He names two things learning from data has not delivered. One is transfer: his reinforcement learning agents start from scratch at each new level of a video game, while people keep building on what they know [2]. The other is social understanding. Teaching his children to drive, he found that they began far ahead of any neural network, already understanding roads and steering, and that the hard part was something else: "the degree to which driving is a social interaction activity" [2]. One son had to be taught explicitly to think about what other drivers were thinking [2]. On language models he makes a related point: learning from what people have written is not enough, and "I think you really need to be pushed back at" by people to discover nuance [2].

Marc Raibert: athletic intelligence first, then cognition

Raibert, founder of Boston Dynamics and now head of the AI Institute, divides intelligence in two. Athletic intelligence is the mechanical design, real-time control and energetics that let a machine balance, run and flip [4]. Cognitive intelligence is planning of the kind that gets a person out of the house at nine for a 9:30 meeting [4]. Most robots, he says, are pretty dumb, which is why skilled people spend a long time programming everything they do [4].

His career is the case for what engineering can do. The control recipe for his first hopping robots had three parts: regulate the bounce by adding or removing energy, place the foot in the right spot relative to the center of mass, and keep the body upright by applying torque while the foot is on the ground [4]. The natural look of his later machines comes, he says, partly from good hardware and partly from predictive control that looks ahead instead of reacting, about a second or two for Spot, recalculated every tenth of a second, and much further for a somersault [4]. He thinks people who believe hardware innovation is finished are wrong [4].

He shares Tedrake's view of the body. He cites the same passive dynamic walkers as evidence that the body participates in motion and that good designs account for what the machine naturally wants to do [4]. And he shares Tedrake's complaint about timidity: after 50 years of careful geometric grasping, robots are nowhere near human dexterity, while people juggle objects in their hands [4].

On learning, he reports the optimism since ChatGPT and gives a reason for caution: physical robotics differs from language because pixels are not like word tokens [4]. His institute has several people working on learning and opened a Zurich office led by Marco Hutter, a leader in reinforcement learning for robots [4]. But he notes that the most impressive performances, including most of Atlas's, still come from so-called traditional control such as model predictive control, and he expects the best results to come from combining the two [4].

The institute's projects show where he thinks learning belongs. In one, watch-understand-do, a robot watches a person perform a task, segments it into components, maps them to skills and then does it, starting with simple bicycle repair [4]. In another, inspect-diagnose-fix, a robot would do what a repair visit does; Spot already collects thermal readings, gauge data and sounds, but interpretation and repair are not yet done by robots [4]. He organizes the work as "stepping stones to moonshots," never going more than about a year without a tangible result [4].

He does not dismiss the learned-first competitors. He admires Musk as a technologist and would not count him out, while saying he does not think Optimus is where Atlas is today [4].

Where They Agree, and Where They Don't

They agree that learning is part of the answer. Tedrake uses deep learning heavily and pursues fleet learning [1]. Raibert has hired for reinforcement learning [4]. Littman built a career on it [2]. Nobody here argues for hand-coding everything.

They agree that rare cases decide whether a robot can be trusted. Karpathy's data engine exists to find the rare cases where the model struggles, and he calls driving hard mainly because of its tail of unusual situations [3]. Tedrake builds a generator of rare failures and wants fleets of robots to cover states a single machine never reaches [1]. Littman worries about parts that fail when the data shifts [2]. They differ on the remedy, more data from more machines in Karpathy's case, deliberate search and analysis in Tedrake's, and those remedies are compatible [1][3].

They agree, mostly, on the human form. Tedrake says the most compelling argument for humanoids is that "the world is built for the human form," though he thinks that may fade as environments are built for robots [1]. Karpathy makes the same argument for Optimus [3].

They disagree on who should write the program. Karpathy's line that humans are not good at writing software and Tedrake's that you should not train a network to invert a matrix are the two ends [1][3]. Karpathy reasons from a history in which each removal of human design improved results [3]. Tedrake reasons from the value of knowing why something works and from the cases where mature methods already do [1]. Raibert stands with the evidence from his own machines: so far, the best physical performances are engineered [4].

They disagree on hardware. Karpathy treats each additional sensor as a liability and argues that vision is enough [3]. Tedrake says robots are built wrong and need soft surfaces and tactile sensing, partly because the hand blocks the cameras at the very moment of contact: "I think we're building robots wrong and being more soft and more contact is important." [1] Raibert says hardware innovation is far from finished [4]. The gap follows the task. Karpathy's argument was made about driving, where the question was which sensors a car needs; Tedrake's is about manipulation [1][3].

They disagree on whether the body is part of the intelligence. Tedrake and Raibert both say it is, and both point to machines that walk with no computer [1][4]. Karpathy describes a neural network as mathematics, "a very complicated alien artifact," and locates intelligence in what is learned [3]. He does, however, allow that intelligence may require acting in the physical world, which is why he calls the humanoid a hedge [3].

They disagree on how far scale will go. Karpathy's instinct is to keep the architecture fixed and scale the data [3]. Littman suspects a sigmoid [2]. Raibert doubts that the language recipe transfers directly, because pixels are not tokens [4].

They differ on how much credit the method deserves. Littman's neural net whisperers are a reminder that learned systems succeed in the hands of particular skilled people [2]. Karpathy does not really dispute this: he says the hard part of the data engine is execution, and he built a thousand-person labeling operation to feed it [3].

Why It Works

The sources explain why learning has delivered in some places and not yet in others [1][2][3][4].

Why learning won in perception. Hand-written vision features were repeatedly beaten by learned ones, and at Tesla networks took over tasks, fusing cameras and tracking through time, that people had been coding by hand [3]. Large, accurate, diverse data did what explicit rules could not [3].

Why prior knowledge matters. Agents trained from scratch never found a reward by chance; agents starting from a pretrained model could [3]. Littman's children started driving far ahead of a network for the same reason: they already understood the world [2].

Why contact resists both approaches. Control theory assumes smoothness, and contact is discontinuous [1]. The physics itself becomes ambiguous under rigid-body assumptions, and the number of contact configurations grows combinatorially [1]. Tedrake's point about representation applies here: the difficulty is in how the problem is described, so a method that does not find a good description inherits the explosion [1].

Why engineered control still leads in athletic tasks. Raibert's machines succeed through hardware that suits the motion and controllers that predict the next second or two using a model of the machine [4]. Physics supplies much of the behavior, as the passive walkers and the trout show, which leaves less for a controller, learned or written, to do [1][4].

Why pure plans fail. The MIT fall shows the weakness of the engineered side: a plan covers the situations its authors thought of [1]. That is the gap learning from many machines and many trials is meant to fill, and Tedrake's response, more testing, rare-event search and fleets, moves in Karpathy's direction [1][3].

Why data from the physical world is expensive. A real robot cannot run the same experiment twice, one robot cannot run enough hours, and Raibert's team needed 109 filmed attempts over about six weeks for one stair-climbing video [1][4]. Littman notes a parallel in language: interaction with people is costly, so it cannot supply data at the scale self-play does [2]. Tesla's answer was a fleet of cars already on the road; Karpathy's hedge imagines the same with millions of humanoids [3].

Why understanding still pays. Guarantees let parts be combined safely, in Littman's argument [2]. Simple explanations let knowledge build, in Tedrake's [1]. Karpathy himself insists on building concepts from scratch when he teaches, on the ground that "The source of truth is the code," [3].

Where It Applies, and Where It Doesn't

Driving is not manipulation. Karpathy's evidence comes mainly from a car on a road, a problem he says is hard because of rare cases and predicting other people [3]. Tedrake's hard problem is touching and handling objects [1]. Success in the first does not settle the second, and Karpathy offers the humanoid as a bet, not a result [3].

Games are not the world. Littman's caution about self-play applies: a system that improves without limit at Go is improving at Go [2].

Mature methods should stay. Where a well-understood solution exists, Tedrake sees no reason to replace it with a trained one [1].

The engineered record has its own limits. Raibert's robots are athletic and, by his own account, not smart; people program what they do [4]. Fifty years of careful grasping have not produced dexterity [4].

The picture is a snapshot. Each speaker describes the field as it stood when he spoke, and Raibert's statement about traditional control is explicitly about performances so far [4].

No one claims a finished answer. Tedrake wants another Newton [1]. Raibert expects a combination [4]. Karpathy will not give a date for full self-driving [3]. Littman once predicted a gradual rollout of autonomous cars and says he was wrong about what would happen [2].

What To Do With It

1. Ask what kind of problem it is. Perception from cameras has rewarded learning at scale; planning through contact has not yet [1][3].

2. Do not learn what is already known. Use the mature method where one exists, as with Tedrake's matrix [1].

3. Start learning from prior knowledge. Karpathy's World of Bits story and Littman's driving lessons both say that learning from nothing is the hard way [2][3].

4. Build the loop, not just the model. Deploy, find the failures, recover the truth, retrain; Karpathy says the difficulty is in executing this, not in the idea [3].

5. Hunt for the rare failure on purpose. Search for the cases where the controller breaks, and test on more than one machine [1].

6. Let the body do some of the work. Design hardware for the motion, and consider soft surfaces and touch where the task involves contact [1][4].

7. Ask for guarantees where parts are combined. A component that only works on its training distribution may fail inside a larger system [2].

8. Be wary of extrapolating from one success. TD-Gammon did not generalize for decades, and exponentials can be the front half of an S-curve [2].

9. Plan in stepping stones. Raibert's rule is a tangible result at least once a year on the way to a larger goal; Karpathy gives similar advice about earning revenue along the way [3][4].

10. Expect a combination. The roboticist with the longest record of working machines expects the best results from joining learned and engineered methods [4].

Sources

1. Russ Tedrake on Underactuated Robotics, Contact, Touch and the DARPA Robotics Challenge · Lex Fridman Podcast
2. Michael Littman on Reinforcement Learning, AI Risk Skepticism, and the Future of AI · Lex Fridman Podcast
3. Andrej Karpathy on Neural Nets, Tesla Autopilot, Optimus, Aliens and the Road to AGI · Lex Fridman Podcast
4. Marc Raibert: Boston Dynamics and the Future of Robotics · Lex Fridman Podcast