Why robots still cannot do the dishes: contact, cost and generality

The Short Version

Robots still cannot do the dishes because handling objects means making and breaking contact with the world, and contact is where today's control methods, sensing and training data are weakest [1][2][3]. Russ Tedrake says most robots are "afraid" of touching anything except with their feet or fingertips, and that the mathematics of which surfaces touch when explodes into more cases than anyone can enumerate [1]. Robert Playter and Marc Raibert, who built the most capable legged machines in the world at Boston Dynamics, add the economics: a general-purpose robot is perfect at nothing, costs a great deal to make reliable, and has to sell in the thousands to survive, which is why their robots work in factories and not kitchens [2][3]. Kate Darling says we are far behind where people think in physical robotics, and Fei-Fei Li explains why the recipe behind chatbots does not transfer: there is no internet-sized record of physical handling to learn from [4][5]. None of them says home robots are impossible, and Tedrake expects kitchen helpers before long, but all of them describe a slower and harder road than the one language models took [1][2][3][5].

Table of contents

What They Said

Russ Tedrake: contact is the hard part

Tedrake, an MIT professor and head of robotics research at the Toyota Research Institute, puts the problem in one observation. Most robots touch the world only with their feet or fingertips, and "we mostly in robotics are afraid of contact on the rest of our body, which is crazy." [1] There is a whole field of collision-free motion planning devoted to making sure a robot dances around the world without touching it, while a person who is pushed simply puts a hand out [1].

He explains why engineers avoid contact. Classical control grew up stabilizing fighter jets and power grids, systems where small changes produce small effects [1]. Contact is discontinuous: making or breaking it, or switching from sticking to sliding, changes the system [1]. The usual assumption that objects are perfectly rigid even produces paradoxes. A three-legged stool at rest has uniquely determined leg forces, but a perfectly machined four-legged table does not, and since friction depends on those forces, the equations cannot say which way the table will slide when pushed [1]. The result is a set of possible outcomes and not a single prediction [1].

Then the counting problem. Every contact configuration looks like a separate system, and the combinations explode [1]. A flat foot alone has four configurations, and enumerating every order in which a hand could touch a water bottle is hopeless [1]. The equations for a hand grasping an object blow up not because of the hand's motion but because of the combinatorics of which surfaces touch when [1]. When Lex Fridman asks about a robot planning quickly through rich contact, as a wrestler does, Tedrake grants there are impressive learned controllers and reasons for optimism, but says "the challenge you just offered there still eludes us" [1].

Household objects add a perception problem. He asks "what's the state of my shirt?" [1] Chopping carrots turns one object into a hundred pieces, and a controller cannot track the pose of each [1]. Manipulation is also underactuated, meaning the robot has no motor for the cup it is holding, let alone for every fold of a shirt it is buttoning [1]. As he puts it, "The world gives us underactuated robots, whether we like it or not." [1] The hand itself blocks the head cameras at exactly the moment of contact, which is why he cares about tactile sensors such as a camera behind a soft skin [1]. He names tying anybody's shoelaces as a great challenge problem [1].

His own experience shows how fragile general robots are. At the DARPA Robotics Challenge, one unticked box on a checklist left a foot controller running while MIT's Atlas climbed out of a vehicle; the robot's tailbone touched the seat, a foot lifted, and "nothing in our plans had ever told us what to do if you're butts on the seat and your feet are in the air." [1] It fell face first [1]. With one robot running constantly, his team still saw new failures days before the competition [1].

That leads to what he calls the real challenge for a home robot, which is validation: taking a system with learned perception, complex dynamics, controllers and planners and gaining enough confidence to ship it into anybody's kitchen [1]. His institute builds simulators good enough for tasks like loading a dishwasher, runs algorithms that hunt for where a controller fails ("We joke and we call it the black swan generator."), and pursues fleet learning, since one robot cannot visit all its states in a lifetime but many similar robots together might [1].

Robert Playter: the curse of general purpose, and the math of selling robots

Playter, the chief executive of Boston Dynamics, supplies the view from a company that has to ship. The 2015 DARPA challenge used tasks that are simple for a person, such as driving a vehicle, opening a door and closing a valve, and mostly the robots failed and fell down [2]. He calls it humbling [2]. Atlas was also a little too big to get in and out of the car, which he calls the curse of a general-purpose robot: it is not perfect at any one thing, though it can do many things and cope with uncertainty [2]. One failure stayed with him. A robot reached for a door handle, missed the grasp, and when it tried to turn the knob it threw itself over, because it expected the door to push back and nothing did [2]. The first time you push on the world with a robot, he says, you forget that the world pushes back [2].

Progress is real but slow. Natural-looking walking on Atlas, he says, took from a 2008 prototype until about 2022 [2]. Handling an object changes the robot's center of mass, so it must anticipate the weight and think a couple of seconds ahead [2]. Reliable motion requires layered control loops running at roughly a thousand times a second at each joint, a hundred for the whole body, and slower loops for perception [2]. He names mobile manipulation as the hard problem of the next ten years, and says the next steps for Atlas are more dexterous hands and moving heavy, unwieldy objects, not fine work [2].

Then cost. Under Google's ownership, Larry Page pressed for a consumer product costing a few thousand dollars and used every day, while Boston Dynamics was building machines costing hundreds of thousands [2]. Playter describes a philosophical difference: build the best hardware you can, so that hard problems are not made harder by a poor machine [2]. That is why the company sells to industry, which can afford a robot at a hundred thousand dollars or more, whereas at the consumer level "no one's going to buy a robot like that" [2]. He states the survival rule plainly: "If you only sell hundreds, you will commercially fail." [2] He adds that this is "where most of the small robot companies have died" [2]. Reliability must be earned in the field; a fleet of Spot robots runs missions around the office for over a thousand hours a week to surface rare failures, and each fix changes the design and frustrates the supply chain [2].

He also explains why specialization keeps winning. If the company knew on day one that a robot would work in one factory with three sensors, it could integrate everything into a package that costs less and fails less, and he calls this "the difference between the general purpose machine or the platform versus the purpose built machine" [2]. Boston Dynamics dropped Handle, a robot he calls graceful like nothing else they built, because it picked boxes too slowly, and replaced it with Stretch, an arm on a wheeled base that unloads trucks [2].

On the home, he is direct. The path runs through industry and mass manufacturing first; social robot companies that began with cute interactions mostly have not survived; and a home robot needs real safety around children and the ability to handle varied tasks like telling a beer from a cola or clearing dishes without breaking them [2]. He allows that generative AI able to hold a real conversation may reopen the market for robot pets, though that is not his company's path [2].

Marc Raibert: athletic intelligence is not cognitive intelligence

Raibert, who founded Boston Dynamics and now leads its AI Institute, divides the problem in two. Athletic intelligence is mechanical design, real-time control and energetics; cognitive intelligence is planning and understanding [3]. Most robots are pretty dumb, he says, so skilled people spend a long time programming everything they do [3]. Since ChatGPT there is huge optimism about learning, but he cautions that physical robotics differs from language because pixels are not like word tokens, and he notes that the most impressive performances, including most of Atlas's, still come from traditional methods such as model predictive control [3]. He expects the best results from combining the two [3].

On hands he is blunt. After 50 years of careful geometric grasping, robots are nowhere near human dexterity, while people juggle objects in their hands [3]. He cites a colleague's analysis of a Julia Child cooking video, which found many ways she handled things, none of them grasping [3]. He wants manipulation to become dynamic the way legged robots did, and admits that being more aggressive means things will not work well for a while [3].

He confirms the cost of reliability from inside. A video of Atlas climbing three big steps took 109 tries over about six weeks, with collapses, perception misses and hardware failures with oil flying everywhere [3]. He thinks robot costs have a long way to fall and that a carmaker's manufacturing skill can help [3]. The key challenge is realistic, money-making use cases, and warehouses and fixed industrial arms are what make money today [3]. Several social robot companies failed, he says, because performance, safety and cost must all line up at once [3]. And he makes a point that surprises people who have watched his robots hike through mud: homes and offices are harder than woods, because indoors you cannot leave scuff marks or bang into walls [3].

Kate Darling: we overestimate what we can automate

Darling, who studies human-robot interaction at the MIT Media Lab, says we are far behind where people think in physical robotics [4]. Her example of a recent milestone is modest: a warehouse robot that is safe for people to work around, when strong industrial robots have usually had to be fenced off [4]. In the exchange about work that still resists machines she says "I think we overestimate what we can automate." [4] She adds that this is especially true in the physical realm [4]. Fridman offers demining as an example, still done mostly by humans and animals because it requires enormous experience with edge cases, and Darling notes navies still use dolphins for mine detection [4].

On the household question the two agree. Fridman says he thinks washing the dishes is harder than building a robot that can be a companion, and Darling answers "Being useful in the home is actually really tough." [4] She is also skeptical of the humanoid shortcut. The usual argument is that the world is built for human bodies, but factories have already been rebuilt for other kinds of robots, and she asks of arms, "why not have three arms?" [4] For social robotics she calls the humanoid form "a distraction" and says of humanoids generally "I think it's a little overhyped." [4]

Fei-Fei Li: no data, no shortcut

Li, the Stanford computer scientist behind ImageNet, explains why language arrived first. Each leap in AI came from the same recipe of a neural network, a very large dataset and fast chips, first with images, then with text, then with video [5]. A video model produces plausible cat movement because it has seen countless cat videos, not because it understands anatomy: "It's the statistics." [5]

Physical skill lacks that corpus. She describes her father's liver surgery, performed by a surgeon operating a robot, and her question whether AI could someday do it alone; the answer was that the liver is complex and variable, and even all the world's liver surgeries might not provide enough data [5]. Her lesson is that AI learns from patterns, so where patterns are scarce, human and machine working together beat an under-trained robot [5]. She sees spatial intelligence as the next frontier beyond language and speaks of roughly 30 years when she describes robots helping to care for aging parents, fight wildfires and support nurses [5].

Where They Agree, and Where They Don't

The consensus is broad. Tedrake, Playter and Raibert all locate the difficulty in physical interaction, and all three describe failures that came from the world pushing back in a way the plan did not cover [1][2][3]. Playter, Raibert and Darling agree that money is made today by specialized machines in industry [2][3][4]. Raibert and Li agree that language-model methods do not carry over directly [3][5]. Fridman supplies a hopeful corollary that Playter accepts: learning in robots is less worrying than in chatbots because "The physical reality is a good verifier." [2]

They disagree about the humanoid body. Tedrake says the most compelling argument for it is that "the world is built for the human form," and Playter and Raibert have spent careers on Atlas [1][2][3]. Darling thinks it better to build environments and machines that suit each other, and Tedrake himself concedes the argument may fade as environments are built for robots [1][4].

They also differ on timing. Tedrake expects home robots soon, saying it is only a matter of time until kitchen-counter assistants "spring arms" [1]. Playter says the company might eventually get to a consumer product, and Li thinks in decades [2][5]. Fridman is the optimist throughout, arguing to Playter that a conversational model could simply be added to a reliable robot platform [2]. And they differ on how hopeless contact is: Tedrake argues that simulating it is not as intractable as people claim, since the governing equations are known, and that soft hands change the problem for the better [1].

One caution about the consensus itself. Playter and Raibert lead organizations whose business is industrial robots, so their preference for industry first is also their strategy [2][3].

Why It Works

The sources give the mechanisms. Contact is discontinuous, so methods that rely on smooth behavior break down [1]. The number of possible contact sequences grows combinatorially [1]. Rigid-body models are ambiguous about friction [1]. Deformable and divisible objects have no simple state to estimate [1]. A robot holding something is controlling a thing it has no motor on [1]. The world reacts, so a plan that expected resistance can topple the machine [2].

Reliability multiplies the cost. Real experiments can never be repeated exactly, so testing is done by running as many trials as possible, and rare failures surface only after long hours [1][2]. Homes are unstructured and unforgiving of damage [3]. And the training-data route that worked for language is thin for physical tasks [5]. Tedrake adds a reason progress is undercounted: "robotics always has to solve the next problem and doesn't get to celebrate its past successes," since a dishwasher stops being called a robot once it works [1]. He also points to a remedy in the hardware: softness turns point contacts into patch contacts, so a hand can cradle an egg instead of pinching it [1].

Where It Applies, and Where It Doesn't

The argument applies to general household manipulation: dishes, laundry, cooking, anything involving varied, fragile or deformable objects near people [1][2][4]. It does not apply to single-purpose machines, which already work, or to structured industrial jobs such as inspection rounds and truck unloading, where robots are being sold [2]. Nor does it apply to conversation, which Playter thinks may arrive in robot pets well before useful hands do [2].

These conversations were recorded over several years, and the field moves; humanoid programs, learned manipulation and prices may all have changed since, so check current results before treating any limit here as fixed [1][2][3]. The experts themselves leave room for surprise: Tedrake calls the future of soft, contact-rich robots unwritten, and Raibert says he would not count out new competitors [1][3].

What To Do With It

1. When you see a robot demonstration, ask how many attempts it took and whether the failures were shown; Raibert's team needed 109 tries for one stair climb [3].
2. Ask what the robot touches and with what. Whole-body or in-hand contact with varied objects is the hard case; walking and fingertip grasps are the solved-looking ones [1][3].
3. Separate talking from doing. A fluent voice on a robot says nothing about its hands, since Raibert treats those as different kinds of intelligence [3].
4. Check whether a product is purpose-built or general purpose, because Playter says the purpose-built machine is cheaper and more reliable [2].
5. Look at the price and the numbers sold. Playter's rule is that a robot maker selling only hundreds will fail [2].
6. Expect robots to reach homes through industry first, and treat promises of a near-term household helper with the caution Playter and Darling express [2][4].
7. If you build robots, design for contact instead of avoiding it: softer surfaces, tactile sensing, and controllers that expect the world to push back [1][2].
8. Budget for breaking things and for testing. Run a fleet, hunt for rare failures, and keep spare parts and a repair crew [1][2][3].
9. Where data for a physical task is scarce, plan for a person and a machine working together, which is Li's lesson from surgery [5].

Sources

1. Russ Tedrake on Underactuated Robotics, Contact, Touch and the DARPA Robotics Challenge · Lex Fridman Podcast
2. Robert Playter on Boston Dynamics: Atlas, Spot, Stretch and the Business of Legged Robots · Lex Fridman Podcast
3. Marc Raibert: Boston Dynamics and the Future of Robotics · Lex Fridman Podcast
4. Kate Darling on Social Robots, the Animal Analogy, Privacy, Manipulation and Lessons from MIT's Epstein Scandal · Lex Fridman Podcast
5. Dr. Fei-Fei Li on How AI Learns, What It Can't Do, and Keeping Human Agency at the Center · Huberman Lab