Anca Dragan on Human-Robot Interaction, Modeling Humans and Reward Engineering

High-Level Overview

Lex Fridman sits down with Anca Dragan, a Berkeley professor who works on human-robot interaction: the algorithms that let a robot act well around people instead of in isolation. (She also consults for Waymo, but Lex notes she speaks here purely as an academic.) The talk covers why people are so hard for robots to model, how a robot can deliberately act to learn about the humans around it, why self-driving cars would be close to solved if there were no people on the road, the risks of supervise-the-autopilot semi-autonomous driving, and her most striking idea: that designing reward functions is itself a collaboration between human and robot, and that people constantly "leak" information about what they want.

The tone is warm, playful and technical by turns. Lex opens by saying he was exhausted before the recording and that Dragan's energy and excitement were contagious, and that comes through. She mixes formal ideas (inverse reinforcement learning, Boltzmann rationality, Bayesian belief updates, general-sum games) with stories: a Wall-E robot her husband built to propose, a lunar lander game that people keep crashing, and a physics teacher in Romania who changed her life. The conversation moves from personal origins into the theory of modeling humans, then into autonomous driving, then reward design, and ends with mortality, kindness and the meaning of life.

The Discussion Explores:

• How Dragan came to robotics through Romanian math olympiads, optimization and a life-changing ride in an early Google self-driving car
• Expressive robots, Wall-E, and why kids being rude to Alexa hints at a missing piece in how machines respond to us
• The two core problems of human-robot interaction: sharing space with people who also act, and optimizing for the end user's preferences rather than the programmer's
• Modeling humans: from utility maximization and noisy rationality to the idea that people are rational under different assumptions than the robot's
• Robots that take actions to gather information about people, and robots that act to shape what people believe about them
• Why driving is hard mainly because of humans, plus lidar, learning versus planning, simulation and the "out of distribution" problem
• The hidden danger in semi-autonomous systems that assume a supervising human is as safe as a driving one
• Reward engineering as a human-robot collaboration, and how corrections, e-stops and even the state of a room leak preferences

Deep Dive into Main Ideas

From Math Olympiads to Self-Driving Cars

Dragan describes her path into robotics as gradual and "somewhat accidental." She started programming in fourth grade in QBasic (her first program drew a circle), fell in love with math through the Math Olympiad, which she jokes is Romania's national sport, and moved into computer science because it was math you could actually do something with in the world. From a small school in Germany she applied to top US programs; MIT, Berkeley and Stanford said no, and Carnegie Mellon, where she had spent an exchange semester, said yes. She saw robotics as a great way to apply what she loved, especially optimization.

Her path into cars came during her PhD, when she attended the RSS 2014 conference at Berkeley, organized by Pieter Abbeel, and Google gave attendees rides in its self-driving cars. After years of manipulation research, where progress is painfully hard, watching a robot make the right call decision after decision was amazing. She kept the "I rode in a self driving car" sticker on her laptop for years. Lex shares his own magical moments: meeting the Google car and Boston Dynamics' Spot, whose little arm looking at him made him realize robots can form human connections.

Expressive Robots and the Wall-E Proposal

Her favorite fictional robot is Wall-E, which she loves for how expressive its motion and timing are. The story takes a personal turn: "My husband proposed to me by building a Wally," complete with actuated movement and the belly box that opens to reveal a Lego ring box. They still have the robot.

On the harder question of making robots expressive in general, Dragan explains that a narrow case is easy: bring in an animator to hand-design trajectories, as Anki did with its Cozmo robot. The hard part is doing it autonomously, so that a robot placing an object could do it confidently or hesitantly, happily or disappointed. She thinks this matters. She mentions an article about kids being rude to Alexa because it never reacts, and argues that a personal robot should probably behave differently when you are upset than when you are happy. Technically, this means expanding the robot's notion of state to include the human's internal state, meaning what the person perceives and thinks about the robot, and then optimizing over that, which is very hard to write math about.

What Makes Human-Robot Interaction Hard

Dragan narrows her focus to taking a task a robot can do in isolation and asking what changes when its real goal is to help a person. Two things change. First, the robot is no longer the only agent: cars move among people, and an office robot fetching coffee reaches into the same space as others. Second, what the robot optimizes should reflect the end user's preferences (how you want your house cleaned, how close it should come to you) and not just what a programmer wrote down.

She sees predicting what people will do and inferring what they want as deeply linked, because people are not arbitrary: they act to achieve things. The field has gotten far with inverse reinforcement learning, which takes human behavior and infers the reward it seems optimal for. It rests on a simple model. Utility maximization dates to von Neumann and Morgenstern in the 1940s, and Luce and Shepard added noisy, probabilistic choice in the late 1950s, which robotics turned into "Boltzmann rationality." But then came behavioral economics in the 1970s, which said "people are not rational people are messy and emotional and irrational." Tasks like the Atari-style lunar lander game, where most people crash the craft, break these models. Dragan admits there are days she wants to pack her bags and switch jobs because making sense of human behavior feels so daunting.

Humans Are Rational Under Different Assumptions

Her source of recent hope is a reframing. Instead of giving up on people as irrational, give them the benefit of the doubt: maybe they are rational but work with different beliefs about the state of the world, how it changes, or how far ahead to plan. In the lunar lander study, her group modeled players as using a simplified "intuitive physics" (drawing on cognitive science work by Josh Tenenbaum and Tom Griffiths), worked out where the player thought they were sending the craft, and then used real physics to get it there. "I kid you not it worked," she says: people landed between the flags. She calls this evidence, not proof, that we may be underestimating humans.

She describes the approach as taking the Bellman update and breaking it in every way: people may not see the true state, may use a wrong transition model, or may still be learning what they want, like someone browsing Netflix who does not yet know what they like. Lex points out that this is a good life lesson in empathy too: assume there is something you are missing about the constraints another person works under. Dragan notes that most roboticists she knows are, to be honest, "kind of afraid of humans because they're messy and complicated," and only a handful work on this.

Acting to Learn, and Acting to Be Understood

Robots do not have to watch people passively. Dragan describes work by her student Dorsa Sadigh on autonomous cars changing lanes next to a human driver whose style is unknown. If the car plays it safe and just drives alongside, it learns nothing, because aggressive and defensive drivers look the same when going straight. So the robot plans information-gathering actions: it nudges toward the lane, watches whether the other driver slows, and updates its model. Lex calls it "a fascinating dance."

The reverse works too. People inevitably build mental models of robots, for example seeing a car that cuts someone off as aggressive. Dragan's group modeled the person's belief about the robot's internal parameters as part of the state, assumed people update it roughly with Bayes' rule, and had the robot choose actions that steer that belief toward the truth, making it less confusing and more informative. Lex raises his worry that robots will become humanity's next "other" and get shoved around, and suggests they may need some attitude. Dragan offers two thoughts. People respond to vulnerability as well as assertiveness, so expressiveness may help. And interaction is really a general-sum game: a car and a nearby driver both want to avoid a crash but have different goals, and seeing it that way lets a robot understand that it influences people while they influence it. A car stuck at a merge sees no feasible plan because it assumes traffic will not react. Once it realizes that inching forward can make people slow down, new options open up. She calls the human part of this an underactuated system: degrees of freedom you influence but do not control. Lex adds a runner's tip that if you do not make eye contact, people move out of your way, and Dragan names it: civil inattention.

Why Driving Is Hard: It's the People

Lex brings up Jim Keller's view that driving is mostly a ballistics problem. Dragan offers a thought experiment: take downtown San Francisco and remove every human, driver, cyclist and scooter rider. "I think driving at that point is done," she says. Perception is no longer the main obstacle (though she stresses that everything in robotics is hard), assuming lidar is available. She considers not using lidar irresponsible where you have the option, while admitting she doesn't know enough about vision-only systems. Lex sympathizes with Elon Musk's framing of lidar as a crutch but agrees that safety-critical systems should stick with what works. He also describes his first fully driverless ride with Waymo in Phoenix as a magical moment.

On learning, Dragan rejects the choice between hand-written rules and end-to-end learning. Planning and optimization, where a robot searches for how to reach its goal, are themselves AI, not rules. Pure imitation fails once you go off distribution, and reinforcement learning needs either real-world trial and error or a simulator good enough to model people. Human models, she argues, must combine data with structure, because too many policies fit any dataset. Assuming that people have intentions is a useful prior. Asked whether robots need to encode common sense such as the fear of death, she suggests it may be simpler: any agent modeled as pursuing goals implicitly wants to keep existing, a point she credits to Stuart Russell's line that a robot can't fetch the coffee if it is dead. The rationality framework is broken but fruitful, and a more positive view: maybe people "make more sense than we think."

The Trap in Semi-Autonomous Driving

On Level 2 systems such as Tesla Autopilot and Cadillac Super Cruise, Dragan's concern is the assumption that a supervising human is as safe as a driving one. Technically, once the automation takes over, the car ends up in states the person would never have reached driving themselves, and there is no reason to assume they perform as well there. In human-factors terms, passively observing feels very different from actively deciding, much as students who work out answers learn better than those who are told them. This has been studied for airplanes.

Lex, who says he has "gotten a huge amount of heat on this," agrees with the human-factors literature but reports that its predictions have often been wrong in his large data collection: with sufficiently "dumb" systems, supervisors seem more energized and observant. Dragan agrees this is possible. Her point is that doing it naively carries a hidden assumption. The right goal is to design the system so the person stays engaged and is empowered to be much better than alone, a different mindset from asking them not to drive but be ready to take over.

Reward Engineering Is a Collaboration

Asked how to design good reward functions, Dragan's answer is blunt: we don't know. Even without the interactive part, specifying a reward that produces the right behavior in every situation is very hard. In practice she samples representative situations and tunes the cost terms (how far to stay from people, how much efficiency matters) until the optimal behavior looks right there. After ten years of tuning cost functions, she sees that AI moved the hand-tuning from behavior into rewards without removing it. Lex links this to Goodhart's law: once a metric becomes a target, it stops being a good metric.

Her reframing is that the robot and the designer are collaborating. When a person specifies a reward, it is not gospel, "it's not like the letter of the law"; it is evidence about what they want, meant for the situations they had in mind. So the robot should stay uncertain about its true reward and keep learning from other signals. With Pieter Abbeel and Stuart Russell, her group treats a specified reward this way. She also answers Lex's question about Asimov's three laws in the same spirit: they are words that have to become math (what does "harm" mean?), and nobody wants a machine that takes words literally, so she favors continuous adjustment over fixed universal laws.

Leaked Information: Pushes, E-Stops and Tidy Shoes

Lex loves her phrase that humans "leak" reward information. If a robot arm comes too close and you push it away, control theory treats that as a disturbance to reject, but Dragan sees it as a signal: the robot's idea of optimal was wrong. Her group treats the robot's torque plus the person's correction as roughly optimal for what the person wants, and learns from it, for example that you want it to stay farther away. An emergency stop says even more: whatever the robot was about to do was so bad that stopping and staying still was better.

The most surprising source is the environment itself. If a robot walks into her home and sees shoes lined up on the floor, it should infer she wants them aligned, because someone spent effort on that. As Lex puts it, "the environment leaks information." She adds a caution: a messy house does not mean you want it messy; it may mean your effort went elsewhere.

Books, Mortality, Kindness and Meaning

Her formative book was Russell and Norvig's Artificial Intelligence: A Modern Approach, which she found as a PDF in Romania in 12th grade, when The Matrix was her only exposure to AI. The idea that you could give a system a goal and have it find its own way through a messy situation captivated her. Today, she says, much of her lab's work is to "write math about human behavior combined with data and learning" and let robots figure out the right thing to do.

Given five years to live, she says the idea is her "biggest nightmare" because she loves existing: "it's such a great privilege to exist." Yet she would change little beyond maybe more Caribbean trips and less departmental service, since she already follows the Marie Kondo rule of doing what sparks joy. Her remembered act of kindness is a high school physics teacher (her name is unclear in the transcript) who urged her to take the SATs and apply abroad, then tutored her in physics and SAT prep for free when her parents couldn't afford it. On the meaning of life, a planetarium show about the multiverse convinced her there is no cosmic meaning, and that we should impact whatever local things we can and "try to be there for other humans." Lex closes with an Isaac Asimov quote: "your assumptions are your windows in the world scrub them off every once in a while or the light won't come in."

Conclusion

Dragan's central message is that robots have to stop treating people as noise or obstacles and start treating them as agents with goals, beliefs and limits that can be modeled, even when they look irrational at first. That idea connects everything: cars that nudge to learn who they are merging with, robots that move so people can read them, semi-autonomous systems designed to keep drivers engaged rather than assuming they are, and reward functions treated as evidence rather than law, refined through corrections, e-stops and even the tidiness of a room. It is a generous view of humans that fits the Asimov quote Lex ends on: many of the hardest problems in robotics come from assumptions that need to be revisited.

This is a summary of "#81 – Anca Dragan: Human-Robot Interaction and Reward Engineering" from Lex Fridman Podcast, giving you the GIST so you can decide if you want the REST. Listen to the full episode.