Edge cases and black swans: how robot builders hunt rare failures
Russ Tedrake uses simulation to find black swans, Andrej Karpathy describes Tesla's data engine, and Lex Fridman explains why edge cases dominate autonomous driving. The shared lesson is that reliability in the physical world comes from systematically finding the rare situations, not from average performance.
Why is a robot that works in the ordinary case not yet reliable? Show answer
The failures that matter are rare. Reliability comes from finding and fixing unusual situations, and there are a great many of them.
What happened to Russ Tedrake's robot before the competition? Show answer
After running almost every waking minute, it fell in a way it had never fallen before, days before the event.
What did Tedrake conclude from that fall? Show answer
One robot does not give enough hours to test. The winning team had two robots and tested at a rate and discipline his team could not match.
What caused the fall during the competition itself? Show answer
Someone forgot to tick one checklist box, leaving a foot controller on. The robot ended up in a position no plan covered.
How does Tedrake describe ordinary robot testing? Show answer
As somewhat random search. You run as many trials as you can and try to make worrying conditions occur often.
What is the black swan generator? Show answer
Tedrake's joking name for algorithms that hammer on a simulator to find where a controller fails, then generate many examples near that spot.
What does the simulation approach depend on? Show answer
A simulator good enough to trust. The subtle failures are the hardest to reproduce in simulation.
What are the steps of Andrej Karpathy's data engine? Show answer
Deploy the model, find rare cases where it struggles, reconstruct what truly happened, add them to the training set and retrain.
Why is diversity of data so important to Karpathy? Show answer
A large set of ordinary driving contains very few unusual situations, and a learned system improves only on what it is shown.
What makes driving hard, in Karpathy's account? Show answer
The rare tail of cases that involve predictions about other people.
What does Lex Fridman conclude about today's self-driving systems? Show answer
They are semi-autonomous and need human supervision, because machines will always be flawed and learn at the edge of their capability.
How do Tedrake and Karpathy differ on where to hunt failures? Show answer
Tedrake hunts in simulation, where failures cost nothing. Karpathy hunts in a fleet on real roads, where the cases are certain to be real.