Self-play: how AI taught itself Go, poker and Diplomacy
October 6, 2026 · 18 min read
David Silver, Noam Brown, Michael Littman and Lex Fridman explain how systems that play against themselves reached superhuman level, and Aravind Srinivas asks whether the same trick can work where no rule decides the winner. It is the clearest account of both the power and the ceiling of machines that improve without human data.
The rules, numbers and steps worth keeping in front of you.
- Self-play: start from random play, compete against yourself, keep what wins.
- It needs a scoreboard: a clearly defined goal that can be scored automatically.
- Silver: hand-built knowledge is brittle and sets a ceiling; learning removes both.
- Silver: errors disappear only when a system can correct its own errors.
- AlphaGo Zero used no human data and beat the earlier AlphaGo 100 games to zero.
- AlphaZero dropped game-specific knowledge; MuZero learned without being told the rules.
- Brown: poker bots minimize regret and converge on an unbeatable equilibrium in two-player zero-sum games.
- Brown: search at decision time is underrated; AlphaZero without it is below top humans.
- Diplomacy: pure self-play failed among humans because the bot invented its own conventions.
- Cicero anchored self-play to about 50,000 human games.
- Littman: the system gets better at Go only; finite games imply diminishing returns.
- Littman: the front half of an S-curve looks like an exponential.
- Srinivas: games, math and code can be checked; open-ended questions need humans.
- Fridman: a hard-coded objective must match human values.
This is the short form. The full insight has every expert's reasoning, their own words and the sources.