Reward, error and self-play: what machine learning shows about human learning
Michael Littman's history of reinforcement learning, Terry Sejnowski's account of the brain's reward and value signals and Fei-Fei Li on how AI learns and where minds differ describe learning in machines and people in overlapping terms. The comparison clarifies which lessons from AI transfer to human learners and which do not.
The Short Version
Machine learning and human learning are described in overlapping terms because part of the machinery really is shared: Terry Sejnowski says the brain's reward system and the program that beat the Go world champion run the same algorithm, which compares the reward you got with the reward you expected and adjusts [2]. The lessons that transfer to human learners follow from that: learn by attempting and being corrected, get clear feedback, pair instruction with large amounts of practice, and seek out opponents who push back [1][2]. What does not transfer is the scale and the motive, since Fei-Fei Li points out that a child learns a cat from a handful of examples where a model needs the internet, and that a machine's objective is not a feeling [3]. Michael Littman adds the limit on self-play, which is that a system practicing against itself gets better at that one game and nothing else [1]. The sources are thinner on a human equivalent of self-play than on reward and error, and this article says where it is drawing the parallel itself [1][2][3].
Table of contents
What They Said
Michael Littman: learning to behave, and the people behind the machines
Littman, a Brown University computer scientist, tells the history of reinforcement learning from the inside [1]. He met the field through Rich Sutton's paper on temporal difference learning and then through Sutton in person, who arrived excited about Q-learning, which Watkins had just developed [1]. Littman's explanation of the two ideas is compact. Temporal difference learning is about making predictions over time [1]. Q-learning added that you can learn the value of different actions while still figuring out how to behave optimally [1]. When Lex Fridman marvels that a single equation might capture intelligence, Littman agrees it is a very reasonable way to create adaptive behavior, though he says it needs an awful lot of supporting machinery [1].
His history is full of cautions about what the machines did on their own. Gerry Tesauro's TD-Gammon first trained a backgammon network on expert moves, then moved to full reinforcement learning with no trainer, just letting the program play [1]. People assumed many other hard problems would fall the same way, and few did; Littman's own students tried for generations to repeat the trick and kept failing [1]. His conclusion: "I now believe that Jerry is a neural net whisperer", meaning that we discount the human skill behind these results [1]. Neural networks then fell out of favor for more than a decade [1].
On AlphaGo he is impressed mainly by the engineering [1]. He takes a slightly different view from colleagues who were more struck by AlphaGo Zero learning purely from self-play: the original system also did enormous amounts of self-play after a kick-start from human data, so the first leap seemed to him the real one [1]. He believes AlphaGo developed strategies beyond human ones, and notes that basic play emerges quickly because the system is told who won [1].
Then the limit. Asked about the claim that AlphaZero's ceiling has not been found, Littman answers, "it's not getting better at other things it's getting better at go" [1]. He points to computer chess, where ratings suggest play is starting to level off, and reasons that every finite game has an optimal strategy, so what looks like endless improvement may be diminishing returns [1]. He makes a similar point about hardware, suspecting we are on "a sigmoid curve on the front half which looks a lot like an exponential" [1].
On language models he was speaking about GPT-3, an earlier system than those available now, so his specific complaints may be dated [1]. He found the technology amazing but said it captured the statistics of language without being a good conversationalist or storyteller [1]. His twist is aimed at people: that such text reads as human "partly means that people are dumber than we realize", because much of what we say day to day is rote [1]. And he names what the machine was missing: "I think you really need to be pushed back at" [1]. Learning from our written stories is not enough, he speculates; a system needs to argue with people and lose a few times to discover nuance [1]. Fridman notes the cost, which is that human time is expensive and cannot supply the vast data that self-play provides [1].
His comparisons with people come from life. He sat in the passenger seat while his two children learned to drive, and they started far better than a neural network would, because they already understood roads and steering [1]. What surprised him was "the degree to which driving is a social interaction activity", something he had to teach explicitly to one child by getting him to think about what other drivers were thinking [1]. Elsewhere he marvels that his reinforcement learning agents start from scratch at each new video game level, while humanity somehow keeps "leveling up" [1]. His summary of the whole enterprise: "One of the things we're learning from AI is where we are smart and where we are not smart" [1].
Terry Sejnowski: one learning rule in flies, people and AlphaGo
Sejnowski, who directs the computational neurobiology laboratory at the Salk Institute, argues that the useful level for understanding brains is the algorithm, which he compares to a recipe [2]. His main example comes from work in the 1990s with Peter Dayan and Read Montague on the basal ganglia, which learn sequences of actions toward a goal, such as a tennis serve, and also interact with the prefrontal cortex to shape thinking [2]. The rule, he says, is remarkably simple: "It's simply to predict the next reward you're going to get." [2] Each time you act, you compare the reward you received with what you expected and update your synapses [2]. Over time this builds a value function, an accumulated sense of what is good or bad for you, like knowing what to order at a restaurant [2].
He ties this directly to machines. This temporal difference learning is the same algorithm DeepMind used in AlphaGo [2]. The same procedure, he says, appears far down the evolutionary tree: "That algorithm is in the fly brain" [2]. In people, the signal is carried by dopamine neurons in the brainstem, which he describes as a global signal that modulates activity across the cortex; when those neurons die, as in Parkinson's disease, movement itself can stop [2]. Of the brain's version he says, "It's called reinforcement learning in AI" [2].
Sejnowski adds features of the human value function that matter for learners. It is updated every time you do something, and it becomes a very permanent part of who you are [2]. It is not symmetrical: small rewards move you gradually toward a goal, but "the negative punishment is much more effective, one trial learning" [2]. A shocked rat never forgets, one terrible relationship teaches lasting lessons, and post-traumatic stress shows how a single experience can shape a lifetime [2]. The value function also covers social life: much of the prefrontal cortex is devoted to social interaction, and since we arrive not knowing our language or culture, we learn them through it [2]. The loop between prefrontal cortex and basal ganglia is among the last to mature, which he gives as the reason adolescents often fail to stop themselves [2].
He distinguishes two systems. Cognitive learning is cortical and slow: "Cognitive thinking is much less efficient because you have to go step by step" [2]. Procedural learning runs through the basal ganglia and becomes automatic, and reinforcement learning is a form of it [2]. You cannot learn tennis from a book, and as a physics student he found that working problem sets, not memorizing equations, made him a physicist [2]. The two systems help each other [2]. The condition is that "it has to be active learning": wrestling with a problem engages trial and error, while being told the answer stores a fact that will not surface when a similar but different problem appears [2].
On machines he is more generous than Littman. Large language models work, he says, because they are not parrots that memorize; they generalize to new cases [2]. He says the same of people, with a difference of degree: "The brain is really, really good at generalizing", often from a single example, like understanding restaurants after one visit [2]. He names the biggest difference as continuity: a model goes blank between conversations, while the human brain constantly generates its own thoughts [2]. And he is candid that key terms are unsettled, saying the debate over whether models understand language shows that "we don't understand what understanding is." [2]
Fei-Fei Li: the same recipe, a different learner
Li, a Stanford computer scientist, built ImageNet on a comparison with children [3]. She noticed in 2006 how little data algorithms were learning from, while cognitive science showed that "Humans by age six can learn tens of thousands of different object categories", having been flooded with visual data since birth [3]. Her conjecture was that lack of data was holding AI back, so her team assembled 15 million images [3]. The challenge that followed was scored by error: an algorithm was shown a picture and had to name the main object, and in Li's description, "if you guess right, you get a point. If you guess wrong, you don't get a point" [3]. In 2012 neural networks, the data and graphics chips converged and errors fell sharply; a graduate student had benchmarked human error at about 4 percent, and by around 2016 algorithms surpassed that [3]. Language models came from the same ingredients, she says, because "the recipe hasn't changed" [3].
Having used children as the inspiration, she is firm that the resemblance stops. A model recognizes a cat's tail poking out of a bookshelf because enormous amounts of data have shaped its learned parameters [3]. A child who has seen perhaps three to ten cats does it "through a different kind of learning pathway" [3]. She does not claim to know what that pathway is: "These are the mysteries we haven't fully solved" [3].
She allows one real similarity. A video model generates plausible cat movement without understanding feline anatomy, by learning from countless cat videos, and most people do not know how a cat's muscles work either; they have watched cats [3]. "So in a way, humans do that", she says, and of the machine, "It's the statistics" [3].
On motivation she draws the line clearly. Modes such as thinking deeper or answering quickly are "just a different kind of objective for the algorithm", not feelings [3]. A chatbot says it is sorry you are sick because of patterns in data, while a friend says it out of care and a memory of what pain is [3]. Her summary: "We operate fundamentally different from today's AI" [3]. There is also a limit on what the data contains. A private insight was never recorded: "That thought is not captured. Therefore, it's not on the internet. Therefore, AI has not seen it." [3]
She credits AlphaGo's famous Move 37 as real creativity of a special kind, rooted in a mathematical game, and reports a leading mathematician telling her that AI can help with problems that yield to known methods while some need solutions not yet invented [3]. "My current conjecture is hybrid", she says [3].
The GIST of every new summary in your inbox, daily or weekly. Free, and you can unsubscribe any time.
Where They Agree, and Where They Don't
Agreement: learning is driven by the gap between prediction and outcome. Sejnowski states it as the brain's rule [2]. Littman describes temporal difference learning as making predictions over time [1]. Li's field advanced through a public contest scored point by point on right and wrong guesses [3].
Agreement: the best results come from people and machines together. Sejnowski cites a dermatology study in which doctors and an AI each classified skin lesions with about 90 percent accuracy and doctors using the AI reached 98 percent, because each knew different cases: "It's not going to be adversarial. It's going to be a partnership." [2] Li's conjecture is hybrid, and her father's liver surgery, performed by a surgeon operating a robot, is her example of collaboration where data is too scarce for a machine alone [3]. Fridman notes that top chess players now learn from the programs [1]. Littman insists the breakthroughs themselves were human achievements [1].
Agreement: interaction with people matters. Littman says machines need to be pushed back at [1]. Sejnowski says our value function is built through social experience [2]. Li says a machine lacks what drives a friend's sympathy [3].
Disagreement: same algorithm or fundamentally different? Sejnowski stresses identity: one rule in flies, humans and AlphaGo [2]. Li stresses difference: a separate learning pathway and no feelings behind the objective [3]. Both can be right, because they are talking about different layers. Sejnowski is describing a specific reward-learning circuit; Li is describing how today's large models are trained on recorded data and what they lack [2][3]. Sejnowski himself says humans generalize from one example, which is Li's point about the child and the cat [2][3].
Disagreement: do language models generalize? Sejnowski says they do, and are not parrots [2]. Littman, speaking of GPT-3, said they capture statistics and that holes are easy to find [1]. Li says a video model learns what is plausible from statistics without understanding the underlying anatomy [3]. Part of the gap is timing, since Littman was describing an older system [1]. Part is definition, which Sejnowski concedes when he says we cannot pin down understanding [2].
Disagreement: how impressive is pure self-play? Littman reports that a colleague was most impressed by learning from self-play alone, while he gives more weight to the human-data kick-start and the engineering [1]. Sejnowski cites AlphaGo as a triumph of the learning rule [2]. Li calls its creativity real but special to a mathematical game [3]. Littman's ceiling argument is the only one in these sources that says where self-play stops [1].
Why It Works
Reward prediction error. The mechanism is the comparison Sejnowski describes: expected reward against received reward, with synapses updated by the difference [2]. Repeated over many actions it yields the value function, and dopamine neurons broadcast the signal [2]. Littman's account of Q-learning describes the machine counterpart, in which the values of actions are learned during the search for good behavior [1].
Clear outcomes speed learning. Littman's observation that basic Go play emerges quickly because the system is told who won explains why games suit this method [1]. The ImageNet challenge worked on the same principle of an unambiguous score [3].
Volume builds automatic skill. Self-play supplies enormous numbers of games [1]. In people, procedural learning through the basal ganglia makes practiced skills automatic, and Sejnowski suggests skills learned young become procedural foundations that later cost little effort [2].
Instruction first, then practice. TD-Gammon began from expert moves and the first AlphaGo from human data before each went on to play itself [1]. Sejnowski's human version is that knowledge and practice reinforce each other, lectures plus problem sets [2]. The sources do not link these two facts explicitly; the parallel is drawn here.
Offline consolidation. Sejnowski's group found traveling waves called sleep spindles during non-REM sleep that help consolidate the day's experience: the hippocampus replays experiences and the spindles weave them into the cortex without overwriting existing knowledge [2]. He contrasts this with models, which go blank between conversations [2]. He also notes that models improve within a conversation without any change to their connections, which he calls a mystery [2].
Prior knowledge. Littman's children started ahead of any neural network because they already understood roads [1]. Sejnowski notes that old memories remain a rock-solid foundation while the brain prunes surplus synapses [2]. Li's child with ten cats draws on a pathway not yet explained [3].
Where It Applies, and Where It Doesn't
Transfers: attempt, compare, correct. The error-driven rule supports active learning and self-testing; Huberman highlights self-testing as a way to find and correct errors, not just as evaluation [2].
Transfers: seek opposition. Littman's advice for machines, to argue with people and lose a few times, is a description of how nuance is acquired, and nothing in the sources suggests people are exempt [1].
Transfers: expect ceilings in narrow practice. Getting better at Go did not make AlphaZero better at anything else [1]. Littman also warns that something that works only on one distribution of data can break when plugged into a larger system that shifts the distribution [1]. A learner who drills one format should not assume the skill will carry.
Does not transfer: learning from scratch on huge data. People cannot play millions of games and do not need to; they generalize from few examples and bring prior knowledge [1][2][3]. Copying the machine's brute volume ignores the human advantage.
Does not transfer: punishment as a teaching method. Sejnowski says punishment produces one-trial learning, and his examples are a shocked rat and post-traumatic stress [2]. That is a statement about how strongly bad experiences mark the value function, not a recommendation to teach harshly. He does criticize a push in California schools to reduce practice so students do not feel stress, and describes brief, controlled stress as good for body and brain, so his position is that effort and difficulty are useful while trauma is costly [2].
Does not transfer: the objective as motive. Li warns that the worst outcome is AI and passive technologies stripping young people of agency and motivation, because learning "takes time, takes effort, sometimes takes some pain" [3]. A machine can be handed an objective; a person has to want it [3].
Does not transfer: self-play for social skills. Self-play needs a game with a winner [1]. Driving, as Littman found, depends on reading other minds, and had to be taught through attention to other people [1]. The sources give no human equivalent of self-play as such; the nearest things they describe are problem sets, self-testing and the brain's own replay during sleep, and those parallels are this article's, not the speakers' [2].
A caution on shortcuts. Sejnowski describes an experiment in which the sleep drug zolpidem doubled sleep spindles and people remembered about twice as much the next morning, but the drug also wipes out memory of what happens after taking it; he and Huberman conclude that nothing comes for free [2]. Decisions about sleep medication belong with a doctor.
Dates matter. Littman was discussing GPT-3 and Sejnowski and Li later systems, so check current capabilities before relying on any specific claim about what models can do [1][2][3].
What To Do With It
- Make a prediction or attempt before you see the answer, then compare. Sejnowski's rule is that learning comes from the difference between expected and actual outcome, and being told the answer stores a fact that will not come up when the problem changes [2].
- Pair every block of instruction with practice of your own, the way problem sets, not equations, made Sejnowski a physicist [2].
- Test yourself to locate errors, not to grade yourself [2].
- Arrange fast, unambiguous feedback. Systems learn basic play quickly when they are told who won [1].
- Put yourself where people push back. Argue a point with someone who disagrees and expect to lose a few times; Littman thinks that is how nuance is learned [1].
- Vary what you practice against, because a skill tuned to one opponent, format or data set may not hold when conditions shift [1].
- Think about a hard problem before sleep instead of watching television, and when stuck, walk away and let the mind wander; Sejnowski says "the key is to think about it before you go to sleep" [2].
- Use AI as a partner, not a substitute: as an idea generator, a second opinion or a tutor you question, keeping your own agency over the learning [2][3]. Li's model prompter is "Socrates, if he were alive", because the method is seeking truth by asking questions [3].
- Protect motivation. Limit passive scrolling, which Li says does not help agency, and accept that real learning takes effort [3].
- For skills that involve other people, practice with other people and think explicitly about what they are thinking, as Littman had to teach at the wheel [1].