Why LLMs make things up, and three ways builders fix it

The Short Version

Language models make things up because of what they are built to do: continue text in a way that is typical of what people have written, one token at a time, with no separate step that checks the result against the world [1][2]. Stephen Wolfram calls that "a shallow computation on a large amount" of training data, Yann LeCun shows why small per-token errors compound over a long answer, and Jimmy Wales says a model "doesn't really know or care about the truth." [1][2][5] Builders respond in three ways: hand exact questions to a computational system, as Wolfram does; retrieve documents and force the model to cite them, as Aravind Srinivas does at Perplexity; and anchor answers in sources that people have checked, the method Wikipedia has used for two decades [1][4][5]. The experts agree on the cause far more than on the cure. LeCun thinks the flaw belongs to the architecture, Marc Andreessen points out that the same behavior is called creativity when we like it, and each of the three fixes has failure modes of its own [2][3][4][5]. So the practical rule is to trust an answer as far as you can see where it came from [1][4][5].

Table of contents

What They Said

Stephen Wolfram: plausible continuation is not computation

Wolfram, the founder of Wolfram Research, gives the plainest description of the mechanism. ChatGPT's core job is continuing a prompt in a way that is typical of roughly a trillion words humans have written on the web [1]. Words become numbers, those numbers ripple through about 400 layers, and the output is a probability for each of roughly 50,000 possible next tokens [1]. A temperature setting controls how often the model picks less likely words; around 1.2, he says, it sometimes goes bonkers and nobody knows why [1]. The chosen token is fed back in and the process repeats [1].

He contrasts this with his own 40-year project of taking the formal structures civilization has built, from mathematics to systematic knowledge, and using them for arbitrarily deep computation that can produce things never computed before [1]. His summary is that ChatGPT is "wide and shallow" and computation is deep [1]. The limit shows up even in simple structure: he tried teaching a transformer network to match parentheses and found it can manage small cases, but once the problem needs a deeper computation "it's hopeless" [1].

Because the model writes fiction and fact with the same machinery, his advice to newcomers such as journalists is blunt: "the idea that you're going to get factual output is not a very good idea." [1] He prefers to think of it as a "linguistic user interface." [1] His stories make the point. In a word problem it solved flawlessly, it got the last two lines wrong [1]. Asked to play the song HAL sings in the film 2001 while being disconnected, it knew the scene and produced convincing notes, which turned out to be "Mary Had a Little Lamb" [1]. In his words, it did an amazing thing, "except it's the wrong song" [1]. A persuasive essay about a supposedly blue Tibetan wolf was plausible enough that he went looking for the animal [1]. The model even invents Wolfram Language code that looks right but is not [1].

Wolfram also explains a quirk users notice. The outer loop of feeding output back in is why ChatGPT can go down a wrong path yet immediately spot the error when shown its whole answer [1].

His remedy is the first of the three. Human language describes the world but does not plug directly into computation, so the goal of Wolfram Language is to turn things we care about into precise symbolic structures you can compute from [1]. Wolfram|Alpha converts natural-language queries into that language with, he says, 98 to 99% success [1]. In the combined workflow a person roughly describes what they want, the model writes a small piece of code, and the human runs it and reads it, since the language was designed to be readable like math notation [1]. The ChatGPT plugin even debugs itself, noticing implausible results, reading error messages and consulting documentation when it has invented options that do not exist [1].

He is careful about what truth means even here. Who is the best actress is unknowable, but who won the Oscar is a definite fact, and even strong facts depend on how you define and measure them [1]. His standard is procedural: we have a procedure, we follow it, we might get it wrong, but we will not be corrupt about getting it wrong [1].

Yann LeCun: errors compound, and the tail is endless

LeCun, Meta's Chief AI Scientist, describes the same mechanism under its technical name. An autoregressive model outputs a probability distribution over a finite vocabulary, samples a token, feeds it back and repeats [2]. It does not plan its answer; it produces one word after another, a bit like answering a question automatically while distracted [2].

From this he derives hallucination. Each token carries some chance of leaving the set of reasonable answers, and if those errors are independent, which he calls a strong assumption, the probability of staying correct shrinks exponentially with the length of the answer [2]. Fine-tuning can cover perhaps 80% of the questions people commonly ask, but the space of possible prompts is gigantic with a very long tail [2]. Random character strings, or swapping a few words into another language, can derail a model, which to him means the system is essentially a giant lookup table [2]. He adds that a model spends the same computation per token whether a question is trivial or impossible, unlike people, who think longer about harder problems [2].

Why do we believe the output anyway? "we're fooled by their fluency." [2] We assume fluent language implies human intelligence, and he says that impression is false [2].

LeCun does not endorse any of the three fixes as a cure. His remedy is a different design: systems that "think about their answer, plan their answer by optimization before turning it into text." [2] He imagines an energy-based model that scores how well an answer fits a prompt and searches an abstract representation space for a good one before decoding it into words [2]. He does list tools such as search engines and calculators among the actions a planning system could learn to use [2].

Marc Andreessen: a bug that is also the feature

Andreessen, the investor and browser pioneer, reframes the problem: "hallucination is what we call it when we don't like it. Creativity is what we call it when we do like it." [3] A big law firm told his team it wants creative models to explore legal theories, and then checks the citations separately [3]. That one example contains the whole logic of the fixes: let the generator generate, and verify with something else [3].

He names the candidate verifiers: fact-checking plugins like Wolfram Alpha, or one model checking another [3]. He also reports that LeCun argues the problem is unsolvable [3]. The bar he sets is Wikipedia's: not perfect, but more likely to be right than the alternatives [3]. He expects direct answers to replace search as we know it, with the model able to regenerate ten blue links or cited sources whenever users want them [3].

His caution is about who decides what is true. He asks for enormous humility and suspicion of anyone claiming a monopoly on truth, and poses a friend's thought experiment: had GPT-4 existed at Galileo's trial, trained on the consensus of the time and tuned by the church's human feedback, would it have declared him wrong? [3] He also flags a longer-term problem. If people stop making web pages, a major source of training data dries up, and whether synthetic data adds real signal or just empty calories is what he calls a trillion-dollar question [3].

Aravind Srinivas: retrieve, then cite everything

Srinivas, the chief executive of Perplexity, built a company on the second fix. Every part of an answer is backed by sources, "like how an academic writes a paper." [4] The system runs a traditional search, reads the relevant links, extracts relevant paragraphs and feeds them to a model instructed to write a concise answer with footnotes [4]. The principle came from his first research paper, when senior colleagues told him every sentence should be backed by a citation from a peer-reviewed paper or his own experiments, and anything else is opinion [4].

The product began with a failure. The founders built a Slack bot on GPT-3.5 to explain health insurance terms, and it sometimes said incorrect things [4]. They asked what would happen if a chatbot were forced to cite everything, and realized that is essentially how Wikipedia works [4]. The rule he states for retrieval-augmented generation is strict: "you're not supposed to say anything that you don't retrieve," and if there is not enough information, the system should say so [4].

He is specific about why retrieval does not end hallucination. It still happens in four ways: the model is not smart enough to understand the query and the paragraphs, the snippets are stale or thin, too much irrelevant detail confuses the model, or the retrieved documents are irrelevant [4]. Each points to a lever: better models, fresher and richer indexes, better snippet selection, better retrieval [4]. He notes that pure vector embeddings are not enough for ranking and that an older term-based method still beats many of them, so real search needs a hybrid [4]. His own launch showed the risk: the first viral moment came from people typing their Twitter handles and getting sometimes hallucinated summaries of themselves [4]. Early on the team considered showing links in a side panel in case answers were wrong, and bet that models and indexes would improve [4]. He also turned down investors who told him to embrace hallucination by building AI companions [4].

Jimmy Wales: sources that people have checked

Wales, the Wikipedia co-founder, describes asking a model about sports analogies and receiving three entirely fictitious but plausible examples; his verdict is that it "doesn't really know or care about the truth." [5] He thinks chatbots should report how much of an answer is unreliable, and half-jokes that he would trust newspapers more if they ran banners saying an article's neutrality is disputed [5].

Wikipedia is the third fix in practice. He prefers verifiability to notability as the test for what belongs, and treats proper attribution as simply good intellectual practice [5]. The community grades its sources, deprecating one tabloid not by banning it but by signaling that better sources probably exist [5]. He is encouraged by work on grounding model answers in quality sources, and describes a weekend demo by Wikimedia's head of machine learning that retrieves relevant articles and answers only from them [5].

He is honest that human-checked sources fail too. A magazine once published that he enjoys playing chess with friends, a claim that came from vandalism on his own page, and the printed article could then serve as a source for the false claim [5]. Wikipedia's reliance on journalism is complicated by the decline of local media and the rise of clickbait [5]. He also sees modest, useful roles for models inside the process: flagging biased wording, checking whether an article accurately reflects its sources, and summarizing disputes [5].

Sam Altman: what tuning does and does not fix

Altman, the OpenAI chief executive, adds the view from the lab. After training on a huge amount of text, the base model knows a lot but is not easy to use; in reinforcement learning with human feedback, people are shown two outputs, asked which is better, and that preference is fed back [6]. "So RLHF is how we align the model to what humans want it to do." [6] That makes the model more usable, which is a different thing from making it check facts [6]. He suspects too much of the model's processing goes into using it as a database instead of a reasoning engine [6]. He also notes that some things that seem easy are hard by design: "counting characters, counting words, that sort of stuff, that is hard for these models to do well the way they're architected" [6].

He supplies a useful correction to scare stories. Ask the same question 10,000 times and rank the answers, and most people see something near the middle, but "the output that gets all of the Twitter attention is output 10,000." [6]

Where They Agree, and Where They Don't

All six describe the same cause. The model generates likely text; nothing in that process consults reality [1][2][5]. All of them also agree the fluent result is persuasive enough to fool careful people, Wolfram and Wales included [1][5].

They disagree about whether it can be fixed. LeCun holds that the weakness comes from token-by-token generation itself and that the answer is a new architecture; Andreessen reports that LeCun considers the problem unsolvable [2][3]. Srinivas bets the opposite way commercially, expecting better models, fresher indexes and better retrieval to keep driving errors down, though he grants there may be truth in LeCun's claim that reasoning needs to happen in an abstract space [4]. Wolfram sits between them: he does not expect the language model to become factual, so he pairs it with a system that is [1].

They also differ on whether making things up is a defect at all. Andreessen treats it as the source of the model's usefulness for brainstorming; Wales and Srinivas, whose products are judged on accuracy, treat it as a failure [3][4][5]. And the Galileo question cuts against the third fix: anchoring a model in human feedback or approved sources makes it as right, and as wrong, as the people doing the approving [3]. Readers should also note that Wolfram and Srinivas are describing their own products, and that LeCun is an investor in Perplexity [1][4].

Why It Works

Each fix works by moving the factual claim out of the generator. With computation, the model only translates a request into precise symbolic form, and a deterministic system produces the answer, so the result follows from a procedure and not from likelihood [1]. With retrieval, the model is restricted to summarizing paragraphs that a search system fetched, and the footnote lets a reader check each claim [4]. With human-checked sources, the claim has already passed through editors applying verifiability rules, and a model that answers only from those articles inherits that work [5].

This matches LeCun's arithmetic: a short span of generation tied to a retrieved passage or a computed result leaves less room for errors to accumulate than a long free-running answer [2]. Srinivas adds that language models tolerate imperfect ranking, finding the right link even when it is ninth [4]. The evidence in these sources is the builders' experience and reasoning, not controlled measurement of error rates [1][2][4][5].

Where It Applies, and Where It Doesn't

Computation fits questions with definite answers, such as arithmetic, data and anything with a formal definition; it does not fit judgments such as whether someone is a good person, which Wolfram calls not really computable [1]. Retrieval fits questions the web has answered well, and Perplexity's own answer concedes a traditional search engine is still faster and better for navigational queries and real-time information such as sports scores [4]. Human-checked sources are strongest on well-covered topics and weaker in obscure corners, where Wales says bias is more likely because only enthusiasts write the entries [5].

None of the fixes removes the need to look. Code written by a model must still be read, a citation can point to an irrelevant or stale page, and a cited source can itself be wrong [1][4][5]. Where an error would be costly, as in legal, medical or financial matters, check the primary source or ask a qualified professional, as the law firm in Andreessen's example does with its citations [3]. For brainstorming, drafting and other work where a rough answer is already a win, invention costs little [1][3]. Finally, the models and products named in these conversations have moved on, so check current capabilities before assuming any specific limit still holds [1][4][6].

What To Do With It

1. Treat a bare chatbot answer as a fluent draft, not a fact, which is Wolfram's advice to journalists [1].
2. Send exact questions to a tool that computes, and read the code or query the model wrote before trusting the result [1].
3. Prefer answers with footnotes, and open the sources for any claim you plan to rely on [4].
4. When a cited answer looks wrong, check Srinivas's four failure points: was the question misread, is the snippet stale or thin, is there irrelevant detail, or is the source off topic [4].
5. Be more suspicious of long answers and unusual questions, where LeCun says errors compound and the long tail begins [2].
6. Show the model its whole answer and ask it to check, since Wolfram notes it can spot errors it made along the way [1].
7. Separate creating from checking, as the law firm does: use the model to explore, then verify independently [3].
8. Ask where a cited claim originally came from, remembering how a vandalized line about Wales became a published fact [5].
9. Do not let fluency stand in for accuracy, and do not judge a tool by one viral failure either; try the question yourself [2][6].

Sources

1. Stephen Wolfram on ChatGPT, Computational Irreducibility, Truth and the Second Law · Lex Fridman Podcast
2. Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI · Lex Fridman Podcast
3. Marc Andreessen on the Future of the Internet, Why AI Will Save the World, and the Birth of the Browser · Lex Fridman Podcast
4. Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet · Lex Fridman Podcast
5. Jimmy Wales on Building Wikipedia, Neutrality, Community Health and the Business of Trust · Lex Fridman Podcast
6. Sam Altman on GPT-4, ChatGPT, AI Safety and the Road to AGI · Lex Fridman Podcast