Why LLMs know things

Why did ChatGPT invent six court cases?

In 2023 two New York lawyers filed a brief in Mata v. Avianca, a personal-injury suit against an airline. The brief cited court decisions that neither the airline’s lawyers nor the judge could find. One of the lawyers had found them by asking ChatGPT, and before filing he asked it whether one of them was real. It replied:

Asked for its source, it added that the case could be found on legal research databases such as Westlaw and LexisNexis. There is no such case. The judge found six invented decisions in the brief, with invented quotations and invented internal citations, and in June 2023 he fined the lawyers and their firm $5,000. The citation has the right shape, down to the volume and page number, and the model stated it as calmly as it would state a fact it had read many times. To see where an answer like that comes from, ask a model two simpler questions.

Pause and predict Knowing “Paris” and inventing a paper look like opposite behaviours. Are they two different mechanisms inside the model?

A trained model is a fixed set of numbers plus a fixed recipe of arithmetic. Everything it “knows” must be somewhere in those numbers.

The problem: there is nothing inside to look things up in

If ChatGPT had copied that citation from a document, the document would have to be somewhere inside the model file. So consider what a model file actually contains.

Start with what survives training. The text is read in batches, each batch nudges the weights a little, and then the batch is discarded. When training ends, you ship one file.

The problem

A model answers factual questions, but contains no documents and no database.

The naive fix

Assume the facts are stored as records somewhere inside: one neuron or one row per fact, which the model looks up.

Why it fails

Nobody has found such records. Remove individual neurons and, as a rule, no single fact cleanly disappears. And a lookup would fail visibly on a missing key, whereas a model never does.

The idea

Facts are statistical regularities of the training text, absorbed as small adjustments to many shared weights, because getting them right lowered next-token loss.

The trade-off

Compact and able to generalise, but there is no way to list what is known, no clean way to update one fact, and no built-in signal for “this is not in here”.

Recall the surprising idea from the start of the course: the model only predicts the next token. To predict the token after “The capital of France is”, it helps enormously to have absorbed that Paris goes there. Knowledge is a side effect of prediction.

A mental model: knowledge as learned tendencies

Training leaves three kinds of trace in the weights. They are points on one scale.

• Verbatim memorisation. Text seen many times can be reproduced word for word: famous quotes, licence boilerplate, well-known poems.
• Learned patterns. “Capital of X is Y”, how a citation is formatted, how a Python function starts. The model has seen each of these in thousands of variations and keeps it as a reusable shape.
• Generalisation. Applying those patterns to text never seen before. This is most of what a model does: almost every prompt you type is new.

Memorisation is established: researchers have extracted verbatim training text, and it rises with how often a passage was repeated and with model size. You created the extreme case yourself if you ever over-trained tiny_gpt.py on a tiny corpus: training loss keeps falling while validation loss rises.

A useful picture: the weights are a lossy compression of the training text, like a JPEG of a library. Repeated content survives almost exactly; rare content is blurred, and zooming into a blur still draws sharp-looking pixels. A JPEG, though, only gives back what it compressed, while a model recombines patterns into sentences that never existed. That is why it is useful, and why its inventions look exactly like its recollections.

Which part of the Transformer does what?

You have built all three components. Here is what is currently understood about their role in recalling a fact. The confidence drops from top to bottom.

Here is a simplified mental model, useful for thinking but not a wiring diagram. Read the MLP as a soft key-value memory. Its first matrix is a bank of pattern detectors (“does this vector look like France + capital-of?”), its second is what each detector writes back (“push toward Paris”). Between the two, the activation function silences detectors that matched only weakly. This reading explains part of what MLP layers do; most detectors respond to many unrelated things.

Distributed, superposed representation

One fact is spread over many weights, and one weight takes part in many facts. Models even pack more features into a layer than it has dimensions, by using directions that are almost, but not exactly, perpendicular.

For example: In the lab below, 8 facts share 80 weights. Answering any one question uses all 80.

Let's see the numbers

Two calculations. First, the toy hallucination from tab 4, by hand. These are the actual logits the trained lab model produces (seeded, so yours match).

Pause and predict The lab model was trained on France but never saw Austria. How sure is its (wrong) answer about Austria, compared with its right answer about France?

Why making things up is structural

The Austria row is a hallucination in miniature, and nothing malfunctioned to produce it. Three facts you have already met explain why a model produces answers like that.

1. The output is always a distribution over plausible next tokens. Softmax makes the probabilities add up to 1, so some token always comes out on top.
2. There is no built-in “I do not know” signal. The model can emit the words “I do not know” only when those words are the likely continuation. In most training text, a question is followed by an answer. Its probabilities do carry some information about whether it is right, but nothing turns that into “I do not know” unless training made those words likely.
3. Training rewards fluent continuation. Pretraining loss is lower when the output looks like the training text. For well-covered facts, looking right and being right coincide. For thinly covered ones, they come apart, and the loss barely notices. Later training and benchmarks that score a confident guess above “I do not know” push the same way.

In a real LLM, point 1 is not the whole story: “I do not know” is in the vocabulary, so the deciding factor is what points 2 and 3 made likely. So a hallucination is the generalisation machinery doing its normal job (produce the most plausible continuation) where plausible and true differ. In the lab, “Austria → Berlin” came from exactly the same arithmetic as “France → Paris”.

Where the parameters are

Second calculation: where are the parameters in the model you built? Take one block of tiny_gpt.py with n_embd = 128.

131,712 / 198,272 = 66%. About two thirds of every block is MLP. With four blocks plus embeddings (65-character vocabulary), the whole model is 809,856 numbers: the “0.81M” that the script prints. Everything your Shakespeare model knows about English is in those 809,856 numbers.

The math: why bigger models know more

Why did making models larger work so well? The answer starts with a measurement. The theory came later and is still incomplete.

Train a family of models of different sizes on plenty of data, and plot final loss against size. On log-log axes, the points fall close to a straight line over many orders of magnitude. A straight line on log-log axes is a power law:

Read it like this: multiply the parameters by 10 and the removable part of the loss is multiplied by 10^−α. With the 2020 study’s α = 0.076 that is 0.84: about 16% lower, every time, for every factor of 10. Each step is a small gain, but it is predictable, and that predictability is why labs could justify very large training runs before running them.

The scaling laws themselves are established: empirical, and reproduced many times. Loss falls smoothly as a power law in parameters, data and compute, as long as none of the three is the bottleneck. A follow-up (“Chinchilla”, 2022) showed that parameters and training tokens should grow together, around 20 tokens per parameter for a compute-optimal run.

What you should remember

1. Only parameters persist. There are no documents, no database and no index inside the model. Knowledge is a side effect of learning to predict the next token.
2. Knowledge is distributed: one fact touches many weights, one weight serves many facts. Hence gradual degradation, no way to list what is known, and no clean single-fact update. MLP layers are heavily involved in recall, but “facts live in the FFN” is a simplification.
3. Hallucination is structural: the model always outputs a distribution, has no built-in “I do not know” signal, and was trained for fluent continuation. Retrieval, tools and training to abstain reduce it, but none of them removes it.
4. Scaling laws are established as measurements (loss falls as a power law). Why they hold and what “emergence” really is are active research.

Where this appears in a real LLM

In this lesson

• One linear layer, 80 weights, 8 facts
• Subject vectors are random and fixed
• You can see every weight and retrain in a second
• Output: 8 capitals, nothing else

In a real system

• Billions of weights, dozens of blocks with attention and MLPs
• Subject representations are built up over several layers from the context
• Nobody can read the weights directly; interpretability is a research field
• Output: the whole vocabulary, so “I am not sure” is possible, if training made it likely

What reduces hallucination, and why nothing removes it

How well models “know what they know” is still research: a model’s internal probabilities carry real but imperfect information about whether its answer is correct (base models are fairly well calibrated on many question sets, Kadavath and colleagues, 2022), less so on unfamiliar topics. Chat tuning can make that calibration worse, as the GPT-4 report showed. And common benchmarks may make things worse by scoring a confident guess above an honest “I do not know”; one 2025 analysis (Kalai and colleagues) argues this incentive, together with facts seen only once in training, explains much of why hallucination persists.

This is why production systems put retrieval and tools around the model for anything that must be current, exact or auditable, and why fine-tuning is a good way to change behaviour and a poor way to install facts. A model’s knowledge cut-off follows from the same point: the weights stopped changing when training ended. Reading the weights directly is the job of interpretability research, the subject of Looking inside the model; treat any “neuron for Paris” diagram as an illustration.

So the model in Mata v. Avianca did not look anything up, and a model should not be trusted with citations until it is given the real documents and its answers are checked against them. Retrieval is how it gets them.