Why simple language models break

How far back does a language model need to look?

Text often depends on words far back. In “The keys that I left on the kitchen table are gone”, the verb “are” agrees with “keys”, eight words earlier, and the nearer word “table” would suggest “is”. In a story, a “she” on one page refers to someone named pages before.

The character model from the last lesson cannot use any of this. It sees one character at a time, so it cannot remember even the previous word.

Here is what that model wrote when the repository’s Python file ran:

Every pair of neighbouring characters is plausible English: “th”, “he”, “qu”, “in”. The text as a whole is nonsense.

The code has no bug. The model is working exactly as designed, and the design lets it see one character. When it writes the “e” of “the”, it has already forgotten the “t”. It cannot finish a word it cannot remember starting, so agreement across a sentence is far out of its reach.

A language model is only as good as the context it can use, meaning the earlier text it looks at when it makes each prediction. The obvious next step is to let it look further back.

The problem: why not just remember longer phrases?

The count table worked, so the first idea is to give it more to look at. Instead of one row per previous character, use one row per pair of previous characters. Or one row per five characters, or per twenty words. A longer context should give better guesses.

Pause and predict Suppose the count table uses the last n tokens as its row key. What goes wrong as n grows?

Both failures can be measured with a few lines of code.

The measurements count contexts in the 493-character training text from the last lesson, taken once. (The Python file repeats it three times, as CORPUS = (...) * 3, which adds no new contexts.)

A mental model: a phrasebook versus knowing the language

A count table can only answer for a context it has stored. Consider what that means when a new sentence differs from a stored one by a single word.

Pause and predict A table has seen “the cat sat” many times, and never “the dog sat”. What does it know about the word after “the dog sat”?

As a developer you know this as the difference between a cache and a function. A cache keyed on the exact input has a 0% hit rate on inputs it has never seen. A function computes an answer for any input. The n-gram table is a cache of the training text.

Let's see the numbers

The measurements on the single 493-character copy of the text, side by side:

Pause and predict Our text is 493 characters long. At most how many different contexts of length 10 can it hold?

Now the neural alternative. The Python file’s Model C also looks at 3 characters, but it uses a network instead of a table. It looks up an embedding for each of the 3 characters, joins the three vectors end to end into one longer vector, and feeds that to the small two-layer network from Part 3.

Pause and predict The count table for 3 characters of context stores 531,441 numbers. About how many does Model C need?

And the result, from running the file:

The loss falls from 1.75 to 0.35. In perplexity, the model goes from about 5.8 “effective choices” per character to e^0.348 = 1.4.

No amount of cleverness with one character of context could do that, because 1.7518 was already the best possible score with one character. The improvement comes from the extra context.

The math

The wall, as a formula:

Compare how the two approaches grow when you add one more token of context:

• Table: rows multiply by V. One more character: 27 times more rows. One more real token: 50,000 times more.
• Model C: the first layer gets d × hidden more weights (here 16 × 64 = 1,024). So the network grows by adding weights, while the table grows by multiplying rows.

That switch from multiplying to adding is what embeddings bought us. Every context, seen or unseen, is a point in the same space, and nearby points get similar predictions.

What you should remember

1. One token of context gives text in which each pair of neighbouring characters is plausible but the whole is nonsense. More context helps enormously: 1.75 → 0.35 with three characters.
2. Count tables cannot scale: possible contexts = Vⁿ, and almost every long context in real text occurs once or never (data sparsity).
3. Long-context counting memorises the training text and cannot generalise to contexts it has not seen. Fluent output on training text proves nothing. Judge models on unseen text.
4. Embeddings + a neural network share strength across similar contexts. Their parameter count grows by addition as the context gets longer, while a table’s grows by multiplication. But a concatenated window is fixed in length, bound to positions, and cannot choose what matters. Open question: how can a model use relationships between distant tokens, when which tokens matter changes every sentence?

Where this appears in a real LLM

In this lesson

• Characters, 3 of them, 493 characters of text
• Loss measured on the training text
• Runs in seconds

In a real system

• Word n-gram tables (typically up to 5 words, plus tricks for unseen contexts) powered speech recognition and machine translation for decades
• The fixed-window neural language model of 2003 is Model C at scale: it introduced learned word embeddings for exactly the reason in this lesson
• Neither could use long-range context

The sentence that breaks all of them

To continue this sentence sensibly, a model must know what “it” is. The answer, “animal”, is 6 words back. In a real document the relevant word might be 200 tokens back. And change “tired” to “flooded” and the relevant word becomes “road”: which earlier token matters depends on the sentence itself.

A fixed window either does not reach that far, or reaches it through weights bound to “slot 6”, which are useless when the next sentence has the noun in slot 9.

The pre-2017 answer: read left to right and keep a summary

Recurrent neural networks (RNNs, and their improved form, LSTMs) removed the fixed window. They read one token at a time and keep a running summary vector: new summary = f(old summary, new token). The same weights are used at every position, and in principle the summary can carry information any distance.

They worked, and they ran into two flaws of their own:

• A fixed-size bottleneck. Everything read so far must be squeezed into one vector of fixed size. By the time the model reaches “it”, “animal” has been overwritten many times. Distant details fade.
• Strictly sequential. Step 500 needs the summary from step 499. You cannot compute the positions of a sequence in parallel. GPUs are fast because they do thousands of things at once, so training on long texts makes poor use of them.

The question you should now be asking

The next lesson, Attention, builds that mechanism.