Predicting the next token

How close can counting characters get to English?

Give a short Python program 1,479 characters of plain English. For each character, it counts which character came next. Then it writes new text one character at a time, choosing each next character in proportion to those counts. One run prints this:

Most of it is nonsense, yet it is plainly closer to English than random letters would be. Spaces fall at roughly word-sized gaps, “is the” appears, and “h” follows “t” again and again. The program knows nothing about words or spelling. It only counted which character followed which, and the same loop with a far better predictor inside it is what writes a chatbot’s reply.

Start with the question the program answers. Finish this sentence:

Pause and predict What word comes next?

This is the task every LLM performs: given the tokens so far, give a probability to every possible next token. GPT-2 did this, and the largest models today do the same.

The problem: how does guessing one token become writing?

A program that only predicts the next token sounds like the autocomplete on a phone keyboard. Two things about that definition are puzzling.

The problem

We want a program that can continue any text.

The naive fix

Write rules: grammar, facts, style. Or store whole sentences and look them up.

Why it fails

Rules never end, and almost every sentence anyone types is new, so lookup finds nothing.

The idea

Learn one small skill from raw text: the probability of each next token. Then apply that skill over and over, feeding each chosen token back in.

The trade-off

Nothing in the loop plans a paragraph. Each turn only picks the next token, and every pick is a roll of the dice.

We will start absurdly small: the tokens are single characters, and the model may look at only one previous character.

A mental model: count what comes next, then keep pressing

Forget neural networks for a moment. How would you guess the next character after “q” in English? You would say “u”, because that is what you have always seen.

The counting program at the top works the same way. A useful guess about what comes next can be built just by counting what came next before.

Where do the examples come from? From the text itself

Take any text. Slide it one position to the left. Every position is now a labelled example: input = this token, correct answer = the token that actually came next.

Eight characters gave seven training examples, and no human labelled anything. This is why LLMs can be trained on a large fraction of the public internet: every position of every text is a free exam question with the answer attached.

Let's see the numbers

Where did the bar for “h” after “t” come from? From counting. In the training text, “t” is followed by another character 135 times:

That row of probabilities is the model’s prediction after “t”. Twenty-seven rows like it, one per character, are the whole model: a 27 × 27 table.

Plain counts have one problem. “t” was never followed by “z”. A count of 0 means probability 0, which means “impossible”. If the model later meets “tz”, its surprise is −log(0) = infinity and the average loss explodes. So the code adds a tiny 0.01 to every cell before dividing. This is called smoothing:

P(h | t) = (78 + 0.01) ÷ (135 + 27 × 0.01) = 78.01 ÷ 135.27 = 0.5767

That is the 57.7% you see in the playground.

How good is the model? Use the same score as in Softmax: cross-entropy, the average surprise at the character that really came next. For the two predictions inside the word “the”:

Average that over all 1,478 predictions in the text and you get 1.7518. That is the loss.

Now raise e to that number: e^1.7518 = 5.76. This second number is the perplexity, and it is the one people usually report, because it is easier to feel. It means that, on average, the model is as unsure as someone choosing among about 6 equally likely characters.

Perplexity = e^loss. When a paper or a leaderboard quotes “perplexity 5.76”, this is the calculation behind it.

A model that knows nothing gives each of the 27 characters the same probability, so it is choosing among all 27. Its perplexity is 27 and its cross-entropy is ln 27 = 3.30. Ours is torn between about 6 choices instead of 27, so the counts have taught it a good deal about which characters follow which.

Two conventions, so other people’s numbers do not confuse you:

• Nats or bits. We use the natural logarithm (ln), so the loss is in “nats”. Some papers use log₂ and report “bits”; then perplexity is 2^loss. The unit changes the loss, but the perplexity comes out the same.
• Per token. Perplexity is counted per token. Two models can only be compared on it if they cut the text into the same tokens.

The math

The table in one line:

And the score:

A real LLM changes exactly one thing in these formulas: the condition. Instead of P(next | one previous token) it computes P(next | all previous tokens in the context). The loss that trains GPT-class models is this same average surprise.

What you should remember

1. A language model does one thing: P(next token | previous tokens), a probability for every token in the vocabulary.
2. Generation is repeated prediction: predict, sample one token, append, repeat (autoregressive). The loop itself does not plan or look ahead.
3. Training data is free: shift any text by one position and every position is a labelled example.
4. Quality is measured by cross-entropy (average surprise) or perplexity = e^loss (effective number of choices). Ours: 1.7518 and 5.76.
5. A neural network trained by gradient descent converges to the count table. Training is a way of finding these statistics when a table is impossible.

Where this appears in a real LLM

In this lesson

• Tokens are 27 characters
• Sees 1 previous token
• The “model” is a 27 × 27 table of counts
• 1,479 characters of training text
• Perplexity 5.76 per character

In a real system

• Tokens are roughly 30,000 to 260,000 subwords
• Sees thousands to millions of previous tokens
• The model is a Transformer with billions of learned weights
• Trillions of tokens of training text
• Same loss: average −log P(actual next token)

This explains several things you see as an API user. Output streams token by token because it is produced token by token. The same prompt can give different answers because each token is a random draw. And output tokens cost more than input tokens partly because each one needs its own pass through the model.

So a language model is autocomplete run in a loop. The counted gibberish at the top and a chatbot’s fluent reply come from the same loop. What differs is how much of the earlier text each prediction can see, and the next lesson tries to give the model more of it.