What happens when I type a prompt?

What happens when you press Enter?

Open any chatbot and type a test question:

A moment later, words begin to appear. “A cat is a small furry animal…” They arrive one piece at a time, as if someone were typing.

If you write software for a living, you probably use these tools every day. Try to say what happened in that one second. Most programmers can’t, and the usual guess, that it looked the answer up, is wrong.

Pause and predict What happened in that one second?

The wrong picture: “it looks the answer up”

Most programmers reach for what they know. Text goes in, a relevant answer comes out: that smells like a database or a search index.

The guess is natural and wrong, and it will mislead you about what the model does well and what it does badly.

The problem

We need a working picture of what sits between the prompt and the answer.

The naive fix

Somewhere there is a giant store of questions and answers. The model finds the closest match and returns it.

Why it fails

Ask for a limerick about a Kubernetes pod. Nobody has ever written that, so there is nothing to find, yet you get one. And ask the same question twice: you often get two differently worded answers. A lookup would return the same row.

The idea

Nothing is looked up. The answer does not exist anywhere until it is produced, one small piece at a time, by a calculation.

The trade-off

A calculation that can write anything can also write things that are false. There is no stored row to check against.

Pause and predict Send “What is a cat?” twice in a row. A lookup would return the same row both times. What do chat products actually do?

The pipeline below explains all three.

A pipeline with a loop at the end

Here is what happens when you press Enter, drawn with the actual data at each stage. The diagram has no maths and no code. It shows the whole machine from the outside, once, so that every later piece has a place to sit. Some of the words will be new, and each one is explained properly further on.

Follow the numbers 1 to 8, then the dashed arrow. The chosen token “A” is glued onto the text. Then the whole trip runs again, to choose the token after it.

The trip runs once for every token until the answer is finished, which is why the words arrive one by one.

Two pieces of vocabulary are worth fixing now, because everything later leans on them.

Token

A piece of text from a fixed list the model was built with. Often a whole word, sometimes part of a word, sometimes a punctuation mark.

For example: “What is a cat?” → What·is·a·cat? (5 tokens; the dot marks a space that belongs to the token).

Autoregressive generation

Writing one token at a time, where each new token is chosen after reading everything written so far, including the model’s own earlier output.

For example: “What is a cat?” → “ A”. Then “What is a cat? A” → “ cat”. Then “What is a cat? A cat” → “ is”. And so on.

What you should remember

1. An LLM does not look answers up. The answer does not exist until it is generated.
2. The pipeline: text → tokens → ids → embeddings → Transformer → probabilities → sampling → next token → repeat.
3. The network’s only output is a probability for every possible next token. Everything else is plumbing around that.
4. Each pass through the model produces one token. The model’s own output is appended and fed back in (autoregressive generation).
5. Randomness lives in one place: the sampling step. Nothing in the loop checks whether the text is true.

Where this appears in a real LLM

The explorer is a cardboard model of the machine. Here is what is cardboard and what is real.

In this lesson

• 18 tokens, split on spaces
• 4 hand-picked numbers per token
• “Transformer” = a hand-written table that looks at the last 1 to 2 tokens
• Plain weighted random pick

In a real system

• Roughly 30,000 to 260,000 tokens, learned from data; rare words split into pieces
• Hundreds to thousands of learned numbers per token
• Dozens of Transformer layers, billions of learned numbers, reading the entire text so far
• The same pick, with knobs such as temperature and top-k

Every step of that second gets its own lesson later, and you will write the code for each one.

If nothing is stored and looked up, then where does “Paris is the capital of France” live? The surprising idea takes up that question.