Attention

Which bank did the customer call?

A support ticket reads: “The customer called the bank because it was closed.” You read “bank” and think of money without trying. In “they sat on the river bank”, the same word means the ground beside a river, and you would not hesitate there either. You also know that “it” in the ticket means the bank, not the customer.

The word is the same and the meanings are different, and you picked the right meaning from the words around it. The embedding table from Embeddings cannot do that. It gives “bank” one vector, whatever the sentence.

Here is the sentence from the end of the last lesson:

In this sentence “it” refers to the animal. Change one word, so that it ends “…because it was flooded”, and “it” now refers to the road.

You worked that out by looking at other words in the sentence. A model that predicts the next token needs the same ability. The meaning of one token depends on the tokens around it, and which of those tokens matter changes from sentence to sentence.

Pause and predict What does a model need so that “bank” and “it” get the right meaning in each sentence?

The problem

You arrive with two loose ends from earlier lessons:

Pause and predict Before 2017, language models read text left to right and squeezed everything seen so far into one running summary vector. What goes wrong with that approach?

A mental model

Take the phrase “the river bank” and put yourself in the place of “bank”. To know which meaning you have, you need something from the other two words. Before reading on, think about what that exchange between “bank” and the other words requires.

Pause and predict For one token to fetch information from the other tokens, how many different vectors does the model need to make from each token?

These three views have names. You will see them everywhere, always in these colours:

Query, Key, Value

Three vectors made from each token. The query is its search request, the key is the label others match against, the value is the content it hands over.

For example: query of “bank” · key of “river” = high score → “bank” takes a lot of “river”’s value.

Let's see the numbers

Here is the same computation for “bank” in “the river bank”, done by hand. Only two dimensions of the query and keys are non-zero, so we can ignore the rest.

Pause and predict The query of “bank” is [2, 2]. The keys are “the” [0, 0], “river” [2, 0], “bank” [1, 1]. Which token will “bank” listen to least?

The math

You have now done every step. The famous formula is those same steps, written on one line, for all tokens at once:

Why a dot product for matching? Because it is the cheapest way to ask “do these two vectors point the same way?”, as you saw in Vectors and the dot product. Attention uses the raw dot product, without dividing by the lengths as cosine similarity does. So a long key or query also raises the score, and a trained model can make use of that.

Why softmax? Because we want a weighted average. Weights that are positive and sum to one keep the blended vector in the same range as the values, however many tokens there are.

What you should remember

1. Attention exists because a token’s meaning depends on other tokens, and which ones matter changes every time.
2. Query = what I am looking for. Key = what I can be matched on. Value = what I pass along. All three are learned projections of the same token vector.
3. The recipe: score (q·k) → scale (÷√d) → softmax (weights sum to 1) → blend (weights × V).
4. The output is a context-aware version of each token: “bank” near “river” ends up as a different vector from “bank” near “money”.
5. Cost: every token looks at every token, so work grows with T². There is no loop over positions, so training and prompt reading run in parallel across all tokens. (Generating is still one token at a time.)

Where this appears in a real LLM

In this lesson

• 4 numbers per token, hand-picked so you can read them
• Wq and Wk written by hand to make the point
• 3 tokens
• one attention operation

In a real system

• Thousands of numbers per token; no dimension has a clean human meaning
• All weights learned from data; nobody chooses what to attend to
• Thousands to millions of tokens
• Dozens of heads in each of dozens of layers, plus a causal mask

See it in real GPT-2: the GPT-2 Explainer runs the full model in your browser. Open any block to see the queries, keys, values and attention of all 12 heads on your own prompt.

The next lesson, Causal masks and multiple heads, stops tokens from looking at later tokens and runs several attention operations side by side.