The Transformer block

Why can’t attention tell “Dog bites man” from “Man bites dog”?

Feed two headlines to the attention you built in the last two lessons:

Pause and predict Same three words, different order. Can the attention you have built so far tell the two headlines apart?

That is one of four gaps between “attention” and a working model. Each fix is small. Together they form the Transformer block, the unit that GPT repeats 12, 32 or 80 times. Nothing in it is decoration: for every part we ask what goes wrong without it, and you can remove it and watch.

Four things attention cannot do alone

You have masked multi-head attention from the last lesson. Here is what is still missing.

1. It is blind to word order

The score between “dog” and “bites” depends on those two vectors and on nothing else. Move “dog” to the end and its vector is the same, so the score is the same.

The problem

Attention treats the sentence as a bag of vectors. “dog bites man” and “man bites dog” produce the same vectors for the same words.

The naive fix

Hope the causal mask is enough: after all, it makes each token see a different set of earlier tokens.

Why it fails

The mask tells a token which words came before it, but not in what order, or how far away. In the first block, “dog bites” and “bites dog” look identical to a third token.

The idea

Give every slot in the sequence (0, 1, 2, …) its own learned vector, and add it to the token’s embedding before the first block. Now “dog in slot 0” and “dog in slot 2” are different inputs.

The trade-off

A learned table has a fixed number of slots, so the model has a hard maximum length: the context window. Most modern models use rotary positions (RoPE) instead, which have no table to run out of. But they still work poorly beyond the length they were trained on, unless extended with scaling tricks such as position interpolation or YaRN.

2. It mostly moves information, it barely processes it

Look at what attention outputs: a weighted average of value vectors. The weights come out of a nonlinear softmax, but once they are fixed, the mixing itself is linear in V.

A weighted sum moves information between tokens. It cannot build new features that bend or combine the gathered values in nonlinear ways.

Once “it” has gathered information from “animal”, something must process the result. You already own the tool: the MLP from the neurons lesson, with its bend in the middle. In a block it is applied to each token separately.

3. Deep stacks garble the signal

Your first instinct may be to stack more blocks. That is half right: one round of “communicate, then compute” is not enough, so real models do stack many. But a deep stack brings a problem of its own.

Recall backpropagation: the gradient reaches an early layer only after being multiplied by something at every layer above it. Multiply 24 numbers that are mostly below 1 and almost nothing arrives.

The problem

In a deep stack, each layer overwrites its input. Going forward, the original token information gets garbled. Going backward, the gradient shrinks or blows up on its way to the early layers.

The naive fix

x = layer(x), over and over. Each layer must reproduce everything worth keeping AND add something new.

Why it fails

A freshly initialised layer outputs noise, so layer 1’s careful work is destroyed by layer 2. And the early layers receive almost no gradient, so they barely learn.

The idea

x = x + layer(x). Keep the input and only ADD to it. Each layer now learns a correction, and the plus sign gives the gradient a direct road down to every layer.

The trade-off

Every layer must keep the same width D, because you can only add vectors of equal length. That is why the shape (T, D) never changes inside a Transformer.

4. Numbers drift in size

If every block adds something to x, the numbers in x tend to grow with depth. Large, uneven inputs make a sub-layer’s outputs and gradients large and uneven too, so a learning rate that is safe for one layer is too big for another. Training becomes jumpy, or fails.

The fix is unglamorous: before each sub-layer, rescale each token’s vector to a standard size. That operation is called layer normalisation.

A mental model

Put the four fixes around attention and you get this picture. Read it once now, from the bottom up, following the coloured line. The interactive below lets you open every part.

Notice what that line never does: it never passes through a sub-layer. Attention and the MLP each read a normalised copy, and what they produce is added back at a ⊕. Everything a block does is a correction to the same running x.

Residual connection

Instead of replacing x with a layer’s output, add the layer’s output to x. The layer only has to work out what to change.

For example: x = [1.0, 2.0], the layer outputs [0.1, −0.2]. New x = [1.1, 1.8]. If the layer outputs zeros, x passes through untouched.

Layer normalisation (LayerNorm)

Take one token’s vector. Shift it so its numbers average 0, scale it so their spread is 1. Then let the model stretch and shift it again with two learned vectors, in case a different size works better.

For example: [10, 20, 60] → subtract the mean 30 → [−20, −10, 30] → divide by the spread 21.6 → [−0.93, −0.46, 1.39].

For a developer, a residual block has the same contract as middleware: same type in, same type out, (T, D) → (T, D). That is why blocks can be stacked, removed, or counted with a single config number.

Let's see the numbers

LayerNorm by hand

One token’s vector: [1, 3, 5, 7].

Now try [100, 300, 500, 700]. Mean 400, spread 224, result: [−1.34, −0.45, 0.45, 1.34]. Exactly the same. LayerNorm throws away the overall size and offset and keeps only the pattern. Whatever the blocks below have done to the scale, the next sub-layer sees something familiar.

Finally the learned gain γ and bias β are applied, number by number. They start at 1 and 0, so at the start of training they change nothing.

Why the plus sign rescues the gradient

Suppose each layer passes on only half of the gradient it receives.

Pause and predict Without residual connections, how much of the gradient reaches the bottom after 12 such layers?

The math

The whole block is two lines. Everything else is the definition of the pieces.

What you should remember

1. A block is communicate, then compute: masked self-attention moves information between tokens, the feed-forward MLP processes each token alone. Shape in = shape out = (T, D), so blocks stack: “block × N”.
2. Positional information exists because attention is order-blind. Our GPT adds a learned vector per slot to each token embedding, once, before the first block.
3. Residual connections, x = x + f(x): every sub-layer adds a correction instead of overwriting, and gradients get a direct highway to the early layers.
4. LayerNorm rescales each token’s vector to mean 0, spread 1 (then a learned gain and bias) so every sub-layer sees inputs of a standard size. GPT does this before each sub-layer (pre-norm).

Where this appears in a real LLM

In this lesson

• 3 tokens, D = 4, one block, random weights
• ReLU in the demo; GELU in tiny_gpt.py
• Learned position table, LayerNorm
• 4 blocks of width 128 in the repo’s tiny GPT

In a real system

• The same two-line block, repeated 12 times (GPT-2 small) to 80 or more times (70B-class models), with D in the thousands
• Gated MLP variants such as SwiGLU
• Rotary positions (RoPE) applied inside attention, and RMSNorm, a cheaper LayerNorm without the mean subtraction
• Still pre-norm, still residual, still “attention then MLP”

About two thirds of a block’s parameters are in the MLP (8D² against 4D² for attention). The next lesson stacks N of these blocks into a complete GPT, and you will count those parameters yourself.

See it in real GPT-2: in the GPT-2 Explainer, open any of its 12 blocks to watch LayerNorm, the two residual adds and the 768 → 3,072 → 768 MLP work on your own prompt.