Backpropagation

How does a weight deep inside a network learn its share of the blame?

A release breaks a payments service, and forty customers are refunded twice. The post-mortem does not start at the database. It starts at the symptom and walks backward. The payout service sent two requests because its retry layer saw a timeout. The retry layer saw a timeout because the ledger service answered slowly. The ledger answered slowly because a new query in it skipped an index.

Each team only has to answer one question: “given what reached you, what did you do with it?” Then it passes the question upstream. Every service ends up knowing its share of the blame, and nobody had to understand the whole system. Training a neural network needs the same kind of backward walk.

You now have two things that do not yet fit together.

• A learning loop. Gradient descent: you get the slope of the loss for every parameter and step each parameter against its slope. For a line we derived the slope formula by hand.
• A model that can bend. A neural network: layers of weighted sums and hinges, with thousands of weights. Nobody can derive a separate formula for each of those weights by hand.

The loss is measured at the very end of the network. The weights are spread through every layer. When the prediction is wrong, how does a weight in the first layer, several steps away from the output, find out its share of the blame?

Pause and predict In the post-mortem each team answered one question and passed it upstream. A network has thousands of weights over many layers. How should a weight in the first layer learn its share of the blame?

The problem

The first method to try is the one you already trust: the nudge experiment from Derivatives. You nudge one parameter, run the model again, see how far the loss moved, and divide by the size of the nudge. A gradient measured this way is called a numerical gradient. It gives correct answers. The problem is its cost, so count the forward passes it needs.

Pause and predict If you measure the gradient by nudging, how many forward passes does one update of a language model with 7 billion parameters need?

The problem

Gradient descent needs the slope of the loss for every weight, in every layer, at every step.

The naive fix

Nudge each weight in turn and re-run the network to see how the loss changes.

Why it fails

It needs one forward pass per weight, on every step. And almost all of that work is repeated: nudging two weights in the same layer re-computes every later layer twice, identically.

The idea

Compute the sensitivities from the loss backward, layer by layer. Each layer’s result is computed once and reused by everything upstream of it.

The trade-off

The backward pass needs the values from the forward pass, so they must all be kept in memory until it is done. Training costs far more memory than running the model.

A mental model

A network is a pipeline. The chain rule told you how nudges travel through a pipeline: each stage multiplies the nudge by its own sensitivity.

The way out is to ask the question starting from the far end, next to the loss, and work back toward the input.

1. How does the loss react to a nudge in the logits? That is easy: they are right next to the loss.
2. How does it react to a nudge in layer 2’s input? Take the answer from step 1 and pass it back through layer 2. There is no need to recompute anything from step 1.
3. Do the same for layer 1, reusing the answer from step 2.

Each layer receives one message from downstream: “this is how much the loss cares about each of your outputs”. It uses that message for two jobs: work out the gradient of its own weights, and produce the same kind of message for the layer before it.

Forward pass, backward pass

The forward pass runs the network left to right to get a prediction and a loss. The backward pass then walks right to left, handing each layer the sensitivity of the loss to its outputs, and collecting a gradient for every weight on the way.

For example: Forward: x → h → logits → probs → loss = 2.13. Backward: d_logits → d_h → d_x, picking up d_W2 and d_W1 as it goes.

One clarification that prevents a common confusion: backpropagation does not learn anything. Learning is still w -= lr * grad. Backpropagation is only the fast way to get grad.

Where the backward pass starts: probabilities minus the correct answer

Our network is a classifier, so its output goes through the pair you met in Softmax: softmax turns the scores (logits) into probabilities, and cross-entropy measures the surprise: minus the log of the probability given to the correct class.

The backward pass must start with the sensitivity of that loss to each logit. With exponentials and logs involved, you would expect something ugly.

Pause and predict The model predicted [0.67, 0.24, 0.09] and the truth was the first class. Which is the gradient at the logits?

The whole backward pass, by hand

Same network, same input. The forward pass gave: h = [2, 0], logits = [1, −1], probs = [0.881, 0.119]. The correct class is class 2, so the loss is −ln(0.119) = 2.13.

Now walk backward. “d_something” means: how much the loss changes per unit of nudge to “something”.

Two things in this table are worth noticing.

First, every result was reused. Step 3 used the result of step 1. Step 5 used the result of step 4. Nothing was computed twice. With nudging, each of the 8 weights would have needed its own complete forward pass.

Second, look at the shut hinge. Hidden unit 2 output 0, so its incoming weights get zero gradient, and so do its outgoing ones. That is correct: for this input, tiny changes to those weights really do not change the loss.

Pause and predict Apply one update with learning rate 0.1 and run the forward pass again. Where does the loss of 2.13 go?

The math

This is the densest point in the course. Worked by hand, it turns into three small rules you can hold in your head. We take it in two passes: first one example with real numbers, then the batch version the code uses.

In every rule, d_out is the message arriving from downstream: the sensitivity of the loss to this operation’s output.

First pass: one example, the output layer

A reminder of the convention. In this course and in the repository, a layer computes out = x @ W + b. The input x is a row. W has one row per input and one column per output, so W[i, j] is the wire from input i to output j. (Some books write W x instead; their rules look transposed, but they say the same thing.)

Our output layer, with the numbers from the table above:

Weights. Each weight gets (the input that flowed into it) × (the blame at the output it fed). That fills a grid with every input times every blame, called an outer product:

Inputs. Each input collects blame from every output it fed, through the weight it used: d_h₁ = 0.88×0.5 + (−0.88)×(−0.5) = 0.88, and d_h₂ = 0.88×0.5 + (−0.88)×1.0 = −0.44. That is row i of W, dotted with d_out. In matrix form: d_out @ W2.T.

Bias. A bias is added straight onto its output, so it gets that output’s blame unchanged: d_b = [0.88, −0.88].

These are exactly steps 2 and 3 of the hand-worked table, written as rules.

Second pass: a whole batch at once

Now stack many examples as the rows of X, and their messages as the rows of d_out. Each example contributes its own outer product, and the weight gradient is their sum. X.T @ d_out is precisely “the outer product for every example, added up”. The bias gradient likewise adds up the blame over all rows. The input rule works row by row, as before.

That gives the full list, exactly as it is written at the top of the repository file:

The transpose has a plain reason. Going backward, you travel the same wires as the forward pass in the opposite direction, from outputs back to inputs. Flipping rows and columns is what that looks like when the wires are written as a matrix. You already did it by hand in step 3 of the table above.

ReLU’s slope is 1 on the open side and 0 on the flat side. The chain rule says to multiply by the local slope, so the backward rule multiplies the incoming blame by 1 where the hinge was open and by 0 where it was shut.

What you should remember

1. Backpropagation answers “which weights are to blame, and by how much?” for every weight in one backward sweep, by running the chain rule from the loss toward the input and reusing intermediate results. It only measures gradients. Learning is still W -= lr * d_W.
2. Three local rules cover our whole network. Linear: d_W = X.T @ d_out, d_X = d_out @ W.T. ReLU: pass blame where the hinge was open. Softmax + cross-entropy: d_logits = probs − one_hot: the gradient at the logits is how far each predicted probability missed the correct answer.
3. The backward pass needs the forward pass’s values, so training caches activations. That is why training uses far more memory than inference.
4. Always gradient-check hand-written backward code against nudging. In PyTorch, loss.backward() does all of this for you.

Where this appears in a real LLM

In this lesson

• Three backward rules, written by hand in NumPy
• 4,547 parameters, 300 data points, the whole dataset in every step
• Classes: 3 spiral arms
• Activations cached in a Python list

In a real system

• The same rules plus a few dozen more (attention, normalisation, embeddings…), generated by an autograd system
• Billions of parameters, mini-batches of text, many GPUs each computing part of the gradient
• Classes: every token in the vocabulary, tens of thousands of them. The backward pass still starts with probs − one_hot, at every position in the text
• Cached activations are a main limit on batch size and context length. A common workaround is to store only some of them and recompute the rest during the backward pass, trading time for memory

Every neural language model is trained with backpropagation plus a gradient-descent-style update. When you read that training a model took thousands of GPUs for months, this is what those GPUs were doing: forward pass, backward pass, update, on batch after batch of text.

The next part starts working with language, beginning with how text is cut into tokens.