Gradient descent

How can a program find good numbers without being told them?

Here is a function with two numbers in it, the kind you could write in ten seconds:

You chose the 3.0 and the −1.5. A language model is the same kind of function with billions of such numbers, and nobody chooses any of them. Llama 3 8B has about 8 billion; typing one per second, day and night, would take more than 250 years. The numbers have to come from examples instead.

Pause and predict Pretend you do not know 3.0 and −1.5. You only get examples: for x = 1 the answer was 1.4, for x = 2 it was 4.6. How can a program find the numbers?

The problem

First, two words you have met in passing: parameters in The surprising idea, loss in Softmax. Here they become precise.

Parameters

The adjustable numbers inside a model. The code around them is fixed; only these numbers change during learning. When people say “the model”, they really mean this set of numbers.

For example: In y = w*x + b the parameters are w and b. Set w = 3, b = −1.5 and you have one model. Set w = 0, b = 0 and you have a different (worse) one.

Loss

One number that says how wrong the model currently is. When the predictions are far from the true answers the loss is big, and when they are close it is small.

For example: Predictions [2, 5], true answers [3, 3]. Errors are −1 and 2. Square them (1 and 4) and average: loss = 2.5.

In Softmax the loss was “surprise” (cross-entropy), which fits when the answer is a choice among options. Here the answer is a number, so we measure the squared distance instead. Everything else in this lesson works the same for both.

Why must the loss be a single number? Because we are about to ask “did that change make things better or worse?”. With one number, that question always has an answer.

With a list of 40 separate errors, one change might improve some and worsen others. Then there would be no clear answer to whether the change was “better”.

The quickest plan is a spreadsheet: try every w and every b, and keep the best row. For two numbers that works. Count what happens as the numbers multiply.

Pause and predict Suppose you try 100 values for each parameter. Two parameters then need 100 × 100 = 10,000 rows. How many rows would 1,000 parameters need?

A mental model

In Derivatives you nudged an input and watched how much the output moved. That ratio was the derivative.

Now do the same thing with a parameter as the thing you nudge, and the loss as the thing you watch:

Ask that question once per parameter. The list of answers is the gradient you met in Derivatives: one slope reading per knob. Each reading says which way is uphill for the loss, and how steep it is.

Pause and predict The gradient is [grad_w, grad_b] = [−8.4, 4.1]. To lower the loss, which way do you turn w?

A knob with a steep slope gets a big correction. A knob that barely matters gets a tiny one. Nobody has to decide which parameters are important: the slopes already say so.

Each part of the update has its own job. The loss is one verdict on the whole model. The gradient says, for each parameter, which way to correct it and how strongly. The learning rate sets how large every correction is.

The full loop, which you will see again in every remaining part of this course:

One full update, by hand

Here is one full update on paper. Three data points, all exactly on the hidden line y = 3x − 1.5. We start knowing nothing: w = 0, b = 0. The learning rate is 0.1.

Read the gradient in words. grad_w = −10 means that turning w up lowers the loss steeply. grad_b = −1 means that turning b up lowers the loss too, but only gently.

Pause and predict The hidden b is −1.5, and b starts at 0. After this one step, does b move toward −1.5 or away from it?

Each step only uses the slope where you stand right now. On the next step grad_b already turns positive (0.53), and b heads back down. The path wiggles while the loss keeps falling.

The math

Where did “2 · error · x” come from? From the chain rule you already know. First the loss itself:

For one data point this is a two-stage pipeline: parameters → error → error². The chain rule says: multiply the sensitivities of the stages.

Read the formula for w as a sentence: “a point has a lot to say about w when its error is big and its x is big”. A point at x = 0 says nothing about the slope w, because changing w does not move the prediction there at all.

What you should remember

1. Parameters are the adjustable numbers. The model is those numbers; learning means changing them.
2. The loss squeezes “how wrong are we?” into one number, so that “better” has a direction.
3. The gradient is one slope per parameter: which way is uphill, and how steeply. We take a small step the other way, w -= lr * grad, because a slope is only valid near where it was measured. If the learning rate is too small, training is slow; if it is too large, the loss explodes.
4. The loop predict → loss → gradient → update → repeat, on random mini-batches, is the same loop that trains GPT.

Where this appears in a real LLM

In this lesson

• 2 parameters: w and b
• Loss: mean squared error on a number
• Gradient from a formula we derived by hand
• Plain update: w -= lr * grad
• 40 data points, hundreds of steps

In a real system

• Billions of parameters, all updated on every step
• Loss: cross-entropy on the next token (the “surprise” from Softmax)
• Gradients computed automatically by backpropagation
• A refined update rule (Adam / AdamW) that adapts the step size per parameter, with a learning rate that changes over time
• Trillions of tokens, in batches of millions of tokens, for hundreds of thousands of steps or more

Run on billions of knobs, the same loop of measuring, correcting a little and measuring again is how every chatbot was made. What is still missing is a model worth training: a line is too plain for anything interesting, as Neurons and layers shows.