Neurons and layers

Why does training longer leave a spiral classifier at 54%?

Take 300 dots on a plane, in three spiral arms that curl around each other, and train a model to say which arm each dot belongs to.

Use the loop from Gradient descent: a linear classifier, gradient descent, 300 steps. Accuracy is 54%. Train for 3,000 steps and it is still 54%. After twenty thousand steps it is still 54%.

Pause and predict Twenty thousand steps of training still give 54%. What is stopping the model from doing better?

This is the wall the straight line hit at the end of Gradient descent. Gradient descent finds the best line, and if the data follows a curve the best line is still a bad answer. The model cannot express the shape.

Almost nothing interesting is a straight line. Is this email spam? Which word comes next? Which spiral arm is this dot on? None of them is “the output goes up steadily as the input goes up”.

We need a model that can bend, and whose bends can be learned by the same gradient descent loop. A neural network is exactly that: a collection of adjustable functions whose parameters are learned from examples.

You already own the parts: the dot product and the matrix multiply. There is exactly one new ingredient, and it is one line of code.

The problem: a line cannot bend, and stacking lines does not help

The common first fix is more layers. If one linear step is too simple, do two in a row: multiply by a matrix W1, then by another matrix W2. Two transformations ought to be more powerful than one.

Check with real numbers. Take the input x = [1, 2] (as a column) and two small matrices:

Now multiply the two matrices together first, once, ahead of time:

You get the same answer, and you would for every x, because W2(W1 x) = (W2 W1) x. That is how matrix multiplication works: you are allowed to regroup a chain of multiplies.

So two linear layers do exactly the job of one linear layer, and so do a hundred. The stack can never do more than one layer. It can even do less: if a middle layer is narrow, say one number wide, everything must squeeze through that one number, and some information is lost on the way.

The problem

A linear model can only draw straight lines (or flat planes). Real patterns bend.

The naive fix

Stack several linear layers to get a more powerful model.

Why it fails

The matrices multiply together into one matrix. The stack computes nothing a single layer could not. Depth alone adds nothing.

The idea

Put a tiny non-linear function, a bend, after each layer. Now the layers cannot be merged, and every extra layer adds real expressive power.

The trade-off

The loss surface is no longer one smooth bowl, and the gradient can no longer be derived on a napkin. Computing it efficiently needs a new algorithm: backpropagation, next lesson.

A mental model

The only new idea in this lesson is a hinge: a function whose graph runs flat along zero, bends once, and then rises in a straight line. Everything else is something you already know. A network is built up from it in three steps.

1. One neuron: a weighted sum, then a bend

The weighted sum is a dot product: multiply each input by its weight, add up. You have done this since Vectors and the dot product. It asks “how strongly does the input match the pattern stored in my weights?”

Weights and bias

A weight says how much one input counts, and in which direction (negative = counts against). The bias is a fixed offset added at the end: it sets how easily the neuron switches on.

For example: inputs [1, 2], weights [1.0, 0.5], bias 0: sum = 1×1.0 + 2×0.5 + 0 = 2.0.

Then comes the new ingredient: the bend.

Activation function, ReLU

A simple fixed function applied to the weighted sum. The standard one, ReLU, says: if the number is negative, make it 0; otherwise leave it alone.

For example: relu(2.0) = 2.0, relu(−0.5) = 0. Drawn as a graph it is a hinge: flat, then a straight slope, with one bend.

The weight decides how steep the sloped side is and which way it faces. The bias slides the bend left or right. So a neuron is an adjustable hinge.

2. One layer: many neurons, one matrix multiply

A layer is several neurons looking at the same inputs, each with its own weights. That means several dot products with the same vector, which is exactly what a matrix multiply is. Put each neuron’s weights in one column of a matrix W, and the whole layer is:

In words: multiply by the weights, shift by the biases, then bend with ReLU.

3. A network: layers, stacked

Feed the outputs of one layer in as the inputs of the next. The second layer builds hinges out of things that are already bent, so complexity compounds.

The last layer usually has no activation. It only produces the final scores.

Running an input through all layers to get the output is called the forward pass. It is ordinary function evaluation.

A forward pass, by hand

Take a tiny network with 2 inputs, a hidden layer of 2 ReLU neurons, and 2 output scores. All biases are 0 to keep the arithmetic short. The input is x = [1, 2].

The hidden layer outputs h = [2, 0]. That becomes the input to the output layer:

The network’s output is [1, −1]. If these are class scores (logits), softmax turns them into probabilities: [0.88, 0.12].

Notice neuron 2. For this input its hinge is shut, so its output weights (0.5 and 1.0) had no effect at all. For a different input it may be open. That is the non-linearity at work: which parts of the network are active depends on the input. We will meet this exact network again in the next lesson, where we work out how to improve its weights.

The math

One layer, for all its neurons at once:

A network with two hidden layers is this, nested: scores = relu( relu( x W₁ + b₁ ) W₂ + b₂ ) W₃ + b₃. Remove the two relu’s and the three matrices collapse into one, as you saw above.

What about sigmoid? You will meet another activation in older material: the sigmoid, 1 / (1 + e^−z). It squashes any number smoothly into the range 0 to 1, and early networks used it everywhere.

Its weakness is what happens at the edges. For large positive or negative inputs the sigmoid is almost flat, so a nudge to the input barely moves the output.

Now remember the chain rule: a nudge crossing many layers gets multiplied by each layer’s slope in turn. The sigmoid’s slope is at most 0.25, even at its steepest point, and much less than that near the edges. Multiply a handful of those together and the nudge has nearly vanished by the time it reaches the early layers. Deep sigmoid networks therefore learned very slowly.

ReLU’s slope is exactly 1 wherever the hinge is open, and it is cheaper to compute. Modern LLMs use smooth relatives of ReLU: GELU, the same hinge idea with a rounded corner, or gated variants such as SwiGLU, which multiply a smooth hinge by a second learned signal.

What you should remember

1. A neuron takes the dot product of its inputs with its weights, adds a bias, and applies an activation. A layer is many neurons side by side, computed as one matrix multiply: h = relu(x @ W + b).
2. Without activations, stacked layers collapse into one matrix: W2(W1 x) = (W2 W1) x. Without the bends, extra depth adds nothing.
3. ReLU, max(0, z), is a hinge. The weight sets its steepness and direction, the bias moves the bend. Sums of hinges can approximate any curve.
4. On the spiral’s training points: linear model 54%, the same loop with ReLU layers 98.7% (97.7% on fresh points), with every weight learned by the same loop.

Where this appears in a real LLM

Every Transformer block contains two parts: attention, and a feed-forward network. The feed-forward network is this lesson’s network with one hidden layer.

In GPT-2 it widens each token’s vector from d numbers to 4d numbers, applies the bend, and projects the result back down to d numbers.

In this lesson

• 2 → 64 → 64 → 3, about 4,500 parameters
• ReLU
• Input: a 2-D point. Output: 3 class scores
• One network is the whole model

In a real system

• In GPT-2: d → 4d → d inside every block, e.g. 768 → 3,072 → 768 in the smallest GPT-2: about 4.7 million parameters per block, in each of 12 blocks
• GELU (a hinge with a rounded corner) or SwiGLU, which multiplies such a hinge by a second, learned gate
• Input and output: one token’s vector, processed independently of the other tokens
• One such network per block, alternating with attention. Mixture-of-experts models have many per block and use only a few for each token

Modern models such as Llama use SwiGLU instead. It has three matrices instead of two (one of them acts as a gate), so the textbook choice shrinks the hidden width to about 8/3 · d, which keeps the count near 8d². Real models often go wider. Llama 3 8B has d = 4,096 and a hidden width of 14,336, which is 3.5 · d, so its three matrices hold 3 × 4,096 × 14,336 ≈ 176 million numbers in every block, about 10.5d². The shape of the idea is unchanged: widen, bend, project back. More in Modern LLM architecture.

With the hinges in, the spiral reaches 98.7%. That leaves a question for Backpropagation: with 4,547 weights, how did the loop know which way to turn each one?