Looking inside the model

If every number is visible, why can’t you see what a model is thinking?

GPT-2 small is a real public model with three times as many layers as the GPT you built. Type “The cat sat on the” and it answers “floor”.

You can get more than the answer. Print the tensors and you have 768 numbers per token after each of its 12 layers, pages of decimals. Nothing is hidden. Yet reading those pages tells you nothing about why the answer was “floor”.

Some of what those numbers mean can be read today, and a lot of it still cannot.

Pause and predict You can see all 768 numbers per token after each of the 12 layers. What stops you reading what the model is thinking?

The problem: visible is not readable

You met this in Why LLMs know things: there is no table of facts inside, only weights and activations. So the natural first plan is to look at the activations one neuron at a time and ask what each one means.

Pause and predict Read a model one neuron at a time. Does each neuron turn out to stand for one idea?

You could also ask the model why it answered as it did, since it will happily explain its reasoning. The end of this lesson shows why that explanation is not a look inside the model.

A mental model

1. The residual stream is a shared workspace

In a Transformer block, every block adds its output to the token’s vector: x = x + attn(x), then x = x + mlp(x). That running vector is the residual stream.

Think of it as a shared bus: no layer replaces it, each adds a message that later layers can read. The framing comes from Anthropic’s 2021 “Mathematical Framework for Transformer Circuits”, and the arithmetic is exact: the final vector is the embedding plus the sum of every block’s output.

2. The logit lens: read the workspace early

Since the stream is one shared vector, you can take it at any layer and push it through the model’s final norm and output matrix, as if the model stopped there. This trick (from a 2020 blog post) is the logit lens. Here is a real run on GPT-2 small (12 layers, 768 numbers per token) with “The cat sat on the”.

Pause and predict Read the stream after layer 0, the embedding alone. Where does “ floor” rank among GPT-2’s 50,257 tokens?

Early layers give a confident, generic guess (“the same”). Around layers 6 to 8 the right kind of answer rises: “ floor”, then “ ground”, then “ floor” again. The last layer spreads its bets (floor 7.6%, bed 6.5%, couch 5.4%, ground 5.2%). You can watch the prediction form.

The logit lens is an approximation. It assumes middle layers speak the output’s “language”. That assumption holds well enough for GPT-2, and for some other models it produces gibberish. The tuned lens (2023) learns a small translator per layer and is more reliable.

Try it on the course’s tiny Shakespeare GPT: type a prompt and open the layer-by-layer table. Its head scores are here too: look for previous-token heads, and notice what this small model does not have.

3. Some attention heads have readable jobs

A few heads jump out. A previous-token head always looks one position back. An induction head is cleverer: if the text contains “Nimbu Pay … Nimbu”, it looks at what followed the earlier “Nimbu” and pushes the model to predict “Pay” again.

Induction heads (Olsson et al., Anthropic, 2022) work as a two-head circuit: a previous-token head in an earlier layer writes “the token before me was Nimbu” into each position, and the induction head in a later layer uses that to find where the current token appeared before. They appear fairly suddenly early in training, together with a jump in the model’s use of context. The evidence is strongest in small attention-only models, partly correlational in large ones. Most heads in a large model have no one-line job description.

4. Features and superposition

If single neurons are not the unit of meaning, what is? The current best answer is directions.

Feature

A direction in activation space that corresponds to one recognisable property of the input. The more the activation points that way, the more that property is present.

For example: A “this text is in French” direction, or a “Golden Gate Bridge” direction, in the middle of a model.

The catch: a model seems to track far more features than it has dimensions. How do many directions fit into few numbers?

Superposition

Storing more features than there are dimensions, by giving them directions that are almost perpendicular. It works when features are rarely active at the same time, so their small overlaps rarely collide.

For example: In the lab below: 5 features in 2 numbers, arranged as a pentagon.

Picture a Bengaluru bus with forty seats and sixty regulars: it works because they are never all on board at once. Where the picture stops: when two overlapping features are active together, each reading is a little wrong. That error is the price of packing features so tightly.

5. Sparse autoencoders: un-mixing the features

Can we find those directions? A sparse autoencoder (SAE) is a second, small network trained on a model’s activations. It must rebuild each activation from a very large list of candidate directions, using only a few at a time. The directions it learns are called features.

In “Towards Monosemanticity” (2023), Anthropic trained SAEs on a one-layer model with 512 MLP neurons, with dictionaries from 512 up to 131,072 features. Most features were far easier to interpret than the neurons: Arabic script, DNA sequences, base64, legal language. “Scaling Monosemanticity” (2024) used about 1, 4 and 34 million features on the middle of Claude 3 Sonnet, and found features for cities, people, code bugs, sycophantic praise and deception, many working across languages and in images.

The famous demo: a Golden Gate Bridge feature was clamped to a high value during generation. The model, released for a day as “Golden Gate Claude”, brought the bridge into almost every answer and even described itself as the bridge. Turning a feature up changed behaviour: evidence that the model really uses that direction.

6. Testing by intervening: patching, probes, steering

A feature that lights up for bridges might not be used. Three families of tools test this.

7. Circuit tracing: following one answer through the model

In 2025 Anthropic published attribution graphs: replace a model’s MLPs with a more interpretable stand-in made of features (a “cross-layer transcoder”), then trace, for one prompt, which features pushed which others toward the final token.

Asked for “the capital of the state containing Dallas”, Claude 3.5 Haiku showed an internal “Texas” step before “Austin”: a two-hop path, done inside one forward pass. Writing a rhyming couplet, it picked the rhyme word before writing the line that ends in it: evidence of planning ahead.

Let's see the numbers

Check the pentagon by hand. The trained sparse model (seed 1) has five arrows about 1.13 long, 72° apart, and every bias about −0.25. Switch on feature 1 alone with value 1.

The hidden vector h is feature 1’s arrow. Output i is ReLU(arrow_i · h + b_i), and a dot product of two arrows is length × length × cos(angle between them).

Pause and predict Feature 1 is on alone. What do the two far features, 144° away, output?

Why sparsity decides

Measured loss on fresh data, same model size, same training:

When every feature is always on, any overlap interferes on every example, so the model keeps two clean directions and predicts the other three by their average (bias near 0.5). When features are rarely on together, keeping all five pays. Across 12 seeds at S = 0.95, 7 runs found the pentagon and 5 got stuck in the higher-loss square: gradient descent finds a good arrangement, which is sometimes not the best one.

The math

The toy model you trained, in three lines:

A sparse autoencoder is the same shape turned inside out. The small thing is now the model’s activation a, and the SAE blows it up into many features, then rebuilds it:

This is the teaching form, the 2023 “vanilla” SAE. The λ penalty also shrinks the features that should be on, so newer SAEs mostly swap it out: TopK keeps exactly the k largest features (OpenAI, 2024), JumpReLU gives each feature its own threshold (used for Gemma Scope), and Anthropic weights each feature’s penalty by the length of its decoder direction.

And the logit lens is one line: take the residual stream h_ℓ after layer ℓ and apply the model’s own last two steps.

What you should remember

1. Visible is not readable. Every activation can be printed (and the logit lens reads a rough prediction from the residual stream at any layer); the work is finding which directions mean what, and proving the model uses them.
2. Superposition: with sparse features, a model can store more features than dimensions as almost-perpendicular directions, paying a little interference (the toy pentagon).
3. Sparse autoencoders un-mix activations into many more interpretable features (Golden Gate Bridge). Patching, probes and steering test whether a direction is present and used.
4. Limits: attribution graphs explain some prompts, partially. No one can yet fully read a large model, and a chain of thought is a claim, not a trace.

Where this appears in a real LLM

In this lesson

• 5 features in 2 dimensions, chosen by us
• Synthetic data with known sparsity
• You can see the true answer (the pentagon) and check it by hand
• Trains in under a second

In a real system

• Unknown number of features in thousands of dimensions per layer, across dozens of layers
• Features must be discovered, not listed; SAEs with millions of entries
• No ground truth: every interpretation needs an intervention to test it
• SAE training and circuit tracing need large compute and expert time

Real, reproducible results exist: induction heads in small models, the name-finding circuit in GPT-2 small, board-state probes in Othello-GPT, SAE features that steer behaviour when clamped, and a refusal direction in chat models. Open tools and pretrained SAEs (for example Gemma Scope for Google’s Gemma 2 models, and Gemma Scope 2, released in December 2025 for the Gemma 3 family with transcoders and crosscoders as well) let anyone repeat parts of this work.

The next lesson, Reasoning models, turns to models that write out their working before they answer, and to how far that written working can be trusted.