Small models from big ones

How does a 3-billion-parameter model learn from a 600-billion-parameter one?

About 45 minutes · open lesson · LLM engineering

How did a 32B model learn from a 671B one?

In January 2025 DeepSeek released R1, a reasoning model with 671 billion parameters. Its authors then took Qwen2.5-32B, an open model about a twentieth of that size, and compared two ways of teaching it to reason.

The first way was the expensive recipe that had produced R1 itself: large-scale reinforcement learning, in which the model attempts maths problems, a checker marks each answer, and the weights move toward the attempts that scored well. After that training the 32B model solved 47.0% of the problems from AIME 2024, a hard American maths competition, on a single attempt.

The second way was far cheaper. They had R1 write out about 800,000 worked solutions and fine-tuned the 32B model to reproduce that text, using ordinary supervised training and no reinforcement learning at all. That model scored 72.6% on the same problems.

So the small model gained more from imitating a big model’s output than from practising on the problems itself. The same pattern holds well beyond reasoning: a small model trained on raw data learns less from it than a big one does, and letting the big model teach it closes much of the gap.

Pause and predict Besides the right answer, what can a big model hand to a small one?

The problem

Look at what ordinary training gives the model. In Training GPT, every position had one correct next token, taken from real text. After “The cat sat on the”, the data says mat. That is all it says.

It does not say that “sofa” and “floor” would also have been fine, or that “moon” would be absurd. A label like that is called a hard label: one right answer, everything else equally wrong.

A big model trained on trillions of tokens has seen “sat on the” thousands of times. Its softmax output already holds the whole picture: mat likely, sofa and floor plausible, moon nearly impossible. That picture cost a fortune in GPU time to learn.

Pause and predict Train the small model from scratch, on the same data and the same hard labels. How does it do?

Distillation can sound like compressing a model, the way a zip file compresses a document. It works differently. The student has its own architecture and its own weights, often of a different shape entirely. It copies the teacher’s behaviour through training, and it never copies the teacher’s weights. (Shrinking the weights themselves is a different family of tricks, in Making the model itself cheaper.)

A mental model

Soft targets: The teacher’s full list of probabilities, used as the thing the student should reproduce, instead of one correct token.; For example: Instead of “the answer is mat”, the student is told: mat 70%, sofa 12%, floor 10%, bed 6%, roof 2%, moon 0.2%.

Why would soft targets teach more? Look at what one example carries. A hard label names one token. A soft target also ranks every other token. “Sofa is much closer than moon” is a fact about the world, and the teacher gives it away for free with every example.

Geoffrey Hinton, who popularised the method with Oriol Vinyals and Jeff Dean in 2015, called this hidden information dark knowledge: what the teacher knows that shows up only in the small probabilities below its top answer.

Turning up the volume on small numbers: temperature

Soft targets have one problem. A confident teacher puts 99% on the top token. The useful information, that sofa is much more plausible than moon, then sits in numbers like 0.3% and 0.001%. Numbers that small barely change the loss, so the student hardly learns from them.

Pause and predict How could you make those tiny numbers big enough to learn from, without changing which token ranks first?

Three ways to learn from a teacher

name	what the student learns from	what you need
Logit distillation (token-level)	The teacher’s full distribution at every position of the training text.	The teacher’s probabilities, so you must be able to run it yourself.
Sequence-level distillation (Kim and Rush, 2016)	Answers the teacher wrote, with ordinary SFT.	Only the teacher’s text, even through an API.
On-policy distillation	Its own answers, with the teacher scoring every token: “I would have given this token 40%, you gave it 90%”.	The teacher’s probabilities on the student’s text. The student learns from its own mistakes.

The second is the one you will meet most often, under a name you have probably heard already: synthetic data.

When the teacher writes the textbook

A synthetic data pipeline is sequence-level distillation run as a factory:

The filter is where most of the quality comes from. For maths, keep answers whose final number is right. For code, keep programs that pass tests. For everything else, a judge model scores the answers and the low scorers are dropped.

This is how a large share of today’s fine-tuning data is made. It is cheap and it scales, but it has real risks:

Model collapse. Suppose you train a model on its own output, then train the next model on that model’s output, and never add fresh human data. Rare things disappear first: a phrase with 1% probability may not appear in the sample at all, and a model fitted to that sample gives it 0%. Each generation loses more of the tails and drifts towards bland, repetitive text.
Contamination and inherited errors. A teacher that saw benchmark questions can quietly reproduce them, so the student scores well on a test it has half memorised (see Evals). The student also inherits the teacher’s wrong facts, biases and habits, with no human in the loop to catch them.

Let's see the numbers

Here are three small calculations. The code section further down runs each of them in Python, so you can check the numbers.

1. Temperature. A teacher has logits [3, 1, −1] for mat, sofa and moon.

Pause and predict Raise T from 1 to 2. Does moon’s share go up or down from its 1.6%?

2. How far is the student from the teacher? Take a teacher p = [0.7, 0.2, 0.1] and a student q = [0.5, 0.3, 0.2]. The distance used in distillation is the KL divergence (you met it as the “leash” in From raw text to assistant). For each token, multiply the teacher’s probability by the log of the teacher’s probability divided by the student’s, then add up the results.

token	p	q	p × ln(p ÷ q)
mat	0.7	0.5	0.7 × ln 1.4 = 0.7 × 0.3365 = 0.2355
sofa	0.2	0.3	0.2 × ln 0.667 = 0.2 × (−0.4055) = −0.0811
moon	0.1	0.2	0.1 × ln 0.5 = 0.1 × (−0.6931) = −0.0693
KL(p ‖ q)	0.0851

Some terms are negative, but the total never is. It is 0 only when q equals p exactly.

Compare the loss a hard label would give. The label is “mat”, so the loss is cross-entropy on one token: −ln 0.5 = 0.693. It says nothing about sofa or moon.

3. What each loss tells the student to change. For softmax, the gradient on the student’s logits is always “student probabilities minus target”.

	mat	sofa	moon
hard label: q − [1, 0, 0]	−0.5	+0.3	+0.2
soft target: q − p	−0.2	+0.1	+0.1

The hard label pushes sofa down harder than moon (0.3 against 0.2), only because the student currently rates sofa higher. Nothing in the hard label says that sofa is nearly right. The soft target pushes each token towards the teacher’s own number. Keep taking hard-label steps and every token except mat heads towards 0. That is exactly the rising dashed curve in the lab.

The math

First, softmax with a temperature, which you have used since softmax:

Then the distillation loss. Teacher logits give p(T), student logits give q(T), both at the same temperature:

Often this is mixed with the ordinary hard-label loss on the real next token: total = α × loss_KD + (1 − α) × cross-entropy, with α somewhere between 0 and 1.

One useful identity. Cross-entropy with soft targets, −Σ p_i ln q_i, equals the teacher’s own entropy plus the KL. In the worked example: 0.8869 = 0.8018 + 0.0851. The entropy depends only on the teacher, so it cannot be changed by training the student. Minimising soft cross-entropy and minimising KL move the student in exactly the same way.

Why multiply by T²?

The gradient of KL(p(T) ‖ q(T)) with respect to the student’s logit z_i is (q_i(T) − p_i(T)) ÷ T. The ÷ T comes from the chain rule, because the logits were divided by T.

At high temperature both distributions flatten towards uniform, so the difference q − p itself also shrinks roughly like 1/T. Together the gradient falls roughly like 1/T². Multiplying the loss by T² undoes that, so the soft loss keeps about the same weight against the hard-label loss whatever T you choose. Hinton and colleagues give this argument in the 2015 paper. In the lab, the gradient is T × (q(T) − p(T)), which is exactly this.

Why not always use a huge T? In the limit, matching soft targets becomes matching the teacher’s logits directly, including very negative logits that the teacher’s own training barely constrained, which can be noise. The 2015 paper reports that when the student is much smaller than the teacher, intermediate temperatures worked best. In LLM practice T is usually around 1 to 2.

Which way round? Forward and reverse KL

KL is not symmetric. In the worked example, KL(p ‖ q) = 0.0851 but KL(q ‖ p) = 0.0920.

Forward KL(teacher ‖ student), as above, is large wherever the teacher has probability and the student does not. It punishes the student for missing anything the teacher might say, so the student spreads itself to cover all of it (“mode-covering”). A small student that cannot fit everything ends up vague.

Reverse KL(student ‖ teacher) is large wherever the student puts probability the teacher would not. It punishes the student for saying things the teacher would not say, so a small student concentrates on a few answers the teacher likes (“mode-seeking”).

On-policy distillation (MiniLLM by Gu and colleagues; GKD by Agarwal and colleagues, both published in 2024) works on text the student wrote itself. MiniLLM uses the reverse direction; GKD tries a family of divergences between the two and often finds reverse KL or a blend works best. Which divergence works best depends on the task and is still studied. Newer 2026 papers (MOPD by Ma and colleagues; TrustMOPD by Sun and colleagues) distil several specialist teachers into one student this way, and TrustMOPD weights each teacher token by token by how reliable it looks at that point.

What you should remember

Distillation trains a small student to imitate a big teacher’s behaviour, never its weights. The student can differ in size and shape.
A hard label names one right token. A soft target is the teacher’s whole distribution, and its small probabilities (the “dark knowledge”) say which wrong answers are nearly right.
Loss: T² × KL(p(T) ‖ q(T)), both at temperature T, often mixed with ordinary cross-entropy. Temperature above 1 makes the small probabilities loud enough to learn.
Three flavours: match probabilities (logit), fine-tune on the teacher’s text (sequence-level, i.e. synthetic data), or let the student write and the teacher grade (on-policy). Synthetic data brings three risks: model collapse when a model’s own outputs replace real data, contamination, and inherited errors. Pure imitation roughly caps the student at its teacher; filtering and RL can lift that cap.

Where this appears in a real LLM

In this lesson

One context, six tokens
A student of six numbers that can match the teacher exactly
Exact gradients, 200 steps
Sampled labels from a seeded random generator

In a real system

Billions of contexts and a vocabulary of 100,000 or more tokens
A student network with far fewer weights than the teacher, so it can only approximate it
Teacher probabilities computed (or stored, often only the top few per position) for trillions of tokens
Teacher-written datasets of hundreds of thousands to millions of examples, heavily filtered

The idea predates LLMs: Buciluă, Caruana and Niculescu-Mizil compressed large ensembles into small models in 2006, and Hinton, Vinyals and Dean introduced the temperature form in 2015. DistilBERT (2019) applied it to Transformers. Google’s Gemma 2 report (2024) says the 2B and 9B models were trained with distillation from a larger model instead of plain next-token prediction. Meta’s Llama 3.2 1B and 3B models (2024) combined both families of tricks: they started from a pruned Llama 3.1 8B and used the logits of Llama 3.1 8B and 70B as token-level targets during pretraining.

It also works for reasoning models, which write out long step-by-step working before they answer (Reasoning models). DeepSeek-R1 (2025) has 671 billion parameters in total (a mixture-of-experts model that uses about 37 billion per token). Its authors fine-tuned six much smaller open models (Qwen2.5 models from 1.5B to 32B, and Llama models of 8B and 70B) on about 800,000 examples written by R1, with SFT only. Those are the runs behind the comparison at the top of this page: 72.6% on AIME 2024 (pass@1) for the distilled 32B model, against 47.0% for the same 32B base trained by large-scale RL directly. Their conclusion: for small models, distilling a stronger model beat the expensive RL recipe, while pushing beyond the teacher still needs a stronger base model and RL.

One practical warning: before you distil from a commercial model, read its terms of service. Several major providers forbid using their outputs to train models that compete with theirs. Open-weight teachers come with their own licence, which may also say something about derived models.

A student that only imitates its teacher learns the teacher’s uncertainty along with its knowledge. It tends to be unsure, and wrong, in the same places the teacher was, and it is rarely better than the teacher.

Open the interactive lesson

The interactive lesson adds the predictions with their answers, the labs, and graded practice with spaced review.