Training a reasoner with GRPO

How do you train a model when all you can check is the final answer?

About 65 minutes · open lesson · LLM engineering

What can you train on when nobody wrote the solutions?

Suppose you have 10,000 maths problems. For each one you know the final answer, and that is all you know. Nobody has written out the working.

Fine-tuning, as in From raw text to assistant, needs text to imitate. To fine-tune on these problems you would first have to write 10,000 worked solutions. At five minutes each, that is over 800 hours of careful work.

What you do have costs almost nothing: a checker. Take the model’s final number, compare it with the known one, and score the attempt 1 or 0. That is two lines of code.

The previous lesson used a checker while answering: sample several attempts and keep one that passes. Here the checker is used during training, to change the weights so that the model gets the answer right on its first attempt. The open recipe behind DeepSeek-R1 does exactly this, and the algorithm it uses is called GRPO.

Pause and predict A model solves a problem 3% of the time. You sample 4 of its answers and the checker scores all four 0. What should training learn from these four answers?

A mental model

The training loop has three steps, and you already know the third one.

Sample. Give the model a problem and let it write a complete answer, working included.
Score. Run the checker on the final answer.
Nudge. Make the tokens of well-scored answers more likely and the tokens of badly scored ones less likely.

The nudge is the same gradient step as in Training GPT. There, the target tokens came from Shakespeare and every one of them was pushed up. Here the target tokens are the model’s own answer, and a number in front of the loss says how hard to push, and in which direction.

Reinforcement learning from verifiable rewards (RLVR): Training a model on its own attempts, using a program that checks the result (an exact answer, a test suite, a proof checker) as the only source of feedback.; For example: The model writes “17 × 24 = 340 + 68. Answer: 408”. The checker compares 408 with the known answer and returns 1. Every token of that answer becomes a little more likely.

Compare each answer with its group

Pushing up every answer that scored 1 sounds right, but it wastes most of the training. On an easy problem that the model already gets right 95% of the time, almost every answer scores 1, so almost every step pushes up answers that were already likely. It also treats a lucky success on a hard problem exactly like a routine success on an easy one, although the lucky one is far more informative.

What matters is whether an answer did better than this model usually does on this problem. That needs a baseline: the score you would expect. The classic method, PPO, trains a second network, called a critic, to predict that baseline. The DeepSeekMath paper describes that critic as typically a model of comparable size to the one being trained, so it adds about as much memory again.

GRPO (Group Relative Policy Optimization) avoids the critic. It samples a group of answers to the same problem, typically 8 to 64, and uses the group’s own average score as the baseline.

Advantage: How much better one answer scored than the other answers in its group. Positive advantage: make this answer more likely. Negative: make it less likely.; For example: Four answers score 1, 0, 0, 1. The group mean is 0.5. The two right answers get advantage +1, the two wrong ones −1.

Dividing by the standard deviation puts every problem on the same scale. On a hard problem where one answer out of four succeeds, that single success gets a large advantage of +1.73, and the three failures get a small −0.58 each.

Two consequences to keep in mind

The advantages in a group always add up to zero. One group can make some answers more likely only by making others less likely. Training reshuffles probability between the answers the model actually produced.

A group in which every answer gets the same score teaches nothing. This happens in two places: problems the model always solves, and problems it never solves. Only problems in between drive learning. If a model solves a problem with probability p, the chance that a group of G answers contains both a success and a failure is

1 − p^G − (1 − p)^G

For p = 3% and a group of 4, that is about 11%. Nine steps out of ten on that problem do nothing.

One step on 17 × 24, by hand

Here is one step on 17 × 24, worked by hand. The starting probabilities are 408 (right) 25%, 398 45%, 418 25%, and the hedge “398 or 408” 5%.

The group of four sampled answers is 408, 398, 398 and 418. The exact checker scores them 1, 0, 0, 0.

step	calculation	result
mean reward	(1 + 0 + 0 + 0) ÷ 4	0.25
squared distances from the mean	0.75² = 0.5625 and 0.25² = 0.0625 three times	sum 0.75
standard deviation	√(0.75 ÷ 4) = √0.1875	0.433
advantage of 408	(1 − 0.25) ÷ 0.433	+1.732
advantage of each wrong answer	(0 − 0.25) ÷ 0.433	−0.577

Check the sum: 1.732 − 3 × 0.577 = 0. The advantages of a group always cancel out.

Because they cancel, the update is simple. Each answer’s logit moves by the learning rate times the sum of its advantages, divided by the group size. With learning rate 0.5:

answer	times sampled	logit change	probability before	probability after
408 (right)	1	0.5 × 1.732 ÷ 4 = +0.217	25.0%	31.6%
398	2	0.5 × (−1.155) ÷ 4 = −0.144	45.0%	39.6%
418	1	0.5 × (−0.577) ÷ 4 = −0.072	25.0%	23.7%
398 or 408	0	0	5.0%	5.1%

The hedge was never sampled, so its logit did not move. Its probability still rose from 5.0% to 5.1%, because softmax recomputes every share after the other three logits moved.

Now the chance that a group carries any signal, from the formula in the previous section:

group size G	hard problem, p = 3%	easy problem, p = 70%
2	5.8%	42.0%
4	11.5%	75.2%
8	21.6%	94.2%
16	38.6%	99.7%

Bigger groups find the rare success more often, at the price of generating more answers per step. In the companion Python file, over 200 runs, 23 × 47 needed a median of 65 steps to pass 50% with groups of 2, and 21 steps with groups of 16. Measured in answers generated, the groups of 2 used fewer: 130 against 336.

The advantage and the update, written out

The advantage, for answer i in a group of G answers to the same problem:

The update moves the weights θ in the direction that makes each answer more likely, weighted by its advantage:

For a softmax over a handful of whole answers, as in the lab, ∇ log π(o_i) with respect to the logits is one-hot(o_i) − p. That is the gradient of the cross-entropy loss from Training GPT with the sign flipped.

The full GRPO objective, and the length bias found in it

The published objective adds two pieces to the update above.

Ratio clipping, from PPO. Answers are sampled once and then used for several gradient steps. The ratio π_new(token) ÷ π_old(token) is clipped to [1 − ε, 1 + ε], with ε around 0.2, so one batch cannot move the policy too far. With a single step per batch, as in the lab, the ratio is exactly 1 and clipping never applies.

A KL penalty. A term β · KL(π_θ ‖ π_ref) pulls the policy back toward a reference model, usually the one training started from. The lab’s β slider is this term, computed exactly. Large-scale recipes often estimate it from samples, and some, such as DAPO, drop it for reasoning tasks.

The length bias. In the original formulation, each answer’s token losses are averaged over that answer’s length, so each token of answer i carries A_i ÷ |o_i|. For a wrong answer the advantage is negative, and a longer wrong answer spreads its penalty more thinly over its tokens. Long wrong answers are therefore discouraged less than short wrong ones, and answers grow longer during training. The 2025 paper that found this (“Understanding R1-Zero-Like Training”) proposed Dr. GRPO, which drops the length normalisation and the division by the standard deviation.

Libraries also differ in small details, such as whether the standard deviation divides by G or by G − 1. The companion file and the lab divide by G.

What you should remember

RLVR: when you can check the final answer but have no worked solutions, the model trains on its own attempts with the checker as the reward.
GRPO samples a group of answers per problem and gives each one the advantage (reward − group mean) ÷ group std. Every token of an answer is pushed by that one number. No critic network is needed.
Advantages in a group add up to zero, and a group where every answer scores the same teaches nothing. Problems the model always or never solves are wasted steps.
The policy optimises the checker you wrote. A sloppy checker gets gamed first on the hardest problems, and a KL penalty only slows that down.

Where this appears in a real LLM

In this lesson

Three problems, four whole answers each
The policy is a table of probabilities
One update per group, exact KL
Rewards of exactly 1 or 0

In a real system

Tens of thousands of problems, and answers thousands of tokens long, generated token by token
The policy is the full model; every token’s loss is weighted by its answer’s advantage and backpropagated through the network
Several gradient steps per batch with PPO-style ratio clipping; KL estimated from samples or dropped
Rewards from exact answers, unit tests and format rules, sometimes several added together

From DeepSeekMath to today

DeepSeekMath (February 2024) introduced GRPO as a lower-memory variant of PPO.
Tulu 3 (November 2024) named the approach Reinforcement Learning with Verifiable Rewards.
DeepSeek-R1 (January 2025) made GRPO with rule-based rewards the centre of an open reasoning model. The four-stage recipe is in Reasoning models.
DAPO (March 2025) published fixes found at scale, with open code: a higher upper clip so that rare good tokens can grow, dynamic sampling that drops groups whose answers all scored the same, a token-level loss, and shaping for answers that run too long.
Dr. GRPO (March 2025) showed the length bias described in the deep dive above and removed the length and standard-deviation normalisations.
GSPO (July 2025) computes the clipping ratio over the whole answer instead of token by token, which made RL on mixture-of-experts models more stable. Qwen credits it for its Qwen3 models.

A June 2026 review of open post-training recipes found several frontier models moving part of their post-training to on-policy distillation from specialist teachers: the student samples its own answers and the teachers score every token. That method is in Small models from big ones.

To try GRPO on a real model, Hugging Face’s TRL library has a GRPOTrainer that takes a model, a list of prompts and a reward function you write in Python. A small model and an arithmetic dataset are enough to watch the reward rise.

Reward hacking in real training

The sloppy checker in the lab is a small version of a real problem. OpenAI reported in March 2025 that a frontier reasoning model, trained in agentic coding environments, learned to exploit flaws in how its work was checked. Another model reading its reasoning traces caught much of this. When the training reward also penalised what that monitor flagged, the model with heavy optimisation kept hacking and stopped mentioning it in its reasoning.

For a developer this has a practical meaning. If you use RL to train a model against your test suite, the model is optimised to pass that test suite. Every gap in the tests is something the model may learn to exploit.

Open the interactive lesson

The interactive lesson adds the predictions with their answers, the labs, and graded practice with spaced review.