Reasoning models

Why do you reach for a pencil at 47 × 86?

Ask yourself what seven times eight is, and 56 arrives before you have finished reading the question. Ask yourself what 47 times 86 is, and most people reach for a pencil.

On paper the hard question turns into three easy ones. 47 × 80 = 3,760. 47 × 6 = 282. Adding the two gives 3,760 + 282 = 4,042.

Picking up the pencil did not make you any cleverer. It let you break one hard step into several easy ones, and the paper held the partial results so that you did not have to keep them in your head.

Since late 2024 (OpenAI’s o1, which hid its working, then the open DeepSeek-R1, which shows it), some models do the same: they write a long stretch of “thinking” before the answer. They are slower, they cost more, and they score clearly higher on maths, code and multi-step problems. Why should writing help a model at all?

Pause and predict The model must write the next token after “2 + 2 =” and after “the 40th prime number is”. How much compute does it spend on each?

One fix is to use a bigger model. A bigger model makes every token more expensive, the easy ones included, and its number of layers is still fixed, while a problem can need any number of steps. The other way out needs no new architecture: let the model write its steps first.

A mental model

A Transformer has no scratchpad that survives from one token to the next, apart from the KV cache of what is already on the page. So the text the model has already written is its only scratchpad: anything it writes down, later tokens can read back.

Chain of thought (a reasoning trace)

Intermediate steps that the model writes out, as ordinary tokens, before it writes its final answer.

For example: “17 × 3 = 51. 51 + 28 = 79. …” and only then “Answer: 113”.

Writing steps does two things at once. It buys more compute: fifty tokens of working are fifty forward passes aimed at your question. And it gives working memory: a partial result that is written down does not have to be held inside the vectors, because later tokens attend to it.

Let's see the numbers

A curve is easier to trust once you have computed one point of it yourself. Here are two points of the lab, worked by hand.

Take p = 0.4 and four wrong answers, so each wrong answer turns up 0.6 ÷ 4 = 0.15 of the time. Use N = 3 attempts.

Best-of-3 with a perfect checker. It fails only if all three attempts are wrong:

0.6 × 0.6 × 0.6 = 0.216 → accuracy = 1 − 0.216 = 0.784

Majority vote over 3. Count the ways the right answer can win:

(The ¾: given two wrong attempts, the second one differs from the first with probability 3 out of 4.)

So for the same 3× token bill: single attempt 40%, voting 46%, a perfect checker 78%. Set the lab to p = 40%, m = 4, N = 3, verifier 100% and you will see exactly these values.

Now suppose every wrong attempt gives the same wrong answer, so that one wrong answer turns up 60% of the time (m = 1). A vote over 3 attempts now picks the right answer only when at least 2 of the 3 are right: 0.064 + 0.288 = 0.352. That is below 40%. Voting amplifies whatever is most common, right or wrong.

The math

The perfect-checker case sets the upper limit for any way of choosing among N attempts:

The bill is simpler: tokens spent = N × tokens per attempt, a straight line. The failure rate shrinks by the same factor (1 − p) with each attempt, so each extra attempt buys less. Going from 1 to 4 attempts at p = 0.4 buys 47 points. Going from 8 to 16 buys less than 2.

What you should remember

1. One token = one forward pass = fixed compute. A model that must answer at once has a fixed budget for the hard part.
2. A reasoning trace buys more forward passes and uses the model’s own output as working memory, through the same generation loop you built.
3. Majority vote helps only if the right answer is the most common one. A good verifier helps much more; a poor one sets a ceiling.
4. Reasoning models are trained for this, in open recipes by reinforcement learning on checkable answers. Thinking is not free, and a trace is not a guaranteed faithful explanation.

Where this appears in a real LLM

In this lesson

• Each attempt is right with a fixed probability p
• Attempts are independent; wrong answers are spread evenly
• The verifier is a biased coin
• “Tokens per attempt” is a slider

In a real system

• p differs for every question, and nobody tells you what it is
• Errors are correlated: a model tends to repeat the same misconception, the “one popular wrong answer” case
• Verifiers are tests, proof checkers, exact answers, or another model with its own blind spots
• The model decides how long to think, often within an effort or budget setting chosen by the caller

The open reference: DeepSeek-R1

DeepSeek-R1 (2025) published its paper and weights. Its core is reinforcement learning on problems with automatically checkable results: maths answers, and code that must pass tests. The rewards are simple rules: an accuracy reward for a correct final answer, a format reward for putting the thinking between the required tags, and later a language-consistency reward, because traces kept switching language mid-thought. There is no learned reward model for the reasoning tasks.

Pure RL was only the research variant, R1-Zero. R1 itself came from four stages: a small SFT on long, readable reasoning examples (the “cold start”), then this reasoning RL, then SFT on about 800,000 samples kept by rejection sampling (reasoning and ordinary tasks), and a final RL stage that added learned reward models for helpfulness and harmlessness.

The algorithm is GRPO: for each problem, sample a group of answers and judge each against how the rest of the group did. An answer that scored better than the rest of its group is made more likely, and one that scored worse is made less likely. Concretely, each answer’s advantage is its reward minus the group’s mean reward, divided by the group’s standard deviation, and a KL term keeps the policy near the reference. No separate value network is needed. (GRPO was introduced a year earlier, in DeepSeekMath, 2024.) For the variant trained by RL alone (R1-Zero), the authors report that responses grew longer and that re-checking and trying a second approach became more frequent without being scripted.

Since R1

• Better GRPO. DAPO, Dr. GRPO and GSPO fix flaws found in practice, such as a length normalisation that quietly favoured long wrong answers. These fixes are published, with open code.
• RL on agent tasks. One attempt is now many steps with tools (search, a terminal, a code runner), rewarded at the end, for example when a software task’s tests pass. Open 2025 reports such as Kimi K2 and GLM-4.5 describe it.
• Rubric rewards. For tasks with no checker, another model grades against a written rubric. It inherits the noisy-verifier ceiling: the policy can learn to please the judge.
• Hybrid thinking. One model that can answer at once or think first, with a thinking budget set by the caller. Qwen3 did this openly in 2025, then went back to separate Instruct and Thinking models a few months later for quality; DeepSeek-V3.1 and Claude’s extended thinking keep both modes in one model. Whether one model or two is better is still an open design choice. Many APIs now expose an effort setting.

For a support bot this suggests a practical setting. Turn the thinking budget off for questions like “what is my balance”, which need no working, and leave it on for disputed refunds, which do. Thinking tokens are billed like any other output, so every unnecessary one costs money. Make that choice when a conversation starts, though: on many APIs, changing the thinking or effort setting partway through a conversation invalidates the cached messages, and the next call pays full price for the whole history unless the provider offers a cache-preserving way to change it.