From raw text to assistant

How does a next-token predictor become a helpful assistant?

About 65 minutes · open lesson · LLM engineering

Why did the raw model answer a question with more questions?

Take a real open model in its raw form, straight out of pretraining, before any of the training that turns it into a chat assistant. Nobody puts this version in an app, but every assistant starts from one.

Type in a warm-up question, the sort any assistant answers in a second:

“What is the capital of France?”

Pause and predict This model has read more text than any person alive. What do you expect it to reply?

The model has almost certainly seen the answer many times, so why does it ask questions back? Because it did exactly what it was trained to do: it continued the document. On the web, a quiz question is very often followed by more quiz questions, so more questions are a likely continuation.

Yet the assistant on your phone answers the same question, and it has the same architecture and the same kind of weights. Something happens between the raw model and the assistant.

An assistant is made in stages. Pretraining produces a model that can continue any text. Supervised fine-tuning teaches it that the text to continue is a conversation, and that its part is the helpful reply. Preference optimisation teaches it which replies people actually prefer. Every stage is gradient descent on the same weights. Only the data and the loss change.

The problem

First, where did that raw model come from? You ran the training loop yourself in Training GPT and watched the loss fall. At scale the loop itself is the same, but almost everything around it changes.

Data quality matters a great deal. Groups that train open models report that filtering and de-duplication were some of the most effective things they did, and controlled experiments agree. The sizes are hard to picture: Meta reports more than 15 trillion tokens for Llama 3. Your Shakespeare file has about one million characters.

The result is called a base model. The model at the top of this page, the one that answered with more questions, is a base model, and the rest of this page is about turning it into an assistant.

The problem: A base model continues text. Users want their question answered, in a helpful manner, and they want the model to stop afterwards.
The naive fix: Prompt engineering: write the beginning of a document in which a helpful answer is the likely continuation, for example a fake FAQ page or an interview transcript.
Why it fails: It is fragile. Every task needs its own disguise, the model may continue with another question or never stop, and nothing pushes it towards honesty or away from harmful output.
The idea: Keep training the same weights, first on demonstrations of good conversations, then on human judgements about which answers are better.
The trade-off: The model now optimises for what demonstrators wrote and what raters liked. That is a stand-in for “good”, and stand-ins can be gamed: flattery, padding, and confident tone can score well.

A mental model

Making an assistant uses three kinds of teacher, one after another. Each one takes over when the one before it has taught what it can.

teacher	stage	where the right answer comes from
1 · Read everything	Pretraining	The next token of a real document. No labelling, so trillions of tokens.
2 · Watch demonstrations	Supervised fine-tuning (SFT)	The next token of a demonstration reply: written by people, or today often by a strong model and then filtered. Far fewer than pretraining tokens.
3 · Feedback on your own attempts	Preference optimisation	No single right token for “write a haiku”, but a person can say which of two attempts is better.

A common summary is “pretraining gives knowledge, the later stages shape behaviour”. It is a good first approximation with real exceptions: fine-tuning can teach some new facts or damage existing abilities, and the newer reinforcement-learning stages (in Reasoning models) appear to improve problem-solving as well as manners.

Stage 2 needs one new piece of plumbing. A model takes a single sequence of tokens, but a conversation has roles and turns. If you have written backend code you have met this problem before: it is serialisation, turning structured data into one flat stream.

Chat template: A fixed recipe for flattening a list of messages into one long token sequence, using special marker tokens to say “a new message by this role starts here” and “this message ends here”.; For example: <|im_start|>user⏎What is a cat?<|im_end|>⏎<|im_start|>assistant⏎ … and the model continues from there.

Let's see the numbers

First, the loss mask. Take the default conversation from the template viewer. Flattened, it is 34 tokens in our toy tokenizer.

Pause and predict Supervised fine-tuning grades some of those 34 tokens and not others. Which are graded?

Second, one preference comparison. Suppose the reward model currently scores answer A at 2.0 and answer B at 0.5. How confident is it that a person prefers A?

difference of rewards	2.0 − 0.5 = 1.5
squash to a probability	1 / (1 + e^−1.5) = 1 / (1 + 0.223) = 0.818 ≈ 0.82
loss if the person did prefer A	−ln(0.818) = 0.20 (small: it expected this)
loss if the person preferred B	−ln(1 − 0.818) = −ln(0.182) = 1.70 (large: it was surprised)

This is the same “minus log of the probability given to what actually happened” that you have used since softmax. Only the event is different. Before, it was which token came next; here, it is which answer the person picked.

Third, one update. In the lab, all three weights start at 0. In comparison 1 you prefer “Paris.” with features [0.03, 0, 1] over the long non-answer with features [1, 1, 0]. The model gave A a probability of 0.5, so the surprise is 1 − 0.5 = 0.5. Each weight moves by 0.5 × (chosen feature − rejected feature):

w = [0.5 × (0.03 − 1), 0.5 × (0 − 1), 0.5 × (1 − 0)] = [−0.485, −0.50, +0.50]

Click “Paris.” in a freshly reset lab and you will see these weights, rounded to two decimals (the lab displays −0.485 as −0.48).

The math

You have already computed it. Written down, the reward model’s assumption about people is:

Notice that only the difference of rewards matters. A reward of 7 means nothing on its own. It only means “better than an answer that scores 5”.

This way of turning pairwise choices into scores is the Bradley-Terry model (Bradley and Terry, 1952). It is decades older than LLMs; the same model ranks sports teams and chess players from match results.

Then the policy (the language model) is tuned. The objective has two parts, pulling in opposite directions:

Why the leash? The reward model was only trained on answers that look like what the SFT model writes. Far from those, its scores are unreliable guesses, and without the leash the optimiser wanders off to strange text that happens to score well. You saw that in step 3 of the lab with β near 0.1.

Reinforcement learning, in the four words you need: Learning from a score for what you did, instead of from a worked example of what you should have done.; For example: The policy is the language model. The state is the text so far. An action is choosing the next token. The reward is one number that arrives only after the whole answer is finished. (In practice the KL penalty below is often charged token by token along the way.)

DPO: the same data without a reward model or a reinforcement learning loop

RLHF as described has many moving parts: a separate reward model, sampling answers from the model during training, and a reinforcement learning algorithm to turn those scores into weight updates. The classic choice is PPO, proximal policy optimization. It works, and it is fussy to tune.

DPO, direct preference optimization (Rafailov et al., 2023), starts from a mathematical observation. For the objective above, the best possible policy can be written in closed form: policy(y) ∝ SFT(y) × e^r(y)/β. (That is precisely the formula the lab uses for its “tuned model” column.) Turn that around and the reward can be expressed through the policy: r(y) = β × log( policy(y) / SFT(y) ), plus a term that cancels when you subtract two rewards.

Substitute that into the Bradley-Terry loss and the reward model disappears:

loss = −log σ( β × [ log(π(chosen)/π_ref(chosen)) − log(π(rejected)/π_ref(rejected)) ] )

Here π is the model being tuned and π_ref is the frozen SFT model. In words: make the chosen answer more likely, relative to the reference, than the rejected answer. It is an ordinary supervised loss on fixed pairs: no sampling, no separate reward network.

Trade-off: plain (offline) DPO learns only from the fixed pairs in the dataset, not from fresh attempts of the current model. Iterative DPO fixes part of that by sampling new pairs from the current model each round, as Llama 3 did. Which approach gives better models, and when, is an active debate, and many published recipes combine several methods.

What you should remember

Pretraining is the loop you know (batch, forward, loss, backprop, update) at scale, with data filtering and de-duplication doing much of the work. It gives a base model that continues text and holds most of the knowledge, but does not follow instructions.
SFT is the same next-token loss on demonstrations. A chat template flattens the conversation into one token sequence with role markers, and the loss is usually counted only on assistant tokens, including the end marker.
Preference optimisation exists because many prompts have no single right answer but people can compare two. RLHF fits a reward model with P(A ≻ B) = σ(r(A) − r(B)), then tunes the model against it on a KL leash. DPO uses the same pairs with no separate reward model.
The reward is a stand-in for what people want. Optimising a stand-in invites reward hacking: length, flattery and confident tone can win. These stages shape behaviour; they do not give the model a way to check whether what it says is true.

Where this appears in a real LLM

In this lesson

Illustrative, hand-written outputs for each stage
A toy tokenizer and one ChatML-style template
A reward model with 3 readable features and 8 comparisons
The tuned policy computed exactly over 5 candidate answers

In a real system

Base and instruction-tuned versions of many open models are both published, so you can observe the difference yourself
Each model family has its own template and special tokens, shipped with the tokenizer
The reward model is a full Transformer trained on tens of thousands of comparisons or more, increasingly with AI-generated feedback guided by written principles
The policy is tuned by many small gradient steps on its own sampled answers, often in several rounds mixing methods

“Pretrain, then SFT, then preference tuning” is the textbook order. Real pipelines blur it: several rounds, instruction-like data mixed into late pretraining, many variants of the preference loss. A growing share of the data is synthetic, written or judged by another, usually stronger, model (the subject of Small models from big ones). The exact recipes of commercial assistants are mostly not public, so do not trust anyone’s detailed diagram of them.

One stage is still missing from this picture. When the task has a checkable answer, such as a maths result or code that must pass tests, the reward does not need to be learned from people at all. That changes what RL can do, and it is the subject of Reasoning models, at the end of this part.

If you type the France question into the chat version of the same model, it answers “Paris.” and stops. The network underneath is the same one; it has been through two more stages of training.

Open the interactive lesson

The interactive lesson adds the predictions with their answers, the labs, and graded practice with spaced review.