Fine-tuning

Why does the bot still sound like a form letter?

With retrieval, a payments company’s support bot now finds the right policy. It still does not sound like the company.

The brand team’s rule is short: two warm sentences, no “Dear Valued Customer”, and no promise the policy does not allow. Read a sample of the bot’s answers and one begins “We deeply regret any inconvenience this may have caused” and goes on for a whole paragraph.

The rules are already in the prompt, and they mostly work. The prompt has grown to 800 tokens of tone instructions and sample replies, and one reply in fifty still drifts back into the stiff, apologetic paragraph.

Two problems that used to push teams towards fine-tuning are now handled by the API. If the app needs each reply wrapped in JSON, structured outputs handle it: you pass a JSON schema in strict mode, and the provider constrains decoding so that only tokens that fit the schema can be sampled. Apart from refusals and replies cut off by the token limit, the envelope always matches. Prompt caching bills a repeated 800-token prefix at about a tenth of the normal input price. Neither of them reaches the voice inside the envelope. A schema can require a string field called reply, but it cannot make the sentence in that field sound like the company.

Pause and predict The JSON envelope is guaranteed and the long prefix is cached, yet one reply in fifty still ignores the tone rules. What would fix the habit itself?

The problem

You have already seen fine-tuning once. In From raw text to assistant, SFT turned a raw text predictor into something that answers questions. Here you do the same thing to a model, for your task.

Fine-tuning

Continue training a model that is already trained, on a small set of examples of the behaviour you want.

For example: 2,000 pairs of (customer message → ideal reply in your house style). Run the usual training loop for a few passes with a small learning rate.

Mechanically, nothing new is needed: it is the same model, the same loss and the same gradient descent.

Three things differ. The data is yours. The starting point is a trained model. The learning rate is small, because you want to nudge a working system without rebuilding it.

If you wrote the training loop in Training GPT, you have the mechanics. The loop is the easy part. The hard problems are what it costs and what the model forgets.

Pause and predict You want one tuned model per business customer. What goes wrong if you train every weight of a 7B model for each one? Name one thing.

A mental model

First decide whether you should fine-tune at all

A plan that comes up in almost every team: fine-tune the model on all the product docs, so that it knows everything and the search step can go.

Pause and predict Is that a good plan?

Each of the three tools changes something different. Reach for them in this order, because each costs more than the one before.

The data is the specification

The loss rewards one thing: predicting your examples. So the model imitates everything in them.

The tone you wanted, yes. Also the typos, the inconsistent greetings, the agent who always wrote “pls advise”, and any wrong answers.

That is why a few hundred excellent, consistent examples usually beat many thousands of sloppy ones.

Why models forget

In Why LLMs know things you saw that abilities are spread across shared weights. The same weight helps with many things at once.

Now look at the loss during fine-tuning. It contains only your new examples. Nothing in it mentions French, or Python, or Shakespeare.

Gradient descent moves every weight in whatever direction lowers the new loss. It pays no attention to what else that weight was doing.

Catastrophic forgetting

While learning the new task, the model gets measurably worse at things it used to do, because the same weights were overwritten.

For example: Fine-tune a Shakespeare model on office English: office loss falls, Shakespeare loss rises.

Let's see the numbers

Now count what it costs. Take one weight matrix from a 7-billion-parameter-class model: 4096 inputs, 4096 outputs. Full fine-tuning trains 4096 × 4096 = 16,777,216 numbers in it.

Pause and predict LoRA with rank r = 8 trains a 4096 × 8 matrix and an 8 × 4096 matrix instead. What share of the 16.8 million is that?

The whole saving comes from one fact: the count grows with r × (d_in + d_out), a sum, instead of d_in × d_out, a product. Try other shapes:

The same arithmetic explains the number in the real run above. Our tiny GPT has 4 blocks. In each, LoRA wraps the qkv layer (128 → 384) and the proj layer (128 → 128), with r = 4:

4 × [ 4 × (128 + 384) + 4 × (128 + 128) ] = 4 × [ 2,048 + 1,024 ] = 12,288

The math

A normal layer multiplies its input by a weight matrix. LoRA adds a second, thin path beside it and sums the two results.

Rank

How many independent directions a matrix can write into. The product A·B is forced through a bottleneck of r numbers, so it can only ever make r independent kinds of change.

For example: With r = 1, every row of A·B is a multiple of the same single row. With r = 2, every row is a mix of two rows.

Why would a low-rank correction be enough? The model already knows the language. Fine-tuning only nudges a capable system toward a new habit.

The empirical finding behind LoRA is that the change needed for such nudges is often close to low-rank, even though W itself is not. The lab above showed both sides: a rank-2 change is captured perfectly at r = 2, and an arbitrary change still has 42% error at r = 2.

Merge or swap. After training, you can compute W′ = W + A·B·(α/r) once and serve W′: same outputs, zero extra cost per token. Or keep W shared and swap small (A, B) pairs per customer or task.

What you should remember

1. Fine-tuning changes the weights, so the prompt can stay short. The same training loop on new data, from trained weights, with a small learning rate. RAG for knowledge that changes, fine-tuning for behaviour (style, tone, narrow skills). Try prompting first, and let structured outputs handle the format.
2. The model imitates everything in your examples, including your mistakes. A few hundred excellent examples beat thousands of sloppy ones.
3. Catastrophic forgetting: nothing in the new loss protects old abilities, and they share weights. Always evaluate on a held-out set and re-test old capabilities.
4. LoRA: freeze W, train a low-rank correction A·B with B = 0 at the start. About 1% of the numbers; base model preserved exactly; adapters can be merged or swapped.

Where this appears in a real LLM

In this lesson

• A 0.8M-parameter character GPT; a 27×27 bigram in the browser
• Two tiny synthetic corpora standing in for “old skill” and “new task”
• Rank 4 on the attention layers: 12,288 trainable numbers
• Forgetting measured as one loss on one old corpus

In a real system

• Billions of parameters; the pretrained weights are usually downloaded, not trained by you
• Prompt → response pairs (SFT), often followed by preference tuning such as DPO (see Alignment and safety). The ideal replies are often written by a bigger model rather than by people (see Small models from big ones)
• LoRA on attention and often feed-forward layers, commonly combined with a quantised (compressed) frozen base so it fits on one GPU
• Forgetting checked with a battery of benchmarks and task-specific regression tests

The assistant you chat with is itself a fine-tuned model: pretraining, then SFT, then preference tuning and reinforcement learning, as in From raw text to assistant. How those later stages teach a model what it should not do is the subject of Alignment and safety. Serving many LoRA adapters over one shared frozen base is standard practice, because swapping a small adapter is far cheaper than loading another full model.

What practitioners use

Almost nobody trains every weight of a large model for a product feature. The usual choice is LoRA, or QLoRA when GPU memory is tight. QLoRA stores the frozen base in a 4-bit quantised format and trains LoRA adapters on top of it in higher precision. Its authors used it to fine-tune a 65-billion-parameter model on a single 48 GB GPU (Dettmers et al., 2023).

For open models, the common tools are Hugging Face’s TRL library, whose trainers cover SFT and DPO and use the PEFT library for LoRA, and Unsloth, which makes LoRA and QLoRA runs on one GPU faster and lighter on memory. If you would rather not manage GPUs, hosted services take a file of examples and return a tuned model or adapter: Google’s Vertex AI does this for Gemini models, and providers such as Together AI do it for open models. Check what your provider offers before you plan around it. OpenAI, for example, has said it is winding down its fine-tuning platform and no longer accepts new users.

SFT is often followed by preference tuning with DPO (direct preference optimisation, Rafailov et al., 2023). Each example holds two replies to the same prompt, one you prefer and one you do not, and the loss raises the probability of the first relative to the second. It needs no separate reward model. It also suits the problem this lesson started with, because “this reply sounds like us and that one does not” is easier to label than a perfect reply is to write.

Before you ship an adapter, run the held-out examples through it and re-test the old abilities too, including ones nobody has complained about, because the training loss never looked at them.