Evals: how you know it works

If a bot answers four demo questions correctly, what does that prove?

A team demos its support bot. Someone types four questions, and the bot gives four clean answers, each with a citation. The room is ready to ship it.

Those four questions are the whole test. Two hundred real tickets from last month are sitting in a folder, and nobody has run them. Before running them, it is worth seeing what four questions can hide. The toy RAG bot from the RAG lesson and the evaluation harness that ships with the repository will show it.

Suppose someone proposes a change: split the documents into chunks of one sentence each, on the theory that smaller chunks give sharper matches. You try the four demo questions, and all four still answer correctly. The change looks free.

Pause and predict All four demo questions still pass after the change. Should you ship it?

There is a second problem, and it points the other way: 16 versus 14 out of 24 is not enough evidence that the system got worse at all.

An eval is a repeatable measurement of whether your LLM system does its job: a fixed set of inputs, a rule for marking outputs, and a number that comes with an honest statement of its uncertainty. It is how you replace “it looked fine when I tried it” with evidence. It has a price: building the set is real work, and a small set gives a number that is mostly noise.

The problem

Pause and predict Your first instinct as a programmer may be to treat an eval as a regression suite. Name one way in which an eval differs from a unit test.

Golden set

A fixed list of inputs for which people have decided what a good outcome is. It is the dataset your system is graded on.

For example: Question: “what is the wifi password”. Expected: a refusal, because no document contains it.

The anatomy of an eval

Every eval, from a 20-line script to a public benchmark, has the same four parts. Here they are with real rows from this lesson’s harness:

1. The dataset decides what you can find out

An eval only sees failures its questions can trigger. Our 24 items come in four categories, and that is deliberate:

Two rules protect the dataset’s value.

Keep a held-out part. If you tune the threshold until the score on these 24 items peaks, you have fitted the system to these 24 items, exactly like a model that memorises its training set. It is the validation loss idea again: keep some items you never look at while tuning, and check them at the end.

Watch for leakage. If an eval question (or its answer) also sits in the few-shot examples of your prompt, in your fine-tuning data, or in the model’s pretraining data, a high score tells you what the system has memorised and very little about what it can do.

2. The task calls the system like a user would

The eval sends the question in and records everything that comes back: the answer, the retrieved chunks, the citation. The intermediate results are what let you locate a failure later.

3. The scorer is where most evals go wrong

Pause and predict The reference is “within 5 minutes”. The bot answers “Primary oncall must acknowledge pages within five minutes”. Does a substring scorer pass it?

LLM-as-judge

Using a language model as the scorer: give it the question, the answer and a marking scheme, and ask for a verdict.

For example: “Does the answer state the same fact as the reference? Wording and number format do not matter. Reply PASS or FAIL with one sentence of reasoning.”

So a judge is usable, on three conditions. Give it a specific rubric and a reference answer, not “rate this 1 to 10”. For pairwise comparisons, run both orders and count a flip as a tie. And calibrate it: have people label a sample, and measure how often the judge agrees with them. If it agrees with people less than people agree with each other, fix the rubric before you trust a single score.

A note on this lesson’s harness: it has no LLM in it. Its “rubric judge” is a deterministic function that applies three checks a judge prompt would ask for, so every number is reproducible offline. It has none of a real judge’s flexibility and none of its biases.

4. Component metrics tell you where, end-to-end metrics tell you whether

When an answer is wrong, the first question is: which part failed? A RAG system has at least three separately measurable properties:

• Retrieval: was the chunk that holds the answer among the k retrieved? Measurable without any LLM.
• Faithfulness (groundedness): is every claim in the answer supported by the retrieved text? An answer can be faithful and still wrong, if retrieval brought the wrong text.
• Answer correctness: does the final answer match what a person expects? This is the end-to-end number users feel.

recall@k

Of the passages that should have been retrieved, what share showed up in the top k results?

For example: 17 answerable questions, one right passage each. The right passage is in the top 3 for 16 of them: recall@3 = 16/17 = 94%.

Pause and predict In our harness, retrieval recall is 94% while answer correctness is 67%. Where should the next week of work go?

Let's see the numbers

Pause and predict Your eval has 50 items and the system passes 40. You report 80%. How far from the truth could that be, for a 95% interval?

Treat each item as a coin that comes up “pass” with the system’s true accuracy p. Fifty flips of such a coin do not always give the same count. The typical wobble of the measured rate is called the standard error:

So “80%” means “somewhere between about 69% and 91%”. A change that moves the score from 80% to 83% is one and a half items. It is far inside the noise.

The square root is the painful part. To halve the interval you need four times the items:

Now the real output of python phase6-engineering/eval_harness.py on our 24 items, which is also what the lab shows:

The last column is Cohen’s kappa: agreement with the human after subtracting the agreement two careless raters would reach by luck alone. 1 is perfect, 0 is coin-flipping. The deep dive below works it out.

Same system, same answers, and the score runs from 17% to 67% depending on the scorer.

The rubric judge is closest to the human, and still disagrees on one item. Asked “does the oncall rotation change every friday”, the system quotes “The oncall rotation changes every Monday at 10am”. A person accepts that.

The reference says “no, every Monday”, the word “no” is missing, and the judge fails it. The system was right and the eval was wrong. Reading your failures will regularly end with fixing the eval.

Now put an interval on the rubric score. The bootstrap is the lazy way to get one: draw 24 marks at random from your 24 marks, with repeats allowed, average them, and do that ten thousand times. The middle 95% of those averages is your interval. Here it runs from 45.8% to 83.3% around a score of 66.7%.

Now return to the regression check from the opening. A scores 66.7% and B scores 58.3%. Two items pass only under A, and no item passes only under B.

The sign test asks whether a split of two to nothing would be surprising if each disagreement were a fair coin flip. It would not: p = 0.50. So the verdict is that you cannot tell, because two fair coin flips land the same way half the time.

Do the same arithmetic for the 200 real tickets: at 80% accuracy, the 95% interval is still ± 5.5 points. That is far narrower than the ± 16 points you get from 24 items, and it is still a range.

This is the same discipline as latency work. You would not compare two builds on one request each, and you would not report a p99 from 20 samples (System Design builds that habit in Design metrics and monitoring). A pass rate is a statistic. Give it the treatment you give every other statistic on your dashboards.

The math

This assumes items are independent draws from the questions you care about. If your 500 items are 50 documents with 10 questions each, they are not independent, and the true uncertainty is larger than the formula says.

Comparing two systems. Run both on the same items and look at each item’s difference. Items that both pass, or both fail, cancel out. Only the items where they disagree carry information, and the question becomes: of those disagreements, is the split lopsided enough that a fair coin would rarely produce it? That is the sign test in the harness. Comparing two separately reported scores throws the pairing away and keeps all the noise of both. Evan Miller’s “Adding Error Bars to Evals” (2024) makes this the central recommendation, noting that models tend to get the same questions right and wrong.

Repeated runs. If your system samples with temperature above 0, one run per item adds a second source of noise on top of the choice of items. Run each item several times and average, or the same commit will score differently on two CI runs.

What you should remember

1. An eval has four parts: dataset, task, scorer, aggregate. Each can be wrong independently. Build the dataset from real traffic, past failures and adversarial cases, and keep a part of it held out.
2. The scorer is a design decision. Exact match is too strict, embedding similarity too lenient, an LLM judge is flexible and has measured biases (position, verbosity). Calibrate any scorer against human labels and report the agreement.
3. Separate component metrics from end-to-end metrics. Retrieval recall@k, faithfulness and answer correctness fail for different reasons. Measure each so a failure can be located.
4. A score is a range. SE = √(p(1−p)/n). At n = 50 and 80%, that is ± 11 points. A 3-point change is noise. Quadruple the items to halve the interval. Compare paired: same items through both systems, count the disagreements, test them, and put that test, plus per-category floors, in CI.

Where this appears in a real LLM

In this lesson

• 24 hand-written items, one tiny corpus
• A deterministic pipeline, so one run per item is enough
• A rule-based stand-in for the judge
• One author’s labels as “the human”

In a real system

• Hundreds to thousands of items, sampled from production logs and refreshed
• Sampling noise: several runs per item, and results averaged
• LLM judges with rubrics, validated against human labels on a sample
• Several annotators, with their agreement measured before anyone trusts the labels

Offline evals and online signals

An offline eval answers “did this change break what we already know about?” before users see it. It cannot answer “is this what users need?”, because its questions are yesterday’s.

Production systems pair it with online signals: A/B tests on a business metric, thumbs up and down, how often users rephrase or abandon, how often guardrails fire. Those signals are noisy and slow, and they are the only ones that measure the real distribution. The loop closes when a production failure becomes a new golden item.

What public benchmarks measure, and why they keep dying

A leaderboard might show a model scoring 92% on MMLU. That number comes from a public benchmark, which is someone else’s golden set built for someone else’s task. Everything above applies to public benchmarks too, plus two problems that only show up at internet scale.

Saturation. When every strong model scores near the top, the remaining gaps are smaller than the benchmark’s noise, and the rest is label errors. The benchmark stops telling models apart. It is your ± 11 points again, at the scale of a whole field.

Contamination. Test questions sit on the web, and the web is training data. A high score may then reflect questions the model memorised during training. It is the few-shot leak from the exercise above, at internet scale.

And a leaderboard measures a different task: exam questions are not your support tickets. On contamination, one controlled check, GSM1k (Zhang et al., 2024), wrote fresh look-alikes of a popular grade-school maths benchmark and found accuracy drops of up to 8% for some model families, while many frontier models showed little sign of overfitting. Contamination is real, uneven, and hard to rule out from the outside.

So the field keeps building harder, fresher sets, and every one of them will saturate or leak in its turn. The names will change. The two questions to ask of any score stay the same: is there room left at the top? and could the model have seen these questions?

The practical order of operations: use public benchmarks to shortlist two or three models. Build your own golden set from your own traffic, starting with 50 items this week and not 5,000 next quarter. Read the failures by hand before automating anything. Add the scorer that would have caught what you read. Put it in CI with a paired test. The tooling is the easy part. The hard part, and the one most teams skip, is the afternoon spent writing down what “good” means for each item.

Once you can show that the system works, the next question is whether it keeps working when a thousand users arrive at once, which is the subject of Serving many users at once.