How to read an LLM paper

Why does a line on page 4 look familiar?

“Attention Is All You Need” is the 15-page paper that eight researchers working at Google posted in June 2017. Every model in this course descends from it.

Most developers never open it, and the first page gives them a reason to close it: the abstract is full of BLEU scores, a measure of translation quality you have not met. Keep reading to page 4 and you reach this line:

Pause and predict Where have you seen this line before?

Papers are hard for two reasons that have nothing to do with intelligence: the notation is unfamiliar, and the format hides the important parts. Both can be learned in an afternoon.

The problem

A paper is an argument addressed to expert reviewers. It states a claim, gives the evidence for it, and explains why earlier work falls short. It was written to convince reviewers, so it does not stop to teach you anything along the way.

Few people read a paper from the first word to the last, and the method below makes that a deliberate choice: you read in several passes and decide after each one whether the paper deserves another.

The problem

New ideas in this field appear as papers first, often years before a good explanation exists. An applied engineer has to be able to read them.

The naive fix

Start at the first word and read to the last, understanding each sentence before moving on.

Why it fails

You spend your energy on the related-work section and on notation before you know what the paper claims. You cannot tell the one equation that matters from the ten that do not. Most people stop on page 3.

The idea

Read in several passes, each with a different goal and a time limit. Find the claim first, then the evidence, and only then the details. Decide after each pass whether the paper deserves the next one.

The trade-off

The early passes feel like cheating, and you must tolerate not understanding things yet. The last pass is expensive, so you can afford it for a few papers a year.

A method: three passes

The method comes from a two-page note by S. Keshav, “How to Read a Paper”, published in ACM SIGCOMM Computer Communication Review in 2007. It is worth reading in full. Here is a summary of the three passes it describes:

After the first pass the note asks you for five answers, the “five Cs”: the paper’s category, its context (what it builds on), whether its assumptions look correct, its contributions, and its clarity.

Where the content hides in a machine-learning paper

The note was written for networking research. For ML papers, add this to the first and second pass: look at the pictures first. Five places carry most of the information; here they are in the paper we are about to read, at their real page numbers.

1. The architecture figure is the authors’ own summary of the method in one picture.
2. The one central equation carries the idea. A paper may print twenty equations, but usually only one is the idea, and the rest define terms or report settings.
3. The main results table holds the evidence for the claim in the abstract.
4. The ablation table is often the most useful part for a practitioner, because it shows which pieces of the method actually matter.
5. The hyperparameters, in an appendix or a training section, are what you need to reproduce anything, and they show what the authors had to tune.

Ablation

An experiment that removes or changes one piece of the system and measures what happens. It is how authors show that each piece matters.

For example: Table 3 of the Transformer paper: keep everything fixed and vary the number of heads. One head scores 0.9 BLEU below the best setting. Too many heads also scores lower.

Reading a results table like a sceptic

A number printed in bold in a results table is the authors’ claim, and you should check it before you believe it. Ask these questions, in this order:

• What is the baseline, and who ran it? Numbers copied from older papers may come from weaker tuning or less compute.
• Is the compute comparable? The Transformer paper is exemplary here: Table 2 has a training-cost column next to the scores.
• What was tuned, and on which data? If choices were made by looking at the test set, the score is optimistic.
• Are there error bars or repeated runs? Often there are none, and then a gap of a few tenths of a point may be noise. In Evals you computed how wide such intervals are.
• Could the benchmark be in the training data? For LLMs trained on the web, contamination is a standing risk. It is the same leak you saw in Evals and From raw text to assistant, at internet scale.
• What is missing? Look for the comparison that was not run, the task where the method lost, and the cost nobody mentions.

One convention that makes formulas look backwards

This course writes a layer as x @ W + b: the vector is a row and stands on the left. Many papers write Wx + b: the vector is a column and stands on the right, so their W has shape (out, in), the transpose of ours.

Nothing is different except the bookkeeping, but shapes and the order of matrices flip. The LoRA paper writes W₀x + BAx with B of shape d × r. Your code computes (x @ A) @ B with B of shape r × d. The maths is the same. The Transformer paper happens to use rows, like this course: its FFN is max(0, xW₁ + b₁)W₂ + b₂.

PyTorch straddles both worlds: nn.Linear stores its weight as (out, in), the paper convention, and computes x @ W.T + b, the row convention. You met the consequences in the previous lesson.

Let’s see the numbers: a third pass in miniature

Table 3 of the paper says the base model has 65 × 10⁶ parameters. A third-pass reader does not take that on trust. You have counted parameters since Build GPT, so rebuild it from the paper’s own hyperparameters: d_model = 512, d_ff = 2048, N = 6, a shared vocabulary of about 37,000.

Pause and predict Before you count: how close will your own count come to the paper’s 65 million?

What did that buy you? You now know the decoder layer is a third larger than the encoder layer, because of cross-attention. You know a third of the base model is vocabulary. And you know the count depends on sharing one embedding matrix three ways, which section 3.4 mentions in a single sentence.

The math

Two pieces of the paper that are new to you, in the course’s usual form.

Label smoothing

During training, do not ask for 100% on the correct token. Ask for a little less, and spread the remainder over the other tokens. The model is discouraged from becoming extremely confident.

For example: Vocabulary of 5, correct token is the third, smoothing 0.1. Instead of the target [0, 0, 1, 0, 0], train against [0.02, 0.02, 0.92, 0.02, 0.02].

What you should remember

1. Three passes (from “How to Read a Paper”, 2007): find the claim in ten minutes; grasp the argument and its evidence in an hour, skipping proofs; re-create the work only for the papers that matter. In an ML paper, look first at the architecture figure, the one central equation, the main results table, the ablation table and the hyperparameters.
2. Notation is code you already wrote: ∈ ℝ^n×d is a shape, Σ is a loop, 𝔼 is .mean(), ∇ is .backward(), ⊙ is *. Many papers use Wx + b with column vectors: the transpose of this course’s x @ W + b.
3. The original Transformer is an encoder-decoder: unmasked self-attention over the source, masked self-attention over the target, and cross-attention with queries from the decoder and keys and values from the encoder. GPT is the decoder alone, without cross-attention.
4. Treat results as claims, and check the baselines, the compute, the tuning, the error bars, possible contamination, and what is missing. Even this paper says “we suspect”, and has a typo in a headline number.

Where this appears in a real LLM

In this lesson

• Decoder-only, pre-norm, learned positions, GELU, constant learning rate
• One sequence: prompt and answer together
• Evaluated by validation loss and by reading samples
• 0.8M parameters, minutes on a CPU

In a real system

• The 2017 paper: encoder-decoder, post-norm, sinusoidal positions, ReLU, warm-up then decay, label smoothing 0.1
• Two sequences joined by cross-attention
• Evaluated by BLEU on WMT 2014: 28.4 and 41.8 for the big model
• 65M and 213M parameters; 12 hours and 3.5 days on 8 P100 GPUs

What to read next, and in what order

You are prepared for every paper on this list. Use the three passes. Do a third pass on at most two or three of them.

The order is by dependency, not by date: the GPT line first, then how size and data trade off, then how a base model becomes an assistant, then the engineering papers behind the models you can download (DeepSeek-V3 pulls several of them together), then systems built around a model, then a look inside a trained model, and reasoning models last.

If you followed this lesson with the paper open, you have done a first pass, most of a second, and a small third, because you counted the parameters yourself. The next lesson turns from reading about models to running an agent for real users.