Modern LLM architecture
What is different inside a 2020s model compared with our tiny GPT?
How much of DeepSeek-V3 have you already built?
DeepSeek-V3 is an open-weight model released in December 2024, and many models since have borrowed its design. Its weights come with a config.json file that describes the architecture in a few dozen lines. Here are nine of them:
You can read the first lines already. num_hidden_layers is your n_layer: 61 blocks, where your tiny GPT has 4. hidden_size is your n_embd: 7,168 numbers per token, where yours has 128, so the model is 56 times wider. silu is a smooth activation function, a close relative of the GELU in your MLP.
The rest is unfamiliar. rms_norm_eps belongs to a norm called RMSNorm, which your model does not have. rope_scaling refers to a way of encoding position that needs no position table. kv_lora_rank: 512 means the KV cache keeps one compressed vector of 512 numbers per token in each layer, where your model would keep a full key and a full value for every head. And the last three lines say that the MLP in most blocks has been replaced by 256 smaller MLPs called experts, plus one shared expert, and that only 8 of the 256 run for any one token.
Pause and predict Those unfamiliar lines look like a different design with far more parameters. How much of this config do you expect to be new?
Same skeleton, swapped parts
The whole comparison fits in one table. The left column is what you built. The right column is what you would find in most open models today.
| Part | Our tiny GPT (GPT-2 style) | Typical open model, 2024 to 2026 | Fixes |
|---|---|---|---|
| Normalisation | LayerNorm | RMSNorm (root mean square norm), often also on q and k (QK-norm) | cost, stability |
| Position | learned table, added to embeddings | RoPE (rotary position embedding): rotate q and k | position |
| MLP | 2 matrices + GELU | 3 matrices, gated (SwiGLU) | quality per parameter |
| Attention heads | one K/V head per query head | shared K/V heads (GQA), a compressed cache (MLA), or a sliding window in some layers | memory |
| Attention execution | build the full T×T table | FlashAttention: same result, tiled | memory, speed |
| Context window | 64 (GPT-2: 1,024) | about 128 thousand is common; open models reached 1 million in 2025 (Qwen2.5-1M, Llama 4), a year after Google’s closed Gemini 1.5 | usefulness |
| Size | 0.8M parameters, all used | billions; dense or, increasingly, mixture-of-experts (MoE) | capability per unit of compute |
The Size row hides the most interesting fact: in the biggest open models, most parameters sit idle for any one token. The part on mixture-of-experts below explains how.
Several of these swaps came from trying things and measuring the loss, with no theory behind them at the time, and the explanations below say which ones.
Let's see the numbers
The KV cache of two real models, by hand
For every token we store one key and one value (that is the 2), in every layer, for every K/V head, head_dim numbers each. In 16-bit precision each number is 2 bytes.
| Llama-2 7B (MHA) | Llama-2 70B (GQA) | 70B if it used MHA | |
|---|---|---|---|
| layers | 32 | 80 | 80 |
| query heads | 32 | 64 | 64 |
| K/V heads | 32 | 8 | 64 |
| head_dim | 128 | 128 | 128 |
| bytes per token 2 × layers × KV heads × head_dim × 2 | 524,288 = 0.5 MiB | 327,680 = 0.31 MiB | 2,621,440 = 2.5 MiB |
| × 4,096 tokens | 2.00 GiB | 1.25 GiB | 10.0 GiB |
Read the middle column again. The 70B model is ten times larger than the 7B model, yet its cache per token is smaller, because 8 K/V heads replaced 64. The saving is exactly 64 ÷ 8 = 8×.
Stretch the context to 131,072 tokens and the 70B cache is 40 GiB with GQA, or 320 GiB with plain multi-head attention. For comparison, the 70B weights themselves take about 140 GB in 16-bit. Check all of these in the calculator.
Now MLA. DeepSeek-V3 has 61 layers and caches 512 + 64 = 576 numbers per token per layer, with no separate key and value and no K/V heads:
61 layers × 576 × 2 bytes = 70,272 bytes ≈ 68.6 KiB per token
That is about 4.7 times less than Llama-2 70B’s 327,680 bytes with GQA, in a model with roughly ten times the parameters. At 131,072 tokens it comes to about 8.6 GiB, against 40 GiB. (The calculator only models GQA, so check this one by hand.)
RoPE by hand
Take the simplest vectors: q = [1, 0] and k = [1, 0], with θ = 30° per position. Put the query at position 5 and the key at position 3.
| positions (5, 3) | positions (105, 103) | |
|---|---|---|
| q rotated by | 5 × 30° = 150° → [−0.866, 0.5] | 105 × 30° = 3150° = 270° → [0, −1] |
| k rotated by | 3 × 30° = 90° → [0, 1] | 103 × 30° = 3090° = 210° → [−0.866, −0.5] |
| dot product | −0.866×0 + 0.5×1 = 0.5 | 0×(−0.866) + (−1)×(−0.5) = 0.5 |
| angle between them | 150° − 90° = 60° | 270° − 210° = 60° |
Both vectors have length 1, so the dot product is cos(60°) = 0.5. The angle between them is (5 − 3) × 30° in both cases. One hundred positions later, the score is identical.
RMSNorm by hand
For x = [2, 4, 6, 8]: the squares are 4, 16, 36, 64. Their mean is 30. √30 = 5.477. Divide: [0.365, 0.730, 1.095, 1.461].
LayerNorm on the same vector: mean 5, centred [−3, −1, 1, 3], spread √5 = 2.236, result [−1.342, −0.447, 0.447, 1.342]. The numbers differ, but both do the same job: the output has a standard size no matter how large the input was.
The math
Three short formulas. You have already computed each of them.
LayerNorm would first replace x by x − mean(x), and add a learned bias at the end. That is the entire difference.
In words: rotating both vectors and then comparing them gives the same answer as leaving the query alone and rotating the key by the difference. Rotating the whole picture does not change any angle inside it.
What you should remember
- A modern open model is the same skeleton you built: blocks of attention + MLP with residuals. The differences are targeted repairs, each with a cost. RMSNorm is LayerNorm minus the mean and bias; SwiGLU is a gated MLP that measured better, for reasons that are mostly empirical.
- RoPE rotates q and k by position × θ, so the score depends only on the offset between two tokens. It needs no position table and adds no parameters, but it does not make unlimited length free.
- KV cache bytes = 2 × layers × KV heads × head_dim × tokens × bytes. GQA shrinks the “KV heads” factor (64 query heads on 8 K/V heads = 8× less cache); MLA, sliding windows and hybrid layers shrink it further. FlashAttention is exact attention with a better memory access pattern.
- Long context costs T² compute and linear cache memory, and fitting in the window does not mean being used well. MoE, now the usual design for the largest open-weight models, buys more parameters at the compute of its active parameters, and pays in memory and routing complexity.
Where this appears in a real LLM
One block, side by side
| Step inside a block | Our tiny GPT block | A Llama-style block |
|---|---|---|
| before the block | x = tok_emb + pos_emb | x = tok_emb |
| 1. normalise | LayerNorm(x) | RMSNorm(x) |
| 2. make q, k, v | 4 heads each for q, k, v | many q heads, fewer k/v heads (GQA) |
| 3. position | (already added at the input) | rotate q and k with RoPE |
| 4. attention | softmax(QKᵀ/√d)V, full T×T table | softmax(QKᵀ/√d)V, FlashAttention kernel, K/V read from cache |
| 5. residual | x = x + attn | x = x + attn |
| 6. normalise | LayerNorm(x) | RMSNorm(x) |
| 7. MLP | Linear → GELU → Linear (4d wide) | SwiGLU (3 matrices, ≈ 8d/3 wide), or a router + experts (MoE) |
| 8. residual | x = x + mlp | x = x + mlp |
| biases, dropout | biases everywhere, dropout 0.1 | usually no biases (some keep them on q, k, v, as Qwen2 does), no dropout in pretraining |
Rows 5 and 8, the residual stream, did not change at all. Neither did the attention formula, the causal mask, the training loss or the generation loop.
- In this lesson
- 4 blocks, 128 numbers per token, 4 heads of 32
- 64-token window from a learned table
- KV cache (if you add one, in 32-bit): 256 KiB
- every parameter used for every token
- In a real system
- Llama-3 70B: 80 blocks, 8,192 numbers per token, 64 query heads of 128 sharing 8 K/V heads
- 8,192 to 128,000+ tokens via RoPE
- KV cache: gigabytes per conversation, the main serving constraint
- the largest open-weight models are mostly mixture-of-experts: DeepSeek-V3 runs about 37B of its 671B parameters per token
“Modern LLM = GPT-2 + these repairs” is a simplification. Real models combine them differently (GQA or MLA, all-global or interleaved local and global layers, dense or MoE, pure attention or a hybrid) and also differ in tokenizer, norm placement, initialisation and quantisation. Every mechanism here is published and readable in open model code and configs.
You can now read every line of the DeepSeek-V3 config from the top of this page and name the problem it fixes. The model it describes still only continues text; the next lesson is about how training turns it into an assistant that answers.