LLM From First Principles

From one dot product to your own GPT, and the systems built around it.

40 open lessons · practice and review with a pass

Part 0: What is an LLM?

See the whole machine before we open it up.

What happens when I type a prompt?: What happens between typing "What is a cat?" and seeing an answer?
The surprising idea: If it is not looking answers up, what is it doing?
Your map of the course: What are the boxes inside a language model, and in what order will you build them?

Four ideas, each one a picture before it is a formula.

Vectors and the dot product: How do you ask two lists of numbers "do you agree?"
Matrices: many dot products at once: Why is everything in an LLM a matrix multiply?
Scores into probabilities: softmax: How does a model turn raw scores into "70% sure"?
Nudges: derivatives and the chain rule: If I nudge this number, how much does the result move?

Prediction, error, and the one loop that trains everything.

Adjustable functions, stacked, and how blame flows backward.

You cannot multiply the word "cat". So what do we do?

Next-token prediction, and why the simple version hits a wall.

The heart of the course. Take your time here.

Attention: How can one token look at the other tokens?
Causal masks and multiple heads: How do we stop tokens peeking at the future, and why run attention several times?
The Transformer block: What else does a block need besides attention?

Assemble the parts, watch it learn, make it talk.

Build GPT: Can you trace one token from text to prediction?
Training GPT: What actually happens during "training"?
Inference: sampling and the KV cache: What do temperature and top-p really do, and why is generation slow?

What changed since 2019, and what we honestly do not know.

Why LLMs know things: Where is the knowledge, and why do models make things up?
Modern LLM architecture: What is different inside a 2020s model compared with our tiny GPT?
From raw text to assistant: How does a next-token predictor become a helpful assistant?
Small models from big ones: How does a 3-billion-parameter model learn from a 600-billion-parameter one?
Alignment and safety: How do you teach a model what it should not do, and how do you check that it learned?
Models that see and hear: How does a model that only reads tokens look at a photo?
Looking inside the model: Can we read what the numbers inside a model mean?
Reasoning models: What changes when a model is allowed to think before answering?
Training a reasoner with GRPO: How do you train a model when all you can check is the final answer?

RAG, fine-tuning and agents: what each one really changes.

Retrieval-augmented generation: What if the information is not inside the model?
Fine-tuning: What does fine-tuning change that a prompt cannot?
Agents: How can a text generator use a tool?

What it takes to run this in production, and to keep learning from papers.

Evals: how you know it works: The demo looked great. How do you know the system actually works?
Serving many users at once: One user is easy. What changes when a thousand arrive at once?
Making the model itself cheaper: The batch is as full as it can get. What is left to change?
The PyTorch and Hugging Face bridge: Can you open a real open model and recognise everything inside it?
How to read an LLM paper: Can you read “Attention Is All You Need” and recognise every section?
Context engineering and production agents: Your agent works on your laptop. What breaks when real users and real money are involved?

Build the whole system, then explain it from memory.