Build your own mini LLM system

Can you wire every piece together yourself?

So far, every program you wrote ran alone. The tokenizer never met the GPT. The GPT never met the retriever. The agent loop called a scripted stand-in instead of a model.

This capstone joins them into one pipeline, in files of your own:

• your tokenizer feeds your Transformer;
• you train it, then improve how it generates;
• you put retrieved text into its prompt;
• finally you hand it to an agent loop as a tool.

Pause and predict You wire your own tokenizer, GPT, retriever and agent together. The model has under a million parameters. Put the retrieved policy text that holds the answer into its prompt: will it answer a customer correctly?

So the last step swaps the model out. A hosted model takes its place behind the same function call, and you run your eval, your budget and your trace against it. Everything except the model stays as you wrote it, so you can see exactly what a large model changes and what it leaves alone.

What you should remember

1. The whole stack is six small programs with narrow interfaces: strings ↔ ids, ids → scores, scores → one id, question → prompt, text → action.
2. The model never sees text and never acts. Tokenizers, samplers, retrievers and agent loops are ordinary code around it.
3. RAG changes the prompt. Training and fine-tuning change the weights. An agent adds a loop around the model.
4. Test properties that must hold: lossless round trip, rows sum to 1, no peeking at the future, cached equals naive, weights unchanged by retrieval.
5. Your model is tiny and cannot answer questions. What separates it from a real LLM is scale and training data; the architecture is the same, which is why a hosted model can take its place behind the same function.

Where this appears in a real LLM

In this lesson

• BPE over characters, a few hundred merges, pure Python
• Under 1M parameters, 64-token context, minutes on a CPU
• TF-IDF-style embeddings, exact search over a few chunks
• Regex-parsed TOOL/ARGS text, scripted brain

In a real system

• BPE over bytes, around 100,000 merges, optimised native code
• Billions of parameters, context of 100,000+ tokens, months on GPU clusters, then instruction and preference tuning
• Learned Transformer embeddings, approximate search over millions of chunks
• Structured tool-call formats the model was trained to emit, with the same loop around them

What a production assistant has that your six pieces do not is mostly more training; the plumbing is the same. A big model can teach a small one (Small models from big ones). Post-training teaches it what it should refuse (Alignment and safety). An image encoder lets it read a payment screenshot (Models that see and hear). And none of it is fully readable from the inside yet (Looking inside the model).

When everything is connected, your tiny GPT, called as a tool by your own agent loop, writes a few lines of verse that make no sense. Every character of it passed through code you wrote. With a hosted model behind the same function, the same loop and guards can answer the oncall question, your eval says how often the answers are right, and the trace shows every step.

The last lesson asks whether you can rebuild the whole picture from memory.