The PyTorch and Hugging Face bridge

Would you recognise a real model if you opened one?

Every model in this course so far has been your own code. Open a real one: go to huggingface.co/openai-community/gpt2 and click “Files”. You see this:

Those six files, and nothing else, make up a working language model.

Pause and predict Which of these files contains the code that runs the model?

You have already built every part of it:

• the tokenizer in Tokenization,
• the GPT in Build GPT,
• the sampling loop in Inference,
• LoRA in Fine-tuning.

What you have not done is match your parts to theirs. A real open model is your tiny_gpt.py with different names, bigger numbers and a file format.

What stands between your GPT and theirs?

The first surprise is that the repository holds no program. None of the ideas is new to you. What stands between your GPT and this one is four practical things:

• Names. Your fused QKV layer is attn.qkv. GPT-2 calls it attn.c_attn. Llama splits it into q_proj, k_proj, v_proj. The maths is the same in all three; only the names differ.
• Files. A model is shipped as data. Which file holds what, and which can hurt you?
• Number formats. At 8 billion parameters the format decides whether the model fits on your GPU at all.
• Libraries. from_pretrained, generate, get_peft_model each wrap something you wrote by hand.

The problem

A trained model must be handed to other people: a few hundred megabytes to hundreds of gigabytes of numbers, plus everything needed to use them.

The naive fix

Save the Python object. torch.save(model) uses Python’s pickle, which can store any object.

Why it fails

A pickle is a small program that rebuilds the object, and loading it runs that program. A malicious checkpoint can execute any code on your machine. The Python documentation opens its pickle page with that warning: only unpickle data you trust. It also ties the file to one exact class definition.

The idea

Split the model into plain data. A JSON file for the architecture’s numbers, plain files for the tokenizer, and a tensor file that holds only names, shapes and raw bytes. The code that gives them meaning lives in an open library.

The trade-off

Data without code is safe but mute. You need library code that matches the architecture, and every model family names its tensors its own way.

A model repository, file by file

Each kind of file answers one question:

1. Tokenizer files answer “how does text become ids?”. merges.txt is the ordered merge list you trained in Tokenization; vocab.json maps each piece to its id. GPT-2 has 50,000 merges, 256 byte tokens and one special token: 50,257 ids.
2. config.json answers “what is the skeleton?”. It is your Config class as JSON. It contains no weights.
3. model.safetensors answers “what are the numbers?”. It is a dictionary from tensor name to array.
4. generation_config.json answers “how should the loop behave by default?”: which id ends generation, sometimes a default temperature or top-p.

safetensors

A file format for tensors that holds data only. It starts with a JSON header that lists every tensor’s name, number type, shape and byte range. After the header come the raw bytes. Loading it cannot run code.

For example: GPT-2’s file: 8 bytes of header length, 14,283 bytes of JSON, then 137,022,720 numbers × 4 bytes. Total 548,105,171 bytes, which is exactly the size the Hub shows.

The tokenizer is a separate artefact, and it must match

The model only ever sees integers. Row 3797 of the embedding table means “␣cat” because GPT-2’s tokenizer says so. Feed the same text through another tokenizer and you look up the wrong rows. The model then produces fluent nonsense, and nothing raises an error.

The same holds for special tokens and chat formats. In From raw text to assistant you saw that an assistant is trained on conversations flattened by a chat template. That template ships with the tokenizer, and apply_chat_template applies it for you. We run it below.

Two dialects, one machine

Here is what you wrote, next to GPT-2 and a Llama-style model (Llama, Mistral, Qwen and many others share these names).

Every shape in the GPT-2 column is printed by the script in this lesson. The Llama shapes were read from the header of the published checkpoint.

Let’s see the numbers: will it fit?

The first thing to ask of any model is whether it fits. Take Llama 3 8B: 8,030,261,248 parameters, published in bf16.

Pause and predict Inference in bf16 needs the weights. Full fine-tuning with Adam needs much more. Which fits on one 80 GB GPU?

Where do 16 bytes per parameter come from?

Inference needs the weights. Training needs the weights, a gradient for every weight, and Adam’s two running averages for every weight (you met them in Training GPT). In the usual mixed-precision setup:

This accounting is from the ZeRO paper (Rajbhandari et al., 2019), which uses it to show that a 1.5B-parameter GPT-2 needs at least 24 GB of model state. It is the standard back-of-envelope estimate, not a law: activations come on top, and 8-bit optimisers or pure-bf16 training lower it.

Now LoRA on GPT-2 from this lesson’s script. The frozen base needs only its weights: 124.4M × 4 bytes = 498 MB in fp32. Gradients and Adam state exist only for the adapter’s 442,368 numbers: about 5 MB. Activations still have to be stored for backprop, so LoRA does not make training free. It removes the largest fixed cost.

The math

The whole sizing exercise is three multiplications, worth writing down once:

What you should remember

1. A model repo is data, not code: config.json is the skeleton, tokenizer files turn text into ids, model.safetensors holds named tensors, generation_config.json holds loop defaults. The forward pass lives in the library.
2. Two dialects, one machine: wte / c_attn / c_proj / c_fc / ln_f (GPT-2) and embed_tokens / q_proj k_proj v_proj o_proj / gate_proj up_proj down_proj / norm (Llama-style). Name plus shape identifies any tensor.
3. From the config alone you can compute parameters exactly (GPT-2: 124,439,808; Llama 3 8B: 8,030,261,248), weights = N × bytes, KV cache per token = 2 · L · H_kv · d_head · bytes, and training ≈ 16 bytes per parameter plus activations.
4. generate() is your loop with a KV cache, and peft’s LoRA is your LoRALinear plus bookkeeping. model.eval() (dropout off) and torch.no_grad() (recording off) are independent switches. Prefer safetensors: a pickle can run code.

Where this appears in a real LLM

In this lesson

• tiny_gpt.py: 0.8M parameters, fp32, one file, names you chose
• GPT-2 small in this lesson: 124M, fp32, runs on a laptop CPU
• LoRA on 40 tickets, 8 held out, checked by eye
• Your loop re-runs the whole sequence for each new token

In a real system

• Billions of parameters in bf16, sharded across several safetensors files with an index
• Served in bf16, FP8 or a 4-bit format (int4, MXFP4, NVFP4), by engines such as the ones in Inference systems
• Thousands to millions of examples, a held-out set sized as in Evals, often LoRA on all linear layers over a 4-bit base (QLoRA)
• KV cache, batching, stop tokens and chat templates handled by the serving stack

The names and shapes in this lesson are read from the published files, and the architecture code is open: you can read modeling_gpt2.py and modeling_llama.py in the transformers repository and find every line of your own GPT in them. For closed models such as Claude, GPT or Gemini none of this is published; what you can say is that openly released models from many labs share this layout.

Make the two-column list for yourself, your names on the left and theirs on the right, and check that no row is a mystery. The next lesson reads the paper this architecture came from.