Making the model itself cheaper

How do you halve the GPU bill once the batch is full?

A support bot’s monthly GPU quote is twice what the business wants to pay. The batch is already packed as tightly as scheduling allows: continuous batching keeps the slots full, and paged allocation of the KV cache fits more conversations into the same memory.

All of that took the model as given. What is left is to change the model.

One fact from Serving many users at once carries this lesson: decode is memory-bound. To produce one token the GPU reads every weight once, and the arithmetic takes roughly 1% of the time the read takes. For one stream, tokens per second is at most bandwidth ÷ weight bytes. (Prefill is the opposite case: it processes the whole prompt in one pass, so there the arithmetic units are the limit.)

Why not buy more bandwidth? It did grow, from about 2 TB/s on an A100 to about 8 TB/s on a B200, four times in two generations. But 16-bit arithmetic grew about seven times over the same span, and models grew faster than either. No bandwidth helps a model that does not fit in the memory you have.

Pause and predict Bandwidth grew four times in two generations, about 2 TB/s to 8 TB/s, and decode waits on it. If you cannot buy your way out, what can you change?

Fewer bytes, more tokens per pass, more GPUs

1. Quantization: fewer bytes per weight

Quantization

Store each weight in fewer bits plus a shared scale, instead of a 16-bit float. The small number can be an integer (int8, int4) or a tiny float (FP8, FP4). 8 bits halves the bytes, 4 bits quarters them.

For example: Eight weights with largest magnitude 0.035. Scale = 0.035 ÷ 7 = 0.005. The weight 0.021 is stored as round(0.021 ÷ 0.005) = 4 and read back as 4 × 0.005 = 0.020.

int8 and int4 have 256 and 16 possible values per weight; a 16-bit float has tens of thousands. Because decode is limited by bytes moved, this makes serving smaller and faster at once. It is lossy compression of a table you read in full on every request, and decompression is almost free because the arithmetic units were idle anyway.

Pause and predict Eight weights share one int4 scale. One of them is 25 times larger than the rest. What happens to the ordinary weights?

Low-precision floats: FP8 and FP4

In data centres today the main route is tiny floating-point formats, because recent GPUs do their arithmetic on them directly. A float spends some bits on an exponent (how big) and the rest on a mantissa (how precise); with 8 or 4 bits, you choose the split.

FP8 (E4M3 and E5M2)

An 8-bit float. E4M3 has 4 exponent bits and 3 mantissa bits: more precision, largest value 448. E5M2 has 5 and 2: less precision, but reaches 57,344.

For example: Weights and activations usually use E4M3. Gradients in training, which swing across a wide range, have often used E5M2.

FP8 is now a standard way to serve large open models: the weights halve, and the matrix multiplies themselves run in 8 bits. It is used in training too: DeepSeek-V3 (December 2024) trained most of its matrix multiplies in FP8 E4M3, one of the first openly documented runs at that scale to do so.

Below 8 bits, FP4 formats such as MXFP4 give every block of 32 weights its own shared scale, about 4.25 bits per weight. OpenAI’s open-weight gpt-oss models (August 2025) shipped their mixture-of-experts weights in MXFP4, which is how the 120B-parameter model fits on a single 80 GB GPU.

The KV cache can be stored in low precision too, which raises the number of sequences that fit. vLLM offers an FP8 cache, DeepSeek-V4.1-Flash (2026) ships with its main KV cache in FP4, and research such as KIVI (ICML 2024) goes as low as 2 bits.

2. Speculative decoding: spend the idle arithmetic

The big model can score 5 positions in one pass in nearly the time it scores 1, exactly like a tiny prefill. It only needs to know which 5 tokens to score.

Think of a teacher marking a sum a student has written out in full. Five lines can be ticked in the time it takes to work out one. At the first wrong line the teacher writes the correction and crosses out everything below it. The analogy stops at the tick: a teacher’s check is yes or no, and the real check is a probability rule.

Pause and predict A draft proposes “sat on the mat”. The big model accepts “sat”, “on” and “the”, and rejects “mat”. How many tokens does that one pass of the big model produce?

Speculative decoding

A small, fast draft model guesses the next few tokens. The big target model checks all the guesses in one pass. Guesses are kept or replaced by a rule that makes the final text distributed exactly as if the big model had written every token itself.

For example: The draft proposes “sat on the mat”. The target accepts “sat”, “on”, “the”, rejects “mat” and replaces it with “sofa”. Four tokens for one pass of the big model.

The target model is the one whose answers you want. The drafter makes the guesses, and there are three common kinds. The acceptance rule is the same for all three.

• A separate small model, a few hundred million parameters against the target’s tens of billions: the original recipe, and our demo’s.
• A draft head on the target itself, one light extra layer that reads its hidden vectors and guesses ahead. EAGLE-3 (Li et al., 2025) is the best-known.
• Multi-token prediction heads trained with the model. DeepSeek-V3’s report says its extra token was accepted 85% to 90% of the time.

It is a latency technique that only works because decode is memory-bound. It does not help when the drafter is often wrong (little is accepted, and its own time is wasted), or when the server already runs large batches (the arithmetic is no longer idle, so verifying extra positions is no longer free). The gain shrinks as batches grow, but it does not vanish: with long contexts the cache reads keep even a big batch memory-bound, and speculation can still pay.

3. When the model does not fit on one GPU

Tensor parallelism

Split every weight matrix across several GPUs. Each GPU holds a slice, computes its part of every layer, and the parts are combined.

For example: A 70B model in 16-bit is 140 GB. Across 4 GPUs of 80 GB each, every GPU holds 35 GB of weights and a quarter of each KV cache.

Pause and predict Split the model by layers instead: GPU 1 runs layers 1 to 20, GPU 2 runs 21 to 40. What happens to one request’s latency?

A large deployment typically combines them: tensor parallel inside a server so the model fits and responds quickly, then as many replicas as the traffic needs. So the three techniques divide the work: quantization shrinks the bytes, speculative decoding buys more tokens per read of those bytes, and parallelism spreads the bytes over more memory links. A real engine does all of it at once, on top of the scheduler from the last lesson.

What int4 buys, on the same GPU

Take the capacity plan from the last lesson and change one thing: store the weights in int4 with group scales instead of 16-bit floats. Everything else is the same. 8B parameters, an 80 GB GPU with 2 TB/s of bandwidth, 10% of memory kept back, a KV cache of 131,072 bytes per token and 1,300 tokens per sequence at its longest.

Pause and predict The weights are now a quarter of the bytes. How many more conversations fit on a fully loaded server?

One change gives two very different gains.

• For one user, almost four times the speed. The weight read is the whole step, and it shrank by the full factor. This is the gain you notice on a laptop.
• For a loaded server, 21% more sequences. The weights were only 16 of the 72 usable gigabytes, and the caches were already most of the rest. Freeing 12 GB helps, but it cannot help by four times, because the caches did not shrink.

That gap is why quantizing the KV cache is a separate and increasingly important lever at scale. The right column is also a more optimistic ceiling than the left, because real int4 kernels spend arithmetic unpacking weights. Read 480 as the point where the weight read stops being the limit, and do not promise it to anyone.

The math

Two formulas, both about the acceptance rule, because that is the surprising part. Here p is the target’s probability for the proposed token and q the draft’s:

With α = 0.8 and γ = 4 that is 3.36 tokens per pass. It is a geometric series capped at γ + 1 (Leviathan et al., equation 1). The speed-up is smaller than this number, because the draft’s γ steps also take time.

What you should remember

1. Quantization stores each weight in fewer bits plus a shared scale: a small integer (int8, int4) or a tiny float (FP8, and FP4 formats such as MXFP4 with one scale per block of 32). Fewer bytes shrink memory and speed up decode. Outliers are what breaks it: more scales, per row or per group of 32 to 128, contain the damage, at about 4.1 to 4.5 bits per weight.
2. The gain is large for one stream and modest for a full server, because a big batch is dominated by KV cache reads that quantizing the weights leaves untouched. Quantize the cache too if you need more.
3. In speculative decoding something cheap (a small model, an EAGLE-style head, a multi-token-prediction head) guesses the next few tokens, and the target model checks all the guesses in one pass. The accept-or-replace rule keeps the target’s distribution exactly. It buys latency only while the arithmetic units are idle, so it fades at large batch sizes.
4. When the model does not fit: tensor parallelism splits every matrix and cuts latency at the cost of two all-reduces per layer, pipeline parallelism splits by layers and needs bubbles filled, data parallelism is replicas behind a load balancer.

Where this appears in a real LLM

In this lesson

• Round-to-nearest quantization of one random matrix
• Error measured on one layer’s output, not on any task
• Speculative decoding between two 6-token Markov chains
• Parallelism described, not run: one process, one machine

In a real system

• FP8 and FP4 (MXFP4, NVFP4) run natively by the hardware, plus GPTQ, AWQ and similar integer methods
• Quality measured with task evals on the quantized model before it ships
• EAGLE-style draft heads, native multi-token-prediction heads, draft models or n-gram lookups as the proposer, verified in batches
• Tensor parallel inside a server over fast links, replicas across servers

Where you will meet these: a model-card name like “FP8”, “MXFP4”, “GPTQ int4, group 128” or “AWQ 4-bit” tells you the format, the method and the granularity. llama.cpp’s GGUF schemes, which range from 1.5 to 8 bits (Q4_K_M averages about 4.8), are what make a 7B model run on a laptop. vLLM, SGLang and TensorRT-LLM all serve FP8 and 4-bit weights, offer a low-precision KV cache and support speculative decoding. And “tensor parallel size”, a launch flag on every serving engine, set to the number of GPUs in one server, is the usual first answer to “the model does not fit”.

For the support bot at the start, a realistic plan is FP8 weights and an FP8 cache, a draft head for the quiet hours, and a full eval run before either ships. That lowers the bill, though by less than half, and you can account for every step of the saving.