Design LLM serving with RAG
How do you serve a support bot to thousands of customers at a sane cost?
Why does the bot take a minute to say hello?
By winter the support desk takes about 1,500 calls on a festival day, and most of them are one question: where is my dabba? So the kitchen builds a support bot. It rents one GPU machine, loads an open-weights language model on it, and writes a small loop: collect up to 32 questions, run them together, send each answer when the batch is done.
The help articles go into a vector index, and, to save a lookup, every order’s status too, refreshed each night.
At 12:41 on Diwali a customer types her question. The bot shows dots for 50 seconds before the first word appears. The phones ring anyway: people ask the bot, wait, give up and call.
Then a customer forwards a screenshot. The bot told him his low-sugar box was “being prepared”. It had left the kitchen an hour earlier.
The GPU costs the same every hour whether it talks or not, and the dashboard says it is busy all lunch. So why is it slow? Look inside one batch. The GPU is busy, but most of the batch is waiting: one long answer about a refund keeps 31 finished short answers from being sent. And the batch rarely reaches 32 anyway, because the loop sets aside memory for the longest answer any request might write.
Do the sum before reading on. A refund answer runs to 800 tokens and a typical answer to 60. A decode step is one round in which the model writes one more token for every answer in the batch. For how many decode steps does a short answer sit finished, waiting for the refund to end?
Two proposals come up at once: four more GPUs, or fine-tuning the model on all the orders so it knows them. Four more machines would each spend most of their time waiting in the same way. And no model can learn an order that was placed ten minutes ago.
Pause and predict In the bot’s loop, one long refund answer holds back 31 finished short ones, and every request has memory set aside for the longest answer it might write. Without adding a GPU, which change helps most?
What must never go wrong?
One rule comes first: the bot may say “I do not know”, but it must never state an order fact that is not true now.
Core entities, in two minutes
A Tenant with a quota; a Conversation of Turns, each with tokens in and out; Documents (help articles) cut into Chunks with embeddings; and daily Usage per tenant. Orders are not stored in this system: the bot asks the order system for them when it needs them.
The estimate that changes the design
- Requests: 50,000 orders x 10% x 3 turns = 15,000 a day. If 30% fall in the lunch hour, that is 4,500 / 3,600 s = 1.25 a second; three times that at a festival is about 4 a second.
- The knowledge base: about 400 help articles, five chunks each, is 2,000 chunks. With embeddings of 768 numbers in 4 bytes each, that is 2,000 x 768 x 4 = 6.1 MB.
Pause and predict The help articles come to 6.1 MB of embeddings. What does searching them need?
The first estimate, about 4 questions a second, says one GPU machine is enough at the peak if it batches well. The rest of the design is about making it batch well.
The obvious version, and how it breaks
Three separate faults are hiding in the first version.
- Static batches. A batch runs until its longest answer is finished. A 60-token “It left at 12:31” sits finished for hundreds of steps behind one 800-token refund explanation, and the next batch cannot start.
- Reserved memory. The loop sets aside room for the prompt plus the longest answer allowed, for every request. Most answers use a small part of it, so the batch is limited by memory nobody uses.
- Live facts in a nightly index. Order status changes every few minutes; an embedding computed last night describes the order as it was last night. That is why a box already on the road was “being prepared”.
The first two are about how the GPU is shared, and the batching lab in the logical design shows them. The third is about where facts come from, and the logical design fixes it.
A mental model
Two phases happen inside the model for every request, and they behave differently.
- Prefill and decode
- Prefill reads the whole prompt at once and produces the first output token. Decode then produces one token per step, each step needing everything before it.
- For example: A 1,500-token prompt is one prefill; a 64-token answer is 64 decode steps, shared with every other sequence in the batch.
- KV cache
- The keys and values the model computed for every token so far, kept in GPU memory so each new token does not recompute the whole conversation.
- For example: For OPT-13B, the vLLM paper computes 800 KB per token: 2 (keys and values) x 5,120 x 40 layers x 2 bytes.
The LLM course builds this cache inside the model, one token at a time: the KV cache in LLM From First Principles.
Run the model, or rent it by the token?
With the engine in place, the bot keeps up. One question remains: should the kitchen run GPUs at all? Two goals compete: a bill that moves with use and no machines to run, or a fixed cost and customer data that never leaves the kitchen’s own cloud account.
The deciding number is volume. At 15,000 questions a day, the fixed cost of two machines is mostly paid for idle hours; a per-token price grows with every question. The crossover depends on prices that change every quarter, so the decision record names a measurement to repeat, tokens a day, instead of a price.
What to remember
- Model serving has two phases: prefill (the whole prompt at once) and decode (one token per step). Batching many sequences into each decode step is where throughput comes from.
- Continuous batching lets requests join and leave every step; static batches wait for their slowest member.
- The KV cache grows with every token: 2 x layers x KV heads x head size x bytes per token. Reserving it for max_tokens wastes most of it; paging allocates it as it is used.
- Stream the reply and measure time to first token and time between tokens separately.
- Retrieve stable knowledge; fetch live facts by tool call, with the customer’s identity checked by code. Meter tokens per tenant.
Batch per decode step, allocate KV memory as it is used, and fetch live facts at the moment of asking.
In the real world
The simulator is a toy engine with illustrative step costs and one GPU. Real engines add much more: splitting long prefills into chunks so they do not stall decoding, sharing KV blocks between requests with the same prefix (every conversation here starts with the same system text), speculative decoding, quantised weights and KV caches, and models spread across several GPUs. Their scheduling knobs exist to hold the same two targets you met here: time to first token and time between tokens.