Retrieval-augmented generation

Why did the bot make up a dispute window?

A payments company’s support bot has invented a policy.

A customer asked how long they have to dispute a failed payment. The bot replied, very politely, with a number it made up. The real answer sits on page 14 of a policy PDF that the bot has never seen.

Telling the model to read the PDFs does not help by itself. A model can read text that is put in front of it, but it has no way to go and find that text. Finding the right passage is the job of the code around the model.

Before the policy PDFs, start with a smaller rehearsal: an engineering wiki of four short documents, and a question whose answer you already know:

The correct answer is in oncall.md on the wiki: five minutes. The model has never seen that file. It was not in the training data, so it is not in the weights.

You know from Why LLMs know things what happens next. The model has no “I have no record of this” mechanism. It produces the most plausible continuation: “Typically within 15 minutes”. The answer is fluent and confident, and it is wrong.

The same gap appears for anything private (your documents), anything recent (after training stopped), and anything rare (seen too few times to be stored reliably).

Pause and predict Policies change every month, and retraining a model each time is slow and costly. What is the cheapest change that lets the bot answer from page 14?

The problem

So far, “How does an LLM know?” had one answer: patterns stored in weights during training.

This lesson asks the opposite question. What if the information is not inside the model at all?

The problem

The model must answer from information it was never trained on: private, recent, or rare.

The naive fix

Train it on that information (fine-tune on the company wiki).

Why it fails

Retraining is slow and costly to repeat every time a document changes. Facts seen a few times are stored unreliably. And you still cannot tell which document an answer came from.

The idea

Do not store the facts in the model at all. At question time, find the relevant text and put it into the prompt. The model’s job changes from remembering to reading.

The trade-off

You now own a search system. If search brings back the wrong text, the model answers from the wrong text, or from its own guesses.

Retrieval-augmented generation (RAG)

Before calling the model, search your own documents for passages related to the question, and paste them into the prompt with the instruction “answer from this”.

For example: Question: “when do deploy freezes apply?” → search finds a passage from deploy-policy.md → prompt = instruction + that passage + the question.

A mental model

The pipeline has two halves that run at different times. Ingest runs once, offline. Query runs for every question. They meet at one table.

Follow the highlighted rows. The three best-scoring chunks leave the table, land under “Context:” in the prompt, and come out as the citation. Nothing else in the picture is new to the model: the box marked “LLM, unchanged” is the same model with the same weights.

Only two pieces of this pipeline are new: chunks and the vector database. You have met everything else in earlier lessons.

A note on the name. The original RAG paper (Lewis and colleagues, 2020) also fine-tuned the question encoder and the generator together. Today “RAG” almost always means the version in this lesson: a frozen model and a search step in front of it.

Chunk

A passage of a document, small enough to be about one thing. Documents are cut into chunks because retrieval works on passages.

For example: Two sentences of oncall.md. Neighbouring chunks share a sentence (“overlap”) so a fact that straddles a cut survives in at least one chunk.

Vector database

A store of (vector, text) pairs that answers one query: “which stored vectors point in the most similar direction to this one?”

For example: Embed the question, take the dot product with every stored vector, sort, keep the top 3.

“Top-k” here means the k best-scoring chunks. It is the same idea as top-k sampling in Inference (keep the k best, drop the rest), applied to chunks instead of tokens.

The vectors come from an embedding model. In Embeddings one word became a vector. Here a whole passage becomes one vector, so that passages about similar things land close together.

Let's see the numbers

Retrieval has to decide which stored chunk points in the most similar direction to the question. “Similar direction” is measured by cosine similarity, which you met in Vectors and the dot product: the dot product with both lengths divided out. Here it is by hand, with a vocabulary of just three words: [pages, acknowledge, budget].

Pause and predict The question is [1, 1, 0]. The paging chunk is [2, 1, 0]; the expenses chunk is [0, 1, 2]. Which is closer in direction, and by roughly how much?

Where do the numbers in the vectors come from? In our crude embedder, each word’s weight is how rare it is. With 19 chunks:

A word that appears almost everywhere tells you almost nothing about which chunk you want, so it gets almost no weight.

This scheme has a name, TF-IDF (term frequency × inverse document frequency). The rarity weight is the IDF part, which dates from the 1970s (Spärck Jones, 1972). Counting how often the word occurs in the chunk is the TF part.

The math

The whole of query time fits on one line. Nothing in it touches the model’s parameters.

What you should remember

1. RAG changes the prompt and leaves the weights alone. It is retrieval + context injection around an unchanged model. It moves the model’s job from recall (unreliable) to reading comprehension (strong), and makes knowledge instantly updatable and citable.
2. Ingest time: documents → chunks → embeddings → vector database. Query time: embed the question → top-k by cosine → paste into the prompt → LLM → answer with citation.
3. Exact search is one matrix multiply. Approximate indexes such as IVF search only the nearest clusters: a recall vs speed knob.
4. RAG does not stop hallucination. If retrieval returns the wrong text, the model answers from the wrong text. When it fails, check chunking, embedding and k before the model.

Where this appears in a real LLM

In this lesson

• 4 documents, 19 chunks of 2 sentences
• Bag-of-words TF-IDF embedder with 123 dimensions, one per word
• Exact search over a Python list
• An extractive stand-in that can only quote one sentence

In a real system

• Millions of documents; chunks of a few hundred tokens that respect headings
• A Transformer encoder producing dense vectors, often combined with keyword search (“hybrid”) and a slower re-ranking model over the top candidates
• An approximate index (IVF, or the graph-based HNSW) with metadata filters
• A real LLM that can combine passages, and can also ignore or misread them

The shape is the same in every production system: chunk, embed, index, retrieve, assemble a prompt, generate. “Search the web” and “chat with your PDF” features are this pipeline. Long context windows reduce how much you must retrieve, but do not remove the need: someone still has to decide what goes in the prompt, and you pay for every token.

What to build first

If you are building retrieval for real documents, start with hybrid search and a reranker. Run a keyword search next to the vector search. The standard one is BM25, which scores a chunk by how often the question’s words appear in it, giving rare words more weight and correcting for chunk length. Merge the two ranked lists. The common simple method is reciprocal rank fusion, which gives a chunk 1 ÷ (60 + rank) for each list it appears in and adds those up. Then pass the top few dozen to a reranker. A reranker is a cross-encoder: a model that reads the question and one chunk together and scores how well the chunk answers it. It is too slow to run over the whole index, which is why it only sees the shortlist. Keyword search catches exact strings that embeddings blur, such as error codes, product names and transaction ids. Vector search catches paraphrases, such as “sick” for “medical leave”. Each covers the other’s misses, and the reranker fixes the order.

Retrieval is also where access control belongs. When the index holds documents that not every user may read, store the permissions with each chunk as metadata (team, customer id, the document’s access list) and filter on them inside the search query, so a chunk the user may not see is never a candidate. Filtering after the model has answered is too late. Once a chunk is in the prompt, the model can quote it, and no instruction in the prompt reliably stops that.

Serving this pipeline to thousands of customers at once, with a vector index, quotas and a cost budget, is a system design problem in its own right. Design LLM serving with RAG

One change is now common. Instead of your code running one search before the model call, the model is given search as a tool. It writes its own queries, reads the results, and searches again if the first results were poor. This is often called agentic search. It is the same pipeline, driven by a loop, and you will build that loop in Agents.

With retrieval in place, the support bot answers the dispute question from page 14 of the policy PDF and shows the file’s name beside the answer. The model did not learn the policy. It read it, and nothing inside the model changed. When you do want to change what is inside the model, the next lesson, Fine-tuning, takes that on.