RAG: making a model answer from your own documents

A language model read a great deal of text during training, but it has not read your documents: your internal documentation, your contracts, your blog posts, your support knowledge base. It also does not know what changed last week. Ask it about those subjects and it will still answer, with its usual confidence, and it will make up what it does not know.

RAG, short for retrieval-augmented generation, is the most common answer to this problem. The idea fits in a sentence: before asking the model the question, you find the most relevant passages in your documents and put them in the prompt. The model no longer answers from memory, it answers from a text you show it.

Why not simply train the model on your documents?

You can fine-tune a model on your own data, but it is not the right tool for injecting facts. Fine-tuning mostly changes style and behaviour, it is expensive to redo every time the documents change, and it cannot tell you where a piece of information came from. With RAG, updating knowledge means updating a document, and every answer can point to its source.

The four stages of a RAG system

Every RAG system, from prototype to platform, breaks down into four stages.

1. Ingestion. You extract the text of each document (PDFs, web pages, office files, tickets) and clean it: repeated headers and footers, menus and layout artefacts are removed.
2. Chunking. The text is cut into passages, called chunks, of a reasonable size.
3. Indexing. Each passage is turned into a vector by an embedding model, then stored with its text and metadata.
4. Querying. For a question, you retrieve the closest passages, assemble them in the prompt, and the model writes the answer.

Chunking, the most underrated setting

A passage that is too long dilutes the information: its vector represents a bit of everything and strongly resembles nothing. A passage that is too short loses its context: an isolated sentence such as “it is valid for one year” does not say what it is about. There is no universal size, but a few principles hold up well.

• Cut at natural boundaries: headings, paragraphs, list items, rather than in the middle of a sentence every thousand characters.
• Keep a little overlap between neighbouring passages so an idea is not separated from what follows.
• Attach context to the passage: document title, section heading, date. This metadata serves both search and citation.
• Handle tables and lists of values separately, which naive chunking destroys.

Embeddings, what they know and what they do not

An embedding model turns a text into a long series of numbers, so that two texts about the same thing end up with nearby vectors. Closeness is measured with a similarity, most often the cosine. That is what lets you find “cancel my subscription” when the document says “terminate the contract”.

But semantic closeness is fuzzy. An embedding finds paraphrases very well, and much less well anything that hinges on one exact term: an order number, an error code, a rare proper noun, an article number in a statute. For those queries, a classic search engine does better.

Hybrid search

The solution is to combine both. On one side, a lexical search, for instance the BM25 algorithm, which ranks passages by the query words they contain, weighted by how rare those words are. On the other, a vector search. You get two ranked lists, which you merge.

A simple, robust merging method is called reciprocal rank fusion: each passage gets a score based on its rank in each list, and the scores are added. It avoids comparing scores of different natures, which cannot easily be put on the same scale. In practice, hybrid search finds both paraphrases and exact terms, and it is rarely a bad starting point.

You can then add a reranking step: a second model rereads the best ten or twenty candidates alongside the question and reorders them more finely. It costs compute time and is only worth it once the earlier stages are well tuned.

Making the model answer, and cite its sources

Once the passages are retrieved, the instruction has to be unambiguous: answer only from the excerpts provided, say plainly when they are not enough, add nothing from memory. A system that can answer “I cannot find that in the documents” is more useful than one that always answers.

Citations are what make an answer verifiable. A good practice is to number the passages in the prompt and ask the model to cite the numbers it used. But you should not take its word for it: it is the code that turns those numbers into links, and that discards any number that does not match a passage actually supplied. An invented citation then leads nowhere, instead of leading to a fake but credible-looking source.

The model writes, the code attributes the sources.

Access rights: at query time, not afterwards

If your documents do not all have the same audience, permission filtering has to happen during retrieval, before anything reaches the prompt. Filtering the answer afterwards is too late: the confidential passage has already been read by the model, which can reuse a piece of it in rephrased form. Every indexed passage must therefore carry the information of who is allowed to read it.

What breaks most often

• Outdated documents, never removed from the index, which the system cites with aplomb.
• Questions that require crossing several documents or computing over tables, which plain passage retrieval handles badly.
• Scanned PDFs whose extraction returns noise or nothing at all, and that nobody checked.
• An index that is never refreshed when the documents change.
• Too many passages in the prompt: more context is not always better, and the useful part gets lost in the middle.

Measure before you tune

Without measurement, you tune at random. Build a small set of real questions, with for each one the passage or passages that contain the answer. Measure retrieval alone first: is the right passage among the top five results? That stage decides everything. If retrieval is poor, no prompt will save the answer. Then measure answer quality, preferably by hand on a few dozen cases before automating.

A disappointing RAG is almost always a retrieval problem, not a generation problem. Look at the retrieved passages before blaming the model.