Retrieval-Augmented Generation (RAG)

A trained language model stores what it learned in its weights. That knowledge stops at the training cutoff, it never included your company's private documents, and you cannot point to the sentence a fact came from. Ask it your refund window and you get a plausible number that may not be yours.

Retrieval-augmented generation (RAG) splits the job in two. A retriever searches a collection you control and returns a handful of passages. A generator, the language model (a transformer, covered in the transformers and self-attention topic), writes the answer with those passages in its prompt. The model supplies language and reasoning; the facts live in an index you can edit in seconds.

RAG fits when:

• The facts change often (prices, policies, tickets). Re-indexing a page is cheap; retraining is not.
• Answers must cite a source a person can check.
• Different users may see different documents.

Fine-tuning is better for a format, a tone or a narrow skill; it is a poor way to load facts that go stale, and leaves no citation trail.

The name comes from Lewis and colleagues at Facebook AI Research, UCL and NYU (NeurIPS 2020). Their parametric memory was a pre-trained BART-large generator of about 400 million parameters. The non-parametric memory was a dense vector index of Wikipedia, about 21 million 100-word passages, searched by DPR, a BERT-base bi-encoder. The query encoder and generator were fine-tuned together. RAG-Sequence used the same passages for the whole answer; RAG-Token could switch passage per token. Swapping the index changed what the model knew, with no retraining.

Four real designs side by side:

• Lewis et al. 2020. Dense retrieval only, trained jointly with the generator, one corpus, research benchmarks.
• Dense Passage Retrieval (Karpukhin et al. 2020). The retriever alone, with separate question and passage encoders. On top-20 passage retrieval it beat a Lucene BM25 baseline by 9 to 19 points absolute on several open-domain QA datasets. Those are Wikipedia-style benchmarks, not proof that dense wins everywhere.
• Anthropic's contextual retrieval guide (vendor cookbook). BM25 plus embeddings with rank fusion, an optional reranker, and a model-written note on each chunk's place in its document. On their code-heavy test set (9 codebases, 248 queries) Pass@10 rose from 87.15% to 92.34% with those notes and to 95.26% with reranking. A vendor ran it on its own data: a direction, not a guarantee.
• The common production stack. A frozen off-the-shelf embedding model, a keyword index beside it, a hosted LLM, and prompt rules for citing and refusing.

In interviews "RAG" usually means the last one.

Offline, whenever documents change. Documents are split into chunks, and each chunk keeps metadata: title, source link and the user groups allowed to read it. Every chunk goes into two indexes. The keyword index holds term statistics for BM25, the ranking function from the full-text search topic. The vector index holds one embedding per chunk: a fixed-length list of numbers where texts with similar meaning land close together (the tokenization and embeddings topic shows how these vectors are made).

Online, for every question. The question goes to both indexes, and each search sees only chunks the user may read. Rank fusion merges the two lists. An optional reranker rescores the top few dozen with a slower, more accurate model. A guard checks whether the best match is good enough and, if not, hands the question to a person. Otherwise the chunks go into a prompt that tells the model to answer from them and cite them.

Precomputed embeddings make this affordable. Per the Sentence-BERT paper, finding the most similar pair among 10,000 sentences with a BERT cross-encoder takes about 50 million model runs (roughly 65 hours) versus about 5 seconds with one embedding per sentence. Embedding the 32 articles of the test corpus below took well under a second on an ordinary CPU (0.28 s on the test machine).

The test corpus is a toy help centre I wrote for a made-up home goods store: 32 short articles (1,411 words), four of them staff-only. There are 28 questions: 12 reuse the articles' wording, 12 are paraphrased the way customers talk ("Will you put the bed together for me?"), and 4 quote an error or product code.

For chunking, the public articles were glued into 5 long pages of 140 to 348 words, like a CMS export, then split into windows with 25% overlap. A hit means the exact answer phrase (say "within 7 days") is inside the retrieved chunk.

Results for all 28 questions (answer phrase in the top-1 chunk, then in the top 3):

Ten-word windows rarely hold a whole answer: the phrase is cut across two chunks. Whole pages look best on paper, but with only 5 pages the top 3 is 60% of the corpus, about 789 words for a one-sentence answer. That costs tokens and buries the answer mid-prompt. Dense search with 50-word chunks put the answer in the top chunk for 24 of 28 questions while sending about 146 words.

Read this as a mechanism, not a rule. With 28 questions one hit is 3.6 points, and the best size depends on your documents. Tables, code and lists break under fixed word windows; splitting on headings or paragraphs and attaching the page title to each chunk are common fixes.

Here each article was one chunk. The dense retriever is all-MiniLM-L6-v2, a small 384-dimension sentence-embedding model run locally with the fastembed library. Hybrid uses reciprocal rank fusion (RRF): each list gives each item 1 / (k + rank), and votes are summed. BM25 and cosine scores live on different scales; RRF uses only ranks, so it does not care.

Items high in both lists win; with weights, dense decides close calls. k = 60 here, untuned. On the toy corpus (correct article first / in top 3):

Paraphrases break BM25: "My vase showed up smashed" shares no useful word with "If a parcel arrives broken", so BM25 put the holiday-hours article first. The embedding model got 8 of 12. Equal-weight fusion dragged that down to 4, because BM25's list on those questions was mostly noise and RRF gave the noise equal votes. Weighting dense at 80% recovered some ground (6 of 12) but still trailed dense alone.

Exact codes such as E-417 did not trip the embedding model here, but 4 questions prove little; with thousands of similar part numbers, keyword matching is safer. Hybrid is a sensible default, but measure it and set weights from your own queries.

A reranker reads the question and each candidate together (a cross-encoder). That is more accurate than comparing separate vectors and much slower, so it only rescores the top 20 to 50. Anthropic's guide estimates 100 to 200 ms of added latency. No reranker was run here.

Three rules belong in code, not the prompt.

Run on the toy corpus (DOCS maps an id to title, text and allowed groups; vecs holds one normalised embedding per article):

Permissions. A customer asking about agent refund limits never sees the staff-only article; a support agent gets it first at 0.56. Filtering after retrieval, or asking the model to "ignore documents the user may not see", lets a confidential chunk leak. Filter inside the search.

Refusal. The off-topic question's best score was 0.21, under the 0.30 cut-off, so it is handed off. The margin is thin (the real vase match scored 0.38), so calibrate on real questions and keep "say you do not know" as a second guard.

Position. Liu et al. ("Lost in the Middle", 2023) found models used information at the start or end of a long context better than in the middle; GPT-3.5-Turbo's accuracy fell by more than 20 points depending on where the answer sat. Those were mid-2023 models and newer ones may do better, but the cheap defence holds: send a few good chunks, strongest first.

Retrieved text is untrusted input. A page can contain instructions aimed at the model, so never let retrieved text widen what the system may do.

A wrong answer has two causes with unrelated fixes: the retriever missed, or the chunk was in the prompt and the model ignored or garbled it. Log retrieved chunk ids with every answer so you can tell which.

Evaluate in that order:

• Retrieval. Collect real questions with the chunk that answers each, and measure recall@k: how often it is in the top k. No LLM is needed, so you can sweep chunk size, fusion weights and embedding models in seconds, as the tables above do.
• Generation. Where retrieval succeeded, check the answer is correct, grounded and cites a supporting chunk.

Labels can be wrong too. For "I want you to forget everything you know about me" I marked the account article (deletion) as correct; the model chose the privacy article (stored data). Either is defensible, so read failures before trusting a percentage.

Where to look when it goes wrong:

• Right document, wrong chunk: chunking split the answer.
• Paraphrases miss: improve dense retrieval or rewrite the query.
• Right chunk, wrong answer: fewer chunks, best first, a stricter prompt.

The RAG paper's gains were on specific benchmarks. Retrieval only helps when it finds the right text, and a model can still misread a correct chunk.

Q: RAG or fine-tuning for a support bot over company docs?
A: RAG for facts that change, need citations or are restricted per user. Fine-tuning for format, tone or a narrow skill. They combine well.

Q: How do you choose a chunk size?
A: Measure it. Too small splits answers (with 10-word chunks the top chunk held the answer for 8 of 28 questions with dense search and 5 of 28 with BM25 here); too large wastes tokens and buries the answer. Start with paragraphs plus overlap and the page title, then sweep against recall@k.

Q: Dense, BM25 or hybrid?
A: BM25 for exact terms and codes, dense for paraphrase (8 of 12 versus 1 of 12 here). Hybrid with rank fusion is a common default, but equal weights hurt when one list was noise, so tune on your data.

Q: Why not put the whole corpus in a long context?
A: Every question pays for every token, models use the middle of long contexts less reliably, and you lose per-user filtering. For a small, static, single-user corpus it can be fine.

Q: How do you enforce document permissions?
A: Store allowed groups on each chunk and filter inside the search, before ranking, never in the prompt.

Q: Design a support bot over 50,000 internal docs that must cite sources and respect access rights.
A: Chunk by heading with source URL and ACL groups, re-index on change; BM25 plus vector search, both ACL-filtered; RRF, then rerank the top 30; a score threshold that hands off to a human; a prompt demanding [doc-id] citations; logged retrieval ids; recall@k and answer checks on labelled questions.