DL0084 RAG: Retrieval-Augmented Generation

Retrieval-augmented generation conditions a frozen language model on text fetched at query time from an external corpus, so the answer is grounded in documents the model never memorized. Nothing about the weights changes: the retriever supplies evidence, the prompt carries it, and the decoder writes an answer that should be traceable to that evidence. This buys three things a fine-tuned model cannot easily give you, namely freshness (reindex a document and the next query sees it), attribution (each claim can cite a chunk ID), and access control (metadata filters keep a user out of documents they may not read). A practical pipeline has an offline half and an online half: the offline half runs chunk → embed → index, and the online half runs retrieve → rerank → assemble prompt → generate. The parts that decide whether the system works in production are rarely the LLM; they are the chunking policy, the hybrid retrieval that combines dense and lexical matching, and the cross-encoder reranker that decides which five of fifty candidates actually reach the context window.

(1) Ingestion and Chunking: parse the source into text, then split it into passages of roughly 400 to 800 tokens with modest overlap, attaching metadata (document ID, section, timestamp, ACL tags) that later powers filtering and citation.
(2) Embedding and Indexing: a bi-encoder maps each chunk to a fixed vector stored in an ANN index (HNSW or IVF-PQ) alongside a sparse BM25 or SPLADE index, because dense and lexical retrieval fail on different queries.
(3) Query Processing and Retrieval: the query is optionally rewritten (decontextualized against chat history, expanded, or decomposed for multi-hop), encoded once, and used to pull a wide candidate set of 30 to 100 chunks from both indexes, fused by reciprocal rank fusion.
(4) Reranking and Context Assembly: a cross-encoder scores each query-chunk pair jointly and keeps the top 3 to 8, which are deduplicated, ordered, and packed into the prompt with instructions to answer only from the provided context and to cite chunk IDs.
(5) Generation, Evaluation, and Guardrails: the LLM generates a grounded answer, and an offline harness tracks retrieval recall@k separately from answer faithfulness, since a wrong answer with perfect retrieval and a right answer with broken retrieval demand opposite fixes.

Figure 1: The offline half builds a hybrid index once per document revision; the online half spends its latency budget on retrieval, reranking, and decoding, and only the reranked top-5 chunks ever enter the context window.

Most RAG failures are retrieval failures, so it helps to measure each stage separately. A useful decomposition is recall@50 for the first-stage retriever (did the answer-bearing chunk survive at all?) and recall@5 after reranking (did it reach the prompt?). Dense bi-encoders miss exact identifiers such as error codes, part numbers, and rare acronyms, which BM25 handles trivially; BM25 misses paraphrases, which the bi-encoder handles. Fusing the two lists and then letting a cross-encoder read query and chunk together typically removes a large share of the remaining misses, at the cost of one extra model call over 50 pairs, which is why the reranker runs on the shortlist and never on the corpus.

Figure 2: Illustrative stage-by-stage gains: hybrid fusion fixes the queries dense retrieval alone drops, and the cross-encoder reranker plus context-aware chunk embeddings cut the residual failure rate further.

Mathematical Formulation:

Where:

• is the cosine similarity between the query embedding and the chunk embedding , both produced by the same bi-encoder.
• is the chunked corpus with passages, and is the retrieved set of size that the prompt will carry.
• fuses ranked lists (dense, lexical, and any filtered variants); is the position of chunk in list , and the constant damps the influence of top ranks so one confident list cannot dominate.
• is the frozen decoder, the token at step of a -token answer; the retrieved set enters only as prompt context, so is never updated by indexing new documents.
• In practice the ANN index returns an approximate : HNSW recall is tuned by the search-width parameter, and quantized indexes trade a small recall loss for roughly to less memory.