DL0085 RAG vs Fine-Tuning

Reach for RAG when the failure is a knowledge gap, and fine-tune when the failure is a behavior gap. Knowledge gaps look like this: the required facts change (pricing, inventory, incident history), they are private to a tenant, they are too numerous to memorize (millions of documents), or the answer must carry a citation a human can audit. Behavior gaps look different: the model has the right context in the prompt and still emits the wrong JSON schema, the wrong tone, the wrong clinical or legal register, or a malformed tool call. The diagnostic that settles most arguments is a context-injection test: paste the gold document into the prompt by hand, and if the answer becomes correct, you have a retrieval problem, not a weights problem. Update economics reinforce the split, because a corrected fact ships to a RAG system in minutes by reindexing one chunk, while the same correction in a fine-tuned checkpoint means a training run plus a full regression eval, and controlled comparisons have repeatedly found that continued pretraining on new documents injects facts less reliably than simply retrieving them.

(1) Knowledge vs Behavior Test: if hand-pasting the gold document fixes the output, choose retrieval; if the model still misbehaves with perfect context, the deficit is in the weights and only fine-tuning (or better prompting) moves it.
(2) Update Latency And Provenance: RAG updates at index write speed and can return source spans for audit, whereas a fine-tuned model has no pointer back to evidence and its knowledge is frozen at the last training run.
(3) Cost Moves From Training To Serving: retrieval adds a lookup plus prefill tokens on every query, while fine-tuning pays once up front and then serves short prompts, which matters under a tight token budget or a strict p99 latency SLO.
(4) Failure Modes Are Different: RAG fails by retrieval miss or by grounding on a plausible distractor chunk; fine-tuning fails by confidently asserting stale facts and by catastrophic forgetting of general ability.
(5) The Honest Answer Is Often Both: use RAG for the facts and a small LoRA adapter for format, jargon, and the skill of reading retrieved context (including ignoring irrelevant chunks), which is what retrieval-aware fine-tuning trains explicitly.

Figure 1: The decision starts from a measured failure, not from a technology preference: classify the gap first, then accept the update path and per-query cost that come with the chosen option.

The serving arithmetic is worth doing before committing. A 60-token question with retrieved chunks of 400 tokens each becomes a 2060-token prefill, roughly 34 times the baseline prompt, plus 20 to 80 ms of vector search and reranking. That is usually cheaper than a training run, but it is a recurring per-request tax, and it is why teams with a small static corpus sometimes skip retrieval entirely and rely on long context with prompt caching. Fine-tuning inverts the profile: one-off data curation and GPU hours, then thin prompts forever. A common production shape is retrieve → rerank → generate on a live index, with a LoRA adapter handling the output contract, so the two mechanisms address the two gaps independently instead of competing.

Mathematical Formulation:

Where:

• is the generated answer and the user query; the first line is the RAG marginalization, where new knowledge enters through the retrieved set rather than through the weights.
• is a retrieved passage from the top- set , scored by the retriever , and is the generator; only changes when you reindex.
• is the fine-tuned parameter set obtained by minimizing loss on the curated set ; with LoRA only a low-rank delta is trained, so itself stays frozen.
• is the question length, the tokens per chunk, and the per-query context the RAG path must pay for; fine-tuning leaves .
• , and are the latency terms; a hard SLO such as ms with chunks is where retrieval budgets usually break.

Figure 2: Illustrative fact churn of about 15 accuracy points per quarter: a frozen fine-tuned checkpoint decays, quarterly retraining buys a sawtooth that is stale by construction between runs, and a live index holds accuracy flat with no training at all.

Dimension	RAG	Fine-Tuning	Both
Best for	Changing, private, or high-volume facts that need citations	Output format, tone, domain jargon, tool-call syntax, latency cuts	Grounded answers in a strict contract, with noisy retrieval
Update path	Reindex the changed chunk, minutes, no GPU	New data, training run, regression eval, redeploy	Reindex for facts, retrain only when behavior shifts
Provenance	Source spans returned with the answer	None; knowledge is diffuse in weights	Same as RAG, with better citation discipline
Per-query cost	Retrieval latency plus prefill tokens	Short prompt only; cheapest at high QPS	Same as RAG, plus adapter serving
Main failure mode	Retrieval miss, distractor chunks, lost-in-the-middle	Stale confident facts, catastrophic forgetting, overfit style	Two systems to version, debug, and evaluate together
Build effort	Chunking, embeddings, index, reranker, eval set	Labeled pairs, hyperparameters, GPU time, eval harness	Highest, so justify it with measured gains over RAG alone