Tag: LLM

DL0088 PEFT: Parameter-Efficient Fine-Tuning

Compare Parameter-Efficient Fine-Tuning (PEFT) methods: LoRA, adapters, and prompt/prefix tuning. When would you choose each?

Answer

All three families freeze the pretrained weights and train a tiny set of new parameters, but they differ in where those parameters live, and that single structural choice decides inference cost, serving flexibility, and achievable quality. LoRA adds a low-rank update in parallel to existing weight matrices, so after training the update folds back into $W$ and costs exactly zero extra latency. Serial adapters insert a small bottleneck MLP inside the block’s forward path, which keeps task modules cleanly composable but adds depth, so every decoded token pays for them. Prefix and prompt tuning touch no weight matrix at all: prefix tuning prepends trainable key/value vectors at every layer, prompt tuning prepends trainable vectors only at the input embeddings, which is the cheapest option in parameter count but the least stable and it consumes context length. In practice LoRA is the default (paired with 4-bit quantization as QLoRA when GPU memory is the binding constraint), adapters win when modular composition matters more than latency, and prefix/prompt tuning wins when the base weights must stay byte-identical or when you want per-request conditioning.

(1) Where The Parameters Live: LoRA sits parallel to chosen projections, adapters sit serially between sublayers, prefix tuning sits in the KV cache rather than in any weight tensor.
(2) Mergeability Decides Latency: only LoRA is exactly mergeable into $W_0$ ; adapters add two matmuls and a nonlinearity per sublayer that cannot be folded away, which is a 10-25% decode overhead at batch 1 on a 7B model.
(3) Capacity Is A Knob, Except For Prompts: rank $r$ and bottleneck width $b$ scale capacity smoothly, while prompt tuning’s capacity saturates and only becomes competitive with full fine-tuning above roughly 10B parameters.
(4) Multi-Tenant Serving: unmerged LoRA and prefix tuning both support heterogeneous batching (per-request low-rank gather, or per-request KV prefix) from one frozen base copy; merged LoRA needs a separate weight copy per task and adapters need a weight swap per request.

Diagram of one transformer block: input, LayerNorm, self-attention, LayerNorm, FFN, output. A blue LoRA box below attention taps the same input and adds its output back in parallel, a green prefix-tuning box above attention prepends trainable key and value vectors, and an orange serial adapter box sits in the path between the FFN and the block output

Figure 1: The three injection sites in one block. LoRA is parallel and therefore mergeable, the adapter is serial and therefore always in the critical path (real adapter recipes place one after attention and one after the FFN), and prefix tuning changes only K and V, leaving every weight untouched.

Two consequences are worth stating precisely because interviewers probe them. First, the merge is algebraically exact, not an approximation: $W_0 + (\alpha/r)BA$ is just another matrix of the same shape, so a merged LoRA checkpoint is indistinguishable from a normally fine-tuned one at serving time, and the price is that you can no longer swap tasks without unmerging. Second, PEFT saves optimizer and gradient memory, not activation memory: with Adam you drop from roughly 12-16 bytes per trainable parameter of state to almost none, which is why a 7B model fine-tunes on a single 24GB card, but activations still scale with batch size and sequence length, and QLoRA’s 4-bit NF4 base plus paged optimizers is what pushes a 65B model onto one 48GB GPU. Prefix tuning has a third quirk: because the prefix occupies KV slots, a 20-token prefix at every layer permanently shortens the usable context and its quality is non-monotonic in prefix length, which is why the original work needed a reparameterization MLP to train stably at all.

Mathematical Formulation:
$h = W_0 x + \frac{\alpha}{r} B A x$
$W' = W_0 + \frac{\alpha}{r} B A$
$h = x + W_{\mathrm{up}} \sigma(W_{\mathrm{down}} x)$
$\tilde{K} = \mathrm{concat}(P_k, K)$
$\tilde{V} = \mathrm{concat}(P_v, V)$

Where:

$h$ is the sublayer output and $x$ the input; $W_0$ is the frozen pretrained matrix of shape $d_{out} \times d_{in}$ .
$B \in \mathbb{R}^{d_{out} \times r}$ and $A \in \mathbb{R}^{r \times d_{in}}$ are the LoRA factors with rank $r \ll \min(d_{in}, d_{out})$ ; $A$ is Gaussian-initialized and $B$ is zero-initialized so training starts from the pretrained function.
$\alpha$ is a scaling constant, and the ratio $\alpha/r$ keeps the effective update magnitude roughly stable when you sweep $r$ ; $W'$ is the merged weight used at inference.
$W_{\mathrm{down}} \in \mathbb{R}^{b \times d}$ and $W_{\mathrm{up}} \in \mathbb{R}^{d \times b}$ form the adapter bottleneck of width $b$ , with nonlinearity $\sigma$ ; the residual term $x$ plus near-zero initialization makes the adapter an approximate identity at step 0.
$P_k, P_v \in \mathbb{R}^{p \times d_k}$ are the per-layer, per-head prefix key and value vectors of length $p$ ; $K, V$ are the ordinary projected keys and values, so attention runs over $p + n$ positions for a sequence of length $n$ .

Two-panel bar chart. Left panel, log scale: trainable parameters for full fine-tuning 6738M, adapters 33.6M, LoRA 8.4M, prefix tuning 5.2M. Right panel: added per-token decode latency at batch 1, merged LoRA 0 percent, unmerged LoRA 6 percent, serial adapters 18 percent, prefix tuning 4 percent

Figure 2: Illustrative budgets for a 7B decoder with $d = 4096$ and 32 layers. Parameter counts span three orders of magnitude, yet the latency picture is unrelated to them: the smallest method is not the fastest, and merged LoRA is free.

Dimension	LoRA	Serial Adapters	Prefix / Prompt Tuning
Insertion point	Parallel to chosen linear layers (q,v by default; all attention and FFN projections for hard tasks)	Serial bottleneck after the attention and FFN sublayers	Extra K/V vectors at every layer (prompt tuning: input embeddings only)
Trainable share (7B)	0.05-0.5% (8.4M at r=16 on q,v)	0.5-3% (33.6M at b=64, two per layer)	Under 0.1% (5.2M for a 20-token prefix)
Mergeable	Yes, algebraically exact	No, extra depth is irreducible	No, but no weight is modified either
Inference cost	0% merged; roughly 5-10% unmerged at batch 1	Roughly 10-25% at batch 1, worst at small batch	Small KV cost, but the prefix permanently shortens usable context
Many tasks, one base	Strong: keep unmerged and batch a per-request low-rank gather	Weak: needs a per-request module swap, though AdapterFusion composes trained modules	Strong: a prefix is just cached K/V, trivially per-request
Quality on hard tasks	Matches full fine-tuning closely at adequate rank; underfits at very low r on large domain shifts	Comparable to LoRA, with a well-studied bottleneck-width knob	Lags on generation and reasoning; unstable below roughly 10B parameters
Choose it when	Default choice; memory-bound runs pair it with 4-bit QLoRA; hundreds of task variants over one base	You want modular, composable, independently versioned task modules and have latency headroom	Base weights must stay immutable, or you need cheap per-request conditioning at very large scale

August 15, 2026

DL0087 QLoRA: 4-bit Quantized LoRA

What is QLoRA, and how does it combine 4-bit quantization with LoRA?

Answer

QLoRA (Dettmers et al., 2023) is a fine-tuning recipe that keeps the entire base model frozen in 4-bit precision and trains only LoRA adapters in 16-bit on top of it, so the memory that would have held gradients and optimizer state for billions of parameters simply never gets allocated. During the forward pass each 4-bit weight block is dequantized to bf16 on the fly, used for the matrix multiply, and discarded; the backward pass dequantizes again to propagate gradients through the frozen weights into the adapters, which are the only tensors that receive updates. Three components make the 4-bit base workable: the NF4 data type, which places its 16 levels at quantiles of a normal distribution rather than uniformly, double quantization of the per-block scaling constants, and paged optimizers that offload optimizer state through unified memory when a long-sequence step spikes. The headline result was fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit LoRA quality on instruction-following benchmarks. Note that QLoRA quantizes the frozen base, never the adapters or the gradients, so all learning still happens in bf16.

(1) NF4 (4-bit NormalFloat): pretrained weights within a block are approximately zero-centered Gaussian, so NF4 spaces its levels at quantiles of a standard normal instead of uniformly, which is information-theoretically optimal for that assumption and beats 4-bit float or int at equal bit width.
(2) Blockwise Absmax Scaling: weights are quantized in blocks of 64 with one absmax constant per block, which localizes outliers so a single large weight cannot crush the resolution of an entire tensor.
(3) Double Quantization: those constants are themselves quantized to 8-bit with a second-level fp32 scale per 256 constants, cutting the metadata overhead from 0.5 to about 0.127 bits per parameter (roughly 3 GB on a 65B model).
(4) Only Adapters Train: gradients pass through the frozen 4-bit weights but are stored only for $A$ and $B$ , so optimizer state scales with the adapter rank, not with model size.
(5) Paged Optimizers: optimizer states live in NVIDIA unified memory and are paged to host RAM during transient spikes, which is what keeps a 33B or 65B single-GPU run from OOM-ing on a long batch.

Block diagram of a QLoRA linear layer: input X feeds a frozen NF4 base weight with its double-quantized absmax constants, which is dequantized to bf16 for the matmul, while a parallel bf16 LoRA A and B path scaled by alpha over r is added to produce output Y, with dashed gradient arrows returning only into A and B

Figure 1: One QLoRA linear layer. The NF4 weight and its quantization constants are read-only, the bf16 copy exists only for the duration of the matmul, and the dashed path shows that gradients terminate at the low-rank adapters.

Mathematical Formulation:
$Y = X\,\mathrm{dq}(W_4) + \frac{\alpha}{r}\,X A B$
$\mathrm{dq}(W_4)_{ij} = c_i\,z_{q_{ij}}$
$c_i = \mathrm{absmax}(W_i)$
$b = 4 + \frac{8}{64} + \frac{32}{64 \cdot 256}$
$b \approx 4.127\ \text{bits per parameter}$

Where:

$Y$ is the layer output and $X$ the bf16 input activation; both stay in 16-bit throughout.
$W_4$ is the frozen base weight stored as NF4 indices, and $\mathrm{dq}$ is the dequantization that reconstructs a bf16 tile just before the matmul.
$i$ indexes the blocks of 64 weights, $q_{ij} \in \{0,\ldots,15\}$ is the stored 4-bit code, and $z_k$ are the fixed NF4 levels, obtained from normal quantiles and rescaled to $[-1, 1]$ with an exact zero.
$c_i$ is the per-block absmax constant; double quantization stores it in 8 bits with one fp32 scale per 256 constants.
$A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$ are the trainable bf16 adapters of rank $r$ (the paper uses $r = 64$ on every linear layer), scaled by $\alpha / r$ ; $B$ starts at zero so the layer initially reproduces the quantized base model.
$b$ is the effective storage cost per base parameter: 4 bits of payload plus 0.125 bits of 8-bit constants plus 0.002 bits of second-level scales.

Horizontal stacked bar chart of training memory for a 7B model: full fine-tuning totals 112 GB with 14 GB weights, 14 GB gradients and 84 GB optimizer states; 16-bit LoRA totals 16.6 GB; QLoRA totals 6.2 GB with a 3.6 GB NF4 base

Figure 2: Illustrative state memory for a 7B model with rank-64 adapters on all linear layers. LoRA removes the optimizer and gradient bulk, and QLoRA then shrinks the remaining frozen weights from 14 GB to 3.6 GB; activation memory is excluded.

Aspect	QLoRA (NF4 base)	16-bit LoRA	Full Fine-Tuning
Base weight storage	NF4, about 4.13 bits per parameter with double quantization	bf16, 16 bits per parameter	bf16 weights plus an fp32 master copy
Trainable parameters (7B)	About 160M in bf16 at rank 64 on all linear layers	Identical adapter count	All 7B parameters
State memory (7B)	About 6 GB	About 17 GB	About 112 GB, needs sharding across GPUs
Step time	Slowest: dequantization runs in both forward and backward	Fastest of the two adapter methods	Highest total compute and communication
Instruction-tuning quality	Matches 16-bit LoRA when adapters cover every linear layer	Reference point for adapter tuning	Preferred with very large in-domain corpora or heavy domain shift
Deployment	Adapter cannot fold into NF4 losslessly; merge into the 16-bit base, then requantize	Merge into bf16 weights for zero added latency	One merged checkpoint per task

August 15, 2026

DL0086 LoRA: Low-Rank Adaptation

What is LoRA, and can you explain the low-rank decomposition idea and why it is parameter-efficient?

Answer

LoRA (Low-Rank Adaptation) freezes every pretrained weight matrix and learns only a rank-constrained update beside it: instead of training $W_0 + \Delta W$ with a full $d \times k$ update, it factors the update as $\Delta W = BA$ with $B \in \mathbb{R}^{d \times r}$ , $A \in \mathbb{R}^{r \times k}$ and $r \ll \min(d,k)$ . The motivation is the intrinsic-dimension hypothesis: adapting a large pretrained model to a downstream task moves the weights along a very low-dimensional subspace, so a rank-8 or rank-16 correction captures almost all of the useful change. For one $4096 \times 4096$ attention projection, a rank-16 factorization trains 131,072 numbers instead of 16.8M, roughly 0.78% of the matrix. The efficiency that matters in practice is not forward FLOPs, which barely change, but optimizer and gradient state: Adam keeps two moments plus an fp32 master copy per trainable parameter, so shrinking the trainable set by two or three orders of magnitude is what turns a multi-node job into a single-GPU job. At inference the product can be folded back with $W' = W_0 + (\alpha/r)BA$ , so a merged LoRA adds zero latency compared with the original model.

(1) Frozen Base, Additive Branch: the pretrained matrix never receives a gradient; all learning happens in the parallel $BA$ branch whose output is added to the frozen path.
(2) Rank Is The Budget Knob: parameter count is $r(d+k)$ and grows linearly in $r$ , while the full matrix costs $dk$ , so the ratio is tiny for any practical rank.
(3) Zero Initialization On One Factor: $A$ gets a random Gaussian init and $B$ is initialized to zeros, so $\Delta W = 0$ at step 0 and training starts exactly at the pretrained function.
(4) Memory Savings Come From Optimizer State: gradients and Adam moments exist only for adapter parameters, which is where the 8x reduction in peak training memory comes from, not from cheaper matrix multiplies.
(5) Mergeable And Swappable: a task adapter is a few tens of MB, so many tasks can share one frozen base, either merged per deployment or batched as separate adapters in a single server.

Diagram of a LoRA layer: the input feeds a frozen 4096 by 4096 projection on the upper path and a rank-16 bottleneck A then B then alpha over r scaling on the lower path, with both paths summed into the output

Figure 1: A LoRA layer keeps the frozen 16.8M-parameter projection untouched and routes the input through a rank-16 bottleneck whose 131,072 trainable parameters are added back, scaled by $\alpha/r$ .

Two implementation details do most of the work. The scaling factor $\alpha/r$ decouples the effective update magnitude from the rank, so raising $r$ to add capacity does not silently multiply the learning rate on the adapter branch; the common convention is to fix $\alpha$ and sweep $r$ . The choice of target modules matters more than the rank in most experiments: the original paper adapted only the attention query and value projections, while current practice applies LoRA to all linear layers including the MLP, which raises trainable parameters to about 42M on an 8B model but consistently closes the gap to full fine-tuning. One caveat on merging: if the base weights are quantized, folding a bf16 adapter into a 4-bit matrix reintroduces quantization error, so quantized deployments usually keep the adapter separate.

Mathematical Formulation:
$h = W_0 x + \Delta W x$
$\Delta W = \frac{\alpha}{r} B A$
$|\Theta| = r(d + k)$
$\rho = \frac{r(d + k)}{d k}$
$W' = W_0 + \frac{\alpha}{r} B A$

Where:

$h$ is the layer output and $x \in \mathbb{R}^{k}$ the input activation for one token.
$W_0 \in \mathbb{R}^{d \times k}$ is the frozen pretrained matrix, and $\Delta W$ is the learned correction.
$B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ are the trainable factors; the rank $r$ satisfies $r \ll \min(d,k)$ , which forces $\mathrm{rank}(\Delta W) \leq r$ .
$\alpha$ is a fixed scaling constant, so $\alpha/r$ keeps the update magnitude roughly rank-independent.
$|\Theta|$ is the trainable count per adapted matrix and $\rho$ its fraction of the full matrix; with $d = k = 4096$ and $r = 16$ this gives 131,072 parameters and $\rho \approx 0.0078$ .
Required initial condition: $B = 0$ with $A$ Gaussian, so $\Delta W = 0$ before the first step; the last line is the merge applied after training.

Stacked bar chart of training memory in gigabytes for an 8B model: full fine-tuning totals 128 GB from 16 GB base weights, 16 GB gradients and 96 GB optimizer state, LoRA rank 16 totals 16.6 GB, and QLoRA with a 4-bit base totals 4.6 GB

Figure 2: Illustrative training memory for an 8B model before activations: full fine-tuning spends 96 GB on Adam state alone, LoRA with 41.9M trainable parameters needs about 16.6 GB, and a 4-bit frozen base drops that to about 4.6 GB.

Property	LoRA	Full Fine-Tuning	Bottleneck Adapters
Trainable parameters (8B model)	41.9M at r=16 on all linear layers, about 0.5%	8.0B, all of them	Comparable order, set by bottleneck width
Added inference latency	None once merged into the base weights	None	Extra sequential layers, noticeable at small batch
Peak training memory	About 17 GB, or 5 GB with a 4-bit base	About 128 GB with Adam in mixed precision	Similar to LoRA
Multi-task serving	Hundreds of small adapters over one frozen base, batchable	One 16 GB checkpoint per task	Swappable but cannot be merged away
Quality ceiling	Matches full tuning on most instruction and style adaptation	Upper bound, clearly better for large domain shift	Roughly on par, less used today

August 15, 2026

DL0085 RAG vs Fine-Tuning

When should you use RAG instead of fine-tuning?

Answer

Reach for RAG when the failure is a knowledge gap, and fine-tune when the failure is a behavior gap. Knowledge gaps look like this: the required facts change (pricing, inventory, incident history), they are private to a tenant, they are too numerous to memorize (millions of documents), or the answer must carry a citation a human can audit. Behavior gaps look different: the model has the right context in the prompt and still emits the wrong JSON schema, the wrong tone, the wrong clinical or legal register, or a malformed tool call. The diagnostic that settles most arguments is a context-injection test: paste the gold document into the prompt by hand, and if the answer becomes correct, you have a retrieval problem, not a weights problem. Update economics reinforce the split, because a corrected fact ships to a RAG system in minutes by reindexing one chunk, while the same correction in a fine-tuned checkpoint means a training run plus a full regression eval, and controlled comparisons have repeatedly found that continued pretraining on new documents injects facts less reliably than simply retrieving them.

(1) Knowledge vs Behavior Test: if hand-pasting the gold document fixes the output, choose retrieval; if the model still misbehaves with perfect context, the deficit is in the weights and only fine-tuning (or better prompting) moves it.
(2) Update Latency And Provenance: RAG updates at index write speed and can return source spans for audit, whereas a fine-tuned model has no pointer back to evidence and its knowledge is frozen at the last training run.
(3) Cost Moves From Training To Serving: retrieval adds a lookup plus $k \cdot n_{chunk}$ prefill tokens on every query, while fine-tuning pays once up front and then serves short prompts, which matters under a tight token budget or a strict p99 latency SLO.
(4) Failure Modes Are Different: RAG fails by retrieval miss or by grounding on a plausible distractor chunk; fine-tuning fails by confidently asserting stale facts and by catastrophic forgetting of general ability.
(5) The Honest Answer Is Often Both: use RAG for the facts and a small LoRA adapter for format, jargon, and the skill of reading retrieved context (including ignoring irrelevant chunks), which is what retrieval-aware fine-tuning trains explicitly.

Flowchart: a failure measured on a held-out eval set fans out into three branches, knowledge gap, behavior gap, and both gaps, which route to RAG updated by reindexing, a LoRA fine-tune updated by retraining, and retrieval-aware fine-tuning that runs both pipelines, with per-query cost notes beneath each option

Figure 1: The decision starts from a measured failure, not from a technology preference: classify the gap first, then accept the update path and per-query cost that come with the chosen option.

The serving arithmetic is worth doing before committing. A 60-token question with $k = 5$ retrieved chunks of 400 tokens each becomes a 2060-token prefill, roughly 34 times the baseline prompt, plus 20 to 80 ms of vector search and reranking. That is usually cheaper than a training run, but it is a recurring per-request tax, and it is why teams with a small static corpus sometimes skip retrieval entirely and rely on long context with prompt caching. Fine-tuning inverts the profile: one-off data curation and GPU hours, then thin prompts forever. A common production shape is retrieve → rerank → generate on a live index, with a LoRA adapter handling the output contract, so the two mechanisms address the two gaps independently instead of competing.

Mathematical Formulation:
$p(y \mid x) = \sum_{z \in \mathcal{Z}_k} p_{\eta}(z \mid x) p_{\theta}(y \mid x, z)$
$\theta^{*} = \arg\min_{\theta} \mathcal{L}(\theta; \mathcal{D}_{ft})$
$n_{prefill} = n_{q} + k \cdot n_{chunk}$
$t_{total} = t_{retrieve} + t_{prefill} + t_{decode}$

Where:

$y$ is the generated answer and $x$ the user query; the first line is the RAG marginalization, where new knowledge enters through the retrieved set rather than through the weights.
$z$ is a retrieved passage from the top- $k$ set $\mathcal{Z}_k$ , scored by the retriever $p_{\eta}$ , and $p_{\theta}$ is the generator; only $\mathcal{Z}_k$ changes when you reindex.
$\theta^{*}$ is the fine-tuned parameter set obtained by minimizing loss $\mathcal{L}$ on the curated set $\mathcal{D}_{ft}$ ; with LoRA only a low-rank delta is trained, so $\theta$ itself stays frozen.
$n_{q}$ is the question length, $n_{chunk}$ the tokens per chunk, and $n_{prefill}$ the per-query context the RAG path must pay for; fine-tuning leaves $n_{prefill} = n_{q}$ .
$t_{retrieve}$ , $t_{prefill}$ and $t_{decode}$ are the latency terms; a hard SLO such as $t_{total} \leq 300$ ms with $k = 20$ chunks is where retrieval budgets usually break.

Line chart of fact-level answer accuracy over 365 days after a training cutoff: a fine-tuned checkpoint with no retraining decays from 0.88 to about 0.30, quarterly retraining produces a sawtooth that recovers to 0.88 every 91 days, and RAG over a live index stays flat near 0.86

Figure 2: Illustrative fact churn of about 15 accuracy points per quarter: a frozen fine-tuned checkpoint decays, quarterly retraining buys a sawtooth that is stale by construction between runs, and a live index holds accuracy flat with no training at all.

Dimension	RAG	Fine-Tuning	Both
Best for	Changing, private, or high-volume facts that need citations	Output format, tone, domain jargon, tool-call syntax, latency cuts	Grounded answers in a strict contract, with noisy retrieval
Update path	Reindex the changed chunk, minutes, no GPU	New data, training run, regression eval, redeploy	Reindex for facts, retrain only when behavior shifts
Provenance	Source spans returned with the answer	None; knowledge is diffuse in weights	Same as RAG, with better citation discipline
Per-query cost	Retrieval latency plus $k \cdot n_{chunk}$ prefill tokens	Short prompt only; cheapest at high QPS	Same as RAG, plus adapter serving
Main failure mode	Retrieval miss, distractor chunks, lost-in-the-middle	Stale confident facts, catastrophic forgetting, overfit style	Two systems to version, debug, and evaluate together
Build effort	Chunking, embeddings, index, reranker, eval set	Labeled pairs, hyperparameters, GPU time, eval harness	Highest, so justify it with measured gains over RAG alone

August 15, 2026

DL0084 RAG: Retrieval-Augmented Generation

What is RAG (retrieval-augmented generation), and describe the components of a RAG pipeline?

Answer

Retrieval-augmented generation conditions a frozen language model on text fetched at query time from an external corpus, so the answer is grounded in documents the model never memorized. Nothing about the weights changes: the retriever supplies evidence, the prompt carries it, and the decoder writes an answer that should be traceable to that evidence. This buys three things a fine-tuned model cannot easily give you, namely freshness (reindex a document and the next query sees it), attribution (each claim can cite a chunk ID), and access control (metadata filters keep a user out of documents they may not read). A practical pipeline has an offline half and an online half: the offline half runs chunk → embed → index, and the online half runs retrieve → rerank → assemble prompt → generate. The parts that decide whether the system works in production are rarely the LLM; they are the chunking policy, the hybrid retrieval that combines dense and lexical matching, and the cross-encoder reranker that decides which five of fifty candidates actually reach the context window.

(1) Ingestion and Chunking: parse the source into text, then split it into passages of roughly 400 to 800 tokens with modest overlap, attaching metadata (document ID, section, timestamp, ACL tags) that later powers filtering and citation.
(2) Embedding and Indexing: a bi-encoder maps each chunk to a fixed vector stored in an ANN index (HNSW or IVF-PQ) alongside a sparse BM25 or SPLADE index, because dense and lexical retrieval fail on different queries.
(3) Query Processing and Retrieval: the query is optionally rewritten (decontextualized against chat history, expanded, or decomposed for multi-hop), encoded once, and used to pull a wide candidate set of 30 to 100 chunks from both indexes, fused by reciprocal rank fusion.
(4) Reranking and Context Assembly: a cross-encoder scores each query-chunk pair jointly and keeps the top 3 to 8, which are deduplicated, ordered, and packed into the prompt with instructions to answer only from the provided context and to cite chunk IDs.
(5) Generation, Evaluation, and Guardrails: the LLM generates a grounded answer, and an offline harness tracks retrieval recall@k separately from answer faithfulness, since a wrong answer with perfect retrieval and a right answer with broken retrieval demand opposite fixes.

Two-row diagram: the offline row runs corpus to chunker to embedding model to a hybrid index of HNSW vectors plus BM25 postings; the online row runs user query to rewrite and encode to hybrid retrieval of top-50 candidates to a cross-encoder rerank to top-5 to prompt assembly to LLM answer with citations, with a dashed arrow from the index down to the retrieval stage

Figure 1: The offline half builds a hybrid index once per document revision; the online half spends its latency budget on retrieval, reranking, and decoding, and only the reranked top-5 chunks ever enter the context window.

Most RAG failures are retrieval failures, so it helps to measure each stage separately. A useful decomposition is recall@50 for the first-stage retriever (did the answer-bearing chunk survive at all?) and recall@5 after reranking (did it reach the prompt?). Dense bi-encoders miss exact identifiers such as error codes, part numbers, and rare acronyms, which BM25 handles trivially; BM25 misses paraphrases, which the bi-encoder handles. Fusing the two lists and then letting a cross-encoder read query and chunk together typically removes a large share of the remaining misses, at the cost of one extra model call over 50 pairs, which is why the reranker runs on the shortlist and never on the corpus.

Bar chart of top-20 retrieval failure rate for five configurations: BM25 only at 9.2 percent, dense only at 5.7 percent, hybrid fusion at 4.5 percent, hybrid plus reranker at 3.2 percent, and contextual chunk embeddings plus hybrid plus reranker at 2.4 percent

Figure 2: Illustrative stage-by-stage gains: hybrid fusion fixes the queries dense retrieval alone drops, and the cross-encoder reranker plus context-aware chunk embeddings cut the residual failure rate further.

Mathematical Formulation:
$s(q,c) = \frac{e_q^{\top} e_c}{\|e_q\|\,\|e_c\|}$
$\mathcal{R}_k(q) = \text{top-}k\{\, s(q,c) : c \in \mathcal{D} \,\}$
$\mathrm{RRF}(c) = \sum_{r=1}^{R} \frac{1}{K + \mathrm{rank}_r(c)}$
$p(y \mid q) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{1:t-1}, q, \mathcal{R}_k(q))$

Where:

$s(q,c)$ is the cosine similarity between the query embedding $e_q$ and the chunk embedding $e_c$ , both produced by the same bi-encoder.
$\mathcal{D} = \{c_1,\ldots,c_M\}$ is the chunked corpus with $M$ passages, and $\mathcal{R}_k(q)$ is the retrieved set of size $k$ that the prompt will carry.
$\mathrm{RRF}(c)$ fuses $R$ ranked lists (dense, lexical, and any filtered variants); $\mathrm{rank}_r(c)$ is the position of chunk $c$ in list $r$ , and the constant $K \approx 60$ damps the influence of top ranks so one confident list cannot dominate.
$p_\theta$ is the frozen decoder, $y_t$ the token at step $t$ of a $T$ -token answer; the retrieved set enters only as prompt context, so $\theta$ is never updated by indexing new documents.
In practice the ANN index returns an approximate $\mathcal{R}_k$ : HNSW recall is tuned by the search-width parameter, and quantized indexes trade a small recall loss for roughly $4\times$ to $8\times$ less memory.

Dimension	RAG	Fine-tuning	Long-Context Stuffing
Best for	Large, changing, permissioned corpora	Style, format, task behavior, domain jargon	Small corpora that fit in one window
Update cost	Reindex the changed chunks, seconds	A new training run per refresh	None, but every request pays for the tokens
Attribution	Native: cite the retrieved chunk ID	None, facts are diffused into weights	Possible, but no ranking signal to audit
Main failure mode	Retriever misses the answer-bearing chunk	Confident stale answers, catastrophic forgetting	Lost-in-the-middle, cost and latency growth
Serving complexity	High: index, retriever, reranker, eval harness	Low at serve time, high at train time	Low, especially with prompt caching

August 15, 2026

DL0083 LLM Hallucination and Reduction

What is hallucination in LLMs, and what techniques reduce it?

Answer

A hallucination is fluent, confident model output that is not supported by the provided source or by verifiable fact. The standard split is intrinsic (the output contradicts a document that was placed in the context) versus extrinsic (the output adds unsupported content, such as an invented citation, API argument, or date). Hallucination is structural rather than a bug in any single checkpoint: next-token maximum likelihood rewards plausible continuations, not true ones, parametric memory of rare facts is lossy, and benchmarks scored on plain accuracy give zero credit for saying “I don’t know,” so a guess strictly dominates an abstention during both evaluation and preference tuning. Reduction therefore works on three fronts at once: put the evidence in the context (retrieval grounding with citations), check the produced claims against that evidence (claim-level verification), and change the objective so the model is allowed to abstain when its confidence is below the scoring threshold.

(1) Grounding Beats Recall: retrieving the passage and requiring an inline citation converts a memory problem into a reading-comprehension problem, which is what cuts unsupported claims the most for factual queries.
(2) Verify At Claim Granularity: decompose the answer into atomic claims and run an entailment (NLI) check of each claim against its cited span; a single unsupported claim in an otherwise correct paragraph is exactly what whole-answer scoring misses.
(3) Uncertainty Signals: sampling several answers and measuring disagreement over meanings rather than token strings (semantic entropy, SelfCheckGPT-style consistency checks) flags confabulation without any external corpus.
(4) Calibrated Abstention: a fixed answer-or-abstain threshold derived from the scoring rule trades coverage for precision, and is the only technique that removes hallucinations the retriever never had evidence for.
(5) Decoding And Tuning Help At The Margin: contrastive decoding against earlier layers or against the ungrounded prior, plus fine-tuning that rewards “insufficient evidence” responses, reduce sycophancy and copy-drift but do not manufacture missing knowledge.

Grouped bar chart comparing hallucination rate and answer coverage for five configurations: closed-book baseline at 42 percent hallucination and 100 percent coverage, chain of thought with self-consistency at 31 and 100, RAG top-5 at 18 and 98, RAG plus claim verification at 11 and 95, and RAG plus verification plus abstention at 4 percent hallucination but only 78 percent coverage

Figure 1: Illustrative long-tail QA sweep: grounding and verification push the hallucination rate from 42% to 11% while coverage stays near 95%, and only abstention reaches 4%, at the cost of dropping coverage to 78%.

In production the pieces are wired as one loop rather than as independent tricks: retrieve → generate with citations → extract claims → entail each claim against its cited span, and route anything unsupported to a second retrieval pass or to an explicit refusal. Two details decide whether the loop actually helps. First, the verifier must judge support, not plausibility, so an NLI model or a judge restricted to the cited span is required; asking the same generator “is this correct?” mostly reproduces its original error. Second, a retrieval miss must be visible: if the top-k passages are off-topic and the prompt still demands an answer, the model quietly falls back to parametric memory and attaches a real-looking citation to a fabricated statement, which is the worst failure mode because it survives casual review. Systems that expose this path well (Google’s Vertex AI grounding returns per-claim support scores alongside the answer) make the unsupported fraction a monitorable metric instead of an anecdote.

Left to right pipeline of six boxes: user query, retrieve top-k passages, generate answer with citations, split into atomic claims, NLI check versus cited passage, return answer plus evidence, with a dashed feedback path from the NLI check back to the retriever labelled unsupported claim, re-retrieve or abstain

Figure 2: The verification loop: every atomic claim is entailment-checked against the span it cites, and unsupported claims trigger re-retrieval or abstention instead of being returned with a decorative citation.

Mathematical Formulation:
$\mathbb{E}[S] = p - c\,(1 - p)$
$p \geq \frac{c}{1 + c}$
$p(C_k \mid x) = \sum_{s \in C_k} p(s \mid x)$
$H_{sem}(x) = -\sum_{k=1}^{K} p(C_k \mid x)\,\log p(C_k \mid x)$

Where:

$\mathbb{E}[S]$ is the expected score of answering under a rule that pays $+1$ for correct, $-c$ for wrong, and $0$ for abstaining; $p$ is the model’s probability that its answer is correct.
The second line is the abstention threshold: answer only when $p$ clears it. Plain accuracy scoring is the case $c = 0$ , where the threshold collapses to $0$ and guessing is never penalized, which is the incentive that trains hallucination into the model.
$x$ is the prompt and $s$ a sampled generation; $C_k$ is a semantic equivalence class of generations grouped by bidirectional entailment, with $k \in \{1,\ldots,K\}$ indexing the $K$ distinct meanings observed.
$H_{sem}(x)$ is the semantic entropy: it is near $0$ when all samples paraphrase one meaning and large when the model spreads mass over mutually contradictory meanings, so a high value is a practical confabulation flag even when token-level entropy is low.

Technique	What It Actually Fixes	Cost	When to Reach for It
Retrieval grounding with citations	Missing or stale parametric knowledge; long-tail entities	Index build plus retrieval latency and a much longer prompt	Default for any factual, enterprise, or time-sensitive query
Claim extraction plus NLI verifier	Answers that cite a real source but overstate what it says	One extra model call per claim; adds hundreds of ms	Regulated or high-stakes surfaces where citations are shown to users
Sampling self-checks and semantic entropy	Confabulation on questions with no retrievable source	5x to 20x tokens; misses confident systematic errors	Offline audits, dataset cleaning, and routing to human review
Calibrated abstention and refusal tuning	Guessing when evidence is absent; sycophantic agreement	Lower coverage and more “I cannot verify this” responses	When a wrong answer costs far more than a missing one
Contrastive decoding (DoLa, context-aware)	Drift back to the parametric prior while a document is in context	Cheap, but needs logit access and per-model tuning	Summarization and closed-context QA on self-hosted models

August 15, 2026

DL0082 LLM Temperature

What is LLM temperature?

Answer

Temperature is a single positive scalar that divides the model’s logits before the softmax at every decoding step, so it reshapes the next-token distribution without touching the model weights. At $T = 1$ you sample from exactly the distribution the model was trained to produce; values below 1 sharpen that distribution toward the highest-scoring token, and values above 1 flatten it toward uniform over the vocabulary. The transformation is monotone in the logits, so temperature never changes which token is ranked first, only how much probability the gaps between logits are worth. The practical effect is a variance knob: low temperature gives repetitive but stable text, high temperature gives diverse text with a much fatter tail of implausible tokens. Because it is applied independently at every step of an autoregressive loop, a small per-token change in tail mass compounds across a long generation.

(1) Logit Rescaling, Not Reranking: dividing all logits by the same $T$ preserves their order, so the argmax token is invariant and only the sampling probabilities move.
(2) The Two Limits: as $T$ approaches 0 sampling collapses to greedy decoding, and as $T$ grows the distribution approaches uniform; frameworks special-case $T = 0$ as argmax because the division itself is undefined.
(3) Tails Gain Disproportionately: temperature is a power transform of the base distribution, so unlikely tokens change by the largest relative factor; with logits $(4, 3, 2, 1)$ the top token falls from 0.644 to 0.455 at $T = 2$ while the weakest rises from 0.032 to 0.102.
(4) Compounding Across Steps: the model conditions on its own samples, so one low-quality token drawn from the inflated tail can steer every subsequent token, which is why high temperature degrades long generations faster than short ones.
(5) Not A Truthfulness Knob: lowering $T$ reduces variance, not error; if the mode is wrong, greedy decoding returns the wrong answer with total confidence.

Grouped bar chart of next-token probabilities for four tokens with logits 4.0, 3.0, 2.0 and 1.0 at temperatures 0.5, 1.0 and 2.0, showing the top token falling from 0.865 to 0.455 and the weakest token rising from 0.002 to 0.102

Figure 1: The same four logits under three temperatures. Entropy grows from 0.66 bits at $T = 0.5$ to 1.80 bits at $T = 2$ , and the weakest token gains roughly 47x probability across that range while the top token loses less than half.

Temperature never acts alone in a real decoder. The usual pipeline is logits → temperature → top-k or top-p truncation → renormalize → sample, and the ordering matters: because temperature runs first, raising it inflates the tail that nucleus sampling then has to cut, so the same $p = 0.95$ threshold admits a larger candidate set than it did at $T = 1$ . That is why “high temperature plus top-p” is not a safety net, and why many production defaults pair a moderate temperature with a fixed truncation instead of pushing either one hard. A separate but easily confused use of the same formula is temperature scaling for calibration, where a single $T$ is fit on held-out data to correct an overconfident classifier; there the goal is matching predicted confidence to observed accuracy, not generating diverse text, and the fitted value is a property of the model rather than a user-facing creativity dial.

Mathematical Formulation:
$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^{V} \exp(z_j/T)}$
$p_i(T) \propto p_i(1)^{1/T}$
$i^{*} = \arg\max_j z_j$
$\lim_{T \to 0^{+}} p_{i^{*}}(T) = 1$
$H(T) = -\sum_{i=1}^{V} p_i(T) \log_2 p_i(T)$

Where:

$p_i(T)$ is the sampling probability of vocabulary token $i$ at temperature $T$ , and $z_i$ is that token’s raw logit from the final linear layer.
$V$ is the vocabulary size and $j$ indexes every candidate in the softmax denominator, so all probabilities are renormalized after rescaling.
$T > 0$ is the temperature: values below 1 sharpen the distribution, values above 1 flatten it, and $T = 1$ is an identity operation.
The second line shows temperature is a power transform with exponent $1/T$ applied to the base distribution, which is the compact reason tail tokens move by the largest relative factor.
$i^{*}$ is the argmax token, and the fourth line is the greedy limit; implementations therefore branch to argmax at $T = 0$ instead of evaluating the ratio.
$H(T)$ is the entropy of the next-token distribution in bits; it increases monotonically with $T$ and is bounded above by $\log_2 V$ .

Two panels versus temperature from 0.05 to 5: left panel shows entropy in bits rising monotonically from near 0 toward the 2-bit uniform limit and passing 1.37 bits at T equals 1; right panel shows top-token probability falling from 1 to about 0.33 while the mass of the two weakest tokens rises from near 0 to about 0.40

Figure 2: Sweeping $T$ over the same four logits. Entropy is monotonically increasing and saturates at $\log_2 4 = 2$ bits, while the tail mass of the two weakest tokens more than doubles between $T = 1$ and $T = 2$ .

Knob	Temperature	Top-k	Top-p (Nucleus)
What it changes	Rescales all logits; every token keeps nonzero probability	Keeps the k highest-scoring tokens, zeroes the rest	Keeps the smallest set whose mass reaches p, zeroes the rest
Adapts to context?	No, the same scalar at every step regardless of confidence	No, a fixed candidate count	Yes, the set shrinks when the model is confident
Typical failure mode	Too low: loops and boilerplate. Too high: off-topic or invented tokens	Fixed k is too tight on flat distributions and too loose on peaked ones	A high temperature inflates the tail, so the same p admits far more junk
When to reach for it	Global variance control, and generating diverse samples for self-consistency voting	Cheap hard cap on the candidate set, useful as a safety floor	Default truncation for open-ended text, paired with a modest temperature

August 15, 2026

DL0081 Tokenizer Vocabulary Size Effects

How does the tokenizer vocabulary size affect LLM model quality, memory, and throughput?

Answer

Vocabulary size $V$ is one of the few hyperparameters that moves quality, memory, and throughput in opposite directions at the same time, so there is a genuine optimum rather than a “bigger is better” rule. On the cost side, the embedding and unembedding tables grow linearly in $V$ (up to $2Vd$ parameters when untied), the output projection adds $2Vd$ FLOPs per token, and the training-time logits tensor of shape $b \times T \times V$ in fp32 becomes one of the largest activations in the whole model. On the benefit side, a larger vocabulary compresses text into fewer tokens, which shortens sequences, shrinks the KV cache, cuts the $O(T^2)$ attention term, and reduces the number of autoregressive decode steps needed to emit the same text. The catch is that compression improves only logarithmically in $V$ while cost grows linearly, and each added token gets a smaller slice of the training signal, so rare embeddings end up undertrained. Empirically the optimum grows with model size: 32k was reasonable for GPT-2-era models, Meta’s Llama 3 moved to 128,256 tokens, and Google’s Gemma 2 uses 256,128, while byte-level and byte-patch models sit at the opposite extreme.

(1) Memory Cost Is Linear: at $d = 4096$ and $V = 128\text{k}$ , untied tables hold about 1.05B parameters, roughly 13% of an 8B model, and 2.1 GB of bf16 weights before optimizer state.
(2) Compression Gain Is Logarithmic: going 32k → 128k buys only about 11% fewer tokens per document in English, while the embedding tables quadruple.
(3) The Output Softmax Dominates Small Models: the $d \times V$ projection is about 7% of forward FLOPs for an 8B model but roughly a third for a 1B model with $d = 2048$ .
(4) Quality Is Non-Monotonic: too small and sequences are long and multilingual fertility explodes; too large and rare rows receive too few gradient updates, producing dead or glitch tokens.
(5) Throughput Has Two Units: tokens per second falls slightly as $V$ grows, but bytes (or words) per second usually rises because fewer steps are needed for the same text.

Two bar panels: left panel shows bytes per token rising from 3.1 at 8k vocabulary to 3.9 at 32k, 4.1 at 50k, 4.4 at 128k and 4.6 at 256k; right panel shows relative sequence length for a fixed document falling from 1.26 at 8k to 1.00 at 32k, 0.95 at 50k, 0.89 at 128k and 0.85 at 256k

Figure 1: Illustrative English compression: quadrupling the vocabulary from 32k to 128k lifts bytes per token from 3.9 to 4.4, which shortens sequences by only about 11%, while the embedding tables grow 4x.

The most under-appreciated cost is not the weights but the logits activation. Cross-entropy is normally computed in fp32, so a single 8192-token sequence against a 128k vocabulary materializes $8192 \times 128000 \times 4$ bytes, about 4.2 GB, and a microbatch of four pushes past 16 GB before the backward pass. This is why large-vocabulary training runs adopt chunked or fused cross-entropy kernels that never hold the full logits matrix, and why teams sometimes cap $V$ purely for activation-memory reasons. At inference the picture flips: the vocabulary contributes a fixed $d \times V$ GEMM per decode step, which is cheap relative to the KV-cache reads in a large model, and the shorter sequence means fewer steps overall, so end-to-end latency for generating a fixed passage typically improves. A separate trap is evaluation: validation loss is not comparable across tokenizers, because the per-token loss is defined over a different unit of text, so comparisons must be normalized to bits per byte or bits per character.

Mathematical Formulation:
$P_{emb} = 2 V d$
$T = B / c(V)$
$c(V) \approx a + \beta \log V$
$C = 6 (N_{nv} + P_{emb}) T$
$M_{logits} = b \, T \, V \, p$
$V^{\star} \propto N_{nv}^{0.83}$

Where:

$P_{emb}$ is the parameter count of the input embedding plus output unembedding; the factor 2 drops to 1 when the two are tied.
$V$ is the vocabulary size and $d$ the model width; $N_{nv}$ is the non-vocabulary (attention plus MLP) parameter count.
$B$ is the raw text size in bytes, $c(V)$ the compression ratio in bytes per token, and $T$ the resulting token count; $a$ and $\beta$ are corpus- and language-dependent fit constants.
$C$ is approximate training FLOPs under the $6ND$ rule, which is why a larger $V$ both adds parameters and removes tokens.
$M_{logits}$ is the logits activation in bytes, with microbatch $b$ and precision $p$ (4 for fp32); $V^{\star}$ is the compute-optimal vocabulary, whose exponent near 0.83 means it grows more slowly than the rest of the model.

Two U-shaped curves of relative loss in bits per byte versus vocabulary size on a log-2 axis from 4k to 256k: the 300M non-embedding parameter curve bottoms out near 24k, and the 7B curve sits lower overall and bottoms out near 96k, showing the optimum shifting right with model size

Figure 2: Illustrative shape of the trade-off at fixed compute: loss measured in bits per byte is U-shaped in $V$ , and the minimum shifts right as non-embedding capacity grows, because a bigger body can afford to spend parameters on the vocabulary.

Property	Small BPE (32k)	Large BPE (128k-256k)	Byte level (256)
Bytes per token (English)	about 3.9	about 4.4 to 4.6	1.0
Embedding params at d=2048, untied	131M	525M to 1.05B	1M
fp32 logits per 8192-token sequence	1.05 GB	4.2 GB to 8.4 GB	8 MB
Average updates per row over 300B tokens	about 9M	about 1M to 2M	over 1B
Sequence length for a fixed document	baseline	0.85x to 0.89x	about 3.9x
Main weakness	High fertility on non-English text and code	Undertrained rare rows, huge logits activation	Long sequences, many decode steps
Best fit	Sub-1B on-device models, single-language domains	Multilingual frontier models with long contexts	Noisy input, character tasks, patch-based research

August 15, 2026

DL0080 Tokenization in LLMs

Explain tokenization in large language models.

Answer

Tokenization is the reversible mapping between raw text and the sequence of integer ids a language model actually consumes, drawn from a fixed vocabulary of size $V$ . Modern LLMs use subword vocabularies that sit between characters (short vocab, very long sequences) and words (huge vocab, unavoidable out-of-vocabulary gaps), and the dominant recipe is byte-level BPE: count adjacent symbol pairs over a training corpus, greedily merge the most frequent pair, and repeat until the vocabulary reaches its target size. Because the base alphabet is the 256 possible bytes rather than Unicode characters, any input is representable and there is no UNK token, at the cost of spending several tokens per character on scripts the merge table never learned. The tokenizer is frozen before pretraining and is effectively part of the architecture: it fixes the embedding matrix and output projection ( $2Vd$ parameters), it decides how many model passes a sentence costs, and it determines whether the model sees 1024 as one symbol or three. Most surprising LLM behaviors around arithmetic, non-English cost, and trailing whitespace trace back to this layer rather than to the transformer itself.

(1) Subword Is A Middle Ground: character-level models keep the vocabulary tiny but inflate sequence length, and attention is $O(N^2)$ in that length; word-level vocabularies need hundreds of thousands of entries and still miss rare words and typos.
(2) How BPE Is Trained And Applied: training produces an ordered merge list; encoding replays those merges greedily on each pretokenized chunk, so segmentation is deterministic and not a search over the best split.
(3) Pretokenization Comes First: in the GPT-style recipe a regex splits text on whitespace, punctuation, and digit groups before BPE runs, which is why the leading space belongs to the token (" the" and "the" are different ids) and why, with standard pretokenization, merges do not cross word boundaries; that constraint comes from the pretokenizer rather than from BPE itself, and implementations that skip or relax it (see SuperBPE below) do learn multi-word tokens.
(4) Vocabulary Size Is A Compute Knob: raising $V$ lowers fertility (tokens per word) so each sentence needs fewer forward passes, but it grows the embedding and softmax cost, which is why the field drifted from 32k to 128k vocabularies (Llama 2 → Llama 3) as models got larger.
(5) The Failure Modes Users Notice: digit grouping degrading arithmetic, non-English prompts costing two to three times more tokens for the same content, glitch tokens whose embeddings were barely trained, and special chat-template ids that must never be injectable from user text.

Bar chart of tokens per whitespace word for eight languages under one English-heavy tokenizer: English 1.15, French 1.40, Spanish 1.45, German 1.55, Russian 1.90, Turkish 2.15, Swahili 2.35, Hindi 2.70

Figure 1: Illustrative fertility gap: with a merge table dominated by English text, the same content in Hindi consumes roughly $2.3\times$ more tokens, so it costs more per request and reaches the context limit sooner.

Two implementation details matter more in production than the choice of merge algorithm. First, digit handling: tokenizers that merge arbitrary digit runs give the model inconsistent units for numbers, so 2024, 202, and 24 share no stable structure and column-wise arithmetic has to be learned per token; Llama 3 and several recent tokenizers instead force digits into groups of at most three, and some research tokenizers split every digit. Second, special tokens such as beginning-of-text and end-of-turn markers are inserted out of band by the chat template, not produced by BPE over user text; if a serving layer lets a user string encode into those ids, the model’s turn structure can be rewritten from inside a prompt. Related traps include unnormalized Unicode (NFC versus NFKC changes ids for accented text), tokenizer and checkpoint version skew, which silently shifts every embedding lookup, and untrained vocabulary rows that surface as glitch tokens the model cannot repeat back.

Mathematical Formulation:
$(a,b)^{*} = \arg\max_{(a,b)} c(a,b)$
$F = \frac{T}{W}$
$P_{\text{emb}} = 2 V d$
$C_{\text{word}} = F \cdot (C_{\text{body}} + 2 V d)$

Where:

$(a,b)^{*}$ is the pair chosen at each BPE training step, and $c(a,b)$ is its frequency as adjacent symbols in the corpus; the merge is appended to an ordered list that encoding later replays.
$F$ is fertility: $T$ tokens produced for $W$ whitespace words, measured per language and per corpus, so it is a property of the tokenizer and the text, not of the model.
$P_{\text{emb}}$ counts the input embedding plus the untied output projection for vocabulary $V$ and hidden width $d$ ; tying the two halves this term but couples input and output geometry.
$C_{\text{word}}$ is the compute per word of text, where $C_{\text{body}}$ is the per-token cost of everything except the vocabulary layers; the two $V$ -dependent effects pull in opposite directions, which is why an interior optimum in $V$ exists and grows with model size.

Two panels versus vocabulary size on a log axis: left panel shows tokens per word falling from about 1.72 at 4k vocab to 1.27 at 512k with diminishing returns, right panel shows relative compute per word dipping to a minimum near 33k vocabulary and rising to about 1.65 at 512k

Figure 2: Illustrative trade-off for a 1B-scale model with $d = 1024$ : shorter sequences from a larger vocabulary stop paying for themselves once the $2Vd$ output-layer cost rivals the transformer body, and the minimum shifts right as $d$ and depth grow.

Feature	Byte-level BPE	WordPiece	Unigram LM
How the vocab is learned	Greedily merge the most frequent adjacent pair, recording an ordered merge list	Merge the pair that most increases corpus likelihood under a unigram model	Start from a large candidate set and prune the tokens whose removal costs the least likelihood (EM)
How text is segmented	Replay merges in training order; deterministic and fast	Longest-match-first greedy scan, with `##` marking continuations	Viterbi search for the most probable segmentation, and it can sample splits for subword regularization
Unseen input	No UNK is possible; the 256-byte alphabet covers all of Unicode	Emits UNK for characters outside the alphabet unless byte fallback is added	Byte fallback is optional in SentencePiece and usually enabled
Where you see it	GPT-2 onward, Llama, Mistral, Qwen	BERT and the encoder family that followed it	T5, ALBERT, and many multilingual encoders

August 15, 2026

DL0079 Scaling Laws Size vs Data

How do scaling laws relate model size to training data requirements?

Answer

Neural scaling laws say that test loss falls as a power law in both parameter count $N$ and training tokens $D$ , plus an irreducible floor, so the two are not independent knobs: a fixed compute budget $C \approx 6ND$ forces a trade between them. Kaplan et al. (2020) fit the loss surface and concluded that most extra compute should go into parameters, which is why GPT-3 used 175B parameters on only about 300B tokens (roughly 1.7 tokens per parameter). Hoffmann et al. (2022, “Chinchilla”) re-ran the sweep with properly decayed learning-rate schedules over more than 400 runs and found the compute-optimal frontier scales both quantities at roughly the same rate, $N_{opt} \propto C^{0.5}$ and $D_{opt} \propto C^{0.5}$ , giving the well-known heuristic of about 20 tokens per parameter. Under that law a 70B model trained on 1.4T tokens beat a 280B model trained on 300B tokens at equal compute. The practical answer to the interview question is therefore: data requirements grow roughly linearly with model size along the compute-optimal frontier, and any deviation from that ratio should be justified by a constraint the loss law does not model, such as a finite corpus or the cost of serving.

(1) Joint Power Law: loss decomposes into an entropy floor plus one shrinking term per resource, $L = E + AN^{-\alpha} + BD^{-\beta}$ , so starving either resource leaves its term dominant and no amount of the other resource can fix it.
(2) Compute Couples The Two: with $C \approx 6ND$ FLOPs (2 for the forward pass and 4 for the backward pass per parameter per token), choosing $N$ fixes $D$ , which is why the question is always “what split”, never “how big”.
(3) IsoFLOP Minima Define The Frontier: sweeping $N$ at fixed $C$ traces a shallow U-shaped curve; the locus of minima across budgets is the compute-optimal frontier, and its slope in log-log space is the exponent you actually care about.
(4) Kaplan Versus Chinchilla: the earlier exponents ( $N_{opt} \propto C^{0.73}$ ) came from a fixed-length cosine schedule truncated early and from parameter counts that excluded embeddings, both of which penalize the long-token runs and bias the fit toward oversized models.
(5) Optimal Does Not Mean Correct: the law optimizes training loss per training FLOP; it says nothing about inference cost, so serving-heavy models are deliberately overtrained far past 20 tokens per parameter.

The empirical procedure behind these numbers is worth stating precisely, because interviewers probe it. You pick a grid of compute budgets, and for each budget you train several models of different widths and depths, always to the token count that exhausts exactly that budget, with the learning rate decayed to its minimum at the end of each run. Plotting final loss against $N$ for one budget yields an IsoFLOP curve whose minimum is flat enough that a factor of two in parameter count often costs under 1% in loss, which is exactly the slack engineers exploit when memory or latency constraints bite.

Four U-shaped IsoFLOP curves of loss versus parameter count on a log x-axis for compute budgets of 1e19, 1e20, 1e21 and 1e22 FLOPs, with minima marked and joined by a dashed frontier line moving right and down

Figure 1: Illustrative IsoFLOP curves from $L = E + AN^{-0.34} + BD^{-0.34}$ with $C = 6ND$ . Each budget’s minimum sits at $D/N \approx 20$ , moving from 0.29B parameters / 5.8B tokens at $10^{19}$ FLOPs to 9.1B parameters / 183B tokens at $10^{22}$ FLOPs, and the basins are shallow near the optimum.

Mathematical Formulation:
$L(N,D) = E + A N^{-\alpha} + B D^{-\beta}$
$C \approx 6ND$
$N_{opt} \propto C^{a}$
$D_{opt} \propto C^{b}$
$a = \frac{\beta}{\alpha + \beta}$
$b = \frac{\alpha}{\alpha + \beta}$
$D_{opt} / N_{opt} \approx 20$

Where:

$L(N,D)$ is expected test loss in nats per token, and $E$ is the irreducible term: the entropy of the data plus whatever the architecture can never represent, so no scaling drives loss to zero.
$N$ is the number of trainable parameters (embeddings included, which is where the Kaplan fit differed) and $D$ is the number of training tokens.
$A, B$ are fitted scale constants and $\alpha, \beta$ are the power-law exponents; Chinchilla’s fit put both near $0.3$ , meaning the two resources have comparable marginal value.
$C$ is training compute in FLOPs; the factor 6 comes from roughly 2 FLOPs per parameter per token forward and 4 backward, and it ignores attention’s quadratic term, which is minor while $d_{model}$ dominates sequence length.
$a, b$ are the frontier exponents obtained by minimizing $L$ subject to the compute constraint; $a + b = 1$ always holds, so if parameters take a larger share of compute growth, tokens must take a smaller one.
The ratio $D_{opt}/N_{opt}$ is constant only when $\alpha = \beta$ ; the “20 tokens per parameter” rule is that special case, and it drifts slowly with budget under asymmetric fits.

Log-log plot of compute-optimal parameter count and token count versus training compute from 1e19 to 1e25 FLOPs, with Chinchilla curves scaling as C to the 0.5 for both quantities and Kaplan curves scaling as C to the 0.73 for parameters and C to the 0.27 for tokens, diverging by more than an order of magnitude at high compute

Figure 2: Illustrative comparison of the two prescriptions, normalized to agree at $10^{19}$ FLOPs. Because the exponents differ ( $0.5$ versus $0.73$ ), the gap compounds: at $10^{24}$ FLOPs one recipe asks for 91B parameters on 1.8T tokens and the other for 1.3T parameters on 130B tokens, about a $14\times$ disagreement in both directions.

Modern frontier practice has moved past the compute-optimal point on purpose. If a model will serve billions of tokens, total lifetime FLOPs are dominated by inference, and a smaller model trained on far more data is cheaper end to end even though its training run is nominally suboptimal. Meta’s Llama 3 8B was trained on roughly 15T tokens, near 1,875 tokens per parameter, almost two orders of magnitude past the Chinchilla ratio, and the authors report the loss was still improving log-linearly at that point. The scaling law is still doing the work here; the objective being minimized has simply changed from training loss per training FLOP to quality per unit of deployment cost.

Regime	Tokens Per Parameter	What It Optimizes	When It Is The Right Call
Kaplan-style (parameter-heavy)	About 1 to 2	Loss under a truncated schedule and an embedding-free parameter count	Essentially never today; understand it to explain why GPT-3 looked the way it did
Chinchilla-optimal	About 20	Best loss for a fixed training budget	Research runs, ablations, and any model whose main cost is the training job
Deliberately overtrained	150 to 2,000 or more	Quality per unit of training plus inference cost	High-traffic serving, on-device targets, fixed memory or latency budgets
Data-constrained	Set by the corpus, not by compute	Loss given a finite unique-token pool with repeated epochs	Specialist domains (code, medical, low-resource languages) where fresh tokens run out

August 15, 2026