Tag: LLM

DL0109 Mixture-of-Experts MoE Routing

Explain mixture-of-experts (MoE) and compare Sparse Mixture-of-Experts (MoE) routing strategies, as used in models like Mixtral and DeepSeek-V3.

Answer

A sparse mixture-of-experts layer replaces one feed-forward block with $N$ parallel feed-forward blocks (the experts) plus a tiny linear router, and sends each token to only $k \ll N$ of them. This is conditional computation: parameter count grows with $N$ while the FLOPs per token grow only with $k$ , so Mixtral 8x7B holds 46.7B parameters but activates about 12.9B per token, and DeepSeek-V3 holds 671B while activating 37B. The interesting engineering is not the experts, which are ordinary MLPs, but the routing strategy, because a learned router is free to collapse onto a few favorite experts and then most of the capacity you paid for is never trained. The dominant strategy is token-choice top-k: every token picks its own $k$ experts, each expert has a fixed buffer set by a capacity factor, and tokens that arrive at a full expert are dropped through the residual connection. The main alternatives invert or remove that choice: expert-choice routing lets each expert select its top tokens (perfect load balance, but some tokens get no expert), hash layers assign tokens deterministically with no learned router at all, and BASE layers solve a global linear assignment per batch. The trade-off axis is always the same, namely how much balance you buy and what you pay for it in dropped tokens, extra loss terms, batch dependence, and all-to-all communication.

(1) Parameters Decouple From FLOPs: with $N=8, k=2$ a token touches 25% of the expert weights, so quality scales with total parameters while latency scales with active parameters.
(2) The Router Is One Matrix: a single $d \times N$ projection followed by top-k selection and a renormalized softmax over the selected logits, which is why gradients reach the router only through the chosen experts.
(3) Load Balancing Is The Central Problem: Switch-style training adds an auxiliary balance loss (typically $\alpha = 0.01$ ) and a router z-loss to keep logits small and the assignment spread out.
(4) Capacity Factor Controls Token Dropping: a buffer of $c \cdot T \cdot k / N$ slots per expert means $c$ near 1.0 is cheap but discards tokens whenever the router skews.
(5) Routing Families Differ In Who Chooses: token-choice, expert-choice, deterministic hashing, and global assignment sit on a spectrum from fully learned and imbalanced to fully balanced and inflexible.
(6) Serving Cost Is Memory, Not Compute: every expert must be resident in HBM even though each token uses $k$ of them, and expert parallelism adds two all-to-all collectives per MoE layer.

Two-panel bipartite diagram: on the left, token-choice top-2 routing where four tokens each select two experts, expert one is oversubscribed and two edges are dropped at capacity; on the right, expert-choice routing where each expert selects its top two tokens, giving perfect expert load but leaving one token with no expert

Figure 1: The two families differ in the direction of selection. Token-choice top-k guarantees every token gets $k$ experts but not that experts get equal load, so overflow is dropped at capacity; expert-choice guarantees equal expert load but not that every token is served.

Token-choice top-k stays the production default for autoregressive language models for one structural reason: it is a per-token function, so the routing decision for token $t$ does not depend on any other token in the batch. Expert-choice and BASE layers both rank or match tokens against each other, which makes the forward pass batch-dependent, breaks the causal guarantee during teacher-forced training, and cannot be reproduced at decode time when the batch is a single token. Hash layers show how much of MoE’s benefit comes from capacity rather than from clever routing, since a fixed hash of the token ID is perfectly balanced by construction and still recovers a large part of the gain, but it can never learn semantic specialization. In practice the fix for token-choice imbalance is not to abandon it: DeepSeek-V3 keeps token-choice top-8 over 256 fine-grained experts, adds one always-on shared expert to absorb common knowledge, restricts each token to experts on at most 4 nodes to bound communication, and replaces the auxiliary loss with a per-expert bias that is nudged up or down to equalize load.

Mathematical Formulation:
$h(x) = W_r x \in \mathbb{R}^{N}$
$\mathcal{T}(x) = \mathrm{TopK}(h(x), k)$
$g_i(x) = \frac{\exp(h_i)}{\sum_{j \in \mathcal{T}} \exp(h_j)}$
$y = x + \sum_{i \in \mathcal{T}} g_i(x)\, E_i(x)$
$\mathcal{L}_{\mathrm{bal}} = \alpha N \sum_{i=1}^{N} f_i P_i$
$C = \left\lceil \frac{c\, T\, k}{N} \right\rceil$

Where:

$x \in \mathbb{R}^{d}$ is the token hidden state and $y$ the MoE layer output, written with the residual path that a dropped token falls back to.
$W_r \in \mathbb{R}^{N \times d}$ is the router, $h(x)$ its logits, and $E_i$ the $i$ -th expert MLP; $N$ is the expert count and $k$ the number kept.
$\mathcal{T}(x)$ is the selected index set and $g_i$ the gate weight, renormalized over the selected logits only so the weights sum to 1.
$f_i$ is the fraction of tokens in the batch dispatched to expert $i$ (piecewise constant, no gradient) and $P_i$ the mean router probability for expert $i$ (differentiable), so $\mathcal{L}_{\mathrm{bal}}$ pushes probability mass away from overloaded experts.
$\alpha$ is the balance-loss coefficient, commonly $\alpha = 0.01$ ; the product $N \sum_i f_i P_i$ equals 1 under a perfectly uniform assignment and grows toward $N$ under total collapse.
$T$ is the tokens per device, $c$ the capacity factor (usually $1.0 \leq c \leq 1.25$ in training, larger at eval), and $C$ the per-expert buffer; any token beyond $C$ is dropped.

Line chart of the fraction of routed tokens dropped versus capacity factor from 1.0 to 2.0 for eight experts under uniform, mildly skewed, and heavily skewed router distributions; the uniform curve is flat at zero while the heavily skewed curve stays above twenty percent even at capacity factor two

Figure 2: Token dropping is a joint function of the capacity factor and the router’s skew. Under a balanced router, $c = 1.0$ drops nothing; under a collapsed router, even doubling capacity leaves a quarter of the assignments discarded, which is why the balance loss matters more than the buffer size.

Architecture diagram of a sparse mixture-of-experts layer showing the data flow from input token through a linear router and top-k softmax selection to two selected expert FFNs out of N, with the remaining N-k non-selected experts shown folded in a dashed box, then a weighted sum with gate weights and a residual connection producing the output

Figure 3: The full MoE layer in one picture. A single linear router produces logits over all $N$ experts, the top-k selection picks only $k$ of them, and the selected expert outputs are combined with renormalized gate weights and added back through the residual connection. Non-selected experts stay resident in memory but consume zero FLOPs for this token, which is why total parameters scale with $N$ while active compute scales with $k$ .

Property	Token-choice top-k	Expert-choice	Hash / global assignment
Who selects	Each token picks its k highest-scoring experts	Each expert picks its C highest-scoring tokens	A fixed hash of the token ID, or a linear-assignment solver over the batch
Load balance	Not guaranteed; needs an auxiliary loss or a bias correction	Exact by construction, every expert receives C tokens	Exact by construction, with no learned router to collapse
Failure mode	Overflow tokens are dropped to the residual and get no expert compute	A token can be selected by zero experts, and tokens compete across the batch	No semantic specialization (hash) or expensive, batch-coupled solves (BASE)
Per-token independence	Yes; identical decision at batch size 1 and at batch size 1M	No; ranking couples tokens, so causal decoding does not match training	Yes for hashing, no for assignment-based methods
Where it is used	GShard, Switch (k=1), Mixtral (k=2), DeepSeek-V3 (k=8), OLMoE	Encoder-style and vision MoE, and research settings with full-sequence visibility	Baselines and ablations that isolate capacity from learned specialization

August 17, 2026

DL0108 PPO vs DPO vs GRPO

Explain the architectural and mathematical differences between PPO, DPO (Direct Preference Optimization), and GRPO (Group Relative Policy Optimization).

Answer

All three optimize the same underlying target, maximize a preference-derived reward while staying close to a frozen reference policy, and they differ in how many networks must be resident, how the advantage is estimated, and whether the training data is sampled from the current policy. PPO is the full actor-critic loop used in InstructGPT-style RLHF: it keeps four networks (trained policy, frozen reference, frozen reward model, trained critic), samples rollouts on-policy, and updates with a clipped importance ratio against token-level GAE advantages produced by the critic. DPO deletes the RL loop entirely by inverting the closed-form solution of the KL-constrained objective: the implied reward is $\beta \log(\pi_\theta / \pi_{ref})$ , the partition function cancels inside a Bradley-Terry pairwise likelihood, and what remains is a supervised binary-classification loss on fixed $(y_w, y_l)$ pairs with only two networks and no sampling. GRPO keeps online sampling and the clipped surrogate but removes the critic: for each prompt it draws a group of $G$ completions, uses the group’s mean reward as the baseline, and z-scores the rewards to get one scalar advantage that is broadcast to every token of its completion. So the axis is not “better versus worse” but which piece of machinery you are willing to pay for: PPO buys fine-grained credit assignment with a learned value function, DPO buys simplicity by giving up exploration, and GRPO buys on-policy learning with a Monte Carlo baseline.

(1) Networks Resident: PPO needs policy, reference, reward model, and critic; GRPO drops the critic; DPO drops both the critic and the reward model.
(2) Advantage Estimation: PPO uses GAE over a learned $V_\psi$ , GRPO uses a group-relative z-score of sequence rewards, and DPO never forms an advantage at all, only a reward margin between two responses.
(3) On-Policy Versus Offline: PPO and GRPO resample from the current policy every iteration, so the clip and importance ratio are meaningful; DPO trains on a static dataset and is therefore exposed to distribution shift.
(4) KL Control: PPO and GRPO add an explicit KL penalty (GRPO commonly uses the low-variance k3 estimator), while DPO’s KL constraint is baked into the log-ratio parameterization and controlled solely by $\beta$ .
(5) Credit Granularity: only PPO assigns different advantages to different tokens; GRPO gives every token in a completion the same scalar, and DPO gives a whole-sequence gradient.
(6) Where Each Fits: verifiable rewards from a checker or unit test favor GRPO, a cheap single-pass alignment on collected preferences favors DPO, and a nuanced learned reward model with long generations favors PPO.

Three side-by-side data-flow panels: PPO with prompt, trained policy, rollout, frozen reward model plus trained critic, and GAE advantage; DPO with a static preference pair scored by the trained policy and frozen reference into a logistic loss; GRPO with a prompt, trained policy, a group of G rollouts, a frozen verifier, and a group z-scored advantage

Figure 1: The three objectives differ mainly in what sits between the policy and the loss: PPO inserts a reward model plus a trained critic, GRPO replaces the critic with a group of sampled rollouts, and DPO removes the sampling stage so the frozen reference is the only extra network.

DPO’s derivation is what makes the contrast precise. The KL-constrained bandit objective has the closed-form optimum $\pi^*(y \mid x) \propto \pi_{ref}(y \mid x) \exp(r(x,y)/\beta)$ ; solving for the reward gives $r = \beta \log(\pi^*/\pi_{ref}) + \beta \log Z(x)$ , and because the Bradley-Terry likelihood depends only on reward differences for the same prompt, the intractable partition function $Z(x)$ cancels. The reward model therefore never has to be materialized, since its optimal policy is the object you were going to train anyway. The price is that this equivalence is exact only when the preference pairs come from $\pi_{ref}$ ; on off-policy pairs the loss can raise the margin while pushing down the probability of the chosen response as well, the failure mode usually called likelihood displacement. GRPO takes the opposite trade, keeping the on-policy ratio and clip but swapping the critic’s learned variance reduction for a Monte Carlo baseline over $G$ samples of the same prompt, which is cheap and well-behaved when the reward is a verifiable 0/1 signal from a math checker or unit test. That is exactly the regime DeepSeek used when introducing GRPO for DeepSeekMath and then scaling it for R1.

Mathematical Formulation:
$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{old}(a_t \mid s_t)}$
$\tilde r_t = \mathrm{clip}(r_t, 1 - \epsilon, 1 + \epsilon)$
$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t A_t,\ \tilde r_t A_t)\right]$
$A_t = \sum_{l \geq 0} (\gamma \lambda)^l \delta_{t+l}$
$\delta_t = R_t + \gamma V_\psi(s_{t+1}) - V_\psi(s_t)$
$\hat r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{ref}(y \mid x)}$
$\Delta = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)$
$\mathcal{L}_{DPO} = -\mathbb{E}\left[\log \sigma(\Delta)\right]$
$A_i = \frac{R_i - \mathrm{mean}(R_{1:G})}{\mathrm{std}(R_{1:G})}$
$\mathcal{L}_{GRPO} = -\mathbb{E}\left[\min(r_i A_i,\ \tilde r_i A_i)\right]$
$\qquad + \beta \, \mathbb{D}_{KL}\left[\pi_\theta \,\|\, \pi_{ref}\right]$

Where:

$\pi_\theta$ is the trained policy, $\pi_{ref}$ the frozen reference (normally the SFT checkpoint), and $\pi_{old}$ the policy that generated the current batch of rollouts.
$s_t$ is the prompt-plus-prefix state and $a_t$ the token emitted at position $t$ ; $R_t$ is the per-step reward, which in RLHF is usually nonzero only at the final token.
$V_\psi$ is the learned critic, $\delta_t$ the TD residual, and $A_t$ the GAE advantage with discount $\gamma$ and trace decay $\lambda$ .
$\epsilon$ is the clip half-width (typically 0.1 to 0.2) and $\beta$ is the KL coefficient in PPO and GRPO, or the implicit-reward temperature in DPO (typically 0.01 to 0.5).
$y_w$ and $y_l$ are the preferred and rejected responses for prompt $x$ , $\sigma$ is the logistic function, and $\hat r_\theta$ is the reward implied by the policy itself.
$i \in \{1, \ldots, G\}$ indexes the group of completions sampled per prompt (commonly $G = 8$ to $64$ ), $R_i$ is its sequence-level reward, and $A_i$ is shared by every token of completion $i$ .
Required initial condition in all three: $\pi_\theta$ is initialized from $\pi_{ref}$ , otherwise the KL term and the log-ratio reward have no meaningful anchor.

Stacked bar chart of resident bf16 weight memory for a 7B policy: PPO 56 GB with 28 GB trained, GRPO with a reward model 42 GB with 14 GB trained, GRPO with a rule verifier 28 GB, and DPO 28 GB with 14 GB trained

Figure 2: Counting only weights at 7B scale in bf16, PPO holds four copies and trains two of them, so its Adam state is roughly double GRPO’s; a rule-based verifier removes the reward model entirely, making GRPO as light as DPO in weight memory while still sampling online.

Property	PPO	DPO	GRPO
Networks resident	Policy, reference, reward model, critic (two trained)	Policy and frozen reference (one trained)	Policy, reference, reward model or verifier (one trained)
Data source	Fresh on-policy rollouts from prompts	Static offline preference pairs, no generation	Groups of G on-policy rollouts per prompt
Baseline for the gradient	Learned value function, GAE per token	The rejected response acts as the baseline	Group mean reward, normalized by group std
Credit granularity	Per token, values differ along the sequence	Whole sequence, one margin per pair	One scalar per completion, broadcast to all its tokens
KL control	Explicit penalty or reward shaping, often adaptive	Implicit in the log-ratio, tuned only through beta	Explicit term with the k3 estimator, sometimes dropped
Cost per update	Highest: generation plus four forward passes plus critic training	Lowest: two forward passes on cached text	Generation dominates, G completions per prompt
Main failure mode	Critic instability and reward hacking, many coupled hyperparameters	Off-policy shift and likelihood displacement on unseen responses	Degenerate groups with zero reward variance, length and difficulty bias

August 17, 2026

DL0107 Multi-head Latent Attention MLA

Explain Multi-head Latent Attention (MLA), as used in DeepSeek-V2 and DeepSeek-V3.

Answer

Multi-head Latent Attention (MLA) attacks the KV-cache bottleneck of autoregressive decoding by low-rank joint compression of keys and values instead of by deleting KV heads. Each token is projected down to a single shared latent vector $c_t^{KV}$ of dimension $d_c = 512$ , and that latent is the only thing cached; per-head keys and values are reconstructed on the fly by up-projection matrices $W_i^{UK}$ and $W_i^{UV}$ , so every head still gets its own distinct K and V, unlike GQA and MQA which force heads to share. Because RoPE does not commute with the up-projection, MLA adds a small decoupled RoPE key of dimension $d_h^R = 64$ that is shared across heads and cached directly, giving a total cache of $512 + 64 = 576$ values per token per layer against $2 \cdot 128 \cdot 128 = 32{,}768$ for the equivalent MHA configuration, a roughly 57x reduction. The reason this is not simply lossy compression is the absorption trick: at inference the up-projections fold into the query and output projections, so the model never materializes the full keys and values, and the attention score becomes a bilinear form on the two latents. The result reported in DeepSeek-V2 is unusual for a cache-reduction technique, since MLA matched or beat full MHA on benchmarks while caching about 1/57 as much, which is why DeepSeek-V3 kept it unchanged at 61 layers and 128 heads.

(1) Joint Low-Rank KV Compression: one down-projection produces a single latent per token that serves all heads and both K and V, so cache cost is $(d_c + d_h^R) \cdot l$ rather than $2 n_h d_h l$ .
(2) Head Diversity Is Preserved: per-head up-projections mean MLA reduces cache without reducing the number of effective K/V heads, which is the quality cost GQA and MQA pay.
(3) Decoupled RoPE Path: position information travels in a separate small key dimension because applying RoPE inside the compressed path would make the up-projection position-dependent and destroy absorption.
(4) Absorption At Inference: $W_i^{UK}$ folds into $W_i^{UQ}$ and $W_i^{UV}$ folds into $W^O$ , so decoding reads only the cached latent and never reconstructs full K or V tensors.
(5) Query Compression Is A Separate Win: queries are also routed through a latent of dimension 1536, which cuts training activation memory but has no effect on the KV cache.
(6) Memory-Bandwidth Trade For FLOPs: the absorbed score is a dot product over 576 dimensions instead of 128, so MLA raises arithmetic intensity, exactly the right direction for memory-bound batch decoding.

MLA architecture diagram: the hidden state is projected into a query latent, a shared 512-dimensional KV latent that is cached, and a 64-dimensional decoupled RoPE key that is also cached; per-head keys and values are reconstructed by up-projections and attention runs over a 192-dimensional concatenated head

Figure 1: Only the two boxes inside the dashed region enter the cache: the 512-dimensional KV latent and the 64-dimensional decoupled RoPE key. Per-head keys and values are reconstructed from the latent, and at inference those up-projections are absorbed so they are never computed explicitly.

The decoupled RoPE dimension is the part candidates most often miss, and it follows from a single algebraic fact. The score between query $t$ and key $j$ in the compressed path is $(W_i^{UQ} c_t^{Q})^\top (W_i^{UK} c_j^{KV})$ , which lets you precompute the constant matrix $M_i = (W_i^{UQ})^\top W_i^{UK}$ . If RoPE were applied to $k_j^C$ , that matrix would become $(W_i^{UQ})^\top R_{t-j} W_i^{UK}$ , which depends on the relative position and therefore differs for every cached token, so nothing can be precomputed and every prefix key would have to be rebuilt at each step. MLA sidesteps this by carrying position in a separate 64-dimensional key that is cached in rotated form and shared by all heads, so the query concatenates a 128-dimensional content part with a 64-dimensional positional part, giving an effective head dimension of 192 during training. A practical consequence is that the value head dimension (128) differs from the query and key dimension (192), which is why MLA needs purpose-built kernels such as FlashMLA rather than a stock FlashAttention call.

Mathematical Formulation:
$c_t^{KV} = W^{DKV} h_t$
$k_{t,i}^C = W_i^{UK} c_t^{KV}$
$v_{t,i}^C = W_i^{UV} c_t^{KV}$
$k_t^R = \mathrm{RoPE}(W^{KR} h_t)$
$q_{t,i} = [\,q_{t,i}^C ; q_{t,i}^R\,]$
$k_{t,i} = [\,k_{t,i}^C ; k_t^R\,]$
$(q_{t,i}^C)^\top k_{j,i}^C = (c_t^{Q})^\top M_i c_j^{KV}$
$M_i = (W_i^{UQ})^\top W_i^{UK}$
$d_c + d_h^R = 512 + 64 = 576$

Where:

$c_t^{KV} \in \mathbb{R}^{d_c}$ is the cached KV latent and $k_t^R \in \mathbb{R}^{d_h^R}$ the cached decoupled RoPE key; together they are the entire per-token cache.
$h_t \in \mathbb{R}^{d}$ is the layer input ( $d = 7168$ in DeepSeek-V3), and $c_t^{Q}$ is the query latent of dimension 1536.
$t$ indexes the current token, $j$ ranges over cached prefix positions with $j \leq t$ , and $i \in \{1,\ldots,n_h\}$ indexes the $n_h = 128$ heads.
$d_h = 128$ is the content head dimension, $d_h^R = 64$ the positional dimension, so the concatenated query and key have dimension 192 while values keep 128.
$W^{DKV}, W^{DQ}$ are down-projections, $W_i^{UK}, W_i^{UV}, W_i^{UQ}$ up-projections, $W^{KR}, W^{QR}$ the RoPE-path projections, and $W^O$ the output projection.
$M_i$ is the absorbed score matrix, valid only because no rotation sits between the two up-projections; $\mathrm{RoPE}$ applies the rotary transform, and $[\,\cdot;\cdot\,]$ denotes concatenation.

Log-scale bar chart of KV cache per token per layer: MHA with 128 KV heads at 32768 values and 524 GB at 128K context, GQA with 8 groups at 2048 values and 32.8 GB, MLA at 576 values and 9.2 GB, MQA at 256 values and 4.1 GB

Figure 2: Totals assume 61 layers, bf16, and a 128K-token context. MLA lands between MQA and 8-group GQA on cache size, but reaches that point by compressing rather than by removing KV heads, which is why it does not carry their quality penalty.

Property	MLA	GQA (8 groups)	MQA
What is cached	One 512-dim latent plus a 64-dim RoPE key	Full K and V for 8 shared KV heads	Full K and V for a single KV head
Values per token per layer	576	2,048	256
Effective K/V per head	Distinct for all 128 heads, reconstructed by up-projection	16 query heads share one K and V	All 128 query heads share one K and V
Decode compute per key	Dot product over 576 dims after absorption, higher FLOPs	128 dims, plus broadcast of shared heads	128 dims, lowest FLOPs
Kernel and tooling support	Needs custom kernels (FlashMLA) because query/key is 192 and value is 128	Supported by every stock attention kernel	Supported by every stock attention kernel
Tensor-parallel behavior	Latent is shared by all heads, so it is replicated per rank	KV heads shard cleanly across ranks	Single KV head must be replicated
Quality relative to MHA	Matched or slightly better in the DeepSeek-V2 ablations	Small but measurable degradation	Largest degradation, especially on long context

August 17, 2026

DL0095 Zero-Shot vs Few-Shot

How do zero-shot and few-shot prompting differ, and when does few-shot prompting beat fine-tuning?

Answer

Both are inference-time conditioning: the weights $\theta$ never change, only the tokens placed before the query. Zero-shot prompting gives an instruction and the input, so the model must map the task description onto a behavior it already learned during pretraining or instruction tuning. Few-shot prompting (in-context learning) prepends $K$ solved demonstrations $(x_i, y_i)$ that pin down the output format, the label space, and the input distribution before the real query arrives, which is why it helps most on tasks with an unusual schema or a strict output contract. In the original GPT-3 study, TriviaQA accuracy for the 175B model moved 64.3% → 71.2% going from zero-shot to 64-shot, and the gap between the two settings shrinks as models get better instruction tuning. Few-shot prompting beats fine-tuning when labeled data is scarce (roughly tens of examples), the task spec is still changing weekly, one frozen base must serve many tasks, or no training infrastructure exists; fine-tuning wins once you have thousands of clean labels, need the lowest possible per-request cost and latency, or need behavior that no prompt reliably elicits. The decision is mostly economics plus label count, not model quality: demonstrations are paid for on every request, while a fine-tune is a one-time cost amortized over traffic.

(1) Same Weights, Different Context: neither method computes a gradient; few-shot differs from zero-shot only by the demonstration block $S_K$ inserted into the prompt.
(2) Demonstrations Teach Format, Not Mostly Facts: the label space, input distribution, and output template drive most of the gain, which is why even partly incorrect labels in the exemplars often still work.
(3) Cost Is Recurring: $K$ exemplars add $K T_{ex}$ prefill tokens per call, inflating time-to-first-token and input spend on every request forever.
(4) Fine-Tuning Trades Setup For Marginal Cost: a LoRA run costs money once and then serves a short prompt, so it wins above a traffic break-even point.
(5) Data Volume Decides The Ceiling: with a handful of labels in-context learning is usually ahead; with thousands, parameter updates reach accuracy no prompt matches.

The mechanism is worth stating precisely because it predicts the failure modes. Demonstrations act as a task locator rather than a training set: replacing gold labels with random ones from the correct label set degrades few-shot accuracy far less than removing the labels entirely, which shows the exemplars are mostly specifying which distribution to condition on. That same conditioning creates strong biases: majority-label bias (a class over-represented in the exemplars gets over-predicted), recency bias (the last exemplar dominates), and ordering sensitivity that can swing accuracy by tens of points across permutations of the same $K$ examples. Calibration on a content-free input and stratified, order-shuffled exemplar selection recover most of that variance, and any prompt tuned on a large validation set is no longer honestly “few-shot” because the selection itself consumed labels.

Line chart of task accuracy against the number of in-context examples for an 8B and a 70B frozen model, with two dashed horizontal reference lines for an 8B model fine-tuned on 100 and on 5000 labels

Figure 1: Illustrative shot-scaling behavior: most of the in-context gain arrives by $K = 4$ and flattens after $K = 16$ , a model fine-tuned on only 100 labels sits near the few-shot curve, and 5,000 labels put the fine-tuned small model above the frozen large model.

Mathematical Formulation:
$p_{\theta}(y \mid I, x)$
$p_{\theta}(y \mid I, S_K, x)$
$S_K = ((x_1, y_1), \ldots, (x_K, y_K))$
$T_{ctx} = T_I + K T_{ex} + T_x$
$40 + 32 \times 60 + 30 = 1990$
$R^{*} = \frac{C_{ft}}{K T_{ex} c_{in}}$

Where:

$y$ is the generated answer, $x$ the query, and $I$ the instruction text; the first two lines are the zero-shot and few-shot predictive distributions under identical $\theta$ .
$S_K$ is the demonstration block and $K$ the shot count, with $K = 0$ recovering the zero-shot case exactly.
$T_I$ , $T_{ex}$ , and $T_x$ are token lengths of the instruction, one exemplar, and the query; $T_{ctx}$ is the prefill length that sets time-to-first-token.
The numeric line instantiates $T_{ctx}$ for $K = 32$ exemplars of 60 tokens each, giving 1,990 prompt tokens against 70 for the zero-shot prompt.
$c_{in}$ is the price per input token, $C_{ft}$ the one-time fine-tuning cost, and $R^{*}$ the break-even request volume above which the fine-tune is cheaper; it assumes both options serve the same output length and per-token price.

Plugging in real numbers makes the trade-off concrete. At $c_{in} = \$0.30$ per million input tokens, the 1,920 extra tokens from 32 exemplars cost about $\$0.00058$ per request, so a $\$60$ LoRA job pays for itself after roughly 104,000 requests. Prompt caching changes that arithmetic sharply: because the exemplar block is a fixed prefix, cached reads billed near 10% of the input rate push the break-even beyond a million requests and also cut the prefill latency penalty. That is why the honest answer to “few-shot or fine-tune” depends on traffic volume, prefix stability, and whether your serving stack caches, not on which technique sounds more advanced.

Line chart of cumulative extra cost in dollars against requests served, comparing few-shot prompting with and without prompt caching against a flat one-time fine-tuning cost, with the crossover marked near 104 thousand requests

Figure 2: Illustrative cost crossover: the few-shot line grows linearly with traffic because the exemplar prefix is re-billed per call, the fine-tune is a flat one-time charge, and prompt caching flattens the few-shot slope by roughly an order of magnitude.

Property	Zero-shot	Few-shot (in-context)	Fine-tuning (LoRA)
Labeled examples needed	None, only a clear instruction	Typically 4 to 64, plus a small set for prompt selection	Hundreds to tens of thousands
Where task knowledge lives	Pretraining and instruction tuning only	In the prompt, re-sent or cached per request	In adapter weights, prompt stays short
Prompt tokens per request	70 in the worked example	1,990 at K=32, so higher time-to-first-token	70, same as zero-shot
Iteration speed	Seconds, edit the instruction	Seconds, swap or reorder exemplars	Hours per run plus eval and deploy
Main failure mode	Wrong output schema, task misread	Majority-label and recency bias, ordering variance, context limits	Overfits small or noisy label sets, forgets off-task behavior
Multi-task serving	One model, one endpoint	One model, per-task prompt template	One adapter per task over a frozen base

August 16, 2026

DL0094 Evaluating an LLM

How do you evaluate an LLM?

Answer

There is no single number, so I treat evaluation as a stack of layers chosen to match how the model will actually be used: intrinsic likelihood (perplexity or bits per byte) at the bottom, static capability benchmarks above it, automated graders such as unit tests and LLM-as-judge, human pairwise preference, and finally an online A/B test on the real product metric. Each layer up the stack is closer to user value and more expensive, so cheap layers act as fast regression gates and expensive layers make ship decisions. The practical work is defining the task metric first, then building a private evaluation set drawn from production traffic, because public leaderboards measure a distribution that is rarely yours. Three things decide whether an evaluation is worth anything: contamination control, judge calibration against human labels, and error bars. Most reported “improvements” of one or two points on a 1000-item benchmark are inside the noise band and would not survive a paired significance test.

(1) Start From The Deployment Task: a RAG assistant is scored on groundedness, citation precision, and refusal correctness, a coding model on pass@k against hidden tests, an agent on end-to-end task completion; a generic leaderboard score answers none of these.
(2) Intrinsic Versus Extrinsic: perplexity is cheap and useful for pretraining and quantization regressions, but it is not comparable across tokenizers and correlates weakly with instruction-following quality.
(3) Static Benchmarks Decay: MMLU-Pro, GPQA, MATH, HumanEval, and SWE-bench Verified are reproducible and cheap, but they saturate and they leak into pretraining corpora, so contamination checks (n-gram overlap, held-out variants, release-date filtering) are mandatory.
(4) Automated Graders Scale, With Bias: the loop is sample responses → grade with a rubric or unit tests → aggregate with intervals; a judge model shows position, verbosity, and self-preference bias, so swap the answer order and calibrate agreement with humans before trusting it.
(5) Human Preference Is The Open-Ended Gold Standard: pairwise votes fitted with a Bradley-Terry (Elo) model give a single latent quality score, at the cost of slow turnaround and annotator disagreement.
(6) Report Uncertainty, Safety, And Cost: every score needs a confidence interval and a paired test against the baseline, plus a separate axis for jailbreak resistance, toxicity, PII leakage, and the p95 latency and cost per request that decide whether the win is affordable.

Five stacked layers of LLM evaluation from intrinsic perplexity at the bottom through static benchmarks, automated graders, human preference, and production A/B testing at the top, with cost and fidelity increasing upward

Figure 1: The evaluation stack. Cheap layers at the bottom run on every commit as regression gates; the expensive layers at the top are the only ones that measure user value, so a release decision needs evidence from both ends.

Two failure modes dominate real evaluation work. The first is contamination: a model that has memorized a benchmark scores high without generalizing, which you detect by rewriting items into paraphrases or numeric variants and watching the score collapse, or by using benchmarks whose items post-date the training cutoff. The second is judge miscalibration: an LLM judge that prefers longer, more confidently worded answers will happily rank a verbose model above a correct one, so I always measure judge-human agreement (Cohen’s kappa above roughly 0.6 before the judge is allowed to gate a release) and control for response length. On top of that, generation is stochastic, so a benchmark run at temperature 0.7 has run-to-run variance of its own; fix seeds and decoding parameters, or report the mean over several runs. Finally, offline wins must be confirmed online, because the production metric that matters (resolution rate, edit distance to the accepted answer, escalation rate) frequently moves in the opposite direction from a benchmark score.

Property	Static benchmark	LLM-as-judge	Human preference
What it measures	Closed-form correctness on a fixed item set	Rubric compliance and pairwise win rate on open-ended prompts	Latent quality as perceived by real users or experts
Turnaround per model	Minutes to hours, fully automated	Hours, bounded by judge API throughput and cost	Days to weeks, bounded by annotator supply
Main failure mode	Contamination and saturation; near-zero headroom on old suites	Position, verbosity, and self-preference bias	Annotator disagreement and prompt-mix drift
Reproducible	Yes, if decoding and prompt template are pinned	Only against a pinned judge version; judge upgrades shift scores	No, each round is a fresh sample of raters
Best used for	CI regression gates and capability screening	Scaling open-ended comparison between candidate checkpoints	Final ranking and calibrating the automated judge

Mathematical Formulation:
$\mathrm{PPL} = \exp\left(-\frac{1}{T}\sum_{t=1}^{T}\log p(x_t \mid x_{1:t-1})\right)$
$\mathrm{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$
$P(a \succ b) = \frac{1}{1 + 10^{(R_b - R_a)/400}}$
$\mathrm{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{N}}$

Where:

$\mathrm{PPL}$ is perplexity over a held-out sequence $x_1,\ldots,x_T$ , with $t$ indexing tokens; it depends on the tokenizer, so bits per byte is the cross-model comparable form.
$n$ is the number of samples drawn per coding problem, $c$ the number that pass the hidden tests, and $k$ the budget of attempts scored; this is the unbiased pass@k estimator, averaged over problems.
$P(a \succ b)$ is the modelled probability that response $a$ beats $b$ , and $R_a, R_b$ are the fitted Bradley-Terry ratings on the Elo scale, where a 100-point gap implies about a 64% win rate.
$\hat{p}$ is the observed accuracy on $N$ independent items and $\mathrm{SE}$ its standard error; the 95% interval is $\hat{p} \pm 1.96\,\mathrm{SE}$ , which for $\hat{p}=0.7$ and $N=1000$ is roughly $\pm 2.8$ points.
Required condition: items must be independent and not contaminated; when two models are scored on the same items, use a paired test (McNemar or a paired bootstrap) rather than comparing two independent intervals.

Bar chart of three model accuracies of 71.2, 73.5 and 78.9 percent on a 1000-item benchmark with 95 percent confidence interval error bars of about 2.8 points, showing the first two intervals overlapping

Figure 2: Accuracy with 95% intervals on a 1000-item benchmark. The 2.3-point gap between the first two models sits well inside the noise band, so an unpaired claim of improvement is unsupported; only the third model separates cleanly.

August 16, 2026

DL0093 Distillation vs Serving

When is distilling a large model into a small one a better strategy than serving the large model directly, for example behind a high-traffic assistant API?

Answer

Distillation wins when the recurring cost of inference, multiplied by sustained traffic, dominates the one-time cost of teacher labeling plus student training, and when the task distribution is narrow enough that the teacher’s surplus capability is never exercised. The decision is a break-even calculation: if the teacher costs $2.00 per 1k requests and an 8B student costs $0.25, and distillation costs $1,400 once (500k teacher-labeled prompts at $1,000 plus 200 GPU-hours at $400), the crossover is 800k requests, roughly 18 hours at 12 QPS. Two other conditions make distillation the only option rather than the cheapest one: a hard latency SLO that no batching strategy can meet with a 70B decoder, and a fixed memory envelope such as a phone NPU or a single 24 GB GPU, where quantization alone cannot close a 10x parameter gap. Serve the large model directly when traffic is low, when the input distribution is open-ended, or when the win is available more cheaply through quantization, speculative decoding, prompt caching, or a small-to-large cascade, since those preserve behavior and skip a new eval and re-qualification cycle. Distillation also carries a maintenance tax: every teacher upgrade reopens the labeling job, and the student inherits the teacher’s errors without the capacity to recover from them.

(1) Break-Even Volume: divide the one-time distillation cost by the per-request savings; below that volume, serving the teacher is strictly cheaper, and the payback period, not the compression ratio, is the number that decides the project.
(2) Latency Floor, Not Throughput: autoregressive decoding is memory-bandwidth bound and sequential, so per-token latency scales with parameter count and cannot be batched away; shrinking the model is one of the few levers that moves p99 rather than QPS per GPU.
(3) Task Breadth Decides Feasibility: on a stable, narrow task a student 8x smaller typically recovers 98-99% of teacher quality, while an open-ended assistant distribution loses many points because the teacher’s broad competence is exactly what is being compressed away.
(4) Hard Deployment Constraints: on-device, air-gapped, or offline targets impose a memory and power budget that no serving trick satisfies, so distillation becomes a requirement rather than an optimization.
(5) Cheaper Alternatives First: speculative decoding is distribution-preserving and needs no re-qualification, quantization is a one-line config change, and a router that sends easy traffic to a small model keeps the teacher available for the tail; distillation is the only lever that actually reduces the footprint, and it is the most expensive to maintain.

Mathematical Formulation:
$Q^{*} = \frac{C_{0}}{c_{L} - c_{S}}$
$C_{0} = N c_{L} + C_{\text{train}}$
$\mathcal{L} = (1-\lambda)\mathcal{L}_{\text{CE}} + \lambda \tau^{2} \mathcal{L}_{\text{KD}}$
$\mathcal{L}_{\text{KD}} = \mathrm{KL}(p^{T}_{\tau} \| p^{S}_{\tau})$

Where:

$Q^{*}$ is the break-even request volume, the point at which cumulative distilled-serving cost equals cumulative teacher-serving cost.
$c_{L}$ and $c_{S}$ are the marginal serving costs per request for the large teacher and the small student; the difference $c_{L} - c_{S}$ must be positive for distillation to ever pay back.
$C_{0}$ is the one-time cost, composed of teacher inference over $N$ unlabeled prompts plus student training $C_{\text{train}}$ ; engineering and eval time belong here too and usually exceed the compute.
$\mathcal{L}$ is the student objective, mixing hard-label cross-entropy $\mathcal{L}_{\text{CE}}$ with the distillation term $\mathcal{L}_{\text{KD}}$ under mixing weight $\lambda \in [0,1]$ .
$\tau$ is the softmax temperature applied to both teacher and student logits, and the $\tau^{2}$ factor rescales the gradient so $\lambda$ stays meaningful as $\tau$ changes; $p^{T}_{\tau}$ and $p^{S}_{\tau}$ are the tempered teacher and student distributions.
Required condition for the whole strategy: the student must also satisfy the latency and memory constraints, $t_{S} \leq \text{SLO}$ and $m_{S} \leq m_{\max}$ , which no amount of traffic volume can buy.

Line chart of cumulative serving cost in dollars against cumulative requests in millions: serving the 70B teacher rises linearly at 2 dollars per thousand requests from the origin, while the distillation path starts at 1400 dollars and rises at 0.25 dollars per thousand, crossing at 800 thousand requests with the savings region shaded beyond the crossing

Figure 1: Illustrative cost curves: the distilled path carries a $1.4k fixed intercept but an eight-times shallower slope, so the break-even sits at 800k requests and every request after that is pure savings.

The break-even math is only valid if the student actually holds quality on the traffic you will receive, and that depends far more on task breadth than on the compression ratio. A single-intent classifier, a fixed extraction schema, or one product domain compresses well because the teacher’s decision boundary in that slice is simple and you can generate unlimited soft targets for it; an open-ended assistant does not, because the surplus capacity is the product. When breadth is the problem, a cascade is usually the better answer than a bigger student: route every request to the distilled model, escalate the tail flagged by a confidence or verifier signal to the teacher, and you capture most of the cost win while keeping the quality ceiling. Whatever you choose, budget for the maintenance tax, since a teacher upgrade invalidates the student, drifting inputs silently move outside the distilled distribution, and you now own two eval suites instead of one.

Semi-log chart of percentage of teacher quality retained against student parameter count from 0.5B to 70B, showing a narrow single-domain task staying above the 98 percent floor from about 1B upward while an open-ended assistant task falls to roughly 71 percent at 1B and only reaches the floor near 20B

Figure 2: Illustrative quality retention: on a narrow task a 1B student already clears a 98% floor, while the same student on open-ended traffic gives up roughly 27 points, which is why breadth, not the compression factor, gates the decision.

Property	Distill to a small student	Serve large + quantization	Serve large + speculative decoding
Memory footprint	8-20x smaller, can drop from 8 GPUs to 1 or to a phone	2-4x smaller weights, KV cache mostly unchanged	Slightly larger, the drafter is an extra resident model
Quality guarantee	None, must be re-measured per task; tail regressions are common	Small measurable drift, worse at 4-bit and on long context	Distribution-preserving, output matches the teacher exactly
Typical latency gain	4-6x on per-token latency, the largest available lever	1.5-2.5x, bounded by bandwidth saved on weights	2-3x at low batch, degrades as batch size grows
One-time cost	Label generation, training runs, a new eval harness	Calibration pass plus a quality spot-check	Serving-stack work, no retraining of the target model
Cost when the teacher is upgraded	Full re-distillation and re-qualification	Re-run calibration only	Swap the target, optionally refresh the drafter
Wins when	High sustained traffic, narrow task, or a fixed device budget	You need a fast, low-risk cost cut on broad traffic	Latency matters, batches are small, and quality cannot move

August 16, 2026

DL0092 Reward Hacking in RLHF

What is reward hacking, and how does it manifest in RLHF?

Answer

Reward hacking is what happens when a policy maximizes the measured reward instead of the intended objective, exploiting the gap between the two. In RLHF that gap is concrete and unavoidable: the reward model $r_{\phi}$ is a finite-capacity network fit to a fixed set of human preference pairs, so it equals the true human utility $r^{*}$ plus an error term $\epsilon$ . PPO does not distinguish the two, and since it actively searches for the highest-scoring completions, it lands precisely in the regions where $\epsilon$ is largest, which are the regions the reward model never saw during training. The empirical signature is the overoptimization curve: proxy reward climbs monotonically while gold reward measured by fresh human raters rises, peaks, and then falls. The visible symptoms are familiar to anyone who has run RLHF: verbosity, heavy markdown formatting, confident-sounding hedges, and sycophancy, all of which are cheap ways to increase $r_{\phi}$ without increasing helpfulness.

(1) Goodhart Under Active Search: the reward model is only a proxy, and RL is an adversary that optimizes against its error surface rather than sampling from it.
(2) Distribution Shift Is The Mechanism: preference labels are collected on SFT-model samples, so as the policy moves, its outputs leave the reward model’s training support and the error term stops being small.
(3) Length Is The Canonical Exploit: human raters mildly prefer longer answers, the reward model amplifies that correlation, and a large share of RLHF win-rate gains disappears once responses are length-controlled.
(4) The KL Term Is A Leash, Not A Fix: the $\beta$ -weighted KL penalty bounds how far the policy can travel from the reference model, which delays hacking but trades away real gains at the same time.
(5) Detection Requires An Off-Proxy Signal: you cannot see hacking in the training curve, because the training curve is the thing being hacked; it takes fresh human labels, a held-out reward model, or a verifiable check to notice.

Diagram of the RLHF loop: fixed preference pairs train a proxy reward model, the reward model scores policy samples, PPO or GRPO updates the policy under a KL penalty, and fresh samples feed back into the reward model while drifting off its training support

Figure 1: The RLHF loop (preferences → reward model → policy update → fresh samples) closes on a fixed, off-policy reward model, so every optimization step pushes the sampled distribution further from where the proxy was ever validated.

It helps to separate the failure modes by where the exploit lives. Spurious-feature hacking targets shallow correlates of quality that the reward model learned from labeler habits: length, bullet lists, bold headers, apologetic openers, restating the question. Sycophancy targets the labelers themselves, since annotators reward answers that agree with them, so the policy learns to mirror the stated opinion rather than correct it. Off-distribution spikes are the strangest class: degenerate strings and odd punctuation patterns that no human ever rated, which the reward model happens to score near its maximum. In agentic and code settings the same pressure produces test gaming, where the model special-cases unit tests or edits the test file instead of fixing the function.

Mathematical Formulation:
$J(\pi_{\theta}) = \mathbb{E}[r_{\phi}(x,y)] - \beta\,\mathcal{D}$
$\mathcal{D} = D_{\mathrm{KL}}(\pi_{\theta} \| \pi_{\mathrm{ref}})$
$r_{\phi}(x,y) = r^{*}(x,y) + \epsilon(x,y)$
$d = \sqrt{\mathcal{D}}$
$R_{\mathrm{gold}}(d) = d\,(\alpha - \beta_{o}\log d)$

Where:

$J(\pi_{\theta})$ is the RLHF objective actually optimized, and $R_{\mathrm{gold}}$ is the true human-judged quality, which is never in the gradient.
$x$ is the prompt, $y \sim \pi_{\theta}(\cdot\mid x)$ the sampled completion, and $\pi_{\mathrm{ref}}$ the frozen SFT reference policy.
$r_{\phi}$ is the learned proxy reward, $r^{*}$ the intended reward, and $\epsilon$ the model error whose magnitude grows off the preference-data support.
$\beta$ is the KL coefficient (typical range 0.01 to 0.1), so $\beta \to 0$ removes the leash and $\beta$ large pins the policy to the reference.
$d$ is the square-root KL distance used as the x-axis of overoptimization scaling laws; $\alpha$ and $\beta_{o}$ are fitted constants, distinct from the KL coefficient $\beta$ .
Required initial condition: $\pi_{\theta} = \pi_{\mathrm{ref}}$ at step 0, so $d = 0$ and $R_{\mathrm{gold}}$ starts at the SFT baseline; the peak of the fitted curve sits at $\log d = (\alpha - \beta_{o})/\beta_{o}$ .

Line chart with square-root KL distance on the x axis showing proxy reward rising monotonically while gold reward rises to a peak near d equals 3.1 and then declines below zero, with the peak marked as the early stopping point

Figure 2: Illustrative overoptimization: the proxy score the trainer logs never stops improving, while gold reward turns over around $d \approx 3.1$ and eventually falls below the SFT baseline.

Mitigation	Mechanism	What it buys	Cost or limitation
KL penalty and early stopping	Bounds the distance travelled from the reference policy	Keeps the policy inside the region where the proxy was validated	Caps genuine gains too; the right KL budget is only knowable from gold evals
Reward model ensembles	Score with several models, optimize a pessimistic aggregate such as the minimum	Suppresses idiosyncratic per-model error spikes	Members share preference data, so correlated biases like length survive; N times the scoring cost
Iterative on-policy relabeling	Collect fresh preferences on current-policy samples and refit the reward model	Directly closes the distribution shift that creates the error term	Slow and expensive; each round needs new human annotation
Explicit bias disentangling	Factor the reward into a quality head and a length or format head, then discard the latter	Removes the single largest known spurious correlate	Only fixes biases you already named; new exploits appear elsewhere
Verifiable or rule-based reward	Replace the learned model with a checker: unit tests, a symbolic solver, format rules	No learned error surface to climb on checkable tasks	Only applies where ground truth exists; test gaming and reward tampering remain

August 16, 2026

DL0091 Quantization Formats

What is quantization (FP16/BF16/INT8/INT4), and how does it affect model memory and accuracy?

Answer

Quantization stores weights, activations, or KV-cache entries in a numeric format with fewer bits than the FP32 values the model was defined in, trading numerical resolution for memory and bandwidth. Memory falls almost exactly linearly in bit width: a 70B-parameter model occupies 280 GB in FP32, 140 GB in FP16 or BF16, 70 GB in INT8, and roughly 37 GB in 4-bit with group scales. The two 16-bit floating formats differ only in how they split the 16 bits: FP16 spends 5 bits on the exponent and 10 on the mantissa (max magnitude 65,504, so gradients can overflow without loss scaling), while BF16 keeps FP32’s 8 exponent bits and truncates the mantissa to 7, giving the full $10^{38}$ dynamic range at coarser resolution. Below 16 bits the formats become integers plus a scale, so a real number is recovered as $\hat{w} = s(w_q - z)$ , and quality now depends on how finely those scales are estimated. In practice INT8 weights are essentially free in accuracy with per-channel scales, INT4 weight-only costs a few tenths of a perplexity point on a 70B model with a good calibration method, and quantizing activations is the hard part because transformer activations contain a handful of channels with magnitudes 20x to 100x the median.

(1) Exponent Versus Mantissa: FP16 and BF16 use the same 16 bits, but BF16’s 8 exponent bits buy dynamic range at the cost of precision, which is why BF16 is the default training format on modern accelerators and needs no loss scaling.
(2) Memory Is Linear In Bits: parameter memory is $N b / 8$ bytes, so each halving of the bit width halves the checkpoint, the resident weights, and the bytes moved per token.
(3) Integers Need A Scale And A Granularity: per-tensor scaling is cheapest, per-channel is the practical minimum for weights, and group-wise (typically 128 weights per scale) is what makes 4 bits usable.
(4) Weight-Only Helps Because Decode Is Bandwidth-Bound: single-stream generation reads every weight per token, so 4-bit weights cut latency even when the matmul itself runs in FP16 after dequantization.
(5) Error Concentrates In Outliers: a single extreme value stretches the scale and destroys resolution for every other value in the group, which is why outlier handling matters more than the nominal bit count.

Bit-field diagram comparing FP32, FP16, BF16, FP8 E4M3, INT8 and INT4: each format is drawn as a proportional bar split into sign, exponent and mantissa fields, with dynamic-range notes on the right

Figure 1: Bit budgets drawn to scale. BF16 keeps FP32’s 8 exponent bits and pays with 7 mantissa bits, FP16 does the reverse, and the sub-8-bit formats drop the exponent entirely in favour of an external scale factor.

An integer format is defined by an affine map from the real line onto $2^b$ evenly spaced levels. Symmetric quantization fixes the zero-point at $z = 0$ and is the standard choice for weights, whose distribution is roughly zero-centred; asymmetric quantization keeps a learned $z$ and suits post-ReLU or post-GELU activations that sit mostly on one side of zero. The granularity of $s$ is the real design knob: one scale per tensor stores nothing extra but is destroyed by a single outlier channel, whereas one scale per group of $g = 128$ weights adds a 16-bit scale and a 16-bit zero-point per group, which raises an INT4 tensor from 4.0 to 4.25 effective bits. Calibration methods such as GPTQ and AWQ then choose the rounding of each weight to minimise the error of the layer output on a small calibration set rather than the error of the weight itself, which is what closes most of the remaining gap at 4 bits.

Mathematical Formulation:
$\hat{w} = s (w_q - z)$
$w_q = \mathrm{clip}(\mathrm{round}(w/s) + z,\ 0,\ 2^b - 1)$
$s = \frac{\max(w) - \min(w)}{2^b - 1}$
$b_{eff} = b + \frac{32}{g}$
$M = \frac{N \, b_{eff}}{8}$

Where:

$\hat{w}$ is the dequantized value actually used in the matmul, and $w$ is the original FP32 or BF16 weight.
$w_q$ is the stored integer code, clipped into $\{0,\ldots,2^b-1\}$ ; values outside the calibrated range are saturated rather than wrapped.
$s$ is the scale (step size) and $z$ the zero-point; symmetric quantization sets $z = 0$ and uses $s = \max|w| / (2^{b-1} - 1)$ .
$b$ is the nominal bit width and $g$ the group size, the number of weights sharing one $(s, z)$ pair stored in FP16, hence the $32/g$ overhead.
$b_{eff}$ is the effective bits per weight; with $b = 4$ and $g = 128$ this gives $b_{eff} = 4.25$ .
$N$ is the parameter count and $M$ the weight memory in bytes; for $N = 7 \times 10^{10}$ this is about 37 GB. Required condition: $s > 0$ must be computed from calibration data before any activation quantization is applied.

Bar chart of weight memory in gigabytes for a 70 billion parameter model: 280 GB in FP32, 140 GB in FP16 or BF16, 70 GB in INT8 and 37 GB in INT4 with group size 128, with a dashed line marking the 80 GB capacity of one GPU

Figure 2: Weight memory for a 70B model. Only the 4-bit variant fits one 80 GB GPU with room left for the KV cache, and because decoding is memory-bandwidth bound the same reduction shows up directly in tokens per second.

Property	FP16	BF16	INT8	INT4 (g=128)
Layout	1 sign, 5 exponent, 10 mantissa	1 sign, 8 exponent, 7 mantissa	8-bit integer plus a per-channel FP16 scale	4-bit integer plus a scale per 128 weights
Dynamic range	Max 65,504; underflows below 6e-5	Same as FP32, about 3.4e38	256 levels inside the calibrated range	16 levels per group; clipping is the main risk
70B weights	140 GB	140 GB	70 GB	About 37 GB at 4.25 effective bits
Typical role	Inference, and training with loss scaling	Default training and master-weight format	W8A8 serving at high batch, compute-bound prefill	Weight-only decode on capacity-limited GPUs
Accuracy cost	None measurable for inference	None measurable; the lost mantissa bits rarely matter	Under 1% with per-channel scales and outlier handling	A few tenths of perplexity on 70B, clearly worse below 7B
Hardware support	Tensor cores on every recent GPU	Ampere onward, plus TPUs	Roughly 2x the FP16 matmul throughput	No native 4-bit matmul on most GPUs; kernels dequantize in registers

August 16, 2026

DL0090 Post-Training vs Quantization-Aware Training

What is the difference between post-training quantization and quantization-aware training, and how would you choose between them when shipping a model to an on-device NPU?

Answer

Post-training quantization (PTQ) takes a finished float checkpoint and converts it to low precision after the fact: a few hundred unlabeled samples are pushed through the network so the tool can observe activation ranges, pick a scale and zero point per tensor or per channel, and round the weights. No labels, no loss, no backward pass, and typically minutes to a couple of hours on a single GPU. Quantization-aware training (QAT) instead inserts fake-quantization nodes into the graph and continues training, so every forward pass sees rounded values while gradients flow through the non-differentiable rounding step via the straight-through estimator (STE). The one-line distinction worth memorizing: PTQ fits the quantizer to fixed weights, while QAT moves the weights to fit the quantizer. At INT8 on a well-behaved network the two land within a few tenths of a point of each other, so PTQ wins on cost; at 4 bits and below, or on outlier-heavy and depthwise-separable models, PTQ falls off a cliff and QAT recovers most of the loss.

(1) Where It Happens: PTQ is a post-processing step on a frozen checkpoint; QAT is a fine-tuning stage that must run inside your training pipeline with the original data loader and loss.
(2) What Data It Needs: PTQ needs only a small calibration set (roughly 128 to 1024 unlabeled samples) that is representative of deployment traffic; QAT needs labeled data, or at least a teacher model for distillation.
(3) The STE Trick: rounding has zero gradient almost everywhere, so QAT pretends the quantizer is the identity inside the clipping range and passes the gradient straight through, which is what lets weights drift toward values that round well.
(4) Where PTQ Breaks: per-tensor scales collapse when channel ranges differ by orders of magnitude (depthwise convolutions), when activations carry massive outliers (transformer residual streams), or when the bit width drops to 4 or fewer.
(5) Decision Rule: always try PTQ first because it is cheap and reversible; escalate to QAT only when a measured accuracy gap survives per-channel scales, better range selection, and bias correction.

Mechanically, a fake-quant node applies quantize → dequantize in the forward pass, so tensors stay in float during QAT but carry exactly the values the integer kernel will produce at inference. That means QAT does not speed up training; it slows it down by 20 to 40 percent while simulating the deployment numerics. The payoff is that the optimizer sees the rounding error as part of the loss surface and settles into flatter minima where a few least significant bits do not matter. PTQ has no such feedback: whatever error the rounding introduces is simply propagated forward, which is why its failure mode is a sudden collapse rather than a graceful slide.

Two horizontal pipelines. Top row, post-training quantization: trained FP32 checkpoint, calibration pass on 128 to 1024 unlabeled samples, fit scale and zero point, export INT model. Bottom row, quantization-aware training: trained FP32 checkpoint, insert fake-quant nodes, fine-tune with labels and STE gradients, fold scales and export INT model.

Figure 1: The same checkpoint, two routes to integer inference. PTQ adds one forward-only calibration pass; QAT adds a full fine-tuning loop whose gradients reach the weights through the straight-through estimator.

Mathematical Formulation:
$s = \frac{x_{max} - x_{min}}{2^{b} - 1}$
$q = \mathrm{clip}(\lfloor x/s \rceil + z, 0, 2^{b} - 1)$
$\hat{x} = s\,(q - z)$
$\frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial \mathcal{L}}{\partial \hat{x}} \cdot \mathbf{1}[x_{min} \leq x \leq x_{max}]$

Where:

$\hat{x}$ is the dequantized value the network actually computes with, and $x$ is the original float weight or activation.
$s$ is the scale and $z$ the zero point (the integer that maps to exactly 0.0); $q$ is the stored integer code.
$b$ is the bit width, so $b = 8$ gives 256 levels and $b = 4$ only 16; $x_{min}, x_{max}$ are the calibrated clipping bounds, chosen per channel for weights and per tensor or per token for activations.
$\lfloor \cdot \rceil$ is round-to-nearest and $\mathrm{clip}$ saturates values outside the representable range; both are the source of the error being managed.
The last line is the straight-through estimator: the indicator $\mathbf{1}[\cdot]$ passes the gradient unchanged inside the clipping range and zeroes it outside, which is the only reason QAT can backpropagate through rounding.
PTQ solves for $s, z$ with $x$ held fixed; QAT keeps the whole chain in the graph and updates $x$ (and, with learned-step methods, $s$ itself).

Line chart of top-1 accuracy versus bit width from INT8 down to INT2. PTQ tracks the FP32 baseline at INT8 and INT6 but drops to 68 at INT4, 41 at INT3 and 6 at INT2, while QAT stays at 74.5 at INT4, 71 at INT3 and 62 at INT2.

Figure 2: Illustrative accuracy versus bit width. The two methods are indistinguishable at INT8, which is why PTQ dominates production INT8 pipelines; the gap opens abruptly at 4 bits and below, where the rounding error stops behaving like small additive noise.

Dimension	Post-Training Quantization	Quantization-Aware Training
Data needed	128 to 1024 unlabeled calibration samples	Labeled training data or a teacher for distillation
Compute	Minutes to hours, one GPU, forward passes only	Hours to days, often multi-GPU, full training loop
Pipeline access	Works on a vendor or third-party checkpoint	Requires the original recipe, loss and hyperparameters
Typical INT8 result	Within roughly 0.5 points of FP32 with per-channel weights	Essentially lossless, rarely worth the cost
Typical INT4 result	Large drop unless advanced methods (GPTQ, AWQ) are used	Recovers most of the gap, the standard choice below 4 bits
Iteration speed	Cheap to sweep many bit widths and granularities	Each configuration is a separate training run

August 15, 2026

DL0089 RLHF: Human Feedback Training

What is Reinforcement Learning from Human Feedback?

Answer

Reinforcement Learning from Human Feedback (RLHF) is a fine-tuning procedure that optimizes a language model against a learned model of human preferences instead of against a hand-written loss. It exists because the behaviors we actually want (helpful, honest, non-toxic, well-formatted answers) have no differentiable definition: next-token cross-entropy on scraped text rewards imitating the corpus, not satisfying a user. The standard recipe has three stages: supervised fine-tuning on demonstrations, training a reward model on human comparisons of pairs of responses, then optimizing the policy with an RL algorithm such as PPO against that reward, with a KL penalty pinning the policy near its starting point. Humans never grade individual tokens; they rank whole responses, and the reward model is the mechanism that turns those sparse rankings into a dense per-sequence training signal. InstructGPT is the canonical demonstration: labelers preferred outputs from the 1.3B InstructGPT model to those from the 175B pretrained GPT-3 base model despite roughly 100x fewer parameters, so the gain came from the objective rather than from scale.

(1) Three-Stage Recipe: SFT → reward model → RL fine-tuning, where each stage consumes a different kind of human data (demonstrations, then comparisons, then only prompts).
(2) Preferences Are Cheaper Than Demonstrations: asking “which of these two answers is better?” is faster and gives higher inter-annotator agreement than asking someone to write the ideal answer, and the Bradley-Terry model converts those binary comparisons into a scalar reward.
(3) The KL Anchor Is Load-Bearing: the reward being optimized is a learned proxy, so the objective subtracts $\beta$ times the KL divergence to the SFT reference policy; remove it and the policy walks off-distribution into text the reward model scores highly and humans hate.
(4) Reward Hacking Is The Default Failure: proxy reward keeps rising while true human preference peaks and then declines, which is why production pipelines collect fresh comparisons on the current policy’s own samples rather than training once on a frozen dataset.

Three-stage RLHF pipeline diagram: supervised fine-tuning on prompts plus demonstrations produces the reference policy, a reward model is trained on pairwise preference labels with the Bradley-Terry loss, and PPO optimizes the policy on prompts only against the frozen reward model minus a beta-weighted KL term, with a dashed path showing the SFT policy also serving as the KL anchor

Figure 1: The three stages and the data each one needs. Stage 3 requires no new labels, only prompts, because the reward model has absorbed the human judgments, and the SFT checkpoint serves double duty as initialization and as the KL reference.

The reward model is a copy of the transformer with the language-modeling head replaced by a scalar head, trained so that the preferred response scores higher than the rejected one. Because only differences of rewards appear in the loss, the reward scale and offset are unidentifiable, which is why RLHF implementations whiten or normalize rewards per batch before computing advantages. In stage 3 the policy samples completions for a prompt, the frozen reward model scores each one, and PPO takes a clipped policy-gradient step; the KL term is usually implemented as a per-token penalty folded into the reward rather than as a hard constraint. A naive PPO setup keeps four networks resident (policy, reference, reward model, value head), so memory and orchestration cost is the practical reason many teams reach for a direct preference method instead.

Mathematical Formulation:
$\mathcal{L}_{RM} = -\log \sigma(r_\phi(x,y_w) - r_\phi(x,y_l))$
$R(x,y) = r_\phi(x,y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}$
$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}[R(x,y)]$

Where:

$\mathcal{L}_{RM}$ is the Bradley-Terry pairwise loss and $\sigma$ is the logistic function, so the reward model is trained as a binary classifier over response pairs.
$x$ is a prompt from the dataset $\mathcal{D}$ , and $y_w$ and $y_l$ are the human-preferred and rejected responses to it.
$r_\phi$ is the learned scalar reward model with parameters $\phi$ ; only reward differences are identified, so its absolute scale is arbitrary.
$\pi_\theta$ is the policy being optimized and $\pi_{\mathrm{ref}}$ is the frozen SFT policy; $\beta > 0$ sets how far the policy may drift, with typical values near $0.01$ to $0.1$ .
$R(x,y)$ is the KL-shaped reward actually fed to PPO, and $J(\theta)$ is the on-policy objective, with the expectation taken over responses sampled from the current policy.

Chart of reward versus KL divergence from the reference policy: the proxy reward model score rises monotonically with KL while the gold human preference score rises, peaks around 12 nats, and then declines, with a marker at the peak

Figure 2: Illustrative overoptimization curve: the reward model score climbs without bound as the policy drifts, while measured human preference peaks and falls. $\beta$ and early stopping exist to keep training left of that peak.

Aspect	PPO-based RLHF	DPO	GRPO
Separate reward model	Yes, trained and frozen	No, the policy is its own implicit reward	Yes, or a programmatic verifier
Sampling	On-policy generation every step	Offline, fixed preference pairs	On-policy groups of responses per prompt
Networks in memory	Four: policy, reference, reward, value	Two: policy and reference	Three: no value network, group mean is the baseline
Main failure mode	Reward hacking plus brittle PPO tuning	Stale data, drifts off the current policy distribution	Noisy advantages when a whole group scores alike
Best fit	Large budget, fresh labels, general helpfulness	Fixed preference dataset, limited compute	Reasoning tasks with checkable answers

August 15, 2026