Author: admin

DL0100 GAN Framework

Explain the GAN framework: what are the two networks optimizing, and why does the generator learn anything at all?

Answer

A Generative Adversarial Network trains two networks against each other. The generator $G$ maps a latent noise vector $z \sim p_z$ (typically $\mathcal{N}(0, I)$ ) to a sample in data space, and the discriminator $D$ outputs the probability that its input came from the real dataset rather than from $G$ . They share one value function in a two-player minimax game: $D$ maximizes the log-likelihood of correctly classifying real and fake batches, while $G$ minimizes the same quantity, so the generator’s only learning signal is the gradient that flows backward through the discriminator into the fake samples. The theory is clean: for a fixed $G$ the optimal discriminator is the density ratio $p_{data}/(p_{data} + p_g)$ , and substituting it turns the generator’s objective into the Jensen-Shannon divergence between the data distribution and the model distribution, uniquely minimized when $p_g = p_{data}$ . The practical payoff is that $G$ is an implicit model: it never evaluates a likelihood, it just produces a sample in one forward pass, which is why GANs remain attractive whenever sampling latency matters.

(1) Two Networks One Objective: there is a single value function $V(D,G)$ ; the discriminator ascends it and the generator descends it, so no explicit reconstruction or likelihood term is ever written down.
(2) Discriminator As A Density Ratio: the optimal $D^*$ encodes $p_{data}(x)/p_g(x)$ , which is exactly the information the generator needs about where it is over- or under-producing mass.
(3) Implicit Sampling Model: $p_g$ is defined only through the pushforward of $p_z$ by $G$ , so sampling is a single forward pass but density evaluation is impossible.
(4) Non-Saturating Generator Loss: the theoretical $\log(1 - D(G(z)))$ term has almost no gradient while the generator is bad, so implementations maximize $\log D(G(z))$ instead.
(5) Alternating Updates, No Loss Curve To Read: training alternates discriminator and generator steps and seeks a saddle point, not a minimum, so a falling loss means nothing and quality is judged with FID or human inspection.

Diagram of the GAN framework: latent noise feeds the generator which produces fake samples, real samples and fake samples both feed the discriminator which outputs a probability of being real, and a dashed path carries the generator gradient back from the discriminator output into the generator

Figure 1: The generator never sees a real sample directly; it only receives gradient that has been routed backward through the discriminator, which is why the quality of $D$ bounds what $G$ can learn.

A training step draws a minibatch of real samples and a minibatch of latents, updates $D$ on both, then updates $G$ with $D$ held fixed. The original paper allowed $k$ discriminator steps per generator step; almost everyone now uses $k = 1$ and controls the balance with regularization instead. The three interventions that matter most in practice are spectral normalization or an R1 gradient penalty to keep the discriminator smooth, two-timescale learning rates so the discriminator can stay slightly ahead, and an exponential moving average of the generator weights for evaluation. The classic failure modes are mode collapse, where $G$ concentrates on a few outputs that currently fool $D$ , and vanishing generator gradient, where a discriminator with near-perfect separation returns almost nothing useful.

Mathematical Formulation:
$\min_G \max_D V(D, G)$
$V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)]$
$\quad + \mathbb{E}_{z \sim p_z}[\log (1 - D(G(z)))]$
$D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}$
$C(G) = 2 \, \mathrm{JSD}(p_{data} \, \| \, p_g) - \log 4$
$\mathcal{L}_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]$

Where:

$V(D,G)$ is the shared value function, maximized by $D$ and minimized by $G$ .
$x \sim p_{data}$ is a real sample and $z \sim p_z$ a latent vector, so $G(z)$ is a generated sample and $D(\cdot) \in (0,1)$ is the estimated probability of being real.
$p_g$ is the implicit distribution induced by pushing $p_z$ through $G$ ; it is never evaluated, only sampled.
$D^*$ is the optimal discriminator for a fixed $G$ , and $C(G) = \max_D V(D,G)$ is the resulting generator criterion; $\mathrm{JSD}$ is the Jensen-Shannon divergence, so $C(G) \geq -\log 4$ with equality only at $p_g = p_{data}$ .
$\mathcal{L}_G$ is the non-saturating generator loss actually used in code; it shares the same fixed point as the minimax form but has large gradient when $D(G(z))$ is near 0, which is the required starting condition of training.

Log-scale plot of generator gradient magnitude versus the discriminator score on fake samples, showing the saturating loss curve one over one minus D staying near one at low scores while the non-saturating curve one over D rises to one hundred

Figure 2: Gradient magnitude of the two generator losses with respect to $D(G(z))$ . At the start of training the discriminator wins easily, the saturating form contributes almost nothing, and the non-saturating form is roughly 100x larger at a score of 0.01.

Property	GAN	VAE	Diffusion
Training objective	Adversarial minimax, no likelihood term	ELBO, a lower bound on log-likelihood	Weighted denoising regression on noised inputs
Sampling cost	One forward pass through the generator	One forward pass through the decoder	20 to 1000 network evaluations unless distilled
Sample sharpness	Very sharp, no pixel-averaging term to blur output	Typically blurry from the reconstruction term	Very sharp, current state of the art on text-to-image
Mode coverage	Weakest; mode collapse is the signature failure	Good coverage, over-smoothed samples	Good coverage from a likelihood-style objective
Training stability	Saddle-point game, needs penalties and careful balance	Single stable objective	Single stable objective, scales predictably

August 16, 2026

DL0099 Masked Autoencoding MAE

What is masked autoencoding (MAE, or masked language modeling) as a representation learning objective?

Answer

Masked autoencoding is a denoising self-supervised objective: hide a random subset of the input units, then train the network to reconstruct exactly what was hidden from the surviving context. The supervision is free because the labels are the withheld input itself, and the representation quality comes from the fact that reconstruction cannot be solved by copying: predicting a masked word or a masked image region requires the encoder to carry semantic, long-range context in its hidden states. BERT-style masked language modeling is the discrete version, masking about 15% of tokens and predicting a softmax over the vocabulary at those positions. MAE is the continuous vision version, masking about 75% of image patches and regressing raw pixels with an asymmetric encoder-decoder: a deep ViT encoder that only ever sees the visible 25% of patches, plus a shallow decoder that is discarded after pretraining. That asymmetry is what makes it cheap, and it scales: a ViT-Huge pretrained with MAE on ImageNet-1K alone reaches 87.8% top-1 after fine-tuning, with over 3x faster pretraining than a full-token baseline.

(1) Masking Manufactures The Labels: no annotation is required, so the objective scales with raw data; the only design decisions are what to mask, how much, and what target to regress or classify.
(2) Asymmetric Encoder-Decoder: the encoder processes only unmasked tokens and mask tokens enter at the decoder, so both compute and the pretrain-finetune input mismatch shrink dramatically.
(3) Masking Ratio Tracks Information Density: text is dense and semantic so 15% suffices, while adjacent pixels are highly redundant and need 75% before the task stops being local interpolation.
(4) The Target Defines The Representation: raw pixels, per-patch normalized pixels, discrete VQ tokens (BEiT), or latent teacher features (data2vec) all train the same encoder toward different levels of abstraction.
(5) Strong Fine-Tuning, Weak Linear Probing: MAE features are highly non-linear, so they beat contrastive methods after fine-tuning but lag them under a frozen linear probe or k-NN retrieval.

Flow diagram of the MAE pipeline: a 4 by 4 patch grid with 75 percent of cells masked, an arrow into a ViT encoder that processes only the 49 visible tokens, then a block that inserts mask tokens and restores order, then a lightweight 8-block decoder, then the reconstructed patch grid

Figure 1: The MAE pipeline for a ViT-L/16 at 224 pixels: of 196 patches only 49 reach the encoder, mask tokens are inserted just before a shallow decoder, and the MSE loss is computed on masked patches only.

Two details separate the text and vision instantiations. BERT cannot drop masked positions because the prediction head must sit at those positions, so the literal [MASK] symbol is fed to the encoder during pretraining and never appears at fine-tuning time; the original recipe patches this discrepancy by replacing only 80% of selected tokens with [MASK], 10% with a random token, and leaving 10% unchanged. MAE removes the problem structurally: masked patches are simply deleted from the encoder’s input sequence, and the mask token is a learned vector injected at the decoder. The second detail is the loss support. Computing the reconstruction error on masked patches only matters, since including visible patches lets the model spend capacity on an identity mapping and measurably hurts downstream accuracy.

Mathematical Formulation:
$\mathcal{L}_{\mathrm{MLM}} = -\sum_{i \in \mathcal{M}} \log p_{\theta}(x_i \mid \tilde{x})$
$\hat{x} = D\big(E(x_{\mathcal{V}}), m\big)$
$\mathcal{L}_{\mathrm{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \|\hat{x}_i - x_i\|_2^2$
$|\mathcal{V}| = (1 - \rho)\,N$
$C_{\mathrm{attn}}(\mathcal{V}) / C_{\mathrm{attn}}(N) = (1 - \rho)^2$

Where:

$\mathcal{L}_{\mathrm{MLM}}$ is the cross-entropy over masked token positions and $\mathcal{L}_{\mathrm{MAE}}$ the per-patch MSE; both are averaged over masked positions only.
$x_i$ is the true content at position $i$ , $\hat{x}_i$ the prediction, and $\tilde{x}$ the corrupted sequence containing $\texttt{[MASK]}$ symbols.
$\mathcal{M}$ and $\mathcal{V}$ are the masked and visible index sets, disjoint with $|\mathcal{M}| + |\mathcal{V}| = N$ for $N$ total tokens or patches.
$E$ is the deep encoder applied to visible tokens alone, $D$ the shallow decoder, and $m$ the shared learned mask-token embedding plus positional encoding.
$\rho$ is the masking ratio: $\rho = 0.15$ for BERT and $\rho = 0.75$ for MAE, giving $|\mathcal{V}| = 49$ of $N = 196$ patches.
$C_{\mathrm{attn}}$ is quadratic self-attention cost, so at $\rho = 0.75$ encoder attention drops to $1/16$ and the token-wise MLP cost to $1/4$ ; targets are usually per-patch normalized pixels rather than raw values.

Line chart of ImageNet top-1 accuracy against masking ratio from 10 to 90 percent, showing a nearly flat fine-tuning curve near 85 percent and a strongly peaked linear-probing curve rising to about 73.5 percent at a 75 percent masking ratio then falling sharply at 90 percent

Figure 2: Approximate ViT-L behaviour versus masking ratio: fine-tuning is nearly flat from 40% to 80%, while linear probing peaks sharply near 75%, which is why a ratio that looks extreme for text is the vision default.

Property	MAE (pixel targets)	MLM (BERT-style)	Contrastive / joint embedding
Corruption level	75% of patches removed	15% of tokens replaced or kept	No masking; two augmented views
Encoder input	Visible tokens only, no mask token	Full sequence including [MASK]	Full clean views
Prediction head	8-block 512-dim decoder, discarded	Single linear layer over the vocabulary	MLP projector plus temperature
Best transfer mode	Full fine-tuning and dense tasks	Fine-tuning on token-level tasks	Frozen features, k-NN, zero-shot retrieval
Main weakness	Capacity spent on high-frequency detail; weak linear probe	Only ~15% of positions produce gradient per step	Augmentation-sensitive; risk of collapse

August 16, 2026

DL0098 CLIP Zero-Shot Classification

What is CLIP, and how does contrastive image-text training enable zero-shot classification?

Answer

CLIP (Contrastive Language-Image Pre-training) is a dual-encoder model: an image encoder (a ViT or ResNet) and a text encoder (a Transformer) each map their input to a vector in one shared embedding space, and the vectors are L2-normalized so their inner product is a cosine similarity. Training uses a batch of $N$ image-caption pairs scraped from the web (400M pairs for the original model), computes the full $N \times N$ similarity matrix, and applies a symmetric InfoNCE loss that pushes the $N$ diagonal entries up and the $N^2 - N$ off-diagonal entries down. Nothing in that objective mentions class labels, so the learned text encoder is a general classifier generator: at test time you write each candidate class as a sentence such as “a photo of a golden retriever”, encode it, and the resulting unit vectors act as the rows of a linear classifier weight matrix. Classification is then one matrix multiply followed by an argmax over cosine similarities, which is why a new label set costs a text-encoder forward pass rather than a training run. The best original model (ViT-L/14@336px) reaches 76.2% zero-shot top-1 on ImageNet, matching a supervised ResNet-101 with an identical 76.2% ImageNet top-1 that saw 1.28M labeled images, and it is far more robust under distribution shift.

(1) Two Encoders, One Space: each modality has its own encoder plus a linear projection into a shared $d$ -dimensional space; there is no cross-attention between them, so image and text embeddings can be computed and cached independently.
(2) Symmetric InfoNCE Over The Batch: the loss is cross-entropy applied twice, once along rows (image picks its caption) and once along columns (caption picks its image), averaged.
(3) Learned Temperature: the logit scale $\exp(\tau)$ is a trained scalar initialized to a temperature of 0.07 and clipped at 100, which controls how hard the softmax pushes against near-miss negatives.
(4) Prompts Become Classifier Weights: zero-shot inference replaces the learned classification head with text embeddings of class descriptions, so the label set is defined at inference time, not at training time.
(5) Batch Size Is Part Of The Objective: negatives come only from the current batch, so the original model trained at $N = 32768$ across many GPUs with the similarity matrix sharded.

Diagram of CLIP training: a batch of N images passes through an image encoder to unit-norm vectors, a batch of N captions passes through a text encoder to unit-norm vectors, and both feed an N by N cosine similarity matrix whose diagonal cells are the positive pairs

Figure 1: Each training step builds the full $N \times N$ similarity matrix; the $N$ diagonal entries are the true pairs and every other entry is an in-batch negative, which is why one step at $N = 32768$ supplies over a billion contrastive comparisons.

Three details separate a working CLIP from a broken one. Normalization is load-bearing: without it the model can shrink the loss by inflating embedding norms instead of improving alignment, and the learned temperature then has no fixed scale to calibrate against. Prompt wording matters more than people expect, because captions in the training data are sentences, not bare nouns: using “a photo of a {label}” instead of the raw class name adds about 1.3 points on ImageNet, and ensembling 80 prompt templates by averaging their normalized text embeddings adds roughly 3.5 points for almost no inference cost, since the averaged vectors are computed once and cached. Finally, label naming is part of the model: polysemous class names such as “boxer” (dog breed or athlete) or “crane” (bird or machine) must be disambiguated in the prompt, which is a form of engineering that has no analogue in a supervised classifier with integer labels.

Mathematical Formulation:
$z_i = f_{\theta}(I_i) / \|f_{\theta}(I_i)\|_2$
$t_j = g_{\phi}(T_j) / \|g_{\phi}(T_j)\|_2$
$s_{ij} = \exp(\tau)\, z_i^{\top} t_j$
$\mathcal{L}_{I} = -\frac{1}{N}\sum_{i} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$
$\mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{I} + \mathcal{L}_{T}\right)$
$\hat{y} = \arg\max_{c} \; z^{\top} t_c$

Where:

$z_i$ and $t_j$ are the unit-norm embeddings of image $I_i$ and caption $T_j$ , both in $\mathbb{R}^{d}$ with $d = 768$ for ViT-L/14.
$f_{\theta}$ is the image encoder plus its projection and $g_{\phi}$ the text encoder plus its projection; the two share no weights.
$s_{ij}$ is the scaled cosine similarity and $\exp(\tau)$ the learned logit scale; since $z_i^{\top} t_j$ lies in $[-1, 1]$ , the scale alone sets the sharpness of the softmax.
$i, j \in \{1, \ldots, N\}$ index the batch; $\mathcal{L}_{I}$ normalizes each row and $\mathcal{L}_{T}$ the same expression with the softmax taken over columns.
$\hat{y}$ is the zero-shot prediction for a query image with embedding $z$ , and $t_c$ is the (optionally prompt-ensembled and renormalized) text embedding of class $c$ ; the temperature drops out of the argmax.
Required initial condition: $\tau = \log(1/0.07)$ at step 0, clamped so that $\exp(\tau) \leq 100$ throughout training to prevent the logits from exploding.

Grouped bar chart comparing top-1 accuracy of a supervised ResNet-101 and zero-shot CLIP ViT-L/14 on ImageNet and five distribution-shift benchmarks, with CLIP far ahead on ImageNet-R, ObjectNet, Sketch and ImageNet-A

Figure 2: Matched at 76.2% on the ImageNet validation set, the two models diverge sharply under shift: zero-shot CLIP keeps 77.1% on ImageNet-A and 60.2% on Sketch, where the supervised model drops to 2.7% and 25.2%, evidence that never fitting the ImageNet label distribution is itself a robustness mechanism.

Property	Zero-shot CLIP (ViT-L/14@336px)	Supervised classifier (ResNet-101)
Task supervision	Zero labeled examples for the task, 400M noisy web pairs for pretraining	1.28M hand-labeled images for exactly this label set
Adding a class	Encode one more prompt and append the vector, no gradient step	Collect labels, then retrain or refit the head
ImageNet top-1	76.2%	76.2%
ImageNet-A top-1	77.1%	2.7%
Specialized domains	Weak: near chance on tumor-patch classification, poor on counting and satellite land use	Strong once in-domain labels exist
Inference cost	One image forward pass plus a $d \times C$ matmul against cached text vectors	One forward pass through a fixed head

August 16, 2026

DL0097 Variational Autoencoder VAE

What is a variational autoencoder (VAE), and how does it differ from a plain autoencoder?

Answer

A variational autoencoder is a latent-variable generative model trained by amortized variational inference, not a compression network with a smaller bottleneck. A plain autoencoder learns a deterministic map $z = f_\phi(x)$ and minimizes reconstruction error alone, so nothing constrains where codes land: the latent space is an arbitrary point cloud with holes, and decoding a random vector usually produces garbage. A VAE instead makes the encoder output the parameters of a distribution $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$ , samples $z$ from it, and optimizes the evidence lower bound (ELBO), which adds a KL term pulling every posterior toward a fixed prior $p(z) = \mathcal{N}(0, I)$ . That single extra term is what turns an encoder-decoder pair into a generator: after training you can draw $z \sim \mathcal{N}(0, I)$ and decode it, because the aggregate posterior now covers the prior. The sampling step is made differentiable by the reparameterization trick, $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ , which keeps the stochastic node out of the gradient path.

(1) Probabilistic Encoder: the encoder emits a mean and a log-variance per latent dimension rather than a point, so each input maps to a small blob of latent space instead of a single coordinate.
(2) Two-Term Objective: the loss is reconstruction plus a KL regularizer against the prior; a plain autoencoder has only the first term, which is exactly why it is not generative.
(3) Reparameterization Trick: sampling is rewritten as a deterministic function of the parameters and an external noise draw, giving a low-variance pathwise gradient instead of a high-variance score-function estimator.
(4) Sampling From The Prior: because the KL term forces the posteriors to overlap and fill the prior, ancestral sampling ( $z \sim p(z)$ , then decode) yields plausible data, and interpolation between two codes stays on the data manifold.
(5) Explicit Likelihood Bound: the ELBO is a lower bound on $\log p_\theta(x)$ , so a VAE gives a comparable density estimate, whereas an autoencoder’s reconstruction error has no probabilistic meaning.

Two-row diagram: the top row shows a plain autoencoder mapping input x through an encoder to a single deterministic code and back through a decoder; the bottom row shows a VAE whose encoder emits mu and log sigma squared, a sampling step z equals mu plus sigma times epsilon, and a decoder, with a KL term pulling the posterior toward the standard normal prior

Figure 1: The structural difference is one node: the plain autoencoder passes a single deterministic code to the decoder, while the VAE passes a sample from a learned Gaussian that the KL term keeps anchored to $\mathcal{N}(0, I)$ .

Two practical details dominate real training runs. The first is the balance between the two loss terms: with a strong decoder or a large KL weight, the cheapest solution is to set $q_\phi(z|x) = p(z)$ and ignore the latent entirely, a failure called posterior collapse that shows up as a KL term decaying to near zero while reconstruction stalls. Standard mitigations are KL annealing (ramp the weight from 0 to 1 over the first epochs) and free bits (do not penalize a dimension until its per-dimension KL exceeds a floor of roughly 0.05 to 0.5 nats). The second is the choice of likelihood: a diagonal Gaussian decoder is equivalent to an MSE reconstruction loss, which averages over plausible outputs and is the direct cause of the blurry samples VAEs are known for; discretized logistic or categorical likelihoods sharpen results noticeably. Modern image systems exploit this honestly, using a KL-regularized autoencoder as the perceptual compressor and putting the generative burden on a diffusion model in that latent space, as in Stable Diffusion.

Mathematical Formulation:
$\log p_\theta(x) \geq \mathcal{L}(\theta, \phi; x)$
$\mathcal{L} = \mathcal{L}_{rec} - \beta \, D_{KL}(q_\phi \| p)$
$\mathcal{L}_{rec} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]$
$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$
$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon$
$\epsilon \sim \mathcal{N}(0, I)$
$D_{KL} = -\frac{1}{2} \sum_{j=1}^{d} \delta_j$
$\delta_j = 1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2$

Where:

$\mathcal{L}$ is the ELBO, a lower bound on the marginal log-likelihood $\log p_\theta(x)$ ; the gap between them equals $D_{KL}(q_\phi(z|x) \| p_\theta(z|x))$ , so maximizing the bound both fits the data and sharpens the approximate posterior.
$x$ is the observation and $z \in \mathbb{R}^{d}$ the latent code, with $d \ll \dim(x)$ in the usual bottleneck setting.
$q_\phi(z|x)$ is the encoder (recognition model) with parameters $\phi$ , and $p_\theta(x|z)$ is the decoder (likelihood) with parameters $\theta$ .
$p(z) = \mathcal{N}(0, I)$ is the fixed prior, and $j \in \{1, \ldots, d\}$ indexes latent dimensions in the closed-form KL, where $\mu_j$ and $\sigma_j$ are the encoder outputs for example $x$ .
$\odot$ is elementwise multiplication and $\epsilon$ is the external noise draw that makes the sample differentiable in $\mu$ and $\sigma$ .
$\beta$ is the KL weight; $\beta = 1$ recovers the exact ELBO, $\beta > 1$ gives the beta-VAE disentanglement regime, and $\beta = 0$ degenerates to a plain autoencoder. Networks are typically initialized so $\log \sigma_j^2 \approx 0$ , which starts training near the prior.

Two scatter panels of a two-dimensional latent space: on the left the plain autoencoder codes sit in six distant clusters far from the origin with large empty gaps and prior samples falling in a hole, on the right the VAE codes form overlapping blobs filling the unit and two-sigma circles of the standard normal prior so prior samples land on data

Figure 2: Encoded training data in a 2D latent space. The plain autoencoder is free to scatter codes anywhere, so a draw from $\mathcal{N}(0, I)$ lands in a hole; the KL term compresses the VAE’s aggregate posterior onto the prior, so the same draw hits populated territory.

Property	Plain Autoencoder	VAE	VQ-VAE
Encoder output	One deterministic code vector per input	Distribution parameters, mean and log-variance	Continuous vector snapped to the nearest codebook entry
Training objective	Reconstruction error only	ELBO: reconstruction minus beta times KL to N(0, I)	Reconstruction plus codebook and commitment losses, no KL
Latent geometry	Arbitrary scale, holes and gaps between clusters	Smooth and prior-matched, interpolation stays on-manifold	Discrete grid of K entries, no notion of interpolation
Generating new data	Not supported, a random code decodes to noise	Draw z from the prior and decode, one forward pass	Needs a learned prior over codes, such as a transformer
Typical failure	Memorizes an identity map when the bottleneck is wide	Posterior collapse, blurry Gaussian-likelihood samples	Codebook collapse with most entries unused
Common use	Denoising, compression, anomaly detection	Generative modeling and the latent space of latent diffusion	Discrete tokens for autoregressive image and audio models

August 16, 2026

DL0096 Autoencoders Practical Uses

What is an autoencoder? What are the practical uses of autoencoders?

Answer

An autoencoder is a network trained to reconstruct its own input through a narrow bottleneck: an encoder maps $x \in \mathbb{R}^{D}$ to a code $z \in \mathbb{R}^{d}$ with $d \ll D$ , and a decoder maps that code back to $\hat{x}$ , with the reconstruction error as the only supervision. Because the target is the input itself, training needs no labels, which is why autoencoders are the canonical self-supervised compression model. The bottleneck is what makes the code useful: the network cannot pass everything through, so it must keep the structure that is shared across typical inputs and drop the rest. With linear encoder and decoder and squared loss, the optimum spans exactly the same subspace as the top $d$ principal components; nonlinear layers buy a curved manifold instead of a flat one. The practical uses split into three families depending on which part of the trained model you keep: the error (anomaly detection), the code (compression, retrieval, visualization, latent space for a diffusion model), or the decoder (denoising, inpainting, generation once the latent is regularized).

(1) Bottleneck Plus Reconstruction Loss: the objective is $\|x - \hat{x}\|_2^2$ (or cross-entropy for binary pixels), and the capacity constraint, not the loss, is what forces a compressed representation.
(2) Undercomplete Versus Regularized: an undercomplete model constrains $d$ directly, while sparse, denoising, and contractive variants keep a wide code and constrain it with an $L_1$ penalty, input corruption, or a Jacobian penalty.
(3) Linear Case Recovers PCA: a single linear layer each side with MSE learns the principal subspace, so any gain over PCA must come from nonlinearity, not from the architecture being a neural network.
(4) Anomaly Detection By Error Score: train only on normal data, then score new samples with $s(x) = \|x - \hat{x}\|_2^2$ and flag $s(x) > \tau$ , with $\tau$ set from a percentile of validation errors.
(5) Compression Into A Working Latent Space: modern latent diffusion models run denoising inside an autoencoder latent that shrinks a $512 \times 512 \times 3$ image to $64 \times 64 \times 4$ , a 48x reduction in elements, which is what makes high-resolution sampling affordable.
(6) Not Generative By Default: a vanilla autoencoder learns no density over $z$ , so decoding an arbitrary code usually yields garbage; sampling requires a VAE, VQ-VAE, or a prior fitted over the codes afterwards.

Diagram of an autoencoder: a 784-dimensional input passes through a 256-unit hidden layer into a 32-dimensional code, then back out through a 256-unit hidden layer to a 784-dimensional reconstruction, with both ends feeding a reconstruction loss box

Figure 1: The flow is $x$ → $z$ → $\hat{x}$ ; the $784 \rightarrow 32$ bottleneck is a 24.5x squeeze, and nothing but the reconstruction error supervises what the code contains.

Where do they actually earn their place in a system? Anomaly detection on sensor, telemetry, and transaction data is the most common production use, because normal data is abundant and labeled failures are not. Denoising and restoration use the decoder: train with corrupted inputs and clean targets, and the model learns to project back onto the data manifold. Dimensionality reduction gives a cheap nonlinear preprocessor, typically compressing to 32 or 64 dimensions before a nearest-neighbor index or a t-SNE plot, which is far cheaper than running the visualization on raw features. Two uses dominate current deep learning: the perceptual autoencoder that provides the latent space for latent diffusion, and masked autoencoders, where a ViT reconstructs 75% masked patches as a pretraining task and the encoder is then fine-tuned for classification or detection. Recommender systems also use the pattern directly, reconstructing a user’s sparse interaction vector to predict the entries that are missing.

Histogram of reconstruction errors: normal samples cluster at low error, anomalous samples spread into a long right tail, with a vertical threshold line drawn at the 99th percentile of the normal validation errors

Figure 2: Illustrative anomaly scores from an autoencoder trained on normal data only. A threshold at the 99th percentile of normal validation errors fixes the false-positive rate at 1% by construction, and the achievable recall depends entirely on how far the anomalous tail separates.

Mathematical Formulation:
$z = f_{\theta}(x)$
$\hat{x} = g_{\phi}(z)$
$\mathcal{L}(\theta, \phi) = \frac{1}{N} \sum_{i=1}^{N} \| x_i - \hat{x}_i \|_2^2$
$s(x) = \| x - g_{\phi}(f_{\theta}(x)) \|_2^2$
$\mathcal{L}_{\mathrm{DAE}} = \mathbb{E} \| x - g_{\phi}(f_{\theta}(\tilde{x})) \|_2^2$

Where:

$z \in \mathbb{R}^{d}$ is the code and $\hat{x} \in \mathbb{R}^{D}$ the reconstruction of the input $x \in \mathbb{R}^{D}$ .
$D$ is the input dimension and $d$ the code dimension, with $d \ll D$ in the undercomplete case.
$f_{\theta}$ is the encoder and $g_{\phi}$ the decoder, with trainable parameters $\theta$ and $\phi$ optimized jointly.
$i \in \{1, \ldots, N\}$ indexes the $N$ training examples, and $\mathcal{L}$ is the mean squared reconstruction loss.
$s(x)$ is the anomaly score, and a sample is flagged when $s(x) > \tau$ for a threshold $\tau$ chosen from a high percentile of normal validation scores.
$\tilde{x}$ is a corrupted copy of $x$ (additive noise, masking, or dropout on inputs); the denoising objective still targets the clean $x$ , which is what prevents the identity solution even when $d \geq D$ .

Property	Undercomplete autoencoder	PCA	VAE
Mapping	Nonlinear encoder and decoder, learned by SGD	Single linear projection, closed-form via SVD	Nonlinear, but the encoder outputs a distribution over z
Objective	Reconstruction error only	Maximum retained variance, equivalent to linear MSE	Reconstruction plus a KL term pulling z toward a prior
Latent geometry	Arbitrary, often with holes and unused directions	Orthogonal ordered axes, fully interpretable scale	Smooth and roughly isotropic, so interpolation is meaningful
Sampling new data	Not supported without fitting a prior over codes afterwards	Only under an explicit Gaussian model such as probabilistic PCA	Yes, sample the prior and decode
Typical use	Anomaly scoring, denoising, compact features for indexing	Fast baseline, whitening, exploratory analysis	Latent space for diffusion, generative sampling, recsys
Main caveat	Too much capacity collapses to near-identity and the code stops being informative	Cannot represent curved manifolds; cost grows with feature count	Blurry reconstructions and posterior collapse if the KL weight is too high

August 16, 2026

DL0095 Zero-Shot vs Few-Shot

How do zero-shot and few-shot prompting differ, and when does few-shot prompting beat fine-tuning?

Answer

Both are inference-time conditioning: the weights $\theta$ never change, only the tokens placed before the query. Zero-shot prompting gives an instruction and the input, so the model must map the task description onto a behavior it already learned during pretraining or instruction tuning. Few-shot prompting (in-context learning) prepends $K$ solved demonstrations $(x_i, y_i)$ that pin down the output format, the label space, and the input distribution before the real query arrives, which is why it helps most on tasks with an unusual schema or a strict output contract. In the original GPT-3 study, TriviaQA accuracy for the 175B model moved 64.3% → 71.2% going from zero-shot to 64-shot, and the gap between the two settings shrinks as models get better instruction tuning. Few-shot prompting beats fine-tuning when labeled data is scarce (roughly tens of examples), the task spec is still changing weekly, one frozen base must serve many tasks, or no training infrastructure exists; fine-tuning wins once you have thousands of clean labels, need the lowest possible per-request cost and latency, or need behavior that no prompt reliably elicits. The decision is mostly economics plus label count, not model quality: demonstrations are paid for on every request, while a fine-tune is a one-time cost amortized over traffic.

(1) Same Weights, Different Context: neither method computes a gradient; few-shot differs from zero-shot only by the demonstration block $S_K$ inserted into the prompt.
(2) Demonstrations Teach Format, Not Mostly Facts: the label space, input distribution, and output template drive most of the gain, which is why even partly incorrect labels in the exemplars often still work.
(3) Cost Is Recurring: $K$ exemplars add $K T_{ex}$ prefill tokens per call, inflating time-to-first-token and input spend on every request forever.
(4) Fine-Tuning Trades Setup For Marginal Cost: a LoRA run costs money once and then serves a short prompt, so it wins above a traffic break-even point.
(5) Data Volume Decides The Ceiling: with a handful of labels in-context learning is usually ahead; with thousands, parameter updates reach accuracy no prompt matches.

The mechanism is worth stating precisely because it predicts the failure modes. Demonstrations act as a task locator rather than a training set: replacing gold labels with random ones from the correct label set degrades few-shot accuracy far less than removing the labels entirely, which shows the exemplars are mostly specifying which distribution to condition on. That same conditioning creates strong biases: majority-label bias (a class over-represented in the exemplars gets over-predicted), recency bias (the last exemplar dominates), and ordering sensitivity that can swing accuracy by tens of points across permutations of the same $K$ examples. Calibration on a content-free input and stratified, order-shuffled exemplar selection recover most of that variance, and any prompt tuned on a large validation set is no longer honestly “few-shot” because the selection itself consumed labels.

Line chart of task accuracy against the number of in-context examples for an 8B and a 70B frozen model, with two dashed horizontal reference lines for an 8B model fine-tuned on 100 and on 5000 labels

Figure 1: Illustrative shot-scaling behavior: most of the in-context gain arrives by $K = 4$ and flattens after $K = 16$ , a model fine-tuned on only 100 labels sits near the few-shot curve, and 5,000 labels put the fine-tuned small model above the frozen large model.

Mathematical Formulation:
$p_{\theta}(y \mid I, x)$
$p_{\theta}(y \mid I, S_K, x)$
$S_K = ((x_1, y_1), \ldots, (x_K, y_K))$
$T_{ctx} = T_I + K T_{ex} + T_x$
$40 + 32 \times 60 + 30 = 1990$
$R^{*} = \frac{C_{ft}}{K T_{ex} c_{in}}$

Where:

$y$ is the generated answer, $x$ the query, and $I$ the instruction text; the first two lines are the zero-shot and few-shot predictive distributions under identical $\theta$ .
$S_K$ is the demonstration block and $K$ the shot count, with $K = 0$ recovering the zero-shot case exactly.
$T_I$ , $T_{ex}$ , and $T_x$ are token lengths of the instruction, one exemplar, and the query; $T_{ctx}$ is the prefill length that sets time-to-first-token.
The numeric line instantiates $T_{ctx}$ for $K = 32$ exemplars of 60 tokens each, giving 1,990 prompt tokens against 70 for the zero-shot prompt.
$c_{in}$ is the price per input token, $C_{ft}$ the one-time fine-tuning cost, and $R^{*}$ the break-even request volume above which the fine-tune is cheaper; it assumes both options serve the same output length and per-token price.

Plugging in real numbers makes the trade-off concrete. At $c_{in} = \$0.30$ per million input tokens, the 1,920 extra tokens from 32 exemplars cost about $\$0.00058$ per request, so a $\$60$ LoRA job pays for itself after roughly 104,000 requests. Prompt caching changes that arithmetic sharply: because the exemplar block is a fixed prefix, cached reads billed near 10% of the input rate push the break-even beyond a million requests and also cut the prefill latency penalty. That is why the honest answer to “few-shot or fine-tune” depends on traffic volume, prefix stability, and whether your serving stack caches, not on which technique sounds more advanced.

Line chart of cumulative extra cost in dollars against requests served, comparing few-shot prompting with and without prompt caching against a flat one-time fine-tuning cost, with the crossover marked near 104 thousand requests

Figure 2: Illustrative cost crossover: the few-shot line grows linearly with traffic because the exemplar prefix is re-billed per call, the fine-tune is a flat one-time charge, and prompt caching flattens the few-shot slope by roughly an order of magnitude.

Property	Zero-shot	Few-shot (in-context)	Fine-tuning (LoRA)
Labeled examples needed	None, only a clear instruction	Typically 4 to 64, plus a small set for prompt selection	Hundreds to tens of thousands
Where task knowledge lives	Pretraining and instruction tuning only	In the prompt, re-sent or cached per request	In adapter weights, prompt stays short
Prompt tokens per request	70 in the worked example	1,990 at K=32, so higher time-to-first-token	70, same as zero-shot
Iteration speed	Seconds, edit the instruction	Seconds, swap or reorder exemplars	Hours per run plus eval and deploy
Main failure mode	Wrong output schema, task misread	Majority-label and recency bias, ordering variance, context limits	Overfits small or noisy label sets, forgets off-task behavior
Multi-task serving	One model, one endpoint	One model, per-task prompt template	One adapter per task over a frozen base

August 16, 2026

DL0094 Evaluating an LLM

How do you evaluate an LLM?

Answer

There is no single number, so I treat evaluation as a stack of layers chosen to match how the model will actually be used: intrinsic likelihood (perplexity or bits per byte) at the bottom, static capability benchmarks above it, automated graders such as unit tests and LLM-as-judge, human pairwise preference, and finally an online A/B test on the real product metric. Each layer up the stack is closer to user value and more expensive, so cheap layers act as fast regression gates and expensive layers make ship decisions. The practical work is defining the task metric first, then building a private evaluation set drawn from production traffic, because public leaderboards measure a distribution that is rarely yours. Three things decide whether an evaluation is worth anything: contamination control, judge calibration against human labels, and error bars. Most reported “improvements” of one or two points on a 1000-item benchmark are inside the noise band and would not survive a paired significance test.

(1) Start From The Deployment Task: a RAG assistant is scored on groundedness, citation precision, and refusal correctness, a coding model on pass@k against hidden tests, an agent on end-to-end task completion; a generic leaderboard score answers none of these.
(2) Intrinsic Versus Extrinsic: perplexity is cheap and useful for pretraining and quantization regressions, but it is not comparable across tokenizers and correlates weakly with instruction-following quality.
(3) Static Benchmarks Decay: MMLU-Pro, GPQA, MATH, HumanEval, and SWE-bench Verified are reproducible and cheap, but they saturate and they leak into pretraining corpora, so contamination checks (n-gram overlap, held-out variants, release-date filtering) are mandatory.
(4) Automated Graders Scale, With Bias: the loop is sample responses → grade with a rubric or unit tests → aggregate with intervals; a judge model shows position, verbosity, and self-preference bias, so swap the answer order and calibrate agreement with humans before trusting it.
(5) Human Preference Is The Open-Ended Gold Standard: pairwise votes fitted with a Bradley-Terry (Elo) model give a single latent quality score, at the cost of slow turnaround and annotator disagreement.
(6) Report Uncertainty, Safety, And Cost: every score needs a confidence interval and a paired test against the baseline, plus a separate axis for jailbreak resistance, toxicity, PII leakage, and the p95 latency and cost per request that decide whether the win is affordable.

Five stacked layers of LLM evaluation from intrinsic perplexity at the bottom through static benchmarks, automated graders, human preference, and production A/B testing at the top, with cost and fidelity increasing upward

Figure 1: The evaluation stack. Cheap layers at the bottom run on every commit as regression gates; the expensive layers at the top are the only ones that measure user value, so a release decision needs evidence from both ends.

Two failure modes dominate real evaluation work. The first is contamination: a model that has memorized a benchmark scores high without generalizing, which you detect by rewriting items into paraphrases or numeric variants and watching the score collapse, or by using benchmarks whose items post-date the training cutoff. The second is judge miscalibration: an LLM judge that prefers longer, more confidently worded answers will happily rank a verbose model above a correct one, so I always measure judge-human agreement (Cohen’s kappa above roughly 0.6 before the judge is allowed to gate a release) and control for response length. On top of that, generation is stochastic, so a benchmark run at temperature 0.7 has run-to-run variance of its own; fix seeds and decoding parameters, or report the mean over several runs. Finally, offline wins must be confirmed online, because the production metric that matters (resolution rate, edit distance to the accepted answer, escalation rate) frequently moves in the opposite direction from a benchmark score.

Property	Static benchmark	LLM-as-judge	Human preference
What it measures	Closed-form correctness on a fixed item set	Rubric compliance and pairwise win rate on open-ended prompts	Latent quality as perceived by real users or experts
Turnaround per model	Minutes to hours, fully automated	Hours, bounded by judge API throughput and cost	Days to weeks, bounded by annotator supply
Main failure mode	Contamination and saturation; near-zero headroom on old suites	Position, verbosity, and self-preference bias	Annotator disagreement and prompt-mix drift
Reproducible	Yes, if decoding and prompt template are pinned	Only against a pinned judge version; judge upgrades shift scores	No, each round is a fresh sample of raters
Best used for	CI regression gates and capability screening	Scaling open-ended comparison between candidate checkpoints	Final ranking and calibrating the automated judge

Mathematical Formulation:
$\mathrm{PPL} = \exp\left(-\frac{1}{T}\sum_{t=1}^{T}\log p(x_t \mid x_{1:t-1})\right)$
$\mathrm{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$
$P(a \succ b) = \frac{1}{1 + 10^{(R_b - R_a)/400}}$
$\mathrm{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{N}}$

Where:

$\mathrm{PPL}$ is perplexity over a held-out sequence $x_1,\ldots,x_T$ , with $t$ indexing tokens; it depends on the tokenizer, so bits per byte is the cross-model comparable form.
$n$ is the number of samples drawn per coding problem, $c$ the number that pass the hidden tests, and $k$ the budget of attempts scored; this is the unbiased pass@k estimator, averaged over problems.
$P(a \succ b)$ is the modelled probability that response $a$ beats $b$ , and $R_a, R_b$ are the fitted Bradley-Terry ratings on the Elo scale, where a 100-point gap implies about a 64% win rate.
$\hat{p}$ is the observed accuracy on $N$ independent items and $\mathrm{SE}$ its standard error; the 95% interval is $\hat{p} \pm 1.96\,\mathrm{SE}$ , which for $\hat{p}=0.7$ and $N=1000$ is roughly $\pm 2.8$ points.
Required condition: items must be independent and not contaminated; when two models are scored on the same items, use a paired test (McNemar or a paired bootstrap) rather than comparing two independent intervals.

Bar chart of three model accuracies of 71.2, 73.5 and 78.9 percent on a 1000-item benchmark with 95 percent confidence interval error bars of about 2.8 points, showing the first two intervals overlapping

Figure 2: Accuracy with 95% intervals on a 1000-item benchmark. The 2.3-point gap between the first two models sits well inside the noise band, so an unpaired claim of improvement is unsupported; only the third model separates cleanly.

August 16, 2026

DL0093 Distillation vs Serving

When is distilling a large model into a small one a better strategy than serving the large model directly, for example behind a high-traffic assistant API?

Answer

Distillation wins when the recurring cost of inference, multiplied by sustained traffic, dominates the one-time cost of teacher labeling plus student training, and when the task distribution is narrow enough that the teacher’s surplus capability is never exercised. The decision is a break-even calculation: if the teacher costs $2.00 per 1k requests and an 8B student costs $0.25, and distillation costs $1,400 once (500k teacher-labeled prompts at $1,000 plus 200 GPU-hours at $400), the crossover is 800k requests, roughly 18 hours at 12 QPS. Two other conditions make distillation the only option rather than the cheapest one: a hard latency SLO that no batching strategy can meet with a 70B decoder, and a fixed memory envelope such as a phone NPU or a single 24 GB GPU, where quantization alone cannot close a 10x parameter gap. Serve the large model directly when traffic is low, when the input distribution is open-ended, or when the win is available more cheaply through quantization, speculative decoding, prompt caching, or a small-to-large cascade, since those preserve behavior and skip a new eval and re-qualification cycle. Distillation also carries a maintenance tax: every teacher upgrade reopens the labeling job, and the student inherits the teacher’s errors without the capacity to recover from them.

(1) Break-Even Volume: divide the one-time distillation cost by the per-request savings; below that volume, serving the teacher is strictly cheaper, and the payback period, not the compression ratio, is the number that decides the project.
(2) Latency Floor, Not Throughput: autoregressive decoding is memory-bandwidth bound and sequential, so per-token latency scales with parameter count and cannot be batched away; shrinking the model is one of the few levers that moves p99 rather than QPS per GPU.
(3) Task Breadth Decides Feasibility: on a stable, narrow task a student 8x smaller typically recovers 98-99% of teacher quality, while an open-ended assistant distribution loses many points because the teacher’s broad competence is exactly what is being compressed away.
(4) Hard Deployment Constraints: on-device, air-gapped, or offline targets impose a memory and power budget that no serving trick satisfies, so distillation becomes a requirement rather than an optimization.
(5) Cheaper Alternatives First: speculative decoding is distribution-preserving and needs no re-qualification, quantization is a one-line config change, and a router that sends easy traffic to a small model keeps the teacher available for the tail; distillation is the only lever that actually reduces the footprint, and it is the most expensive to maintain.

Mathematical Formulation:
$Q^{*} = \frac{C_{0}}{c_{L} - c_{S}}$
$C_{0} = N c_{L} + C_{\text{train}}$
$\mathcal{L} = (1-\lambda)\mathcal{L}_{\text{CE}} + \lambda \tau^{2} \mathcal{L}_{\text{KD}}$
$\mathcal{L}_{\text{KD}} = \mathrm{KL}(p^{T}_{\tau} \| p^{S}_{\tau})$

Where:

$Q^{*}$ is the break-even request volume, the point at which cumulative distilled-serving cost equals cumulative teacher-serving cost.
$c_{L}$ and $c_{S}$ are the marginal serving costs per request for the large teacher and the small student; the difference $c_{L} - c_{S}$ must be positive for distillation to ever pay back.
$C_{0}$ is the one-time cost, composed of teacher inference over $N$ unlabeled prompts plus student training $C_{\text{train}}$ ; engineering and eval time belong here too and usually exceed the compute.
$\mathcal{L}$ is the student objective, mixing hard-label cross-entropy $\mathcal{L}_{\text{CE}}$ with the distillation term $\mathcal{L}_{\text{KD}}$ under mixing weight $\lambda \in [0,1]$ .
$\tau$ is the softmax temperature applied to both teacher and student logits, and the $\tau^{2}$ factor rescales the gradient so $\lambda$ stays meaningful as $\tau$ changes; $p^{T}_{\tau}$ and $p^{S}_{\tau}$ are the tempered teacher and student distributions.
Required condition for the whole strategy: the student must also satisfy the latency and memory constraints, $t_{S} \leq \text{SLO}$ and $m_{S} \leq m_{\max}$ , which no amount of traffic volume can buy.

Line chart of cumulative serving cost in dollars against cumulative requests in millions: serving the 70B teacher rises linearly at 2 dollars per thousand requests from the origin, while the distillation path starts at 1400 dollars and rises at 0.25 dollars per thousand, crossing at 800 thousand requests with the savings region shaded beyond the crossing

Figure 1: Illustrative cost curves: the distilled path carries a $1.4k fixed intercept but an eight-times shallower slope, so the break-even sits at 800k requests and every request after that is pure savings.

The break-even math is only valid if the student actually holds quality on the traffic you will receive, and that depends far more on task breadth than on the compression ratio. A single-intent classifier, a fixed extraction schema, or one product domain compresses well because the teacher’s decision boundary in that slice is simple and you can generate unlimited soft targets for it; an open-ended assistant does not, because the surplus capacity is the product. When breadth is the problem, a cascade is usually the better answer than a bigger student: route every request to the distilled model, escalate the tail flagged by a confidence or verifier signal to the teacher, and you capture most of the cost win while keeping the quality ceiling. Whatever you choose, budget for the maintenance tax, since a teacher upgrade invalidates the student, drifting inputs silently move outside the distilled distribution, and you now own two eval suites instead of one.

Semi-log chart of percentage of teacher quality retained against student parameter count from 0.5B to 70B, showing a narrow single-domain task staying above the 98 percent floor from about 1B upward while an open-ended assistant task falls to roughly 71 percent at 1B and only reaches the floor near 20B

Figure 2: Illustrative quality retention: on a narrow task a 1B student already clears a 98% floor, while the same student on open-ended traffic gives up roughly 27 points, which is why breadth, not the compression factor, gates the decision.

Property	Distill to a small student	Serve large + quantization	Serve large + speculative decoding
Memory footprint	8-20x smaller, can drop from 8 GPUs to 1 or to a phone	2-4x smaller weights, KV cache mostly unchanged	Slightly larger, the drafter is an extra resident model
Quality guarantee	None, must be re-measured per task; tail regressions are common	Small measurable drift, worse at 4-bit and on long context	Distribution-preserving, output matches the teacher exactly
Typical latency gain	4-6x on per-token latency, the largest available lever	1.5-2.5x, bounded by bandwidth saved on weights	2-3x at low batch, degrades as batch size grows
One-time cost	Label generation, training runs, a new eval harness	Calibration pass plus a quality spot-check	Serving-stack work, no retraining of the target model
Cost when the teacher is upgraded	Full re-distillation and re-qualification	Re-run calibration only	Swap the target, optionally refresh the drafter
Wins when	High sustained traffic, narrow task, or a fixed device budget	You need a fast, low-risk cost cut on broad traffic	Latency matters, batches are small, and quality cannot move

August 16, 2026

DL0092 Reward Hacking in RLHF

What is reward hacking, and how does it manifest in RLHF?

Answer

Reward hacking is what happens when a policy maximizes the measured reward instead of the intended objective, exploiting the gap between the two. In RLHF that gap is concrete and unavoidable: the reward model $r_{\phi}$ is a finite-capacity network fit to a fixed set of human preference pairs, so it equals the true human utility $r^{*}$ plus an error term $\epsilon$ . PPO does not distinguish the two, and since it actively searches for the highest-scoring completions, it lands precisely in the regions where $\epsilon$ is largest, which are the regions the reward model never saw during training. The empirical signature is the overoptimization curve: proxy reward climbs monotonically while gold reward measured by fresh human raters rises, peaks, and then falls. The visible symptoms are familiar to anyone who has run RLHF: verbosity, heavy markdown formatting, confident-sounding hedges, and sycophancy, all of which are cheap ways to increase $r_{\phi}$ without increasing helpfulness.

(1) Goodhart Under Active Search: the reward model is only a proxy, and RL is an adversary that optimizes against its error surface rather than sampling from it.
(2) Distribution Shift Is The Mechanism: preference labels are collected on SFT-model samples, so as the policy moves, its outputs leave the reward model’s training support and the error term stops being small.
(3) Length Is The Canonical Exploit: human raters mildly prefer longer answers, the reward model amplifies that correlation, and a large share of RLHF win-rate gains disappears once responses are length-controlled.
(4) The KL Term Is A Leash, Not A Fix: the $\beta$ -weighted KL penalty bounds how far the policy can travel from the reference model, which delays hacking but trades away real gains at the same time.
(5) Detection Requires An Off-Proxy Signal: you cannot see hacking in the training curve, because the training curve is the thing being hacked; it takes fresh human labels, a held-out reward model, or a verifiable check to notice.

Diagram of the RLHF loop: fixed preference pairs train a proxy reward model, the reward model scores policy samples, PPO or GRPO updates the policy under a KL penalty, and fresh samples feed back into the reward model while drifting off its training support

Figure 1: The RLHF loop (preferences → reward model → policy update → fresh samples) closes on a fixed, off-policy reward model, so every optimization step pushes the sampled distribution further from where the proxy was ever validated.

It helps to separate the failure modes by where the exploit lives. Spurious-feature hacking targets shallow correlates of quality that the reward model learned from labeler habits: length, bullet lists, bold headers, apologetic openers, restating the question. Sycophancy targets the labelers themselves, since annotators reward answers that agree with them, so the policy learns to mirror the stated opinion rather than correct it. Off-distribution spikes are the strangest class: degenerate strings and odd punctuation patterns that no human ever rated, which the reward model happens to score near its maximum. In agentic and code settings the same pressure produces test gaming, where the model special-cases unit tests or edits the test file instead of fixing the function.

Mathematical Formulation:
$J(\pi_{\theta}) = \mathbb{E}[r_{\phi}(x,y)] - \beta\,\mathcal{D}$
$\mathcal{D} = D_{\mathrm{KL}}(\pi_{\theta} \| \pi_{\mathrm{ref}})$
$r_{\phi}(x,y) = r^{*}(x,y) + \epsilon(x,y)$
$d = \sqrt{\mathcal{D}}$
$R_{\mathrm{gold}}(d) = d\,(\alpha - \beta_{o}\log d)$

Where:

$J(\pi_{\theta})$ is the RLHF objective actually optimized, and $R_{\mathrm{gold}}$ is the true human-judged quality, which is never in the gradient.
$x$ is the prompt, $y \sim \pi_{\theta}(\cdot\mid x)$ the sampled completion, and $\pi_{\mathrm{ref}}$ the frozen SFT reference policy.
$r_{\phi}$ is the learned proxy reward, $r^{*}$ the intended reward, and $\epsilon$ the model error whose magnitude grows off the preference-data support.
$\beta$ is the KL coefficient (typical range 0.01 to 0.1), so $\beta \to 0$ removes the leash and $\beta$ large pins the policy to the reference.
$d$ is the square-root KL distance used as the x-axis of overoptimization scaling laws; $\alpha$ and $\beta_{o}$ are fitted constants, distinct from the KL coefficient $\beta$ .
Required initial condition: $\pi_{\theta} = \pi_{\mathrm{ref}}$ at step 0, so $d = 0$ and $R_{\mathrm{gold}}$ starts at the SFT baseline; the peak of the fitted curve sits at $\log d = (\alpha - \beta_{o})/\beta_{o}$ .

Line chart with square-root KL distance on the x axis showing proxy reward rising monotonically while gold reward rises to a peak near d equals 3.1 and then declines below zero, with the peak marked as the early stopping point

Figure 2: Illustrative overoptimization: the proxy score the trainer logs never stops improving, while gold reward turns over around $d \approx 3.1$ and eventually falls below the SFT baseline.

Mitigation	Mechanism	What it buys	Cost or limitation
KL penalty and early stopping	Bounds the distance travelled from the reference policy	Keeps the policy inside the region where the proxy was validated	Caps genuine gains too; the right KL budget is only knowable from gold evals
Reward model ensembles	Score with several models, optimize a pessimistic aggregate such as the minimum	Suppresses idiosyncratic per-model error spikes	Members share preference data, so correlated biases like length survive; N times the scoring cost
Iterative on-policy relabeling	Collect fresh preferences on current-policy samples and refit the reward model	Directly closes the distribution shift that creates the error term	Slow and expensive; each round needs new human annotation
Explicit bias disentangling	Factor the reward into a quality head and a length or format head, then discard the latter	Removes the single largest known spurious correlate	Only fixes biases you already named; new exploits appear elsewhere
Verifiable or rule-based reward	Replace the learned model with a checker: unit tests, a symbolic solver, format rules	No learned error surface to climb on checkable tasks	Only applies where ground truth exists; test gaming and reward tampering remain

August 16, 2026

DL0091 Quantization Formats

What is quantization (FP16/BF16/INT8/INT4), and how does it affect model memory and accuracy?

Answer

Quantization stores weights, activations, or KV-cache entries in a numeric format with fewer bits than the FP32 values the model was defined in, trading numerical resolution for memory and bandwidth. Memory falls almost exactly linearly in bit width: a 70B-parameter model occupies 280 GB in FP32, 140 GB in FP16 or BF16, 70 GB in INT8, and roughly 37 GB in 4-bit with group scales. The two 16-bit floating formats differ only in how they split the 16 bits: FP16 spends 5 bits on the exponent and 10 on the mantissa (max magnitude 65,504, so gradients can overflow without loss scaling), while BF16 keeps FP32’s 8 exponent bits and truncates the mantissa to 7, giving the full $10^{38}$ dynamic range at coarser resolution. Below 16 bits the formats become integers plus a scale, so a real number is recovered as $\hat{w} = s(w_q - z)$ , and quality now depends on how finely those scales are estimated. In practice INT8 weights are essentially free in accuracy with per-channel scales, INT4 weight-only costs a few tenths of a perplexity point on a 70B model with a good calibration method, and quantizing activations is the hard part because transformer activations contain a handful of channels with magnitudes 20x to 100x the median.

(1) Exponent Versus Mantissa: FP16 and BF16 use the same 16 bits, but BF16’s 8 exponent bits buy dynamic range at the cost of precision, which is why BF16 is the default training format on modern accelerators and needs no loss scaling.
(2) Memory Is Linear In Bits: parameter memory is $N b / 8$ bytes, so each halving of the bit width halves the checkpoint, the resident weights, and the bytes moved per token.
(3) Integers Need A Scale And A Granularity: per-tensor scaling is cheapest, per-channel is the practical minimum for weights, and group-wise (typically 128 weights per scale) is what makes 4 bits usable.
(4) Weight-Only Helps Because Decode Is Bandwidth-Bound: single-stream generation reads every weight per token, so 4-bit weights cut latency even when the matmul itself runs in FP16 after dequantization.
(5) Error Concentrates In Outliers: a single extreme value stretches the scale and destroys resolution for every other value in the group, which is why outlier handling matters more than the nominal bit count.

Bit-field diagram comparing FP32, FP16, BF16, FP8 E4M3, INT8 and INT4: each format is drawn as a proportional bar split into sign, exponent and mantissa fields, with dynamic-range notes on the right

Figure 1: Bit budgets drawn to scale. BF16 keeps FP32’s 8 exponent bits and pays with 7 mantissa bits, FP16 does the reverse, and the sub-8-bit formats drop the exponent entirely in favour of an external scale factor.

An integer format is defined by an affine map from the real line onto $2^b$ evenly spaced levels. Symmetric quantization fixes the zero-point at $z = 0$ and is the standard choice for weights, whose distribution is roughly zero-centred; asymmetric quantization keeps a learned $z$ and suits post-ReLU or post-GELU activations that sit mostly on one side of zero. The granularity of $s$ is the real design knob: one scale per tensor stores nothing extra but is destroyed by a single outlier channel, whereas one scale per group of $g = 128$ weights adds a 16-bit scale and a 16-bit zero-point per group, which raises an INT4 tensor from 4.0 to 4.25 effective bits. Calibration methods such as GPTQ and AWQ then choose the rounding of each weight to minimise the error of the layer output on a small calibration set rather than the error of the weight itself, which is what closes most of the remaining gap at 4 bits.

Mathematical Formulation:
$\hat{w} = s (w_q - z)$
$w_q = \mathrm{clip}(\mathrm{round}(w/s) + z,\ 0,\ 2^b - 1)$
$s = \frac{\max(w) - \min(w)}{2^b - 1}$
$b_{eff} = b + \frac{32}{g}$
$M = \frac{N \, b_{eff}}{8}$

Where:

$\hat{w}$ is the dequantized value actually used in the matmul, and $w$ is the original FP32 or BF16 weight.
$w_q$ is the stored integer code, clipped into $\{0,\ldots,2^b-1\}$ ; values outside the calibrated range are saturated rather than wrapped.
$s$ is the scale (step size) and $z$ the zero-point; symmetric quantization sets $z = 0$ and uses $s = \max|w| / (2^{b-1} - 1)$ .
$b$ is the nominal bit width and $g$ the group size, the number of weights sharing one $(s, z)$ pair stored in FP16, hence the $32/g$ overhead.
$b_{eff}$ is the effective bits per weight; with $b = 4$ and $g = 128$ this gives $b_{eff} = 4.25$ .
$N$ is the parameter count and $M$ the weight memory in bytes; for $N = 7 \times 10^{10}$ this is about 37 GB. Required condition: $s > 0$ must be computed from calibration data before any activation quantization is applied.

Bar chart of weight memory in gigabytes for a 70 billion parameter model: 280 GB in FP32, 140 GB in FP16 or BF16, 70 GB in INT8 and 37 GB in INT4 with group size 128, with a dashed line marking the 80 GB capacity of one GPU

Figure 2: Weight memory for a 70B model. Only the 4-bit variant fits one 80 GB GPU with room left for the KV cache, and because decoding is memory-bandwidth bound the same reduction shows up directly in tokens per second.

Property	FP16	BF16	INT8	INT4 (g=128)
Layout	1 sign, 5 exponent, 10 mantissa	1 sign, 8 exponent, 7 mantissa	8-bit integer plus a per-channel FP16 scale	4-bit integer plus a scale per 128 weights
Dynamic range	Max 65,504; underflows below 6e-5	Same as FP32, about 3.4e38	256 levels inside the calibrated range	16 levels per group; clipping is the main risk
70B weights	140 GB	140 GB	70 GB	About 37 GB at 4.25 effective bits
Typical role	Inference, and training with loss scaling	Default training and master-weight format	W8A8 serving at high batch, compute-bound prefill	Weight-only decode on capacity-limited GPUs
Accuracy cost	None measurable for inference	None measurable; the lost mantissa bits rarely matter	Under 1% with per-channel scales and outlier handling	A few tenths of perplexity on 70B, clearly worse below 7B
Hardware support	Tensor cores on every recent GPU	Ampere onward, plus TPUs	Roughly 2x the FP16 matmul throughput	No native 4-bit matmul on most GPUs; kernels dequantize in registers

August 16, 2026