Category: Hard

DL0111 SigLIP vs CLIP

What are the differences between SigLIP and CLIP?

Answer

CLIP and SigLIP share the same dual-encoder architecture, the same cosine-similarity scoring, and the same zero-shot classification recipe; the substantive difference is the training objective. CLIP optimizes a softmax InfoNCE loss in which each image’s logit row is normalized over every text in the batch and each text’s column over every image, so the loss contribution of a single pair depends on the entire batch, and the distributed implementation needs an all-gather plus a global normalization pass. SigLIP replaces this with a pairwise sigmoid loss that treats each of the $N^2$ cells of the similarity matrix as an independent binary decision (“is this a real pair?”), using a shared learnable temperature plus a learnable bias $b$ initialized to $-10$ to absorb the fact that only $N$ of the $N^2$ pairs are positive. Because nothing is normalized across the batch, the loss decomposes into $b \times b$ blocks, which lets SigLIP be implemented as a memory-efficient ring over devices and trained at batch sizes a softmax implementation cannot fit. Empirically the sigmoid loss is clearly better at small and moderate batch sizes, the two objectives converge above roughly 32k examples per step, and the million-example batch study SigLIP made possible showed that accuracy saturates long before that. That efficiency, plus the shape-optimized SoViT-400m/14 backbone released with it, is why SigLIP checkpoints became the default vision tower in many recent vision-language models such as PaliGemma.

(1) Loss Form: CLIP uses two symmetric softmax cross-entropies over the batch; SigLIP uses one binary logistic loss per pair, summed over the full $N \times N$ grid.
(2) Batch Coupling: softmax logits are only meaningful after a global normalization, so every device must see all $2N$ embeddings; sigmoid logits are independent, so blocks can be scored and reduced locally.
(3) Extra Bias Parameter: the sigmoid objective faces an extreme $N^2 - N$ negatives against $N$ positives imbalance, which the learnable bias fixes by starting every logit near “not a pair”.
(4) Batch-Size Behavior: sigmoid wins at 4k to 16k, matches softmax above roughly 32k, and both saturate rather than improving toward one million.
(5) Everything Else Is Unchanged: image tower, text tower, unit-norm embeddings, and prompt-based zero-shot evaluation are identical, so SigLIP is effectively a drop-in replacement at the loss layer.
(6) Practical Differences In The Checkpoints: public English SigLIP models use a 32k SentencePiece vocabulary with a 64-token text context (the original paper’s initial experiments used 16, but the released checkpoints increased it to 64), shorter than CLIP’s 77-token BPE context, which matters when captions are long.

Two 4 by 4 image-text similarity matrices. On the left, arrows show softmax normalization along every row and every column, coupling all pairs. On the right, each cell is an independent binary label scored by a sigmoid with learnable temperature and bias.

Figure 1: Both objectives see the same similarity matrix. CLIP normalizes each row and column, so a single logit’s gradient depends on the whole batch; SigLIP scores each cell as an independent binary label with a shared temperature and bias, which removes the coupling entirely.

Mathematical Formulation:
$s_{ij} = t\, x_i^{\top} y_j$
$\mathcal{L}_{\mathrm{i2t}} = -\frac{1}{N}\sum_{i} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$
$\mathcal{L}_{\mathrm{t2i}} = -\frac{1}{N}\sum_{j} \log \frac{e^{s_{jj}}}{\sum_{i} e^{s_{ij}}}$
$\mathcal{L}_{\mathrm{CLIP}} = \tfrac{1}{2}\left(\mathcal{L}_{\mathrm{i2t}} + \mathcal{L}_{\mathrm{t2i}}\right)$
$\mathcal{L}_{\mathrm{SigLIP}} = \frac{1}{N}\sum_{i}\sum_{j}\log\left(1 + e^{-z_{ij}(s_{ij} + b)}\right)$

Where:

$x_i$ and $y_j$ are the unit-norm image and text embeddings, so $x_i^{\top} y_j$ is a cosine similarity in $[-1, 1]$ .
$s_{ij}$ is the scaled logit and $t = \exp(t')$ is the learnable temperature, parameterized in log space in both methods.
$i, j \in \{1, \ldots, N\}$ index the global batch; the diagonal $i = j$ holds the $N$ true pairs and the off-diagonal holds $N^2 - N$ negatives.
$z_{ij} = +1$ for a matched pair and $z_{ij} = -1$ otherwise, which is the binary label the sigmoid loss regresses on.
$b$ is the learnable bias unique to SigLIP, with the required initialization $b = -10$ and $t' = \log 10$ ; CLIP has no analogue because its softmax is shift-invariant.

The engineering consequence is the part interviewers usually probe. In a softmax implementation the batch is a single indivisible unit: after the all-gather each of the $D$ devices holds all $2N$ embeddings and materializes an $N \times b$ logit slab, and the row and column sums must be reduced across devices before any gradient exists. SigLIP instead keeps each device’s image chunk local and passes text chunks around a ring, accumulating loss from one $b \times b$ block at a time, so peak logit memory drops by a factor of $D$ and no cross-device normalization is needed. This is exactly what allowed the authors to sweep batch size up to one million and demonstrate saturation, and it is also why the SigLiT variant, which locks a pretrained image tower and trains only the text side, reached 84.5% ImageNet zero-shot accuracy in two days on four TPUv4 chips. The trade-off is two extra hyperparameters to get right: a badly initialized bias makes the first thousands of steps a wasted fight against the negative-pair prior.

Top panel: four devices each holding a local chunk feed into an all-gather box where every device holds all N embeddings for a global softmax. Bottom panel: four devices keep their image chunk local while text chunks rotate around a ring, scoring one b by b block per step.

Figure 2: The softmax loss forces an all-gather and a global normalization, so each device carries an $N \times b$ logit slab. The sigmoid loss decomposes, so a ring of $D$ steps covers every pair while only $b \times b$ logits exist at once.

Property	SigLIP (sigmoid loss)	CLIP (softmax InfoNCE)
Objective	One binary logistic loss per image-text pair over the full grid	Two symmetric cross-entropies over row-wise and column-wise softmax
Batch coupling	None; the loss is a sum of independent terms	Global; every logit is normalized against the whole batch
Extra parameters	Learnable temperature plus a learnable bias, initialized to log 10 and -10	Learnable temperature only; a bias would cancel in the softmax
Distributed cost	Chunked ring; peak logit memory b x b per device, no normalization reduce	All-gather of all embeddings plus an N x b logit slab per device
Small batch (4k to 16k)	Clearly stronger zero-shot accuracy at equal examples seen	Degrades noticeably; the normalization has few negatives to work with
Very large batch	Feasible up to one million, but accuracy saturates near 32k	Comparable above roughly 32k, but memory-bound before that
Text context	64 tokens (16 in the original paper), 32k SentencePiece in the English releases	77 tokens, 49k byte-pair vocabulary
Typical role today	Default frozen vision tower for many VLMs, notably SoViT-400m/14	Legacy ecosystem: diffusion text conditioning, CLIPScore, many distilled models

August 17, 2026

DL0109 Mixture-of-Experts MoE Routing

Explain mixture-of-experts (MoE) and compare Sparse Mixture-of-Experts (MoE) routing strategies, as used in models like Mixtral and DeepSeek-V3.

Answer

A sparse mixture-of-experts layer replaces one feed-forward block with $N$ parallel feed-forward blocks (the experts) plus a tiny linear router, and sends each token to only $k \ll N$ of them. This is conditional computation: parameter count grows with $N$ while the FLOPs per token grow only with $k$ , so Mixtral 8x7B holds 46.7B parameters but activates about 12.9B per token, and DeepSeek-V3 holds 671B while activating 37B. The interesting engineering is not the experts, which are ordinary MLPs, but the routing strategy, because a learned router is free to collapse onto a few favorite experts and then most of the capacity you paid for is never trained. The dominant strategy is token-choice top-k: every token picks its own $k$ experts, each expert has a fixed buffer set by a capacity factor, and tokens that arrive at a full expert are dropped through the residual connection. The main alternatives invert or remove that choice: expert-choice routing lets each expert select its top tokens (perfect load balance, but some tokens get no expert), hash layers assign tokens deterministically with no learned router at all, and BASE layers solve a global linear assignment per batch. The trade-off axis is always the same, namely how much balance you buy and what you pay for it in dropped tokens, extra loss terms, batch dependence, and all-to-all communication.

(1) Parameters Decouple From FLOPs: with $N=8, k=2$ a token touches 25% of the expert weights, so quality scales with total parameters while latency scales with active parameters.
(2) The Router Is One Matrix: a single $d \times N$ projection followed by top-k selection and a renormalized softmax over the selected logits, which is why gradients reach the router only through the chosen experts.
(3) Load Balancing Is The Central Problem: Switch-style training adds an auxiliary balance loss (typically $\alpha = 0.01$ ) and a router z-loss to keep logits small and the assignment spread out.
(4) Capacity Factor Controls Token Dropping: a buffer of $c \cdot T \cdot k / N$ slots per expert means $c$ near 1.0 is cheap but discards tokens whenever the router skews.
(5) Routing Families Differ In Who Chooses: token-choice, expert-choice, deterministic hashing, and global assignment sit on a spectrum from fully learned and imbalanced to fully balanced and inflexible.
(6) Serving Cost Is Memory, Not Compute: every expert must be resident in HBM even though each token uses $k$ of them, and expert parallelism adds two all-to-all collectives per MoE layer.

Two-panel bipartite diagram: on the left, token-choice top-2 routing where four tokens each select two experts, expert one is oversubscribed and two edges are dropped at capacity; on the right, expert-choice routing where each expert selects its top two tokens, giving perfect expert load but leaving one token with no expert

Figure 1: The two families differ in the direction of selection. Token-choice top-k guarantees every token gets $k$ experts but not that experts get equal load, so overflow is dropped at capacity; expert-choice guarantees equal expert load but not that every token is served.

Token-choice top-k stays the production default for autoregressive language models for one structural reason: it is a per-token function, so the routing decision for token $t$ does not depend on any other token in the batch. Expert-choice and BASE layers both rank or match tokens against each other, which makes the forward pass batch-dependent, breaks the causal guarantee during teacher-forced training, and cannot be reproduced at decode time when the batch is a single token. Hash layers show how much of MoE’s benefit comes from capacity rather than from clever routing, since a fixed hash of the token ID is perfectly balanced by construction and still recovers a large part of the gain, but it can never learn semantic specialization. In practice the fix for token-choice imbalance is not to abandon it: DeepSeek-V3 keeps token-choice top-8 over 256 fine-grained experts, adds one always-on shared expert to absorb common knowledge, restricts each token to experts on at most 4 nodes to bound communication, and replaces the auxiliary loss with a per-expert bias that is nudged up or down to equalize load.

Mathematical Formulation:
$h(x) = W_r x \in \mathbb{R}^{N}$
$\mathcal{T}(x) = \mathrm{TopK}(h(x), k)$
$g_i(x) = \frac{\exp(h_i)}{\sum_{j \in \mathcal{T}} \exp(h_j)}$
$y = x + \sum_{i \in \mathcal{T}} g_i(x)\, E_i(x)$
$\mathcal{L}_{\mathrm{bal}} = \alpha N \sum_{i=1}^{N} f_i P_i$
$C = \left\lceil \frac{c\, T\, k}{N} \right\rceil$

Where:

$x \in \mathbb{R}^{d}$ is the token hidden state and $y$ the MoE layer output, written with the residual path that a dropped token falls back to.
$W_r \in \mathbb{R}^{N \times d}$ is the router, $h(x)$ its logits, and $E_i$ the $i$ -th expert MLP; $N$ is the expert count and $k$ the number kept.
$\mathcal{T}(x)$ is the selected index set and $g_i$ the gate weight, renormalized over the selected logits only so the weights sum to 1.
$f_i$ is the fraction of tokens in the batch dispatched to expert $i$ (piecewise constant, no gradient) and $P_i$ the mean router probability for expert $i$ (differentiable), so $\mathcal{L}_{\mathrm{bal}}$ pushes probability mass away from overloaded experts.
$\alpha$ is the balance-loss coefficient, commonly $\alpha = 0.01$ ; the product $N \sum_i f_i P_i$ equals 1 under a perfectly uniform assignment and grows toward $N$ under total collapse.
$T$ is the tokens per device, $c$ the capacity factor (usually $1.0 \leq c \leq 1.25$ in training, larger at eval), and $C$ the per-expert buffer; any token beyond $C$ is dropped.

Line chart of the fraction of routed tokens dropped versus capacity factor from 1.0 to 2.0 for eight experts under uniform, mildly skewed, and heavily skewed router distributions; the uniform curve is flat at zero while the heavily skewed curve stays above twenty percent even at capacity factor two

Figure 2: Token dropping is a joint function of the capacity factor and the router’s skew. Under a balanced router, $c = 1.0$ drops nothing; under a collapsed router, even doubling capacity leaves a quarter of the assignments discarded, which is why the balance loss matters more than the buffer size.

Architecture diagram of a sparse mixture-of-experts layer showing the data flow from input token through a linear router and top-k softmax selection to two selected expert FFNs out of N, with the remaining N-k non-selected experts shown folded in a dashed box, then a weighted sum with gate weights and a residual connection producing the output

Figure 3: The full MoE layer in one picture. A single linear router produces logits over all $N$ experts, the top-k selection picks only $k$ of them, and the selected expert outputs are combined with renormalized gate weights and added back through the residual connection. Non-selected experts stay resident in memory but consume zero FLOPs for this token, which is why total parameters scale with $N$ while active compute scales with $k$ .

Property	Token-choice top-k	Expert-choice	Hash / global assignment
Who selects	Each token picks its k highest-scoring experts	Each expert picks its C highest-scoring tokens	A fixed hash of the token ID, or a linear-assignment solver over the batch
Load balance	Not guaranteed; needs an auxiliary loss or a bias correction	Exact by construction, every expert receives C tokens	Exact by construction, with no learned router to collapse
Failure mode	Overflow tokens are dropped to the residual and get no expert compute	A token can be selected by zero experts, and tokens compete across the batch	No semantic specialization (hash) or expensive, batch-coupled solves (BASE)
Per-token independence	Yes; identical decision at batch size 1 and at batch size 1M	No; ranking couples tokens, so causal decoding does not match training	Yes for hashing, no for assignment-based methods
Where it is used	GShard, Switch (k=1), Mixtral (k=2), DeepSeek-V3 (k=8), OLMoE	Encoder-style and vision MoE, and research settings with full-sequence visibility	Baselines and ablations that isolate capacity from learned specialization

August 17, 2026

DL0108 PPO vs DPO vs GRPO

Explain the architectural and mathematical differences between PPO, DPO (Direct Preference Optimization), and GRPO (Group Relative Policy Optimization).

Answer

All three optimize the same underlying target, maximize a preference-derived reward while staying close to a frozen reference policy, and they differ in how many networks must be resident, how the advantage is estimated, and whether the training data is sampled from the current policy. PPO is the full actor-critic loop used in InstructGPT-style RLHF: it keeps four networks (trained policy, frozen reference, frozen reward model, trained critic), samples rollouts on-policy, and updates with a clipped importance ratio against token-level GAE advantages produced by the critic. DPO deletes the RL loop entirely by inverting the closed-form solution of the KL-constrained objective: the implied reward is $\beta \log(\pi_\theta / \pi_{ref})$ , the partition function cancels inside a Bradley-Terry pairwise likelihood, and what remains is a supervised binary-classification loss on fixed $(y_w, y_l)$ pairs with only two networks and no sampling. GRPO keeps online sampling and the clipped surrogate but removes the critic: for each prompt it draws a group of $G$ completions, uses the group’s mean reward as the baseline, and z-scores the rewards to get one scalar advantage that is broadcast to every token of its completion. So the axis is not “better versus worse” but which piece of machinery you are willing to pay for: PPO buys fine-grained credit assignment with a learned value function, DPO buys simplicity by giving up exploration, and GRPO buys on-policy learning with a Monte Carlo baseline.

(1) Networks Resident: PPO needs policy, reference, reward model, and critic; GRPO drops the critic; DPO drops both the critic and the reward model.
(2) Advantage Estimation: PPO uses GAE over a learned $V_\psi$ , GRPO uses a group-relative z-score of sequence rewards, and DPO never forms an advantage at all, only a reward margin between two responses.
(3) On-Policy Versus Offline: PPO and GRPO resample from the current policy every iteration, so the clip and importance ratio are meaningful; DPO trains on a static dataset and is therefore exposed to distribution shift.
(4) KL Control: PPO and GRPO add an explicit KL penalty (GRPO commonly uses the low-variance k3 estimator), while DPO’s KL constraint is baked into the log-ratio parameterization and controlled solely by $\beta$ .
(5) Credit Granularity: only PPO assigns different advantages to different tokens; GRPO gives every token in a completion the same scalar, and DPO gives a whole-sequence gradient.
(6) Where Each Fits: verifiable rewards from a checker or unit test favor GRPO, a cheap single-pass alignment on collected preferences favors DPO, and a nuanced learned reward model with long generations favors PPO.

Three side-by-side data-flow panels: PPO with prompt, trained policy, rollout, frozen reward model plus trained critic, and GAE advantage; DPO with a static preference pair scored by the trained policy and frozen reference into a logistic loss; GRPO with a prompt, trained policy, a group of G rollouts, a frozen verifier, and a group z-scored advantage

Figure 1: The three objectives differ mainly in what sits between the policy and the loss: PPO inserts a reward model plus a trained critic, GRPO replaces the critic with a group of sampled rollouts, and DPO removes the sampling stage so the frozen reference is the only extra network.

DPO’s derivation is what makes the contrast precise. The KL-constrained bandit objective has the closed-form optimum $\pi^*(y \mid x) \propto \pi_{ref}(y \mid x) \exp(r(x,y)/\beta)$ ; solving for the reward gives $r = \beta \log(\pi^*/\pi_{ref}) + \beta \log Z(x)$ , and because the Bradley-Terry likelihood depends only on reward differences for the same prompt, the intractable partition function $Z(x)$ cancels. The reward model therefore never has to be materialized, since its optimal policy is the object you were going to train anyway. The price is that this equivalence is exact only when the preference pairs come from $\pi_{ref}$ ; on off-policy pairs the loss can raise the margin while pushing down the probability of the chosen response as well, the failure mode usually called likelihood displacement. GRPO takes the opposite trade, keeping the on-policy ratio and clip but swapping the critic’s learned variance reduction for a Monte Carlo baseline over $G$ samples of the same prompt, which is cheap and well-behaved when the reward is a verifiable 0/1 signal from a math checker or unit test. That is exactly the regime DeepSeek used when introducing GRPO for DeepSeekMath and then scaling it for R1.

Mathematical Formulation:
$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{old}(a_t \mid s_t)}$
$\tilde r_t = \mathrm{clip}(r_t, 1 - \epsilon, 1 + \epsilon)$
$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t A_t,\ \tilde r_t A_t)\right]$
$A_t = \sum_{l \geq 0} (\gamma \lambda)^l \delta_{t+l}$
$\delta_t = R_t + \gamma V_\psi(s_{t+1}) - V_\psi(s_t)$
$\hat r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{ref}(y \mid x)}$
$\Delta = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)$
$\mathcal{L}_{DPO} = -\mathbb{E}\left[\log \sigma(\Delta)\right]$
$A_i = \frac{R_i - \mathrm{mean}(R_{1:G})}{\mathrm{std}(R_{1:G})}$
$\mathcal{L}_{GRPO} = -\mathbb{E}\left[\min(r_i A_i,\ \tilde r_i A_i)\right]$
$\qquad + \beta \, \mathbb{D}_{KL}\left[\pi_\theta \,\|\, \pi_{ref}\right]$

Where:

$\pi_\theta$ is the trained policy, $\pi_{ref}$ the frozen reference (normally the SFT checkpoint), and $\pi_{old}$ the policy that generated the current batch of rollouts.
$s_t$ is the prompt-plus-prefix state and $a_t$ the token emitted at position $t$ ; $R_t$ is the per-step reward, which in RLHF is usually nonzero only at the final token.
$V_\psi$ is the learned critic, $\delta_t$ the TD residual, and $A_t$ the GAE advantage with discount $\gamma$ and trace decay $\lambda$ .
$\epsilon$ is the clip half-width (typically 0.1 to 0.2) and $\beta$ is the KL coefficient in PPO and GRPO, or the implicit-reward temperature in DPO (typically 0.01 to 0.5).
$y_w$ and $y_l$ are the preferred and rejected responses for prompt $x$ , $\sigma$ is the logistic function, and $\hat r_\theta$ is the reward implied by the policy itself.
$i \in \{1, \ldots, G\}$ indexes the group of completions sampled per prompt (commonly $G = 8$ to $64$ ), $R_i$ is its sequence-level reward, and $A_i$ is shared by every token of completion $i$ .
Required initial condition in all three: $\pi_\theta$ is initialized from $\pi_{ref}$ , otherwise the KL term and the log-ratio reward have no meaningful anchor.

Stacked bar chart of resident bf16 weight memory for a 7B policy: PPO 56 GB with 28 GB trained, GRPO with a reward model 42 GB with 14 GB trained, GRPO with a rule verifier 28 GB, and DPO 28 GB with 14 GB trained

Figure 2: Counting only weights at 7B scale in bf16, PPO holds four copies and trains two of them, so its Adam state is roughly double GRPO’s; a rule-based verifier removes the reward model entirely, making GRPO as light as DPO in weight memory while still sampling online.

Property	PPO	DPO	GRPO
Networks resident	Policy, reference, reward model, critic (two trained)	Policy and frozen reference (one trained)	Policy, reference, reward model or verifier (one trained)
Data source	Fresh on-policy rollouts from prompts	Static offline preference pairs, no generation	Groups of G on-policy rollouts per prompt
Baseline for the gradient	Learned value function, GAE per token	The rejected response acts as the baseline	Group mean reward, normalized by group std
Credit granularity	Per token, values differ along the sequence	Whole sequence, one margin per pair	One scalar per completion, broadcast to all its tokens
KL control	Explicit penalty or reward shaping, often adaptive	Implicit in the log-ratio, tuned only through beta	Explicit term with the k3 estimator, sometimes dropped
Cost per update	Highest: generation plus four forward passes plus critic training	Lowest: two forward passes on cached text	Generation dominates, G completions per prompt
Main failure mode	Critic instability and reward hacking, many coupled hyperparameters	Off-policy shift and likelihood displacement on unseen responses	Degenerate groups with zero reward variance, length and difficulty bias

August 17, 2026

DL0107 Multi-head Latent Attention MLA

Explain Multi-head Latent Attention (MLA), as used in DeepSeek-V2 and DeepSeek-V3.

Answer

Multi-head Latent Attention (MLA) attacks the KV-cache bottleneck of autoregressive decoding by low-rank joint compression of keys and values instead of by deleting KV heads. Each token is projected down to a single shared latent vector $c_t^{KV}$ of dimension $d_c = 512$ , and that latent is the only thing cached; per-head keys and values are reconstructed on the fly by up-projection matrices $W_i^{UK}$ and $W_i^{UV}$ , so every head still gets its own distinct K and V, unlike GQA and MQA which force heads to share. Because RoPE does not commute with the up-projection, MLA adds a small decoupled RoPE key of dimension $d_h^R = 64$ that is shared across heads and cached directly, giving a total cache of $512 + 64 = 576$ values per token per layer against $2 \cdot 128 \cdot 128 = 32{,}768$ for the equivalent MHA configuration, a roughly 57x reduction. The reason this is not simply lossy compression is the absorption trick: at inference the up-projections fold into the query and output projections, so the model never materializes the full keys and values, and the attention score becomes a bilinear form on the two latents. The result reported in DeepSeek-V2 is unusual for a cache-reduction technique, since MLA matched or beat full MHA on benchmarks while caching about 1/57 as much, which is why DeepSeek-V3 kept it unchanged at 61 layers and 128 heads.

(1) Joint Low-Rank KV Compression: one down-projection produces a single latent per token that serves all heads and both K and V, so cache cost is $(d_c + d_h^R) \cdot l$ rather than $2 n_h d_h l$ .
(2) Head Diversity Is Preserved: per-head up-projections mean MLA reduces cache without reducing the number of effective K/V heads, which is the quality cost GQA and MQA pay.
(3) Decoupled RoPE Path: position information travels in a separate small key dimension because applying RoPE inside the compressed path would make the up-projection position-dependent and destroy absorption.
(4) Absorption At Inference: $W_i^{UK}$ folds into $W_i^{UQ}$ and $W_i^{UV}$ folds into $W^O$ , so decoding reads only the cached latent and never reconstructs full K or V tensors.
(5) Query Compression Is A Separate Win: queries are also routed through a latent of dimension 1536, which cuts training activation memory but has no effect on the KV cache.
(6) Memory-Bandwidth Trade For FLOPs: the absorbed score is a dot product over 576 dimensions instead of 128, so MLA raises arithmetic intensity, exactly the right direction for memory-bound batch decoding.

MLA architecture diagram: the hidden state is projected into a query latent, a shared 512-dimensional KV latent that is cached, and a 64-dimensional decoupled RoPE key that is also cached; per-head keys and values are reconstructed by up-projections and attention runs over a 192-dimensional concatenated head

Figure 1: Only the two boxes inside the dashed region enter the cache: the 512-dimensional KV latent and the 64-dimensional decoupled RoPE key. Per-head keys and values are reconstructed from the latent, and at inference those up-projections are absorbed so they are never computed explicitly.

The decoupled RoPE dimension is the part candidates most often miss, and it follows from a single algebraic fact. The score between query $t$ and key $j$ in the compressed path is $(W_i^{UQ} c_t^{Q})^\top (W_i^{UK} c_j^{KV})$ , which lets you precompute the constant matrix $M_i = (W_i^{UQ})^\top W_i^{UK}$ . If RoPE were applied to $k_j^C$ , that matrix would become $(W_i^{UQ})^\top R_{t-j} W_i^{UK}$ , which depends on the relative position and therefore differs for every cached token, so nothing can be precomputed and every prefix key would have to be rebuilt at each step. MLA sidesteps this by carrying position in a separate 64-dimensional key that is cached in rotated form and shared by all heads, so the query concatenates a 128-dimensional content part with a 64-dimensional positional part, giving an effective head dimension of 192 during training. A practical consequence is that the value head dimension (128) differs from the query and key dimension (192), which is why MLA needs purpose-built kernels such as FlashMLA rather than a stock FlashAttention call.

Mathematical Formulation:
$c_t^{KV} = W^{DKV} h_t$
$k_{t,i}^C = W_i^{UK} c_t^{KV}$
$v_{t,i}^C = W_i^{UV} c_t^{KV}$
$k_t^R = \mathrm{RoPE}(W^{KR} h_t)$
$q_{t,i} = [\,q_{t,i}^C ; q_{t,i}^R\,]$
$k_{t,i} = [\,k_{t,i}^C ; k_t^R\,]$
$(q_{t,i}^C)^\top k_{j,i}^C = (c_t^{Q})^\top M_i c_j^{KV}$
$M_i = (W_i^{UQ})^\top W_i^{UK}$
$d_c + d_h^R = 512 + 64 = 576$

Where:

$c_t^{KV} \in \mathbb{R}^{d_c}$ is the cached KV latent and $k_t^R \in \mathbb{R}^{d_h^R}$ the cached decoupled RoPE key; together they are the entire per-token cache.
$h_t \in \mathbb{R}^{d}$ is the layer input ( $d = 7168$ in DeepSeek-V3), and $c_t^{Q}$ is the query latent of dimension 1536.
$t$ indexes the current token, $j$ ranges over cached prefix positions with $j \leq t$ , and $i \in \{1,\ldots,n_h\}$ indexes the $n_h = 128$ heads.
$d_h = 128$ is the content head dimension, $d_h^R = 64$ the positional dimension, so the concatenated query and key have dimension 192 while values keep 128.
$W^{DKV}, W^{DQ}$ are down-projections, $W_i^{UK}, W_i^{UV}, W_i^{UQ}$ up-projections, $W^{KR}, W^{QR}$ the RoPE-path projections, and $W^O$ the output projection.
$M_i$ is the absorbed score matrix, valid only because no rotation sits between the two up-projections; $\mathrm{RoPE}$ applies the rotary transform, and $[\,\cdot;\cdot\,]$ denotes concatenation.

Log-scale bar chart of KV cache per token per layer: MHA with 128 KV heads at 32768 values and 524 GB at 128K context, GQA with 8 groups at 2048 values and 32.8 GB, MLA at 576 values and 9.2 GB, MQA at 256 values and 4.1 GB

Figure 2: Totals assume 61 layers, bf16, and a 128K-token context. MLA lands between MQA and 8-group GQA on cache size, but reaches that point by compressing rather than by removing KV heads, which is why it does not carry their quality penalty.

Property	MLA	GQA (8 groups)	MQA
What is cached	One 512-dim latent plus a 64-dim RoPE key	Full K and V for 8 shared KV heads	Full K and V for a single KV head
Values per token per layer	576	2,048	256
Effective K/V per head	Distinct for all 128 heads, reconstructed by up-projection	16 query heads share one K and V	All 128 query heads share one K and V
Decode compute per key	Dot product over 576 dims after absorption, higher FLOPs	128 dims, plus broadcast of shared heads	128 dims, lowest FLOPs
Kernel and tooling support	Needs custom kernels (FlashMLA) because query/key is 192 and value is 128	Supported by every stock attention kernel	Supported by every stock attention kernel
Tensor-parallel behavior	Latent is shared by all heads, so it is replicated per rank	KV heads shard cleanly across ranks	Single KV head must be replicated
Quality relative to MHA	Matched or slightly better in the DeepSeek-V2 ablations	Small but measurable degradation	Largest degradation, especially on long context

August 17, 2026

DL0104 Score-Based Diffusion

What is the score-based / score-matching interpretation of diffusion models, as used in systems like Stable Diffusion?

Answer

The score-based view says a diffusion model never learns a density; it learns the score, the gradient of the log density $s(x) = \nabla_x \log p(x)$ . This is attractive because the gradient annihilates the intractable normalizing constant: if $p(x) = e^{-E(x)}/Z$ then $\nabla_x \log p(x) = -\nabla_x E(x)$ with no $Z$ anywhere, so an unnormalized energy model becomes trainable. Direct score matching still needs the intractable $\nabla_x \log p_{\text{data}}(x)$ , and the fix is denoising score matching: perturb the data with a known Gaussian kernel, and the score of the perturbed conditional is available in closed form, which reduces the loss to predicting the noise that was added. Because a single small noise level leaves the score undefined off the data manifold and unreliable in low-density regions, the model is trained over a continuum of noise levels, which turns into the forward SDE of Song et al. and makes sampling the simulation of a reverse-time SDE or its deterministic probability-flow ODE. The punchline is that DDPM’s noise-prediction network is the same object in different clothing: $s_\theta(x_t,t) = -\epsilon_\theta(x_t,t)/\sigma_t$ , so the discrete-time denoising story and the continuous-time score story are two parameterizations of one model.

(1) Model The Gradient, Not The Density: learning $\nabla_x \log p(x)$ removes the partition function, so the network output is an unconstrained vector field of the same shape as the data.
(2) Denoising Score Matching Makes It Tractable: Vincent’s identity shows that regressing onto $\nabla_{x_t}\log p(x_t \mid x_0)$ has the same minimizer as matching the true marginal score, and that target is just the scaled added noise.
(3) Multiple Noise Scales Are Essential: real data lies near a low-dimensional manifold where the score is ill-defined, and large noise levels smooth the density so the vector field is informative everywhere and connects separated modes.
(4) Sampling Is Numerical Integration: annealed Langevin dynamics, the reverse SDE, and the probability-flow ODE all consume only the learned score; the ODE additionally gives exact likelihoods and deterministic, invertible latents.
(5) Unifies VE And VP Diffusions: NCSN’s variance-exploding noise ladder and DDPM’s variance-preserving chain are two discretizations of the same SDE family, differing in $f$ , $g$ , and the loss weighting $\lambda_t$ .

The geometric picture is worth holding onto. The score at a point is the direction of steepest increase in log density, so the learned field points from wherever you are toward nearby high-probability regions, and its magnitude grows as you move into the tails. Sampling then means dropping a Gaussian sample somewhere in space and following that field while injecting just enough noise to keep the chain from collapsing onto a single mode. The reason this needs annealing is that a score trained only at tiny noise is accurate on the data manifold and near-arbitrary far from it, where every initial sample actually starts.

Contour plot of a two-mode density with an overlaid arrow field pointing toward the modes, and a jagged Langevin sampling path starting in the empty lower-left corner and ending inside the left mode

Figure 1: The learned score is a vector field pointing toward high-density regions; a Langevin chain follows it from a random start and ends up distributed according to $p(x)$ , never evaluating the density itself.

Mathematical Formulation:
$s_\theta(x,t) \approx \nabla_x \log p_t(x)$
$x_t = \alpha_t x_0 + \sigma_t \epsilon$
$\nabla_{x_t} \log p(x_t \mid x_0) = -\frac{\epsilon}{\sigma_t}$
$\mathcal{L}(\theta) = \mathbb{E}\left[ \lambda_t \left\| s_\theta(x_t,t) + \frac{\epsilon}{\sigma_t} \right\|^2 \right]$
$dx = f(x,t)\,dt + g(t)\,dw$
$dx = \left[ f - g^2 s_\theta \right] dt + g\,d\bar{w}$
$dx = \left[ f - \tfrac{1}{2} g^2 s_\theta \right] dt$
$s_\theta(x_t,t) = -\frac{\epsilon_\theta(x_t,t)}{\sigma_t}$

Where:

$s_\theta(x,t)$ is the network output, a vector with the same shape as $x$ , approximating the score of the perturbed marginal $p_t$ .
$x_0$ is a clean data point, $x_t$ its noisy version, and $\epsilon \sim \mathcal{N}(0,I)$ the Gaussian draw.
$\alpha_t$ and $\sigma_t$ define the noise schedule; variance-preserving keeps $\alpha_t^2 + \sigma_t^2 = 1$ , variance-exploding fixes $\alpha_t = 1$ and grows $\sigma_t$ .
$t \in [0,T]$ indexes the noise level, with $p_T$ essentially pure Gaussian noise; $\lambda_t$ is the loss weighting, and the choice $\lambda_t = \sigma_t^2$ turns the objective into the plain noise-prediction MSE.
$f(x,t)$ is the drift and $g(t)$ the diffusion coefficient of the forward SDE; $w$ is a Wiener process and $\bar{w}$ its reverse-time counterpart, so the second SDE is integrated from $t = T$ down to $t = 0$ .
The final line is the reparameterization identity linking the score view to DDPM’s $\epsilon_\theta$ ; required initial condition for sampling: $x_T \sim \mathcal{N}(0, \sigma_T^2 I)$ for the VE case.

Three panels showing the same two-mode density smoothed by increasing Gaussian noise, with score arrows that are confined near the modes at small noise and fill the whole plane at large noise

Figure 2: Why a ladder of noise levels is required: at $\sigma = 0.05$ the score carries no usable signal away from the data, while at $\sigma = 1.5$ the smoothed field points inward from everywhere and bridges the two modes.

Property	Score SDE (VE, NCSN)	DDPM (VP, noise-pred)	Flow matching
Network output	Score, magnitude scales like 1/sigma_t	Unit-variance noise estimate	Velocity field of a probability path
Corruption	Add noise, variance grows to data scale	Shrink signal, keep total variance at 1	Straight interpolation between data and noise
Loss	Weighted denoising score matching	Same loss with weighting sigma_t squared	Regression onto the conditional velocity
Sampler	Annealed Langevin or predictor-corrector	Ancestral chain, or DDIM as an ODE	ODE solver on a near-straight path
Exact likelihood	Yes, via the probability-flow ODE	Only an ELBO in discrete time	Yes, continuous normalizing flow change of variables

August 16, 2026

DL0102 Diffusion vs GAN vs VAE

Explain the fundamental principles of Diffusion Models, and compare their pros and cons with GANs, VAEs, and Flow-based models.

Answer

A diffusion model defines a fixed forward process that gradually corrupts data into Gaussian noise over $T$ steps, then learns the reverse process that removes a little noise at a time. The forward chain has no parameters and admits a closed form, so training reduces to a plain regression: sample a clean image $x_0$ , sample a timestep $t$ and noise $\epsilon$ , build the noisy state $x_t$ , and ask the network to predict $\epsilon$ . This denoising objective is a weighted variational bound on the log-likelihood and is also equivalent to score matching, which is why the same network can be plugged into an SDE or ODE sampler. The practical consequence is that diffusion buys stable training and excellent mode coverage at the price of many network evaluations per sample, whereas a GAN generates in one forward pass but fights an unstable min-max game, a VAE trains stably but produces blurry samples under a Gaussian decoder, and a normalizing flow gives exact likelihoods but pays for invertibility with architectural constraints and parameter count.

(1) Fixed Corruption, Learned Reversal: only the reverse direction has parameters, so there is no adversary and no discriminator to balance, which removes the main source of GAN training instability.
(2) Closed-Form Training Target: because $q(x_t \mid x_0)$ is Gaussian, any timestep can be sampled directly without simulating the chain, making each training step $O(1)$ in $T$ .
(3) Score Interpretation: the noise predictor is a rescaled estimate of $\nabla_{x_t}\log q(x_t)$ , which links diffusion to Langevin dynamics and to probability-flow ODE samplers such as DDIM.
(4) Sampling Is The Bottleneck: quality scales with the number of function evaluations, so a 1000-step DDPM sampler costs three orders of magnitude more compute per image than a GAN generator.
(5) Coverage Versus Speed: the likelihood-flavored objective penalizes dropped modes, which is exactly the failure GANs are known for, so the two families sit on opposite corners of the quality/diversity/speed trade-off.

Diagram of a diffusion chain: five states from a clean image to pure noise, with upper arrows showing the fixed forward noising process q and lower arrows showing the learned reverse denoiser p theta, plus a dashed arc marking the closed-form jump from x0 to any xt

Figure 1: The forward process is fixed and parameter-free, and the dashed arc is the closed form that lets training jump straight to any $x_t$ ; every arrow in the lower reverse path costs one network evaluation at sampling time.

The four families differ mostly in how they pay for tractability. A VAE keeps an explicit encoder and decoder and optimizes a lower bound, so the mismatch between the true posterior and the Gaussian approximation shows up as blur and posterior collapse. A normalizing flow refuses any approximation and computes exact likelihoods, but every layer must be invertible with a cheap Jacobian determinant, which caps expressiveness per parameter. A GAN drops likelihood entirely and optimizes a discriminator-defined divergence, which produces sharp samples in one step but offers no signal that the whole data distribution is covered. Diffusion sidesteps all three constraints by turning generation into a long sequence of easy denoising problems, and then spends compute at inference to pay for it.

Mathematical Formulation:
$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\, x_{t-1}, \beta_t I)$
$\bar{\alpha}_t = \prod_{s=1}^{t} (1 - \beta_s)$
$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$
$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon} [ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 ]$
$s_\theta(x_t, t) = -\frac{\epsilon_\theta(x_t, t)}{\sqrt{1 - \bar{\alpha}_t}}$

Where:

$x_0$ is a clean data sample and $x_t$ the noised state at step $t$ , with $x_T$ approximately standard Gaussian.
$\epsilon \sim \mathcal{N}(0, I)$ is the injected noise and $\epsilon_\theta$ the learned noise predictor with parameters $\theta$ .
$t \in \{1, \ldots, T\}$ is sampled uniformly during training, and $s$ indexes the product defining the cumulative signal retention.
$\beta_t$ is the noise schedule, typically small and increasing, and $\bar{\alpha}_t$ is the cumulative signal factor giving signal-to-noise ratio $\bar{\alpha}_t / (1 - \bar{\alpha}_t)$ .
$s_\theta$ is the implied score function, the quantity an SDE or probability-flow ODE sampler integrates.
Required initial condition: sampling starts from $x_T \sim \mathcal{N}(0, I)$ , which is only valid if the schedule drives $\bar{\alpha}_T$ to (near) zero.

Property	Diffusion	GAN	VAE	Normalizing Flow
Training objective	Denoising regression on a fixed forward process, no adversary	Min-max game against a discriminator	ELBO: reconstruction plus KL to the prior	Exact log-likelihood via change of variables
Network calls per sample	10 to 1000, or 1 to 4 after distillation	1	1	1
Sample fidelity	State of the art for images, video, and audio	Very sharp, still strong under a one-step budget	Blurry with a Gaussian decoder	Below GAN and diffusion at equal compute
Mode coverage	Broad, the objective penalizes missing mass	Mode collapse is the classic failure	Broad but over-smoothed	Broad, likelihood forbids ignoring regions
Density estimate	Bound or ODE-based estimate, not exact in closed form	None	Lower bound only	Exact and tractable
Architecture constraint	Free, any denoiser such as a U-Net or a DiT	Free	Free	Invertible layers with cheap Jacobian determinant
Typical failure	Slow sampling, schedule and guidance artifacts	Unstable training, collapsed diversity	Blur and posterior collapse	Very large models for modest sample quality

Triangle diagram of the generative learning trilemma with vertices high sample quality, fast sampling, and mode coverage; GAN sits on the quality-speed edge, VAE and normalizing flow on the speed-coverage edge, and diffusion on the quality-coverage edge

Figure 2: The generative learning trilemma: each family wins two corners and pays on the third, and most modern work on diffusion (distillation, consistency training, rectified flow) is an attempt to drag it toward the fast-sampling corner without losing coverage.

August 16, 2026

DL0101 GAN Loss Functions

What loss functions do the GAN generator and discriminator optimize?

Answer

The discriminator optimizes an ordinary binary cross-entropy objective: label real samples 1, generated samples 0, and minimize the negative log-likelihood of that two-class problem. The generator is defined by the same value function played in the opposite direction, so the textbook formulation is the minimax game $\min_{G} \max_{D} V(D,G)$ , but almost nobody implements the generator that way. Minimizing $\log(1 - D(G(z)))$ saturates exactly when the generator is bad, which is precisely when it needs signal, so Goodfellow’s original paper already proposed the non-saturating form: maximize $\log D(G(z))$ , or equivalently minimize $-\log D(G(z))$ . Both variants share the same fixed point, where the optimal discriminator is $D^{*}(x) = p_{r}(x)/(p_{r}(x)+p_{g}(x))$ and the outer objective reduces to the Jensen-Shannon divergence between data and model. In current practice the exact loss shape matters less than the regularizer attached to it, and the common defaults are the non-saturating logistic loss or the hinge loss combined with an R1 gradient penalty on real data.

(1) Discriminator Is Just A Classifier: its loss is standard BCE over a batch of half real and half fake samples, so any logit-based classification loss can be dropped in without changing the game structure.
(2) Saturating Vs Non-Saturating Generator: the two forms have identical optima but very different gradient magnitudes; the non-saturating loss keeps a large gradient when $D(G(z))$ is near 0.
(3) Optimal Discriminator Reveals A Divergence: substituting $D^{*}$ turns the value function into $2\,\mathrm{JSD}(p_{r} \,\|\, p_{g}) - \log 4$ , which is why a perfect discriminator gives no useful gradient on disjoint supports.
(4) Alternatives Change The Metric: LSGAN uses a squared error, WGAN replaces the classifier with a 1-Lipschitz critic estimating the Earth-Mover distance, and hinge loss caps the penalty once a margin is met.
(5) Regularization Dominates Stability: zero-centered gradient penalties such as R1 are what make training converge locally, and swapping loss families with the same regularizer usually moves FID far less than practitioners expect.

Two panel chart: left panel plots the saturating generator loss log of one minus D and the non-saturating loss minus log D against the discriminator score on a fake sample, right panel plots the magnitude of their gradients on a logarithmic axis showing the non-saturating gradient is far larger when the score is near zero

Figure 1: Early in training the discriminator confidently rejects fakes, so $D(G(z))$ sits near 0. At $D(G(z)) = 0.05$ the saturating loss supplies a gradient magnitude of 1.05 while the non-saturating loss supplies 20, about 19x more signal in exactly the regime where the generator is weakest.

Mathematical Formulation:
$\min_{G} \max_{D} V(D, G)$
$V(D,G) = \mathbb{E}_{x \sim p_{r}}[\log D(x)]$
$+ \mathbb{E}_{z \sim p_{z}}[\log (1 - D(G(z)))]$
$\mathcal{L}_{G}^{\mathrm{sat}} = \mathbb{E}_{z}[\log (1 - D(G(z)))]$
$\mathcal{L}_{G}^{\mathrm{ns}} = -\mathbb{E}_{z}[\log D(G(z))]$
$D^{*}(x) = \frac{p_{r}(x)}{p_{r}(x) + p_{g}(x)}$
$V(D^{*}, G) = 2\,\mathrm{JSD}(p_{r} \,\|\, p_{g}) - \log 4$

Where:

$V(D,G)$ is the value function; the discriminator maximizes it and the generator minimizes it, so $\mathcal{L}_{D} = -V$ is the batch-level binary cross-entropy actually coded up.
$x \sim p_{r}$ is a real sample and $D(x) \in [0,1]$ is the estimated probability that its input is real.
$z \sim p_{z}$ is a latent drawn from a fixed prior, usually $\mathcal{N}(0, I)$ ; $G(z)$ is the generated sample and $p_{g}$ the distribution it induces.
$\mathcal{L}_{G}^{\mathrm{sat}}$ is the literal minimax generator loss and $\mathcal{L}_{G}^{\mathrm{ns}}$ the non-saturating heuristic; they share the same optimum but differ in gradient scale by a factor of $(1 - D)/D$ .
$D^{*}$ is the optimal discriminator for a fixed $G$ , obtained by maximizing the integrand pointwise; it equals $1/2$ everywhere when $p_{g} = p_{r}$ .
$\mathrm{JSD}$ is bounded in $[0, \log 2]$ , so $V(D^{*},G)$ ranges over $[-\log 4, 0]$ and hits its global minimum $-\log 4 \approx -1.386$ only at $p_{g} = p_{r}$ .
Practical condition: $D^{*}$ is never reached, since the discriminator takes $k$ steps (typically $k = 1$ ) per generator step, and the whole derivation assumes optimization in function space rather than over network parameters.

The main alternatives keep the same alternating structure and change only the per-sample penalty. LSGAN replaces the log with a squared error, which pushes fake samples toward the decision boundary and stops the loss from flattening, at the cost of also penalizing samples that are already comfortably on the correct side. WGAN drops the sigmoid entirely: the critic maximizes $\mathbb{E}[D(x)] - \mathbb{E}[D(G(z))]$ under a 1-Lipschitz constraint, enforced in WGAN-GP by a two-sided penalty on the gradient norm at interpolated points, and the generator simply maximizes $\mathbb{E}[D(G(z))]$ . Hinge loss, the default in BigGAN and many diffusion-era discriminators, uses $\max(0, 1 - D(x))$ on reals and $\max(0, 1 + D(G(z)))$ on fakes, so a correctly classified sample beyond the margin contributes exactly zero gradient and the discriminator stops overfitting easy examples. Independently of the family chosen, the R1 penalty $\frac{\gamma}{2}\mathbb{E}_{x \sim p_{r}}[\|\nabla_{x} D(x)\|^{2}]$ is what turns the local dynamics from oscillatory into convergent.

Line chart of the discriminator loss contributed by a single fake sample as a function of its logit, comparing the smooth softplus curve of binary cross-entropy, the piecewise linear hinge loss that becomes exactly zero below minus one, and the unbounded linear Wasserstein critic loss

Figure 2: Loss contributed by one fake sample as a function of its logit $t$ . BCE uses $\log(1 + e^{t})$ and decays smoothly but never quite to zero, hinge is exactly flat once the sample clears the $-1$ margin, and the Wasserstein critic stays linear so it never saturates and must be constrained by a Lipschitz penalty.

Property	Non-saturating BCE	Hinge	WGAN-GP
Discriminator term on a fake	softplus of the logit, smooth and always positive	max(0, 1 + logit), exactly zero past the margin	the raw critic score, unbounded and linear
Generator term	minus log of the fake probability	minus the critic score, no hinge applied	minus the critic score
Behavior when D wins	gradient shrinks but stays usable via the non-saturating form	D stops learning from easy samples, which limits overfitting	score gap keeps growing, so the generator still gets signal
Extra cost per step	none beyond an optional R1 penalty, often lazily applied every 16 steps	same as BCE	a double backward pass for the penalty plus roughly 5 critic steps per generator step
Typical use	StyleGAN family with R1, and most adversarial auxiliary losses	BigGAN, SAGAN, and many feature-space discriminators	settings where a meaningful loss-versus-quality correlation is wanted

August 16, 2026

DL0098 CLIP Zero-Shot Classification

What is CLIP, and how does contrastive image-text training enable zero-shot classification?

Answer

CLIP (Contrastive Language-Image Pre-training) is a dual-encoder model: an image encoder (a ViT or ResNet) and a text encoder (a Transformer) each map their input to a vector in one shared embedding space, and the vectors are L2-normalized so their inner product is a cosine similarity. Training uses a batch of $N$ image-caption pairs scraped from the web (400M pairs for the original model), computes the full $N \times N$ similarity matrix, and applies a symmetric InfoNCE loss that pushes the $N$ diagonal entries up and the $N^2 - N$ off-diagonal entries down. Nothing in that objective mentions class labels, so the learned text encoder is a general classifier generator: at test time you write each candidate class as a sentence such as “a photo of a golden retriever”, encode it, and the resulting unit vectors act as the rows of a linear classifier weight matrix. Classification is then one matrix multiply followed by an argmax over cosine similarities, which is why a new label set costs a text-encoder forward pass rather than a training run. The best original model (ViT-L/14@336px) reaches 76.2% zero-shot top-1 on ImageNet, matching a supervised ResNet-101 with an identical 76.2% ImageNet top-1 that saw 1.28M labeled images, and it is far more robust under distribution shift.

(1) Two Encoders, One Space: each modality has its own encoder plus a linear projection into a shared $d$ -dimensional space; there is no cross-attention between them, so image and text embeddings can be computed and cached independently.
(2) Symmetric InfoNCE Over The Batch: the loss is cross-entropy applied twice, once along rows (image picks its caption) and once along columns (caption picks its image), averaged.
(3) Learned Temperature: the logit scale $\exp(\tau)$ is a trained scalar initialized to a temperature of 0.07 and clipped at 100, which controls how hard the softmax pushes against near-miss negatives.
(4) Prompts Become Classifier Weights: zero-shot inference replaces the learned classification head with text embeddings of class descriptions, so the label set is defined at inference time, not at training time.
(5) Batch Size Is Part Of The Objective: negatives come only from the current batch, so the original model trained at $N = 32768$ across many GPUs with the similarity matrix sharded.

Diagram of CLIP training: a batch of N images passes through an image encoder to unit-norm vectors, a batch of N captions passes through a text encoder to unit-norm vectors, and both feed an N by N cosine similarity matrix whose diagonal cells are the positive pairs

Figure 1: Each training step builds the full $N \times N$ similarity matrix; the $N$ diagonal entries are the true pairs and every other entry is an in-batch negative, which is why one step at $N = 32768$ supplies over a billion contrastive comparisons.

Three details separate a working CLIP from a broken one. Normalization is load-bearing: without it the model can shrink the loss by inflating embedding norms instead of improving alignment, and the learned temperature then has no fixed scale to calibrate against. Prompt wording matters more than people expect, because captions in the training data are sentences, not bare nouns: using “a photo of a {label}” instead of the raw class name adds about 1.3 points on ImageNet, and ensembling 80 prompt templates by averaging their normalized text embeddings adds roughly 3.5 points for almost no inference cost, since the averaged vectors are computed once and cached. Finally, label naming is part of the model: polysemous class names such as “boxer” (dog breed or athlete) or “crane” (bird or machine) must be disambiguated in the prompt, which is a form of engineering that has no analogue in a supervised classifier with integer labels.

Mathematical Formulation:
$z_i = f_{\theta}(I_i) / \|f_{\theta}(I_i)\|_2$
$t_j = g_{\phi}(T_j) / \|g_{\phi}(T_j)\|_2$
$s_{ij} = \exp(\tau)\, z_i^{\top} t_j$
$\mathcal{L}_{I} = -\frac{1}{N}\sum_{i} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$
$\mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{I} + \mathcal{L}_{T}\right)$
$\hat{y} = \arg\max_{c} \; z^{\top} t_c$

Where:

$z_i$ and $t_j$ are the unit-norm embeddings of image $I_i$ and caption $T_j$ , both in $\mathbb{R}^{d}$ with $d = 768$ for ViT-L/14.
$f_{\theta}$ is the image encoder plus its projection and $g_{\phi}$ the text encoder plus its projection; the two share no weights.
$s_{ij}$ is the scaled cosine similarity and $\exp(\tau)$ the learned logit scale; since $z_i^{\top} t_j$ lies in $[-1, 1]$ , the scale alone sets the sharpness of the softmax.
$i, j \in \{1, \ldots, N\}$ index the batch; $\mathcal{L}_{I}$ normalizes each row and $\mathcal{L}_{T}$ the same expression with the softmax taken over columns.
$\hat{y}$ is the zero-shot prediction for a query image with embedding $z$ , and $t_c$ is the (optionally prompt-ensembled and renormalized) text embedding of class $c$ ; the temperature drops out of the argmax.
Required initial condition: $\tau = \log(1/0.07)$ at step 0, clamped so that $\exp(\tau) \leq 100$ throughout training to prevent the logits from exploding.

Grouped bar chart comparing top-1 accuracy of a supervised ResNet-101 and zero-shot CLIP ViT-L/14 on ImageNet and five distribution-shift benchmarks, with CLIP far ahead on ImageNet-R, ObjectNet, Sketch and ImageNet-A

Figure 2: Matched at 76.2% on the ImageNet validation set, the two models diverge sharply under shift: zero-shot CLIP keeps 77.1% on ImageNet-A and 60.2% on Sketch, where the supervised model drops to 2.7% and 25.2%, evidence that never fitting the ImageNet label distribution is itself a robustness mechanism.

Property	Zero-shot CLIP (ViT-L/14@336px)	Supervised classifier (ResNet-101)
Task supervision	Zero labeled examples for the task, 400M noisy web pairs for pretraining	1.28M hand-labeled images for exactly this label set
Adding a class	Encode one more prompt and append the vector, no gradient step	Collect labels, then retrain or refit the head
ImageNet top-1	76.2%	76.2%
ImageNet-A top-1	77.1%	2.7%
Specialized domains	Weak: near chance on tumor-patch classification, poor on counting and satellite land use	Strong once in-domain labels exist
Inference cost	One image forward pass plus a $d \times C$ matmul against cached text vectors	One forward pass through a fixed head

August 16, 2026

DL0097 Variational Autoencoder VAE

What is a variational autoencoder (VAE), and how does it differ from a plain autoencoder?

Answer

A variational autoencoder is a latent-variable generative model trained by amortized variational inference, not a compression network with a smaller bottleneck. A plain autoencoder learns a deterministic map $z = f_\phi(x)$ and minimizes reconstruction error alone, so nothing constrains where codes land: the latent space is an arbitrary point cloud with holes, and decoding a random vector usually produces garbage. A VAE instead makes the encoder output the parameters of a distribution $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$ , samples $z$ from it, and optimizes the evidence lower bound (ELBO), which adds a KL term pulling every posterior toward a fixed prior $p(z) = \mathcal{N}(0, I)$ . That single extra term is what turns an encoder-decoder pair into a generator: after training you can draw $z \sim \mathcal{N}(0, I)$ and decode it, because the aggregate posterior now covers the prior. The sampling step is made differentiable by the reparameterization trick, $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ , which keeps the stochastic node out of the gradient path.

(1) Probabilistic Encoder: the encoder emits a mean and a log-variance per latent dimension rather than a point, so each input maps to a small blob of latent space instead of a single coordinate.
(2) Two-Term Objective: the loss is reconstruction plus a KL regularizer against the prior; a plain autoencoder has only the first term, which is exactly why it is not generative.
(3) Reparameterization Trick: sampling is rewritten as a deterministic function of the parameters and an external noise draw, giving a low-variance pathwise gradient instead of a high-variance score-function estimator.
(4) Sampling From The Prior: because the KL term forces the posteriors to overlap and fill the prior, ancestral sampling ( $z \sim p(z)$ , then decode) yields plausible data, and interpolation between two codes stays on the data manifold.
(5) Explicit Likelihood Bound: the ELBO is a lower bound on $\log p_\theta(x)$ , so a VAE gives a comparable density estimate, whereas an autoencoder’s reconstruction error has no probabilistic meaning.

Two-row diagram: the top row shows a plain autoencoder mapping input x through an encoder to a single deterministic code and back through a decoder; the bottom row shows a VAE whose encoder emits mu and log sigma squared, a sampling step z equals mu plus sigma times epsilon, and a decoder, with a KL term pulling the posterior toward the standard normal prior

Figure 1: The structural difference is one node: the plain autoencoder passes a single deterministic code to the decoder, while the VAE passes a sample from a learned Gaussian that the KL term keeps anchored to $\mathcal{N}(0, I)$ .

Two practical details dominate real training runs. The first is the balance between the two loss terms: with a strong decoder or a large KL weight, the cheapest solution is to set $q_\phi(z|x) = p(z)$ and ignore the latent entirely, a failure called posterior collapse that shows up as a KL term decaying to near zero while reconstruction stalls. Standard mitigations are KL annealing (ramp the weight from 0 to 1 over the first epochs) and free bits (do not penalize a dimension until its per-dimension KL exceeds a floor of roughly 0.05 to 0.5 nats). The second is the choice of likelihood: a diagonal Gaussian decoder is equivalent to an MSE reconstruction loss, which averages over plausible outputs and is the direct cause of the blurry samples VAEs are known for; discretized logistic or categorical likelihoods sharpen results noticeably. Modern image systems exploit this honestly, using a KL-regularized autoencoder as the perceptual compressor and putting the generative burden on a diffusion model in that latent space, as in Stable Diffusion.

Mathematical Formulation:
$\log p_\theta(x) \geq \mathcal{L}(\theta, \phi; x)$
$\mathcal{L} = \mathcal{L}_{rec} - \beta \, D_{KL}(q_\phi \| p)$
$\mathcal{L}_{rec} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]$
$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$
$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon$
$\epsilon \sim \mathcal{N}(0, I)$
$D_{KL} = -\frac{1}{2} \sum_{j=1}^{d} \delta_j$
$\delta_j = 1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2$

Where:

$\mathcal{L}$ is the ELBO, a lower bound on the marginal log-likelihood $\log p_\theta(x)$ ; the gap between them equals $D_{KL}(q_\phi(z|x) \| p_\theta(z|x))$ , so maximizing the bound both fits the data and sharpens the approximate posterior.
$x$ is the observation and $z \in \mathbb{R}^{d}$ the latent code, with $d \ll \dim(x)$ in the usual bottleneck setting.
$q_\phi(z|x)$ is the encoder (recognition model) with parameters $\phi$ , and $p_\theta(x|z)$ is the decoder (likelihood) with parameters $\theta$ .
$p(z) = \mathcal{N}(0, I)$ is the fixed prior, and $j \in \{1, \ldots, d\}$ indexes latent dimensions in the closed-form KL, where $\mu_j$ and $\sigma_j$ are the encoder outputs for example $x$ .
$\odot$ is elementwise multiplication and $\epsilon$ is the external noise draw that makes the sample differentiable in $\mu$ and $\sigma$ .
$\beta$ is the KL weight; $\beta = 1$ recovers the exact ELBO, $\beta > 1$ gives the beta-VAE disentanglement regime, and $\beta = 0$ degenerates to a plain autoencoder. Networks are typically initialized so $\log \sigma_j^2 \approx 0$ , which starts training near the prior.

Two scatter panels of a two-dimensional latent space: on the left the plain autoencoder codes sit in six distant clusters far from the origin with large empty gaps and prior samples falling in a hole, on the right the VAE codes form overlapping blobs filling the unit and two-sigma circles of the standard normal prior so prior samples land on data

Figure 2: Encoded training data in a 2D latent space. The plain autoencoder is free to scatter codes anywhere, so a draw from $\mathcal{N}(0, I)$ lands in a hole; the KL term compresses the VAE’s aggregate posterior onto the prior, so the same draw hits populated territory.

Property	Plain Autoencoder	VAE	VQ-VAE
Encoder output	One deterministic code vector per input	Distribution parameters, mean and log-variance	Continuous vector snapped to the nearest codebook entry
Training objective	Reconstruction error only	ELBO: reconstruction minus beta times KL to N(0, I)	Reconstruction plus codebook and commitment losses, no KL
Latent geometry	Arbitrary scale, holes and gaps between clusters	Smooth and prior-matched, interpolation stays on-manifold	Discrete grid of K entries, no notion of interpolation
Generating new data	Not supported, a random code decodes to noise	Draw z from the prior and decode, one forward pass	Needs a learned prior over codes, such as a transformer
Typical failure	Memorizes an identity map when the bottleneck is wide	Posterior collapse, blurry Gaussian-likelihood samples	Codebook collapse with most entries unused
Common use	Denoising, compression, anomaly detection	Generative modeling and the latent space of latent diffusion	Discrete tokens for autoregressive image and audio models

August 16, 2026

DL0092 Reward Hacking in RLHF

What is reward hacking, and how does it manifest in RLHF?

Answer

Reward hacking is what happens when a policy maximizes the measured reward instead of the intended objective, exploiting the gap between the two. In RLHF that gap is concrete and unavoidable: the reward model $r_{\phi}$ is a finite-capacity network fit to a fixed set of human preference pairs, so it equals the true human utility $r^{*}$ plus an error term $\epsilon$ . PPO does not distinguish the two, and since it actively searches for the highest-scoring completions, it lands precisely in the regions where $\epsilon$ is largest, which are the regions the reward model never saw during training. The empirical signature is the overoptimization curve: proxy reward climbs monotonically while gold reward measured by fresh human raters rises, peaks, and then falls. The visible symptoms are familiar to anyone who has run RLHF: verbosity, heavy markdown formatting, confident-sounding hedges, and sycophancy, all of which are cheap ways to increase $r_{\phi}$ without increasing helpfulness.

(1) Goodhart Under Active Search: the reward model is only a proxy, and RL is an adversary that optimizes against its error surface rather than sampling from it.
(2) Distribution Shift Is The Mechanism: preference labels are collected on SFT-model samples, so as the policy moves, its outputs leave the reward model’s training support and the error term stops being small.
(3) Length Is The Canonical Exploit: human raters mildly prefer longer answers, the reward model amplifies that correlation, and a large share of RLHF win-rate gains disappears once responses are length-controlled.
(4) The KL Term Is A Leash, Not A Fix: the $\beta$ -weighted KL penalty bounds how far the policy can travel from the reference model, which delays hacking but trades away real gains at the same time.
(5) Detection Requires An Off-Proxy Signal: you cannot see hacking in the training curve, because the training curve is the thing being hacked; it takes fresh human labels, a held-out reward model, or a verifiable check to notice.

Diagram of the RLHF loop: fixed preference pairs train a proxy reward model, the reward model scores policy samples, PPO or GRPO updates the policy under a KL penalty, and fresh samples feed back into the reward model while drifting off its training support

Figure 1: The RLHF loop (preferences → reward model → policy update → fresh samples) closes on a fixed, off-policy reward model, so every optimization step pushes the sampled distribution further from where the proxy was ever validated.

It helps to separate the failure modes by where the exploit lives. Spurious-feature hacking targets shallow correlates of quality that the reward model learned from labeler habits: length, bullet lists, bold headers, apologetic openers, restating the question. Sycophancy targets the labelers themselves, since annotators reward answers that agree with them, so the policy learns to mirror the stated opinion rather than correct it. Off-distribution spikes are the strangest class: degenerate strings and odd punctuation patterns that no human ever rated, which the reward model happens to score near its maximum. In agentic and code settings the same pressure produces test gaming, where the model special-cases unit tests or edits the test file instead of fixing the function.

Mathematical Formulation:
$J(\pi_{\theta}) = \mathbb{E}[r_{\phi}(x,y)] - \beta\,\mathcal{D}$
$\mathcal{D} = D_{\mathrm{KL}}(\pi_{\theta} \| \pi_{\mathrm{ref}})$
$r_{\phi}(x,y) = r^{*}(x,y) + \epsilon(x,y)$
$d = \sqrt{\mathcal{D}}$
$R_{\mathrm{gold}}(d) = d\,(\alpha - \beta_{o}\log d)$

Where:

$J(\pi_{\theta})$ is the RLHF objective actually optimized, and $R_{\mathrm{gold}}$ is the true human-judged quality, which is never in the gradient.
$x$ is the prompt, $y \sim \pi_{\theta}(\cdot\mid x)$ the sampled completion, and $\pi_{\mathrm{ref}}$ the frozen SFT reference policy.
$r_{\phi}$ is the learned proxy reward, $r^{*}$ the intended reward, and $\epsilon$ the model error whose magnitude grows off the preference-data support.
$\beta$ is the KL coefficient (typical range 0.01 to 0.1), so $\beta \to 0$ removes the leash and $\beta$ large pins the policy to the reference.
$d$ is the square-root KL distance used as the x-axis of overoptimization scaling laws; $\alpha$ and $\beta_{o}$ are fitted constants, distinct from the KL coefficient $\beta$ .
Required initial condition: $\pi_{\theta} = \pi_{\mathrm{ref}}$ at step 0, so $d = 0$ and $R_{\mathrm{gold}}$ starts at the SFT baseline; the peak of the fitted curve sits at $\log d = (\alpha - \beta_{o})/\beta_{o}$ .

Line chart with square-root KL distance on the x axis showing proxy reward rising monotonically while gold reward rises to a peak near d equals 3.1 and then declines below zero, with the peak marked as the early stopping point

Figure 2: Illustrative overoptimization: the proxy score the trainer logs never stops improving, while gold reward turns over around $d \approx 3.1$ and eventually falls below the SFT baseline.

Mitigation	Mechanism	What it buys	Cost or limitation
KL penalty and early stopping	Bounds the distance travelled from the reference policy	Keeps the policy inside the region where the proxy was validated	Caps genuine gains too; the right KL budget is only knowable from gold evals
Reward model ensembles	Score with several models, optimize a pessimistic aggregate such as the minimum	Suppresses idiosyncratic per-model error spikes	Members share preference data, so correlated biases like length survive; N times the scoring cost
Iterative on-policy relabeling	Collect fresh preferences on current-policy samples and refit the reward model	Directly closes the distribution shift that creates the error term	Slow and expensive; each round needs new human annotation
Explicit bias disentangling	Factor the reward into a quality head and a length or format head, then discard the latter	Removes the single largest known spurious correlate	Only fixes biases you already named; new exploits appear elsewhere
Verifiable or rule-based reward	Replace the learned model with a checker: unit tests, a symbolic solver, format rules	No learned error surface to climb on checkable tasks	Only applies where ground truth exists; test gaming and reward tampering remain

August 16, 2026