Author: admin

DL0130 MoE Router Collapse and Auxiliary Loss

What is Router Collapse in Sparse Mixture-of-Experts (MoE) LLMs? Derive the auxiliary load-balancing loss and Router Z-loss used to stabilize MoE training.

Answer

Router collapse is the failure mode in which the learned router of a sparse MoE layer stops using most of its experts and concentrates almost all token assignments on a small subset, so a layer you paid $N$ experts’ worth of memory for behaves like a much smaller model. The cause is a positive feedback loop: an expert that happens to win slightly more tokens early in training receives more gradient signal, becomes genuinely better on those tokens, so the router raises its logit further, while starved experts never see enough tokens to become useful. Under token-choice top-k the visible symptoms are a skewed load histogram, a large dropped-token fraction once the hot experts hit their capacity buffer, and a validation loss that tracks a dense model of the active-parameter size rather than the total-parameter size. Production training stacks suppress it with two extra loss terms rather than with a new routing algorithm: an auxiliary load-balancing loss that pushes router probability mass away from overloaded experts, and a router z-loss that penalizes the squared log-sum-exp of the router logits so the logits cannot grow without bound and destabilize the softmax in low precision. Both are differentiable surrogates for quantities that are not: the actual assignment counts are piecewise constant, and the actual numerical blow-up is a hardware property, so each loss attacks a proxy the gradient can reach.

(1) Collapse Is Self-Reinforcing: a small early advantage in router logits compounds through the gradient the winning expert receives, which is why collapse typically happens in the first few thousand steps and is nearly irreversible afterwards.
(2) The Assignment Is Non-Differentiable: the load fraction $f_i$ comes from a top-k selection and has zero gradient almost everywhere, so the balance loss must pair it with the differentiable mean gate probability $P_i$ .
(3) Balance Loss Is A Normalized Dot Product: $N \sum_i f_i P_i$ equals 1 under a uniform assignment and grows toward $N$ under total collapse, giving a scale-free objective independent of expert count.
(4) Its Gradient Is Load-Proportional: the derivative with respect to $P_i$ is exactly $\alpha N f_i$ , so probability mass is pushed down in proportion to how overloaded each expert already is.
(5) Z-Loss Bounds The Logits: penalizing the squared log-partition function caps $\max_j h_j$ , which matters because the relative round-off in $e^{h}$ grows linearly with $|h|$ in bfloat16 and can flip top-k selection.
(6) Coefficients Are A Quality Trade: $\alpha \approx 10^{-2}$ and $\beta \approx 10^{-3}$ are the common settings, because a large $\alpha$ buys perfect balance by actively fighting the language-modeling objective.

Cycle diagram of router collapse: router logits for one expert edge above the others, top-k dispatches more tokens to it, that expert receives most of the expert gradient, the starved experts stay undertrained and score lower, which feeds back into the logit gap; two green intervention boxes on the right show the router z-loss acting on the logits node and the auxiliary balance loss acting on the dispatch node

Figure 1: Router collapse is a closed loop, not a single bad step: the logit gap, the dispatch skew, and the gradient imbalance each amplify the next. The two stabilizers cut the loop at different points, with the z-loss constraining the logit magnitudes and the balance loss constraining the dispatch distribution.

The derivation of the balance loss starts from what you actually want to penalize, namely the variance of the per-expert token counts, and then asks which part of that quantity carries a gradient. The counts themselves come from a top-k over the router logits, so they are a step function of the parameters and give nothing to backpropagation. The fix used by GShard and simplified by Switch Transformers is to pair the non-differentiable load vector $f$ with the differentiable importance vector $P$ , the mean softmax probability per expert, and minimize their inner product. Treating $f$ as a constant, the objective is linear in $P$ with coefficient $N f_i$ , so each step lowers the router’s probability for exactly the experts that were overloaded in the current batch, and because $f$ is recomputed every step this becomes a self-correcting controller whose fixed point is the uniform assignment. The factor $N$ is a normalization choice: it makes the minimum value 1 regardless of how many experts you have, so the same $\alpha$ transfers from an 8-expert layer to a 256-expert layer. The z-loss has an entirely different motivation: it is a numerical guard, derived from the observation that a logit stored with relative precision $\epsilon$ produces an absolute error of about $|h|\epsilon$ , and exponentiation converts that absolute error into a relative error of the same size, so large logits make the softmax and therefore the selected expert set unreliable.

Mathematical Formulation:
$h_t = W_r x_t, \quad p_t = \mathrm{softmax}(h_t)$
$f_i = \frac{1}{T}\sum_{t=1}^{T} \mathbb{1}[i \in \mathcal{T}_t]$
$P_i = \frac{1}{T}\sum_{t=1}^{T} p_{t,i}$
$\mathcal{L}_{\mathrm{bal}} = N \sum_{i=1}^{N} f_i P_i$
$\frac{\partial \mathcal{L}_{\mathrm{bal}}}{\partial P_i} = N f_i$
$1 \leq \mathcal{L}_{\mathrm{bal}} \leq N$
$\mathcal{L}_{z} = \frac{1}{T}\sum_{t=1}^{T}\left(\log \sum_{j=1}^{N} e^{h_{t,j}}\right)^{2}$
$\max_j h_{t,j} \leq \mathrm{lse}(h_t) \leq \max_j h_{t,j} + \log N$
$\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \alpha \mathcal{L}_{\mathrm{bal}} + \beta \mathcal{L}_{z}$

Where:

$x_t \in \mathbb{R}^{d}$ is the hidden state of token $t$ , $W_r \in \mathbb{R}^{N \times d}$ the router matrix, and $h_t$ the router logits before any selection.
$T$ is the number of tokens the statistics are aggregated over (micro-batch, device batch, or global batch), $N$ the expert count, and $\mathcal{T}_t$ the set of $k$ experts selected for token $t$ .
$f_i$ is the load, the fraction of tokens dispatched to expert $i$ ; it is piecewise constant in the parameters and therefore contributes no gradient.
$P_i$ is the importance, the mean router probability assigned to expert $i$ ; it is smooth, so all of the balance-loss gradient flows through it and into $W_r$ .
$\mathcal{L}_{\mathrm{bal}} = 1$ exactly when $f_i = P_i = 1/N$ for all $i$ , and approaches $N$ when one expert takes every token, so the value is directly readable as an imbalance factor.
$\mathrm{lse}(h_t) = \log\sum_j e^{h_{t,j}}$ is the log-partition function; squaring it penalizes large logits in either direction and, by the sandwich bound, keeps $\max_j h_{t,j}$ within $\log N$ of a small target.
$\alpha$ and $\beta$ are the balance and z-loss coefficients, commonly $\alpha = 10^{-2}$ and $\beta = 10^{-3}$ ; $\mathcal{L}_{\mathrm{LM}}$ is the ordinary next-token cross-entropy.

Two-panel chart: left panel plots the maximum expert load fraction against training step for an eight-expert layer with balance-loss coefficients zero, one thousandth, and one hundredth, showing full collapse toward one expert without the loss and a curve close to the uniform 0.125 line with alpha one hundredth; right panel plots relative round-off error in exp of a router logit against logit magnitude on a log scale for bfloat16 and float32 storage, with bfloat16 crossing one percent error at a logit magnitude near 2.6

Figure 2: The two losses guard different quantities. Left: without a balance term the maximum load fraction runs from the uniform $1/N = 0.125$ to near 1.0 within a few thousand steps, while $\alpha = 10^{-2}$ holds it close to uniform. Right: the relative round-off in $e^{h}$ grows linearly with $|h|$ , which is why bfloat16 routers become unreliable at moderate logits and the z-loss keeps $\mathrm{lse}(h)$ near $O(1)$ .

Property	Auxiliary balance loss	Router z-loss	Bias-based loss-free balancing
Quantity penalized	The dot product of load and importance, scaled by the expert count	The squared log-sum-exp of the router logits, averaged over tokens	Nothing; a per-expert bias is added to the selection logits only
Failure it prevents	Expert collapse, wasted parameters, and dropped tokens at the capacity buffer	Logit blow-up, low-precision softmax round-off, and loss spikes	Expert collapse, without perturbing the gate weights used in the output
Typical setting	Coefficient 1e-2, aggregated per device batch or per global batch	Coefficient 1e-3, with the router itself computed in float32	Bias update rate around 1e-3, driven by observed per-expert load error
Effect on the LM objective	Adds an interference gradient that trades some quality for balance	Mild regularizer, usually neutral or slightly positive for quality	No interference term at all, since the bias is not part of the output gate
Where it is used	GShard, Switch, Mixtral, OLMoE	ST-MoE and most later open MoE training stacks, including OLMoE	DeepSeek-V3 and its loss-free-balancing follow-ups

August 17, 2026

DL0129 SwiGLU, GELU, and ReLU FFNs

Compare SwiGLU, GeLU, and ReLU activation functions in the Feed-Forward Network (FFN) blocks of LLMs, as used in models like Llama 3 and GPT-3.

Answer

All three sit in the same place, namely the two-layer FFN that follows attention in every transformer block, but they differ in smoothness and in whether the nonlinearity is elementwise or multiplicative. ReLU is $\max(0,x)$ : cheapest, produces exact zeros (roughly 90-95% of FFN units are zero on a typical token), but has a kink at the origin and zero gradient for negative inputs, which gives dead units. GELU is $x\Phi(x)$ : smooth, non-monotonic, keeps a small negative response, and was the default through BERT, GPT-2, and GPT-3. SwiGLU is not an elementwise function at all but a gated layer: the up-projection is split into a gate branch passed through SiLU ( $x\sigma(x)$ ) and a value branch left linear, and the two are multiplied elementwise. That third weight matrix is why practitioners shrink the hidden width by $2/3$ to stay parameter- and FLOP-neutral, and the payoff reported by Shazeer’s GLU-variants ablation is a consistent but modest gain, about 0.05 nats of pretraining log-perplexity on T5’s span-corruption objective at matched budget, which is enough that PaLM, Llama, Mistral, and Qwen2 all adopted it. The cost is one extra matmul, one extra saved activation for the backward pass, loss of exploitable sparsity, and a wider activation range that makes low-bit quantization harder.

(1) Two Matrices Versus Three: a dense FFN is $W_{up}$ then $W_{down}$ ; a gated FFN adds $W_g$ , so parameters go from $2 d\, d_{ff}$ to $3 d\, d_{ff}$ at the same width.
(2) The 2/3 Width Rule: to keep the block budget fixed you set $d_{ff} = \frac{8}{3}d$ instead of $4d$ , which is exactly why Llama 2 7B uses 11008 with $d = 4096$ .
(3) Smoothness Changes Optimization: ReLU’s derivative is a step with no gradient for negative inputs, while GELU and SiLU are $C^\infty$ and pass a small negative signal, so units recover instead of dying.
(4) Gating Is A Multiplicative Interaction: the output is a product of two learned projections of the same input, giving a data-dependent, per-channel scaling that no elementwise function can express.
(5) Sparsity Is A ReLU-Only Asset: exact zeros let inference systems skip entire rows of $W_{down}$ ; GELU and SwiGLU produce values that are small but almost never exactly zero.
(6) SwiGLU Costs Memory Traffic And Range: three matmuls plus an extra stored tensor per layer, and the product of two unbounded projections widens the dynamic range that quantization must cover.

Two-panel line chart: left panel plots ReLU, GELU and SiLU over the range minus four to four, showing ReLU flat at zero for negative inputs while GELU and SiLU dip slightly below zero; right panel plots their derivatives, showing ReLU's step from zero to one at the origin versus the smooth GELU and SiLU derivatives that briefly exceed one and go slightly negative

Figure 1: Shape (left) and derivative (right). ReLU’s derivative is a step function with a kink at the origin and no gradient for negative inputs; GELU and SiLU are smooth, non-monotonic, and dip below zero (SiLU’s minimum is $-0.278$ at $x = -1.28$ ), so a unit pushed into the negative region still receives gradient and can come back.

The historical progression is ReLU → GELU → gated GLU variants, and it is worth being honest that only the first step has a clean story. Replacing ReLU with GELU removes the dead-unit problem and gives a slightly better-conditioned loss surface. Replacing an elementwise nonlinearity with a gate is different in kind: the value branch stays linear, and the gate branch modulates it, so the layer computes a second-order interaction of the input with itself. Shazeer’s paper explicitly declines to explain why this helps, attributing the gain to luck rather than theory, and later analyses mostly note that gating gives the block a cheap multiplicative pathway similar in spirit to the gates in an LSTM. What made SwiGLU the production default is that the gain survives at scale, needs no extra hyperparameters, and is free once you apply the $2/3$ width rule. Llama 3 8B keeps the gated form but widens back out to $d_{ff} = 14336$ with $d = 4096$ , a deliberate choice to spend more parameters in the FFN rather than a parameter-neutral swap.

Mathematical Formulation:
$\mathrm{FFN}_{\mathrm{ReLU}}(x) = \max(0, xW_{up})\,W_{down}$
$\mathrm{GELU}(x) = x\,\Phi(x)$
$\mathrm{SiLU}(x) = x\,\sigma(x)$
$\mathrm{SwiGLU}(x) = \mathrm{SiLU}(xW_g) \odot (xW_u)$
$\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \mathrm{SwiGLU}(x)\,W_{down}$
$d_{ff} = \frac{2}{3}\cdot 4d = \frac{8}{3}d$

Where:

$x \in \mathbb{R}^{d}$ is the post-norm hidden state for one token, and the FFN output has the same dimension $d$ so it can be added back through the residual connection.
$W_{up}, W_g, W_u \in \mathbb{R}^{d \times d_{ff}}$ are up-projections and $W_{down} \in \mathbb{R}^{d_{ff} \times d}$ the down-projection; the dense block uses $W_{up}$ only, the gated block uses both $W_g$ (gate) and $W_u$ (value).
$\Phi$ is the standard normal CDF and $\sigma$ the logistic sigmoid; GELU is often shipped as the tanh approximation $0.5x(1+\tanh(\sqrt{2/\pi}(x+0.044715x^3)))$ , which is why two frameworks can disagree in the last few digits.
$\odot$ is the elementwise product over the $d_{ff}$ hidden channels, the operation that turns a gate into a multiplicative interaction rather than a pointwise map.
$d_{ff}$ is the FFN hidden width; the last line is the parameter-neutral setting, since $3 d \cdot \frac{8}{3}d = 8d^2 = 2 d \cdot 4d$ , and real implementations round the result to a hardware-friendly multiple such as 256.

Architecture diagram with two stacked left-to-right flows: the top flow is a dense FFN with hidden state, one up-projection, an elementwise ReLU or GELU, a down-projection and output, annotated with two d times d_ff parameters; the bottom flow is a gated SwiGLU FFN where the hidden state branches into a SiLU gate path and a linear value path, the two are combined by an elementwise product, then a down-projection and output, annotated with three d times d_ff parameters and the two-thirds width rule

Figure 2: The structural difference is a branch, not a curve. A dense FFN applies one elementwise function between two matrices; a gated FFN splits the up-projection into a SiLU gate path and a linear value path whose elementwise product feeds $W_{down}$ . The third matrix is what forces the $8d/3$ width if the block budget must stay fixed.

Property	ReLU	GELU	SwiGLU
Form	Elementwise, piecewise linear	Elementwise, smooth, non-monotonic	Gated layer: SiLU branch times linear branch
Matrices per FFN	2	2	3, so width drops to 8d/3 to stay budget-neutral
Gradient behaviour	0 or 1; dead units never recover	Smooth, briefly exceeds 1, small negative region	Smooth, plus a gradient path through the gate itself
Exact-zero sparsity	Yes, typically 90-95% of units, exploitable at inference	No; values are small but nonzero	No; near-zero mass only, so row skipping needs a threshold and loses accuracy
Quantization friendliness	Best: bounded below, no multiplicative amplification	Good	Hardest: the product of two unbounded projections widens outliers into W_down
Representative models	Original Transformer, T5 v1.0, OPT	BERT, GPT-2, GPT-3, ViT	PaLM, Llama 2 and Llama 3, Mistral, Qwen2 (Gemma uses the GELU-gated GeGLU)

August 17, 2026

DL0128 Subword Regularization

What is subword regularization, and why can it help machine translation quality?

Answer

Subword regularization is the practice of training a translation model on multiple stochastically sampled segmentations of the same sentence instead of the one deterministic segmentation a tokenizer normally emits. Any subword vocabulary is ambiguous: the word “smaller” can be a single piece, or small plus er, or sm plus all plus er, and a standard tokenizer always returns the same choice, so the model never sees the alternatives. Two mechanisms implement the sampling: unigram-LM sampling (Kudo 2018, shipped in SentencePiece), which draws a path from the segmentation lattice with probability proportional to $P(\mathbf{x})^{\alpha}$ , and BPE-dropout (Provilkov 2020), which keeps the ordinary BPE merge table but skips each merge with probability $p$ during application. The quality gain has three sources: it is free data augmentation over the same parallel corpus, it trains the embeddings of rare pieces that a deterministic segmenter almost never produces, and it makes the encoder robust to the segmentation shifts caused by typos, morphology, and domain change. Reported gains are roughly 1 BLEU on average, shrinking toward noise on large in-domain high-resource benchmarks and reaching 2 BLEU or more on low-resource pairs and out-of-domain test sets. Sampling is a training-time-only device: decoding uses the deterministic segmentation, or optionally averages scores over $k$ sampled segmentations of the source.

(1) Segmentation Is Ambiguous: a fixed vocabulary admits exponentially many segmentations per sentence, and deterministic tokenization silently commits to one of them in every epoch.
(2) Unigram-LM Sampling: SentencePiece samples from the $l$ -best lattice paths with a temperature $\alpha$ , typically $l = 64$ and $\alpha$ near 0.1 to 0.2, where small $\alpha$ is near-uniform and $\alpha \to \infty$ recovers the Viterbi segmentation.
(3) BPE-Dropout: drop each merge operation independently with $p = 0.1$ on both source and target; the vocabulary and merge table are unchanged, so it is a drop-in change to the tokenizer call.
(4) Approximate Marginalization: the model is pushed to score a sentence pair well under any plausible segmentation, which is closer to the true marginal $P(Y \mid X)$ than a single-path proxy.
(5) Robustness Beats In-Domain Accuracy: the largest wins appear on low-resource, morphologically rich, noisy, and out-of-domain data, exactly where the test-time segmentation drifts from the training-time one.
(6) Costs Are Real: average sequence length grows, tokenization moves into the data loader and cannot be cached, and convergence is slower in wall-clock terms for the same number of updates.

Segmentation lattice for the word smaller: boundary nodes under the characters, with three bracketed paths drawn above them, one spanning the whole word as a single piece, one splitting into small plus er, and one splitting into sm plus all plus er, each annotated with its unigram probability and the sampling weight at alpha equal to 0.2

Figure 1: One word, three legal paths through the segmentation lattice. Deterministic tokenization always returns the top path, so the -er morpheme boundary is never shown to the encoder; sampling at $\alpha = 0.2$ returns the top path only about half the time and spends the rest of the probability mass on the finer decompositions.

The reason the same trick helps two different segmenters is that both are inverting a latent variable the corpus never labels. A unigram language model gives an explicit probability to each path, so its lattice can be sampled cleanly and the temperature $\alpha$ gives a continuous dial from “uniform over the lattice” to “deterministic”. BPE has no probability model at all, only an ordered merge list, so BPE-dropout injects randomness procedurally instead: skipping a merge with probability $p$ produces a shorter piece and therefore a longer, more character-like segmentation, and stacking many independent skips over a long word yields a wide distribution of segmentations. Both settings need a knob that is gentle: too much randomness pushes segmentation toward characters, inflating sequence length, wasting attention on re-learning spelling, and destroying the frequency statistics that made the vocabulary useful in the first place.

Mathematical Formulation:
$P(\mathbf{x}) = \prod_{i=1}^{M} p(x_i)$
$P_{\alpha}(\mathbf{x} \mid X) \propto P(\mathbf{x})^{\alpha}$
$\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x}, \mathbf{y}}\left[\log P(\mathbf{y} \mid \mathbf{x}; \theta)\right]$
$P(Y \mid X) = \sum_{\mathbf{x}} P(\mathbf{x} \mid X)\, P(Y \mid \mathbf{x})$
$\hat{Y} = \arg\max_{Y} \frac{1}{k} \sum_{j=1}^{k} \log P(Y \mid \mathbf{x}_j)$

Where:

$X$ and $Y$ are the raw source and target sentences, while $\mathbf{x} = (x_1, \ldots, x_M)$ and $\mathbf{y}$ are particular segmentations of them into vocabulary pieces.
$p(x_i)$ is the unigram probability of piece $x_i$ , estimated by EM when the SentencePiece unigram vocabulary is built, and $M$ is the number of pieces on that path.
$\alpha$ is the smoothing exponent of the sampling distribution: $\alpha \to 0$ gives a near-uniform draw over the $l$ -best lattice paths and $\alpha \to \infty$ collapses to the single Viterbi path, so deterministic tokenization is the limiting case.
The expectation in $\mathcal{L}(\theta)$ is taken over $\mathbf{x} \sim P_{\alpha}(\cdot \mid X)$ and $\mathbf{y} \sim P_{\alpha}(\cdot \mid Y)$ , approximated by one fresh sample per example per epoch rather than by an inner loop.
$P(Y \mid X)$ is the true segmentation-marginalized translation probability that the training objective approximates; the sum over $\mathbf{x}$ is intractable, which is exactly why sampling is used.
$k$ is the number of source segmentations used in optional n-best decoding, where $k = 1$ with the Viterbi path is the standard, cheapest inference setting.

Two-panel line chart: left panel shows the entropy in bits of the segmentation sampling distribution decreasing from near-maximum to near zero as alpha grows from 0.02 to 3 for a three-path and a four-path lattice; right panel shows the expected number of pieces per word falling toward one over the same alpha range, both panels marking the default alpha near 0.2

Figure 2: The temperature $\alpha$ controls exactly two things that matter. Left: the entropy of the sampling distribution, which is how much augmentation diversity the model receives per epoch. Right: the expected pieces per word, which is the sequence-length tax you pay for that diversity. Beyond roughly $\alpha = 1.5$ both collapse to the deterministic Viterbi behaviour and the method does nothing.

Property	Deterministic BPE / unigram	Unigram-LM sampling	BPE-dropout
Randomness source	None; one path per sentence forever	Draw from the l-best lattice paths of a unigram LM	Independently skip each merge while applying the merge table
Knob and typical value	Vocabulary size only	l = 64, alpha near 0.1 to 0.2	p = 0.1 on both sides, lower for high-resource pairs
Segmenter changes	n/a	Needs a probabilistic unigram vocabulary, not a merge list	None; reuses an existing BPE model unchanged
Main cost	Rare pieces stay under-trained	Lattice sampling per example; longer average sequences	Re-merging per example; length inflation grows fast with p
Inference	Same path as training	Viterbi path, or average log-scores over k sampled sources	Standard BPE with dropout disabled

August 17, 2026

DL0127 DDPM Training Objective

What is the DDPM training objective, and why do we train the network to predict the noise rather than the clean image directly?

Answer

A denoising diffusion probabilistic model is trained by maximizing a variational bound on the data log-likelihood, and because both the forward corruption and the reverse posterior are Gaussian, that bound collapses into a sum of KL divergences whose only learnable content is a mean-matching term at each noise level. Reparameterizing that mean through the closed-form forward marginal turns every term into a weighted squared error between the injected noise $\epsilon$ and a network prediction, and Ho et al. then drop the per-step weight $\lambda_t$ to get the simple objective $L_{\mathrm{simple}} = \mathbb{E}\|\epsilon - \epsilon_\theta(x_t,t)\|^2$ . Predicting the noise is not a different model from predicting the clean image: $\epsilon$ -prediction, $x_0$ -prediction, and direct mean prediction are affine reparameterizations of one another given $x_t$ and $t$ , so they define the same optimum but different implicit loss weightings and different numerical conditioning. The practical reasons to regress noise are that the target is always unit-variance no matter how noisy the input is, that the residual structure lets the network pass the low-frequency content through instead of re-synthesizing it, and that discarding $\lambda_t$ under this parameterization down-weights the easy low-noise steps by roughly an order of magnitude, which empirically buys much better FID. It also makes the network a rescaled score estimator, which is what connects DDPM to score matching and to every ODE and SDE sampler built on top of it.

(1) The Objective Is One Scalar MSE: sample $x_0$ , sample $t \sim \mathcal{U}\{1,\ldots,T\}$ and $\epsilon \sim \mathcal{N}(0,I)$ , form $x_t$ in closed form, and regress $\epsilon$ . No per-step model, no adversarial term, no sequential rollout during training.
(2) It Comes From The ELBO, Not From Heuristics: each $L_{t-1}$ is a Gaussian KL whose learnable part is a mean difference, and substituting the forward marginal converts it into $\lambda_t\|\epsilon-\epsilon_\theta\|^2$ .
(3) Dropping $\lambda_t$ Is A Reweighting Choice: $\lambda_t$ is largest at small $t$ , so setting all weights to 1 shifts capacity toward the high-noise steps that decide global structure.
(4) The Target Is Scale-Stable: $\epsilon$ is standard normal for every $t$ , so a single output head with fixed normalization works across the whole schedule, whereas the difficulty of predicting $x_0$ varies enormously with $t$ .
(5) Residual Prediction Is Easier For A U-Net: at low noise, most of $x_t$ is already the answer, so predicting the small perturbation avoids forcing the network to reconstruct an image it was handed.
(6) Conditioning Flips At The Two Ends: converting $\epsilon_\theta$ to $\hat x_0$ amplifies error by $1/\sqrt{\mathrm{SNR}_t}$ , which is tiny at low noise and roughly 150x at $t=T$ , the reason v-prediction exists.

Left-to-right flow diagram of one DDPM training step: sample a clean image, sample a timestep and Gaussian noise, form the corrupted sample in closed form, pass it through the U-Net to predict the noise, compute the squared error against the sampled noise, and a dashed feedback arrow carrying the gradient step back to the U-Net

Figure 1: One training step. The closed-form forward marginal means an arbitrary timestep can be sampled directly, so training never simulates the chain; the loss is a single unit-variance regression shared by all $T$ noise levels.

The reweighting argument is the one that most candidates miss. Under the ELBO, the term at $t=1$ carries roughly 50x the weight of the term at $t=T$ for the standard linear schedule, and those low- $t$ terms correspond to removing almost imperceptible noise, a task that contributes little to perceptual quality but a lot to likelihood. Setting every weight to 1 in $\epsilon$ -space is therefore a deliberate trade of likelihood for sample quality, and it is why DDPM reports strong FID with mediocre bits-per-dimension. The conditioning argument cuts the other way at high noise: since $\hat x_0$ is recovered by dividing by $\sqrt{\bar\alpha_t}$ , a small noise error becomes a large image error near $t=T$ , so pure $\epsilon$ -prediction is a poor target for few-step or distilled samplers that must produce a usable $\hat x_0$ from the very first step. Finally, the identity $\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\,s_\theta$ shows the trained network is a scaled score function, which is exactly what a probability-flow ODE or an annealed Langevin sampler needs, so the same checkpoint serves DDPM, DDIM, and higher-order solvers.

Mathematical Formulation:
$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$
$L_{t-1} = \lambda_t\,\|\epsilon - \epsilon_\theta(x_t,t)\|^2$
$\lambda_t = \frac{\beta_t^2}{2\sigma_t^2\alpha_t(1-\bar\alpha_t)}$
$L_{\mathrm{simple}} = \mathbb{E}\left[\|\epsilon - \epsilon_\theta(x_t,t)\|^2\right]$
$\hat x_0 = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta}{\sqrt{\bar\alpha_t}}$
$\epsilon_\theta(x_t,t) = -\sqrt{1-\bar\alpha_t}\,s_\theta(x_t,t)$

Where:

$x_0$ is a clean training sample, $x_t$ its corrupted version at step $t$ , and $\epsilon \sim \mathcal{N}(0,I)$ the injected noise that serves as the regression target.
$t \in \{1,\ldots,T\}$ is sampled uniformly, $\beta_t$ is the schedule, $\alpha_t = 1-\beta_t$ , and $\bar\alpha_t = \prod_{s=1}^{t}\alpha_s$ gives the closed-form marginal that makes single-step training possible.
$\epsilon_\theta$ is the U-Net with parameters $\theta$ , conditioned on $t$ through a sinusoidal timestep embedding.
$\lambda_t$ is the ELBO weight and $\sigma_t^2$ the reverse-process variance (commonly $\sigma_t^2 = \beta_t$ ); $L_{\mathrm{simple}}$ is $L_{t-1}$ with $\lambda_t$ replaced by 1.
$\mathrm{SNR}_t = \bar\alpha_t/(1-\bar\alpha_t)$ is the signal-to-noise ratio, which decreases monotonically in $t$ and sets the error amplification $1/\sqrt{\mathrm{SNR}_t}$ from noise space into image space.
$s_\theta(x_t,t) \approx \nabla_{x_t}\log q(x_t)$ is the score, so a trained $\epsilon_\theta$ doubles as a score model for ODE and SDE samplers.

Two-panel chart: left panel plots the ELBO per-step weight relative to its value at the final timestep on a log axis against the diffusion step, showing roughly fifty times more weight at small t than at large t, next to the flat unit weight used by the simple objective; right panel plots on a log axis the factor by which a prediction error is amplified into clean-image space for noise prediction, clean-image prediction, and v-prediction, with noise prediction rising from about 0.01 to about 150

Figure 2: The two quantitative reasons the parameterization matters. Left: dropping $\lambda_t$ removes a roughly 50x preference for the easiest low-noise steps. Right: the same trained error costs almost nothing in image space at low noise but is amplified by $1/\sqrt{\mathrm{SNR}_t}$ near $t=T$ , which is where v-prediction stays bounded.

Property	Noise prediction (DDPM)	Clean-image prediction	v-prediction
Target	The sampled noise, standard normal at every t	The data sample itself, with data-dependent statistics	A schedule-dependent mix of noise and data
Implicit weighting under uniform MSE	Down-weights low-noise steps, favors perceptual quality over likelihood	Up-weights low-noise steps by the SNR, closer to the raw ELBO	Roughly balanced across the schedule, close to SNR-plus-one weighting
Numerical weak spot	Error into image space amplified by about 150x near t = T	Error into noise space amplified near t = 0, where the residual is tiny	Amplification stays at or below 1 in both directions
Typical use	Pixel and latent diffusion with many sampling steps, Stable Diffusion 1.x and 2.0-base	Very short schedules, consistency-style objectives, some latent decoders	Progressive distillation, high-resolution and upscaling models, SD 2.1-v

August 17, 2026

DL0126 Contrastive Learning Pairs

What is contrastive learning, and what are positive and negative pairs?

Answer

Contrastive learning trains an encoder by comparison instead of by prediction of a label: it pulls the embeddings of things that should mean the same thing together and pushes everything else apart. A positive pair is two views of the same underlying content (two augmentations of one photo in SimCLR, an image and its caption in CLIP, two nearby audio segments in CPC), and a negative pair is an anchor paired with content assumed to be different (any other item in the batch). The standard objective, InfoNCE, turns this into a softmax classification problem: given an anchor, identify its single positive among one positive and $K$ negatives, using scaled cosine similarity as the logit. Positives define what the representation should be invariant to, and negatives are what prevents the trivial solution where the encoder maps every input to the same vector, a failure called representational collapse. The whole design problem of a contrastive system is therefore the pair-construction policy: a positive that is too easy teaches nothing, and a negative that is secretly a positive actively teaches the wrong thing.

(1) Positives Encode The Invariance You Want: whatever transformation you apply to build a positive pair is exactly the factor the encoder learns to discard, so color jitter buys color invariance and destroys any task where color is the signal.
(2) Negatives Are The Anti-Collapse Term: without a repulsive term the constant map $f(x) = c$ is a global minimum of the attraction loss, so negatives provide the uniformity pressure that spreads embeddings over the hypersphere.
(3) InfoNCE Is A $K+1$ -Way Classification: one positive logit in the numerator, the positive plus all negatives in the denominator, which is why the loss is bounded below by $\log(K+1)$ under a random encoder.
(4) Temperature Sets Hard-Negative Focus: small $\tau$ (0.05 to 0.1 is typical) concentrates almost all repulsive gradient on the few most similar negatives, while large $\tau$ treats all negatives nearly equally.
(5) Negative Count Is A Systems Problem: SimCLR needs batch sizes of 4096 or more to get enough in-batch negatives, MoCo decouples them with a momentum-encoder queue, and CLIP trained with a 32768 global batch across many GPUs.
(6) False Negatives Are The Main Bias: in-batch negatives assume every other sample is semantically different, which is false on class-imbalanced or deduplicated-poorly data, and the loss then pushes apart items that should be close.

Pairs do not have to come from augmentation. The general recipe is: find a cheap source of known agreement and treat everything else as disagreement. Augmentation gives agreement between two crops of one image; multimodal alignment gives agreement between an image and the alt-text scraped alongside it; temporal or spatial context gives agreement between adjacent frames, patches, or sentences; and labels give agreement between any two samples of the same class, which is what SupCon exploits to allow many positives per anchor. The negatives are almost always just the rest of the batch, because sampling them explicitly is expensive and in-batch negatives come for free with the forward pass already computed.

Two-panel diagram: left panel shows one source image producing two augmented views labeled anchor and positive while two other images in the batch produce negative views; right panel shows the same points on a unit circle with an attraction arrow from anchor to positive and dashed repulsion arrows pushing the negatives away

Figure 1: Pair construction and its geometric effect. Each anchor has exactly one positive (the other view of the same source) and $2N-2$ negatives (all views of the other images in the batch); on the unit hypersphere the loss is a single attractive force toward the positive balanced against repulsive forces from every negative.

Mathematical Formulation:
$s(u,v) = \frac{u^\top v}{\|u\|\,\|v\|}$
$\ell_i = -\log \frac{\exp(s_{i,i^+}/\tau)}{\sum_{k \neq i} \exp(s_{i,k}/\tau)}$
$\mathcal{L} = \frac{1}{2N}\sum_{i=1}^{2N} \ell_i$
$I(u;v) \geq \log K - \mathcal{L}_{\mathrm{InfoNCE}}$

Where:

$s(u,v)$ is the cosine similarity used as the logit, computed on L2-normalized embeddings $u = f(v)$ produced by the encoder (and, in SimCLR, a discarded projection head).
$\ell_i$ is the per-anchor NT-Xent loss and $\mathcal{L}$ its average over all $2N$ views of an $N$ -image batch.
$i$ indexes the anchor view, $i^{+}$ its unique positive, and $k$ ranges over every other view, so the denominator holds 1 positive and $2N-2$ negatives.
$\tau > 0$ is the temperature; the gradient weight on negative $k$ is its softmax probability, so shrinking $\tau$ sharpens that distribution onto the hardest negatives.
$I(u;v)$ is the mutual information between the two views and $K$ the number of negatives, giving the standard InfoNCE lower bound: the bound saturates at $\log K$ , which is one reason large batches help.

Log-x line chart of the cumulative share of repulsive gradient carried by the hardest fraction of negatives, for temperature 0.05, 0.1 and 0.5 over 1024 negatives; the low-temperature curve rises almost vertically showing that the hardest one percent of negatives absorbs most of the gradient

Figure 2: Temperature decides which negatives matter. At $\tau = 0.05$ the hardest 1% of negatives absorbs most of the repulsive gradient, making training an implicit hard-negative miner that is also maximally sensitive to false negatives; at $\tau = 0.5$ the pressure is spread nearly uniformly and the embedding space stays smoother but less discriminative.

Aspect	Augmentation (SimCLR, MoCo)	Multimodal (CLIP, ALIGN)	Supervised (SupCon)
Anchor	One augmented view of an image	An image embedding	A labeled sample
Positive	A second augmentation of the same image (exactly one)	The paired caption text (exactly one, symmetric loss both directions)	Every other sample sharing the label (many per anchor)
Negatives	All 2N-2 other views, or a momentum queue of 65k stale keys	All other captions in the global batch (32k in CLIP)	Only samples with a different label, so false negatives vanish
Invariance learned	To the augmentation family you chose (crop, color, blur)	To modality and phrasing, giving a shared image-text space	To everything within a class, which can over-collapse fine detail
Main failure mode	Augmentation removes task-relevant signal; shortcut solutions from crop statistics	Noisy or generic captions produce weak positives; batch size dominates cost	Needs labels, so it is not self-supervised and inherits label noise

August 17, 2026

DL0125 Action-Conditioned World Model

Explain the architecture of an Action-Conditioned World Model for autonomous driving, of the kind Wayve describes for its GAIA models.

Answer

An action-conditioned world model is a generative model of future sensor data whose predictions are explicitly steered by the ego vehicle’s own control commands, so it can answer “what would the road look like in three seconds if I brake at 5 m/s² instead of holding speed?”. Architecturally it factors into four blocks that are trained in two stages: a video encoder that compresses multi-camera frames into a small latent grid, an action and context encoder that embeds the low-dimensional control signal, a latent dynamics model that rolls the compressed state forward in time, and a decoder plus auxiliary heads that render latents back to pixels or BEV occupancy. Almost every production-scale system follows this shape and differs only in the dynamics block: GAIA-1 uses a 6.5B-parameter causal transformer over discrete video tokens with a separate diffusion video decoder, GAIA-2 replaces it with a flow-matching diffusion transformer over continuous latents so five surround views stay geometrically consistent, and Dreamer-style recurrent state-space models keep a compact deterministic-plus-stochastic state for reinforcement learning in imagination. The reason compression comes first is the same as in latent diffusion: at 6.25 Hz a 26-frame context of $288 \times 512$ frames is 3.8M pixels per frame-stack but only 14,976 latent tokens, and the dynamics model’s cost is quadratic in that number. What makes the model a simulator rather than a video generator is the rollout loop plus faithful action adherence, which is precisely where these models are weakest and where interviewers push.

(1) Observation Encoder: a VQ or KL-regularized video autoencoder with spatial factor $f = 16$ to $32$ maps each frame to an $18 \times 32$ grid; it is trained first with perceptual and adversarial losses, then frozen.
(2) Action And Context Conditioning: a 2-D action (curvature and acceleration) is embedded by a small MLP and injected per frame at every block, as prefix tokens or through AdaLN modulation, alongside text prompts, agent bounding boxes, and camera calibration.
(3) Latent Dynamics Backbone: a causal spacetime transformer predicting the next token, or a diffusion/flow-matching transformer denoising a whole latent chunk conditioned on the past; this is where nearly all parameters live.
(4) Decoder And Auxiliary Heads: a diffusion or convolutional decoder for video, plus cheap heads for BEV occupancy, collision cost, and value that planning actually consumes.
(5) Two-Stage Training With Teacher Forcing: the dynamics model always sees ground-truth latents during training, which creates the exposure-bias gap that dominates long-horizon rollouts.
(6) Three Consumers: closed-loop policy evaluation against counterfactual actions, planning or RL in imagination, and generation of rare corner-case scenarios that fleets rarely log.

Left to right pipeline: five surround cameras feed a video tokenizer producing an 18 by 32 latent per frame, a dynamics transformer conditioned on action, text, agent boxes and camera pose predicts future latents with an autoregressive rollout loop, and a decoder renders predicted video and BEV occupancy

Figure 1: The four blocks of an action-conditioned world model. Only the frozen codec ever touches pixels, the action modulates every block of the dynamics model, and the rollout loop is what turns a video generator into a driving simulator.

How the action enters is the part worth rehearsing. A two-number control signal has to influence a token stream dominated by appearance, so injecting it once at the input is not enough: it is embedded per frame and re-applied at every layer, and it must be time-aligned to the interval it causes rather than the frame it was logged with. Two failure modes follow directly. First, shortcut learning: logged actions are almost perfectly predictable from the visible road geometry, so a model can minimize training loss while ignoring $a_t$ entirely, and you only detect this by rolling out counterfactual actions that contradict the scene. Second, weak controllability at sampling time, which is why conditioning is dropped for 10-20% of training samples so that classifier-free guidance can later amplify action adherence. Evaluation therefore needs an action-following metric (does the rendered ego trajectory match the commanded one?) next to FID or FVD, because generative realism and control fidelity move independently.

Mathematical Formulation:
$z_t = \mathcal{E}(o_t)$
$h_t = f_\theta(h_{t-1}, z_{t-1}, a_{t-1})$
$\hat{z}_t \sim p_\theta(z_t \mid h_t, a_{t-1})$
$\hat{o}_t = \mathcal{D}(h_t, \hat{z}_t)$
$\mathcal{L} = \mathcal{L}_{\text{obs}} + \beta \, \mathcal{L}_{\text{dyn}}$
$\mathcal{L}_{\text{obs}} = -\log p_\theta(o_t \mid h_t, z_t)$
$\mathcal{L}_{\text{dyn}} = D_{KL}\!\left(q_\phi(z_t \mid h_t, o_t) \,\|\, p_\theta(z_t \mid h_t)\right)$
$p(z_{1:T} \mid a_{1:T}) = \prod_{t=1}^{T} p(z_t \mid z_{1:t-1}, a_{1:t-1})$
$N = T \cdot (H/f) \cdot (W/f)$
$N = 26 \cdot 18 \cdot 32 = 14{,}976$

Where:

$o_t$ is the multi-camera observation at step $t$ and $\hat{o}_t$ its prediction; $a_t$ is the ego action (curvature and acceleration, or steering and pedal).
$z_t$ is the compressed observation latent and $h_t$ the deterministic recurrent or attention-carried state that summarizes the past.
$\mathcal{E}$ and $\mathcal{D}$ are the frozen encoder and decoder, $f_\theta$ the dynamics backbone, $p_\theta$ the action-conditioned prior, and $q_\phi$ the posterior that also sees the true observation.
$\beta$ balances reconstruction against the dynamics (KL) term; in practice it is annealed and often free-bits clipped so the prior does not collapse onto the posterior.
$T$ is the context length in frames, $H \times W$ the frame resolution, $f$ the spatial compression factor, and $N$ the token count whose square drives attention cost.
Required initial conditions for a rollout: $h_0 = 0$ and a real context $z_{1:k}$ encoded from logged frames, after which only actions are supplied and $j > k$ steps are pure imagination.

Left panel: bird's-eye view of three ego trajectories branching from the same start under hold-speed, brake and lane-change action sequences, with a braking lead vehicle and a predicted collision marker. Right panel: prediction error versus rollout horizon for one-step re-encoding, free-running rollout, and a variant with action tokens ablated

Figure 2: Left: fixing the initial latent and varying only the action sequence produces counterfactual rollouts, the property that makes the model usable for closed-loop evaluation. Right: because training is teacher-forced, free-running error compounds superlinearly, and ablating the action tokens makes it worse still.

Dynamics block	Discrete-token autoregressive	Latent diffusion / flow matching	Recurrent state-space (RSSM)
Latent representation	VQ codebook indices, 576 tokens per frame	Continuous latent grid, tens of channels, temporally compressed	Small vector state: deterministic GRU plus categorical stochastic units
How the action enters	Per-frame prefix tokens in the causal sequence	AdaLN modulation or cross-attention at every block, with conditioning dropout	Concatenated into the recurrent transition at each step
Rollout cost	Hundreds of sequential token decodes per frame; slowest	Tens of denoising steps per chunk, parallel across positions	One cheap matrix step per frame; fast enough for RL in imagination
Strength	Exact likelihoods, easy long-context scaling, LLM tooling reuse	Best photorealism and multi-view consistency; controllable via guidance	Compact enough to train a policy on millions of imagined steps
Typical failure mode	Quantization artifacts and drift after a few seconds of rollout	Plausible but unfaithful scenes; ignores the action unless guided	Blurry reconstructions; posterior collapse hides rare agents
Representative system	GAIA-1	GAIA-2, Vista, NVIDIA Cosmos	DreamerV3, MILE

August 17, 2026

DL0124 3D Occupancy Flow

What is 3D Occupancy Flow? What is the tradeoff for predicting dense spatiotemporal occupancy grids instead of discrete 3D bounding-box trajectories?

Answer

3D Occupancy Flow is a joint perception-and-forecasting output format that replaces the list of tracked objects with a dense grid: for every voxel (or BEV cell) and every future waypoint the network predicts an occupancy probability plus a flow vector describing how the mass in that cell moves. Waymo’s Occupancy Flow Fields formulation predicts three quantities per waypoint on a $256 \times 256$ BEV grid covering roughly $80\ \text{m} \times 80\ \text{m}$ : observed occupancy, occluded occupancy, and backward flow. Camera-only 3D variants such as Occ3D on nuScenes predict a $200 \times 200 \times 16$ voxel grid at $0.4\ \text{m}$ resolution, and Tesla presented an occupancy network with an occupancy-flow head at its 2022 AI Day. The appeal is that free space and obstacle geometry become class-agnostic and non-parametric: a tipped-over mattress, an articulated trailer, a swinging crane boom, and an overhanging branch all get represented without appearing in a detector taxonomy, and probability mass can sit on both branches of a fork at once without a mode head or non-maximum suppression. The cost is that a grid has no notion of an object, so instance identity, track continuity, and per-agent attributes disappear, and the output tensor grows by roughly two orders of magnitude, which pushes cost onto compute, memory, label pipelines, and the loss function’s handling of a grid where the overwhelming majority of voxels are empty. In practice this is why most production stacks run occupancy flow alongside a box pipeline rather than as a drop-in replacement.

(1) Dense Spatiotemporal Output: the head emits $o_t(v)$ and $f_t(v)$ for every cell and every future timestep, not a parametric box with a heading and a velocity per agent.
(2) Class-Agnostic Geometry: anything that occupies space is representable, which removes the long-tail detection failure where an unlisted object class becomes invisible to the planner.
(3) Backward Flow, Not Forward: predicting motion from $t$ back to $t-1$ makes warping a gather with one source per cell, so mass never collides during the warp and a flow-grounded occupancy consistency check becomes well defined.
(4) Non-Parametric Multimodality: a grid holds several futures simultaneously as spread probability mass, but that same property makes averaged modes look like blur or ghost occupancy rather than a ranked set of hypotheses.
(5) Identity Is What You Give Up: without instance IDs, right-of-way logic, interaction-aware conditioning, and per-object intent signals such as turn indicators lose their handle on the scene.
(6) Cost Scales With The Grid: output size grows as $H W D T$ , dense labels require multi-sweep LiDAR accumulation and voxelization, and roughly 95% or more of voxels are empty, so class imbalance dominates the occupancy loss.

Diagram contrasting two forecasting pipelines from the same sensor input: an object-centric branch running detector, tracker, and trajectory predictor to emit sparse box waypoints, and an occupancy-centric branch running a BEV or voxel encoder with occupancy and backward-flow heads to emit a dense spatiotemporal grid

Figure 1: The object-centric branch (detect → track → forecast) produces a sparse, identity-carrying output limited by its taxonomy; the occupancy branch skips detection and data association entirely and produces a dense, class-agnostic grid with no instance IDs.

The deeper tradeoff is not really compute, it is what the downstream planner can express. A box trajectory is a commitment: this vehicle, with this ID, will be here in 3 seconds with this probability, which lets a planner reason about yielding to a specific agent, replay a scenario in simulation, and produce an auditable explanation for a maneuver. Occupancy flow is a statement about space, which is exactly what collision checking and drivable-free-space queries want, but a grid that hedges between “the cyclist goes straight” and “the cyclist turns” paints both corridors at moderate probability, and a naive cost function that treats any occupancy above a threshold as blocked yields the freezing-robot behavior. Occupancy flow partially recovers correspondence without identity: warping the previous occupancy along the predicted backward flow and multiplying it against the current occupancy gives a differentiable consistency term, so the model is penalized for teleporting mass even though it never names an object.

Mathematical Formulation:
$0 \leq o_t(v) \leq 1$
$\hat{o}_t = o_t \odot \mathcal{W}(o_{t-1}, f_t)$
$\mathcal{L} = \mathcal{L}_{occ} + \lambda \mathcal{L}_{flow}$
$\mathcal{L}_{occ} = \sum_{t=1}^{T} \sum_{v} \mathrm{BCE}(o_t(v), y_t(v))$
$\mathcal{L}_{flow} = \sum_{t=1}^{T} \sum_{v \in \Omega_t} \lVert f_t(v) - f_t^{*}(v) \rVert_1$
$N_{occ} = 200 \cdot 200 \cdot 16 \cdot 8 = 5.12 \times 10^{6}$
$N_{box} = 50 \cdot 6 \cdot 16 \cdot 5 = 2.40 \times 10^{4}$

Where:

$o_t(v)$ is the predicted occupancy probability of cell $v$ at future waypoint $t$ , and $f_t(v)$ is the backward flow vector pointing to where that mass sat at $t-1$ .
$\mathcal{W}$ is the warp operator that gathers $o_{t-1}$ along $f_t$ (bilinear or trilinear), and $\hat{o}_t$ is the flow-grounded occupancy used both as a loss term and as an evaluation metric.
$y_t(v)$ is the voxelized ground-truth occupancy label, $f_t^{*}$ the ground-truth flow, and $\Omega_t$ the set of genuinely occupied cells to which the flow loss is masked.
$t \in \{1, \ldots, T\}$ indexes waypoints and $v$ indexes the $H \times W \times D$ grid; $\lambda$ balances the two terms, and $\mathrm{BCE}$ is usually replaced by a focal or class-balanced variant because empty voxels dominate.
$N_{occ}$ counts predicted occupancy values for a $200 \times 200 \times 16$ grid over 8 waypoints, and $N_{box}$ counts a comparable box head with 50 agents, 6 modes, 16 waypoints, and 5 numbers per waypoint.
Required initial condition: $o_0$ and the whole grid must be expressed in the ego frame at $t = 0$ , so ego motion is compensated before flow is interpreted as agent motion.

Grid diagram showing occupancy at time t minus one in light dashed cells and predicted occupancy at time t in solid cells, with backward flow arrows drawn from each occupied cell at time t to the cell its mass came from, alongside the flow-grounded occupancy product equation

Figure 2: Backward flow assigns each occupied cell at time $t$ a single source cell at $t-1$ , so the warp is a gather rather than a scatter and the consistency product $o_t \odot \mathcal{W}(o_{t-1}, f_t)$ penalizes mass that appears without a plausible origin.

Resolution and horizon are the two knobs that make or break the design. Halving the voxel size multiplies the tensor by 8 in 3D, and every additional waypoint is another full grid, so a $0.2\ \text{m}$ grid over a 5-second horizon at $2\ \text{Hz}$ is far beyond a real vehicle compute budget once flow channels and semantics are added. Going the other way is not free either: at $0.4\ \text{m}$ voxels a pedestrian walking at $1.4\ \text{m/s}$ moves less than one voxel per $0.2\ \text{s}$ frame, so the flow target is sub-voxel and the occupancy channel alone cannot express the motion, which is precisely why the flow head is kept as a continuous regression rather than a discrete cell-to-cell assignment.

Log-scale bar chart of predicted output size per inference: 24000 numbers for multi-modal box trajectories, 524288 for a 256 by 256 BEV occupancy grid over 8 waypoints, 5.12 million for a 200 by 200 by 16 voxel grid over 8 waypoints, and 20.48 million once three flow channels are added

Figure 3: Output size is where the tradeoff becomes concrete: the 3D voxel grid predicts about 213x more numbers than a multi-modal box head, and adding a 3-channel flow field multiplies that by another 4 before any semantic classes are included.

Property	3D occupancy flow	Box trajectories	Hybrid stack
Output per inference	Millions of per-cell values, occupancy plus flow per waypoint	Tens of thousands of numbers, a few modes per tracked agent	Both, sharing one BEV or voxel backbone
Unlisted geometry	Represented, since occupancy is class-agnostic	Dropped if no detector class fits it	Covered by the occupancy branch
Instance identity	None; only flow-based correspondence	Explicit IDs, attributes, and track history	IDs from the box branch, geometry from the grid
Multimodality	Implicit in the probability field, no mode count to tune	Explicit ranked modes with confidences	Explicit modes for interaction, field for collision checks
Supervision	Accumulated multi-sweep LiDAR voxelized into dense labels	Human box and track annotations only	Both label pipelines must be maintained
Typical failure mode	Blurred or ghost occupancy that makes the planner over-conservative	Missed detection or ID switch removes an obstacle entirely	Disagreement between branches needs an arbitration policy

August 17, 2026

DL0123 3D-Aware Visual Pretraining

How does 3D-aware visual pre-training bridge spatial intelligence between 2D generative video models and 3D physical world simulators?

Answer

3D-aware visual pre-training trains a visual backbone or a video generator with objectives that can only be solved by recovering scene geometry: per-pixel pointmaps, metric depth, camera rays, and cross-view correspondence, instead of appearance statistics alone. The two ends it connects have complementary deficits. A 2D video diffusion model absorbs enormous visual and dynamic diversity from internet footage, but its geometry is implicit, so it has no queryable state: you cannot ask it for the distance to a mug in centimeters, and a long rollout quietly changes room layout because nothing anchors the scene. A physics simulator has the opposite profile, exact meshes, contacts, and metric scale, but its asset library and rendering diversity are tiny compared with the real world. Pre-training with geometric targets creates the shared interface: the same representation gives the generator a camera and depth channel it can be conditioned on, and gives the simulator a path to ingest real scenes as reconstructions, so the pipeline runs video → pointmaps → 3D Gaussians or mesh → simulator asset in one direction and simulator depth, pose, and contact labels back into the pre-training loss in the other.

(1) Geometry-Grounded Pretext Tasks: pointmap, ray, and depth regression plus multi-view matching force the backbone to encode where surfaces are, not just what they look like; ordinary video supplies the multiple views for free.
(2) Camera As A First-Class Input: Plücker ray embeddings or pose tokens turn a text-to-video model into a controllable renderer, which is exactly the interface a simulator viewport exposes.
(3) Persistent State Beats Frame Memory: caching an explicit 3D representation and re-projecting it into each new frame removes the layout drift and object-identity loss that pure 2D autoregressive rollouts accumulate.
(4) Two-Way Data Flow: real-to-sim reconstruction fills the simulator’s asset gap, while simulator renders provide perfect metric depth, pose, and contact supervision that no internet video can give.
(5) 3D Awareness Is Measurable: linear probes on frozen features show that self-supervised 2D backbones encode single-view depth surprisingly well yet fail at multi-view consistency, and 3D-aware fine-tuning lifts both plus downstream segmentation.
(6) Metric Scale Is The Robot Requirement: scale-invariant depth is enough for a nice-looking novel view and useless for a grasp, so metric anchoring is the part embodied policies actually consume.

Diagram: internet video corpora and simulator renders both feed a shared backbone whose pointmap, camera-pose, and differentiable-render heads serve two consumers, a camera-controlled video generator and a 3D physics simulator

Figure 1: One backbone, two supervision sources, two consumers. Unlabeled video supplies visual diversity and self-consistency signals, simulator renders supply exact depth and pose, and the geometry heads are what the video generator and the simulator both read from.

The concrete form most systems settle on is a feed-forward multi-view transformer that predicts a pointmap per frame in a common coordinate frame, so camera intrinsics, extrinsics, depth, and correspondence all fall out of one prediction rather than a fragile SfM plus MVS chain. DUSt3R established the pointmap formulation for two views, MASt3R added a matching head, and VGGT scaled it to many frames with attention alternating between frame-local and global blocks. That output is directly liftable: initialize 3D Gaussians from the points for a renderable scene, or run meshing plus material estimation for a body a rigid-body engine can collide against. Two caveats decide whether the bridge is load-bearing. First, monocular training recovers geometry only up to an unknown similarity transform, so metric scale has to be injected from calibrated stereo, a known camera baseline, simulator ground truth, or object-size priors. Second, the reprojection and correspondence losses assume a static scene, so dynamic content needs motion masks or a 4D formulation or the model folds object motion into depth as flying pixels.

Mathematical Formulation:
$u = \pi(K, T_c, X)$
$\bar{X}_i = X_i / s$
$s = \frac{1}{N}\sum_i \lVert X_i \rVert_2$
$\mathcal{L}_{geo} = \sum_i \lVert \bar{X}_i - \bar{X}^{*}_i \rVert_1$
$\mathcal{L}_{corr} = \sum_i \lVert u'_i - \pi(K', T_{c'}, X_i) \rVert_2$
$\mathcal{L}_{rgb} = \lVert I_c - \mathcal{R}(G, K, T_c) \rVert_1$
$\mathcal{L} = \mathcal{L}_{2D} + \lambda_g \mathcal{L}_{geo}$
$+\ \lambda_c \mathcal{L}_{corr} + \lambda_r \mathcal{L}_{rgb}$

Where:

$u$ is a pixel coordinate and $X \in \mathbb{R}^3$ the predicted scene point for that pixel, expressed in the reference camera’s frame.
$\pi$ is the pinhole projection, $K$ the intrinsics, and $T_c$ the pose of camera $c$ ; primed symbols refer to a second view.
$i \in \{1, \ldots, N\}$ indexes valid pixels, and $X^{*}$ is the ground-truth pointmap from a simulator, LiDAR, or a reconstruction pipeline.
$s$ is the mean point norm used for scale normalization, so $\mathcal{L}_{geo}$ on $\bar{X}$ is scale-invariant; dropping the division makes the loss metric and requires metric labels.
$\mathcal{L}_{corr}$ is the reprojection error against a matched pixel $u'$ in the other view, and it is valid only where the scene is static between the two frames.
$\mathcal{R}$ is a differentiable renderer (splatting or volume rendering) over a lifted representation $G$ , letting 2D images supervise 3D structure through $\mathcal{L}_{rgb}$ .
$\mathcal{L}_{2D}$ is the original appearance objective (diffusion denoising or masked prediction) and $\lambda_g, \lambda_c, \lambda_r \geq 0$ weight the geometric terms.

Log-scale bar chart of global attention cost relative to a single view for 1, 2, 4, 8, 16, and 32 views at 1369 tokens per view, growing as the square of the view count

Figure 2: Joint multi-view geometry is what makes a representation 3D-aware, but attention over all views at once costs $O((VN)^2)$ , so a 32-view window is roughly 1,000x a single view. This is why practical models alternate frame-local and global attention and slide a bounded view window.

Property	2D generative video model	3D-aware pre-trained representation	3D physics simulator
Scene state	Implicit in activations and past frames; not queryable	Explicit pointmaps, depth, poses, optionally 3D Gaussians	Exact meshes, transforms, joints, contact manifolds
Camera control	Text prompts only, unless pose conditioning is added	Ray or pose embeddings make trajectory a direct input	Arbitrary and exact by construction
Metric scale	Absent	Available only if metric supervision or calibration is injected	Exact, in SI units
Visual and asset diversity	Internet scale, effectively unbounded	Inherits video-scale diversity, adds geometry heads	Limited to authored assets and shaders
Typical failure mode	Layout drift, object teleporting, impossible occlusions	Scale ambiguity, flying pixels on moving objects, cost of joint views	Appearance gap and unmodeled deformable or fluid behavior

August 17, 2026

DL0122 V-JEPA vs Video World Models

Compare V-JEPA (Joint-Embedding Predictive Architecture) with video generation world models such as OpenAI’s Sora or NVIDIA Cosmos. What does each objective actually learn, and when would you deploy one over the other?

Answer

Both families are trained by predicting the unseen part of a video, and both are called world models, but they differ in where the prediction target lives. V-JEPA masks a large 3D region of a clip and trains a predictor to regress the embeddings of that region produced by an EMA target encoder, using an L1 loss in representation space; there is no decoder anywhere in the system, so the model is never asked to reproduce a pixel. Sora and Cosmos Predict instead train a generative model (a diffusion transformer over spacetime latent patches, or an autoregressive token model) to reconstruct the actual frames, which forces the network to spend capacity on every detail, including detail that is fundamentally unpredictable, such as the exact texture of foliage or the phase of a reflection. That single design choice cascades: V-JEPA gets a compact motion-aware representation and a rollout that costs one forward pass per step, which is what makes energy-based planning tractable, while generative world models get a renderable future that can serve as a simulator, a synthetic-data generator, or an input to an existing camera-based perception stack. Neither dominates; the question to ask in an interview is whether your downstream consumer is a policy or a pixel pipeline.

(1) Target Space: V-JEPA minimizes distance between predicted and EMA-encoded latents; Sora and Cosmos minimize a reconstruction or denoising loss defined on tokenizer latents that decode back to RGB.
(2) Discarded Information Is A Feature: because the target is learned, JEPA can drop aleatoric high-frequency detail instead of averaging over it, which is exactly what produced blurry futures in older pixel-space predictors.
(3) Collapse Risk: a learned target can be gamed by a constant function, so JEPA needs stop-gradient plus an EMA teacher and aggressive 3D block masking; a generative model has a fixed data target and cannot collapse.
(4) No Rendering: V-JEPA cannot show you its prediction, so debugging and human review happen through probes and downstream task metrics, not by watching a video.
(5) Rollout Cost: one predictor call per latent step versus tens of denoiser calls plus a decoder pass, a difference of one to two orders of magnitude inside a model-predictive control loop.
(6) Evaluation Protocol: JEPA is scored with frozen attentive probes on motion-heavy benchmarks and with planning success rate; generative world models are scored with FVD, human preference, physics-consistency suites, and sim2real transfer of policies trained on their output.

Two stacked architecture rows: V-JEPA encodes a masked clip, runs a predictor, and compares predicted latents against stop-gradient EMA target latents with an L1 loss; a video world model encodes a clip with a causal tokenizer, iterates a DiT denoiser for tens of steps, and decodes back to frames

Figure 1: The two families share a video encoder but differ in the prediction target: V-JEPA regresses EMA target embeddings and never instantiates a pixel, while a generative world model passes through a tokenizer, an iterative denoiser, and a decoder that must reconstruct every frame.

The masking strategy is what makes the latent objective non-trivial. V-JEPA masks large 3D blocks that span the full temporal extent of the clip, removing on the order of 90 percent of the tubelet tokens, so the predictor cannot solve the task by interpolating from neighboring patches and is pushed toward object identity, motion, and rough physics. V-JEPA 2 scaled this recipe to a ViT-g encoder over more than a million hours of internet video and then post-trained an action-conditioned predictor on roughly 62 hours of unlabeled robot video, after which planning is just a search over action sequences whose predicted latent lands closest to a goal image embedding. Cosmos takes the opposite bet at similar scale, curating on the order of tens of millions of hours of video into diffusion and autoregressive world foundation models plus tokenizers, precisely because a physical-AI developer wants renderable, controllable rollouts they can feed into an existing autonomy stack. The two are complementary in practice: Cosmos ships a separate reasoning model alongside its generators, and several robotics stacks now use a generative model for data augmentation and a latent predictor for the control loop.

Mathematical Formulation:
$s_y = E_{\bar{\theta}}(y)$
$\hat{s}_y = P_{\phi}(E_{\theta}(x), m)$
$\mathcal{L}_{\mathrm{JEPA}} = \lVert \hat{s}_y - \mathrm{sg}(s_y) \rVert_1$
$\bar{\theta} \leftarrow \tau \bar{\theta} + (1 - \tau)\,\theta$
$\mathcal{L}_{\mathrm{gen}} = \mathbb{E}\left[\lVert \epsilon - \epsilon_{\theta}(z_t, t, c) \rVert_2^2\right]$
$a^{*} = \arg\min_{a} \lVert P_{\phi}(s_t, a) - s_g \rVert_1$

Where:

$x$ is the visible (context) part of a clip and $y$ the masked target region; $m$ carries the positional mask tokens telling the predictor where to predict.
$E_{\theta}$ is the online encoder, $E_{\bar{\theta}}$ the EMA target encoder, and $P_{\phi}$ the narrow predictor discarded after pretraining.
$\mathrm{sg}(\cdot)$ is the stop-gradient that blocks the trivial solution, and $\tau$ is the EMA momentum, typically ramped from about 0.998 toward 1.
$z_t$ is the noised tokenizer latent at diffusion step $t$ , $\epsilon$ the sampled Gaussian noise, $\epsilon_{\theta}$ the denoiser, and $c$ the conditioning (text, past frames, or actions).
$s_t$ is the current latent state, $a$ a candidate action sequence, and $s_g$ the goal embedding; the planner requires an initial condition $s_t = E_{\theta}(o_t)$ from the current observation.

Log-scale bar chart of forward passes per planning step for a 200-candidate 16-step CEM search: 3200 for a one-pass latent predictor, 12800 for a distilled four-step video diffusion model, and 96000 for a thirty-step video diffusion world model

Figure 2: For the same 200-candidate, 16-step CEM search, a single-pass latent predictor needs 3,200 network calls while a 30-step generative rollout needs 96,000, before the decoder is even invoked; this 30x gap is why planning loops favor latent prediction.

Property	V-JEPA (latent predictive)	Sora / Cosmos (generative world model)
Training target	Embeddings from an EMA copy of the encoder, L1 loss	Noise or next token on VAE latents that decode to RGB
Handles stochastic detail by	Dropping it from the representation	Sampling it, which costs capacity and steps
Can render a future	No decoder; futures exist only as vectors	Yes, watchable and consumable by any vision stack
Cost per rollout step	One predictor forward pass	Tens of denoiser passes plus a decode
Main pathology	Representation collapse and shortcut solutions under weak masking	Physically implausible but photoreal rollouts, plus error accumulation over long horizons
Natural deployment	Frozen backbone for recognition and anticipation; goal-conditioned planning	Synthetic data, neural simulators, scenario replay, content generation

August 17, 2026

DL0121 Open-Loop vs Closed-Loop Validation

Explain why open-loop validation metrics (e.g., MSE on offline trajectories) often fail to correlate with closed-loop success rates in physical robotics or driving environments.

Answer

Open-loop evaluation replays a logged trajectory and asks the policy to reproduce the expert action at states the expert visited, with ground-truth history fed back in at every step, so each prediction error is scored and then discarded. Closed-loop execution feeds every error into the next observation, so the policy’s own state distribution drifts off the demonstration manifold and errors compound: the classic behavior-cloning result turns a per-step error $\epsilon$ into an excess cost of up to $T^2 \epsilon$ over a $T$ -step episode, a bound that is already vacuous for a 200-step driving episode at $\epsilon = 0.01$ . Two further effects break the correlation before compounding even starts. MSE is mean-seeking: its minimizer is the conditional mean $\mathbb{E}[a \mid s]$ , and averaging the two valid modes of “swerve left or swerve right around the obstacle” produces a straight-line action that is optimal under the metric and a collision in the world. And metric mass is not risk mass: most logged frames are trivial lane-following, so average displacement error is dominated by how well a model extrapolates its own velocity, which is why an MLP fed only ego status with no perception input at all scored competitively on nuScenes open-loop L2 while being worthless as a planner.

(1) Covariate Shift: open loop measures the loss under $d_{\pi^*}$ , the expert’s state distribution, while success is determined under $d_{\pi}$ , the policy’s own induced distribution; the two diverge as soon as the policy acts.
(2) Compounding Error: teacher forcing caps the deviation at one step’s worth of error, whereas rollout integrates it, giving the quadratic-in-horizon gap that a single-step regression number cannot express.
(3) Mode Averaging: a squared-error objective on a multimodal action distribution returns an interpolation of the modes, which is frequently the one infeasible action available.
(4) Long-Tail Mismatch: safety outcomes are decided by a fraction of a percent of frames (cut-ins, occluded pedestrians, contact-rich grasps), and those frames contribute almost nothing to a dataset-averaged MSE.
(5) Non-Reactive Logs: logged agents never yield, brake, or negotiate, so open loop cannot score any behavior whose correctness depends on how the world responds to the ego.
(6) Shortcut Exploitation: ground-truth history leaks the answer; extrapolating the logged ego velocity minimizes displacement error without any scene understanding, and that shortcut vanishes the moment the policy controls its own history.

Two panels: left shows a logged expert trajectory with short prediction-error arrows at each sampled state and the state reset to the log after every step; right shows the same per-step error accumulating into a rollout that curves away from the dashed expert reference into off-distribution states

Figure 1: Open-loop scoring resets the policy onto the logged state after every prediction, so a bounded per-step error stays bounded; closed-loop rollout feeds each error into the next observation, and the visited states leave the training distribution where the policy has no guarantees at all.

The theory is unusually clean here. Ross and Bagnell showed that supervised imitation with per-step loss $\epsilon$ under the expert distribution admits an excess cost that grows as $T^2 \epsilon$ , because a mistake at step $t$ can put the agent in a state where it makes mistakes for all remaining $T - t$ steps; on-policy correction such as DAgger restores the linear $T \epsilon$ rate precisely by collecting labels on $d_{\pi}$ . The practical consequence is that ranking two policies by offline $\epsilon$ tells you almost nothing about their closed-loop ordering, since the multiplier between them differs by a factor of $T$ and depends on recovery behavior that the offline data never contains. Codevilla and colleagues measured this directly on vision-based driving models and found offline prediction error to be a weak predictor of on-road driving quality, and the nuPlan and NAVSIM benchmarks were built specifically because open-loop leaderboard position stopped tracking closed-loop driving score.

Mathematical Formulation:
$\epsilon = \mathbb{E}_{s \sim d_{\pi^*}}\left[\ell(s, \pi(s))\right]$
$J(\pi) - J(\pi^*) \leq T^2 \epsilon$
$\lVert d_{\pi} - d_{\pi^*} \rVert_1 \leq 2 T \epsilon$
$\pi_{\mathrm{mse}}(s) = \mathbb{E}[a \mid s]$
$\pi_{\mathrm{mse}}(s) = 0.5 a_L + 0.5 a_R$

Where:

$\epsilon$ is the offline per-step error that an open-loop MSE actually reports, and $\ell$ is the per-state surrogate loss (squared action error, displacement error).
$\pi$ is the learned policy, $\pi^*$ the expert, and $J$ the closed-loop episode cost, with per-step cost bounded in $[0, 1]$ .
$d_{\pi^*}$ and $d_{\pi}$ are the state distributions induced by the expert and by the policy; open loop samples the first, deployment samples the second.
$T$ is the episode horizon in control steps, the multiplier that an offline metric never sees; the bound becomes vacuous once $T^2 \epsilon \geq T$ .
$a_L$ and $a_R$ are two equally valid expert modes (pass left, pass right) at the same state $s$ ; their MSE-optimal average drives straight into the obstacle.
Required condition for the bounds: the offline data is drawn from $d_{\pi^*}$ with no on-policy correction, which is exactly the assumption behind pure behavior cloning.

Log-scale chart of excess closed-loop cost against episode length T for a fixed per-step error of 0.01: the offline metric is a flat line at 0.01, on-policy correction grows linearly as T times epsilon, and behavior cloning grows quadratically as T squared times epsilon

Figure 2: A single offline number $\epsilon$ is consistent with wildly different closed-loop outcomes, because the horizon $T$ is the multiplier and it is invisible to the metric; on-policy data collection is what changes the exponent from 2 to 1.

Property	Open-loop replay	Closed loop, log-replay agents	Closed loop, reactive agents or hardware
States visited	Expert distribution only	Policy distribution, but in a world frozen to the log	Policy distribution with a world that responds to it
Error feedback	None; state is reset each step	Ego error compounds; other agents do not react	Full two-way feedback including other agents
Typical metric	Action MSE, ADE/FDE, L2 at 1/2/3 s	Route completion, collision rate, comfort sub-scores	Task success rate, interventions per kilometer or per trial
Cost per evaluation	One forward pass per frame, fully parallel, seconds	Sequential rollout, hundreds of scenarios, minutes to hours	Highest; wall-clock hardware time or heavy sim agents
Main blind spot	Recovery, mode collapse to the mean, ego-status shortcuts	False collisions from behind, merges and nudges scored unfairly	Sim-to-real gap in sensing, or low statistical power on real hardware

August 17, 2026