Tag: Transformer

DL0132 LLM Serving Latency: TTFT and ITL

How do you measure and reduce Time to First Token (TTFT), Inter-Token Latency (ITL), and latency variance in a production LLM API such as a streaming chat endpoint served with vLLM?

Answer

TTFT is the wall-clock time from request arrival to the first streamed token and ITL is the gap between consecutive tokens after that, and they are separate metrics because they are produced by two phases with two different hardware bottlenecks. Prefill processes the whole prompt in one or a few forward passes and is compute-bound, so $T_{ttft}$ grows roughly linearly with prompt length; decode generates one token per forward pass and is memory-bandwidth-bound, so $T_{itl}$ is floored by the time to stream the weights and KV cache out of HBM. Measurement must happen at the streaming boundary and be reported as a distribution, namely p50, p95 and p99 merged from histograms and bucketed by input and output length, because a p99 computed over a mix of 200-token and 100k-token prompts is meaningless. Reduction is metric-specific: prefix caching and chunked prefill cut TTFT, FP8 quantization and speculative decoding cut ITL, and the variance work is almost entirely about removing things that can stall the decode loop, namely unchunked prefills, KV cache preemption, and queueing above roughly 70% utilization. The metric that actually decides capacity is goodput, the request rate you can sustain while both the TTFT and ITL SLOs hold at p99, not raw tokens per second.

(1) Two Phases, Two Bottlenecks: prefill is compute-bound and sets TTFT, decode is bandwidth-bound and sets ITL, so a change that helps one often does nothing for the other.
(2) Measure Distributions, Not Means: record one TTFT scalar plus an ITL vector of length $N_{out} - 1$ per request, and merge t-digest or HDR histograms across replicas instead of averaging per-replica p99s.
(3) TTFT Is Mostly Queueing Under Load: split server-side spans into $T_{queue}$ and $T_{prefill}$ , because past about 70% utilization the queue term dominates and no kernel optimization will help.
(4) The ITL Tail Is Interference, Not Steady State: p50 ITL reflects the decode step, while p99 ITL reflects the longest single stall, typically a colliding prefill or a preempted request being recomputed.
(5) Knobs Map To Metrics: prefix caching, chunked prefill and admission control for TTFT; quantization, GQA and speculative decoding for ITL; batch caps and prefill-decode disaggregation for variance.
(6) Optimize Goodput: report the sustained rate under both SLOs simultaneously, since raising batch size always trades TTFT and ITL tails for throughput.

Two timeline rows for one decoding request. In the baseline row, queue and prefill produce the first token, then four decode steps are interrupted by one wide unchunked prefill block that stalls the decode loop, creating a large inter-token gap. In the chunked-prefill row, the same prefill is split into six narrow chunks interleaved with decode steps, so every inter-token gap stays small.

Figure 1: TTFT is queue time plus prefill time; ITL is whatever happens between two decode steps of the same request. A single unchunked prefill scheduled into the shared loop stalls every decoding stream at once, which is the dominant source of p99 ITL. Splitting it into token-budgeted chunks bounds the worst stall to one chunk, at a small throughput cost from re-reading weights more often.

Instrument at the streaming boundary rather than inside the model: timestamp request arrival, the first SSE data event, and every subsequent event, then derive TTFT and the ITL vector client-side while the server emits its own queue, prefill and decode spans for the same request ID. The two views disagree in informative ways, because proxy buffering (an nginx deployment with proxy_buffering on) and a slow client event loop both inflate measured ITL with zero GPU involvement. For load testing, drive the endpoint with realistic input and output length distributions, since synthetic fixed-length prompts hide exactly the length-mixing effects that create the real tail; the vLLM benchmark_serving script and NVIDIA GenAI-Perf both report TTFT and ITL percentiles at a fixed request rate, which is the right shape of experiment. The variance checklist beyond scheduling is mundane but produces most of the surprises in practice: KV cache preemption and recompute when cache headroom runs out, ITL growing with batch size as decode drifts toward compute-bound, CUDA graph misses on uncaptured batch shapes, tensor-parallel all-reduce jitter, autoscaler cold starts that load tens of gigabytes of weights, and speculative decoding turning ITL bimodal because accepted-draft steps emit several tokens at once while rejected steps emit one.

Mathematical Formulation:
$T_{e2e} = T_{ttft} + (N_{out} - 1)\, \overline{T_{itl}}$
$T_{ttft} = T_{queue} + T_{prefill}$
$T_{prefill} \approx \frac{2 N_{in} P}{C_{eff}}$
$T_{itl} \geq \frac{B_{w} + B_{kv}}{BW}$
$T_{itl}^{spec} \approx \frac{T_{step}}{\mathbb{E}[n_{acc}]}$
$G = \lambda \cdot \Pr(T_{ttft} \leq S_{t},\ T_{itl} \leq S_{i})$

Where:

$T_{e2e}$ is end-to-end request latency, $T_{ttft}$ the time to first token, and $\overline{T_{itl}}$ the mean inter-token latency (its reciprocal is the perceived tokens per second of one stream).
$T_{queue}$ is scheduler wait time and $T_{prefill}$ the prompt forward pass; separating them is the single most useful piece of instrumentation, because they need opposite fixes.
$N_{in}$ and $N_{out}$ are prompt and completion token counts, $P$ the parameter count, and $C_{eff}$ the achieved (not peak) FLOP/s; the factor 2 counts one multiply and one add per parameter per token.
$B_{w}$ and $B_{kv}$ are the weight and KV cache bytes read per decode step and $BW$ is per-device HBM bandwidth, so this inequality is the bandwidth floor on ITL that no batching removes.
$T_{step}$ is one verification forward pass and $\mathbb{E}[n_{acc}] \geq 1$ the expected accepted tokens per step under speculative decoding, which divides effective ITL but widens its distribution.
$\lambda$ is offered request rate, $S_{t}$ and $S_{i}$ the TTFT and ITL SLO thresholds, and $G$ the resulting goodput, the only capacity number worth putting in a dashboard.

Plugging in real hardware makes the two bottlenecks concrete. A 70B model in BF16 needs roughly 140 GB of weights, so on eight H100s with tensor parallelism each device reads about 17.5 GB per decode step against roughly 3.35 TB/s of HBM, giving a floor near 5 ms per token before all-reduce and KV reads, which is why a single stream tops out around 100 to 150 tokens per second in practice. The same eight GPUs prefill a 2048-token prompt in about $2 \cdot 2048 \cdot 70$ B FLOPs, roughly 287 TFLOPs, which at an achieved 3.2 PFLOP/s is about 90 ms; a 32k-token prompt is 16 times that, which is exactly why long-context traffic destroys a shared TTFT SLO unless prefixes are cached or prefill is disaggregated. Note that FP8 weights halve $B_w$ and therefore the ITL floor, but barely move TTFT, while prefix caching can cut TTFT by an order of magnitude on a 1500-token shared system prompt and does nothing for ITL.

Two line charts against offered request rate. Left panel shows p50 and p99 TTFT in milliseconds rising sharply near capacity with a dotted 1500 ms SLO line and a vertical marker at the crossing rate. Right panel shows p50 and p99 inter-token latency in milliseconds with a dotted 80 ms SLO line crossed at a lower request rate, marking the binding constraint.

Figure 2: Both p50 curves stay flat while the p99 curves bend upward, because queueing and interference hit the tail first. Here the ITL SLO is crossed at a lower rate than the TTFT SLO, so goodput is ITL-bound and the correct response is to cap batch size or disaggregate, not to add prefill compute.

Technique	Metric it moves	What it costs
Prefix / KV cache reuse	TTFT p50 and p99 on shared system prompts, often 5x to 10x	HBM held by cached blocks, eviction policy tuning, cross-tenant isolation concerns
Chunked prefill	ITL p99 (bounds the worst decode stall to one chunk)	Slightly higher TTFT and a few percent lower throughput from repeated weight reads
Prefill-decode disaggregation	Both tails, by giving each phase its own pool and SLO	A KV cache transfer per request, duplicated weights, needs fast interconnect
Speculative decoding	ITL p50, roughly divided by accepted tokens per step	Bimodal ITL, wasted FLOPs on rejection, gains shrink at large batch sizes
FP8 or INT4 weight quantization	ITL floor, by halving or quartering bytes read per step	Accuracy regression that must be evaluated per task, calibration pipeline
Batch cap plus admission control	Latency variance, by refusing to accept work that will miss the SLO	Lower throughput and visible queueing or 429s, so it needs a priority policy

August 17, 2026

DL0131 DoRA: Weight-Decomposed LoRA

Explain DoRA (Weight-Decomposed Low-Rank Adaptation). How does separating weight magnitude from direction improve LoRA fine-tuning stability?

Answer

DoRA reparameterizes every pretrained weight matrix into a magnitude vector and a directional matrix, and then lets LoRA update only the direction. Concretely it writes $W_0 = m \frac{V}{\|V\|_c}$ , where $\|\cdot\|_c$ is the column-wise L2 norm, the trainable vector $m \in \mathbb{R}^{1 \times k}$ is initialized to $\|W_0\|_c$ , and the low-rank product $BA$ is added to the direction before renormalization. The motivation comes from a weight decomposition analysis: when full fine-tuning updates a layer, the magnitude change $\Delta M$ and the direction change $\Delta D$ are negatively correlated (a large rotation with only slight rescaling, or the reverse), while plain LoRA shows a positive, roughly proportional relation, so it can essentially only push both quantities in the same direction. Giving magnitude its own $k$ parameters removes that coupling, and the renormalization makes the gradient reaching the directional component orthogonal to the current direction, which is the classic weight-normalization conditioning effect: the low-rank branch can rotate the weight without simultaneously inflating its scale. In practice this buys a learning pattern that mirrors full fine-tuning, noticeably better accuracy at low rank (the DoRA paper reports about a 3 to 4 point average gain over LoRA on the eight commonsense-reasoning benchmarks with LLaMA-7B, and matching LoRA at half the rank in its ablations), and zero inference overhead, because $m V' / \|V'\|_c$ collapses back into one merged matrix.

(1) Two Degrees Of Freedom Instead Of One: LoRA has a single additive update $W_0 + BA$ ; DoRA splits the same layer into a scalar-per-column scale and a normalized direction, so scale and rotation are optimized by separate tensors.
(2) Exact Identity At Initialization: with $B = 0$ and $m_0 = \|W_0\|_c$ the adapted layer reproduces $W_0$ bit-for-bit, so training starts from the pretrained function with no loss spike.
(3) Gradient Is Projected, Not Just Scaled: differentiating through the column norm inserts the projector $I - V'V'^{\top} / \|V'\|_c^2$ , which removes the radial component of the directional gradient and bounds its effective step.
(4) Learning Pattern Matches Full Fine-Tuning: measured over training, DoRA reproduces the negative $\Delta M$ versus $\Delta D$ slope of full fine-tuning, whereas LoRA’s slope is positive.
(5) Cheapest Where LoRA Hurts Most: the extra magnitude vector costs $k$ parameters per layer, roughly 6% on top of a rank-8 adapter for a $4096 \times 4096$ projection, and pays off most at $r = 4$ to $r = 8$ .
(6) Training Overhead, Not Serving Overhead: the column norm of $W_0 + BA$ must be recomputed each step, which raises training memory; detaching that norm from the backward graph recovers roughly a quarter of the overhead with negligible accuracy change, and inference is unaffected after merging.

Data-flow diagram of a DoRA layer: the frozen pretrained weight is decomposed into a trainable magnitude vector and a frozen normalized direction, the low-rank product BA is added to the direction, the sum is renormalized column-wise, then scaled by the magnitude vector to produce the merged adapted weight

Figure 1: A DoRA layer. Only the magnitude vector $m$ and the LoRA factors $A, B$ receive gradients; the frozen direction is perturbed by $BA$ , renormalized per column, and rescaled by $m$ . Because the final expression is a single matrix, the adapter still merges away at inference.

The stability argument is easier to see from what LoRA cannot express. In LoRA the only knob is the additive term, so any attempt to rescale a column also rotates it, and any attempt to rotate it also changes its norm; the two effects are welded together by a single rank- $r$ product, which is why its measured $(\Delta D, \Delta M)$ points fall on a line with positive slope. Full fine-tuning has $dk$ free parameters and shows the opposite pattern, with subtle magnitude changes accompanying substantial directional change. DoRA recovers that behavior with $k$ extra parameters, and the practical consequences are the ones interviewers care about: lower sensitivity to the learning rate and to the LoRA scaling factor $\alpha / r$ , because a badly scaled low-rank branch is renormalized away instead of blowing up the weight norm, and much flatter accuracy-versus-rank curves, because a rank-4 direction plus a full-rank magnitude still spans useful updates.

Scatter plot of magnitude change versus direction change per layer for full fine-tuning, LoRA, and DoRA with fitted regression lines; full fine-tuning and DoRA have negative slopes while LoRA has a positive slope

Figure 2: Measured magnitude change $\Delta M$ against direction change $\Delta D$ per adapted module. Full fine-tuning and DoRA trace a negative slope, meaning scale and rotation are traded against each other, while LoRA‘s positive slope shows the two are coupled by its single additive term.

Mathematical Formulation:
$W_0 = m_0 \frac{V_0}{\|V_0\|_c}$
$m_0 = \|W_0\|_c$
$W' = m \frac{W_0 + BA}{\|W_0 + BA\|_c}$
$\nabla_{V'}\mathcal{L} = \frac{m}{C}\left(I - \frac{V'V'^{\top}}{C^2}\right)\nabla_{W'}\mathcal{L}$
$\Delta M_t = \frac{1}{k}\sum_{j} |m_t^{j} - m_0^{j}|$
$\Delta D_t = \frac{1}{k}\sum_{j} \left(1 - \cos(V_t^{j}, W_0^{j})\right)$

Where:

$W'$ is the adapted weight actually used in the forward pass, and $W_0 \in \mathbb{R}^{d \times k}$ is the frozen pretrained weight it starts from.
$m \in \mathbb{R}^{1 \times k}$ is the trainable magnitude vector and $V' = W_0 + BA$ the unnormalized direction, with $V_0 = W_0$ and $m_0$ the initialization that makes $W' = W_0$ at step 0.
$B \in \mathbb{R}^{d \times r}$ (initialized to zero) and $A \in \mathbb{R}^{r \times k}$ are the LoRA factors with rank $r \ll \min(d,k)$ .
$j \in \{1,\ldots,k\}$ indexes output columns, $t$ indexes training checkpoints, and $\|\cdot\|_c$ takes the L2 norm of each column independently, returning a $1 \times k$ row vector.
$C = \|V'\|_c$ is that column norm, and $I - V'V'^{\top}/C^2$ is the orthogonal projector that strips the radial part of the gradient, so the directional update is norm-preserving to first order.
$\Delta M_t$ and $\Delta D_t$ are the magnitude and direction change metrics whose correlation is plotted in Figure 2, with $\Delta D_t \in [0,2]$ because it is one minus a cosine similarity.

Property	DoRA	LoRA	Full fine-tuning
Trainable tensors per layer	A, B, and the magnitude vector m	A and B only	The entire weight matrix
Parameters added	r(d + k) + k, about 6% more than LoRA at r = 8, k = 4096	r(d + k)	dk, plus optimizer state for all of it
Magnitude vs direction coupling	Decoupled; negative correlation like full fine-tuning	Coupled through one additive term; positive correlation	Fully free, the reference behavior
Behavior at very low rank	Degrades slowly; competitive at r = 4 to 8	Degrades quickly below r = 8 on harder tasks	Not applicable
Training and serving cost	Extra column-norm recompute each step; merges cleanly, no inference cost	Cheapest to train; merges cleanly, no inference cost	Full optimizer and gradient memory; one checkpoint per task

August 17, 2026

DL0130 MoE Router Collapse and Auxiliary Loss

What is Router Collapse in Sparse Mixture-of-Experts (MoE) LLMs? Derive the auxiliary load-balancing loss and Router Z-loss used to stabilize MoE training.

Answer

Router collapse is the failure mode in which the learned router of a sparse MoE layer stops using most of its experts and concentrates almost all token assignments on a small subset, so a layer you paid $N$ experts’ worth of memory for behaves like a much smaller model. The cause is a positive feedback loop: an expert that happens to win slightly more tokens early in training receives more gradient signal, becomes genuinely better on those tokens, so the router raises its logit further, while starved experts never see enough tokens to become useful. Under token-choice top-k the visible symptoms are a skewed load histogram, a large dropped-token fraction once the hot experts hit their capacity buffer, and a validation loss that tracks a dense model of the active-parameter size rather than the total-parameter size. Production training stacks suppress it with two extra loss terms rather than with a new routing algorithm: an auxiliary load-balancing loss that pushes router probability mass away from overloaded experts, and a router z-loss that penalizes the squared log-sum-exp of the router logits so the logits cannot grow without bound and destabilize the softmax in low precision. Both are differentiable surrogates for quantities that are not: the actual assignment counts are piecewise constant, and the actual numerical blow-up is a hardware property, so each loss attacks a proxy the gradient can reach.

(1) Collapse Is Self-Reinforcing: a small early advantage in router logits compounds through the gradient the winning expert receives, which is why collapse typically happens in the first few thousand steps and is nearly irreversible afterwards.
(2) The Assignment Is Non-Differentiable: the load fraction $f_i$ comes from a top-k selection and has zero gradient almost everywhere, so the balance loss must pair it with the differentiable mean gate probability $P_i$ .
(3) Balance Loss Is A Normalized Dot Product: $N \sum_i f_i P_i$ equals 1 under a uniform assignment and grows toward $N$ under total collapse, giving a scale-free objective independent of expert count.
(4) Its Gradient Is Load-Proportional: the derivative with respect to $P_i$ is exactly $\alpha N f_i$ , so probability mass is pushed down in proportion to how overloaded each expert already is.
(5) Z-Loss Bounds The Logits: penalizing the squared log-partition function caps $\max_j h_j$ , which matters because the relative round-off in $e^{h}$ grows linearly with $|h|$ in bfloat16 and can flip top-k selection.
(6) Coefficients Are A Quality Trade: $\alpha \approx 10^{-2}$ and $\beta \approx 10^{-3}$ are the common settings, because a large $\alpha$ buys perfect balance by actively fighting the language-modeling objective.

Cycle diagram of router collapse: router logits for one expert edge above the others, top-k dispatches more tokens to it, that expert receives most of the expert gradient, the starved experts stay undertrained and score lower, which feeds back into the logit gap; two green intervention boxes on the right show the router z-loss acting on the logits node and the auxiliary balance loss acting on the dispatch node

Figure 1: Router collapse is a closed loop, not a single bad step: the logit gap, the dispatch skew, and the gradient imbalance each amplify the next. The two stabilizers cut the loop at different points, with the z-loss constraining the logit magnitudes and the balance loss constraining the dispatch distribution.

The derivation of the balance loss starts from what you actually want to penalize, namely the variance of the per-expert token counts, and then asks which part of that quantity carries a gradient. The counts themselves come from a top-k over the router logits, so they are a step function of the parameters and give nothing to backpropagation. The fix used by GShard and simplified by Switch Transformers is to pair the non-differentiable load vector $f$ with the differentiable importance vector $P$ , the mean softmax probability per expert, and minimize their inner product. Treating $f$ as a constant, the objective is linear in $P$ with coefficient $N f_i$ , so each step lowers the router’s probability for exactly the experts that were overloaded in the current batch, and because $f$ is recomputed every step this becomes a self-correcting controller whose fixed point is the uniform assignment. The factor $N$ is a normalization choice: it makes the minimum value 1 regardless of how many experts you have, so the same $\alpha$ transfers from an 8-expert layer to a 256-expert layer. The z-loss has an entirely different motivation: it is a numerical guard, derived from the observation that a logit stored with relative precision $\epsilon$ produces an absolute error of about $|h|\epsilon$ , and exponentiation converts that absolute error into a relative error of the same size, so large logits make the softmax and therefore the selected expert set unreliable.

Mathematical Formulation:
$h_t = W_r x_t, \quad p_t = \mathrm{softmax}(h_t)$
$f_i = \frac{1}{T}\sum_{t=1}^{T} \mathbb{1}[i \in \mathcal{T}_t]$
$P_i = \frac{1}{T}\sum_{t=1}^{T} p_{t,i}$
$\mathcal{L}_{\mathrm{bal}} = N \sum_{i=1}^{N} f_i P_i$
$\frac{\partial \mathcal{L}_{\mathrm{bal}}}{\partial P_i} = N f_i$
$1 \leq \mathcal{L}_{\mathrm{bal}} \leq N$
$\mathcal{L}_{z} = \frac{1}{T}\sum_{t=1}^{T}\left(\log \sum_{j=1}^{N} e^{h_{t,j}}\right)^{2}$
$\max_j h_{t,j} \leq \mathrm{lse}(h_t) \leq \max_j h_{t,j} + \log N$
$\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \alpha \mathcal{L}_{\mathrm{bal}} + \beta \mathcal{L}_{z}$

Where:

$x_t \in \mathbb{R}^{d}$ is the hidden state of token $t$ , $W_r \in \mathbb{R}^{N \times d}$ the router matrix, and $h_t$ the router logits before any selection.
$T$ is the number of tokens the statistics are aggregated over (micro-batch, device batch, or global batch), $N$ the expert count, and $\mathcal{T}_t$ the set of $k$ experts selected for token $t$ .
$f_i$ is the load, the fraction of tokens dispatched to expert $i$ ; it is piecewise constant in the parameters and therefore contributes no gradient.
$P_i$ is the importance, the mean router probability assigned to expert $i$ ; it is smooth, so all of the balance-loss gradient flows through it and into $W_r$ .
$\mathcal{L}_{\mathrm{bal}} = 1$ exactly when $f_i = P_i = 1/N$ for all $i$ , and approaches $N$ when one expert takes every token, so the value is directly readable as an imbalance factor.
$\mathrm{lse}(h_t) = \log\sum_j e^{h_{t,j}}$ is the log-partition function; squaring it penalizes large logits in either direction and, by the sandwich bound, keeps $\max_j h_{t,j}$ within $\log N$ of a small target.
$\alpha$ and $\beta$ are the balance and z-loss coefficients, commonly $\alpha = 10^{-2}$ and $\beta = 10^{-3}$ ; $\mathcal{L}_{\mathrm{LM}}$ is the ordinary next-token cross-entropy.

Two-panel chart: left panel plots the maximum expert load fraction against training step for an eight-expert layer with balance-loss coefficients zero, one thousandth, and one hundredth, showing full collapse toward one expert without the loss and a curve close to the uniform 0.125 line with alpha one hundredth; right panel plots relative round-off error in exp of a router logit against logit magnitude on a log scale for bfloat16 and float32 storage, with bfloat16 crossing one percent error at a logit magnitude near 2.6

Figure 2: The two losses guard different quantities. Left: without a balance term the maximum load fraction runs from the uniform $1/N = 0.125$ to near 1.0 within a few thousand steps, while $\alpha = 10^{-2}$ holds it close to uniform. Right: the relative round-off in $e^{h}$ grows linearly with $|h|$ , which is why bfloat16 routers become unreliable at moderate logits and the z-loss keeps $\mathrm{lse}(h)$ near $O(1)$ .

Property	Auxiliary balance loss	Router z-loss	Bias-based loss-free balancing
Quantity penalized	The dot product of load and importance, scaled by the expert count	The squared log-sum-exp of the router logits, averaged over tokens	Nothing; a per-expert bias is added to the selection logits only
Failure it prevents	Expert collapse, wasted parameters, and dropped tokens at the capacity buffer	Logit blow-up, low-precision softmax round-off, and loss spikes	Expert collapse, without perturbing the gate weights used in the output
Typical setting	Coefficient 1e-2, aggregated per device batch or per global batch	Coefficient 1e-3, with the router itself computed in float32	Bias update rate around 1e-3, driven by observed per-expert load error
Effect on the LM objective	Adds an interference gradient that trades some quality for balance	Mild regularizer, usually neutral or slightly positive for quality	No interference term at all, since the bias is not part of the output gate
Where it is used	GShard, Switch, Mixtral, OLMoE	ST-MoE and most later open MoE training stacks, including OLMoE	DeepSeek-V3 and its loss-free-balancing follow-ups

August 17, 2026

DL0129 SwiGLU, GELU, and ReLU FFNs

Compare SwiGLU, GeLU, and ReLU activation functions in the Feed-Forward Network (FFN) blocks of LLMs, as used in models like Llama 3 and GPT-3.

Answer

All three sit in the same place, namely the two-layer FFN that follows attention in every transformer block, but they differ in smoothness and in whether the nonlinearity is elementwise or multiplicative. ReLU is $\max(0,x)$ : cheapest, produces exact zeros (roughly 90-95% of FFN units are zero on a typical token), but has a kink at the origin and zero gradient for negative inputs, which gives dead units. GELU is $x\Phi(x)$ : smooth, non-monotonic, keeps a small negative response, and was the default through BERT, GPT-2, and GPT-3. SwiGLU is not an elementwise function at all but a gated layer: the up-projection is split into a gate branch passed through SiLU ( $x\sigma(x)$ ) and a value branch left linear, and the two are multiplied elementwise. That third weight matrix is why practitioners shrink the hidden width by $2/3$ to stay parameter- and FLOP-neutral, and the payoff reported by Shazeer’s GLU-variants ablation is a consistent but modest gain, about 0.05 nats of pretraining log-perplexity on T5’s span-corruption objective at matched budget, which is enough that PaLM, Llama, Mistral, and Qwen2 all adopted it. The cost is one extra matmul, one extra saved activation for the backward pass, loss of exploitable sparsity, and a wider activation range that makes low-bit quantization harder.

(1) Two Matrices Versus Three: a dense FFN is $W_{up}$ then $W_{down}$ ; a gated FFN adds $W_g$ , so parameters go from $2 d\, d_{ff}$ to $3 d\, d_{ff}$ at the same width.
(2) The 2/3 Width Rule: to keep the block budget fixed you set $d_{ff} = \frac{8}{3}d$ instead of $4d$ , which is exactly why Llama 2 7B uses 11008 with $d = 4096$ .
(3) Smoothness Changes Optimization: ReLU’s derivative is a step with no gradient for negative inputs, while GELU and SiLU are $C^\infty$ and pass a small negative signal, so units recover instead of dying.
(4) Gating Is A Multiplicative Interaction: the output is a product of two learned projections of the same input, giving a data-dependent, per-channel scaling that no elementwise function can express.
(5) Sparsity Is A ReLU-Only Asset: exact zeros let inference systems skip entire rows of $W_{down}$ ; GELU and SwiGLU produce values that are small but almost never exactly zero.
(6) SwiGLU Costs Memory Traffic And Range: three matmuls plus an extra stored tensor per layer, and the product of two unbounded projections widens the dynamic range that quantization must cover.

Two-panel line chart: left panel plots ReLU, GELU and SiLU over the range minus four to four, showing ReLU flat at zero for negative inputs while GELU and SiLU dip slightly below zero; right panel plots their derivatives, showing ReLU's step from zero to one at the origin versus the smooth GELU and SiLU derivatives that briefly exceed one and go slightly negative

Figure 1: Shape (left) and derivative (right). ReLU’s derivative is a step function with a kink at the origin and no gradient for negative inputs; GELU and SiLU are smooth, non-monotonic, and dip below zero (SiLU’s minimum is $-0.278$ at $x = -1.28$ ), so a unit pushed into the negative region still receives gradient and can come back.

The historical progression is ReLU → GELU → gated GLU variants, and it is worth being honest that only the first step has a clean story. Replacing ReLU with GELU removes the dead-unit problem and gives a slightly better-conditioned loss surface. Replacing an elementwise nonlinearity with a gate is different in kind: the value branch stays linear, and the gate branch modulates it, so the layer computes a second-order interaction of the input with itself. Shazeer’s paper explicitly declines to explain why this helps, attributing the gain to luck rather than theory, and later analyses mostly note that gating gives the block a cheap multiplicative pathway similar in spirit to the gates in an LSTM. What made SwiGLU the production default is that the gain survives at scale, needs no extra hyperparameters, and is free once you apply the $2/3$ width rule. Llama 3 8B keeps the gated form but widens back out to $d_{ff} = 14336$ with $d = 4096$ , a deliberate choice to spend more parameters in the FFN rather than a parameter-neutral swap.

Mathematical Formulation:
$\mathrm{FFN}_{\mathrm{ReLU}}(x) = \max(0, xW_{up})\,W_{down}$
$\mathrm{GELU}(x) = x\,\Phi(x)$
$\mathrm{SiLU}(x) = x\,\sigma(x)$
$\mathrm{SwiGLU}(x) = \mathrm{SiLU}(xW_g) \odot (xW_u)$
$\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \mathrm{SwiGLU}(x)\,W_{down}$
$d_{ff} = \frac{2}{3}\cdot 4d = \frac{8}{3}d$

Where:

$x \in \mathbb{R}^{d}$ is the post-norm hidden state for one token, and the FFN output has the same dimension $d$ so it can be added back through the residual connection.
$W_{up}, W_g, W_u \in \mathbb{R}^{d \times d_{ff}}$ are up-projections and $W_{down} \in \mathbb{R}^{d_{ff} \times d}$ the down-projection; the dense block uses $W_{up}$ only, the gated block uses both $W_g$ (gate) and $W_u$ (value).
$\Phi$ is the standard normal CDF and $\sigma$ the logistic sigmoid; GELU is often shipped as the tanh approximation $0.5x(1+\tanh(\sqrt{2/\pi}(x+0.044715x^3)))$ , which is why two frameworks can disagree in the last few digits.
$\odot$ is the elementwise product over the $d_{ff}$ hidden channels, the operation that turns a gate into a multiplicative interaction rather than a pointwise map.
$d_{ff}$ is the FFN hidden width; the last line is the parameter-neutral setting, since $3 d \cdot \frac{8}{3}d = 8d^2 = 2 d \cdot 4d$ , and real implementations round the result to a hardware-friendly multiple such as 256.

Architecture diagram with two stacked left-to-right flows: the top flow is a dense FFN with hidden state, one up-projection, an elementwise ReLU or GELU, a down-projection and output, annotated with two d times d_ff parameters; the bottom flow is a gated SwiGLU FFN where the hidden state branches into a SiLU gate path and a linear value path, the two are combined by an elementwise product, then a down-projection and output, annotated with three d times d_ff parameters and the two-thirds width rule

Figure 2: The structural difference is a branch, not a curve. A dense FFN applies one elementwise function between two matrices; a gated FFN splits the up-projection into a SiLU gate path and a linear value path whose elementwise product feeds $W_{down}$ . The third matrix is what forces the $8d/3$ width if the block budget must stay fixed.

Property	ReLU	GELU	SwiGLU
Form	Elementwise, piecewise linear	Elementwise, smooth, non-monotonic	Gated layer: SiLU branch times linear branch
Matrices per FFN	2	2	3, so width drops to 8d/3 to stay budget-neutral
Gradient behaviour	0 or 1; dead units never recover	Smooth, briefly exceeds 1, small negative region	Smooth, plus a gradient path through the gate itself
Exact-zero sparsity	Yes, typically 90-95% of units, exploitable at inference	No; values are small but nonzero	No; near-zero mass only, so row skipping needs a threshold and loses accuracy
Quantization friendliness	Best: bounded below, no multiplicative amplification	Good	Hardest: the product of two unbounded projections widens outliers into W_down
Representative models	Original Transformer, T5 v1.0, OPT	BERT, GPT-2, GPT-3, ViT	PaLM, Llama 2 and Llama 3, Mistral, Qwen2 (Gemma uses the GELU-gated GeGLU)

August 17, 2026

DL0128 Subword Regularization

What is subword regularization, and why can it help machine translation quality?

Answer

Subword regularization is the practice of training a translation model on multiple stochastically sampled segmentations of the same sentence instead of the one deterministic segmentation a tokenizer normally emits. Any subword vocabulary is ambiguous: the word “smaller” can be a single piece, or small plus er, or sm plus all plus er, and a standard tokenizer always returns the same choice, so the model never sees the alternatives. Two mechanisms implement the sampling: unigram-LM sampling (Kudo 2018, shipped in SentencePiece), which draws a path from the segmentation lattice with probability proportional to $P(\mathbf{x})^{\alpha}$ , and BPE-dropout (Provilkov 2020), which keeps the ordinary BPE merge table but skips each merge with probability $p$ during application. The quality gain has three sources: it is free data augmentation over the same parallel corpus, it trains the embeddings of rare pieces that a deterministic segmenter almost never produces, and it makes the encoder robust to the segmentation shifts caused by typos, morphology, and domain change. Reported gains are roughly 1 BLEU on average, shrinking toward noise on large in-domain high-resource benchmarks and reaching 2 BLEU or more on low-resource pairs and out-of-domain test sets. Sampling is a training-time-only device: decoding uses the deterministic segmentation, or optionally averages scores over $k$ sampled segmentations of the source.

(1) Segmentation Is Ambiguous: a fixed vocabulary admits exponentially many segmentations per sentence, and deterministic tokenization silently commits to one of them in every epoch.
(2) Unigram-LM Sampling: SentencePiece samples from the $l$ -best lattice paths with a temperature $\alpha$ , typically $l = 64$ and $\alpha$ near 0.1 to 0.2, where small $\alpha$ is near-uniform and $\alpha \to \infty$ recovers the Viterbi segmentation.
(3) BPE-Dropout: drop each merge operation independently with $p = 0.1$ on both source and target; the vocabulary and merge table are unchanged, so it is a drop-in change to the tokenizer call.
(4) Approximate Marginalization: the model is pushed to score a sentence pair well under any plausible segmentation, which is closer to the true marginal $P(Y \mid X)$ than a single-path proxy.
(5) Robustness Beats In-Domain Accuracy: the largest wins appear on low-resource, morphologically rich, noisy, and out-of-domain data, exactly where the test-time segmentation drifts from the training-time one.
(6) Costs Are Real: average sequence length grows, tokenization moves into the data loader and cannot be cached, and convergence is slower in wall-clock terms for the same number of updates.

Segmentation lattice for the word smaller: boundary nodes under the characters, with three bracketed paths drawn above them, one spanning the whole word as a single piece, one splitting into small plus er, and one splitting into sm plus all plus er, each annotated with its unigram probability and the sampling weight at alpha equal to 0.2

Figure 1: One word, three legal paths through the segmentation lattice. Deterministic tokenization always returns the top path, so the -er morpheme boundary is never shown to the encoder; sampling at $\alpha = 0.2$ returns the top path only about half the time and spends the rest of the probability mass on the finer decompositions.

The reason the same trick helps two different segmenters is that both are inverting a latent variable the corpus never labels. A unigram language model gives an explicit probability to each path, so its lattice can be sampled cleanly and the temperature $\alpha$ gives a continuous dial from “uniform over the lattice” to “deterministic”. BPE has no probability model at all, only an ordered merge list, so BPE-dropout injects randomness procedurally instead: skipping a merge with probability $p$ produces a shorter piece and therefore a longer, more character-like segmentation, and stacking many independent skips over a long word yields a wide distribution of segmentations. Both settings need a knob that is gentle: too much randomness pushes segmentation toward characters, inflating sequence length, wasting attention on re-learning spelling, and destroying the frequency statistics that made the vocabulary useful in the first place.

Mathematical Formulation:
$P(\mathbf{x}) = \prod_{i=1}^{M} p(x_i)$
$P_{\alpha}(\mathbf{x} \mid X) \propto P(\mathbf{x})^{\alpha}$
$\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x}, \mathbf{y}}\left[\log P(\mathbf{y} \mid \mathbf{x}; \theta)\right]$
$P(Y \mid X) = \sum_{\mathbf{x}} P(\mathbf{x} \mid X)\, P(Y \mid \mathbf{x})$
$\hat{Y} = \arg\max_{Y} \frac{1}{k} \sum_{j=1}^{k} \log P(Y \mid \mathbf{x}_j)$

Where:

$X$ and $Y$ are the raw source and target sentences, while $\mathbf{x} = (x_1, \ldots, x_M)$ and $\mathbf{y}$ are particular segmentations of them into vocabulary pieces.
$p(x_i)$ is the unigram probability of piece $x_i$ , estimated by EM when the SentencePiece unigram vocabulary is built, and $M$ is the number of pieces on that path.
$\alpha$ is the smoothing exponent of the sampling distribution: $\alpha \to 0$ gives a near-uniform draw over the $l$ -best lattice paths and $\alpha \to \infty$ collapses to the single Viterbi path, so deterministic tokenization is the limiting case.
The expectation in $\mathcal{L}(\theta)$ is taken over $\mathbf{x} \sim P_{\alpha}(\cdot \mid X)$ and $\mathbf{y} \sim P_{\alpha}(\cdot \mid Y)$ , approximated by one fresh sample per example per epoch rather than by an inner loop.
$P(Y \mid X)$ is the true segmentation-marginalized translation probability that the training objective approximates; the sum over $\mathbf{x}$ is intractable, which is exactly why sampling is used.
$k$ is the number of source segmentations used in optional n-best decoding, where $k = 1$ with the Viterbi path is the standard, cheapest inference setting.

Two-panel line chart: left panel shows the entropy in bits of the segmentation sampling distribution decreasing from near-maximum to near zero as alpha grows from 0.02 to 3 for a three-path and a four-path lattice; right panel shows the expected number of pieces per word falling toward one over the same alpha range, both panels marking the default alpha near 0.2

Figure 2: The temperature $\alpha$ controls exactly two things that matter. Left: the entropy of the sampling distribution, which is how much augmentation diversity the model receives per epoch. Right: the expected pieces per word, which is the sequence-length tax you pay for that diversity. Beyond roughly $\alpha = 1.5$ both collapse to the deterministic Viterbi behaviour and the method does nothing.

Property	Deterministic BPE / unigram	Unigram-LM sampling	BPE-dropout
Randomness source	None; one path per sentence forever	Draw from the l-best lattice paths of a unigram LM	Independently skip each merge while applying the merge table
Knob and typical value	Vocabulary size only	l = 64, alpha near 0.1 to 0.2	p = 0.1 on both sides, lower for high-resource pairs
Segmenter changes	n/a	Needs a probabilistic unigram vocabulary, not a merge list	None; reuses an existing BPE model unchanged
Main cost	Rare pieces stay under-trained	Lattice sampling per example; longer average sequences	Re-merging per example; length inflation grows fast with p
Inference	Same path as training	Viterbi path, or average log-scores over k sampled sources	Standard BPE with dropout disabled

August 17, 2026

DL0127 DDPM Training Objective

What is the DDPM training objective, and why do we train the network to predict the noise rather than the clean image directly?

Answer

A denoising diffusion probabilistic model is trained by maximizing a variational bound on the data log-likelihood, and because both the forward corruption and the reverse posterior are Gaussian, that bound collapses into a sum of KL divergences whose only learnable content is a mean-matching term at each noise level. Reparameterizing that mean through the closed-form forward marginal turns every term into a weighted squared error between the injected noise $\epsilon$ and a network prediction, and Ho et al. then drop the per-step weight $\lambda_t$ to get the simple objective $L_{\mathrm{simple}} = \mathbb{E}\|\epsilon - \epsilon_\theta(x_t,t)\|^2$ . Predicting the noise is not a different model from predicting the clean image: $\epsilon$ -prediction, $x_0$ -prediction, and direct mean prediction are affine reparameterizations of one another given $x_t$ and $t$ , so they define the same optimum but different implicit loss weightings and different numerical conditioning. The practical reasons to regress noise are that the target is always unit-variance no matter how noisy the input is, that the residual structure lets the network pass the low-frequency content through instead of re-synthesizing it, and that discarding $\lambda_t$ under this parameterization down-weights the easy low-noise steps by roughly an order of magnitude, which empirically buys much better FID. It also makes the network a rescaled score estimator, which is what connects DDPM to score matching and to every ODE and SDE sampler built on top of it.

(1) The Objective Is One Scalar MSE: sample $x_0$ , sample $t \sim \mathcal{U}\{1,\ldots,T\}$ and $\epsilon \sim \mathcal{N}(0,I)$ , form $x_t$ in closed form, and regress $\epsilon$ . No per-step model, no adversarial term, no sequential rollout during training.
(2) It Comes From The ELBO, Not From Heuristics: each $L_{t-1}$ is a Gaussian KL whose learnable part is a mean difference, and substituting the forward marginal converts it into $\lambda_t\|\epsilon-\epsilon_\theta\|^2$ .
(3) Dropping $\lambda_t$ Is A Reweighting Choice: $\lambda_t$ is largest at small $t$ , so setting all weights to 1 shifts capacity toward the high-noise steps that decide global structure.
(4) The Target Is Scale-Stable: $\epsilon$ is standard normal for every $t$ , so a single output head with fixed normalization works across the whole schedule, whereas the difficulty of predicting $x_0$ varies enormously with $t$ .
(5) Residual Prediction Is Easier For A U-Net: at low noise, most of $x_t$ is already the answer, so predicting the small perturbation avoids forcing the network to reconstruct an image it was handed.
(6) Conditioning Flips At The Two Ends: converting $\epsilon_\theta$ to $\hat x_0$ amplifies error by $1/\sqrt{\mathrm{SNR}_t}$ , which is tiny at low noise and roughly 150x at $t=T$ , the reason v-prediction exists.

Left-to-right flow diagram of one DDPM training step: sample a clean image, sample a timestep and Gaussian noise, form the corrupted sample in closed form, pass it through the U-Net to predict the noise, compute the squared error against the sampled noise, and a dashed feedback arrow carrying the gradient step back to the U-Net

Figure 1: One training step. The closed-form forward marginal means an arbitrary timestep can be sampled directly, so training never simulates the chain; the loss is a single unit-variance regression shared by all $T$ noise levels.

The reweighting argument is the one that most candidates miss. Under the ELBO, the term at $t=1$ carries roughly 50x the weight of the term at $t=T$ for the standard linear schedule, and those low- $t$ terms correspond to removing almost imperceptible noise, a task that contributes little to perceptual quality but a lot to likelihood. Setting every weight to 1 in $\epsilon$ -space is therefore a deliberate trade of likelihood for sample quality, and it is why DDPM reports strong FID with mediocre bits-per-dimension. The conditioning argument cuts the other way at high noise: since $\hat x_0$ is recovered by dividing by $\sqrt{\bar\alpha_t}$ , a small noise error becomes a large image error near $t=T$ , so pure $\epsilon$ -prediction is a poor target for few-step or distilled samplers that must produce a usable $\hat x_0$ from the very first step. Finally, the identity $\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\,s_\theta$ shows the trained network is a scaled score function, which is exactly what a probability-flow ODE or an annealed Langevin sampler needs, so the same checkpoint serves DDPM, DDIM, and higher-order solvers.

Mathematical Formulation:
$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$
$L_{t-1} = \lambda_t\,\|\epsilon - \epsilon_\theta(x_t,t)\|^2$
$\lambda_t = \frac{\beta_t^2}{2\sigma_t^2\alpha_t(1-\bar\alpha_t)}$
$L_{\mathrm{simple}} = \mathbb{E}\left[\|\epsilon - \epsilon_\theta(x_t,t)\|^2\right]$
$\hat x_0 = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta}{\sqrt{\bar\alpha_t}}$
$\epsilon_\theta(x_t,t) = -\sqrt{1-\bar\alpha_t}\,s_\theta(x_t,t)$

Where:

$x_0$ is a clean training sample, $x_t$ its corrupted version at step $t$ , and $\epsilon \sim \mathcal{N}(0,I)$ the injected noise that serves as the regression target.
$t \in \{1,\ldots,T\}$ is sampled uniformly, $\beta_t$ is the schedule, $\alpha_t = 1-\beta_t$ , and $\bar\alpha_t = \prod_{s=1}^{t}\alpha_s$ gives the closed-form marginal that makes single-step training possible.
$\epsilon_\theta$ is the U-Net with parameters $\theta$ , conditioned on $t$ through a sinusoidal timestep embedding.
$\lambda_t$ is the ELBO weight and $\sigma_t^2$ the reverse-process variance (commonly $\sigma_t^2 = \beta_t$ ); $L_{\mathrm{simple}}$ is $L_{t-1}$ with $\lambda_t$ replaced by 1.
$\mathrm{SNR}_t = \bar\alpha_t/(1-\bar\alpha_t)$ is the signal-to-noise ratio, which decreases monotonically in $t$ and sets the error amplification $1/\sqrt{\mathrm{SNR}_t}$ from noise space into image space.
$s_\theta(x_t,t) \approx \nabla_{x_t}\log q(x_t)$ is the score, so a trained $\epsilon_\theta$ doubles as a score model for ODE and SDE samplers.

Two-panel chart: left panel plots the ELBO per-step weight relative to its value at the final timestep on a log axis against the diffusion step, showing roughly fifty times more weight at small t than at large t, next to the flat unit weight used by the simple objective; right panel plots on a log axis the factor by which a prediction error is amplified into clean-image space for noise prediction, clean-image prediction, and v-prediction, with noise prediction rising from about 0.01 to about 150

Figure 2: The two quantitative reasons the parameterization matters. Left: dropping $\lambda_t$ removes a roughly 50x preference for the easiest low-noise steps. Right: the same trained error costs almost nothing in image space at low noise but is amplified by $1/\sqrt{\mathrm{SNR}_t}$ near $t=T$ , which is where v-prediction stays bounded.

Property	Noise prediction (DDPM)	Clean-image prediction	v-prediction
Target	The sampled noise, standard normal at every t	The data sample itself, with data-dependent statistics	A schedule-dependent mix of noise and data
Implicit weighting under uniform MSE	Down-weights low-noise steps, favors perceptual quality over likelihood	Up-weights low-noise steps by the SNR, closer to the raw ELBO	Roughly balanced across the schedule, close to SNR-plus-one weighting
Numerical weak spot	Error into image space amplified by about 150x near t = T	Error into noise space amplified near t = 0, where the residual is tiny	Amplification stays at or below 1 in both directions
Typical use	Pixel and latent diffusion with many sampling steps, Stable Diffusion 1.x and 2.0-base	Very short schedules, consistency-style objectives, some latent decoders	Progressive distillation, high-resolution and upscaling models, SD 2.1-v

August 17, 2026

DL0126 Contrastive Learning Pairs

What is contrastive learning, and what are positive and negative pairs?

Answer

Contrastive learning trains an encoder by comparison instead of by prediction of a label: it pulls the embeddings of things that should mean the same thing together and pushes everything else apart. A positive pair is two views of the same underlying content (two augmentations of one photo in SimCLR, an image and its caption in CLIP, two nearby audio segments in CPC), and a negative pair is an anchor paired with content assumed to be different (any other item in the batch). The standard objective, InfoNCE, turns this into a softmax classification problem: given an anchor, identify its single positive among one positive and $K$ negatives, using scaled cosine similarity as the logit. Positives define what the representation should be invariant to, and negatives are what prevents the trivial solution where the encoder maps every input to the same vector, a failure called representational collapse. The whole design problem of a contrastive system is therefore the pair-construction policy: a positive that is too easy teaches nothing, and a negative that is secretly a positive actively teaches the wrong thing.

(1) Positives Encode The Invariance You Want: whatever transformation you apply to build a positive pair is exactly the factor the encoder learns to discard, so color jitter buys color invariance and destroys any task where color is the signal.
(2) Negatives Are The Anti-Collapse Term: without a repulsive term the constant map $f(x) = c$ is a global minimum of the attraction loss, so negatives provide the uniformity pressure that spreads embeddings over the hypersphere.
(3) InfoNCE Is A $K+1$ -Way Classification: one positive logit in the numerator, the positive plus all negatives in the denominator, which is why the loss is bounded below by $\log(K+1)$ under a random encoder.
(4) Temperature Sets Hard-Negative Focus: small $\tau$ (0.05 to 0.1 is typical) concentrates almost all repulsive gradient on the few most similar negatives, while large $\tau$ treats all negatives nearly equally.
(5) Negative Count Is A Systems Problem: SimCLR needs batch sizes of 4096 or more to get enough in-batch negatives, MoCo decouples them with a momentum-encoder queue, and CLIP trained with a 32768 global batch across many GPUs.
(6) False Negatives Are The Main Bias: in-batch negatives assume every other sample is semantically different, which is false on class-imbalanced or deduplicated-poorly data, and the loss then pushes apart items that should be close.

Pairs do not have to come from augmentation. The general recipe is: find a cheap source of known agreement and treat everything else as disagreement. Augmentation gives agreement between two crops of one image; multimodal alignment gives agreement between an image and the alt-text scraped alongside it; temporal or spatial context gives agreement between adjacent frames, patches, or sentences; and labels give agreement between any two samples of the same class, which is what SupCon exploits to allow many positives per anchor. The negatives are almost always just the rest of the batch, because sampling them explicitly is expensive and in-batch negatives come for free with the forward pass already computed.

Two-panel diagram: left panel shows one source image producing two augmented views labeled anchor and positive while two other images in the batch produce negative views; right panel shows the same points on a unit circle with an attraction arrow from anchor to positive and dashed repulsion arrows pushing the negatives away

Figure 1: Pair construction and its geometric effect. Each anchor has exactly one positive (the other view of the same source) and $2N-2$ negatives (all views of the other images in the batch); on the unit hypersphere the loss is a single attractive force toward the positive balanced against repulsive forces from every negative.

Mathematical Formulation:
$s(u,v) = \frac{u^\top v}{\|u\|\,\|v\|}$
$\ell_i = -\log \frac{\exp(s_{i,i^+}/\tau)}{\sum_{k \neq i} \exp(s_{i,k}/\tau)}$
$\mathcal{L} = \frac{1}{2N}\sum_{i=1}^{2N} \ell_i$
$I(u;v) \geq \log K - \mathcal{L}_{\mathrm{InfoNCE}}$

Where:

$s(u,v)$ is the cosine similarity used as the logit, computed on L2-normalized embeddings $u = f(v)$ produced by the encoder (and, in SimCLR, a discarded projection head).
$\ell_i$ is the per-anchor NT-Xent loss and $\mathcal{L}$ its average over all $2N$ views of an $N$ -image batch.
$i$ indexes the anchor view, $i^{+}$ its unique positive, and $k$ ranges over every other view, so the denominator holds 1 positive and $2N-2$ negatives.
$\tau > 0$ is the temperature; the gradient weight on negative $k$ is its softmax probability, so shrinking $\tau$ sharpens that distribution onto the hardest negatives.
$I(u;v)$ is the mutual information between the two views and $K$ the number of negatives, giving the standard InfoNCE lower bound: the bound saturates at $\log K$ , which is one reason large batches help.

Log-x line chart of the cumulative share of repulsive gradient carried by the hardest fraction of negatives, for temperature 0.05, 0.1 and 0.5 over 1024 negatives; the low-temperature curve rises almost vertically showing that the hardest one percent of negatives absorbs most of the gradient

Figure 2: Temperature decides which negatives matter. At $\tau = 0.05$ the hardest 1% of negatives absorbs most of the repulsive gradient, making training an implicit hard-negative miner that is also maximally sensitive to false negatives; at $\tau = 0.5$ the pressure is spread nearly uniformly and the embedding space stays smoother but less discriminative.

Aspect	Augmentation (SimCLR, MoCo)	Multimodal (CLIP, ALIGN)	Supervised (SupCon)
Anchor	One augmented view of an image	An image embedding	A labeled sample
Positive	A second augmentation of the same image (exactly one)	The paired caption text (exactly one, symmetric loss both directions)	Every other sample sharing the label (many per anchor)
Negatives	All 2N-2 other views, or a momentum queue of 65k stale keys	All other captions in the global batch (32k in CLIP)	Only samples with a different label, so false negatives vanish
Invariance learned	To the augmentation family you chose (crop, color, blur)	To modality and phrasing, giving a shared image-text space	To everything within a class, which can over-collapse fine detail
Main failure mode	Augmentation removes task-relevant signal; shortcut solutions from crop statistics	Noisy or generic captions produce weak positives; batch size dominates cost	Needs labels, so it is not self-supervised and inherits label noise

August 17, 2026

DL0125 Action-Conditioned World Model

Explain the architecture of an Action-Conditioned World Model for autonomous driving, of the kind Wayve describes for its GAIA models.

Answer

An action-conditioned world model is a generative model of future sensor data whose predictions are explicitly steered by the ego vehicle’s own control commands, so it can answer “what would the road look like in three seconds if I brake at 5 m/s² instead of holding speed?”. Architecturally it factors into four blocks that are trained in two stages: a video encoder that compresses multi-camera frames into a small latent grid, an action and context encoder that embeds the low-dimensional control signal, a latent dynamics model that rolls the compressed state forward in time, and a decoder plus auxiliary heads that render latents back to pixels or BEV occupancy. Almost every production-scale system follows this shape and differs only in the dynamics block: GAIA-1 uses a 6.5B-parameter causal transformer over discrete video tokens with a separate diffusion video decoder, GAIA-2 replaces it with a flow-matching diffusion transformer over continuous latents so five surround views stay geometrically consistent, and Dreamer-style recurrent state-space models keep a compact deterministic-plus-stochastic state for reinforcement learning in imagination. The reason compression comes first is the same as in latent diffusion: at 6.25 Hz a 26-frame context of $288 \times 512$ frames is 3.8M pixels per frame-stack but only 14,976 latent tokens, and the dynamics model’s cost is quadratic in that number. What makes the model a simulator rather than a video generator is the rollout loop plus faithful action adherence, which is precisely where these models are weakest and where interviewers push.

(1) Observation Encoder: a VQ or KL-regularized video autoencoder with spatial factor $f = 16$ to $32$ maps each frame to an $18 \times 32$ grid; it is trained first with perceptual and adversarial losses, then frozen.
(2) Action And Context Conditioning: a 2-D action (curvature and acceleration) is embedded by a small MLP and injected per frame at every block, as prefix tokens or through AdaLN modulation, alongside text prompts, agent bounding boxes, and camera calibration.
(3) Latent Dynamics Backbone: a causal spacetime transformer predicting the next token, or a diffusion/flow-matching transformer denoising a whole latent chunk conditioned on the past; this is where nearly all parameters live.
(4) Decoder And Auxiliary Heads: a diffusion or convolutional decoder for video, plus cheap heads for BEV occupancy, collision cost, and value that planning actually consumes.
(5) Two-Stage Training With Teacher Forcing: the dynamics model always sees ground-truth latents during training, which creates the exposure-bias gap that dominates long-horizon rollouts.
(6) Three Consumers: closed-loop policy evaluation against counterfactual actions, planning or RL in imagination, and generation of rare corner-case scenarios that fleets rarely log.

Left to right pipeline: five surround cameras feed a video tokenizer producing an 18 by 32 latent per frame, a dynamics transformer conditioned on action, text, agent boxes and camera pose predicts future latents with an autoregressive rollout loop, and a decoder renders predicted video and BEV occupancy

Figure 1: The four blocks of an action-conditioned world model. Only the frozen codec ever touches pixels, the action modulates every block of the dynamics model, and the rollout loop is what turns a video generator into a driving simulator.

How the action enters is the part worth rehearsing. A two-number control signal has to influence a token stream dominated by appearance, so injecting it once at the input is not enough: it is embedded per frame and re-applied at every layer, and it must be time-aligned to the interval it causes rather than the frame it was logged with. Two failure modes follow directly. First, shortcut learning: logged actions are almost perfectly predictable from the visible road geometry, so a model can minimize training loss while ignoring $a_t$ entirely, and you only detect this by rolling out counterfactual actions that contradict the scene. Second, weak controllability at sampling time, which is why conditioning is dropped for 10-20% of training samples so that classifier-free guidance can later amplify action adherence. Evaluation therefore needs an action-following metric (does the rendered ego trajectory match the commanded one?) next to FID or FVD, because generative realism and control fidelity move independently.

Mathematical Formulation:
$z_t = \mathcal{E}(o_t)$
$h_t = f_\theta(h_{t-1}, z_{t-1}, a_{t-1})$
$\hat{z}_t \sim p_\theta(z_t \mid h_t, a_{t-1})$
$\hat{o}_t = \mathcal{D}(h_t, \hat{z}_t)$
$\mathcal{L} = \mathcal{L}_{\text{obs}} + \beta \, \mathcal{L}_{\text{dyn}}$
$\mathcal{L}_{\text{obs}} = -\log p_\theta(o_t \mid h_t, z_t)$
$\mathcal{L}_{\text{dyn}} = D_{KL}\!\left(q_\phi(z_t \mid h_t, o_t) \,\|\, p_\theta(z_t \mid h_t)\right)$
$p(z_{1:T} \mid a_{1:T}) = \prod_{t=1}^{T} p(z_t \mid z_{1:t-1}, a_{1:t-1})$
$N = T \cdot (H/f) \cdot (W/f)$
$N = 26 \cdot 18 \cdot 32 = 14{,}976$

Where:

$o_t$ is the multi-camera observation at step $t$ and $\hat{o}_t$ its prediction; $a_t$ is the ego action (curvature and acceleration, or steering and pedal).
$z_t$ is the compressed observation latent and $h_t$ the deterministic recurrent or attention-carried state that summarizes the past.
$\mathcal{E}$ and $\mathcal{D}$ are the frozen encoder and decoder, $f_\theta$ the dynamics backbone, $p_\theta$ the action-conditioned prior, and $q_\phi$ the posterior that also sees the true observation.
$\beta$ balances reconstruction against the dynamics (KL) term; in practice it is annealed and often free-bits clipped so the prior does not collapse onto the posterior.
$T$ is the context length in frames, $H \times W$ the frame resolution, $f$ the spatial compression factor, and $N$ the token count whose square drives attention cost.
Required initial conditions for a rollout: $h_0 = 0$ and a real context $z_{1:k}$ encoded from logged frames, after which only actions are supplied and $j > k$ steps are pure imagination.

Left panel: bird's-eye view of three ego trajectories branching from the same start under hold-speed, brake and lane-change action sequences, with a braking lead vehicle and a predicted collision marker. Right panel: prediction error versus rollout horizon for one-step re-encoding, free-running rollout, and a variant with action tokens ablated

Figure 2: Left: fixing the initial latent and varying only the action sequence produces counterfactual rollouts, the property that makes the model usable for closed-loop evaluation. Right: because training is teacher-forced, free-running error compounds superlinearly, and ablating the action tokens makes it worse still.

Dynamics block	Discrete-token autoregressive	Latent diffusion / flow matching	Recurrent state-space (RSSM)
Latent representation	VQ codebook indices, 576 tokens per frame	Continuous latent grid, tens of channels, temporally compressed	Small vector state: deterministic GRU plus categorical stochastic units
How the action enters	Per-frame prefix tokens in the causal sequence	AdaLN modulation or cross-attention at every block, with conditioning dropout	Concatenated into the recurrent transition at each step
Rollout cost	Hundreds of sequential token decodes per frame; slowest	Tens of denoising steps per chunk, parallel across positions	One cheap matrix step per frame; fast enough for RL in imagination
Strength	Exact likelihoods, easy long-context scaling, LLM tooling reuse	Best photorealism and multi-view consistency; controllable via guidance	Compact enough to train a policy on millions of imagined steps
Typical failure mode	Quantization artifacts and drift after a few seconds of rollout	Plausible but unfaithful scenes; ignores the action unless guided	Blurry reconstructions; posterior collapse hides rare agents
Representative system	GAIA-1	GAIA-2, Vista, NVIDIA Cosmos	DreamerV3, MILE

August 17, 2026

DL0124 3D Occupancy Flow

What is 3D Occupancy Flow? What is the tradeoff for predicting dense spatiotemporal occupancy grids instead of discrete 3D bounding-box trajectories?

Answer

3D Occupancy Flow is a joint perception-and-forecasting output format that replaces the list of tracked objects with a dense grid: for every voxel (or BEV cell) and every future waypoint the network predicts an occupancy probability plus a flow vector describing how the mass in that cell moves. Waymo’s Occupancy Flow Fields formulation predicts three quantities per waypoint on a $256 \times 256$ BEV grid covering roughly $80\ \text{m} \times 80\ \text{m}$ : observed occupancy, occluded occupancy, and backward flow. Camera-only 3D variants such as Occ3D on nuScenes predict a $200 \times 200 \times 16$ voxel grid at $0.4\ \text{m}$ resolution, and Tesla presented an occupancy network with an occupancy-flow head at its 2022 AI Day. The appeal is that free space and obstacle geometry become class-agnostic and non-parametric: a tipped-over mattress, an articulated trailer, a swinging crane boom, and an overhanging branch all get represented without appearing in a detector taxonomy, and probability mass can sit on both branches of a fork at once without a mode head or non-maximum suppression. The cost is that a grid has no notion of an object, so instance identity, track continuity, and per-agent attributes disappear, and the output tensor grows by roughly two orders of magnitude, which pushes cost onto compute, memory, label pipelines, and the loss function’s handling of a grid where the overwhelming majority of voxels are empty. In practice this is why most production stacks run occupancy flow alongside a box pipeline rather than as a drop-in replacement.

(1) Dense Spatiotemporal Output: the head emits $o_t(v)$ and $f_t(v)$ for every cell and every future timestep, not a parametric box with a heading and a velocity per agent.
(2) Class-Agnostic Geometry: anything that occupies space is representable, which removes the long-tail detection failure where an unlisted object class becomes invisible to the planner.
(3) Backward Flow, Not Forward: predicting motion from $t$ back to $t-1$ makes warping a gather with one source per cell, so mass never collides during the warp and a flow-grounded occupancy consistency check becomes well defined.
(4) Non-Parametric Multimodality: a grid holds several futures simultaneously as spread probability mass, but that same property makes averaged modes look like blur or ghost occupancy rather than a ranked set of hypotheses.
(5) Identity Is What You Give Up: without instance IDs, right-of-way logic, interaction-aware conditioning, and per-object intent signals such as turn indicators lose their handle on the scene.
(6) Cost Scales With The Grid: output size grows as $H W D T$ , dense labels require multi-sweep LiDAR accumulation and voxelization, and roughly 95% or more of voxels are empty, so class imbalance dominates the occupancy loss.

Diagram contrasting two forecasting pipelines from the same sensor input: an object-centric branch running detector, tracker, and trajectory predictor to emit sparse box waypoints, and an occupancy-centric branch running a BEV or voxel encoder with occupancy and backward-flow heads to emit a dense spatiotemporal grid

Figure 1: The object-centric branch (detect → track → forecast) produces a sparse, identity-carrying output limited by its taxonomy; the occupancy branch skips detection and data association entirely and produces a dense, class-agnostic grid with no instance IDs.

The deeper tradeoff is not really compute, it is what the downstream planner can express. A box trajectory is a commitment: this vehicle, with this ID, will be here in 3 seconds with this probability, which lets a planner reason about yielding to a specific agent, replay a scenario in simulation, and produce an auditable explanation for a maneuver. Occupancy flow is a statement about space, which is exactly what collision checking and drivable-free-space queries want, but a grid that hedges between “the cyclist goes straight” and “the cyclist turns” paints both corridors at moderate probability, and a naive cost function that treats any occupancy above a threshold as blocked yields the freezing-robot behavior. Occupancy flow partially recovers correspondence without identity: warping the previous occupancy along the predicted backward flow and multiplying it against the current occupancy gives a differentiable consistency term, so the model is penalized for teleporting mass even though it never names an object.

Mathematical Formulation:
$0 \leq o_t(v) \leq 1$
$\hat{o}_t = o_t \odot \mathcal{W}(o_{t-1}, f_t)$
$\mathcal{L} = \mathcal{L}_{occ} + \lambda \mathcal{L}_{flow}$
$\mathcal{L}_{occ} = \sum_{t=1}^{T} \sum_{v} \mathrm{BCE}(o_t(v), y_t(v))$
$\mathcal{L}_{flow} = \sum_{t=1}^{T} \sum_{v \in \Omega_t} \lVert f_t(v) - f_t^{*}(v) \rVert_1$
$N_{occ} = 200 \cdot 200 \cdot 16 \cdot 8 = 5.12 \times 10^{6}$
$N_{box} = 50 \cdot 6 \cdot 16 \cdot 5 = 2.40 \times 10^{4}$

Where:

$o_t(v)$ is the predicted occupancy probability of cell $v$ at future waypoint $t$ , and $f_t(v)$ is the backward flow vector pointing to where that mass sat at $t-1$ .
$\mathcal{W}$ is the warp operator that gathers $o_{t-1}$ along $f_t$ (bilinear or trilinear), and $\hat{o}_t$ is the flow-grounded occupancy used both as a loss term and as an evaluation metric.
$y_t(v)$ is the voxelized ground-truth occupancy label, $f_t^{*}$ the ground-truth flow, and $\Omega_t$ the set of genuinely occupied cells to which the flow loss is masked.
$t \in \{1, \ldots, T\}$ indexes waypoints and $v$ indexes the $H \times W \times D$ grid; $\lambda$ balances the two terms, and $\mathrm{BCE}$ is usually replaced by a focal or class-balanced variant because empty voxels dominate.
$N_{occ}$ counts predicted occupancy values for a $200 \times 200 \times 16$ grid over 8 waypoints, and $N_{box}$ counts a comparable box head with 50 agents, 6 modes, 16 waypoints, and 5 numbers per waypoint.
Required initial condition: $o_0$ and the whole grid must be expressed in the ego frame at $t = 0$ , so ego motion is compensated before flow is interpreted as agent motion.

Grid diagram showing occupancy at time t minus one in light dashed cells and predicted occupancy at time t in solid cells, with backward flow arrows drawn from each occupied cell at time t to the cell its mass came from, alongside the flow-grounded occupancy product equation

Figure 2: Backward flow assigns each occupied cell at time $t$ a single source cell at $t-1$ , so the warp is a gather rather than a scatter and the consistency product $o_t \odot \mathcal{W}(o_{t-1}, f_t)$ penalizes mass that appears without a plausible origin.

Resolution and horizon are the two knobs that make or break the design. Halving the voxel size multiplies the tensor by 8 in 3D, and every additional waypoint is another full grid, so a $0.2\ \text{m}$ grid over a 5-second horizon at $2\ \text{Hz}$ is far beyond a real vehicle compute budget once flow channels and semantics are added. Going the other way is not free either: at $0.4\ \text{m}$ voxels a pedestrian walking at $1.4\ \text{m/s}$ moves less than one voxel per $0.2\ \text{s}$ frame, so the flow target is sub-voxel and the occupancy channel alone cannot express the motion, which is precisely why the flow head is kept as a continuous regression rather than a discrete cell-to-cell assignment.

Log-scale bar chart of predicted output size per inference: 24000 numbers for multi-modal box trajectories, 524288 for a 256 by 256 BEV occupancy grid over 8 waypoints, 5.12 million for a 200 by 200 by 16 voxel grid over 8 waypoints, and 20.48 million once three flow channels are added

Figure 3: Output size is where the tradeoff becomes concrete: the 3D voxel grid predicts about 213x more numbers than a multi-modal box head, and adding a 3-channel flow field multiplies that by another 4 before any semantic classes are included.

Property	3D occupancy flow	Box trajectories	Hybrid stack
Output per inference	Millions of per-cell values, occupancy plus flow per waypoint	Tens of thousands of numbers, a few modes per tracked agent	Both, sharing one BEV or voxel backbone
Unlisted geometry	Represented, since occupancy is class-agnostic	Dropped if no detector class fits it	Covered by the occupancy branch
Instance identity	None; only flow-based correspondence	Explicit IDs, attributes, and track history	IDs from the box branch, geometry from the grid
Multimodality	Implicit in the probability field, no mode count to tune	Explicit ranked modes with confidences	Explicit modes for interaction, field for collision checks
Supervision	Accumulated multi-sweep LiDAR voxelized into dense labels	Human box and track annotations only	Both label pipelines must be maintained
Typical failure mode	Blurred or ghost occupancy that makes the planner over-conservative	Missed detection or ID switch removes an obstacle entirely	Disagreement between branches needs an arbitration policy

August 17, 2026

DL0123 3D-Aware Visual Pretraining

How does 3D-aware visual pre-training bridge spatial intelligence between 2D generative video models and 3D physical world simulators?

Answer

3D-aware visual pre-training trains a visual backbone or a video generator with objectives that can only be solved by recovering scene geometry: per-pixel pointmaps, metric depth, camera rays, and cross-view correspondence, instead of appearance statistics alone. The two ends it connects have complementary deficits. A 2D video diffusion model absorbs enormous visual and dynamic diversity from internet footage, but its geometry is implicit, so it has no queryable state: you cannot ask it for the distance to a mug in centimeters, and a long rollout quietly changes room layout because nothing anchors the scene. A physics simulator has the opposite profile, exact meshes, contacts, and metric scale, but its asset library and rendering diversity are tiny compared with the real world. Pre-training with geometric targets creates the shared interface: the same representation gives the generator a camera and depth channel it can be conditioned on, and gives the simulator a path to ingest real scenes as reconstructions, so the pipeline runs video → pointmaps → 3D Gaussians or mesh → simulator asset in one direction and simulator depth, pose, and contact labels back into the pre-training loss in the other.

(1) Geometry-Grounded Pretext Tasks: pointmap, ray, and depth regression plus multi-view matching force the backbone to encode where surfaces are, not just what they look like; ordinary video supplies the multiple views for free.
(2) Camera As A First-Class Input: Plücker ray embeddings or pose tokens turn a text-to-video model into a controllable renderer, which is exactly the interface a simulator viewport exposes.
(3) Persistent State Beats Frame Memory: caching an explicit 3D representation and re-projecting it into each new frame removes the layout drift and object-identity loss that pure 2D autoregressive rollouts accumulate.
(4) Two-Way Data Flow: real-to-sim reconstruction fills the simulator’s asset gap, while simulator renders provide perfect metric depth, pose, and contact supervision that no internet video can give.
(5) 3D Awareness Is Measurable: linear probes on frozen features show that self-supervised 2D backbones encode single-view depth surprisingly well yet fail at multi-view consistency, and 3D-aware fine-tuning lifts both plus downstream segmentation.
(6) Metric Scale Is The Robot Requirement: scale-invariant depth is enough for a nice-looking novel view and useless for a grasp, so metric anchoring is the part embodied policies actually consume.

Diagram: internet video corpora and simulator renders both feed a shared backbone whose pointmap, camera-pose, and differentiable-render heads serve two consumers, a camera-controlled video generator and a 3D physics simulator

Figure 1: One backbone, two supervision sources, two consumers. Unlabeled video supplies visual diversity and self-consistency signals, simulator renders supply exact depth and pose, and the geometry heads are what the video generator and the simulator both read from.

The concrete form most systems settle on is a feed-forward multi-view transformer that predicts a pointmap per frame in a common coordinate frame, so camera intrinsics, extrinsics, depth, and correspondence all fall out of one prediction rather than a fragile SfM plus MVS chain. DUSt3R established the pointmap formulation for two views, MASt3R added a matching head, and VGGT scaled it to many frames with attention alternating between frame-local and global blocks. That output is directly liftable: initialize 3D Gaussians from the points for a renderable scene, or run meshing plus material estimation for a body a rigid-body engine can collide against. Two caveats decide whether the bridge is load-bearing. First, monocular training recovers geometry only up to an unknown similarity transform, so metric scale has to be injected from calibrated stereo, a known camera baseline, simulator ground truth, or object-size priors. Second, the reprojection and correspondence losses assume a static scene, so dynamic content needs motion masks or a 4D formulation or the model folds object motion into depth as flying pixels.

Mathematical Formulation:
$u = \pi(K, T_c, X)$
$\bar{X}_i = X_i / s$
$s = \frac{1}{N}\sum_i \lVert X_i \rVert_2$
$\mathcal{L}_{geo} = \sum_i \lVert \bar{X}_i - \bar{X}^{*}_i \rVert_1$
$\mathcal{L}_{corr} = \sum_i \lVert u'_i - \pi(K', T_{c'}, X_i) \rVert_2$
$\mathcal{L}_{rgb} = \lVert I_c - \mathcal{R}(G, K, T_c) \rVert_1$
$\mathcal{L} = \mathcal{L}_{2D} + \lambda_g \mathcal{L}_{geo}$
$+\ \lambda_c \mathcal{L}_{corr} + \lambda_r \mathcal{L}_{rgb}$

Where:

$u$ is a pixel coordinate and $X \in \mathbb{R}^3$ the predicted scene point for that pixel, expressed in the reference camera’s frame.
$\pi$ is the pinhole projection, $K$ the intrinsics, and $T_c$ the pose of camera $c$ ; primed symbols refer to a second view.
$i \in \{1, \ldots, N\}$ indexes valid pixels, and $X^{*}$ is the ground-truth pointmap from a simulator, LiDAR, or a reconstruction pipeline.
$s$ is the mean point norm used for scale normalization, so $\mathcal{L}_{geo}$ on $\bar{X}$ is scale-invariant; dropping the division makes the loss metric and requires metric labels.
$\mathcal{L}_{corr}$ is the reprojection error against a matched pixel $u'$ in the other view, and it is valid only where the scene is static between the two frames.
$\mathcal{R}$ is a differentiable renderer (splatting or volume rendering) over a lifted representation $G$ , letting 2D images supervise 3D structure through $\mathcal{L}_{rgb}$ .
$\mathcal{L}_{2D}$ is the original appearance objective (diffusion denoising or masked prediction) and $\lambda_g, \lambda_c, \lambda_r \geq 0$ weight the geometric terms.

Log-scale bar chart of global attention cost relative to a single view for 1, 2, 4, 8, 16, and 32 views at 1369 tokens per view, growing as the square of the view count

Figure 2: Joint multi-view geometry is what makes a representation 3D-aware, but attention over all views at once costs $O((VN)^2)$ , so a 32-view window is roughly 1,000x a single view. This is why practical models alternate frame-local and global attention and slide a bounded view window.

Property	2D generative video model	3D-aware pre-trained representation	3D physics simulator
Scene state	Implicit in activations and past frames; not queryable	Explicit pointmaps, depth, poses, optionally 3D Gaussians	Exact meshes, transforms, joints, contact manifolds
Camera control	Text prompts only, unless pose conditioning is added	Ray or pose embeddings make trajectory a direct input	Arbitrary and exact by construction
Metric scale	Absent	Available only if metric supervision or calibration is injected	Exact, in SI units
Visual and asset diversity	Internet scale, effectively unbounded	Inherits video-scale diversity, adds geometry heads	Limited to authored assets and shaders
Typical failure mode	Layout drift, object teleporting, impossible occlusions	Scale ambiguity, flying pixels on moving objects, cost of joint views	Appearance gap and unmodeled deformable or fluid behavior

August 17, 2026