Category: Hard

DL0172 Attention Collapse in Ultra-Deep Transformers

What actually causes attention collapse in ultra-deep Transformers with 100+ layers, uniform attention scores or rank degeneration, and how do register tokens, coupled attention, and SkipNet-style gated skipping address it?

Answer

Both phenomena are real, but they are not the same object and only one of them is the root cause. Rank degeneration is a property of the token matrix: after enough layers every row of $X^{(l)}$ converges to the same vector, so the representation is effectively rank-1 and no head can distinguish positions any more. Uniform attention scores are one route to that state but not a requirement, because every softmax attention map is row-stochastic, and a product of row-stochastic matrices contracts any mean-zero component of the token set by the second-largest eigenvalue at each step, even when each individual map looks sharp and interpretable. Dong et al. proved the sharper version of this: pure self-attention without residual connections or MLPs loses rank doubly exponentially in depth, so the residual stream, the FFN nonlinearity, and the normalization placement are the load-bearing defenses that let 100-layer stacks exist at all. A third, opposite pathology is attention entropy collapse, where rows become nearly one-hot as the spectral norm of the query-key map grows; that destroys training stability rather than smoothing representations, and it is the failure that spectral reparameterization targets. Register tokens, cross-layer attention coupling, and gated layer skipping each attack a different factor of the product that drives the representation toward rank-1.

(1) Two Different Objects: entropy of the attention rows measures how a layer mixes, while the rank of $X^{(l)}$ measures what survives after mixing. A model can have healthy per-layer entropy and still be rank-collapsed at layer 90.
(2) Doubly Exponential Rank Loss: for attention-only stacks the distance to rank-1 obeys a cubic recursion, so it is numerically zero within a handful of layers, not after 100.
(3) Non-Uniform Attention Still Collapses: because $A^{(l)}\mathbf{1} = \mathbf{1}$ , the all-ones vector is always a fixed point, and repeated application contracts everything else toward the Perron eigenvector.
(4) The Opposite Failure Mode: entropy collapse to near one-hot rows correlates with exploding query-key spectral norm and produces loss spikes; $\sigma$ Reparam bounds it by spectral-normalizing the logit map with a learned scalar.
(5) Register Tokens: a handful of learnable non-content tokens (typically 4 to 16) give softmax a place to dump probability mass, so content rows are not forced to spread uniformly and the emergent high-norm artifact tokens disappear.
(6) Coupled Attention: adding the previous layer’s raw logits to the current layer’s logits means the effective attention is no longer a fresh independent stochastic matrix per layer, which preserves diversity and gives gradients a direct path across depth.
(7) Gated Skipping: a SkipNet-style gate that drops a block reduces the effective mixing depth below the nominal $L$ , shortening the product of contraction factors and cutting sequential latency at the same time.

The reason a 100-layer Transformer is not already dead is that a residual block computes $X + \mathrm{Attn}(X)$ rather than $\mathrm{Attn}(X)$ . In path-decomposition terms, the identity path carries the full-rank input straight through, and only paths that traverse many attention modules are strongly contracted. The FFN nonlinearity adds a second defense by increasing the Lipschitz constant of the layer map away from a pure average, which is why the empirical decay of the distance to rank-1 in a working model looks geometric with a factor close to 1 instead of doubly exponential. The remaining problem at extreme depth is that these defenses only slow the contraction: with a per-layer factor of 0.95, a 120-layer stack still retains only about 0.2% of the initial token spread, which shows up as flat similarity matrices, near-duplicate hidden states in the last third of the network, and layers whose removal barely changes the loss.

Two panel chart. Left panel plots distance from rank-1 on a log scale against layer index for four settings: pure attention plunging off the chart by layer five, pre-LN residual only decaying geometrically to about 1e-4 at layer 120, residual plus MLP decaying to about 0.03, and a stabilized configuration staying above 0.5. Right panel plots attention entropy normalized by log N against layer index, showing one curve rising toward the uniform limit of 1.0, one curve falling toward zero labeled entropy collapse, and a stabilized curve staying inside a shaded healthy band.

Figure 1: The two collapse modes are measured on different axes. Left: rank degeneration is catastrophic for attention-only stacks, already below $10^{-8}$ by layer 5, and merely slow once residuals and FFNs are present. Right: entropy can fail in either direction, drifting up toward the uniform limit $\log N$ (over-smoothing) or down toward zero (entropy collapse and loss spikes), and a healthy deep model must stay in the band between them.

Mathematical Formulation:
$\mathrm{res}(X) = X - \mathbf{1}x^{\top}$
$\|\mathrm{res}(X^{l+1})\| \leq c\,\|\mathrm{res}(X^{l})\|^{3}$
$c^{1/2}\|\mathrm{res}(X^{L})\| \leq \left(c^{1/2}\|\mathrm{res}(X^{0})\|\right)^{3^{L}}$
$A^{(l)}\mathbf{1} = \mathbf{1}$
$\|A^{(l)}u\| \leq \lambda_2^{(l)}\|u\|$
$H(a_i) = -\sum_{j=1}^{N} a_{ij}\log a_{ij}$
$0 \leq H(a_i) \leq \log N$

Where:

$X^{(l)} \in \mathbb{R}^{N \times d}$ is the token matrix at layer $l$ , with $N$ tokens of width $d$ , and $L$ is the total depth.
$\mathrm{res}(X)$ is the distance to the nearest rank-1 matrix whose rows are all equal; $\mathbf{1}$ is the all-ones vector and $x$ the common row it would collapse to.
$c$ collects the head geometry, roughly $4\gamma\beta/\sqrt{d_{qk}}$ , where $\gamma$ and $\beta$ bound the value and query-key weight norms and $d_{qk}$ is the head dimension.
The cubic recursion compounds into a doubly exponential bound with exponent $3^{L}$ , which is why attention-only depth is hopeless while residual depth is merely expensive.
$A^{(l)} \in \mathbb{R}^{N \times N}$ is the row-stochastic attention map, so $\mathbf{1}$ is always its eigenvector with eigenvalue 1, and $u$ is any mean-zero deviation across tokens, $\mathbf{1}^{\top}u = 0$ .
$\lambda_2^{(l)}$ is the second-largest eigenvalue modulus of $A^{(l)}$ , strictly less than 1 whenever all entries are positive; the surviving spread after $L$ layers scales like $\prod_l \lambda_2^{(l)}$ .
$H(a_i)$ is the entropy of attention row $i$ ; the upper bound $\log N$ is the uniform row (maximal mixing) and 0 is the one-hot row (entropy collapse).

Collapse Speed, Two Regimes:
$0.9^{3^{5}} = 0.9^{243} \approx 7 \times 10^{-12}$
$0.95^{120} \approx 2.1 \times 10^{-3}$

The first line is the attention-only regime: five layers are enough to destroy the representation. The second is the realistic regime for a 120-layer residual stack with a mild per-layer contraction, and it is the number that motivates the three interventions. Register tokens change the geometry of each individual $A^{(l)}$ so that content rows keep structure instead of hedging uniformly; the attention sink observed in decoder-only LLMs, where the first token absorbs a large share of the mass, is the same phenomenon arising without being designed. Coupled attention changes the product itself, since adding the previous layer’s logits makes consecutive maps correlated rather than independent draws. Gated skipping changes the number of factors in the product, and it also attacks a separate ultra-deep pathology: under Pre-LN the output variance grows with depth, so late blocks approach the identity and contribute almost nothing, which means paying their latency buys no capacity.

Left to right architecture diagram of a deep Transformer segment. An input sequence box containing N content tokens plus R register tokens feeds block l minus 1, then block l, then a skip gate, then block l plus one, then an output box. A dashed arc above the stack shows attention logits from block l minus 1 being added to block l as coupled attention, an annotation above the input box explains that registers give softmax a non-content place to dump attention mass, and a routed path below the stack shows the gate bypassing block l plus one when its gate value is zero, reducing effective mixing depth.

Figure 2: Three interventions at three different levels of the same product. Registers reshape each individual attention map, coupled attention correlates consecutive maps so the stack stops re-averaging from scratch, and the skip gate removes factors entirely by lowering the effective depth. The residual stream and FFN remain the baseline defense underneath all three.

Property	Register tokens	Coupled attention	SkipNet-style gating
What it changes	The geometry of each single attention map	The correlation between consecutive maps	The number of maps in the product
Mechanism	Extra learnable non-content tokens absorb attention mass, so content rows need not spread uniformly	Pre-softmax logits of layer l minus 1 are added to layer l, giving a residual path through attention itself	A learned gate executes or bypasses a block per input, so effective depth is data dependent
Cost	Sequence grows to N plus R, so prefill cost grows quadratically in that length; registers are discarded at the head	Must retain the previous layer’s logit tensor, which is $O(hN^2)$ activation memory per block	Gate parameters plus non-differentiable routing, usually trained with a straight-through or RL estimator
Where it fails	Cannot be bolted on after pretraining, and too many registers waste context without adding capacity	Incompatible with kernels that never materialize the score matrix, so it fights FlashAttention-style fusion	Ragged per-example depth hurts batched throughput, and gates can collapse to always-on or always-off
Diagnostic it fixes	High-norm artifact tokens and noisy attention maps	Entropy drifting toward the uniform limit in late layers	Late blocks that behave as the identity and can be pruned for free

August 19, 2026

DL0171 Jamba Hybrid SSM-Transformer Architecture

Explain Jamba and hybrid SSM-Transformer architectures. Why blend Mamba blocks with attention blocks instead of using either alone?

Answer

Jamba (AI21 Labs, 2024) is a decoder-only language model whose 32 layers are built from a repeating 8-layer block that contains seven Mamba mixers and one attention mixer, with a mixture-of-experts feed-forward on every other layer (16 experts, top-2 routing, 52B total and 12B active parameters). Every layer still writes into the same residual stream, so the only thing that changes from layer to layer is which token-mixing operator runs: a selective state space recurrence that is linear in sequence length and keeps a fixed-size state, or full self-attention that is quadratic in prefill and keeps a growing KV cache. The blend exists because the two operators fail in opposite directions. A pure Mamba stack of this size does not reliably form induction heads, so it degrades at few-shot in-context learning, format copying, and verbatim retrieval, while a pure Transformer of the same depth carries a KV cache eight times larger and loses long-context throughput. Four attention layers out of thirty-two are enough to restore the copy and in-context behaviour, and the remaining twenty-eight Mamba layers deliver the memory and speed: at a 256K-token context the KV cache is 4 GB instead of 32 GB, and long-context throughput is roughly 3x that of a comparable all-attention MoE model.

(1) Two Mixers, One Residual Stream: Mamba and attention layers are interchangeable drop-ins at the token-mixing position, so no fusion or adapter machinery is needed to combine them.
(2) A 1:7 Attention Ratio: one attention layer per eight-layer block was chosen after ablations showed 1:3 and 1:7 score alike, so the cheaper ratio wins.
(3) MoE Buys Capacity Without FLOPs: replacing the MLP on every second layer with 16 experts and top-2 routing raises total parameters to 52B while keeping 12B active per token.
(4) Attention Supplies Exact Recall: the few attention layers are the only components that can address an arbitrary earlier token exactly, which is what induction heads and few-shot copying need.
(5) Mamba Supplies Position: the recurrence is inherently ordered, so Jamba ships with no explicit positional encoding and RoPE gave no measurable gain.
(6) The Payoff Is Memory, Not Perplexity: a 256K context fits in 4 GB of KV cache, letting a 52B model serve 140K tokens on a single 80 GB GPU in int8.

A ribbon of 32 layer cells split into four identical eight-layer blocks, with the fourth cell of every block shaded as an attention mixer and the other twenty-eight shaded as Mamba mixers, above an expanded view of one block showing each layer as a mixer cell plus a feed-forward cell where MLP alternates with a 16-expert MoE

Figure 1: The whole architecture is one repeated block. Attention appears at layer 4 of every 8, giving 4 attention layers out of 32, and the MoE feed-forward alternates with a dense MLP so that capacity grows without raising the per-token FLOP count.

The reason neither operator survives alone is a difference in what they can store. A selective SSM compresses the entire prefix into a state of fixed size, so its recall is lossy and content-addressed by whatever the gating learned to keep. Theoretical and empirical work on copying shows that a fixed-state recurrent model needs state size proportional to the string it must reproduce, whereas attention copies with a constant number of heads. That is exactly the gap seen in Jamba’s ablations: a pure Mamba model trained on the same 250B tokens tracks the Transformer on log-probability benchmarks yet collapses on few-shot tasks where the model must imitate the label format shown in the prompt, because it never develops induction heads. Inserting attention into one layer in eight repairs this, and the repaired model then inherits Mamba’s cost profile for the other seven eighths of its depth. Ordering matters too, since spreading the attention layers evenly through the stack lets every group of Mamba layers be followed by an exact-lookup step rather than concentrating all lookup capability at one depth.

Mathematical Formulation:
$h_t = \bar{A}_t h_{t-1} + \bar{B}_t x_t$
$y_t = C_t h_t + D x_t$
$\bar{A}_t = \exp(\Delta_t A)$
$C_{\mathrm{ssm}} = O(L d N)$
$C_{\mathrm{attn}} = O(L^2 d)$
$M_{kv} = 2 b L n_a h_{kv} d_h$

Where:

$h_t$ is the SSM state at step $t$ , $x_t$ the layer input and $y_t$ the layer output; $D$ is the skip term.
$\bar{A}_t, \bar{B}_t, C_t$ are the discretized transition, input, and output matrices. In a selective SSM they depend on the current token, which is what lets the layer decide what to keep and what to forget.
$\Delta_t > 0$ is the input-dependent step size; a large $\Delta_t$ overwrites the state with the new token, a small one carries the old state forward.
$L$ is sequence length, $d$ the model width, and $N$ the state dimension (16 in Mamba), so SSM cost is linear in $L$ while attention prefill is quadratic.
$b$ is bytes per element (2 in fp16), $n_a$ the number of attention layers only, $h_{kv}$ the KV heads under GQA, and $d_h$ the head dimension; the leading 2 counts keys and values.
The hybrid changes exactly one factor in $M_{kv}$ , namely $n_a$ , which drops from 32 to 4.

KV Cache At A 256K Context (fp16, 8 KV heads, $d_h = 128$ ):
$M_{\mathrm{hybrid}} = 16\ \mathrm{KB} \times 262144$
$M_{\mathrm{hybrid}} = 4\ \mathrm{GB}$
$M_{\mathrm{full}} = 128\ \mathrm{KB} \times 262144$
$M_{\mathrm{full}} = 32\ \mathrm{GB}$

The 28 Mamba layers contribute a per-sequence state of only a few megabytes that does not grow with $L$ at all, so the entire cache curve of the hybrid is set by its four attention layers. That is what turns long context from a memory problem into a compute problem: batch size at 256K stops being limited by cache residency, and decode throughput at 128K measures roughly 3x a comparable all-attention MoE because far fewer bytes move per generated token.

Log-log chart of KV cache memory in gigabytes versus context length in tokens, with a steep line for a 32-layer pure Transformer at 128 KB per token, a parallel line eight times lower for the Jamba hybrid at 16 KB per token, a flat line near seven megabytes for a pure Mamba model with fixed state, a horizontal marker for a single 80 GB GPU, and an annotation at 256K tokens showing 4 GB versus 32 GB

Figure 2: Only the attention layers create a cache that grows with context. Cutting them from 32 to 4 shifts the whole line down by a constant factor of 8x, while a pure SSM stack is flat because its state size is independent of $L$ . The hybrid buys most of the SSM memory profile at the price of four layers.

Property	Pure Transformer	Pure Mamba	Hybrid (1 attention per 8)
Prefill cost	Quadratic in every layer	Linear via hardware-aware parallel scan	Linear in 28 layers, quadratic in 4
Per-sequence memory at 256K	32 GB of KV cache	A few MB of fixed state	4 GB, an 8x reduction
Exact copy and few-shot format following	Strong, induction heads emerge reliably	Weak at scale, degrades on in-context label imitation	Matches the Transformer with 4 attention layers
Position information	Requires RoPE or ALiBi	Implicit in the recurrence	No explicit encoding needed, RoPE gave no gain
Serving ecosystem	Mature: paged attention, prefix caching, speculative decoding	Custom scan kernels, thin tooling	Needs both paths, plus state handling in the scheduler

August 19, 2026

DL0170 Mamba Selective SSM vs Attention

What is Mamba’s Selective State Space Model (S6), and how does its input-dependent gating compare to Transformer self-attention in computational complexity and long-range recall?

Answer

Mamba’s S6 layer is a linear recurrence that carries a fixed-size hidden state $h_t$ per channel and reads it out with a learned projection, exactly like the earlier S4 state space model. The one change that matters is selection: the timestep $\Delta_t$ and the input and output maps $B_t, C_t$ are linear functions of the current token instead of fixed parameters, which turns the discretized decay $\bar{A}_t = \exp(\Delta_t A)$ into an input-dependent forget gate. That single change gives the model the ability to skip filler tokens and write informative ones into the state, but it destroys linear time invariance, so the layer can no longer be evaluated as one long FFT convolution and instead needs a hardware-aware parallel associative scan. On complexity the comparison is lopsided in Mamba’s favor: sequence mixing is $O(L d N)$ instead of $O(L^2 d)$ , and decoding needs a constant $O(d N)$ state rather than a KV cache that grows with context. On recall the comparison reverses, because a bounded state is a lossy summary of the past, so tasks that need verbatim retrieval of an arbitrary earlier span degrade in a way that attention’s exact, if expensive, cache does not.

(1) Selection Is The Whole Idea: $\Delta_t, B_t, C_t$ become projections of $x_t$ , while $A$ stays a learned diagonal matrix whose effective decay $\exp(\Delta_t A)$ is nonetheless input-dependent.
(2) $\Delta_t$ Is A Forget Gate: small $\Delta_t$ gives $\bar{A}_t \approx I$ and the token is ignored, large $\Delta_t$ drives $\bar{A}_t$ toward zero and overwrites the state with the new input.
(3) Time Invariance Is Lost: S4 could precompute one global convolution kernel, Mamba cannot, so training uses a work-efficient associative scan in SRAM with activation recomputation instead of an FFT.
(4) Linear Compute: mixing costs $O(L d N)$ with $N = 16$ in Mamba-1, so the quadratic attention term dominates once $L$ approaches the model width $d$ .
(5) Bounded Decode State: $2 d N$ values per layer regardless of context length, versus $2 L d$ for an MHA KV cache, which is what makes million-token streaming decode cheap.
(6) Recall Is The Trade: fixed capacity caps exact copying and multi-query associative recall, which is precisely why production long-context models are usually hybrids with a few full-attention layers.

Diagram of four input tokens the, key, is, 7Q4 each producing an input-dependent timestep Delta_t shown as 0.04, 1.6, 0.05 and 2.1 with skip or write annotations, feeding a chain of fixed-size state boxes h_1 through h_4 linked by input-dependent decay arrows, and a readout row of outputs y_1 through y_4, with a side box noting the state holds 2dN values for any sequence length

Figure 1: The recurrence is ordinary, the gate is not. Because $\Delta_t$ is computed from the token itself, a filler word produces $\bar{A}_t \approx I$ and passes through without disturbing the state, while a rare identifier produces a large step that writes into the state and forgets older content. The state width never depends on $L$ , which is both the efficiency win and the recall ceiling.

The engineering consequence of dropping time invariance is that the layer becomes memory-bound rather than FLOP-bound. Materializing the expanded states, which have shape $(B, L, 2d, N)$ , would move far more bytes through HBM than the arithmetic justifies, so the reference implementation fuses discretization, scan, and readout into one kernel that keeps states in SRAM and recomputes them during the backward pass. The recall side has a cleaner theoretical story. Attention stores every key and value, so retrieving a specific earlier token is a lookup, whereas a selective SSM must have decided at write time to keep that token, and its $2 d N$ slots bound how many distinct key-value associations can survive. This is why Mamba matches or beats Transformers on language modeling perplexity, audio, and DNA, yet lags on induction-head style copying and needle-in-a-haystack retrieval, and why Mamba-2 raises $N$ from 16 to as much as 256 while hybrids keep a small number of attention layers to do the exact lookups.

Mathematical Formulation:
$h'(t) = A h(t) + B x(t)$
$y(t) = C h(t)$
$\Delta_t = \mathrm{softplus}(W_{\Delta} x_t + b)$
$B_t = W_B x_t, \quad C_t = W_C x_t$
$\bar{A}_t = \exp(\Delta_t A)$
$h_t = \bar{A}_t h_{t-1} + \Delta_t B_t x_t$
$y_t = C_t^{\top} h_t$

Where:

$y_t$ is the layer output for token $t$ and $h_t \in \mathbb{R}^{N}$ is the hidden state held for one channel, so the full layer keeps $2 d N$ values under the standard expansion factor of 2.
$x_t$ is the input activation, $t \in \{1, \ldots, L\}$ indexes the sequence, and $h_0 = 0$ is the required initial condition.
$A$ is a learned diagonal matrix parameterized as $A = -\exp(A_{\log})$ so every eigenvalue is negative and the discrete decay stays stable.
$\Delta_t > 0$ is the input-dependent timestep, and $W_{\Delta}, W_B, W_C$ are the low-rank projections that make the layer selective; $b$ is initialized so that $\Delta_t$ starts in a useful timescale range.
$\bar{A}_t$ is the zero-order-hold discretization of $A$ , and $\Delta_t B_t x_t$ is the simplified input term Mamba uses in place of the exact $\bar{B}_t$ .
$d$ is the model width, $N$ the state dimension (16 in Mamba-1), and $L$ the sequence length; the scan is $O(L d N)$ work with $O(\log L)$ depth.

Decode Memory At 32k Context (values per layer):
$L = 32768, \quad d = 2048, \quad N = 16$
$\mathrm{cache} = 2 L d = 1.34 \times 10^{8}$
$\mathrm{state} = 2 d N = 6.55 \times 10^{4}$
$\mathrm{ratio} = L / N = 2048$

Those are element counts, so in bytes the gap narrows a little when the cache is fp16 and the SSM state is fp32, and it narrows further with grouped-query attention, which divides the cache by the query-to-key-value head ratio. The structural point survives every such adjustment: the attention term is proportional to $L$ and the SSM term is not, so past a few thousand tokens the recurrent model is decoding from a constant working set while the Transformer is streaming a cache that eventually dominates both memory and bandwidth.

Two log-log panels: the left panel plots forward-pass FLOPs per layer against sequence length for a self-attention block with a quadratic term and a Mamba block with a linear term, with a dash-dotted vertical crossover line near two thousand tokens, and the right panel plots decode state in megabytes showing a rising fp16 KV cache line against a flat constant Mamba state line at 0.26 megabytes

Figure 2: Below roughly $L \approx d$ the wider Mamba block, which expands the channel dimension by 2, actually costs more than attention, because both are dominated by their projection matmuls. Above it the $O(L^2 d)$ term takes over, and at 128k tokens the attention layer needs about 20 times the FLOPs and a per-layer cache about 4,000 times larger than the constant SSM state.

Property	Selective SSM (Mamba S6)	Self-attention (MHA)
Sequence-mixing cost	O(L d N), linear in L	O(L^2 d), quadratic in L
Cost per decoded token	O(d N), independent of context	O(L d), grows with context
State carried between tokens	2 d N values per layer, fixed	2 L d values per layer, unbounded
Training parallelism	Associative scan with recomputation; no FFT convolution, since the recurrence is time-varying	Pure matmuls, no sequential dependence at all
Exact retrieval and copying	Lossy; bounded by state capacity, degrades as the number of stored associations grows	Exact within the window; copying long spans is easy
Natural fit	Audio, DNA, streaming, very long inputs summarized rather than quoted	In-context retrieval, many-shot prompts, verbatim citation

August 19, 2026

DL0169 Softmax-1 Off-by-One Attention

What is the Softmax-1 (off-by-one) modification to attention, and why does subtracting one from the denominator improve length generalization and register-token behavior in recent Transformer architectures?

Answer

Softmax-1, also called off-by-one or quiet attention, replaces the attention normalizer $\sum_j e^{z_j}$ with $1 + \sum_j e^{z_j}$ . That extra constant is exactly what you get by appending a virtual key with logit 0 and value vector 0, so the weights over real tokens now sum to $S/(1+S)$ , which is strictly below one, and a head that finds nothing relevant can emit the zero vector. Ordinary softmax has no such option, because its weights are forced to sum to one whether or not any key matches. Heads that want to be a no-op therefore learn to dump their leftover mass on a low-information token, usually the first token or a delimiter, and to inflate that token’s residual to enormous magnitude: the massive activations that dominate activation-quantization error and the artifact tokens that pollute ViT attention maps. The name refers to the denominator being off by one; the operational reading is that one unit of attention budget is subtracted from the real keys and parked in a null option that costs nothing to select.

(1) One Constant, No New Parameters: adding 1 to the denominator is algebraically identical to concatenating a zero-logit, zero-value key, so the change touches one line of the kernel and adds no weights.
(2) Mass Below One Is The Whole Point: the head gains an explicit “abstain” action, and the output norm $\lVert o \rVert$ can go to zero without any token’s value vector having to be zero.
(3) Absolute Instead Of Relative Scores: ordinary softmax is invariant to a constant shift of all logits, so “nothing is similar” is inexpressible; the fixed zero reference makes the logit level itself meaningful.
(4) The Sink Becomes A Mechanism, Not A Token: vanilla models route no-op mass through a specific KV slot that a sliding window may evict, whereas the constant is always present, which is what stabilizes long-context and windowed decoding.
(5) Registers Stop Being Garbage Dumps: once abstention is free, register and background tokens keep their global-storage role without absorbing surplus attention, so their norms stay moderate and attention maps stay readable.
(6) Production Form Is A Learned Sink Logit: replacing the 1 with $e^{s}$ for a per-head learned $s$ strictly generalizes Softmax-1, which is the $s = 0$ special case.

Two-panel grouped bar chart of attention weights over six keys plus a null slot. In panel A every logit is below zero and ordinary softmax still renormalizes to sum one, while softmax-1 keeps only 0.17 of the budget and sends 0.83 to the null slot. In panel B one logit is six and the two methods are nearly identical, with the null slot receiving only a quarter of a percent.

Figure 1: The gate is selective, not a uniform rescaling. When no key matches (panel A), ordinary softmax renormalizes a row of weak scores into a confident-looking distribution, while Softmax-1 keeps 17% of the budget on real keys and abstains with the rest. When a real match exists (panel B), the constant 1 is negligible against $e^{6} \approx 403$ and the two are indistinguishable, so the modification only fires where the head has nothing to say.

Mathematical Formulation:
$\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{L} e^{z_j}}$
$\mathrm{softmax}_1(z)_i = \frac{e^{z_i}}{1 + \sum_{j=1}^{L} e^{z_j}}$
$S = \sum_{j=1}^{L} e^{z_j}$
$m = \frac{S}{1 + S}$
$m = \sigma(z + \log L)$
$p_i = \frac{e^{z_i}}{e^{s} + \sum_{j=1}^{L} e^{z_j}}$
$o = \sum_{i=1}^{L} p_i v_i$

Where:

$z_i = q^{\top} k_i / \sqrt{d}$ is the scaled dot-product logit for key $i$ , and $L$ is the number of visible keys after masking.
$S$ is the ordinary denominator, so the added 1 is just $e^{0}$ : the virtual key’s exponentiated logit.
$m$ is the total attention mass retained by real keys; the complement $1/(1+S)$ is the abstention share, and it is positive for every finite logit row.
The uniform-logit identity uses $\sigma$ for the logistic function: with all $L$ logits tied at $z$ , the mass is a logistic gate whose half-way point sits at $z = -\log L$ .
$s$ is a learned per-head sink logit replacing the constant, and $s = 0$ recovers Softmax-1 exactly.
$o$ is the head output; because $\sum_i p_i$ can be near zero, $o \approx 0$ is reachable without constraining any $v_i$ .

The length-generalization argument has two halves. First, in ordinary softmax a no-op head’s surplus mass is $1 - m_{\mathrm{rel}}$ , and where it lands depends on how many irrelevant keys exist, so a head calibrated at a 4k training length re-partitions its budget when the context reaches 32k; the model compensates by pinning a token whose residual magnitude was tuned at the training length, and Sun et al. measured such coordinates in the thousands while the median activation sits near 0.1. Second, that sink is a KV slot, not an architectural constant. StreamingLLM showed the consequence directly: evict the first few tokens from a sliding window and perplexity explodes, while pinning four sink tokens keeps generation stable out to millions of tokens. Softmax-1 turns the sink into a term of the denominator that no eviction policy can delete, which is why windowed and extrapolated decoding stop depending on cache bookkeeping. The honest caveat is visible in the uniform-logit identity above: the abstention gate is absolute but its threshold still drifts as $-\log L$ , so pushing a head to stay quiet across a 256x context increase costs about 5.5 nats of logit headroom, which is precisely why deployed variants learn $s$ per head rather than freezing it at zero.

Line chart of total attention mass on real keys versus a uniform attention logit, for context lengths 512, 8192 and 131072. Each softmax-1 curve is a logistic whose half-way point sits at minus log L, marked by dotted vertical lines, while a flat dash-dotted line at mass one shows that ordinary softmax retains the whole budget regardless of logit level or length.

Figure 2: Softmax-1 converts the normalizer into a logistic gate on the absolute logit level, whereas ordinary softmax pins the mass at one for every logit row and every length. The gate’s half-way point moves left by $\log L$ , so a head that abstains comfortably at 512 keys leaks over half its budget at 131k unless it lowers its logits by about 5.5 nats, which is the argument for a learned sink logit rather than a hard-coded 1.

The register story is the same pressure seen in vision. Darcet et al. found that DINOv2, CLIP, and DeiT-III spontaneously repurpose a small fraction of low-information background patches as global scratch space, giving those patches norms roughly an order of magnitude above their neighbours and visibly corrupting attention maps and dense-prediction features; adding explicit register tokens removed the artifacts. Registers supply somewhere to write, while Softmax-1 supplies permission not to write, and the two are complementary rather than competing. Bondarenko et al. reached the identical diagnosis from the quantization side, naming the problem “helping attention heads do nothing” and reporting that suppressing the outliers is what makes per-tensor INT8 activation quantization viable. Reported perplexity gains from the off-by-one change alone are small and setup-dependent, so the case for it rests on outlier suppression and long-context robustness, not on loss curves.

Property	Ordinary softmax	Softmax-1	Learned sink logit	Register tokens
How a head does nothing	Dumps mass on a learned sink token and shrinks its value contribution	Pushes every logit below zero and abstains	Same, with the abstention threshold trained per head	Attends to a dedicated token reserved for scratch space
Weights sum to	Exactly 1	$S/(1+S)$ , strictly below 1	$S/(e^{s}+S)$ , strictly below 1	Exactly 1, including the register slots
Added cost	None	One constant in the denominator, zero parameters	One scalar per head	Extra sequence positions, so extra KV and quadratic prefill
Massive activations	Emerge reliably; block per-tensor INT8 activation quantization	Strongly reduced when trained from scratch	Reduced, and the head can tune how quiet it stays	Confined to registers instead of random patches, not removed
Sliding window safety	Fragile: evicting the sink token collapses perplexity	Safe: the null option is architectural, nothing to pin	Safe, and adapts to the window length during training	Safe only if the registers are never evicted
Retrofit to a trained model	Baseline	Needs continued pretraining; a drop-in swap breaks calibration	Same, though $s$ can be initialized from observed sink mass	Requires retraining or at least adapter tuning of the new tokens

August 19, 2026

DL0168 NoPE: Causal Positional Bias Without Encodings

What is NoPE (No Positional Embeddings), and how does a causal mask alone induce implicit positional bias in decoder-only Transformers without explicit position encodings?

Answer

NoPE is a decoder-only Transformer trained with no position information injected anywhere: no learned or sinusoidal absolute embedding added to the token embeddings, no RoPE rotation inside attention, and no ALiBi distance bias on the logits. The only order-dependent structure left in the network is the causal mask, and that turns out to be enough. A bidirectional self-attention layer without positional encodings is permutation equivariant, so it can only represent a bag of tokens, but causal masking destroys that symmetry because query $i$ attends over exactly $i+1$ keys. The size of each token’s receptive field is therefore itself a strictly monotone function of absolute position, and any head with near-uniform logits converts that count into a readable $1/(i+1)$ signal in the residual stream. Haviv et al. showed that absolute position can be linearly probed out of NoPE hidden states with high accuracy while perplexity stays close to models with explicit encodings, and Kazemnejad et al. proved constructively that a NoPE decoder can represent both absolute and relative position, and that it length-generalizes at least as well as RoPE or ALiBi on small-scale algorithmic tasks.

(1) Definition: NoPE removes every explicit position term and keeps only the lower-triangular attention mask, so position is an emergent property rather than an injected feature.
(2) Broken Permutation Equivariance: without a mask, $f(PX) = P f(X)$ for any permutation $P$ , but the causal mask satisfies $P^\top M P \neq M$ , which is exactly the asymmetry the model exploits.
(3) Counting Is The Mechanism: uniform attention over an $(i+1)$ -token window puts $1/(i+1)$ mass on each visible key, so the output norm encodes absolute position.
(4) Absolute First, Relative Later: layer 1 materializes an absolute code, usually anchored on a dominant BOS sink, and deeper layers subtract two codes to obtain the relative offset $i-j$ .
(5) Resolution Decays Quadratically: the gap between adjacent positions is $O(i^{-2})$ , so the counting code loses discriminative power long before the context window ends.
(6) Implicit Recency Bias: trained NoPE heads develop distance-decaying attention that resembles a learned relative encoding, but with no principled extrapolation knob, which is why long-context NoPE needs attention-temperature scaling or hybrid layers.

The mechanism is easiest to read as counting. Row $i$ of the causal attention matrix has exactly $i+1$ unmasked entries, so a head whose logits are roughly constant across its window spreads $1/(i+1)$ of the probability mass onto each visible key. If the value vectors are dominated by one distinguished token, in practice the BOS token that every position can see, then the output norm at position $i$ is proportional to $1/(i+1)$ : a strictly decreasing, invertible function of absolute position that the next layer can consume as a positional feature. Kazemnejad et al. turn this observation into a theorem, and once an absolute code sits in the residual stream a later layer’s $q_i^\top k_j$ term can compute $i-j$ , recovering relative position too. Real NoPE models show the fingerprints of this construction: very heavy mass on the first token, first-layer heads with near-flat windows, and probes that decode absolute position from early activations.

Three panels: a ten by ten causal attention mask heatmap whose row i has i+1 shaded cells with the visible-key count printed beside each row, the same size bidirectional mask where every row sees all ten keys, and a line chart showing that uniform causal attention yields an output norm of one over i plus one that decreases strictly with position while bidirectional uniform attention yields a flat one over n line carrying no positional information

Figure 1: The causal mask is the position signal. Counting unmasked entries per row gives $1{,}2{,}\ldots{,}n$ under causal masking but a constant $n$ under bidirectional attention, so the same uniform head produces a strictly decreasing code in one case and a flat, uninformative constant in the other.

Mathematical Formulation:
$\alpha_{ij} = \frac{\exp(q_i^\top k_j)}{\sum_{m=0}^{i} \exp(q_i^\top k_m)}$
$\alpha_{ij} = 0 \ \text{for}\ j > i$
$q_i^\top k_j = c \Rightarrow \alpha_{ij} = \tfrac{1}{i+1}$
$o_i = \sum_{j=0}^{i} \alpha_{ij} v_j = \frac{v_0}{i+1}$
$i = \|o_i\|^{-1} - 1$
$f_{\mathrm{bi}}(PX) = P f_{\mathrm{bi}}(X)$
$P^\top M P \neq M$

Where:

$o_i \in \mathbb{R}^{d}$ is the attention output at query position $i$ , and $\alpha_{ij}$ the attention weight it places on key position $j$ .
$q_i, k_j, v_j$ are the query, key, and value vectors, containing no positional term at all under NoPE.
$j, m \in \{0,\ldots,i\}$ index only the unmasked keys; the second line is the causal mask, which is the sole source of order sensitivity.
$c$ is a constant logit, the degenerate case that makes attention uniform over the window; the third and fourth lines assume $v_j = 0$ for $j \geq 1$ and a nonzero anchor value $v_0$ at BOS.
The fifth line inverts the code with $\|v_0\| = 1$ , showing that absolute position is exactly recoverable from a single layer’s output norm.
$P$ is a permutation matrix, $X$ the input sequence, $f_{\mathrm{bi}}$ an unmasked layer, and $M$ the lower-triangular mask; the last two lines state why encoders need explicit position encodings and causal decoders do not.

The same construction explains why NoPE is fragile at long context. Adjacent positions are separated by $\Delta_i = 1/((i+1)(i+2))$ , a relative spacing of only $1/(i+2)$ , so the code that cleanly distinguishes position $i$ from position $i+1$ must resolve relative differences near $8\times10^{-3}$ around token 126 and near $10^{-3}$ around token 1022. Under bf16 activations those differences sit at the edge of machine epsilon, and the second pressure is entropic: uniform attention over a growing window has entropy $H_i \leq \ln(i+1)$ , so a head calibrated to be discriminative at 2K tokens is comparatively diffuse at 32K. Real models do not use the naive counting head alone, but both effects push in the same direction, which is why raw NoPE degrades past its training length while RoPE offers explicit rescaling recipes such as NTK interpolation and YaRN.

Two panels: a log-log plot of the relative gap one over i plus two between adjacent positional codes falling below bf16 machine epsilon around position 126 and below fp16 machine epsilon around position 1022, and a semi-log plot of the maximum attention entropy log of i plus one growing from about 7.6 nats at 2048 tokens to about 10.4 nats at 32768 tokens

Figure 2: Two length pressures on an implicit code. The relative spacing between neighbouring positions shrinks like $1/(i+2)$ , crossing bf16 machine epsilon in the low hundreds of tokens, while the entropy ceiling grows like $\ln(i+1)$ , so the same attention temperature that is sharp at 2K is diffuse at 32K.

Property	NoPE	RoPE	ALiBi
Source of position	Causal mask only; window size acts as an implicit counter	Rotation of queries and keys by angle proportional to index	Fixed per-head linear penalty on the distance i minus j
Works without a mask	No; a bidirectional layer becomes permutation equivariant	Yes; encoders use it directly	Yes, with a symmetric distance bias
Extension knob	Attention temperature or entropy scaling; no standard recipe	Base frequency rescaling, NTK interpolation, YaRN	Slope schedule; extrapolates but truncates effective range
Extra cost	None; consumes capacity and layer-1 heads instead	Two elementwise ops per head; cached keys are pre-rotated	One additive bias term per attention score
Dominant failure mode	Code resolution and softmax sharpness collapse past training length	Unseen rotation phases out of distribution beyond training length	Strong recency bias suppresses genuine long-range retrieval

August 19, 2026

DL0167 Video Generation Physical Consistency Evaluation

How do you evaluate physical consistency (gravity, collision, object permanence, and temporal coherence) in video and world generation models beyond standard distributional metrics like FID and FVD?

Answer

FID and FVD compare summary statistics of clip embeddings from appearance-biased backbones (InceptionV3, I3D), so they answer whether a set of generated videos resembles a reference set, never whether one clip obeys physics. They are dominated by per-frame content rather than motion, they are nearly insensitive to short-range frame reordering, and they can improve while dynamics get worse. Physical consistency therefore needs per-clip, falsifiable probes. The three that work in practice are: condition the model on the opening frames of a real recorded event and score the predicted continuation against the actual future; ask a calibrated judge (human, or a VLM validated against human labels) for a binary physical-commonsense verdict on each clip; and extract a symbolic readout (track → lift to 3D → fit dynamics) so gravity, momentum, permanence, and flow coherence become numbers with known correct values. For a world model you add a fourth layer, closed-loop rollout, because the only decision-relevant question is whether a policy trained or planned inside the model still succeeds in the real environment.

(1) Distributional vs Per-Clip: FVD is a set-level Frechet distance and cannot label a single video as physically wrong, so it belongs in a regression suite, not in a physics report.
(2) Real-Continuation Probes: give the model the first frames of a genuine recorded event and compare the continuation to ground truth with spatial IoU of the motion mask and masked pixel MSE.
(3) Calibrated Judges: report physical commonsense (PC) and semantic adherence (SA) separately plus their joint pass rate, and publish the judge’s agreement with humans before trusting it.
(4) Symbolic Readout: tracking plus depth lifting turns physics into scalars: fitted gravity residual, momentum error at collisions, occlusion recovery rate, and optical-flow warp error.
(5) Object Permanence Is A Timeline Test: it is only measurable across an occlusion window, so probes must be authored with an occluder and scored on identity, count, and attribute preservation after reappearance.
(6) Closed-Loop For World Models: action-conditioned rollouts are scored by drift over horizon and downstream task success, which is the metric that actually correlates with usefulness.

Diagram of a four-tier evaluation stack: a generated clip with its conditioning fans out to tier one distributional FID and FVD over I3D features, tier two real-continuation probe comparing to a recorded future, tier three calibrated human or VLM judge giving a binary per-clip verdict, and tier four symbolic readout that tracks objects and fits dynamics; each tier emits its own metrics and tiers two to four feed a per-clip physics scorecard while tier one connects with a dashed arrow

Figure 1: The evaluation stack. Only tiers 2 to 4 return a verdict for an individual clip; tier 1 is a distribution-level sanity check whose score can fall (improve) while gravity and collision behaviour degrade, which is exactly why it is drawn with a dashed contribution.

Each of the four physical axes needs a different probe design. Gravity is measured by tracking a free-falling centroid, fitting a quadratic to its height, and comparing the implied acceleration to the reference: generated video overwhelmingly fails toward floaty, slow-motion dynamics with a fitted value well below the true one. Collision is measured at the impact frame, where you check momentum and energy: a coefficient of restitution above 1, interpenetration, or objects passing through each other are hard violations that no appearance metric sees. Object permanence requires a prompt authored with an occluder, then a check that the object reappears with the same identity, count, and colour; the natural scores are the recovery rate and identity-switch count borrowed from multi-object tracking. Temporal coherence is the cheapest to automate: warp consecutive frames with optical flow, mask out occlusions and disocclusions, and average the residual, which catches flicker, texture swimming, and morphing that a per-frame FID is blind to. Crucially, all four scores are reported as per-clip pass rates over a curated probe set, not as an average of a continuous score, because averaging hides the fact that most failures are categorical.

Mathematical Formulation:
$d_F^2 = \| \mu_r - \mu_g \|_2^2 + T_{\Sigma}$
$T_{\Sigma} = \mathrm{Tr}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{1/2})$
$y(t) = y_0 + v_0 t + \frac{1}{2} \hat{g} t^2$
$e_g = | \hat{g} - g | / g$
$e_p = \| p^{+} - p^{-} \| / \| p^{-} \|$
$R_p = N_{\mathrm{rec}} / N_{\mathrm{occ}}$
$\tilde{I}_t = \mathcal{W}(I_t, f_{t \to t+1})$
$\epsilon_t = \| m_t \odot (I_{t+1} - \tilde{I}_t) \|_1$
$E_w = \frac{1}{T-1} \sum_{t=1}^{T-1} \epsilon_t$
$J = \frac{1}{N} \sum_{i=1}^{N} c_i^{\mathrm{pc}} c_i^{\mathrm{sa}}$

Where:

$d_F^2$ is the FVD value, with $\mu_r, \Sigma_r$ and $\mu_g, \Sigma_g$ the mean and covariance of real and generated clip features from a fixed video backbone; it is defined only over sets, which is the root of its uselessness for per-clip physics.
$y(t)$ is the tracked height of an object at time $t$ , $y_0$ and $v_0$ the fitted initial height and velocity, and $\hat{g}$ the least-squares acceleration recovered from the generated clip; $e_g$ is its relative error against the reference $g$ .
$p^{-}$ and $p^{+}$ are total momentum immediately before and after a contact frame, so $e_p$ is the relative momentum violation; the same construction on kinetic energy exposes a restitution above 1.
$N_{\mathrm{occ}}$ is the number of authored occlusion events and $N_{\mathrm{rec}}$ the number where the object reappears with correct identity, count, and attributes, giving the permanence recovery rate $R_p$ .
$\mathcal{W}$ is backward warping by the estimated flow $f_{t \to t+1}$ , $m_t$ the binary validity mask that removes occluded and out-of-frame pixels, $\odot$ elementwise product, and $E_w$ the averaged warp error over the $T$ frames.
$c_i^{\mathrm{pc}}, c_i^{\mathrm{sa}} \in \{0,1\}$ are the physical-commonsense and semantic-adherence verdicts on clip $i$ , so $J$ is the joint pass rate: a clip counts only if it is both on-prompt and physically plausible.

Two-panel chart. Left panel plots tracked height versus time for a reference bouncing ball following ballistic arcs with decaying apex heights and a generated clip that falls more slowly, dips below the floor line, and rebounds to an apex higher than its release height. Right panel plots speed magnitude versus time, showing the reference speed dropping from about 9.9 to 7.4 metres per second at impact while the generated clip's speed increases from about 6.6 to 7.6 metres per second at impact

Figure 2: The symbolic readout on one probe clip. The fitted acceleration is 4.2 instead of 9.8, the object interpenetrates the floor, and the post-impact speed exceeds the pre-impact speed (restitution above 1, so energy is created). Every one of these is a numeric, falsifiable failure; all of them are invisible to a distributional metric computed on the same clip.

Metric family	Granularity	What it catches	What it misses
FID / FVD	Set level, needs hundreds of clips	Gross artefacts, blur, mode collapse, distribution shift	All per-clip physics; content bias makes it track texture more than motion
Real-continuation probe	Per clip, against recorded ground truth	Wrong outcome of a near-deterministic event (where a ball lands, whether paint mixes)	Legitimate stochastic futures are penalised unless you score a sampled set
Calibrated human or VLM judge	Per clip, binary PC and SA	Open-ended violations on arbitrary prompts, including material and state-change errors	Sub-second and geometric violations; judge agreement with humans is far from perfect
Symbolic trajectory readout	Per clip, per object, per frame	Gravity residual, momentum and energy errors, interpenetration, identity switches, warp error	Anything the tracker or depth model cannot follow: fluids, smoke, cloth, heavy occlusion
Closed-loop rollout	Per episode, action-conditioned	Compounding drift and any error that actually changes a policy’s decision	Expensive, needs an action-conditioned model plus a real or simulated environment

August 19, 2026

DL0166 3D Gaussian and NeRF Neural Simulator

How do 3D Gaussians and Neural Radiance Fields serve as dense physical representations inside differentiable neural simulators?

Answer

A differentiable neural simulator needs one scene description that works twice: as geometry the solver can push around, and as an image the photometric loss can compare against real video. NeRF supplies the second half well, storing the scene as an implicit field that maps a position and view direction to density and radiance, rendered by differentiable volume quadrature along rays. What it lacks is handles, because an MLP field has no particle to give mass, no covariance to advect, and no obvious place to attach a constitutive model. 3D Gaussian Splatting removes that obstacle by storing the scene as an explicit set of anisotropic Gaussians with mean, covariance, opacity, and spherical-harmonic color, so every primitive doubles as a material point in a Lagrangian solver. The simulator advances those points with differentiable physics, usually MLS-MPM, the rasterizer splats them back to pixels, and gradients from the rendering loss flow through both the rollout and the renderer into material parameters such as Young’s modulus, Poisson ratio, yield stress, and friction. The loop is therefore video → physical parameters → rollout of unseen dynamics, with pixels as the only supervision.

(1) Dense Means Physics Everywhere: unlike a mesh, both representations describe the interior volume, so mass, velocity, and stress are defined at every sampled point rather than only on a surface shell.
(2) Gaussians Are Already Particles: a splat’s mean is a particle position and its covariance is a local volume element, which is exactly the state an MPM particle carries, so the reconstruction and the simulation share one data structure.
(3) NeRF Needs A Bridge: a continuous field has to be voxelized or resampled onto particles each step, the Eulerian-Lagrangian conversion that PAC-NeRF introduced, which costs an extra transfer and an extra source of gradient noise.
(4) Covariance Follows The Deformation Gradient: kinematics are preserved by advecting the mean through the flow map and conjugating the covariance with $F$ , otherwise stretched material still renders as undeformed blobs.
(5) The Loss Is Purely Photometric: no ground-truth stiffness is available, so system identification is driven by rendered-versus-observed pixel error over a multi-frame rollout.
(6) The Cost Lives In The Backward Pass: memory grows as $O(TN)$ in substeps and particles, and gradients through contact and friction are non-smooth, so checkpointing and softened contact models are mandatory.

Left-to-right pipeline: multi-view video with camera poses feeds a dense reconstruction stage producing 3D Gaussians or a NeRF field, which becomes a physical particle state with mass, velocity and deformation gradient, advanced by a differentiable MPM simulator, rendered by a differentiable renderer and compared to the observed frames, with a dashed backward gradient path into a material parameter block that feeds the simulator

Figure 1: The identification loop. The dense representation appears twice in the graph, once as the initial physical state and once as the thing being rendered, which is why gradients from a single photometric loss can reach material parameters that were never observed directly.

The reason 3D Gaussians displaced radiance fields in this role is mechanical rather than aesthetic. In a Lagrangian solver the unknowns are particle positions and their local deformation, and a Gaussian already stores a mean plus a covariance factored as scale and rotation, so PhysGaussian can drive splats directly with a continuum solver and keep rendering in real time. A radiance field, by contrast, is a function of space with no identity attached to any location, so tracking material requires either a learned deformation field mapping observation time back to a canonical frame, or a particle proxy that carries the physics while the field carries appearance. Two consequences matter in practice. First, the spherical-harmonic coefficients are expressed in world coordinates, so when a splat rotates by $R$ its appearance basis must be rotated too, otherwise highlights stay welded to the world frame while the geometry turns. Second, lighting is baked, so shadows and specular reflections do not respond to motion, which biases the photometric gradient exactly on the frames where an object moves most.

Two panels of ellipse grids: the left panel shows a three by three grid of identical circular Gaussians in the rest state, the right panel shows the same grid after a shear and vertical compression, where each ellipse is stretched and tilted consistently with the displaced particle centers, connected by an arrow labelled F

Figure 2: Kinematics of a splat. The mean rides the flow map while the covariance is conjugated by the deformation gradient, so shear and compression change both where a Gaussian sits and how it is shaped; skipping the covariance update leaves visibly isotropic blobs inside sheared material and corrupts the rendering gradient.

Mathematical Formulation:
$\alpha_i = 1 - \exp(-\sigma_i \delta_i)$
$T_i = \prod_{j=1}^{i-1}(1 - \alpha_j)$
$C(r) = \sum_{i=1}^{N} T_i \alpha_i c_i$
$\Sigma = R S S^\top R^\top$
$\mu_p^{t} = \phi_t(\mu_p^{0})$
$\Sigma_p^{t} = F_p^{t}\,\Sigma_p^{0}\,(F_p^{t})^\top$
$s_{t+1} = \mathcal{S}(s_t;\theta)$
$\mathcal{L} = \sum_t \lVert \mathcal{R}(s_t) - I_t \rVert^2$
$\nabla_{\theta}\mathcal{L} = \sum_t \frac{\partial \mathcal{L}}{\partial s_t}\frac{\partial s_t}{\partial \theta}$

Where:

$C(r)$ is the rendered color of ray $r$ , accumulated over $N$ ordered samples or splats with per-sample color $c_i$ ; this same alpha-compositing form covers NeRF quadrature and Gaussian splatting.
$\sigma_i$ is density, $\delta_i$ the sample spacing along the ray, $\alpha_i$ the resulting opacity, and $T_i$ the transmittance reaching sample $i$ .
$\Sigma$ is a splat’s covariance, parameterized by rotation $R$ and diagonal scale $S$ so that it stays positive semi-definite under gradient updates.
$p$ indexes particles, $\phi_t$ is the flow map from rest to time $t$ , and $F_p^{t} = \partial \phi_t / \partial \mu$ is the deformation gradient whose determinant gives local volume change.
$s_t$ is the full simulator state (positions, velocities, $F$ , affine momentum) and $\mathcal{S}$ one differentiable substep.
$\theta$ collects the unknown physical parameters such as Young’s modulus $E$ , Poisson ratio $\nu$ , density $\rho$ , yield stress, and friction coefficient.
$\mathcal{R}$ is the differentiable renderer and $I_t$ the observed frame, so $\mathcal{L}$ is a multi-frame photometric residual and its gradient is a product of per-substep Jacobians.

That last product is where the engineering happens. Explicit MPM needs a CFL-limited timestep, so one video frame at 30 fps typically hides a few hundred substeps, and reverse-mode differentiation over a one-second clip means backpropagating through several thousand Jacobians. Storing every intermediate state is what kills the run: 200k particles at roughly 96 bytes of state per particle is about 19 MB per substep, so a 1,000-substep rollout needs 19 GB of tape. The standard remedies are gradient checkpointing at an interval near $\sqrt{T}$ , truncating the loss window to a handful of frames, and recomputing forward segments during the backward pass. Numerically, the same product of Jacobians makes gradients explode for stiff materials and vanish once the trajectory has been dissipated by friction, which is why practitioners fit a log-parameterized stiffness and soften contact before trusting any gradient at all.

Log-log chart of backward-pass memory in bytes versus number of simulator substeps for 200 thousand particles, comparing storing the full trajectory which grows linearly and reaches 40 gigabytes near 2000 substeps, against square-root gradient checkpointing which stays under 3 gigabytes, with a dash-dotted horizontal line marking a 40 gigabyte HBM budget

Figure 3: Why differentiable rollouts are short. At 19 MB of particle state per substep, the naive tape exhausts a 40 GB HBM budget after roughly 2,000 substeps, about one second of simulated time, while square-root checkpointing keeps memory in the low gigabytes at the price of one extra forward pass.

Property	NeRF (implicit field)	3D Gaussians (explicit primitives)
Primitive	MLP or hash grid queried at continuous positions; no persistent identity	Set of anisotropic Gaussians, each with mean, covariance, opacity, SH color
Coupling to solver	Needs a deformation field or an Eulerian-Lagrangian resampling step onto particles	Splat is used directly as an MPM material point, no conversion layer
Render cost per frame	Hundreds of network queries per ray; typically tens of ms to seconds	Tile-based rasterization, real time at 1080p on one GPU
Handles topology change	Naturally, since density is just a field value that can appear or vanish	Poorly; fracture needs splitting or spawning new Gaussians outside the solver
Dominant failure mode	Slow rollouts, floaters in unobserved regions, noisy gradients through resampling	Baked lighting and unrotated SH, plus spiky splats that behave like bad particles

August 19, 2026

DL0165 End-to-End Driving Trajectory Scoring

How do end-to-end autonomous driving architectures combine world model generation with trajectory scoring and cost function evaluation to select the safest driving path?

Answer

Competitive end-to-end stacks do not regress one trajectory and drive it. They propose many candidates, imagine the future under each one, and then take the argmax of an explicit cost, which turns planning into a ranking problem over a finite action set. The world model is what makes the ranking meaningful: it is an ego-conditioned forecaster that answers a counterfactual question, namely what the agents, the occupancy grid, or the camera views would look like if this particular trajectory were executed. The cost function then reads that imagined rollout and produces bounded sub-metrics for collision, drivable-area compliance, time-to-collision, comfort, and progress, which are combined so that hard safety terms multiply and soft preference terms are averaged. Because rolling out a simulator for thousands of candidates is far too slow at a 10 Hz replanning rate, production designs run the expensive scorer offline as a teacher and distill it into a small scoring head that evaluates the whole vocabulary in one batched forward pass.

(1) Proposal Set Instead Of Regression: a fixed trajectory vocabulary (8192 clustered 4-second trajectories in Hydra-MDP, 4096 in VADv2) or a small set of diffusion anchors preserves multi-modality and gives the cost function something concrete to compare.
(2) Ego-Conditioned World Model: the rollout must be conditioned on the candidate action, otherwise every candidate shares one predicted future and the scores carry no causal information about the ego’s own choice.
(3) Multiplicative Gates Plus Weighted Soft Terms: collision and off-road terms enter as multiplicative factors so a single violation zeroes the score, while time-to-collision, comfort, and progress enter as a weighted average that only re-ranks the survivors.
(4) Simulator As Teacher, Head As Student: a rule-based closed-loop metric computed with LQR tracking on a bicycle model labels every candidate offline, and a cross-attention scoring head learns those labels, converting a multi-second simulation into a millisecond-scale inference.
(5) Reactivity Decides Validity: non-reactive rollouts hold other agents on their logged paths, which systematically over-rewards cutting in and under-penalizes aggressive gap acceptance.
(6) Latency Sets The Design: a 4-second horizon at 10 Hz is 40 poses per candidate and the whole loop must close inside about 100 ms, which rules out per-candidate video generation online.

Left-to-right pipeline diagram: multi-view camera BEV scene encoder, then a proposal generator producing a vocabulary of 8192 four-second trajectories, then an ego-conditioned world model rollout predicting agent futures and occupancy for each candidate, then metric heads for no at-fault collision, drivable area compliance, time-to-collision, comfort and progress, then score aggregation and argmax selecting one trajectory, with an offline rule-based simulator below feeding distillation targets into the metric heads

Figure 1: One scene encoder, many candidates, one score. The world model rollout is evaluated per candidate, and the metric heads are trained against an offline rule-based simulator, so the simulator’s judgement is available at inference without running it.

The world model shows up in three distinct roles, and confusing them is a common interview mistake. In the in-loop role, a generative model such as Drive-WM synthesizes multi-view future frames conditioned on each candidate and an image-space reward built from map and object cues picks the branch, which is expressive but so expensive that only a shallow tree of a few branches is affordable. In the representation role, future-occupancy or future-agent forecasting is an auxiliary head that shapes the BEV latent, so cost terms are computed on predicted geometry rather than pixels, which is the practical choice inside a 100 ms budget. In the offline role, high-fidelity generators supply rare scenarios and closed-loop training environments, so the cost function is validated against near-misses that almost never appear in logged data. Whatever the role, the failure mode is the same: if the rollout ignores that other agents respond to the ego, the cost function rewards trajectories that would have been vetoed by any reactive simulator.

Bird's-eye-view schematic of a two-lane road with an ego vehicle, a slow lead vehicle ahead in the same lane, and a vehicle in the left lane. Four candidate trajectories are drawn: an overtake into the left lane that collides with the left-lane vehicle and scores zero, a swerve onto the non-drivable shoulder that also scores zero, a full-stop brake that stays legal but has near-zero progress and scores 0.48, and a lane-keeping follow trajectory that scores 0.92 and is selected

Figure 2: Why the aggregation is multiplicative rather than additive. Candidates A and B are gated to exactly zero by collision and drivable-area violations no matter how much progress they buy, and only then do the soft terms separate the timid full stop from the trajectory that actually makes progress.

Mathematical Formulation:
$\hat{s}_{t+1:t+H} \sim p_{\phi}(\cdot \mid o_{\leq t}, \tau)$
$f_m(\tau) = g_m(\hat{s}_{t+1:t+H})$
$P(\tau) = \prod_{m \in \mathcal{M}_g} f_m(\tau)$
$Q(\tau) = \frac{\sum_m w_m f_m(\tau)}{\sum_m w_m}$
$S(\tau) = P(\tau) \cdot Q(\tau)$
$\tau^{*} = \arg\max_{\tau \in \mathcal{V}} S(\tau)$

Where:

$\tau^{*}$ is the executed trajectory and $\mathcal{V}$ the candidate vocabulary, typically a few thousand clustered 4-second trajectories sampled at 10 Hz.
$o_{\leq t}$ is the sensor history, $p_{\phi}$ the ego-conditioned world model, and $\hat{s}_{t+1:t+H}$ the imagined future over horizon $H$ under that specific $\tau$ .
$g_m$ extracts metric $m$ from the rollout and $f_m(\tau)$ is its value, bounded to the unit interval so that scores are comparable across scenes.
$\mathcal{M}_g$ are the gate metrics (no at-fault collision, drivable-area compliance) whose product $P(\tau)$ vetoes a candidate outright.
$w_m$ are the weights of the soft metrics (time-to-collision, comfort, ego progress) aggregated by $Q(\tau)$ , which only re-ranks candidates that passed every gate.

NAVSIM PDM Score Instantiation:
$S = \mathrm{NC} \cdot \mathrm{DAC} \cdot Q$
$Q = (5\,\mathrm{TTC} + 2\,\mathrm{C} + 5\,\mathrm{EP}) / 12$

Here NC and DAC are binary gates, while time-to-collision, comfort, and ego progress carry weights 5, 2, and 5. The weights are the policy: raising the progress weight produces a stack that squeezes through gaps, and raising comfort produces one that refuses to brake late. The distillation objective simply asks the student head to reproduce every teacher sub-metric per candidate.

Distillation And Latency Budget:
$\mathcal{L} = \sum_{\tau} \sum_m \mathrm{BCE}(\hat{f}_m(\tau), f_m^{\mathrm{sim}})$
$t_{perc} + t_{gen} + t_{score} \leq 100\ \mathrm{ms}$
$40\ \mathrm{ms} + 25\ \mathrm{ms} + 15\ \mathrm{ms} = 80\ \mathrm{ms}$

Log-log chart of scoring latency in milliseconds against the number of scored candidate trajectories, with three curves: a distilled learned scoring head that stays nearly flat from 14 to about 32 milliseconds up to 8192 candidates, a rule-based closed-loop simulator that grows linearly at roughly 0.45 milliseconds per candidate, and a generative video world model rollout at roughly 320 milliseconds per candidate, plus a dashed horizontal line marking a 100 millisecond replanning budget

Figure 3: Representative per-candidate costs against the 100 ms replanning budget. A distilled scoring head is almost flat in the number of candidates because scoring is one batched forward pass, a rule-based simulator caps out near a hundred candidates, and a per-candidate video rollout blows the budget on its first branch.

Property	Rule-based simulator in the loop	Generative world model in the loop	Distilled learned scorer
Proposal source	Centerline offsets crossed with target speeds, about 15 proposals	A shallow tree of a few branches per replan	Clustered vocabulary of 4096 to 8192 trajectories, or 20 diffusion anchors
How futures are obtained	Bicycle model plus LQR tracking with map and collision checks	Diffusion rollout of multi-view frames or occupancy conditioned on the action	No explicit rollout at inference, the head predicts each sub-metric directly
Agent reactivity	Configurable, IDM background traffic reacts to the ego	Learned and implicit, quality depends entirely on training data	Inherited from whatever the teacher assumed, usually non-reactive logs
Per-candidate cost	Sub-millisecond but CPU-bound and linear in candidates	Hundreds of milliseconds of GPU time per branch	Microseconds, batched inside one transformer forward pass
Dominant failure mode	Hand-written dynamics and a coarse proposal set miss creative maneuvers	Hallucinated geometry, action leakage, and unusable latency	Copies the teacher’s blind spots and can latch onto ego-status shortcuts

August 19, 2026

DL0164 BEV Transformation: Lift-Splat-Shoot

Explain Bird’s-Eye-View transformation methods such as Lift-Splat-Shoot and transformer cross-attention for mapping multi-view camera video into a unified 3D world representation, as used in camera-only autonomous driving stacks and benchmarked on nuScenes.

Answer

A BEV transformation converts $N_{c}$ perspective images, each of which has thrown away the depth of every pixel, into a single metric grid in the ego frame where one cell always means the same physical patch of ground. Every method must invent the missing depth, and the two families differ only in which direction they move information. Forward projection, introduced by Lift-Splat-Shoot (LSS), predicts a categorical depth distribution per pixel, lifts each pixel into a frustum of $D$ candidate 3D points weighted by that distribution, then splats the points into BEV pillars with sum pooling. Backward projection, popularised by BEVFormer, starts from a fixed set of learned BEV queries, projects each query’s 3D anchor points into every camera using the known intrinsics and extrinsics, and pulls features back with deformable cross-attention so no explicit depth prediction is required. Both produce the identical output contract, typically a $200 \times 200 \times 256$ feature map at roughly 0.5 m resolution, which is why detection, map segmentation, occupancy and planning heads can be shared, and both add temporal fusion by warping the previous BEV feature into the current ego frame before merging.

(1) The Core Difficulty Is Depth, Not Geometry: the pixel-to-ray mapping $K^{-1}$ and the camera-to-ego rigid transform are exactly known, so the only unknown is the scalar range along each ray.
(2) Forward Projection (Push): LSS predicts $\alpha_{u,v,d}$ over $D$ discrete depth bins, takes an outer product with the context feature, and voxel-pools the resulting frustum point cloud into pillars.
(3) Backward Projection (Pull): BEV queries carry their own 3D position, project into the cameras that actually see them, and sample features with deformable attention, which sidesteps depth estimation entirely.
(4) Shared Output Contract: both write into the same ego-frame grid, so the transformation is a swappable module rather than an architecture commitment.
(5) Temporal Recurrence Is Not Optional: warping $B_{t-1}$ by the ego pose delta and fusing it into $B_{t}$ is what makes velocity estimation and short-occlusion memory possible from cameras alone.
(6) Depth Supervision Decides Accuracy: BEVDepth showed that supervising $\alpha$ with LiDAR-projected depth, rather than letting the detection loss shape it, is the single largest quality lever for the forward family.

Pipeline diagram: six surround cameras feed a shared 2D backbone with FPN, which splits into a top lane predicting a per-pixel depth distribution over 59 bins followed by an outer product and voxel pooling of roughly one million frustum points, and a bottom lane of learned 200 by 200 BEV queries whose 3D anchors are projected into the hit cameras for deformable cross-attention over image features used as keys and values; both lanes write into one 200 by 200 by 256 unified BEV feature map that feeds detection, map and occupancy heads

Figure 1: Two directions, one destination. The push lane commits to a depth distribution and scatters features outward; the pull lane keeps the grid fixed and gathers features inward. Everything downstream of the unified BEV feature map is identical, which is why these modules are interchangeable in practice.

The forward path is best understood as a soft, differentiable version of unprojecting a depth map. A pixel with context feature $c_{u,v} \in \mathbb{R}^{C}$ does not pick one depth; it spreads that feature over all $D$ bins in proportion to $\alpha_{u,v,d}$ , so a confident pixel deposits nearly all of its mass in one pillar while an ambiguous pixel smears a faint trail along its ray. Because the splat is a sum, the operation is permutation-invariant and handles overlapping camera fields of view for free, and because it is differentiable, gradients reach the depth head through the pooling. The engineering cost is the frustum point count, which is why production implementations replace the naive scatter with a sorted cumulative-sum pooling kernel or a preallocated BEVPoolv2 index table that skips materialising the point cloud at all.

Three panels: a grid of image pixels with one highlighted pixel carrying a 256-dimensional context feature; a bar chart of the predicted probability over 59 depth bins showing a sharp peak near 18 metres for a confident pixel and a broad dashed curve for an ambiguous pixel; and a top-down bird's-eye-view grid with the camera at the origin, a ray fanning outward, circles along the ray whose size is proportional to the depth probability, and one highlighted 3 by 3 metre pillar where the points are sum-pooled

Figure 2: One pixel becomes $D$ weighted 3D points. The width of the depth distribution is literally the width of the smear in BEV, so a flat $\alpha$ over a textureless road or a night-time scene produces a long low-confidence streak instead of a localised object.

Forward Projection (Lift-Splat-Shoot):
$\tilde{u} = (u, v, 1)^{T}$
$p_{c} = d\, K^{-1} \tilde{u}$
$p_{e} = R\, p_{c} + t$
$F_{u,v,d} = \alpha_{u,v,d}\, c_{u,v}$
$\sum_{d=1}^{D} \alpha_{u,v,d} = 1$
$B(x,y) = \sum_{p \in \Pi(x,y)} F(p)$

Where:

$B(x,y) \in \mathbb{R}^{C}$ is the BEV feature at grid cell $(x,y)$ in the ego frame, and $\Pi(x,y)$ is the set of frustum points whose ego coordinates fall inside that pillar.
$\tilde{u}$ is the homogeneous pixel coordinate, $K$ the camera intrinsic matrix, and $(R, t)$ the camera-to-ego extrinsic rotation and translation.
$d$ indexes the depth bins, with $d \in \{1, \ldots, D\}$ over a fixed range such as 1 m to 60 m in 1 m steps, giving $D = 59$ .
$\alpha_{u,v,d}$ is the softmax depth distribution for pixel $(u,v)$ and $c_{u,v}$ its context feature, so $F_{u,v,d}$ is the outer-product lift.
The splat is sum pooling, which keeps the operation order-free across cameras and differentiable with respect to both $\alpha$ and $c$ .

The backward path inverts the flow of information. A query at grid cell $(x,y)$ is lifted to $N_{z}$ anchor heights along a vertical pillar, each anchor is projected into every camera, and only the cameras whose image plane actually contains the projection contribute. Deformable attention then samples a handful of learned offsets around each projected location, so cost scales with the number of queries rather than with image resolution times depth bins, and a query near a lane boundary can shift its sampling points to where the evidence is instead of trusting a predicted depth. The trade-off is that the geometry is now an attention prior rather than a hard constraint: if extrinsics are wrong, the network can still learn to compensate, which is convenient during training and dangerous during deployment because the failure is silent.

Backward Projection (Cross-Attention):
$q_{xy} = Q(x,y) + \mathrm{PE}(x,y)$
$r_{j} = (x, y, z_{j})$
$\hat{p}_{ij} = \pi_{i}(r_{j})$
$A_{i} = \sum_{j=1}^{N_{z}} \mathrm{DA}(q_{xy}, \hat{p}_{ij}, F_{i})$
$\mathrm{CA}(q_{xy}) = \frac{1}{|V_{xy}|} \sum_{i \in V_{xy}} A_{i}$

Where:

$\mathrm{CA}(q_{xy})$ is the cross-attention output written into BEV cell $(x,y)$ , and $q_{xy}$ is the learned query plus its 2D positional encoding.
$r_{j}$ is the $j$ -th pillar anchor at height $z_{j}$ , with $N_{z} = 4$ a common choice spanning roughly -5 m to 3 m.
$\pi_{i}$ is the full projection of camera $i$ , so $\hat{p}_{ij}$ is a sub-pixel image location and $F_{i}$ the multi-scale feature map of that camera.
$V_{xy}$ is the set of cameras whose frustum contains at least one anchor, so the average is taken only over hit views and empty views contribute nothing.
$\mathrm{DA}$ is deformable attention, which samples a few learned offsets around $\hat{p}_{ij}$ with bilinear interpolation instead of attending to all pixels.

Frustum Cost At A Typical Configuration:
$N_{pts} = N_{c} \cdot H \cdot W \cdot D$
$6 \times 32 \times 88 \times 59 = 996864$
$200 \times 200 = 40000$

Roughly one million frustum points collapse into forty thousand pillars, an average of about 25 points per cell, and that ratio is exactly why the pooling kernel rather than the backbone is often the latency bottleneck in the forward family. It also exposes the accuracy story: because every point sits on a known ray, a lateral mistake of one pixel at 50 m is only about 4 cm, while a 5% depth mistake at the same range is 2.5 m. BEV error is dominated by range error, and it grows linearly with distance.

Log-scale line chart of bird's-eye-view position error in metres against range from the ego vehicle from 2 to 80 metres, showing straight rising lines for 10 percent, 5 percent and 2 percent relative depth error, and a much lower line for a one-pixel lateral error at focal length 1266 pixels, with a dashed horizontal line at the 2 metre matching threshold and an annotation noting that a 5 percent depth error at 60 metres displaces the box by 3 metres

Figure 3: Range error, not image-plane error, sets BEV quality. A one-pixel lateral error stays under 10 cm across the whole working range, while a modest relative depth error crosses the 2 m matching threshold somewhere between 20 m and 100 m depending on the depth head, which is why depth supervision and long-baseline temporal stereo pay off so heavily.

Property	Forward push (LSS, BEVDet, BEVDepth)	Backward pull (BEVFormer)	Implicit 3D encoding (PETR)
Depth handling	Explicit categorical distribution over D bins, optionally LiDAR-supervised	No depth head; anchors at fixed pillar heights sample every hit view	3D coordinates baked into image position encodings, depth learned implicitly
Dominant cost	Frustum scatter of about 1M points; needs a cumsum or index-table kernel	40,000 queries times layers times sampling points, quadratic in grid side	Global attention over all image tokens, no explicit BEV grid to build
Calibration sensitivity	Hard geometric constraint, so extrinsic drift shifts features into wrong pillars	Attention can partly absorb drift, which hides the fault instead of surfacing it	Most tolerant, but least interpretable when a single camera goes bad
Temporal fusion	Warp and concatenate past BEV grids, or run temporal stereo across frames	Recurrent BEV self-attention on the ego-warped previous grid	Propagate sparse object queries forward in time, no grid to warp
Typical failure	Flat depth distribution smears a distant object along its ray	Empty cells still consume compute, and unseen regions hallucinate from priors	Weak spatial locality makes small distant objects easy to miss

August 19, 2026

DL0163 Map-Based vs Mapless World Models

How do map-based world models conditioned on HD-maps differ from end-to-end mapless world models that predict from raw sensor observations, for a driving stack like Waymo’s or Wayve’s?

Answer

Both families learn a latent transition model that rolls the scene forward under a candidate ego action, so the architectural difference is not the dynamics head but what the dynamics head is allowed to condition on. A map-based world model receives a local HD-map crop $m_t$ , a rasterized or vectorized patch of lane centerlines, stop lines, crosswalks, and traffic-light-to-lane associations, indexed out of a global map by the pose that the localization stack estimates. That crop enters as a hard geometric prior, so the network never has to learn road topology from pixels and can spend capacity on agent behavior. A mapless model sees only the sensor history $o_{t-k:t}$ and must reconstruct that same topology inside its latent on every frame, which is statistically much harder but works on roads nobody has surveyed. The practical result is a swap of error sources rather than a strict improvement: the map-based system inherits map staleness and localization error, the mapless system inherits data hunger and weaker long-horizon geometric consistency.

(1) Conditioning Set: map-based factorizes as $p(z_{t+1} \mid z_t, a_t, m_t)$ with an extra exogenous input, while mapless factorizes as $p(z_{t+1} \mid z_{t-k:t}, a_t)$ and must carry topology in the recurrent state.
(2) Where The Prior Lives: in one system lane geometry is a surveyed asset refreshed by a mapping fleet, in the other it is weights learned from driving video.
(3) Sample Efficiency: handing the model the road graph removes an enormous nuisance factor, so map-conditioned models converge on thousands of hours where mapless generative models are trained on tens of thousands to millions.
(4) Failure Signature: a stale or misaligned map produces confidently wrong rollouts, whereas a mapless model facing an ambiguous road typically produces a wide, visibly uncertain distribution.
(5) Pose Coupling: the map crop is only meaningful in the correct frame, so map-based prediction quality is bounded by localization accuracy; mapless prediction has no pose dependency at all.
(6) Scaling And ODD: map-based scales with surveyed kilometers and re-survey cadence, mapless scales with data and compute, which is why geofenced robotaxi and everywhere-consumer programs made opposite choices.

Two stacked pipelines: the upper map-based pipeline encodes surround camera and LiDAR into a latent state, takes an HD map crop selected by ego pose as an extra conditioning input to the transition model, and decodes BEV occupancy, agent futures, and an ego plan; the lower mapless pipeline tokenizes surround camera video, feeds a spatio-temporal transformer or diffusion prior conditioned only on past tokens and the action, and decodes future frames, an implicit map, and an ego plan

Figure 1: The two stacks share an encoder, a latent transition model, and a decoder. Only one extra edge differs: the HD map crop selected by the estimated pose feeding the transition model. That single edge is what buys sample efficiency and what imports two new failure sources, map staleness and localization drift.

The interesting behavior appears when the map and the world disagree. Because the map is treated as ground truth during training, the model learns to trust it, and at inference it has no mechanism to discount it: a lane that was closed for construction last week is still a valid lane inside $m_t$ , so the imagined rollout drives straight through the closure with low predictive variance. A mapless model never had that crutch, so its estimate of drivable space comes from the same pixels that show the cones. It pays for this with weaker long-horizon geometry, since nothing anchors an eight-second rollout to a globally consistent lane graph and errors compound in the latent. This is also why the honest comparison is not accuracy on a nominal benchmark, where the map-conditioned model almost always wins, but accuracy conditioned on map validity.

Two bird's-eye-view panels of the same two-lane road with a lane closure marked by cones. In the left map-based panel a blue HD map centerline runs straight through the closure and the predicted rollout follows it with a narrow uncertainty band, ending in a red cross at the cones. In the right mapless panel there is no map centerline, the predicted rollout bends into the adjacent lane around the cones, and the uncertainty band is visibly wider.

Figure 2: Same scene, same closure, different conditioning. The map prior keeps the rollout tight and correct whenever the map matches reality, and keeps it tight and wrong when it does not. The mapless rollout is wider but reactive, because the only evidence it ever had for drivable space is the current image.

Mathematical Formulation:
$z_t = E_\theta(o_{t-k:t})$
$m_t = \Pi(M, \hat{T}_t), \quad \hat{T}_t = T_t + \epsilon_{\mathrm{loc}}$
$p_{\mathrm{map}} = p_\theta(z_{t+1} \mid z_t, a_t, m_t)$
$p_{\mathrm{free}} = p_\phi(z_{t+1} \mid z_{t-k:t}, a_t)$
$\mathcal{L} = \sum_{h=1}^{H} \ell(D(z_{t+h}), o_{t+h})$

Where:

$z_t \in \mathbb{R}^{d}$ is the latent scene state produced by encoder $E_\theta$ from the observation window $o_{t-k:t}$ of surround camera, radar, and optionally LiDAR frames.
$M$ is the global HD map, $T_t \in SE(3)$ the true ego pose, $\hat{T}_t$ the estimate produced by localization, and $\Pi$ the operator that crops and transforms $M$ into the ego frame.
$\epsilon_{\mathrm{loc}}$ is localization error. It is not observation noise: it rigidly shifts the entire conditioning input, so a 0.5 m lateral bias moves every lane boundary by 0.5 m.
$a_t$ is the candidate ego action, which is what makes both models action-conditioned and therefore usable for planning rather than passive video prediction.
$\theta$ and $\phi$ are the two parameter sets; $\phi$ must additionally encode road topology that $\theta$ receives for free through $m_t$ .
$D$ is the decoder (future frames, BEV occupancy, or agent boxes), $H$ the rollout horizon in steps, and $\ell$ the per-step loss, typically reconstruction plus a KL or diffusion denoising term.

Schematic line chart of mean lateral rollout error in metres against prediction horizon in seconds, with three curves: a map-based model with a fresh map staying lowest, a mapless model rising moderately faster, and a map-based model with a stale or misaligned map rising steepest and crossing the others within about one second, plus a dashed horizontal line at 0.5 metres marking lane-keeping tolerance

Figure 3: Schematic, but the ordering is the point. With a valid map the conditioned model dominates at every horizon; with a stale or misaligned map the same model becomes the worst of the three within roughly a second, because the prior is applied with full confidence in the wrong place. A mapless model has no best case that good and no worst case that bad.

Property	Map-based (HD-map conditioned)	Mapless (raw sensor)
Conditioning input	Sensor latent plus a pose-indexed vector or raster map crop	Sensor history and the ego action only
What must be learned	Agent behavior and interaction; road topology is given	Topology, drivable space, and behavior, jointly from pixels
Data appetite	Thousands of hours plus a surveyed and maintained map	Tens of thousands of hours upward; Cosmos-class pretraining uses about 20M hours of video
Dominant failure mode	Stale map or localization drift produces low-variance wrong rollouts	Ambiguous or occluded geometry produces drifting, globally inconsistent rollouts
Scaling axis	Surveyed kilometers and re-survey cadence, so expansion is operational	Data and compute, so expansion is a training-run problem
Verifiability	Map is an auditable artifact; behavior can be certified per mapped intersection	Topology lives in weights, so guarantees come only from aggregate evaluation
Traffic-light and rule handling	Light-to-lane association and speed limits come from the map	Association must be inferred visually, a known long-tail weakness

August 19, 2026