Author: admin

DL0170 Mamba Selective SSM vs Attention

What is Mamba’s Selective State Space Model (S6), and how does its input-dependent gating compare to Transformer self-attention in computational complexity and long-range recall?

Answer

Mamba’s S6 layer is a linear recurrence that carries a fixed-size hidden state $h_t$ per channel and reads it out with a learned projection, exactly like the earlier S4 state space model. The one change that matters is selection: the timestep $\Delta_t$ and the input and output maps $B_t, C_t$ are linear functions of the current token instead of fixed parameters, which turns the discretized decay $\bar{A}_t = \exp(\Delta_t A)$ into an input-dependent forget gate. That single change gives the model the ability to skip filler tokens and write informative ones into the state, but it destroys linear time invariance, so the layer can no longer be evaluated as one long FFT convolution and instead needs a hardware-aware parallel associative scan. On complexity the comparison is lopsided in Mamba’s favor: sequence mixing is $O(L d N)$ instead of $O(L^2 d)$ , and decoding needs a constant $O(d N)$ state rather than a KV cache that grows with context. On recall the comparison reverses, because a bounded state is a lossy summary of the past, so tasks that need verbatim retrieval of an arbitrary earlier span degrade in a way that attention’s exact, if expensive, cache does not.

(1) Selection Is The Whole Idea: $\Delta_t, B_t, C_t$ become projections of $x_t$ , while $A$ stays a learned diagonal matrix whose effective decay $\exp(\Delta_t A)$ is nonetheless input-dependent.
(2) $\Delta_t$ Is A Forget Gate: small $\Delta_t$ gives $\bar{A}_t \approx I$ and the token is ignored, large $\Delta_t$ drives $\bar{A}_t$ toward zero and overwrites the state with the new input.
(3) Time Invariance Is Lost: S4 could precompute one global convolution kernel, Mamba cannot, so training uses a work-efficient associative scan in SRAM with activation recomputation instead of an FFT.
(4) Linear Compute: mixing costs $O(L d N)$ with $N = 16$ in Mamba-1, so the quadratic attention term dominates once $L$ approaches the model width $d$ .
(5) Bounded Decode State: $2 d N$ values per layer regardless of context length, versus $2 L d$ for an MHA KV cache, which is what makes million-token streaming decode cheap.
(6) Recall Is The Trade: fixed capacity caps exact copying and multi-query associative recall, which is precisely why production long-context models are usually hybrids with a few full-attention layers.

Diagram of four input tokens the, key, is, 7Q4 each producing an input-dependent timestep Delta_t shown as 0.04, 1.6, 0.05 and 2.1 with skip or write annotations, feeding a chain of fixed-size state boxes h_1 through h_4 linked by input-dependent decay arrows, and a readout row of outputs y_1 through y_4, with a side box noting the state holds 2dN values for any sequence length

Figure 1: The recurrence is ordinary, the gate is not. Because $\Delta_t$ is computed from the token itself, a filler word produces $\bar{A}_t \approx I$ and passes through without disturbing the state, while a rare identifier produces a large step that writes into the state and forgets older content. The state width never depends on $L$ , which is both the efficiency win and the recall ceiling.

The engineering consequence of dropping time invariance is that the layer becomes memory-bound rather than FLOP-bound. Materializing the expanded states, which have shape $(B, L, 2d, N)$ , would move far more bytes through HBM than the arithmetic justifies, so the reference implementation fuses discretization, scan, and readout into one kernel that keeps states in SRAM and recomputes them during the backward pass. The recall side has a cleaner theoretical story. Attention stores every key and value, so retrieving a specific earlier token is a lookup, whereas a selective SSM must have decided at write time to keep that token, and its $2 d N$ slots bound how many distinct key-value associations can survive. This is why Mamba matches or beats Transformers on language modeling perplexity, audio, and DNA, yet lags on induction-head style copying and needle-in-a-haystack retrieval, and why Mamba-2 raises $N$ from 16 to as much as 256 while hybrids keep a small number of attention layers to do the exact lookups.

Mathematical Formulation:
$h'(t) = A h(t) + B x(t)$
$y(t) = C h(t)$
$\Delta_t = \mathrm{softplus}(W_{\Delta} x_t + b)$
$B_t = W_B x_t, \quad C_t = W_C x_t$
$\bar{A}_t = \exp(\Delta_t A)$
$h_t = \bar{A}_t h_{t-1} + \Delta_t B_t x_t$
$y_t = C_t^{\top} h_t$

Where:

$y_t$ is the layer output for token $t$ and $h_t \in \mathbb{R}^{N}$ is the hidden state held for one channel, so the full layer keeps $2 d N$ values under the standard expansion factor of 2.
$x_t$ is the input activation, $t \in \{1, \ldots, L\}$ indexes the sequence, and $h_0 = 0$ is the required initial condition.
$A$ is a learned diagonal matrix parameterized as $A = -\exp(A_{\log})$ so every eigenvalue is negative and the discrete decay stays stable.
$\Delta_t > 0$ is the input-dependent timestep, and $W_{\Delta}, W_B, W_C$ are the low-rank projections that make the layer selective; $b$ is initialized so that $\Delta_t$ starts in a useful timescale range.
$\bar{A}_t$ is the zero-order-hold discretization of $A$ , and $\Delta_t B_t x_t$ is the simplified input term Mamba uses in place of the exact $\bar{B}_t$ .
$d$ is the model width, $N$ the state dimension (16 in Mamba-1), and $L$ the sequence length; the scan is $O(L d N)$ work with $O(\log L)$ depth.

Decode Memory At 32k Context (values per layer):
$L = 32768, \quad d = 2048, \quad N = 16$
$\mathrm{cache} = 2 L d = 1.34 \times 10^{8}$
$\mathrm{state} = 2 d N = 6.55 \times 10^{4}$
$\mathrm{ratio} = L / N = 2048$

Those are element counts, so in bytes the gap narrows a little when the cache is fp16 and the SSM state is fp32, and it narrows further with grouped-query attention, which divides the cache by the query-to-key-value head ratio. The structural point survives every such adjustment: the attention term is proportional to $L$ and the SSM term is not, so past a few thousand tokens the recurrent model is decoding from a constant working set while the Transformer is streaming a cache that eventually dominates both memory and bandwidth.

Two log-log panels: the left panel plots forward-pass FLOPs per layer against sequence length for a self-attention block with a quadratic term and a Mamba block with a linear term, with a dash-dotted vertical crossover line near two thousand tokens, and the right panel plots decode state in megabytes showing a rising fp16 KV cache line against a flat constant Mamba state line at 0.26 megabytes

Figure 2: Below roughly $L \approx d$ the wider Mamba block, which expands the channel dimension by 2, actually costs more than attention, because both are dominated by their projection matmuls. Above it the $O(L^2 d)$ term takes over, and at 128k tokens the attention layer needs about 20 times the FLOPs and a per-layer cache about 4,000 times larger than the constant SSM state.

Property	Selective SSM (Mamba S6)	Self-attention (MHA)
Sequence-mixing cost	O(L d N), linear in L	O(L^2 d), quadratic in L
Cost per decoded token	O(d N), independent of context	O(L d), grows with context
State carried between tokens	2 d N values per layer, fixed	2 L d values per layer, unbounded
Training parallelism	Associative scan with recomputation; no FFT convolution, since the recurrence is time-varying	Pure matmuls, no sequential dependence at all
Exact retrieval and copying	Lossy; bounded by state capacity, degrades as the number of stored associations grows	Exact within the window; copying long spans is easy
Natural fit	Audio, DNA, streaming, very long inputs summarized rather than quoted	In-context retrieval, many-shot prompts, verbatim citation

August 19, 2026

DL0169 Softmax-1 Off-by-One Attention

What is the Softmax-1 (off-by-one) modification to attention, and why does subtracting one from the denominator improve length generalization and register-token behavior in recent Transformer architectures?

Answer

Softmax-1, also called off-by-one or quiet attention, replaces the attention normalizer $\sum_j e^{z_j}$ with $1 + \sum_j e^{z_j}$ . That extra constant is exactly what you get by appending a virtual key with logit 0 and value vector 0, so the weights over real tokens now sum to $S/(1+S)$ , which is strictly below one, and a head that finds nothing relevant can emit the zero vector. Ordinary softmax has no such option, because its weights are forced to sum to one whether or not any key matches. Heads that want to be a no-op therefore learn to dump their leftover mass on a low-information token, usually the first token or a delimiter, and to inflate that token’s residual to enormous magnitude: the massive activations that dominate activation-quantization error and the artifact tokens that pollute ViT attention maps. The name refers to the denominator being off by one; the operational reading is that one unit of attention budget is subtracted from the real keys and parked in a null option that costs nothing to select.

(1) One Constant, No New Parameters: adding 1 to the denominator is algebraically identical to concatenating a zero-logit, zero-value key, so the change touches one line of the kernel and adds no weights.
(2) Mass Below One Is The Whole Point: the head gains an explicit “abstain” action, and the output norm $\lVert o \rVert$ can go to zero without any token’s value vector having to be zero.
(3) Absolute Instead Of Relative Scores: ordinary softmax is invariant to a constant shift of all logits, so “nothing is similar” is inexpressible; the fixed zero reference makes the logit level itself meaningful.
(4) The Sink Becomes A Mechanism, Not A Token: vanilla models route no-op mass through a specific KV slot that a sliding window may evict, whereas the constant is always present, which is what stabilizes long-context and windowed decoding.
(5) Registers Stop Being Garbage Dumps: once abstention is free, register and background tokens keep their global-storage role without absorbing surplus attention, so their norms stay moderate and attention maps stay readable.
(6) Production Form Is A Learned Sink Logit: replacing the 1 with $e^{s}$ for a per-head learned $s$ strictly generalizes Softmax-1, which is the $s = 0$ special case.

Two-panel grouped bar chart of attention weights over six keys plus a null slot. In panel A every logit is below zero and ordinary softmax still renormalizes to sum one, while softmax-1 keeps only 0.17 of the budget and sends 0.83 to the null slot. In panel B one logit is six and the two methods are nearly identical, with the null slot receiving only a quarter of a percent.

Figure 1: The gate is selective, not a uniform rescaling. When no key matches (panel A), ordinary softmax renormalizes a row of weak scores into a confident-looking distribution, while Softmax-1 keeps 17% of the budget on real keys and abstains with the rest. When a real match exists (panel B), the constant 1 is negligible against $e^{6} \approx 403$ and the two are indistinguishable, so the modification only fires where the head has nothing to say.

Mathematical Formulation:
$\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{L} e^{z_j}}$
$\mathrm{softmax}_1(z)_i = \frac{e^{z_i}}{1 + \sum_{j=1}^{L} e^{z_j}}$
$S = \sum_{j=1}^{L} e^{z_j}$
$m = \frac{S}{1 + S}$
$m = \sigma(z + \log L)$
$p_i = \frac{e^{z_i}}{e^{s} + \sum_{j=1}^{L} e^{z_j}}$
$o = \sum_{i=1}^{L} p_i v_i$

Where:

$z_i = q^{\top} k_i / \sqrt{d}$ is the scaled dot-product logit for key $i$ , and $L$ is the number of visible keys after masking.
$S$ is the ordinary denominator, so the added 1 is just $e^{0}$ : the virtual key’s exponentiated logit.
$m$ is the total attention mass retained by real keys; the complement $1/(1+S)$ is the abstention share, and it is positive for every finite logit row.
The uniform-logit identity uses $\sigma$ for the logistic function: with all $L$ logits tied at $z$ , the mass is a logistic gate whose half-way point sits at $z = -\log L$ .
$s$ is a learned per-head sink logit replacing the constant, and $s = 0$ recovers Softmax-1 exactly.
$o$ is the head output; because $\sum_i p_i$ can be near zero, $o \approx 0$ is reachable without constraining any $v_i$ .

The length-generalization argument has two halves. First, in ordinary softmax a no-op head’s surplus mass is $1 - m_{\mathrm{rel}}$ , and where it lands depends on how many irrelevant keys exist, so a head calibrated at a 4k training length re-partitions its budget when the context reaches 32k; the model compensates by pinning a token whose residual magnitude was tuned at the training length, and Sun et al. measured such coordinates in the thousands while the median activation sits near 0.1. Second, that sink is a KV slot, not an architectural constant. StreamingLLM showed the consequence directly: evict the first few tokens from a sliding window and perplexity explodes, while pinning four sink tokens keeps generation stable out to millions of tokens. Softmax-1 turns the sink into a term of the denominator that no eviction policy can delete, which is why windowed and extrapolated decoding stop depending on cache bookkeeping. The honest caveat is visible in the uniform-logit identity above: the abstention gate is absolute but its threshold still drifts as $-\log L$ , so pushing a head to stay quiet across a 256x context increase costs about 5.5 nats of logit headroom, which is precisely why deployed variants learn $s$ per head rather than freezing it at zero.

Line chart of total attention mass on real keys versus a uniform attention logit, for context lengths 512, 8192 and 131072. Each softmax-1 curve is a logistic whose half-way point sits at minus log L, marked by dotted vertical lines, while a flat dash-dotted line at mass one shows that ordinary softmax retains the whole budget regardless of logit level or length.

Figure 2: Softmax-1 converts the normalizer into a logistic gate on the absolute logit level, whereas ordinary softmax pins the mass at one for every logit row and every length. The gate’s half-way point moves left by $\log L$ , so a head that abstains comfortably at 512 keys leaks over half its budget at 131k unless it lowers its logits by about 5.5 nats, which is the argument for a learned sink logit rather than a hard-coded 1.

The register story is the same pressure seen in vision. Darcet et al. found that DINOv2, CLIP, and DeiT-III spontaneously repurpose a small fraction of low-information background patches as global scratch space, giving those patches norms roughly an order of magnitude above their neighbours and visibly corrupting attention maps and dense-prediction features; adding explicit register tokens removed the artifacts. Registers supply somewhere to write, while Softmax-1 supplies permission not to write, and the two are complementary rather than competing. Bondarenko et al. reached the identical diagnosis from the quantization side, naming the problem “helping attention heads do nothing” and reporting that suppressing the outliers is what makes per-tensor INT8 activation quantization viable. Reported perplexity gains from the off-by-one change alone are small and setup-dependent, so the case for it rests on outlier suppression and long-context robustness, not on loss curves.

Property	Ordinary softmax	Softmax-1	Learned sink logit	Register tokens
How a head does nothing	Dumps mass on a learned sink token and shrinks its value contribution	Pushes every logit below zero and abstains	Same, with the abstention threshold trained per head	Attends to a dedicated token reserved for scratch space
Weights sum to	Exactly 1	$S/(1+S)$ , strictly below 1	$S/(e^{s}+S)$ , strictly below 1	Exactly 1, including the register slots
Added cost	None	One constant in the denominator, zero parameters	One scalar per head	Extra sequence positions, so extra KV and quadratic prefill
Massive activations	Emerge reliably; block per-tensor INT8 activation quantization	Strongly reduced when trained from scratch	Reduced, and the head can tune how quiet it stays	Confined to registers instead of random patches, not removed
Sliding window safety	Fragile: evicting the sink token collapses perplexity	Safe: the null option is architectural, nothing to pin	Safe, and adapts to the window length during training	Safe only if the registers are never evicted
Retrofit to a trained model	Baseline	Needs continued pretraining; a drop-in swap breaks calibration	Same, though $s$ can be initialized from observed sink mass	Requires retraining or at least adapter tuning of the new tokens

August 19, 2026

DL0168 NoPE: Causal Positional Bias Without Encodings

What is NoPE (No Positional Embeddings), and how does a causal mask alone induce implicit positional bias in decoder-only Transformers without explicit position encodings?

Answer

NoPE is a decoder-only Transformer trained with no position information injected anywhere: no learned or sinusoidal absolute embedding added to the token embeddings, no RoPE rotation inside attention, and no ALiBi distance bias on the logits. The only order-dependent structure left in the network is the causal mask, and that turns out to be enough. A bidirectional self-attention layer without positional encodings is permutation equivariant, so it can only represent a bag of tokens, but causal masking destroys that symmetry because query $i$ attends over exactly $i+1$ keys. The size of each token’s receptive field is therefore itself a strictly monotone function of absolute position, and any head with near-uniform logits converts that count into a readable $1/(i+1)$ signal in the residual stream. Haviv et al. showed that absolute position can be linearly probed out of NoPE hidden states with high accuracy while perplexity stays close to models with explicit encodings, and Kazemnejad et al. proved constructively that a NoPE decoder can represent both absolute and relative position, and that it length-generalizes at least as well as RoPE or ALiBi on small-scale algorithmic tasks.

(1) Definition: NoPE removes every explicit position term and keeps only the lower-triangular attention mask, so position is an emergent property rather than an injected feature.
(2) Broken Permutation Equivariance: without a mask, $f(PX) = P f(X)$ for any permutation $P$ , but the causal mask satisfies $P^\top M P \neq M$ , which is exactly the asymmetry the model exploits.
(3) Counting Is The Mechanism: uniform attention over an $(i+1)$ -token window puts $1/(i+1)$ mass on each visible key, so the output norm encodes absolute position.
(4) Absolute First, Relative Later: layer 1 materializes an absolute code, usually anchored on a dominant BOS sink, and deeper layers subtract two codes to obtain the relative offset $i-j$ .
(5) Resolution Decays Quadratically: the gap between adjacent positions is $O(i^{-2})$ , so the counting code loses discriminative power long before the context window ends.
(6) Implicit Recency Bias: trained NoPE heads develop distance-decaying attention that resembles a learned relative encoding, but with no principled extrapolation knob, which is why long-context NoPE needs attention-temperature scaling or hybrid layers.

The mechanism is easiest to read as counting. Row $i$ of the causal attention matrix has exactly $i+1$ unmasked entries, so a head whose logits are roughly constant across its window spreads $1/(i+1)$ of the probability mass onto each visible key. If the value vectors are dominated by one distinguished token, in practice the BOS token that every position can see, then the output norm at position $i$ is proportional to $1/(i+1)$ : a strictly decreasing, invertible function of absolute position that the next layer can consume as a positional feature. Kazemnejad et al. turn this observation into a theorem, and once an absolute code sits in the residual stream a later layer’s $q_i^\top k_j$ term can compute $i-j$ , recovering relative position too. Real NoPE models show the fingerprints of this construction: very heavy mass on the first token, first-layer heads with near-flat windows, and probes that decode absolute position from early activations.

Three panels: a ten by ten causal attention mask heatmap whose row i has i+1 shaded cells with the visible-key count printed beside each row, the same size bidirectional mask where every row sees all ten keys, and a line chart showing that uniform causal attention yields an output norm of one over i plus one that decreases strictly with position while bidirectional uniform attention yields a flat one over n line carrying no positional information

Figure 1: The causal mask is the position signal. Counting unmasked entries per row gives $1{,}2{,}\ldots{,}n$ under causal masking but a constant $n$ under bidirectional attention, so the same uniform head produces a strictly decreasing code in one case and a flat, uninformative constant in the other.

Mathematical Formulation:
$\alpha_{ij} = \frac{\exp(q_i^\top k_j)}{\sum_{m=0}^{i} \exp(q_i^\top k_m)}$
$\alpha_{ij} = 0 \ \text{for}\ j > i$
$q_i^\top k_j = c \Rightarrow \alpha_{ij} = \tfrac{1}{i+1}$
$o_i = \sum_{j=0}^{i} \alpha_{ij} v_j = \frac{v_0}{i+1}$
$i = \|o_i\|^{-1} - 1$
$f_{\mathrm{bi}}(PX) = P f_{\mathrm{bi}}(X)$
$P^\top M P \neq M$

Where:

$o_i \in \mathbb{R}^{d}$ is the attention output at query position $i$ , and $\alpha_{ij}$ the attention weight it places on key position $j$ .
$q_i, k_j, v_j$ are the query, key, and value vectors, containing no positional term at all under NoPE.
$j, m \in \{0,\ldots,i\}$ index only the unmasked keys; the second line is the causal mask, which is the sole source of order sensitivity.
$c$ is a constant logit, the degenerate case that makes attention uniform over the window; the third and fourth lines assume $v_j = 0$ for $j \geq 1$ and a nonzero anchor value $v_0$ at BOS.
The fifth line inverts the code with $\|v_0\| = 1$ , showing that absolute position is exactly recoverable from a single layer’s output norm.
$P$ is a permutation matrix, $X$ the input sequence, $f_{\mathrm{bi}}$ an unmasked layer, and $M$ the lower-triangular mask; the last two lines state why encoders need explicit position encodings and causal decoders do not.

The same construction explains why NoPE is fragile at long context. Adjacent positions are separated by $\Delta_i = 1/((i+1)(i+2))$ , a relative spacing of only $1/(i+2)$ , so the code that cleanly distinguishes position $i$ from position $i+1$ must resolve relative differences near $8\times10^{-3}$ around token 126 and near $10^{-3}$ around token 1022. Under bf16 activations those differences sit at the edge of machine epsilon, and the second pressure is entropic: uniform attention over a growing window has entropy $H_i \leq \ln(i+1)$ , so a head calibrated to be discriminative at 2K tokens is comparatively diffuse at 32K. Real models do not use the naive counting head alone, but both effects push in the same direction, which is why raw NoPE degrades past its training length while RoPE offers explicit rescaling recipes such as NTK interpolation and YaRN.

Two panels: a log-log plot of the relative gap one over i plus two between adjacent positional codes falling below bf16 machine epsilon around position 126 and below fp16 machine epsilon around position 1022, and a semi-log plot of the maximum attention entropy log of i plus one growing from about 7.6 nats at 2048 tokens to about 10.4 nats at 32768 tokens

Figure 2: Two length pressures on an implicit code. The relative spacing between neighbouring positions shrinks like $1/(i+2)$ , crossing bf16 machine epsilon in the low hundreds of tokens, while the entropy ceiling grows like $\ln(i+1)$ , so the same attention temperature that is sharp at 2K is diffuse at 32K.

Property	NoPE	RoPE	ALiBi
Source of position	Causal mask only; window size acts as an implicit counter	Rotation of queries and keys by angle proportional to index	Fixed per-head linear penalty on the distance i minus j
Works without a mask	No; a bidirectional layer becomes permutation equivariant	Yes; encoders use it directly	Yes, with a symmetric distance bias
Extension knob	Attention temperature or entropy scaling; no standard recipe	Base frequency rescaling, NTK interpolation, YaRN	Slope schedule; extrapolates but truncates effective range
Extra cost	None; consumes capacity and layer-1 heads instead	Two elementwise ops per head; cached keys are pre-rotated	One additive bias term per attention score
Dominant failure mode	Code resolution and softmax sharpness collapse past training length	Unseen rotation phases out of distribution beyond training length	Strong recency bias suppresses genuine long-range retrieval

August 19, 2026

DL0167 Video Generation Physical Consistency Evaluation

How do you evaluate physical consistency (gravity, collision, object permanence, and temporal coherence) in video and world generation models beyond standard distributional metrics like FID and FVD?

Answer

FID and FVD compare summary statistics of clip embeddings from appearance-biased backbones (InceptionV3, I3D), so they answer whether a set of generated videos resembles a reference set, never whether one clip obeys physics. They are dominated by per-frame content rather than motion, they are nearly insensitive to short-range frame reordering, and they can improve while dynamics get worse. Physical consistency therefore needs per-clip, falsifiable probes. The three that work in practice are: condition the model on the opening frames of a real recorded event and score the predicted continuation against the actual future; ask a calibrated judge (human, or a VLM validated against human labels) for a binary physical-commonsense verdict on each clip; and extract a symbolic readout (track → lift to 3D → fit dynamics) so gravity, momentum, permanence, and flow coherence become numbers with known correct values. For a world model you add a fourth layer, closed-loop rollout, because the only decision-relevant question is whether a policy trained or planned inside the model still succeeds in the real environment.

(1) Distributional vs Per-Clip: FVD is a set-level Frechet distance and cannot label a single video as physically wrong, so it belongs in a regression suite, not in a physics report.
(2) Real-Continuation Probes: give the model the first frames of a genuine recorded event and compare the continuation to ground truth with spatial IoU of the motion mask and masked pixel MSE.
(3) Calibrated Judges: report physical commonsense (PC) and semantic adherence (SA) separately plus their joint pass rate, and publish the judge’s agreement with humans before trusting it.
(4) Symbolic Readout: tracking plus depth lifting turns physics into scalars: fitted gravity residual, momentum error at collisions, occlusion recovery rate, and optical-flow warp error.
(5) Object Permanence Is A Timeline Test: it is only measurable across an occlusion window, so probes must be authored with an occluder and scored on identity, count, and attribute preservation after reappearance.
(6) Closed-Loop For World Models: action-conditioned rollouts are scored by drift over horizon and downstream task success, which is the metric that actually correlates with usefulness.

Diagram of a four-tier evaluation stack: a generated clip with its conditioning fans out to tier one distributional FID and FVD over I3D features, tier two real-continuation probe comparing to a recorded future, tier three calibrated human or VLM judge giving a binary per-clip verdict, and tier four symbolic readout that tracks objects and fits dynamics; each tier emits its own metrics and tiers two to four feed a per-clip physics scorecard while tier one connects with a dashed arrow

Figure 1: The evaluation stack. Only tiers 2 to 4 return a verdict for an individual clip; tier 1 is a distribution-level sanity check whose score can fall (improve) while gravity and collision behaviour degrade, which is exactly why it is drawn with a dashed contribution.

Each of the four physical axes needs a different probe design. Gravity is measured by tracking a free-falling centroid, fitting a quadratic to its height, and comparing the implied acceleration to the reference: generated video overwhelmingly fails toward floaty, slow-motion dynamics with a fitted value well below the true one. Collision is measured at the impact frame, where you check momentum and energy: a coefficient of restitution above 1, interpenetration, or objects passing through each other are hard violations that no appearance metric sees. Object permanence requires a prompt authored with an occluder, then a check that the object reappears with the same identity, count, and colour; the natural scores are the recovery rate and identity-switch count borrowed from multi-object tracking. Temporal coherence is the cheapest to automate: warp consecutive frames with optical flow, mask out occlusions and disocclusions, and average the residual, which catches flicker, texture swimming, and morphing that a per-frame FID is blind to. Crucially, all four scores are reported as per-clip pass rates over a curated probe set, not as an average of a continuous score, because averaging hides the fact that most failures are categorical.

Mathematical Formulation:
$d_F^2 = \| \mu_r - \mu_g \|_2^2 + T_{\Sigma}$
$T_{\Sigma} = \mathrm{Tr}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{1/2})$
$y(t) = y_0 + v_0 t + \frac{1}{2} \hat{g} t^2$
$e_g = | \hat{g} - g | / g$
$e_p = \| p^{+} - p^{-} \| / \| p^{-} \|$
$R_p = N_{\mathrm{rec}} / N_{\mathrm{occ}}$
$\tilde{I}_t = \mathcal{W}(I_t, f_{t \to t+1})$
$\epsilon_t = \| m_t \odot (I_{t+1} - \tilde{I}_t) \|_1$
$E_w = \frac{1}{T-1} \sum_{t=1}^{T-1} \epsilon_t$
$J = \frac{1}{N} \sum_{i=1}^{N} c_i^{\mathrm{pc}} c_i^{\mathrm{sa}}$

Where:

$d_F^2$ is the FVD value, with $\mu_r, \Sigma_r$ and $\mu_g, \Sigma_g$ the mean and covariance of real and generated clip features from a fixed video backbone; it is defined only over sets, which is the root of its uselessness for per-clip physics.
$y(t)$ is the tracked height of an object at time $t$ , $y_0$ and $v_0$ the fitted initial height and velocity, and $\hat{g}$ the least-squares acceleration recovered from the generated clip; $e_g$ is its relative error against the reference $g$ .
$p^{-}$ and $p^{+}$ are total momentum immediately before and after a contact frame, so $e_p$ is the relative momentum violation; the same construction on kinetic energy exposes a restitution above 1.
$N_{\mathrm{occ}}$ is the number of authored occlusion events and $N_{\mathrm{rec}}$ the number where the object reappears with correct identity, count, and attributes, giving the permanence recovery rate $R_p$ .
$\mathcal{W}$ is backward warping by the estimated flow $f_{t \to t+1}$ , $m_t$ the binary validity mask that removes occluded and out-of-frame pixels, $\odot$ elementwise product, and $E_w$ the averaged warp error over the $T$ frames.
$c_i^{\mathrm{pc}}, c_i^{\mathrm{sa}} \in \{0,1\}$ are the physical-commonsense and semantic-adherence verdicts on clip $i$ , so $J$ is the joint pass rate: a clip counts only if it is both on-prompt and physically plausible.

Two-panel chart. Left panel plots tracked height versus time for a reference bouncing ball following ballistic arcs with decaying apex heights and a generated clip that falls more slowly, dips below the floor line, and rebounds to an apex higher than its release height. Right panel plots speed magnitude versus time, showing the reference speed dropping from about 9.9 to 7.4 metres per second at impact while the generated clip's speed increases from about 6.6 to 7.6 metres per second at impact

Figure 2: The symbolic readout on one probe clip. The fitted acceleration is 4.2 instead of 9.8, the object interpenetrates the floor, and the post-impact speed exceeds the pre-impact speed (restitution above 1, so energy is created). Every one of these is a numeric, falsifiable failure; all of them are invisible to a distributional metric computed on the same clip.

Metric family	Granularity	What it catches	What it misses
FID / FVD	Set level, needs hundreds of clips	Gross artefacts, blur, mode collapse, distribution shift	All per-clip physics; content bias makes it track texture more than motion
Real-continuation probe	Per clip, against recorded ground truth	Wrong outcome of a near-deterministic event (where a ball lands, whether paint mixes)	Legitimate stochastic futures are penalised unless you score a sampled set
Calibrated human or VLM judge	Per clip, binary PC and SA	Open-ended violations on arbitrary prompts, including material and state-change errors	Sub-second and geometric violations; judge agreement with humans is far from perfect
Symbolic trajectory readout	Per clip, per object, per frame	Gravity residual, momentum and energy errors, interpenetration, identity switches, warp error	Anything the tracker or depth model cannot follow: fluids, smoke, cloth, heavy occlusion
Closed-loop rollout	Per episode, action-conditioned	Compounding drift and any error that actually changes a policy’s decision	Expensive, needs an action-conditioned model plus a real or simulated environment

August 19, 2026

DL0166 3D Gaussian and NeRF Neural Simulator

How do 3D Gaussians and Neural Radiance Fields serve as dense physical representations inside differentiable neural simulators?

Answer

A differentiable neural simulator needs one scene description that works twice: as geometry the solver can push around, and as an image the photometric loss can compare against real video. NeRF supplies the second half well, storing the scene as an implicit field that maps a position and view direction to density and radiance, rendered by differentiable volume quadrature along rays. What it lacks is handles, because an MLP field has no particle to give mass, no covariance to advect, and no obvious place to attach a constitutive model. 3D Gaussian Splatting removes that obstacle by storing the scene as an explicit set of anisotropic Gaussians with mean, covariance, opacity, and spherical-harmonic color, so every primitive doubles as a material point in a Lagrangian solver. The simulator advances those points with differentiable physics, usually MLS-MPM, the rasterizer splats them back to pixels, and gradients from the rendering loss flow through both the rollout and the renderer into material parameters such as Young’s modulus, Poisson ratio, yield stress, and friction. The loop is therefore video → physical parameters → rollout of unseen dynamics, with pixels as the only supervision.

(1) Dense Means Physics Everywhere: unlike a mesh, both representations describe the interior volume, so mass, velocity, and stress are defined at every sampled point rather than only on a surface shell.
(2) Gaussians Are Already Particles: a splat’s mean is a particle position and its covariance is a local volume element, which is exactly the state an MPM particle carries, so the reconstruction and the simulation share one data structure.
(3) NeRF Needs A Bridge: a continuous field has to be voxelized or resampled onto particles each step, the Eulerian-Lagrangian conversion that PAC-NeRF introduced, which costs an extra transfer and an extra source of gradient noise.
(4) Covariance Follows The Deformation Gradient: kinematics are preserved by advecting the mean through the flow map and conjugating the covariance with $F$ , otherwise stretched material still renders as undeformed blobs.
(5) The Loss Is Purely Photometric: no ground-truth stiffness is available, so system identification is driven by rendered-versus-observed pixel error over a multi-frame rollout.
(6) The Cost Lives In The Backward Pass: memory grows as $O(TN)$ in substeps and particles, and gradients through contact and friction are non-smooth, so checkpointing and softened contact models are mandatory.

Left-to-right pipeline: multi-view video with camera poses feeds a dense reconstruction stage producing 3D Gaussians or a NeRF field, which becomes a physical particle state with mass, velocity and deformation gradient, advanced by a differentiable MPM simulator, rendered by a differentiable renderer and compared to the observed frames, with a dashed backward gradient path into a material parameter block that feeds the simulator

Figure 1: The identification loop. The dense representation appears twice in the graph, once as the initial physical state and once as the thing being rendered, which is why gradients from a single photometric loss can reach material parameters that were never observed directly.

The reason 3D Gaussians displaced radiance fields in this role is mechanical rather than aesthetic. In a Lagrangian solver the unknowns are particle positions and their local deformation, and a Gaussian already stores a mean plus a covariance factored as scale and rotation, so PhysGaussian can drive splats directly with a continuum solver and keep rendering in real time. A radiance field, by contrast, is a function of space with no identity attached to any location, so tracking material requires either a learned deformation field mapping observation time back to a canonical frame, or a particle proxy that carries the physics while the field carries appearance. Two consequences matter in practice. First, the spherical-harmonic coefficients are expressed in world coordinates, so when a splat rotates by $R$ its appearance basis must be rotated too, otherwise highlights stay welded to the world frame while the geometry turns. Second, lighting is baked, so shadows and specular reflections do not respond to motion, which biases the photometric gradient exactly on the frames where an object moves most.

Two panels of ellipse grids: the left panel shows a three by three grid of identical circular Gaussians in the rest state, the right panel shows the same grid after a shear and vertical compression, where each ellipse is stretched and tilted consistently with the displaced particle centers, connected by an arrow labelled F

Figure 2: Kinematics of a splat. The mean rides the flow map while the covariance is conjugated by the deformation gradient, so shear and compression change both where a Gaussian sits and how it is shaped; skipping the covariance update leaves visibly isotropic blobs inside sheared material and corrupts the rendering gradient.

Mathematical Formulation:
$\alpha_i = 1 - \exp(-\sigma_i \delta_i)$
$T_i = \prod_{j=1}^{i-1}(1 - \alpha_j)$
$C(r) = \sum_{i=1}^{N} T_i \alpha_i c_i$
$\Sigma = R S S^\top R^\top$
$\mu_p^{t} = \phi_t(\mu_p^{0})$
$\Sigma_p^{t} = F_p^{t}\,\Sigma_p^{0}\,(F_p^{t})^\top$
$s_{t+1} = \mathcal{S}(s_t;\theta)$
$\mathcal{L} = \sum_t \lVert \mathcal{R}(s_t) - I_t \rVert^2$
$\nabla_{\theta}\mathcal{L} = \sum_t \frac{\partial \mathcal{L}}{\partial s_t}\frac{\partial s_t}{\partial \theta}$

Where:

$C(r)$ is the rendered color of ray $r$ , accumulated over $N$ ordered samples or splats with per-sample color $c_i$ ; this same alpha-compositing form covers NeRF quadrature and Gaussian splatting.
$\sigma_i$ is density, $\delta_i$ the sample spacing along the ray, $\alpha_i$ the resulting opacity, and $T_i$ the transmittance reaching sample $i$ .
$\Sigma$ is a splat’s covariance, parameterized by rotation $R$ and diagonal scale $S$ so that it stays positive semi-definite under gradient updates.
$p$ indexes particles, $\phi_t$ is the flow map from rest to time $t$ , and $F_p^{t} = \partial \phi_t / \partial \mu$ is the deformation gradient whose determinant gives local volume change.
$s_t$ is the full simulator state (positions, velocities, $F$ , affine momentum) and $\mathcal{S}$ one differentiable substep.
$\theta$ collects the unknown physical parameters such as Young’s modulus $E$ , Poisson ratio $\nu$ , density $\rho$ , yield stress, and friction coefficient.
$\mathcal{R}$ is the differentiable renderer and $I_t$ the observed frame, so $\mathcal{L}$ is a multi-frame photometric residual and its gradient is a product of per-substep Jacobians.

That last product is where the engineering happens. Explicit MPM needs a CFL-limited timestep, so one video frame at 30 fps typically hides a few hundred substeps, and reverse-mode differentiation over a one-second clip means backpropagating through several thousand Jacobians. Storing every intermediate state is what kills the run: 200k particles at roughly 96 bytes of state per particle is about 19 MB per substep, so a 1,000-substep rollout needs 19 GB of tape. The standard remedies are gradient checkpointing at an interval near $\sqrt{T}$ , truncating the loss window to a handful of frames, and recomputing forward segments during the backward pass. Numerically, the same product of Jacobians makes gradients explode for stiff materials and vanish once the trajectory has been dissipated by friction, which is why practitioners fit a log-parameterized stiffness and soften contact before trusting any gradient at all.

Log-log chart of backward-pass memory in bytes versus number of simulator substeps for 200 thousand particles, comparing storing the full trajectory which grows linearly and reaches 40 gigabytes near 2000 substeps, against square-root gradient checkpointing which stays under 3 gigabytes, with a dash-dotted horizontal line marking a 40 gigabyte HBM budget

Figure 3: Why differentiable rollouts are short. At 19 MB of particle state per substep, the naive tape exhausts a 40 GB HBM budget after roughly 2,000 substeps, about one second of simulated time, while square-root checkpointing keeps memory in the low gigabytes at the price of one extra forward pass.

Property	NeRF (implicit field)	3D Gaussians (explicit primitives)
Primitive	MLP or hash grid queried at continuous positions; no persistent identity	Set of anisotropic Gaussians, each with mean, covariance, opacity, SH color
Coupling to solver	Needs a deformation field or an Eulerian-Lagrangian resampling step onto particles	Splat is used directly as an MPM material point, no conversion layer
Render cost per frame	Hundreds of network queries per ray; typically tens of ms to seconds	Tile-based rasterization, real time at 1080p on one GPU
Handles topology change	Naturally, since density is just a field value that can appear or vanish	Poorly; fracture needs splitting or spawning new Gaussians outside the solver
Dominant failure mode	Slow rollouts, floaters in unobserved regions, noisy gradients through resampling	Baked lighting and unrotated SH, plus spiky splats that behave like bad particles

August 19, 2026

DL0165 End-to-End Driving Trajectory Scoring

How do end-to-end autonomous driving architectures combine world model generation with trajectory scoring and cost function evaluation to select the safest driving path?

Answer

Competitive end-to-end stacks do not regress one trajectory and drive it. They propose many candidates, imagine the future under each one, and then take the argmax of an explicit cost, which turns planning into a ranking problem over a finite action set. The world model is what makes the ranking meaningful: it is an ego-conditioned forecaster that answers a counterfactual question, namely what the agents, the occupancy grid, or the camera views would look like if this particular trajectory were executed. The cost function then reads that imagined rollout and produces bounded sub-metrics for collision, drivable-area compliance, time-to-collision, comfort, and progress, which are combined so that hard safety terms multiply and soft preference terms are averaged. Because rolling out a simulator for thousands of candidates is far too slow at a 10 Hz replanning rate, production designs run the expensive scorer offline as a teacher and distill it into a small scoring head that evaluates the whole vocabulary in one batched forward pass.

(1) Proposal Set Instead Of Regression: a fixed trajectory vocabulary (8192 clustered 4-second trajectories in Hydra-MDP, 4096 in VADv2) or a small set of diffusion anchors preserves multi-modality and gives the cost function something concrete to compare.
(2) Ego-Conditioned World Model: the rollout must be conditioned on the candidate action, otherwise every candidate shares one predicted future and the scores carry no causal information about the ego’s own choice.
(3) Multiplicative Gates Plus Weighted Soft Terms: collision and off-road terms enter as multiplicative factors so a single violation zeroes the score, while time-to-collision, comfort, and progress enter as a weighted average that only re-ranks the survivors.
(4) Simulator As Teacher, Head As Student: a rule-based closed-loop metric computed with LQR tracking on a bicycle model labels every candidate offline, and a cross-attention scoring head learns those labels, converting a multi-second simulation into a millisecond-scale inference.
(5) Reactivity Decides Validity: non-reactive rollouts hold other agents on their logged paths, which systematically over-rewards cutting in and under-penalizes aggressive gap acceptance.
(6) Latency Sets The Design: a 4-second horizon at 10 Hz is 40 poses per candidate and the whole loop must close inside about 100 ms, which rules out per-candidate video generation online.

Left-to-right pipeline diagram: multi-view camera BEV scene encoder, then a proposal generator producing a vocabulary of 8192 four-second trajectories, then an ego-conditioned world model rollout predicting agent futures and occupancy for each candidate, then metric heads for no at-fault collision, drivable area compliance, time-to-collision, comfort and progress, then score aggregation and argmax selecting one trajectory, with an offline rule-based simulator below feeding distillation targets into the metric heads

Figure 1: One scene encoder, many candidates, one score. The world model rollout is evaluated per candidate, and the metric heads are trained against an offline rule-based simulator, so the simulator’s judgement is available at inference without running it.

The world model shows up in three distinct roles, and confusing them is a common interview mistake. In the in-loop role, a generative model such as Drive-WM synthesizes multi-view future frames conditioned on each candidate and an image-space reward built from map and object cues picks the branch, which is expressive but so expensive that only a shallow tree of a few branches is affordable. In the representation role, future-occupancy or future-agent forecasting is an auxiliary head that shapes the BEV latent, so cost terms are computed on predicted geometry rather than pixels, which is the practical choice inside a 100 ms budget. In the offline role, high-fidelity generators supply rare scenarios and closed-loop training environments, so the cost function is validated against near-misses that almost never appear in logged data. Whatever the role, the failure mode is the same: if the rollout ignores that other agents respond to the ego, the cost function rewards trajectories that would have been vetoed by any reactive simulator.

Bird's-eye-view schematic of a two-lane road with an ego vehicle, a slow lead vehicle ahead in the same lane, and a vehicle in the left lane. Four candidate trajectories are drawn: an overtake into the left lane that collides with the left-lane vehicle and scores zero, a swerve onto the non-drivable shoulder that also scores zero, a full-stop brake that stays legal but has near-zero progress and scores 0.48, and a lane-keeping follow trajectory that scores 0.92 and is selected

Figure 2: Why the aggregation is multiplicative rather than additive. Candidates A and B are gated to exactly zero by collision and drivable-area violations no matter how much progress they buy, and only then do the soft terms separate the timid full stop from the trajectory that actually makes progress.

Mathematical Formulation:
$\hat{s}_{t+1:t+H} \sim p_{\phi}(\cdot \mid o_{\leq t}, \tau)$
$f_m(\tau) = g_m(\hat{s}_{t+1:t+H})$
$P(\tau) = \prod_{m \in \mathcal{M}_g} f_m(\tau)$
$Q(\tau) = \frac{\sum_m w_m f_m(\tau)}{\sum_m w_m}$
$S(\tau) = P(\tau) \cdot Q(\tau)$
$\tau^{*} = \arg\max_{\tau \in \mathcal{V}} S(\tau)$

Where:

$\tau^{*}$ is the executed trajectory and $\mathcal{V}$ the candidate vocabulary, typically a few thousand clustered 4-second trajectories sampled at 10 Hz.
$o_{\leq t}$ is the sensor history, $p_{\phi}$ the ego-conditioned world model, and $\hat{s}_{t+1:t+H}$ the imagined future over horizon $H$ under that specific $\tau$ .
$g_m$ extracts metric $m$ from the rollout and $f_m(\tau)$ is its value, bounded to the unit interval so that scores are comparable across scenes.
$\mathcal{M}_g$ are the gate metrics (no at-fault collision, drivable-area compliance) whose product $P(\tau)$ vetoes a candidate outright.
$w_m$ are the weights of the soft metrics (time-to-collision, comfort, ego progress) aggregated by $Q(\tau)$ , which only re-ranks candidates that passed every gate.

NAVSIM PDM Score Instantiation:
$S = \mathrm{NC} \cdot \mathrm{DAC} \cdot Q$
$Q = (5\,\mathrm{TTC} + 2\,\mathrm{C} + 5\,\mathrm{EP}) / 12$

Here NC and DAC are binary gates, while time-to-collision, comfort, and ego progress carry weights 5, 2, and 5. The weights are the policy: raising the progress weight produces a stack that squeezes through gaps, and raising comfort produces one that refuses to brake late. The distillation objective simply asks the student head to reproduce every teacher sub-metric per candidate.

Distillation And Latency Budget:
$\mathcal{L} = \sum_{\tau} \sum_m \mathrm{BCE}(\hat{f}_m(\tau), f_m^{\mathrm{sim}})$
$t_{perc} + t_{gen} + t_{score} \leq 100\ \mathrm{ms}$
$40\ \mathrm{ms} + 25\ \mathrm{ms} + 15\ \mathrm{ms} = 80\ \mathrm{ms}$

Log-log chart of scoring latency in milliseconds against the number of scored candidate trajectories, with three curves: a distilled learned scoring head that stays nearly flat from 14 to about 32 milliseconds up to 8192 candidates, a rule-based closed-loop simulator that grows linearly at roughly 0.45 milliseconds per candidate, and a generative video world model rollout at roughly 320 milliseconds per candidate, plus a dashed horizontal line marking a 100 millisecond replanning budget

Figure 3: Representative per-candidate costs against the 100 ms replanning budget. A distilled scoring head is almost flat in the number of candidates because scoring is one batched forward pass, a rule-based simulator caps out near a hundred candidates, and a per-candidate video rollout blows the budget on its first branch.

Property	Rule-based simulator in the loop	Generative world model in the loop	Distilled learned scorer
Proposal source	Centerline offsets crossed with target speeds, about 15 proposals	A shallow tree of a few branches per replan	Clustered vocabulary of 4096 to 8192 trajectories, or 20 diffusion anchors
How futures are obtained	Bicycle model plus LQR tracking with map and collision checks	Diffusion rollout of multi-view frames or occupancy conditioned on the action	No explicit rollout at inference, the head predicts each sub-metric directly
Agent reactivity	Configurable, IDM background traffic reacts to the ego	Learned and implicit, quality depends entirely on training data	Inherited from whatever the teacher assumed, usually non-reactive logs
Per-candidate cost	Sub-millisecond but CPU-bound and linear in candidates	Hundreds of milliseconds of GPU time per branch	Microseconds, batched inside one transformer forward pass
Dominant failure mode	Hand-written dynamics and a coarse proposal set miss creative maneuvers	Hallucinated geometry, action leakage, and unusable latency	Copies the teacher’s blind spots and can latch onto ego-status shortcuts

August 19, 2026

DL0164 BEV Transformation: Lift-Splat-Shoot

Explain Bird’s-Eye-View transformation methods such as Lift-Splat-Shoot and transformer cross-attention for mapping multi-view camera video into a unified 3D world representation, as used in camera-only autonomous driving stacks and benchmarked on nuScenes.

Answer

A BEV transformation converts $N_{c}$ perspective images, each of which has thrown away the depth of every pixel, into a single metric grid in the ego frame where one cell always means the same physical patch of ground. Every method must invent the missing depth, and the two families differ only in which direction they move information. Forward projection, introduced by Lift-Splat-Shoot (LSS), predicts a categorical depth distribution per pixel, lifts each pixel into a frustum of $D$ candidate 3D points weighted by that distribution, then splats the points into BEV pillars with sum pooling. Backward projection, popularised by BEVFormer, starts from a fixed set of learned BEV queries, projects each query’s 3D anchor points into every camera using the known intrinsics and extrinsics, and pulls features back with deformable cross-attention so no explicit depth prediction is required. Both produce the identical output contract, typically a $200 \times 200 \times 256$ feature map at roughly 0.5 m resolution, which is why detection, map segmentation, occupancy and planning heads can be shared, and both add temporal fusion by warping the previous BEV feature into the current ego frame before merging.

(1) The Core Difficulty Is Depth, Not Geometry: the pixel-to-ray mapping $K^{-1}$ and the camera-to-ego rigid transform are exactly known, so the only unknown is the scalar range along each ray.
(2) Forward Projection (Push): LSS predicts $\alpha_{u,v,d}$ over $D$ discrete depth bins, takes an outer product with the context feature, and voxel-pools the resulting frustum point cloud into pillars.
(3) Backward Projection (Pull): BEV queries carry their own 3D position, project into the cameras that actually see them, and sample features with deformable attention, which sidesteps depth estimation entirely.
(4) Shared Output Contract: both write into the same ego-frame grid, so the transformation is a swappable module rather than an architecture commitment.
(5) Temporal Recurrence Is Not Optional: warping $B_{t-1}$ by the ego pose delta and fusing it into $B_{t}$ is what makes velocity estimation and short-occlusion memory possible from cameras alone.
(6) Depth Supervision Decides Accuracy: BEVDepth showed that supervising $\alpha$ with LiDAR-projected depth, rather than letting the detection loss shape it, is the single largest quality lever for the forward family.

Pipeline diagram: six surround cameras feed a shared 2D backbone with FPN, which splits into a top lane predicting a per-pixel depth distribution over 59 bins followed by an outer product and voxel pooling of roughly one million frustum points, and a bottom lane of learned 200 by 200 BEV queries whose 3D anchors are projected into the hit cameras for deformable cross-attention over image features used as keys and values; both lanes write into one 200 by 200 by 256 unified BEV feature map that feeds detection, map and occupancy heads

Figure 1: Two directions, one destination. The push lane commits to a depth distribution and scatters features outward; the pull lane keeps the grid fixed and gathers features inward. Everything downstream of the unified BEV feature map is identical, which is why these modules are interchangeable in practice.

The forward path is best understood as a soft, differentiable version of unprojecting a depth map. A pixel with context feature $c_{u,v} \in \mathbb{R}^{C}$ does not pick one depth; it spreads that feature over all $D$ bins in proportion to $\alpha_{u,v,d}$ , so a confident pixel deposits nearly all of its mass in one pillar while an ambiguous pixel smears a faint trail along its ray. Because the splat is a sum, the operation is permutation-invariant and handles overlapping camera fields of view for free, and because it is differentiable, gradients reach the depth head through the pooling. The engineering cost is the frustum point count, which is why production implementations replace the naive scatter with a sorted cumulative-sum pooling kernel or a preallocated BEVPoolv2 index table that skips materialising the point cloud at all.

Three panels: a grid of image pixels with one highlighted pixel carrying a 256-dimensional context feature; a bar chart of the predicted probability over 59 depth bins showing a sharp peak near 18 metres for a confident pixel and a broad dashed curve for an ambiguous pixel; and a top-down bird's-eye-view grid with the camera at the origin, a ray fanning outward, circles along the ray whose size is proportional to the depth probability, and one highlighted 3 by 3 metre pillar where the points are sum-pooled

Figure 2: One pixel becomes $D$ weighted 3D points. The width of the depth distribution is literally the width of the smear in BEV, so a flat $\alpha$ over a textureless road or a night-time scene produces a long low-confidence streak instead of a localised object.

Forward Projection (Lift-Splat-Shoot):
$\tilde{u} = (u, v, 1)^{T}$
$p_{c} = d\, K^{-1} \tilde{u}$
$p_{e} = R\, p_{c} + t$
$F_{u,v,d} = \alpha_{u,v,d}\, c_{u,v}$
$\sum_{d=1}^{D} \alpha_{u,v,d} = 1$
$B(x,y) = \sum_{p \in \Pi(x,y)} F(p)$

Where:

$B(x,y) \in \mathbb{R}^{C}$ is the BEV feature at grid cell $(x,y)$ in the ego frame, and $\Pi(x,y)$ is the set of frustum points whose ego coordinates fall inside that pillar.
$\tilde{u}$ is the homogeneous pixel coordinate, $K$ the camera intrinsic matrix, and $(R, t)$ the camera-to-ego extrinsic rotation and translation.
$d$ indexes the depth bins, with $d \in \{1, \ldots, D\}$ over a fixed range such as 1 m to 60 m in 1 m steps, giving $D = 59$ .
$\alpha_{u,v,d}$ is the softmax depth distribution for pixel $(u,v)$ and $c_{u,v}$ its context feature, so $F_{u,v,d}$ is the outer-product lift.
The splat is sum pooling, which keeps the operation order-free across cameras and differentiable with respect to both $\alpha$ and $c$ .

The backward path inverts the flow of information. A query at grid cell $(x,y)$ is lifted to $N_{z}$ anchor heights along a vertical pillar, each anchor is projected into every camera, and only the cameras whose image plane actually contains the projection contribute. Deformable attention then samples a handful of learned offsets around each projected location, so cost scales with the number of queries rather than with image resolution times depth bins, and a query near a lane boundary can shift its sampling points to where the evidence is instead of trusting a predicted depth. The trade-off is that the geometry is now an attention prior rather than a hard constraint: if extrinsics are wrong, the network can still learn to compensate, which is convenient during training and dangerous during deployment because the failure is silent.

Backward Projection (Cross-Attention):
$q_{xy} = Q(x,y) + \mathrm{PE}(x,y)$
$r_{j} = (x, y, z_{j})$
$\hat{p}_{ij} = \pi_{i}(r_{j})$
$A_{i} = \sum_{j=1}^{N_{z}} \mathrm{DA}(q_{xy}, \hat{p}_{ij}, F_{i})$
$\mathrm{CA}(q_{xy}) = \frac{1}{|V_{xy}|} \sum_{i \in V_{xy}} A_{i}$

Where:

$\mathrm{CA}(q_{xy})$ is the cross-attention output written into BEV cell $(x,y)$ , and $q_{xy}$ is the learned query plus its 2D positional encoding.
$r_{j}$ is the $j$ -th pillar anchor at height $z_{j}$ , with $N_{z} = 4$ a common choice spanning roughly -5 m to 3 m.
$\pi_{i}$ is the full projection of camera $i$ , so $\hat{p}_{ij}$ is a sub-pixel image location and $F_{i}$ the multi-scale feature map of that camera.
$V_{xy}$ is the set of cameras whose frustum contains at least one anchor, so the average is taken only over hit views and empty views contribute nothing.
$\mathrm{DA}$ is deformable attention, which samples a few learned offsets around $\hat{p}_{ij}$ with bilinear interpolation instead of attending to all pixels.

Frustum Cost At A Typical Configuration:
$N_{pts} = N_{c} \cdot H \cdot W \cdot D$
$6 \times 32 \times 88 \times 59 = 996864$
$200 \times 200 = 40000$

Roughly one million frustum points collapse into forty thousand pillars, an average of about 25 points per cell, and that ratio is exactly why the pooling kernel rather than the backbone is often the latency bottleneck in the forward family. It also exposes the accuracy story: because every point sits on a known ray, a lateral mistake of one pixel at 50 m is only about 4 cm, while a 5% depth mistake at the same range is 2.5 m. BEV error is dominated by range error, and it grows linearly with distance.

Log-scale line chart of bird's-eye-view position error in metres against range from the ego vehicle from 2 to 80 metres, showing straight rising lines for 10 percent, 5 percent and 2 percent relative depth error, and a much lower line for a one-pixel lateral error at focal length 1266 pixels, with a dashed horizontal line at the 2 metre matching threshold and an annotation noting that a 5 percent depth error at 60 metres displaces the box by 3 metres

Figure 3: Range error, not image-plane error, sets BEV quality. A one-pixel lateral error stays under 10 cm across the whole working range, while a modest relative depth error crosses the 2 m matching threshold somewhere between 20 m and 100 m depending on the depth head, which is why depth supervision and long-baseline temporal stereo pay off so heavily.

Property	Forward push (LSS, BEVDet, BEVDepth)	Backward pull (BEVFormer)	Implicit 3D encoding (PETR)
Depth handling	Explicit categorical distribution over D bins, optionally LiDAR-supervised	No depth head; anchors at fixed pillar heights sample every hit view	3D coordinates baked into image position encodings, depth learned implicitly
Dominant cost	Frustum scatter of about 1M points; needs a cumsum or index-table kernel	40,000 queries times layers times sampling points, quadratic in grid side	Global attention over all image tokens, no explicit BEV grid to build
Calibration sensitivity	Hard geometric constraint, so extrinsic drift shifts features into wrong pillars	Attention can partly absorb drift, which hides the fault instead of surfacing it	Most tolerant, but least interpretable when a single camera goes bad
Temporal fusion	Warp and concatenate past BEV grids, or run temporal stereo across frames	Recurrent BEV self-attention on the ego-warped previous grid	Propagate sparse object queries forward in time, no grid to warp
Typical failure	Flat depth distribution smears a distant object along its ray	Empty cells still consume compute, and unseen regions hallucinate from priors	Weak spatial locality makes small distant objects easy to miss

August 19, 2026

DL0163 Map-Based vs Mapless World Models

How do map-based world models conditioned on HD-maps differ from end-to-end mapless world models that predict from raw sensor observations, for a driving stack like Waymo’s or Wayve’s?

Answer

Both families learn a latent transition model that rolls the scene forward under a candidate ego action, so the architectural difference is not the dynamics head but what the dynamics head is allowed to condition on. A map-based world model receives a local HD-map crop $m_t$ , a rasterized or vectorized patch of lane centerlines, stop lines, crosswalks, and traffic-light-to-lane associations, indexed out of a global map by the pose that the localization stack estimates. That crop enters as a hard geometric prior, so the network never has to learn road topology from pixels and can spend capacity on agent behavior. A mapless model sees only the sensor history $o_{t-k:t}$ and must reconstruct that same topology inside its latent on every frame, which is statistically much harder but works on roads nobody has surveyed. The practical result is a swap of error sources rather than a strict improvement: the map-based system inherits map staleness and localization error, the mapless system inherits data hunger and weaker long-horizon geometric consistency.

(1) Conditioning Set: map-based factorizes as $p(z_{t+1} \mid z_t, a_t, m_t)$ with an extra exogenous input, while mapless factorizes as $p(z_{t+1} \mid z_{t-k:t}, a_t)$ and must carry topology in the recurrent state.
(2) Where The Prior Lives: in one system lane geometry is a surveyed asset refreshed by a mapping fleet, in the other it is weights learned from driving video.
(3) Sample Efficiency: handing the model the road graph removes an enormous nuisance factor, so map-conditioned models converge on thousands of hours where mapless generative models are trained on tens of thousands to millions.
(4) Failure Signature: a stale or misaligned map produces confidently wrong rollouts, whereas a mapless model facing an ambiguous road typically produces a wide, visibly uncertain distribution.
(5) Pose Coupling: the map crop is only meaningful in the correct frame, so map-based prediction quality is bounded by localization accuracy; mapless prediction has no pose dependency at all.
(6) Scaling And ODD: map-based scales with surveyed kilometers and re-survey cadence, mapless scales with data and compute, which is why geofenced robotaxi and everywhere-consumer programs made opposite choices.

Two stacked pipelines: the upper map-based pipeline encodes surround camera and LiDAR into a latent state, takes an HD map crop selected by ego pose as an extra conditioning input to the transition model, and decodes BEV occupancy, agent futures, and an ego plan; the lower mapless pipeline tokenizes surround camera video, feeds a spatio-temporal transformer or diffusion prior conditioned only on past tokens and the action, and decodes future frames, an implicit map, and an ego plan

Figure 1: The two stacks share an encoder, a latent transition model, and a decoder. Only one extra edge differs: the HD map crop selected by the estimated pose feeding the transition model. That single edge is what buys sample efficiency and what imports two new failure sources, map staleness and localization drift.

The interesting behavior appears when the map and the world disagree. Because the map is treated as ground truth during training, the model learns to trust it, and at inference it has no mechanism to discount it: a lane that was closed for construction last week is still a valid lane inside $m_t$ , so the imagined rollout drives straight through the closure with low predictive variance. A mapless model never had that crutch, so its estimate of drivable space comes from the same pixels that show the cones. It pays for this with weaker long-horizon geometry, since nothing anchors an eight-second rollout to a globally consistent lane graph and errors compound in the latent. This is also why the honest comparison is not accuracy on a nominal benchmark, where the map-conditioned model almost always wins, but accuracy conditioned on map validity.

Two bird's-eye-view panels of the same two-lane road with a lane closure marked by cones. In the left map-based panel a blue HD map centerline runs straight through the closure and the predicted rollout follows it with a narrow uncertainty band, ending in a red cross at the cones. In the right mapless panel there is no map centerline, the predicted rollout bends into the adjacent lane around the cones, and the uncertainty band is visibly wider.

Figure 2: Same scene, same closure, different conditioning. The map prior keeps the rollout tight and correct whenever the map matches reality, and keeps it tight and wrong when it does not. The mapless rollout is wider but reactive, because the only evidence it ever had for drivable space is the current image.

Mathematical Formulation:
$z_t = E_\theta(o_{t-k:t})$
$m_t = \Pi(M, \hat{T}_t), \quad \hat{T}_t = T_t + \epsilon_{\mathrm{loc}}$
$p_{\mathrm{map}} = p_\theta(z_{t+1} \mid z_t, a_t, m_t)$
$p_{\mathrm{free}} = p_\phi(z_{t+1} \mid z_{t-k:t}, a_t)$
$\mathcal{L} = \sum_{h=1}^{H} \ell(D(z_{t+h}), o_{t+h})$

Where:

$z_t \in \mathbb{R}^{d}$ is the latent scene state produced by encoder $E_\theta$ from the observation window $o_{t-k:t}$ of surround camera, radar, and optionally LiDAR frames.
$M$ is the global HD map, $T_t \in SE(3)$ the true ego pose, $\hat{T}_t$ the estimate produced by localization, and $\Pi$ the operator that crops and transforms $M$ into the ego frame.
$\epsilon_{\mathrm{loc}}$ is localization error. It is not observation noise: it rigidly shifts the entire conditioning input, so a 0.5 m lateral bias moves every lane boundary by 0.5 m.
$a_t$ is the candidate ego action, which is what makes both models action-conditioned and therefore usable for planning rather than passive video prediction.
$\theta$ and $\phi$ are the two parameter sets; $\phi$ must additionally encode road topology that $\theta$ receives for free through $m_t$ .
$D$ is the decoder (future frames, BEV occupancy, or agent boxes), $H$ the rollout horizon in steps, and $\ell$ the per-step loss, typically reconstruction plus a KL or diffusion denoising term.

Schematic line chart of mean lateral rollout error in metres against prediction horizon in seconds, with three curves: a map-based model with a fresh map staying lowest, a mapless model rising moderately faster, and a map-based model with a stale or misaligned map rising steepest and crossing the others within about one second, plus a dashed horizontal line at 0.5 metres marking lane-keeping tolerance

Figure 3: Schematic, but the ordering is the point. With a valid map the conditioned model dominates at every horizon; with a stale or misaligned map the same model becomes the worst of the three within roughly a second, because the prior is applied with full confidence in the wrong place. A mapless model has no best case that good and no worst case that bad.

Property	Map-based (HD-map conditioned)	Mapless (raw sensor)
Conditioning input	Sensor latent plus a pose-indexed vector or raster map crop	Sensor history and the ego action only
What must be learned	Agent behavior and interaction; road topology is given	Topology, drivable space, and behavior, jointly from pixels
Data appetite	Thousands of hours plus a surveyed and maintained map	Tens of thousands of hours upward; Cosmos-class pretraining uses about 20M hours of video
Dominant failure mode	Stale map or localization drift produces low-variance wrong rollouts	Ambiguous or occluded geometry produces drifting, globally inconsistent rollouts
Scaling axis	Surveyed kilometers and re-survey cadence, so expansion is operational	Data and compute, so expansion is a training-run problem
Verifiability	Map is an auditable artifact; behavior can be certified per mapped intersection	Topology lives in weights, so guarantees come only from aggregate evaluation
Traffic-light and rule handling	Light-to-lane association and speed limits come from the map	Association must be inferred visually, a known long-tail weakness

August 19, 2026

DL0162 World Model Object Permanence

How do world models maintain object permanence over long video generation horizons when an object is fully occluded behind another vehicle or building?

Answer

Object permanence is not a property a next-frame predictor gets for free. During a full occlusion the pixels that evidence the object are simply absent, so nothing in the current observation constrains what should re-emerge, and the only thing that can carry the object across the gap is state that outlives the pixels. Production world models therefore rely on three memory mechanisms, usually in combination: a long context of past frame latents that the generator can re-attend to, a recurrent latent state whose prior propagates unobserved objects forward, and an explicit pose-indexed geometric memory (object slots, 3D boxes, or a bird’s-eye-view layout) that is re-projected into the image when the occluder clears. Google DeepMind’s Genie 3 makes the first mechanism explicit, generating 720p frames at 24 fps with a reported visual memory extending roughly one minute into the past, while driving world models in the GAIA line lean on structured conditioning such as other agents’ 3D boxes and the ego trajectory. The practical question in an interview is therefore never “does the model understand permanence” but “does the information survive the occlusion window, and at what cost”, because a 2.5 second occlusion at 24 fps is 60 frames of pure extrapolation and the memory horizon either covers it or the object is re-invented rather than remembered.

(1) Occlusion Is A Missing-Evidence Problem: the observation likelihood contributes nothing about the hidden object, so generation must fall back on the dynamics prior rather than on perception.
(2) Long Context Buys Re-Readable Memory: keeping the last $H$ frames of latents lets attention look back at the pre-occlusion view, at the price of a KV cache linear in $H$ and a quadratic prefill.
(3) Recurrent State Buys Constant Cost: a fixed-size latent $s_t$ carries the object at $O(1)$ memory per step, but its finite capacity makes identity attributes drift instead of vanishing abruptly.
(4) Explicit 3D Memory Buys Correct Re-Projection: storing object records with poses turns re-emergence into geometry, so the vehicle reappears at the right pixel and the right moment.
(5) The Failure Signature Differs Per Mechanism: a hard cliff when $T_{\mathrm{occ}}$ exceeds $H$ , a smooth decay for recurrent state, and mis-registration under ego-pose drift for geometric memory.
(6) Rollout Stability Is A Separate Axis: teacher-forced training with fully autoregressive inference creates exposure bias, so small per-frame errors compound and corrupt identity even when the object is never occluded.

Architecture diagram in which an observation, an action, and a video tokenizer feed three parallel memory blocks: a long-context attention cache over the last H frames, a recurrent latent state whose prior propagates hidden objects, and a pose-indexed 3D or bird's-eye-view memory holding explicit object slots; all three condition a diffusion or autoregressive generator that predicts the next frame, which is appended back into the context as an autoregressive loop

Figure 1: Three places a hidden object can live. The generator is identical in all three cases; what changes is which memory still contains the occluded vehicle at frame $t$ , and therefore whether the model re-renders the same car or invents a new one.

The cleanest way to reason about the recurrent path is the standard latent-dynamics factorization used by Dreamer-style world models. A posterior $q(s_t \mid s_{t-1}, a_t, o_t)$ fuses the new observation, while a prior $p(s_t \mid s_{t-1}, a_t)$ predicts the next state without one. Under full occlusion the object’s contribution to the posterior disappears, so the model is effectively running the prior for $T_{\mathrm{occ}}$ steps on that part of the scene, which is exactly the regime where capacity limits and accumulated error show up. Long-context attention avoids that open-loop integration by re-reading the original evidence, but only if the pre-occlusion frames are still inside the window: once they are evicted, the model has no way to distinguish “a red hatchback was there” from “some vehicle may exist”, and it samples a plausible fresh instance. Explicit memory sidesteps both problems by storing a symbol rather than a distributed code, which is why driving-domain world models condition on agent boxes and map layouts instead of hoping attention rediscovers them.

Line chart of a target vehicle's lateral road position in metres against frame index from 0 to 140, with a shaded band marking a 60-frame full occlusion. The ground-truth trajectory is a dotted grey line; an explicit 3D memory model tracks it closely after the occlusion, a long-context model re-emerges about 1.2 metres off-lane and slowly converges back, and a short-context model produces no object after the occlusion until a spurious new vehicle appears far from the true trajectory

Figure 2: What re-emergence actually looks like. The occlusion window is identical for all three models; the difference appears only after the occluder clears, where the short-context rollout hallucinates a fresh instance in the wrong lane and the long-context rollout keeps the object but mis-registers its position, while pose-indexed memory lands on the true trajectory.

Mathematical Formulation:
$p(x_{1:T} \mid a_{1:T}) = \prod_{t=1}^{T} p(x_t \mid m_t, a_t)$
$m_t = (x_{t-H:t-1},\; s_{t-1},\; \mathcal{M}_{t-1})$
$q_t = q(s_t \mid s_{t-1}, a_t, o_t)$
$p_t = p(s_t \mid s_{t-1}, a_t)$
$T_{\mathrm{occ}} = \Delta t \cdot \mathrm{fps}$
$H \geq T_{\mathrm{occ}}$

Where:

$x_t$ is the generated frame latent at step $t$ and $a_t$ the conditioning action or control (steering, camera pose, text instruction).
$m_t$ is everything the generator may condition on: the context window of past latents, the recurrent state, and the persistent geometric memory.
$H$ is the memory horizon in frames and $N_p$ the number of latent tokens per frame, so the window holds $H N_p$ tokens.
$q_t$ is the posterior that uses the observation $o_t$ and $p_t$ the prior that does not; under full occlusion the object is carried only by $p_t$ .
$\mathcal{M}_t$ is the explicit memory: object records with identity, extent, and pose, plus the camera pose needed to re-project them.
$T_{\mathrm{occ}}$ is the occlusion length in frames, obtained from its duration $\Delta t$ and the frame rate; the last line is the necessary condition for a pure context window to bridge it.
$L$ and $d$ are the number of attention layers and the model width, which set the cache and prefill costs below.

Bridging A 2.5 Second Occlusion At 24 fps:
$T_{\mathrm{occ}} = 2.5 \times 24 = 60$
$H N_p = 60 \times 256 = 15360$
$\mathrm{cache} = O(H N_p L d)$
$\mathrm{prefill} = O((H N_p)^2 d)$
$R(600) = 0.995^{600} \approx 0.05$

Sixty frames of latents at 256 tokens per frame is 15,360 tokens of context bought for a single occlusion, and every additional second of horizon costs another 6,144 tokens of cache plus a quadratic increase in prefill work, which is precisely why real-time interactive world models cap $H$ aggressively and then need a cheaper memory. The last line models the orthogonal failure: if identity is corrupted independently at each step with probability $\delta = 0.005$ , the retention rate $R(T) = (1-\delta)^{T}$ leaves only about 5% of objects intact after a 600-frame (25 second) rollout even with no occlusion at all. Permanence over long horizons is therefore two bugs wearing one coat: a horizon bug that shows up as a cliff, and a drift bug that shows up as exponential decay.

Chart of correct re-emergence rate against occlusion duration in frames from 0 to 240 for four memory mechanisms: a 16-frame context window collapses sharply just after 16 frames, a 96-frame context window stays high then collapses just after 96 frames, a recurrent latent state decays smoothly and geometrically, and a pose-indexed 3D memory stays near 0.9 across the whole range, with vertical markers at the two context horizons and at the 60-frame mark

Figure 3: Two distinct shapes of failure. Finite context gives a step cliff exactly at $T_{\mathrm{occ}} = H$ , recurrent state gives geometric decay from capacity limits and compounding error, and explicit pose-indexed memory is nearly flat because duration costs it almost nothing.

Property	Long-context attention	Recurrent latent state	Explicit 3D / object memory
What survives the gap	Raw pre-occlusion frame latents, re-readable by attention	A compressed distributed code of the whole scene	Symbolic object records: identity, extent, pose, velocity
Cost of a 60-frame occlusion	About 15,360 cached tokens, with quadratic prefill growth	One fixed-size state vector, independent of duration	A few object records plus one camera pose per frame
Hard limit	The horizon itself: nothing outside H frames exists	State dimensionality and compounding prior error	Detector, tracker, and pose accuracy; assumes rigid objects
Dominant failure mode	Cliff: a fresh instance with new colour, size, and lane	Drift: right object class, wrong attributes and timing	Mis-registration under pose drift, or duplicated instances
Typical home	Video diffusion transformers and interactive frame models	Dreamer-style latent world models for control	Driving world models conditioned on agent boxes and layouts

August 19, 2026

DL0161 World Model Safety Edge Cases

How do world models simulate low-probability safety-critical edge cases such as tire blowouts or sudden pedestrian jaywalking for closed-loop planner stress testing?

Answer

A driving world model is an action-conditioned generative simulator $p_\theta(z_{t+1} \mid z_t, a_t, c)$ that rolls forward latent state and decodes multi-view sensor observations, so the planner under test acts inside the rollout instead of replaying a fixed log. Rare events are never obtained by sampling the model unconditionally, because the learned distribution matches the fleet distribution in which a severe collision appears roughly once per $10^{8}$ miles. Instead the scenario is authored through conditioning: a low-dimensional parameter vector $c$ holding occlusion geometry, jaywalk onset time, pedestrian speed, and surface friction is drawn from a biased proposal $q(c)$ fitted by a search loop that maximizes criticality, then each rollout is reweighted by the likelihood ratio $w = p(c)/q(c)$ so the resulting failure-rate estimate remains unbiased. A tire blowout is handled differently from a jaywalk, because it is a vehicle-dynamics discontinuity rather than an appearance change and almost no camera dataset contains it, so production stacks inject it in an analytic dynamics layer (collapsed cornering stiffness on one corner plus a yaw moment) while the world model supplies the surrounding traffic and imagery.

(1) Closed Loop Requires Action Conditioning: the planner’s own control enters the model at every step, so log replay and open-loop video prediction cannot be used, since the ego trajectory diverges from the recorded one within a second or two.
(2) Controllability Comes From Structured Conditioning: text, road layout, 3D agent boxes, and ego action are all conditioning channels, which turns “pedestrian steps out from behind the parked van 0.8 s before ego arrival” into an explicit specification rather than a hopeful prompt.
(3) Rarity Comes From Biased Sampling: importance sampling with a cross-entropy-method proposal concentrates rollouts near the failure boundary, cutting the sample count for a fixed confidence by three to five orders of magnitude.
(4) Unbiasedness Comes From Likelihood Ratios: reporting the raw failure fraction under $q$ overstates real-world risk by exactly the sampling bias, so every rollout carries its weight $w_i$ into the estimator.
(5) Hybrid Physics For Fault Injection: blowouts, brake fade, and actuator latency live in a validated dynamics model, and the world model consumes the resulting ego state as a conditioning signal.
(6) Plausibility Must Be Constrained: unconstrained adversarial search produces kinematically impossible agents and physically unavoidable collisions, so scenarios are filtered by a likelihood floor on $\log p(c)$ and by an avoidability check against a reference planner.
(7) Rollout Stability Bounds The Horizon: autoregressive generation accumulates drift, so useful stress tests run 10 s to 30 s at 10 Hz rather than minutes, and geometric consistency across views is monitored as a validity metric.

Diagram of a closed-loop stress-testing system: real driving logs pretrain a generative world model, an analytic vehicle dynamics block injects tire blowout as reduced cornering stiffness and a yaw moment, the world model emits observations to the planner under test which returns actions, the rollout produces a criticality score, a scenario optimizer fits a biased proposal over scenario parameters, and a reweighted risk estimate combines failure indicators with importance weights

Figure 1: Two loops, not one. The inner loop (world model and planner exchanging $o_t$ and $a_t$ ) provides reactivity, while the outer loop (criticality score, proposal update, resampled $c$ ) provides rarity. Fault injection enters through the analytic dynamics block because the generative model has essentially no blowout data to learn from.

The two example events stress different parts of the stack. A jaywalk is fundamentally a behavior and occlusion problem, and the world model is well suited to it: the pedestrian is spawned behind a parked vehicle, the onset time is set relative to the ego’s projected arrival, and the generated views must keep the occluder and the emerging body consistent across cameras so that the planner’s detector sees a realistic partial reveal. A blowout is a plant fault: within roughly 100 ms the affected corner loses most of its lateral capability and an asymmetric longitudinal force produces a yaw moment the driver did not command, so what the planner must handle is a sudden mismatch between commanded and achieved trajectory, not a novel image. Injecting it analytically also gives the parameter sweep something meaningful to search over, since severity, onset speed, and which corner fails are continuous knobs, whereas asking a video model to “show a blowout” yields visually plausible frames with no correct dynamics underneath them.

Mathematical Formulation:
$z_{t+1} \sim p_\theta(z_{t+1} \mid z_t, a_t, c)$
$o_t = g_\theta(z_t)$
$a_t = \pi(o_{1:t})$
$P_F = \mathbb{E}_{c \sim p}[\mathbf{1}\{F(c, \pi)\}]$
$\hat P_F = \frac{1}{N}\sum_{i=1}^{N} w_i \mathbf{1}\{F(c_i, \pi)\}$
$w_i = p(c_i) / q(c_i)$
$\mathrm{RSE}_{\mathrm{MC}} = 1 / \sqrt{N P_F}$

Where:

$z_t$ is the latent scene state, $o_t = g_\theta(z_t)$ the decoded multi-view observation, and $a_t$ the control emitted by the planner under test $\pi$ .
$c$ is the scenario parameter vector (occlusion offset, jaywalk onset, agent speeds, friction, fault severity), held fixed for a rollout and sampled per rollout.
$p(c)$ is the nominal distribution estimated from fleet logs and $q(c)$ the biased proposal from which $c_i \sim q$ is actually drawn.
$F(c, \pi)$ is the failure event for the closed-loop rollout (collision, or time-to-collision below a threshold) and $\mathbf{1}\{\cdot\}$ its indicator.
$w_i$ is the importance weight that removes the sampling bias, and $i \in \{1,\ldots,N\}$ indexes the $N$ rollouts in the campaign.
$\mathrm{RSE}_{\mathrm{MC}}$ is the relative standard error of naive Monte Carlo, which is what makes small $P_F$ unaffordable without a proposal.

Blowout As A Dynamics Perturbation:
$\tilde C_{\alpha} = \beta C_{\alpha}$
$M_z = \tfrac{1}{2} t_w \Delta F_x$

Here $C_{\alpha}$ is the nominal cornering stiffness of the failed tire, $\beta$ its residual fraction after deflation (often taken near 0.3 and ramped over about 100 ms), $t_w$ the track width, and $\Delta F_x$ the left-right longitudinal force asymmetry that generates the uncommanded yaw moment $M_z$ . Both $\beta$ and the onset ramp are entries in $c$ , so the same search machinery that finds the worst jaywalk timing also finds the worst blowout severity at the worst road curvature.

Sampling Budget For A $10^{-7}$ Event:
$N_{\mathrm{MC}} = 100 / 10^{-7} = 10^{9}$
$10^{9} \times 30\ \mathrm{s} = 3 \times 10^{10}\ \mathrm{s}$
$3 \times 10^{10} / 1000 = 3 \times 10^{7}\ \mathrm{s}$
$N_{\mathrm{IS}} = 10^{9} / 10^{4} = 10^{5}$
$10^{5} \times 30\ \mathrm{s} / 1000 = 3 \times 10^{3}\ \mathrm{s}$

Reaching a 10% relative standard error on a $10^{-7}$ event needs $10^{9}$ nominal rollouts, which at 30 s of simulated time each and 1000 parallel workers is about 347 days of wall clock. A proposal that delivers a $10^{4}$ variance-reduction factor brings the same confidence down to $10^{5}$ rollouts, or roughly 50 minutes. That ratio, not visual fidelity, is why rare-event machinery is the core of a stress-testing system.

Two panels: the left panel plots the nominal density of time-to-collision at pedestrian entry peaking near three seconds against a biased proposal density peaking near one second, with the region below 0.6 seconds shaded as the failure region and annotated with its tiny nominal probability mass; the right panel is a log-log plot of estimator relative standard error against number of rollouts for naive Monte Carlo and for importance sampling, with a dashed ten percent target line crossed at one billion rollouts and one hundred thousand rollouts respectively

Figure 2: Left: the nominal distribution puts almost no mass in the failure region, so the proposal is shifted onto the boundary and every sample is reweighted by $p/q$ . Right: because naive relative error scales as $1/\sqrt{N P_F}$ , a $10^{4}$ variance-reduction factor moves the 10% confidence target from $10^{9}$ rollouts to $10^{5}$ .

Property	Log replay	Procedural simulator	Generative world model
Closed-loop reactivity	None, other agents ignore the ego	Scripted or rule-based reaction	Learned reaction conditioned on ego action
Sensor realism	Perfect for the original trajectory only	Graphics-engine gap in texture and noise	High photometric realism, geometry can drift
Authoring a rare event	Impossible, you can only wait for it	Hand-written scenario scripts	Conditioning plus a searchable parameter vector
Tire blowout fidelity	Only if a logged vehicle actually blew a tire	Good, validated vehicle dynamics model	Poor alone, needs an analytic dynamics layer
Rollout throughput	Very high, no synthesis cost	High, especially at bounding-box abstraction	Low, diffusion or autoregressive decoding per frame
Dominant failure mode	Divergence invalidates the log after ~1 s	Unrealistic agent behavior distribution	Hallucinated geometry and long-horizon drift

August 19, 2026