Category: Hard

DL0162 World Model Object Permanence

How do world models maintain object permanence over long video generation horizons when an object is fully occluded behind another vehicle or building?

Answer

Object permanence is not a property a next-frame predictor gets for free. During a full occlusion the pixels that evidence the object are simply absent, so nothing in the current observation constrains what should re-emerge, and the only thing that can carry the object across the gap is state that outlives the pixels. Production world models therefore rely on three memory mechanisms, usually in combination: a long context of past frame latents that the generator can re-attend to, a recurrent latent state whose prior propagates unobserved objects forward, and an explicit pose-indexed geometric memory (object slots, 3D boxes, or a bird’s-eye-view layout) that is re-projected into the image when the occluder clears. Google DeepMind’s Genie 3 makes the first mechanism explicit, generating 720p frames at 24 fps with a reported visual memory extending roughly one minute into the past, while driving world models in the GAIA line lean on structured conditioning such as other agents’ 3D boxes and the ego trajectory. The practical question in an interview is therefore never “does the model understand permanence” but “does the information survive the occlusion window, and at what cost”, because a 2.5 second occlusion at 24 fps is 60 frames of pure extrapolation and the memory horizon either covers it or the object is re-invented rather than remembered.

(1) Occlusion Is A Missing-Evidence Problem: the observation likelihood contributes nothing about the hidden object, so generation must fall back on the dynamics prior rather than on perception.
(2) Long Context Buys Re-Readable Memory: keeping the last $H$ frames of latents lets attention look back at the pre-occlusion view, at the price of a KV cache linear in $H$ and a quadratic prefill.
(3) Recurrent State Buys Constant Cost: a fixed-size latent $s_t$ carries the object at $O(1)$ memory per step, but its finite capacity makes identity attributes drift instead of vanishing abruptly.
(4) Explicit 3D Memory Buys Correct Re-Projection: storing object records with poses turns re-emergence into geometry, so the vehicle reappears at the right pixel and the right moment.
(5) The Failure Signature Differs Per Mechanism: a hard cliff when $T_{\mathrm{occ}}$ exceeds $H$ , a smooth decay for recurrent state, and mis-registration under ego-pose drift for geometric memory.
(6) Rollout Stability Is A Separate Axis: teacher-forced training with fully autoregressive inference creates exposure bias, so small per-frame errors compound and corrupt identity even when the object is never occluded.

Architecture diagram in which an observation, an action, and a video tokenizer feed three parallel memory blocks: a long-context attention cache over the last H frames, a recurrent latent state whose prior propagates hidden objects, and a pose-indexed 3D or bird's-eye-view memory holding explicit object slots; all three condition a diffusion or autoregressive generator that predicts the next frame, which is appended back into the context as an autoregressive loop

Figure 1: Three places a hidden object can live. The generator is identical in all three cases; what changes is which memory still contains the occluded vehicle at frame $t$ , and therefore whether the model re-renders the same car or invents a new one.

The cleanest way to reason about the recurrent path is the standard latent-dynamics factorization used by Dreamer-style world models. A posterior $q(s_t \mid s_{t-1}, a_t, o_t)$ fuses the new observation, while a prior $p(s_t \mid s_{t-1}, a_t)$ predicts the next state without one. Under full occlusion the object’s contribution to the posterior disappears, so the model is effectively running the prior for $T_{\mathrm{occ}}$ steps on that part of the scene, which is exactly the regime where capacity limits and accumulated error show up. Long-context attention avoids that open-loop integration by re-reading the original evidence, but only if the pre-occlusion frames are still inside the window: once they are evicted, the model has no way to distinguish “a red hatchback was there” from “some vehicle may exist”, and it samples a plausible fresh instance. Explicit memory sidesteps both problems by storing a symbol rather than a distributed code, which is why driving-domain world models condition on agent boxes and map layouts instead of hoping attention rediscovers them.

Line chart of a target vehicle's lateral road position in metres against frame index from 0 to 140, with a shaded band marking a 60-frame full occlusion. The ground-truth trajectory is a dotted grey line; an explicit 3D memory model tracks it closely after the occlusion, a long-context model re-emerges about 1.2 metres off-lane and slowly converges back, and a short-context model produces no object after the occlusion until a spurious new vehicle appears far from the true trajectory

Figure 2: What re-emergence actually looks like. The occlusion window is identical for all three models; the difference appears only after the occluder clears, where the short-context rollout hallucinates a fresh instance in the wrong lane and the long-context rollout keeps the object but mis-registers its position, while pose-indexed memory lands on the true trajectory.

Mathematical Formulation:
$p(x_{1:T} \mid a_{1:T}) = \prod_{t=1}^{T} p(x_t \mid m_t, a_t)$
$m_t = (x_{t-H:t-1},\; s_{t-1},\; \mathcal{M}_{t-1})$
$q_t = q(s_t \mid s_{t-1}, a_t, o_t)$
$p_t = p(s_t \mid s_{t-1}, a_t)$
$T_{\mathrm{occ}} = \Delta t \cdot \mathrm{fps}$
$H \geq T_{\mathrm{occ}}$

Where:

$x_t$ is the generated frame latent at step $t$ and $a_t$ the conditioning action or control (steering, camera pose, text instruction).
$m_t$ is everything the generator may condition on: the context window of past latents, the recurrent state, and the persistent geometric memory.
$H$ is the memory horizon in frames and $N_p$ the number of latent tokens per frame, so the window holds $H N_p$ tokens.
$q_t$ is the posterior that uses the observation $o_t$ and $p_t$ the prior that does not; under full occlusion the object is carried only by $p_t$ .
$\mathcal{M}_t$ is the explicit memory: object records with identity, extent, and pose, plus the camera pose needed to re-project them.
$T_{\mathrm{occ}}$ is the occlusion length in frames, obtained from its duration $\Delta t$ and the frame rate; the last line is the necessary condition for a pure context window to bridge it.
$L$ and $d$ are the number of attention layers and the model width, which set the cache and prefill costs below.

Bridging A 2.5 Second Occlusion At 24 fps:
$T_{\mathrm{occ}} = 2.5 \times 24 = 60$
$H N_p = 60 \times 256 = 15360$
$\mathrm{cache} = O(H N_p L d)$
$\mathrm{prefill} = O((H N_p)^2 d)$
$R(600) = 0.995^{600} \approx 0.05$

Sixty frames of latents at 256 tokens per frame is 15,360 tokens of context bought for a single occlusion, and every additional second of horizon costs another 6,144 tokens of cache plus a quadratic increase in prefill work, which is precisely why real-time interactive world models cap $H$ aggressively and then need a cheaper memory. The last line models the orthogonal failure: if identity is corrupted independently at each step with probability $\delta = 0.005$ , the retention rate $R(T) = (1-\delta)^{T}$ leaves only about 5% of objects intact after a 600-frame (25 second) rollout even with no occlusion at all. Permanence over long horizons is therefore two bugs wearing one coat: a horizon bug that shows up as a cliff, and a drift bug that shows up as exponential decay.

Chart of correct re-emergence rate against occlusion duration in frames from 0 to 240 for four memory mechanisms: a 16-frame context window collapses sharply just after 16 frames, a 96-frame context window stays high then collapses just after 96 frames, a recurrent latent state decays smoothly and geometrically, and a pose-indexed 3D memory stays near 0.9 across the whole range, with vertical markers at the two context horizons and at the 60-frame mark

Figure 3: Two distinct shapes of failure. Finite context gives a step cliff exactly at $T_{\mathrm{occ}} = H$ , recurrent state gives geometric decay from capacity limits and compounding error, and explicit pose-indexed memory is nearly flat because duration costs it almost nothing.

Property	Long-context attention	Recurrent latent state	Explicit 3D / object memory
What survives the gap	Raw pre-occlusion frame latents, re-readable by attention	A compressed distributed code of the whole scene	Symbolic object records: identity, extent, pose, velocity
Cost of a 60-frame occlusion	About 15,360 cached tokens, with quadratic prefill growth	One fixed-size state vector, independent of duration	A few object records plus one camera pose per frame
Hard limit	The horizon itself: nothing outside H frames exists	State dimensionality and compounding prior error	Detector, tracker, and pose accuracy; assumes rigid objects
Dominant failure mode	Cliff: a fresh instance with new colour, size, and lane	Drift: right object class, wrong attributes and timing	Mis-registration under pose drift, or duplicated instances
Typical home	Video diffusion transformers and interactive frame models	Dreamer-style latent world models for control	Driving world models conditioned on agent boxes and layouts

August 19, 2026

DL0161 World Model Safety Edge Cases

How do world models simulate low-probability safety-critical edge cases such as tire blowouts or sudden pedestrian jaywalking for closed-loop planner stress testing?

Answer

A driving world model is an action-conditioned generative simulator $p_\theta(z_{t+1} \mid z_t, a_t, c)$ that rolls forward latent state and decodes multi-view sensor observations, so the planner under test acts inside the rollout instead of replaying a fixed log. Rare events are never obtained by sampling the model unconditionally, because the learned distribution matches the fleet distribution in which a severe collision appears roughly once per $10^{8}$ miles. Instead the scenario is authored through conditioning: a low-dimensional parameter vector $c$ holding occlusion geometry, jaywalk onset time, pedestrian speed, and surface friction is drawn from a biased proposal $q(c)$ fitted by a search loop that maximizes criticality, then each rollout is reweighted by the likelihood ratio $w = p(c)/q(c)$ so the resulting failure-rate estimate remains unbiased. A tire blowout is handled differently from a jaywalk, because it is a vehicle-dynamics discontinuity rather than an appearance change and almost no camera dataset contains it, so production stacks inject it in an analytic dynamics layer (collapsed cornering stiffness on one corner plus a yaw moment) while the world model supplies the surrounding traffic and imagery.

(1) Closed Loop Requires Action Conditioning: the planner’s own control enters the model at every step, so log replay and open-loop video prediction cannot be used, since the ego trajectory diverges from the recorded one within a second or two.
(2) Controllability Comes From Structured Conditioning: text, road layout, 3D agent boxes, and ego action are all conditioning channels, which turns “pedestrian steps out from behind the parked van 0.8 s before ego arrival” into an explicit specification rather than a hopeful prompt.
(3) Rarity Comes From Biased Sampling: importance sampling with a cross-entropy-method proposal concentrates rollouts near the failure boundary, cutting the sample count for a fixed confidence by three to five orders of magnitude.
(4) Unbiasedness Comes From Likelihood Ratios: reporting the raw failure fraction under $q$ overstates real-world risk by exactly the sampling bias, so every rollout carries its weight $w_i$ into the estimator.
(5) Hybrid Physics For Fault Injection: blowouts, brake fade, and actuator latency live in a validated dynamics model, and the world model consumes the resulting ego state as a conditioning signal.
(6) Plausibility Must Be Constrained: unconstrained adversarial search produces kinematically impossible agents and physically unavoidable collisions, so scenarios are filtered by a likelihood floor on $\log p(c)$ and by an avoidability check against a reference planner.
(7) Rollout Stability Bounds The Horizon: autoregressive generation accumulates drift, so useful stress tests run 10 s to 30 s at 10 Hz rather than minutes, and geometric consistency across views is monitored as a validity metric.

Diagram of a closed-loop stress-testing system: real driving logs pretrain a generative world model, an analytic vehicle dynamics block injects tire blowout as reduced cornering stiffness and a yaw moment, the world model emits observations to the planner under test which returns actions, the rollout produces a criticality score, a scenario optimizer fits a biased proposal over scenario parameters, and a reweighted risk estimate combines failure indicators with importance weights

Figure 1: Two loops, not one. The inner loop (world model and planner exchanging $o_t$ and $a_t$ ) provides reactivity, while the outer loop (criticality score, proposal update, resampled $c$ ) provides rarity. Fault injection enters through the analytic dynamics block because the generative model has essentially no blowout data to learn from.

The two example events stress different parts of the stack. A jaywalk is fundamentally a behavior and occlusion problem, and the world model is well suited to it: the pedestrian is spawned behind a parked vehicle, the onset time is set relative to the ego’s projected arrival, and the generated views must keep the occluder and the emerging body consistent across cameras so that the planner’s detector sees a realistic partial reveal. A blowout is a plant fault: within roughly 100 ms the affected corner loses most of its lateral capability and an asymmetric longitudinal force produces a yaw moment the driver did not command, so what the planner must handle is a sudden mismatch between commanded and achieved trajectory, not a novel image. Injecting it analytically also gives the parameter sweep something meaningful to search over, since severity, onset speed, and which corner fails are continuous knobs, whereas asking a video model to “show a blowout” yields visually plausible frames with no correct dynamics underneath them.

Mathematical Formulation:
$z_{t+1} \sim p_\theta(z_{t+1} \mid z_t, a_t, c)$
$o_t = g_\theta(z_t)$
$a_t = \pi(o_{1:t})$
$P_F = \mathbb{E}_{c \sim p}[\mathbf{1}\{F(c, \pi)\}]$
$\hat P_F = \frac{1}{N}\sum_{i=1}^{N} w_i \mathbf{1}\{F(c_i, \pi)\}$
$w_i = p(c_i) / q(c_i)$
$\mathrm{RSE}_{\mathrm{MC}} = 1 / \sqrt{N P_F}$

Where:

$z_t$ is the latent scene state, $o_t = g_\theta(z_t)$ the decoded multi-view observation, and $a_t$ the control emitted by the planner under test $\pi$ .
$c$ is the scenario parameter vector (occlusion offset, jaywalk onset, agent speeds, friction, fault severity), held fixed for a rollout and sampled per rollout.
$p(c)$ is the nominal distribution estimated from fleet logs and $q(c)$ the biased proposal from which $c_i \sim q$ is actually drawn.
$F(c, \pi)$ is the failure event for the closed-loop rollout (collision, or time-to-collision below a threshold) and $\mathbf{1}\{\cdot\}$ its indicator.
$w_i$ is the importance weight that removes the sampling bias, and $i \in \{1,\ldots,N\}$ indexes the $N$ rollouts in the campaign.
$\mathrm{RSE}_{\mathrm{MC}}$ is the relative standard error of naive Monte Carlo, which is what makes small $P_F$ unaffordable without a proposal.

Blowout As A Dynamics Perturbation:
$\tilde C_{\alpha} = \beta C_{\alpha}$
$M_z = \tfrac{1}{2} t_w \Delta F_x$

Here $C_{\alpha}$ is the nominal cornering stiffness of the failed tire, $\beta$ its residual fraction after deflation (often taken near 0.3 and ramped over about 100 ms), $t_w$ the track width, and $\Delta F_x$ the left-right longitudinal force asymmetry that generates the uncommanded yaw moment $M_z$ . Both $\beta$ and the onset ramp are entries in $c$ , so the same search machinery that finds the worst jaywalk timing also finds the worst blowout severity at the worst road curvature.

Sampling Budget For A $10^{-7}$ Event:
$N_{\mathrm{MC}} = 100 / 10^{-7} = 10^{9}$
$10^{9} \times 30\ \mathrm{s} = 3 \times 10^{10}\ \mathrm{s}$
$3 \times 10^{10} / 1000 = 3 \times 10^{7}\ \mathrm{s}$
$N_{\mathrm{IS}} = 10^{9} / 10^{4} = 10^{5}$
$10^{5} \times 30\ \mathrm{s} / 1000 = 3 \times 10^{3}\ \mathrm{s}$

Reaching a 10% relative standard error on a $10^{-7}$ event needs $10^{9}$ nominal rollouts, which at 30 s of simulated time each and 1000 parallel workers is about 347 days of wall clock. A proposal that delivers a $10^{4}$ variance-reduction factor brings the same confidence down to $10^{5}$ rollouts, or roughly 50 minutes. That ratio, not visual fidelity, is why rare-event machinery is the core of a stress-testing system.

Two panels: the left panel plots the nominal density of time-to-collision at pedestrian entry peaking near three seconds against a biased proposal density peaking near one second, with the region below 0.6 seconds shaded as the failure region and annotated with its tiny nominal probability mass; the right panel is a log-log plot of estimator relative standard error against number of rollouts for naive Monte Carlo and for importance sampling, with a dashed ten percent target line crossed at one billion rollouts and one hundred thousand rollouts respectively

Figure 2: Left: the nominal distribution puts almost no mass in the failure region, so the proposal is shifted onto the boundary and every sample is reweighted by $p/q$ . Right: because naive relative error scales as $1/\sqrt{N P_F}$ , a $10^{4}$ variance-reduction factor moves the 10% confidence target from $10^{9}$ rollouts to $10^{5}$ .

Property	Log replay	Procedural simulator	Generative world model
Closed-loop reactivity	None, other agents ignore the ego	Scripted or rule-based reaction	Learned reaction conditioned on ego action
Sensor realism	Perfect for the original trajectory only	Graphics-engine gap in texture and noise	High photometric realism, geometry can drift
Authoring a rare event	Impossible, you can only wait for it	Hand-written scenario scripts	Conditioning plus a searchable parameter vector
Tire blowout fidelity	Only if a logged vehicle actually blew a tire	Good, validated vehicle dynamics model	Poor alone, needs an analytic dynamics layer
Rollout throughput	Very high, no synthesis cost	High, especially at bounding-box abstraction	Low, diffusion or autoregressive decoding per frame
Dominant failure mode	Divergence invalidates the log after ~1 s	Unrealistic agent behavior distribution	Hallucinated geometry and long-horizon drift

August 19, 2026

DL0160 World Model Multimodal Futures

How do world models handle multi-modal future distributions when a car at an intersection could turn left, turn right, or go straight, as in driving world models like Wayve’s GAIA?

Answer

The future at an intersection is genuinely multi-modal, so the first thing a world model must avoid is regression to the conditional mean. The minimizer of a squared-error objective is $\mathbb{E}[x \mid c]$ , which means a network that has perfectly learned that left, straight, and right are all plausible will output their average: a path down the median that no driver would ever take. The fix is architectural rather than a loss-weighting trick. A world model is built as a conditional sampler, not a point predictor, so drawing from it returns one internally consistent future at a time and the multimodality appears across draws rather than inside a single blurred output. Three mechanisms dominate in practice: a stochastic latent sampled at every rollout step (Dreamer-style RSSM), ancestral sampling of discrete tokens over a learned video codebook (GAIA-1-style), and iterative denoising from Gaussian noise (GAIA-2-style latent diffusion). Trajectory-level stacks add a fourth: a small set of anchors or mixture components with explicit probabilities, trained with a winner-take-all loss so each head owns one maneuver.

(1) Squared Error Averages Modes: any L2-trained deterministic head converges to the mean of the modes, and at a T-junction that mean is a physically impossible maneuver even though it minimizes the loss.
(2) Sample, Do Not Average: the model defines $p(x \mid c)$ and multimodality is expressed by independent draws, each of which must remain a single coherent maneuver from first frame to last.
(3) Stochastic Latents: an RSSM splits state into a deterministic recurrent part $h_t$ and a sampled part $z_t$ (DreamerV3 uses 32 categoricals of 32 classes), so branching happens once per step and is then carried forward consistently.
(4) Discrete Token Sampling: quantize frames into codebook tokens and sample autoregressively with temperature or top-p, which turns mode choice into ordinary categorical sampling at the price of a long token sequence per second of video.
(5) Explicit Mode Heads: $K$ anchored components with probabilities $\pi_k$ give the planner a calibrated, enumerable set of maneuvers instead of an opaque sampler, at the cost of a fixed mode budget.
(6) Metrics Must Reward Coverage: single-sample L2 rewards mode averaging, so evaluation moves to $\mathrm{minADE}_K$ , maneuver recall, and probability calibration, with $K = 6$ the standard budget on the Waymo Open Motion Dataset.

Two top-down views of the same four-way intersection. On the left, three colored ground-truth trajectories leave the ego lane and turn left, continue straight, and turn right, while a thick red dashed line shows the squared-error optimum crawling up the middle of the junction and ending between the modes. On the right, twelve sampled trajectories from a stochastic world model, four per maneuver with varying speed and lateral jitter, each one a legal single maneuver covering all three modes.

Figure 1: The same scene under two objectives. The L2 optimum is the pointwise average of the three maneuvers, so it drifts up the middle of the junction and matches none of them, while 12 draws from a stochastic model are individually legal and jointly cover all three modes. Nothing is wrong with the averaged model’s likelihood estimate of the mean; the problem is that the mean is not an admissible trajectory.

Where the randomness enters decides how the model behaves in a rollout. In a latent state-space model the sample is a small categorical latent drawn once per timestep, so a single decision at the junction propagates through the recurrent state and the decoded frames stay consistent for the rest of the horizon; the risk is posterior collapse, where the KL term is tuned so aggressively that the prior stops carrying maneuver information and rollouts become deterministic again. In a discrete autoregressive model the sample is a token, and mode choice is spread over hundreds of tokens per frame, which makes temperature a global blur-versus-diversity knob: too low and every rollout goes straight, too high and lane geometry falls apart. Diffusion models place the randomness in the initial noise vector, giving the best sample fidelity and the most controllable conditioning, but they pay 20 to 50 network evaluations per sample and cannot easily produce a probability for each maneuver. Explicit mixture heads sit at the opposite end: a Wayformer-style model emits $K = 6$ Gaussian components with softmax weights in one forward pass of a few milliseconds, which is what a downstream planner actually wants, but six is a hard ceiling on expressible futures.

Diagram with a left column of three boxes naming the sources of randomness, categorical latent sampling in a Dreamer-style RSSM, discrete token sampling over a video codebook as in GAIA-1, and Gaussian noise denoised by a diffusion model as in GAIA-2, next to a branching tree on the right where a context box splits into three colored maneuver branches that each split into two leaves labeled with maneuver variants and probabilities summing to one.

Figure 2: All three families implement the same idea with different noise sources, and the tree shows why it works: a draw at the first branch commits to a maneuver, later draws only refine speed and gap acceptance, and the leaf probabilities recover the marginal maneuver distribution (0.28 left, 0.45 straight, 0.27 right). A deterministic model collapses this tree to its centroid.

Mathematical Formulation:
$f^{*}(c) = \mathbb{E}[x \mid c]$
$p(x \mid c) = \sum_{k=1}^{K} \pi_k \mathcal{N}(x; \mu_k, \Sigma_k)$
$p(x \mid c) = \int p(x \mid z, c) p(z \mid c) \, dz$
$p(x_{1:T} \mid c) = \prod_{t=1}^{T} p(x_t \mid x_{1:t-1}, c)$

Where:

$x$ is the future to be predicted (a trajectory of waypoints or a sequence of frames) and $f^{*}(c)$ is the squared-error optimum, which is the conditional mean and therefore not generally a valid maneuver.
$c$ is the conditioning context: past observations, lane graph, ego speed, and traffic-light state.
$k \in \{1, \ldots, K\}$ indexes mixture components or anchors, with weights $\pi_k$ summing to one, means $\mu_k$ , and covariances $\Sigma_k$ ; $K = 6$ is the standard benchmark budget.
$z$ is the stochastic latent whose prior $p(z \mid c)$ carries the mode choice; marginalizing it makes $p(x \mid c)$ multi-modal even when $p(x \mid z, c)$ is unimodal.
$t \in \{1, \ldots, T\}$ indexes rollout steps or tokens, and the product form is the ancestral sampling factorization used by discrete-token world models.

Why L2 Prefers The Impossible Path (two equally likely 8 m lateral outcomes):
$f^{*} = 0.5(-8) + 0.5(+8) = 0$
$\mathrm{MSE}(f^{*}) = 0.5(64) + 0.5(64) = 64$
$\mathrm{MSE}(\mu_1) = 0.5(0) + 0.5(256) = 128$

Committing to a real maneuver scores twice as badly as predicting the physically impossible average, which is the whole reason single-output regression is untrustworthy here. Mode-based training removes the incentive by only penalizing the closest component and learning the weights separately, and rollout models remove it by making the loss a likelihood rather than a distance.

Mode-Based Training And Scoring:
$k^{*} = \arg\min_{k} \lVert x - \mu_k \rVert$
$\mathcal{L}_{\mathrm{reg}} = \lVert x - \mu_{k^{*}} \rVert^2$
$\mathcal{L}_{\mathrm{cls}} = -\log \pi_{k^{*}}$
$\mathrm{minADE}_K = \min_{k} \mathrm{ADE}(\hat{x}^{(k)}, x)$
$\mathrm{ADE}(\hat{x}, x) = \frac{1}{T} \sum_{t=1}^{T} \lVert x_t - \hat{x}_t \rVert$

The evaluation side matters as much as the model. Reporting a single-sample average displacement error silently rewards mode averaging, so benchmarks score $\mathrm{minADE}_K$ and $\mathrm{minFDE}_K$ over $K$ samples, plus a mAP-style metric that requires the probabilities to be calibrated and not just the geometry to be covered. That combination is deliberate: $\mathrm{minADE}_K$ alone can be gamed by spraying diverse but implausible samples, while likelihood alone can be won by a model that puts all its mass on “straight” because straight is the majority class at most intersections. In production the planner consumes both, treating each mode as a separate scenario to cost, so a mode that is missing from the sample set is a scenario the planner never considers.

Left panel: line chart of minADE in meters versus number of sampled futures K from 1 to 24, with a calibrated diverse sampler falling steadily below one meter, a mode-collapsed sampler plateauing near 1.8 meters, and a dashed horizontal line for a single deterministic prediction at 3.1 meters, with a vertical dashed line marking the benchmark budget K equals 6. Right panel: bar chart of maneuver coverage at that budget, 0.34 for the deterministic head, 0.61 for the collapsed generative model, and 0.93 for the calibrated generative model, with value labels above the bars.

Figure 3: Drawing more samples only helps if the samples differ. A mode-collapsed sampler flattens out near 1.8 m because sample 6 repeats sample 1, while a diverse sampler keeps improving; the bar panel shows the consequence the planner feels, namely how often the maneuver the other car actually performed is present anywhere in the predicted set.

Property	Anchored mixture head	Latent state-space (RSSM)	Discrete token AR	Latent diffusion
Source of multimodality	K anchors with softmax weights	Categorical latent drawn each step	Temperature or top-p token sampling	Initial Gaussian noise vector
Cost per future	One forward pass gives all K	T cheap recurrent steps	Hundreds of tokens per frame	20 to 50 denoising steps
Explicit probabilities	Yes, directly usable by a planner	Only via repeated sampling	Sequence likelihood, hard to read	No tractable density
Expressible futures	Capped at K (usually 6)	Combinatorial in latent codes	Unbounded, pixel-level detail	Unbounded, highest fidelity
Dominant failure	Dead heads under winner-take-all	Posterior collapse, deterministic rollouts	Low temperature collapses to majority mode	Latency, and no calibrated mode weights

August 19, 2026

DL0159 Counterfactual Simulation for Driving

What is counterfactual simulation in autonomous driving, and how do world models let safety engineers ask what would happen if the ego vehicle had braked two seconds earlier?

Answer

Counterfactual simulation re-runs one specific logged drive with exactly one thing changed, usually the ego vehicle’s own action sequence, and then asks what the rest of the world would have done in response. It is Pearl’s three-step recipe applied to a driving log: abduction → action → prediction. First infer the latent variables $z$ that explain the observed log, including road geometry, occlusions, and each neighbour’s intent and aggressiveness. Then intervene by substituting a new ego plan $a'$ , such as brake onset two seconds earlier, and roll the scene forward under that intervention. The hard part is not the ego kinematics, which a closed-form formula settles in one line; the hard part is that every other agent’s logged trajectory becomes invalid the instant the ego behaves differently, so a naive non-reactive log replay manufactures collisions that could never have happened. A world model supplies the missing conditional distribution $p(s_{t+1} \mid s_t, a'_t, z)$ , generating plausible reactions for surrounding traffic either as trajectories over a structured scene or as generated sensor frames, and because those reactions are stochastic the answer is a distribution over outcomes estimated from many sampled rollouts rather than a single verdict.

(1) Counterfactual, Not Merely Interventional: $p(Y \mid do(a'))$ averages over all scenes that could occur, while the counterfactual $p(Y_{a'} \mid o_{1:T}, a_{1:T})$ conditions on the evidence of this exact log, so the same pedestrian keeps the same intent.
(2) Abduction Fixes The World: inferring $z$ from the log is what makes the rollout a rerun of that day rather than a generic scenario with similar statistics.
(3) Reactivity Is The Whole Problem: logged responses are only valid for the logged ego action, so frozen playback agents produce fake rear-end contacts and hide genuine near misses.
(4) Divergence Horizon: simulated and logged states separate as the intervention propagates, so a counterfactual is trustworthy over a few seconds and becomes speculation over tens of seconds.
(5) Sample, Never Simulate Once: outcome metrics are estimated from $N$ rollouts with confidence intervals, since one lucky rollout is not evidence of avoidance.
(6) Metrics Decide The Verdict: contact or no contact, impact speed and delta-V, minimum time-to-collision, and whether the newly introduced risk (an in-lane stop, an induced rear-end) offsets the risk removed.

Five-stage pipeline from left to right: observe the logged drive, abduct the latent scene state and agent intents, intervene by replacing the ego plan with a brake two seconds earlier, roll out the world model with reacting neighbours, and score the outcome, with a second row showing the corresponding notation under each stage

Figure 1: A counterfactual is not a fresh simulation. Step 2 abducts the latent scene so the neighbours keep the dispositions they actually had, step 3 changes only the ego action, and step 4 must re-generate every response because the logged responses are no longer admissible evidence.

Three simulator families answer step 4 with very different fidelity. Log replay keeps every neighbour on its recorded trajectory, which is cheap and perfectly grounded for the first fraction of a second and actively misleading afterwards. Rule-based reactive agents (car-following plus lane-change models) give physically consistent responses at negligible cost but with a narrow behavioural repertoire that tends to be too polite, biasing avoidance estimates optimistically. Learned world models sit at the top: trajectory-level sim agents of the kind benchmarked in the Waymo Open Sim Agents Challenge, LiDAR-space models such as Copilot4D, and pixel-space generative models such as Wayve’s GAIA line, which can re-render the camera stream so the perception stack is exercised too rather than being handed ground-truth tracks. Regulatory use of this machinery is already public: Waymo reconstructed fatal crashes inside its operating domain and simulated its driver in place of each human participant, both as crash initiator and as responder, reporting avoided-or-mitigated outcomes against a non-impaired, eyes-always-on reference driver. The credibility of any such claim rests on how well the simulator’s agents were validated, not on how photorealistic the rollout looks.

Two time versus distance panels for the same counterfactual. Left panel shows log replay where the following vehicle keeps a straight logged trajectory and intersects the braking ego, marked as a fake rear-end. Right panel shows a reactive world model where the follower brakes after a reaction delay and stops short of the ego, with the logged ego trajectory reaching the stopped lead vehicle at 100 metres in both panels

Figure 2: The same intervention, two simulators. With frozen playback the follower drives into the decelerating ego and the tool reports an 18.9 m/s rear-end that never existed, while a reactive model has the follower brake 0.9 s after the brake lights and stop 5.3 m short. Both panels agree on the ego: braking two seconds earlier turns a 12.6 m/s frontal impact into a stop 26.7 m short of the obstacle.

Mathematical Formulation:
$z \sim p(z \mid o_{1:T}, a_{1:T})$
$a'_t = \pi'(\hat{s}_t)$
$\hat{s}_{t+1} = f(\hat{s}_t, a'_t, z)$
$Y_{a'} \sim p(Y \mid z, a'_{1:T})$
$\Delta R = R(\tau) - \mathbb{E}[R(\tau')]$

Where:

$o_{1:T}$ and $a_{1:T}$ are the observed log, meaning the recorded sensor stream plus tracks and the ego actions that were actually executed.
$z$ is the abducted latent state: map geometry, occupancy behind occlusions, and per-agent intent and aggressiveness, held fixed across the counterfactual.
$\pi'$ is the counterfactual policy and $a'_t$ its action, here a brake command issued 2 s earlier than in the log; $\hat{s}_t$ is the simulated joint state of ego plus neighbours.
$f$ is the world model transition, the only component that can supply neighbour responses to an action that was never taken.
$Y_{a'}$ is the counterfactual outcome variable (contact, impact speed, minimum TTC) and its distribution is estimated from $N$ independent rollouts $\tau'$ .
$R$ is a scalar risk or severity functional, so $\Delta R$ is the risk actually removed by the intervention, and it can be negative when the new action introduces its own hazard.

Braking Two Seconds Earlier At 20 m/s:
$d_{\mathrm{brake}} = \frac{v^2}{2a}$
$= \frac{20^2}{2 \times 6} = 33.3\ \text{m}$
$\Delta d = v \Delta t = 20 \times 2 = 40\ \text{m}$
$v_{\mathrm{imp}} = \sqrt{v^2 - 2 a d}$
$= \sqrt{400 - 240} = 12.6\ \text{m/s}$

In the logged run the brake came on with only 20 m left to a stopped lead vehicle, so the ego arrived at 12.6 m/s. The intervention adds 40 m of room, giving 60 m against the 33.3 m the ego needs, which is why it stops 26.7 m short. Notice how weak the two-second question actually is: the shortfall was $33.3 - 20 = 13.3$ m, so 0.67 s of extra warning already avoids contact, and everything beyond that buys margin rather than outcome. The engineering value of the simulator therefore lies in the second-order effects that closed-form kinematics cannot see, above all the induced risk from an earlier, harder, cause-free deceleration in front of a following vehicle.

Two stacked charts against how much earlier the brake is applied. The upper chart plots probability of any contact, frontal contact with a confidence band, and rear-end contact over 256 rollouts, with frontal risk falling from near one to near zero and rear-end risk rising slowly. The lower chart plots expected impact speed from sampled reactive rollouts against a dotted single-agent kinematic estimate that reaches zero at 0.67 seconds

Figure 3: Sweeping the intervention turns one question into a curve. Frontal risk collapses within about one second of extra warning while rear-end exposure grows, so past roughly 1.4 s the dominant residual hazard has changed identity. The dotted line shows why sampling matters: the single-agent kinematic estimate declares the crash impossible after 0.67 s, whereas reactive rollouts still assign residual severity out to about 1.5 s.

Property	Log replay (non-reactive)	Rule-based reactive agents	Learned world model
Neighbour behaviour	Frozen to the recorded trajectory, blind to the new ego action	Car-following and lane-change heuristics with hand-set reaction times	Sampled from a learned conditional distribution over joint futures
Valid horizon	Only until the ego state diverges, often under 0.5 s	Seconds, as long as the manoeuvre stays inside the rule set	Seconds to tens of seconds, limited by compounding rollout error
What is simulated	Ego dynamics only, on ground-truth perception	Ego plus abstract agent boxes, still on ground-truth perception	Trajectories, occupancy, LiDAR, or camera frames, so perception can be tested in the loop
Cost per rollout	Negligible, millions of segments per night	Milliseconds, fully parallel across scenarios	Accelerator seconds per simulated second for generative sensor rollouts
Dominant failure mode	Fabricated collisions and hidden near misses	Over-polite traffic that inflates avoidance rates	Hallucinated agents, causal confusion, and quiet drift out of distribution
Best used for	Regression checks on planner outputs in unchanged scenes	Large-scale sweeps and worst-case bounds with auditable assumptions	High-stakes crash reconstruction and long-tail scenario editing

August 19, 2026

DL0158 3D-Aware World Model Geometry

How do 3D-aware latent world models use geometric inductive biases such as 3D Gaussian Splatting and NeRFs to ensure view-consistent camera rollouts?

Answer

A 3D-aware world model never lets a decoder invent pixels directly. Its latent transition predicts a scene state (a set of anisotropic 3D Gaussians, or the weights and features of a radiance field), and the only path from that state to an image is a differentiable renderer that takes the camera extrinsics and intrinsics as explicit arguments. Because every frame of a rollout is a deterministic function of one shared geometry, parallax, occlusion ordering, and disparity are produced by projection math rather than learned from data, so a camera can orbit for hundreds of steps without the scene mutating underneath it. NeRF contributed the volumetric rendering equation that made this pipeline differentiable end to end. 3D Gaussian Splatting keeps exactly the same alpha-compositing math but replaces per-ray marching with tile-based rasterization, which cuts per-frame cost by two to three orders of magnitude and is what makes closed-loop, interactive camera rollouts possible at all. The bias is not free: it guarantees consistency only where geometry is supported, so unobserved regions, topology changes, and strong view-dependent effects still require a generative prior layered on top.

(1) Render, Do Not Predict: the decoder is replaced by $\mathcal{R}(\mathcal{S}_t, \pi_t)$ , so the camera pose enters as a projection matrix instead of a conditioning token the network may ignore.
(2) One State, Many Views: all views share a single geometry, which turns multi-view consistency from a learned behaviour into an architectural invariant up to the representation’s capacity.
(3) Two Renderers, One Equation: NeRF integrates density along rays while 3DGS rasterizes projected Gaussians, but both use the same front-to-back alpha compositing, so the geometric bias is identical and only the cost model differs.
(4) Dynamics Live In Scene Space: time is handled by a deformation field or 4D Gaussians that move primitives, so motion cannot silently destroy the identity of an object the way pixel-space prediction can.
(5) Latency Decides Feasibility: a world model must render inside the control loop, and rasterization at roughly 5-10 ms per 1080p frame versus seconds for vanilla volumetric marching is the difference between a usable simulator and an offline reconstruction.
(6) Where The Bias Stops Helping: unobserved regions, non-rigid topology change, transparency, and monocular scale ambiguity are outside the guarantee and are exactly where a generative prior must take over.

Two-row architecture comparison. The top row shows a latent state feeding a transition function then a 2D pixel decoder that emits a predicted frame, with camera pose supplied only as conditioning tokens through a dashed arrow. The bottom row shows the same latent state and transition producing an explicit scene state of 3D Gaussians with position, covariance, opacity and spherical harmonics, which passes through a differentiable renderer that projects, sorts and alpha-composites, with the camera pose entering as extrinsics and intrinsics, emitting a frame at any queried pose.

Figure 1: The same latent dynamics, two decoders. In the pixel-space path the pose is a soft conditioning signal the decoder may only approximately honour, so geometric errors compound across a long orbit. In the 3D-aware path the pose is an argument of an exact projection, so consistency holds by construction and the model’s remaining job is predicting geometry rather than predicting appearance.

The mechanism that actually enforces consistency is worth stating precisely. A pixel is a function of the primitives that project onto it, sorted by depth, so moving the camera changes the projected position of a primitive by an amount inversely proportional to its depth. That single fact gives the renderer correct parallax, correct occlusion ordering when a near primitive covers a far one, and correct disocclusion when the camera slides sideways, all without any of it appearing in the loss as a special term. Training only needs a photometric loss on the observed views; the geometry that explains several views simultaneously is the only geometry that can drive that loss to zero, which is why a well-fit scene state extrapolates to nearby unseen poses. A 2D video model has to learn the same relationships as correlations over pixels, and it has no mechanism preventing a later frame from contradicting an earlier one.

Left panel is a top-down view of a scene with two camera poses at the bottom, dotted view frustums, a near orange elliptical splat at depth 2.8 and a far blue elliptical splat at depth 5.6, with solid rays drawn from each camera to each splat centre. Right panel shows the two resulting image strips stacked vertically, with orange and blue markers at the projected horizontal positions, and two double-headed arrows measuring that the near splat shifts 0.73 of the frame width between the views while the far splat shifts only 0.37, a ratio of exactly two matching the depth ratio.

Figure 2: Why a shared 3D state is a hard constraint. Both renders come from one set of primitives, so the horizontal shift of each splat is fixed by its depth: the near splat moves exactly twice as far as the splat at twice the distance. A pixel-space decoder must reproduce this ratio from statistics, whereas the renderer cannot violate it, which is the entire content of the phrase geometric inductive bias.

Mathematical Formulation:
$z_{t+1} = f(z_t, a_t)$
$\mathcal{S}_t = g(z_t)$
$I_t = \mathcal{R}(\mathcal{S}_t, \pi_t)$

Shared Rendering Equation:
$C(p) = \sum_{i=1}^{N} T_i \alpha_i c_i$
$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j)$
$\alpha_i^{\mathrm{nerf}} = 1 - \exp(-\sigma_i \delta_i)$
$\alpha_i^{\mathrm{gs}} = o_i \, G_i(p)$
$\Sigma = R S S^{\top} R^{\top}$
$\Sigma' = J W \Sigma W^{\top} J^{\top}$
$\mu_i(t) = \mu_i + \Delta\mu_i(z_t)$

Where:

$z_t$ is the compact latent state, $a_t$ the action, and $f$ the learned transition model; $g$ decodes the latent into a scene state rather than into pixels.
$\mathcal{S}_t$ is the geometric state: for 3DGS a set of primitives $\{\mu_i, \Sigma_i, o_i, \mathrm{SH}_i\}$ , for a NeRF a field returning density and colour at a queried point and direction.
$\pi_t = (R \mid t, K)$ is the camera pose and intrinsics, and $\mathcal{R}$ the differentiable renderer; $I_t$ is the frame, so any pose can be queried at any rollout step.
$C(p)$ is the colour of pixel $p$ , $i$ indexes the $N$ contributions sorted front to back, $c_i$ is the view-dependent colour, and $T_i$ is the accumulated transmittance that produces occlusion for free.
$\sigma_i$ is the volume density at sample $i$ and $\delta_i$ the spacing between adjacent samples along the ray; $o_i$ is a Gaussian’s opacity and $G_i(p)$ its projected 2D density evaluated at the pixel.
$\Sigma$ is the world-space covariance factored into rotation $R$ and scale $S$ so it stays positive semi-definite under gradient descent, and $\Sigma'$ is its screen-space projection through the viewing transform $W$ and the affine Jacobian $J$ .
$\Delta\mu_i(z_t)$ is the predicted per-primitive displacement from a canonical configuration, which is how dynamics enter without breaking the static consistency guarantee at each instant.

The cost model is what selects the representation in practice. Volumetric marching evaluates a network at every sample on every ray, so a vanilla NeRF pays roughly $2 \times 10^{6}$ rays times 192 samples for one 1080p frame, which is hundreds of millions of queries. Hash-grid encodings shrink the per-query cost by orders of magnitude but keep the same per-ray structure. Rasterization instead touches each Gaussian once, splats it into the tiles it covers, and blends sorted contributions, which turns rendering into a bandwidth-bound pass rather than a compute-bound integral. That is the reason 3D-aware world models became practical after 2023 rather than after 2020.

Log-log line chart of milliseconds per frame versus output resolution in megapixels for three renderers: vanilla NeRF with 192 samples per ray at about 47000 milliseconds per megapixel, a hash-grid accelerated field at about 15 milliseconds per megapixel, and 3D Gaussian rasterization at about 3.7 milliseconds per megapixel, with horizontal reference lines at the 33 millisecond thirty frames per second budget and the 11 millisecond ninety frames per second budget, and an annotation marking that rasterization renders 1080p in roughly 8 milliseconds.

Figure 3: The geometric bias is only usable if you can afford to render. All three curves are linear in pixel count, so the gaps are constant multipliers: rasterization clears the 33 ms interactive budget at 1080p with room to spare, a hash-grid field sits near the edge, and vanilla volumetric marching is four orders of magnitude away, which restricts it to offline reconstruction rather than in-the-loop camera rollouts.

Property	2D latent video world model	NeRF-style radiance field	3D Gaussian splats
Scene representation	Latent tokens or 2D feature maps, no explicit geometry	Implicit continuous field returning density and colour	Explicit primitives with position, covariance, opacity, spherical harmonics
How the camera enters	Conditioning embedding, a soft constraint the decoder can bend	Defines ray origins and directions used for sampling	Enters the viewing transform and the projection Jacobian
Cost per 1080p frame	One pass of a large decoder, tens to hundreds of ms	Seconds for a vanilla MLP field, tens of ms with hash grids	Single rasterization pass, roughly 5-10 ms
View consistency	Statistical, drifts as the orbit lengthens	Exact for supported geometry	Exact for supported geometry, same compositing math
Dynamics	Native, learned directly in latent or pixel space	Time-conditioned field or canonical space plus deformation	Per-primitive deformation or 4D Gaussians
Dominant failure mode	Hallucinated geometry and object identity flicker on return views	Slow to fit and render, blurry thin structures, hard to edit	Floaters and popping in unobserved regions, memory grows with primitive count

August 19, 2026

DL0156 Force-Torque and Visual Fusion in VLA

How do force-torque sensor inputs and joint currents fuse with visual features inside VLA cross-attention layers to handle physical contact tasks?

Answer

Fusion begins with tokenization, not with attention: a 6-axis wrench sampled at 1 kHz and the per-joint motor currents are cut into short windows (typically 50-200 ms), pushed through a small 1D CNN or spectrogram encoder, pooled down to roughly one token per 10 ms, and projected by a learned matrix into the same $d$ -dimensional space that holds the visual patch tokens. Inside the block, the language and vision residual stream supplies the queries while the concatenated visual and proprioceptive tokens form the key/value memory, so softmax attention decides layer by layer how much contact evidence to read instead of averaging modalities with fixed weights. Because the pretrained VLM has never seen a wrench token, the cross-attention output is normally added through a zero-initialized tanh gate, which makes the new modality an exact no-op at initialization and lets it grow in during robot fine-tuning without destroying the language prior. The two streams also carry different information: vision says where and what, force says when and how hard, and force is the only channel that survives occlusion once the tool is inside the hole. What this fusion cannot do is close the contact loop, since one VLA forward pass costs 50-200 ms while an impact transient decays in a few milliseconds, so the policy emits setpoints, target wrenches, and impedance gains for a 1 kHz low-level controller that does the actual reacting.

(1) Window Then Tokenize: a 1 kHz stream cannot enter a 5 Hz backbone sample by sample, so it is windowed and pooled into about 10 tokens per 100 ms, which keeps the added sequence length near 2 percent of a two-camera observation.
(2) Shared Embedding Space: a per-modality encoder plus a linear projection is what makes wrench tokens, current tokens, and patch tokens legal keys in the same attention operation.
(3) Queries From The Stream, Keys From The Sensors: cross-attention is asymmetric on purpose, so the policy can ignore force during free-space motion and attend heavily to it during contact.
(4) Zero-Init Gating: $\tanh(\alpha)$ with $\alpha = 0$ preserves the pretrained VLM at step zero and is the standard fix for adding a modality that was absent from web-scale pretraining.
(5) Action Space Must Be Contact-Aware: position-only outputs cannot express compliance; the head has to emit a target wrench or stiffness and damping for an impedance or admittance law.
(6) Frequency Separation: the transformer supervises at 5-10 Hz and the 1 kHz controller reflexes, because policy latency is roughly 30 times longer than the impact transient it would need to catch.

Architecture diagram with three input lanes for the language instruction, wrist and scene cameras, and a 6-axis force-torque sensor with joint currents; each lane passes through its own encoder plus linear projection to the model width, the vision and force tokens form a shared key-value memory, the language and vision hidden states act as queries into a gated cross-attention block inside the pretrained VLM, and the policy emits an action chunk, a target wrench, and impedance gains

Figure 1: Only the encoders and projections are modality-specific. After projection, the wrench and current tokens are ordinary keys and values, and the zero-initialized gate controls how much of that memory reaches the residual stream. The head deliberately outputs gains and wrenches, not just poses, because a stiff position command in contact is how robots break parts.

Two practical details decide whether this works on hardware. The first is signal conditioning: a raw wrench includes tool gravity, payload inertia, and thermal bias drift, so the encoder must be fed a gravity-compensated and re-zeroed signal, otherwise the same physical contact produces different tokens after every payload change. Joint current is an even noisier torque proxy, since $\tau_j = k_t i_j$ holds only after subtracting harmonic-drive friction and stiction, which are velocity-dependent and hysteretic; current is therefore best used for coarse collision and jam detection across the whole arm rather than for fine wrench estimation at the tool. The second is bandwidth. Slip, chatter, and impact live in the 100-1000 Hz band, so pooling to 10 Hz destroys exactly the evidence you added the sensor for, which is why many implementations feed short-time spectral features or per-window statistics (peak, RMS, jerk) alongside the mean rather than a naive average.

Four stacked time plots over 400 milliseconds: the top shows a 1 kHz force-torque channel that jumps to 22 newtons at contact and rings down within about 4 milliseconds to an 8 newton steady value, the second shows camera frames as ticks every 100 milliseconds, the third shows VLA forward passes as 115 millisecond blocks repeating at 5 hertz, and the bottom shows the commanded stiffness as a staircase that only changes when a new policy output lands

Figure 2: The rate mismatch is the whole design constraint. The impact transient starts and finishes inside a single policy cycle, and the camera may not even sample it, so the VLA can only choose a compliance policy in advance and let the 1 kHz loop execute it. Anything that must react in under 10 ms cannot live in the transformer.

Mathematical Formulation:
$w_t = (f_x, f_y, f_z, \tau_x, \tau_y, \tau_z)$
$\tau_j = k_t i_j - \tau_{f}(\dot q_j)$
$U_{ft} = W_{ft}\, \phi_{ft}(w_{t-K+1:t})$
$M = [\, U_{vis} ; U_{ft} ; U_{cur} \,]$
$C = \mathrm{Attn}(h W_q,\; M W_k,\; M W_v)$
$h \leftarrow h + \tanh(\alpha)\, C$
$f_{cmd} = K(x_d - x) + D(\dot x_d - \dot x)$

Where:

$w_t \in \mathbb{R}^{6}$ is the wrench at time $t$ , with three forces $f$ and three moments $\tau$ expressed in the tool frame after gravity and payload compensation.
$i_j$ is the measured current of joint $j$ , $k_t$ the motor torque constant, and $\tau_f(\dot q_j)$ the friction and stiction term that makes current a biased torque estimate.
$\phi_{ft}$ is the window encoder over the last $K$ samples and $W_{ft}$ its projection to the backbone width $d$ ; $U_{ft}$ is the resulting small set of soft tokens.
$M$ is the concatenated key/value memory holding visual, force-torque, and joint-current tokens, and $h$ is the language and vision hidden state that provides queries through $W_q$ .
$\alpha$ is the learned scalar gate, initialized at $0$ so that $\tanh(\alpha) = 0$ and the block reduces to the pretrained VLM on the first step.
$K$ and $D$ are the commanded stiffness and damping matrices, $x_d$ and $\dot x_d$ the desired pose and velocity, and $f_{cmd}$ the wrench the 1 kHz controller actually applies.

Rate And Token Budget:
$K = 0.1 \times 1000 = 100$
$L_{ft} = 100 / 10 = 10$
$t_{vla} \approx 115\ \mathrm{ms}$
$t_{contact} \approx 4\ \mathrm{ms}$

A 100 ms window at 1 kHz gives 100 raw samples, pooled to 10 tokens, so about 100 force tokens per second ride alongside thousands of visual tokens. Meanwhile roughly 29 impact transients fit inside one policy forward pass, which is the quantitative reason the fusion is about situational awareness and gain scheduling rather than reflexes. The most common training pathology follows from the same asymmetry: proprioceptive inputs are low-dimensional and almost perfectly predict the next demonstrated action, so behaviour cloning happily learns to regress the previous command from state and stops looking at the camera. That is textbook causal confusion, and the usual defenses are proprioception dropout, delta actions instead of absolute targets, action chunking, and holdout episodes with the object moved.

Line chart of the share of cross-attention probability mass over normalized task progress for a peg insertion, with vision tokens starting near 0.62 and falling to about 0.30 during the shaded contact and insertion phase, force-torque tokens rising from 0.05 to about 0.56 across that same phase, and joint-current tokens staying below 0.18 throughout, with phase labels for reach, approach, contact plus insert, and retract

Figure 3: Illustrative attention-mass trace for a peg-insertion policy. During free-space reaching the wrench tokens are near zero and attract almost no mass, and once the peg is inside the hole vision is largely occluded so the wrench becomes the dominant evidence. The shares do not sum to one because language and history tokens hold the remainder.

Property	Early token concat (self-attention)	Gated cross-attention adapter	Hierarchical (force outside the model)
Where force enters	Appended to the input sequence, every layer sees it	As keys and values in an inserted cross-attention block	Only in the 1 kHz impedance loop, plus a scalar contact flag
Parameters touched	Full backbone fine-tune in practice	Encoder, projection, and adapter only; backbone can stay frozen	None; the controller is hand-designed
Robot data needed	Largest, since force was absent from pretraining	Moderate, the zero-init gate keeps the prior intact	Smallest, contact behaviour is not learned
Contact reaction latency	One policy cycle, 100-200 ms	One policy cycle, 100-200 ms	About 1-2 ms
Dominant failure mode	Proprioception shortcut and catastrophic forgetting of language grounding	Gate saturates near zero and the sensor is silently ignored	Cannot express task-dependent compliance or recover from a jam

August 19, 2026

DL0152 VLA Frequency Gap Bridge

How do you bridge the execution frequency gap between a low-frequency VLA policy running at 3 to 10 Hz, as in a pi-0 or GR00T N1 class model, and a high-frequency joint controller demanding 500 to 1000 Hz torque or impedance updates?

Answer

You never let the VLA command joints directly. The frequency gap is closed by a three-tier cascade in which each tier runs at its own clock: the VLA emits an action chunk covering hundreds of milliseconds of future motion instead of a single next action, a mid-rate head or buffer replays that chunk at 20 to 100 Hz, and a real-time layer upsamples the waypoints to the servo rate and closes an impedance or torque loop at 1 kHz against live encoder feedback. Chunking alone is not enough, because a blocking call stalls the controller for the whole inference window, so the chunk is computed asynchronously while the previous one is still executing, and the first few actions of the new chunk are frozen or blended to match what the robot has already committed to. Between waypoints you interpolate with a cubic or minimum-jerk segment rather than holding the last value, since a zero-order hold turns every chunk step into a velocity impulse the drives cannot follow. The real-time layer also owns safety: a watchdog that decays to gravity compensation when the buffer underruns, plus joint limits and torque saturation that the neural policy never sees.

(1) Chunks, Not Single Actions: the policy predicts $H$ future actions per forward pass, converting a 3 to 10 Hz decision rate into a continuous 20 to 100 Hz stream of targets.
(2) Horizon Must Cover Latency: the chunk has to be long enough to keep the buffer fed through one full inference plus transport delay, otherwise execution stutters at every boundary.
(3) Asynchronous Inference: issue the next forward pass while the current chunk still has actions left, so compute overlaps execution instead of interrupting it.
(4) Frozen Prefix And Blending: the first $d$ actions of a fresh chunk are already stale on arrival, so they are discarded or soft-constrained to the committed trajectory to avoid a jump at the splice.
(5) Interpolate, Never Zero-Order Hold: a spline between waypoints spreads each step over the $r$ controller ticks in between and keeps commanded velocity and acceleration bounded.
(6) The 1 kHz Layer Owns Safety: impedance gains, torque limits, and a buffer-underrun watchdog run on a real-time thread that never waits on a GPU.

Vertical three-tier diagram: a VLA backbone at 3 to 10 Hz with 100 to 300 ms inference passes features down to an action expert that emits a chunk of 50 joint targets at 50 Hz, which feeds a real-time layer performing cubic upsampling and a 1 kHz impedance law before reaching the robot joints, with a feedback path returning observations resampled to 5 Hz

Figure 1: Three clocks on one command path. Each tier only has to meet the deadline of the tier below it, so the 200 ms VLA period never appears as a 200 ms hole in the torque loop. Only the bottom tier is hard real time, and it is the only tier that reads encoders at 1 kHz.

The scheduling detail is what separates a demo from a deployed system. In the naive loop you observe, block on the network and the GPU for 100 to 300 ms, then execute the chunk, which means the controller spends a large fraction of every cycle replaying a stale target or holding still, and the robot visibly pauses at each boundary. Running inference asynchronously removes the hole but introduces a second problem: the chunk that arrives at time $t$ was conditioned on the observation from $t - t_{lat}$ , so its early actions describe a state the robot has already left. Real-time chunking handles this by treating the overlap as an inpainting constraint, keeping the first $d$ actions pinned to the trajectory already in flight and letting the sampler adjust only the free tail. The cheaper approximation used by ACT is temporal ensembling: keep every overlapping prediction for the current timestep and average them with weights $w_k = \exp(-mk)$ , which smooths the splice but adds no latency compensation and biases the command toward older observations.

Timing diagram with two lanes: the upper blocking lane alternates 130 ms inference bars with 200 ms execution blocks separated by red hatched 130 ms hold gaps, while the lower asynchronous lane overlaps inference bars with execution so that chunk blocks butt against each other with no gap

Figure 2: With a 130 ms forward pass and a 200 ms chunk, blocking inference leaves the controller starved for 130 ms out of every 330, roughly 39% dead time. Overlapping the next forward pass with the current execution removes the gaps entirely, and the price is that every chunk acts on an observation that is one cycle old, which is exactly what the frozen prefix compensates for.

Mathematical Formulation:
$T_{chunk} = H / f_a$
$d = \lceil f_a \, t_{lat} \rceil$
$H \geq d + \lceil f_a / f_{vla} \rceil$
$r = f_c / f_a$
$\tau = K_p (q_d - q) + K_d (\dot q_d - \dot q) + g(q)$

Where:

$T_{chunk}$ is the wall-clock horizon a single chunk covers, $H$ is the number of actions in the chunk, and $f_a$ is the rate at which those actions are consumed.
$t_{lat}$ is the end-to-end delay from shutter to first usable action, covering encoding, network transport, and the forward pass; $d$ is the resulting number of stale leading actions.
$f_{vla}$ is the policy replan rate, so $\lceil f_a / f_{vla} \rceil$ is how many actions are consumed per replan and the third relation is the no-underrun condition.
$f_c$ is the servo rate and $r$ the upsampling ratio, the number of interpolated setpoints emitted between two consecutive policy waypoints.
$\tau$ is the joint torque, $q_d$ and $\dot q_d$ the interpolated position and velocity setpoints, $q$ and $\dot q$ the measured state, and $g(q)$ the gravity term.
$K_p$ and $K_d$ set the mechanical impedance; low gains make the arm compliant and forgiving of a slightly wrong setpoint, high gains make it track hard and punish every command discontinuity.

Budget For A 5 Hz VLA On A 1 kHz Arm:
$d = \lceil 50 \times 0.13 \rceil = 7$
$\lceil f_a / f_{vla} \rceil = 50 / 5 = 10$
$H \geq 7 + 10 = 17$
$r = 1000 / 50 = 20$

With a 130 ms latency, a 50 Hz action rate, and a replan every 200 ms, the chunk needs at least 17 actions, and shipping $H = 50$ (a 1.0 s horizon) buys margin for a GPU hiccup or a dropped packet. The controller then produces 20 interpolated setpoints per waypoint, so the neural policy is responsible for shape and the real-time layer for smoothness. Note that a longer horizon is not free: everything past the next replan is open-loop motion, so the chunk length trades buffer robustness against reaction time to disturbances, and only the first $f_a / f_{vla}$ actions of a 50-action chunk are normally executed at all.

Two-panel chart: left panel plots a reference joint trajectory against a 5 Hz zero-order-hold staircase, a 5 Hz linear ramp, and a 50 Hz waypoint sequence cubically upsampled to 1 kHz; right panel is a log-scale bar chart of peak commanded joint velocity for the three command paths with a dashed joint velocity limit line

Figure 3: The same intended motion, three command paths. A zero-order hold asks for a finite position jump inside one 1 ms tick, so the implied velocity sits two orders of magnitude above the joint limit and the drive answers with a torque spike. Both interpolated paths stay under the limit, but only the 50 Hz waypoints actually reproduce the reference shape; upsampling a 5 Hz command smooths the command at the cost of cutting the corners of the trajectory.

Property	Blocking chunk + hold	Overlapping chunks + temporal ensembling	Async chunking + frozen prefix
Controller starvation	One dead window per cycle, about 39% duty loss at 130 ms latency	None if the ensemble buffer stays full	None, inference always overlaps execution
Boundary smoothness	Jump whenever the new chunk disagrees with the held target	Smooth, the exponential average filters the disagreement	Smooth by construction, the prefix is pinned to committed actions
Latency compensation	None, the whole chunk is stale by $t_{lat}$	None, and averaging biases toward older observations	Explicit, the first $d$ actions are skipped or constrained
Extra cost	Cheapest, one forward pass per executed chunk	Keeps several chunks in memory, needs a weighting hyperparameter	Needs a client-server split, a chunk buffer, and guided sampling
Reasonable use	Quasi-static pick and place, teleop replay, offline evaluation	On-board policies with low, stable latency	Remote or large models, dynamic tasks, anything with jittery latency

August 19, 2026

DL0151 Diffusion Policy and pi0 Flow Matching

How does Diffusion Policy generate continuous robot action chunks through denoising, and how does Physical Intelligence’s π0 instantiate the same idea as a flow-matching action expert on top of a pretrained vision-language backbone?

Answer

Neither model emits one action per forward pass. Both treat a whole chunk of $H$ future actions as a single high-dimensional sample from a conditional generative model, and they produce it by starting from Gaussian noise and running an iterative sampler conditioned on the current observation. Diffusion Policy does this with a DDPM: a 1D temporal U-Net (or a transformer variant) predicts the noise inside a noisy action sequence, and $K$ denoising steps turn $A^K \sim \mathcal{N}(0, I)$ into an executable chunk, of which only the first $T_a$ actions are executed before replanning. π0 keeps that output object and changes two things. The sampler becomes conditional flow matching along a straight noise-to-action path integrated with about 10 Euler steps, and the denoiser becomes a 300M-parameter action expert placed inside a PaliGemma 3B VLM as a second set of weights in one transformer. Images and language flow through the VLM weights, the robot state and the 50 noisy action tokens flow through the expert weights, and a single shared self-attention operation joins them, which is why the backbone is initialized from a VLM and fine-tuned rather than kept literally frozen.

(1) Chunks, Not Single Actions: the policy models $p(A_t \mid O_t)$ over an $H \times d$ matrix, which suppresses per-step jitter and makes long idle or contact phases survivable.
(2) Denoising Is The Policy: sampling replaces regression, so the network never has to collapse several valid demonstrated behaviors into their average.
(3) Multimodality Is Preserved: an MSE regressor asked to pass left or right of an obstacle outputs the mean of the two, which hits the obstacle; a denoiser draws one mode per rollout.
(4) Receding Horizon Closes The Loop: predict $H$ , execute $T_a \leq H$ , re-observe, resample. This is the only feedback mechanism the chunk has.
(5) Flow Matching Straightens The Path: a linear interpolation between noise and data gives an almost constant velocity field, so 10 integration steps suffice for a 50-step chunk at 50 Hz.
(6) Two Experts, One Attention: π0 routes tokens to modality-specific weights but keeps one attention operation, and the prefix KV cache is computed once per observation while only the small expert runs on every integration step.

Top row shows three line plots of two action dimensions over the fifty steps of a chunk, starting as pure Gaussian noise, then partially denoised, then a smooth executable trajectory, with denoise arrows between them. Bottom panel shows three overlapping horizontal bars representing chunks predicted at successive replanning times, each with a shaded leading segment marking the executed portion.

Figure 1: The sampler operates on the entire chunk at once, so temporal smoothness is a property of the generated sample rather than something enforced by a filter. At the bottom, only the leading $T_a$ actions of each chunk are executed, so the replanning period sets the reaction latency to anything the model did not anticipate.

The reason to pay for an iterative sampler is the shape of the demonstration data. Teleoperated demonstrations are multimodal and idle-heavy: the same scene is solved in several ways, and a maximum-likelihood Gaussian head trained with MSE returns the conditional mean, which is frequently not a valid action. Discretizing each dimension independently avoids averaging but breaks cross-dimension coordination, and a joint discretization is exponential in $d$ . Diffusion Policy’s published recipe uses observation horizon 2, prediction horizon 16, execution horizon 8, 100 DDPM training steps with 10 DDIM inference steps, FiLM conditioning of the observation embedding into a 1D temporal convolutional U-Net, and end-effector position control rather than velocity control, reporting an average 46.9% relative improvement over prior behavior-cloning baselines across 15 tasks.

Diffusion Policy (DDPM formulation):
$A_t = (a_t, a_{t+1}, \ldots, a_{t+H-1})$
$\hat{\epsilon} = \epsilon_{\theta}(O_t, A_t^k, k)$
$A_t^{k-1} = \alpha_k (A_t^k - \gamma_k \hat{\epsilon}) + \sigma_k z$
$\mathcal{L}_{\mathrm{DP}} = \mathbb{E}\|\epsilon - \hat{\epsilon}\|^2$

The same object, an $H \times d$ chunk, is what π0 produces, but the generative process is a continuous-time flow rather than a discrete Markov chain. Training samples a noise vector and a time $\tau \in [0,1]$ from a beta distribution that deliberately over-weights the noisy end of the path, forms the linear interpolant, and regresses the network onto the constant velocity that carries noise to data. Because the path is straight by construction, inference integrates with a fixed step $\delta = 0.1$ from $\tau = 0$ to $\tau = 1$ , which is 10 network evaluations for a chunk of 50 actions at 50 Hz. Cross-embodiment training is handled crudely and effectively: every state and action vector is zero-padded to the largest action dimension in the mixture (18 in the released model), and robots with fewer joints simply ignore the padded slots.

Architecture diagram with four input lanes for camera images, language instruction, robot state, and the noisy action chunk with its tau embedding, each passing through its own encoder into a single transformer stack that contains two weight sets: PaliGemma VLM weights for the prefix tokens and a 300 million parameter action expert for state and action tokens, joined by shared self-attention, producing a velocity field that is integrated by ten Euler steps into the final action chunk.

Figure 2: π0 is a mixture of two experts inside one transformer: the token type decides which weight matrices are applied, while attention is computed jointly over the whole sequence. Only the small expert is re-run per integration step, so the reported cost of a chunk is one 3B prefill plus ten passes over 300M parameters, roughly 73 ms in the released report.

Flow-matching action expert:
$A_t^{\tau} = \tau A_t + (1-\tau)\epsilon$
$u(A_t^{\tau} \mid A_t) = A_t - \epsilon$
$\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\|v_{\theta}(A_t^{\tau}, o_t) - u\|^2$
$A_t^{\tau+\delta} = A_t^{\tau} + \delta\, v_{\theta}(A_t^{\tau}, o_t)$
$H \Delta t = 50 \times 20\ \text{ms} = 1000\ \text{ms}$
$C = C_{\mathrm{VLM}} + 10\, C_{\mathrm{expert}}$

Where:

$A_t \in \mathbb{R}^{H \times d}$ is the action chunk starting at time $t$ , $a_i$ one action, $H$ the prediction horizon (16 in Diffusion Policy, 50 in π0), and $d$ the padded action dimension.
$O_t$ and $o_t$ are the conditioning observations: for Diffusion Policy a short stack of image features and proprioception, for π0 the image tokens, language tokens, and the state token.
$k \in \{K, \ldots, 1\}$ indexes discrete denoising steps and $\epsilon_{\theta}$ is the noise-prediction network; $\alpha_k, \gamma_k, \sigma_k$ come from the noise schedule and $z \sim \mathcal{N}(0, I)$ is the injected sampling noise.
$\tau \in [0,1]$ is the continuous flow time, $\epsilon \sim \mathcal{N}(0, I)$ the noise endpoint, and $A_t^{\tau}$ the linear interpolant between them.
$u$ is the target velocity field of the straight path, $v_{\theta}$ the action expert’s prediction of it, and $\delta = 0.1$ the Euler step, giving 10 evaluations per chunk.
$\Delta t = 20\ \text{ms}$ is the control period at 50 Hz, so one chunk covers one second; $C_{\mathrm{VLM}}$ and $C_{\mathrm{expert}}$ are the per-pass costs of the 3B prefix and the 300M expert.

Three by three block attention matrix with rows as query blocks and columns as key blocks. The prefix row attends only to the prefix column and is marked bidirectional and cached. The state row attends to prefix and state. The action row attends to all three blocks and is bidirectional within the chunk. The remaining upper right cells are hatched as masked.

Figure 3: The blockwise causal mask is what makes the cost structure possible. Because the image and text prefix never attends to the noisy action tokens, its keys and values do not depend on $\tau$ and stay valid across all 10 integration steps, while the action block attends bidirectionally within the chunk so every predicted timestep sees every other.

Property	Diffusion Policy (2023)	π0 (2024)
Generative process	Discrete-time DDPM, epsilon-prediction, cosine or squared-cosine schedule	Continuous-time conditional flow matching on the straight interpolant
Sampler steps	100 training steps, 10 DDIM steps at inference	10 forward Euler steps with fixed step 0.1
Denoiser	1D temporal U-Net with FiLM, or a small transformer variant	300M action expert sharing attention with PaliGemma 3B weights
Conditioning	ResNet image features plus proprioception, no language	Up to 3 camera views, natural-language instruction, state token
Chunk and rate	Predict 16, execute 8, typically about 10 Hz control	Predict 50 actions, one second of control at 50 Hz
Embodiment scope	One robot and task per trained policy	Cross-embodiment mixture, all vectors zero-padded to 18 dimensions
Dominant failure mode	No semantic generalization; the visual encoder is trained from a few hundred demos	Open-loop within a chunk, and prefill latency dominates the control budget

August 19, 2026

DL0150 Generative Multi-View World Model

How do generative world models generate realistic, high-resolution multi-view camera outputs (e.g., 8 ring cameras on a vehicle) with spatial and temporal coherence?

Answer

Modern driving world models are latent video diffusion models rather than per-camera image generators. A causal video tokenizer compresses every camera stream (roughly 16x spatially and 4x temporally), and one diffusion transformer denoises a single latent tensor that holds all 8 views and all frames of the clip simultaneously, so coherence is a property of the sampling process instead of a post-hoc fix. Spatial agreement between neighbouring cameras comes from two ingredients: cross-view attention inside every block, and camera-geometry conditioning that tells each token where its camera points, usually as per-pixel Plücker ray embeddings built from the intrinsics and extrinsics of that rig. Temporal agreement comes from a temporal attention axis plus a shared ego-action and agent-layout condition that all 8 views must obey, so a braking maneuver or a crossing pedestrian appears in every camera at the same instant. High resolution is bought with a cascade instead of brute force: the transformer works in a small latent grid, and a separate diffusion upsampler restores full per-view pixels, while long horizons come from autoregressive rollout over overlapping context frames.

(1) Latent, Not Pixel, Diffusion: a learned video tokenizer removes most spatial and temporal redundancy first, so the generator never denoises 8 megapixel-scale streams directly.
(2) One Joint Tensor Over Views And Time: the state being denoised is $z \in \mathbb{R}^{V \times T \times N \times c}$ , which is why views cannot drift into independent hallucinations.
(3) Factorized 4D Attention: spatial attention within a view-frame, cross-view attention across the ring at a fixed time, and temporal attention across time within a view, roughly 126x cheaper than full joint attention at 8 views and 16 frames.
(4) Geometry As A First-Class Condition: Plücker rays per pixel (plus optional epipolar-constrained attention) let one model serve rigs with different camera counts, mounting positions, and fields of view.
(5) Shared Layout And Action Control: 3D agent boxes, an HD map or BEV raster, and the ego trajectory are broadcast to every view, so all cameras render the same world state rather than eight plausible worlds.
(6) Cascade And Rollout: a super-resolution diffusion stage supplies pixels and an overlapped sliding window supplies duration, at the cost of error accumulation over long horizons.

Pipeline diagram: eight-view video enters a causal video tokenizer with 16x spatial and 4x temporal compression, producing a latent grid of V by T by N tokens that feeds a diffusion transformer of L blocks, then a latent decoder and a diffusion upsampler back to 448 by 960 per view; an expanded panel shows one block containing spatial self-attention within a view-frame, cross-view attention across the eight views at one time step, and temporal attention across frames of one view, with camera Plucker rays, ego action, agent layout, and noise level injected on the right

Figure 1: One tokenizer, one transformer, one upsampler. The only thing that makes this a multi-view model rather than eight video models is the cross-view attention axis and the shared conditioning bundle; the token budget of 215,040 latents per clip is what forces the attention to be factorized.

The cross-view axis is where naive designs fail, and geometry explains why. Ring cameras overlap only with their immediate neighbours, so attending from the rear camera to the front camera buys almost nothing while costing the same as a useful pair; restricting cross-view attention to ring-adjacent views keeps essentially all of the benefit. Inside an adjacent pair, a point visible in view $i$ cannot be anywhere in view $j$ : the fundamental matrix of the calibrated rig confines it to a single epipolar line, which is why epipolar-band attention is a legitimate sparsification rather than a heuristic. Conditioning carries the rest of the load. Because the ego action and the 3D agent boxes are shared, the front-left and front cameras are both being asked to render the same car at the same metric position, and the overlap region then agrees on colour, exposure, geometry, and occlusion because both views were denoised from the same layout under attention that could compare them at every layer.

Left panel: top-down view of a vehicle with eight camera field-of-view wedges arranged in a ring, labelled front, front-right, right, rear-right, rear, rear-left, left, and front-left, with a 15 degree overlap marked between neighbours and an octagon of lines connecting adjacent camera nodes to show which pairs exchange cross-view attention. Right panel: two image-plane rectangles for view i and view j, a marked point x in view i and the corresponding epipolar band drawn as a shaded stripe in view j, with the constraint that a matching token lies on that single line.

Figure 2: Coherence is a geometry problem before it is a modelling problem. Only ring-adjacent pairs share a field of view, and within a pair the epipolar constraint bounds where a matching token can live, so both the sparsity pattern and the conditioning signal are dictated by the calibrated rig.

Mathematical Formulation:
$z \in \mathbb{R}^{V \times T \times N \times c}$
$N = (H/p)(W/p) = 28 \times 60 = 1680$
$L = V T N = 8 \times 16 \times 1680 = 215040$
$\mathcal{L}(\theta) = \mathbb{E}\left[\lVert \epsilon - \epsilon_{\theta}(z_t, t, c) \rVert^2\right]$
$c = (r, a, b, z_{\mathrm{ctx}})$
$r_{uv} = (d_{uv},\ o \times d_{uv}) \in \mathbb{R}^{6}$

Where:

$z$ is the joint latent tensor being denoised, with $V=8$ views, $T$ latent frames, $N$ tokens per view-frame, and $c$ latent channels.
$H \times W = 448 \times 960$ is the per-view pixel resolution and $p = 16$ the combined tokenizer and patch downsampling factor, giving a $28 \times 60$ latent map.
$L$ is the total sequence length if the clip were flattened, the number that decides whether full attention is affordable.
$\epsilon$ is the sampled noise, $z_t$ the noised latent at level $t$ , and $\epsilon_{\theta}$ the diffusion transformer.
$c$ bundles the conditioning: per-view rays $r$ , ego action $a$ (speed and curvature), agent and map layout $b$ , and the overlapping context latents $z_{\mathrm{ctx}}$ from the previous window.
$r_{uv}$ is the Plücker embedding of the ray through pixel $(u,v)$ , with unit direction $d_{uv}$ and camera origin $o$ , so extrinsics and intrinsics enter as a 6-channel image instead of a flat vector.

Attention Cost Per Block:
$C_{\mathrm{full}} = L^2 = 4.62 \times 10^{10}$
$C_{\mathrm{spa}} = V T N^2 = 3.61 \times 10^{8}$
$C_{\mathrm{view}} = T N V^2 = 1.72 \times 10^{6}$
$C_{\mathrm{tmp}} = V N T^2 = 3.44 \times 10^{6}$
$C_{\mathrm{spa}} + C_{\mathrm{view}} + C_{\mathrm{tmp}} = 3.66 \times 10^{8}$

Factorization therefore costs about 0.8% of full joint attention at this shape, and almost all of the remaining cost is ordinary within-frame spatial attention. The cross-view and temporal axes together add under 1.5% on top of the spatial term, which is the practical reason multi-view coherence is affordable at all: the expensive part of the model is generating each image, not keeping the images consistent.

Log-scale line chart of attention pairs per transformer block against the number of camera views from 1 to 8, with 16 latent frames and 1680 tokens per view-frame. The full joint 4D attention curve rises quadratically from about 7.2e8 to 4.62e10 pairs, while the factorized spatial plus cross-view plus temporal curve rises nearly linearly from about 2.9e7 to 3.66e8 pairs, an annotation noting the roughly 126x gap at eight views.

Figure 3: Full joint attention over the flattened clip grows as $V^2$ while the factorized form grows close to linearly in $V$ , so the gap widens with every camera added to the rig. Adding a ninth or tenth view is a routine cost in the factorized design and prohibitive in the joint one.

Property	Independent per-view generation	Factorized view + time attention	Full joint 4D attention
Attention pairs per block	3.61e8 (spatial only)	3.66e8, about 1.4% overhead	4.62e10, roughly 126x more
Overlap agreement	None; only the shared layout condition, so seams disagree	Enforced on ring-adjacent pairs, optionally epipolar-banded	Enforced globally, mostly on pairs that never overlap
Temporal behaviour	Per-view only; the eight views desynchronize over rollout	Temporal axis plus one shared action keeps views in lockstep	Strongest in principle, but memory caps the clip length
Failure signature	Objects appear in one camera and vanish in its neighbour	Slow drift of far-field content and appearance across rollout	Out-of-memory or a clip too short to be useful for simulation
Practical verdict	Useless for BEV or occupancy training data	The production choice for ring-camera world models	Research ablation at small resolution or few frames

August 18, 2026

DL0149 World Models from Unlabeled Video

How do world models learn interactive, action-controllable environment rollouts from unlabeled video datasets without ground-truth action labels?

Answer

The missing action labels are invented by the model itself. A latent action model (LAM) is an inverse-dynamics encoder that looks at the past frames and the next frame together and emits a single discrete code $z_t$ through a vector-quantized bottleneck; a forward dynamics model is then required to predict that next frame from the past frames plus $z_t$ alone. Both halves are trained jointly with next-frame reconstruction as the only supervision, so the code is pushed to carry exactly the part of the transition that the past cannot explain, which for gameplay and egocentric video is precisely the agent’s action. Genie made this concrete at scale, training an 11B-parameter model on roughly 30,000 hours of filtered 2D platformer video with a codebook of only $K = 8$ latent actions, and the tiny codebook is not a detail but the mechanism: at 3 bits per step the code physically cannot smuggle the next frame through it. At inference the LAM encoder is thrown away, a key press is mapped onto one of the $K$ learned codes, and the dynamics model rolls the world forward autoregressively, which turns passive video into a playable environment.

(1) Latent Action Model: an inverse-dynamics encoder over $(x_{1:t}, x_{t+1})$ produces one discrete code per transition, standing in for the action label that the dataset never had.
(2) The Bottleneck Is The Whole Trick: a codebook of size $K = 8$ caps the code at $\log_2 8 = 3$ bits, forcing it to describe a control decision rather than pixels.
(3) Reconstruction Is The Only Loss: the forward model’s next-frame error trains the encoder through the quantizer, so no human ever annotates a button.
(4) Tokenize, Then Roll Out: a spatiotemporal VQ-VAE turns frames into tokens and a masked-token or diffusion dynamics model generates the next frame conditioned on the past tokens and $z_t$ .
(5) Codes Are Shared Across Videos: because one codebook serves the whole corpus, the same index means the same intent in unseen scenes, which is what makes the latent space a controller instead of a per-clip artifact.
(6) Grounding To Real Controls Is Cheap: a handful of labeled clips, or human probing of each code, suffices to attach semantics; VPT instead spent 2,000 labeled hours to train an inverse-dynamics model and pseudo-labeled 70,000 web hours with it.
(7) Rollouts Drift: generation is autoregressive on its own output, so compounding error, memory loss, and entangled camera-versus-agent motion dominate the failure list.

Two-panel diagram. The training panel shows past frames and the next frame both entering a latent action model with a VQ bottleneck of eight codes, producing a 3-bit code, while the past frames are separately tokenized by a spatiotemporal VQ-VAE; both streams enter a masked video-token dynamics model whose only loss is reconstruction of the next frame. The inference panel shows a key press selecting one of the eight codes without the latent action model, feeding the same dynamics weights, decoding and rendering a frame that loops back as the next context.

Figure 1: The asymmetry between the two panels is the point. During training the next frame is visible to the latent action model, which is why an unlabeled corpus can supervise a controller; at inference that encoder is discarded and the codebook itself becomes the input device, with the rendered frame fed back as context for the next step.

Everything hinges on how much information the quantizer lets through. If the latent is generous (a wide continuous vector, or a codebook of thousands of entries), the encoder discovers the shortcut of encoding the next frame directly, the dynamics model degenerates into a decoder of the latent, reconstruction looks excellent, and controllability collapses because a user-chosen code no longer corresponds to anything an agent could do. If the latent is too narrow, distinct moves collapse into one code and the environment responds to only a couple of coarse commands. The standard diagnostic is a controllability delta: roll the model forward with the inferred code, then with a random code, and measure the gap in frame quality. A large gap means the code is doing real work; a gap near zero means either leakage or a code the model ignores.

Mathematical Formulation:
$z_t = q(f_{\phi}(x_{1:t+1}))$
$\hat{x}_{t+1} = g_{\theta}(x_{1:t}, z_t)$
$\mathcal{L}(\phi,\theta) = \| x_{t+1} - \hat{x}_{t+1} \|_2^2$
$H(z_t) \leq \log_2 K$
$\Delta = \mathrm{PSNR}(z_t) - \mathrm{PSNR}(\tilde{z})$

Where:

$z_t$ is the latent action for the transition at step $t$ , and $q$ is vector quantization onto a codebook of $K$ entries.
$f_{\phi}$ is the inverse-dynamics encoder, which is the only component allowed to see the future frame, and $\phi$ are its parameters.
$x_{1:t}$ are the observed past frames (in practice their VQ tokens), $x_{t+1}$ is the true next frame, and $\hat{x}_{t+1}$ is the prediction.
$g_{\theta}$ is the forward dynamics model, a masked video-token transformer or a diffusion decoder, conditioned on the past plus $z_t$ and nothing else.
$\mathcal{L}$ is the reconstruction objective, extended in practice by the usual commitment and codebook terms; no action label appears in it, which is why unlabeled video is sufficient.
$H(z_t) \leq \log_2 K$ is the capacity bound that prevents frame leakage, giving 3 bits per step when $K = 8$ .
$\Delta$ is the controllability metric, comparing a rollout driven by the inferred code against one driven by a random code $\tilde{z}$ drawn from the same codebook.

Why 3 Bits Cannot Encode A Frame:
$\log_2 K = \log_2 8 = 3 \ \text{bits}$
$256 \times 10 = 2560 \ \text{bits per frame}$

A $16 \times 16$ token grid over a 1024-entry visual codebook already carries about 2,560 bits, roughly three orders of magnitude more than the action code, so the encoder has no route to cheat and the dynamics model must genuinely keep a model of the scene in its context. That ratio is the design knob you actually tune: too small and the environment is unresponsive, too large and it stops being an environment at all.

Schematic line chart with latent action capacity in bits per transition on the x-axis and normalized quality on the y-axis. Next-frame reconstruction quality rises monotonically and saturates, while controllability rises to a peak near three bits and then decays toward zero as capacity grows, with a shaded band marking the usable control regime between about two and five bits and a dashed vertical line at three bits labeled as an eight-entry codebook.

Figure 2: Schematic view of the tension that a single reconstruction loss hides. Reconstruction quality never warns you about failure because it keeps improving as the latent widens, whereas controllability peaks and then collapses once the code can carry the frame; only the two curves together tell you the bottleneck is set correctly.

Property	Latent action model (no labels)	Inverse-dynamics pseudo-labeling	Logged true actions
Label requirement	None during training; a few clips only to name the codes	A labeled seed set, such as VPT’s 2,000 contractor hours	Every frame paired with the true control input
Action space	Discovered discrete codes, semantics unknown a priori	The real action API, recovered by a trained predictor	The real action API, exactly
Data ceiling	Any internet video of the domain	Any video, but the seed set must match the domain	Only instrumented environments and agents
Dominant failure mode	Frame leakage through a wide latent, or codes entangling camera and agent motion	Pseudo-label noise on rare actions propagates into the dynamics	Coverage limited to the logging policy’s behaviour
Representative system	Genie, LAPO	VPT	GameNGen, Dreamer-style agents

August 18, 2026