DL0150 Generative Multi-View World Model

Modern driving world models are latent video diffusion models rather than per-camera image generators. A causal video tokenizer compresses every camera stream (roughly 16x spatially and 4x temporally), and one diffusion transformer denoises a single latent tensor that holds all 8 views and all frames of the clip simultaneously, so coherence is a property of the sampling process instead of a post-hoc fix. Spatial agreement between neighbouring cameras comes from two ingredients: cross-view attention inside every block, and camera-geometry conditioning that tells each token where its camera points, usually as per-pixel Plücker ray embeddings built from the intrinsics and extrinsics of that rig. Temporal agreement comes from a temporal attention axis plus a shared ego-action and agent-layout condition that all 8 views must obey, so a braking maneuver or a crossing pedestrian appears in every camera at the same instant. High resolution is bought with a cascade instead of brute force: the transformer works in a small latent grid, and a separate diffusion upsampler restores full per-view pixels, while long horizons come from autoregressive rollout over overlapping context frames.

(1) Latent, Not Pixel, Diffusion: a learned video tokenizer removes most spatial and temporal redundancy first, so the generator never denoises 8 megapixel-scale streams directly.
(2) One Joint Tensor Over Views And Time: the state being denoised is , which is why views cannot drift into independent hallucinations.
(3) Factorized 4D Attention: spatial attention within a view-frame, cross-view attention across the ring at a fixed time, and temporal attention across time within a view, roughly 126x cheaper than full joint attention at 8 views and 16 frames.
(4) Geometry As A First-Class Condition: Plücker rays per pixel (plus optional epipolar-constrained attention) let one model serve rigs with different camera counts, mounting positions, and fields of view.
(5) Shared Layout And Action Control: 3D agent boxes, an HD map or BEV raster, and the ego trajectory are broadcast to every view, so all cameras render the same world state rather than eight plausible worlds.
(6) Cascade And Rollout: a super-resolution diffusion stage supplies pixels and an overlapped sliding window supplies duration, at the cost of error accumulation over long horizons.

Figure 1: One tokenizer, one transformer, one upsampler. The only thing that makes this a multi-view model rather than eight video models is the cross-view attention axis and the shared conditioning bundle; the token budget of 215,040 latents per clip is what forces the attention to be factorized.

The cross-view axis is where naive designs fail, and geometry explains why. Ring cameras overlap only with their immediate neighbours, so attending from the rear camera to the front camera buys almost nothing while costing the same as a useful pair; restricting cross-view attention to ring-adjacent views keeps essentially all of the benefit. Inside an adjacent pair, a point visible in view cannot be anywhere in view : the fundamental matrix of the calibrated rig confines it to a single epipolar line, which is why epipolar-band attention is a legitimate sparsification rather than a heuristic. Conditioning carries the rest of the load. Because the ego action and the 3D agent boxes are shared, the front-left and front cameras are both being asked to render the same car at the same metric position, and the overlap region then agrees on colour, exposure, geometry, and occlusion because both views were denoised from the same layout under attention that could compare them at every layer.

Figure 2: Coherence is a geometry problem before it is a modelling problem. Only ring-adjacent pairs share a field of view, and within a pair the epipolar constraint bounds where a matching token can live, so both the sparsity pattern and the conditioning signal are dictated by the calibrated rig.

Mathematical Formulation:

Where:

• is the joint latent tensor being denoised, with views, latent frames, tokens per view-frame, and latent channels.
• is the per-view pixel resolution and the combined tokenizer and patch downsampling factor, giving a latent map.
• is the total sequence length if the clip were flattened, the number that decides whether full attention is affordable.
• is the sampled noise, the noised latent at level , and the diffusion transformer.
• bundles the conditioning: per-view rays , ego action (speed and curvature), agent and map layout , and the overlapping context latents from the previous window.
• is the Plücker embedding of the ray through pixel , with unit direction and camera origin , so extrinsics and intrinsics enter as a 6-channel image instead of a flat vector.

Attention Cost Per Block:

Factorization therefore costs about 0.8% of full joint attention at this shape, and almost all of the remaining cost is ordinary within-frame spatial attention. The cross-view and temporal axes together add under 1.5% on top of the spatial term, which is the practical reason multi-view coherence is affordable at all: the expensive part of the model is generating each image, not keeping the images consistent.

Figure 3: Full joint attention over the flattened clip grows as while the factorized form grows close to linearly in , so the gap widens with every camera added to the rig. Adding a ninth or tenth view is a routine cost in the factorized design and prohibitive in the joint one.