DL0158 3D-Aware World Model Geometry

A 3D-aware world model never lets a decoder invent pixels directly. Its latent transition predicts a scene state (a set of anisotropic 3D Gaussians, or the weights and features of a radiance field), and the only path from that state to an image is a differentiable renderer that takes the camera extrinsics and intrinsics as explicit arguments. Because every frame of a rollout is a deterministic function of one shared geometry, parallax, occlusion ordering, and disparity are produced by projection math rather than learned from data, so a camera can orbit for hundreds of steps without the scene mutating underneath it. NeRF contributed the volumetric rendering equation that made this pipeline differentiable end to end. 3D Gaussian Splatting keeps exactly the same alpha-compositing math but replaces per-ray marching with tile-based rasterization, which cuts per-frame cost by two to three orders of magnitude and is what makes closed-loop, interactive camera rollouts possible at all. The bias is not free: it guarantees consistency only where geometry is supported, so unobserved regions, topology changes, and strong view-dependent effects still require a generative prior layered on top.

(1) Render, Do Not Predict: the decoder is replaced by , so the camera pose enters as a projection matrix instead of a conditioning token the network may ignore.
(2) One State, Many Views: all views share a single geometry, which turns multi-view consistency from a learned behaviour into an architectural invariant up to the representation’s capacity.
(3) Two Renderers, One Equation: NeRF integrates density along rays while 3DGS rasterizes projected Gaussians, but both use the same front-to-back alpha compositing, so the geometric bias is identical and only the cost model differs.
(4) Dynamics Live In Scene Space: time is handled by a deformation field or 4D Gaussians that move primitives, so motion cannot silently destroy the identity of an object the way pixel-space prediction can.
(5) Latency Decides Feasibility: a world model must render inside the control loop, and rasterization at roughly 5-10 ms per 1080p frame versus seconds for vanilla volumetric marching is the difference between a usable simulator and an offline reconstruction.
(6) Where The Bias Stops Helping: unobserved regions, non-rigid topology change, transparency, and monocular scale ambiguity are outside the guarantee and are exactly where a generative prior must take over.

Figure 1: The same latent dynamics, two decoders. In the pixel-space path the pose is a soft conditioning signal the decoder may only approximately honour, so geometric errors compound across a long orbit. In the 3D-aware path the pose is an argument of an exact projection, so consistency holds by construction and the model’s remaining job is predicting geometry rather than predicting appearance.

The mechanism that actually enforces consistency is worth stating precisely. A pixel is a function of the primitives that project onto it, sorted by depth, so moving the camera changes the projected position of a primitive by an amount inversely proportional to its depth. That single fact gives the renderer correct parallax, correct occlusion ordering when a near primitive covers a far one, and correct disocclusion when the camera slides sideways, all without any of it appearing in the loss as a special term. Training only needs a photometric loss on the observed views; the geometry that explains several views simultaneously is the only geometry that can drive that loss to zero, which is why a well-fit scene state extrapolates to nearby unseen poses. A 2D video model has to learn the same relationships as correlations over pixels, and it has no mechanism preventing a later frame from contradicting an earlier one.

Figure 2: Why a shared 3D state is a hard constraint. Both renders come from one set of primitives, so the horizontal shift of each splat is fixed by its depth: the near splat moves exactly twice as far as the splat at twice the distance. A pixel-space decoder must reproduce this ratio from statistics, whereas the renderer cannot violate it, which is the entire content of the phrase geometric inductive bias.

Mathematical Formulation:

Shared Rendering Equation:

Where:

• is the compact latent state, the action, and the learned transition model; decodes the latent into a scene state rather than into pixels.
• is the geometric state: for 3DGS a set of primitives , for a NeRF a field returning density and colour at a queried point and direction.
• is the camera pose and intrinsics, and the differentiable renderer; is the frame, so any pose can be queried at any rollout step.
• is the colour of pixel , indexes the contributions sorted front to back, is the view-dependent colour, and is the accumulated transmittance that produces occlusion for free.
• is the volume density at sample and the spacing between adjacent samples along the ray; is a Gaussian’s opacity and its projected 2D density evaluated at the pixel.
• is the world-space covariance factored into rotation and scale so it stays positive semi-definite under gradient descent, and is its screen-space projection through the viewing transform and the affine Jacobian .
• is the predicted per-primitive displacement from a canonical configuration, which is how dynamics enter without breaking the static consistency guarantee at each instant.

The cost model is what selects the representation in practice. Volumetric marching evaluates a network at every sample on every ray, so a vanilla NeRF pays roughly rays times 192 samples for one 1080p frame, which is hundreds of millions of queries. Hash-grid encodings shrink the per-query cost by orders of magnitude but keep the same per-ray structure. Rasterization instead touches each Gaussian once, splats it into the tiles it covers, and blends sorted contributions, which turns rendering into a bandwidth-bound pass rather than a compute-bound integral. That is the reason 3D-aware world models became practical after 2023 rather than after 2020.

Figure 3: The geometric bias is only usable if you can afford to render. All three curves are linear in pixel count, so the gaps are constant multipliers: rasterization clears the 33 ms interactive budget at 1080p with room to spare, a hash-grid field sits near the edge, and vanilla volumetric marching is four orders of magnitude away, which restricts it to offline reconstruction rather than in-the-loop camera rollouts.