DL0123 3D-Aware Visual Pretraining

3D-aware visual pre-training trains a visual backbone or a video generator with objectives that can only be solved by recovering scene geometry: per-pixel pointmaps, metric depth, camera rays, and cross-view correspondence, instead of appearance statistics alone. The two ends it connects have complementary deficits. A 2D video diffusion model absorbs enormous visual and dynamic diversity from internet footage, but its geometry is implicit, so it has no queryable state: you cannot ask it for the distance to a mug in centimeters, and a long rollout quietly changes room layout because nothing anchors the scene. A physics simulator has the opposite profile, exact meshes, contacts, and metric scale, but its asset library and rendering diversity are tiny compared with the real world. Pre-training with geometric targets creates the shared interface: the same representation gives the generator a camera and depth channel it can be conditioned on, and gives the simulator a path to ingest real scenes as reconstructions, so the pipeline runs video → pointmaps → 3D Gaussians or mesh → simulator asset in one direction and simulator depth, pose, and contact labels back into the pre-training loss in the other.

(1) Geometry-Grounded Pretext Tasks: pointmap, ray, and depth regression plus multi-view matching force the backbone to encode where surfaces are, not just what they look like; ordinary video supplies the multiple views for free.
(2) Camera As A First-Class Input: Plücker ray embeddings or pose tokens turn a text-to-video model into a controllable renderer, which is exactly the interface a simulator viewport exposes.
(3) Persistent State Beats Frame Memory: caching an explicit 3D representation and re-projecting it into each new frame removes the layout drift and object-identity loss that pure 2D autoregressive rollouts accumulate.
(4) Two-Way Data Flow: real-to-sim reconstruction fills the simulator’s asset gap, while simulator renders provide perfect metric depth, pose, and contact supervision that no internet video can give.
(5) 3D Awareness Is Measurable: linear probes on frozen features show that self-supervised 2D backbones encode single-view depth surprisingly well yet fail at multi-view consistency, and 3D-aware fine-tuning lifts both plus downstream segmentation.
(6) Metric Scale Is The Robot Requirement: scale-invariant depth is enough for a nice-looking novel view and useless for a grasp, so metric anchoring is the part embodied policies actually consume.

Figure 1: One backbone, two supervision sources, two consumers. Unlabeled video supplies visual diversity and self-consistency signals, simulator renders supply exact depth and pose, and the geometry heads are what the video generator and the simulator both read from.

The concrete form most systems settle on is a feed-forward multi-view transformer that predicts a pointmap per frame in a common coordinate frame, so camera intrinsics, extrinsics, depth, and correspondence all fall out of one prediction rather than a fragile SfM plus MVS chain. DUSt3R established the pointmap formulation for two views, MASt3R added a matching head, and VGGT scaled it to many frames with attention alternating between frame-local and global blocks. That output is directly liftable: initialize 3D Gaussians from the points for a renderable scene, or run meshing plus material estimation for a body a rigid-body engine can collide against. Two caveats decide whether the bridge is load-bearing. First, monocular training recovers geometry only up to an unknown similarity transform, so metric scale has to be injected from calibrated stereo, a known camera baseline, simulator ground truth, or object-size priors. Second, the reprojection and correspondence losses assume a static scene, so dynamic content needs motion masks or a 4D formulation or the model folds object motion into depth as flying pixels.

Mathematical Formulation:

Where:

• is a pixel coordinate and the predicted scene point for that pixel, expressed in the reference camera’s frame.
• is the pinhole projection, the intrinsics, and the pose of camera ; primed symbols refer to a second view.
• indexes valid pixels, and is the ground-truth pointmap from a simulator, LiDAR, or a reconstruction pipeline.
• is the mean point norm used for scale normalization, so on is scale-invariant; dropping the division makes the loss metric and requires metric labels.
• is the reprojection error against a matched pixel in the other view, and it is valid only where the scene is static between the two frames.
• is a differentiable renderer (splatting or volume rendering) over a lifted representation , letting 2D images supervise 3D structure through .
• is the original appearance objective (diffusion denoising or masked prediction) and weight the geometric terms.

Figure 2: Joint multi-view geometry is what makes a representation 3D-aware, but attention over all views at once costs , so a 32-view window is roughly 1,000x a single view. This is why practical models alternate frame-local and global attention and slide a bounded view window.