DL0192 Depth Anything 3 Cross-View Depth Estimation

Depth Anything 3 keeps the monocular recipe of its predecessors and changes essentially one thing inside the network: the scope of self-attention. Each of the input views is patchified by the same plain DINOv2 encoder, the per-view token sequences are concatenated into a single sequence, and the stack then alternates between within-view attention, where a query sees only its own view and fine monocular detail survives, and cross-view attention, where every token attends to every token of every view and correspondence, relative pose, and a common scale are learned. No fusion module, cross-attention adapter, or cost volume is introduced. The pretrained attention weights are simply given a wider window, so a single-view input degenerates exactly to the original monocular model, which is why multi-view capability does not cost single-image quality. The second change is the output: instead of the per-image affine-invariant disparity of Depth Anything V1 and V2, DA3 predicts a depth-ray target per view (a depth map plus a ray map), and depth along predicted rays back-projects to one point cloud and one set of camera poses under a single global scale. That one target replaces the multi-head, multi-task output of VGGT-style geometry transformers, and the reported gains over VGGT are roughly 44% on camera pose accuracy and 25% on geometric accuracy, while monocular depth still improves over DA2.

(1) Attention Scope, Not A New Module: cross-view interaction is implemented by concatenating view tokens and letting the existing self-attention layers run over the union, so no modality-specific or view-specific parameters are added.
(2) Interleaved Within-View And Cross-View Layers: within-view layers protect high-frequency monocular detail, cross-view layers enforce geometric agreement, and the two are alternated through the stack.
(3) Input-Adaptive Degeneration: at the cross-view layer is numerically identical to the single-view layer, so the model is a strict superset of its monocular predecessor rather than a compromise.
(4) Single Depth-Ray Target: one head predicts depth plus a ray map per view; depth along rays → point cloud, and camera pose is read out of the ray field instead of a dedicated pose head.
(5) One Global Scale For The Whole Set: normalization is fitted once over all views rather than a free scale and shift per image, which is precisely what removes per-frame flicker and non-overlapping point clouds.
(6) Plain Backbone, Teacher-Student Data: a vanilla DINOv2 transformer with a DPT-style dense head is enough; the accuracy comes from the target and from teacher-student pseudo-labelling, not from architectural specialisation.
(7) The Price Is Quadratic: a cross-view layer costs times a within-view layer, so view count, not image resolution, becomes the dominant memory term.

Figure 1: One encoder, two attention scopes, one target. The only cross-view machinery is the wider attention window, and the only output is a depth map plus a ray map per view, from which the point cloud and the camera poses are derived rather than predicted by separate heads.

It helps to look at the attention mask directly. With views of tokens each, a within-view layer is a block-diagonal mask: independent blocks, exactly what a monocular model computes, repeated in parallel. A cross-view layer fills in the off-diagonal blocks, and those off-diagonal entries are the whole mechanism, because a token on a wall corner in view 3 can now match the same corner in view 1 and inherit its depth ordering. Because the projection matrices are unchanged, the same weights serve both scopes and the model never has to learn a separate matching operator. The cost of filling those blocks is the reason view count dominates the budget, and it is also the reason non-overlapping views buy nothing: the off-diagonal blocks exist, but there is no correspondence for them to find, so the relative scale between two disjoint clusters of views stays unconstrained.

Figure 2: The same layer, two masks. Single-view models compute only the block diagonal; DA3 fills the off-diagonal blocks, and cross-view correspondence lives entirely there. The pair count grows from to , a factor of exactly .

Mathematical Formulation:

Where:

• holds the tokens of view and is the single concatenated sequence the transformer actually sees.
• are the projections restricted to one view, while are the same projections applied to all of ; the weights are shared, only the scope differs.
• indexes pixels, indexes views, is tokens per view for patch size , and is the model width.
• is the predicted depth and the predicted ray map (origin and unit direction) expressed in a shared frame, so is a 3D point in that frame and the camera pose follows from fitting .
• is a single global scale estimated jointly over all views and pixels, replacing the per-image scale and shift used by affine-invariant monocular training.
• and count attention token pairs per layer, so their ratio is and the KV cache of a cross-view layer grows linearly in .

Token Budget At 518 Pixels And Patch Size 14:

Thirty-two views at a modest resolution already put nearly 44k tokens in one sequence, and a single cross-view layer touches about 1.9 billion token pairs against 60 million for a within-view layer. This is why the interleaving ratio is a real design knob rather than a detail: every cross-view layer you insert buys consistency and pays times the attention cost, and it is why the practical deployment question for any-view geometry models is not accuracy but how many views fit on the device.

Figure 3: Consistency is not free. A within-view layer scales linearly in view count while a cross-view layer scales quadratically, and the two curves meet at , which is the formal statement of the input-adaptive property that keeps monocular quality intact.

The target change matters as much as the attention change. A per-image affine-invariant prediction is ambiguous by construction: two frames of the same room can be individually excellent and still disagree by a factor of two in scale, so stitching them produces a doubled wall. Fitting one scale over the whole view set turns depth from a per-image ranking problem into a set-level geometry problem, and the ray map supplies the missing piece by encoding where each pixel’s viewing ray points in the shared frame. Camera intrinsics and extrinsics then fall out of the ray field by a least-squares fit rather than from a separate pose head, which is the concrete sense in which DA3 collapses a multi-task output into a single one.