DL0139 Multi-Camera Video VLM

Fusion is three separate problems solved in a fixed order: time alignment, per-view encoding, then spatial merging. Production rigs first put every camera on one clock (PTP or GPS-locked triggers) and stamp each frame with its own capture time, because software receive time is meaningless at 30 fps. Each view is then encoded independently by a weight-shared ViT and compressed to a small token set per frame, since the raw budget is multiplicative: 6 cameras at 8 frames of 256 tokens each is visual tokens before a single word of the prompt. The merge itself comes in two families. Token-level fusion concatenates all view tokens into one LLM context and tags each token with a camera-identity, extrinsic-pose, and timestamp embedding, letting attention discover the cross-view correspondence itself; this is the route taken by surround-view multimodal models such as Waymo’s EMMA. Geometric fusion instead defines 3D query points (BEV cells or object queries), projects each one into every camera using calibrated intrinsics and extrinsics warped by ego motion to a common query time, and merges the sampled features with per-camera weights, so a point seen by two overlapping cameras is fused once rather than described twice.

(1) One Clock Before Anything Else: hardware-triggered PTP or GPS sync plus a per-frame capture timestamp is the prerequisite; without it every downstream geometric step inherits an unknown offset.
(2) Shared Encoder Per View: one ViT with tied weights runs on all views, followed by token reduction (pooling, a Q-Former, or a perceiver resampler) to keep the context finite.
(3) Token-Level Fusion Is Implicit: concatenate everything and add camera-ID, pose, and time embeddings; flexible and calibration-tolerant, but the model must learn that two tokens describe the same object.
(4) Geometric Fusion Is Explicit: project each 3D query point into every camera and merge samples with confidence weights, which deduplicates the overlap region by construction.
(5) Asynchrony Is A Pose Problem: each view is warped by the ego transform between its own capture time and the query time , so odometry or IMU quality bounds fusion accuracy.
(6) Token Budget Dominates Cost: tokens grow as while self-attention grows as , so doubling the camera count quadruples attention FLOPs.
(7) Overlap Weighting Prevents Double Counting: weights derived from depth uncertainty and viewing angle favor the near, front-facing view over the oblique one.

The choice between the two families is not stylistic. Geometric fusion needs metric extrinsics, reasonable depth, and a rigid rig, and in exchange it gives a metrically consistent scene where 3D detection, tracking, and occupancy prediction are natural. Token-level fusion needs almost nothing beyond a camera index, survives loose or drifting calibration, and keeps the full appearance detail that a projection into a coarse BEV grid throws away, which matters for open-ended questions such as reading a sign visible in only one view. Most deployed stacks are hybrid: encode per view, warp features to a common timebase, lift to a shared 3D representation for the geometric heads, and pass a reduced token set of the same features into the language decoder for reasoning.

Figure 1: The pipeline is shared up to the fusion point: timestamped feeds → weight-tied ViT → token reduction, then either Route A (concatenate all tokens and let attention resolve the overlap) or Route B (project a 3D grid into every camera and merge with weights). Route A pays in context length, Route B pays in calibration and depth accuracy.

Mathematical Formulation:

Where:

• is the visual token count entering the decoder, with cameras, frames kept per camera, and tokens per frame after reduction.
• is a 3D query point in the ego frame (a BEV cell center or a learned object query) and is the same point expressed in camera ‘s frame at that camera’s own capture time.
• is the fixed extrinsic from ego to camera , and is the ego-motion transform from the query time to the capture time, obtained from wheel odometry, IMU, or visual odometry.
• is the intrinsic matrix, the perspective projection, the resulting pixel location, and the encoder feature sampled there by bilinear interpolation.
• is the visibility mask (point inside the frustum and unoccluded), the per-camera confidence weight from depth uncertainty and incidence angle, and a guard so uncovered cells stay zero instead of dividing by zero.
• is ego speed and the timestamp offset, so is the spatial error incurred by ignoring compensation; the numeric line uses m/s (60 km/h) and a 25 ms offset expressed in seconds.

Figure 2: Why geometric fusion handles overlap cleanly. Each BEV cell queries every camera whose frustum contains it, so a cell in the pairwise overlap receives two samples that are averaged with confidence weights instead of appearing as two separate objects. Cells outside every frustum stay explicitly empty, which token-level concatenation cannot represent.

Asynchrony bites hardest at the selection step. With free-running 30 fps sensors, the newest available frame from each camera can be up to one full period old, so a naive “latest frame per camera” gather can mix captures spread across 33 ms. At 60 km/h the rig moves 0.55 m in that window, which is larger than the BEV cell size most stacks use, and the symptom is a smeared or duplicated object in the overlap region rather than an obvious crash. Two fixes are standard: warp features per view with before sampling, and feed the residual offset to the model as a timestamp embedding so it can learn how much to trust a stale view. Hardware-triggered synchronous shutters remove the problem at the source and are worth the wiring cost on any rig that must produce metric output.

Figure 3: Free-running feeds mean the frames you gather at can span nearly a full frame period (left), and that spread converts directly into ego displacement (right). At highway speed even a 20 ms offset exceeds a typical 0.3 m fusion tolerance, which is why the ego-motion warp is not optional.