Category: Hard

DL0148 Action Chunking Transformer (ACT)

Explain the Action Chunking Transformer (ACT). How does predicting sequences of future action vectors ( $k$ -step horizons) solve temporal inconsistency and compounding error drift?

Answer

ACT is the imitation-learning policy introduced with the ALOHA bimanual setup: a transformer encoder-decoder that takes the current multi-view images plus the 14-dimensional joint state and emits, in a single forward pass, a whole chunk of $k=100$ future joint-position targets rather than one action. It is trained as a conditional VAE (CVAE) with an $L_1$ reconstruction loss on the chunk plus a KL term on a small style latent $z$ , which lets one deterministic-at-test-time policy absorb the multimodality of human teleoperation instead of averaging it away. Chunking attacks drift arithmetically: at 50 Hz a 20-second task is 1,000 control steps but only $1000/100 = 10$ chunk decisions, so the number of places where the policy can leave the training distribution shrinks by a factor of $k$ and the classic $O(\epsilon T^2)$ behaviour-cloning bound falls to roughly $O(\epsilon T^2 / k)$ . Chunking also removes per-step temporal inconsistency, because the actions inside a chunk are generated jointly from one latent and one observation, so the policy cannot flip between two valid modes on consecutive 20 ms steps. To avoid a visible discontinuity when a new chunk starts, ACT still queries the network every step and blends the overlapping predictions with temporal ensembling.

(1) The Chunk Is The Output Unit: the policy models $\pi(a_{t:t+k-1} \mid o_t)$ instead of $\pi(a_t \mid o_t)$ , so $k$ correlated actions are predicted together as one object.
(2) Effective Horizon Divided By $k$ : the agent makes $H = \lceil T/k \rceil$ sequential decisions, and compounding error grows with the number of decisions, not with the number of motor commands.
(3) CVAE For Multimodality: a BERT-style encoder compresses the demonstrated chunk into a 32-dimensional latent $z$ during training and is discarded at test time, where $z = 0$ gives one clean decisive mode.
(4) Non-Markovian Demos Stop Being Fatal: a single-step policy standing at a human pause sees a bimodal target (hold still or move) and can freeze forever; a chunk that contains the pause and the following motion resolves the ambiguity.
(5) Temporal Ensembling For Smoothness: re-query every step and average the overlapping chunk predictions with weights $w_i = \exp(-m i)$ , which keeps reaction latency at one control step while removing chunk-boundary jumps.
(6) $L_1$ Over $L_2$ : the $L_1$ loss on absolute joint targets produces sharper, less smeared trajectories, which matters for millimetre-scale contact tasks.

Architecturally there is nothing exotic. Each of the four RGB streams (two wrist cameras, two static cameras at 480×640) goes through a ResNet-18 whose final feature map is flattened into roughly 300 tokens, the joint vector becomes one more token, and the style latent becomes one more; a 4-layer transformer encoder with width 512 mixes them, and a 7-layer decoder attends to that memory from $k$ fixed learned position embeddings, one per future step, so the $k \times 14$ output is produced non-autoregressively in about 0.01 s. The whole policy is roughly 80M parameters trained from scratch per task on about 50 demonstrations, which is the interesting part: the gain does not come from scale or pretraining, it comes from changing what a single prediction means.

Architecture diagram of ACT: four RGB cameras feed per-camera ResNet-18 backbones producing about 300 tokens each, a joint-position token joins them, a 4-layer transformer encoder mixes the tokens, a 7-layer decoder with k learned queries emits a k by 14 action chunk, and above the main lane a training-only CVAE encoder maps the demonstrated action sequence and joint state to a 32-dimensional style latent z that is injected into the encoder and set to zero at test time

Figure 1: ACT is a CVAE whose decoder is a chunk predictor. The dashed lane exists only during training: it sees the ground-truth action sequence and squeezes the demonstrator’s stylistic choice into $z \in \mathbb{R}^{32}$ . At test time $z$ is fixed to the prior mean, so the same network becomes a deterministic policy that outputs $k \times 14$ absolute joint targets in one pass.

Mathematical Formulation:
$\pi_\theta(\hat a_{t:t+k-1} \mid o_t, q_t, z)$
$\mathcal{L}_1 = \sum_{j=0}^{k-1} \| \hat a_{t+j} - a_{t+j} \|_1$
$\mathcal{L} = \mathcal{L}_1 + \beta \, D_{\mathrm{KL}}(q_\phi \, \| \, \mathcal{N}(0, I))$

Where:

$\hat a_{t:t+k-1}$ is the predicted action chunk, here $k=100$ absolute target joint configurations in $\mathbb{R}^{14}$ for a bimanual 6-DoF-plus-gripper arm pair.
$o_t$ are the four camera images at time $t$ and $q_t$ the measured joint positions, together forming the only observation the chunk is conditioned on.
$z \in \mathbb{R}^{32}$ is the style latent, sampled from the encoder posterior $q_\phi(z \mid a_{t:t+k-1}, q_t)$ in training and set to $z = 0$ at inference.
$j \in \{0, \ldots, k-1\}$ indexes positions inside the chunk, each produced by its own learned decoder query.
$\beta$ weights the KL term; a large $\beta$ collapses $z$ to noise while a small one lets the encoder leak the answer and hurts test-time behaviour.
The $L_1$ norm is deliberate: it penalises large joint errors less quadratically than $L_2$ and therefore averages competing modes less aggressively.

The drift argument is worth writing down. Standard behaviour cloning with per-step error $\epsilon$ suffers covariate shift: a mistake moves the robot to a state the demonstrations never covered, the next prediction is worse, and the regret bound is quadratic in the horizon. Chunking does not make the policy immune, it reduces how many times the loop is closed. With $H = T/k$ decision points, each contributing error $\epsilon$ that persists over the remaining decisions, and each decision covering $k$ timesteps of cost, the accumulated term scales as $\epsilon H^2 k = \epsilon T^2 / k$ . The same factor cuts inference calls, which is why a chunked policy can afford a much heavier network per decision.

Horizon Arithmetic At 50 Hz:
$T = 20 \times 50 = 1000$
$H = \lceil 1000 / 100 \rceil = 10$
$\mathcal{E}_{\mathrm{BC}} = O(\epsilon T^2)$
$\mathcal{E}_{\mathrm{chunk}} = O(\epsilon T^2 / k)$

Semi-log chart of a compounding-error bound scale against episode length in control steps at 50 Hz, with three parabolic curves for chunk size k equal to 1, 10, and 100, showing the k equals 100 curve two orders of magnitude below the single-step curve, annotated with the point where a 20 second task of 1000 steps requires only 10 chunk decisions

Figure 2: The bound is still quadratic in task length, so chunking does not abolish drift, it buys two orders of magnitude at $k=100$ . The practical reading is that a long-horizon task becomes as hard as a short one only if the chunk itself remains executable open-loop, which is exactly the assumption that breaks when the scene moves during the 2 seconds a chunk spans.

Naive open-loop execution of the chunk creates a new problem: every 2 seconds a fresh observation produces a fresh chunk that need not start where the previous one ended, and the robot jerks. ACT therefore runs the policy at the full control rate and aggregates. At timestep $t$ there are up to $k$ different chunks that predict an action for $t$ , one from each of the previous $k$ queries, and the executed command is their exponentially weighted mean with $m = 0.01$ , where index $i = 0$ is the oldest prediction. That value of $m$ keeps the weights nearly uniform (1.00, 0.99, 0.98, …), so new observations are folded in immediately while the average stays smooth; a larger $m$ leans on the oldest chunk and reacts more slowly.

Temporal Ensembling:
$w_i = \exp(-m i)$
$\bar a_t = \sum_i w_i \hat a_t^{(i)} / \sum_i w_i$

Timeline diagram with four horizontal rows of six cells each, one row per chunk issued at timesteps 0, 1, 2 and 3, staggered so that all four rows cover timestep 4; the four cells covering timestep 4 are highlighted and labelled with exponential weights 1.00, 0.99, 0.98 and 0.97, and an arrow leads down to a box stating that the executed action at timestep 4 is the weighted mean of the four overlapping predictions

Figure 3: Chunks overlap because the policy is queried every step, so each control command is a vote of up to $k$ predictions made from $k$ different observations. This is what keeps ACT closed-loop at 50 Hz while its prediction horizon stays 2 seconds long, and it costs one extra forward pass per step rather than any extra training.

Property	Single-step BC (k = 1)	Chunk, open-loop execution	ACT: chunk + temporal ensembling
Policy output	One 14-d joint target	100 x 14 targets, all executed	100 x 14 targets, blended with earlier chunks
Queries per 20 s task	1,000	10	1,000 (each cheap, 0.01 s)
Decision points that can drift	1,000, bound scales as eps T^2	10, bound scales as eps T^2 / k	10 committed motions, continuously re-averaged
Reaction to a disturbance	20 ms	Up to 2 s of stale commands	20 ms, damped by the weighted mean
Human pauses in demos	Bimodal target, policy can freeze	Pause is inside the chunk, resolved	Same, plus no boundary discontinuity
Dominant failure	Jitter and mode switching between steps	Jerk at chunk boundaries, blindness mid-chunk	Averaging across modes can blur a decisive motion

August 18, 2026

DL0147 Multi-Task VLA Negative Transfer

How do multi-task VLA policies, such as the RT-X models trained on the Open X-Embodiment dataset, avoid negative task transfer when trained across heterogeneous robot arms, grippers, and kinematic chains?

Answer

Negative transfer in a cross-embodiment VLA is not a mysterious optimization pathology; it is the direct consequence of the same observation-instruction pair mapping to different correct action vectors on different robots, so the per-embodiment gradients on shared weights partially cancel. The fix is to decide, layer by layer, what genuinely transfers and what does not: vision-language grounding transfers and stays fully shared, while action semantics do not and are pushed into a canonical action space plus embodiment-specific output parameters. In practice that means four moves applied together: canonicalize actions into a common frame with per-dataset percentile normalization, condition the policy on an explicit embodiment token so the target is identifiable, decode through a per-embodiment head or a padded action expert so incompatible outputs never share a final linear layer, and control the data mixture so one 100k-episode domain does not dominate the average gradient. Only after those are in place is gradient surgery (PCGrad, CAGrad) worth its cost, because most measured conflict in early cross-embodiment runs comes from unnormalized, mis-framed action labels rather than from a real task disagreement.

(1) Shared Trunk, Split Output: the ViT plus language backbone sees every dataset, and all embodiment-specific incompatibility is confined to the last block, which is where the conflicting gradients would otherwise meet.
(2) Canonical Action Space: relative end-effector deltas in a fixed camera or base frame, with a normalized gripper channel, make “move 2 cm right” mean the same thing on a Franka and a WidowX.
(3) Per-Dimension Percentile Normalization: mapping each action dimension by its 1st and 99th dataset percentiles into $[-1,1]$ removes the scale mismatch between a 0.08 m gripper stroke and a 0.03 m one.
(4) Explicit Embodiment Conditioning: an embodiment id, proprioceptive state, and control-rate token make the label identifiable; without them the policy is asked to regress a multi-modal target from an ambiguous input.
(5) Mixture Weighting And Capacity: per-domain sampling caps stop large datasets from monopolizing updates, and added capacity (more parameters, or modality/embodiment-aware routing) converts interference into specialization.
(6) Measure Conflict Before Fixing It: log pairwise gradient cosine similarity between embodiment groups and per-domain validation loss, then apply projection methods only to the pairs that are genuinely negative.

The heterogeneity is concrete rather than abstract. A 7-DoF arm commanded in end-effector deltas at 3 Hz, a 6-DoF arm commanded in joint velocities at 5 Hz, and a 14-DoF bimanual rig commanded in absolute joint positions at 50 Hz produce label vectors of different dimension, unit, frame, and scale. Stack them into one tensor and regress with one output layer, and that layer must fit a multi-modal conditional distribution whose modes are mutually contradictory, so the mean-seeking L1 or MSE solution is a blurred command that satisfies nobody. The RT-X study made the asymmetry visible: the small 35M-parameter policy gained substantially on data-poor domains while losing to the original single-domain policies on some data-rich ones, whereas the far larger vision-language-backboned variant did not, which is the classic capacity-limited interference signature. Action chunking makes the stakes higher still, because a head predicting a horizon of $H$ steps outputs $H \times d_a$ numbers whose meaning is embodiment-dependent at every position.

Architecture diagram with three input boxes for RGB observations, the language instruction, and an embodiment id plus proprioceptive spec, all feeding one tall shared vision-language trunk, which produces a shared latent that fans out to three separate action heads for a 7-DoF arm at 3 Hz, a 14-DoF bimanual rig at 50 Hz, and an 8-DoF mobile manipulator, each emitting an action vector of different dimension

Figure 1: Where heterogeneity is absorbed. Everything to the left of the latent is shared across every robot and is exactly the part that benefits from pooled data; everything to the right is embodiment-specific, so a 14-DoF joint-position label and a 7-DoF end-effector delta label never contend for the same output weights. The embodiment id enters as a token in the trunk and also selects the head.

Mathematical Formulation:
$z = f_{\theta}(o_{1:n}, \ell, c_e)$
$\hat{a} = h_{\phi_e}(z)$
$\mathcal{L}(\theta, \phi) = \sum_{e=1}^{E} w_e \mathcal{L}_e$
$g_e = \nabla_{\theta} \mathcal{L}_e$
$\cos(g_i, g_j) = \frac{g_i^{\top} g_j}{\lVert g_i \rVert \lVert g_j \rVert}$
$g_i' = g_i - \frac{g_i^{\top} g_j}{\lVert g_j \rVert^2} g_j$

Where:

$z$ is the shared latent produced by the trunk $f_{\theta}$ from the camera views $o_{1:n}$ , the instruction $\ell$ , and the embodiment context $c_e$ (robot id, proprioceptive state, control rate).
$h_{\phi_e}$ is the head for embodiment group $e$ , with its own output dimension $d_a^{(e)}$ and chunk horizon, so $\hat{a} \in \mathbb{R}^{H \times d_a^{(e)}}$ .
$w_e$ are the mixture weights over $E$ embodiment groups, set by capped sampling rather than raw episode counts, and $\mathcal{L}_e$ is the behaviour-cloning loss on group $e$ .
$g_e$ is that group’s gradient with respect to the shared parameters only; head parameters $\phi_e$ receive gradient from one group and cannot conflict by construction.
The cosine is the diagnostic: a negative value means the two groups disagree about the shared update, and its magnitude tells you how much of each step is being cancelled.
$g_i'$ is the PCGrad projection of $g_i$ onto the normal plane of $g_j$ , applied only when the cosine is negative and symmetrically for $g_j'$ .

Per-Dimension Action Normalization:
$\tilde{a}^{(d)} = 2 \frac{a^{(d)} - q_{1}^{(d)}}{q_{99}^{(d)} - q_{1}^{(d)}} - 1$

Percentiles rather than min-max are used because teleoperation logs contain jitter spikes that would otherwise compress the useful range into a few percent of the interval. Note also what normalization cannot repair: it aligns scales, not semantics. If one dataset logs joint velocities and another logs end-effector deltas, both normalized to $[-1,1]$ , the shared head still sees two incompatible meanings for the same slot, which is precisely the residual conflict that per-embodiment decoding removes.

Two-panel vector diagram in the plane. Left panel shows a bimanual gradient pointing up and a single-arm gradient pointing down-left with an angle of about 117 degrees between them, and their sum drawn as a short vector, illustrating cancellation. Right panel shows the same two gradients as faint dashed arrows plus their PCGrad projections drawn solid, whose sum is a longer vector that still descends both losses

Figure 2: Negative transfer, geometrically. When two embodiment groups have gradient cosine near -0.45, their raw sum is shorter than either gradient, so the shared trunk barely moves while both per-domain losses stall. After projecting each gradient onto the other’s normal plane, the combined step is longer and its inner product with both original gradients stays positive, which is the formal statement of “descends both tasks”.

Choosing where the embodiment-specific parameters live is the main design decision, and three families are in use. A single padded head defines one action vector of maximum width and zero-pads unused dimensions, with the loss masked over the padding, which keeps the model monolithic and transfers well when the padded dimensions are physically comparable. Per-embodiment heads require no cross-robot alignment at all and are the most robust to genuinely different control interfaces, at the cost of one head per group and no head for an unseen robot. Latent action spaces learn a discrete or continuous code from video by inverse dynamics, train the policy in that code, and attach a small decoder per robot, which unlocks human video and action-free data but adds an entire quantization stage that can lose fine-grained precision.

Property	Single padded head	Per-embodiment heads	Latent action space
How heterogeneity is handled	One max-width action vector, unused slots zero-padded and masked in the loss	One output module per action space, no alignment across robots needed	Policy predicts a robot-agnostic code, small per-robot decoder maps it to motors
Negative-transfer risk	Moderate: slots must be semantically comparable or the head averages modes	Lowest at the output layer, residual conflict only in the shared trunk	Low if the code is truly embodiment-invariant, high if it leaks robot identity
Unseen robot at test time	Works if its dimensions fit the padded layout	Needs a new head plus a short fine-tune on target data	Needs only a cheap decoder, which is the main selling point
Extra cost	Wasted output width and careful mask bookkeeping	Linear growth in heads, and per-group data must be sufficient	A separate latent-action pretraining stage and possible precision loss
Representative systems	Flow-matching action experts with padded dimensions, unified-action diffusion policies	Readout-head generalist policies and cross-embodied transformers spanning manipulation and navigation	Latent-action pretraining from human and web video

August 18, 2026

DL0146 VLA Continuous Action Representation

How does a transformer decoder represent multi-modal continuous robot actions (e.g., 7-DoF arm pose, gripper state, base velocity) alongside language tokens, as in Google’s RT-2 and the OpenVLA models?

Answer

The decoder never sees a float. Heterogeneous degrees of freedom are first concatenated into one fixed-width vector (end-effector deltas in metres and radians, a gripper command, base velocities, and a discrete mode flag that says which sub-system is being driven this step), then each dimension is independently normalized by its own 1st and 99th percentile statistics so that metres, radians, and a binary gripper all live on $[-1, 1]$ . In the RT-2 and OpenVLA line each normalized dimension is uniformly discretized into 256 bins and the bin index is mapped onto reserved token ids, typically the 256 least-frequently-used entries of the existing SentencePiece or Llama vocabulary, so an action literally becomes a short string of tokens appended to the same causal sequence that holds the image patches and the instruction. Nothing about the architecture changes: the same causal self-attention, the same softmax, and the same cross-entropy loss that predict the next sub-word now predict the next joint delta, which is exactly what lets a VLA inherit internet-scale pretraining. The cost is a quantization floor plus one sequential decoder step per degree of freedom, which is why newer systems keep the shared prefix but replace the discrete head with a continuous action expert that emits a whole chunk of future actions through diffusion or flow matching in a single pass.

(1) One Flat Vector, Fixed Slots: arm, gripper, and base share a single $D$ -dimensional action vector with fixed slot semantics, and unused slots are zero-padded so one model can serve several embodiments.
(2) A Mode Dimension, Not Separate Heads: RT-1 carries an explicit mode variable that switches between controlling the arm, controlling the base, and terminating the episode, so mutually exclusive sub-spaces are disambiguated inside the same vector.
(3) Per-Dimension Quantile Normalization: statistics are computed per dimension and per dataset, using percentiles rather than min and max, because a handful of teleoperation jerks would otherwise consume most of the dynamic range.
(4) 256 Bins Onto Reserved Vocabulary Ids: the action alphabet is carved out of the language vocabulary, so no new embedding table, output head, or loss function is introduced.
(5) Autoregressive Factorization Across DoFs: the chain rule over dimensions lets the yaw bin depend on the already-emitted $x$ and $y$ bins, which is what keeps the joint action coherent rather than per-axis independent.
(6) Action Chunking And Continuous Heads: predicting $H$ future steps at once fights compounding error and raises the effective control rate, and a flow-matching expert removes the bin grid entirely at the price of extra weights and solver steps.

Architecture diagram showing camera frames, instruction text, and proprioception each tokenized into a shared prefix sequence feeding a decoder-only transformer, which branches into a discrete head that emits 256-bin indices as reserved vocabulary tokens and a continuous action expert that emits an H by D chunk by flow matching, both producing the same 11-dimensional action vector of six arm deltas, one gripper command, three base velocities, and one mode flag

Figure 1: The prefix is modality-agnostic; only the head differs. The discrete head reuses the language softmax and spends one sequential decoder step per degree of freedom, while the continuous action expert attends to the same prefix but emits an entire $H \times D$ chunk of real numbers. Both write into the identical fixed-slot action vector, including the mode flag that selects arm, base, or episode termination.

Normalization is where most of the real accuracy is won or lost, and it is the step engineers most often get wrong. Raw teleoperation logs are heavy-tailed: a few reset motions or dropped-controller frames produce end-effector deltas an order of magnitude larger than anything the policy needs at test time. If the bin edges are set by the observed minimum and maximum, those rare samples stretch the grid so far that the entire working range of the robot collapses into a few dozen of the 256 available levels, and the policy inherits a positional resolution floor of several millimetres that no amount of extra data can fix. Clipping to the 1st and 99th percentile before binning, as OpenVLA does, spends the full alphabet on the range the robot actually operates in and pushes the residual error below the mechanical repeatability of most arms. The same argument applies to the gripper, which is near-binary in the data, so it is usually thresholded into open or closed rather than treated as a smooth continuous axis.

Two histograms of the same end-effector delta-z distribution. The left panel bins over the raw min-max range, where rare large outliers stretch the grid and only a small fraction of the 256 bins are ever occupied. The right panel clips to the 1st and 99th percentile before binning, and nearly all 256 bins are occupied with a much smaller bin width in millimetres.

Figure 2: The same action dimension, two bin grids. Min-max binning lets rare teleoperation outliers dictate the grid, so most of the 256 levels are never used and the effective step size is coarse. Quantile binning throws away 2% of the samples and buys back an order of magnitude in resolution per bin, which is why percentile statistics are standard in production VLA data pipelines.

Mathematical Formulation:
$a_t = (a^{\mathrm{arm}}_t, a^{\mathrm{grip}}_t, a^{\mathrm{base}}_t, m_t)$
$\tilde{a}_i = \mathrm{clip}(a_i, q^{i}_{1}, q^{i}_{99})$
$u_i = 2 (\tilde{a}_i - q^{i}_{1}) / (q^{i}_{99} - q^{i}_{1}) - 1$
$b_i = \lfloor 255 (u_i + 1) / 2 \rfloor$
$\mathrm{id}_i = V - 1 - b_i$
$p(a_t \mid c) = \prod_{i=1}^{D} p(b_i \mid c, b_{1:i-1})$

Where:

$a_t$ is the action at control step $t$ , built from an arm delta pose (three translations and three rotations), a gripper command, a base velocity triple, and a discrete mode flag $m_t$ .
$a_i$ is the raw value of dimension $i$ and $\tilde{a}_i$ its clipped version; $q^{i}_{1}$ and $q^{i}_{99}$ are the per-dimension percentiles estimated on the training corpus.
$u_i \in [-1, 1]$ is the normalized value and $b_i \in \{0, \ldots, 255\}$ the bin index, so the grid holds 256 levels per dimension.
$V$ is the language vocabulary size, and $\mathrm{id}_i$ overwrites the 256 least-used token ids at the tail of that vocabulary.
$c$ is the prefix (image patch tokens, instruction sub-words, optional proprioception) and $i \in \{1, \ldots, D\}$ indexes the action dimensions in a fixed canonical order.
The product is the autoregressive factorization: each bin is conditioned on all previously emitted bins of the same action, so the joint distribution is not a product of independent marginals.

Resolution And Token Budget:
$\Delta_i = (q^{i}_{99} - q^{i}_{1}) / 256$
$\Delta = 0.1\ \mathrm{m} / 256 \approx 0.39\ \mathrm{mm}$
$N = H \times D = 8 \times 11 = 88$

Those two lines set the entire engineering trade-off. A translation axis clipped to $\pm 0.05$ m resolves to about 0.39 mm per bin, which is fine for pick and place and marginal for connector insertion, but an eight-step chunk of an 11-dimensional action costs 88 sequential decoder passes, which caps a 7B VLA at roughly 3 to 10 Hz on a single accelerator. The alternative keeps the shared prefix and attaches a small action expert whose tokens are continuous rather than discrete: starting from noise, it integrates a learned velocity field for a handful of solver steps and emits the whole chunk at once, which is how flow-matching policies reach 50 Hz dexterous control. Crucially, both the categorical head and the flow head are distributional, and that matters more than resolution: when two different behaviours are equally valid, a plain L2 or Gaussian regression head averages them into an action that belongs to neither mode.

$A_t = (a_t, a_{t+1}, \ldots, a_{t+H-1})$
$A^{\tau + \delta} = A^{\tau} + \delta\, v_{\theta}(A^{\tau}, \tau, c)$

Left panel shows a top-down workspace with an obstacle and two valid detour trajectories, one above and one below, plus a straight-line mean trajectory that drives into the obstacle. Right panel shows a bimodal action density with a 256-way softmax step approximation covering both peaks and a red dashed vertical line marking the L2 optimum located in the empty valley between the modes.

Figure 3: Why the head must be distributional. Two demonstrators route around the same obstacle in opposite directions, so the conditional action distribution is bimodal; a regression head minimizing L2 lands on the mean, which is a collision. A 256-way categorical over bins, or a diffusion or flow head over the chunk, keeps both peaks and samples one of them.

Property	Discrete bins as vocabulary tokens	Diffusion / flow action expert	Direct continuous regression
Output form	256-way categorical per DoF over reserved token ids	Real-valued chunk of shape H by D, denoised from noise	One real vector, the predicted mean action
Training objective	Cross-entropy, identical to language pretraining	Flow-matching or denoising regression on noised chunks	L2 or L1 on the action vector
Cost per chunk	H times D sequential decoder steps (88 at H=8, D=11)	One prefix pass plus about 10 solver steps of a small expert	One forward pass
Multimodal actions	Yes, per-DoF peaks are tied together by the chain rule	Yes, jointly over the whole chunk	No, collapses to the mean
Resolution	Bin width, about 0.39 mm on a 0.1 m range	Continuous, limited only by data noise	Continuous
Dominant failure mode	Quantization floor plus slow autoregressive decoding	Extra weights and solver steps, harder to co-train with text	Mode averaging produces an invalid middle action

August 18, 2026

DL0145 ITC, ITM, and MLM Pretraining Losses

Compare Image-Text Contrastive (ITC), Image-Text Matching (ITM), and Masked Language Modeling (MLM) losses used in foundational vision-language pre-training, as in models like OpenAI’s CLIP and Salesforce’s BLIP.

Answer

The three objectives differ along three axes: how much cross-modal interaction they permit, which parameters receive gradient, and what the pretrained model can actually do afterwards. ITC is a dual-encoder loss that compresses each image and each caption into a single L2-normalized vector and maximizes the cosine similarity of matched pairs against the other pairs in the batch, so it produces embeddings that can be precomputed and indexed, at the price of never letting a word look at a patch. ITM is a cross-encoder loss: image and text tokens are fused with cross-attention and a binary head decides match or no-match, which gives token-level discrimination but requires one joint forward pass per candidate pair, so it can only ever be a reranker. MLM masks about 15% of the text tokens and reconstructs them conditioned on the image, which is the objective that forces word-to-region grounding and trains the language head that downstream VQA and captioning heads reuse. CLIP trains ITC alone at batch 32,768; ALBEF and BLIP train all three at equal weight because each one supplies something the others cannot.

(1) ITC Learns Global Alignment: an InfoNCE objective over the $N \times N$ in-batch similarity matrix, with $N-1$ negatives per anchor and a learned temperature, optimized symmetrically image-to-text and text-to-image.
(2) ITM Learns Fine-Grained Verification: binary cross-entropy on the fused $\mathrm{[CLS]}$ state, and it is nearly useless unless the negatives are hard negatives mined from the ITC similarity distribution.
(3) MLM Learns Grounded Language: reconstructing masked words from surrounding text plus image evidence ties nouns and attributes to specific patches and yields a vocabulary-sized output head that pure ITC models do not have.
(4) Different Architectural Footprint: ITC updates only the two unimodal towers and their projections, while ITM and MLM update the cross-attention fusion layers; only ITC leaves behind vectors you can put in an ANN index.
(5) Different Compute Profile: ITC is one forward pass per modality but needs a huge batch or a momentum queue (65,536 in ALBEF), whereas ITM and MLM add fusion passes, so BLIP runs its text stack three times per step.
(6) They Compose Into A Pipeline: ITC → hard-negative mining → ITM rerank is the standard retrieval stack, and MLM is the branch that makes the same weights usable for generation-flavored tasks.

Three side-by-side computation graphs. The ITC panel shows separate image and text encoders feeding L2-normalized projections, then an N by N similarity matrix, then an InfoNCE loss. The ITM panel shows both encoders feeding cross-attention fusion layers, then a joint CLS hidden state, then a two-way match versus no-match head using hard negatives. The MLM panel shows the image encoder and a text stream with 15 percent of tokens masked feeding the same fusion layers, then hidden states at masked positions, then a softmax over a 30k word-piece vocabulary.

Figure 1: The same two towers, three different graphs. ITC stops at a single pooled vector per modality, so no gradient ever reaches a fusion layer; ITM and MLM share the cross-attention stack and differ only in the head placed on top, which is why adding MLM to an ITM model is cheap while adding either to a CLIP-style model requires new parameters.

The three losses are not interchangeable because they fail in different directions. An ITC-only model scores “a man in a red shirt riding a bike” and “a man in a blue shirt riding a bike” almost identically, since a single 512-dimensional vector cannot preserve every attribute binding; this is the classic bag-of-words behavior that compositionality benchmarks expose. ITM fixes exactly that case because the fused representation can compare the color word against the actual patches, but its cost is $O(NM)$ joint forward passes for a gallery of $N$ images and $M$ captions, which is why ALBEF and BLIP use ITC to shortlist $k$ candidates and only then run ITM. MLM contributes something neither of the other two does: because the target is a token id rather than a pair label, it forces the model to represent which region licenses which word, and it produces the token-level head that VQA fine-tuning reuses. The coupling runs the other way too, since the ITM negative sampler reads its distribution from the ITC matrix, so a weak ITC head starves ITM of informative negatives.

Left panel: an eight by eight grayscale similarity matrix for one batch, with bright diagonal cells outlined in blue as positive pairs and one bright off-diagonal cell at image four versus text seven outlined in red as the hardest negative. Right panel: a bar chart of the softmax distribution over the eight texts for image four, with a tall blue bar for the true caption, a nearly as tall red bar for the hard negative, and six short gray bars for the easy negatives.

Figure 2: One batch, two uses of one matrix. ITC consumes the whole row and the whole column as a softmax classification problem, while ITM samples a single off-diagonal entry with high similarity and feeds that pair to the fusion encoder labelled no-match. The six near-zero bars are the reason random negatives teach ITM almost nothing: the easy cases already carry negligible gradient.

Mathematical Formulation:
$s_{ij} = z_i^{\top} t_j / \tau$
$\mathcal{L}_{\mathrm{i2t}} = -\log \frac{\exp(s_{ii})}{\sum_{j} \exp(s_{ij})}$
$\mathcal{L}_{\mathrm{itc}} = \tfrac{1}{2}(\mathcal{L}_{\mathrm{i2t}} + \mathcal{L}_{\mathrm{t2i}})$
$\mathcal{L}_{\mathrm{itm}} = -\log p(y \mid h_{\mathrm{cls}})$
$\mathcal{L}_{\mathrm{mlm}} = -\sum_{m \in M} \log p(t_m \mid I, T_{\setminus M})$
$\mathcal{L} = \mathcal{L}_{\mathrm{itc}} + \mathcal{L}_{\mathrm{itm}} + \mathcal{L}_{\mathrm{mlm}}$

Where:

$z_i$ and $t_j$ are the L2-normalized projections of the image $i$ and caption $j$ taken from the two unimodal towers, so $z_i^{\top} t_j$ is a cosine similarity in $[-1, 1]$ .
$\tau$ is the learned temperature (CLIP initializes it at 0.07 and clamps it), which controls how sharply the softmax concentrates on the hardest negatives.
$i, j \in \{1, \ldots, N\}$ index the batch, and the diagonal $s_{ii}$ holds the positives; $\mathcal{L}_{\mathrm{t2i}}$ is the same expression with the softmax taken down the column.
$h_{\mathrm{cls}}$ is the fused $\mathrm{[CLS]}$ state after cross-attention and $y \in \{0, 1\}$ the match label, with the negative pair drawn from the ITC row rather than uniformly.
$M$ is the set of masked positions (about 15% of tokens), $T_{\setminus M}$ the unmasked context, and $t_m$ the true word piece at position $m$ out of roughly 30k classes.
The total loss in ALBEF and BLIP uses equal weights, and MLM replaces the plain text-encoder forward pass rather than adding an independent one.

Retrieval Cost For 1k Images And 5k Captions:
$N_{\mathrm{itc}} = 1000 + 5000 = 6000$
$N_{\mathrm{full}} = 5000 \times 1000 = 5000000$
$N_{\mathrm{rerank}} = 5000 \times 32 = 160000$

ITC needs 6,000 encoder passes total and then only dot products, which is what makes billion-scale ANN search possible. A pure cross-encoder needs 5,000,000 fusion passes on the same Flickr30k-sized benchmark, while reranking the top $k = 32$ ITC candidates needs 160,000, about 31 times cheaper than the exhaustive version and still recovering most of the accuracy gain. That arithmetic, not any statement about representational quality, is why production retrieval systems keep ITC in the serving path and treat ITM as an offline or second-stage model.

Property	ITC	ITM	MLM
Objective form	Softmax over in-batch similarities (InfoNCE)	Binary cross-entropy on a fused pair	Cross-entropy over a 30k word-piece vocabulary
Cross-modal interaction	None until the final dot product	Full token-to-patch cross-attention	Full cross-attention, per masked position
Parameters trained	Two towers plus linear projections	Towers plus fusion layers plus 2-way head	Towers plus fusion layers plus vocab head
Negatives	All other pairs in the batch or momentum queue	One or two hard negatives sampled from the ITC row	Implicit: every other word in the vocabulary
Batch-size sensitivity	High: 32,768 in CLIP, or a 65,536 queue in ALBEF	Low, but negative quality depends on batch diversity	Low, like ordinary text pretraining
Retrieval role	Indexable embeddings, first-stage ANN search	Second-stage reranker over top-k candidates	No pair score at all; helps only as auxiliary signal
Typical failure mode	Bag-of-words behavior, attribute and relation blindness, false negatives from duplicate captions	Saturates near 100% accuracy with easy negatives and stops learning	Bidirectional masking cannot generate text, so captioning needs a different head

August 18, 2026

DL0140 Temporal Visual Grounding

What are the challenges of temporal visual grounding (predicting the start and end timestamps of a language query inside a long, untrimmed video, as in video search features built on models like InternVideo2 or TimeChat)?

Answer

Temporal visual grounding (also called temporal sentence grounding or moment retrieval) maps one natural-language query onto a single interval $(s, e)$ inside an untrimmed video, and almost every difficulty follows from one mismatch: the metric is far more precise than the signal you can afford to look at. Benchmarks report R@1 at IoU 0.7 against a human-drawn span, yet the target moment is often a few percent of a video that runs for minutes, its boundaries are genuinely ambiguous (annotators disagree by seconds about when “opens the fridge” starts), supervision is a single positive span among thousands of candidate windows, and compute forces you to subsample frames so hard that the achievable temporal resolution can be coarser than the tolerance the metric grants. Worse, the standard datasets carry strong temporal location priors, so a model that never looks at the video can score respectably and mask the fact that no grounding is happening at all. Cross-modal features add their own problem, because image-level encoders describe objects well and describe verbs, ordering, and completion poorly, which is exactly the information a query like “right after he sits down” depends on.

(1) Tolerance Scales With The Moment, Not The Video: at $m = 0.7$ a correctly sized 4-second moment tolerates a rigid boundary shift of only 0.71 s, so short moments demand sub-second precision while long ones are almost free.
(2) Boundary Ambiguity Lives In The Labels: event onsets and offsets are not crisp, so inter-annotator IoU sits well below 1.0 and an L1 boundary regression fits annotation noise past a certain accuracy.
(3) Sampling Stride Trades Precision Against Cost: sampling at $f$ fps sets a hard error floor near $1/(2f)$ seconds, but keeping $f$ high makes $N = fT$ large and early cross-modal fusion costs $O(N^2)$ .
(4) Features Are Weak On Temporal Semantics: frozen CLIP-style per-frame embeddings encode appearance, so verbs, order, and “before/after” relations are nearly invisible without motion-aware or video-pretrained features.
(5) Location Priors Let Blind Models Win: on Charades-STA and ActivityNet Captions the ground-truth spans cluster in predictable places, so query-only or prior-only baselines are competitive and out-of-distribution splits expose the gap.
(6) Supervision Is Sparse And Assumes One Span: a single positive interval creates extreme foreground/background imbalance, and queries matching several disjoint intervals or none at all break single-span heads outright.

Line chart of the largest rigid boundary shift still counted correct versus ground-truth moment duration, for IoU thresholds 0.3, 0.5 and 0.7, with dotted horizontal lines marking the resolution floors of 1 fps and 0.5 fps sampling; short moments at threshold 0.7 fall below the 1 fps floor

Figure 1: The tolerance the metric grants is proportional to the moment duration, so a strict threshold on short moments asks for a precision that the frame sampling stride cannot deliver. Below the dotted lines, the demanded accuracy is finer than one sampled frame, and no amount of head tuning recovers it.

Mathematical Formulation:
$I = \max(0,\ \min(\hat{e}, e) - \max(\hat{s}, s))$
$U = \max(\hat{e}, e) - \min(\hat{s}, s)$
$\mathrm{IoU} = I / U$
$\delta \leq \frac{1 - m}{1 + m}\, L$
$\mathrm{Acc}_{m} = \frac{1}{Q} \sum_{q=1}^{Q} \mathbf{1}[\mathrm{IoU}_{q} \geq m]$
$\mathcal{L} = \lambda_{1} \lVert \hat{b} - b \rVert_{1} + \lambda_{2} (1 - \mathrm{IoU})$
$N = f\,T$

Where:

$(s, e)$ is the annotated span and $(\hat{s}, \hat{e})$ the prediction, both in seconds; $I$ and $U$ are the 1D intersection and union, and the $\max(0, \cdot)$ handles disjoint spans.
$L = e - s$ is the ground-truth duration, $m$ the IoU threshold, and $\delta$ the largest rigid shift of both boundaries that still passes, obtained from $(L - \delta)/(L + \delta) \geq m$ .
$q \in \{1, \ldots, Q\}$ indexes queries and $\mathrm{Acc}_{m}$ is the usual R@1, IoU=m metric, a hard indicator that is flat inside the tolerance band and gives no gradient-like signal about how close a miss was.
$b = (s, e)$ normalized by video length, $\lambda_{1}$ and $\lambda_{2}$ the loss weights; the L1 term is scale-sensitive and the IoU term is scale-free, which is why both appear in DETR-style grounding heads.
$T$ is the video duration, $f$ the sampling rate, and $N$ the number of frame tokens; a 20-minute video at $f = 1$ gives $N = 1200$ , and self-attention over it costs $O(N^2)$ per query.

The design space has moved through three families and is now entering a fourth: sliding-window proposal ranking (dense candidate spans scored against the query, as in 2D-TAN), proposal-free span prediction (per-frame start and end distributions, as in VSLNet), DETR-style set prediction (learnable moment queries plus Hungarian matching with an L1 and IoU loss, as in Moment-DETR and CG-DETR), and video LLMs that emit timestamps as text or as dedicated time tokens. Each family inherits the same structural problems in a different shape: proposal methods make the imbalance explicit and cap resolution at the window grid, span prediction is cheap but assumes exactly one contiguous answer, set prediction handles multiple moments and a no-moment class but needs enough data to learn the matching, and LLM-based grounders quantize time into a token vocabulary and therefore trade fine boundary precision for reasoning ability. Scaling to hour-long input is a separate axis, because early fusion concatenates query and video tokens and blows up memory with $N$ , while late fusion keeps the video encoding query-independent so features can be cached and reused across thousands of queries.

Pipeline diagram of a temporal grounding model: untrimmed video, frame sampling at f fps, frozen per-frame visual encoder, cross-modal encoder with quadratic attention, and a span head emitting start, end and score, with the query sentence and text encoder feeding into the cross-modal stage; a challenge callout sits under each stage, and a bottom timeline compares a ground-truth 16 second moment against a prediction shifted by 3.8 seconds giving IoU 0.62

Figure 2: Each pipeline stage contributes its own failure mode: the sampling stride fixes the resolution floor, the frozen per-frame encoder discards motion and ordering, early fusion makes cost quadratic in frame count, and a single-span head cannot express multiple or absent moments. The bottom timeline shows why the threshold choice dominates reported numbers: the same prediction passes at $m = 0.5$ and fails at $m = 0.7$ .

Property	Charades-STA	ActivityNet Captions	QVHighlights	MAD
Typical video length	about 30 s	about 120 s	150 s clips	full movies, about 110 min
Typical moment length	about 8 s	about 36 s	tens of seconds	about 4 s
Moment share of video	roughly a quarter	roughly a third	small but non-trivial	well under 0.1 percent
Moments per query	one	one	often several disjoint	one
Dominant difficulty	strong location prior, blind baselines score well	long vague spans, annotation bias	set prediction plus saliency, no-moment cases	extreme needle in a haystack, memory and caching

August 17, 2026

DL0139 Multi-Camera Video VLM

How do multi-camera video VLMs fuse visual inputs across asynchronous camera feeds with overlapping fields of view, as in Waymo’s surround-camera perception stack?

Answer

Fusion is three separate problems solved in a fixed order: time alignment, per-view encoding, then spatial merging. Production rigs first put every camera on one clock (PTP or GPS-locked triggers) and stamp each frame with its own capture time, because software receive time is meaningless at 30 fps. Each view is then encoded independently by a weight-shared ViT and compressed to a small token set per frame, since the raw budget is multiplicative: 6 cameras at 8 frames of 256 tokens each is $12{,}288$ visual tokens before a single word of the prompt. The merge itself comes in two families. Token-level fusion concatenates all view tokens into one LLM context and tags each token with a camera-identity, extrinsic-pose, and timestamp embedding, letting attention discover the cross-view correspondence itself; this is the route taken by surround-view multimodal models such as Waymo’s EMMA. Geometric fusion instead defines 3D query points (BEV cells or object queries), projects each one into every camera using calibrated intrinsics and extrinsics warped by ego motion to a common query time, and merges the sampled features with per-camera weights, so a point seen by two overlapping cameras is fused once rather than described twice.

(1) One Clock Before Anything Else: hardware-triggered PTP or GPS sync plus a per-frame capture timestamp is the prerequisite; without it every downstream geometric step inherits an unknown offset.
(2) Shared Encoder Per View: one ViT with tied weights runs on all $N$ views, followed by token reduction (pooling, a Q-Former, or a perceiver resampler) to keep the context finite.
(3) Token-Level Fusion Is Implicit: concatenate everything and add camera-ID, pose, and time embeddings; flexible and calibration-tolerant, but the model must learn that two tokens describe the same object.
(4) Geometric Fusion Is Explicit: project each 3D query point into every camera and merge samples with confidence weights, which deduplicates the overlap region by construction.
(5) Asynchrony Is A Pose Problem: each view is warped by the ego transform between its own capture time and the query time $t_q$ , so odometry or IMU quality bounds fusion accuracy.
(6) Token Budget Dominates Cost: tokens grow as $N \cdot T \cdot P$ while self-attention grows as $O(L^2)$ , so doubling the camera count quadruples attention FLOPs.
(7) Overlap Weighting Prevents Double Counting: weights derived from depth uncertainty and viewing angle favor the near, front-facing view over the oblique one.

The choice between the two families is not stylistic. Geometric fusion needs metric extrinsics, reasonable depth, and a rigid rig, and in exchange it gives a metrically consistent scene where 3D detection, tracking, and occupancy prediction are natural. Token-level fusion needs almost nothing beyond a camera index, survives loose or drifting calibration, and keeps the full appearance detail that a projection into a coarse BEV grid throws away, which matters for open-ended questions such as reading a sign visible in only one view. Most deployed stacks are hybrid: encode per view, warp features to a common timebase, lift to a shared 3D representation for the geometric heads, and pass a reduced token set of the same features into the language decoder for reasoning.

Top-to-bottom architecture diagram: three asynchronous camera feeds with different capture timestamps feed a shared ViT encoder with per-frame token reduction, which then branches into two fusion routes, token-level concatenation with camera-ID and pose embeddings on the left and geometric lifting into a shared BEV grid on the right, both feeding a vision-language decoder that emits text output

Figure 1: The pipeline is shared up to the fusion point: timestamped feeds → weight-tied ViT → token reduction, then either Route A (concatenate all tokens and let attention resolve the overlap) or Route B (project a 3D grid into every camera and merge with weights). Route A pays in context length, Route B pays in calibration and depth accuracy.

Mathematical Formulation:
$L_{vis} = N \cdot T \cdot P = 6 \cdot 8 \cdot 256 = 12288$
$\hat{X}_c = T_{c \leftarrow e}\, \Delta T(t_q, t_c)\, X_p$
$u_c = \pi(K_c\, \hat{X}_c)$
$B(p) = \frac{\sum_c m_c w_c\, F_c(u_c)}{\epsilon + \sum_c m_c w_c}$
$\Delta d = v\, |t_c - t_q|$
$\Delta d = 16.7 \cdot 0.025 = 0.42\ \text{m}$

Where:

$L_{vis}$ is the visual token count entering the decoder, with $N$ cameras, $T$ frames kept per camera, and $P$ tokens per frame after reduction.
$X_p$ is a 3D query point in the ego frame (a BEV cell center or a learned object query) and $\hat{X}_c$ is the same point expressed in camera $c$ ‘s frame at that camera’s own capture time.
$T_{c \leftarrow e}$ is the fixed extrinsic from ego to camera $c$ , and $\Delta T(t_q, t_c)$ is the ego-motion transform from the query time to the capture time, obtained from wheel odometry, IMU, or visual odometry.
$K_c$ is the intrinsic matrix, $\pi$ the perspective projection, $u_c$ the resulting pixel location, and $F_c(u_c)$ the encoder feature sampled there by bilinear interpolation.
$m_c \in \{0,1\}$ is the visibility mask (point inside the frustum and unoccluded), $w_c$ the per-camera confidence weight from depth uncertainty and incidence angle, and $\epsilon$ a guard so uncovered cells stay zero instead of dividing by zero.
$v$ is ego speed and $|t_c - t_q|$ the timestamp offset, so $\Delta d$ is the spatial error incurred by ignoring compensation; the numeric line uses $v = 16.7$ m/s (60 km/h) and a 25 ms offset expressed in seconds.

Top-down BEV diagram of an ego vehicle with three forward camera frustums drawn as shaded wedges that overlap pairwise, laid over a square BEV grid; one highlighted cell falls inside two wedges and is labeled as fused with two weights, another cell falls inside one wedge only, and a cell at the far left falls outside all wedges and is labeled as having no coverage

Figure 2: Why geometric fusion handles overlap cleanly. Each BEV cell queries every camera whose frustum contains it, so a cell in the pairwise overlap receives two samples that are averaged with confidence weights instead of appearing as two separate objects. Cells outside every frustum stay explicitly empty, which token-level concatenation cannot represent.

Asynchrony bites hardest at the selection step. With free-running 30 fps sensors, the newest available frame from each camera can be up to one full period old, so a naive “latest frame per camera” gather can mix captures spread across 33 ms. At 60 km/h the rig moves 0.55 m in that window, which is larger than the BEV cell size most stacks use, and the symptom is a smeared or duplicated object in the overlap region rather than an obvious crash. Two fixes are standard: warp features per view with $\Delta T(t_q, t_c)$ before sampling, and feed the residual offset to the model as a timestamp embedding so it can learn how much to trust a stale view. Hardware-triggered synchronous shutters remove the problem at the source and are worth the wiring cost on any rig that must produce metric output.

Left panel: timeline of four free-running 30 fps camera feeds with staggered capture phases, showing the newest frame available before a query time of 40 milliseconds for each camera and a double-headed arrow marking a 25 millisecond spread. Right panel: line chart of ego displacement in meters versus timestamp offset from 0 to 50 milliseconds for speeds of 10, 30, 60 and 100 kilometres per hour, with a dotted horizontal line at a 0.3 metre fusion tolerance

Figure 3: Free-running feeds mean the frames you gather at $t_q$ can span nearly a full frame period (left), and that spread converts directly into ego displacement (right). At highway speed even a 20 ms offset exceeds a typical 0.3 m fusion tolerance, which is why the ego-motion warp is not optional.

Property	Token-level concatenation	Dense BEV lifting	Sparse 3D queries
Shared representation	The LLM context itself, with camera-ID, pose and time embeddings	A metric BEV or voxel grid in the ego frame	A few hundred learned 3D queries with reference points
Overlap handling	Implicit; attention must learn that two tokens are one object	Explicit; one cell averages all cameras that see it	Explicit; one query attends to all views that contain its point
Asynchrony handling	Timestamp embedding only; residual error stays in the features	Ego-motion warp per view before sampling	Query propagation across frames with a motion-compensated pose
Calibration requirement	Low; a camera index and rough pose are enough	High; extrinsic drift of a degree visibly smears the grid	High for the projection, but errors stay local to each query
Cost scaling	Quadratic in N x T x P through self-attention	Linear in cameras, but grid resolution cubed in 3D	Linear in queries times cameras, cheapest at high N
Best for	Open-ended VQA, captioning, uncalibrated or ad hoc rigs	Occupancy, map and free-space prediction on a rigid rig	Streaming 3D detection and tracking under a latency budget

August 17, 2026

DL0138 Point-Informed Region VLM

Explain how Point-Informed or Region-Based Vision-Language Models incorporate bounding boxes or visual prompts directly into self-attention, as in Ferret and Molmo.

Answer

A region-based VLM has to turn a geometric object (a box, a point, a scribble) into something a transformer can consume, and there are only three places to put it: the token sequence, the positional encoding, or the attention logits. The cheapest route serializes the geometry as coordinate tokens, either quantized bins added to the vocabulary (Kosmos-2 uses 1024 location tokens over a 32×32 grid, written like <loc_0512>) or plain digit strings (Shikra), so the box reaches self-attention as ordinary keys and values with zero architecture change. The second route extracts region features with RoIAlign or a learned sampler and projects them into the language model’s embedding space as extra tokens interleaved into the prompt (GPT4RoI, Ferret), which every text token can attend to like a word. The third route never adds tokens at all: it adds a bias matrix $B$ to $QK^{\top}$ before the softmax, so patches inside the referred region get a boost and patches outside get suppressed or hard-masked to $-\infty$ . A fourth, architecture-free trick draws the prompt in pixel space (ViP-LLaVA’s overlaid arrows and circles, Set-of-Mark’s numbered masks) and lets the vision encoder’s own self-attention pick it up, which is the only option when the model is a frozen API. The choice is a trade among spatial precision, sequence growth, and how much grounding supervision you can afford to train on.

(1) Three Injection Points: geometry enters as tokens, as positional/coordinate embeddings, or as an additive term inside the attention logits, and most systems combine at least two.
(2) Coordinate Tokens Are Free But Quantized: a 1000-bin normalized grid on a 1024 px image gives roughly 1 px resolution, but the same bins on a 4K crop after tiling lose fidelity and shift under aspect-ratio padding.
(3) Region Tokens Buy Content, Not Just Location: a pooled RoIAlign vector carries appearance of the referent, so the model can describe a region it could not have isolated from coordinates alone.
(4) Attention Bias Adds Zero Parameters: $B_{ij}$ is added elementwise to the pre-softmax scores, leaving the $O(N^2 d)$ cost and the KV cache unchanged, and a per-head $\lambda_h$ can be learned.
(5) Hard Masks Destroy Context: setting $B_{ij} = -\infty$ outside the region removes exactly the surrounding evidence that relational questions need, and a fully masked row produces a degenerate softmax.
(6) Points Are Cheaper Supervision Than Boxes: Molmo’s pointing data shows a single $(x, y)$ pair is enough for counting and referring, and it avoids the box-regression ambiguity for deformable objects.
(7) Pixel-Space Prompts Work On Frozen Models: drawing the mark changes the image, not the architecture, but it occludes content and fails on thin or tiny structures.

Diagram of a vision-language model token sequence containing text tokens, quantized coordinate tokens, projected region-feature tokens and image patch tokens, all feeding a self-attention block whose pre-softmax logits receive an additive region bias matrix from the side, with a separate pixel-space visual prompt path feeding the patch tokens

Figure 1: Four ways a box reaches attention. Mechanisms (1) and (2) extend the token sequence so geometry becomes ordinary keys and values, mechanism (3) leaves the sequence untouched and edits the pre-softmax logits, and mechanism (4) modifies the pixels so the vision encoder does the work. Only (3) avoids sequence growth entirely.

The region-feature path is the one that most resembles classical detection heads: pool the backbone feature map over the box (RoIAlign → linear projector → language-model token), then concatenate a coordinate embedding so the model knows where the appearance came from, not only what it looks like. Ferret generalizes this with a spatial-aware visual sampler that samples and pools features inside an arbitrary free-form shape, so a point, a box, and a scribble all reduce to the same fixed-size token, and Groma pushes localization into the tokenizer itself by proposing regions up front and giving each one a referable ID token. The attention-bias path is complementary and is the literal reading of the question: because the softmax is invariant to nothing but its own inputs, a single additive term reweights how much probability mass a text query spends on patches inside versus outside the referent, and a moderate $\lambda_h$ concentrates mass without deleting surrounding context. In practice, training matters more than the plumbing: none of these mechanisms produces reliable grounding without a large corpus of region-text pairs, which is why grounded VLMs are trained on visual genome, RefCOCO-style referring expressions, and synthetically generated region captions.

Mathematical Formulation:
$e_r = W_v\, \mathrm{RoIAlign}(F, b) + W_c\, \psi(b)$
$\psi(b) = [\, x_1,\, y_1,\, x_2,\, y_2 \,] / S$
$Z^{(0)} = [\, E_{txt};\; e_r;\; E_{img} \,]$
$A = \mathrm{softmax}\!\left( \frac{QK^{\top}}{\sqrt{d_h}} + B \right)$
$B_{ij} = \lambda_h \, \mathbf{1}[\, j \in \mathcal{R}(i) \,]$
$\mathrm{cost} = O\big( (N + M)^2 d \big)$

Where:

$e_r$ is the region token appended to the prompt, and $Z^{(0)}$ is the full input sequence of text, region, and image-patch embeddings.
$F$ is the vision-encoder feature map, $b = (x_1, y_1, x_2, y_2)$ the box in pixels, $S$ the image side used for normalization, and $\psi(b)$ the normalized coordinate vector.
$W_v$ projects the pooled region feature into the language-model width and $W_c$ embeds the coordinates; both are the only new parameters this path needs.
$Q, K$ are the query and key projections of $Z$ , $d_h$ the per-head dimension, and $A$ the attention matrix over all $N + M$ positions.
$B$ is the additive region bias, $\mathcal{R}(i)$ the set of positions that query $i$ is encouraged to attend to (the patches overlapping the referred box), and $\lambda_h \geq 0$ the per-head bias strength; $\lambda_h = 0$ recovers plain attention and $\lambda_h \to \infty$ gives a hard mask.
$N$ is the number of visual plus text tokens, $M$ the number of added region or coordinate tokens, and $d$ the model width, so token-based injection is quadratic in $M$ while the bias term is free.

Three grayscale heatmaps of a text query's attention over an eight by eight patch grid, with a blue rectangle marking the referred region: the first with no bias shows diffuse mass spread across the image, the second with a soft additive bias concentrates mass inside the rectangle while keeping some outside, and the third with hard masking puts all mass inside the rectangle and exactly zero outside

Figure 2: The same query, the same keys, three values of $\lambda_h$ . A soft bias raises in-region mass from a small baseline to the majority while preserving outside context, which is what relational questions need; hard masking reaches 100% in-region mass and throws that context away.

Property	Coordinate tokens	Region-feature tokens	Attention bias / mask	Pixel-space prompt
Path into attention	New vocabulary or digit tokens as keys/values	Pooled RoI feature projected to an embedding	Added to pre-softmax logits	Altered patch embeddings
New parameters	Embedding rows only	Sampler plus two projections	None, or one scalar per head	None
Sequence growth	4 to 8 tokens per box	1 to 49 tokens per region	Zero	Zero
Precision limit	Bin width and tiling frame	RoIAlign grid and patch stride	Patch stride (14 or 16 px)	Stroke width and image resolution
Main failure mode	Coordinate frame drift after padding or crops	Token blowup with many regions	Loses outside context; needs a region per query	Occludes the referent; fails on small objects
Representative systems	Pix2Seq, Kosmos-2, Shikra, Qwen2.5-VL	GPT4RoI, Ferret, Groma	Region-conditioned decoders, SAM-style prompting	ViP-LLaVA, Set-of-Mark

August 17, 2026

DL0137 Early vs Late Fusion Multimodal

Compare Early Fusion vs. Late Fusion in multimodal models. What are the scaling limitations of early fusion architectures when scaling parameters past 100B?

Answer

The two designs differ in where the modalities meet. Late fusion (the modular VLM recipe behind BLIP-2, LLaVA, and Qwen2-VL) keeps a separately pretrained vision encoder, compresses its output through a small connector (an MLP projector, a Q-Former, or gated cross-attention), and feeds the resulting soft tokens into a pretrained text LLM, so nearly all capacity is inherited and only the connector plus a light fine-tune is trained. Early fusion (native multimodal, as in Fuyu-8B, Chameleon, and Llama 4) discretizes or linearly projects raw patches into the same token stream as text and runs one shared transformer from layer 0, so every layer attends across modalities and the model can emit image tokens as well as text. Early fusion is representationally stronger and, per recent scaling-law work, not worse per FLOP, so its problems past 100B parameters are systems and data problems rather than a representational ceiling. You cannot reuse a trained text-only 100B checkpoint, a Chinchilla-scale run now wants roughly 2T interleaved tokens that do not exist at text quality, the shared dense weights must be split between modalities, and every image inflates the sequence that attention pays $O(T^2)$ for.

(1) Meeting Point Defines The Family: late fusion joins modalities after a frozen encoder has already compressed the image; early fusion joins them at the token level, before any transformer layer.
(2) Checkpoint Reuse Is The Real Asymmetry: late fusion amortizes the trillions of text tokens already spent on the LLM, while a native 100B model pays that bill again on scarcer interleaved data.
(3) Modality Competition In Shared Weights: a dense early-fusion stack allocates the same parameters to pixels and text, so raising the image-token fraction $r$ costs text-benchmark quality and turns $r$ into a hyperparameter you cannot sweep cheaply at 100B.
(4) Token Budget Dominates The Context: one 512px image is about 1024 VQ tokens, so a four-image document spends 4096 tokens on pixels and quadratic attention absorbs the difference.
(5) Optimization Instability Grows With Width: heterogeneous token statistics drive logit growth and loss spikes, which is why Chameleon needed QK-Norm, reordered normalization, and dropout to keep a 34B run stable.
(6) Modularity Versus Capability: late fusion lets you swap the encoder or raise resolution for the price of a connector re-train, while early fusion buys interleaved any-to-any generation that a frozen tower cannot express.

Two-panel architecture diagram: the left panel shows late fusion where an image passes through a frozen ViT encoder and a trainable projector before entering a pretrained 100B LLM alongside text tokens; the right panel shows early fusion where image patches and text are tokenized into a single interleaved stream that a shared transformer trained from scratch consumes

Figure 1: Late fusion (left) is an assembly of pretrained parts, so the trainable surface is a projector of roughly 0.02B parameters and the alignment budget is on the order of 1B image-text tokens. Early fusion (right) is one homogeneous stack from random init, which is simpler to shard but means the whole 100B must be pretrained on interleaved data.

The scaling wall past 100B is mostly arithmetic. A dense 100B model trained compute-optimally wants about 2T tokens and roughly $1.2 \times 10^{24}$ FLOPs, and in native early fusion those tokens must be interleaved image-text documents; the supply of such data at web-text quality is far smaller than the text corpus, so teams either repeat data or dilute the text share and watch reasoning benchmarks regress. Late fusion sidesteps this entirely because the expensive part is already paid: aligning a connector on a couple of billion tokens with a frozen backbone is three orders of magnitude cheaper than a from-scratch native run at the same parameter count. On top of the data problem, dense early fusion suffers capacity contention, since gradients from visual reconstruction and from language modeling compete for the same MLP weights, and sparsity is the standard fix: a mixture-of-experts stack with modality-aware routing gives each modality its own parameters while keeping active FLOPs fixed, which is why recent native models are almost always sparse. Early fusion does win on the systems side (no separate vision tower to shard, no encoder-LLM pipeline bubble, and a single tokenizer path), and it is the only option when the target task requires interleaved generation or genuinely fine-grained grounding that a pooled 576-token projection has already destroyed.

Mathematical Formulation:
$h_v = W_p\, E_v(I)$
$y = \mathrm{LLM}_{\phi}([\,h_v;\, E_t(x)\,])$
$y = T_{\theta}([\,z_1, z_2, \ldots, z_T\,])$
$C \approx 6ND$
$D \approx 20N$
$N = 10^{11} \Rightarrow C \approx 1.2 \times 10^{24}$
$T = N_{img} + N_{txt}$
$C_{attn} = \Theta(L\, T^{2}\, d)$
$D_{txt} = (1 - r)\, D$

Where:

$y$ is the generated output; the first two equations are the late-fusion path and the third is the early-fusion path, where a single stack $T_{\theta}$ consumes the whole interleaved sequence.
$I$ is the image and $x$ the text; $E_v$ is the pretrained vision encoder, $E_t$ the text embedding, and $W_p$ the connector, typically the only trained matrix during alignment.
$z_t$ for $t \in \{1,\ldots,T\}$ are the interleaved tokens, either VQ image codes or linear patch projections placed in the same sequence as text tokens.
$N$ is the parameter count, $D$ the training tokens, and $C$ the training FLOPs; $D \approx 20N$ is the Chinchilla compute-optimal ratio, which for $N = 10^{11}$ demands about 2T tokens.
$L$ is the layer count and $d$ the model width, so $C_{attn}$ shows that image tokens entering the shared stack are charged quadratically, not linearly.
$r \in [0,1]$ is the image-token fraction of the pretraining mixture, so $D_{txt}$ is the surviving text budget; at fixed $C$ , every point of $r$ is taken directly out of language modeling.

Two-panel chart: left panel is a log-log plot of training FLOPs versus parameter count comparing native early fusion trained from scratch at 120 N squared against late-fusion connector alignment at four N times two billion tokens, with the gap at one hundred billion parameters annotated; right panel is a log-scale bar chart of image tokens per image for pooled projectors, SigLIP 384, Chameleon VQ, Qwen2-VL dynamic resolution, and native 1024 pixel patches

Figure 2: Left: from-scratch native training scales as $120N^{2}$ under the compute-optimal token rule, while connector alignment scales linearly in $N$ , so the gap at 100B is roughly three orders of magnitude. Right: the number of tokens an image consumes spans about an order of magnitude across tokenizers, and in early fusion every one of those tokens enters the shared quadratic attention rather than being pooled away first.

Property	Early fusion (native)	Late fusion (modular)
Fusion point	Token level, before layer 1; all layers are cross-modal	After a pretrained encoder, via a projector or cross-attention
Trainable surface	Every parameter, from random init	Connector plus optional LLM fine-tune; encoder often frozen
Data requirement at 100B	About 2T interleaved tokens, a corpus that barely exists at text quality	Order 1B to 10B image-text pairs plus instruction data
Dominant failure mode	Modality competition for dense capacity, text regression, loss spikes	Information already discarded by the frozen encoder (OCR, small objects, counting)
Upgrade path	Resolution or tokenizer changes touch the pretraining recipe	Swap the encoder or the LLM and re-align the connector
Generation ability	Interleaved any-to-any output, since image tokens are in the vocabulary	Text out only, unless a separate image decoder is bolted on
Representative systems	Fuyu-8B, Chameleon, Transfusion, Llama 4	Flamingo, BLIP-2, LLaVA-1.5, Qwen2-VL, InternVL

August 17, 2026

DL0136 Unified Multimodal Gemini Architecture

How do unified multi-modal models such as Google’s Gemini handle simultaneous natively tokenized audio, vision, and text input streams?

Answer

A unified model does not bolt encoders onto a finished language model at inference time; it is pretrained from step zero on interleaved sequences in which every modality has already been converted into tokens that live in the same $d$ -dimensional embedding space. Text passes through a SentencePiece vocabulary, images and sampled video frames through a ViT-style patch encoder, and audio through a speech encoder in the Universal Speech Model (USM) lineage operating on 16 kHz waveforms; each front end emits discrete ids or continuous soft tokens that are projected to the model width and concatenated in timestamp order into one causal sequence. From that point there is no separate fusion module: ordinary self-attention in every layer is the fusion mechanism, so a text token attends to an audio window from second 12 and to the patch tokens of the frame shown during that second at the same cost as attending to another word. The consequences of native tokenization are therefore budgetary rather than architectural, because Gemini’s published token rates are roughly 258 tokens per image tile or sampled frame and 32 tokens per second of audio, so one hour of video with its soundtrack consumes about 1M tokens and long context stops being a feature and becomes a precondition.

(1) Native Tokenization, Not An Adapter: multimodal data is present in pretraining from the first step, so the shared representation is learned jointly rather than stitched together by a projector trained on top of two frozen towers.
(2) One Shared Embedding Space: a per-modality encoder plus a linear projection maps sub-words, patches, and audio frames into the same $\mathbb{R}^{d}$ , which is what makes concatenation legal.
(3) Self-Attention Does The Cross-Modal Work: there is no cross-attention adapter per modality, and the price is a $O(L^2 d)$ prefill over the combined length.
(4) Timestamp-Ordered Packing: tokens from the same second of an audio-visual clip are placed adjacent, so co-occurring events are a few hundred positions apart instead of hundreds of thousands.
(5) Token Rates Set The Budget: 258 tokens per frame, 32 tokens per audio second, and 1 fps frame sampling as the lossy compression knob that decides what the model can even see.
(6) The Output Side Can Be Multimodal Too: a single next-token head over a vocabulary extended with discrete image and audio codec tokens lets one decoder emit text, pixels, or speech without a separate generator.

Architecture diagram with three input lanes for text, image or video, and audio, each passing through its own tokenizer or encoder and a linear projection into a shared d-dimensional space, then merging into one interleaved token sequence in timestamp order that feeds a single decoder-only transformer with full self-attention, which emits text tokens, discrete image tokens, and audio codec tokens

Figure 1: Three front ends, one sequence, one stack. The only modality-specific parameters sit in the encoders and projections; after that the tokens are indistinguishable to the transformer, and the output modality is decided purely by which token ids the single next-token head emits.

Two design decisions do most of the work. The first is early fusion: because all three streams enter the same stack as tokens, cross-modal alignment is learned by the same attention weights that learn syntax, which is what lets the model join a spoken sentence to whatever was on screen while it was said. A late-fusion alternative, where a frozen vision tower is glued to a frozen LLM through a projector or Flamingo-style cross-attention adapter, is far cheaper to train but confines modality interaction to the adapter, while a cascade (audio → ASR → LLM → TTS) discards everything the transcript does not carry, including speaker identity, prosody, laughter, and overlapping non-speech events. The second decision is packing order. Placing the frame tokens and the audio tokens of the same second next to each other keeps relative position encodings informative and keeps local attention patterns useful, whereas a modality-blocked layout pushes a frame and its own soundtrack thousands of positions apart. Frame rate is then the compression knob: 1 fps is adequate for scene-level questions but structurally unable to represent a 200 ms gesture, a single flashed frame, or the exact moment a door closes.

Two token-sequence strips compared: the top strip alternates a wide block of 258 image tokens with a thin slice of 32 audio tokens for each of four consecutive seconds in timestamp order, and the bottom strip places all four image blocks first followed by all four audio slices, with a double-headed arrow showing that frame zero and its own audio are now 1032 tokens apart

Figure 2: The same tokens, two layouts. Time-interleaved packing keeps each frame beside the 32 audio tokens recorded during it, while modality-blocked packing separates them by 1,032 positions, which is why audio-visual grounding degrades even though attention can technically still reach across the whole sequence. The width ratio also shows the real cost structure: vision dominates the budget roughly 8 to 1 over audio.

Mathematical Formulation:
$u_i = W_m \phi_m(x_i)$
$S = (u_1, u_2, \ldots, u_L)$
$L = L_{\mathrm{txt}} + 258 N_f + 32 T_a$
$\mathrm{prefill} = O(L^2 d)$
$p(y_{1:M}) = \prod_{j=1}^{M} p(y_j \mid S, y_{1:j-1})$

Where:

$u_i \in \mathbb{R}^{d}$ is the $i$ -th token embedding in the shared space and $x_i$ the raw unit behind it (a sub-word, an image tile, or a short audio window).
$\phi_m$ is the front end for modality $m \in \{\mathrm{txt}, \mathrm{img}, \mathrm{aud}\}$ and $W_m$ its projection to the model width $d$ ; text is an embedding lookup over discrete ids, vision and audio produce continuous soft tokens.
$S$ is the single causal sequence, ordered by timestamp rather than by modality, and $L$ is its length.
$L_{\mathrm{txt}}$ is the text token count, $N_f$ the number of image tiles or sampled frames at 258 tokens each, and $T_a$ the audio duration in seconds at 32 tokens each.
The prefill term is quadratic in $L$ while the KV cache grows linearly, so audio and video inflate both compute and memory before a single output token is produced.
$y_{1:M}$ is the generated output drawn from one next-token distribution whose vocabulary may include discrete image and audio codes alongside sub-words.

Token Budget For One Hour Of Video With Sound:
$3600 \times 258 = 928800$
$3600 \times 32 = 115200$
$L = 928800 + 115200 = 1044000$

A single hour of ordinary video therefore saturates a 1M-token window, which explains why native multimodality and million-token context arrived together in Gemini 1.5 rather than as separate features. It also explains where production effort actually goes: not into inventing a fusion block, but into deciding frame rate, tile count, and audio span so that the useful evidence survives tokenization, and into paying the quadratic prefill only for the segments that matter.

Log-scale line chart of tokens consumed versus input duration in minutes for four streams: a text transcript at about 200 tokens per minute, audio only at 1920 tokens per minute, video frames only at 15480 tokens per minute, and video plus audio at 17400 tokens per minute, with a dashed horizontal line marking a one-million-token context that the video plus audio curve reaches at about 57 minutes

Figure 3: Native tokens are not cheap. At published rates, a spoken transcript costs about 200 tokens per minute while the same minute of video plus audio costs roughly 17,400, so the entire 1M-token context is spent after about 57 minutes. Every design choice about frame rate, tiling, or clip trimming is a move along this line.

Property	Text	Vision (image / video)	Audio
Front end	SentencePiece sub-word vocabulary, discrete ids	ViT-style patch encoder emitting continuous soft tokens	Speech encoder in the USM lineage over 16 kHz audio
Token rate	About 1 token per 4 characters	About 258 tokens per tile, and per sampled frame at 1 fps	32 tokens per second, independent of content
Temporal handling	Sequence order only, no clock	Frames sampled at a fixed rate and packed in timestamp order	Continuous, packed beside the frames of the same second
Native generation	Standard next-token softmax	Discrete image tokens in Gemini 2.0 native image output	Codec tokens in the native-audio and Live streaming models
Dominant failure mode	Tokenizer fragments rare words, digits, and code	1 fps sampling misses sub-second events, and tiling explodes the budget	Coarse rate blurs fine prosody, and long clips crowd out the prompt

August 17, 2026

DL0134 Interleaved Vision-Language Architectures

Explain Interleaved Vision-Language Architectures. How do models pre-trained on mixed sequences of raw image tokens and text tokens differ from cross-attention models?

Answer

An interleaved vision-language architecture flattens a document into one sequence that alternates text spans and images, projects each image’s patch features into the language model’s embedding space, and trains a single decoder with ordinary next-token prediction over the mixed stream. This is early fusion: visual tokens occupy real sequence positions, so they are keys and values for every text token in every layer, and text tokens are keys and values for the image. The alternative family, cross-attention conditioning (Flamingo, Idefics1), keeps vision outside the text sequence: a frozen ViT plus a Perceiver resampler compresses each image to a fixed $R = 64$ latents, and newly inserted gated cross-attention layers let the frozen LM read those latents without ever lengthening its own sequence. The consequences are structural rather than cosmetic. Early fusion makes cost quadratic in the joint length $L = N_t + M N_v$ and demands full multimodal pre-training, but it gives uniform bidirectional-within-causal mixing, unlimited image ordering, and (with discrete VQ image tokens) the ability to generate images from the same head. Cross-attention decouples image count from the text sequence and protects a strong frozen LLM, at the price of a hand-designed conditioning path, per-image attention masks that weaken multi-image relational reasoning, and no route to image output.

(1) Where Fusion Happens: early fusion concatenates projected patch embeddings into the token stream, so mixing occurs in every self-attention layer; cross-attention confines mixing to a handful of inserted layers reading a fixed latent set.
(2) Sequence Length Is The Cost Model: a ViT-L/14 at 336 px yields $N_v = 576$ tokens per image, so eight images add 4608 positions and roughly $100\times$ the attention cost of a text-only forward pass.
(3) Attention Pattern And Multi-Image Reasoning: Flamingo masks each text token to cross-attend only to the most recent preceding image, while early fusion lets any token attend to all earlier images and their surrounding text.
(4) Parameter Budget And Text Regression: cross-attention adds new trainable blocks around a frozen LM and preserves text-only quality by construction; early fusion updates the whole backbone and risks catastrophic forgetting unless text data is replayed.
(5) Generation Symmetry: only the interleaved token-in-sequence form can emit images, because image outputs must live in the same vocabulary the head predicts (Chameleon, Emu3).
(6) Data Requirement: both families need interleaved web corpora such as M3W or OBELICS rather than caption pairs, since few-shot in-context learning is what interleaving buys.

Two-panel architecture diagram: left panel shows interleaved early fusion where text token groups and 576-token image groups alternate in one row feeding a single causal self-attention stack and a joint-vocabulary next-token head; right panel shows cross-attention conditioning where an image passes through a frozen ViT and a Perceiver resampler producing 64 latents that are read by gated cross-attention layers inserted between frozen language model blocks

Figure 1: The two conditioning paths. In early fusion the image becomes sequence positions inside the shared stack, so every layer mixes modalities and the sequence grows with each image. In cross-attention conditioning the image stays outside the text sequence as a fixed set of $R$ latents, read through gated layers whose gate starts at zero so the frozen LM is initially unchanged.

Practice has converged on early fusion, and the reason is mostly negative: the cross-attention path is an extra design surface with its own hyperparameters (how many layers to insert, resampler depth, latent count, gate schedule) and it caps the visual information at $R$ latents regardless of image resolution, which destroys dense OCR and chart reading. Ablations in MM1 found the interleaved token-in-sequence recipe matched or beat cross-attention at equal scale while being simpler, and the lineage of the same lab’s open models makes the shift explicit: Idefics1 (Flamingo-style cross-attention) → Idefics2 (early fusion with pooled visual tokens). Early fusion then imports the cross-attention family’s one real advantage, token compression, without its plumbing: Idefics3 pixel-shuffles $2 \times 2$ patch blocks into one token, Qwen2-VL merges adjacent patches in the projector, and both keep the compressed tokens in the main sequence. What remains genuinely hard in early fusion is positional encoding for a 2D object embedded in a 1D stream, handled by explicit row-separator tokens in Fuyu or by multimodal rotary embeddings that give an image token separate height, width, and time indices.

Mathematical Formulation:
$N_v = (H/P)\,(W/P)$
$L = N_t + M\,N_v$
$C_{\mathrm{self}} = O(L^2 d)$
$Z = \mathrm{Resampler}(\mathrm{ViT}(I)) \in \mathbb{R}^{R \times d}$
$C_{\mathrm{cross}} = O(N_t\, M R\, d)$
$y_\ell = h_\ell + \tanh(\alpha_\ell)\,\mathrm{XAttn}(h_\ell, Z)$

Where:

$H, W$ are the image height and width and $P$ the patch size, so $N_v$ is the visual tokens per image: at $336/14$ this is $24 \times 24 = 576$ .
$N_t$ is the number of text tokens, $M$ the number of images in the document, and $L$ the joint sequence length the early-fusion decoder actually processes; $d$ is the model width.
$C_{\mathrm{self}}$ is the self-attention cost, quadratic in $L$ , so each added image contributes both its own $N_v^2$ block and cross terms with all existing tokens.
$I$ is the raw image, $Z$ the resampled latents, and $R \ll N_v$ the latent count ( $R = 64$ in Flamingo), which is why $C_{\mathrm{cross}}$ is linear in $M$ and leaves the LM’s own $O(N_t^2 d)$ untouched.
$h_\ell$ are the frozen LM hidden states at layer $\ell$ and $\alpha_\ell$ a learned scalar gate initialized to zero, so $\tanh(\alpha_\ell) = 0$ makes the inserted block an identity at step 0 and training degrades the language model gradually rather than abruptly.

Line chart of attention cost normalized to a text-only forward pass versus the number of images from one to eight, with three curves: early fusion at 576 tokens per image rising quadratically to about one hundred times, early fusion with four-times pooling at 144 tokens per image rising to about ten times, and cross-attention with 64 latents per image rising linearly to about two times

Figure 2: Why token count dominates the choice. Early fusion pays $O(L^2)$ in the joint sequence, so eight full-resolution images cost roughly $100\times$ a text-only pass, while cross-attention stays near $2\times$ because its cost is linear in $M R$ . Patch pooling recovers most of the gap without leaving the sequence, which is the modern compromise.

Property	Interleaved early fusion	Cross-attention conditioning
Image position	Real sequence positions inside the decoder	Outside the sequence, as fixed keys and values
Cost in image count	Quadratic through the joint length L	Linear, and only in the inserted layers
Trainable surface	Projector plus the whole backbone, usually full pre-training	Resampler and gated cross-attention only, LM frozen
Visual detail ceiling	Scales with resolution; supports OCR and dense charts	Capped at R latents regardless of resolution
Multi-image reasoning	All prior images visible to every token	Typically masked to the most recent image per text token
Image generation	Possible with discrete VQ image tokens in one vocabulary	Not possible; the path is input-only
Representative models	Fuyu-8B, Chameleon, Idefics2/3, MM1, Qwen2-VL, Emu3	Flamingo, OpenFlamingo, Idefics1, Llama 3.2 Vision

August 17, 2026