Author: admin

DL0150 Generative Multi-View World Model

How do generative world models generate realistic, high-resolution multi-view camera outputs (e.g., 8 ring cameras on a vehicle) with spatial and temporal coherence?

Answer

Modern driving world models are latent video diffusion models rather than per-camera image generators. A causal video tokenizer compresses every camera stream (roughly 16x spatially and 4x temporally), and one diffusion transformer denoises a single latent tensor that holds all 8 views and all frames of the clip simultaneously, so coherence is a property of the sampling process instead of a post-hoc fix. Spatial agreement between neighbouring cameras comes from two ingredients: cross-view attention inside every block, and camera-geometry conditioning that tells each token where its camera points, usually as per-pixel Plücker ray embeddings built from the intrinsics and extrinsics of that rig. Temporal agreement comes from a temporal attention axis plus a shared ego-action and agent-layout condition that all 8 views must obey, so a braking maneuver or a crossing pedestrian appears in every camera at the same instant. High resolution is bought with a cascade instead of brute force: the transformer works in a small latent grid, and a separate diffusion upsampler restores full per-view pixels, while long horizons come from autoregressive rollout over overlapping context frames.

(1) Latent, Not Pixel, Diffusion: a learned video tokenizer removes most spatial and temporal redundancy first, so the generator never denoises 8 megapixel-scale streams directly.
(2) One Joint Tensor Over Views And Time: the state being denoised is $z \in \mathbb{R}^{V \times T \times N \times c}$ , which is why views cannot drift into independent hallucinations.
(3) Factorized 4D Attention: spatial attention within a view-frame, cross-view attention across the ring at a fixed time, and temporal attention across time within a view, roughly 126x cheaper than full joint attention at 8 views and 16 frames.
(4) Geometry As A First-Class Condition: Plücker rays per pixel (plus optional epipolar-constrained attention) let one model serve rigs with different camera counts, mounting positions, and fields of view.
(5) Shared Layout And Action Control: 3D agent boxes, an HD map or BEV raster, and the ego trajectory are broadcast to every view, so all cameras render the same world state rather than eight plausible worlds.
(6) Cascade And Rollout: a super-resolution diffusion stage supplies pixels and an overlapped sliding window supplies duration, at the cost of error accumulation over long horizons.

Pipeline diagram: eight-view video enters a causal video tokenizer with 16x spatial and 4x temporal compression, producing a latent grid of V by T by N tokens that feeds a diffusion transformer of L blocks, then a latent decoder and a diffusion upsampler back to 448 by 960 per view; an expanded panel shows one block containing spatial self-attention within a view-frame, cross-view attention across the eight views at one time step, and temporal attention across frames of one view, with camera Plucker rays, ego action, agent layout, and noise level injected on the right

Figure 1: One tokenizer, one transformer, one upsampler. The only thing that makes this a multi-view model rather than eight video models is the cross-view attention axis and the shared conditioning bundle; the token budget of 215,040 latents per clip is what forces the attention to be factorized.

The cross-view axis is where naive designs fail, and geometry explains why. Ring cameras overlap only with their immediate neighbours, so attending from the rear camera to the front camera buys almost nothing while costing the same as a useful pair; restricting cross-view attention to ring-adjacent views keeps essentially all of the benefit. Inside an adjacent pair, a point visible in view $i$ cannot be anywhere in view $j$ : the fundamental matrix of the calibrated rig confines it to a single epipolar line, which is why epipolar-band attention is a legitimate sparsification rather than a heuristic. Conditioning carries the rest of the load. Because the ego action and the 3D agent boxes are shared, the front-left and front cameras are both being asked to render the same car at the same metric position, and the overlap region then agrees on colour, exposure, geometry, and occlusion because both views were denoised from the same layout under attention that could compare them at every layer.

Left panel: top-down view of a vehicle with eight camera field-of-view wedges arranged in a ring, labelled front, front-right, right, rear-right, rear, rear-left, left, and front-left, with a 15 degree overlap marked between neighbours and an octagon of lines connecting adjacent camera nodes to show which pairs exchange cross-view attention. Right panel: two image-plane rectangles for view i and view j, a marked point x in view i and the corresponding epipolar band drawn as a shaded stripe in view j, with the constraint that a matching token lies on that single line.

Figure 2: Coherence is a geometry problem before it is a modelling problem. Only ring-adjacent pairs share a field of view, and within a pair the epipolar constraint bounds where a matching token can live, so both the sparsity pattern and the conditioning signal are dictated by the calibrated rig.

Mathematical Formulation:
$z \in \mathbb{R}^{V \times T \times N \times c}$
$N = (H/p)(W/p) = 28 \times 60 = 1680$
$L = V T N = 8 \times 16 \times 1680 = 215040$
$\mathcal{L}(\theta) = \mathbb{E}\left[\lVert \epsilon - \epsilon_{\theta}(z_t, t, c) \rVert^2\right]$
$c = (r, a, b, z_{\mathrm{ctx}})$
$r_{uv} = (d_{uv},\ o \times d_{uv}) \in \mathbb{R}^{6}$

Where:

$z$ is the joint latent tensor being denoised, with $V=8$ views, $T$ latent frames, $N$ tokens per view-frame, and $c$ latent channels.
$H \times W = 448 \times 960$ is the per-view pixel resolution and $p = 16$ the combined tokenizer and patch downsampling factor, giving a $28 \times 60$ latent map.
$L$ is the total sequence length if the clip were flattened, the number that decides whether full attention is affordable.
$\epsilon$ is the sampled noise, $z_t$ the noised latent at level $t$ , and $\epsilon_{\theta}$ the diffusion transformer.
$c$ bundles the conditioning: per-view rays $r$ , ego action $a$ (speed and curvature), agent and map layout $b$ , and the overlapping context latents $z_{\mathrm{ctx}}$ from the previous window.
$r_{uv}$ is the Plücker embedding of the ray through pixel $(u,v)$ , with unit direction $d_{uv}$ and camera origin $o$ , so extrinsics and intrinsics enter as a 6-channel image instead of a flat vector.

Attention Cost Per Block:
$C_{\mathrm{full}} = L^2 = 4.62 \times 10^{10}$
$C_{\mathrm{spa}} = V T N^2 = 3.61 \times 10^{8}$
$C_{\mathrm{view}} = T N V^2 = 1.72 \times 10^{6}$
$C_{\mathrm{tmp}} = V N T^2 = 3.44 \times 10^{6}$
$C_{\mathrm{spa}} + C_{\mathrm{view}} + C_{\mathrm{tmp}} = 3.66 \times 10^{8}$

Factorization therefore costs about 0.8% of full joint attention at this shape, and almost all of the remaining cost is ordinary within-frame spatial attention. The cross-view and temporal axes together add under 1.5% on top of the spatial term, which is the practical reason multi-view coherence is affordable at all: the expensive part of the model is generating each image, not keeping the images consistent.

Log-scale line chart of attention pairs per transformer block against the number of camera views from 1 to 8, with 16 latent frames and 1680 tokens per view-frame. The full joint 4D attention curve rises quadratically from about 7.2e8 to 4.62e10 pairs, while the factorized spatial plus cross-view plus temporal curve rises nearly linearly from about 2.9e7 to 3.66e8 pairs, an annotation noting the roughly 126x gap at eight views.

Figure 3: Full joint attention over the flattened clip grows as $V^2$ while the factorized form grows close to linearly in $V$ , so the gap widens with every camera added to the rig. Adding a ninth or tenth view is a routine cost in the factorized design and prohibitive in the joint one.

Property	Independent per-view generation	Factorized view + time attention	Full joint 4D attention
Attention pairs per block	3.61e8 (spatial only)	3.66e8, about 1.4% overhead	4.62e10, roughly 126x more
Overlap agreement	None; only the shared layout condition, so seams disagree	Enforced on ring-adjacent pairs, optionally epipolar-banded	Enforced globally, mostly on pairs that never overlap
Temporal behaviour	Per-view only; the eight views desynchronize over rollout	Temporal axis plus one shared action keeps views in lockstep	Strongest in principle, but memory caps the clip length
Failure signature	Objects appear in one camera and vanish in its neighbour	Slow drift of far-field content and appearance across rollout	Out-of-memory or a clip too short to be useful for simulation
Practical verdict	Useless for BEV or occupancy training data	The production choice for ring-camera world models	Research ablation at small resolution or few frames

August 18, 2026

DL0149 World Models from Unlabeled Video

How do world models learn interactive, action-controllable environment rollouts from unlabeled video datasets without ground-truth action labels?

Answer

The missing action labels are invented by the model itself. A latent action model (LAM) is an inverse-dynamics encoder that looks at the past frames and the next frame together and emits a single discrete code $z_t$ through a vector-quantized bottleneck; a forward dynamics model is then required to predict that next frame from the past frames plus $z_t$ alone. Both halves are trained jointly with next-frame reconstruction as the only supervision, so the code is pushed to carry exactly the part of the transition that the past cannot explain, which for gameplay and egocentric video is precisely the agent’s action. Genie made this concrete at scale, training an 11B-parameter model on roughly 30,000 hours of filtered 2D platformer video with a codebook of only $K = 8$ latent actions, and the tiny codebook is not a detail but the mechanism: at 3 bits per step the code physically cannot smuggle the next frame through it. At inference the LAM encoder is thrown away, a key press is mapped onto one of the $K$ learned codes, and the dynamics model rolls the world forward autoregressively, which turns passive video into a playable environment.

(1) Latent Action Model: an inverse-dynamics encoder over $(x_{1:t}, x_{t+1})$ produces one discrete code per transition, standing in for the action label that the dataset never had.
(2) The Bottleneck Is The Whole Trick: a codebook of size $K = 8$ caps the code at $\log_2 8 = 3$ bits, forcing it to describe a control decision rather than pixels.
(3) Reconstruction Is The Only Loss: the forward model’s next-frame error trains the encoder through the quantizer, so no human ever annotates a button.
(4) Tokenize, Then Roll Out: a spatiotemporal VQ-VAE turns frames into tokens and a masked-token or diffusion dynamics model generates the next frame conditioned on the past tokens and $z_t$ .
(5) Codes Are Shared Across Videos: because one codebook serves the whole corpus, the same index means the same intent in unseen scenes, which is what makes the latent space a controller instead of a per-clip artifact.
(6) Grounding To Real Controls Is Cheap: a handful of labeled clips, or human probing of each code, suffices to attach semantics; VPT instead spent 2,000 labeled hours to train an inverse-dynamics model and pseudo-labeled 70,000 web hours with it.
(7) Rollouts Drift: generation is autoregressive on its own output, so compounding error, memory loss, and entangled camera-versus-agent motion dominate the failure list.

Two-panel diagram. The training panel shows past frames and the next frame both entering a latent action model with a VQ bottleneck of eight codes, producing a 3-bit code, while the past frames are separately tokenized by a spatiotemporal VQ-VAE; both streams enter a masked video-token dynamics model whose only loss is reconstruction of the next frame. The inference panel shows a key press selecting one of the eight codes without the latent action model, feeding the same dynamics weights, decoding and rendering a frame that loops back as the next context.

Figure 1: The asymmetry between the two panels is the point. During training the next frame is visible to the latent action model, which is why an unlabeled corpus can supervise a controller; at inference that encoder is discarded and the codebook itself becomes the input device, with the rendered frame fed back as context for the next step.

Everything hinges on how much information the quantizer lets through. If the latent is generous (a wide continuous vector, or a codebook of thousands of entries), the encoder discovers the shortcut of encoding the next frame directly, the dynamics model degenerates into a decoder of the latent, reconstruction looks excellent, and controllability collapses because a user-chosen code no longer corresponds to anything an agent could do. If the latent is too narrow, distinct moves collapse into one code and the environment responds to only a couple of coarse commands. The standard diagnostic is a controllability delta: roll the model forward with the inferred code, then with a random code, and measure the gap in frame quality. A large gap means the code is doing real work; a gap near zero means either leakage or a code the model ignores.

Mathematical Formulation:
$z_t = q(f_{\phi}(x_{1:t+1}))$
$\hat{x}_{t+1} = g_{\theta}(x_{1:t}, z_t)$
$\mathcal{L}(\phi,\theta) = \| x_{t+1} - \hat{x}_{t+1} \|_2^2$
$H(z_t) \leq \log_2 K$
$\Delta = \mathrm{PSNR}(z_t) - \mathrm{PSNR}(\tilde{z})$

Where:

$z_t$ is the latent action for the transition at step $t$ , and $q$ is vector quantization onto a codebook of $K$ entries.
$f_{\phi}$ is the inverse-dynamics encoder, which is the only component allowed to see the future frame, and $\phi$ are its parameters.
$x_{1:t}$ are the observed past frames (in practice their VQ tokens), $x_{t+1}$ is the true next frame, and $\hat{x}_{t+1}$ is the prediction.
$g_{\theta}$ is the forward dynamics model, a masked video-token transformer or a diffusion decoder, conditioned on the past plus $z_t$ and nothing else.
$\mathcal{L}$ is the reconstruction objective, extended in practice by the usual commitment and codebook terms; no action label appears in it, which is why unlabeled video is sufficient.
$H(z_t) \leq \log_2 K$ is the capacity bound that prevents frame leakage, giving 3 bits per step when $K = 8$ .
$\Delta$ is the controllability metric, comparing a rollout driven by the inferred code against one driven by a random code $\tilde{z}$ drawn from the same codebook.

Why 3 Bits Cannot Encode A Frame:
$\log_2 K = \log_2 8 = 3 \ \text{bits}$
$256 \times 10 = 2560 \ \text{bits per frame}$

A $16 \times 16$ token grid over a 1024-entry visual codebook already carries about 2,560 bits, roughly three orders of magnitude more than the action code, so the encoder has no route to cheat and the dynamics model must genuinely keep a model of the scene in its context. That ratio is the design knob you actually tune: too small and the environment is unresponsive, too large and it stops being an environment at all.

Schematic line chart with latent action capacity in bits per transition on the x-axis and normalized quality on the y-axis. Next-frame reconstruction quality rises monotonically and saturates, while controllability rises to a peak near three bits and then decays toward zero as capacity grows, with a shaded band marking the usable control regime between about two and five bits and a dashed vertical line at three bits labeled as an eight-entry codebook.

Figure 2: Schematic view of the tension that a single reconstruction loss hides. Reconstruction quality never warns you about failure because it keeps improving as the latent widens, whereas controllability peaks and then collapses once the code can carry the frame; only the two curves together tell you the bottleneck is set correctly.

Property	Latent action model (no labels)	Inverse-dynamics pseudo-labeling	Logged true actions
Label requirement	None during training; a few clips only to name the codes	A labeled seed set, such as VPT’s 2,000 contractor hours	Every frame paired with the true control input
Action space	Discovered discrete codes, semantics unknown a priori	The real action API, recovered by a trained predictor	The real action API, exactly
Data ceiling	Any internet video of the domain	Any video, but the seed set must match the domain	Only instrumented environments and agents
Dominant failure mode	Frame leakage through a wide latent, or codes entangling camera and agent motion	Pseudo-label noise on rare actions propagates into the dynamics	Coverage limited to the logging policy’s behaviour
Representative system	Genie, LAPO	VPT	GameNGen, Dreamer-style agents

August 18, 2026

DL0148 Action Chunking Transformer (ACT)

Explain the Action Chunking Transformer (ACT). How does predicting sequences of future action vectors ( $k$ -step horizons) solve temporal inconsistency and compounding error drift?

Answer

ACT is the imitation-learning policy introduced with the ALOHA bimanual setup: a transformer encoder-decoder that takes the current multi-view images plus the 14-dimensional joint state and emits, in a single forward pass, a whole chunk of $k=100$ future joint-position targets rather than one action. It is trained as a conditional VAE (CVAE) with an $L_1$ reconstruction loss on the chunk plus a KL term on a small style latent $z$ , which lets one deterministic-at-test-time policy absorb the multimodality of human teleoperation instead of averaging it away. Chunking attacks drift arithmetically: at 50 Hz a 20-second task is 1,000 control steps but only $1000/100 = 10$ chunk decisions, so the number of places where the policy can leave the training distribution shrinks by a factor of $k$ and the classic $O(\epsilon T^2)$ behaviour-cloning bound falls to roughly $O(\epsilon T^2 / k)$ . Chunking also removes per-step temporal inconsistency, because the actions inside a chunk are generated jointly from one latent and one observation, so the policy cannot flip between two valid modes on consecutive 20 ms steps. To avoid a visible discontinuity when a new chunk starts, ACT still queries the network every step and blends the overlapping predictions with temporal ensembling.

(1) The Chunk Is The Output Unit: the policy models $\pi(a_{t:t+k-1} \mid o_t)$ instead of $\pi(a_t \mid o_t)$ , so $k$ correlated actions are predicted together as one object.
(2) Effective Horizon Divided By $k$ : the agent makes $H = \lceil T/k \rceil$ sequential decisions, and compounding error grows with the number of decisions, not with the number of motor commands.
(3) CVAE For Multimodality: a BERT-style encoder compresses the demonstrated chunk into a 32-dimensional latent $z$ during training and is discarded at test time, where $z = 0$ gives one clean decisive mode.
(4) Non-Markovian Demos Stop Being Fatal: a single-step policy standing at a human pause sees a bimodal target (hold still or move) and can freeze forever; a chunk that contains the pause and the following motion resolves the ambiguity.
(5) Temporal Ensembling For Smoothness: re-query every step and average the overlapping chunk predictions with weights $w_i = \exp(-m i)$ , which keeps reaction latency at one control step while removing chunk-boundary jumps.
(6) $L_1$ Over $L_2$ : the $L_1$ loss on absolute joint targets produces sharper, less smeared trajectories, which matters for millimetre-scale contact tasks.

Architecturally there is nothing exotic. Each of the four RGB streams (two wrist cameras, two static cameras at 480×640) goes through a ResNet-18 whose final feature map is flattened into roughly 300 tokens, the joint vector becomes one more token, and the style latent becomes one more; a 4-layer transformer encoder with width 512 mixes them, and a 7-layer decoder attends to that memory from $k$ fixed learned position embeddings, one per future step, so the $k \times 14$ output is produced non-autoregressively in about 0.01 s. The whole policy is roughly 80M parameters trained from scratch per task on about 50 demonstrations, which is the interesting part: the gain does not come from scale or pretraining, it comes from changing what a single prediction means.

Architecture diagram of ACT: four RGB cameras feed per-camera ResNet-18 backbones producing about 300 tokens each, a joint-position token joins them, a 4-layer transformer encoder mixes the tokens, a 7-layer decoder with k learned queries emits a k by 14 action chunk, and above the main lane a training-only CVAE encoder maps the demonstrated action sequence and joint state to a 32-dimensional style latent z that is injected into the encoder and set to zero at test time

Figure 1: ACT is a CVAE whose decoder is a chunk predictor. The dashed lane exists only during training: it sees the ground-truth action sequence and squeezes the demonstrator’s stylistic choice into $z \in \mathbb{R}^{32}$ . At test time $z$ is fixed to the prior mean, so the same network becomes a deterministic policy that outputs $k \times 14$ absolute joint targets in one pass.

Mathematical Formulation:
$\pi_\theta(\hat a_{t:t+k-1} \mid o_t, q_t, z)$
$\mathcal{L}_1 = \sum_{j=0}^{k-1} \| \hat a_{t+j} - a_{t+j} \|_1$
$\mathcal{L} = \mathcal{L}_1 + \beta \, D_{\mathrm{KL}}(q_\phi \, \| \, \mathcal{N}(0, I))$

Where:

$\hat a_{t:t+k-1}$ is the predicted action chunk, here $k=100$ absolute target joint configurations in $\mathbb{R}^{14}$ for a bimanual 6-DoF-plus-gripper arm pair.
$o_t$ are the four camera images at time $t$ and $q_t$ the measured joint positions, together forming the only observation the chunk is conditioned on.
$z \in \mathbb{R}^{32}$ is the style latent, sampled from the encoder posterior $q_\phi(z \mid a_{t:t+k-1}, q_t)$ in training and set to $z = 0$ at inference.
$j \in \{0, \ldots, k-1\}$ indexes positions inside the chunk, each produced by its own learned decoder query.
$\beta$ weights the KL term; a large $\beta$ collapses $z$ to noise while a small one lets the encoder leak the answer and hurts test-time behaviour.
The $L_1$ norm is deliberate: it penalises large joint errors less quadratically than $L_2$ and therefore averages competing modes less aggressively.

The drift argument is worth writing down. Standard behaviour cloning with per-step error $\epsilon$ suffers covariate shift: a mistake moves the robot to a state the demonstrations never covered, the next prediction is worse, and the regret bound is quadratic in the horizon. Chunking does not make the policy immune, it reduces how many times the loop is closed. With $H = T/k$ decision points, each contributing error $\epsilon$ that persists over the remaining decisions, and each decision covering $k$ timesteps of cost, the accumulated term scales as $\epsilon H^2 k = \epsilon T^2 / k$ . The same factor cuts inference calls, which is why a chunked policy can afford a much heavier network per decision.

Horizon Arithmetic At 50 Hz:
$T = 20 \times 50 = 1000$
$H = \lceil 1000 / 100 \rceil = 10$
$\mathcal{E}_{\mathrm{BC}} = O(\epsilon T^2)$
$\mathcal{E}_{\mathrm{chunk}} = O(\epsilon T^2 / k)$

Semi-log chart of a compounding-error bound scale against episode length in control steps at 50 Hz, with three parabolic curves for chunk size k equal to 1, 10, and 100, showing the k equals 100 curve two orders of magnitude below the single-step curve, annotated with the point where a 20 second task of 1000 steps requires only 10 chunk decisions

Figure 2: The bound is still quadratic in task length, so chunking does not abolish drift, it buys two orders of magnitude at $k=100$ . The practical reading is that a long-horizon task becomes as hard as a short one only if the chunk itself remains executable open-loop, which is exactly the assumption that breaks when the scene moves during the 2 seconds a chunk spans.

Naive open-loop execution of the chunk creates a new problem: every 2 seconds a fresh observation produces a fresh chunk that need not start where the previous one ended, and the robot jerks. ACT therefore runs the policy at the full control rate and aggregates. At timestep $t$ there are up to $k$ different chunks that predict an action for $t$ , one from each of the previous $k$ queries, and the executed command is their exponentially weighted mean with $m = 0.01$ , where index $i = 0$ is the oldest prediction. That value of $m$ keeps the weights nearly uniform (1.00, 0.99, 0.98, …), so new observations are folded in immediately while the average stays smooth; a larger $m$ leans on the oldest chunk and reacts more slowly.

Temporal Ensembling:
$w_i = \exp(-m i)$
$\bar a_t = \sum_i w_i \hat a_t^{(i)} / \sum_i w_i$

Timeline diagram with four horizontal rows of six cells each, one row per chunk issued at timesteps 0, 1, 2 and 3, staggered so that all four rows cover timestep 4; the four cells covering timestep 4 are highlighted and labelled with exponential weights 1.00, 0.99, 0.98 and 0.97, and an arrow leads down to a box stating that the executed action at timestep 4 is the weighted mean of the four overlapping predictions

Figure 3: Chunks overlap because the policy is queried every step, so each control command is a vote of up to $k$ predictions made from $k$ different observations. This is what keeps ACT closed-loop at 50 Hz while its prediction horizon stays 2 seconds long, and it costs one extra forward pass per step rather than any extra training.

Property	Single-step BC (k = 1)	Chunk, open-loop execution	ACT: chunk + temporal ensembling
Policy output	One 14-d joint target	100 x 14 targets, all executed	100 x 14 targets, blended with earlier chunks
Queries per 20 s task	1,000	10	1,000 (each cheap, 0.01 s)
Decision points that can drift	1,000, bound scales as eps T^2	10, bound scales as eps T^2 / k	10 committed motions, continuously re-averaged
Reaction to a disturbance	20 ms	Up to 2 s of stale commands	20 ms, damped by the weighted mean
Human pauses in demos	Bimodal target, policy can freeze	Pause is inside the chunk, resolved	Same, plus no boundary discontinuity
Dominant failure	Jitter and mode switching between steps	Jerk at chunk boundaries, blindness mid-chunk	Averaging across modes can blur a decisive motion

August 18, 2026

DL0147 Multi-Task VLA Negative Transfer

How do multi-task VLA policies, such as the RT-X models trained on the Open X-Embodiment dataset, avoid negative task transfer when trained across heterogeneous robot arms, grippers, and kinematic chains?

Answer

Negative transfer in a cross-embodiment VLA is not a mysterious optimization pathology; it is the direct consequence of the same observation-instruction pair mapping to different correct action vectors on different robots, so the per-embodiment gradients on shared weights partially cancel. The fix is to decide, layer by layer, what genuinely transfers and what does not: vision-language grounding transfers and stays fully shared, while action semantics do not and are pushed into a canonical action space plus embodiment-specific output parameters. In practice that means four moves applied together: canonicalize actions into a common frame with per-dataset percentile normalization, condition the policy on an explicit embodiment token so the target is identifiable, decode through a per-embodiment head or a padded action expert so incompatible outputs never share a final linear layer, and control the data mixture so one 100k-episode domain does not dominate the average gradient. Only after those are in place is gradient surgery (PCGrad, CAGrad) worth its cost, because most measured conflict in early cross-embodiment runs comes from unnormalized, mis-framed action labels rather than from a real task disagreement.

(1) Shared Trunk, Split Output: the ViT plus language backbone sees every dataset, and all embodiment-specific incompatibility is confined to the last block, which is where the conflicting gradients would otherwise meet.
(2) Canonical Action Space: relative end-effector deltas in a fixed camera or base frame, with a normalized gripper channel, make “move 2 cm right” mean the same thing on a Franka and a WidowX.
(3) Per-Dimension Percentile Normalization: mapping each action dimension by its 1st and 99th dataset percentiles into $[-1,1]$ removes the scale mismatch between a 0.08 m gripper stroke and a 0.03 m one.
(4) Explicit Embodiment Conditioning: an embodiment id, proprioceptive state, and control-rate token make the label identifiable; without them the policy is asked to regress a multi-modal target from an ambiguous input.
(5) Mixture Weighting And Capacity: per-domain sampling caps stop large datasets from monopolizing updates, and added capacity (more parameters, or modality/embodiment-aware routing) converts interference into specialization.
(6) Measure Conflict Before Fixing It: log pairwise gradient cosine similarity between embodiment groups and per-domain validation loss, then apply projection methods only to the pairs that are genuinely negative.

The heterogeneity is concrete rather than abstract. A 7-DoF arm commanded in end-effector deltas at 3 Hz, a 6-DoF arm commanded in joint velocities at 5 Hz, and a 14-DoF bimanual rig commanded in absolute joint positions at 50 Hz produce label vectors of different dimension, unit, frame, and scale. Stack them into one tensor and regress with one output layer, and that layer must fit a multi-modal conditional distribution whose modes are mutually contradictory, so the mean-seeking L1 or MSE solution is a blurred command that satisfies nobody. The RT-X study made the asymmetry visible: the small 35M-parameter policy gained substantially on data-poor domains while losing to the original single-domain policies on some data-rich ones, whereas the far larger vision-language-backboned variant did not, which is the classic capacity-limited interference signature. Action chunking makes the stakes higher still, because a head predicting a horizon of $H$ steps outputs $H \times d_a$ numbers whose meaning is embodiment-dependent at every position.

Architecture diagram with three input boxes for RGB observations, the language instruction, and an embodiment id plus proprioceptive spec, all feeding one tall shared vision-language trunk, which produces a shared latent that fans out to three separate action heads for a 7-DoF arm at 3 Hz, a 14-DoF bimanual rig at 50 Hz, and an 8-DoF mobile manipulator, each emitting an action vector of different dimension

Figure 1: Where heterogeneity is absorbed. Everything to the left of the latent is shared across every robot and is exactly the part that benefits from pooled data; everything to the right is embodiment-specific, so a 14-DoF joint-position label and a 7-DoF end-effector delta label never contend for the same output weights. The embodiment id enters as a token in the trunk and also selects the head.

Mathematical Formulation:
$z = f_{\theta}(o_{1:n}, \ell, c_e)$
$\hat{a} = h_{\phi_e}(z)$
$\mathcal{L}(\theta, \phi) = \sum_{e=1}^{E} w_e \mathcal{L}_e$
$g_e = \nabla_{\theta} \mathcal{L}_e$
$\cos(g_i, g_j) = \frac{g_i^{\top} g_j}{\lVert g_i \rVert \lVert g_j \rVert}$
$g_i' = g_i - \frac{g_i^{\top} g_j}{\lVert g_j \rVert^2} g_j$

Where:

$z$ is the shared latent produced by the trunk $f_{\theta}$ from the camera views $o_{1:n}$ , the instruction $\ell$ , and the embodiment context $c_e$ (robot id, proprioceptive state, control rate).
$h_{\phi_e}$ is the head for embodiment group $e$ , with its own output dimension $d_a^{(e)}$ and chunk horizon, so $\hat{a} \in \mathbb{R}^{H \times d_a^{(e)}}$ .
$w_e$ are the mixture weights over $E$ embodiment groups, set by capped sampling rather than raw episode counts, and $\mathcal{L}_e$ is the behaviour-cloning loss on group $e$ .
$g_e$ is that group’s gradient with respect to the shared parameters only; head parameters $\phi_e$ receive gradient from one group and cannot conflict by construction.
The cosine is the diagnostic: a negative value means the two groups disagree about the shared update, and its magnitude tells you how much of each step is being cancelled.
$g_i'$ is the PCGrad projection of $g_i$ onto the normal plane of $g_j$ , applied only when the cosine is negative and symmetrically for $g_j'$ .

Per-Dimension Action Normalization:
$\tilde{a}^{(d)} = 2 \frac{a^{(d)} - q_{1}^{(d)}}{q_{99}^{(d)} - q_{1}^{(d)}} - 1$

Percentiles rather than min-max are used because teleoperation logs contain jitter spikes that would otherwise compress the useful range into a few percent of the interval. Note also what normalization cannot repair: it aligns scales, not semantics. If one dataset logs joint velocities and another logs end-effector deltas, both normalized to $[-1,1]$ , the shared head still sees two incompatible meanings for the same slot, which is precisely the residual conflict that per-embodiment decoding removes.

Two-panel vector diagram in the plane. Left panel shows a bimanual gradient pointing up and a single-arm gradient pointing down-left with an angle of about 117 degrees between them, and their sum drawn as a short vector, illustrating cancellation. Right panel shows the same two gradients as faint dashed arrows plus their PCGrad projections drawn solid, whose sum is a longer vector that still descends both losses

Figure 2: Negative transfer, geometrically. When two embodiment groups have gradient cosine near -0.45, their raw sum is shorter than either gradient, so the shared trunk barely moves while both per-domain losses stall. After projecting each gradient onto the other’s normal plane, the combined step is longer and its inner product with both original gradients stays positive, which is the formal statement of “descends both tasks”.

Choosing where the embodiment-specific parameters live is the main design decision, and three families are in use. A single padded head defines one action vector of maximum width and zero-pads unused dimensions, with the loss masked over the padding, which keeps the model monolithic and transfers well when the padded dimensions are physically comparable. Per-embodiment heads require no cross-robot alignment at all and are the most robust to genuinely different control interfaces, at the cost of one head per group and no head for an unseen robot. Latent action spaces learn a discrete or continuous code from video by inverse dynamics, train the policy in that code, and attach a small decoder per robot, which unlocks human video and action-free data but adds an entire quantization stage that can lose fine-grained precision.

Property	Single padded head	Per-embodiment heads	Latent action space
How heterogeneity is handled	One max-width action vector, unused slots zero-padded and masked in the loss	One output module per action space, no alignment across robots needed	Policy predicts a robot-agnostic code, small per-robot decoder maps it to motors
Negative-transfer risk	Moderate: slots must be semantically comparable or the head averages modes	Lowest at the output layer, residual conflict only in the shared trunk	Low if the code is truly embodiment-invariant, high if it leaks robot identity
Unseen robot at test time	Works if its dimensions fit the padded layout	Needs a new head plus a short fine-tune on target data	Needs only a cheap decoder, which is the main selling point
Extra cost	Wasted output width and careful mask bookkeeping	Linear growth in heads, and per-group data must be sufficient	A separate latent-action pretraining stage and possible precision loss
Representative systems	Flow-matching action experts with padded dimensions, unified-action diffusion policies	Readout-head generalist policies and cross-embodied transformers spanning manipulation and navigation	Latent-action pretraining from human and web video

August 18, 2026

DL0146 VLA Continuous Action Representation

How does a transformer decoder represent multi-modal continuous robot actions (e.g., 7-DoF arm pose, gripper state, base velocity) alongside language tokens, as in Google’s RT-2 and the OpenVLA models?

Answer

The decoder never sees a float. Heterogeneous degrees of freedom are first concatenated into one fixed-width vector (end-effector deltas in metres and radians, a gripper command, base velocities, and a discrete mode flag that says which sub-system is being driven this step), then each dimension is independently normalized by its own 1st and 99th percentile statistics so that metres, radians, and a binary gripper all live on $[-1, 1]$ . In the RT-2 and OpenVLA line each normalized dimension is uniformly discretized into 256 bins and the bin index is mapped onto reserved token ids, typically the 256 least-frequently-used entries of the existing SentencePiece or Llama vocabulary, so an action literally becomes a short string of tokens appended to the same causal sequence that holds the image patches and the instruction. Nothing about the architecture changes: the same causal self-attention, the same softmax, and the same cross-entropy loss that predict the next sub-word now predict the next joint delta, which is exactly what lets a VLA inherit internet-scale pretraining. The cost is a quantization floor plus one sequential decoder step per degree of freedom, which is why newer systems keep the shared prefix but replace the discrete head with a continuous action expert that emits a whole chunk of future actions through diffusion or flow matching in a single pass.

(1) One Flat Vector, Fixed Slots: arm, gripper, and base share a single $D$ -dimensional action vector with fixed slot semantics, and unused slots are zero-padded so one model can serve several embodiments.
(2) A Mode Dimension, Not Separate Heads: RT-1 carries an explicit mode variable that switches between controlling the arm, controlling the base, and terminating the episode, so mutually exclusive sub-spaces are disambiguated inside the same vector.
(3) Per-Dimension Quantile Normalization: statistics are computed per dimension and per dataset, using percentiles rather than min and max, because a handful of teleoperation jerks would otherwise consume most of the dynamic range.
(4) 256 Bins Onto Reserved Vocabulary Ids: the action alphabet is carved out of the language vocabulary, so no new embedding table, output head, or loss function is introduced.
(5) Autoregressive Factorization Across DoFs: the chain rule over dimensions lets the yaw bin depend on the already-emitted $x$ and $y$ bins, which is what keeps the joint action coherent rather than per-axis independent.
(6) Action Chunking And Continuous Heads: predicting $H$ future steps at once fights compounding error and raises the effective control rate, and a flow-matching expert removes the bin grid entirely at the price of extra weights and solver steps.

Architecture diagram showing camera frames, instruction text, and proprioception each tokenized into a shared prefix sequence feeding a decoder-only transformer, which branches into a discrete head that emits 256-bin indices as reserved vocabulary tokens and a continuous action expert that emits an H by D chunk by flow matching, both producing the same 11-dimensional action vector of six arm deltas, one gripper command, three base velocities, and one mode flag

Figure 1: The prefix is modality-agnostic; only the head differs. The discrete head reuses the language softmax and spends one sequential decoder step per degree of freedom, while the continuous action expert attends to the same prefix but emits an entire $H \times D$ chunk of real numbers. Both write into the identical fixed-slot action vector, including the mode flag that selects arm, base, or episode termination.

Normalization is where most of the real accuracy is won or lost, and it is the step engineers most often get wrong. Raw teleoperation logs are heavy-tailed: a few reset motions or dropped-controller frames produce end-effector deltas an order of magnitude larger than anything the policy needs at test time. If the bin edges are set by the observed minimum and maximum, those rare samples stretch the grid so far that the entire working range of the robot collapses into a few dozen of the 256 available levels, and the policy inherits a positional resolution floor of several millimetres that no amount of extra data can fix. Clipping to the 1st and 99th percentile before binning, as OpenVLA does, spends the full alphabet on the range the robot actually operates in and pushes the residual error below the mechanical repeatability of most arms. The same argument applies to the gripper, which is near-binary in the data, so it is usually thresholded into open or closed rather than treated as a smooth continuous axis.

Two histograms of the same end-effector delta-z distribution. The left panel bins over the raw min-max range, where rare large outliers stretch the grid and only a small fraction of the 256 bins are ever occupied. The right panel clips to the 1st and 99th percentile before binning, and nearly all 256 bins are occupied with a much smaller bin width in millimetres.

Figure 2: The same action dimension, two bin grids. Min-max binning lets rare teleoperation outliers dictate the grid, so most of the 256 levels are never used and the effective step size is coarse. Quantile binning throws away 2% of the samples and buys back an order of magnitude in resolution per bin, which is why percentile statistics are standard in production VLA data pipelines.

Mathematical Formulation:
$a_t = (a^{\mathrm{arm}}_t, a^{\mathrm{grip}}_t, a^{\mathrm{base}}_t, m_t)$
$\tilde{a}_i = \mathrm{clip}(a_i, q^{i}_{1}, q^{i}_{99})$
$u_i = 2 (\tilde{a}_i - q^{i}_{1}) / (q^{i}_{99} - q^{i}_{1}) - 1$
$b_i = \lfloor 255 (u_i + 1) / 2 \rfloor$
$\mathrm{id}_i = V - 1 - b_i$
$p(a_t \mid c) = \prod_{i=1}^{D} p(b_i \mid c, b_{1:i-1})$

Where:

$a_t$ is the action at control step $t$ , built from an arm delta pose (three translations and three rotations), a gripper command, a base velocity triple, and a discrete mode flag $m_t$ .
$a_i$ is the raw value of dimension $i$ and $\tilde{a}_i$ its clipped version; $q^{i}_{1}$ and $q^{i}_{99}$ are the per-dimension percentiles estimated on the training corpus.
$u_i \in [-1, 1]$ is the normalized value and $b_i \in \{0, \ldots, 255\}$ the bin index, so the grid holds 256 levels per dimension.
$V$ is the language vocabulary size, and $\mathrm{id}_i$ overwrites the 256 least-used token ids at the tail of that vocabulary.
$c$ is the prefix (image patch tokens, instruction sub-words, optional proprioception) and $i \in \{1, \ldots, D\}$ indexes the action dimensions in a fixed canonical order.
The product is the autoregressive factorization: each bin is conditioned on all previously emitted bins of the same action, so the joint distribution is not a product of independent marginals.

Resolution And Token Budget:
$\Delta_i = (q^{i}_{99} - q^{i}_{1}) / 256$
$\Delta = 0.1\ \mathrm{m} / 256 \approx 0.39\ \mathrm{mm}$
$N = H \times D = 8 \times 11 = 88$

Those two lines set the entire engineering trade-off. A translation axis clipped to $\pm 0.05$ m resolves to about 0.39 mm per bin, which is fine for pick and place and marginal for connector insertion, but an eight-step chunk of an 11-dimensional action costs 88 sequential decoder passes, which caps a 7B VLA at roughly 3 to 10 Hz on a single accelerator. The alternative keeps the shared prefix and attaches a small action expert whose tokens are continuous rather than discrete: starting from noise, it integrates a learned velocity field for a handful of solver steps and emits the whole chunk at once, which is how flow-matching policies reach 50 Hz dexterous control. Crucially, both the categorical head and the flow head are distributional, and that matters more than resolution: when two different behaviours are equally valid, a plain L2 or Gaussian regression head averages them into an action that belongs to neither mode.

$A_t = (a_t, a_{t+1}, \ldots, a_{t+H-1})$
$A^{\tau + \delta} = A^{\tau} + \delta\, v_{\theta}(A^{\tau}, \tau, c)$

Left panel shows a top-down workspace with an obstacle and two valid detour trajectories, one above and one below, plus a straight-line mean trajectory that drives into the obstacle. Right panel shows a bimodal action density with a 256-way softmax step approximation covering both peaks and a red dashed vertical line marking the L2 optimum located in the empty valley between the modes.

Figure 3: Why the head must be distributional. Two demonstrators route around the same obstacle in opposite directions, so the conditional action distribution is bimodal; a regression head minimizing L2 lands on the mean, which is a collision. A 256-way categorical over bins, or a diffusion or flow head over the chunk, keeps both peaks and samples one of them.

Property	Discrete bins as vocabulary tokens	Diffusion / flow action expert	Direct continuous regression
Output form	256-way categorical per DoF over reserved token ids	Real-valued chunk of shape H by D, denoised from noise	One real vector, the predicted mean action
Training objective	Cross-entropy, identical to language pretraining	Flow-matching or denoising regression on noised chunks	L2 or L1 on the action vector
Cost per chunk	H times D sequential decoder steps (88 at H=8, D=11)	One prefix pass plus about 10 solver steps of a small expert	One forward pass
Multimodal actions	Yes, per-DoF peaks are tied together by the chain rule	Yes, jointly over the whole chunk	No, collapses to the mean
Resolution	Bin width, about 0.39 mm on a 0.1 m range	Continuous, limited only by data noise	Continuous
Dominant failure mode	Quantization floor plus slow autoregressive decoding	Extra weights and solver steps, harder to co-train with text	Mode averaging produces an invalid middle action

August 18, 2026

DL0145 ITC, ITM, and MLM Pretraining Losses

Compare Image-Text Contrastive (ITC), Image-Text Matching (ITM), and Masked Language Modeling (MLM) losses used in foundational vision-language pre-training, as in models like OpenAI’s CLIP and Salesforce’s BLIP.

Answer

The three objectives differ along three axes: how much cross-modal interaction they permit, which parameters receive gradient, and what the pretrained model can actually do afterwards. ITC is a dual-encoder loss that compresses each image and each caption into a single L2-normalized vector and maximizes the cosine similarity of matched pairs against the other pairs in the batch, so it produces embeddings that can be precomputed and indexed, at the price of never letting a word look at a patch. ITM is a cross-encoder loss: image and text tokens are fused with cross-attention and a binary head decides match or no-match, which gives token-level discrimination but requires one joint forward pass per candidate pair, so it can only ever be a reranker. MLM masks about 15% of the text tokens and reconstructs them conditioned on the image, which is the objective that forces word-to-region grounding and trains the language head that downstream VQA and captioning heads reuse. CLIP trains ITC alone at batch 32,768; ALBEF and BLIP train all three at equal weight because each one supplies something the others cannot.

(1) ITC Learns Global Alignment: an InfoNCE objective over the $N \times N$ in-batch similarity matrix, with $N-1$ negatives per anchor and a learned temperature, optimized symmetrically image-to-text and text-to-image.
(2) ITM Learns Fine-Grained Verification: binary cross-entropy on the fused $\mathrm{[CLS]}$ state, and it is nearly useless unless the negatives are hard negatives mined from the ITC similarity distribution.
(3) MLM Learns Grounded Language: reconstructing masked words from surrounding text plus image evidence ties nouns and attributes to specific patches and yields a vocabulary-sized output head that pure ITC models do not have.
(4) Different Architectural Footprint: ITC updates only the two unimodal towers and their projections, while ITM and MLM update the cross-attention fusion layers; only ITC leaves behind vectors you can put in an ANN index.
(5) Different Compute Profile: ITC is one forward pass per modality but needs a huge batch or a momentum queue (65,536 in ALBEF), whereas ITM and MLM add fusion passes, so BLIP runs its text stack three times per step.
(6) They Compose Into A Pipeline: ITC → hard-negative mining → ITM rerank is the standard retrieval stack, and MLM is the branch that makes the same weights usable for generation-flavored tasks.

Three side-by-side computation graphs. The ITC panel shows separate image and text encoders feeding L2-normalized projections, then an N by N similarity matrix, then an InfoNCE loss. The ITM panel shows both encoders feeding cross-attention fusion layers, then a joint CLS hidden state, then a two-way match versus no-match head using hard negatives. The MLM panel shows the image encoder and a text stream with 15 percent of tokens masked feeding the same fusion layers, then hidden states at masked positions, then a softmax over a 30k word-piece vocabulary.

Figure 1: The same two towers, three different graphs. ITC stops at a single pooled vector per modality, so no gradient ever reaches a fusion layer; ITM and MLM share the cross-attention stack and differ only in the head placed on top, which is why adding MLM to an ITM model is cheap while adding either to a CLIP-style model requires new parameters.

The three losses are not interchangeable because they fail in different directions. An ITC-only model scores “a man in a red shirt riding a bike” and “a man in a blue shirt riding a bike” almost identically, since a single 512-dimensional vector cannot preserve every attribute binding; this is the classic bag-of-words behavior that compositionality benchmarks expose. ITM fixes exactly that case because the fused representation can compare the color word against the actual patches, but its cost is $O(NM)$ joint forward passes for a gallery of $N$ images and $M$ captions, which is why ALBEF and BLIP use ITC to shortlist $k$ candidates and only then run ITM. MLM contributes something neither of the other two does: because the target is a token id rather than a pair label, it forces the model to represent which region licenses which word, and it produces the token-level head that VQA fine-tuning reuses. The coupling runs the other way too, since the ITM negative sampler reads its distribution from the ITC matrix, so a weak ITC head starves ITM of informative negatives.

Left panel: an eight by eight grayscale similarity matrix for one batch, with bright diagonal cells outlined in blue as positive pairs and one bright off-diagonal cell at image four versus text seven outlined in red as the hardest negative. Right panel: a bar chart of the softmax distribution over the eight texts for image four, with a tall blue bar for the true caption, a nearly as tall red bar for the hard negative, and six short gray bars for the easy negatives.

Figure 2: One batch, two uses of one matrix. ITC consumes the whole row and the whole column as a softmax classification problem, while ITM samples a single off-diagonal entry with high similarity and feeds that pair to the fusion encoder labelled no-match. The six near-zero bars are the reason random negatives teach ITM almost nothing: the easy cases already carry negligible gradient.

Mathematical Formulation:
$s_{ij} = z_i^{\top} t_j / \tau$
$\mathcal{L}_{\mathrm{i2t}} = -\log \frac{\exp(s_{ii})}{\sum_{j} \exp(s_{ij})}$
$\mathcal{L}_{\mathrm{itc}} = \tfrac{1}{2}(\mathcal{L}_{\mathrm{i2t}} + \mathcal{L}_{\mathrm{t2i}})$
$\mathcal{L}_{\mathrm{itm}} = -\log p(y \mid h_{\mathrm{cls}})$
$\mathcal{L}_{\mathrm{mlm}} = -\sum_{m \in M} \log p(t_m \mid I, T_{\setminus M})$
$\mathcal{L} = \mathcal{L}_{\mathrm{itc}} + \mathcal{L}_{\mathrm{itm}} + \mathcal{L}_{\mathrm{mlm}}$

Where:

$z_i$ and $t_j$ are the L2-normalized projections of the image $i$ and caption $j$ taken from the two unimodal towers, so $z_i^{\top} t_j$ is a cosine similarity in $[-1, 1]$ .
$\tau$ is the learned temperature (CLIP initializes it at 0.07 and clamps it), which controls how sharply the softmax concentrates on the hardest negatives.
$i, j \in \{1, \ldots, N\}$ index the batch, and the diagonal $s_{ii}$ holds the positives; $\mathcal{L}_{\mathrm{t2i}}$ is the same expression with the softmax taken down the column.
$h_{\mathrm{cls}}$ is the fused $\mathrm{[CLS]}$ state after cross-attention and $y \in \{0, 1\}$ the match label, with the negative pair drawn from the ITC row rather than uniformly.
$M$ is the set of masked positions (about 15% of tokens), $T_{\setminus M}$ the unmasked context, and $t_m$ the true word piece at position $m$ out of roughly 30k classes.
The total loss in ALBEF and BLIP uses equal weights, and MLM replaces the plain text-encoder forward pass rather than adding an independent one.

Retrieval Cost For 1k Images And 5k Captions:
$N_{\mathrm{itc}} = 1000 + 5000 = 6000$
$N_{\mathrm{full}} = 5000 \times 1000 = 5000000$
$N_{\mathrm{rerank}} = 5000 \times 32 = 160000$

ITC needs 6,000 encoder passes total and then only dot products, which is what makes billion-scale ANN search possible. A pure cross-encoder needs 5,000,000 fusion passes on the same Flickr30k-sized benchmark, while reranking the top $k = 32$ ITC candidates needs 160,000, about 31 times cheaper than the exhaustive version and still recovering most of the accuracy gain. That arithmetic, not any statement about representational quality, is why production retrieval systems keep ITC in the serving path and treat ITM as an offline or second-stage model.

Property	ITC	ITM	MLM
Objective form	Softmax over in-batch similarities (InfoNCE)	Binary cross-entropy on a fused pair	Cross-entropy over a 30k word-piece vocabulary
Cross-modal interaction	None until the final dot product	Full token-to-patch cross-attention	Full cross-attention, per masked position
Parameters trained	Two towers plus linear projections	Towers plus fusion layers plus 2-way head	Towers plus fusion layers plus vocab head
Negatives	All other pairs in the batch or momentum queue	One or two hard negatives sampled from the ITC row	Implicit: every other word in the vocabulary
Batch-size sensitivity	High: 32,768 in CLIP, or a 65,536 queue in ALBEF	Low, but negative quality depends on batch diversity	Low, like ordinary text pretraining
Retrieval role	Indexable embeddings, first-stage ANN search	Second-stage reranker over top-k candidates	No pair score at all; helps only as auxiliary signal
Typical failure mode	Bag-of-words behavior, attribute and relation blindness, false negatives from duplicate captions	Saturates near 100% accuracy with easy negatives and stops learning	Bidirectional masking cannot generate text, so captioning needs a different head

August 18, 2026

DL0144 Synthetic Captions vs Alt-Text

What is the impact of synthetic image captions generated by strong VLMs, as OpenAI did when training DALL-E 3, versus raw web alt-text during multi-modal pre-training?

Answer

Raw alt-text is a noisy channel: it averages roughly 10 words, frequently describes the page rather than the pixels, and is polluted by filenames, SEO keywords, stock-photo boilerplate, and product codes. Running a strong captioner over the corpus replaces that with a dense, grounded description of roughly 50 words that actually mentions attributes, counts, spatial relations, and background objects, which sharply improves text-to-image alignment and prompt following in generative models and improves retrieval in contrastive models. The cost is that the captioner can only describe what it can see and what it already knows, so recaptioning silently deletes the named entities, brands, landmarks, and long-tail vocabulary that only alt-text carries, and it stamps every sample with a single writing style, which collapses caption diversity and imports the captioner’s own hallucinations as ground truth. The practical result reported across DALL-E 3, DataComp, and Recap-DataComp-1B is that neither source wins outright: the tuned quantity is the mixing ratio, with generative text-to-image training favouring almost pure synthetic data (DALL-E 3 used a 95% blend) while contrastive CLIP-style training usually peaks at a genuine mix and degrades toward 100% synthetic.

(1) Alt-Text Is Noisy But Unbiased: it is written by humans for arbitrary purposes, so it is wrong or irrelevant often, yet its errors are not correlated with any single model’s blind spots.
(2) Synthetic Captions Raise Density: a 50-word grounded description supplies far more supervised text tokens per image than a 10-word alt string, which is what drives the gain in compositional and attribute-level alignment.
(3) Recaptioning Deletes World Knowledge: a captioner that cannot name a specific landmark, celebrity, or product writes “a tall building” and the entity vanishes from the training signal permanently.
(4) Style Collapse And Inherited Hallucination: every caption inherits one syntax template and one error distribution, so the student model learns the captioner’s biases as if they were facts.
(5) The Mixing Ratio Is The Real Knob: sample the synthetic caption with probability $p$ and the alt-text otherwise; $p$ near 1 suits text-to-image generation, intermediate $p$ suits contrastive pre-training.
(6) Context Length And One-Time Compute: CLIP’s text encoder truncates at 77 tokens, so dense captions are partially discarded, and recaptioning a billion images is a fixed preprocessing bill of order $10^4$ GPU-hours.

Pipeline diagram: a crawled web page supplies both an alt attribute and image pixels; the alt attribute becomes a short noisy raw caption while the pixels pass through a VLM captioner that emits a dense fifty-word synthetic caption, and both streams feed a mixing sampler that selects the synthetic caption with probability p before contrastive or text-to-image pre-training

Figure 1: The two text streams come from different places. Alt-text is a property of the page, synthetic captions are a property of the pixels plus the captioner’s knowledge, and the only place the two are reconciled is the mixing sampler that draws each training pair’s caption with probability $p$ .

The mechanism behind the improvement is easy to state: the contrastive or captioning objective is unchanged, only the text marginal moves. A short alt string gives the model very few positive constraints, so many wrong images remain compatible with it, whereas a dense caption pins down attributes, counts, and relations and therefore produces a much sharper positive. That is exactly why generative text-to-image models benefit most: their failure mode is ignoring adjectives, counts, and spatial prepositions in a user prompt, and dense captions are the only supervision that ever mentions those. It is also why the gains shrink for discriminative zero-shot classification at scale: with a 1B-scale pool, alt-text’s lexical diversity and entity coverage begin to matter more than its per-sample precision, and studies on DataComp report that generated captions dominate at small and medium pool sizes while a raw-plus-synthetic mixture wins at the large scale.

Two panels: the left panel plots downstream metric against the share of synthetic captions p, with a retrieval and alignment curve rising monotonically toward p equal to one and a zero-shot classification curve peaking near p equal to zero point five five and falling afterwards; the right panel overlays two caption length histograms, raw alt-text concentrated near ten tokens and synthetic captions centred near fifty-eight tokens, with a dashed vertical line at the seventy-seven token CLIP context limit

Figure 2: Two views of the same trade-off. Panel (a) shows why one blend cannot serve both objectives: alignment keeps improving with $p$ while zero-shot classification turns over once entity-bearing alt-text is crowded out. Panel (b) shows the second-order problem: dense captions push the length distribution against the 77-token text-encoder limit, so part of the extra supervision is truncated before it is ever used.

Mathematical Formulation:
$c_i \sim q_{\phi}(c \mid v_i)$
$P(t_i = c_i) = p, \quad P(t_i = a_i) = 1 - p$
$s_{ij} = f(v_i)^{\top} g(t_j) / \tau$
$\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$

Where:

$c_i$ is the synthetic caption sampled from the captioner $q_{\phi}$ conditioned on image $v_i$ , and $a_i$ is the raw alt-text scraped alongside that image.
$t_i$ is the caption actually used for example $i$ , and $p \in [0,1]$ is the mixing ratio, the single hyper-parameter that decides how much of the corpus the captioner rewrites.
$f$ and $g$ are the image and text towers producing normalized embeddings, $\tau$ is the learned temperature, and $B$ is the batch size supplying the in-batch negatives.
Nothing in $\mathcal{L}$ changes when you recaption; the entire effect flows through the conditional distribution of $t_i$ given $v_i$ , which becomes lower-noise but also lower-entropy and model-biased.

One-Time Recaptioning Cost For 1B Images:
$T = 10^{9} / 20 = 5 \times 10^{7}$
$5 \times 10^{7} / 3600 \approx 1.4 \times 10^{4}$

At a sustained 20 images per second per GPU for a 7B-class captioner emitting about 50 tokens, one billion images take $5 \times 10^{7}$ seconds of single-GPU time, roughly 14,000 GPU-hours, or about half a day on a 1,000-GPU cluster. That is a real but one-time preprocessing cost, amortized over every subsequent training run on the corpus, which is why recaptioning is usually cheaper than the ablation sweeps it replaces. The recurring costs are subtler: longer captions mean more text-encoder tokens per step, and a frozen captioner freezes a snapshot of one model’s competence into the dataset.

Property	Raw web alt-text	VLM synthetic caption	Mixture at ratio p
Typical length	About 10 words, often a fragment	About 50 words of dense description	Bimodal, which also teaches the model short prompts
Image grounding	Frequently describes the page, not the pixels	Grounded by construction, with residual hallucination	Grounded on the synthetic draw, noisy on the raw draw
World knowledge	Carries brands, landmarks, people, rare nouns	Entities collapse to generic categories	Entity coverage preserved by the raw fraction
Diversity	High lexical and syntactic variety	Single style template, reduced noun variety	Diversity recovered without giving up density
Best fit	Very large pools where scale beats precision	Text-to-image generation and prompt following	Contrastive pre-training and general-purpose encoders
Marginal cost	Free, already in the crawl	Order 10,000 GPU-hours per billion images	Same captioning bill, plus storage for two text fields

August 18, 2026

DL0143 VLM Pretraining Data Curation

Walk through the data curation pipeline for pre-training large VLMs, including web image-text filtering, synthetic re-captioning (e.g., LLaVA-1.5/1.6), and visual instruction tuning.

Answer

Pre-training a large VLM is mostly a data engineering problem, and the pipeline produces three qualitatively different corpora: a heavily filtered web corpus for breadth, a synthetically re-captioned corpus for description quality, and a small hand-assembled instruction corpus for behaviour. Stage one takes a raw crawl on the order of 10B alt-text pairs, removes NSFW, PII and duplicate URLs, drops images below roughly 200 px and captions outside a 5 to 64 token window, keeps about the top 30% by CLIP image-text cosine, and intersects that with a cluster-based balancing filter, which is how DataComp’s 12.8B CommonPool collapses to the ~1.4B pairs of DataComp-1B. Stage two attacks the fact that surviving alt-text is still short, keyword-shaped and frequently describes the page rather than the pixels: a captioner VLM, itself trained on a small set of 100K high-quality dense captions in the ShareGPT4V style, rewrites each image into a 50 to 100 word caption, and the load-bearing detail is that the best recipes mix synthetic and original captions instead of replacing one with the other, because pure synthetic text launders away proper nouns and world knowledge. Stage three is tiny by comparison, LLaVA-1.5’s 665K instruction mixture growing to roughly 760K in LLaVA-1.6 with DocVQA, ChartQA and AI2D added, and it buys instruction following, short grounded answers, OCR and chart reading rather than new visual knowledge. The three stages differ by four orders of magnitude in scale and by roughly the same factor in cost per example, which is why the filtering stage is optimised for throughput and the instruction stage for mixture ratios.

(1) Cascade Order Is Cost Order: run cheap deterministic filters (decode check, resolution, aspect ratio, caption length, exact and near-duplicate hashing, NSFW and PII removal) before any model forward pass, since a CLIP score on 12.8B pairs is the single most expensive step in the pipeline.
(2) CLIP Score Is A Precision Knob: the cosine gate raises image-text agreement but systematically deletes long compositional captions, rare entities and text-heavy images, so a threshold tuned for zero-shot classification quietly damages OCR and document tasks.
(3) Cluster Balancing Beats Raw Score: DataComp’s winning filtering-track recipe intersects the CLIP-score gate with an image-embedding cluster filter, keeping pairs whose visual cluster resembles curated concept distributions, which fixes the head-heavy topical skew of the crawl.
(4) Re-Captioning Changes The Supervision, Not The Images: the same 1.4B images are re-labelled by a captioner VLM, so the corpus gains dense spatial, attribute and relational description at roughly the cost of one VLM forward pass per image and zero new crawling.
(5) Mix, Do Not Replace: a mixing probability of about $\alpha \approx 0.8$ synthetic to 0.2 original, or an LLM fusion of both strings as in CapsFusion, retains the named entities and factual hooks that only alt-text carries.
(6) Instruction Data Is Ratio-Sensitive, Not Scale-Sensitive: at the 665K scale the composition (VQA, OCR, region grounding, text-only chat) matters far more than the count, and dropping the text-only share collapses multi-turn conversational quality while adding no visual skill.
(7) Decontamination Is Mandatory: near-duplicate removal against the images and questions of VQAv2, TextVQA, MMMU and friends must run at every stage, because a captioner trained on benchmark-adjacent data will otherwise leak answers into the pre-training corpus.

Three-band pipeline diagram: band A shows a raw Common Crawl pool of 12.8B image-text pairs passing through safety and PII removal, basic resolution and caption-length filters, a CLIP cosine gate keeping the top 30 percent, and cluster balancing plus decontamination to yield 1.4B pairs; band B shows a captioner VLM trained on 100K high-quality captions producing dense synthetic captions, an LLM fusing alt-text with the synthetic caption, and a mixed corpus at alpha near 0.8; band C shows projector alignment on 558K pairs, caption plus interleaved pre-training, visual instruction tuning on 665K to 760K examples, and evaluation on VQAv2, TextVQA, DocVQA and MMMU

Figure 1: The pipeline as three chained corpora rather than one dataset. Band A is a cheap-to-expensive filter cascade that discards about 89% of the crawl, band B re-labels the survivors with a captioner VLM and fuses the result with the original alt-text, and band C spends a four-orders-of-magnitude smaller budget on alignment then instruction tuning.

The filtering stage is best understood as trading recall for precision under a fixed compute budget. DataComp’s central result is that the winning entry is not the largest pool but the most aggressively filtered one: at a fixed number of training samples seen, a 1.4B subset beats the 12.8B pool it came from, because gradient steps spent on mismatched pairs are worse than wasted. The failure mode of that logic is that the CLIP scorer used to filter was itself trained on similarly filtered data, so its notion of “matching” is circular and biased against exactly the long, unusual, or text-dense captions that document and chart understanding require. Production pipelines therefore keep separate sub-pools with different thresholds, plus an explicitly retained OCR-heavy shard, rather than applying one global $\tau$ to everything.

Mathematical Formulation:
$s(I,T) = \cos(f_I, f_T)$
$D_1 = \{(I,T) \in D_0 : s(I,T) \geq \tau\}$
$D_2 = D_1 \cap C_{\mathrm{clust}}$
$T' \sim p_{\phi}(T \mid I)$
$P(\tilde{T} = T') = \alpha$
$\mathcal{L} = -\sum_{t \in A} \log p_{\theta}(y_t \mid I, x, y_{1:t-1})$

Where:

$s(I,T)$ is the CLIP cosine between the image embedding $f_I$ and caption embedding $f_T$ , and $\tau$ is the gate, historically about $0.28$ for a ViT-B/32 scorer or a percentile such as the top 30%.
$D_0$ is the raw pool (12.8B pairs in CommonPool), $D_1$ the CLIP-gated set, and $D_2$ the final corpus after intersecting with the cluster filter $C_{\mathrm{clust}}$ and benchmark decontamination, giving roughly $1.4 \times 10^{9}$ pairs.
$p_{\phi}$ is the captioner VLM and $T'$ the dense synthetic caption it samples for image $I$ ; $\phi$ is trained on a small human or GPT-4V-labelled seed set, typically around 100K captions.
$\alpha \in [0,1]$ is the mixing probability of using the synthetic caption instead of the original alt-text $T$ for a given training sample; $\alpha = 1$ is pure synthetic and $\alpha = 0$ is the raw web baseline.
$\mathcal{L}$ is the instruction-tuning objective over the answer token set $A$ only, with the image $I$ and instruction $x$ as context and loss masked on the prompt, which is what prevents the model from learning to hallucinate its own questions.

The instruction stage also decides the inference bill, because resolution enters through the token count rather than the parameter count. LLaVA-1.5 uses a CLIP ViT-L/14 at 336 px, so each image becomes $(336/14)^2 = 576$ visual tokens, while LLaVA-1.6’s AnyRes scheme tiles a high-resolution image into four crops plus a global thumbnail, giving $5 \times 576 = 2880$ tokens. That five-fold increase is what unlocks DocVQA and ChartQA, and it also means the instruction mixture must contain enough high-resolution document data to justify the tokens, otherwise the model pays the cost without learning to use the detail.

Line chart with synthetic caption mixing ratio alpha on the x axis from 0 to 1 and relative benchmark score on the y axis: a retrieval and captioning curve rises monotonically from 100 to about 118, an entity and world-knowledge curve rises to a peak near alpha 0.65 then falls back to about 100 at alpha 1, and their average peaks near alpha 0.8, marked by a vertical dashed line labelled common operating point

Figure 2: Why re-captioning is a mixture and not a replacement. Synthetic captions monotonically improve retrieval and description because they are fluent and pixel-grounded, but entity and knowledge accuracy peaks well before $\alpha = 1$ and then decays, since a captioner cannot invent the proper nouns, brands, and dates that only human alt-text supplied. The reported operating point in VeCLIP, CapsFusion and Recap-DataComp ablations lands near $\alpha \approx 0.8$ .

Property	Filtered web pairs	Synthetic re-captions	Visual instruction data
Typical scale	1B to 5B pairs after filtering 10B+ raw	100K seed captions, then 1M to 1.3B generated	665K in LLaVA-1.5, about 760K in LLaVA-1.6
Source	Common Crawl alt-text, HTML attributes	A captioner VLM run over already-filtered images	Academic VQA/OCR/grounding sets plus LLM-written dialogue
Cost per example	Crawl plus one CLIP forward pass	One VLM generation of 50 to 100 tokens	Human annotation or strong-model distillation, orders of magnitude higher
What it teaches	Concept coverage, entities, long-tail visual vocabulary	Dense attributes, spatial relations, fluent grounded description	Answer format, instruction following, refusal and multi-turn behaviour
Dominant failure mode	Mismatched or page-level captions, topical head skew	Hallucinated details, lost proper nouns, uniform caption style	Wrong mixture ratio, benchmark overfitting, short-answer bias
Where it enters training	Projector alignment and large-scale pre-training	Mixed into the same pre-training stream at ratio $\alpha$	Final supervised stage, full LLM and projector unfrozen

August 18, 2026

DL0142 OCR-Free Document VLM

How do OCR-free Document VLMs process complex multi-column PDFs, tables, and infographics compared to multi-stage OCR pipeline setups?

Answer

A multi-stage pipeline turns a page into a text document before any reasoning happens: rasterize → detect text regions → recognize each crop → classify layout blocks → sort them into reading order → recover table cell structure → serialize to Markdown or HTML → feed a text LLM. An OCR-free Document VLM deletes that entire chain and treats the page as an image: a dynamic-resolution ViT encoder cuts the raster into 14×14 patches, a 2×2 pixel-shuffle merge collapses them into one visual token per 28×28 pixel block, and a decoder-only LLM attends over those tokens to emit the answer, the Markdown, or the HTML table directly. The consequence is that self-attention itself becomes the layout model: column boundaries, cell alignment, chart axes, and legend-to-series association are learned from pixels rather than reconstructed by six independently trained components whose errors multiply. What you gain is robustness on infographics and rotated or borderless tables, where reported scores such as Qwen2.5-VL-72B’s roughly 96 ANLS on DocVQA and roughly 87 on InfographicVQA are far out of reach for a serialized-text pipeline. What you lose is character-level coordinates, per-token confidences, and cheap per-page cost, because a single A4 page at 150 DPI already costs about 2,835 visual tokens and the prefill over them is quadratic.

(1) Pixels In, Structure Out: the model never sees a text layer, so scanned pages, screenshots, and born-digital PDFs with broken embedded fonts all take the identical path.
(2) Dynamic Resolution Tokenization: instead of squashing every page to 224×224, the encoder keeps native aspect ratio and resolution, so an 8 pt footnote survives as its own tokens rather than being blurred away.
(3) Attention Replaces The Reading-Order Module: a three-column paper needs no LayoutReader-style sorter, because the decoder learns column continuation the way a language model learns syntax.
(4) Tables As Generated Markup: structure recognition becomes ordinary autoregressive decoding of HTML or Markdown, which handles borderless and spanning cells but has no per-cell confidence.
(5) No Error Compounding, No Coordinates: the pipeline’s five or six stages multiply their error rates, while the VLM has one loss and one failure surface but cannot tell you where on the page an answer came from.
(6) Token Budget Is The Real Constraint: visual tokens grow with the square of DPI, so resolution, tiling, and page count trade directly against context and $O(N^2 d)$ prefill.

Two horizontal lanes compared on the same rasterized page: the upper lane shows a six-stage OCR pipeline running text detection, crop recognition, layout analysis with reading order, table structure recognition, and Markdown serialization into a text LLM, annotated with compounding per-stage error; the lower lane shows an OCR-free VLM with a dynamic-resolution ViT patch encoder, a 2x2 pixel-shuffle merge producing 2,835 visual tokens, and a decoder-only LLM emitting the answer or an HTML table

Figure 1: The same pixels, two failure surfaces. The pipeline produces an intermediate text document with coordinates that any downstream model can consume and any auditor can overlay, at the cost of five models whose accuracies multiply. The VLM is one differentiable stack with one loss, and its output carries no character boxes at all unless the model was explicitly trained to emit them.

The three hard document classes fail differently. On multi-column PDFs, a pipeline’s mistake is almost never recognition, it is serialization: a two-column paper with a full-width figure caption in the middle gets flattened into interleaved half-sentences, and the LLM downstream has no way to recover the intended order because the evidence, the geometry, was discarded. On tables, borderless layouts and spanning header cells break rule-based and detection-based structure recognition, whereas a VLM trained on HTML targets can emit rowspan and colspan because it saw the whole grid at once. On infographics and charts, OCR returns a bag of strings with no relations, so “which bar is tallest” or “what does the dashed series do after 2021” is unanswerable from the transcript; this is precisely where the ChartQA and InfographicVQA gaps are widest. The pipeline still wins wherever the requirement is verbatim fidelity plus provenance, since a hallucinated digit inside a generated table cell is indistinguishable from a correct one, while a low-confidence OCR crop announces itself.

Mathematical Formulation:
$N_{tok} = \lceil H/p \rceil \times \lceil W/p \rceil$
$N_{tok} = 63 \times 45 = 2835$
$C_{prefill} = O(N_{tok}^2 d)$
$A_{pipe} = \prod_{k=1}^{K} a_k$
$A_{pipe} = 0.95^5 \approx 0.77$

Where:

$N_{tok}$ is the number of visual tokens the encoder emits for one page, which is what actually enters the LLM context.
$H$ and $W$ are the rasterized page height and width in pixels ( $1754 \times 1240$ for A4 at 150 DPI), and $p = 28$ is the effective patch stride after a 2×2 merge of 14×14 patches.
$C_{prefill}$ is the attention cost before the first output token, quadratic in $N_{tok}$ and linear in model width $d$ ; the KV cache grows linearly, so a 20-page document is a memory problem as well as a compute one.
$a_k$ is the per-page success rate of pipeline stage $k$ and $K$ the number of stages, with $k \in \{1,\ldots,K\}$ running detection, recognition, layout, reading order, and table structure.
$A_{pipe}$ is the end-to-end page accuracy: five stages that each succeed 95% of the time leave only about 77% of pages fully clean, and this multiplicative compounding is the structural argument for a single-stage model.

Log-scale chart of tokens per A4 page versus rasterization DPI from 72 to 420: a curve for visual tokens after 2x2 pixel-shuffle merge rising quadratically from about 640 at 72 DPI to about 21500 at 420 DPI, a four-times-higher curve for raw 14x14 patches without merging, and a flat line at about 800 tokens for serialized OCR text, with a horizontal marker at the 16384-token per-image cap and a vertical dashed line at 150 DPI

Figure 2: Resolution is the cost knob. The same A4 page costs 2,835 visual tokens at 150 DPI and 11,214 at 300 DPI, roughly 4x the tokens and 16x the prefill FLOPs, while a serialized OCR transcript of that page stays near 800 tokens whatever the DPI. The 2×2 pixel-shuffle merge is what keeps a full page under the common 16,384-token per-image cap at all.

Property	Multi-stage OCR pipeline	OCR-free Document VLM
Multi-column reading order	Explicit sorter over layout blocks; interleaves columns when a full-width element splits the page	Learned implicitly by attention over the whole page at once
Tables	Dedicated structure model emitting cell boxes; weak on borderless and spanning cells	Generates HTML with rowspan and colspan; can silently drop or invent rows in long tables
Charts and infographics	Returns unrelated strings; visual relations such as legend-to-series are lost	Reads axes, bar heights, and legends jointly, which is where the accuracy gap is largest
Provenance	Character and word boxes plus per-crop confidence, usable for redaction and highlighting	None by default; needs grounding training to emit absolute coordinates
Dominant failure mode	Compounding stage errors and serialization scrambling; degrades visibly	Fluent hallucination and repetition loops; degrades invisibly
Cost per page	Small CNN and CTC models, CPU-viable, millions of pages per day cheaply	Thousands of visual tokens through a multi-billion-parameter decoder with quadratic prefill
Adapting to a new form type	Retrain or rewrite whichever stage broke, with per-stage labels	Fine-tune once on image and target-string pairs, no intermediate annotation

August 18, 2026

DL0141 Visual Token Compression for OCR

How do visual token compression techniques reduce the sequence length of visual inputs without degrading fine-grained OCR performance?

Answer

The techniques that survive contact with documents all compress along the channel axis rather than the token axis: a pixel unshuffle (InternVL) or a strided convolutional reducer (DocOwl 1.5) folds each $s \times s$ block of patch embeddings into one vector of width $s^2 d$ and projects it back to $d$ , so four patches become one token while the patch-to-region mapping stays bijective and no pixel is discarded. The second half of the recipe is that the token budget must keep scaling with input resolution: dynamic tiling and native-resolution patching (Qwen2-VL) give a 1275×1650 scan roughly 2,600 tokens of 28×28 pixels each, whereas a fixed- $K$ resampler hands the same page 64 tokens no matter how many glyphs it contains. What actually kills OCR is rarely the compressor itself but the resize step in front of it: squeezing a page into 336×336 makes an 11 pt glyph thinner than one 14-pixel patch, and no downstream module can recover strokes the encoder never sampled. The useful mental model is glyph density: keep the number of glyphs covered by a single visual token near one, and compression is nearly free; push it toward ten and character-level accuracy falls off a cliff while scene-level captioning barely moves.

(1) Compress Channels, Not Positions: pixel unshuffle and conv reducers move information into the feature dimension, so a $4\times$ length reduction still lets every output token point at a known rectangle of the page.
(2) Keep The Budget Resolution-Dependent: a compression ratio is safe, a compression target is not; dynamic tiling plus native-resolution patching lets a dense page buy more tokens than a photo of a beach.
(3) Respect Glyph Nyquist: the binding constraint is stroke width versus patch size in the resized image, which is why 336-pixel inputs cap document accuracy regardless of the connector.
(4) Fixed-Query Resamplers Lose The Wrong Thing First: $K$ learned queries cross-attend to all patches without a positional index, so reading order and rare characters degrade before object-level semantics do.
(5) Two-Scale Views Are Cheap: a global thumbnail supplies layout while local tiles supply glyphs, which is why AnyRes-style designs use $4 \times 576 + 576 = 2880$ tokens rather than one giant grid.
(6) Prune Late And Query-Aware: dropping half the visual tokens after LLM layer 2 (FastV) saves about 45% of prefill FLOPs on scene VQA but deletes whole text lines when the question has not yet been attended to.

Three-row diagram comparing visual token compression families on the same 8 by 4 patch grid: the top row folds each 2 by 2 block of patches into one token on the channel axis and produces a 4 by 2 output grid with matching tints, the middle row sends all patches through cross-attention into four fixed learned query tokens with no positional index, and the bottom row keeps the original grid but marks half the cells as dropped by an attention score

Figure 1: Same patch grid, three compression axes. Only the top row keeps an exact mapping from output token back to page rectangle, which is what OCR decoding depends on; the middle row replaces that mapping with $K$ content-addressed slots, and the bottom row keeps positions but deletes evidence.

Why the distinction matters becomes obvious once you count information. Natural images are locally redundant, so averaging neighbouring patches costs almost nothing; a page of text is close to the opposite, since each glyph is a high-entropy symbol whose identity cannot be inferred from its neighbours and whose position carries the reading order. A learned resampler is a query-agnostic bottleneck: it must decide what to keep before the question arrives, and a fixed 64-slot budget forces it to summarise, which is exactly the wrong operation for text. Structured merging instead makes a bounded, uniform trade, and empirically the boundary sits near one glyph per token. The design lineage of production VLMs follows that logic directly: Q-Former resamplers → pixel unshuffle with dynamic tiles → native dynamic resolution with a 2×2 patch merger, each step trading a smaller guaranteed budget for a budget that grows with how much text is actually on the page.

Mathematical Formulation:
$N_p = \dfrac{HW}{p^2}$
$N_v = \dfrac{N_p}{s^2} = \dfrac{HW}{p^2 s^2}$
$A = \dfrac{H_0 W_0}{N_v}$
$c = \dfrac{G}{N_v}$
$\mathrm{prefill} = O((N_v + N_t)^2 d)$

Where:

$N_p$ is the patch count the vision encoder produces from a resized input of size $H \times W$ with patch size $p$ , and $N_v$ is the number of tokens actually handed to the language model.
$s$ is the spatial merge factor; pixel unshuffle with $s=2$ concatenates 4 patch embeddings into width $4d$ and projects back to $d$ , so length drops $4\times$ with no averaging.
$H_0 \times W_0$ is the original page resolution and $A$ the original pixels covered by one visual token, which is the honest measure of compression because the resize is itself a compressor.
$G$ is the glyph count on the page and $c$ the glyphs per visual token; $c \approx 1$ is the practical safety line for character-accurate reading.
$N_t$ is the text prompt length and $d$ the model width, so prefill is quadratic in the combined sequence while the KV cache grows linearly.

Worked Example, One Dense A4 Page At 150 DPI:
$N_v = 4 \times 576 + 576 = 2880$
$c = 3200 / 2880 \approx 1.1$
$c = 3200 / 576 \approx 5.6$
$(2880 / 576)^2 = 25$

Assuming about 3,200 glyphs on the page, an AnyRes layout of four 336-pixel tiles plus a thumbnail lands at roughly one glyph per token, while a single 336-pixel view lands at 5.6 and reads only headlines. The last line is the bill: those extra tokens cost 25 times the attention work in prefill, which is precisely why the compressor exists and why the interesting engineering is choosing the smallest $N_v$ that still keeps $c$ near 1 for the document class you serve.

Figure 2: Compression is only meaningful relative to glyph density. Configurations inside the band give each visual token roughly one character and read reliably; a 64-slot resampler asks one token to encode about 50 glyphs. Systems trained specifically for optical text compression can operate right of the band, but at a measured precision cost, and below the band extra tokens buy nothing.

Property	Spatial channel merge	Fixed-query resampler	In-LLM pruning or merging
Mechanism	Pixel unshuffle or strided conv over the patch grid, then a linear projection	Cross-attention from K learned queries into all patch embeddings	Rank tokens by attention received in an early layer, drop or merge the tail
Budget vs resolution	Grows linearly with pixels, fixed ratio of 4x or 16x	Constant at K (typically 32 to 256) whatever the input size	Grows with pixels, then cut by a fixed keep-rate
Spatial index kept	Yes, one token maps to one known rectangle	No, slots are content-addressed and order must be relearned	Yes for survivors, but dropped regions leave holes
Training cost	One small projection, trained with the connector	A full extra transformer stage plus alignment pretraining	Usually training-free, applied at inference
Dominant failure	Long context and quadratic prefill on multi-page inputs	Reading order and rare glyphs collapse on dense pages	Whole text lines vanish when the query is not yet visible to the scorer

August 18, 2026