Category: Medium

DL0173 Continuous vs Static Batching

How does continuous batching (iteration-level scheduling) in serving systems like vLLM and TGI differ from static request batching, and why does it improve GPU utilization under heterogeneous request lengths?

Answer

Static batching schedules at request granularity: the server collects up to $B$ requests, launches one generation loop, and that loop runs until the longest sequence in the group finishes. Every shorter sequence keeps holding its slot, its KV memory, and its lane in every matmul while emitting nothing useful, and a request that arrives one iteration after launch waits for the entire group to drain. Continuous batching, introduced as iteration-level scheduling in Orca and now the default in vLLM and TGI, moves the scheduling decision inside the loop: after every single forward pass the scheduler retires finished sequences, frees their KV blocks, and admits queued requests into the next pass. The batch is re-formed at each of the thousands of decode steps a request lives through, so the number of sequences actually producing a token stays pinned near the memory limit instead of decaying toward one. Because decoding is memory-bandwidth bound, that sustained batch size converts almost directly into tokens per second, which is why the win grows with the variance of output lengths.

(1) Scheduling Granularity: static batching makes one admission decision per batch, continuous batching makes one per forward pass, which is the entire conceptual difference.
(2) No Head-Of-Line Blocking: a finished sequence is evicted and a waiting request admitted on the next iteration, so queueing delay stops scaling with the longest generation in the current group.
(3) Padding Disappears: Orca’s selective batching batches the position-independent linear layers over flattened tokens and runs attention per sequence with variable-length kernels, so no pad tokens are ever computed.
(4) Paged KV Cache Makes Admission Cheap: PagedAttention allocates KV in fixed blocks rather than reserving a contiguous max-length buffer, so freed blocks immediately become admission capacity for a new prompt.
(5) Decode Is Bandwidth Bound: one decode step reads all model weights once regardless of $B$ , so a sustained batch amortizes that read across more tokens and lifts arithmetic intensity roughly linearly.
(6) The Cost Is Prefill Interference: admitting a long prompt injects a compute-heavy prefill into the loop and stalls every decoding sequence, producing inter-token latency spikes that chunked prefill was designed to remove.

Two stacked Gantt charts of four GPU batch slots across decode iterations for the same eight requests. The top chart shows static batching: the first four requests start together, three of them finish early and leave hatched idle slots until the longest request finishes at iteration 16, at which point the second group of four is admitted and finishes at iteration 23. The bottom chart shows continuous batching: as soon as a short request finishes, a queued request is admitted into the freed slot, so all eight requests complete by iteration 16 with far fewer idle cells.

Figure 1: The same eight requests and the same 50 sequence-iterations of useful decode work, scheduled two ways. Static batching spreads them over 23 iterations at 54% slot occupancy because the group cannot retire until its longest member does; continuous batching backfills every freed slot on the next iteration and finishes in 16 at 78% occupancy. The hatched cells are the entire cost of request-level scheduling: paid compute and reserved KV memory that produce no tokens.

The mechanics that make iteration-level scheduling possible are as important as the policy. A naive implementation would need all sequences in a batch to sit at the same generation position so the whole thing is one dense padded tensor, which is exactly why static batching pads. Selective batching breaks the batch apart: the QKV projections, MLP, and output head act on tokens independently, so they run over a flattened ragged tensor, while attention is dispatched per sequence with its own context length. PagedAttention then removes the second obstacle, memory fragmentation, by storing KV in fixed-size blocks (typically 16 tokens) that need not be contiguous, so a sequence grows block by block and a new arrival can be admitted whenever a handful of blocks are free. With both pieces in place, admission is bounded by free KV blocks rather than by a pre-declared batch shape, and the scheduler’s job becomes a per-iteration packing problem over a memory budget.

Why this shows up as GPU utilization is a roofline argument rather than a scheduling one. Generating one token for one sequence requires reading every weight from HBM, roughly 16 GB for an 8B model in fp16, but only about $2P$ FLOPs of arithmetic, so a batch of one runs at a tiny fraction of peak FLOPs and the GPU is idle waiting on memory. Adding sequences to the same forward pass reuses that single weight read for more tokens, so throughput climbs steeply until the KV-cache reads and finally the matmuls take over. Static batching’s effective batch size decays as its short members retire, so it spends most of its time in the low-intensity regime; continuous batching holds the batch near $B_{\max}$ and stays in the high-intensity regime. Reported end-to-end gains follow from this: Orca measured up to 36.9x throughput over a static FasterTransformer baseline at matched latency, and vLLM measured a further 2x to 4x from paged memory alone.

Line chart of aggregate decode throughput in tokens per second versus the number of sequences in the decode batch, from one to one hundred twenty-eight, for an eight-billion-parameter fp16 model with a one-thousand-token context on a two-terabyte-per-second GPU. The curve rises steeply and then bends as KV cache reads grow, with two marked points: an average effective batch of seventeen for static batching at about eighteen hundred tokens per second, and a sustained batch of thirty-two for continuous batching at about thirty-two hundred tokens per second.

Figure 2: Decode throughput against sustained batch size for an 8B fp16 model with 1K-token contexts on a 2 TB/s GPU. The curve is bandwidth-bound everywhere in this range, so throughput rises almost linearly at small $B$ and bends only as KV reads start to rival the 16 GB weight read. Continuous batching does not move the curve; it moves the operating point, from the average effective batch a draining static group achieves to the memory-limited maximum.

Mathematical Formulation:
$U_{\mathrm{static}} = \frac{\sum_{i=1}^{B} L_i}{B \, L_{\max}}$
$b_{\mathrm{kv}} = 2 \, n_l \, h_{kv} \, d_h \, s$
$t_{\mathrm{step}}(B) \approx \frac{2P + B \bar{L} b_{\mathrm{kv}}}{\mathrm{BW}}$
$\lambda(B) = B \, / \, t_{\mathrm{step}}(B)$
$B_{\max} = M_{\mathrm{free}} \, / \, (\bar{L} \, b_{\mathrm{kv}})$

Where:

$U_{\mathrm{static}}$ is the fraction of batch slot-iterations that produce a token under static batching, with $L_i$ the output length of request $i$ and $L_{\max} = \max_i L_i$ ; it is exactly 1 only when all lengths are equal.
$b_{\mathrm{kv}}$ is KV bytes per token, where $n_l$ is layers, $h_{kv}$ key/value heads (small under GQA), $d_h$ head dimension, $s$ bytes per element, and the leading 2 counts K and V.
$P$ is the parameter count, so $2P$ bytes is the fp16 weight read paid once per iteration regardless of $B$ , and $\mathrm{BW}$ is achievable HBM bandwidth.
$\bar{L}$ is the mean context length in the running batch and $B$ the number of sequences decoding in one forward pass.
$\lambda(B)$ is aggregate decode throughput in tokens per second; it is concave in $B$ because the KV term grows with $B$ while the weight term does not.
$M_{\mathrm{free}}$ is HBM left after weights and activations, so $B_{\max}$ is the admission ceiling the scheduler targets; the formula assumes $\bar{L}$ stays bounded, which it does not for sequences still growing.

Throughput At Two Operating Points (8B fp16, 1K context, 2 TB/s):
$\bar{L} \, b_{\mathrm{kv}} = 1024 \times 128\ \mathrm{KB} = 0.13\ \mathrm{GB}$
$t_{\mathrm{step}}(17) = 18.2 / 2000 = 9.1\ \mathrm{ms}$
$t_{\mathrm{step}}(32) = 20.2 / 2000 = 10.1\ \mathrm{ms}$
$\lambda(17) \approx 1870$
$\lambda(32) \approx 3170$

The byte totals are the 16 GB weight read plus $B \times 0.13$ GB of KV, divided by 2000 GB/s. Nearly doubling the sustained batch costs only 11% more time per step and yields 1.7x the tokens per second, which is the whole economic case for iteration-level scheduling: the extra sequences ride along in memory traffic that was already being paid.

Property	Static (request-level) batching	Continuous (iteration-level) batching	Continuous + chunked prefill
Scheduling unit	One whole generation loop per group	One forward pass	One forward pass with a token budget split across prefill and decode
Wait for a new arrival	Until the longest sequence in the current group finishes	One iteration, if KV blocks are free	One iteration, and its prefill is spread over several
Wasted compute	Pad tokens plus idle slots, growing with length variance	Near zero: ragged attention, no padding	Near zero, with better SM occupancy on decode-only steps
KV memory model	Contiguous buffer reserved for max length per slot	Paged blocks allocated on demand, freed on retirement	Same paged blocks, filled incrementally during prefill
Inter-token latency	Stable within a group, terrible queueing before it	Spikes whenever a long prompt is admitted	Bounded by the chunk size, at slightly higher TTFT
Dominant failure mode	Throughput collapse under heavy-tailed output lengths	KV exhaustion causing preemption and recompute thrash	Chunk size mistuned, trading TTFT against throughput

August 19, 2026

DL0157 VLA Zero-Shot and Few-Shot Transfer

How do you evaluate zero-shot and few-shot transfer of generalist VLA policies to novel physical environments and unseen manipulation objects, for a checkpoint such as Google DeepMind’s RT-2 or the open OpenVLA model?

Answer

Evaluating transfer is a measurement design problem, not a demo reel. The protocol must vary one generalization axis per evaluation cell, hold everything else fixed with a scripted scene reset and matched initial states, and report a per-axis success rate with a confidence interval instead of one aggregate number. Zero-shot cells run the frozen checkpoint at $k = 0$ demonstrations, while few-shot cells sweep $k \in \{1, 5, 10, 25\}$ target demonstrations and report a curve plus the retention on the original seen-task suite after adaptation. The trial budget sets the resolution of the entire experiment, because 20 trials per cell give a 95% interval of roughly $\pm 22$ points at $\hat{p} = 0.5$ , which cannot resolve the 10 to 15 point differences people actually argue about. Cheap simulation suites such as LIBERO and SIMPLER buy statistical power and reproducibility, and real-robot paired trials buy validity, so a credible report uses both and checks that they rank policies the same way.

(1) Factor-Isolated Axes: Split “novel” into unseen object instance, unseen object category, unseen scene and background, added distractors and camera pose, and rephrased language. A single mixed “hard eval” set cannot attribute a failure to any one of them.
(2) Matched Initial States: Every policy under comparison sees the same object poses, lighting, and clutter, logged from a reset script or a fixture template, which turns a noisy independent comparison into a paired test with far more power.
(3) Trial Budget Sets Resolution: Detecting a 10-point difference at 95% confidence needs roughly 190 trials per arm, so 20-trial evals should be reported as pilots, never as rankings.
(4) Progress Score, Not Only Binary Success: Score reach, grasp, transport, and place as staged partial credit in $[0, 1]$ , which exposes whether a novel object breaks perception or grasp geometry.
(5) Few-Shot Is A Curve: Report $\hat{p}(k)$ over several $k$ values and fit a demo-efficiency constant, because a single $k = 10$ number hides whether the policy needed 2 demos or 50.
(6) Retention And Blind Operation: Re-run the seen-task suite after fine-tuning to measure catastrophic forgetting, and keep the operator unaware of which checkpoint is running to remove reset and intervention bias.

Protocol diagram with five generalization axis lanes on the left (unseen object instances, unseen object categories, unseen scene, unseen distractors with camera pose shift, rephrased instructions) feeding a tall paired-trial harness box that specifies scripted resets, matched initial states, a blind operator, randomized trial order, N trials per cell and a staged progress score, which then splits into a zero-shot arm at k equals zero demos and a few-shot arm at k equals 1, 5, 10, 25 demos, each producing its own report block

Figure 1: One checkpoint, one factor per cell. The harness is the part people skip: without scripted resets and matched initial states, the operator becomes an uncontrolled variable, and the same policy can swing 20 points between two afternoons. Each cell also carries a seen-object control in the new scene, so a gap can be attributed to the object rather than to the room.

The protocol hygiene matters more than the model comparison. Each trial follows a fixed loop, reset → verify pose against a reference photo → run for a hard time limit → score stages → log the episode, and every episode is kept, including ones the operator considers unfair, because discarding “bad setups” is how a 55% policy becomes an 80% policy on paper. Interventions must be recorded rather than silently corrected, since a policy that needs one nudge per episode is not transferring. On the few-shot side, the demonstrations for a novel object must be collected under the same distribution the evaluation will use, and the evaluation objects must be held out from those demonstrations; reusing the same physical mug for teaching and testing measures memorization, not transfer. Finally, adaptation method is part of the result: LoRA on the action head, full fine-tuning, and fine-tuning with a replay mix of pretraining data produce very different retention curves at the same novel-task success.

Mathematical Formulation:
$\hat{p} = \frac{1}{N}\sum_{i=1}^{N} s_i$
$\mathrm{CI}_{95} = \hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/N}$
$G_a = \hat{p}_{\mathrm{seen}} - \hat{p}_a$
$N \geq 2 z^2 \bar{p}(1-\bar{p}) / \Delta^2$
$\hat{p}(k) = p_{\infty} - (p_{\infty} - p_0)e^{-k/\tau}$

Where:

$\hat{p}$ is the estimated success rate of one evaluation cell and $s_i$ the outcome of trial $i$ , either binary or a staged progress score in $[0, 1]$ .
$N$ is the number of trials in that cell, the single quantity that decides whether the reported number can support a claim.
$G_a$ is the generalization gap on axis $a$ , measured against the seen-task control run in the same session and the same scene.
$z = 1.96$ at 95% confidence, $\bar{p}$ is the pooled rate of the two arms, and $\Delta$ is the minimum detectable difference you are willing to claim.
$k$ is the number of target-domain demonstrations, $p_0$ the zero-shot rate, $p_{\infty}$ the saturation rate, and $\tau$ the demo-efficiency constant in demonstrations, so smaller $\tau$ means faster adaptation.
The unpaired formula for $N$ is an upper bound; matched initial states reduce the variance of the difference and can cut the required $N$ by a factor of two or more.

Trial Budget For A 10-Point Claim:
$N \geq 2(1.96)^2(0.5)(0.5)/(0.10)^2$
$N \geq 192$

At roughly 90 seconds per real trial including reset, 192 trials per arm is about 5 hours of operator time for a single cell, and a five-axis grid with two policies multiplies that by ten. That arithmetic, not modelling taste, is why teams push generalization sweeps into simulation and reserve real hardware for a small number of decisive paired comparisons, and why crowd-sourced or autonomous evaluation has become an active research direction.

Two panel chart: left panel shows per-axis success rate bars for seen tasks, novel object instance, novel object category, novel scene, added distractors and rephrased instruction, each with a wide red 95 percent interval for 20 trials and a narrow black interval for 200 trials, showing that the 20 trial intervals overlap across axes; right panel plots success rate against number of demonstrations per novel task at k equals 0, 1, 3, 5, 10 and 25 for novel object and novel scene curves rising toward saturation, plus a declining dashed curve for retention on the original seen task suite after naive fine-tuning

Figure 2: Left, the same measurements at two trial budgets. With 20 trials per cell the intervals for novel instance, distractors, and rephrasing all overlap, so the axis ranking is unsupported; at 200 trials the ordering becomes real. Right, few-shot transfer is a curve with a second axis nobody reports: novel-task success climbs with $k$ while seen-task retention decays under naive fine-tuning.

Property	Simulation suites (LIBERO, SIMPLER, CALVIN)	Real-robot paired trials
Cost of 100 trials	Minutes of wall clock, fully parallel across workers	2 to 3 hours of operator time plus fixtures and resets
Axes varied cheaply	Object mesh and texture, lighting, camera pose, distractor count, instruction text	Anything physically buildable, but each new factor costs a scene rebuild
What it misses	Contact dynamics, deformables and liquids, actuator lag, real camera noise and exposure	Nothing by construction, since it is the deployment distribution
Achievable precision	Thousands of trials, interval within a few points	20 to 50 trials is typical, giving intervals of 15 to 22 points
Dominant validity risk	Rank inversion versus hardware, and policies tuned to the renderer	Operator drift, inconsistent resets, unlogged interventions, cherry-picked episodes
Right role	Wide axis sweeps, regression gates, hyperparameter and checkpoint selection	A few decisive head-to-head comparisons and the final transfer claim

August 19, 2026

DL0155 Crowdsourced Teleoperation Quality

How do you handle dataset quality variance in crowdsourced teleoperation data when training large VLA policies, for a collection like DROID or Open X-Embodiment?

Answer

Crowdsourced teleoperation data is a mixture over operators, rigs, and sessions, not a single distribution. DROID’s 76k trajectories were collected by about 50 operators across 13 institutions, and Open X-Embodiment pools 60 datasets over 22 embodiments, so per-episode quality varies more than the task labels do. The working assumption is that quality is a measured per-episode variable, not a binary success flag, so the first engineering step is a scoring pass that attaches provenance plus cheap kinematic proxies to every episode and stores them as metadata rather than deleting anything. Training then spends that metadata at three separate points: group weights over operator or site during pretraining, a quality token in the conditioning so the model learns the difference between clean and sloppy behaviour, and a short anneal on a curated gold subset that decides what the policy actually imitates at test time. Hard filtering is the last resort, because imitation error compounds as $O(\epsilon H^2)$ and most of that $\epsilon$ comes from states the policy has never visited, which is exactly what the messy tail supplies. Two mechanical fixes matter as much as the curation: percentile-based action normalization so one jerky operator cannot rescale the action space, and an expressive action head over action chunks so contradictory strategies are not averaged into a mean action.

(1) Quality Is A Variable, Not A Flag: score every episode continuously and keep the score, because a policy trained on a thresholded boolean cannot be re-mixed later without a second scoring pass.
(2) Cheap Automatic Proxies: action jerk, path-length ratio, idle-time fraction, regrasp and retry count, teleop latency spikes, and VLM-based verification of the success label and the language annotation.
(3) Provenance Is The Grouping Unit: operator, site, and rig ids define the groups used for reweighting and, critically, for held-out evaluation splits so a strong operator’s style does not leak across train and test.
(4) Coverage Versus Purity: the sloppy tail carries the recovery states and rare scenes that keep compounding error small, so aggressive filtering can lower success even while raising average demo quality.
(5) Pretrain Broad, Post-Train Narrow: the data-pyramid recipe used by recent VLA foundation models trains on nearly everything and then anneals on a small, verified, high-quality set.
(6) Quality-Conditioned Cloning: feed the tier as a token during training and sample at the gold setting at inference, which uses bad data as negative evidence instead of throwing it away.
(7) Robustness Plumbing: 1st/99th percentile action normalization as in OpenVLA, plus flow-matching or diffusion heads, so outliers and multimodality do not corrupt the regression target.

Pipeline diagram: three crowdsourced collection sites with per-operator episode counts feed an automatic episode scoring stage using VLM success checks, action jerk, path ratio, idle fraction, regrasps and latency spikes; the scorer assigns gold, silver and bronze tiers; all three tiers feed one training recipe box listing group-DRO pretraining weights, a quality token in the prompt, and a final anneal on the gold tier; the recipe trains a VLA policy that is evaluated on held-out sites, with a feedback arrow returning operator scorecards and re-collection requests to the sites

Figure 1: The topology is what matters: scoring is a metadata pass, not a delete pass, all three tiers reach the pretraining stage, and only the last stage is restricted to gold. The closing arrow is the part teams forget, since per-operator scorecards turn quality measurement into targeted re-collection rather than a one-time filter.

The scoring pass should stay cheap enough to run over every episode. Kinematic proxies are computed directly from the recorded action stream: mean squared jerk over the trajectory, the ratio of executed end-effector path length to the straight-line distance, the fraction of timesteps with near-zero commanded velocity, and the number of gripper open/close reversals as a regrasp counter. Semantic proxies need a model: a VLM watches the final frames and votes on whether the stated goal was reached, and a second pass checks that the language annotation matches the video, which catches the common crowdsourcing failure where the instruction and the demonstration disagree. These proxies are then aggregated into one score per episode and, separately, per operator. Reweighting uses the group statistic because per-episode weights are noisy and because the real covariate shift is at the operator and rig level, where calibration offsets, camera mounts, and latency are shared.

Mathematical Formulation:
$q_j = \sigma(\beta^{\top} \phi(\tau_j))$
$\mathcal{L}_g(\theta) = \mathbb{E}_{(o,a) \sim D_g}[\ell(a, \pi_{\theta}(o))]$
$\min_{\theta} \max_{w \in \Delta_G} \sum_{g=1}^{G} w_g \mathcal{L}_g(\theta)$
$J(\pi^{*}) - J(\hat{\pi}) \leq C \epsilon H^2$
$\tilde{a} = 2 (a - q_{01}) / (q_{99} - q_{01}) - 1$

Where:

$q_j \in (0,1)$ is the quality score of episode $\tau_j$ , with $\phi$ the proxy feature vector (jerk, path ratio, idle fraction, regrasps, VLM verdict) and $\beta$ fitted on a few hundred human-labelled episodes.
$\mathcal{L}_g$ is the loss on group $g$ , where a group is one operator, site, or source dataset, and $D_g$ its observation-action pairs with observation $o$ (images plus instruction plus proprioception) and action chunk $a$ .
$g \in \{1,\ldots,G\}$ indexes groups, $\Delta_G$ is the simplex of mixture weights, and the inner maximization is group DRO, which upweights whichever group is currently worst-fit instead of trusting a hand-tuned mixture.
$\pi_{\theta}$ is the policy, $\ell$ the per-sample cloning loss, $H$ the rollout horizon, and $\epsilon$ the per-step error under the expert’s state distribution; the $H^2$ factor is why coverage of off-nominal states is worth more than average demo neatness.
$q_{01}$ and $q_{99}$ are the 1st and 99th percentiles of each action dimension over the training mixture, so $\tilde{a} \in [-1,1]$ is a quantile-normalized target that a single spiking operator cannot stretch.

The trade-off curve is the part that surprises people. Sorting episodes by score and keeping only the top fraction improves the average target the policy regresses onto, but it also removes scenes, lighting conditions, object instances, and above all recovery segments: the moment where an operator overshot, re-approached, and succeeded is precisely the state a deployed policy will find itself in. Empirically the filter-only curve is an inverted U, peaking somewhere around half the data and then falling below the train-on-everything baseline once coverage collapses. Reweighting dominates filtering because it keeps every state in the support while shrinking the gradient contribution of noisy actions, and the gold anneal captures most of the remaining gain at almost no cost since it touches only the final few percent of optimization steps.

Line chart of real-robot success rate against the fraction of training data retained when keeping the highest-quality episodes first. The hard-filter curve is an inverted U peaking near 55 percent retained and dropping sharply below the baseline at 10 percent retained, while the reweight-plus-gold-anneal curve rises monotonically and is highest when all data is retained. A dash-dotted horizontal line marks the uniform train-on-everything baseline.

Figure 2: Two different shapes from the same scores. Hard filtering trades coverage for purity and therefore has an interior optimum you must search for, while reweighting plus a gold anneal keeps the full state support and improves monotonically as more data is retained. The gap at the right edge is the cost of deleting data you could have downweighted instead.

Quality variance also shows up as multimodality, which is a modelling problem rather than a data problem. When two operators route around the same obstacle in opposite directions, the conditional action distribution given the observation is bimodal, and any head trained with a plain mean-squared-error objective converges to $\mathbb{E}[a \mid o]$ , the average of the two modes. That average is often an invalid action, so the policy fails on the exact scenes where it had the most data. The fixes are an expressive head (diffusion or flow matching over an action chunk, as in the pi-zero family) which samples a single coherent mode, action chunking so a commitment persists for several timesteps instead of flip-flopping per step, and conditioning on style or operator id when the modes are genuinely operator-specific. This is also why “smooth” should never be the only quality criterion: a smooth demonstration that solves the task differently from the rest of the corpus still injects a competing mode.

Two panels sharing the same scene with a start point, a goal, and a rectangular obstacle between them. Left panel: one operator's demonstrations arc above the obstacle, another operator's arc below, and a red dashed straight line at the midline shows that the mean-squared-error head predicts the average of both modes and drives into the obstacle. Right panel: a flow-matching or diffusion head produces three sampled trajectories above and three below, each committing to a single side and clearing the obstacle.

Figure 3: Mixed-operator data is multimodal in action space. A unimodal regression head collapses two valid strategies onto their mean, which here is a straight line through the obstacle, while a sampling head over action chunks commits to one mode per rollout. No amount of filtering fixes this if both strategies are high quality.

Property	Hard filtering	Group reweighting + gold anneal	Quality-conditioned cloning
Mechanism	Drop every episode below a score threshold before training	Per-group loss weights during pretraining, then a short fine-tune on the gold tier	Tier token in the conditioning; sample at the gold setting at inference
Effect on coverage	Shrinks the state support, removes rare scenes and recovery segments	Full support retained; only the gradient share of noisy groups shrinks	Full support retained; bad data acts as contrastive evidence
Extra machinery	One threshold, tuned by expensive real-robot sweeps	Group ids, a weight optimizer such as group DRO, a second training stage	Score discretization, a token in the prompt, calibrated tier boundaries
When it wins	Labels are corrupt rather than merely sloppy, or the episode is unsafe to imitate at all	Large heterogeneous corpora where coverage is the scarce resource	Scores are reliable and the sloppy modes are still physically valid
Main failure mode	Proxy measures task difficulty, so filtering silently deletes the hard tasks	Weights overfit one noisy group; the anneal overfits and forgets the broad prior	Miscalibrated tiers make the gold token meaningless, and the policy ignores it

August 19, 2026

DL0154 Synthetic Data and Isaac Sim

How do synthetic data generators such as Isaac Sim and Omniverse produce realistic tactile, force sensor, and RGB-D depth outputs for VLA pre-training?

Answer

Nothing in the stack emits a realistic sensor reading directly. Each modality comes out of a different subsystem with its own fidelity ceiling, and realism is added afterwards as an explicit sensor model. RGB and depth are produced by the RTX renderer and read back through Omniverse Replicator annotators, where distance_to_image_plane returns per-pixel metric depth with no holes, no quantization, and no stereo shadows. Joint and contact forces are read out of the PhysX 5 reduced-coordinate articulation solver, so a “measured” wrist force is really a constraint impulse divided by the physics timestep. Tactile is not a first-class sensor at all: it is reconstructed from the signed-distance penetration between a gel collider and the object, then either rendered as a GelSight-style image or collapsed into a taxel normal-force map. Everything the simulator hands you is exact, and the actual engineering is deciding how to break it so a VLA trained on it survives a real RealSense and a real six-axis load cell.

(1) Ground Truth First, Noise Second: the renderer and the solver give error-free signals, so every realism claim rests on a post-processing layer that injects noise, quantization, dropout, bandwidth limits, and latency.
(2) RGB-D Is A Rendering Plus Annotator Problem: PBR materials and HDRI domes handle appearance, while tiled rendering makes many cameras affordable, but no renderer reproduces stereo matching failure on specular and transparent surfaces unless you model it.
(3) Forces Are Solver Readouts: get_measured_joint_forces and the contact reporter expose impulses at the physics rate, so magnitudes depend on timestep, substeps, solver iterations, and contact offset rather than on any sensor.
(4) Tactile Is Reconstructed Geometry: the penetration field against the object SDF drives both the rendered gel image and the taxel forces, with shear obtained by clipping tangential displacement to the Coulomb friction cone.
(5) Physics Calibration Sets Force Fidelity: friction, mass, inertia, restitution, and SDF colliders instead of convex hulls matter more to contact realism than any amount of visual polish.
(6) Scale Comes From Trajectories, Not Pixels: Replicator randomizes lighting, materials, poses, and per-environment physics while MimicGen-style augmentation turns a few dozen human demos into hundreds of thousands of contact-rich episodes.

Pipeline diagram in which a USD scene feeds both a PhysX 5 GPU solver and the RTX renderer, which in turn feed three sensor lanes for joint and contact forces, a tactile penetration field, and RGB-D annotators, all passing through a shared sensor realism layer of noise, quantization, invalid masks, filtering and latency before a dataset writer and VLA pre-training

Figure 1: Three sensor lanes, one scene graph. Only the RGB-D lane is a genuine sensor simulation; the force lane is a solver readout and the tactile lane is reconstructed from geometry, which is why the shared realism layer does most of the sim-to-real work.

Depth is where naive pipelines fail first. The annotator gives exact metric depth, so a policy trained on it learns to trust razor-sharp object boundaries, reads valid depth off glass and polished steel, and never sees the occlusion shadow that a stereo baseline creates on the left edge of every object. A usable depth channel is therefore built by emulating the device: convert depth to disparity, quantize it, add matching noise that grows as $z^2$ , invalidate pixels where the simulated left and right views disagree or where the material is specular or transparent, quantize to the 1 mm uint16 grid, and finally apply the frame delay and rolling-shutter smear of the real driver. Two annotator traps show up in review: distance_to_camera returns Euclidean range while distance_to_image_plane returns the $z$ component that RGB-D APIs expect, and a policy trained on the wrong one develops a radial bias that grows toward the image corners.

Line chart of injected axial depth noise in millimetres versus range in metres for stereo baselines of 50, 95 and 120 millimetres, all growing quadratically with range, plus a flat zero-error line for the simulator raw depth and a dash-dotted line marking the one millimetre uint16 quantization step

Figure 2: Depth realism is arithmetic, not rendering. With $f = 600$ px and $\sigma_d = 0.1$ px, a 50 mm baseline is submillimeter at 0.5 m and roughly 13 mm at 2 m, while the simulator’s raw depth sits on the flat zero-error line at every range.

Mathematical Formulation:
$z = \frac{f b}{d}$
$\sigma_z = \frac{z^2}{f b}\,\sigma_d$
$\tilde z = q\,\mathrm{round}(z/q) + \eta$
$F_{\mathrm{meas}} = \lambda_c / \Delta t$
$d(u,v) = \max(0,\, -\phi(x_{uv}))$
$\|f_t\| \leq \mu f_n$

Where:

$z$ is metric depth, $f$ the focal length in pixels, $b$ the stereo baseline, and $d$ the disparity in pixels.
$\sigma_d$ is subpixel matching noise, typically 0.05 to 0.2 px, and the $z^2$ factor is why one camera is millimeter-accurate up close and centimeter-accurate at 2 m.
$q$ is the depth quantization step (1 mm for uint16 output) and $\eta$ the residual additive noise; a separate invalid mask zeroes stereo-shadow, specular, and transparent pixels.
$\lambda_c$ is the constraint impulse the solver applied at a joint or contact and $\Delta t$ the physics step, so the reported force scales as $1/\Delta t$ and is rate and solver dependent rather than physical.
$\phi$ is the object’s signed distance function evaluated at gel sample $x_{uv}$ , and $d(u,v)$ is the penetration depth field on the tactile grid that drives both the gel image and the taxel normal force.
$f_n$ and $f_t$ are normal and tangential contact force with friction coefficient $\mu$ ; marker flow exists only inside the cone, and slip appears when the bound is reached.

On the force side the honest framing is that PhysX gives you a well-behaved dynamics readout, not a load cell. A stiff impact resolved in one 8.3 ms step at 120 Hz appears as a single enormous impulse, whereas a real 1 kHz sensor reports a damped ringdown shaped by its own mechanical resonance plus a slowly drifting bias and a few percent of cross-axis coupling. The practical recipe is more substeps or a smaller $\Delta t$ for contact-rich phases, a contact history buffer in the sensor wrapper, then a low-pass filter to the real sensor bandwidth followed by injected bias, drift, and colored noise. Tactile needs the same discipline plus a calibration step: the elastomer in simulation is rigid geometry, so penetration depth substitutes for deformation, and the mapping from $d(u,v)$ to gel pixel intensity has to be fitted against real presses on known indenters. Hysteresis, adhesion, and creep are simply absent, which bounds how far a purely simulated tactile channel can carry a slip-detection policy.

Two panels: a cross-section showing an object surface pressing 1.2 millimetres into a 3 millimetre gel layer with the penetration region shaded and the contact radius marked, and a heatmap of the resulting penetration depth field over a 12 by 12 millimetre tactile grid with radial shear arrows and a dashed contact boundary

Figure 3: Tactile output is reconstructed, not sensed. The penetration field $d(u,v) = \max(0, -\phi)$ against the object SDF supplies both the rendered gel image and the taxel normal-force map, while tangential flow is only valid inside $\|f_t\| \leq \mu f_n$ .

Property	RGB-D	Joint and contact force	Tactile
Source subsystem	RTX renderer plus Replicator annotators	PhysX 5 articulation and contact solver	SDF penetration query plus a gel renderer (TacSL, TACTO)
Native output	Exact metric depth, segmentation, normals, no holes	6D joint reaction force and net contact force at the physics rate	Penetration depth field on the gel grid, plus a shear field
Fidelity ceiling	Asset and light-transport quality; stereo and ToF failure modes are not rendered	Friction, mass, inertia, solver iterations, contact and rest offset	Gel treated as rigid geometry, so hysteresis, adhesion and creep are missing
Mandatory post-processing	Disparity quantization, $z^2$ noise, invalid mask, blur, frame latency	Low-pass to sensor bandwidth, bias drift, cross-axis coupling, spike clipping	Illumination and marker calibration on real presses, Coulomb-clipped shear
Real data still needed	Little for RGB, moderate for depth on shiny and transparent scenes	Small for quasi-static tasks, large for impacts and insertion	Large: a real fine-tuning set of presses, rolls and slips is usually unavoidable

August 19, 2026

DL0153 Sim-to-Real Transfer

What is sim-to-real transfer in robot learning, and how do domain randomization, system identification, and visual domain adaptation each close a different part of the reality gap?

Answer

Sim-to-real transfer is training a control policy in a physics simulator, where data is fast, safe, and infinitely resettable, and then deploying it on hardware that obeys different physics and produces different pixels. The reality gap is the mismatch between the simulated MDP and the real one, and it decomposes into at least three channels: dynamics parameters that are simply mistuned (mass, friction, joint damping, motor gains), unmodeled effects that the simulator has no term for (cable drag, gear backlash, control latency, deformable contact), and the observation gap between rendered and captured images. The three classic techniques are not competitors; each attacks a different channel. Domain randomization trains one policy over a distribution of simulators so the real robot behaves like just another sample from that distribution, system identification measures the real robot and moves the simulator’s parameters onto it, and visual domain adaptation leaves dynamics alone and instead makes sim and real images look identical to the encoder. Production pipelines for legged locomotion and dexterous manipulation compose them: identify what you can measure, randomize what you cannot, adapt the residual online with a history-conditioned policy.

(1) The Gap Is Not One Number: dynamics, unmodeled effects, and observations fail independently, so a policy can transfer perfectly in torque space and still collapse because the real camera has motion blur.
(2) Domain Randomization Widens The Simulator: sampling $\xi \sim p(\xi)$ each episode buys robustness without any real data, provided the true parameters lie inside the support.
(3) System Identification Shrinks The Gap Itself: fitting $\xi$ to logged real trajectories gives a sharper simulator and a higher-performance policy, but only for effects the simulator can represent.
(4) Visual Domain Adaptation Works In Pixel Space: feature alignment, randomized-to-canonical translation, or image-to-image GANs remove appearance shift that no amount of physics tuning touches.
(5) Support Coverage Is The Binding Constraint: if $\xi_{\mathrm{real}}$ falls outside the randomization support, the policy has literally never trained on the real robot’s physics and no amount of averaging helps.
(6) Online Adaptation Recovers The Lost Performance: conditioning on a short history of states and actions lets the policy infer the latent parameters at run time, which is why teacher-student distillation and rapid motor adaptation beat blind robustness.

Two-panel conceptual diagram. Left panel shows a dynamics parameter space with axes friction coefficient and payload mass, a blue dot for the simulator nominal parameters, a red star for the real robot parameters, a large shaded rectangle marking the randomization support that contains the star, and a dashed arrow from the nominal point to the star labeled system identification. Right panel shows a visual feature space with a tight cluster of sim render points and a separate cluster of real camera image points, a large tilted ellipse marking the spread produced by visual randomization that envelops both clusters, and a curved arrow mapping the real cluster onto the sim cluster labeled domain adaptation.

Figure 1: The same gap, two spaces and two directions of correction. In parameter space randomization inflates the simulator’s support until it contains the real robot, while system identification translates the nominal point onto it. In feature space randomization spreads the sim distribution until it envelops the real look, while adaptation pulls real observations back onto the canonical sim appearance.

The mechanisms differ in what they need and what they cost. Domain randomization needs zero real data: you pick ranges for mass, friction, terrain roughness, motor strength, sensor noise, latency, and texture, then resample every episode so the policy cannot memorize one dynamics model. Its price is conservatism, because a memoryless policy maximizing expected return over a wide $p(\xi)$ converges to behavior that is safe for the average simulator and optimal for none, which is why Automatic Domain Randomization grows the ranges only as fast as the policy can absorb them. System identification runs in the opposite direction: excite the hardware, log $(s_t, a_t, s_{t+1})$ , and minimize prediction residuals to recover $\hat{\xi}$ . It buys back performance and shrinks the randomization ranges you still need, but it can only fit parameters the simulator exposes, so backlash or a compliant tendon that has no corresponding term stays invisible no matter how good the fit is. Visual domain adaptation is orthogonal to both: RCAN-style translation maps heavily randomized or real images to one canonical rendering before the policy sees them, GraspGAN-style translation pushes sim images toward realism, and feature-level alignment penalizes the discrepancy between sim and real encoder activations. The standard modern recipe is a pipeline: identify → randomize the residual uncertainty → train a privileged teacher with access to $\xi$ → distill into a student that reads only onboard history → deploy.

Mathematical Formulation:
$\Delta(\pi) = J_{\mathrm{real}}(\pi) - J_{\mathrm{sim}}(\pi)$
$\pi_{\mathrm{DR}} = \arg\max_{\pi} E_{\xi \sim p(\xi)} [ J_{\xi}(\pi) ]$
$\xi_{\mathrm{real}} \in \mathrm{supp}(p(\xi))$
$\hat{\xi} = \arg\min_{\xi} \sum_{t=1}^{T} \| e_t(\xi) \|^2$
$e_t(\xi) = s_{t+1} - f_{\xi}(s_t, a_t)$
$\min_{g} \; d( g(o_{\mathrm{sim}}), g(o_{\mathrm{real}}) )$

Where:

$J_{\mathrm{real}}$ and $J_{\mathrm{sim}}$ are the expected returns of policy $\pi$ on hardware and in simulation, and $\Delta(\pi)$ is the reality gap measured in task performance rather than in physics units.
$\xi$ is the vector of simulator parameters (masses, friction coefficients, motor gains, latencies, texture and lighting seeds) and $J_{\xi}$ is the return in the simulator instantiated with $\xi$ .
$p(\xi)$ is the randomization distribution; the support condition is the precondition for transfer, since a policy trained on $p(\xi)$ has no guarantee at all for parameters outside $\mathrm{supp}(p(\xi))$ .
$\hat{\xi}$ is the identified parameter estimate, $f_{\xi}$ the simulator’s one-step transition model, and $e_t$ the one-step prediction residual against a logged real transition from state $s_t$ under action $a_t$ .
$t \in \{1, \ldots, T\}$ indexes the real excitation trajectory; residual that cannot be driven to zero by any $\xi$ is unmodeled physics, and it is exactly what must be randomized or adapted away instead.
$g$ is the visual encoder, $o_{\mathrm{sim}}$ and $o_{\mathrm{real}}$ are observations from the two domains, and $d$ is a distribution distance such as an adversarial discriminator loss or a maximum mean discrepancy.

Line chart of task success rate versus randomization half-width as a fraction of the nominal parameter value. The sim-domain success curve starts near 0.97 at zero width and decays gently to about 0.73 at full width. The real-world success curve starts near zero, rises steeply once the support begins to contain the true parameters around a width of 0.3, peaks near 0.89 around 0.45, then declines to about 0.39 at full width. A dotted vertical line marks where the real parameter enters the support and a shaded band marks the practical operating range.

Figure 2: Randomization width has an inverted-U effect on real-world success while sim success decays monotonically, so sim reward is an actively misleading model-selection signal. Too narrow and $\xi_{\mathrm{real}}$ sits outside the support; too wide and the policy averages over incompatible dynamics. System identification moves the useful band left by removing uncertainty you no longer need to cover.

Property	Domain randomization	System identification	Visual domain adaptation
Gap channel it closes	Unknown dynamics and appearance, covered by breadth	Mistuned but representable dynamics parameters	Observation shift between rendered and captured images
Real data required	None, only sensible ranges	Excitation trajectories on the target robot	Unlabeled real images, no reward or actions needed
Main cost	Sample complexity plus a conservative policy	Hardware time, and it must be redone per unit and as parts wear	A second generative or adversarial model to train and maintain
Dominant failure mode	Support misses the true parameters, or breadth destroys performance	Overfits to one robot and cannot fit unmodeled effects	Translation hallucinates or drops task-critical detail; dynamics untouched
Best fit	Contact-rich locomotion and manipulation across a fleet	High-precision tasks on one well-instrumented platform	Image-based grasping and navigation with cheap renderers

August 19, 2026

DL0144 Synthetic Captions vs Alt-Text

What is the impact of synthetic image captions generated by strong VLMs, as OpenAI did when training DALL-E 3, versus raw web alt-text during multi-modal pre-training?

Answer

Raw alt-text is a noisy channel: it averages roughly 10 words, frequently describes the page rather than the pixels, and is polluted by filenames, SEO keywords, stock-photo boilerplate, and product codes. Running a strong captioner over the corpus replaces that with a dense, grounded description of roughly 50 words that actually mentions attributes, counts, spatial relations, and background objects, which sharply improves text-to-image alignment and prompt following in generative models and improves retrieval in contrastive models. The cost is that the captioner can only describe what it can see and what it already knows, so recaptioning silently deletes the named entities, brands, landmarks, and long-tail vocabulary that only alt-text carries, and it stamps every sample with a single writing style, which collapses caption diversity and imports the captioner’s own hallucinations as ground truth. The practical result reported across DALL-E 3, DataComp, and Recap-DataComp-1B is that neither source wins outright: the tuned quantity is the mixing ratio, with generative text-to-image training favouring almost pure synthetic data (DALL-E 3 used a 95% blend) while contrastive CLIP-style training usually peaks at a genuine mix and degrades toward 100% synthetic.

(1) Alt-Text Is Noisy But Unbiased: it is written by humans for arbitrary purposes, so it is wrong or irrelevant often, yet its errors are not correlated with any single model’s blind spots.
(2) Synthetic Captions Raise Density: a 50-word grounded description supplies far more supervised text tokens per image than a 10-word alt string, which is what drives the gain in compositional and attribute-level alignment.
(3) Recaptioning Deletes World Knowledge: a captioner that cannot name a specific landmark, celebrity, or product writes “a tall building” and the entity vanishes from the training signal permanently.
(4) Style Collapse And Inherited Hallucination: every caption inherits one syntax template and one error distribution, so the student model learns the captioner’s biases as if they were facts.
(5) The Mixing Ratio Is The Real Knob: sample the synthetic caption with probability $p$ and the alt-text otherwise; $p$ near 1 suits text-to-image generation, intermediate $p$ suits contrastive pre-training.
(6) Context Length And One-Time Compute: CLIP’s text encoder truncates at 77 tokens, so dense captions are partially discarded, and recaptioning a billion images is a fixed preprocessing bill of order $10^4$ GPU-hours.

Pipeline diagram: a crawled web page supplies both an alt attribute and image pixels; the alt attribute becomes a short noisy raw caption while the pixels pass through a VLM captioner that emits a dense fifty-word synthetic caption, and both streams feed a mixing sampler that selects the synthetic caption with probability p before contrastive or text-to-image pre-training

Figure 1: The two text streams come from different places. Alt-text is a property of the page, synthetic captions are a property of the pixels plus the captioner’s knowledge, and the only place the two are reconciled is the mixing sampler that draws each training pair’s caption with probability $p$ .

The mechanism behind the improvement is easy to state: the contrastive or captioning objective is unchanged, only the text marginal moves. A short alt string gives the model very few positive constraints, so many wrong images remain compatible with it, whereas a dense caption pins down attributes, counts, and relations and therefore produces a much sharper positive. That is exactly why generative text-to-image models benefit most: their failure mode is ignoring adjectives, counts, and spatial prepositions in a user prompt, and dense captions are the only supervision that ever mentions those. It is also why the gains shrink for discriminative zero-shot classification at scale: with a 1B-scale pool, alt-text’s lexical diversity and entity coverage begin to matter more than its per-sample precision, and studies on DataComp report that generated captions dominate at small and medium pool sizes while a raw-plus-synthetic mixture wins at the large scale.

Two panels: the left panel plots downstream metric against the share of synthetic captions p, with a retrieval and alignment curve rising monotonically toward p equal to one and a zero-shot classification curve peaking near p equal to zero point five five and falling afterwards; the right panel overlays two caption length histograms, raw alt-text concentrated near ten tokens and synthetic captions centred near fifty-eight tokens, with a dashed vertical line at the seventy-seven token CLIP context limit

Figure 2: Two views of the same trade-off. Panel (a) shows why one blend cannot serve both objectives: alignment keeps improving with $p$ while zero-shot classification turns over once entity-bearing alt-text is crowded out. Panel (b) shows the second-order problem: dense captions push the length distribution against the 77-token text-encoder limit, so part of the extra supervision is truncated before it is ever used.

Mathematical Formulation:
$c_i \sim q_{\phi}(c \mid v_i)$
$P(t_i = c_i) = p, \quad P(t_i = a_i) = 1 - p$
$s_{ij} = f(v_i)^{\top} g(t_j) / \tau$
$\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$

Where:

$c_i$ is the synthetic caption sampled from the captioner $q_{\phi}$ conditioned on image $v_i$ , and $a_i$ is the raw alt-text scraped alongside that image.
$t_i$ is the caption actually used for example $i$ , and $p \in [0,1]$ is the mixing ratio, the single hyper-parameter that decides how much of the corpus the captioner rewrites.
$f$ and $g$ are the image and text towers producing normalized embeddings, $\tau$ is the learned temperature, and $B$ is the batch size supplying the in-batch negatives.
Nothing in $\mathcal{L}$ changes when you recaption; the entire effect flows through the conditional distribution of $t_i$ given $v_i$ , which becomes lower-noise but also lower-entropy and model-biased.

One-Time Recaptioning Cost For 1B Images:
$T = 10^{9} / 20 = 5 \times 10^{7}$
$5 \times 10^{7} / 3600 \approx 1.4 \times 10^{4}$

At a sustained 20 images per second per GPU for a 7B-class captioner emitting about 50 tokens, one billion images take $5 \times 10^{7}$ seconds of single-GPU time, roughly 14,000 GPU-hours, or about half a day on a 1,000-GPU cluster. That is a real but one-time preprocessing cost, amortized over every subsequent training run on the corpus, which is why recaptioning is usually cheaper than the ablation sweeps it replaces. The recurring costs are subtler: longer captions mean more text-encoder tokens per step, and a frozen captioner freezes a snapshot of one model’s competence into the dataset.

Property	Raw web alt-text	VLM synthetic caption	Mixture at ratio p
Typical length	About 10 words, often a fragment	About 50 words of dense description	Bimodal, which also teaches the model short prompts
Image grounding	Frequently describes the page, not the pixels	Grounded by construction, with residual hallucination	Grounded on the synthetic draw, noisy on the raw draw
World knowledge	Carries brands, landmarks, people, rare nouns	Entities collapse to generic categories	Entity coverage preserved by the raw fraction
Diversity	High lexical and syntactic variety	Single style template, reduced noun variety	Diversity recovered without giving up density
Best fit	Very large pools where scale beats precision	Text-to-image generation and prompt following	Contrastive pre-training and general-purpose encoders
Marginal cost	Free, already in the crawl	Order 10,000 GPU-hours per billion images	Same captioning bill, plus storage for two text fields

August 18, 2026

DL0143 VLM Pretraining Data Curation

Walk through the data curation pipeline for pre-training large VLMs, including web image-text filtering, synthetic re-captioning (e.g., LLaVA-1.5/1.6), and visual instruction tuning.

Answer

Pre-training a large VLM is mostly a data engineering problem, and the pipeline produces three qualitatively different corpora: a heavily filtered web corpus for breadth, a synthetically re-captioned corpus for description quality, and a small hand-assembled instruction corpus for behaviour. Stage one takes a raw crawl on the order of 10B alt-text pairs, removes NSFW, PII and duplicate URLs, drops images below roughly 200 px and captions outside a 5 to 64 token window, keeps about the top 30% by CLIP image-text cosine, and intersects that with a cluster-based balancing filter, which is how DataComp’s 12.8B CommonPool collapses to the ~1.4B pairs of DataComp-1B. Stage two attacks the fact that surviving alt-text is still short, keyword-shaped and frequently describes the page rather than the pixels: a captioner VLM, itself trained on a small set of 100K high-quality dense captions in the ShareGPT4V style, rewrites each image into a 50 to 100 word caption, and the load-bearing detail is that the best recipes mix synthetic and original captions instead of replacing one with the other, because pure synthetic text launders away proper nouns and world knowledge. Stage three is tiny by comparison, LLaVA-1.5’s 665K instruction mixture growing to roughly 760K in LLaVA-1.6 with DocVQA, ChartQA and AI2D added, and it buys instruction following, short grounded answers, OCR and chart reading rather than new visual knowledge. The three stages differ by four orders of magnitude in scale and by roughly the same factor in cost per example, which is why the filtering stage is optimised for throughput and the instruction stage for mixture ratios.

(1) Cascade Order Is Cost Order: run cheap deterministic filters (decode check, resolution, aspect ratio, caption length, exact and near-duplicate hashing, NSFW and PII removal) before any model forward pass, since a CLIP score on 12.8B pairs is the single most expensive step in the pipeline.
(2) CLIP Score Is A Precision Knob: the cosine gate raises image-text agreement but systematically deletes long compositional captions, rare entities and text-heavy images, so a threshold tuned for zero-shot classification quietly damages OCR and document tasks.
(3) Cluster Balancing Beats Raw Score: DataComp’s winning filtering-track recipe intersects the CLIP-score gate with an image-embedding cluster filter, keeping pairs whose visual cluster resembles curated concept distributions, which fixes the head-heavy topical skew of the crawl.
(4) Re-Captioning Changes The Supervision, Not The Images: the same 1.4B images are re-labelled by a captioner VLM, so the corpus gains dense spatial, attribute and relational description at roughly the cost of one VLM forward pass per image and zero new crawling.
(5) Mix, Do Not Replace: a mixing probability of about $\alpha \approx 0.8$ synthetic to 0.2 original, or an LLM fusion of both strings as in CapsFusion, retains the named entities and factual hooks that only alt-text carries.
(6) Instruction Data Is Ratio-Sensitive, Not Scale-Sensitive: at the 665K scale the composition (VQA, OCR, region grounding, text-only chat) matters far more than the count, and dropping the text-only share collapses multi-turn conversational quality while adding no visual skill.
(7) Decontamination Is Mandatory: near-duplicate removal against the images and questions of VQAv2, TextVQA, MMMU and friends must run at every stage, because a captioner trained on benchmark-adjacent data will otherwise leak answers into the pre-training corpus.

Three-band pipeline diagram: band A shows a raw Common Crawl pool of 12.8B image-text pairs passing through safety and PII removal, basic resolution and caption-length filters, a CLIP cosine gate keeping the top 30 percent, and cluster balancing plus decontamination to yield 1.4B pairs; band B shows a captioner VLM trained on 100K high-quality captions producing dense synthetic captions, an LLM fusing alt-text with the synthetic caption, and a mixed corpus at alpha near 0.8; band C shows projector alignment on 558K pairs, caption plus interleaved pre-training, visual instruction tuning on 665K to 760K examples, and evaluation on VQAv2, TextVQA, DocVQA and MMMU

Figure 1: The pipeline as three chained corpora rather than one dataset. Band A is a cheap-to-expensive filter cascade that discards about 89% of the crawl, band B re-labels the survivors with a captioner VLM and fuses the result with the original alt-text, and band C spends a four-orders-of-magnitude smaller budget on alignment then instruction tuning.

The filtering stage is best understood as trading recall for precision under a fixed compute budget. DataComp’s central result is that the winning entry is not the largest pool but the most aggressively filtered one: at a fixed number of training samples seen, a 1.4B subset beats the 12.8B pool it came from, because gradient steps spent on mismatched pairs are worse than wasted. The failure mode of that logic is that the CLIP scorer used to filter was itself trained on similarly filtered data, so its notion of “matching” is circular and biased against exactly the long, unusual, or text-dense captions that document and chart understanding require. Production pipelines therefore keep separate sub-pools with different thresholds, plus an explicitly retained OCR-heavy shard, rather than applying one global $\tau$ to everything.

Mathematical Formulation:
$s(I,T) = \cos(f_I, f_T)$
$D_1 = \{(I,T) \in D_0 : s(I,T) \geq \tau\}$
$D_2 = D_1 \cap C_{\mathrm{clust}}$
$T' \sim p_{\phi}(T \mid I)$
$P(\tilde{T} = T') = \alpha$
$\mathcal{L} = -\sum_{t \in A} \log p_{\theta}(y_t \mid I, x, y_{1:t-1})$

Where:

$s(I,T)$ is the CLIP cosine between the image embedding $f_I$ and caption embedding $f_T$ , and $\tau$ is the gate, historically about $0.28$ for a ViT-B/32 scorer or a percentile such as the top 30%.
$D_0$ is the raw pool (12.8B pairs in CommonPool), $D_1$ the CLIP-gated set, and $D_2$ the final corpus after intersecting with the cluster filter $C_{\mathrm{clust}}$ and benchmark decontamination, giving roughly $1.4 \times 10^{9}$ pairs.
$p_{\phi}$ is the captioner VLM and $T'$ the dense synthetic caption it samples for image $I$ ; $\phi$ is trained on a small human or GPT-4V-labelled seed set, typically around 100K captions.
$\alpha \in [0,1]$ is the mixing probability of using the synthetic caption instead of the original alt-text $T$ for a given training sample; $\alpha = 1$ is pure synthetic and $\alpha = 0$ is the raw web baseline.
$\mathcal{L}$ is the instruction-tuning objective over the answer token set $A$ only, with the image $I$ and instruction $x$ as context and loss masked on the prompt, which is what prevents the model from learning to hallucinate its own questions.

The instruction stage also decides the inference bill, because resolution enters through the token count rather than the parameter count. LLaVA-1.5 uses a CLIP ViT-L/14 at 336 px, so each image becomes $(336/14)^2 = 576$ visual tokens, while LLaVA-1.6’s AnyRes scheme tiles a high-resolution image into four crops plus a global thumbnail, giving $5 \times 576 = 2880$ tokens. That five-fold increase is what unlocks DocVQA and ChartQA, and it also means the instruction mixture must contain enough high-resolution document data to justify the tokens, otherwise the model pays the cost without learning to use the detail.

Line chart with synthetic caption mixing ratio alpha on the x axis from 0 to 1 and relative benchmark score on the y axis: a retrieval and captioning curve rises monotonically from 100 to about 118, an entity and world-knowledge curve rises to a peak near alpha 0.65 then falls back to about 100 at alpha 1, and their average peaks near alpha 0.8, marked by a vertical dashed line labelled common operating point

Figure 2: Why re-captioning is a mixture and not a replacement. Synthetic captions monotonically improve retrieval and description because they are fluent and pixel-grounded, but entity and knowledge accuracy peaks well before $\alpha = 1$ and then decays, since a captioner cannot invent the proper nouns, brands, and dates that only human alt-text supplied. The reported operating point in VeCLIP, CapsFusion and Recap-DataComp ablations lands near $\alpha \approx 0.8$ .

Property	Filtered web pairs	Synthetic re-captions	Visual instruction data
Typical scale	1B to 5B pairs after filtering 10B+ raw	100K seed captions, then 1M to 1.3B generated	665K in LLaVA-1.5, about 760K in LLaVA-1.6
Source	Common Crawl alt-text, HTML attributes	A captioner VLM run over already-filtered images	Academic VQA/OCR/grounding sets plus LLM-written dialogue
Cost per example	Crawl plus one CLIP forward pass	One VLM generation of 50 to 100 tokens	Human annotation or strong-model distillation, orders of magnitude higher
What it teaches	Concept coverage, entities, long-tail visual vocabulary	Dense attributes, spatial relations, fluent grounded description	Answer format, instruction following, refusal and multi-turn behaviour
Dominant failure mode	Mismatched or page-level captions, topical head skew	Hallucinated details, lost proper nouns, uniform caption style	Wrong mixture ratio, benchmark overfitting, short-answer bias
Where it enters training	Projector alignment and large-scale pre-training	Mixed into the same pre-training stream at ratio $\alpha$	Final supervised stage, full LLM and projector unfrozen

August 18, 2026

DL0142 OCR-Free Document VLM

How do OCR-free Document VLMs process complex multi-column PDFs, tables, and infographics compared to multi-stage OCR pipeline setups?

Answer

A multi-stage pipeline turns a page into a text document before any reasoning happens: rasterize → detect text regions → recognize each crop → classify layout blocks → sort them into reading order → recover table cell structure → serialize to Markdown or HTML → feed a text LLM. An OCR-free Document VLM deletes that entire chain and treats the page as an image: a dynamic-resolution ViT encoder cuts the raster into 14×14 patches, a 2×2 pixel-shuffle merge collapses them into one visual token per 28×28 pixel block, and a decoder-only LLM attends over those tokens to emit the answer, the Markdown, or the HTML table directly. The consequence is that self-attention itself becomes the layout model: column boundaries, cell alignment, chart axes, and legend-to-series association are learned from pixels rather than reconstructed by six independently trained components whose errors multiply. What you gain is robustness on infographics and rotated or borderless tables, where reported scores such as Qwen2.5-VL-72B’s roughly 96 ANLS on DocVQA and roughly 87 on InfographicVQA are far out of reach for a serialized-text pipeline. What you lose is character-level coordinates, per-token confidences, and cheap per-page cost, because a single A4 page at 150 DPI already costs about 2,835 visual tokens and the prefill over them is quadratic.

(1) Pixels In, Structure Out: the model never sees a text layer, so scanned pages, screenshots, and born-digital PDFs with broken embedded fonts all take the identical path.
(2) Dynamic Resolution Tokenization: instead of squashing every page to 224×224, the encoder keeps native aspect ratio and resolution, so an 8 pt footnote survives as its own tokens rather than being blurred away.
(3) Attention Replaces The Reading-Order Module: a three-column paper needs no LayoutReader-style sorter, because the decoder learns column continuation the way a language model learns syntax.
(4) Tables As Generated Markup: structure recognition becomes ordinary autoregressive decoding of HTML or Markdown, which handles borderless and spanning cells but has no per-cell confidence.
(5) No Error Compounding, No Coordinates: the pipeline’s five or six stages multiply their error rates, while the VLM has one loss and one failure surface but cannot tell you where on the page an answer came from.
(6) Token Budget Is The Real Constraint: visual tokens grow with the square of DPI, so resolution, tiling, and page count trade directly against context and $O(N^2 d)$ prefill.

Two horizontal lanes compared on the same rasterized page: the upper lane shows a six-stage OCR pipeline running text detection, crop recognition, layout analysis with reading order, table structure recognition, and Markdown serialization into a text LLM, annotated with compounding per-stage error; the lower lane shows an OCR-free VLM with a dynamic-resolution ViT patch encoder, a 2x2 pixel-shuffle merge producing 2,835 visual tokens, and a decoder-only LLM emitting the answer or an HTML table

Figure 1: The same pixels, two failure surfaces. The pipeline produces an intermediate text document with coordinates that any downstream model can consume and any auditor can overlay, at the cost of five models whose accuracies multiply. The VLM is one differentiable stack with one loss, and its output carries no character boxes at all unless the model was explicitly trained to emit them.

The three hard document classes fail differently. On multi-column PDFs, a pipeline’s mistake is almost never recognition, it is serialization: a two-column paper with a full-width figure caption in the middle gets flattened into interleaved half-sentences, and the LLM downstream has no way to recover the intended order because the evidence, the geometry, was discarded. On tables, borderless layouts and spanning header cells break rule-based and detection-based structure recognition, whereas a VLM trained on HTML targets can emit rowspan and colspan because it saw the whole grid at once. On infographics and charts, OCR returns a bag of strings with no relations, so “which bar is tallest” or “what does the dashed series do after 2021” is unanswerable from the transcript; this is precisely where the ChartQA and InfographicVQA gaps are widest. The pipeline still wins wherever the requirement is verbatim fidelity plus provenance, since a hallucinated digit inside a generated table cell is indistinguishable from a correct one, while a low-confidence OCR crop announces itself.

Mathematical Formulation:
$N_{tok} = \lceil H/p \rceil \times \lceil W/p \rceil$
$N_{tok} = 63 \times 45 = 2835$
$C_{prefill} = O(N_{tok}^2 d)$
$A_{pipe} = \prod_{k=1}^{K} a_k$
$A_{pipe} = 0.95^5 \approx 0.77$

Where:

$N_{tok}$ is the number of visual tokens the encoder emits for one page, which is what actually enters the LLM context.
$H$ and $W$ are the rasterized page height and width in pixels ( $1754 \times 1240$ for A4 at 150 DPI), and $p = 28$ is the effective patch stride after a 2×2 merge of 14×14 patches.
$C_{prefill}$ is the attention cost before the first output token, quadratic in $N_{tok}$ and linear in model width $d$ ; the KV cache grows linearly, so a 20-page document is a memory problem as well as a compute one.
$a_k$ is the per-page success rate of pipeline stage $k$ and $K$ the number of stages, with $k \in \{1,\ldots,K\}$ running detection, recognition, layout, reading order, and table structure.
$A_{pipe}$ is the end-to-end page accuracy: five stages that each succeed 95% of the time leave only about 77% of pages fully clean, and this multiplicative compounding is the structural argument for a single-stage model.

Log-scale chart of tokens per A4 page versus rasterization DPI from 72 to 420: a curve for visual tokens after 2x2 pixel-shuffle merge rising quadratically from about 640 at 72 DPI to about 21500 at 420 DPI, a four-times-higher curve for raw 14x14 patches without merging, and a flat line at about 800 tokens for serialized OCR text, with a horizontal marker at the 16384-token per-image cap and a vertical dashed line at 150 DPI

Figure 2: Resolution is the cost knob. The same A4 page costs 2,835 visual tokens at 150 DPI and 11,214 at 300 DPI, roughly 4x the tokens and 16x the prefill FLOPs, while a serialized OCR transcript of that page stays near 800 tokens whatever the DPI. The 2×2 pixel-shuffle merge is what keeps a full page under the common 16,384-token per-image cap at all.

Property	Multi-stage OCR pipeline	OCR-free Document VLM
Multi-column reading order	Explicit sorter over layout blocks; interleaves columns when a full-width element splits the page	Learned implicitly by attention over the whole page at once
Tables	Dedicated structure model emitting cell boxes; weak on borderless and spanning cells	Generates HTML with rowspan and colspan; can silently drop or invent rows in long tables
Charts and infographics	Returns unrelated strings; visual relations such as legend-to-series are lost	Reads axes, bar heights, and legends jointly, which is where the accuracy gap is largest
Provenance	Character and word boxes plus per-crop confidence, usable for redaction and highlighting	None by default; needs grounding training to emit absolute coordinates
Dominant failure mode	Compounding stage errors and serialization scrambling; degrades visibly	Fluent hallucination and repetition loops; degrades invisibly
Cost per page	Small CNN and CTC models, CPU-viable, millions of pages per day cheaply	Thousands of visual tokens through a multi-billion-parameter decoder with quadratic prefill
Adapting to a new form type	Retrain or rewrite whichever stage broke, with per-stage labels	Fine-tune once on image and target-string pairs, no intermediate annotation

August 18, 2026

DL0141 Visual Token Compression for OCR

How do visual token compression techniques reduce the sequence length of visual inputs without degrading fine-grained OCR performance?

Answer

The techniques that survive contact with documents all compress along the channel axis rather than the token axis: a pixel unshuffle (InternVL) or a strided convolutional reducer (DocOwl 1.5) folds each $s \times s$ block of patch embeddings into one vector of width $s^2 d$ and projects it back to $d$ , so four patches become one token while the patch-to-region mapping stays bijective and no pixel is discarded. The second half of the recipe is that the token budget must keep scaling with input resolution: dynamic tiling and native-resolution patching (Qwen2-VL) give a 1275×1650 scan roughly 2,600 tokens of 28×28 pixels each, whereas a fixed- $K$ resampler hands the same page 64 tokens no matter how many glyphs it contains. What actually kills OCR is rarely the compressor itself but the resize step in front of it: squeezing a page into 336×336 makes an 11 pt glyph thinner than one 14-pixel patch, and no downstream module can recover strokes the encoder never sampled. The useful mental model is glyph density: keep the number of glyphs covered by a single visual token near one, and compression is nearly free; push it toward ten and character-level accuracy falls off a cliff while scene-level captioning barely moves.

(1) Compress Channels, Not Positions: pixel unshuffle and conv reducers move information into the feature dimension, so a $4\times$ length reduction still lets every output token point at a known rectangle of the page.
(2) Keep The Budget Resolution-Dependent: a compression ratio is safe, a compression target is not; dynamic tiling plus native-resolution patching lets a dense page buy more tokens than a photo of a beach.
(3) Respect Glyph Nyquist: the binding constraint is stroke width versus patch size in the resized image, which is why 336-pixel inputs cap document accuracy regardless of the connector.
(4) Fixed-Query Resamplers Lose The Wrong Thing First: $K$ learned queries cross-attend to all patches without a positional index, so reading order and rare characters degrade before object-level semantics do.
(5) Two-Scale Views Are Cheap: a global thumbnail supplies layout while local tiles supply glyphs, which is why AnyRes-style designs use $4 \times 576 + 576 = 2880$ tokens rather than one giant grid.
(6) Prune Late And Query-Aware: dropping half the visual tokens after LLM layer 2 (FastV) saves about 45% of prefill FLOPs on scene VQA but deletes whole text lines when the question has not yet been attended to.

Three-row diagram comparing visual token compression families on the same 8 by 4 patch grid: the top row folds each 2 by 2 block of patches into one token on the channel axis and produces a 4 by 2 output grid with matching tints, the middle row sends all patches through cross-attention into four fixed learned query tokens with no positional index, and the bottom row keeps the original grid but marks half the cells as dropped by an attention score

Figure 1: Same patch grid, three compression axes. Only the top row keeps an exact mapping from output token back to page rectangle, which is what OCR decoding depends on; the middle row replaces that mapping with $K$ content-addressed slots, and the bottom row keeps positions but deletes evidence.

Why the distinction matters becomes obvious once you count information. Natural images are locally redundant, so averaging neighbouring patches costs almost nothing; a page of text is close to the opposite, since each glyph is a high-entropy symbol whose identity cannot be inferred from its neighbours and whose position carries the reading order. A learned resampler is a query-agnostic bottleneck: it must decide what to keep before the question arrives, and a fixed 64-slot budget forces it to summarise, which is exactly the wrong operation for text. Structured merging instead makes a bounded, uniform trade, and empirically the boundary sits near one glyph per token. The design lineage of production VLMs follows that logic directly: Q-Former resamplers → pixel unshuffle with dynamic tiles → native dynamic resolution with a 2×2 patch merger, each step trading a smaller guaranteed budget for a budget that grows with how much text is actually on the page.

Mathematical Formulation:
$N_p = \dfrac{HW}{p^2}$
$N_v = \dfrac{N_p}{s^2} = \dfrac{HW}{p^2 s^2}$
$A = \dfrac{H_0 W_0}{N_v}$
$c = \dfrac{G}{N_v}$
$\mathrm{prefill} = O((N_v + N_t)^2 d)$

Where:

$N_p$ is the patch count the vision encoder produces from a resized input of size $H \times W$ with patch size $p$ , and $N_v$ is the number of tokens actually handed to the language model.
$s$ is the spatial merge factor; pixel unshuffle with $s=2$ concatenates 4 patch embeddings into width $4d$ and projects back to $d$ , so length drops $4\times$ with no averaging.
$H_0 \times W_0$ is the original page resolution and $A$ the original pixels covered by one visual token, which is the honest measure of compression because the resize is itself a compressor.
$G$ is the glyph count on the page and $c$ the glyphs per visual token; $c \approx 1$ is the practical safety line for character-accurate reading.
$N_t$ is the text prompt length and $d$ the model width, so prefill is quadratic in the combined sequence while the KV cache grows linearly.

Worked Example, One Dense A4 Page At 150 DPI:
$N_v = 4 \times 576 + 576 = 2880$
$c = 3200 / 2880 \approx 1.1$
$c = 3200 / 576 \approx 5.6$
$(2880 / 576)^2 = 25$

Assuming about 3,200 glyphs on the page, an AnyRes layout of four 336-pixel tiles plus a thumbnail lands at roughly one glyph per token, while a single 336-pixel view lands at 5.6 and reads only headlines. The last line is the bill: those extra tokens cost 25 times the attention work in prefill, which is precisely why the compressor exists and why the interesting engineering is choosing the smallest $N_v$ that still keeps $c$ near 1 for the document class you serve.

Figure 2: Compression is only meaningful relative to glyph density. Configurations inside the band give each visual token roughly one character and read reliably; a 64-slot resampler asks one token to encode about 50 glyphs. Systems trained specifically for optical text compression can operate right of the band, but at a measured precision cost, and below the band extra tokens buy nothing.

Property	Spatial channel merge	Fixed-query resampler	In-LLM pruning or merging
Mechanism	Pixel unshuffle or strided conv over the patch grid, then a linear projection	Cross-attention from K learned queries into all patch embeddings	Rank tokens by attention received in an early layer, drop or merge the tail
Budget vs resolution	Grows linearly with pixels, fixed ratio of 4x or 16x	Constant at K (typically 32 to 256) whatever the input size	Grows with pixels, then cut by a fixed keep-rate
Spatial index kept	Yes, one token maps to one known rectangle	No, slots are content-addressed and order must be relearned	Yes for survivors, but dropped regions leave holes
Training cost	One small projection, trained with the connector	A full extra transformer stage plus alignment pretraining	Usually training-free, applied at inference
Dominant failure	Long context and quadratic prefill on multi-page inputs	Reading order and rare glyphs collapse on dense pages	Whole text lines vanish when the query is not yet visible to the scorer

August 18, 2026

DL0135 Dynamic High-Resolution ViT

How do Dynamic High-Resolution ViT slicing architectures handle non-standard aspect ratios and high-resolution images without spatial distortion?

Answer

A pretrained vision tower expects one fixed square input, usually $336 \times 336$ or $448 \times 448$ , with exactly one learned position embedding per patch. Squashing a $1500 \times 500$ receipt or slide into that square applies anisotropic scaling (0.30x horizontally against 0.90x vertically), which shears glyphs and bar charts before the encoder sees a single pixel, while interpolating the position-embedding grid up to native resolution is both off-distribution and quadratically expensive in attention. Dynamic high-resolution slicing, the AnyRes family used by LLaVA-NeXT, UReader, Monkey, MiniCPM-V and InternVL 1.5, avoids both by never changing the encoder’s input size: it picks a tile grid whose aspect ratio best matches the image, resizes the image to exactly $n_w S \times n_h S$ , cuts it into $n_h n_w$ native-resolution tiles that are encoded independently, and appends a downsampled full-image thumbnail to restore the global layout that slicing destroys. Because the target width and height are both integer multiples of the same tile size chosen to match the input ratio, the horizontal and vertical scale factors are (nearly) equal, so the resize is isotropic and no geometry is distorted. What remains is a budgeting problem: the token count grows linearly in tiles, so token compression and a cap $N_{\max}$ on the tile product decide how much resolution you can actually afford.

(1) Aspect-Ratio Matched Grid: enumerate all grids with $n_h n_w \leq N_{\max}$ and pick the one minimizing the log-ratio mismatch against $W/H$ , which is exactly the quantity that becomes anisotropic shear if you get it wrong.
(2) Tiles Stay At Native Resolution: every tile is precisely $S \times S$ , so the frozen CLIP or SigLIP tower runs at its pretraining resolution with no position-embedding interpolation and no distribution shift.
(3) Thumbnail Carries Global Context: tiles are encoded independently, so no attention crosses a seam; a low-resolution whole-image view is concatenated to supply page layout and object-scale cues.
(4) Linear Instead Of Quadratic Attention: encoding $K$ tiles of $M$ patches costs $K \cdot O(M^2)$ rather than $O(K^2M^2)$ for one giant grid, a $K$ -fold saving that is why slicing scales to 4K pages.
(5) Token Compression Is Mandatory: a pixel-unshuffle of stride 2 folds each $2 \times 2$ patch neighborhood into one embedding, cutting 1024 patches per 448 tile to 256 tokens; without it a 12-tile page costs over 13k tokens.
(6) The LLM Must Be Told The Layout: the flattened tile sequence is ambiguous, so implementations insert row separator tokens or 2D tile indices, otherwise a $3 \times 1$ and a $1 \times 3$ arrangement look identical downstream.

Two-panel diagram of a 1500 by 500 panoramic input containing a reference circle. The left panel resizes it into a single 448 by 448 square, where the circle becomes a narrow vertical ellipse because horizontal scale is 0.30x and vertical scale is 0.90x. The right panel matches a 3 by 1 tile grid, resizes to 1344 by 448 with both axes at 0.90x so the circle stays circular, cuts it into three native-resolution 448 tiles, and adds a low-resolution global thumbnail.

Figure 1: The reference circle is the whole story. A single square resize applies different scale factors per axis, so the circle becomes an ellipse and every glyph shears; the aspect-ratio-matched grid scales both axes by 0.90x, keeps each tile at the encoder’s native $448 \times 448$ , and pays for the lost cross-tile view with one global thumbnail.

Two details separate a correct implementation from a leaky one. First, the resize to $n_w^\ast S \times n_h^\ast S$ is only exactly isotropic when the chosen grid ratio equals the image ratio; otherwise a residual anisotropy survives, equal to the log mismatch, and it is bounded by how fine your candidate set is. With $N_{\max} = 6$ a 4:3 photo still carries 0.29 of residual log-anisotropy because the exact $3 \times 4$ grid needs 12 tiles, which is why InternVL 1.5 raises $N_{\max}$ to 12 and InternLM-XComposer2-4KHD pushes to 55 tiles, and why some systems pad to the grid instead of stretching. Second, slicing changes what the encoder can see: a table row spanning a vertical seam is split across two independently encoded tiles, so the fusion must happen inside the language model rather than in the vision tower, and this is the main reason answers to fine-grained OCR questions degrade near tile boundaries. Token cost is the binding constraint in production: LLaVA-NeXT at $S=336$ with at most 4 tiles plus a base view spends $5 \times 576 = 2880$ tokens per image, and InternVL 1.5 with 12 tiles plus a thumbnail spends $13 \times 256 = 3328$ only because pixel shuffle compresses each tile fourfold.

Mathematical Formulation:
$\mathcal{G} = \{(n_h, n_w) : n_h n_w \leq N_{\max}\}$
$(n_h^\ast, n_w^\ast) = \arg\min_{\mathcal{G}} \left| \log \frac{n_w}{n_h} - \log \frac{W}{H} \right|$
$s_x = \frac{n_w^\ast S}{W}$
$s_y = \frac{n_h^\ast S}{H}$
$D = \left| \log \frac{s_x}{s_y} \right|$
$T = (n_h^\ast n_w^\ast + 1) \frac{S^2}{p^2 r^2}$

Where:

$(n_h^\ast, n_w^\ast)$ is the selected tile grid, and $\mathcal{G}$ the candidate set of grids whose tile product stays within the budget $N_{\max}$ .
$W$ and $H$ are the original image width and height, and $S$ is the encoder’s native side length (336 for LLaVA-NeXT, 448 for InternVL 1.5).
$s_x$ and $s_y$ are the per-axis scale factors of the resize, and $D$ is the residual anisotropy: $D = 0$ means the resize is isotropic and geometry is preserved exactly.
$T$ is the vision-token count fed to the LLM, with the $+1$ accounting for the global thumbnail.
$p$ is the patch size (typically 14) and $r$ the pixel-shuffle stride, so $S=448, p=14, r=2$ gives $1024/4 = 256$ tokens per tile.
The objective is stated in log space because minimizing $|\log(n_w/n_h) - \log(W/H)|$ is identical to minimizing $D$ , which makes aspect matching and distortion control the same problem.

Two charts. Left: residual log-anisotropy versus the maximum tile product for images of aspect ratio 4:3, 3:1, 5:1 and 8:1, showing step decreases to zero when an exactly matching grid enters the candidate set, with the 4:3 curve stuck at 0.29 until twelve tiles are allowed. Right: vision-token count versus tile count for pixel-shuffle stride 1 and stride 2, with a horizontal 4096-token image budget line that stride 1 crosses at three tiles while stride 2 stays under it through twelve tiles.

Figure 2: The two knobs pull against each other. Raising $N_{\max}$ is what drives residual anisotropy to zero for awkward ratios (left), but each extra tile buys a fixed block of tokens, so only pixel-shuffle compression keeps a 12-tile page inside a realistic per-image token budget (right).

Property	Square resize + PE interpolation	Dynamic tiling (AnyRes)	Native dynamic resolution
Aspect handling	Anisotropic stretch, or padding that wastes most of the input area	Isotropic up to grid quantization; residual anisotropy shrinks as N_max grows	Exact: the patch grid itself takes the image’s shape, no quantization
Encoder reuse	Frozen tower reused, but interpolated position embeddings go off-distribution	Frozen tower reused unchanged at its exact pretraining resolution	Requires training or adapting the tower with 2D RoPE and variable-length packing
Attention cost	Quadratic in total patches once resolution is raised	Linear in tiles: K independent O(M squared) passes, easy to batch	Quadratic over the full image unless windowed or block-diagonal attention is used
Global context	Complete but low detail; small text is unrecoverable	Broken at seams; restored approximately by the thumbnail and separator tokens	Intact, every patch can attend to every other patch in the image
Representative systems	Original LLaVA, BLIP-2, early CLIP-based VLMs	UReader, Monkey, LLaVA-NeXT, MiniCPM-V, InternVL 1.5 and 2.5	NaViT, Idefics2, Qwen2-VL and Qwen2.5-VL, Pixtral

August 17, 2026