Author: admin

DL0120 VLA Video Pretraining

How do you leverage internet-scale human video datasets (e.g., Ego4D, Something-Something) for pre-training robotic VLA policies? How do you overcome the embodiment/kinematic gap?

Answer

Human video is enormous but action-free: Ego4D contributes roughly 3,670 hours of egocentric recording across 74 locations and Something-Something V2 adds about 220,000 clips over 174 manipulation-style classes, yet none of it carries joint commands, gripper states, or proprioception. The practical recipe is therefore to never try to learn a robot action space directly from video, but to choose an intermediate interface that video can supervise and that a small teleoperated dataset can later decode into commands. Three interfaces dominate: visual representation pretraining (R3M, VIP, VC-1 style encoders trained on Ego4D with time-contrastive or masked objectives), latent action pretraining (a VQ inverse-dynamics model turns each frame pair into a discrete code, the VLA is pretrained to predict those codes from image plus instruction, and only the final decoder head sees real actions), and embodiment-free intermediate predictions such as future 2D/3D point tracks, subgoal images, or hand-contact affordance maps. The embodiment gap is then attacked on four separate axes rather than as one problem: appearance (human arms and hands in frame), kinematics (a roughly 27-DoF hand versus a 1-DoF parallel gripper), viewpoint and calibration (head-mounted camera with unknown scale versus a fixed wrist or shoulder camera), and dynamics (contact forces and speeds a rigid gripper cannot reproduce). In every working system the last mile is still robot data, typically 50 to 200 hours of teleoperation, mixed into training with a weight that keeps human video as a prior rather than a target.

(1) Pick The Interface, Not The Dataset: transferability and actionability trade off directly, so the design decision is which quantity human video supervises, not whether to use it.
(2) Latent Actions As A Shared Vocabulary: a quantized inverse-dynamics model maps $(o_t, o_{t+k})$ to a discrete code that describes observed scene change, which both a hand and a gripper can produce.
(3) Inverse Dynamics Pseudo-Labeling: the VPT approach trains an IDM on a small labeled set and pseudo-labels the rest, which works when the action space is shared and fails badly across embodiments.
(4) Retargeting Where Geometry Allows: a MANO hand fit gives wrist pose plus a scalar aperture, enough to drive a parallel gripper but not a multi-finger in-hand manipulation.
(5) Visual Alignment Beats Visual Luck: masking or inpainting human hands, matching camera placement with head-mounted rigs, and cropping to the same field of view remove most of the appearance gap cheaply.
(6) Co-Training Ratio Is A Hyperparameter: human video enters with weight $\lambda$ alongside robot batches; too high and the policy predicts plausible video instead of executable actions.

Pipeline diagram: human video from Ego4D and Something-Something enters a VQ inverse-dynamics latent action model, the resulting discrete codes pretrain a VLA backbone, and a small robot demonstration set finetunes an action head that outputs joint-space commands

Figure 1: The latent-action route: human video → discrete latent codes → VLA pretraining, with the embodiment gap crossed only in the final stage, where a few tens of hours of teleoperated demonstrations train the head that emits real commands.

Treating the gap as four separate problems makes each one tractable. For appearance, hand segmentation plus inpainting, or observation-space alignment using head-mounted capture on the human side and a matching camera on the robot, is usually enough that the frozen backbone stops keying on skin texture. For kinematics, the honest move is to restrict what human video is asked to teach: it supplies where to go, what to touch, and in what order, while grasp synthesis and force control come from robot data or from a classical controller. For viewpoint and scale, monocular depth or 3D point tracking lifts predictions into a metric frame, and predicting relative motion rather than absolute pose removes the calibration dependency. For dynamics, the temporal stride $k$ matters more than people expect: a stride of a few frames captures human motion an order of magnitude faster than the robot will execute, so codes must be interpreted as subgoals and re-planned, not replayed. Cross-embodiment action tokenization, as used in the Open X-Embodiment and RT-X work over 22 embodiments, is the complementary trick on the robot side: a shared tokenizer plus per-embodiment heads lets one backbone absorb heterogeneous action spaces.

Mathematical Formulation:
$z_t = q\big(E(o_t, o_{t+k})\big)$
$\hat{o}_{t+k} = D(o_t, z_t)$
$\mathcal{L}_{\mathrm{LAM}} = \lVert o_{t+k} - \hat{o}_{t+k} \rVert^2 + \beta \mathcal{L}_{\mathrm{VQ}}$
$\mathcal{L}_{\mathrm{pre}} = -\log \pi_\theta(z_t \mid o_t, \ell)$
$\mathcal{L}_{\mathrm{ft}} = \lVert a_t - g_\phi(h_\theta(o_t, \ell)) \rVert^2$
$\mathcal{L} = \mathcal{L}_{\mathrm{robot}} + \lambda \mathcal{L}_{\mathrm{human}}$

Where:

$o_t$ is the observation (usually a single RGB frame) and $\ell$ the language instruction; $a_t$ is the true robot action, such as an end-effector delta or joint target.
$z_t$ is the discrete latent action for the transition, produced by encoder $E$ and quantizer $q$ against a codebook of size $C$ with $m$ tokens per transition, giving $C^m$ distinct codes.
$D$ is the forward decoder that must reconstruct $o_{t+k}$ from $o_t$ and $z_t$ alone, which is what forces $z_t$ to carry the motion information.
$k > 0$ is the temporal stride in frames and $\beta$ weights the VQ commitment loss; small $k$ captures noise, large $k$ makes the code ambiguous.
$\pi_\theta$ is the VLA backbone during latent pretraining, $h_\theta$ its features after pretraining, and $g_\phi$ the small action head trained only on robot data.
$\lambda$ is the co-training weight on the human-video term; a typical schedule anneals $\lambda$ from near 1 toward 0 so that the deployed policy is dominated by real action supervision.

Qualitative scatter plot of transfer interfaces: joint torques and end-effector poses are directly executable but robot-specific, latent action codes and subgoal images sit in a shaded sweet spot, and frozen visual features are fully embodiment-agnostic but need a full policy on top

Figure 2: The interface ladder. Human video can only supervise quantities that are not tied to a specific body, so the useful designs sit in the middle band, where latent codes, point tracks, and subgoal images are both learnable from video and decodable into commands with modest robot data.

Property	Frozen visual representations	Latent action pretraining	Tracks and subgoal images
What human video supervises	Encoder weights and a temporal value or progress signal	A discrete motion vocabulary shared by hands and grippers	Future 2D or 3D point trajectories, or the next subgoal frame
Robot data still required	A full policy trained on top of the frozen features	A small action head mapping codes to commands, tens of hours	A track-following or goal-reaching low-level policy
Handles hand versus gripper by	Ignoring actions entirely, so the mismatch never appears	Absorbing it in the quantizer: codes describe scene change, not joints	Staying in task space, where points and pixels carry no body
Main failure mode	Task-agnostic features help perception but little on long horizons	Codes drift toward camera motion and background dynamics	Tracker failure under occlusion, and no notion of contact force
Best fit	Small robot datasets and cluttered scenes needing better perception	Generalist VLAs that must follow unseen instructions	Quasi-static rearrangement and tool use with visible motion

August 17, 2026

DL0119 VLA Imitation vs RL

Explain imitation learning vs RL for VLA training. When would you reach for each one on a real robot, for example when post-training a model like Physical Intelligence’s $\pi_0$ ?

Answer

Imitation learning trains a vision-language-action model by maximum likelihood on teleoperated demonstrations: every timestep carries a ground-truth action label, so the objective is the same supervised next-token or flow-matching loss used to pretrain the underlying VLM, and no reward, reset, or simulator is required. Reinforcement learning replaces that per-step label with a scalar return and optimizes the policy on its own state distribution, which is the only way to fix the failure mode imitation cannot fix and to exceed the demonstrator’s performance. The asymmetry is practical rather than philosophical: demonstrations are cheap to collect and trivially parallel across operators, while a single real-robot RL run needs a success detector, an automated reset, and $10^4$ to $10^6$ environment steps of physically risky exploration. That is why essentially every published generalist VLA (RT-2, OpenVLA, $\pi_0$ ) is imitation-pretrained on cross-embodiment data such as Open X-Embodiment, and RL appears only as a narrow post-training stage on a handful of tasks where reward is machine-checkable. The interview answer is therefore not “which is better” but “imitation for coverage and language grounding, RL for the last 10 to 20 points of success rate on tasks you can actually score”.

(1) Different Objectives: imitation maximizes $\log \pi_\theta(a \mid o)$ under a fixed data distribution, while RL maximizes expected discounted return under the policy’s own visitation distribution.
(2) Compounding Error: behavior cloning suffers covariate shift, and its regret against the expert grows as $O(\epsilon T^2)$ in the horizon, whereas interactive or on-policy training reduces this to $O(\epsilon T)$ .
(3) Supervision Density: a demonstration gives a full action vector at every control step; a sparse success reward gives one bit per episode, so RL’s credit assignment problem is orders of magnitude harder.
(4) Performance Ceiling: imitation is capped at the demonstrator and inherits their pauses, jerk, and inconsistent strategies; RL can discover non-human solutions but will hack a badly shaped reward.
(5) Multimodality: human demos are multimodal, so a naive MSE head averages incompatible strategies, which is exactly why modern VLAs use discretized action tokens, diffusion, or flow-matching action experts plus action chunking.
(6) Hybrid Is The Default: production recipes pretrain with imitation, then apply human-in-the-loop corrections (HG-DAgger), offline RL with a conservatism penalty, or advantage-weighted fine-tuning with a KL leash to the cloned policy.

Flow diagram: teleoperated demonstrations feed imitation pretraining of a VLA, the resulting policy produces on-robot rollouts, a success detector or human takeover produces a reward signal, and an RL fine-tuning stage with advantage weighting and a KL penalty updates the policy in a loop

Figure 1: The standard VLA recipe is imitation first, RL second: demonstrations supply dense per-step labels with no reward machinery at all, and the RL loop is only viable once the cloned policy already succeeds often enough for a scalar success signal to be informative.

The sample-cost gap dominates every design decision. Collecting 100 demonstrations of a new task takes a single operator under an hour and immediately yields a policy that succeeds sometimes; reaching the same point with online RL from a random initialization means exploring a 7-DoF continuous action space with sparse reward, which is hopeless on hardware and merely expensive in simulation. Once an imitation-pretrained policy exists, RL becomes tractable because exploration starts near the solution manifold: human-in-the-loop RL methods that keep demonstrations in the replay buffer and let an operator intervene on impending failures report near-perfect success on precise, contact-rich assembly tasks after roughly one to three hours of real-world training. The corresponding risk is catastrophic forgetting: unconstrained RL on one task will happily destroy the language grounding and cross-task generalization that the imitation stage paid for, so practitioners freeze most of the VLM backbone, fine-tune the action expert, and add an explicit KL term back to the behavior-cloned reference policy.

Mathematical Formulation:
$\theta_{BC} = \arg\max_{\theta} \mathbb{E}_{(o,a) \sim \mathcal{D}} \left[\log \pi_{\theta}(a \mid o)\right]$
$J(\pi) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{T} \gamma^{t} r(s_t, a_t)\right]$
$J(\pi^{*}) - J(\pi_{BC}) = O(\epsilon T^{2})$
$J(\pi^{*}) - J(\pi_{DAgger}) = O(\epsilon T)$
$\max_{\theta} \mathbb{E}_{\mathcal{D}}\left[\log \pi_{\theta}(a \mid o) \exp\!\left(A(o,a)/\beta\right)\right]$

Where:

$\pi_{\theta}(a \mid o)$ is the VLA policy mapping an observation $o$ (images, proprioception, language instruction) to an action or action chunk $a$ .
$\mathcal{D}$ is the fixed demonstration dataset, and $\pi^{*}$ is the human expert whose behavior it samples.
$s_t$ and $t \in \{0, \ldots, T\}$ are the environment state and control step, $r$ the reward, and $\gamma \in (0,1]$ the discount.
$\epsilon$ is the per-step supervised action error under the expert’s distribution; the quadratic term comes from errors pushing the policy into states absent from $\mathcal{D}$ .
$A(o,a)$ is the estimated advantage and $\beta > 0$ the temperature; as $\beta \to \infty$ the advantage-weighted objective degenerates to plain behavior cloning, which is why the hybrid is a continuum rather than a binary choice.
Required initial condition for the RL stage: $\pi_{\theta}$ is initialized at $\theta_{BC}$ and the task admits an automated reset plus a reward oracle, otherwise $J$ is not estimable on hardware.

Line chart of regret against the expert versus episode horizon for a per-step error of 0.02, comparing a quadratic behavior-cloning bound that reaches 200 at horizon 100 with a linear interactive-training bound that reaches only 2

Figure 2: With a fixed per-step error of $\epsilon = 0.02$ , the quadratic behavior-cloning bound separates from the linear on-policy bound by a factor of $T$ , which is why long-horizon manipulation degrades far faster than single-step prediction accuracy suggests.

Property	Imitation learning (BC)	Offline RL	Online RL
Supervision per episode	One action label at every control step	Logged actions plus reward or success labels	Scalar reward, often a single success bit
What it needs to run	Teleoperation rig only; no reset, reward, or simulator	A labeled buffer of mixed-quality data	Automated reset, success detector, safety envelope
Typical budget for one task	50 to 2,000 demonstrations, hours of human time	Reuses existing logs; compute-bound, not robot-bound	$10^4$ to $10^6$ real steps, or a sim-to-real gap to close
Distribution optimized on	Expert states only, so covariate shift is unaddressed	Buffer states, with pessimism outside their support	The policy’s own states, which is the point
Performance ceiling	The demonstrator, minus compounding error	Best behavior recoverable from the logged data	Can exceed the human on speed and precision
Main failure mode	Drift into unseen states; mode averaging on multimodal demos	Value overestimation on out-of-support actions	Reward hacking, hardware damage, forgetting language grounding

August 17, 2026

DL0118 VLA vs VLM

What is a VLA model and how does it differ from a VLM?

Answer

A VLA (Vision-Language-Action) model is a policy that maps camera images plus a natural-language instruction directly to robot actions, typically a short sequence of end-effector or joint deltas plus a gripper command, emitted at a fixed control rate. A VLM (Vision-Language Model) maps the same image-plus-text input to text tokens. Architecturally the two are close relatives: almost every modern VLA (RT-2, OpenVLA, $\pi_0$ ) starts from a pretrained VLM backbone and is fine-tuned on teleoperated demonstration trajectories, either by discretizing each action dimension into vocabulary tokens or by attaching a continuous action expert head. The real difference is not the encoder, it is everything downstream of it: the output lives in a continuous, embodiment-specific action space, the model runs inside a closed feedback loop where its own outputs change the next observation, and errors therefore compound over the rollout instead of being independent per query. That single fact drives the different data (robot demos, not web image-text pairs), the different latency budget (tens of milliseconds, not seconds), and the different metric (physical task success rate, not benchmark accuracy).

(1) Output Space: a VLM produces a distribution over a discrete text vocabulary; a VLA produces a vector in $\mathbb{R}^{d}$ per timestep, usually predicted as an action chunk covering the next $H$ control steps.
(2) Closed Loop Versus Open Loop: a VLA’s action changes the world and therefore its own next input, so covariate shift makes behavior-cloning error grow roughly with the square of the horizon; a VLM answer is scored once and never fed back through a robot.
(3) Training Data: web-scale image-caption and VQA corpora for the VLM, versus expensive teleoperated trajectories (Open X-Embodiment aggregates roughly one million episodes) with synchronized proprioception for the VLA.
(4) Action Representation Is A Design Choice: RT-2 and OpenVLA quantize each dimension into 256 bins and reuse rarely-used text tokens; $\pi_0$ instead trains a flow-matching action expert that emits continuous chunks at 50 Hz.
(5) Latency Is A Correctness Constraint: a 2 s VLM response is fine, but a controller starved of fresh actions produces jerky or unsafe motion, so chunk horizon and inference time must be budgeted together.
(6) Embodiment Coupling: VLM weights transfer across any image; VLA action heads are tied to a specific DoF count, camera mount, and control convention, which is why cross-embodiment training is an active research problem.

Diagram showing a shared ViT plus LLM backbone taking an RGB observation and a language instruction, branching into a text decoder head producing answer tokens for a VLM and an action expert head producing a 50 by 7 action chunk executed by a robot at 50 Hz, with a feedback arrow returning the new observation to the encoder

Figure 1: Both models share the same perception stack; the VLA replaces or augments the text head with an action head and runs inside the loop observe → predict chunk → execute → observe, so its own predictions determine the next input distribution.

The token-budget arithmetic explains why action chunking is universal. A 7-DoF arm predicted one step at a time at 50 Hz would need a full autoregressive forward pass every 20 ms, which no 3B-parameter backbone can sustain. Predicting a chunk of $H = 50$ actions amortizes one forward pass across a full second of motion, at the price of running open loop within the chunk. Shorter chunks mean tighter feedback and better disturbance rejection but more compute and more jitter at chunk boundaries; longer chunks are smoother but blind to anything that happens mid-chunk. Naive autoregressive decoding of 350 discrete action tokens is also slow, which is exactly the bottleneck that continuous action experts and frequency-domain tokenizers were built to remove.

Mathematical Formulation:
$p_\theta(y_{1:T} \mid I, \ell) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{1:t-1}, I, \ell)$
$a_{t:t+H-1} \sim \pi_\theta(\cdot \mid o_t, s_t, \ell)$
$k_j = \mathrm{round}\left(\frac{a_j - a_{\min}}{a_{\max} - a_{\min}}(B-1)\right)$
$N_{tok} = H \cdot d = 50 \cdot 7 = 350$
$t_{infer} \leq H / f_{ctrl}$
$50 / 50\ \text{Hz} = 1\ \text{s}$
$J(\pi_\theta) - J(\pi^{*}) = O(\epsilon T^{2})$

Where:

$y_{1:T}$ are the text tokens a VLM emits for image $I$ and instruction $\ell$ ; the factorization is the only thing the VLA keeps unchanged.
$a_{t:t+H-1} \in \mathbb{R}^{H \times d}$ is the action chunk, $o_t$ the current camera observation, and $s_t$ the proprioceptive state that a VLM never receives.
$d$ is the action dimension (7 for a 6-DoF pose delta plus gripper) and $H$ the chunk horizon in control steps.
$k_j \in \{0, \ldots, B-1\}$ is the discrete bin for dimension $j$ , with $B = 256$ and $[a_{\min}, a_{\max}]$ set from per-dimension training quantiles so outliers do not collapse the resolution.
$f_{ctrl}$ is the control frequency and $t_{infer}$ the policy latency; the required deployment condition is that a new chunk arrives before the previous one is exhausted.
$\epsilon$ is the per-step imitation error, $T$ the rollout length, and $J$ the task cost; the quadratic bound is the classical behavior-cloning compounding result.

Log-scale chart of accumulated error against rollout length, comparing a linear curve for independent per-query error and a quadratic curve for closed-loop behavior cloning, with the quadratic curve 400 times higher at 400 steps

Figure 2: A VLM’s mistakes are independent per query and accumulate linearly; a VLA’s mistakes move the robot off the demonstration distribution, so the worst-case cost grows as $O(\epsilon T^{2})$ and a 1% per-step error is fatal over a 400-step manipulation.

Property	VLA (Vision-Language-Action)	VLM (Vision-Language Model)
Output	Continuous action chunk, typically 50 steps by 7 dimensions, as bin tokens or a flow-matching head	Text tokens from a fixed vocabulary of roughly 32k to 256k entries
Extra inputs	Proprioception, gripper state, often multiple synchronized camera views	Images and text only
Training data	Teleoperated demonstrations (Open X-Embodiment scale is about 1M episodes), usually co-trained with web data to keep semantics	Billions of web image-text pairs plus instruction tuning
Latency budget	Chunk must arrive before the previous one runs out, so tens to a few hundred milliseconds	Seconds; streaming hides most of it from the user
Evaluation	Physical or simulated rollout success rate over many trials, with high variance and slow iteration	Static benchmark accuracy or preference scores, reproducible offline
Dominant failure mode	Compounding covariate shift, unrecoverable states, embodiment mismatch, control jitter	Hallucination and grounding errors, recoverable by re-prompting

August 17, 2026

DL0117 3D Attention vs Frame Pooling

Compare 3D convolutional and space-time attention modules (e.g., TimeSformer) against frame-level pooling for temporal token aggregation in video encoders. When is each the right choice?

Answer

Both families turn $T$ frames of $N$ patch tokens into one clip representation, and they differ only in where temporal information is allowed to mix. Frame-level pooling runs a purely spatial encoder per frame and then averages or attention-pools the $T$ frame vectors (frames → per-frame ViT → pool → head), so no token ever sees another frame and the aggregation is permutation invariant over time. 3D modules mix earlier: a 3D convolution gives each token a local spatiotemporal receptive field of size $k_t$ , joint space-time attention lets all $NT$ tokens attend to each other, and TimeSformer’s divided space-time attention factorizes that into a temporal MSA over the same spatial position across frames followed by a spatial MSA inside each frame. The cost separation is the first thing to state in an interview: joint attention is $O(N^2T^2)$ in the pair count while divided is $O(N^2T + NT^2)$ , a ratio of $NT/(N+T)$ that reaches roughly 64x at 96 frames. The accuracy separation depends almost entirely on whether the label actually depends on frame order: on scene-biased Kinetics-400 dropping temporal attention costs about a point, while on Something-Something V2 the same ablation costs roughly 23 points.

(1) Pooling Is Order-Blind By Construction: mean or max pooling over frame embeddings is symmetric, so “opening a door” and “closing a door” produce the identical clip vector no matter how good the image backbone is.
(2) 3D Convolution Buys Locality Cheaply: cost is linear in $T$ , but the temporal receptive field grows only $k_t - 1$ frames per layer, so long-range order needs depth or a slow/fast dual pathway.
(3) Joint Attention Is Global But Quadratic: with ViT-B at 8 frames the token sequence is $196 \times 8 = 1568$ , and every added frame inflates the attention matrix quadratically.
(4) Factorization Is The Practical Default: divided space-time attention beat both space-only and joint attention in the TimeSformer ablations while being far cheaper than joint, which is why factorized variants dominate video ViTs.
(5) Benchmark Bias Decides The Verdict: appearance-biased datasets reward a strong image backbone, and temporally-ordered datasets punish any aggregator that discards order.
(6) Pooling Keeps System Properties Attention Destroys: per-frame embeddings can be cached, indexed, and streamed independently, which is why large-scale video retrieval still ships CLIP-style pooled encoders.

Three grids of tokens arranged as spatial patches by frames; in the first panel a query token connects only to tokens in its own frame, in the second it connects to its own frame plus the same spatial position in all frames, and in the third it connects to every token in the clip

Figure 1: The three aggregation schemes differ only in the attended set of a query token. Space-only attention plus pooling never crosses a frame boundary, divided space-time adds a one-dimensional temporal pass over the same spatial position, and joint attention connects all $NT$ tokens at quadratic cost.

A subtlety that separates mid from senior candidates is that divided attention is not simply “cheaper joint attention”. Its temporal MSA only compares a patch with the same spatial coordinate in other frames, so a fast-moving object that shifts several patches between frames is matched indirectly, through the spatial MSA that follows. That works because the two passes alternate at every block, but it is also why divided attention benefits from higher frame rates and larger patch strides, and why 3D convolutions with a spatial kernel remain competitive on motion-heavy, short-horizon tasks. Practically, inflating an image-pretrained ViT into a divided model requires zero-initializing the temporal projection so the network starts as an exact image model and the pretrained features survive the first epochs.

Mathematical Formulation:
$z = \frac{1}{T}\sum_{t=1}^{T} f(x_t)$
$C_{pool} = O(N^2 T D)$
$C_{3D} = O(k_t k_s^2 N T D^2)$
$C_{joint} = O(N^2 T^2 D)$
$C_{div} = O(N^2 T D + N T^2 D)$
$\frac{C_{joint}}{C_{div}} = \frac{NT}{N + T}$
$\frac{196 \cdot 96}{196 + 96} \approx 64$

Where:

$z$ is the clip embedding and $f$ the frozen or fine-tuned per-frame encoder applied to frame $x_t$ ; because the sum is symmetric, $z$ is unchanged by any permutation of the frames.
$t \in \{1, \ldots, T\}$ indexes frames, $N$ is the number of patch tokens per frame ( $196$ for ViT-B at $224 \times 224$ with patch 16), and $D$ is the model width.
$C_{pool}$ , $C_{joint}$ , and $C_{div}$ are the attention costs of space-only, joint space-time, and divided space-time blocks; all three exclude the identical per-token MLP term.
$k_t$ and $k_s$ are the temporal and spatial kernel sizes of a 3D convolution, whose cost is linear in $T$ but whose temporal receptive field after $L$ layers is only about $L(k_t - 1) + 1$ frames.
The ratio $NT/(N+T)$ holds whenever $T$ is at least 2; it grows toward $N$ as $T$ grows, so the saving is bounded above by the token count per frame.

Grouped bar chart of top-1 accuracy for space-only, joint space-time, and divided space-time attention on Kinetics-400 and Something-Something V2, showing a small gap on Kinetics and a very large gap on Something-Something

Figure 2: TimeSformer ViT-B ablations at 8 frames. Removing temporal mixing costs only 1.1 points on Kinetics-400, whose classes are largely identifiable from a single frame, but 22.9 points on Something-Something V2, where the label is defined by the direction of motion.

Property	Frame-level pooling	3D convolution (I3D, X3D, SlowFast)	Divided space-time attention
Temporal receptive field	None inside the encoder; one symmetric average at the end	Local, grows by k_t minus 1 frames per layer	Global over all T frames from the first block
Cost scaling in T	Linear, and trivially parallel across frames	Linear, with a constant factor of k_t	Linear plus a small quadratic term N T squared
Sensitive to frame order	No; shuffled clips give identical embeddings	Yes, within the local window	Yes, with temporal position embeddings
Image pretraining transfer	Perfect; the backbone is unchanged	Via kernel inflation and rescaling	Strong if the temporal projection is zero-initialized
Per-frame embedding caching	Yes; embeddings are reusable across clips and queries	No; features depend on neighboring frames	No; every block mixes across the whole clip
Typical failure	Collapses on reversible actions and counting tasks	Misses long-horizon structure without deep stacks	Weak on fast motion that leaves the shared patch column
Where it wins	Retrieval, tagging, zero-shot with image-text encoders	Short motion-heavy clips on constrained hardware	Order-sensitive recognition over dozens of frames

August 17, 2026

DL0116 Video VLM Temporal Sampling

How do video VLMs balance temporal sampling resolution against KV cache memory limits when processing multi-hour video streams, as in Google’s Gemini video understanding?

Answer

A video VLM has no separate temporal-resolution dial: it flattens the stream into one token sequence, so frame rate, tokens per frame, and clip length all collapse into a single token count $N = f \cdot T \cdot t_f$ , and that count fixes both the KV cache size (linear in $N$ ) and the prefill cost (quadratic in $N$ ). Two hours sampled at 1 fps with 64 tokens per frame is 460,800 tokens; on a Llama-3-8B-class decoder with 32 layers, 8 KV heads of head dimension 128, and a bf16 cache, each token costs 128 KB of KV, so the cache alone is roughly 60 GB, against the roughly 62 GB an 80 GB H100 has free after weights and activations. Multiply the frame rate by four or keep the encoder’s full 256 tokens per frame and you are at 240 GB, which no single device holds. Production systems therefore invert the question: fix a token budget from the memory and latency SLO first, then decide how to spend it across time and space. The three levers are temporal subsampling (fewer frames), token compression (pooling, merging, or resampling each frame), and bounded state (a sliding window plus a compressed memory bank or an offline retrieve-then-read pass), and only the third one makes memory independent of stream length.

(1) Tokens Are The Currency: memory scales as $O(N)$ and prefill attention as $O(N^2)$ , so a 4x frame-rate increase is a 4x memory bill and a 16x prefill bill.
(2) Per-Token KV Cost Is Architectural: $m = 2 L n_{kv} d_h b$ bytes, so GQA with 8 KV heads instead of 32 already cuts it 4x, and INT4 KV quantization cuts it another 4x before any sampling change.
(3) Equal Memory, Different Failures: halving $f$ and quartering $t_f$ can cost the same tokens, but the first causes temporal aliasing of short events while the second destroys small text and fine spatial detail.
(4) Slow-Fast Splitting: strong systems decouple the two axes, keeping many frames at very few tokens for motion and a handful of keyframes at full spatial resolution for detail.
(5) Adaptive Beats Uniform At Fixed Budget: dropping near-duplicate frames by feature similarity and selecting query-relevant segments spends the same tokens on the informative part of the timeline.
(6) Time Grounding Must Survive Downsampling: position encodings need absolute timestamps, not token indices, or a variable frame rate makes the model’s answers about “when” drift.

Log-scale bar chart of KV cache size for a two-hour video under six configurations: 242 GB at 1 fps with 256 tokens per frame, 60 GB at 1 fps with 64 tokens, 30 GB at 0.5 fps, 15 GB at 16 tokens per frame, 15 GB with INT4 cache, and under 1 GB for a streaming window plus memory bank, against a 62 GB device budget line

Figure 1: KV cache for the same two-hour video under six budgets. Memory depends only on the product $f \cdot T \cdot t_f$ and the per-token cost, so sampling changes and cache quantization are interchangeable for memory, but only a bounded streaming state stops growth with stream length.

The reason the two sampling axes are not interchangeable for accuracy is that they alias different things. Uniform sampling at 1 fps is enough for plot-level questions over an hour of footage but silently deletes any event shorter than a second, which is why counting, ordering, and “who handed what to whom” questions collapse at low frame rates. Cutting tokens per frame is safe for gist and scene recognition and catastrophic for anything requiring OCR of on-screen text or small-object detail. Public systems make these trade-offs explicit: Gemini samples video at 1 fps with about 258 tokens per frame by default, which is roughly 0.93M tokens per hour and effectively consumes a 1M-token context, and offers a low-resolution mode at about 66 tokens per frame that stretches the same window to several hours. For genuinely unbounded streams the only stable design is bounded state: keep the last few tens of seconds as an exact KV window with attention sinks so the distribution stays in-domain, merge evicted frames into a fixed-size memory bank, and for offline archives replace dense prefill with a two-pass pipeline that indexes cheap summaries and then re-decodes only candidate segments at high frame rate.

Mathematical Formulation:
$N = f \cdot T \cdot t_f$
$m = 2 L n_{kv} d_h b$
$M_{kv} = m \cdot N$
$C_{prefill} = O(N^2 d)$
$f_{max} = \frac{M_{budget}}{m \cdot T \cdot t_f}$
$m = 2 \cdot 32 \cdot 1024 \cdot 2 = 131072$
$N = 1 \cdot 7200 \cdot 64 = 460800$

Where:

$N$ is the total visual token count fed to the decoder, $f$ the sampling rate in frames per second, $T$ the stream duration in seconds, and $t_f$ the tokens retained per frame after pooling or resampling.
$m$ is the KV bytes per token, with $L$ layers, $n_{kv}$ key-value heads, head dimension $d_h$ , and $b$ bytes per stored element; the factor 2 counts keys and values.
$M_{kv}$ is cache size and $M_{budget}$ the free HBM after weights, activations, and the text prompt; $f_{max}$ is the highest frame rate a dense prefill can afford at a given $t_f$ .
$C_{prefill}$ is prefill attention cost with model width $d$ ; it usually binds before memory does, since doubling $N$ quadruples time to first token.
The numeric example uses $L = 32$ , $n_{kv} d_h = 1024$ , and $b = 2$ bytes, giving 128 KB per token, and $T = 7200$ s at $f = 1$ , $t_f = 64$ , so $M_{kv} \approx 60$ GB.
A streaming design replaces $N$ with $N_w + N_{mem}$ , the exact window plus a fixed memory bank, which is $O(1)$ in $T$ .

Pipeline diagram: a two-hour 30 fps stream is decoded at 1 fps, encoded per frame into 256 patch tokens, reduced to 64 tokens per frame, then fed into a sliding-window KV cache whose evicted frames are merged into a fixed-size compressed memory bank, with both feeding an LLM decoder that answers a timestamped user query

Figure 2: A streaming video VLM: decode rate and per-frame token count shrink the sequence by two orders of magnitude, then eviction into a fixed memory bank makes resident state constant in stream length while the exact sliding window preserves fine detail for recent seconds.

Property	Uniform dense sampling	Query-adaptive keyframe retrieval	Streaming window plus memory bank
Token growth with duration	Linear and unbounded	Bounded at read time, but the index still scales with duration	Constant resident state
KV cache for 2 h	60 GB at 1 fps and 64 tokens per frame; 240 GB at full 256 tokens	A few GB, set by the number of retrieved segments	Under 1 GB for roughly 5,900 live tokens
Where information is lost	Between sampled frames, uniformly across the timeline	In segments the retriever scored as irrelevant	In merged history: detail decays with age
Best fit	Clips of a few minutes and dense temporal grounding	Offline archives with a known question per request	Live monitoring, assistants, always-on capture
Typical failure	Out-of-memory, or time to first token in the tens of seconds	Retriever misses the one relevant second, so the answer is confidently wrong	Questions about hour-old fine detail that the merge already discarded
Latency profile	One huge quadratic prefill per request	Cheap index build offline, small prefill per query	Amortized per frame, answers available at any instant

August 17, 2026

DL0115 Visual Chain-of-Thought

Explain how Chain-of-Thought (CoT) prompting with visual cropping (Visual CoT) enhances multi-step spatial reasoning in complex visual problem-solving.

Answer

Text-only CoT helps a vision-language model because it lets the model spend more tokens on intermediate reasoning, but it cannot help with detail the vision encoder never tokenized. A CLIP ViT-L/14 at $336 \times 336$ produces a fixed $24 \times 24$ grid of 576 patch tokens, so a $1500 \times 1500$ photograph is downsampled by $4.5$ x before a single transformer layer runs and a 60-pixel license plate lands inside less than one patch. Visual CoT closes that gap by making localization an explicit step in the reasoning chain: the model first emits a bounding box for the region the sub-question depends on, that crop is re-encoded at the encoder’s native resolution, the new visual tokens are appended to the context alongside the original global tokens, and only then does the model answer. The chain becomes localize → crop → re-encode → reason, and it can iterate, which is exactly what multi-step spatial questions need: “what does the sign to the left of the red truck say?” decomposes into finding the truck, resolving left of in the global view, then reading text at a magnification where the glyphs actually exist. The VisCoT work trains this behavior with 438k question-answer pairs carrying intermediate bounding-box annotations, and reports its largest gains on text-rich and small-object splits, precisely where fixed-resolution encoding destroys the evidence.

(1) Perception Bottleneck, Not Reasoning Bottleneck: on high-resolution inputs the failure is usually that the target occupies a fraction of a patch, so extra reasoning tokens cannot recover it.
(2) Crop As A Resolution Amplifier: re-encoding a $w \times h$ crop at $S \times S$ raises the sampling density on that region by $WH/(wh)$ without changing the encoder.
(3) Global Tokens Must Be Retained: a crop destroys the frame of reference, so relational predicates such as “left of” or “third from the top” require keeping the original 576 tokens in context.
(4) Errors Compound Multiplicatively: the answer is correct only if localization and reading are both correct, so a 0.9 localization accuracy caps a two-hop chain at 0.81.
(5) It Needs Grounding Supervision: unlike text CoT, the intermediate step is a box, so it comes from bbox-annotated data, a detector tool, or a localization reward rather than from prompting alone.
(6) Sequential Cost: each hop is an extra forward pass over the encoder plus a longer prefill, which trades latency for accuracy in a way tiling does not.

Left-to-right pipeline: a 1500 by 1500 image and question are globally encoded into 576 tokens, the model predicts a bounding box, the crop is re-encoded at 336 by 336 into another 576 tokens, and the model reasons over 1152 tokens to produce an answer, with a loop back to the localization step for multi-step questions and a dashed path showing the global tokens retained in context

Figure 1: Localization becomes an explicit reasoning step: the crop is re-encoded at native resolution and concatenated with the global tokens, so each hop of a multi-step spatial question is answered at the magnification it needs while the frame of reference survives.

The cost profile is what makes this a design decision rather than a free win. Visual CoT buys resolution with sequential compute: two or more encoder passes and a growing prefill, so at fixed batch size it roughly doubles time-to-first-token, and every hop is a place where a bad box silently poisons everything downstream. Tiling approaches such as AnyRes buy resolution with parallel compute instead, encoding a fixed grid of sub-images in one pass, which is friendlier to serving but pushes the visual context to 2,880 tokens or more and makes the language model’s $O(N^2)$ attention the new bottleneck. The interesting property of the crop-based chain is that it is adaptive: it spends its extra tokens only on the region the question actually depends on, which is why it scales to $4$ K screenshots and document scans where uniform tiling would need dozens of tiles.

Mathematical Formulation:
$N = (S/p)^2$
$N = (336/14)^2 = 576$
$\rho = \frac{W H}{w h}$
$\rho = \frac{1500 \cdot 1500}{150 \cdot 150} = 100$
$p_{\mathrm{task}} = p_{\mathrm{loc}} \cdot p_{\mathrm{read}}$
$p_{\mathrm{task}} = 0.90 \cdot 0.90 = 0.81$

Where:

$N$ is the number of visual tokens per encoder pass, $S$ the encoder’s square input side, and $p$ the patch size; $S = 336, p = 14$ gives the familiar 576 tokens.
$W \times H$ are the original image dimensions and $w \times h$ the predicted crop, both in pixels, with the crop constrained to lie inside the image.
$\rho$ is the areal sampling gain on the cropped region, so the linear magnification is $\sqrt{\rho} = 10$ for the numbers above.
$p_{\mathrm{loc}}$ is the probability the emitted box actually contains the target and $p_{\mathrm{read}}$ the probability of reading it correctly once magnified; $p_{\mathrm{task}}$ is end-to-end accuracy for a two-hop chain.
For a $k$ -hop chain the same argument gives $p_{\mathrm{task}} = \prod_{i=1}^{k} p_i$ , and total visual tokens grow as $(k+1)N$ because the global view is kept.

Log-scale bar chart of how many ViT patches cover a 60 by 60 pixel target inside a 1500 by 1500 image: under one patch for a single 336 by 336 encode with 576 tokens, about four patches for AnyRes 2 by 2 tiling with 2880 tokens, and about 92 patches for a re-encoded 150 by 150 Visual CoT crop with 1152 tokens

Figure 2: A 60-pixel target inside a $1500 \times 1500$ image gets under one patch from a single low-resolution pass and about four from $2 \times 2$ tiling, but roughly 92 patches after a crop is re-encoded, at half the token cost of tiling.

Property	Visual CoT (crop and re-encode)	Fixed single low-res pass	AnyRes tiling
How fine detail is obtained	Model predicts a bbox, that crop is re-encoded at native encoder resolution	Not obtained; anything below one patch is destroyed at encode time	Image is split into a fixed grid of tiles, each encoded at full resolution
Visual tokens at 1500×1500	1,152 for one hop (576 global + 576 crop)	576	2,880 for four tiles plus a global view
Encoder passes per answer	Two or more, strictly sequential	One	One, tiles batched in parallel
Patches on a 60 px target	About 92	Under 1	About 4
Supervision required	Bbox-annotated chains, a detector tool, or a localization reward	None beyond standard instruction tuning	None; purely an input-packing change
Main failure mode	A wrong box yields a confident answer about the wrong region	Small text, gauges, and distant objects are simply invisible	Quadratic attention growth, and tile borders cut objects in half

August 17, 2026

DL0114 Chain-of-Thought Prompting

What is Chain-of-Thought prompting and why does it improve reasoning?

Answer

Chain-of-Thought (CoT) prompting makes a language model emit intermediate reasoning steps before its final answer, either through few-shot exemplars whose demonstrations contain worked solutions (Wei et al., 2022) or through a zero-shot trigger phrase such as “Let’s think step by step” (Kojima et al., 2022). The gains are large on multi-step problems: on GSM8K, 8-shot CoT lifted PaLM 540B from 17.9% to 56.9%, and the same trigger took GPT-3 (text-davinci-002) from 10.4% to 40.7% with no weight updates. Two mechanisms explain why. The first is serial compute: a transformer with $L$ layers can apply only $O(L)$ sequential operations before it must commit to the next token, so a task needing more sequential steps than the depth provides is simply not representable in one forward pass; every generated token reruns the whole stack, turning the context window into an external scratchpad that raises effective serial depth from $L$ to $L(T+1)$ . Theory backs this up: constant-depth transformers with a polynomially long chain can simulate any polynomial-time computation, while the same model restricted to direct answers cannot. The second mechanism is factorized conditioning: instead of sampling one high-entropy jump from question to answer, the model decomposes the problem into a product of low-entropy conditionals, each conditioned on the already-written steps, which keeps it on the distribution of solution traces seen in pretraining. CoT is not universally free money: it is emergent with scale, and later meta-analysis shows the gains concentrate on math, symbolic, and logical tasks rather than on knowledge or commonsense retrieval.

(1) Two Elicitation Modes: few-shot CoT shows worked examples and controls format tightly, while zero-shot CoT appends one trigger sentence and costs almost no prompt tokens.
(2) Depth Becomes Time: each scratchpad token buys another full pass through the network, so the model trades latency and tokens for sequential computation it structurally lacked.
(3) Easier Conditionals: the chain rewrites one hard prediction as many easy ones, and each step reads all earlier steps as ordinary context.
(4) Emergent, Not Universal: below roughly 10B parameters, CoT often produces fluent but invalid chains and can score below direct prompting.
(5) Chains Are Not Guaranteed Faithful: models shift answers when a prompt carries a bias cue while the written rationale never mentions it, so a chain is an output artifact, not an audit trail.
(6) Cheap Ensembling On Top: because chains are stochastic, sampling $k$ of them and majority-voting the final answers (self-consistency) pushed PaLM 540B on GSM8K to 74.4%.

Two-panel diagram: direct prompting sends the question through one L-layer forward pass to the answer, while chain-of-thought generates T scratchpad tokens, each of which reruns the full L-layer stack before the answer token

Figure 1: Direct prompting caps sequential computation at the network depth $L$ ; CoT reuses the same weights once per generated token, giving $L(T+1)$ sequential steps and letting step $t$ read every earlier step as context.

The costs are as concrete as the benefits. Chains are typically 100 to 400 decode tokens on grade-school math, so a CoT query can cost an order of magnitude more decode passes and time-to-last-token than a direct answer, which matters for anything user-facing. Errors also propagate: the decoder never backtracks, so a slip in the first arithmetic step is carried through every later step, and accuracy behaves roughly like the product of per-step accuracies. Few-shot CoT is additionally prompt-sensitive, since exemplar choice, ordering, and even the formatting of the equations move accuracy by several points. These properties are exactly why the field moved from prompting toward verifying or training the chain rather than just requesting it.

Mathematical Formulation:
$p(a \mid q) = \sum_{z} p(a \mid q, z)\, p(z \mid q)$
$p(z \mid q) = \prod_{t=1}^{T} p(z_t \mid q, z_{1:t-1})$
$D_{\text{direct}} = L$
$D_{\text{CoT}} = L \cdot (T + 1)$
$\hat{a} = \arg\max_{a} \sum_{i=1}^{k} \mathbf{1}[a_i = a]$

Where:

$q$ is the prompt (question plus any exemplars), $a$ the final answer span, and $z = z_{1:T}$ the generated rationale, a latent variable the model writes into its own context.
$t \in \{1, \ldots, T\}$ indexes chain tokens; $T$ is the chain length and $L$ the number of transformer layers.
$D$ counts the sequential layer applications available before the answer is committed, which is the resource CoT actually adds.
$k$ is the number of sampled chains in self-consistency, $a_i$ the answer extracted from sample $i$ , and $\mathbf{1}[\cdot]$ the indicator used for majority voting.
Required condition: the sum over $z$ is intractable, so plain CoT approximates it with a single greedy or sampled $\hat{z}$ , and self-consistency approximates it with $k$ Monte Carlo samples at temperature $T_s > 0$ .

Grouped bar chart of GSM8K solve rate for PaLM at 8B, 62B and 540B parameters under standard prompting and 8-shot chain-of-thought, with the CoT advantage appearing only at 62B and growing at 540B

Figure 2: CoT is an emergent ability: at 8B parameters the chains are fluent but wrong and buy nothing, while at 540B they roughly triple the GSM8K solve rate. Values are approximate figures reported for PaLM by Wei et al. (2022) and Wang et al. (2023).

Property	Zero-shot CoT	Few-shot CoT	CoT + self-consistency
How it is elicited	One trigger sentence appended to the question	A handful of exemplars that show the full reasoning trace	Same CoT prompt sampled k times, majority vote on the extracted answer
Extra prompt tokens	Roughly 10	Hundreds to a few thousand, and it consumes context	Unchanged prompt, but k independent decodes
Reported GSM8K result	10.4% to 40.7% on text-davinci-002	17.9% to 56.9% on PaLM 540B, 8-shot	74.4% on PaLM 540B with 40 sampled chains
Main weakness	Format drift: the model may skip steps or answer immediately	Sensitive to exemplar choice, order, and formatting	k times the decode cost; needs a comparable, extractable final answer
Best fit	Quick baseline and open-ended chat traffic	A fixed task where output format must be stable	High-value math or code queries where accuracy beats latency

August 17, 2026

DL0113 VLM Object Hallucination

Explain the primary causes of Object Hallucination in VLMs, for a captioning feature built on a model like LLaVA or Qwen2-VL.

Answer

Object hallucination is the case where a vision-language model asserts an object that is not present in the image, and it is not one bug but the sum of four largely independent failure sources stacked along the encode → project → decode pipeline. The vision tower is usually a frozen contrastively-trained ViT, so it encodes globally discriminative semantics and is measurably weak on small objects, counts, duplicates, and absence; the projector then compresses the image to a fixed budget of visual tokens (576 for CLIP ViT-L/14 at 336 px, only 32 query tokens for a BLIP-2 style Q-Former), discarding spatial detail the decoder can never recover. On the language side, the decoder was pretrained on text alone and carries a strong object co-occurrence prior, so whenever the visual evidence for a token is weak the prior decides: “dining table” pulls in “fork”, “kitchen” pulls in “refrigerator”. Instruction tuning makes this worse rather than better, because much visual SFT data is written by a text-only model from captions and bounding boxes and therefore contains ungrounded details, and the training objective rewards confident, fluent, detailed answers with no penalty term for an unsupported noun. Finally, decoding itself drifts: attention to visual tokens concentrates on a few early anchor positions and decays with sequence position, so late sentences are generated nearly blind, and once a wrong noun is emitted it conditions everything after it, producing the well-documented snowball effect.

(1) Weak Visual Features: a contrastive image-text objective optimizes for retrieval-level discrimination, not localization, so paired images differing in one small detail can receive nearly identical embeddings.
(2) Token And Resolution Budget: a fixed low-resolution grid plus projector compression means thin, small, or crowded objects arrive at the decoder as a few blurred features.
(3) Language Co-occurrence Prior: the text-pretrained decoder completes plausible scenes, and objects that frequently co-occur with the true content are the ones hallucinated first.
(4) Instruction Data Noise And Yes-Bias: SFT data generated from captions and boxes contains details never visible, and presence questions are answered affirmatively far more often than chance.
(5) Objective Mismatch: next-token cross-entropy has no grounding loss, and helpfulness-oriented preference tuning rewards verbosity, which mechanically increases the number of nouns at risk.
(6) Autoregressive Drift: visual attention mass decays over the generated sequence and hallucinated nouns become context, so error rate grows with output length rather than with model size.

Pipeline diagram: a 336 by 336 image enters a frozen ViT-L/14 encoder, a projector emits 576 visual tokens, a text-pretrained LLM decoder generates a caption, with four labelled causes attached to the encoder, projector, decoder, and the autoregressive feedback loop

Figure 1: Each stage can independently drop or overwrite visual evidence: the frozen contrastive encoder loses fine detail, the projector loses spatial resolution, the text-only prior fills the gap, and the sampling loop recycles its own mistakes.

The two benchmark families separate these causes reasonably well. CHAIR parses generated captions against a fixed object vocabulary and reports the fraction of mentioned objects that are absent, which makes drift visible because the metric worsens as captions get longer. POPE instead asks balanced yes/no presence questions and splits the negatives three ways: random objects, popular objects, and adversarial objects chosen because they co-occur most often with the ground-truth objects. The gap between the random and adversarial splits is close to a direct measurement of language-prior dominance, and the reported yes-ratios above 90% for several 2023-era models are a direct measurement of the SFT affirmation bias. A useful diagnostic in the same spirit is to score a candidate token twice, once with the image and once with the image removed, since a near-zero difference means the prior, not the pixels, chose that word.

Mathematical Formulation:
$s_t = \log p_\theta(y_t \mid v, x, y_{1:t-1})$
$s_t^{\mathrm{lang}} = \log p_\theta(y_t \mid x, y_{1:t-1})$
$\Delta_t = s_t - s_t^{\mathrm{lang}}$
$s_t^{\mathrm{cd}} = (1 + \alpha)\, s_t - \alpha\, s_t^{\mathrm{dist}}$
$\mathrm{CHAIR}_i = \frac{|H|}{|M|}$

Where:

$y_t$ is the token generated at step $t$ and $y_{1:t-1}$ the tokens already committed, which is why an early mistake is irreversible.
$v$ is the sequence of projected visual tokens (576 for a ViT-L/14 336 px grid, 32 for a Q-Former) and $x$ is the text prompt.
$s_t^{\mathrm{lang}}$ is the same score with the image dropped, so $\Delta_t$ is the visual grounding margin; $\Delta_t \approx 0$ means the token was chosen by the language prior alone.
$s_t^{\mathrm{dist}}$ is the score under a distorted or noised image and $\alpha \geq 0$ the contrast strength, giving the contrastive decoding adjustment used by VCD-style mitigations.
$M$ is the set of object mentions parsed from a caption and $H \subseteq M$ the subset absent from the annotation, so $\mathrm{CHAIR}_i$ is an instance-level hallucination rate in $[0, 1]$ .
Required condition for the margin test: both scores must be computed at the same step with identical $y_{1:t-1}$ , otherwise the two distributions are not comparable.

Two panel schematic chart: left panel shows attention mass on visual tokens declining from 0.30 to 0.06 across six generated sentence indices, right panel shows hallucinated object rate rising from 0.03 to 0.40 across the same indices

Figure 2: Schematic of the drift pattern reported by attention-analysis studies and by CHAIR-versus-length ablations: as generation proceeds, attention mass on visual tokens falls while the hallucinated-object rate rises, so long free-form captions hallucinate far more than short answers from the same model.

Aspect	Visual encoding bottleneck	Language prior dominance	Decoding-time drift
Typical symptom	Misses or confuses small, thin, or duplicated objects; counting and absence errors	Invents objects that usually co-occur with what is really there, such as a fork beside a dining table	First sentence accurate, later sentences increasingly invented; repeated nouns
Diagnostic probe	MMVP-style image pairs differing in one visual detail; linear probes on frozen features	POPE adversarial split versus random split; the grounding margin with and without the image	CHAIR plotted against caption length or max_new_tokens
Effect of a bigger decoder	Little help; the same weak features are described more confidently	Often worse, because a stronger text prior overrides weak visual evidence	Roughly unchanged; this is a sequence-length effect, not a capacity effect
Cheapest effective fix	Higher input resolution or dynamic tiling, more visual tokens, unfreezing the tower late in training	Contrastive decoding against a distorted image, plus preference tuning on grounded and ungrounded response pairs	Shorter outputs, beam-level over-trust penalties, post-hoc detector verification

August 17, 2026

DL0112 VLM Visual Grounding

How do VLMs perform Visual Grounding (predicting 2D/3D bounding boxes)?

Answer

Visual grounding asks the model to return the image region a phrase refers to, and current VLMs differ mainly in where the coordinates are physically produced. The dominant generalist recipe treats a box as text: coordinates are normalized and quantized onto a fixed grid (Kosmos-2 adds 1024 dedicated location tokens, Qwen-VL emits integers from 0 to 999 inside <box> markers, Qwen2.5-VL moved to absolute pixel values inside JSON) and the language model decodes them autoregressively under ordinary cross-entropy, so detection, referring expression comprehension, and grounded captioning all become one sequence task. The specialist recipe keeps a DETR-style decoder with learnable queries, a box regression head, and L1 plus GIoU losses, fusing text into the visual features and aligning region embeddings to word embeddings contrastively (MDETR, GLIP, Grounding DINO, Florence-2). A third, hybrid recipe uses an LLM hidden state as a prompt for an external mask or box decoder, as LISA does by feeding a <SEG> embedding to SAM. Whichever decoder is used, localization quality is mostly decided upstream: the connector has to hand the LLM patch tokens with their 2D layout and position information intact, at a resolution where the target object still covers several patches. 3D grounding extends the output to a 7 to 9 DoF box and introduces the real difficulty, depth and scale ambiguity, resolved either from point clouds (ScanRefer-style proposal-and-match pipelines) or monocularly with camera intrinsics, as in Cube-LLM.

(1) Coordinates As Text: a box becomes four discrete symbols on a normalized grid, so the only training signal is token cross-entropy, with no IoU-aware term and no set matching.
(2) The Connector Decides Localization: raster-ordered ViT patch tokens through an MLP projector preserve geometry, while abstractors that pool everything into a few dozen learned queries discard the spatial layout a box depends on.
(3) Resolution Beats Bin Count: quantization error at 1000 bins is sub-pixel, but a small object that occupies one patch cannot be localized tightly at any bin resolution, which is why dynamic resolution and tiling (AnyRes, Qwen2-VL M-RoPE) matter more than the tokenizer.
(4) Detection-Head Decoding: query-based decoders with region-word contrastive alignment still lead on tight-IoU metrics and emit hundreds of boxes in a single forward pass.
(5) Hidden State As Prompt: a referent or <SEG> token embedding can drive SAM or a 3D mask decoder, decoupling language reasoning from pixel-level decoding.
(6) 3D Adds Scale, Not Just Dimensions: monocular predictions are only consistent if camera intrinsics enter the model or the normalization, otherwise depth estimates do not transfer across datasets.
(7) Metrics And Supervision: RefCOCO family Acc@0.5 for 2D, [email protected] on ScanRefer for 3D, and every box target must be expressed in the same resized or tiled frame the encoder sees.

Diagram of a vision language model grounding pipeline: image, ViT encoder, MLP projector, LLM decoder, and three output paths producing location tokens, a referent hidden state for SAM or a DETR box head, and a 3D head using camera intrinsics

Figure 1: One shared perception stack, three places to produce coordinates. The path image → ViT patches → projector → LLM is identical; only the last stage differs, and the projector is where grounding is usually won or lost.

It is worth doing the arithmetic on the tokenizer, because interviewers often assume quantization is the limiting factor. With a 1000-bin normalized grid on a 1344 px side, the worst-case error per edge is about 0.67 px, far below what Acc@0.5 can detect. The real sensitivity is object size: a square object of side $s$ whose four edges each shift by $d$ has $\mathrm{IoU} = (s-d)^2 / (2s^2 - (s-d)^2)$ , so a 16 px object falls below 0.5 IoU once edges move about 3 px, while a 128 px object tolerates roughly 24 px. That is why generalist grounding scores jumped with higher effective input resolution rather than with finer coordinate grids: the earliest text-token generalists sat near 52 Acc@0.5 on RefCOCO val, Qwen-VL reached roughly 89, and Florence-2-L about 93, close to dedicated grounding detectors.

Mathematical Formulation:
$t_k = \mathrm{round}\left(\frac{c_k}{S}(n-1)\right)$
$p(b \mid I, q) = \prod_{k=1}^{4} p(t_k \mid I, q, t_{1:k-1})$
$\epsilon_{\max} = \frac{S}{2(n-1)}$
$\epsilon_{\max} = \frac{1344}{2 \cdot 999} \approx 0.67$
$\mathcal{L}_{\mathrm{box}} = \lambda_1 \lVert b - \hat{b} \rVert_1 + \lambda_2 (1 - \mathrm{GIoU})$
$B_{3D} = (x, y, z, w, h, l, \theta)$
$(u, v, 1)^{\top} = \frac{1}{z} K (x, y, z)^{\top}$

Where:

$b = (c_1, c_2, c_3, c_4)$ is the target box in the encoder’s resized pixel frame of side $S$ , and $\hat{b}$ is the prediction.
$t_k$ is the $k$ -th coordinate token and $n$ the number of bins (1024 for Kosmos-2 location tokens, 1000 for Qwen-VL integers).
$I$ is the image, $q$ the referring expression, and $t_{1:k-1}$ the previously emitted coordinate tokens, so decoding is strictly autoregressive.
$\epsilon_{\max}$ is the worst-case quantization error per edge in pixels, which scales linearly with input side and inversely with bin count.
$\lambda_1, \lambda_2$ weight the L1 and GIoU terms used by regression-head decoders; text-token models have no analogue of either.
$B_{3D}$ is a 3D box with center $(x, y, z)$ , extents $(w, h, l)$ , and yaw $\theta$ ; full 9 DoF variants add pitch and roll.
$K$ is the camera intrinsic matrix and $(u, v)$ the projected image point; this constraint is the required initial condition for monocular 3D grounding, since without $K$ the depth $z$ and the extents trade off freely.

Two panel chart: left panel shows worst-case per-edge quantization error in pixels versus number of coordinate bins for 448, 1344 and 3840 pixel inputs on log axes; right panel shows IoU versus per-edge shift in pixels for 16, 32 and 128 pixel objects with an IoU equals 0.5 threshold line

Figure 2: Left: a 1000-bin grid costs well under one pixel per edge even at high resolution, so coordinate tokenization is rarely the bottleneck. Right: the Acc@0.5 budget is roughly 3 px of edge error for a 16 px object, 6 px for 32 px, and 24 px for 128 px, which is why small-object grounding is an effective-resolution problem.

Property	Coordinates as text tokens (Kosmos-2, Qwen-VL)	Detection-head decoding (MDETR, Grounding DINO, Florence-2)	Hidden state as decoder prompt (LISA, Grounded 3D-LLM)
Where coordinates appear	In the text stream, as quantized or absolute numbers inside markers or JSON	From query embeddings in a cross-modal decoder with an explicit box head	Implicitly: a referent token embedding conditions an external mask or box decoder
Training signal	Token cross-entropy only, no IoU term	L1 plus GIoU with Hungarian matching, plus region-word contrastive alignment	Mask or box loss backpropagated into the LLM through the prompt embedding
Multi-instance output	Costs about 4 to 8 decoded tokens per box, so dense scenes are slow	Hundreds of boxes in one parallel forward pass	One region per referent token; needs several tokens for several targets
Strength	Unifies grounding with dialogue, counting, OCR, and grounded captioning in one head	Best tight-IoU accuracy and recall under a fixed latency budget	Pixel-accurate masks and 3D shapes without teaching the LLM geometry
Typical failure	Hallucinated coordinates for absent objects, plus parse failures and no calibrated confidence	Weak compositional or relational language, since reasoning capacity is limited	Two-stage error coupling; the frozen decoder caps achievable boundary quality

August 17, 2026

DL0111 SigLIP vs CLIP

What are the differences between SigLIP and CLIP?

Answer

CLIP and SigLIP share the same dual-encoder architecture, the same cosine-similarity scoring, and the same zero-shot classification recipe; the substantive difference is the training objective. CLIP optimizes a softmax InfoNCE loss in which each image’s logit row is normalized over every text in the batch and each text’s column over every image, so the loss contribution of a single pair depends on the entire batch, and the distributed implementation needs an all-gather plus a global normalization pass. SigLIP replaces this with a pairwise sigmoid loss that treats each of the $N^2$ cells of the similarity matrix as an independent binary decision (“is this a real pair?”), using a shared learnable temperature plus a learnable bias $b$ initialized to $-10$ to absorb the fact that only $N$ of the $N^2$ pairs are positive. Because nothing is normalized across the batch, the loss decomposes into $b \times b$ blocks, which lets SigLIP be implemented as a memory-efficient ring over devices and trained at batch sizes a softmax implementation cannot fit. Empirically the sigmoid loss is clearly better at small and moderate batch sizes, the two objectives converge above roughly 32k examples per step, and the million-example batch study SigLIP made possible showed that accuracy saturates long before that. That efficiency, plus the shape-optimized SoViT-400m/14 backbone released with it, is why SigLIP checkpoints became the default vision tower in many recent vision-language models such as PaliGemma.

(1) Loss Form: CLIP uses two symmetric softmax cross-entropies over the batch; SigLIP uses one binary logistic loss per pair, summed over the full $N \times N$ grid.
(2) Batch Coupling: softmax logits are only meaningful after a global normalization, so every device must see all $2N$ embeddings; sigmoid logits are independent, so blocks can be scored and reduced locally.
(3) Extra Bias Parameter: the sigmoid objective faces an extreme $N^2 - N$ negatives against $N$ positives imbalance, which the learnable bias fixes by starting every logit near “not a pair”.
(4) Batch-Size Behavior: sigmoid wins at 4k to 16k, matches softmax above roughly 32k, and both saturate rather than improving toward one million.
(5) Everything Else Is Unchanged: image tower, text tower, unit-norm embeddings, and prompt-based zero-shot evaluation are identical, so SigLIP is effectively a drop-in replacement at the loss layer.
(6) Practical Differences In The Checkpoints: public English SigLIP models use a 32k SentencePiece vocabulary with a 64-token text context (the original paper’s initial experiments used 16, but the released checkpoints increased it to 64), shorter than CLIP’s 77-token BPE context, which matters when captions are long.

Two 4 by 4 image-text similarity matrices. On the left, arrows show softmax normalization along every row and every column, coupling all pairs. On the right, each cell is an independent binary label scored by a sigmoid with learnable temperature and bias.

Figure 1: Both objectives see the same similarity matrix. CLIP normalizes each row and column, so a single logit’s gradient depends on the whole batch; SigLIP scores each cell as an independent binary label with a shared temperature and bias, which removes the coupling entirely.

Mathematical Formulation:
$s_{ij} = t\, x_i^{\top} y_j$
$\mathcal{L}_{\mathrm{i2t}} = -\frac{1}{N}\sum_{i} \log \frac{e^{s_{ii}}}{\sum_{j} e^{s_{ij}}}$
$\mathcal{L}_{\mathrm{t2i}} = -\frac{1}{N}\sum_{j} \log \frac{e^{s_{jj}}}{\sum_{i} e^{s_{ij}}}$
$\mathcal{L}_{\mathrm{CLIP}} = \tfrac{1}{2}\left(\mathcal{L}_{\mathrm{i2t}} + \mathcal{L}_{\mathrm{t2i}}\right)$
$\mathcal{L}_{\mathrm{SigLIP}} = \frac{1}{N}\sum_{i}\sum_{j}\log\left(1 + e^{-z_{ij}(s_{ij} + b)}\right)$

Where:

$x_i$ and $y_j$ are the unit-norm image and text embeddings, so $x_i^{\top} y_j$ is a cosine similarity in $[-1, 1]$ .
$s_{ij}$ is the scaled logit and $t = \exp(t')$ is the learnable temperature, parameterized in log space in both methods.
$i, j \in \{1, \ldots, N\}$ index the global batch; the diagonal $i = j$ holds the $N$ true pairs and the off-diagonal holds $N^2 - N$ negatives.
$z_{ij} = +1$ for a matched pair and $z_{ij} = -1$ otherwise, which is the binary label the sigmoid loss regresses on.
$b$ is the learnable bias unique to SigLIP, with the required initialization $b = -10$ and $t' = \log 10$ ; CLIP has no analogue because its softmax is shift-invariant.

The engineering consequence is the part interviewers usually probe. In a softmax implementation the batch is a single indivisible unit: after the all-gather each of the $D$ devices holds all $2N$ embeddings and materializes an $N \times b$ logit slab, and the row and column sums must be reduced across devices before any gradient exists. SigLIP instead keeps each device’s image chunk local and passes text chunks around a ring, accumulating loss from one $b \times b$ block at a time, so peak logit memory drops by a factor of $D$ and no cross-device normalization is needed. This is exactly what allowed the authors to sweep batch size up to one million and demonstrate saturation, and it is also why the SigLiT variant, which locks a pretrained image tower and trains only the text side, reached 84.5% ImageNet zero-shot accuracy in two days on four TPUv4 chips. The trade-off is two extra hyperparameters to get right: a badly initialized bias makes the first thousands of steps a wasted fight against the negative-pair prior.

Top panel: four devices each holding a local chunk feed into an all-gather box where every device holds all N embeddings for a global softmax. Bottom panel: four devices keep their image chunk local while text chunks rotate around a ring, scoring one b by b block per step.

Figure 2: The softmax loss forces an all-gather and a global normalization, so each device carries an $N \times b$ logit slab. The sigmoid loss decomposes, so a ring of $D$ steps covers every pair while only $b \times b$ logits exist at once.

Property	SigLIP (sigmoid loss)	CLIP (softmax InfoNCE)
Objective	One binary logistic loss per image-text pair over the full grid	Two symmetric cross-entropies over row-wise and column-wise softmax
Batch coupling	None; the loss is a sum of independent terms	Global; every logit is normalized against the whole batch
Extra parameters	Learnable temperature plus a learnable bias, initialized to log 10 and -10	Learnable temperature only; a bias would cancel in the softmax
Distributed cost	Chunked ring; peak logit memory b x b per device, no normalization reduce	All-gather of all embeddings plus an N x b logit slab per device
Small batch (4k to 16k)	Clearly stronger zero-shot accuracy at equal examples seen	Degrades noticeably; the normalization has few negatives to work with
Very large batch	Feasible up to one million, but accuracy saturates near 32k	Comparable above roughly 32k, but memory-bound before that
Text context	64 tokens (16 in the original paper), 32k SentencePiece in the English releases	77 tokens, 49k byte-pair vocabulary
Typical role today	Default frozen vision tower for many VLMs, notably SoViT-400m/14	Legacy ecosystem: diffusion text conditioning, CLIPScore, many distilled models

August 17, 2026