Tag: VLA

Vision-Language-Action models (robotic policies)

DL0121 Open-Loop vs Closed-Loop Validation

Explain why open-loop validation metrics (e.g., MSE on offline trajectories) often fail to correlate with closed-loop success rates in physical robotics or driving environments.

Answer

Open-loop evaluation replays a logged trajectory and asks the policy to reproduce the expert action at states the expert visited, with ground-truth history fed back in at every step, so each prediction error is scored and then discarded. Closed-loop execution feeds every error into the next observation, so the policy’s own state distribution drifts off the demonstration manifold and errors compound: the classic behavior-cloning result turns a per-step error $\epsilon$ into an excess cost of up to $T^2 \epsilon$ over a $T$ -step episode, a bound that is already vacuous for a 200-step driving episode at $\epsilon = 0.01$ . Two further effects break the correlation before compounding even starts. MSE is mean-seeking: its minimizer is the conditional mean $\mathbb{E}[a \mid s]$ , and averaging the two valid modes of “swerve left or swerve right around the obstacle” produces a straight-line action that is optimal under the metric and a collision in the world. And metric mass is not risk mass: most logged frames are trivial lane-following, so average displacement error is dominated by how well a model extrapolates its own velocity, which is why an MLP fed only ego status with no perception input at all scored competitively on nuScenes open-loop L2 while being worthless as a planner.

(1) Covariate Shift: open loop measures the loss under $d_{\pi^*}$ , the expert’s state distribution, while success is determined under $d_{\pi}$ , the policy’s own induced distribution; the two diverge as soon as the policy acts.
(2) Compounding Error: teacher forcing caps the deviation at one step’s worth of error, whereas rollout integrates it, giving the quadratic-in-horizon gap that a single-step regression number cannot express.
(3) Mode Averaging: a squared-error objective on a multimodal action distribution returns an interpolation of the modes, which is frequently the one infeasible action available.
(4) Long-Tail Mismatch: safety outcomes are decided by a fraction of a percent of frames (cut-ins, occluded pedestrians, contact-rich grasps), and those frames contribute almost nothing to a dataset-averaged MSE.
(5) Non-Reactive Logs: logged agents never yield, brake, or negotiate, so open loop cannot score any behavior whose correctness depends on how the world responds to the ego.
(6) Shortcut Exploitation: ground-truth history leaks the answer; extrapolating the logged ego velocity minimizes displacement error without any scene understanding, and that shortcut vanishes the moment the policy controls its own history.

Two panels: left shows a logged expert trajectory with short prediction-error arrows at each sampled state and the state reset to the log after every step; right shows the same per-step error accumulating into a rollout that curves away from the dashed expert reference into off-distribution states

Figure 1: Open-loop scoring resets the policy onto the logged state after every prediction, so a bounded per-step error stays bounded; closed-loop rollout feeds each error into the next observation, and the visited states leave the training distribution where the policy has no guarantees at all.

The theory is unusually clean here. Ross and Bagnell showed that supervised imitation with per-step loss $\epsilon$ under the expert distribution admits an excess cost that grows as $T^2 \epsilon$ , because a mistake at step $t$ can put the agent in a state where it makes mistakes for all remaining $T - t$ steps; on-policy correction such as DAgger restores the linear $T \epsilon$ rate precisely by collecting labels on $d_{\pi}$ . The practical consequence is that ranking two policies by offline $\epsilon$ tells you almost nothing about their closed-loop ordering, since the multiplier between them differs by a factor of $T$ and depends on recovery behavior that the offline data never contains. Codevilla and colleagues measured this directly on vision-based driving models and found offline prediction error to be a weak predictor of on-road driving quality, and the nuPlan and NAVSIM benchmarks were built specifically because open-loop leaderboard position stopped tracking closed-loop driving score.

Mathematical Formulation:
$\epsilon = \mathbb{E}_{s \sim d_{\pi^*}}\left[\ell(s, \pi(s))\right]$
$J(\pi) - J(\pi^*) \leq T^2 \epsilon$
$\lVert d_{\pi} - d_{\pi^*} \rVert_1 \leq 2 T \epsilon$
$\pi_{\mathrm{mse}}(s) = \mathbb{E}[a \mid s]$
$\pi_{\mathrm{mse}}(s) = 0.5 a_L + 0.5 a_R$

Where:

$\epsilon$ is the offline per-step error that an open-loop MSE actually reports, and $\ell$ is the per-state surrogate loss (squared action error, displacement error).
$\pi$ is the learned policy, $\pi^*$ the expert, and $J$ the closed-loop episode cost, with per-step cost bounded in $[0, 1]$ .
$d_{\pi^*}$ and $d_{\pi}$ are the state distributions induced by the expert and by the policy; open loop samples the first, deployment samples the second.
$T$ is the episode horizon in control steps, the multiplier that an offline metric never sees; the bound becomes vacuous once $T^2 \epsilon \geq T$ .
$a_L$ and $a_R$ are two equally valid expert modes (pass left, pass right) at the same state $s$ ; their MSE-optimal average drives straight into the obstacle.
Required condition for the bounds: the offline data is drawn from $d_{\pi^*}$ with no on-policy correction, which is exactly the assumption behind pure behavior cloning.

Log-scale chart of excess closed-loop cost against episode length T for a fixed per-step error of 0.01: the offline metric is a flat line at 0.01, on-policy correction grows linearly as T times epsilon, and behavior cloning grows quadratically as T squared times epsilon

Figure 2: A single offline number $\epsilon$ is consistent with wildly different closed-loop outcomes, because the horizon $T$ is the multiplier and it is invisible to the metric; on-policy data collection is what changes the exponent from 2 to 1.

Property	Open-loop replay	Closed loop, log-replay agents	Closed loop, reactive agents or hardware
States visited	Expert distribution only	Policy distribution, but in a world frozen to the log	Policy distribution with a world that responds to it
Error feedback	None; state is reset each step	Ego error compounds; other agents do not react	Full two-way feedback including other agents
Typical metric	Action MSE, ADE/FDE, L2 at 1/2/3 s	Route completion, collision rate, comfort sub-scores	Task success rate, interventions per kilometer or per trial
Cost per evaluation	One forward pass per frame, fully parallel, seconds	Sequential rollout, hundreds of scenarios, minutes to hours	Highest; wall-clock hardware time or heavy sim agents
Main blind spot	Recovery, mode collapse to the mean, ego-status shortcuts	False collisions from behind, merges and nudges scored unfairly	Sim-to-real gap in sensing, or low statistical power on real hardware

August 17, 2026

DL0120 VLA Video Pretraining

How do you leverage internet-scale human video datasets (e.g., Ego4D, Something-Something) for pre-training robotic VLA policies? How do you overcome the embodiment/kinematic gap?

Answer

Human video is enormous but action-free: Ego4D contributes roughly 3,670 hours of egocentric recording across 74 locations and Something-Something V2 adds about 220,000 clips over 174 manipulation-style classes, yet none of it carries joint commands, gripper states, or proprioception. The practical recipe is therefore to never try to learn a robot action space directly from video, but to choose an intermediate interface that video can supervise and that a small teleoperated dataset can later decode into commands. Three interfaces dominate: visual representation pretraining (R3M, VIP, VC-1 style encoders trained on Ego4D with time-contrastive or masked objectives), latent action pretraining (a VQ inverse-dynamics model turns each frame pair into a discrete code, the VLA is pretrained to predict those codes from image plus instruction, and only the final decoder head sees real actions), and embodiment-free intermediate predictions such as future 2D/3D point tracks, subgoal images, or hand-contact affordance maps. The embodiment gap is then attacked on four separate axes rather than as one problem: appearance (human arms and hands in frame), kinematics (a roughly 27-DoF hand versus a 1-DoF parallel gripper), viewpoint and calibration (head-mounted camera with unknown scale versus a fixed wrist or shoulder camera), and dynamics (contact forces and speeds a rigid gripper cannot reproduce). In every working system the last mile is still robot data, typically 50 to 200 hours of teleoperation, mixed into training with a weight that keeps human video as a prior rather than a target.

(1) Pick The Interface, Not The Dataset: transferability and actionability trade off directly, so the design decision is which quantity human video supervises, not whether to use it.
(2) Latent Actions As A Shared Vocabulary: a quantized inverse-dynamics model maps $(o_t, o_{t+k})$ to a discrete code that describes observed scene change, which both a hand and a gripper can produce.
(3) Inverse Dynamics Pseudo-Labeling: the VPT approach trains an IDM on a small labeled set and pseudo-labels the rest, which works when the action space is shared and fails badly across embodiments.
(4) Retargeting Where Geometry Allows: a MANO hand fit gives wrist pose plus a scalar aperture, enough to drive a parallel gripper but not a multi-finger in-hand manipulation.
(5) Visual Alignment Beats Visual Luck: masking or inpainting human hands, matching camera placement with head-mounted rigs, and cropping to the same field of view remove most of the appearance gap cheaply.
(6) Co-Training Ratio Is A Hyperparameter: human video enters with weight $\lambda$ alongside robot batches; too high and the policy predicts plausible video instead of executable actions.

Pipeline diagram: human video from Ego4D and Something-Something enters a VQ inverse-dynamics latent action model, the resulting discrete codes pretrain a VLA backbone, and a small robot demonstration set finetunes an action head that outputs joint-space commands

Figure 1: The latent-action route: human video → discrete latent codes → VLA pretraining, with the embodiment gap crossed only in the final stage, where a few tens of hours of teleoperated demonstrations train the head that emits real commands.

Treating the gap as four separate problems makes each one tractable. For appearance, hand segmentation plus inpainting, or observation-space alignment using head-mounted capture on the human side and a matching camera on the robot, is usually enough that the frozen backbone stops keying on skin texture. For kinematics, the honest move is to restrict what human video is asked to teach: it supplies where to go, what to touch, and in what order, while grasp synthesis and force control come from robot data or from a classical controller. For viewpoint and scale, monocular depth or 3D point tracking lifts predictions into a metric frame, and predicting relative motion rather than absolute pose removes the calibration dependency. For dynamics, the temporal stride $k$ matters more than people expect: a stride of a few frames captures human motion an order of magnitude faster than the robot will execute, so codes must be interpreted as subgoals and re-planned, not replayed. Cross-embodiment action tokenization, as used in the Open X-Embodiment and RT-X work over 22 embodiments, is the complementary trick on the robot side: a shared tokenizer plus per-embodiment heads lets one backbone absorb heterogeneous action spaces.

Mathematical Formulation:
$z_t = q\big(E(o_t, o_{t+k})\big)$
$\hat{o}_{t+k} = D(o_t, z_t)$
$\mathcal{L}_{\mathrm{LAM}} = \lVert o_{t+k} - \hat{o}_{t+k} \rVert^2 + \beta \mathcal{L}_{\mathrm{VQ}}$
$\mathcal{L}_{\mathrm{pre}} = -\log \pi_\theta(z_t \mid o_t, \ell)$
$\mathcal{L}_{\mathrm{ft}} = \lVert a_t - g_\phi(h_\theta(o_t, \ell)) \rVert^2$
$\mathcal{L} = \mathcal{L}_{\mathrm{robot}} + \lambda \mathcal{L}_{\mathrm{human}}$

Where:

$o_t$ is the observation (usually a single RGB frame) and $\ell$ the language instruction; $a_t$ is the true robot action, such as an end-effector delta or joint target.
$z_t$ is the discrete latent action for the transition, produced by encoder $E$ and quantizer $q$ against a codebook of size $C$ with $m$ tokens per transition, giving $C^m$ distinct codes.
$D$ is the forward decoder that must reconstruct $o_{t+k}$ from $o_t$ and $z_t$ alone, which is what forces $z_t$ to carry the motion information.
$k > 0$ is the temporal stride in frames and $\beta$ weights the VQ commitment loss; small $k$ captures noise, large $k$ makes the code ambiguous.
$\pi_\theta$ is the VLA backbone during latent pretraining, $h_\theta$ its features after pretraining, and $g_\phi$ the small action head trained only on robot data.
$\lambda$ is the co-training weight on the human-video term; a typical schedule anneals $\lambda$ from near 1 toward 0 so that the deployed policy is dominated by real action supervision.

Qualitative scatter plot of transfer interfaces: joint torques and end-effector poses are directly executable but robot-specific, latent action codes and subgoal images sit in a shaded sweet spot, and frozen visual features are fully embodiment-agnostic but need a full policy on top

Figure 2: The interface ladder. Human video can only supervise quantities that are not tied to a specific body, so the useful designs sit in the middle band, where latent codes, point tracks, and subgoal images are both learnable from video and decodable into commands with modest robot data.

Property	Frozen visual representations	Latent action pretraining	Tracks and subgoal images
What human video supervises	Encoder weights and a temporal value or progress signal	A discrete motion vocabulary shared by hands and grippers	Future 2D or 3D point trajectories, or the next subgoal frame
Robot data still required	A full policy trained on top of the frozen features	A small action head mapping codes to commands, tens of hours	A track-following or goal-reaching low-level policy
Handles hand versus gripper by	Ignoring actions entirely, so the mismatch never appears	Absorbing it in the quantizer: codes describe scene change, not joints	Staying in task space, where points and pixels carry no body
Main failure mode	Task-agnostic features help perception but little on long horizons	Codes drift toward camera motion and background dynamics	Tracker failure under occlusion, and no notion of contact force
Best fit	Small robot datasets and cluttered scenes needing better perception	Generalist VLAs that must follow unseen instructions	Quasi-static rearrangement and tool use with visible motion

August 17, 2026

DL0119 VLA Imitation vs RL

Explain imitation learning vs RL for VLA training. When would you reach for each one on a real robot, for example when post-training a model like Physical Intelligence’s $\pi_0$ ?

Answer

Imitation learning trains a vision-language-action model by maximum likelihood on teleoperated demonstrations: every timestep carries a ground-truth action label, so the objective is the same supervised next-token or flow-matching loss used to pretrain the underlying VLM, and no reward, reset, or simulator is required. Reinforcement learning replaces that per-step label with a scalar return and optimizes the policy on its own state distribution, which is the only way to fix the failure mode imitation cannot fix and to exceed the demonstrator’s performance. The asymmetry is practical rather than philosophical: demonstrations are cheap to collect and trivially parallel across operators, while a single real-robot RL run needs a success detector, an automated reset, and $10^4$ to $10^6$ environment steps of physically risky exploration. That is why essentially every published generalist VLA (RT-2, OpenVLA, $\pi_0$ ) is imitation-pretrained on cross-embodiment data such as Open X-Embodiment, and RL appears only as a narrow post-training stage on a handful of tasks where reward is machine-checkable. The interview answer is therefore not “which is better” but “imitation for coverage and language grounding, RL for the last 10 to 20 points of success rate on tasks you can actually score”.

(1) Different Objectives: imitation maximizes $\log \pi_\theta(a \mid o)$ under a fixed data distribution, while RL maximizes expected discounted return under the policy’s own visitation distribution.
(2) Compounding Error: behavior cloning suffers covariate shift, and its regret against the expert grows as $O(\epsilon T^2)$ in the horizon, whereas interactive or on-policy training reduces this to $O(\epsilon T)$ .
(3) Supervision Density: a demonstration gives a full action vector at every control step; a sparse success reward gives one bit per episode, so RL’s credit assignment problem is orders of magnitude harder.
(4) Performance Ceiling: imitation is capped at the demonstrator and inherits their pauses, jerk, and inconsistent strategies; RL can discover non-human solutions but will hack a badly shaped reward.
(5) Multimodality: human demos are multimodal, so a naive MSE head averages incompatible strategies, which is exactly why modern VLAs use discretized action tokens, diffusion, or flow-matching action experts plus action chunking.
(6) Hybrid Is The Default: production recipes pretrain with imitation, then apply human-in-the-loop corrections (HG-DAgger), offline RL with a conservatism penalty, or advantage-weighted fine-tuning with a KL leash to the cloned policy.

Flow diagram: teleoperated demonstrations feed imitation pretraining of a VLA, the resulting policy produces on-robot rollouts, a success detector or human takeover produces a reward signal, and an RL fine-tuning stage with advantage weighting and a KL penalty updates the policy in a loop

Figure 1: The standard VLA recipe is imitation first, RL second: demonstrations supply dense per-step labels with no reward machinery at all, and the RL loop is only viable once the cloned policy already succeeds often enough for a scalar success signal to be informative.

The sample-cost gap dominates every design decision. Collecting 100 demonstrations of a new task takes a single operator under an hour and immediately yields a policy that succeeds sometimes; reaching the same point with online RL from a random initialization means exploring a 7-DoF continuous action space with sparse reward, which is hopeless on hardware and merely expensive in simulation. Once an imitation-pretrained policy exists, RL becomes tractable because exploration starts near the solution manifold: human-in-the-loop RL methods that keep demonstrations in the replay buffer and let an operator intervene on impending failures report near-perfect success on precise, contact-rich assembly tasks after roughly one to three hours of real-world training. The corresponding risk is catastrophic forgetting: unconstrained RL on one task will happily destroy the language grounding and cross-task generalization that the imitation stage paid for, so practitioners freeze most of the VLM backbone, fine-tune the action expert, and add an explicit KL term back to the behavior-cloned reference policy.

Mathematical Formulation:
$\theta_{BC} = \arg\max_{\theta} \mathbb{E}_{(o,a) \sim \mathcal{D}} \left[\log \pi_{\theta}(a \mid o)\right]$
$J(\pi) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{T} \gamma^{t} r(s_t, a_t)\right]$
$J(\pi^{*}) - J(\pi_{BC}) = O(\epsilon T^{2})$
$J(\pi^{*}) - J(\pi_{DAgger}) = O(\epsilon T)$
$\max_{\theta} \mathbb{E}_{\mathcal{D}}\left[\log \pi_{\theta}(a \mid o) \exp\!\left(A(o,a)/\beta\right)\right]$

Where:

$\pi_{\theta}(a \mid o)$ is the VLA policy mapping an observation $o$ (images, proprioception, language instruction) to an action or action chunk $a$ .
$\mathcal{D}$ is the fixed demonstration dataset, and $\pi^{*}$ is the human expert whose behavior it samples.
$s_t$ and $t \in \{0, \ldots, T\}$ are the environment state and control step, $r$ the reward, and $\gamma \in (0,1]$ the discount.
$\epsilon$ is the per-step supervised action error under the expert’s distribution; the quadratic term comes from errors pushing the policy into states absent from $\mathcal{D}$ .
$A(o,a)$ is the estimated advantage and $\beta > 0$ the temperature; as $\beta \to \infty$ the advantage-weighted objective degenerates to plain behavior cloning, which is why the hybrid is a continuum rather than a binary choice.
Required initial condition for the RL stage: $\pi_{\theta}$ is initialized at $\theta_{BC}$ and the task admits an automated reset plus a reward oracle, otherwise $J$ is not estimable on hardware.

Line chart of regret against the expert versus episode horizon for a per-step error of 0.02, comparing a quadratic behavior-cloning bound that reaches 200 at horizon 100 with a linear interactive-training bound that reaches only 2

Figure 2: With a fixed per-step error of $\epsilon = 0.02$ , the quadratic behavior-cloning bound separates from the linear on-policy bound by a factor of $T$ , which is why long-horizon manipulation degrades far faster than single-step prediction accuracy suggests.

Property	Imitation learning (BC)	Offline RL	Online RL
Supervision per episode	One action label at every control step	Logged actions plus reward or success labels	Scalar reward, often a single success bit
What it needs to run	Teleoperation rig only; no reset, reward, or simulator	A labeled buffer of mixed-quality data	Automated reset, success detector, safety envelope
Typical budget for one task	50 to 2,000 demonstrations, hours of human time	Reuses existing logs; compute-bound, not robot-bound	$10^4$ to $10^6$ real steps, or a sim-to-real gap to close
Distribution optimized on	Expert states only, so covariate shift is unaddressed	Buffer states, with pessimism outside their support	The policy’s own states, which is the point
Performance ceiling	The demonstrator, minus compounding error	Best behavior recoverable from the logged data	Can exceed the human on speed and precision
Main failure mode	Drift into unseen states; mode averaging on multimodal demos	Value overestimation on out-of-support actions	Reward hacking, hardware damage, forgetting language grounding

August 17, 2026

DL0118 VLA vs VLM

What is a VLA model and how does it differ from a VLM?

Answer

A VLA (Vision-Language-Action) model is a policy that maps camera images plus a natural-language instruction directly to robot actions, typically a short sequence of end-effector or joint deltas plus a gripper command, emitted at a fixed control rate. A VLM (Vision-Language Model) maps the same image-plus-text input to text tokens. Architecturally the two are close relatives: almost every modern VLA (RT-2, OpenVLA, $\pi_0$ ) starts from a pretrained VLM backbone and is fine-tuned on teleoperated demonstration trajectories, either by discretizing each action dimension into vocabulary tokens or by attaching a continuous action expert head. The real difference is not the encoder, it is everything downstream of it: the output lives in a continuous, embodiment-specific action space, the model runs inside a closed feedback loop where its own outputs change the next observation, and errors therefore compound over the rollout instead of being independent per query. That single fact drives the different data (robot demos, not web image-text pairs), the different latency budget (tens of milliseconds, not seconds), and the different metric (physical task success rate, not benchmark accuracy).

(1) Output Space: a VLM produces a distribution over a discrete text vocabulary; a VLA produces a vector in $\mathbb{R}^{d}$ per timestep, usually predicted as an action chunk covering the next $H$ control steps.
(2) Closed Loop Versus Open Loop: a VLA’s action changes the world and therefore its own next input, so covariate shift makes behavior-cloning error grow roughly with the square of the horizon; a VLM answer is scored once and never fed back through a robot.
(3) Training Data: web-scale image-caption and VQA corpora for the VLM, versus expensive teleoperated trajectories (Open X-Embodiment aggregates roughly one million episodes) with synchronized proprioception for the VLA.
(4) Action Representation Is A Design Choice: RT-2 and OpenVLA quantize each dimension into 256 bins and reuse rarely-used text tokens; $\pi_0$ instead trains a flow-matching action expert that emits continuous chunks at 50 Hz.
(5) Latency Is A Correctness Constraint: a 2 s VLM response is fine, but a controller starved of fresh actions produces jerky or unsafe motion, so chunk horizon and inference time must be budgeted together.
(6) Embodiment Coupling: VLM weights transfer across any image; VLA action heads are tied to a specific DoF count, camera mount, and control convention, which is why cross-embodiment training is an active research problem.

Diagram showing a shared ViT plus LLM backbone taking an RGB observation and a language instruction, branching into a text decoder head producing answer tokens for a VLM and an action expert head producing a 50 by 7 action chunk executed by a robot at 50 Hz, with a feedback arrow returning the new observation to the encoder

Figure 1: Both models share the same perception stack; the VLA replaces or augments the text head with an action head and runs inside the loop observe → predict chunk → execute → observe, so its own predictions determine the next input distribution.

The token-budget arithmetic explains why action chunking is universal. A 7-DoF arm predicted one step at a time at 50 Hz would need a full autoregressive forward pass every 20 ms, which no 3B-parameter backbone can sustain. Predicting a chunk of $H = 50$ actions amortizes one forward pass across a full second of motion, at the price of running open loop within the chunk. Shorter chunks mean tighter feedback and better disturbance rejection but more compute and more jitter at chunk boundaries; longer chunks are smoother but blind to anything that happens mid-chunk. Naive autoregressive decoding of 350 discrete action tokens is also slow, which is exactly the bottleneck that continuous action experts and frequency-domain tokenizers were built to remove.

Mathematical Formulation:
$p_\theta(y_{1:T} \mid I, \ell) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{1:t-1}, I, \ell)$
$a_{t:t+H-1} \sim \pi_\theta(\cdot \mid o_t, s_t, \ell)$
$k_j = \mathrm{round}\left(\frac{a_j - a_{\min}}{a_{\max} - a_{\min}}(B-1)\right)$
$N_{tok} = H \cdot d = 50 \cdot 7 = 350$
$t_{infer} \leq H / f_{ctrl}$
$50 / 50\ \text{Hz} = 1\ \text{s}$
$J(\pi_\theta) - J(\pi^{*}) = O(\epsilon T^{2})$

Where:

$y_{1:T}$ are the text tokens a VLM emits for image $I$ and instruction $\ell$ ; the factorization is the only thing the VLA keeps unchanged.
$a_{t:t+H-1} \in \mathbb{R}^{H \times d}$ is the action chunk, $o_t$ the current camera observation, and $s_t$ the proprioceptive state that a VLM never receives.
$d$ is the action dimension (7 for a 6-DoF pose delta plus gripper) and $H$ the chunk horizon in control steps.
$k_j \in \{0, \ldots, B-1\}$ is the discrete bin for dimension $j$ , with $B = 256$ and $[a_{\min}, a_{\max}]$ set from per-dimension training quantiles so outliers do not collapse the resolution.
$f_{ctrl}$ is the control frequency and $t_{infer}$ the policy latency; the required deployment condition is that a new chunk arrives before the previous one is exhausted.
$\epsilon$ is the per-step imitation error, $T$ the rollout length, and $J$ the task cost; the quadratic bound is the classical behavior-cloning compounding result.

Log-scale chart of accumulated error against rollout length, comparing a linear curve for independent per-query error and a quadratic curve for closed-loop behavior cloning, with the quadratic curve 400 times higher at 400 steps

Figure 2: A VLM’s mistakes are independent per query and accumulate linearly; a VLA’s mistakes move the robot off the demonstration distribution, so the worst-case cost grows as $O(\epsilon T^{2})$ and a 1% per-step error is fatal over a 400-step manipulation.

Property	VLA (Vision-Language-Action)	VLM (Vision-Language Model)
Output	Continuous action chunk, typically 50 steps by 7 dimensions, as bin tokens or a flow-matching head	Text tokens from a fixed vocabulary of roughly 32k to 256k entries
Extra inputs	Proprioception, gripper state, often multiple synchronized camera views	Images and text only
Training data	Teleoperated demonstrations (Open X-Embodiment scale is about 1M episodes), usually co-trained with web data to keep semantics	Billions of web image-text pairs plus instruction tuning
Latency budget	Chunk must arrive before the previous one runs out, so tens to a few hundred milliseconds	Seconds; streaming hides most of it from the user
Evaluation	Physical or simulated rollout success rate over many trials, with high variance and slow iteration	Static benchmark accuracy or preference scores, reproducible offline
Dominant failure mode	Compounding covariate shift, unrecoverable states, embodiment mismatch, control jitter	Hallucination and grounding errors, recoverable by re-prompting

August 17, 2026