DL0118 VLA vs VLM

A VLA (Vision-Language-Action) model is a policy that maps camera images plus a natural-language instruction directly to robot actions, typically a short sequence of end-effector or joint deltas plus a gripper command, emitted at a fixed control rate. A VLM (Vision-Language Model) maps the same image-plus-text input to text tokens. Architecturally the two are close relatives: almost every modern VLA (RT-2, OpenVLA, ) starts from a pretrained VLM backbone and is fine-tuned on teleoperated demonstration trajectories, either by discretizing each action dimension into vocabulary tokens or by attaching a continuous action expert head. The real difference is not the encoder, it is everything downstream of it: the output lives in a continuous, embodiment-specific action space, the model runs inside a closed feedback loop where its own outputs change the next observation, and errors therefore compound over the rollout instead of being independent per query. That single fact drives the different data (robot demos, not web image-text pairs), the different latency budget (tens of milliseconds, not seconds), and the different metric (physical task success rate, not benchmark accuracy).

(1) Output Space: a VLM produces a distribution over a discrete text vocabulary; a VLA produces a vector in per timestep, usually predicted as an action chunk covering the next control steps.
(2) Closed Loop Versus Open Loop: a VLA’s action changes the world and therefore its own next input, so covariate shift makes behavior-cloning error grow roughly with the square of the horizon; a VLM answer is scored once and never fed back through a robot.
(3) Training Data: web-scale image-caption and VQA corpora for the VLM, versus expensive teleoperated trajectories (Open X-Embodiment aggregates roughly one million episodes) with synchronized proprioception for the VLA.
(4) Action Representation Is A Design Choice: RT-2 and OpenVLA quantize each dimension into 256 bins and reuse rarely-used text tokens; instead trains a flow-matching action expert that emits continuous chunks at 50 Hz.
(5) Latency Is A Correctness Constraint: a 2 s VLM response is fine, but a controller starved of fresh actions produces jerky or unsafe motion, so chunk horizon and inference time must be budgeted together.
(6) Embodiment Coupling: VLM weights transfer across any image; VLA action heads are tied to a specific DoF count, camera mount, and control convention, which is why cross-embodiment training is an active research problem.

Figure 1: Both models share the same perception stack; the VLA replaces or augments the text head with an action head and runs inside the loop observe → predict chunk → execute → observe, so its own predictions determine the next input distribution.

The token-budget arithmetic explains why action chunking is universal. A 7-DoF arm predicted one step at a time at 50 Hz would need a full autoregressive forward pass every 20 ms, which no 3B-parameter backbone can sustain. Predicting a chunk of actions amortizes one forward pass across a full second of motion, at the price of running open loop within the chunk. Shorter chunks mean tighter feedback and better disturbance rejection but more compute and more jitter at chunk boundaries; longer chunks are smoother but blind to anything that happens mid-chunk. Naive autoregressive decoding of 350 discrete action tokens is also slow, which is exactly the bottleneck that continuous action experts and frequency-domain tokenizers were built to remove.

Mathematical Formulation:

Where:

• are the text tokens a VLM emits for image and instruction ; the factorization is the only thing the VLA keeps unchanged.
• is the action chunk, the current camera observation, and the proprioceptive state that a VLM never receives.
• is the action dimension (7 for a 6-DoF pose delta plus gripper) and the chunk horizon in control steps.
• is the discrete bin for dimension , with and set from per-dimension training quantiles so outliers do not collapse the resolution.
• is the control frequency and the policy latency; the required deployment condition is that a new chunk arrives before the previous one is exhausted.
• is the per-step imitation error, the rollout length, and the task cost; the quadratic bound is the classical behavior-cloning compounding result.

Figure 2: A VLM’s mistakes are independent per query and accumulate linearly; a VLA’s mistakes move the robot off the demonstration distribution, so the worst-case cost grows as and a 1% per-step error is fatal over a 400-step manipulation.