DL0119 VLA Imitation vs RL

Imitation learning trains a vision-language-action model by maximum likelihood on teleoperated demonstrations: every timestep carries a ground-truth action label, so the objective is the same supervised next-token or flow-matching loss used to pretrain the underlying VLM, and no reward, reset, or simulator is required. Reinforcement learning replaces that per-step label with a scalar return and optimizes the policy on its own state distribution, which is the only way to fix the failure mode imitation cannot fix and to exceed the demonstrator’s performance. The asymmetry is practical rather than philosophical: demonstrations are cheap to collect and trivially parallel across operators, while a single real-robot RL run needs a success detector, an automated reset, and to environment steps of physically risky exploration. That is why essentially every published generalist VLA (RT-2, OpenVLA, ) is imitation-pretrained on cross-embodiment data such as Open X-Embodiment, and RL appears only as a narrow post-training stage on a handful of tasks where reward is machine-checkable. The interview answer is therefore not “which is better” but “imitation for coverage and language grounding, RL for the last 10 to 20 points of success rate on tasks you can actually score”.

(1) Different Objectives: imitation maximizes under a fixed data distribution, while RL maximizes expected discounted return under the policy’s own visitation distribution.
(2) Compounding Error: behavior cloning suffers covariate shift, and its regret against the expert grows as in the horizon, whereas interactive or on-policy training reduces this to .
(3) Supervision Density: a demonstration gives a full action vector at every control step; a sparse success reward gives one bit per episode, so RL’s credit assignment problem is orders of magnitude harder.
(4) Performance Ceiling: imitation is capped at the demonstrator and inherits their pauses, jerk, and inconsistent strategies; RL can discover non-human solutions but will hack a badly shaped reward.
(5) Multimodality: human demos are multimodal, so a naive MSE head averages incompatible strategies, which is exactly why modern VLAs use discretized action tokens, diffusion, or flow-matching action experts plus action chunking.
(6) Hybrid Is The Default: production recipes pretrain with imitation, then apply human-in-the-loop corrections (HG-DAgger), offline RL with a conservatism penalty, or advantage-weighted fine-tuning with a KL leash to the cloned policy.

Figure 1: The standard VLA recipe is imitation first, RL second: demonstrations supply dense per-step labels with no reward machinery at all, and the RL loop is only viable once the cloned policy already succeeds often enough for a scalar success signal to be informative.

The sample-cost gap dominates every design decision. Collecting 100 demonstrations of a new task takes a single operator under an hour and immediately yields a policy that succeeds sometimes; reaching the same point with online RL from a random initialization means exploring a 7-DoF continuous action space with sparse reward, which is hopeless on hardware and merely expensive in simulation. Once an imitation-pretrained policy exists, RL becomes tractable because exploration starts near the solution manifold: human-in-the-loop RL methods that keep demonstrations in the replay buffer and let an operator intervene on impending failures report near-perfect success on precise, contact-rich assembly tasks after roughly one to three hours of real-world training. The corresponding risk is catastrophic forgetting: unconstrained RL on one task will happily destroy the language grounding and cross-task generalization that the imitation stage paid for, so practitioners freeze most of the VLM backbone, fine-tune the action expert, and add an explicit KL term back to the behavior-cloned reference policy.

Mathematical Formulation:

Where:

• is the VLA policy mapping an observation (images, proprioception, language instruction) to an action or action chunk .
• is the fixed demonstration dataset, and is the human expert whose behavior it samples.
• and are the environment state and control step, the reward, and the discount.
• is the per-step supervised action error under the expert’s distribution; the quadratic term comes from errors pushing the policy into states absent from .
• is the estimated advantage and the temperature; as the advantage-weighted objective degenerates to plain behavior cloning, which is why the hybrid is a continuum rather than a binary choice.
• Required initial condition for the RL stage: is initialized at and the task admits an automated reset plus a reward oracle, otherwise is not estimable on hardware.

Figure 2: With a fixed per-step error of , the quadratic behavior-cloning bound separates from the linear on-policy bound by a factor of , which is why long-horizon manipulation degrades far faster than single-step prediction accuracy suggests.

Property	Imitation learning (BC)	Offline RL	Online RL
Supervision per episode	One action label at every control step	Logged actions plus reward or success labels	Scalar reward, often a single success bit
What it needs to run	Teleoperation rig only; no reset, reward, or simulator	A labeled buffer of mixed-quality data	Automated reset, success detector, safety envelope
Typical budget for one task	50 to 2,000 demonstrations, hours of human time	Reuses existing logs; compute-bound, not robot-bound	to real steps, or a sim-to-real gap to close
Distribution optimized on	Expert states only, so covariate shift is unaddressed	Buffer states, with pessimism outside their support	The policy’s own states, which is the point
Performance ceiling	The demonstrator, minus compounding error	Best behavior recoverable from the logged data	Can exceed the human on speed and precision
Main failure mode	Drift into unseen states; mode averaging on multimodal demos	Value overestimation on out-of-support actions	Reward hacking, hardware damage, forgetting language grounding