DL0120 VLA Video Pretraining

Human video is enormous but action-free: Ego4D contributes roughly 3,670 hours of egocentric recording across 74 locations and Something-Something V2 adds about 220,000 clips over 174 manipulation-style classes, yet none of it carries joint commands, gripper states, or proprioception. The practical recipe is therefore to never try to learn a robot action space directly from video, but to choose an intermediate interface that video can supervise and that a small teleoperated dataset can later decode into commands. Three interfaces dominate: visual representation pretraining (R3M, VIP, VC-1 style encoders trained on Ego4D with time-contrastive or masked objectives), latent action pretraining (a VQ inverse-dynamics model turns each frame pair into a discrete code, the VLA is pretrained to predict those codes from image plus instruction, and only the final decoder head sees real actions), and embodiment-free intermediate predictions such as future 2D/3D point tracks, subgoal images, or hand-contact affordance maps. The embodiment gap is then attacked on four separate axes rather than as one problem: appearance (human arms and hands in frame), kinematics (a roughly 27-DoF hand versus a 1-DoF parallel gripper), viewpoint and calibration (head-mounted camera with unknown scale versus a fixed wrist or shoulder camera), and dynamics (contact forces and speeds a rigid gripper cannot reproduce). In every working system the last mile is still robot data, typically 50 to 200 hours of teleoperation, mixed into training with a weight that keeps human video as a prior rather than a target.

(1) Pick The Interface, Not The Dataset: transferability and actionability trade off directly, so the design decision is which quantity human video supervises, not whether to use it.
(2) Latent Actions As A Shared Vocabulary: a quantized inverse-dynamics model maps to a discrete code that describes observed scene change, which both a hand and a gripper can produce.
(3) Inverse Dynamics Pseudo-Labeling: the VPT approach trains an IDM on a small labeled set and pseudo-labels the rest, which works when the action space is shared and fails badly across embodiments.
(4) Retargeting Where Geometry Allows: a MANO hand fit gives wrist pose plus a scalar aperture, enough to drive a parallel gripper but not a multi-finger in-hand manipulation.
(5) Visual Alignment Beats Visual Luck: masking or inpainting human hands, matching camera placement with head-mounted rigs, and cropping to the same field of view remove most of the appearance gap cheaply.
(6) Co-Training Ratio Is A Hyperparameter: human video enters with weight alongside robot batches; too high and the policy predicts plausible video instead of executable actions.

Figure 1: The latent-action route: human video → discrete latent codes → VLA pretraining, with the embodiment gap crossed only in the final stage, where a few tens of hours of teleoperated demonstrations train the head that emits real commands.

Treating the gap as four separate problems makes each one tractable. For appearance, hand segmentation plus inpainting, or observation-space alignment using head-mounted capture on the human side and a matching camera on the robot, is usually enough that the frozen backbone stops keying on skin texture. For kinematics, the honest move is to restrict what human video is asked to teach: it supplies where to go, what to touch, and in what order, while grasp synthesis and force control come from robot data or from a classical controller. For viewpoint and scale, monocular depth or 3D point tracking lifts predictions into a metric frame, and predicting relative motion rather than absolute pose removes the calibration dependency. For dynamics, the temporal stride matters more than people expect: a stride of a few frames captures human motion an order of magnitude faster than the robot will execute, so codes must be interpreted as subgoals and re-planned, not replayed. Cross-embodiment action tokenization, as used in the Open X-Embodiment and RT-X work over 22 embodiments, is the complementary trick on the robot side: a shared tokenizer plus per-embodiment heads lets one backbone absorb heterogeneous action spaces.

Mathematical Formulation:

Where:

• is the observation (usually a single RGB frame) and the language instruction; is the true robot action, such as an end-effector delta or joint target.
• is the discrete latent action for the transition, produced by encoder and quantizer against a codebook of size with tokens per transition, giving distinct codes.
• is the forward decoder that must reconstruct from and alone, which is what forces to carry the motion information.
• is the temporal stride in frames and weights the VQ commitment loss; small captures noise, large makes the code ambiguous.
• is the VLA backbone during latent pretraining, its features after pretraining, and the small action head trained only on robot data.
• is the co-training weight on the human-video term; a typical schedule anneals from near 1 toward 0 so that the deployed policy is dominated by real action supervision.

Figure 2: The interface ladder. Human video can only supervise quantities that are not tied to a specific body, so the useful designs sit in the middle band, where latent codes, point tracks, and subgoal images are both learnable from video and decodable into commands with modest robot data.