DL0157 VLA Zero-Shot and Few-Shot Transfer

Evaluating transfer is a measurement design problem, not a demo reel. The protocol must vary one generalization axis per evaluation cell, hold everything else fixed with a scripted scene reset and matched initial states, and report a per-axis success rate with a confidence interval instead of one aggregate number. Zero-shot cells run the frozen checkpoint at demonstrations, while few-shot cells sweep target demonstrations and report a curve plus the retention on the original seen-task suite after adaptation. The trial budget sets the resolution of the entire experiment, because 20 trials per cell give a 95% interval of roughly points at , which cannot resolve the 10 to 15 point differences people actually argue about. Cheap simulation suites such as LIBERO and SIMPLER buy statistical power and reproducibility, and real-robot paired trials buy validity, so a credible report uses both and checks that they rank policies the same way.

(1) Factor-Isolated Axes: Split “novel” into unseen object instance, unseen object category, unseen scene and background, added distractors and camera pose, and rephrased language. A single mixed “hard eval” set cannot attribute a failure to any one of them.
(2) Matched Initial States: Every policy under comparison sees the same object poses, lighting, and clutter, logged from a reset script or a fixture template, which turns a noisy independent comparison into a paired test with far more power.
(3) Trial Budget Sets Resolution: Detecting a 10-point difference at 95% confidence needs roughly 190 trials per arm, so 20-trial evals should be reported as pilots, never as rankings.
(4) Progress Score, Not Only Binary Success: Score reach, grasp, transport, and place as staged partial credit in , which exposes whether a novel object breaks perception or grasp geometry.
(5) Few-Shot Is A Curve: Report over several values and fit a demo-efficiency constant, because a single number hides whether the policy needed 2 demos or 50.
(6) Retention And Blind Operation: Re-run the seen-task suite after fine-tuning to measure catastrophic forgetting, and keep the operator unaware of which checkpoint is running to remove reset and intervention bias.

Figure 1: One checkpoint, one factor per cell. The harness is the part people skip: without scripted resets and matched initial states, the operator becomes an uncontrolled variable, and the same policy can swing 20 points between two afternoons. Each cell also carries a seen-object control in the new scene, so a gap can be attributed to the object rather than to the room.

The protocol hygiene matters more than the model comparison. Each trial follows a fixed loop, reset → verify pose against a reference photo → run for a hard time limit → score stages → log the episode, and every episode is kept, including ones the operator considers unfair, because discarding “bad setups” is how a 55% policy becomes an 80% policy on paper. Interventions must be recorded rather than silently corrected, since a policy that needs one nudge per episode is not transferring. On the few-shot side, the demonstrations for a novel object must be collected under the same distribution the evaluation will use, and the evaluation objects must be held out from those demonstrations; reusing the same physical mug for teaching and testing measures memorization, not transfer. Finally, adaptation method is part of the result: LoRA on the action head, full fine-tuning, and fine-tuning with a replay mix of pretraining data produce very different retention curves at the same novel-task success.

Mathematical Formulation:

Where:

• is the estimated success rate of one evaluation cell and the outcome of trial , either binary or a staged progress score in .
• is the number of trials in that cell, the single quantity that decides whether the reported number can support a claim.
• is the generalization gap on axis , measured against the seen-task control run in the same session and the same scene.
• at 95% confidence, is the pooled rate of the two arms, and is the minimum detectable difference you are willing to claim.
• is the number of target-domain demonstrations, the zero-shot rate, the saturation rate, and the demo-efficiency constant in demonstrations, so smaller means faster adaptation.
• The unpaired formula for is an upper bound; matched initial states reduce the variance of the difference and can cut the required by a factor of two or more.

Trial Budget For A 10-Point Claim:

At roughly 90 seconds per real trial including reset, 192 trials per arm is about 5 hours of operator time for a single cell, and a five-axis grid with two policies multiplies that by ten. That arithmetic, not modelling taste, is why teams push generalization sweeps into simulation and reserve real hardware for a small number of decisive paired comparisons, and why crowd-sourced or autonomous evaluation has become an active research direction.

Figure 2: Left, the same measurements at two trial budgets. With 20 trials per cell the intervals for novel instance, distractors, and rephrasing all overlap, so the axis ranking is unsupported; at 200 trials the ordering becomes real. Right, few-shot transfer is a curve with a second axis nobody reports: novel-task success climbs with while seen-task retention decays under naive fine-tuning.