DL0155 Crowdsourced Teleoperation Quality

Crowdsourced teleoperation data is a mixture over operators, rigs, and sessions, not a single distribution. DROID’s 76k trajectories were collected by about 50 operators across 13 institutions, and Open X-Embodiment pools 60 datasets over 22 embodiments, so per-episode quality varies more than the task labels do. The working assumption is that quality is a measured per-episode variable, not a binary success flag, so the first engineering step is a scoring pass that attaches provenance plus cheap kinematic proxies to every episode and stores them as metadata rather than deleting anything. Training then spends that metadata at three separate points: group weights over operator or site during pretraining, a quality token in the conditioning so the model learns the difference between clean and sloppy behaviour, and a short anneal on a curated gold subset that decides what the policy actually imitates at test time. Hard filtering is the last resort, because imitation error compounds as and most of that comes from states the policy has never visited, which is exactly what the messy tail supplies. Two mechanical fixes matter as much as the curation: percentile-based action normalization so one jerky operator cannot rescale the action space, and an expressive action head over action chunks so contradictory strategies are not averaged into a mean action.

(1) Quality Is A Variable, Not A Flag: score every episode continuously and keep the score, because a policy trained on a thresholded boolean cannot be re-mixed later without a second scoring pass.
(2) Cheap Automatic Proxies: action jerk, path-length ratio, idle-time fraction, regrasp and retry count, teleop latency spikes, and VLM-based verification of the success label and the language annotation.
(3) Provenance Is The Grouping Unit: operator, site, and rig ids define the groups used for reweighting and, critically, for held-out evaluation splits so a strong operator’s style does not leak across train and test.
(4) Coverage Versus Purity: the sloppy tail carries the recovery states and rare scenes that keep compounding error small, so aggressive filtering can lower success even while raising average demo quality.
(5) Pretrain Broad, Post-Train Narrow: the data-pyramid recipe used by recent VLA foundation models trains on nearly everything and then anneals on a small, verified, high-quality set.
(6) Quality-Conditioned Cloning: feed the tier as a token during training and sample at the gold setting at inference, which uses bad data as negative evidence instead of throwing it away.
(7) Robustness Plumbing: 1st/99th percentile action normalization as in OpenVLA, plus flow-matching or diffusion heads, so outliers and multimodality do not corrupt the regression target.

Figure 1: The topology is what matters: scoring is a metadata pass, not a delete pass, all three tiers reach the pretraining stage, and only the last stage is restricted to gold. The closing arrow is the part teams forget, since per-operator scorecards turn quality measurement into targeted re-collection rather than a one-time filter.

The scoring pass should stay cheap enough to run over every episode. Kinematic proxies are computed directly from the recorded action stream: mean squared jerk over the trajectory, the ratio of executed end-effector path length to the straight-line distance, the fraction of timesteps with near-zero commanded velocity, and the number of gripper open/close reversals as a regrasp counter. Semantic proxies need a model: a VLM watches the final frames and votes on whether the stated goal was reached, and a second pass checks that the language annotation matches the video, which catches the common crowdsourcing failure where the instruction and the demonstration disagree. These proxies are then aggregated into one score per episode and, separately, per operator. Reweighting uses the group statistic because per-episode weights are noisy and because the real covariate shift is at the operator and rig level, where calibration offsets, camera mounts, and latency are shared.

Mathematical Formulation:

Where:

• is the quality score of episode , with the proxy feature vector (jerk, path ratio, idle fraction, regrasps, VLM verdict) and fitted on a few hundred human-labelled episodes.
• is the loss on group , where a group is one operator, site, or source dataset, and its observation-action pairs with observation (images plus instruction plus proprioception) and action chunk .
• indexes groups, is the simplex of mixture weights, and the inner maximization is group DRO, which upweights whichever group is currently worst-fit instead of trusting a hand-tuned mixture.
• is the policy, the per-sample cloning loss, the rollout horizon, and the per-step error under the expert’s state distribution; the factor is why coverage of off-nominal states is worth more than average demo neatness.
• and are the 1st and 99th percentiles of each action dimension over the training mixture, so is a quantile-normalized target that a single spiking operator cannot stretch.

The trade-off curve is the part that surprises people. Sorting episodes by score and keeping only the top fraction improves the average target the policy regresses onto, but it also removes scenes, lighting conditions, object instances, and above all recovery segments: the moment where an operator overshot, re-approached, and succeeded is precisely the state a deployed policy will find itself in. Empirically the filter-only curve is an inverted U, peaking somewhere around half the data and then falling below the train-on-everything baseline once coverage collapses. Reweighting dominates filtering because it keeps every state in the support while shrinking the gradient contribution of noisy actions, and the gold anneal captures most of the remaining gain at almost no cost since it touches only the final few percent of optimization steps.

Figure 2: Two different shapes from the same scores. Hard filtering trades coverage for purity and therefore has an interior optimum you must search for, while reweighting plus a gold anneal keeps the full state support and improves monotonically as more data is retained. The gap at the right edge is the cost of deleting data you could have downweighted instead.

Quality variance also shows up as multimodality, which is a modelling problem rather than a data problem. When two operators route around the same obstacle in opposite directions, the conditional action distribution given the observation is bimodal, and any head trained with a plain mean-squared-error objective converges to , the average of the two modes. That average is often an invalid action, so the policy fails on the exact scenes where it had the most data. The fixes are an expressive head (diffusion or flow matching over an action chunk, as in the pi-zero family) which samples a single coherent mode, action chunking so a commitment persists for several timesteps instead of flip-flopping per step, and conditioning on style or operator id when the modes are genuinely operator-specific. This is also why “smooth” should never be the only quality criterion: a smooth demonstration that solves the task differently from the rest of the corpus still injects a competing mode.

Figure 3: Mixed-operator data is multimodal in action space. A unimodal regression head collapses two valid strategies onto their mean, which here is a straight line through the obstacle, while a sampling head over action chunks commits to one mode per rollout. No amount of filtering fixes this if both strategies are high quality.