DL0121 Open-Loop vs Closed-Loop Validation

Open-loop evaluation replays a logged trajectory and asks the policy to reproduce the expert action at states the expert visited, with ground-truth history fed back in at every step, so each prediction error is scored and then discarded. Closed-loop execution feeds every error into the next observation, so the policy’s own state distribution drifts off the demonstration manifold and errors compound: the classic behavior-cloning result turns a per-step error into an excess cost of up to over a -step episode, a bound that is already vacuous for a 200-step driving episode at . Two further effects break the correlation before compounding even starts. MSE is mean-seeking: its minimizer is the conditional mean , and averaging the two valid modes of “swerve left or swerve right around the obstacle” produces a straight-line action that is optimal under the metric and a collision in the world. And metric mass is not risk mass: most logged frames are trivial lane-following, so average displacement error is dominated by how well a model extrapolates its own velocity, which is why an MLP fed only ego status with no perception input at all scored competitively on nuScenes open-loop L2 while being worthless as a planner.

(1) Covariate Shift: open loop measures the loss under , the expert’s state distribution, while success is determined under , the policy’s own induced distribution; the two diverge as soon as the policy acts.
(2) Compounding Error: teacher forcing caps the deviation at one step’s worth of error, whereas rollout integrates it, giving the quadratic-in-horizon gap that a single-step regression number cannot express.
(3) Mode Averaging: a squared-error objective on a multimodal action distribution returns an interpolation of the modes, which is frequently the one infeasible action available.
(4) Long-Tail Mismatch: safety outcomes are decided by a fraction of a percent of frames (cut-ins, occluded pedestrians, contact-rich grasps), and those frames contribute almost nothing to a dataset-averaged MSE.
(5) Non-Reactive Logs: logged agents never yield, brake, or negotiate, so open loop cannot score any behavior whose correctness depends on how the world responds to the ego.
(6) Shortcut Exploitation: ground-truth history leaks the answer; extrapolating the logged ego velocity minimizes displacement error without any scene understanding, and that shortcut vanishes the moment the policy controls its own history.

Figure 1: Open-loop scoring resets the policy onto the logged state after every prediction, so a bounded per-step error stays bounded; closed-loop rollout feeds each error into the next observation, and the visited states leave the training distribution where the policy has no guarantees at all.

The theory is unusually clean here. Ross and Bagnell showed that supervised imitation with per-step loss under the expert distribution admits an excess cost that grows as , because a mistake at step can put the agent in a state where it makes mistakes for all remaining steps; on-policy correction such as DAgger restores the linear rate precisely by collecting labels on . The practical consequence is that ranking two policies by offline tells you almost nothing about their closed-loop ordering, since the multiplier between them differs by a factor of and depends on recovery behavior that the offline data never contains. Codevilla and colleagues measured this directly on vision-based driving models and found offline prediction error to be a weak predictor of on-road driving quality, and the nuPlan and NAVSIM benchmarks were built specifically because open-loop leaderboard position stopped tracking closed-loop driving score.

Mathematical Formulation:

Where:

• is the offline per-step error that an open-loop MSE actually reports, and is the per-state surrogate loss (squared action error, displacement error).
• is the learned policy, the expert, and the closed-loop episode cost, with per-step cost bounded in .
• and are the state distributions induced by the expert and by the policy; open loop samples the first, deployment samples the second.
• is the episode horizon in control steps, the multiplier that an offline metric never sees; the bound becomes vacuous once .
• and are two equally valid expert modes (pass left, pass right) at the same state ; their MSE-optimal average drives straight into the obstacle.
• Required condition for the bounds: the offline data is drawn from with no on-policy correction, which is exactly the assumption behind pure behavior cloning.

Figure 2: A single offline number is consistent with wildly different closed-loop outcomes, because the horizon is the multiplier and it is invisible to the metric; on-policy data collection is what changes the exponent from 2 to 1.