DL0159 Counterfactual Simulation for Driving

Counterfactual simulation re-runs one specific logged drive with exactly one thing changed, usually the ego vehicle’s own action sequence, and then asks what the rest of the world would have done in response. It is Pearl’s three-step recipe applied to a driving log: abduction → action → prediction. First infer the latent variables that explain the observed log, including road geometry, occlusions, and each neighbour’s intent and aggressiveness. Then intervene by substituting a new ego plan , such as brake onset two seconds earlier, and roll the scene forward under that intervention. The hard part is not the ego kinematics, which a closed-form formula settles in one line; the hard part is that every other agent’s logged trajectory becomes invalid the instant the ego behaves differently, so a naive non-reactive log replay manufactures collisions that could never have happened. A world model supplies the missing conditional distribution , generating plausible reactions for surrounding traffic either as trajectories over a structured scene or as generated sensor frames, and because those reactions are stochastic the answer is a distribution over outcomes estimated from many sampled rollouts rather than a single verdict.

(1) Counterfactual, Not Merely Interventional: averages over all scenes that could occur, while the counterfactual conditions on the evidence of this exact log, so the same pedestrian keeps the same intent.
(2) Abduction Fixes The World: inferring from the log is what makes the rollout a rerun of that day rather than a generic scenario with similar statistics.
(3) Reactivity Is The Whole Problem: logged responses are only valid for the logged ego action, so frozen playback agents produce fake rear-end contacts and hide genuine near misses.
(4) Divergence Horizon: simulated and logged states separate as the intervention propagates, so a counterfactual is trustworthy over a few seconds and becomes speculation over tens of seconds.
(5) Sample, Never Simulate Once: outcome metrics are estimated from rollouts with confidence intervals, since one lucky rollout is not evidence of avoidance.
(6) Metrics Decide The Verdict: contact or no contact, impact speed and delta-V, minimum time-to-collision, and whether the newly introduced risk (an in-lane stop, an induced rear-end) offsets the risk removed.

Figure 1: A counterfactual is not a fresh simulation. Step 2 abducts the latent scene so the neighbours keep the dispositions they actually had, step 3 changes only the ego action, and step 4 must re-generate every response because the logged responses are no longer admissible evidence.

Three simulator families answer step 4 with very different fidelity. Log replay keeps every neighbour on its recorded trajectory, which is cheap and perfectly grounded for the first fraction of a second and actively misleading afterwards. Rule-based reactive agents (car-following plus lane-change models) give physically consistent responses at negligible cost but with a narrow behavioural repertoire that tends to be too polite, biasing avoidance estimates optimistically. Learned world models sit at the top: trajectory-level sim agents of the kind benchmarked in the Waymo Open Sim Agents Challenge, LiDAR-space models such as Copilot4D, and pixel-space generative models such as Wayve’s GAIA line, which can re-render the camera stream so the perception stack is exercised too rather than being handed ground-truth tracks. Regulatory use of this machinery is already public: Waymo reconstructed fatal crashes inside its operating domain and simulated its driver in place of each human participant, both as crash initiator and as responder, reporting avoided-or-mitigated outcomes against a non-impaired, eyes-always-on reference driver. The credibility of any such claim rests on how well the simulator’s agents were validated, not on how photorealistic the rollout looks.

Figure 2: The same intervention, two simulators. With frozen playback the follower drives into the decelerating ego and the tool reports an 18.9 m/s rear-end that never existed, while a reactive model has the follower brake 0.9 s after the brake lights and stop 5.3 m short. Both panels agree on the ego: braking two seconds earlier turns a 12.6 m/s frontal impact into a stop 26.7 m short of the obstacle.

Mathematical Formulation:

Where:

• and are the observed log, meaning the recorded sensor stream plus tracks and the ego actions that were actually executed.
• is the abducted latent state: map geometry, occupancy behind occlusions, and per-agent intent and aggressiveness, held fixed across the counterfactual.
• is the counterfactual policy and its action, here a brake command issued 2 s earlier than in the log; is the simulated joint state of ego plus neighbours.
• is the world model transition, the only component that can supply neighbour responses to an action that was never taken.
• is the counterfactual outcome variable (contact, impact speed, minimum TTC) and its distribution is estimated from independent rollouts .
• is a scalar risk or severity functional, so is the risk actually removed by the intervention, and it can be negative when the new action introduces its own hazard.

Braking Two Seconds Earlier At 20 m/s:

In the logged run the brake came on with only 20 m left to a stopped lead vehicle, so the ego arrived at 12.6 m/s. The intervention adds 40 m of room, giving 60 m against the 33.3 m the ego needs, which is why it stops 26.7 m short. Notice how weak the two-second question actually is: the shortfall was m, so 0.67 s of extra warning already avoids contact, and everything beyond that buys margin rather than outcome. The engineering value of the simulator therefore lies in the second-order effects that closed-form kinematics cannot see, above all the induced risk from an earlier, harder, cause-free deceleration in front of a following vehicle.

Figure 3: Sweeping the intervention turns one question into a curve. Frontal risk collapses within about one second of extra warning while rear-end exposure grows, so past roughly 1.4 s the dominant residual hazard has changed identity. The dotted line shows why sampling matters: the single-agent kinematic estimate declares the crash impossible after 0.67 s, whereas reactive rollouts still assign residual severity out to about 1.5 s.