DL0161 World Model Safety Edge Cases

A driving world model is an action-conditioned generative simulator that rolls forward latent state and decodes multi-view sensor observations, so the planner under test acts inside the rollout instead of replaying a fixed log. Rare events are never obtained by sampling the model unconditionally, because the learned distribution matches the fleet distribution in which a severe collision appears roughly once per miles. Instead the scenario is authored through conditioning: a low-dimensional parameter vector holding occlusion geometry, jaywalk onset time, pedestrian speed, and surface friction is drawn from a biased proposal fitted by a search loop that maximizes criticality, then each rollout is reweighted by the likelihood ratio so the resulting failure-rate estimate remains unbiased. A tire blowout is handled differently from a jaywalk, because it is a vehicle-dynamics discontinuity rather than an appearance change and almost no camera dataset contains it, so production stacks inject it in an analytic dynamics layer (collapsed cornering stiffness on one corner plus a yaw moment) while the world model supplies the surrounding traffic and imagery.

(1) Closed Loop Requires Action Conditioning: the planner’s own control enters the model at every step, so log replay and open-loop video prediction cannot be used, since the ego trajectory diverges from the recorded one within a second or two.
(2) Controllability Comes From Structured Conditioning: text, road layout, 3D agent boxes, and ego action are all conditioning channels, which turns “pedestrian steps out from behind the parked van 0.8 s before ego arrival” into an explicit specification rather than a hopeful prompt.
(3) Rarity Comes From Biased Sampling: importance sampling with a cross-entropy-method proposal concentrates rollouts near the failure boundary, cutting the sample count for a fixed confidence by three to five orders of magnitude.
(4) Unbiasedness Comes From Likelihood Ratios: reporting the raw failure fraction under overstates real-world risk by exactly the sampling bias, so every rollout carries its weight into the estimator.
(5) Hybrid Physics For Fault Injection: blowouts, brake fade, and actuator latency live in a validated dynamics model, and the world model consumes the resulting ego state as a conditioning signal.
(6) Plausibility Must Be Constrained: unconstrained adversarial search produces kinematically impossible agents and physically unavoidable collisions, so scenarios are filtered by a likelihood floor on and by an avoidability check against a reference planner.
(7) Rollout Stability Bounds The Horizon: autoregressive generation accumulates drift, so useful stress tests run 10 s to 30 s at 10 Hz rather than minutes, and geometric consistency across views is monitored as a validity metric.

Figure 1: Two loops, not one. The inner loop (world model and planner exchanging and ) provides reactivity, while the outer loop (criticality score, proposal update, resampled ) provides rarity. Fault injection enters through the analytic dynamics block because the generative model has essentially no blowout data to learn from.

The two example events stress different parts of the stack. A jaywalk is fundamentally a behavior and occlusion problem, and the world model is well suited to it: the pedestrian is spawned behind a parked vehicle, the onset time is set relative to the ego’s projected arrival, and the generated views must keep the occluder and the emerging body consistent across cameras so that the planner’s detector sees a realistic partial reveal. A blowout is a plant fault: within roughly 100 ms the affected corner loses most of its lateral capability and an asymmetric longitudinal force produces a yaw moment the driver did not command, so what the planner must handle is a sudden mismatch between commanded and achieved trajectory, not a novel image. Injecting it analytically also gives the parameter sweep something meaningful to search over, since severity, onset speed, and which corner fails are continuous knobs, whereas asking a video model to “show a blowout” yields visually plausible frames with no correct dynamics underneath them.

Mathematical Formulation:

Where:

• is the latent scene state, the decoded multi-view observation, and the control emitted by the planner under test .
• is the scenario parameter vector (occlusion offset, jaywalk onset, agent speeds, friction, fault severity), held fixed for a rollout and sampled per rollout.
• is the nominal distribution estimated from fleet logs and the biased proposal from which is actually drawn.
• is the failure event for the closed-loop rollout (collision, or time-to-collision below a threshold) and its indicator.
• is the importance weight that removes the sampling bias, and indexes the rollouts in the campaign.
• is the relative standard error of naive Monte Carlo, which is what makes small unaffordable without a proposal.

Blowout As A Dynamics Perturbation:

Here is the nominal cornering stiffness of the failed tire, its residual fraction after deflation (often taken near 0.3 and ramped over about 100 ms), the track width, and the left-right longitudinal force asymmetry that generates the uncommanded yaw moment . Both and the onset ramp are entries in , so the same search machinery that finds the worst jaywalk timing also finds the worst blowout severity at the worst road curvature.

Sampling Budget For A Event:

Reaching a 10% relative standard error on a event needs nominal rollouts, which at 30 s of simulated time each and 1000 parallel workers is about 347 days of wall clock. A proposal that delivers a variance-reduction factor brings the same confidence down to rollouts, or roughly 50 minutes. That ratio, not visual fidelity, is why rare-event machinery is the core of a stress-testing system.

Figure 2: Left: the nominal distribution puts almost no mass in the failure region, so the proposal is shifted onto the boundary and every sample is reweighted by . Right: because naive relative error scales as , a variance-reduction factor moves the 10% confidence target from rollouts to .