DL0163 Map-Based vs Mapless World Models

Both families learn a latent transition model that rolls the scene forward under a candidate ego action, so the architectural difference is not the dynamics head but what the dynamics head is allowed to condition on. A map-based world model receives a local HD-map crop , a rasterized or vectorized patch of lane centerlines, stop lines, crosswalks, and traffic-light-to-lane associations, indexed out of a global map by the pose that the localization stack estimates. That crop enters as a hard geometric prior, so the network never has to learn road topology from pixels and can spend capacity on agent behavior. A mapless model sees only the sensor history and must reconstruct that same topology inside its latent on every frame, which is statistically much harder but works on roads nobody has surveyed. The practical result is a swap of error sources rather than a strict improvement: the map-based system inherits map staleness and localization error, the mapless system inherits data hunger and weaker long-horizon geometric consistency.

(1) Conditioning Set: map-based factorizes as with an extra exogenous input, while mapless factorizes as and must carry topology in the recurrent state.
(2) Where The Prior Lives: in one system lane geometry is a surveyed asset refreshed by a mapping fleet, in the other it is weights learned from driving video.
(3) Sample Efficiency: handing the model the road graph removes an enormous nuisance factor, so map-conditioned models converge on thousands of hours where mapless generative models are trained on tens of thousands to millions.
(4) Failure Signature: a stale or misaligned map produces confidently wrong rollouts, whereas a mapless model facing an ambiguous road typically produces a wide, visibly uncertain distribution.
(5) Pose Coupling: the map crop is only meaningful in the correct frame, so map-based prediction quality is bounded by localization accuracy; mapless prediction has no pose dependency at all.
(6) Scaling And ODD: map-based scales with surveyed kilometers and re-survey cadence, mapless scales with data and compute, which is why geofenced robotaxi and everywhere-consumer programs made opposite choices.

Figure 1: The two stacks share an encoder, a latent transition model, and a decoder. Only one extra edge differs: the HD map crop selected by the estimated pose feeding the transition model. That single edge is what buys sample efficiency and what imports two new failure sources, map staleness and localization drift.

The interesting behavior appears when the map and the world disagree. Because the map is treated as ground truth during training, the model learns to trust it, and at inference it has no mechanism to discount it: a lane that was closed for construction last week is still a valid lane inside , so the imagined rollout drives straight through the closure with low predictive variance. A mapless model never had that crutch, so its estimate of drivable space comes from the same pixels that show the cones. It pays for this with weaker long-horizon geometry, since nothing anchors an eight-second rollout to a globally consistent lane graph and errors compound in the latent. This is also why the honest comparison is not accuracy on a nominal benchmark, where the map-conditioned model almost always wins, but accuracy conditioned on map validity.

Figure 2: Same scene, same closure, different conditioning. The map prior keeps the rollout tight and correct whenever the map matches reality, and keeps it tight and wrong when it does not. The mapless rollout is wider but reactive, because the only evidence it ever had for drivable space is the current image.

Mathematical Formulation:

Where:

• is the latent scene state produced by encoder from the observation window of surround camera, radar, and optionally LiDAR frames.
• is the global HD map, the true ego pose, the estimate produced by localization, and the operator that crops and transforms into the ego frame.
• is localization error. It is not observation noise: it rigidly shifts the entire conditioning input, so a 0.5 m lateral bias moves every lane boundary by 0.5 m.
• is the candidate ego action, which is what makes both models action-conditioned and therefore usable for planning rather than passive video prediction.
• and are the two parameter sets; must additionally encode road topology that receives for free through .
• is the decoder (future frames, BEV occupancy, or agent boxes), the rollout horizon in steps, and the per-step loss, typically reconstruction plus a KL or diffusion denoising term.

Figure 3: Schematic, but the ordering is the point. With a valid map the conditioned model dominates at every horizon; with a stale or misaligned map the same model becomes the worst of the three within roughly a second, because the prior is applied with full confidence in the wrong place. A mapless model has no best case that good and no worst case that bad.