DL0162 World Model Object Permanence

Object permanence is not a property a next-frame predictor gets for free. During a full occlusion the pixels that evidence the object are simply absent, so nothing in the current observation constrains what should re-emerge, and the only thing that can carry the object across the gap is state that outlives the pixels. Production world models therefore rely on three memory mechanisms, usually in combination: a long context of past frame latents that the generator can re-attend to, a recurrent latent state whose prior propagates unobserved objects forward, and an explicit pose-indexed geometric memory (object slots, 3D boxes, or a bird’s-eye-view layout) that is re-projected into the image when the occluder clears. Google DeepMind’s Genie 3 makes the first mechanism explicit, generating 720p frames at 24 fps with a reported visual memory extending roughly one minute into the past, while driving world models in the GAIA line lean on structured conditioning such as other agents’ 3D boxes and the ego trajectory. The practical question in an interview is therefore never “does the model understand permanence” but “does the information survive the occlusion window, and at what cost”, because a 2.5 second occlusion at 24 fps is 60 frames of pure extrapolation and the memory horizon either covers it or the object is re-invented rather than remembered.

(1) Occlusion Is A Missing-Evidence Problem: the observation likelihood contributes nothing about the hidden object, so generation must fall back on the dynamics prior rather than on perception.
(2) Long Context Buys Re-Readable Memory: keeping the last frames of latents lets attention look back at the pre-occlusion view, at the price of a KV cache linear in and a quadratic prefill.
(3) Recurrent State Buys Constant Cost: a fixed-size latent carries the object at memory per step, but its finite capacity makes identity attributes drift instead of vanishing abruptly.
(4) Explicit 3D Memory Buys Correct Re-Projection: storing object records with poses turns re-emergence into geometry, so the vehicle reappears at the right pixel and the right moment.
(5) The Failure Signature Differs Per Mechanism: a hard cliff when exceeds , a smooth decay for recurrent state, and mis-registration under ego-pose drift for geometric memory.
(6) Rollout Stability Is A Separate Axis: teacher-forced training with fully autoregressive inference creates exposure bias, so small per-frame errors compound and corrupt identity even when the object is never occluded.

Figure 1: Three places a hidden object can live. The generator is identical in all three cases; what changes is which memory still contains the occluded vehicle at frame , and therefore whether the model re-renders the same car or invents a new one.

The cleanest way to reason about the recurrent path is the standard latent-dynamics factorization used by Dreamer-style world models. A posterior fuses the new observation, while a prior predicts the next state without one. Under full occlusion the object’s contribution to the posterior disappears, so the model is effectively running the prior for steps on that part of the scene, which is exactly the regime where capacity limits and accumulated error show up. Long-context attention avoids that open-loop integration by re-reading the original evidence, but only if the pre-occlusion frames are still inside the window: once they are evicted, the model has no way to distinguish “a red hatchback was there” from “some vehicle may exist”, and it samples a plausible fresh instance. Explicit memory sidesteps both problems by storing a symbol rather than a distributed code, which is why driving-domain world models condition on agent boxes and map layouts instead of hoping attention rediscovers them.

Figure 2: What re-emergence actually looks like. The occlusion window is identical for all three models; the difference appears only after the occluder clears, where the short-context rollout hallucinates a fresh instance in the wrong lane and the long-context rollout keeps the object but mis-registers its position, while pose-indexed memory lands on the true trajectory.

Mathematical Formulation:

Where:

• is the generated frame latent at step and the conditioning action or control (steering, camera pose, text instruction).
• is everything the generator may condition on: the context window of past latents, the recurrent state, and the persistent geometric memory.
• is the memory horizon in frames and the number of latent tokens per frame, so the window holds tokens.
• is the posterior that uses the observation and the prior that does not; under full occlusion the object is carried only by .
• is the explicit memory: object records with identity, extent, and pose, plus the camera pose needed to re-project them.
• is the occlusion length in frames, obtained from its duration and the frame rate; the last line is the necessary condition for a pure context window to bridge it.
• and are the number of attention layers and the model width, which set the cache and prefill costs below.

Bridging A 2.5 Second Occlusion At 24 fps:

Sixty frames of latents at 256 tokens per frame is 15,360 tokens of context bought for a single occlusion, and every additional second of horizon costs another 6,144 tokens of cache plus a quadratic increase in prefill work, which is precisely why real-time interactive world models cap aggressively and then need a cheaper memory. The last line models the orthogonal failure: if identity is corrupted independently at each step with probability , the retention rate leaves only about 5% of objects intact after a 600-frame (25 second) rollout even with no occlusion at all. Permanence over long horizons is therefore two bugs wearing one coat: a horizon bug that shows up as a cliff, and a drift bug that shows up as exponential decay.

Figure 3: Two distinct shapes of failure. Finite context gives a step cliff exactly at , recurrent state gives geometric decay from capacity limits and compounding error, and explicit pose-indexed memory is nearly flat because duration costs it almost nothing.