DL0149 World Models from Unlabeled Video

The missing action labels are invented by the model itself. A latent action model (LAM) is an inverse-dynamics encoder that looks at the past frames and the next frame together and emits a single discrete code through a vector-quantized bottleneck; a forward dynamics model is then required to predict that next frame from the past frames plus alone. Both halves are trained jointly with next-frame reconstruction as the only supervision, so the code is pushed to carry exactly the part of the transition that the past cannot explain, which for gameplay and egocentric video is precisely the agent’s action. Genie made this concrete at scale, training an 11B-parameter model on roughly 30,000 hours of filtered 2D platformer video with a codebook of only latent actions, and the tiny codebook is not a detail but the mechanism: at 3 bits per step the code physically cannot smuggle the next frame through it. At inference the LAM encoder is thrown away, a key press is mapped onto one of the learned codes, and the dynamics model rolls the world forward autoregressively, which turns passive video into a playable environment.

(1) Latent Action Model: an inverse-dynamics encoder over produces one discrete code per transition, standing in for the action label that the dataset never had.
(2) The Bottleneck Is The Whole Trick: a codebook of size caps the code at bits, forcing it to describe a control decision rather than pixels.
(3) Reconstruction Is The Only Loss: the forward model’s next-frame error trains the encoder through the quantizer, so no human ever annotates a button.
(4) Tokenize, Then Roll Out: a spatiotemporal VQ-VAE turns frames into tokens and a masked-token or diffusion dynamics model generates the next frame conditioned on the past tokens and .
(5) Codes Are Shared Across Videos: because one codebook serves the whole corpus, the same index means the same intent in unseen scenes, which is what makes the latent space a controller instead of a per-clip artifact.
(6) Grounding To Real Controls Is Cheap: a handful of labeled clips, or human probing of each code, suffices to attach semantics; VPT instead spent 2,000 labeled hours to train an inverse-dynamics model and pseudo-labeled 70,000 web hours with it.
(7) Rollouts Drift: generation is autoregressive on its own output, so compounding error, memory loss, and entangled camera-versus-agent motion dominate the failure list.

Figure 1: The asymmetry between the two panels is the point. During training the next frame is visible to the latent action model, which is why an unlabeled corpus can supervise a controller; at inference that encoder is discarded and the codebook itself becomes the input device, with the rendered frame fed back as context for the next step.

Everything hinges on how much information the quantizer lets through. If the latent is generous (a wide continuous vector, or a codebook of thousands of entries), the encoder discovers the shortcut of encoding the next frame directly, the dynamics model degenerates into a decoder of the latent, reconstruction looks excellent, and controllability collapses because a user-chosen code no longer corresponds to anything an agent could do. If the latent is too narrow, distinct moves collapse into one code and the environment responds to only a couple of coarse commands. The standard diagnostic is a controllability delta: roll the model forward with the inferred code, then with a random code, and measure the gap in frame quality. A large gap means the code is doing real work; a gap near zero means either leakage or a code the model ignores.

Mathematical Formulation:

Where:

• is the latent action for the transition at step , and is vector quantization onto a codebook of entries.
• is the inverse-dynamics encoder, which is the only component allowed to see the future frame, and are its parameters.
• are the observed past frames (in practice their VQ tokens), is the true next frame, and is the prediction.
• is the forward dynamics model, a masked video-token transformer or a diffusion decoder, conditioned on the past plus and nothing else.
• is the reconstruction objective, extended in practice by the usual commitment and codebook terms; no action label appears in it, which is why unlabeled video is sufficient.
• is the capacity bound that prevents frame leakage, giving 3 bits per step when .
• is the controllability metric, comparing a rollout driven by the inferred code against one driven by a random code drawn from the same codebook.

Why 3 Bits Cannot Encode A Frame:

A token grid over a 1024-entry visual codebook already carries about 2,560 bits, roughly three orders of magnitude more than the action code, so the encoder has no route to cheat and the dynamics model must genuinely keep a model of the scene in its context. That ratio is the design knob you actually tune: too small and the environment is unresponsive, too large and it stops being an environment at all.

Figure 2: Schematic view of the tension that a single reconstruction loss hides. Reconstruction quality never warns you about failure because it keeps improving as the latent widens, whereas controllability peaks and then collapses once the code can carry the frame; only the two curves together tell you the bottleneck is set correctly.