DL0124 3D Occupancy Flow

3D Occupancy Flow is a joint perception-and-forecasting output format that replaces the list of tracked objects with a dense grid: for every voxel (or BEV cell) and every future waypoint the network predicts an occupancy probability plus a flow vector describing how the mass in that cell moves. Waymo’s Occupancy Flow Fields formulation predicts three quantities per waypoint on a BEV grid covering roughly : observed occupancy, occluded occupancy, and backward flow. Camera-only 3D variants such as Occ3D on nuScenes predict a voxel grid at resolution, and Tesla presented an occupancy network with an occupancy-flow head at its 2022 AI Day. The appeal is that free space and obstacle geometry become class-agnostic and non-parametric: a tipped-over mattress, an articulated trailer, a swinging crane boom, and an overhanging branch all get represented without appearing in a detector taxonomy, and probability mass can sit on both branches of a fork at once without a mode head or non-maximum suppression. The cost is that a grid has no notion of an object, so instance identity, track continuity, and per-agent attributes disappear, and the output tensor grows by roughly two orders of magnitude, which pushes cost onto compute, memory, label pipelines, and the loss function’s handling of a grid where the overwhelming majority of voxels are empty. In practice this is why most production stacks run occupancy flow alongside a box pipeline rather than as a drop-in replacement.

(1) Dense Spatiotemporal Output: the head emits and for every cell and every future timestep, not a parametric box with a heading and a velocity per agent.
(2) Class-Agnostic Geometry: anything that occupies space is representable, which removes the long-tail detection failure where an unlisted object class becomes invisible to the planner.
(3) Backward Flow, Not Forward: predicting motion from back to makes warping a gather with one source per cell, so mass never collides during the warp and a flow-grounded occupancy consistency check becomes well defined.
(4) Non-Parametric Multimodality: a grid holds several futures simultaneously as spread probability mass, but that same property makes averaged modes look like blur or ghost occupancy rather than a ranked set of hypotheses.
(5) Identity Is What You Give Up: without instance IDs, right-of-way logic, interaction-aware conditioning, and per-object intent signals such as turn indicators lose their handle on the scene.
(6) Cost Scales With The Grid: output size grows as , dense labels require multi-sweep LiDAR accumulation and voxelization, and roughly 95% or more of voxels are empty, so class imbalance dominates the occupancy loss.

Figure 1: The object-centric branch (detect → track → forecast) produces a sparse, identity-carrying output limited by its taxonomy; the occupancy branch skips detection and data association entirely and produces a dense, class-agnostic grid with no instance IDs.

The deeper tradeoff is not really compute, it is what the downstream planner can express. A box trajectory is a commitment: this vehicle, with this ID, will be here in 3 seconds with this probability, which lets a planner reason about yielding to a specific agent, replay a scenario in simulation, and produce an auditable explanation for a maneuver. Occupancy flow is a statement about space, which is exactly what collision checking and drivable-free-space queries want, but a grid that hedges between “the cyclist goes straight” and “the cyclist turns” paints both corridors at moderate probability, and a naive cost function that treats any occupancy above a threshold as blocked yields the freezing-robot behavior. Occupancy flow partially recovers correspondence without identity: warping the previous occupancy along the predicted backward flow and multiplying it against the current occupancy gives a differentiable consistency term, so the model is penalized for teleporting mass even though it never names an object.

Mathematical Formulation:

Where:

• is the predicted occupancy probability of cell at future waypoint , and is the backward flow vector pointing to where that mass sat at .
• is the warp operator that gathers along (bilinear or trilinear), and is the flow-grounded occupancy used both as a loss term and as an evaluation metric.
• is the voxelized ground-truth occupancy label, the ground-truth flow, and the set of genuinely occupied cells to which the flow loss is masked.
• indexes waypoints and indexes the grid; balances the two terms, and is usually replaced by a focal or class-balanced variant because empty voxels dominate.
• counts predicted occupancy values for a grid over 8 waypoints, and counts a comparable box head with 50 agents, 6 modes, 16 waypoints, and 5 numbers per waypoint.
• Required initial condition: and the whole grid must be expressed in the ego frame at , so ego motion is compensated before flow is interpreted as agent motion.

Figure 2: Backward flow assigns each occupied cell at time a single source cell at , so the warp is a gather rather than a scatter and the consistency product penalizes mass that appears without a plausible origin.

Resolution and horizon are the two knobs that make or break the design. Halving the voxel size multiplies the tensor by 8 in 3D, and every additional waypoint is another full grid, so a grid over a 5-second horizon at is far beyond a real vehicle compute budget once flow channels and semantics are added. Going the other way is not free either: at voxels a pedestrian walking at moves less than one voxel per frame, so the flow target is sub-voxel and the occupancy channel alone cannot express the motion, which is precisely why the flow head is kept as a continuous regression rather than a discrete cell-to-cell assignment.

Figure 3: Output size is where the tradeoff becomes concrete: the 3D voxel grid predicts about 213x more numbers than a multi-modal box head, and adding a 3-channel flow field multiplies that by another 4 before any semantic classes are included.