DL0164 BEV Transformation: Lift-Splat-Shoot

A BEV transformation converts perspective images, each of which has thrown away the depth of every pixel, into a single metric grid in the ego frame where one cell always means the same physical patch of ground. Every method must invent the missing depth, and the two families differ only in which direction they move information. Forward projection, introduced by Lift-Splat-Shoot (LSS), predicts a categorical depth distribution per pixel, lifts each pixel into a frustum of candidate 3D points weighted by that distribution, then splats the points into BEV pillars with sum pooling. Backward projection, popularised by BEVFormer, starts from a fixed set of learned BEV queries, projects each query’s 3D anchor points into every camera using the known intrinsics and extrinsics, and pulls features back with deformable cross-attention so no explicit depth prediction is required. Both produce the identical output contract, typically a feature map at roughly 0.5 m resolution, which is why detection, map segmentation, occupancy and planning heads can be shared, and both add temporal fusion by warping the previous BEV feature into the current ego frame before merging.

(1) The Core Difficulty Is Depth, Not Geometry: the pixel-to-ray mapping and the camera-to-ego rigid transform are exactly known, so the only unknown is the scalar range along each ray.
(2) Forward Projection (Push): LSS predicts over discrete depth bins, takes an outer product with the context feature, and voxel-pools the resulting frustum point cloud into pillars.
(3) Backward Projection (Pull): BEV queries carry their own 3D position, project into the cameras that actually see them, and sample features with deformable attention, which sidesteps depth estimation entirely.
(4) Shared Output Contract: both write into the same ego-frame grid, so the transformation is a swappable module rather than an architecture commitment.
(5) Temporal Recurrence Is Not Optional: warping by the ego pose delta and fusing it into is what makes velocity estimation and short-occlusion memory possible from cameras alone.
(6) Depth Supervision Decides Accuracy: BEVDepth showed that supervising with LiDAR-projected depth, rather than letting the detection loss shape it, is the single largest quality lever for the forward family.

Figure 1: Two directions, one destination. The push lane commits to a depth distribution and scatters features outward; the pull lane keeps the grid fixed and gathers features inward. Everything downstream of the unified BEV feature map is identical, which is why these modules are interchangeable in practice.

The forward path is best understood as a soft, differentiable version of unprojecting a depth map. A pixel with context feature does not pick one depth; it spreads that feature over all bins in proportion to , so a confident pixel deposits nearly all of its mass in one pillar while an ambiguous pixel smears a faint trail along its ray. Because the splat is a sum, the operation is permutation-invariant and handles overlapping camera fields of view for free, and because it is differentiable, gradients reach the depth head through the pooling. The engineering cost is the frustum point count, which is why production implementations replace the naive scatter with a sorted cumulative-sum pooling kernel or a preallocated BEVPoolv2 index table that skips materialising the point cloud at all.

Figure 2: One pixel becomes weighted 3D points. The width of the depth distribution is literally the width of the smear in BEV, so a flat over a textureless road or a night-time scene produces a long low-confidence streak instead of a localised object.

Forward Projection (Lift-Splat-Shoot):

Where:

• is the BEV feature at grid cell in the ego frame, and is the set of frustum points whose ego coordinates fall inside that pillar.
• is the homogeneous pixel coordinate, the camera intrinsic matrix, and the camera-to-ego extrinsic rotation and translation.
• indexes the depth bins, with over a fixed range such as 1 m to 60 m in 1 m steps, giving .
• is the softmax depth distribution for pixel and its context feature, so is the outer-product lift.
• The splat is sum pooling, which keeps the operation order-free across cameras and differentiable with respect to both and .

The backward path inverts the flow of information. A query at grid cell is lifted to anchor heights along a vertical pillar, each anchor is projected into every camera, and only the cameras whose image plane actually contains the projection contribute. Deformable attention then samples a handful of learned offsets around each projected location, so cost scales with the number of queries rather than with image resolution times depth bins, and a query near a lane boundary can shift its sampling points to where the evidence is instead of trusting a predicted depth. The trade-off is that the geometry is now an attention prior rather than a hard constraint: if extrinsics are wrong, the network can still learn to compensate, which is convenient during training and dangerous during deployment because the failure is silent.

Backward Projection (Cross-Attention):

Where:

• is the cross-attention output written into BEV cell , and is the learned query plus its 2D positional encoding.
• is the -th pillar anchor at height , with a common choice spanning roughly -5 m to 3 m.
• is the full projection of camera , so is a sub-pixel image location and the multi-scale feature map of that camera.
• is the set of cameras whose frustum contains at least one anchor, so the average is taken only over hit views and empty views contribute nothing.
• is deformable attention, which samples a few learned offsets around with bilinear interpolation instead of attending to all pixels.

Frustum Cost At A Typical Configuration:

Roughly one million frustum points collapse into forty thousand pillars, an average of about 25 points per cell, and that ratio is exactly why the pooling kernel rather than the backbone is often the latency bottleneck in the forward family. It also exposes the accuracy story: because every point sits on a known ray, a lateral mistake of one pixel at 50 m is only about 4 cm, while a 5% depth mistake at the same range is 2.5 m. BEV error is dominated by range error, and it grows linearly with distance.

Figure 3: Range error, not image-plane error, sets BEV quality. A one-pixel lateral error stays under 10 cm across the whole working range, while a modest relative depth error crosses the 2 m matching threshold somewhere between 20 m and 100 m depending on the depth head, which is why depth supervision and long-baseline temporal stereo pay off so heavily.