Design the sensor fusion system for an autonomous vehicle that combines camera, LiDAR, and radar into a unified perception of the world, as used by Waymo, Cruise, and Tesla’s Full Self-Driving stack. Each sensor has complementary strengths (camera: dense semantic and color detail; LiDAR: accurate depth and 3D structure; radar: velocity and all-weather range) and weaknesses (camera: depth ambiguity and weather sensitivity; LiDAR: sparse, expensive, poor in fog and rain; radar: low angular resolution and clutter). The fused perception must feed detection, tracking, and prediction at 10-20 Hz with hard latency bounds, and it must degrade gracefully when a sensor fails or is occluded rather than producing a confident wrong answer.
How would you design this system? Cover the per-sensor processing, the fusion architecture (early, late, or mid-level BEV fusion), the time synchronization and calibration, how you handle sensor dropout and degradation, and how you evaluate fusion quality against single-sensor baselines.
The Problem: three sensors look at the same road spray and return three different stories, and the vehicle still has to emit one confident answer every 50 to 100 ms.
Answer
Build a mid-level fusion stack over a shared bird’s-eye-view (BEV) grid. Each sensor keeps its own encoder, every feature is warped into one metric grid at one reference timestamp, and a lightweight transformer fuses them before the detection, occupancy, and tracking heads. Mid-level fusion is the pivotal choice: early raw fusion is brittle under calibration drift and blocks per-sensor pretraining, while late fusion discards exactly the cross-modal evidence that resolves camera depth ambiguity and rescues weak single-sensor detections. Two further decisions carry the safety argument. Train with modality dropout so the fused model remains usable when a sensor disappears, and keep a small non-learned geometric path (radar Doppler tracks plus LiDAR clustering) that can hold the vehicle to a safe stop if the learned stack degrades.
(1) Shared BEV Representation: project all modalities into a common ego-centric grid (roughly 100 m by 100 m at 0.4 m resolution) so fusion is a geometric alignment problem rather than an ad hoc feature concatenation.
(2) Per-Sensor Encoders: depth-distribution lift for images, sparse voxel or pillar encoder for LiDAR, and a point encoder that keeps radar range-rate; each branch trains and validates independently.
(3) Soft-Association Fusion: cross-attention between object queries and per-sensor features instead of hard pixel-to-point matching, which keeps accuracy when extrinsics drift by a fraction of a degree.
(4) Hardware Sync and Online Calibration: PTP-locked triggers, per-point timestamps, and ego-motion warping to one reference time, with reprojection residuals monitored continuously as a calibration health signal.
(5) Fallback Ladder: a health monitor scores each sensor and steps the system from full fusion, to degraded fusion with widened safety margins, to the geometric path plus an ODD exit or minimal risk maneuver.
(6) Ablation-Gated Evaluation: the release rule is that fused perception is never worse than the best single-sensor baseline on any evaluation slice, including night, rain, and forced-dropout slices.
Figure 1: One learned path through a shared BEV grid, plus a health monitor and a non-learned geometric path that stays alive when the learned path should not be trusted.
Clarify Before Designing:
(1) Sensor Suite and Cost Ceiling: how many cameras, what LiDAR (spinning 64-beam, solid state, or none), how many radars, and is 4D imaging radar available?
(2) Operational Design Domain: geofenced urban robotaxi with HD maps, or consumer highway assist without maps; does the ODD include heavy rain, snow, and night?
(3) Compute and Latency Budget: on-vehicle TOPS, power and thermal envelope, target rate (10 Hz or 20 Hz), and the p99 rather than mean latency bound.
(4) Failure Policy: on sensor loss, does the vehicle degrade speed, exit the ODD, hand back to a driver, or execute a minimal risk maneuver; who signs off on that decision?
(5) Label Supply: how much human-labeled 3D data exists, and can an offboard auto-labeling pipeline produce pseudo ground truth from full logs?
(6) Redundancy Requirement: must perception survive the loss of any single sensor, or any single compute unit, and is a second independent stack mandated?
Leave a Reply