Design a multi-camera vehicle and person tracking engine that fuses feeds from dozens of overlapping cameras across a campus, parking lot, or city intersection into a unified, persistent track for every vehicle and pedestrian. Each camera sees only a partial view: objects enter and leave frames, get occluded by each other and by structures, change appearance across lighting and weather, and must be handed off from one camera to the next as they cross camera boundaries.
The engine must run in real time across all cameras, maintain consistent global IDs across views (the same car is ID 42 in camera 3 and camera 7), recover from occlusions and re-identification failures, and scale to hundreds of cameras without linearly growing compute.
How would you design this model? Cover the candidate multi-object tracking architectures (per-camera tracking plus cross-camera re-identification, joint multi-camera tracking, tracking-by-detection with a global association layer), the re-identification feature design and how it survives viewpoint and lighting change, the cross-camera handoff and global ID assignment strategy, how you handle occlusion and track fragmentation, and how you scale inference across many cameras.
The Problem: every camera invents its own local track ID, and the same vehicle looks different in every view. The model’s job is to decide which local tracks are one object and keep that decision stable for hours.
Answer
The design is tracking-by-detection with a two-tier association: a cheap per-camera tracker turns detections into short, high-purity tracklets, and one global association layer stitches tracklets from all cameras into persistent global IDs. Two decisions are pivotal. First, associate at the tracklet level, not the detection level, which shrinks the matching problem by one to two orders of magnitude and gives ReID a multi-frame embedding instead of one blurry crop. Second, gate every candidate pair through a camera-topology graph with travel-time feasibility, so cost grows linearly in cameras rather than quadratically. Geometric bird’s-eye-view fusion is used only where calibrated overlap actually exists, because most real deployments are mostly non-overlapping.
(1) Per-Camera Tracking + Cross-Camera ReID: independent ByteTrack-style Kalman and IoU trackers per camera, then match tracklets across cameras by appearance embedding against a rolling gallery.
(2) Joint Multi-Camera Tracking (BEV): project detections or features onto a shared ground plane through calibrated homographies, fuse them into one occupancy map, and track once in world coordinates.
(3) Tracking-by-Detection + Global Association Layer: keep per-camera tracklets, then solve one hierarchical graph association over all cameras with learned edge costs combining appearance, motion, and topology priors.
(4) Chosen Hybrid: approach (3) as the backbone, borrowing (1)’s tracklet-level ReID features and switching to (2)’s geometric fusion inside the few genuinely overlapping camera clusters.
Figure 1: One cheap tracker per camera, one global association layer for the whole site, and a stored ID book so a returning object reclaims its old identity instead of getting a new one.
Clarify Before Designing:
(1) Calibration and Overlap: do we have intrinsics and extrinsics per camera, and what fraction of camera pairs actually share ground-plane coverage?
(2) Latency Mode: is this live tracking with a sub-second budget, or forensic review where a few seconds of buffering (and thus batch graph solving) is allowed?
(3) Scale and Input Size: how many cameras, at what resolution and frame rate, and can we drop to 10 fps for detection?
(4) Identity Duration: must an ID survive a 90 second walk between two non-overlapping cameras, or only a handoff across an overlapping seam?
(5) Metric and Conditions: are we scored on IDF1 or HOTA, and does the deployment include night, rain, and fleet vehicles that look identical?
Leave a Reply