Design the human detection and avoidance system for warehouse robots at Amazon Robotics. Thousands of mobile robots share the fulfillment center floor with human workers. Each robot must detect humans in its path, predict their trajectory, and adjust its motion to keep a safe distance by slowing down, rerouting, or stopping entirely when a human is too close.
The system must run in real time on onboard sensors (cameras, LiDAR, bump sensors), handle occlusions such as a worker stepping out from behind a storage pod, distinguish humans from shelves, bins, and other robots, and fail safely. A robot that fails to detect a human is a safety incident, not just a performance bug. It must also survive challenging lighting, reflective floors, and the fact that humans behave unpredictably.
How would you design this system? Cover the perception architecture (sensor fusion, human detection model), the trajectory prediction model for human motion, the safety-aware motion planning strategy, how you handle occlusions and sensor failures, and how you test and validate the system for safety-critical deployment.
The Problem: the robot cannot see the worker until roughly a meter before the corner, both are moving at about 1.5 m/s, and the cost of being wrong once is an injury rather than a dropped pod.
Answer
The design is a two-channel safety architecture. A certified channel (a safety-rated 2D scanner with configured protective and warning fields, backed by a contact bumper) owns the stop authority and is hardwired to the brakes, while a learned channel (360-degree camera plus 3D LiDAR fused in a bird’s-eye-view grid, then tracking and trajectory prediction) owns early, smooth slowdowns and routing. The neural stack is never the thing that guarantees the human is not hit, because no learned detector can be argued to the 
per-encounter failure rate a large fleet needs. The two pivotal decisions are to size the protective field from stopping physics rather than from model confidence, and to plan against a worst-case reachable set for occluded and unpredictable humans while using the learned predictor only to make motion smoother inside that envelope.
(1) Two Independent Channels: a safety-rated scanner and bumper stop the robot through certified logic; the ML stack only lowers a speed limit that an arbiter takes the minimum of.
(2) Multi-Modal BEV Perception: RGB and LiDAR fused into a shared bird’s-eye grid gives metric geometry plus semantics, so a person can be separated from a pod, a bin, or another robot.
(3) Unknown Means Human: any occupied cell the classifier cannot label confidently is treated as a person, so odd poses (crouching, lying, on a ladder) never fall through the class taxonomy.
(4) Visibility-Limited Speed: the speed cap is set so the robot can stop before the boundary of the nearest occluded region, which turns blind corners into a planning constraint instead of a detection problem.
(5) Designed Degraded Modes: every sensor fault maps to a lower speed cap or a controlled park, never to continued nominal driving.
(6) Validation Staircase: log replay, simulation, hardware-in-the-loop latency tests, physical stop tests against the standard, then a staged rollout with shadow logging.
Figure 1: Two lanes into one arbiter: the learned channel proposes a comfortable speed, the certified channel can always take it to zero.
Clarify Before Designing:
(1) Zone Type: is this a fenced robotic field that workers enter only under a lockout procedure, or a fully mixed aisle where people walk all shift alongside the fleet?
(2) Speed and Mass: what is the maximum robot speed and the loaded mass, since braking distance and impact energy set the protective field size?
(3) Acceptance Standard: which standard gates deployment (ISO 3691-4, ANSI/RIA R15.08) and what performance level is required for the stop function?
(4) Cost Budget: what sensor bill of materials and power budget per robot is acceptable, given that a $500 sensor becomes $5M across 10,000 robots?
(5) Human Cooperation: may workers wear beacons or high-visibility gear the fleet can read, or must the system assume an unequipped visitor in street clothes?
(6) Throughput Price: how many nuisance stops per robot-hour is operations willing to accept in exchange for a larger detection margin?
Leave a Reply