Interview for Machine Learning

Category: CV

MSD0063 Image Copyright Detection
Design an image copyright violation detection system for Meta. When users upload images to Facebook or Instagram, the platform must detect whether an uploaded image infringes on a copyrighted work (a professional photograph, a movie still, a book cover, artwork) before or shortly after publication. Unlike audio or video fingerprinting, which leans on perceptual hashes for near-duplicate detection, image copyright detection must also handle crops, resizes, color adjustments, partial overlays, and collages where a copyrighted image is one component of a larger composition.
The system must embed both uploaded images and a reference database of registered works into a shared visual feature space, retrieve nearest neighbors with approximate nearest neighbor (ANN) search, and classify whether a match constitutes infringement. It must be robust to adversarial perturbations (a user adding noise or a fake watermark to evade detection), handle partial infringement where only 20 percent of the image is copyrighted, and process hundreds of millions of uploads per day at low latency. Copyright owners must be able to register their works and receive alerts when matches are found.
How would you design this system? Cover the visual embedding architecture (contrastive learning vs triplet loss vs CLIP-style vision encoder), the ANN retrieval index (FAISS vs ScaNN vs HNSW), the infringement classification strategy (similarity threshold vs learned classifier vs human review), how you handle adversarial robustness and partial infringement, and how you evaluate detection precision and recall when ground-truth labels require legal judgment.

The Problem: the same protected photograph arrives cropped, recolored, noised, and buried inside a collage, and the gate has milliseconds and no pixel-exact copy to compare against.
Answer
The design is a retrieve-then-verify matcher built on a copy-detection embedding space, not a semantic one. Every upload is hashed for exact republishes, split into a global view plus a handful of region crops, embedded by a self-supervised copy-detection descriptor trained with aggressive transform augmentation, and queried against a sharded FAISS index of registered works. Retrieved candidates go through geometric verification with local features and a calibrated classifier that scores similarity, matched area, and registry metadata together. The two pivotal decisions are to query with regions rather than one whole-image vector, which is what makes a 20 percent collage insert detectable, and to route scores into enforcement bands so only high-confidence, high-coverage matches act automatically while the ambiguous middle becomes a human review queue.
(1) Copy Descriptor Not CLIP: train a self-supervised descriptor (SSCD-style, with entropy regularization for comparable scores across images) on copy transforms; a CLIP semantic space retrieves same-topic photos and destroys precision.
(2) Multi-Region Queries: query with 1 global view plus 4 quadrants plus 4 half-scale crops, so at least one region is dominated by the protected work even in a dense collage.
(3) Two-Tier Index: a PDQ hash lookup catches pixel-near republishes in microseconds; a FAISS IVF-PQ index over roughly 400M reference vectors handles transformed copies.
(4) Retrieve Then Verify: ANN top-k feeds local-descriptor matching with RANSAC, and a learned classifier fuses inlier count, matched area fraction, descriptor score, and registration signals.
(5) Banded Enforcement: auto-allow, label plus owner alert, hold for human review, or auto-block; the top band alone acts without a human, and every action has an appeal path.
(6) Defense In Depth: adversarial augmentation during training plus three independent signal families (hash, global descriptor, local features) means an evader must break all of them at once.

Figure 1: Two paths meet at the index: uploads are embedded per region and searched, while registered works flow through ownership checks and the same encoder into the reference shards.
Clarify Before Designing:
(1) Timing And Surface: do we block before publication or detect within minutes after, and does that differ for feed posts, Stories, and paid ads?
(2) Registry Shape: how many registered works, growing how fast, and is there a premium catalog (studios, stock agencies) that deserves a stricter path than the long tail?
(3) Enforcement Policy: is the action block, label, claim for monetization, or notify-only, and who owns the appeals and counter-notice flow?
(4) Budget: uploads per day, peak QPS, allowed added latency per upload, and cost ceiling per thousand uploads?
(5) Ground Truth: can legal reviewers label a gold set, at what weekly volume, and can licensed reposts be distinguished from infringing ones in that data?
(6) Retroactivity: when a new work is registered, must we rescan historical uploads, and how far back?
Login to view more content
September 10, 2026
MSD0059 Warehouse Robot Human Avoidance
Design the human detection and avoidance system for warehouse robots at Amazon Robotics. Thousands of mobile robots share the fulfillment center floor with human workers. Each robot must detect humans in its path, predict their trajectory, and adjust its motion to keep a safe distance by slowing down, rerouting, or stopping entirely when a human is too close.
The system must run in real time on onboard sensors (cameras, LiDAR, bump sensors), handle occlusions such as a worker stepping out from behind a storage pod, distinguish humans from shelves, bins, and other robots, and fail safely. A robot that fails to detect a human is a safety incident, not just a performance bug. It must also survive challenging lighting, reflective floors, and the fact that humans behave unpredictably.
How would you design this system? Cover the perception architecture (sensor fusion, human detection model), the trajectory prediction model for human motion, the safety-aware motion planning strategy, how you handle occlusions and sensor failures, and how you test and validate the system for safety-critical deployment.

The Problem: the robot cannot see the worker until roughly a meter before the corner, both are moving at about 1.5 m/s, and the cost of being wrong once is an injury rather than a dropped pod.
Answer
The design is a two-channel safety architecture. A certified channel (a safety-rated 2D scanner with configured protective and warning fields, backed by a contact bumper) owns the stop authority and is hardwired to the brakes, while a learned channel (360-degree camera plus 3D LiDAR fused in a bird’s-eye-view grid, then tracking and trajectory prediction) owns early, smooth slowdowns and routing. The neural stack is never the thing that guarantees the human is not hit, because no learned detector can be argued to the $10^{-9}$ per-encounter failure rate a large fleet needs. The two pivotal decisions are to size the protective field from stopping physics rather than from model confidence, and to plan against a worst-case reachable set for occluded and unpredictable humans while using the learned predictor only to make motion smoother inside that envelope.
(1) Two Independent Channels: a safety-rated scanner and bumper stop the robot through certified logic; the ML stack only lowers a speed limit that an arbiter takes the minimum of.
(2) Multi-Modal BEV Perception: RGB and LiDAR fused into a shared bird’s-eye grid gives metric geometry plus semantics, so a person can be separated from a pod, a bin, or another robot.
(3) Unknown Means Human: any occupied cell the classifier cannot label confidently is treated as a person, so odd poses (crouching, lying, on a ladder) never fall through the class taxonomy.
(4) Visibility-Limited Speed: the speed cap is set so the robot can stop before the boundary of the nearest occluded region, which turns blind corners into a planning constraint instead of a detection problem.
(5) Designed Degraded Modes: every sensor fault maps to a lower speed cap or a controlled park, never to continued nominal driving.
(6) Validation Staircase: log replay, simulation, hardware-in-the-loop latency tests, physical stop tests against the standard, then a staged rollout with shadow logging.

Figure 1: Two lanes into one arbiter: the learned channel proposes a comfortable speed, the certified channel can always take it to zero.
Clarify Before Designing:
(1) Zone Type: is this a fenced robotic field that workers enter only under a lockout procedure, or a fully mixed aisle where people walk all shift alongside the fleet?
(2) Speed and Mass: what is the maximum robot speed and the loaded mass, since braking distance and impact energy set the protective field size?
(3) Acceptance Standard: which standard gates deployment (ISO 3691-4, ANSI/RIA R15.08) and what performance level is required for the stop function?
(4) Cost Budget: what sensor bill of materials and power budget per robot is acceptable, given that a $500 sensor becomes $5M across 10,000 robots?
(5) Human Cooperation: may workers wear beacons or high-visibility gear the fleet can read, or must the system assume an unequipped visitor in street clothes?
(6) Throughput Price: how many nuisance stops per robot-hour is operations willing to accept in exchange for a larger detection margin?
Login to view more content
September 10, 2026
MSD0053 Video Chaptering Keyframe
Design a deep learning system for autonomous video chaptering and keyframe extraction. A video platform (YouTube, Vimeo, Coursera) must automatically segment uploaded videos into chapters and extract representative keyframes so that viewers can navigate to specific sections without scrubbing through the entire video. For a 45-minute lecture, the system must detect topic boundaries (when the speaker moves from “gradient descent” to “backpropagation”), assign a chapter title to each segment, and pick one keyframe per chapter that best represents its content.
The system must handle diverse video types (lectures, vlogs, tutorials, interviews, gameplay) with no manual metadata, detect both visual and audio scene changes (a slide change, a speaker cut, a topic shift in the transcript), and produce chapter titles that are concise and informative. The system must process hundreds of thousands of uploaded videos per day, and the chaptering must be good enough that viewers prefer the auto-generated chapters over no chapters at all.
How would you design this system? Cover the multimodal feature extraction (visual embeddings from frame sampling, audio features, transcript text), the boundary detection model (shot boundary detection vs topic segmentation vs learned segmentation), the keyframe selection strategy (clustering vs diversity sampling vs attention-based), the chapter title generation approach (extractive from transcript vs abstractive summarization vs LLM generation), and how you evaluate chaptering quality when there is no ground truth for most videos.

The Problem: a 45-minute video arrives as raw frames and audio with no table of contents. Find where each topic starts, name it, and pick the one frame that stands for it, for hundreds of thousands of uploads a day.
Answer
The design is an upload-time batch pipeline that turns every video into a per-second sequence of multimodal tokens (frame embeddings with OCR text, speaker-change and music cues from the audio, sentence embeddings from the ASR transcript) and runs a learned boundary transformer over that sequence to predict where chapters start. Shot-boundary detection and transcript topic segmentation are input features to this model rather than the model itself, so a slide flip alone does not cut a chapter while a topic shift inside one static shot still does. The model is trained on creator-supplied chapter timestamps, the millions of videos where creators already typed chapters into the description, filtered to remove noisy chaptering. Each chapter then receives a keyframe chosen as the quality-filtered medoid of its frame embeddings and a title written by an instruction-tuned LLM that reads the chapter’s transcript and slide text under a grounding check, with an extractive fallback. Quality is proven the way the product will be judged: tolerance-window F1 against creator chapters, pairwise human ratings, and online navigation behavior.
(1) One Time Axis: resample every modality to one token per second so boundaries, keyframes, and titles share coordinates; a 45-minute lecture becomes 2,700 tokens.
(2) Weak Labels from Creators: train on creator timestamps that satisfy the platform’s chapter rules (at least three chapters, each 10 s or longer, first at 0:00), reweighted by video type and held out by creator.
(3) Learned Boundary Model over Signals: shot cuts, speaker changes, and transcript cohesion dips are features; a transformer outputs P(boundary at t) and decoding enforces a minimum chapter length.
(4) Quality-Filtered Medoid Keyframe: discard blurry, dark, and transition frames near boundaries, then pick the medoid frame, the one that minimizes total distance to every other frame in the chapter.
(5) Grounded LLM Titles: generate 2-6 word titles from the transcript plus OCR text in one call per video; reject titles whose content words never appear in the source and fall back to the top extractive phrase.
(6) Evaluation Without Truth: tolerance-window boundary F1 on creator-chaptered holdouts, pairwise raters, and chapter click and seek metrics online; withhold chapters when confidence is low.

Figure 1: The chaptering pipeline: three feature streams on one time axis, one learned boundary model, then keyframe and title stages per chapter, with creator edits and navigation logs feeding retraining.
Clarify Before Designing:
(1) Content Mix and Length: what share of uploads is speech-dense (lectures, interviews) versus gameplay, music, or silent screen recordings, and what does the duration tail look like (multi-hour stream replays)?
(2) Freshness Budget: must chapters exist at publish time, or is a delay of minutes to an hour acceptable, and may we reprocess the back catalog when the model improves?
(3) Existing Signals: does the platform already run ASR for captions, and does it store creator-typed chapter timestamps that can serve as labels?
(4) Product Constraints: minimum and maximum chapter counts, minimum chapter length, title length cap, and whether creators can edit or disable auto chapters?
(5) Cost Envelope: how many GPU-hours per thousand video-hours are allowed, which decides frame sampling rate, ASR model size, and whether an LLM runs per video?
(6) Language Coverage: which languages need titles, and how reliable is the transcript for each?
Login to view more content
September 8, 2026
MSD0052 Airbnb Photo Tagging
Design a system that automatically tags photos uploaded by hosts (e.g., “kitchen”, “bedroom”, “pool”, “ocean view”) on Airbnb using a deep image classification model. When a host creates or edits a listing, they upload dozens of photos, and Airbnb must automatically label each photo with the room type or amenity it depicts so that guests can filter by feature (“pool”, “mountain view”) and the listing page can organize photos into structured sections (Living Room, Bedroom, Kitchen, Exterior). The system must handle a wide taxonomy of tags (50+ room types and amenities), work across varying photo quality, angles, and lighting from phone cameras worldwide, distinguish similar categories (“living room” vs “family room”, “balcony” vs “patio”), and handle the long tail of rare amenities (“sauna”, “wine cellar”) that have few training examples. Mislabeling a bathroom as a kitchen degrades search relevance and user trust, so the system must flag low-confidence tags for host confirmation rather than auto-assigning them.
How would you design this system? Cover the classification model architecture (single-label vs multi-label vs hierarchical classification), how you handle the long tail of rare tags with few training examples, how you generate training labels from host-provided metadata and guest clicks, how you calibrate confidence for the host-confirmation workflow, the inference pipeline for processing photo uploads in real time, and how you evaluate tagging accuracy across the full tag taxonomy.

The Problem: Thirty phone photos arrive per listing and each must land on one of fifty-plus tags, where the popular tags have millions of examples, the rare ones have a few hundred, and a confident wrong tag costs more trust than no tag at all.
Answer
The design is a shared vision backbone with two heads: a hierarchical softmax head for the mutually exclusive room or scene type, and a multi-label sigmoid head for amenities that co-occur in one photo. Training labels are mined at scale from host captions, existing photo-section placement, and guest filter clicks, then sharpened by the host confirmations the product itself generates. Rare tags are handled by decoupling representation from classifier: the backbone learns from the head-heavy data, the classifier heads are re-balanced, and tags with too few examples are served through a nearest-prototype path until they graduate. Every prediction passes a per-tag calibrated confidence gate with three outcomes: auto-assign, a one-tap host confirmation card, or leave untagged. Inference runs as a batched GPU service triggered by the upload event, so a 30-photo listing is tagged before the host finishes the flow, and every confirmation flows back into the next training round.
(1) Label Structure: a scene tree (indoor, outdoor, view) with roughly 40 leaves under a hierarchical softmax, plus roughly 30 amenity tags under independent sigmoids; sibling pairs humans confuse (“living room” vs “family room”) share one merged leaf until a reliable cue separates them.
(2) Weak Labels at Scale: host captions, section placement, listing-level amenity declarations, and guest clicks after filtering by an amenity give millions of noisy labels; host confirmations and removals form the small gold layer that drives thresholds and evaluation.
(3) Long Tail by Decoupling: train the backbone on all data, then re-fit the classifier heads with class-balanced sampling and logit adjustment; tags below about 5k labels use a prototype classifier on frozen embeddings.
(4) Calibrated Confidence Gate: temperature scaling per head and per-tag thresholds tuned on the confirmed holdout to a 98% precision floor; high confidence auto-assigns, the gray zone asks the host, low confidence stays untagged.
(5) Async GPU Serving at Upload: an upload event triggers fetch, dedupe, batch inference, and a write to a versioned tag store that feeds search filters and section layout; the catalog backfill is a batch job on the same model.
(6) Confirmation Feedback Loop: confirmations, removals, and re-sectioning are logged as labels with model version, and monthly retraining passes per-region and per-tag gates before release.

Figure 1: One backbone pass per photo, two heads, one gate: confident tags ship, doubtful ones become a host card, and every host answer becomes a training label.
Clarify Before Designing:
(1) Taxonomy Ownership: is the 50-tag list fixed or does the product team add tags quarterly, and does a hierarchy (room vs amenity vs view) already exist?
(2) Latency Expectation: must tags appear inside the upload flow within seconds, or only before the listing goes live minutes later?
(3) Label Sources: what fraction of photos carry a host caption or a section placement, and are guest filter-and-click logs joinable to individual photos?
(4) Error Asymmetry: which hurts more, a missing “pool” or a false “pool”, and is there a precision floor the search team already enforces?
(5) Scale: photos uploaded or edited per day, catalog size for a backfill, and the share of phone versus professional photos?
(6) Confirmation Budget: how many confirmation prompts per upload before hosts disengage, and may a listing publish while tags are pending?
Login to view more content
September 8, 2026
MSD0051 Street View Face Blurring
Design the face and license-plate blurring system for Google Street View using a deep object detection model. Street View captures panoramic street-level imagery from cameras mounted on cars, trekkers, and drones, covering millions of miles of roads across 100+ countries. Every captured panorama must be processed to blur all human faces and vehicle license plates before publication, to comply with privacy laws (GDPR in Europe, CCPA in California, and local regulations worldwide). The system must detect faces and plates at varying scales, angles, lighting, and occlusion (a face in a car window, a plate partially covered by snow), process millions of panoramas per day with minimal compute cost, and keep the false-negative rate near zero because a single unblurred face is a privacy violation. At the same time, false positives that blur non-face objects (signs, manhole covers, building facades) degrade image quality and user experience.
How would you design this system? Cover the detection model architecture (single-stage detector vs two-stage detector vs segmentation-based), how you handle the extreme scale variation and small objects, how you generate and augment training data for rare cases (motorcycle plates, faces in helmets), how you balance the false-negative vs false-positive trade-off given the privacy-first requirement, the inference pipeline for processing millions of panoramas per day, and how you monitor and audit blur quality at scale.

The Problem: every panorama hides a handful of faces and plates at 20 to 200 pixels among signs and facades that look like them; miss one and it is a privacy violation, blur too many and the street turns to mush.
Answer
The design is a recall-first single-stage detector run on overlapping tiles of the full-resolution panorama, feeding a fail-closed blur gate that no image can bypass on its way to publication. An FPN-style anchor-free detector with separate face and plate heads scores every 1024 px tile at native resolution, detections are merged with panorama-level NMS, and the threshold is set far below the mAP-optimal point so recall, not precision, is what the model is tuned for. Boxes are dilated before blurring, low-confidence boxes go to a cheap verifier that promotes rather than vetoes, and a stratified human audit plus the user report path close the loop on the misses no model catches. The pivotal calls are tiling instead of downscaling, an explicit blur budget instead of a precision target, and treating the blur job as a publication gate rather than a post-processing step.
(1) Tiled Native-Resolution Input: the 8k by 4k panorama is cut into overlapping 1024 px tiles so a 20 px face is still 20 px when the detector sees it; a 2x upsampled re-crop handles the smallest bucket.
(2) Single-Stage FPN Detector: an FCOS-class anchor-free model with focal loss, separate face and plate heads, and pyramid levels matched to the 16 to 256 px object range.
(3) Recall-First Operating Point: the threshold is chosen from the recall-versus-false-positives-per-panorama curve, with a blur budget (blurred pixel fraction) as the only cap on precision loss.
(4) Blur If In Doubt: boxes are dilated 15 to 20%, overlapping boxes are unioned, and an uncertain band is re-scored by a crop verifier that can add blur but never remove a confident detection.
(5) Targeted Data Program: country-stratified plate labels, hard-negative mining on facades and signs, and copy-paste augmentation for motorcycle plates, helmets, masks, and faces behind glass.
(6) Fail-Closed Batch Gate: a panorama publishes only with a signed blur receipt; stratified human audit and user reports feed misses and false blurs back into retraining.

Figure 1: The publication path: tile, detect, merge, blur generously, and never publish without a blur receipt; the audit loop underneath is what keeps recall honest over time.
Clarify Before Designing:
(1) Scale and Latency: how many panoramas per day and how many pixels each; is publication latency hours or days, which decides whether this is a batch system or a streaming one?
(2) Legal Bar: which jurisdictions and what counts as a face or plate (statues, posters, reflections, partial plates), and is “near zero” a measured audit rate or a contractual guarantee?
(3) Blur Budget: what fraction of blurred non-target pixels per panorama is acceptable before image quality complaints outweigh privacy gains?
(4) Capture Fleet: how many camera generations and platforms (car, trekker, drone) with different resolutions and viewpoints must one model serve?
(5) Data Assets: do labeled faces and plates from earlier campaigns exist, and can already-published blurred imagery be reused as training signal?
(6) Back-Catalog: must a better model reprocess years of published imagery, which changes the compute plan and the rollout policy?
Login to view more content
September 8, 2026
MSD0046 Multi-Camera Tracking Surveillance
Design a multi-camera vehicle and person tracking engine that fuses feeds from dozens of overlapping cameras across a campus, parking lot, or city intersection into a unified, persistent track for every vehicle and pedestrian. Each camera sees only a partial view: objects enter and leave frames, get occluded by each other and by structures, change appearance across lighting and weather, and must be handed off from one camera to the next as they cross camera boundaries.
The engine must run in real time across all cameras, maintain consistent global IDs across views (the same car is ID 42 in camera 3 and camera 7), recover from occlusions and re-identification failures, and scale to hundreds of cameras without linearly growing compute.
How would you design this model? Cover the candidate multi-object tracking architectures (per-camera tracking plus cross-camera re-identification, joint multi-camera tracking, tracking-by-detection with a global association layer), the re-identification feature design and how it survives viewpoint and lighting change, the cross-camera handoff and global ID assignment strategy, how you handle occlusion and track fragmentation, and how you scale inference across many cameras.

The Problem: every camera invents its own local track ID, and the same vehicle looks different in every view. The model’s job is to decide which local tracks are one object and keep that decision stable for hours.
Answer
The design is tracking-by-detection with a two-tier association: a cheap per-camera tracker turns detections into short, high-purity tracklets, and one global association layer stitches tracklets from all cameras into persistent global IDs. Two decisions are pivotal. First, associate at the tracklet level, not the detection level, which shrinks the matching problem by one to two orders of magnitude and gives ReID a multi-frame embedding instead of one blurry crop. Second, gate every candidate pair through a camera-topology graph with travel-time feasibility, so cost grows linearly in cameras rather than quadratically. Geometric bird’s-eye-view fusion is used only where calibrated overlap actually exists, because most real deployments are mostly non-overlapping.
(1) Per-Camera Tracking + Cross-Camera ReID: independent ByteTrack-style Kalman and IoU trackers per camera, then match tracklets across cameras by appearance embedding against a rolling gallery.
(2) Joint Multi-Camera Tracking (BEV): project detections or features onto a shared ground plane through calibrated homographies, fuse them into one occupancy map, and track once in world coordinates.
(3) Tracking-by-Detection + Global Association Layer: keep per-camera tracklets, then solve one hierarchical graph association over all cameras with learned edge costs combining appearance, motion, and topology priors.
(4) Chosen Hybrid: approach (3) as the backbone, borrowing (1)’s tracklet-level ReID features and switching to (2)’s geometric fusion inside the few genuinely overlapping camera clusters.

Figure 1: One cheap tracker per camera, one global association layer for the whole site, and a stored ID book so a returning object reclaims its old identity instead of getting a new one.
Clarify Before Designing:
(1) Calibration and Overlap: do we have intrinsics and extrinsics per camera, and what fraction of camera pairs actually share ground-plane coverage?
(2) Latency Mode: is this live tracking with a sub-second budget, or forensic review where a few seconds of buffering (and thus batch graph solving) is allowed?
(3) Scale and Input Size: how many cameras, at what resolution and frame rate, and can we drop to 10 fps for detection?
(4) Identity Duration: must an ID survive a 90 second walk between two non-overlapping cameras, or only a handoff across an overlapping seam?
(5) Metric and Conditions: are we scored on IDF1 or HOTA, and does the deployment include night, rain, and fleet vehicles that look identical?
Login to view more content
September 3, 2026
MSD0041 Fitness Pose Estimation
Design a deep learning model for real-time pose estimation in a fitness app that counts reps, corrects exercise form, and runs on a phone. The app uses the phone’s front camera to track the user’s body keypoints (shoulders, elbows, wrists, hips, knees, ankles) while they perform squats, push-ups, and yoga poses.
The model must estimate 2D or 3D pose at 30 fps on-device, recognize which exercise is being performed, count repetitions by detecting the movement cycle, and flag form errors such as knees caving inward during a squat or a back rounding during a deadlift. It has to work across body types, clothing, camera angles, and lighting, and latency must stay low enough for real-time audio feedback.
How would you design this model? Cover the candidate pose estimation architectures (heatmap regression vs direct coordinate regression vs 3D lifting from 2D), the exercise classification and rep counting logic, the form-error detection mechanism, the on-device deployment strategy (quantization, distillation), and how you evaluate pose accuracy and rep-counting reliability.

The Problem: one lens, no depth, and a 33 ms per frame budget, yet every frame must answer three questions at once: where the joints are, where in the rep cycle the body is, and whether the form is safe.
Answer
Run a detector-tracker cascade around a single-person 2D keypoint network whose output head is a coordinate classifier (SimCC-style 1D bins per axis), trained with dense heatmap supervision and distilled from a large teacher. The scale-normalized keypoint sequence then feeds one small temporal head that emits the exercise label, the rep phase, and the form flags. 2D stays the primary signal because monocular depth is ambiguous, and a 27-frame 2D-to-3D lifting head is switched on only for errors that genuinely need depth, such as back rounding or knee valgus seen from an oblique angle. Two decisions carry the design: put the spatial prior in training (heatmaps) while keeping the cheap output at inference (coordinates), and treat rep counting as hysteresis on a phase signal rather than peak-picking on a raw joint angle.
(1) Heatmap Regression: the backbone emits one $H/4 \times W/4$ map per joint, and the joint location is the argmax cell plus a learned sub-pixel offset; strong spatial inductive bias, heavy output tensor.
(2) Direct Coordinate Regression: the head emits $2K$ numbers directly, or classifies each axis into 1D bins (SimCC); tiny output, no argmax, needs heatmap-style supervision or bin targets during training to localize well.
(3) 3D Lifting From 2D: a temporal network consumes a window of 2D keypoints and predicts root-relative 3D joints; it recovers depth-dependent form errors but inherits every 2D error and adds window latency.

Figure 1: One cheap keypoint pass per frame; everything the product actually says out loud is computed from the buffered keypoint sequence, not from raw pixels.
Clarify Before Designing:
(1) Depth Requirement: which form errors are in scope, and can any of them only be judged from depth (back rounding, hip hinge) rather than from an in-plane 2D ratio?
(2) Compute Budget: what is the target device tier, and how much of the 33 ms frame budget is already spent by camera capture, rendering, and audio?
(3) Framing Control: is the phone propped up with the whole body in view, or handheld with limbs leaving the frame; can the app force a calibration pose first?
(4) Exercise Vocabulary: a closed set of 20 named movements, or open-set counting for anything the user does?
(5) Label Availability: do we have keypoint labels on fitness footage (not just COCO street photos), and do we have per-rep form-error labels from coaches?
(6) Teacher Access: can we run a large server-side pose model offline to generate pseudo-labels and distillation targets?
Login to view more content
September 3, 2026
MSD0040 Automated Retail Checkout
Design a deep learning model for a retail store’s automated checkout system that recognizes items from camera feeds and computes totals without barcode scanning. A customer places items on a counter or walks through a gate, overhead and angled cameras capture the items, and the system must identify each product, count multiples, handle occlusion and stacked items, distinguish similar-looking products (two flavors of the same brand), and produce a receipt in seconds.
The system must handle a large product catalog (thousands of SKUs) that changes as inventory rotates, work under varying lighting and camera angles, and keep the error rate low enough that manual corrections are rare.
How would you design this model? Cover the candidate recognition architectures (object detection vs instance segmentation vs metric-learning retrieval against a product catalog), how you handle the large and rotating SKU catalog, how you count multiples and handle occlusion, how you generate training data for new products with few examples, and how you calibrate confidence to flag uncertain items for manual review.

The Problem: the catalog is huge and rotates weekly, the items overlap and stack, and two flavors of the same brand differ by a color band. The model has seconds to turn pixels into an exact receipt.
Answer
Split the problem into localization and identity, and keep identity out of the network weights. A class-agnostic instance segmenter finds every item instance and its mask, a crop encoder trained with a metric-learning loss maps each instance to a unit embedding, and the SKU is decided by cosine nearest-neighbor search against a catalog index of enrolled prototypes. This makes catalog rotation an indexing operation rather than a retraining project, since a new product needs about 20 enrollment crops and no weight update. Masks (not boxes) drive multiple-instance counting under occlusion, counts are fused across the overhead and angled views, and a calibrated margin gate abstains on ambiguous items so the review lane sees only the genuinely hard ones.
(1) Closed-Set Object Detection: one detector with a $K$ -way softmax head over all SKUs; fastest single pass, but the catalog is baked into the weights.
(2) Instance Segmentation With A Class Head: per-item masks give reliable separation of touching and stacked products, but identity still lives in a fixed classifier.
(3) Class-Agnostic Localization Plus Metric-Learning Retrieval: the chosen design; localization is SKU-independent and identity is a nearest-prototype lookup in an index that can be edited nightly.
(4) Open-Vocabulary Detection With Text Prompts: zero-shot on new categories, but it cannot separate two flavors of one brand, so it serves only as a proposal generator or a cold-start fallback.

Figure 1: Localize anything, then look it up: frames → masks → embeddings → catalog index → fused counts → gate. New SKUs enter through the index, never through the weights.
Clarify Before Designing:
(1) Catalog Shape: how many SKUs, how many are added or retired per week, and how many are near-duplicates within a brand family?
(2) Compute Budget: edge box per lane or shared server GPU, and what is the wall-clock budget from last item placed to receipt shown?
(3) Camera Geometry: fixed calibrated overhead plus angled views on a static counter, or a gate where items must be tracked while moving?
(4) Enrollment Data: does the retailer already supply studio packshots per SKU, and can we photograph new products before they reach the shelf?
(5) Error Asymmetry: what is the cost of an undercount (shrinkage) versus an overcharge (trust), and what per-basket review rate is acceptable?
(6) Presentation Rules: may the interface ask the customer to spread items out, or must the model handle arbitrary piles?
Login to view more content
September 3, 2026
MSD0039 Landmark Recognition Travel
Design a landmark recognition model for a travel app that identifies famous landmarks from user photos and returns relevant historical and visitor information. When a traveler points their phone at the Eiffel Tower, the Colosseum, or a lesser-known temple, the model must identify the landmark in seconds, even from unusual angles, partial occlusion, varying lighting, or when the photo is taken from inside rather than the iconic exterior view.
The system must handle the long tail of tens of thousands of landmarks worldwide, many with only a handful of reference images, and it must gracefully say “I don’t recognize this” rather than confidently misidentifying a similar-looking building.
How would you design this model? Cover the candidate recognition approaches (global descriptor retrieval vs local-feature matching vs deep metric learning with classification), how you handle the long tail of rare landmarks with few training images, how you build the reference index, how you calibrate confidence to enable abstention, and how you evaluate recognition accuracy and false-positive rate.

The Problem: the query is rarely the postcard view, the tail landmark has a handful of reference photos, and a confident wrong name costs more trust than an honest “not sure”.
Answer
Treat this as open-set instance retrieval with abstention, not 100k-way classification. A backbone trained with a margin-based classification loss (ArcFace with sub-centers) produces a compact global descriptor; that descriptor retrieves top candidates from an ANN index over all reference photos; local-feature matching with geometric verification re-ranks those candidates and produces an inlier count. The two pivotal decisions are to keep the final decision in the index rather than in a softmax head, so a new temple is added by indexing seven photos instead of retraining, and to build the confidence signal from geometric inliers plus descriptor margin rather than softmax probability, because that is what makes a calibrated “I don’t recognize this” possible.
(1) Global Descriptor Retrieval: GeM-pooled CNN or ViT embedding, PCA-whitened to 512-d, searched with IVF-PQ over every reference image; fast, memory-light, viewpoint-sensitive.
(2) Local Feature Matching: keypoints and descriptors (DELF, SuperPoint) matched pairwise, then RANSAC for a consistent transform; the inlier count is both the score and the explanation.
(3) Deep Metric Learning With Classification Head: train with ArcFace or sub-center ArcFace over the landmark ID vocabulary, discard the head at inference and keep the embedding for retrieval.
(4) Chosen Hybrid: (3) trains the descriptor, (1) does the cheap recall, (2) verifies the top 100, and a calibrated gate over inliers and margin decides answer versus abstain.

Figure 1: Cheap recall first, expensive verification second, and an explicit gate that is allowed to answer nothing.
Clarify Before Designing:
(1) Inventory And Reference Depth: how many landmarks, and what is the reference-count distribution (median photos per landmark, how many with fewer than 10)?
(2) Compute Budget: server-side with a 1 s budget, or on-device with a fixed memory ceiling and no index round trip?
(3) Metadata: is GPS or EXIF available at query time? A 5 km geo prefilter removes most look-alike confusions for free.
(4) Cost Asymmetry: what precision floor does the product require, and what abstention rate is acceptable to hit it?
(5) Definition Of Recognized: do interiors, statues, details, and reconstructions count as the same landmark, and how are near-identical siblings (replicas, chain temples) labeled?
(6) Data Rights: can web-mined and user-submitted photos enter the index, and can queries be logged for hard-negative mining?
Login to view more content
September 3, 2026
MSD0033 Drowsy Driving Detection
Design a drowsy-driving detection system that uses an in-car camera to monitor the driver and warn before they fall asleep, as offered by Volvo’s Driver Alert, Mercedes Attention Assist, and aftermarket dashcam apps. The system must run on an embedded device or a phone mounted on the dashboard, process the camera feed in real time, and detect early signs of drowsiness (eyelid closure duration, blink rate, head nodding, gaze wandering, micro-sleeps) with few false alarms so drivers do not disable it. It must work across drivers of different appearance, in varying lighting (night, tunnels, sunglasses), and it must keep the video on-device.
How would you design this system? Cover the face and landmark detection, the drowsiness signal extraction (PERCLOS, blink dynamics, head pose), the temporal model that combines signals over a window, the on-device deployment and privacy design, and how you calibrate the alert threshold to minimize false alarms without missing real drowsiness.

The Problem: the dangerous state lasts a few seconds and arrives after minutes of visible warning signs, so the system has to fire during the slow decline, at night, through sunglasses, and almost never on an alert driver.
Answer
The design is a landmark-driven feature pipeline with a learned temporal model: a cheap face detector runs on a fraction of frames with a tracker in between, a landmark network gives eye, mouth, and head-pose geometry per frame, hand-designed physiological features (PERCLOS, blink dynamics, nod and gaze statistics) are accumulated over a sliding 60-second window, and a small recurrent model maps that window to a drowsiness score. The two pivotal decisions are to keep interpretable intermediate signals rather than regressing drowsiness end-to-end from pixels, and to treat the alert threshold as a calibrated operating point on a false-alarms-per-hour axis with hysteresis, because a system that chirps at an awake driver gets unplugged on day two. Everything runs on-device on an int8 quantized graph with a near-infrared sensor, and no frame is ever written to storage or uploaded.
(1) Geometric Landmark Pipeline: face detection plus a landmark network feeding eye aspect ratio, PERCLOS, blink timing, and head pose; features are physiologically interpretable and cheap.
(2) End-to-End Video Model: a 3D CNN or video transformer over eye-region crops predicting drowsiness directly, learning subtle cues no geometric feature encodes, at a much higher data and compute cost.
(3) Hybrid Fusion (chosen): geometric features per frame plus a small GRU over the 60-second window, with per-driver normalization and hysteresis on the output score.
(4) Vehicle-Signal Baseline: steering entropy, lane-position variance, and pedal behavior with no camera at all, which is what Volvo’s original Driver Alert used; it is robust to lighting but needs minutes of degraded driving before it reacts.

Figure 1: Per-frame geometry becomes per-window physiology, and only the last stage makes a decision: pixels never leave the device and never leave RAM.
Clarify Before Designing:
(1) Sensor: RGB rolling-shutter phone camera or a dedicated near-infrared global-shutter module with an 850 nm illuminator, and at what resolution and frame rate?
(2) Compute Target: a phone with an NPU delegate, an automotive SoC, or a low-power MCU, and what sustained thermal and power budget do we get on a 4-hour drive?
(3) Alert Budget: how many false alarms per driving hour before drivers disable the feature, and are we targeting a regulated warning profile (UNECE ADDW-style) or a comfort feature?
(4) Labels: do we have annotated drowsy driving data with KSS self-reports or expert video scoring, and how many distinct drivers, or only simulator and public sets?
(5) Occlusion Requirements: must the system work through sunglasses and face masks, or may it declare “sensing unavailable” and fall back to vehicle signals?
(6) Action Space: what can the alert actually do (chime, haptic seat or wheel, escalating voice prompt, rest-stop suggestion), since the escalation ladder shapes the threshold.
Login to view more content
August 22, 2026