Interview for Machine Learning

Author: admin

MSD0064 Notification Ranking System
Design a multi-source notification ranking system for Meta. Facebook and Instagram send notifications from many heterogeneous sources: friend activity (someone tagged you, liked your post), group updates, event reminders, ads and promotions, and system notifications such as security alerts and policy violations. Each source has a different value to the user and a different urgency, but they all compete for the same notification slots on the user’s phone.
The system must rank notifications across all sources by predicted user value, apply frequency capping to avoid notification fatigue (sending too many notifications causes users to disable notifications entirely), deduplicate redundant notifications (three friends liking the same post should become one notification), and decide which notifications to send, suppress, or batch. Ranking must account for the user’s notification engagement history (do they open notifications, do they dismiss them), the time of day (a notification at 3 AM may annoy), and cross-source value normalization (how does a friend’s like compare to an event reminder). Decisions must be served in real time and adapt to changing user preferences.
How would you design this system? Cover the multi-source candidate generation and deduplication, the ranking model that normalizes value across heterogeneous notification types, the frequency capping and fatigue management strategy, the training label generation from user engagement signals, and how you evaluate notification ranking quality when sending fewer notifications might improve long-term engagement.

The Problem: five sources all believe their event deserves a push, the phone has a handful of slots per day, and the user is one tap away from switching the whole channel off forever.
Answer
The design is a funnel with a constrained decision at the end: per-source candidate generation → dedup and aggregation → one shared multi-task value model → a policy layer that emits send, batch, or suppress. Every source is scored by the same model into the same currency, expected long-term value of occupying one slot, so a friend tag and an event reminder become comparable numbers instead of two teams arguing about priorities. The two pivotal decisions are to make the value target include the cost of channel loss (the probability this send pushes the user toward muting or disabling notifications, multiplied by the value of all future notifications), and to implement frequency capping as an adaptive per-user send threshold rather than a fixed daily count. Hard caps and quiet hours remain as safety rails, plus a bypass lane so security and policy notifications are never traded away for engagement.
(1) Per-Source Candidate Generation: each source emits candidates with a TTL and an eligibility check (privacy, blocking, delivery capability); a candidate that expires before its slot opens is dropped rather than delayed.
(2) Dedup and Aggregation First: candidates are collapsed by an aggregation key of (object, action, time window) before ranking, so three likes on one post become one updatable card and the ranker never scores near-duplicates.
(3) Shared Multi-Task Value Model: one model with heads for open, dismiss, downstream action, and notification disable, producing calibrated probabilities that combine into a single cross-source utility.
(4) Adaptive Volume Control: a per-user threshold acts as the shadow price of a slot, tuned by a controller against a volume budget, with quiet hours, hard caps, and a bypass lane for system alerts.
(5) Batch Lane: candidates with real but sub-threshold value go into a digest that is delivered at the user’s historically responsive hour, instead of being dropped or pushed instantly.
(6) Long-Horizon Labels and Holdbacks: short-term opens train the ranker, but a permanent send-fewer holdback measures whether the policy is buying clicks today at the cost of sessions next month.

Figure 1: One scoring path for all sources: collapse duplicates, score into a shared currency, then let a budgeted policy layer decide send, batch, or suppress, with engagement feeding the next model refresh.
Clarify Before Designing:
(1) Objective Horizon: are we optimizing notification opens this week or app sessions over the next 28 days, and who owns the trade-off when they disagree?
(2) Non-Negotiable Sources: which notifications must always be delivered for legal or safety reasons (login alerts, policy enforcement), and therefore sit outside the ranking budget?
(3) Observable Feedback: which signals do the mobile platforms actually give back (delivery, open, dismiss, mute, system-level disable), and are they comparable across iOS and Android?
(4) Latency Requirement: which sources are genuinely time-critical (live video, event starting in 10 minutes) versus fine to batch by hours?
(5) Scale and Budget: how many candidates per user per day, what is the target send volume, and what infrastructure cost per candidate is acceptable?
(6) Experiment Capacity: can we hold out a permanent low-volume population long enough to measure long-term effects, and how large may it be?
Login to view more content
September 10, 2026
MSD0063 Image Copyright Detection
Design an image copyright violation detection system for Meta. When users upload images to Facebook or Instagram, the platform must detect whether an uploaded image infringes on a copyrighted work (a professional photograph, a movie still, a book cover, artwork) before or shortly after publication. Unlike audio or video fingerprinting, which leans on perceptual hashes for near-duplicate detection, image copyright detection must also handle crops, resizes, color adjustments, partial overlays, and collages where a copyrighted image is one component of a larger composition.
The system must embed both uploaded images and a reference database of registered works into a shared visual feature space, retrieve nearest neighbors with approximate nearest neighbor (ANN) search, and classify whether a match constitutes infringement. It must be robust to adversarial perturbations (a user adding noise or a fake watermark to evade detection), handle partial infringement where only 20 percent of the image is copyrighted, and process hundreds of millions of uploads per day at low latency. Copyright owners must be able to register their works and receive alerts when matches are found.
How would you design this system? Cover the visual embedding architecture (contrastive learning vs triplet loss vs CLIP-style vision encoder), the ANN retrieval index (FAISS vs ScaNN vs HNSW), the infringement classification strategy (similarity threshold vs learned classifier vs human review), how you handle adversarial robustness and partial infringement, and how you evaluate detection precision and recall when ground-truth labels require legal judgment.

The Problem: the same protected photograph arrives cropped, recolored, noised, and buried inside a collage, and the gate has milliseconds and no pixel-exact copy to compare against.
Answer
The design is a retrieve-then-verify matcher built on a copy-detection embedding space, not a semantic one. Every upload is hashed for exact republishes, split into a global view plus a handful of region crops, embedded by a self-supervised copy-detection descriptor trained with aggressive transform augmentation, and queried against a sharded FAISS index of registered works. Retrieved candidates go through geometric verification with local features and a calibrated classifier that scores similarity, matched area, and registry metadata together. The two pivotal decisions are to query with regions rather than one whole-image vector, which is what makes a 20 percent collage insert detectable, and to route scores into enforcement bands so only high-confidence, high-coverage matches act automatically while the ambiguous middle becomes a human review queue.
(1) Copy Descriptor Not CLIP: train a self-supervised descriptor (SSCD-style, with entropy regularization for comparable scores across images) on copy transforms; a CLIP semantic space retrieves same-topic photos and destroys precision.
(2) Multi-Region Queries: query with 1 global view plus 4 quadrants plus 4 half-scale crops, so at least one region is dominated by the protected work even in a dense collage.
(3) Two-Tier Index: a PDQ hash lookup catches pixel-near republishes in microseconds; a FAISS IVF-PQ index over roughly 400M reference vectors handles transformed copies.
(4) Retrieve Then Verify: ANN top-k feeds local-descriptor matching with RANSAC, and a learned classifier fuses inlier count, matched area fraction, descriptor score, and registration signals.
(5) Banded Enforcement: auto-allow, label plus owner alert, hold for human review, or auto-block; the top band alone acts without a human, and every action has an appeal path.
(6) Defense In Depth: adversarial augmentation during training plus three independent signal families (hash, global descriptor, local features) means an evader must break all of them at once.

Figure 1: Two paths meet at the index: uploads are embedded per region and searched, while registered works flow through ownership checks and the same encoder into the reference shards.
Clarify Before Designing:
(1) Timing And Surface: do we block before publication or detect within minutes after, and does that differ for feed posts, Stories, and paid ads?
(2) Registry Shape: how many registered works, growing how fast, and is there a premium catalog (studios, stock agencies) that deserves a stricter path than the long tail?
(3) Enforcement Policy: is the action block, label, claim for monetization, or notify-only, and who owns the appeals and counter-notice flow?
(4) Budget: uploads per day, peak QPS, allowed added latency per upload, and cost ceiling per thousand uploads?
(5) Ground Truth: can legal reviewers label a gold set, at what weekly volume, and can licensed reposts be distinguished from infringing ones in that data?
(6) Retroactivity: when a new work is registered, must we rescan historical uploads, and how far back?
Login to view more content
September 10, 2026
MSD0062 Ads Ranking System
When a user opens Facebook or Instagram, Meta’s ad system has a few tens of milliseconds to decide which ads fill the slots in the feed, chosen from millions of active campaigns. Unlike organic ranking, the score must combine the advertiser’s bid with predicted click-through rate (pCTR), predicted conversion rate (pCVR), and ad quality signals, so the platform maximizes long-term revenue without degrading the user experience.
The system runs a real-time auction per impression, calibrates predicted probabilities so that a pCTR of 0.05 really means 5 percent of similar impressions click, respects advertiser budgets and frequency caps, and suppresses low-quality or misleading ads. It must serve in under 50 milliseconds per impression, handle billions of impressions per day, and retrain continuously on fresh click and conversion data.
How would you design this system? Cover the ad candidate retrieval architecture, the CTR/CVR prediction model (logistic regression vs GBDT vs deep learning), the calibration strategy (Platt scaling vs isotonic regression vs temperature scaling), the auction mechanism (first-price vs second-price vs Vickrey-Clarke-Groves), budget pacing and frequency capping, and how you evaluate ads ranking quality beyond revenue.

The Problem: one feed slot, millions of eligible campaigns with different bids and budgets, and under 50 milliseconds to decide who fills it and what they pay.
Answer
The design is a cascaded funnel feeding a per-impression auction: targeting filters plus approximate nearest-neighbour retrieval cut millions of campaigns to about ten thousand, a cheap ranker keeps a few hundred, a heavy multi-task deep model predicts pCTR and pCVR for those, and the auction ranks by expected value per impression. Two decisions dominate everything else. First, the probabilities must be calibrated in absolute terms, not merely well ordered, because the bid is multiplied by them and prices are derived from them. Second, budgets and fatigue are handled by a closed-loop pacing controller and serving-time frequency counters rather than by the model, so spend stays smooth and the ranking model keeps a single clean objective.
(1) Cascaded Retrieval: hard targeting and budget-eligibility filters run first, then embedding retrieval over the remaining inventory; the funnel is $10^{7} \to 10^{4} \to 500 \to 1$ per request.
(2) Multi-Task Deep Ranker: one network with shared sparse embeddings and separate pCTR and pCVR heads, replacing the historical logistic-regression-on-GBDT-leaves stack while keeping trees for the light ranker.
(3) Calibration As A First-Class Stage: per-slice isotonic regression on recent traffic, monitored by expected calibration error, because relative probability biases across ads or slices distort ranking and second-price payments.
(4) Total-Value Auction: rank by bid times predicted action rate minus a quality penalty, bill second-price style so the winner pays the minimum needed to keep its position.
(5) Pacing And Caps Outside The Model: a PID controller adjusts a per-campaign bid multiplier toward the daily budget curve, and a low-latency counter store enforces frequency caps at serving time.
(6) Continuous Retraining: impressions joined with clicks and delayed conversions stream into hourly incremental refreshes, with an exploration budget so new ads can earn data.

Figure 1: The serving funnel: each stage uses a progressively more expensive model on far fewer candidates (10M to 10k to 500 to 1), and every delivered impression flows back as training data.
Clarify Before Designing:
(1) Bid Types: are advertisers bidding per click, per conversion, or on value-based lookalike objectives, and do we owe them an outcome guarantee such as cost-per-result?
(2) Slot Structure: one ad per request or several slots per session, and are ads interleaved with organic content (which makes the opportunity cost an organic post, not another ad)?
(3) Latency And Slot Count: is the 50 ms budget end to end including retrieval, or only model inference, and how many ads must be scored inside it?
(4) Conversion Signal: what is the attribution window and how much conversion feedback is delayed or lost to privacy restrictions on third-party signals?
(5) User Experience Constraint: what is the hard limit on ad load and negative feedback (hide-ad rate, reports), and is it a constraint or part of the objective?
(6) Advertiser Mix: what fraction of spend comes from a few large advertisers versus a long tail of small ones, since that decides how much cold-start exploration matters?
Login to view more content
September 10, 2026
MSD0061 Nearby Place Recommendation
Design a nearby place recommendation system for Meta. When a user opens Facebook or Instagram on their phone, the app can recommend nearby places (restaurants, stores, parks, events) that the user might be interested in right now. Unlike a general recommendation system, the user’s location is constantly changing as they move, and the recommendation must reflect their current or predicted position, not where they were 10 minutes ago.
The system must retrieve candidate places using geohash or H3 spatial indexing, rank them by relevance (place popularity, user preferences, category match, distance, time of day), and serve results in under 200 milliseconds on a mobile device. It must handle cold start for new places with no visit history, handle users who just moved to a new city, and avoid recommending places that are permanently closed or temporarily unavailable. The recommendation must also account for context: a user looking for lunch at noon on a weekday has a different intent than the same user looking for dinner at 8 PM on a Saturday.
How would you design this system? Cover the geospatial retrieval architecture (geohash vs H3 vs quadtree), the ranking model that combines location distance with personalization features, how you handle real-time position updates and predict future position, how you handle cold start for new places and new users, and how you evaluate recommendation quality when the right recommendation depends on unobserved user intent.

The Problem: the user keeps moving, hundreds of places sit within a few blocks, and the intent behind the app open (lunch, coffee, a park) is never observed, yet the first card has to appear in under 200 ms.
Answer
The design is a two-stage geo-recommender. Retrieval pulls a few thousand candidates from an H3 resolution-9 index using a k-ring around the user’s predicted position, an availability gate drops anything closed, and a multi-task ranker scores tap, save, and visit probability from user, place, distance, and context features. Two decisions carry the design. First, retrieval keys on H3 cells rather than geohash strings, because a k-ring is a genuine radius while geohash neighbors sit at uneven distances and cells stretch with latitude. Second, the top few hundred scored candidates are cached on the device and re-ranked locally as the user moves, so walking a block costs a local re-sort instead of a network round trip. Context enters as features rather than as a separate model: hour-of-week crossed with category affinity, and travel time instead of raw haversine distance, because people walk 200 metres for coffee and drive 5 km for dinner.
(1) H3 K-Ring Retrieval: places are indexed by resolution-9 cell, and a query fetches the $k=2$ ring (19 cells, about 2 square km) with k widened automatically in sparse areas until a candidate floor is met.
(2) Predicted Position, Not Last Fix: the query center is the last GPS fix advanced by velocity times expected staleness, with the cell resolution chosen by detected travel mode.
(3) Prefetch Plus On-Device Re-Rank: the server returns roughly 200 scored candidates with their coordinates; the device re-sorts by exact distance every few seconds, which also keeps precise GPS off the server.
(4) Availability Gate Before Ranking: permanent closures, current hours, and temporary-closure signals are applied as a hard filter in retrieval, never as a soft ranking penalty.
(5) Cold Start Priors With Bounded Exploration: new places inherit a prior from category, chain, and cell-level popularity, and reserve one exploration slot per result page with impression-capped bandit selection.

Figure 1: Serving path along the top row, offline loop along the bottom: one cheap spatial fetch, a hard availability gate, one scoring pass, and a cached candidate set the device can re-rank without another round trip.
Clarify Before Designing:
(1) Surface And Cadence: is this a dedicated Nearby tab the user pulls open, or an inline unit rendered on every feed load, and does it refresh continuously as the user walks?
(2) Location Permission: do we have precise foreground location and consent, or only coarse city-level location for most users, and is background position allowed?
(3) Inventory Quality: how complete are hours, category, and closure signals in the Places graph, and how stale is the worst market?
(4) Objective: are we optimizing discovery engagement, confirmed real-world visits, or downstream ad and business-page value, and which one settles ties?
(5) Geography And Scale: global coverage including sparse rural cells, or top metros first, and what peak QPS should the retrieval tier absorb?
(6) Cost Asymmetry: how much worse is recommending a closed restaurant than missing a good open one, since that ratio sets the availability gate’s aggressiveness?
Login to view more content
September 10, 2026
MSD0060 RLHF Alignment Pipeline
Design an RLHF alignment pipeline for aligning a large language model with human preferences. A foundation model team must align a model so that it follows instructions, refuses harmful requests, and produces helpful, harmless, and honest responses.
The alignment must use reinforcement learning from human feedback, but the team must choose between PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and GSPO (Group Sequence Policy Optimization) as the alignment algorithm. Each has different sample efficiency, training stability, and compute requirements. The system must train a reward model from human preference data, run the alignment loop at scale on a distributed cluster, and produce a model that passes safety evaluations while keeping its capability.
How would you design this system? Cover the reward modeling architecture, the candidate RL algorithms (PPO vs GRPO vs GSPO) and their trade-offs, how you handle reward hacking and mode collapse, how you distribute the alignment training across GPUs, and how you evaluate alignment quality.

The Problem: humans can only tell you which of two whole responses they prefer, but gradient descent needs a scalar objective and a stable update signal at every token of every rollout.
Answer
The stack is the standard three stages: an instruction-tuned SFT checkpoint, a Bradley-Terry reward model trained on human preference pairs, and an on-policy RL loop that maximizes reward minus a KL penalty against the frozen SFT reference. Everything above the update rule is shared, so the real design question is the RL objective. PPO learns a separate value network to supply per-token baselines, while GRPO and GSPO delete the critic and use the mean reward of a group of sampled responses instead, removing one trainable model copy from memory. GSPO goes further and computes the importance ratio at the sequence level, matching the granularity of the reward itself, which is what keeps training stable on MoE policies and long rollouts. My default is group-relative advantages with a sequence-level ratio, a KL anchor plus reward-model ensembling to hold back reward hacking, and PPO kept in reserve only where a dense token-level value estimate genuinely earns its extra trainable model.
(1) Shared Substrate: SFT policy, preference-trained reward model, and a frozen reference for the KL anchor; all three candidates optimize the same penalized reward.
(2) PPO: actor-critic; a learned value network produces per-token advantages through GAE, and the importance ratio is clipped once per token.
(3) GRPO: no critic; sample $G$ responses per prompt and standardize their rewards into one advantage per response, still clipped per token.
(4) GSPO: same group baseline, but the importance ratio and the clip act on the whole length-normalized sequence, matching the sequence-level reward.

Figure 1: One alignment loop: preferences train the reward model, reward minus a KL anchor scores group rollouts, and only the update box changes between PPO, GRPO, and GSPO.
Clarify Before Designing:
(1) Policy Architecture: dense or sparse MoE, and at what size? Routing volatility in MoE is the single strongest argument for a sequence-level ratio.
(2) Reward Source: learned reward model only, or are parts of the traffic verifiable (unit tests, math checkers, format rules) so a rule-based reward can replace the model?
(3) Sequence Length: 1k-token chat replies or 32k-token reasoning traces? Long rollouts multiply token-level ratio variance and dominate step time.
(4) Compute Budget: how many GPUs, and does a second trainable critic-sized model fit next to the policy after sharding?
(5) Preference Data: how many pairs exist, what is the annotator agreement rate, and is there a recurring human evaluation loop or only an LLM judge?
(6) Release Bar: what refusal rate on red-team prompts and what maximum capability regression (the alignment tax) gate the release?
Login to view more content
September 10, 2026
MSD0059 Warehouse Robot Human Avoidance
Design the human detection and avoidance system for warehouse robots at Amazon Robotics. Thousands of mobile robots share the fulfillment center floor with human workers. Each robot must detect humans in its path, predict their trajectory, and adjust its motion to keep a safe distance by slowing down, rerouting, or stopping entirely when a human is too close.
The system must run in real time on onboard sensors (cameras, LiDAR, bump sensors), handle occlusions such as a worker stepping out from behind a storage pod, distinguish humans from shelves, bins, and other robots, and fail safely. A robot that fails to detect a human is a safety incident, not just a performance bug. It must also survive challenging lighting, reflective floors, and the fact that humans behave unpredictably.
How would you design this system? Cover the perception architecture (sensor fusion, human detection model), the trajectory prediction model for human motion, the safety-aware motion planning strategy, how you handle occlusions and sensor failures, and how you test and validate the system for safety-critical deployment.

The Problem: the robot cannot see the worker until roughly a meter before the corner, both are moving at about 1.5 m/s, and the cost of being wrong once is an injury rather than a dropped pod.
Answer
The design is a two-channel safety architecture. A certified channel (a safety-rated 2D scanner with configured protective and warning fields, backed by a contact bumper) owns the stop authority and is hardwired to the brakes, while a learned channel (360-degree camera plus 3D LiDAR fused in a bird’s-eye-view grid, then tracking and trajectory prediction) owns early, smooth slowdowns and routing. The neural stack is never the thing that guarantees the human is not hit, because no learned detector can be argued to the $10^{-9}$ per-encounter failure rate a large fleet needs. The two pivotal decisions are to size the protective field from stopping physics rather than from model confidence, and to plan against a worst-case reachable set for occluded and unpredictable humans while using the learned predictor only to make motion smoother inside that envelope.
(1) Two Independent Channels: a safety-rated scanner and bumper stop the robot through certified logic; the ML stack only lowers a speed limit that an arbiter takes the minimum of.
(2) Multi-Modal BEV Perception: RGB and LiDAR fused into a shared bird’s-eye grid gives metric geometry plus semantics, so a person can be separated from a pod, a bin, or another robot.
(3) Unknown Means Human: any occupied cell the classifier cannot label confidently is treated as a person, so odd poses (crouching, lying, on a ladder) never fall through the class taxonomy.
(4) Visibility-Limited Speed: the speed cap is set so the robot can stop before the boundary of the nearest occluded region, which turns blind corners into a planning constraint instead of a detection problem.
(5) Designed Degraded Modes: every sensor fault maps to a lower speed cap or a controlled park, never to continued nominal driving.
(6) Validation Staircase: log replay, simulation, hardware-in-the-loop latency tests, physical stop tests against the standard, then a staged rollout with shadow logging.

Figure 1: Two lanes into one arbiter: the learned channel proposes a comfortable speed, the certified channel can always take it to zero.
Clarify Before Designing:
(1) Zone Type: is this a fenced robotic field that workers enter only under a lockout procedure, or a fully mixed aisle where people walk all shift alongside the fleet?
(2) Speed and Mass: what is the maximum robot speed and the loaded mass, since braking distance and impact energy set the protective field size?
(3) Acceptance Standard: which standard gates deployment (ISO 3691-4, ANSI/RIA R15.08) and what performance level is required for the stop function?
(4) Cost Budget: what sensor bill of materials and power budget per robot is acceptable, given that a $500 sensor becomes $5M across 10,000 robots?
(5) Human Cooperation: may workers wear beacons or high-visibility gear the fleet can read, or must the system assume an unequipped visitor in street clothes?
(6) Throughput Price: how many nuisance stops per robot-hour is operations willing to accept in exchange for a larger detection margin?
Login to view more content
September 10, 2026
MSD0058 Ride-Hailing Matching System
Design the ride-hailing matching and pricing system for a platform like Uber. When a rider requests a ride, the system must find the nearest available drivers, estimate time of arrival, compute a dynamic price based on supply and demand, and match the rider to a driver, all within seconds. The platform operates in hundreds of cities with millions of drivers and riders, and the matching must account for driver availability, traffic conditions, driver preferences (direction, distance), and rider wait time.
The ETA model must predict travel time from the driver to the pickup point and from pickup to destination, accounting for real-time traffic, road network, and time of day. The surge pricing model must balance supply and demand in real time while remaining transparent to riders. The system must handle peak hours when demand far exceeds supply, and it must gracefully degrade when GPS or network connectivity is poor.
How would you design this system? Cover the geospatial indexing architecture (quadtree vs H3 vs geohash), the driver-rider matching algorithm, the ETA prediction model, the dynamic pricing strategy, and how you handle peak load and graceful degradation.

The Problem: the nearest car on the map is rarely the nearest car in traffic, and the right price changes minute by minute. The system must pick a driver, an ETA, and a price before the rider closes the app.
Answer
The design is a city-sharded real-time marketplace built on three layers: an in-memory geospatial index that answers “who is near this point” in tens of milliseconds, a batched assignment solver that matches a small window of requests and drivers jointly, and two learned models (ETA and demand/supply forecasting) that supply the numbers the solver and the pricing engine consume. The pivotal decisions are to index driver locations with H3 hexagons and retrieve candidates by k-ring expansion rather than radius scans, to replace greedy first-come dispatch with a min-cost bipartite match over a 2-5 second window, and to compute ETA as a routed baseline plus a learned residual so the physics and the learning each do what they are good at. Everything sits behind a degradation ladder, so bad GPS or a dead model server produces a worse ride rather than a hard failure in most cases.
(1) H3 Hexagonal Index: driver positions live in a sharded in-memory store keyed by H3 cell at resolution 8-9, and candidate retrieval expands k-rings until enough drivers are found, so neighbor distance is uniform in every direction.
(2) Batched Bipartite Matching: requests accumulate for a few seconds per cell cluster, then a min-cost assignment over pickup ETA, accumulated rider wait, and predicted driver acceptance beats greedy nearest-car dispatch on total wait.
(3) Two-Stage ETA: route the road graph with live segment speeds for the physical baseline, then a learned residual model corrects for pickup friction, time of day, and driver behavior, with a quantile head for the pessimistic bound shown to riders.
(4) Capped Cell-Level Surge: the multiplier comes from a smoothed demand/supply ratio per cell over a short window, is capped and hysteresis-smoothed to stop oscillation, and is folded into a single upfront price the rider sees before confirming.
(5) City-Level Isolation: each city is an independent dispatch shard with its own index, models, and pricing state, so a stadium letting out in one market cannot stall dispatch in another.
(6) Degradation Ladder: map matching and dead reckoning for noisy GPS, cached-speed ETA when the model server is unhealthy, greedy dispatch when the solver is down, and last-known-cell routing during connectivity loss.

Figure 1: The dispatch path: request → cell lookup → candidate filter → ETA scoring → batched assignment → dispatch and quote, with the location stream, road graph, and surge engine feeding each stage from below.
Clarify Before Designing:
(1) Product Scope: single rider-to-driver trips only, or pooled and scheduled rides too, since pooling turns assignment into a vehicle-routing problem with a different solver?
(2) Latency Budget: how long between tap and price quote, and how many seconds may a request sit in a batching window before the rider perceives a stall?
(3) Objective Weights: are we minimizing rider wait, maximizing completed trips, or maximizing marketplace earnings, and how heavily does driver fairness and utilization count?
(4) Pricing Constraints: regulatory multiplier caps, emergency policies, upfront fixed price versus metered fare, and what transparency the rider is legally owed?
(5) Data Coverage: do we have historical GPS traces and a maintained road graph in every city, or are we launching cold-start markets with no traffic history?
(6) Partition Posture: during a network partition, do we prefer to refuse dispatch or risk a double assignment, given the safety and trust cost of two drivers arriving?
Login to view more content
September 10, 2026
MSD0057 LLM Training Data Pipeline
Design a distributed data preprocessing pipeline for training large language models at Amazon. The Bedrock team needs to prepare training corpora from petabytes of raw text (web crawls, code repositories, internal documents, licensed datasets) by tokenizing, deduplicating, filtering for quality, removing harmful content, and sharding into training-ready chunks.
The pipeline must process data arriving in S3 and other stores, handle near-duplicate detection at scale (MinHash, SimHash, exact hash), apply quality classifiers to remove low-quality text, run PII and safety filters, and output sharded tokenized sequences ready for distributed training. It must process hundreds of terabytes per run, be resumable from checkpoints, and produce reproducible datasets with full provenance tracking.
How would you design this system? Cover the ingestion architecture (batch S3 versus streaming), the deduplication strategy (exact versus near-duplicate at scale), the quality and safety filtering pipeline, the tokenization and sharding strategy, and how you ensure reproducibility and provenance across pipeline runs.

The Problem: half a petabyte of messy text arrives per run, most of it duplicated, boilerplate, unsafe, or machine-generated, and the training cluster needs a byte-identical, provenance-tracked shard set within days.
Answer
The design is an immutable, content-addressed data lake on S3 with a cheap-to-expensive filter cascade running as batch Spark or Ray jobs. Every stage reads a frozen snapshot, appends scores instead of deleting rows, and writes a new layer whose path is keyed by the hash of its code plus config, which is what makes runs reproducible and resumable. Ingestion is batch over snapshots rather than streaming, because deduplication is a global operation that needs a full-corpus view. Two decisions are pivotal. First, three-tier deduplication (exact document and line hashes, then MinHash with LSH banding for near duplicates, then substring dedup on high-risk subsets) runs before the model-based quality classifier, so the expensive scorer sees roughly a third of the documents. Second, tokenization is deferred to the last stage so the durable artifact stays text, which means a tokenizer or vocabulary change costs a few hours of CPU rather than a full pipeline rerun.
(1) Immutable Snapshot Ingestion: crawls and repos land in a bronze layer as append-only, content-addressed objects with a snapshot manifest; streaming is used only to buffer arrivals, never to feed the transforms.
(2) Three-Tier Deduplication: exact SHA-256 on documents and lines kills boilerplate, MinHash plus LSH bands catch near duplicates, and union-find keeps one canonical document per cluster.
(3) Score, Never Delete: heuristics, quality classifiers, and safety and PII detectors write scores into columnar tables, and a single declarative policy file turns scores into keep/drop decisions per run.
(4) Cost-Ordered Cascade: language ID and cheap heuristics first, then dedup, then the learned quality classifier, then safety and PII, so each expensive model runs on the smallest surviving corpus.
(5) Tokenize Last, Pack Fixed: BPE tokenization emits packed 8192-token sequences with document-boundary masks into roughly 40,000 one-gigabyte shards plus an index for deterministic, resumable loading.
(6) Provenance And Resumability: a per-document lineage table plus a partition-level task ledger give a Merkle dataset version, replay of only failed partitions, and targeted rebuilds for takedown requests.

Figure 1: The cascade: cheap normalization and dedup shrink the corpus before any learned model runs, tokenization is the final derived stage, and every stage writes lineage into the provenance store that defines the dataset version.
Clarify Before Designing:
(1) Token Target: how many training tokens does the model plan need, and for how many epochs? A 10T-token target with single-epoch training lets us filter aggressively; a 2T-token budget with repetition does not.
(2) Licensing And Compliance: which sources are licensed for training, what is the right-to-delete SLA, and are there regional data-residency constraints on where the pipeline may run?
(3) Cadence: is this a one-off corpus build or a monthly refresh on new crawl snapshots? Incremental refresh forces us to persist MinHash signatures for cross-run dedup.
(4) Tokenizer Stability: is the vocabulary frozen, or still being trained? That decides whether tokenized shards or filtered text is the durable artifact.
(5) Mixture Requirements: what are the target per-domain and per-language proportions (code, multilingual, internal docs), since mixture weights drive per-slice keep-rate floors.
(6) Budget: what wall-clock and dollar budget per run, and how many proxy-model ablations can we afford to validate a corpus change?
Login to view more content
September 10, 2026
MSD0056 Product Search Ranking
Design a product search ranking system for Amazon. When a customer types a query, the search engine must retrieve relevant products from a catalog of hundreds of millions of items and rank them so that the most relevant and purchasable products appear at the top.
The ranking must account for query intent (browsing vs buying), product relevance (text match, category, attributes), business signals (price, availability, seller rating, conversion rate), and personalization (past purchases, browsing history). The system must serve results in under 100 milliseconds, handle millions of queries per day, and support continuous experimentation: every ranking change is validated through online A/B tests measuring not just click-through rate but purchase conversion, revenue, and long-term customer satisfaction.
How would you design this system? Cover the query understanding and product retrieval architecture, the ranking model architecture (pointwise vs pairwise vs listwise), how you handle cold-start products with no click history, how you design and run online A/B experiments at scale, and how you evaluate ranking quality beyond offline metrics.

The Problem: a few typed words must be turned into an ordered page drawn from hundreds of millions of items, in under 100 milliseconds, where “good” means relevant and likely to be bought and kept.
Answer
The design is a cascade: query understanding, hybrid retrieval that unions a lexical inverted index with a two-tower embedding ANN, a cheap L1 ranker that cuts ten thousand candidates to five hundred, and an expensive listwise learning-to-rank model that orders the survivors. The pivotal decisions are the training label and the experiment loop. Labels are graded purchase-weighted utility with position-bias correction, not raw clicks, because a click-optimized ranker sells cheap junk. Every change ships through interleaving for shortlisting and session-randomized A/B tests with revenue and return-rate guardrails for the launch decision. Cold-start products stay competitive through content-only priors plus reserved exploration slots, so a new item is far less likely to remain stuck at zero impressions just because it has never been shown.
(1) Query Understanding: spell correction, intent classification (broad browse vs specific buy), and attribute extraction (brand, size, color) that become hard filters and ranking features.
(2) Hybrid Retrieval: BM25 over an inverted index for exact tokens such as part numbers, unioned with a two-tower ANN trained on query-purchase pairs for vocabulary mismatch.
(3) Two-Stage Ranking: a microsecond-per-document L1 filter feeds a LambdaMART or multi-task DNN L2 model that can afford roughly 60 microseconds per document.
(4) Purchase-Weighted Objective: graded gains (purchase > add-to-cart > click) with inverse propensity weighting for position bias, blended with calibrated business signals.
(5) Cold Start By Design: content embeddings make new items retrievable on day one; hierarchically smoothed category priors plus exploration slots give them their first impressions.
(6) Experimentation Platform: overlapping experiment layers, interleaving for shortlisting, session-randomized A/B for launches, and a permanent long-term holdback.

Figure 1: The cascade: each stage spends more compute on fewer documents, and the logged page feeds the next day’s training set.
Clarify Before Designing:
(1) Traffic And Catalog: queries per second at peak, catalog size, and how many catalog items change price or stock per day?
(2) Success Definition: is the primary metric purchase conversion, revenue per session, or long-term retention, and who owns the trade-off when they disagree?
(3) Latency And Cost: is the 100 ms budget end-to-end at p99 including network, and what GPU or CPU budget per query is acceptable?
(4) Query Mix: what fraction of traffic is head queries versus unseen tail queries, and how much is navigational (a known brand or ASIN)?
(5) Personalization Scope: is signed-in history available at query time, and are there privacy or regional constraints on using it?
(6) Marketplace Constraints: must sponsored slots, seller fairness, or new-seller exposure be guaranteed inside the same page?
Login to view more content
September 10, 2026
MSD0055 Netflix Watch Pause Prediction
Build a system that predicts whether a Netflix user stopped watching a show because they are tired of it or just taking a break, using a deep sequence model. When a user stops a show mid-episode or mid-season, Netflix must distinguish between two very different states: a user who is bored or dissatisfied and is likely to churn, and a user who simply went to bed or to work and will resume tomorrow. The distinction drives different actions: a bored user needs a new recommendation or a nudge, while a sleeping user should be left alone.
The system must model the user’s entire viewing history as a sequence (what they watched, when, for how long, where they stopped, whether they resumed), account for time-of-day and day-of-week patterns (stopping at 11 PM on a Tuesday is different from stopping at 3 PM on a Saturday), handle shows with naturally slow episodes that cause legitimate pauses, and predict the probability of resumption within 24 hours. The label is inherently ambiguous, because you only observe whether the user resumed, not why they stopped, so the system must infer intent from behavioral patterns.
How would you design this system? Cover the sequence model architecture (RNN/LSTM vs Transformer vs temporal convolution), the feature engineering from viewing history (session patterns, completion rates, time gaps, device switches), how you handle the label ambiguity (resumption as a proxy for interest, survival analysis for time-to-resume), how you distinguish seasonal pause patterns from genuine disengagement, and how you evaluate the system when ground-truth intent is never observed.

The Problem: the same paused screen at 11 PM can mean “back tomorrow evening” or “never again”, and the platform only ever observes which one happened, never why. Design the model that reads the viewing history and tells the two apart before the 24 hours are up.
Answer
The design is a survival model over stop events: every time a user stops watching a title, a compact causal transformer reads the last few hundred play/stop events plus the stop-time context and outputs hourly resumption hazards over the next 24 hours, from which the probability and the expected time of resumption both fall out. Three decisions carry the design. First, the label is time-to-resume, not a binary “resumed or not”, so a user who resumes at 8 PM the next day and a user who never returns are separated by the shape of the curve rather than by a single threshold. Second, the model sees the stop relative to the user’s own habits and the title’s own pacing: a stop at 11 PM by a habitual 11 PM stopper, at an episode where 40% of the cohort also paused, is scored as ordinary. Third, the prediction is decoupled from the action: a calibrated probability feeds a nudge policy that is evaluated against a permanent randomized holdout, because intent is never observed and only downstream behavior can validate the system.
(1) Event-Level Sequence Tokens: each play/stop event becomes one token carrying title, device, local hour, weekday, watched fraction, stop position (mid-scene, before credits, autoplay cancelled), and gap to the previous event; the last 512 tokens form the input.
(2) Survival Head Over 24 Hourly Bins: a discrete-time hazard head predicts $h_k$ for each hour, giving P(resume within 24 h) and the resume-time distribution in one pass, with a second 7-day horizon for weekend-only viewers.
(3) Habit and Pacing Normalization: the stop is described by its deviation from the user’s typical stop hour and typical gap duration and by the cohort drop-off curve at that episode, so seasonal and slow-episode pauses stop looking like boredom.
(4) Compact Causal Transformer: 4 layers at width 128 over 512 tokens, about 1 GFLOP per stop, with a GRU as the cheap fallback and a GBDT on aggregate features as the baseline everything must beat.
(5) Decoupled Action Policy: the model emits calibrated probabilities; a separate policy decides “leave alone”, “reorder the homepage”, or “nudge tonight” using the cost asymmetry between annoying a sleeper and losing a bored member.
(6) Randomized Holdout by Design: a permanent no-action slice keeps labels uncontaminated by the nudges and is the only place where “did we read intent right” can be measured.

Figure 1: The end-to-end path: one scored stop event, one calibrated resumption curve, one policy decision, and a no-action feedback loop that keeps the labels honest.
Clarify Before Designing:
(1) Action and Timing: what does the prediction drive, a homepage reorder at the next app open or a same-evening push notification, and how quickly after the stop must the score exist?
(2) Label Definition: is “resumed” the same title within 24 hours, any title, or the same episode; and does starting a different show count as disengagement from the show or as engagement with the service?
(3) Scale and Budget: roughly how many stop events per second, how long a history per profile is available, and is scoring allowed to be asynchronous (minutes) rather than synchronous?
(4) Signal Availability: do we have stop position within the episode, device identity, autoplay-countdown cancellations, and per-title metadata such as episode length and release date?
(5) Cost Asymmetry: how expensive is a wrong nudge (notification fatigue, opt-outs) versus a missed bored user (a lost chance to re-engage before churn)?
(6) Household Sharing: are profiles reliably one person, or must the model tolerate a household where the kid stops and the parent resumes?
Login to view more content
September 8, 2026