Design a product search ranking system for Amazon. When a customer types a query, the search engine must retrieve relevant products from a catalog of hundreds of millions of items and rank them so that the most relevant and purchasable products appear at the top.
The ranking must account for query intent (browsing vs buying), product relevance (text match, category, attributes), business signals (price, availability, seller rating, conversion rate), and personalization (past purchases, browsing history). The system must serve results in under 100 milliseconds, handle millions of queries per day, and support continuous experimentation: every ranking change is validated through online A/B tests measuring not just click-through rate but purchase conversion, revenue, and long-term customer satisfaction.
How would you design this system? Cover the query understanding and product retrieval architecture, the ranking model architecture (pointwise vs pairwise vs listwise), how you handle cold-start products with no click history, how you design and run online A/B experiments at scale, and how you evaluate ranking quality beyond offline metrics.
The Problem: a few typed words must be turned into an ordered page drawn from hundreds of millions of items, in under 100 milliseconds, where “good” means relevant and likely to be bought and kept.
Answer
The design is a cascade: query understanding, hybrid retrieval that unions a lexical inverted index with a two-tower embedding ANN, a cheap L1 ranker that cuts ten thousand candidates to five hundred, and an expensive listwise learning-to-rank model that orders the survivors. The pivotal decisions are the training label and the experiment loop. Labels are graded purchase-weighted utility with position-bias correction, not raw clicks, because a click-optimized ranker sells cheap junk. Every change ships through interleaving for shortlisting and session-randomized A/B tests with revenue and return-rate guardrails for the launch decision. Cold-start products stay competitive through content-only priors plus reserved exploration slots, so a new item is far less likely to remain stuck at zero impressions just because it has never been shown.
(1) Query Understanding: spell correction, intent classification (broad browse vs specific buy), and attribute extraction (brand, size, color) that become hard filters and ranking features.
(2) Hybrid Retrieval: BM25 over an inverted index for exact tokens such as part numbers, unioned with a two-tower ANN trained on query-purchase pairs for vocabulary mismatch.
(3) Two-Stage Ranking: a microsecond-per-document L1 filter feeds a LambdaMART or multi-task DNN L2 model that can afford roughly 60 microseconds per document.
(4) Purchase-Weighted Objective: graded gains (purchase > add-to-cart > click) with inverse propensity weighting for position bias, blended with calibrated business signals.
(5) Cold Start By Design: content embeddings make new items retrievable on day one; hierarchically smoothed category priors plus exploration slots give them their first impressions.
(6) Experimentation Platform: overlapping experiment layers, interleaving for shortlisting, session-randomized A/B for launches, and a permanent long-term holdback.
Figure 1: The cascade: each stage spends more compute on fewer documents, and the logged page feeds the next day’s training set.
Clarify Before Designing:
(1) Traffic And Catalog: queries per second at peak, catalog size, and how many catalog items change price or stock per day?
(2) Success Definition: is the primary metric purchase conversion, revenue per session, or long-term retention, and who owns the trade-off when they disagree?
(3) Latency And Cost: is the 100 ms budget end-to-end at p99 including network, and what GPU or CPU budget per query is acceptable?
(4) Query Mix: what fraction of traffic is head queries versus unseen tail queries, and how much is navigational (a known brand or ASIN)?
(5) Personalization Scope: is signed-in history available at query time, and are there privacy or regional constraints on using it?
(6) Marketplace Constraints: must sponsored slots, seller fairness, or new-seller exposure be guaranteed inside the same page?
Leave a Reply