Interview for Machine Learning

Category: ML

MSD0044 Drug Discovery System
A pharmaceutical company wants to use deep learning to accelerate early-stage drug discovery. Given a disease target (a protein or a pathway), the system must predict which existing molecules are likely to bind (virtual screening), generate novel structures optimized for binding affinity and drug-likeness, and predict ADMET properties (absorption, distribution, metabolism, excretion, toxicity) so that compounds likely to fail later are filtered out early.
The constraints are brutal. Chemical space holds on the order of $10^{60}$ drug-like molecules, only a few thousand experimentally validated binding measurements exist for a typical novel target, and every molecule you nominate costs real money and weeks of chemistry to make and assay. Molecules that look optimal to a model but cannot be synthesized are worthless.
How would you design this system? Cover the molecular representation (SMILES strings vs molecular graphs vs 3D conformations), the property prediction model architecture, the generative model for novel compounds (VAE, GAN, diffusion, or autoregressive), the ADMET prediction pipeline, and how you evaluate generated molecules for validity, novelty, and synthesizability.

The Problem: the search space is astronomically large, the label set is tiny, and the wet lab can only test about a hundred molecules per round, so the system’s real job is choosing which hundred.
Answer
The design is a cost-ordered cascade closed by an active-learning loop: cheap learned scorers rank billions of candidates, progressively more expensive structure-based models rescore the survivors, an ADMET gate with calibrated uncertainty removes likely clinical failures, and a diverse batch of roughly a hundred molecules goes to the assay whose results retrain everything. Two decisions are pivotal. First, generation happens inside synthesizable space, conditioned on reactions and building blocks from a make-on-demand catalog, so a “novel” molecule always comes with a route. Second, every model output is treated as an acquisition function for a expensive experiment, not as truth, so the selection step optimizes expected confirmed hits per round under uncertainty rather than the top predicted potency.
(1) Cost-Ordered Cascade: spend microseconds on billions, seconds on millions, and minutes on thousands; only the final hundred reach the bench.
(2) Representation By Stage: 2D molecular graphs with a pretrained encoder for property prediction, 3D pocket-aware co-folding or docking only for rescoring the shortlist.
(3) Pretrain Then Multitask: self-supervised pretraining on millions of unlabeled molecules plus multitask fine-tuning across related assays beats training a single-target model on 2,000 labels.
(4) Synthesis-Constrained Generation: a pocket-conditioned generator that emits reaction and building-block choices, so validity and synthesizability are structural guarantees, not metrics you hope for.
(5) ADMET As A Gate, Not A Score: multitask ADMET heads plus hard structural alerts, with conformal abstention when a compound sits outside the applicability domain.
(6) Uncertainty-Aware Batch Selection: pick the assay batch for diversity and information gain, and feed every result, especially the inactives, back into training.

Figure 1: The discovery loop: cheap scoring first, structure-based rescoring on survivors, an ADMET and route gate before ordering, and assay results (hits and misses alike) retraining the scorer and the generator.
Clarify Before Designing:
(1) Target Knowledge: is there an experimental structure, a cryo-EM map, or only a predicted model, and is the binding site known?
(2) Starting Data: do we have a known ligand series to grow from, or is this a cold-start target with zero confirmed actives?
(3) Assay Economics: how many compounds per round, at what cost and turnaround, and is the primary assay binding or functional?
(4) Chemistry Constraints: in-house synthesis or catalog only, and which scaffolds are excluded for freedom-to-operate reasons?
(5) Property Profile: oral, CNS-penetrant, or injectable, and what safety margins (for example on hERG) are non-negotiable?
(6) Definition Of Success: is the handoff a confirmed hit rate, a number of novel chemotypes, or a potency threshold at a given selectivity?
Login to view more content
September 3, 2026
MSD0043 Credit Scoring Explainability
Design a credit scoring and underwriting system for a fintech lender using a deep learning model with explainability requirements for regulatory compliance. The system must predict the probability that a loan applicant will default, using features like credit history, income, debt-to-income ratio, employment data, and transaction patterns.
Unlike a pure accuracy-maximizing model, this system must produce per-application explanations that satisfy regulatory requirements (e.g., ECOA, GDPR right to explanation): the lender must tell an applicant which factors increased or decreased their score and by how much. The model must also avoid discriminatory bias across protected groups, handle the severe class imbalance of defaults, and deal with reject inference, since you only observe outcomes for approved loans.
How would you design this system? Cover the model architecture and feature engineering, the explainability approach (SHAP, integrated gradients, attention-based explanations), the fairness and bias mitigation strategy, how you handle reject inference and class imbalance, and how you evaluate both predictive performance and explanation quality.

The Problem: the model can decline an applicant in 20 milliseconds, but the law requires a specific, per-applicant reason, the outcome labels exist only for the people the old model already approved, and any accuracy gain that lands unevenly across groups is a legal liability rather than a win.
Answer
The design makes the scoring core explainable by construction instead of bolting an explainer onto a black box. A monotone additive neural network (a neural GAM with one learned shape function per feature) produces the log-odds as a sum of per-feature terms, so each factor’s contribution in score points falls out of the same forward pass that produced the score. A small transaction sequence encoder feeds the additive layer through a handful of named, bounded cash-flow factors, and those few factors are attributed with integrated gradients under a numerically verified completeness check. Two further decisions carry the compliance load: reject inference is solved primarily with a randomized exploration band just below the cutoff plus bureau performance on declined applicants, and fairness is handled by a documented less-discriminatory-alternative search, because group-specific cutoffs would be illegal disparate treatment in consumer credit.
(1) Explainable-by-Construction Core: a monotone neural GAM over bureau, income, and DTI features, with monotonicity constraints so higher utilization can never lower risk; a GRU over 12 months of transactions contributes 8 named cash-flow factors.
(2) Exact Attribution Contract: per-feature score-point deltas come from the additive decomposition, integrated gradients cover only the encoder factors, and every explanation is checked against completeness before the notice is issued.
(3) Reject Inference by Exploration: a randomized approval band on 1% of near-cutoff declines buys unbiased labels below the cutoff; statistical augmentation and bureau outcomes on declines supplement it but never replace it.
(4) Calibration Over Balance: class weighting during training followed by isotonic recalibration on an unweighted out-of-time holdout, because pricing and expected-loss accounting need a true probability, not a balanced classifier.
(5) Fairness by Search, Not by Threshold: proxy audits, a less-discriminatory-alternative model search, and adverse impact ratio monitoring with proxy group estimates; the search itself is the compliance artifact.
(6) One Versioned Artifact: model, monotonicity spec, explainer, and reason-code map ship and are hashed together, so a notice can never be produced by a version that did not make the decision.

Figure 1: One request produces three artifacts that must agree: a calibrated probability, a decision, and a reason set derived from the same arithmetic that produced the probability.
Clarify Before Designing:
(1) Regulatory Scope: US only (ECOA/Reg B plus FCRA adverse action, up to four principal reasons), or also EU GDPR Article 22, which adds a right to human review and changes the automation design?
(2) Decision Autonomy: is the decline fully automated, or does a human underwriter own a review band where the model is only a recommendation?
(3) Product and Label Lag: what is the term and the performance definition (90 days past due within 12 months?), since that sets how stale the newest usable training vintage is?
(4) Volume and Latency: how many applications per day, and is the decision instant at point of sale or allowed to take minutes?
(5) Reject Data Access: can we pull bureau performance on declined applicants, and does an exploration budget for below-cutoff approvals already exist?
(6) Protected Attributes: are they collected (mortgage-style) or must disparity be measured with proxy estimates, which changes how much we can trust the fairness numbers?
Login to view more content
September 3, 2026
MSD0042 Last-Mile Delivery Routing
Design a deep learning system for last-mile delivery route optimization that predicts travel times, sequences stops, and adapts to real-time traffic and demand. A delivery company such as UPS, FedEx, or Amazon Logistics must route thousands of drivers daily through dense urban and suburban areas, each driver visiting 50-150 stops.
The system must predict travel times between stops accounting for traffic, time of day, weather, and road type; sequence stops to minimize total time while respecting time windows and vehicle capacity; and re-optimize on the fly when a driver falls behind, a new stop is added, or traffic conditions change. The routing problem is NP-hard at scale, so the system must balance solution quality against compute time, and it must learn from historical route data to improve future predictions.
How would you design this system? Cover the travel-time prediction model, the route sequencing optimization algorithm (classical VRP solvers vs learned heuristics vs neural combinatorial optimization), the real-time re-optimization strategy, how you incorporate historical route data to improve predictions, and how you evaluate route quality against ground-truth completion times.

The Problem: one van, 150 stops, hard time windows, and edge costs that change hour by hour. The number of possible orderings is astronomical, so the system has to guess travel times well and then search the orderings cheaply enough to finish before the shift starts.
Answer
The design deliberately splits learning from search. A learned travel-time model turns the road network plus time, weather, and live speed feeds into a time-dependent, distributional cost matrix, and a classical metaheuristic VRP solver (guided local search or hybrid genetic search) sequences the stops against that matrix under time-window and capacity constraints. Deep learning is where it earns its keep, in prediction and in learning driver behavior, not in replacing a solver that must guarantee feasibility. The two pivotal decisions are to predict quantiles rather than mean travel times, so tight windows can be buffered by risk rather than by luck, and to regularize the sequence toward historically executed driver routes, because the Amazon Last Mile Routing Research Challenge showed that cost-optimal sequences are routinely overridden by drivers who know where parking, gates, and one-way alleys actually are. Real-time adaptation is a rolling-horizon re-solve with a frozen prefix, so the plan stays stable in front of the driver while the tail is repaired inside a two-second budget.
(1) Learned ETA, Classical Search: a road-graph neural network predicts edge travel times; a metaheuristic solver does the combinatorial work with hard feasibility guarantees.
(2) Distributional Time-Dependent Costs: per-edge p50 and p90 travel times bucketed by departure hour, so rush-hour edges and risky windows are priced correctly.
(3) Separate Service-Time Model: parking, walking, gates, and elevators are modeled per stop type independently of driving time, because they dominate dense urban routes.
(4) Driver-Preference Regularization: a zone-sequence model learned from executed routes constrains or penalizes orderings drivers reliably reject.
(5) Hierarchical Decomposition: sequence delivery zones first, then stops inside each zone, keeping every solver subproblem in the tens of nodes rather than 150.
(6) Frozen-Prefix Re-optimization: the next few stops are committed; only the tail is re-sequenced by warm-started large neighborhood search under a latency cap.

Figure 1: Overnight the pipeline runs left to right; during the shift the re-optimizer repairs the tail of an existing plan, and executed routes flow back as training data.
Clarify Before Designing:
(1) Scale And Geometry: how many drivers per station per day, how many stops per route, and is the mix dense urban (walking dominated) or suburban (driving dominated)?
(2) Cost Asymmetry: what does a missed delivery window cost relative to an hour of driver overtime or an extra van dispatched?
(3) Planning Cadence: is there an overnight batch window with frozen packages, or does volume keep arriving after dispatch?
(4) Driver Autonomy: must drivers follow the sequence, or may they override, and do we have GPS traces to measure how often they do?
(5) Label Availability: do we get per-stop arrival and departure timestamps and package scans, or only route start and end times?
(6) Compute And Connectivity: what solver budget exists per route overnight, what re-optimization latency is acceptable, and can the driver app fall back offline?
Login to view more content
September 3, 2026
MSD0041 Fitness Pose Estimation
Design a deep learning model for real-time pose estimation in a fitness app that counts reps, corrects exercise form, and runs on a phone. The app uses the phone’s front camera to track the user’s body keypoints (shoulders, elbows, wrists, hips, knees, ankles) while they perform squats, push-ups, and yoga poses.
The model must estimate 2D or 3D pose at 30 fps on-device, recognize which exercise is being performed, count repetitions by detecting the movement cycle, and flag form errors such as knees caving inward during a squat or a back rounding during a deadlift. It has to work across body types, clothing, camera angles, and lighting, and latency must stay low enough for real-time audio feedback.
How would you design this model? Cover the candidate pose estimation architectures (heatmap regression vs direct coordinate regression vs 3D lifting from 2D), the exercise classification and rep counting logic, the form-error detection mechanism, the on-device deployment strategy (quantization, distillation), and how you evaluate pose accuracy and rep-counting reliability.

The Problem: one lens, no depth, and a 33 ms per frame budget, yet every frame must answer three questions at once: where the joints are, where in the rep cycle the body is, and whether the form is safe.
Answer
Run a detector-tracker cascade around a single-person 2D keypoint network whose output head is a coordinate classifier (SimCC-style 1D bins per axis), trained with dense heatmap supervision and distilled from a large teacher. The scale-normalized keypoint sequence then feeds one small temporal head that emits the exercise label, the rep phase, and the form flags. 2D stays the primary signal because monocular depth is ambiguous, and a 27-frame 2D-to-3D lifting head is switched on only for errors that genuinely need depth, such as back rounding or knee valgus seen from an oblique angle. Two decisions carry the design: put the spatial prior in training (heatmaps) while keeping the cheap output at inference (coordinates), and treat rep counting as hysteresis on a phase signal rather than peak-picking on a raw joint angle.
(1) Heatmap Regression: the backbone emits one $H/4 \times W/4$ map per joint, and the joint location is the argmax cell plus a learned sub-pixel offset; strong spatial inductive bias, heavy output tensor.
(2) Direct Coordinate Regression: the head emits $2K$ numbers directly, or classifies each axis into 1D bins (SimCC); tiny output, no argmax, needs heatmap-style supervision or bin targets during training to localize well.
(3) 3D Lifting From 2D: a temporal network consumes a window of 2D keypoints and predicts root-relative 3D joints; it recovers depth-dependent form errors but inherits every 2D error and adds window latency.

Figure 1: One cheap keypoint pass per frame; everything the product actually says out loud is computed from the buffered keypoint sequence, not from raw pixels.
Clarify Before Designing:
(1) Depth Requirement: which form errors are in scope, and can any of them only be judged from depth (back rounding, hip hinge) rather than from an in-plane 2D ratio?
(2) Compute Budget: what is the target device tier, and how much of the 33 ms frame budget is already spent by camera capture, rendering, and audio?
(3) Framing Control: is the phone propped up with the whole body in view, or handheld with limbs leaving the frame; can the app force a calibration pose first?
(4) Exercise Vocabulary: a closed set of 20 named movements, or open-set counting for anything the user does?
(5) Label Availability: do we have keypoint labels on fitness footage (not just COCO street photos), and do we have per-rep form-error labels from coaches?
(6) Teacher Access: can we run a large server-side pose model offline to generate pseudo-labels and distillation targets?
Login to view more content
September 3, 2026
MSD0040 Automated Retail Checkout
Design a deep learning model for a retail store’s automated checkout system that recognizes items from camera feeds and computes totals without barcode scanning. A customer places items on a counter or walks through a gate, overhead and angled cameras capture the items, and the system must identify each product, count multiples, handle occlusion and stacked items, distinguish similar-looking products (two flavors of the same brand), and produce a receipt in seconds.
The system must handle a large product catalog (thousands of SKUs) that changes as inventory rotates, work under varying lighting and camera angles, and keep the error rate low enough that manual corrections are rare.
How would you design this model? Cover the candidate recognition architectures (object detection vs instance segmentation vs metric-learning retrieval against a product catalog), how you handle the large and rotating SKU catalog, how you count multiples and handle occlusion, how you generate training data for new products with few examples, and how you calibrate confidence to flag uncertain items for manual review.

The Problem: the catalog is huge and rotates weekly, the items overlap and stack, and two flavors of the same brand differ by a color band. The model has seconds to turn pixels into an exact receipt.
Answer
Split the problem into localization and identity, and keep identity out of the network weights. A class-agnostic instance segmenter finds every item instance and its mask, a crop encoder trained with a metric-learning loss maps each instance to a unit embedding, and the SKU is decided by cosine nearest-neighbor search against a catalog index of enrolled prototypes. This makes catalog rotation an indexing operation rather than a retraining project, since a new product needs about 20 enrollment crops and no weight update. Masks (not boxes) drive multiple-instance counting under occlusion, counts are fused across the overhead and angled views, and a calibrated margin gate abstains on ambiguous items so the review lane sees only the genuinely hard ones.
(1) Closed-Set Object Detection: one detector with a $K$ -way softmax head over all SKUs; fastest single pass, but the catalog is baked into the weights.
(2) Instance Segmentation With A Class Head: per-item masks give reliable separation of touching and stacked products, but identity still lives in a fixed classifier.
(3) Class-Agnostic Localization Plus Metric-Learning Retrieval: the chosen design; localization is SKU-independent and identity is a nearest-prototype lookup in an index that can be edited nightly.
(4) Open-Vocabulary Detection With Text Prompts: zero-shot on new categories, but it cannot separate two flavors of one brand, so it serves only as a proposal generator or a cold-start fallback.

Figure 1: Localize anything, then look it up: frames → masks → embeddings → catalog index → fused counts → gate. New SKUs enter through the index, never through the weights.
Clarify Before Designing:
(1) Catalog Shape: how many SKUs, how many are added or retired per week, and how many are near-duplicates within a brand family?
(2) Compute Budget: edge box per lane or shared server GPU, and what is the wall-clock budget from last item placed to receipt shown?
(3) Camera Geometry: fixed calibrated overhead plus angled views on a static counter, or a gate where items must be tracked while moving?
(4) Enrollment Data: does the retailer already supply studio packshots per SKU, and can we photograph new products before they reach the shelf?
(5) Error Asymmetry: what is the cost of an undercount (shrinkage) versus an overcharge (trust), and what per-basket review rate is acceptable?
(6) Presentation Rules: may the interface ask the customer to spread items out, or must the model handle arbitrary piles?
Login to view more content
September 3, 2026
MSD0039 Landmark Recognition Travel
Design a landmark recognition model for a travel app that identifies famous landmarks from user photos and returns relevant historical and visitor information. When a traveler points their phone at the Eiffel Tower, the Colosseum, or a lesser-known temple, the model must identify the landmark in seconds, even from unusual angles, partial occlusion, varying lighting, or when the photo is taken from inside rather than the iconic exterior view.
The system must handle the long tail of tens of thousands of landmarks worldwide, many with only a handful of reference images, and it must gracefully say “I don’t recognize this” rather than confidently misidentifying a similar-looking building.
How would you design this model? Cover the candidate recognition approaches (global descriptor retrieval vs local-feature matching vs deep metric learning with classification), how you handle the long tail of rare landmarks with few training images, how you build the reference index, how you calibrate confidence to enable abstention, and how you evaluate recognition accuracy and false-positive rate.

The Problem: the query is rarely the postcard view, the tail landmark has a handful of reference photos, and a confident wrong name costs more trust than an honest “not sure”.
Answer
Treat this as open-set instance retrieval with abstention, not 100k-way classification. A backbone trained with a margin-based classification loss (ArcFace with sub-centers) produces a compact global descriptor; that descriptor retrieves top candidates from an ANN index over all reference photos; local-feature matching with geometric verification re-ranks those candidates and produces an inlier count. The two pivotal decisions are to keep the final decision in the index rather than in a softmax head, so a new temple is added by indexing seven photos instead of retraining, and to build the confidence signal from geometric inliers plus descriptor margin rather than softmax probability, because that is what makes a calibrated “I don’t recognize this” possible.
(1) Global Descriptor Retrieval: GeM-pooled CNN or ViT embedding, PCA-whitened to 512-d, searched with IVF-PQ over every reference image; fast, memory-light, viewpoint-sensitive.
(2) Local Feature Matching: keypoints and descriptors (DELF, SuperPoint) matched pairwise, then RANSAC for a consistent transform; the inlier count is both the score and the explanation.
(3) Deep Metric Learning With Classification Head: train with ArcFace or sub-center ArcFace over the landmark ID vocabulary, discard the head at inference and keep the embedding for retrieval.
(4) Chosen Hybrid: (3) trains the descriptor, (1) does the cheap recall, (2) verifies the top 100, and a calibrated gate over inliers and margin decides answer versus abstain.

Figure 1: Cheap recall first, expensive verification second, and an explicit gate that is allowed to answer nothing.
Clarify Before Designing:
(1) Inventory And Reference Depth: how many landmarks, and what is the reference-count distribution (median photos per landmark, how many with fewer than 10)?
(2) Compute Budget: server-side with a 1 s budget, or on-device with a fixed memory ceiling and no index round trip?
(3) Metadata: is GPS or EXIF available at query time? A 5 km geo prefilter removes most look-alike confusions for free.
(4) Cost Asymmetry: what precision floor does the product require, and what abstention rate is acceptable to hit it?
(5) Definition Of Recognized: do interiors, statues, details, and reconstructions count as the same landmark, and how are near-identical siblings (replicas, chain temples) labeled?
(6) Data Rights: can web-mined and user-submitted photos enter the index, and can queries be logged for hard-negative mining?
Login to view more content
September 3, 2026
MSD0038 LinkedIn PYMK GNN
Design a “People You May Know” feature for LinkedIn using graph neural networks for link prediction over the professional social graph. Given a member and the current graph of connections, the system must recommend people the member likely knows and would connect with, ranking millions of candidates by connection probability.
The graph is massive (hundreds of millions of nodes, billions of edges), sparse for most members, and rich with side features: company, school, job title, location, co-workers, co-alumni. A GNN can propagate information along edges to capture multi-hop structure that simple heuristics (mutual friends, same company) miss, but it must scale to the full graph and stay useful as new connections form every second.
How would you design this model? Cover the candidate GNN architectures (GraphSAGE vs GAT vs PinSage-style random-walk plus convolution), how you handle negative sampling and class imbalance, how you scale training and inference to a graph with hundreds of millions of nodes, how you incorporate node side features, and how you keep recommendations fresh as the graph evolves.

The Problem: a member has a few dozen links inside a graph of hundreds of millions of people, so the model must find the handful of real-world acquaintances hiding in an ocean of strangers.
Answer
The model is a two-layer inductive GraphSAGE encoder trained as a link predictor: for each pair, sample a fixed-fanout neighborhood, aggregate side features along the sampled edges into a 256-dimensional embedding, and score the pair with a dot product. Fixed fanout is the pivotal decision because it makes cost per candidate independent of degree, which matters on a graph where recruiters carry tens of thousands of edges. The second pivotal decision is the negative sampling mixture: random negatives alone make the task trivially easy at a positive rate near $10^{-6}$ , so training mixes in hard two-hop negatives (people who share a company or mutual connections but never connected). Attention is added surgically as a last-layer GAT head on high-degree neighborhoods rather than everywhere, and side features enter as the layer-zero representation so a brand-new member gets an embedding without retraining.
(1) GraphSAGE (chosen encoder): uniform neighbor sampling with fanout $(25, 10)$ and mean or pooling aggregation, inductive by construction because weights act on features rather than node IDs.
(2) GAT (selective upgrade): learned per-edge attention softmax over neighbors, which can down-weight a recruiter or hiring-manager edge that mean aggregation blends into the signal.
(3) PinSage-style random walk plus convolution: neighborhoods defined by random-walk visit counts instead of the raw adjacency, giving importance weights and a hub-resistant top-T neighbor set.
(4) Heuristic and shallow baselines: common neighbors, Adamic-Adar, and matrix factorization set the accuracy floor the GNN must beat, and remain the cheap candidate generator that feeds it.

Figure 1: One shared encoder, two sampled neighborhoods, one dot product: the whole model is a siamese GNN over a bounded receptive field.
Clarify Before Designing:
(1) Label definition: is a positive an invitation sent, an invitation accepted, or a “we actually know each other” signal, and are dismissals usable as negatives?
(2) Inductive requirement: must a member who joined ten minutes ago and has zero edges get an embedding, or may cold-start fall back to a feature-only model?
(3) Freshness budget: how stale may an embedding be after a new connection forms, minutes or a full day?
(4) Compute and storage budget: how many GPUs for training, and can we store $|V| \times d$ embeddings in a serving store?
(5) Graph heterogeneity: do we get typed edges (co-worker, co-alumnus, message, profile view) and reliable side features for all members, or only for complete profiles?
(6) Candidate volume per request: does the GNN score millions of pairs, or a pre-filtered two-hop pool of a few thousand?
Login to view more content
September 3, 2026
MSD0037 Zillow Personalized Search
Design Zillow’s “Homes for You” personalized real estate search system that ranks listings by each user’s preferences and browsing behavior. When a user searches for homes in a city, the system must go beyond simple price and size filters to personalize the ranking: a user who repeatedly views Craftsman-style houses with large yards in quiet neighborhoods should see similar listings first, while a first-time searcher gets a diverse, high-quality mix.
The system must handle cold start for new users, respect hard constraints (price range, location, beds/baths), avoid showing the same listings repeatedly, and balance relevance against inventory freshness, because a listing that went live an hour ago matters more than one that has been sitting for 90 days.
How would you design this system? Cover the candidate generation and ranking funnel, the user preference signal extraction from browsing behavior, how you handle cold start for new users, the feature set (listing attributes, user history, contextual signals), and the online and offline metrics.

The Problem: two million active listings, one ranked list, and a shopper whose real preferences (Craftsman, big yard, quiet street) exist only inside their click history. Everyone with the same filters currently sees the same order.
Answer
The design is a four-stage funnel: strict hard-constraint filtering, multi-source candidate generation, a learned personalized ranker, and a slate re-ranker that enforces diversity, freshness, and seen-listing decay. Personalization never widens or narrows the legal candidate set. Filters behave like a contract, and learning only reorders what already matches. Two decisions carry the design. First, preferences are extracted as a weighted implicit-feedback profile over listing attributes plus a short in-session view sequence, where a contact-agent event counts far more than a click. Second, cold start is a ramp, not a switch: market-quality priors give way to session signals and then to the full profile as events accumulate. A compliance guardrail sits above everything, because housing personalization is regulated and protected-class proxies cannot be features.
(1) Hard Filters First: price, beds, baths, geo polygon, and listing status are strict boolean predicates in the search index, so a personalized model can never hide a listing the user explicitly asked for.
(2) Multi-Source Candidate Generation: union of filter recall (recency-ordered), two-tower ANN retrieval on the user embedding, and co-view/co-save item-item lookups seeded by the session, giving roughly 500 candidates.
(3) Preference Signal Extraction: event weights (contact 10, save 5, long dwell 2, click 1, hide -3) roll up into a decayed attribute-affinity vector plus the last 20 viewed listing embeddings.
(4) Learned Ranker: gradient-boosted trees over ~200 listing, user, context, and cross features, trained on a multi-objective label dominated by save and contact events rather than raw clicks.
(5) Slate Re-Rank: diversity caps per style and block, a bounded new-listing boost, and a seen-decay penalty turn point-wise scores into a slate a human enjoys scrolling.
(6) Cold Start Ladder and Guardrails: market-quality priors and explore slots for new users, and a feature allow-list plus a neutral-ranker kill switch for Fair Housing compliance.

Figure 1: The funnel: strict filters define what is legal to show, three retrieval sources decide what is worth scoring, the ranker orders it, and the re-ranker shapes the slate. Behavior events flow back through the feature store into both retrieval and ranking.
Clarify Before Designing:
(1) Surface and Scale: is this the in-session search results page, the logged-in “Homes for You” home feed, or the daily email? Each has a different latency budget and a different amount of user history available.
(2) Filter Semantics: are price and beds hard constraints or soft preferences? If a $520k home may appear for a $500k filter, the whole candidate layer changes.
(3) Objective: do we optimize saves and agent contacts (monetizable intent) or dwell time? The label choice dictates which behavior the model amplifies.
(4) Freshness Requirement: what fraction of new listings must receive impressions within 24 hours? This is a marketplace supply guarantee, not a modeling detail.
(5) Compliance Constraints: which features are off-limits under Fair Housing rules, and does the audit require identical candidate sets for identical filters?
(6) Data Availability: how much logged-in traffic exists versus anonymous traffic, and can we key profiles on device for anonymous users?
Login to view more content
September 3, 2026
MSD0036 Domain Fine-Tuning Pipeline
Design a fine-tuning pipeline that adapts a foundation model to a company’s domain, including data curation, training strategy, and evaluation. A large enterprise wants to take a general-purpose LLM (e.g., Llama 3 or Mistral) and specialize it for its own support tickets, internal wikis, product manuals, and domain-specific jargon.
The pipeline must curate and clean training data from noisy heterogeneous sources, choose between full fine-tuning, LoRA/QLoRA, and instruction tuning, avoid catastrophic forgetting of general capabilities, and produce a reproducible evaluation that proves the adapted model is better on domain tasks without regressing on general benchmarks.
How would you design this pipeline? Cover the data curation and deduplication strategy, the fine-tuning approach and hyperparameter choices, the evaluation harness (domain-specific benchmarks plus general-capability guardrails), the infrastructure for reproducible runs, and how you decide when the adapted model is ready to replace the base model in production.

The Problem: the base model speaks fluent English but has never read a single one of your change requests, part numbers, or escalation runbooks, and roughly 900M tokens of company text are sitting unused.
Answer
The design is a curate, adapt, gate pipeline: a versioned data factory turns noisy internal sources into a deduplicated, PII-scrubbed corpus plus a hand-reviewed instruction set, a QLoRA adapter is trained on top of a pinned base checkpoint, and a dual-gate evaluation decides promotion. Two decisions dominate. First, adapt with parameter-efficient tuning on a small high-quality mix (domain corpus, instruction pairs, and 5 to 10 percent general replay data) rather than full fine-tuning on a raw dump, because a raw dump buys memorization and forgetting instead of capability. Second, make the promotion decision a two-sided gate: a required lift on a frozen domain benchmark and a bounded regression budget on general benchmarks, both measured by a versioned harness that any engineer can rerun from a run manifest.
(1) Data Factory Before Modeling: exact plus MinHash near-duplicate removal, PII and secret scrubbing, quality filtering, and access-control filtering, all emitting a content-hashed dataset version.
(2) Two-Stage Adaptation: short continued pretraining on the cleaned domain corpus to absorb jargon, then instruction tuning on curated ticket-resolution and doc-QA pairs to restore and shape behavior.
(3) QLoRA as the Default: 4-bit frozen base with rank-16 adapters on attention and MLP projections, so a run costs a single GPU-hour class of compute and the base weights stay bit-identical.
(4) Replay Against Forgetting: mix general instruction data into the SFT set and gate on general benchmarks, so domain gains are never paid for with instruction-following collapse.
(5) Reproducible Runs: pinned base digest, dataset hash, seed, container image, and config in one manifest; every eval number traces back to one manifest.
(6) Adapter Serving with a Kill Switch: serve base plus hot-swappable adapter behind a registry, roll out shadow → 5 percent canary → full, and roll back by unloading the adapter.

Figure 1: The loop: curated and versioned data feeds a cheap adapter run, the harness decides promotion, and production traffic becomes next month’s training data.
Clarify Before Designing:
(1) Target Tasks: which two or three tasks must improve (ticket triage, draft replies, doc QA), and who signs off on the frozen eval set for them?
(2) Corpus Reality: how many tokens survive after access-control and license filtering, and how much of the ticket volume is template boilerplate?
(3) Knowledge vs Behavior: is the gap stale facts (retrieval work) or unfamiliar jargon, format, and tone (fine-tuning work)?
(4) Regression Budget: how many points of MMLU, IFEval, or safety-refusal rate is the business willing to lose for a domain win?
(5) Serving Constraints: self-hosted weights or a closed API, one adapter for everyone or per-business-unit adapters, and what is the latency budget?
(6) Compliance: can customer PII enter training weights at all, and what is the deletion or right-to-forget story once it does?
Login to view more content
September 3, 2026
MSD0035 AI Agent Safeguards
Design the safeguards layer for an AI agent that can take real-world actions on a user’s behalf: booking flights, making purchases, sending messages, or modifying account settings, as envisioned for ChatGPT’s Operator, Anthropic’s computer use, and autonomous agent frameworks. Unlike a chatbot that only produces text, this agent can spend money and send irrevocable messages, so a prompt injection, a misread page, or a goal misinterpretation can cause real harm.
The safeguard system must approve or block each action before execution, verify that the action matches the user’s actual intent, defend against prompt injection from web content the agent reads, and provide a clear audit trail and undo path when something goes wrong.
How would you design this system? Cover the action classification and risk tiers, the intent verification mechanism (confirmation, biometric, second factor for high-risk), the prompt-injection defense for tool-returned content, the human-in-the-loop gate for irreversible actions, and how you evaluate safety without waiting for a real incident.

The Problem: the agent’s inputs include text written by strangers, and its outputs include charges, messages, and settings changes that cannot be taken back. Every action needs a gate before it reaches the world.
Answer
The design is an action firewall that sits between the agent’s planner and every tool that touches the world. Nothing the model “decides” executes directly. Each proposed action is emitted as a typed, structured call, assigned a risk tier by a cheap classifier plus deterministic rules, then evaluated by a policy engine against the capabilities and budget the user actually granted for this session. Two decisions carry most of the safety weight. First, untrusted tool output is quarantined data, never instructions, so page text cannot mint new goals or new capabilities. Second, irreversible actions pass through a human confirmation gate with step-up authentication, and everything else is made cheap to undo through logging, reversible-by-design tool wrappers, and an undo queue.
(1) Typed Action Schema: the agent cannot call raw HTTP or type free-form shell text; it emits validated calls such as purchase(merchant, amount, currency) so risk is computable from arguments, not guessed from prose.
(2) Four Risk Tiers: T0 read-only, T1 reversible writes, T2 spending or external messages, T3 irreversible or identity-changing (password, 2FA, transfers, mass sends); tier fixes the required evidence of intent.
(3) Deterministic Policy Engine: per-session capability grants, allowlists, per-action and per-session spend caps, rate limits, and recipient novelty checks, all enforced outside the model.
(4) Injection Containment: a planner that never sees raw untrusted text plus a quarantined extractor that returns typed values only, with data-flow provenance so tainted values cannot authorize a T2 or T3 action.
(5) Intent Verification Ladder: silent execution for T0/T1, in-context confirmation showing the exact parsed arguments for T2, and passkey or biometric step-up for T3, always rendered from the structured call rather than from model prose.
(6) Audit and Undo: append-only log of prompt, evidence, tier, policy decision, and result; reversible wrappers (hold instead of charge, delayed send, snapshot before settings change) give a real rollback path.

Figure 1: The action firewall: classify → check policy → decide, with untrusted tool output entering as quarantined data and irreversible actions detouring through human confirmation before the sandboxed executor runs.
Clarify Before Designing:
(1) Action Surface: which tools exist on day one (browser clicks, payment API, email, account settings), and can we wrap them in reversible primitives such as authorization holds and delayed sends?
(2) Autonomy Contract: does the user pre-authorize a budget and scope per session, or must every spend be confirmed live? This single answer moves confirmation volume by an order of magnitude.
(3) Harm Asymmetry: what is the cost of a wrongly executed purchase versus a wrongly blocked one, and who eats the loss (user, merchant, or platform)?
(4) Latency and Volume: how many actions per day, and how much of the per-action budget may the safeguard layer consume before the agent feels sluggish?
(5) Identity and Compliance: is a passkey or biometric available on the device, and do payments, health, or messaging rules mandate explicit consent records?
(6) Adversary Model: are we defending against opportunistic web-page injections only, or against attackers who will craft pages specifically targeting our agent?
Login to view more content
August 22, 2026