Interview for Machine Learning

Category: NLP/LLM

MSD0060 RLHF Alignment Pipeline
Design an RLHF alignment pipeline for aligning a large language model with human preferences. A foundation model team must align a model so that it follows instructions, refuses harmful requests, and produces helpful, harmless, and honest responses.
The alignment must use reinforcement learning from human feedback, but the team must choose between PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and GSPO (Group Sequence Policy Optimization) as the alignment algorithm. Each has different sample efficiency, training stability, and compute requirements. The system must train a reward model from human preference data, run the alignment loop at scale on a distributed cluster, and produce a model that passes safety evaluations while keeping its capability.
How would you design this system? Cover the reward modeling architecture, the candidate RL algorithms (PPO vs GRPO vs GSPO) and their trade-offs, how you handle reward hacking and mode collapse, how you distribute the alignment training across GPUs, and how you evaluate alignment quality.

The Problem: humans can only tell you which of two whole responses they prefer, but gradient descent needs a scalar objective and a stable update signal at every token of every rollout.
Answer
The stack is the standard three stages: an instruction-tuned SFT checkpoint, a Bradley-Terry reward model trained on human preference pairs, and an on-policy RL loop that maximizes reward minus a KL penalty against the frozen SFT reference. Everything above the update rule is shared, so the real design question is the RL objective. PPO learns a separate value network to supply per-token baselines, while GRPO and GSPO delete the critic and use the mean reward of a group of sampled responses instead, removing one trainable model copy from memory. GSPO goes further and computes the importance ratio at the sequence level, matching the granularity of the reward itself, which is what keeps training stable on MoE policies and long rollouts. My default is group-relative advantages with a sequence-level ratio, a KL anchor plus reward-model ensembling to hold back reward hacking, and PPO kept in reserve only where a dense token-level value estimate genuinely earns its extra trainable model.
(1) Shared Substrate: SFT policy, preference-trained reward model, and a frozen reference for the KL anchor; all three candidates optimize the same penalized reward.
(2) PPO: actor-critic; a learned value network produces per-token advantages through GAE, and the importance ratio is clipped once per token.
(3) GRPO: no critic; sample $G$ responses per prompt and standardize their rewards into one advantage per response, still clipped per token.
(4) GSPO: same group baseline, but the importance ratio and the clip act on the whole length-normalized sequence, matching the sequence-level reward.

Figure 1: One alignment loop: preferences train the reward model, reward minus a KL anchor scores group rollouts, and only the update box changes between PPO, GRPO, and GSPO.
Clarify Before Designing:
(1) Policy Architecture: dense or sparse MoE, and at what size? Routing volatility in MoE is the single strongest argument for a sequence-level ratio.
(2) Reward Source: learned reward model only, or are parts of the traffic verifiable (unit tests, math checkers, format rules) so a rule-based reward can replace the model?
(3) Sequence Length: 1k-token chat replies or 32k-token reasoning traces? Long rollouts multiply token-level ratio variance and dominate step time.
(4) Compute Budget: how many GPUs, and does a second trainable critic-sized model fit next to the policy after sharding?
(5) Preference Data: how many pairs exist, what is the annotator agreement rate, and is there a recurring human evaluation loop or only an LLM judge?
(6) Release Bar: what refusal rate on red-team prompts and what maximum capability regression (the alignment tax) gate the release?
Login to view more content
September 10, 2026
MSD0057 LLM Training Data Pipeline
Design a distributed data preprocessing pipeline for training large language models at Amazon. The Bedrock team needs to prepare training corpora from petabytes of raw text (web crawls, code repositories, internal documents, licensed datasets) by tokenizing, deduplicating, filtering for quality, removing harmful content, and sharding into training-ready chunks.
The pipeline must process data arriving in S3 and other stores, handle near-duplicate detection at scale (MinHash, SimHash, exact hash), apply quality classifiers to remove low-quality text, run PII and safety filters, and output sharded tokenized sequences ready for distributed training. It must process hundreds of terabytes per run, be resumable from checkpoints, and produce reproducible datasets with full provenance tracking.
How would you design this system? Cover the ingestion architecture (batch S3 versus streaming), the deduplication strategy (exact versus near-duplicate at scale), the quality and safety filtering pipeline, the tokenization and sharding strategy, and how you ensure reproducibility and provenance across pipeline runs.

The Problem: half a petabyte of messy text arrives per run, most of it duplicated, boilerplate, unsafe, or machine-generated, and the training cluster needs a byte-identical, provenance-tracked shard set within days.
Answer
The design is an immutable, content-addressed data lake on S3 with a cheap-to-expensive filter cascade running as batch Spark or Ray jobs. Every stage reads a frozen snapshot, appends scores instead of deleting rows, and writes a new layer whose path is keyed by the hash of its code plus config, which is what makes runs reproducible and resumable. Ingestion is batch over snapshots rather than streaming, because deduplication is a global operation that needs a full-corpus view. Two decisions are pivotal. First, three-tier deduplication (exact document and line hashes, then MinHash with LSH banding for near duplicates, then substring dedup on high-risk subsets) runs before the model-based quality classifier, so the expensive scorer sees roughly a third of the documents. Second, tokenization is deferred to the last stage so the durable artifact stays text, which means a tokenizer or vocabulary change costs a few hours of CPU rather than a full pipeline rerun.
(1) Immutable Snapshot Ingestion: crawls and repos land in a bronze layer as append-only, content-addressed objects with a snapshot manifest; streaming is used only to buffer arrivals, never to feed the transforms.
(2) Three-Tier Deduplication: exact SHA-256 on documents and lines kills boilerplate, MinHash plus LSH bands catch near duplicates, and union-find keeps one canonical document per cluster.
(3) Score, Never Delete: heuristics, quality classifiers, and safety and PII detectors write scores into columnar tables, and a single declarative policy file turns scores into keep/drop decisions per run.
(4) Cost-Ordered Cascade: language ID and cheap heuristics first, then dedup, then the learned quality classifier, then safety and PII, so each expensive model runs on the smallest surviving corpus.
(5) Tokenize Last, Pack Fixed: BPE tokenization emits packed 8192-token sequences with document-boundary masks into roughly 40,000 one-gigabyte shards plus an index for deterministic, resumable loading.
(6) Provenance And Resumability: a per-document lineage table plus a partition-level task ledger give a Merkle dataset version, replay of only failed partitions, and targeted rebuilds for takedown requests.

Figure 1: The cascade: cheap normalization and dedup shrink the corpus before any learned model runs, tokenization is the final derived stage, and every stage writes lineage into the provenance store that defines the dataset version.
Clarify Before Designing:
(1) Token Target: how many training tokens does the model plan need, and for how many epochs? A 10T-token target with single-epoch training lets us filter aggressively; a 2T-token budget with repetition does not.
(2) Licensing And Compliance: which sources are licensed for training, what is the right-to-delete SLA, and are there regional data-residency constraints on where the pipeline may run?
(3) Cadence: is this a one-off corpus build or a monthly refresh on new crawl snapshots? Incremental refresh forces us to persist MinHash signatures for cross-run dedup.
(4) Tokenizer Stability: is the vocabulary frozen, or still being trained? That decides whether tokenized shards or filtered text is the durable artifact.
(5) Mixture Requirements: what are the target per-domain and per-language proportions (code, multilingual, internal docs), since mixture weights drive per-slice keep-rate floors.
(6) Budget: what wall-clock and dollar budget per run, and how many proxy-model ablations can we afford to validate a corpus change?
Login to view more content
September 10, 2026
MSD0056 Product Search Ranking
Design a product search ranking system for Amazon. When a customer types a query, the search engine must retrieve relevant products from a catalog of hundreds of millions of items and rank them so that the most relevant and purchasable products appear at the top.
The ranking must account for query intent (browsing vs buying), product relevance (text match, category, attributes), business signals (price, availability, seller rating, conversion rate), and personalization (past purchases, browsing history). The system must serve results in under 100 milliseconds, handle millions of queries per day, and support continuous experimentation: every ranking change is validated through online A/B tests measuring not just click-through rate but purchase conversion, revenue, and long-term customer satisfaction.
How would you design this system? Cover the query understanding and product retrieval architecture, the ranking model architecture (pointwise vs pairwise vs listwise), how you handle cold-start products with no click history, how you design and run online A/B experiments at scale, and how you evaluate ranking quality beyond offline metrics.

The Problem: a few typed words must be turned into an ordered page drawn from hundreds of millions of items, in under 100 milliseconds, where “good” means relevant and likely to be bought and kept.
Answer
The design is a cascade: query understanding, hybrid retrieval that unions a lexical inverted index with a two-tower embedding ANN, a cheap L1 ranker that cuts ten thousand candidates to five hundred, and an expensive listwise learning-to-rank model that orders the survivors. The pivotal decisions are the training label and the experiment loop. Labels are graded purchase-weighted utility with position-bias correction, not raw clicks, because a click-optimized ranker sells cheap junk. Every change ships through interleaving for shortlisting and session-randomized A/B tests with revenue and return-rate guardrails for the launch decision. Cold-start products stay competitive through content-only priors plus reserved exploration slots, so a new item is far less likely to remain stuck at zero impressions just because it has never been shown.
(1) Query Understanding: spell correction, intent classification (broad browse vs specific buy), and attribute extraction (brand, size, color) that become hard filters and ranking features.
(2) Hybrid Retrieval: BM25 over an inverted index for exact tokens such as part numbers, unioned with a two-tower ANN trained on query-purchase pairs for vocabulary mismatch.
(3) Two-Stage Ranking: a microsecond-per-document L1 filter feeds a LambdaMART or multi-task DNN L2 model that can afford roughly 60 microseconds per document.
(4) Purchase-Weighted Objective: graded gains (purchase > add-to-cart > click) with inverse propensity weighting for position bias, blended with calibrated business signals.
(5) Cold Start By Design: content embeddings make new items retrievable on day one; hierarchically smoothed category priors plus exploration slots give them their first impressions.
(6) Experimentation Platform: overlapping experiment layers, interleaving for shortlisting, session-randomized A/B for launches, and a permanent long-term holdback.

Figure 1: The cascade: each stage spends more compute on fewer documents, and the logged page feeds the next day’s training set.
Clarify Before Designing:
(1) Traffic And Catalog: queries per second at peak, catalog size, and how many catalog items change price or stock per day?
(2) Success Definition: is the primary metric purchase conversion, revenue per session, or long-term retention, and who owns the trade-off when they disagree?
(3) Latency And Cost: is the 100 ms budget end-to-end at p99 including network, and what GPU or CPU budget per query is acceptable?
(4) Query Mix: what fraction of traffic is head queries versus unseen tail queries, and how much is navigational (a known brand or ASIN)?
(5) Personalization Scope: is signed-in history available at query time, and are there privacy or regional constraints on using it?
(6) Marketplace Constraints: must sponsored slots, seller fairness, or new-seller exposure be guaranteed inside the same page?
Login to view more content
September 10, 2026
MSD0050 Document QA Extraction
Design a question-answering system that extracts an answer from a large document collection given a user query using a deep reading comprehension model. A legal or enterprise search product must let users ask natural-language questions (“What is the termination clause in the NDA?”, “What were the Q3 revenue figures for the EMEA region?”) and return exact answer spans extracted from a corpus of millions of documents: contracts, filings, internal wikis, emails.
The system must first retrieve the relevant documents or passages from the corpus (the retrieval stage), then run a reading comprehension model over the retrieved passages to extract the answer span (the reader stage), and finally rank or aggregate answers when multiple passages produce candidates. The retrieval stage must handle lexical mismatch (the query says “termination” but the document says “cancellation”), the reader must handle multi-hop reasoning where the answer requires combining information from two passages, and the system must say “I don’t know” when no passage contains a confident answer rather than hallucinating.
How would you design this system? Cover the retriever architecture (sparse BM25 vs dense bi-encoder vs hybrid retrieval), the reader architecture (extractive span prediction vs generative reading comprehension vs T5/BART-style seq2seq), how you handle multi-hop questions, how you calibrate confidence for abstention, and how you evaluate end-to-end QA accuracy (exact match, F1, answerability detection).

The Problem: one natural-language question, ten million documents that rarely use the asker’s words, and a product that must quote the exact sentence or admit it cannot find one.
Answer
The design is a retrieve-rerank-read pipeline with an explicit abstention gate. A hybrid retriever (BM25 plus a dense bi-encoder, merged by reciprocal rank fusion) pulls about 100 passages from a permission-filtered index, a cross-encoder reranker keeps the top 10, and an extractive reader with a no-answer head scores candidate spans in each passage. Spans are aggregated across passages, and a calibrated confidence threshold decides between returning a cited span and answering “I don’t know”. Multi-hop questions run the same loop twice: the first hop’s reader output becomes a bridge entity that rewrites the query for a second retrieval. The pivotal decisions are hybrid rather than pure dense retrieval (synonyms and exact identifiers both matter in legal and financial text), extractive rather than generative reading (verbatim spans with citations are the product, and a span cannot hallucinate), and a threshold set on the risk-coverage curve rather than on a raw softmax.
(1) Passage Index with Metadata: split documents into 200-300 token passages with overlap, and store document type, date, region, and access-control lists alongside both the inverted index and the ANN index so filters apply at retrieval time.
(2) Hybrid Retrieval: BM25 catches exact identifiers (clause numbers, “Q3”, “EMEA”) while the dense bi-encoder catches paraphrase (“termination” versus “cancellation”); reciprocal rank fusion merges both lists into a top-100.
(3) Cross-Encoder Reranker: joint query-passage encoding narrows 100 → 10 passages and is the largest single quality gain per millisecond in the pipeline.
(4) Extractive Reader with Null Head: start and end logits plus a no-answer score per passage; identical spans from different passages pool their probability.
(5) Iterative Multi-Hop: when hop one yields a bridge entity but no final answer, the query is rewritten and retrieval runs again, capped at two hops to bound latency.
(6) Calibrated Abstention: the null-versus-span margin threshold is chosen on the risk-coverage curve at the product’s tolerated error rate, and abstentions fall back to classic search or a human.

Figure 1: The retrieve-rerank-read path with its two exits, a cited span or an honest abstention, and the dashed multi-hop loop that rewrites the query with a bridge entity.
Clarify Before Designing:
(1) Corpus and Freshness: how many documents and pages, in which languages and formats (scanned PDFs need OCR), and how quickly must a new filing become searchable?
(2) Load and Latency: queries per day, acceptable p95 latency for an interactive answer, and whether a slower two-hop path is tolerable for a minority of questions?
(3) Answer Form: is the product a verbatim span with a citation, or a synthesized paragraph; may the system quote two passages for one answer?
(4) Multi-Hop Share: what fraction of real questions need two passages, and is a bridge through an entity (a counterparty, a region, a fiscal quarter) the dominant pattern?
(5) Labels: do in-domain question-answer pairs exist (support tickets, FAQ pages, past research memos), and what annotation budget do domain experts have?
(6) Error Asymmetry and Access: how much worse is a confidently wrong clause than “I don’t know”, and must answers respect per-user document permissions?
Login to view more content
September 8, 2026
MSD0049 Google Related Searches
When a user enters a query on Google, the search results page shows a list of “related searches” at the bottom: queries that other users have issued in the same session, or queries that are semantically close to the original. These suggestions help users refine their intent, discover alternative phrasings, and navigate to adjacent topics, and they drive a significant fraction of query reformulations. Generating them at scale requires a deep embedding model that maps queries into a semantic space where nearby queries are related, plus a retrieval mechanism that can surface the best candidates from billions of historical queries in milliseconds. The model must handle the long tail of rare queries that have little or no co-occurrence history, distinguish genuinely related queries from merely popular ones, and avoid suggesting queries that lead to harmful or policy-violating content.
How would you design this model? Cover the candidate embedding architectures (dual-encoder trained on query co-occurrence vs contrastive learning on query-query pairs vs LLM-based query expansion), how you build the retrieval index over billions of queries, how you handle the long tail of rare and unseen queries, how you filter harmful or policy-violating suggestions, and how you evaluate related-search quality (click-through rate, reformulation success rate, relevance judgments).

The Problem: billions of past queries could fill the three empty slots under the results, most of them were typed only once, some of them should never be shown, and the choice must be made in a few milliseconds.
Answer
The design is a contrastive query encoder: a BERT-base transformer (110M, 256-d output projection) trained with InfoNCE on query-query pairs harvested from session reformulations and same-click queries, using in-batch negatives plus mined hard negatives and a logQ popularity correction so that cosine similarity means “related” rather than “popular”. Every historical query is embedded once offline into an approximate nearest-neighbor index, and at serve time the live query is encoded and its neighbors are pulled out and filtered. LLM query expansion is deliberately not the online model. It runs offline to synthesize and relabel training pairs and to generate grounded candidates for tail queries that have no usable neighbors. The two pivotal decisions are to train on semantic pairs with explicit popularity debiasing instead of factorizing raw co-occurrence counts, and to keep the LLM offline and grounded so that every displayed suggestion is a real, auditable query.
(1) Dual-Encoder on Co-occurrence: two towers whose training target is the session co-occurrence matrix; a sampled softmax predicts the next or co-session query, which effectively factorizes the count matrix into embeddings.
(2) Contrastive Query-Query Encoder: one shared transformer trained with InfoNCE on curated positive pairs (reformulations, same-click queries, paraphrases), in-batch and mined hard negatives, a temperature, and a logQ correction; retrieval is cosine k-NN over the index.
(3) LLM-Based Query Expansion: a generative model writes related queries directly from the query text, or synthesizes and labels pairs for distillation; generated strings are snapped to real historical queries before use.

Figure 1: The contrastive query encoder is trained once on pairs, embeds the whole historical corpus offline, and serves each live query with one cheap forward pass plus a nearest-neighbor lookup.
Clarify Before Designing:
(1) Candidate Pool: must every suggestion be a real historical query (grounded, auditable), or may the system show generated strings nobody has typed?
(2) Latency and Compute Budget: how many milliseconds and FLOPs per query for encode plus retrieve, and may head queries be precomputed and cached?
(3) Training Signal: which logs exist at what volume: session reformulations, same-click query pairs, human relevance judgments; how many languages?
(4) Definition of Related: refinements, broadenings, lateral topics, or all three, and how much diversity is wanted versus precision?
(5) Tail Share: what fraction of distinct queries and of traffic is seen fewer than a handful of times, and must those queries receive suggestions or is an empty slot acceptable?
(6) Safety Bar: which policy categories apply (health misinformation, harassment, adult content, queries about private individuals), and is any false negative tolerable?
Login to view more content
September 8, 2026
MSD0048 Duolingo Story Difficulty Recommender
Duolingo wants to recommend stories of increasing difficulty as a student learns a language. The app has a library of short reading passages across multiple languages, each with a CEFR level (A1 to C2), and it must serve each learner a story that stretches them just past what they can already read comfortably — challenging enough to push growth but not so hard that the learner gives up. The core challenge is measuring story difficulty and learner ability jointly: a story’s difficulty depends on vocabulary frequency, grammar complexity, sentence length, and concreteness, while a learner’s ability depends on which words and grammar they have mastered so far. Deep knowledge tracing (DKT) can model a learner’s evolving knowledge state from their exercise interactions (correct/wrong, hints used, time spent), and this state can be matched against story difficulty estimates to pick the next story.
How would you design this model? Cover the candidate knowledge tracing architectures (deep knowledge tracing with recurrent networks vs transformer-based knowledge tracing vs item response theory enhanced with neural features), how you estimate story difficulty from text features and historical learner performance, how you combine the learner state with story difficulty to produce a recommendation score, how you handle cold start for new stories and new learners, and how you evaluate recommendation quality beyond simple accuracy.

The Problem: the shelf is sorted by CEFR level, but the learner is not a CEFR level. Which story is one step past what this particular learner can already read, and how would a model know?
Answer
The design is an attention-based knowledge tracer with an IRT-shaped head. A causal transformer reads the learner’s interaction history (which exercise, correct or not, hints used, response time) and emits two outputs: a scalar ability $\theta_t$ on a logit scale and a per-skill mastery vector over the words and grammar points the course teaches. Each story is placed on the same logit scale with a difficulty $\beta_s$ that starts from text features (vocabulary frequency, sentence length, clause depth, concreteness) and is then calibrated on historical learner outcomes. The recommendation score is the predicted comprehension probability $\sigma(\theta_t - \beta_s)$ pulled toward a target band of 0.70 to 0.85, plus a bonus for stories whose skills sit in the learner’s learning zone. The two pivotal decisions are to put ability and difficulty on one shared scale, so a brand-new story can be placed from its text alone, and to window the attention so the per-request cost stays flat as histories grow.
(1) DKT With Recurrent Networks: an LSTM consumes embeddings of (skill, correctness, hint, time) per interaction; its hidden state is the knowledge state and a sigmoid layer reads out per-skill mastery.
(2) Transformer Knowledge Tracing: SAKT, SAINT, or AKT-style causal attention in which the next item queries the interaction history; a distance-aware decay supplies recency and Rasch-style item embeddings supply a difficulty scale.
(3) IRT Enhanced With Neural Features: the Rasch logistic $\sigma(\theta - \beta)$ stays as the output model, while neural front-ends estimate $\theta_t$ from history and $\beta_s$ from story text (Deep-IRT, knowledge tracing machines).

Figure 1: Two estimators on one logit scale: the learner side traces ability and mastery from interactions, the story side places difficulty from text and outcomes, and the match picks the story whose predicted comprehension lands in the target zone.
Clarify Before Designing:
(1) Skill Granularity: is mastery tracked over individual words (thousands per course) or over grammar concepts and word families (hundreds)? This sets the output width and how sparse each skill’s evidence is.
(2) History Length: how many interactions does an active learner accumulate, hundreds or tens of thousands, and must the model attend to all of it or only a recent window?
(3) Inference Budget: is the learner state updated online after every exercise on a CPU budget of a few milliseconds, or recomputed in a nightly batch?
(4) Outcome Signal: do stories carry comprehension checks, or only completion and abandonment, and are hints and response times logged reliably enough to be features?
(5) Per-Language Data: how many learners does the smallest course have, since that number decides whether a transformer is even trainable there?
(6) Difficulty Ground Truth: are the editorial CEFR tags labels the model must respect, or priors it may revise from learner outcomes?
Login to view more content
September 8, 2026
MSD0036 Domain Fine-Tuning Pipeline
Design a fine-tuning pipeline that adapts a foundation model to a company’s domain, including data curation, training strategy, and evaluation. A large enterprise wants to take a general-purpose LLM (e.g., Llama 3 or Mistral) and specialize it for its own support tickets, internal wikis, product manuals, and domain-specific jargon.
The pipeline must curate and clean training data from noisy heterogeneous sources, choose between full fine-tuning, LoRA/QLoRA, and instruction tuning, avoid catastrophic forgetting of general capabilities, and produce a reproducible evaluation that proves the adapted model is better on domain tasks without regressing on general benchmarks.
How would you design this pipeline? Cover the data curation and deduplication strategy, the fine-tuning approach and hyperparameter choices, the evaluation harness (domain-specific benchmarks plus general-capability guardrails), the infrastructure for reproducible runs, and how you decide when the adapted model is ready to replace the base model in production.

The Problem: the base model speaks fluent English but has never read a single one of your change requests, part numbers, or escalation runbooks, and roughly 900M tokens of company text are sitting unused.
Answer
The design is a curate, adapt, gate pipeline: a versioned data factory turns noisy internal sources into a deduplicated, PII-scrubbed corpus plus a hand-reviewed instruction set, a QLoRA adapter is trained on top of a pinned base checkpoint, and a dual-gate evaluation decides promotion. Two decisions dominate. First, adapt with parameter-efficient tuning on a small high-quality mix (domain corpus, instruction pairs, and 5 to 10 percent general replay data) rather than full fine-tuning on a raw dump, because a raw dump buys memorization and forgetting instead of capability. Second, make the promotion decision a two-sided gate: a required lift on a frozen domain benchmark and a bounded regression budget on general benchmarks, both measured by a versioned harness that any engineer can rerun from a run manifest.
(1) Data Factory Before Modeling: exact plus MinHash near-duplicate removal, PII and secret scrubbing, quality filtering, and access-control filtering, all emitting a content-hashed dataset version.
(2) Two-Stage Adaptation: short continued pretraining on the cleaned domain corpus to absorb jargon, then instruction tuning on curated ticket-resolution and doc-QA pairs to restore and shape behavior.
(3) QLoRA as the Default: 4-bit frozen base with rank-16 adapters on attention and MLP projections, so a run costs a single GPU-hour class of compute and the base weights stay bit-identical.
(4) Replay Against Forgetting: mix general instruction data into the SFT set and gate on general benchmarks, so domain gains are never paid for with instruction-following collapse.
(5) Reproducible Runs: pinned base digest, dataset hash, seed, container image, and config in one manifest; every eval number traces back to one manifest.
(6) Adapter Serving with a Kill Switch: serve base plus hot-swappable adapter behind a registry, roll out shadow → 5 percent canary → full, and roll back by unloading the adapter.

Figure 1: The loop: curated and versioned data feeds a cheap adapter run, the harness decides promotion, and production traffic becomes next month’s training data.
Clarify Before Designing:
(1) Target Tasks: which two or three tasks must improve (ticket triage, draft replies, doc QA), and who signs off on the frozen eval set for them?
(2) Corpus Reality: how many tokens survive after access-control and license filtering, and how much of the ticket volume is template boilerplate?
(3) Knowledge vs Behavior: is the gap stale facts (retrieval work) or unfamiliar jargon, format, and tone (fine-tuning work)?
(4) Regression Budget: how many points of MMLU, IFEval, or safety-refusal rate is the business willing to lose for a domain win?
(5) Serving Constraints: self-hosted weights or a closed API, one adapter for everyone or per-business-unit adapters, and what is the latency budget?
(6) Compliance: can customer PII enter training weights at all, and what is the deletion or right-to-forget story once it does?
Login to view more content
September 3, 2026
MSD0035 AI Agent Safeguards
Design the safeguards layer for an AI agent that can take real-world actions on a user’s behalf: booking flights, making purchases, sending messages, or modifying account settings, as envisioned for ChatGPT’s Operator, Anthropic’s computer use, and autonomous agent frameworks. Unlike a chatbot that only produces text, this agent can spend money and send irrevocable messages, so a prompt injection, a misread page, or a goal misinterpretation can cause real harm.
The safeguard system must approve or block each action before execution, verify that the action matches the user’s actual intent, defend against prompt injection from web content the agent reads, and provide a clear audit trail and undo path when something goes wrong.
How would you design this system? Cover the action classification and risk tiers, the intent verification mechanism (confirmation, biometric, second factor for high-risk), the prompt-injection defense for tool-returned content, the human-in-the-loop gate for irreversible actions, and how you evaluate safety without waiting for a real incident.

The Problem: the agent’s inputs include text written by strangers, and its outputs include charges, messages, and settings changes that cannot be taken back. Every action needs a gate before it reaches the world.
Answer
The design is an action firewall that sits between the agent’s planner and every tool that touches the world. Nothing the model “decides” executes directly. Each proposed action is emitted as a typed, structured call, assigned a risk tier by a cheap classifier plus deterministic rules, then evaluated by a policy engine against the capabilities and budget the user actually granted for this session. Two decisions carry most of the safety weight. First, untrusted tool output is quarantined data, never instructions, so page text cannot mint new goals or new capabilities. Second, irreversible actions pass through a human confirmation gate with step-up authentication, and everything else is made cheap to undo through logging, reversible-by-design tool wrappers, and an undo queue.
(1) Typed Action Schema: the agent cannot call raw HTTP or type free-form shell text; it emits validated calls such as purchase(merchant, amount, currency) so risk is computable from arguments, not guessed from prose.
(2) Four Risk Tiers: T0 read-only, T1 reversible writes, T2 spending or external messages, T3 irreversible or identity-changing (password, 2FA, transfers, mass sends); tier fixes the required evidence of intent.
(3) Deterministic Policy Engine: per-session capability grants, allowlists, per-action and per-session spend caps, rate limits, and recipient novelty checks, all enforced outside the model.
(4) Injection Containment: a planner that never sees raw untrusted text plus a quarantined extractor that returns typed values only, with data-flow provenance so tainted values cannot authorize a T2 or T3 action.
(5) Intent Verification Ladder: silent execution for T0/T1, in-context confirmation showing the exact parsed arguments for T2, and passkey or biometric step-up for T3, always rendered from the structured call rather than from model prose.
(6) Audit and Undo: append-only log of prompt, evidence, tier, policy decision, and result; reversible wrappers (hold instead of charge, delayed send, snapshot before settings change) give a real rollback path.

Figure 1: The action firewall: classify → check policy → decide, with untrusted tool output entering as quarantined data and irreversible actions detouring through human confirmation before the sandboxed executor runs.
Clarify Before Designing:
(1) Action Surface: which tools exist on day one (browser clicks, payment API, email, account settings), and can we wrap them in reversible primitives such as authorization holds and delayed sends?
(2) Autonomy Contract: does the user pre-authorize a budget and scope per session, or must every spend be confirmed live? This single answer moves confirmation volume by an order of magnitude.
(3) Harm Asymmetry: what is the cost of a wrongly executed purchase versus a wrongly blocked one, and who eats the loss (user, merchant, or platform)?
(4) Latency and Volume: how many actions per day, and how much of the per-action budget may the safeguard layer consume before the agent feels sluggish?
(5) Identity and Compliance: is a passkey or biometric available on the device, and do payments, health, or messaging rules mandate explicit consent records?
(6) Adversary Model: are we defending against opportunistic web-page injections only, or against attackers who will craft pages specifically targeting our agent?
Login to view more content
August 22, 2026
MSD0034 RAG Customer Support Assistant
Design a RAG-based customer support assistant for a company like Stripe, Shopify, or a large telecom. The assistant is grounded in the company’s own knowledge base: help articles, product documentation, past resolved tickets, and policy pages. It must answer accurately, cite its sources, refuse to fabricate when the answer is not in the knowledge base, escalate to a human when the question is high-stakes (billing, account access, legal) or when its confidence is low, and stay current as the knowledge base is edited every day.
The cost structure is asymmetric: a hallucinated refund policy or a confident wrong answer about account access is far worse than admitting ignorance and handing the conversation to an agent. A wrong answer can create a chargeback, a regulatory complaint, or a churned account, while an escalation costs a few dollars of agent time.
How would you design this system? Cover the retrieval architecture (chunking, embedding, hybrid search, re-ranking), the generation and grounding strategy, the confidence and safety gating that decides when to answer versus escalate, the human handoff path with full context, and how you evaluate answer quality and hallucination rate.

The Problem: the answer either exists somewhere in a knowledge base that changes daily, or it does not exist at all, and the system has to tell those two cases apart before it opens its mouth.
Answer
The design is a retrieve, re-rank, generate pipeline wrapped in an explicit answer policy. Hybrid retrieval over structure-aware chunks pulls 50 candidates, a cross-encoder re-ranker trims them to 8, and the generator may only assert what a cited passage actually says. Two decisions carry the answer. First, abstention and escalation are first-class outputs: a risk classifier runs before generation, a retrieval-confidence threshold guards coverage, and a per-claim groundedness verifier checks the draft, so the default under doubt is a warm human handoff rather than a fluent guess. Second, the index is versioned and refreshed nightly by delta, with product, plan, and region metadata as hard filters, because most support “hallucinations” are really retrieval of the wrong but plausible policy version.
(1) Structure-Aware Chunking: split on headings at roughly 400 tokens with 15% overlap, keep tables and code blocks intact, and prepend the article title and breadcrumb so an isolated chunk is self-describing.
(2) Rich Chunk Metadata: product, plan, region, locale, source type, version, and effective date travel with every chunk and act as hard filters before scoring.
(3) Hybrid Retrieval Plus Re-ranking: BM25 catches error codes, SKUs, and API field names; dense embeddings catch paraphrase; fusion gives 50 candidates and a cross-encoder picks the top 8.
(4) Citation-Constrained Generation: numbered passages in the prompt, sentence-level citation ids in a structured output, low temperature, and an explicit insufficient_context abstain option.
(5) Three-Gate Answer Policy: risk classifier, retrieval-confidence threshold, and claim-level groundedness check; any gate that fires routes to a human instead of shipping the text.
(6) Warm Handoff and Content Loop: the agent inherits the transcript, the exact passages used, the extracted entities, and a draft reply; the agent’s final answer becomes an eval label and, when the gap is real, a new article.

Figure 1: One pass per question: rewrite and risk-classify, retrieve and re-rank, generate only from the surviving passages, then let the gate decide whether the customer sees the text or an agent does.
Clarify Before Designing:
(1) Volume and Channels: how many chat sessions per day, how many turns each, which languages, and is the user authenticated when they ask?
(2) Quality Bar and Deflection Target: what accuracy floor must hold for answers we do give, and who signs off on the refusal rate that buys it?
(3) Knowledge Base State: how much content is written for customers versus internal agents, are policies versioned by region and plan, and are ticket resolutions quality-reviewed?
(4) Account Access: may the assistant read account state (plan, invoices, outage status) through tools, or only public docs? This changes both retrieval and the risk gate.
(5) Latency and Cost Budget: target first token, target full answer, and the per-conversation cost ceiling relative to an agent contact.
(6) High-Stakes Definition: which intents are legally or financially sensitive (billing disputes, cancellation, identity, data deletion) and what retention rules apply to logged conversations?
Login to view more content
August 22, 2026
MSD0026 Multimodal Text-to-Image-Video Search
Design a multimodal search system where a user types a text query and the system returns relevant items from a catalog of billions of images and videos, like Pinterest Visual Search, Google Lens, or TikTok’s visual search. The query may describe objects, scenes, styles, or actions that never appear as literal text anywhere in the catalog (for example “golden retriever catching a frisbee at sunset”), so the system must bridge the modality gap between language and visual content.
Results must return in under 200 ms, the index must stay fresh as new content is uploaded continuously, and the system must handle queries that mix text with an uploaded reference image.
How would you design this system? Cover the shared embedding space and training strategy, the approximate nearest neighbor index at billion scale, the freshness and update strategy, the hybrid text-plus-image query path, and how you evaluate retrieval quality.

The Problem: the words the user types appear nowhere in the catalog, yet the right pixels have to surface out of billions of items inside 200 ms, while new uploads keep arriving.
Answer
The design is a two-tower (dual encoder) retrieval system over one shared embedding space, served as a two-stage retrieve-and-rerank path. A contrastive image-text model of the SigLIP/CLIP family, fine-tuned on in-domain query-engagement pairs, maps text, images, and video clips into the same 512-d space, so “golden retriever catching a frisbee at sunset” becomes a vector whose neighbors are the right pixels even though no matching caption exists. Two decisions carry most of the weight. First, the item side is multi-vector for video (a handful of clip embeddings rather than one averaged vector), because averaging a 60-second video destroys the action the query describes. Second, freshness is solved with a tiered index (hot in-memory buffer, hourly delta shards, nightly rebuilt base) instead of mutating one billion-scale graph in place. The hybrid text-plus-image query runs through a learned combiner that emits a single vector, so the retrieval and ranking path downstream never changes.
(1) Shared Embedding Space: one contrastively trained space for query text, images, and video clips, so retrieval is pure vector search rather than text matching against noisy alt-text.
(2) In-Domain Fine-Tuning: start from a public image-text checkpoint, then fine-tune on click and save pairs from real query logs, with hard negatives mined from the same logs.
(3) Multi-Vector Video Items: sample frames or short clips per video and index several vectors per item, collapsing to one item at merge time; single pooled vectors lose action and scene changes.
(4) Sharded IVF-PQ With Rerank: compress to 64-byte codes so 5 billion vectors fit in roughly 320 GB of RAM across shards, retrieve about 10k candidates, then rescore with full-precision vectors plus business features.
(5) Tiered Freshness: a hot buffer makes new uploads searchable in about a minute, hourly deltas absorb the day, and a nightly base rebuild keeps the big index compact; deletes ride a tombstone list.
(6) Learned Query Combiner: text-plus-image queries are fused into one vector by a small trained combiner, keeping the index path and latency budget identical to text-only search.

Figure 1: One embedding space, two write paths: queries are encoded online, catalog items are encoded on ingest, and the tiered index is the only place they meet.
Clarify Before Designing:
(1) Scale and traffic: how many items (billions of images, or billions of items with heavy video), what peak QPS, and how many uploads per hour?
(2) Latency budget split: is the 200 ms end-to-end including network and thumbnail fetch, or only the retrieval service?
(3) Freshness requirement: must a new upload be findable in seconds (live events, breaking trends) or is an hour acceptable for most content?
(4) Query mix: what fraction is head navigational text, tail descriptive text, action queries over video, and hybrid image-plus-text?
(5) Quality bar and cost asymmetry: is the product optimizing engagement, commercial conversion, or safety-first precision, and what is an acceptable cost per 1k queries?
(6) Available supervision: do we have query-click logs and human relevance ratings, or only alt-text and hashtags?
Login to view more content
August 22, 2026