Interview for Machine Learning

Category: ML

MSD0024 LinkedIn Job Recommendation
Design the job recommendation system for LinkedIn Jobs. When a member visits the Jobs page or receives a job alert email, the system must rank open positions from a catalog of millions by how likely the member is to apply and, ultimately, to get hired.
The ranking must respect hard constraints (location, work authorization, years of experience, salary expectations), handle the cold-start problem for new members and newly posted jobs, and avoid recommending jobs the member has already seen or dismissed. LinkedIn’s two-sided marketplace means the system also affects employers, who pay for applications and churn if candidate quality is poor.
How would you design this system? Cover the candidate generation and ranking funnel, the feature set (member profile, job requirements, interaction history), how you handle cold start, the online and offline metrics, and how you balance member and employer value.

The Problem: one member with hard eligibility constraints, millions of open postings, and a recruiter on the other side who pays for applications and leaves if the applicants are unqualified.
Answer
The design is a standard four-stage recommender funnel wrapped around a two-sided objective: hard-constraint filtering, multi-source candidate generation down to roughly a thousand jobs, a multi-task ranker, then a policy re-rank layer that enforces marketplace and compliance rules. The pivotal design decision is what the ranker optimizes. Optimizing apply probability alone maximizes member clicks and floods a few popular postings with unqualified applications, so the score blends P(apply), P(recruiter treats the application as qualified), and a delayed P(hire) head, which is the objective LinkedIn’s own confirmed-hires framing points at. The second pivotal decision is to treat eligibility as a pre-filter in the retrieval index rather than a learned soft feature, because a job the member cannot legally take is a wasted slot for both sides.
(1) Hard Constraints As Pre-Filters: location radius or remote flag, work authorization, seniority window, and compensation band are boolean index filters applied before scoring, never soft features the ranker can trade away.
(2) Multi-Source Candidate Generation: a two-tower ANN over job embeddings, plus saved searches and alerts, plus graph sources (“members with your title applied here”) and company affinity, unioned to about 1000 candidates.
(3) Multi-Task Ranker: one shared-bottom or MMoE model with heads for apply, qualified-application, and confirmed hire; the serving score is a weighted combination whose weights are the marketplace policy knob.
(4) Cold Start By Content Plus Exploration: both towers are content-based (title, extracted skills, seniority, company), so new members and postings get vectors immediately, and new postings receive a small guaranteed impression budget.
(5) Policy Re-Rank Layer: deduplicate seen and dismissed jobs, cap applicant load per posting, spread traffic beyond the head of the catalog, and apply fairness-aware re-ranking.
(6) Two Serving Paths: the Jobs page runs the funnel online under a ~200 ms budget; alert emails run the same models in a nightly batch with frequency caps and a higher score threshold.

Figure 1: The funnel narrows 20M postings to 25 shown jobs, and the loop back through label joining is what turns applies, recruiter replies, and hires into tomorrow’s model.
Clarify Before Designing:
(1) Surface And Budget: Jobs page (online, latency-bound) or alert email (batch, frequency-bound), and what end-to-end latency budget does the page allow?
(2) Label Availability: do we have recruiter-side signals (application viewed, candidate contacted) and confirmed hires, or only member-side clicks and applies?
(3) Catalog Scale And Freshness: how many open postings, what is the median posting lifetime, and how quickly must closed jobs disappear?
(4) Monetization: are promoted or sponsored jobs blended into the same slots, and is there a separate auction and pacing layer?
(5) Compliance: which fairness and audit requirements apply (employment discrimination law, automated hiring-tool bias audits), and which member attributes are forbidden as features?
(6) Employer Capacity: do postings expose applicant caps or response-time signals we can use to protect candidate quality?
Login to view more content
August 22, 2026
MSD0023 YouTube Copyright Detection
Design the copyright detection system for a video platform like YouTube. Every uploaded video (hundreds of hours per minute) must be scanned against a reference corpus of tens of millions of copyrighted songs, movies, and clips, and matched even when the content is pitch-shifted, cropped, overlaid with voiceover, recorded off a screen, or remixed. The system must return matches within minutes so that rightsholders can block, monetize, or track the use, and it must keep the false-positive rate low enough that legitimate fair use and original content are not wrongly claimed.
How would you design this system? Cover the audio and video fingerprinting representations, the matching index at scale, the latency and throughput budget for the upload firehose, how you handle partial and transformed matches, and how disputes and appeals are routed.

The Problem: the copy is never bit-identical, the corpus is tens of millions of references, the answer is owed in minutes, and a wrong claim silences a legitimate creator.
Answer
The design is a two-stage fingerprint retrieval system: cheap, high-recall candidate generation from compact per-modality fingerprints, then strict temporal verification that only accepts a candidate if many hash hits agree on a single time offset. Audio and video are fingerprinted separately because their attack surfaces differ. Audio uses spectral peak landmark hashes plus a neural fallback for pitch and tempo edits, while video uses learned copy-detection embeddings that survive cropping, overlays, and screen recording. The two pivotal decisions are to make the verification stage, not the recall stage, own precision, and to convert model confidence into tiered actions (auto-claim, human review, track-only) so that an uncertain match costs a reviewer’s time rather than a creator’s channel.
(1) Dual-Modality Fingerprints: audio landmark hashes of the form $(f_1, f_2, \Delta t)$ at roughly 30 hashes per second, and video frame descriptors sampled at 2 to 4 fps after letterbox and border normalization.
(2) Two-Stage Retrieval: a sharded inverted index (plus ANN search over neural embeddings) proposes a few hundred candidate references per query chunk; verification rejects almost all of them.
(3) Offset Voting And Segment Stitching: a real match produces a spike in the histogram of $t_{ref} - t_{query}$ ; adjacent winning chunks are stitched into segments with a minimum matched duration.
(4) Chunked Parallel Ingest: uploads are split into 10-second chunks that fingerprint in parallel, with a priority queue so premieres and large channels finish first.
(5) Tiered Actions: high confidence auto-claims, medium confidence routes to human review or track-only, low confidence is logged and dropped.
(6) Dispute Loop As Training Data: every appeal outcome becomes a label, feeding whitelists, threshold retuning, and reference-corpus cleanup.

Figure 1: One path for uploads (fingerprint → lookup → verify → policy) and one offline path that fingerprints rightsholder references with the same extractor; disputes flow back from the policy engine into verification thresholds and whitelists.
Clarify Before Designing:
(1) Scope Of Rights: do we match sound recordings only, or also compositions and melodies, since a cover song copies the composition without copying any audio?
(2) Verdict SLA: must the scan block publishing, or may the video go live and be claimed minutes later, and does the SLA differ for monetized channels?
(3) Corpus Scale And Registration: how many references, how fast is the corpus growing, and who is allowed to register a reference?
(4) Cost Asymmetry: what false-claim rate is acceptable versus missed detections, and who pays for human review of the uncertain band?
(5) Transformation Catalog: which edits must be caught (crop, pitch shift, speed change, voiceover, screen capture, remix, AI covers), because that choice decides the fingerprint representation?
(6) Available Actions: block, monetize, or track, and are the policies territory-specific?
Login to view more content
August 22, 2026
MSD0022 Model Drift Dashboard
A large platform runs thousands of ML models in production: ranking models, fraud classifiers, content moderators, recommendation embedders, each with its own feature pipeline, serving path, and delayed-label feedback. A single broken upstream feature, a silent model upgrade, or a gradual distribution shift can degrade any of them for days before anyone notices.
The ML platform team needs one real-time monitoring system that surfaces drift, performance, and data-quality issues across the entire fleet, with alerting that catches real regressions without drowning on-call engineers in false alarms. How would you design this system? Cover the metrics each model emits, the streaming and storage architecture for thousands of models, the drift and anomaly detection methods that scale across heterogeneous model types, the alerting and triage workflow, and how you measure the monitoring system itself (mean time to detect, alert precision).

The Problem: one model in a fleet of thousands is quietly getting worse, and the on-call engineer is already ignoring a pager full of alerts that were never real.
Answer
The design is a fleet-wide observability service with three layers. Every serving path emits a uniform telemetry record through a shared logging SDK, a streaming job compresses those records into per model, feature, slice and window sketches, and a detection layer runs cheap statistical guards on everything plus expensive label-joined performance checks on the models that carry real money. The two pivotal decisions are that detection never scans raw events at query time (it reads compact profiles, which is what makes thousands of models affordable), and that alerting is engineered for precision rather than coverage: a trigger must clear an effect-size floor, persist across several windows, and survive root-cause grouping before it pages anyone. The monitoring system is itself a product with SLOs, scored on mean time to detect and alert precision against continuously injected synthetic regressions.
(1) Uniform Telemetry Contract: one prediction record schema (model id and version, feature values or hashes, score, slice keys, join key, latency) emitted asynchronously by a sidecar or SDK, so the platform never reimplements logging per team.
(2) Sketch-Based Aggregation: a streaming job keeps t-digest quantiles, count-min sketches for categoricals, and null/count/cardinality counters per 5-minute window, turning terabytes of events into tens of gigabytes of profiles.
(3) Tiered Detection: tier-1 data-quality and drift guards run on every model and feature; tier-2 label-joined AUC, calibration, and slice regressions run only on business-critical models.
(4) Delayed-Label Handling: proxy signals (score distribution, positive rate, confidence) fire in minutes, while true performance is computed on maturity-aware cohorts so immature labels never trigger a page.
(5) Precision-Engineered Alerting: adaptive seasonal baselines, an effect-size floor, multi-window persistence, and deduplication by feature lineage turn thousands of triggers into a handful of owned incidents.
(6) Self-Measurement: synthetic drift and corrupted-feature injections give a labeled incident corpus, so MTTD and alert precision are measured, not asserted.

Figure 1: Raw predictions are compressed once into profiles; every detector, dashboard, and alert afterwards reads those profiles instead of the event stream.
Clarify Before Designing:
(1) Fleet Shape: how many models, how many features each, aggregate QPS, and how many are business-critical versus experimental?
(2) Detection Budget: is the target time to detect minutes, an hour, or a day, and is it the same for a fraud classifier and a feed ranker?
(3) Label Latency: per use case, how long until ground truth arrives (clicks in seconds, chargebacks in weeks, moderation appeals in days)?
(4) Cost Ceiling: what fraction of serving cost may monitoring consume, and can we sample rather than log every request?
(5) Remediation Authority: may the system auto-rollback a model version or quarantine a feature, or does it only page a human owner?
(6) Data Constraints: which features are PII or regulated, and what is the retention limit on stored feature values?
Login to view more content
August 22, 2026
MSD0021 Hashtag Video Relevance
On a short-video platform like TikTok or Instagram Reels, creators attach hashtags to reach an audience, and a large share of those tags are clickbait: #fyp or a trending sports tag stapled onto a video that has nothing to do with the topic behind it. The core problem is a multimodal relevance scoring mechanism. Given a video (sampled frames, audio, OCR’d on-screen text, creator caption) and a candidate hashtag, produce a relevance score that separates honest tagging from spam.
How would you design this model? Cover the candidate fusion architectures (late fusion of per-modality encoders, early token-level fusion, cross-attention, contrastive hashtag-video embedding), how you represent the hashtag itself (its text, its trending-topic cluster, its co-occurrence graph), how you generate training labels at scale when human-labeled relevance is scarce, and how you handle cold start for a brand-new hashtag with no co-occurrence history.

The Problem: the same video carries an honest tag and a hijacked trending tag, and nothing in the upload tells you which is which. You need a score for every (video, tag) pair, over a tag vocabulary of tens of millions and with almost no human relevance labels.
Answer
The design is a two-stage multimodal relevance model. A contrastive dual tower is pretrained on billions of weakly labeled creator (video, tag) pairs, which gives every tag in the vocabulary a cheap vector and initializes both encoders. A cross-attention scorer then rescores the handful of tags actually attached to a video, letting the tag’s query tokens read the video’s frame, audio, OCR, and caption tokens so a localized mismatch (a football tag on a pasta video, or a single on-screen word borrowed from a trend) becomes visible. Two decisions are pivotal. First, the hashtag is a sum of three slots (subword text, trending-topic cluster centroid, co-occurrence graph embedding) trained with slot dropout, which is what makes cold start work rather than a special case bolted on later. Second, labels come from creator behavior plus frequency-matched hard negatives drawn from the same topic cluster, anchored by a small human gold set used only for calibration and as the release gate.
(1) Late Fusion: one encoder per modality, each pooled to a vector, concatenated with the tag vector into a shallow MLP; everything cacheable, no token-level interaction.
(2) Early Fusion: all modality tokens plus the tag tokens in one transformer from layer 1; maximally expressive and completely uncacheable across candidate tags.
(3) Cross-Attention Scorer: video tokens encoded once as keys and values, hashtag slots as a small query set, a few cross-attention layers plus a sigmoid head.
(4) Contrastive Embedding: dual-tower video and tag encoders trained with InfoNCE or a sigmoid pairwise loss so relevance is a dot product in a shared space.

Figure 1: The scoring mechanism: the video is encoded once into a token set, the hashtag arrives as three composable slots, and cross-attention decides whether the video contains evidence for the tag.
Clarify Before Designing:
(1) Input budget: how many frames per video, is audio and reliable OCR available, and is the caption trustworthy or itself keyword-stuffed?
(2) Compute budget: how many candidate tags are scored per video, and may the video encoder run once at upload so per-tag cost is the only thing multiplied?
(3) Pretrained assets: is there an in-house video-language encoder to freeze, or must the towers be trained from scratch on platform data?
(4) Vocabulary shape: how large is the tag vocabulary, what fraction of traffic is meta-tags such as #fyp, and how fast do new tags appear?
(5) Label reality: how many human-labeled pairs exist today, and can we buy a gold set for calibration even if we cannot buy a training set?
(6) Output contract: is the consumer a hard threshold, a ranking feature, or a reviewer queue, since only the first two need a calibrated probability?
Login to view more content
August 22, 2026
MSD0020 Distributed Training System
Design the distributed training system for a 100B+ parameter language model on a cluster of thousands of GPUs, the way Meta, OpenAI, and DeepSeek run their frontier models. A single run spans weeks to months, costs millions of dollars in compute, and must survive GPU failures, network bottlenecks, and stragglers without restarting from scratch.
The system has to combine data, tensor, and pipeline parallelism, checkpoint efficiently, and keep GPU utilization high enough that the run finishes on budget.
How would you design it? Cover the parallelism strategy and communication topology, fault tolerance and checkpointing, cluster scheduling, and how you diagnose and fix the bottlenecks that keep utilization below target.

The Problem: the model does not fit on one GPU, the cluster is big enough that something breaks every few hours, and every idle GPU-second is billed. The design question is how to keep thousands of accelerators busy on one gradient for a month.
Answer
The design is 3D parallelism mapped onto the bandwidth hierarchy, wrapped in an elastic supervisor that treats hardware failure as a routine event. Tensor parallelism stays inside a node where NVLink is cheap, pipeline parallelism crosses nodes because only stage activations travel, and data parallelism with sharded optimizer state sits outermost where the gradient all-reduce can be overlapped with the backward pass. Fault tolerance is asynchronous distributed checkpointing at a mathematically chosen interval plus a hot-spare pool, so a dead GPU costs minutes rather than the run. The single number the whole system is managed against is MFU (model FLOPs utilization), because at this scale every lost percentage point is real money, and every design choice below is justified by its effect on that number.
(1) Bandwidth-Matched Parallelism: tensor parallel degree capped at the intra-node GPU count, pipeline parallel across nodes, data parallel outermost; context parallelism added only when sequences get long.
(2) Sharded State And Selective Recompute: optimizer states and gradients sharded across the data-parallel group, with selective activation recomputation trading roughly 8% extra FLOPs for the memory that makes the fit possible.
(3) Async Checkpointing At The Optimal Interval: copy state to host memory in seconds, flush to storage in the background, and set the interval from measured cluster MTBF instead of habit.
(4) Elastic Recovery With Hot Spares: a watchdog detects hangs and stragglers, a spare node replaces the failed one in place, and training resumes from the last checkpoint including data-loader position.
(5) Topology-Aware Gang Scheduling: the job is placed inside one network island as an all-or-nothing gang, after a preflight burn-in that rejects slow links and weak GPUs before the run starts.
(6) MFU As The Run’s SLO: per-rank step-time histograms, profiler traces, and a loss budget that attributes every missing point of peak FLOPs to a named cause.

Figure 1: One step is load → forward → backward → reduce → update → stage checkpoint. The lower loop is what makes a month-long run survivable: detect, replace, restore.
Clarify Before Designing:
(1) Hardware And Interconnect: how many GPUs per node, what intra-node link (NVLink or PCIe), and what inter-node fabric bandwidth per GPU?
(2) Token Budget And Deadline: how many training tokens, and is the constraint calendar time, total GPU-hours, or dollars?
(3) Sequence Length: is context 8k or 128k, because long context changes activation memory and adds a whole parallelism axis?
(4) Precision Policy: is FP8 or BF16 allowed for GEMMs, and what loss-curve deviation from a reference run is acceptable?
(5) Storage And Checkpoint SLA: what aggregate write bandwidth exists, and how much lost work per failure is tolerable?
(6) Cluster Ownership: dedicated reservation or shared preemptible pool, and how large a hot-spare pool can be held idle?
Login to view more content
August 21, 2026
MSD0019 On-Device Small LLM Polite
Design a small LLM that runs entirely on a phone for a privacy-preserving assistant, in the style of Apple Intelligence’s on-device foundation model or Gemini Nano on Pixel. The model must fit in roughly 2-4 GB of RAM, respond in under 300 ms on a mobile NPU, and work with the radio off.
It also has to stay polite. It should refuse harmful requests, avoid toxic or biased outputs, and hold a respectful tone even when the user is deliberately provoking it. Battery life and offline capability matter as much as raw benchmark quality, and no user text may leave the handset.
How would you design this system? Cover model compression and quantization for the NPU, the alignment strategy that keeps the model polite at a small scale, the on-device serving path, and how you evaluate safety without sending user data to the cloud.

The Problem: a few gigabytes of RAM, a fraction of a second, no network, and a user who is testing how rude the assistant will get.
Answer
The design is a distilled 3B-parameter model compressed with quantization-aware training to about 2 bits per weight, served on the NPU with per-feature LoRA adapters and a tiny always-resident guard model. Politeness is trained in rather than prompted in: a large aligned teacher supplies refusal and tone supervision, a preference pass sharpens “polite but not preachy”, and safety data stays inside the quantization loss so alignment survives compression. Two decisions are pivotal. First, safety is re-verified after quantization, because low-bit weights erode refusal behavior faster than they erode average benchmark quality. Second, a separate 100M-parameter guard screens the prompt and the streamed output, so a single jailbroken generation is caught by a component that can be patched over the air without retraining the base model.
(1) Distill + QAT + Adapters: prune and distill a large teacher into ~3B parameters, then quantization-aware train to 2-4 bits and ship 16-bit LoRA adapters per feature (summarize, reply, rewrite).
(2) Scratch-Trained Small Model + PTQ: train a 1-2B model on curated data and apply post-training 4-bit quantization; simplest pipeline, weakest quality per byte.
(3) Elastic Nested Model: a MatFormer-style checkpoint whose submodels can be extracted at different sizes, letting the runtime trade quality for RAM under memory pressure.
(4) On-Device Draft + Private Cloud Escalation: answer locally by default and escalate hard prompts to a privacy-hardened server tier with explicit consent.

Figure 1: Everything inside the dashed boundary runs on the handset: guard, decode loop, adapters, and cache. Nothing crosses the boundary, so every check has to happen before pixels appear.
Clarify Before Designing:
(1) Latency Definition: is 300 ms the time to first token or the full reply? Streaming a 60-token answer at 30 tokens per second takes two seconds no matter how good the prefill is.
(2) Task Set and Context: which features ship (reply suggestions, summarization, rewriting), and what is the maximum prompt length? Prefill cost and KV cache size follow directly from that number.
(3) Device Floor: which chip generation is the minimum, and what does its NPU actually execute natively (INT8, INT4, palettized weights, per-channel scales)?
(4) Escalation Policy: is any server path permitted with consent, or is strict offline operation a product promise we cannot break?
(5) Safety Bar: which refusal taxonomy applies, and what false-refusal rate on benign-but-edgy prompts is acceptable? Over-refusal is the failure users complain about.
(6) Telemetry Rules: may we collect differentially private aggregate counters and opt-in donated conversations, or literally nothing?
Login to view more content
August 21, 2026
MSD0018 Ads Policy Violation Detection
Design an end-to-end ML system to detect ads selling weapons and other policy-violating goods on a large ads platform like Meta or Google Ads. Millions of new ad creatives arrive daily as text, images, and video, and advertisers constantly adapt their creative to evade detection. Google’s Ads Safety Report describes blocking or removing billions of ads per year, so this is an industrial-scale enforcement problem, not a one-off classifier.
The system must block prohibited ads before they serve, route borderline cases to human reviewers, and keep the false-positive rate low enough that legitimate advertisers are not churned.
How would you design this system? Cover the multimodal detection model, the human-review queue prioritization, the adversarial adaptation problem, and how you measure recall and precision at platform scale.

The Problem: millions of new creatives arrive every day, roughly one in ten thousand is a prohibited-goods listing, and the decision must be made before the ad is shown even once.
Answer
The design is a pre-serve multimodal gate inside the ad-ingestion path. A cheap first stage kills known-bad creatives by hash and embedding near-duplicate lookup, then a shared vision-language encoder with per-policy heads scores every creative (ad copy, OCR text, landing-page text, image, video keyframes) before the first impression. Each policy class gets three calibrated bands: auto-block, human review, allow. Two decisions carry the design. First, thresholds are set from the asymmetric cost of a served weapon ad versus a wrongly blocked advertiser, per policy and per locale, rather than from a single global score cut. Second, the whole system is treated as an adversarial loop where the time from a new evasion pattern to a deployed block (hours, not quarters) matters more than static offline AUC.
(1) Staged Funnel: creative-hash cache and embedding kNN against known violations resolve most repeat traffic in milliseconds; only novel creatives pay for the full multimodal pass.
(2) Multimodal Model: a frozen vision tower plus a multilingual text tower over copy, OCR, ASR, and landing-page text, fused into multi-label per-policy heads instead of one binary “violating” output.
(3) Three-Band Decision Policy: per-policy calibrated thresholds split traffic into auto-block, review, and allow, so precision on the auto-block band is protected by construction.
(4) Expected-Harm Review Ranking: the reviewer queue is sorted by violation probability times forecast impressions times policy severity, not by score alone, and clusters of near-duplicate creatives collapse into one decision.
(5) Adversarial Response Loop: evasion signals (re-upload similarity, account bursts, appeal patterns) feed hard-negative mining, an embedding blocklist hotfix within hours, and account-level enforcement that raises the attacker’s cost.
(6) Measurement by Sampling: recall is estimated from stratified random audits of allowed traffic, precision from reviewer audits of blocks and from appeal-overturn rates.

Figure 1: Ingest → asset extraction → encoders → per-policy scores → three-band decision, with reviewer labels and the near-duplicate index closing the loop back into the model.
Clarify Before Designing:
(1) Scale and Asset Mix: how many new creatives per day, how many assets each, and what share is video (video dominates compute cost)?
(2) Enforcement SLA: must an ad go live within minutes of upload, or is holding a creative for a deep asynchronous pass acceptable?
(3) Policy Taxonomy: how many prohibited-goods classes, which are legally mandated, and how much do they vary by jurisdiction?
(4) Cost Asymmetry: what is the tolerated false-block rate before advertiser churn becomes the bigger loss, and what is the appeal SLA?
(5) Review Capacity: how many reviewer hours per day, in which languages, and with what accuracy on gold sets?
(6) Label Availability: do we have historical enforcement decisions, appeal outcomes, and account-level history, or are we starting from policy text only?
Login to view more content
August 21, 2026
MSD0017 Meeting Summarization
Design the system behind Zoom AI Companion’s meeting summary: after a 45-minute video meeting, every participant receives a recap, the decisions made, and their action items within a minute or two. The pipeline must handle cross-talk, accents, and bad microphones across ten speakers, and a hallucinated action item is worse than none at all.
How would you design this system? Cover transcription, speaker attribution, the summarization stack, quality control, privacy, and cost at platform scale.

The Problem: forty-five minutes of overlapping speech from ten microphones must become a recap, a decision list, and attributed action items, within minutes, and one invented task destroys trust. Design the pipeline that was actually listening.
Answer
The design is a staged pipeline with a federated summarization stack, the approach Zoom publishes for AI Companion: streaming ASR turns audio into words, speaker diarization attributes them to people, a post-processing stage restores punctuation and domain terms, and the summarization stage drafts on a small in-house LLM, scores the draft with a learned quality model (Zoom’s Z-Scorer), and escalates to a stronger external LLM only when the score is low. The two pivotal decisions are separating “who spoke” from “what was said” (diarization plus ASR reconciled with the meeting roster, not one giant model asked to guess names) and never paying frontier-model prices for meetings a small model summarizes fine (Zoom reports comparable quality at roughly 6% of GPT-4 cost).
(1) ASR Frontend: streaming encoder-decoder (Whisper-class) with custom vocabulary biasing for product and people names; about 95% word accuracy on real meetings.
(2) Speaker Diarization: embedding-based turn detection (pyannote-style) reconciled with the participant roster and per-channel audio when available, so “Speaker 2” becomes “Ada”.
(3) Structured Summarization: chained tasks with schema-checked JSON output: chapter segmentation, recap, decisions, action items with owners.
(4) Federated Quality Control: small model drafts, a learned scorer predicts summary quality, low scores escalate to stronger LLMs, and a committee-of-LLMs cross-check suppresses hallucinated action items.
(5) Privacy by Inheritance: summaries inherit the meeting’s access control and retention; a self-hosted-only model path serves regulated customers.

Figure 1: The pipeline: hear the words, attribute them, then summarize with schema-checked structure; the scorer, not blind trust, decides when a stronger model is worth paying for.
Clarify Before Designing:
(1) Live versus Post: in-meeting “catch me up” queries (seconds, incremental) or post-meeting delivery only (minutes, batch)?
(2) Language Scope: English only, or 30-plus languages via translation assists or multilingual models?
(3) Identity Source: is a participant roster, per-channel audio, or voice enrollment available, or must speakers stay anonymous labels?
(4) Worst Error: which failure is most expensive here: hallucinated action items, missed decisions, or wrong attribution?
(5) Scale and Budget: meetings per day, average length, and the cost ceiling per meeting-minute?
Login to view more content
August 9, 2026
MSD0016 Scanned Document OCR
Design an OCR system that converts millions of scanned documents (contracts, invoices, old books, fax-quality pages) into clean, structured text for a document platform like AWS Textract, Google Document AI, or Mistral’s OCR API. The output must preserve reading order, tables, and equations, not just raw characters, because the text feeds search indexes and RAG pipelines downstream.
How would you design this system? Cover the recognition approach, layout and structure handling, accuracy measurement, and the cost of processing at scale.

The Problem: a skewed, coffee-stained scan of a two-column contract must come out as correctly ordered, structured text, at millions of pages, at a cost a business can pay. Design the reader that never silently scrambles a page.
Answer
The design is a single-pass document VLM pipeline in the style of AI2’s olmOCR: lightweight preprocessing normalizes each page (orientation, deskew, denoise, blank rejection), then a fine-tuned 7B-class vision-language model reads the whole page in one pass and emits structured output (Markdown for headings, HTML for tables, LaTeX for math), and programmatic unit tests verify each page (does the table parse? does the equation recompile? is reading order sane?) before failures are retried or routed to a stronger model or a human queue. The two pivotal decisions are reading whole pages in one pass instead of a detect-then-crop-then-recognize cascade (which accumulates errors and loses reading order), and defining correctness as machine-checkable tests so both evaluation and training optimize what downstream consumers actually need.
(1) Single-Pass VLM Reader: a fine-tuned Qwen2.5-VL-class model turns a page image directly into structured Markdown/HTML/LaTeX, skipping the multi-stage cascade.
(2) Document Anchoring: born-digital PDFs contribute their embedded text and metadata into the prompt alongside the page image, grounding the model and cutting hallucination.
(3) Preprocessing Gate: orientation detection, deskew, denoise, and blank-page rejection run before the expensive model call.
(4) Verifiable QA: per-page unit tests (table parses, math recompiles, reading order monotonic); failures retry with different sampling or escalate to a stronger model or human review.
(5) Batch-First Serving: offline GPU batches with vLLM/SGLang; a tuned open pipeline processes a million pages for roughly $190, about 1/32 of frontier-API pricing.

Figure 1: The pipeline: cheap normalization first, one VLM read per page, then machine-checkable tests decide whether the page ships, retries, or goes to a human.
Clarify Before Designing:
(1) Document Mix: born-digital PDFs versus degraded scans; which languages; is handwriting in scope?
(2) Structure Fidelity: plain text, or RAG-grade structure with tables, equations, and reading order preserved?
(3) Volume and Latency: nightly millions in batch, or interactive upload-and-wait in seconds?
(4) Deployment Constraints: customer data residency or full self-hosting requirements (regulated archives)?
(5) Downstream Contract: does the consumer want Markdown, or structured JSON field extraction (invoice totals, dates) on top?
Login to view more content
August 9, 2026
MSD0015 LLM Request Routing
Design the routing layer for an AI assistant that chooses, per request, between a small cheap model and a large expensive one. ChatGPT’s GPT-5 does exactly this with a real-time router that picks between its fast and reasoning variants, and research frameworks like RouteLLM show a trained router can cut serving cost by over 80% while holding about 95% of the strong model’s quality.
How would you design this system? Cover what the router sees and how it is trained, how answer quality is guarded, the cost accounting, and how the system adapts as models are added, upgraded, or deprecated.

The Problem: most requests are easy, a few are hard, and sending everything to the biggest model burns money on “hello”. Design the gate that sends each request down the cheapest road that still answers it well.
Answer
The design is a trained per-request router: a tiny scorer reads the request and predicts the probability that the strong model’s answer beats the weak one’s, then sends the request to the cheapest model expected to clear the quality bar. This is routing, not cascading, so each request normally pays for exactly one model. The two pivotal decisions are to train the router on human preference data (which model’s answer people actually preferred, as in Chatbot Arena battles) rather than hand-built rules, and to expose an explicit cost-quality operating threshold as a product knob, with an escalation fallback so a routing mistake costs extra compute, not a bad answer.
(1) Difficulty Signals: query embedding and length, task type, tool needs, and conversation state; the router is BERT-class or smaller, ~1/1000 the cost of a strong call.
(2) Preference-Trained Router: learn P(strong model wins | query) from human preference battles, augmented with LLM-judged comparisons; route strong when P passes the threshold.
(3) Explicit Operating Point: the threshold is chosen on the measured cost-quality curve and recalibrated as the traffic mix drifts.
(4) Guarded Fallback: if the small model or a verifier flags low confidence, escalate to the large model; routing errors become cost, not bad answers.
(5) Model Registry & Shadowing: candidates live behind a versioned registry; new models are shadow-scored on live traffic before the router is retrained or recalibrated.

Figure 1: The routing path: one cheap score per request, one paid model call, and an escalation hatch when the cheap path doubts itself.
Clarify Before Designing:
(1) Candidate Set: two models or many; closed APIs, self-hosted weights, or both; is the real split fast-versus-reasoning rather than small-versus-large?
(2) Decision Granularity: per request, per message in a conversation, or pinned per session for style consistency?
(3) Quality Bar: which evaluation defines “good enough” (human preference, task success), and how much loss versus the always-strong policy is tolerable?
(4) Router Budget: how much latency and cost may the router itself add per request?
(5) Traffic Shape: what fraction of traffic is trivial chat versus hard reasoning today, and how fast does that mix drift?
Login to view more content
August 9, 2026