Interview for Machine Learning

Category: CV

MSD0032 End-to-End Self-Driving
Design an end-to-end learning model for self-driving, where a single neural network (or a tightly coupled stack) maps raw sensor inputs directly to driving actions (steering, acceleration, braking), as pursued by Tesla’s FSD v12+, Wayve’s LINGO, and research systems like UniAD and VAD. Unlike the traditional modular pipeline of separate perception, prediction, planning, and control modules, an end-to-end model learns the full policy from data and can in principle optimize the driving objective globally.
The catch is that this policy is a black box. It is hard to interpret, hard to verify for safety, hungry for data and compute, and it only ever sees the long tail of rare driving situations a handful of times.
How would you design this model? Cover the candidate architecture approaches (pure end-to-end policy network vs loosely coupled perception-prediction-planning with learned interfaces vs hybrid with a differentiable cost head), the training data and objective (imitation learning vs reinforcement learning vs inverse RL), how you handle the long tail of rare safety-critical events when learning from data, and how you verify and debug a black-box policy before deployment.

The Problem: one learned function must turn photons into pedal and steering commands, and the events that decide whether it is safe appear once every hundred thousand miles.
Answer
I would build a query-based end-to-end network with a differentiable cost head, not a raw pixels-to-pedal regressor. Multi-camera features are lifted into a BEV representation, a set of scene queries carries agents, map, and occupancy, and an ego query cross-attends to those queries to produce a short-horizon trajectory that a learned cost head scores together with a few hand-authored terms. Everything is trained jointly with one driving loss, so gradients still reach the encoder, but every intermediate is supervised and therefore inspectable. The two pivotal decisions are to keep auxiliary supervision on the intermediate queries (this is what buys sample efficiency and debuggability without cutting the gradient path) and to keep hard safety constraints outside the learned cost, in an independent rule-based guardian that the network cannot argue with.
(1) Pure End-to-End Policy: one network maps sensor history directly to control, trained by behavior cloning on fleet data with no intermediate supervision.
(2) Loosely Coupled Modules with Learned Interfaces: perception → prediction → planning stays as stages, but the interfaces are differentiable query tokens rather than hand-defined object lists, as in UniAD and VAD.
(3) Hybrid with a Differentiable Cost Head: the network emits a learned cost over candidate ego trajectories, combined with explicit collision, comfort, and rule terms, and an optimizer or sampler picks the executed trajectory.
(4) Chosen Combination: approach 2 for the representation plus approach 3 for the output, imitation pretraining followed by closed-loop fine-tuning in simulation.

Figure 1: The chosen architecture. One driving loss backpropagates all the way to the encoder, while auxiliary heads keep the intermediate scene representation readable and the cost head keeps the final decision expressible in physical terms.
Clarify Before Designing:
(1) Sensor Suite And Output Interface: camera-only or camera plus lidar and radar, and does the model emit a 3 second trajectory for a downstream controller or raw actuator commands?
(2) Compute And Latency Budget: how many TOPS on-vehicle and what reaction budget, since that caps token count, temporal context, and model size?
(3) Data Access: fleet miles per day, whether human control signals are time-aligned with sensors, and whether a closed-loop simulator with reactive agents exists.
(4) Operational Design Domain: highway lane keeping, dense urban, or both, and in which geographies and weather, because the ODD size sets the length of the tail.
(5) Safety Architecture: is an independent rule-based guardian or a safety driver allowed downstream, or must the network’s output be the last word before the actuators?
Login to view more content
August 22, 2026
MSD0031 Autonomous Sensor Fusion
Design the sensor fusion system for an autonomous vehicle that combines camera, LiDAR, and radar into a unified perception of the world, as used by Waymo, Cruise, and Tesla’s Full Self-Driving stack. Each sensor has complementary strengths (camera: dense semantic and color detail; LiDAR: accurate depth and 3D structure; radar: velocity and all-weather range) and weaknesses (camera: depth ambiguity and weather sensitivity; LiDAR: sparse, expensive, poor in fog and rain; radar: low angular resolution and clutter). The fused perception must feed detection, tracking, and prediction at 10-20 Hz with hard latency bounds, and it must degrade gracefully when a sensor fails or is occluded rather than producing a confident wrong answer.
How would you design this system? Cover the per-sensor processing, the fusion architecture (early, late, or mid-level BEV fusion), the time synchronization and calibration, how you handle sensor dropout and degradation, and how you evaluate fusion quality against single-sensor baselines.

The Problem: three sensors look at the same road spray and return three different stories, and the vehicle still has to emit one confident answer every 50 to 100 ms.
Answer
Build a mid-level fusion stack over a shared bird’s-eye-view (BEV) grid. Each sensor keeps its own encoder, every feature is warped into one metric grid at one reference timestamp, and a lightweight transformer fuses them before the detection, occupancy, and tracking heads. Mid-level fusion is the pivotal choice: early raw fusion is brittle under calibration drift and blocks per-sensor pretraining, while late fusion discards exactly the cross-modal evidence that resolves camera depth ambiguity and rescues weak single-sensor detections. Two further decisions carry the safety argument. Train with modality dropout so the fused model remains usable when a sensor disappears, and keep a small non-learned geometric path (radar Doppler tracks plus LiDAR clustering) that can hold the vehicle to a safe stop if the learned stack degrades.
(1) Shared BEV Representation: project all modalities into a common ego-centric grid (roughly 100 m by 100 m at 0.4 m resolution) so fusion is a geometric alignment problem rather than an ad hoc feature concatenation.
(2) Per-Sensor Encoders: depth-distribution lift for images, sparse voxel or pillar encoder for LiDAR, and a point encoder that keeps radar range-rate; each branch trains and validates independently.
(3) Soft-Association Fusion: cross-attention between object queries and per-sensor features instead of hard pixel-to-point matching, which keeps accuracy when extrinsics drift by a fraction of a degree.
(4) Hardware Sync and Online Calibration: PTP-locked triggers, per-point timestamps, and ego-motion warping to one reference time, with reprojection residuals monitored continuously as a calibration health signal.
(5) Fallback Ladder: a health monitor scores each sensor and steps the system from full fusion, to degraded fusion with widened safety margins, to the geometric path plus an ODD exit or minimal risk maneuver.
(6) Ablation-Gated Evaluation: the release rule is that fused perception is never worse than the best single-sensor baseline on any evaluation slice, including night, rain, and forced-dropout slices.

Figure 1: One learned path through a shared BEV grid, plus a health monitor and a non-learned geometric path that stays alive when the learned path should not be trusted.
Clarify Before Designing:
(1) Sensor Suite and Cost Ceiling: how many cameras, what LiDAR (spinning 64-beam, solid state, or none), how many radars, and is 4D imaging radar available?
(2) Operational Design Domain: geofenced urban robotaxi with HD maps, or consumer highway assist without maps; does the ODD include heavy rain, snow, and night?
(3) Compute and Latency Budget: on-vehicle TOPS, power and thermal envelope, target rate (10 Hz or 20 Hz), and the p99 rather than mean latency bound.
(4) Failure Policy: on sensor loss, does the vehicle degrade speed, exit the ODD, hand back to a driver, or execute a minimal risk maneuver; who signs off on that decision?
(5) Label Supply: how much human-labeled 3D data exists, and can an offboard auto-labeling pipeline produce pseudo ground truth from full logs?
(6) Redundancy Requirement: must perception survive the loss of any single sensor, or any single compute unit, and is a second independent stack mandated?
Login to view more content
August 22, 2026
MSD0029 Food Calorie Estimation
Design a model that estimates the calorie and macronutrient content of a meal from a single photo, as used by apps like MyFitnessPal, Calorie Mama, or Lose It. The user snaps a photo of their plate. The model must identify the dishes, estimate portion sizes from a 2D image with no depth cue, look up or predict per-gram nutritional values, and return a calorie estimate within a few seconds.
The core technical challenge is portion estimation: a single 2D photo carries no metric depth, so converting pixel area into grams requires a mechanism assumption. The model must also handle a long tail of cuisines and homemade dishes that match no packaged-food database, varying plating and lighting, and the inherent ambiguity of judging a portion from one viewpoint.
How would you design this model? Cover the candidate portion estimation approaches (monocular depth estimation, reference-object scaling, semantic segmentation with food-class density priors, and direct end-to-end calorie regression), how you handle the long tail of homemade and mixed dishes, how you generate training labels at scale when ground-truth portion weights are scarce, and how you calibrate and communicate uncertainty when a single 2D photo is inherently ambiguous.

The Problem: the same mask area can be a thin layer or a tall mound, so one photo maps to a wide range of grams before any model has made a single assumption explicit.
Answer
The design is a segmentation-conditioned multi-task regressor. One shared backbone emits per-dish instance masks, a relative depth map, and a direct per-region regression of grams, calories, and macronutrients, with the physical volume-times-density estimate retained as a parallel sanity check rather than as the primary predictor. Google’s Nutrition5k study drives that ordering: an end-to-end regressor from a single RGB view reaches roughly 26% mean calorie error, and feeding depth — either as an extra channel or as a depth-derived volume scalar — cuts that error substantially, which is why depth is an auxiliary input to the regressor here rather than the front end of a separate segment-then-volume-then-density-lookup chain. Two decisions are pivotal. First, scale is treated as a learned prior anchored by plate and utensil geometry, not as metric information that a single RGB frame secretly contains. Second, the output is a calibrated interval and an editable suggestion, because one viewpoint leaves real ambiguity that no architecture removes.
(1) Monocular Depth Volume: predict per-pixel depth, fit the plate plane, and integrate height over each mask to get volume, then multiply by a food-class density. Physically interpretable, but scale-ambiguous.
(2) Reference-Object Scaling: use a known-size object in frame (bank card, coin, thumb, standard plate rim) to fix millimeters per pixel in the plate plane, which pins down area but not height.
(3) Segmentation With Density Priors: multiply mask area by a learned per-class height prior and a tabulated density in grams per cubic centimeter from food-composition data. Cheapest to build and to explain.
(4) Direct End-to-End Regression: regress grams, kilocalories, and macros straight from region features, letting the network learn scale from plate, cutlery, and hand context. Most accurate where labels exist, least interpretable.

Figure 1: The mechanism: photo → masks and depth → grams per dish, with the physical volume path acting as a guardrail on the learned regressor and a conformal step turning the point estimate into a range.
Clarify Before Designing:
(1) Label Ground Truth: do we have any scale-weighed dishes with per-ingredient masses, or only crowd labels and user-entered corrections?
(2) Capture Protocol: is it strictly one top-down snap, or may we request an oblique angle, a second frame, or a reference object without destroying adoption?
(3) Depth Sensors: what fraction of devices expose LiDAR or ARKit/ARCore depth, which turns scale from a guess into a measurement?
(4) Accuracy Bar: what calorie error is product-acceptable, and does the UX accept a range plus an edit control instead of one number?
(5) Coverage And Budget: which cuisines dominate traffic, what share of meals is packaged or restaurant food with known nutrition, and is inference on-device or server-side?
Login to view more content
August 22, 2026
MSD0028 Deepfake AIGC Detection
Design a deepfake and AI-generated content detection system for a video platform like YouTube or TikTok. Generative video models (Sora, Runway, Kling) and voice cloning tools have made convincing synthetic media cheap, and the platform must label or remove media that misleads viewers, impersonates real people, or spreads disinformation, while leaving legitimate creative and artistic uses of the same tools untouched.
The detector has to keep up with generators that improve every few months, survive uploads that were re-encoded and re-compressed until most forensic cues are gone, and avoid flagging real content that merely looks synthetic (high-quality CGI, heavily graded or filtered footage, screen recordings).
How would you design this system? Cover the detection signals (frequency artifacts, temporal inconsistencies, biometric cues, provenance metadata like C2PA), the model architecture for image, audio, and video, how you handle the adversarial arms race against improving generators, the human-review and appeals path, and how you measure detection rate and false-positive rate.

The Problem: three clips reach the same viewer, one filmed, one generated, one hand-made CGI. Only the misleading one should be acted on, and the pixels have already been re-encoded to 480p before you ever see them.
Answer
The design is a provenance-first, harm-tiered cascade, not a binary “fake or real” classifier. Every upload is first checked for signed C2PA Content Credentials and invisible watermarks, then a cheap multimodal triage model scores sampled frames plus the audio track, and only the riskiest few percent enter a heavy visual, audio, and temporal ensemble. The two pivotal decisions are that the system outputs a calibrated risk score plus a harm class (impersonation of a real person, medical or election claim, synthetic-but-harmless art) which drives graduated enforcement rather than blanket removal, and that the whole training and evaluation loop is built around the arms race: continuous harvesting of every new generator family, red teaming, and a release gate measured on unseen-generator holdouts and in-the-wild data instead of academic benchmarks.
(1) Provenance First, Detection Second: validate C2PA manifests and watermark decoders before any classifier runs; a valid signature is strong positive evidence, but a missing one proves nothing because metadata is stripped by every transcode.
(2) Cheap Triage, Expensive Ensemble: a small model scores 100% of uploads, the deep ensemble runs on roughly 2%, keeping compute inside budget while preserving recall on the tail that matters.
(3) Multimodal Signal Stack: spatial and frequency artifacts, temporal cues (landmark jitter, blink rate, rPPG pulse, lip-sync offset), and audio cues (vocoder phase artifacts) fused late, so no single fragile signal owns the decision.
(4) Compression-Robust Training: synthetic training data is pushed through the platform’s own transcode ladder, because a detector trained on pristine generator output collapses on real uploads.
(5) Harm-Tiered Enforcement: disclosure label, then demotion, then human review, then removal; full automation only for narrow high-confidence classes such as a cloned voice of a verified public figure.
(6) Arms-Race Loop: a red team plus a generation farm produce fresh fakes from every new model release, feeding a 2 to 4 week retrain cadence with an emergency path.

Figure 1: One cheap pass over everything, one expensive pass over the risky tail, and a review loop whose overturns become next week’s training data.
Clarify Before Designing:
(1) Policy Scope: are we labeling all synthetic media, or only synthetic media that misleads (impersonation, fabricated events, health and election claims)? The answer changes the target variable itself.
(2) Scale and Latency: uploads per day and peak factor, and must a verdict land before publish or within minutes after, before the video can go viral?
(3) Provenance Coverage: what fraction of uploads arrive with valid C2PA credentials or a decodable watermark today, and which camera and tool partners are signed up?
(4) Ground Truth: do we have licensed API access to the major generators for in-house fake production, and how many human-confirmed in-the-wild labels per week does review produce?
(5) Cost Asymmetry: what does a missed political deepfake cost versus a wrongly demoted VFX creator, and how large is the review headcount that sets our false-positive budget?
(6) Jurisdiction: which transparency rules apply (EU AI Act disclosure duties, election-period rules), and do they force labels regardless of harm?
Login to view more content
August 22, 2026
MSD0026 Multimodal Text-to-Image-Video Search
Design a multimodal search system where a user types a text query and the system returns relevant items from a catalog of billions of images and videos, like Pinterest Visual Search, Google Lens, or TikTok’s visual search. The query may describe objects, scenes, styles, or actions that never appear as literal text anywhere in the catalog (for example “golden retriever catching a frisbee at sunset”), so the system must bridge the modality gap between language and visual content.
Results must return in under 200 ms, the index must stay fresh as new content is uploaded continuously, and the system must handle queries that mix text with an uploaded reference image.
How would you design this system? Cover the shared embedding space and training strategy, the approximate nearest neighbor index at billion scale, the freshness and update strategy, the hybrid text-plus-image query path, and how you evaluate retrieval quality.

The Problem: the words the user types appear nowhere in the catalog, yet the right pixels have to surface out of billions of items inside 200 ms, while new uploads keep arriving.
Answer
The design is a two-tower (dual encoder) retrieval system over one shared embedding space, served as a two-stage retrieve-and-rerank path. A contrastive image-text model of the SigLIP/CLIP family, fine-tuned on in-domain query-engagement pairs, maps text, images, and video clips into the same 512-d space, so “golden retriever catching a frisbee at sunset” becomes a vector whose neighbors are the right pixels even though no matching caption exists. Two decisions carry most of the weight. First, the item side is multi-vector for video (a handful of clip embeddings rather than one averaged vector), because averaging a 60-second video destroys the action the query describes. Second, freshness is solved with a tiered index (hot in-memory buffer, hourly delta shards, nightly rebuilt base) instead of mutating one billion-scale graph in place. The hybrid text-plus-image query runs through a learned combiner that emits a single vector, so the retrieval and ranking path downstream never changes.
(1) Shared Embedding Space: one contrastively trained space for query text, images, and video clips, so retrieval is pure vector search rather than text matching against noisy alt-text.
(2) In-Domain Fine-Tuning: start from a public image-text checkpoint, then fine-tune on click and save pairs from real query logs, with hard negatives mined from the same logs.
(3) Multi-Vector Video Items: sample frames or short clips per video and index several vectors per item, collapsing to one item at merge time; single pooled vectors lose action and scene changes.
(4) Sharded IVF-PQ With Rerank: compress to 64-byte codes so 5 billion vectors fit in roughly 320 GB of RAM across shards, retrieve about 10k candidates, then rescore with full-precision vectors plus business features.
(5) Tiered Freshness: a hot buffer makes new uploads searchable in about a minute, hourly deltas absorb the day, and a nightly base rebuild keeps the big index compact; deletes ride a tombstone list.
(6) Learned Query Combiner: text-plus-image queries are fused into one vector by a small trained combiner, keeping the index path and latency budget identical to text-only search.

Figure 1: One embedding space, two write paths: queries are encoded online, catalog items are encoded on ingest, and the tiered index is the only place they meet.
Clarify Before Designing:
(1) Scale and traffic: how many items (billions of images, or billions of items with heavy video), what peak QPS, and how many uploads per hour?
(2) Latency budget split: is the 200 ms end-to-end including network and thumbnail fetch, or only the retrieval service?
(3) Freshness requirement: must a new upload be findable in seconds (live events, breaking trends) or is an hour acceptable for most content?
(4) Query mix: what fraction is head navigational text, tail descriptive text, action queries over video, and hybrid image-plus-text?
(5) Quality bar and cost asymmetry: is the product optimizing engagement, commercial conversion, or safety-first precision, and what is an acceptable cost per 1k queries?
(6) Available supervision: do we have query-click logs and human relevance ratings, or only alt-text and hashtags?
Login to view more content
August 22, 2026
MSD0023 YouTube Copyright Detection
Design the copyright detection system for a video platform like YouTube. Every uploaded video (hundreds of hours per minute) must be scanned against a reference corpus of tens of millions of copyrighted songs, movies, and clips, and matched even when the content is pitch-shifted, cropped, overlaid with voiceover, recorded off a screen, or remixed. The system must return matches within minutes so that rightsholders can block, monetize, or track the use, and it must keep the false-positive rate low enough that legitimate fair use and original content are not wrongly claimed.
How would you design this system? Cover the audio and video fingerprinting representations, the matching index at scale, the latency and throughput budget for the upload firehose, how you handle partial and transformed matches, and how disputes and appeals are routed.

The Problem: the copy is never bit-identical, the corpus is tens of millions of references, the answer is owed in minutes, and a wrong claim silences a legitimate creator.
Answer
The design is a two-stage fingerprint retrieval system: cheap, high-recall candidate generation from compact per-modality fingerprints, then strict temporal verification that only accepts a candidate if many hash hits agree on a single time offset. Audio and video are fingerprinted separately because their attack surfaces differ. Audio uses spectral peak landmark hashes plus a neural fallback for pitch and tempo edits, while video uses learned copy-detection embeddings that survive cropping, overlays, and screen recording. The two pivotal decisions are to make the verification stage, not the recall stage, own precision, and to convert model confidence into tiered actions (auto-claim, human review, track-only) so that an uncertain match costs a reviewer’s time rather than a creator’s channel.
(1) Dual-Modality Fingerprints: audio landmark hashes of the form $(f_1, f_2, \Delta t)$ at roughly 30 hashes per second, and video frame descriptors sampled at 2 to 4 fps after letterbox and border normalization.
(2) Two-Stage Retrieval: a sharded inverted index (plus ANN search over neural embeddings) proposes a few hundred candidate references per query chunk; verification rejects almost all of them.
(3) Offset Voting And Segment Stitching: a real match produces a spike in the histogram of $t_{ref} - t_{query}$ ; adjacent winning chunks are stitched into segments with a minimum matched duration.
(4) Chunked Parallel Ingest: uploads are split into 10-second chunks that fingerprint in parallel, with a priority queue so premieres and large channels finish first.
(5) Tiered Actions: high confidence auto-claims, medium confidence routes to human review or track-only, low confidence is logged and dropped.
(6) Dispute Loop As Training Data: every appeal outcome becomes a label, feeding whitelists, threshold retuning, and reference-corpus cleanup.

Figure 1: One path for uploads (fingerprint → lookup → verify → policy) and one offline path that fingerprints rightsholder references with the same extractor; disputes flow back from the policy engine into verification thresholds and whitelists.
Clarify Before Designing:
(1) Scope Of Rights: do we match sound recordings only, or also compositions and melodies, since a cover song copies the composition without copying any audio?
(2) Verdict SLA: must the scan block publishing, or may the video go live and be claimed minutes later, and does the SLA differ for monetized channels?
(3) Corpus Scale And Registration: how many references, how fast is the corpus growing, and who is allowed to register a reference?
(4) Cost Asymmetry: what false-claim rate is acceptable versus missed detections, and who pays for human review of the uncertain band?
(5) Transformation Catalog: which edits must be caught (crop, pitch shift, speed change, voiceover, screen capture, remix, AI covers), because that choice decides the fingerprint representation?
(6) Available Actions: block, monetize, or track, and are the policies territory-specific?
Login to view more content
August 22, 2026
MSD0021 Hashtag Video Relevance
On a short-video platform like TikTok or Instagram Reels, creators attach hashtags to reach an audience, and a large share of those tags are clickbait: #fyp or a trending sports tag stapled onto a video that has nothing to do with the topic behind it. The core problem is a multimodal relevance scoring mechanism. Given a video (sampled frames, audio, OCR’d on-screen text, creator caption) and a candidate hashtag, produce a relevance score that separates honest tagging from spam.
How would you design this model? Cover the candidate fusion architectures (late fusion of per-modality encoders, early token-level fusion, cross-attention, contrastive hashtag-video embedding), how you represent the hashtag itself (its text, its trending-topic cluster, its co-occurrence graph), how you generate training labels at scale when human-labeled relevance is scarce, and how you handle cold start for a brand-new hashtag with no co-occurrence history.

The Problem: the same video carries an honest tag and a hijacked trending tag, and nothing in the upload tells you which is which. You need a score for every (video, tag) pair, over a tag vocabulary of tens of millions and with almost no human relevance labels.
Answer
The design is a two-stage multimodal relevance model. A contrastive dual tower is pretrained on billions of weakly labeled creator (video, tag) pairs, which gives every tag in the vocabulary a cheap vector and initializes both encoders. A cross-attention scorer then rescores the handful of tags actually attached to a video, letting the tag’s query tokens read the video’s frame, audio, OCR, and caption tokens so a localized mismatch (a football tag on a pasta video, or a single on-screen word borrowed from a trend) becomes visible. Two decisions are pivotal. First, the hashtag is a sum of three slots (subword text, trending-topic cluster centroid, co-occurrence graph embedding) trained with slot dropout, which is what makes cold start work rather than a special case bolted on later. Second, labels come from creator behavior plus frequency-matched hard negatives drawn from the same topic cluster, anchored by a small human gold set used only for calibration and as the release gate.
(1) Late Fusion: one encoder per modality, each pooled to a vector, concatenated with the tag vector into a shallow MLP; everything cacheable, no token-level interaction.
(2) Early Fusion: all modality tokens plus the tag tokens in one transformer from layer 1; maximally expressive and completely uncacheable across candidate tags.
(3) Cross-Attention Scorer: video tokens encoded once as keys and values, hashtag slots as a small query set, a few cross-attention layers plus a sigmoid head.
(4) Contrastive Embedding: dual-tower video and tag encoders trained with InfoNCE or a sigmoid pairwise loss so relevance is a dot product in a shared space.

Figure 1: The scoring mechanism: the video is encoded once into a token set, the hashtag arrives as three composable slots, and cross-attention decides whether the video contains evidence for the tag.
Clarify Before Designing:
(1) Input budget: how many frames per video, is audio and reliable OCR available, and is the caption trustworthy or itself keyword-stuffed?
(2) Compute budget: how many candidate tags are scored per video, and may the video encoder run once at upload so per-tag cost is the only thing multiplied?
(3) Pretrained assets: is there an in-house video-language encoder to freeze, or must the towers be trained from scratch on platform data?
(4) Vocabulary shape: how large is the tag vocabulary, what fraction of traffic is meta-tags such as #fyp, and how fast do new tags appear?
(5) Label reality: how many human-labeled pairs exist today, and can we buy a gold set for calibration even if we cannot buy a training set?
(6) Output contract: is the consumer a hard threshold, a ranking feature, or a reviewer queue, since only the first two need a calibrated probability?
Login to view more content
August 22, 2026
MSD0018 Ads Policy Violation Detection
Design an end-to-end ML system to detect ads selling weapons and other policy-violating goods on a large ads platform like Meta or Google Ads. Millions of new ad creatives arrive daily as text, images, and video, and advertisers constantly adapt their creative to evade detection. Google’s Ads Safety Report describes blocking or removing billions of ads per year, so this is an industrial-scale enforcement problem, not a one-off classifier.
The system must block prohibited ads before they serve, route borderline cases to human reviewers, and keep the false-positive rate low enough that legitimate advertisers are not churned.
How would you design this system? Cover the multimodal detection model, the human-review queue prioritization, the adversarial adaptation problem, and how you measure recall and precision at platform scale.

The Problem: millions of new creatives arrive every day, roughly one in ten thousand is a prohibited-goods listing, and the decision must be made before the ad is shown even once.
Answer
The design is a pre-serve multimodal gate inside the ad-ingestion path. A cheap first stage kills known-bad creatives by hash and embedding near-duplicate lookup, then a shared vision-language encoder with per-policy heads scores every creative (ad copy, OCR text, landing-page text, image, video keyframes) before the first impression. Each policy class gets three calibrated bands: auto-block, human review, allow. Two decisions carry the design. First, thresholds are set from the asymmetric cost of a served weapon ad versus a wrongly blocked advertiser, per policy and per locale, rather than from a single global score cut. Second, the whole system is treated as an adversarial loop where the time from a new evasion pattern to a deployed block (hours, not quarters) matters more than static offline AUC.
(1) Staged Funnel: creative-hash cache and embedding kNN against known violations resolve most repeat traffic in milliseconds; only novel creatives pay for the full multimodal pass.
(2) Multimodal Model: a frozen vision tower plus a multilingual text tower over copy, OCR, ASR, and landing-page text, fused into multi-label per-policy heads instead of one binary “violating” output.
(3) Three-Band Decision Policy: per-policy calibrated thresholds split traffic into auto-block, review, and allow, so precision on the auto-block band is protected by construction.
(4) Expected-Harm Review Ranking: the reviewer queue is sorted by violation probability times forecast impressions times policy severity, not by score alone, and clusters of near-duplicate creatives collapse into one decision.
(5) Adversarial Response Loop: evasion signals (re-upload similarity, account bursts, appeal patterns) feed hard-negative mining, an embedding blocklist hotfix within hours, and account-level enforcement that raises the attacker’s cost.
(6) Measurement by Sampling: recall is estimated from stratified random audits of allowed traffic, precision from reviewer audits of blocks and from appeal-overturn rates.

Figure 1: Ingest → asset extraction → encoders → per-policy scores → three-band decision, with reviewer labels and the near-duplicate index closing the loop back into the model.
Clarify Before Designing:
(1) Scale and Asset Mix: how many new creatives per day, how many assets each, and what share is video (video dominates compute cost)?
(2) Enforcement SLA: must an ad go live within minutes of upload, or is holding a creative for a deep asynchronous pass acceptable?
(3) Policy Taxonomy: how many prohibited-goods classes, which are legally mandated, and how much do they vary by jurisdiction?
(4) Cost Asymmetry: what is the tolerated false-block rate before advertiser churn becomes the bigger loss, and what is the appeal SLA?
(5) Review Capacity: how many reviewer hours per day, in which languages, and with what accuracy on gold sets?
(6) Label Availability: do we have historical enforcement decisions, appeal outcomes, and account-level history, or are we starting from policy text only?
Login to view more content
August 21, 2026
MSD0016 Scanned Document OCR
Design an OCR system that converts millions of scanned documents (contracts, invoices, old books, fax-quality pages) into clean, structured text for a document platform like AWS Textract, Google Document AI, or Mistral’s OCR API. The output must preserve reading order, tables, and equations, not just raw characters, because the text feeds search indexes and RAG pipelines downstream.
How would you design this system? Cover the recognition approach, layout and structure handling, accuracy measurement, and the cost of processing at scale.

The Problem: a skewed, coffee-stained scan of a two-column contract must come out as correctly ordered, structured text, at millions of pages, at a cost a business can pay. Design the reader that never silently scrambles a page.
Answer
The design is a single-pass document VLM pipeline in the style of AI2’s olmOCR: lightweight preprocessing normalizes each page (orientation, deskew, denoise, blank rejection), then a fine-tuned 7B-class vision-language model reads the whole page in one pass and emits structured output (Markdown for headings, HTML for tables, LaTeX for math), and programmatic unit tests verify each page (does the table parse? does the equation recompile? is reading order sane?) before failures are retried or routed to a stronger model or a human queue. The two pivotal decisions are reading whole pages in one pass instead of a detect-then-crop-then-recognize cascade (which accumulates errors and loses reading order), and defining correctness as machine-checkable tests so both evaluation and training optimize what downstream consumers actually need.
(1) Single-Pass VLM Reader: a fine-tuned Qwen2.5-VL-class model turns a page image directly into structured Markdown/HTML/LaTeX, skipping the multi-stage cascade.
(2) Document Anchoring: born-digital PDFs contribute their embedded text and metadata into the prompt alongside the page image, grounding the model and cutting hallucination.
(3) Preprocessing Gate: orientation detection, deskew, denoise, and blank-page rejection run before the expensive model call.
(4) Verifiable QA: per-page unit tests (table parses, math recompiles, reading order monotonic); failures retry with different sampling or escalate to a stronger model or human review.
(5) Batch-First Serving: offline GPU batches with vLLM/SGLang; a tuned open pipeline processes a million pages for roughly $190, about 1/32 of frontier-API pricing.

Figure 1: The pipeline: cheap normalization first, one VLM read per page, then machine-checkable tests decide whether the page ships, retries, or goes to a human.
Clarify Before Designing:
(1) Document Mix: born-digital PDFs versus degraded scans; which languages; is handwriting in scope?
(2) Structure Fidelity: plain text, or RAG-grade structure with tables, equations, and reading order preserved?
(3) Volume and Latency: nightly millions in batch, or interactive upload-and-wait in seconds?
(4) Deployment Constraints: customer data residency or full self-hosting requirements (regulated archives)?
(5) Downstream Contract: does the consumer want Markdown, or structured JSON field extraction (invoice totals, dates) on top?
Login to view more content
August 9, 2026
MSD0008 Cancer Cell Segmentation for Digital Pathology

A digital pathology lab scans tissue biopsies into gigapixel whole-slide images (WSI). Pathologists currently hunt for tumor regions by panning and zooming across each slide by eye, which is slow and error-prone, and the lab wants software that highlights regions containing cancer cells for priority review.
How would you design an image segmentation model that detects cancer cells in these slides? Outline the candidate approaches, and explain how you would choose between them.

The Problem: a 100,000 × 100,000-pixel slide, and the cancer signal hides in a tiny fraction of its pixels. Design the segmentation model that fills the question mark.
Answer
The task is pixel-level binary segmentation on gigapixel slides: the model labels every pixel as cancer or non-cancer, and the slide is processed as a grid of patches because no GPU holds 10¹⁰ pixels at once. Three architecture families cover the realistic design space: the U-Net family (encoder-decoder with skip connections), the DeepLab family (atrous convolutions for wide context at full resolution), and Transformer segmenters (SegFormer-style global context). The choice turns on annotation budget, the spatial context the diagnosis needs, and per-slide compute; a strong default is a U-Net variant trained with an overlap-aware loss under heavy stain augmentation.
(1) U-Net Family: an encoder compresses each patch, a decoder upsamples back to full resolution, and skip connections re-inject fine detail so cell boundaries stay sharp; the data-efficient default.
(2) DeepLab / Atrous Family: atrous (dilated) convolutions and ASPP widen the receptive field without shrinking resolution, trading compute for gland-level context.
(3) Transformer Segmenters: self-attention (SegFormer, TransUNet) models long-range tissue architecture, at the price of a much larger appetite for labeled data or pretraining.

Figure 1: The U-Net mechanism on one patch: the encoder gathers context as it downsamples, the decoder rebuilds the full-resolution mask, and skip connections carry boundary detail straight across. Every 256 × 256 patch yields a same-size cancer mask.
Clarify Before Designing:
(1) Imaging Spec: magnification (20× or 40×), typical slide dimensions, and which scanner models feed the pipeline?
(2) Labels: how many slides carry pixel-level pathologist annotations, and are they full masks, outlines, or only slide-level diagnoses?
(3) Domain Spread: how many labs, staining protocols, and organ types must one model cover?
(4) Turnaround Budget: how many minutes per slide may inference take, and on what hardware?
(5) Cost Asymmetry: a missed tumor region versus a false highlight: which error angers pathologists more?
Login to view more content
August 8, 2026