Interview for Machine Learning

Tag: Model

MSD0029 Food Calorie Estimation
Design a model that estimates the calorie and macronutrient content of a meal from a single photo, as used by apps like MyFitnessPal, Calorie Mama, or Lose It. The user snaps a photo of their plate. The model must identify the dishes, estimate portion sizes from a 2D image with no depth cue, look up or predict per-gram nutritional values, and return a calorie estimate within a few seconds.
The core technical challenge is portion estimation: a single 2D photo carries no metric depth, so converting pixel area into grams requires a mechanism assumption. The model must also handle a long tail of cuisines and homemade dishes that match no packaged-food database, varying plating and lighting, and the inherent ambiguity of judging a portion from one viewpoint.
How would you design this model? Cover the candidate portion estimation approaches (monocular depth estimation, reference-object scaling, semantic segmentation with food-class density priors, and direct end-to-end calorie regression), how you handle the long tail of homemade and mixed dishes, how you generate training labels at scale when ground-truth portion weights are scarce, and how you calibrate and communicate uncertainty when a single 2D photo is inherently ambiguous.

The Problem: the same mask area can be a thin layer or a tall mound, so one photo maps to a wide range of grams before any model has made a single assumption explicit.
Answer
The design is a segmentation-conditioned multi-task regressor. One shared backbone emits per-dish instance masks, a relative depth map, and a direct per-region regression of grams, calories, and macronutrients, with the physical volume-times-density estimate retained as a parallel sanity check rather than as the primary predictor. Google’s Nutrition5k study drives that ordering: an end-to-end regressor from a single RGB view reaches roughly 26% mean calorie error, and feeding depth — either as an extra channel or as a depth-derived volume scalar — cuts that error substantially, which is why depth is an auxiliary input to the regressor here rather than the front end of a separate segment-then-volume-then-density-lookup chain. Two decisions are pivotal. First, scale is treated as a learned prior anchored by plate and utensil geometry, not as metric information that a single RGB frame secretly contains. Second, the output is a calibrated interval and an editable suggestion, because one viewpoint leaves real ambiguity that no architecture removes.
(1) Monocular Depth Volume: predict per-pixel depth, fit the plate plane, and integrate height over each mask to get volume, then multiply by a food-class density. Physically interpretable, but scale-ambiguous.
(2) Reference-Object Scaling: use a known-size object in frame (bank card, coin, thumb, standard plate rim) to fix millimeters per pixel in the plate plane, which pins down area but not height.
(3) Segmentation With Density Priors: multiply mask area by a learned per-class height prior and a tabulated density in grams per cubic centimeter from food-composition data. Cheapest to build and to explain.
(4) Direct End-to-End Regression: regress grams, kilocalories, and macros straight from region features, letting the network learn scale from plate, cutlery, and hand context. Most accurate where labels exist, least interpretable.

Figure 1: The mechanism: photo → masks and depth → grams per dish, with the physical volume path acting as a guardrail on the learned regressor and a conformal step turning the point estimate into a range.
Clarify Before Designing:
(1) Label Ground Truth: do we have any scale-weighed dishes with per-ingredient masses, or only crowd labels and user-entered corrections?
(2) Capture Protocol: is it strictly one top-down snap, or may we request an oblique angle, a second frame, or a reference object without destroying adoption?
(3) Depth Sensors: what fraction of devices expose LiDAR or ARKit/ARCore depth, which turns scale from a guess into a measurement?
(4) Accuracy Bar: what calorie error is product-acceptable, and does the UX accept a range plus an edit control instead of one number?
(5) Coverage And Budget: which cuisines dominate traffic, what share of meals is packaged or restaurant food with known nutrition, and is inference on-device or server-side?
Login to view more content
August 22, 2026
MSD0021 Hashtag Video Relevance
On a short-video platform like TikTok or Instagram Reels, creators attach hashtags to reach an audience, and a large share of those tags are clickbait: #fyp or a trending sports tag stapled onto a video that has nothing to do with the topic behind it. The core problem is a multimodal relevance scoring mechanism. Given a video (sampled frames, audio, OCR’d on-screen text, creator caption) and a candidate hashtag, produce a relevance score that separates honest tagging from spam.
How would you design this model? Cover the candidate fusion architectures (late fusion of per-modality encoders, early token-level fusion, cross-attention, contrastive hashtag-video embedding), how you represent the hashtag itself (its text, its trending-topic cluster, its co-occurrence graph), how you generate training labels at scale when human-labeled relevance is scarce, and how you handle cold start for a brand-new hashtag with no co-occurrence history.

The Problem: the same video carries an honest tag and a hijacked trending tag, and nothing in the upload tells you which is which. You need a score for every (video, tag) pair, over a tag vocabulary of tens of millions and with almost no human relevance labels.
Answer
The design is a two-stage multimodal relevance model. A contrastive dual tower is pretrained on billions of weakly labeled creator (video, tag) pairs, which gives every tag in the vocabulary a cheap vector and initializes both encoders. A cross-attention scorer then rescores the handful of tags actually attached to a video, letting the tag’s query tokens read the video’s frame, audio, OCR, and caption tokens so a localized mismatch (a football tag on a pasta video, or a single on-screen word borrowed from a trend) becomes visible. Two decisions are pivotal. First, the hashtag is a sum of three slots (subword text, trending-topic cluster centroid, co-occurrence graph embedding) trained with slot dropout, which is what makes cold start work rather than a special case bolted on later. Second, labels come from creator behavior plus frequency-matched hard negatives drawn from the same topic cluster, anchored by a small human gold set used only for calibration and as the release gate.
(1) Late Fusion: one encoder per modality, each pooled to a vector, concatenated with the tag vector into a shallow MLP; everything cacheable, no token-level interaction.
(2) Early Fusion: all modality tokens plus the tag tokens in one transformer from layer 1; maximally expressive and completely uncacheable across candidate tags.
(3) Cross-Attention Scorer: video tokens encoded once as keys and values, hashtag slots as a small query set, a few cross-attention layers plus a sigmoid head.
(4) Contrastive Embedding: dual-tower video and tag encoders trained with InfoNCE or a sigmoid pairwise loss so relevance is a dot product in a shared space.

Figure 1: The scoring mechanism: the video is encoded once into a token set, the hashtag arrives as three composable slots, and cross-attention decides whether the video contains evidence for the tag.
Clarify Before Designing:
(1) Input budget: how many frames per video, is audio and reliable OCR available, and is the caption trustworthy or itself keyword-stuffed?
(2) Compute budget: how many candidate tags are scored per video, and may the video encoder run once at upload so per-tag cost is the only thing multiplied?
(3) Pretrained assets: is there an in-house video-language encoder to freeze, or must the towers be trained from scratch on platform data?
(4) Vocabulary shape: how large is the tag vocabulary, what fraction of traffic is meta-tags such as #fyp, and how fast do new tags appear?
(5) Label reality: how many human-labeled pairs exist today, and can we buy a gold set for calibration even if we cannot buy a training set?
(6) Output contract: is the consumer a hard threshold, a ranking feature, or a reviewer queue, since only the first two need a calibrated probability?
Login to view more content
August 22, 2026
MSD0019 On-Device Small LLM Polite
Design a small LLM that runs entirely on a phone for a privacy-preserving assistant, in the style of Apple Intelligence’s on-device foundation model or Gemini Nano on Pixel. The model must fit in roughly 2-4 GB of RAM, respond in under 300 ms on a mobile NPU, and work with the radio off.
It also has to stay polite. It should refuse harmful requests, avoid toxic or biased outputs, and hold a respectful tone even when the user is deliberately provoking it. Battery life and offline capability matter as much as raw benchmark quality, and no user text may leave the handset.
How would you design this system? Cover model compression and quantization for the NPU, the alignment strategy that keeps the model polite at a small scale, the on-device serving path, and how you evaluate safety without sending user data to the cloud.

The Problem: a few gigabytes of RAM, a fraction of a second, no network, and a user who is testing how rude the assistant will get.
Answer
The design is a distilled 3B-parameter model compressed with quantization-aware training to about 2 bits per weight, served on the NPU with per-feature LoRA adapters and a tiny always-resident guard model. Politeness is trained in rather than prompted in: a large aligned teacher supplies refusal and tone supervision, a preference pass sharpens “polite but not preachy”, and safety data stays inside the quantization loss so alignment survives compression. Two decisions are pivotal. First, safety is re-verified after quantization, because low-bit weights erode refusal behavior faster than they erode average benchmark quality. Second, a separate 100M-parameter guard screens the prompt and the streamed output, so a single jailbroken generation is caught by a component that can be patched over the air without retraining the base model.
(1) Distill + QAT + Adapters: prune and distill a large teacher into ~3B parameters, then quantization-aware train to 2-4 bits and ship 16-bit LoRA adapters per feature (summarize, reply, rewrite).
(2) Scratch-Trained Small Model + PTQ: train a 1-2B model on curated data and apply post-training 4-bit quantization; simplest pipeline, weakest quality per byte.
(3) Elastic Nested Model: a MatFormer-style checkpoint whose submodels can be extracted at different sizes, letting the runtime trade quality for RAM under memory pressure.
(4) On-Device Draft + Private Cloud Escalation: answer locally by default and escalate hard prompts to a privacy-hardened server tier with explicit consent.

Figure 1: Everything inside the dashed boundary runs on the handset: guard, decode loop, adapters, and cache. Nothing crosses the boundary, so every check has to happen before pixels appear.
Clarify Before Designing:
(1) Latency Definition: is 300 ms the time to first token or the full reply? Streaming a 60-token answer at 30 tokens per second takes two seconds no matter how good the prefill is.
(2) Task Set and Context: which features ship (reply suggestions, summarization, rewriting), and what is the maximum prompt length? Prefill cost and KV cache size follow directly from that number.
(3) Device Floor: which chip generation is the minimum, and what does its NPU actually execute natively (INT8, INT4, palettized weights, per-channel scales)?
(4) Escalation Policy: is any server path permitted with consent, or is strict offline operation a product promise we cannot break?
(5) Safety Bar: which refusal taxonomy applies, and what false-refusal rate on benign-but-edgy prompts is acceptable? Over-refusal is the failure users complain about.
(6) Telemetry Rules: may we collect differentially private aggregate counters and opt-in donated conversations, or literally nothing?
Login to view more content
August 21, 2026
MSD0008 Cancer Cell Segmentation for Digital Pathology

A digital pathology lab scans tissue biopsies into gigapixel whole-slide images (WSI). Pathologists currently hunt for tumor regions by panning and zooming across each slide by eye, which is slow and error-prone, and the lab wants software that highlights regions containing cancer cells for priority review.
How would you design an image segmentation model that detects cancer cells in these slides? Outline the candidate approaches, and explain how you would choose between them.

The Problem: a 100,000 × 100,000-pixel slide, and the cancer signal hides in a tiny fraction of its pixels. Design the segmentation model that fills the question mark.
Answer
The task is pixel-level binary segmentation on gigapixel slides: the model labels every pixel as cancer or non-cancer, and the slide is processed as a grid of patches because no GPU holds 10¹⁰ pixels at once. Three architecture families cover the realistic design space: the U-Net family (encoder-decoder with skip connections), the DeepLab family (atrous convolutions for wide context at full resolution), and Transformer segmenters (SegFormer-style global context). The choice turns on annotation budget, the spatial context the diagnosis needs, and per-slide compute; a strong default is a U-Net variant trained with an overlap-aware loss under heavy stain augmentation.
(1) U-Net Family: an encoder compresses each patch, a decoder upsamples back to full resolution, and skip connections re-inject fine detail so cell boundaries stay sharp; the data-efficient default.
(2) DeepLab / Atrous Family: atrous (dilated) convolutions and ASPP widen the receptive field without shrinking resolution, trading compute for gland-level context.
(3) Transformer Segmenters: self-attention (SegFormer, TransUNet) models long-range tissue architecture, at the price of a much larger appetite for labeled data or pretraining.

Figure 1: The U-Net mechanism on one patch: the encoder gathers context as it downsamples, the decoder rebuilds the full-resolution mask, and skip connections carry boundary detail straight across. Every 256 × 256 patch yields a same-size cancer mask.
Clarify Before Designing:
(1) Imaging Spec: magnification (20× or 40×), typical slide dimensions, and which scanner models feed the pipeline?
(2) Labels: how many slides carry pixel-level pathologist annotations, and are they full masks, outlines, or only slide-level diagnoses?
(3) Domain Spread: how many labs, staining protocols, and organ types must one model cover?
(4) Turnaround Budget: how many minutes per slide may inference take, and on what hardware?
(5) Cost Asymmetry: a missed tumor region versus a false highlight: which error angers pathologists more?
Login to view more content
August 8, 2026
MSD0004 Long Document Attention Scalability

The standard Transformer’s self-attention mechanism has a computational and memory complexity of $O(N^2)$ , where $N$ is the sequence length. For long document classification (e.g., thousands of tokens), this quadratic scaling becomes prohibitive.
Describe one or more attention modifications you would design or choose to enable efficient and effective long document classification.

The Problem: full self-attention costs $O(N^2)$ in compute and memory. Redesign attention so a several-thousand-token document stays affordable without giving up classification accuracy.
Answer
To handle long documents, the quadratic complexity of full self-attention ( $O(N^2)$ ) must be reduced. The two primary directions are Sparse Attention (Longformer, BigBird), which constrains each token to a relevant subset (a local window plus a few global tokens), and Hierarchical Attention (HATN), which applies attention within segments first and then across segment representations. Both preserve classification quality while cutting cost from $O(N^2)$ toward linear; linearized and recurrent variants (Performer, Linformer, Transformer-XL) attack the same wall from other angles.
(1) Sparse Attention (Mechanism): Replace the full attention matrix with a sparse design: local window attention for nearby context plus global tokens (e.g., [CLS]) that exchange information with all tokens (Longformer, BigBird).
(2) Hierarchical Attention (Structure): Split the document into segments; apply token-level attention within each segment, then a document-level attention across segment representations (HATN, LNLF-BERT).
(3) Linearized / Recurrent Variants: Kernel approximations (Performer, Linformer) reach true linear complexity; Transformer-XL / Compressive Transformer carry a recurrent memory across segments.

Figure 1: Full attention (left) fills the whole $N \times N$ matrix; sliding-window attention (right) keeps only a diagonal band: each token attends to its neighbors, and cost drops from $O(N^2)$ to $O(N \cdot w)$ .
Clarify Before Designing:
(1) Document Length: typical and maximum token count: 2k, 8k, 32k?
(2) Starting Point: adapt a pretrained BERT-style encoder, or train from scratch?
(3) Binding Constraint: memory at training, or latency at inference?
(4) Interaction Pattern: does classification need global token interactions (a [CLS] readout), and how long-range are the dependencies?
(5) Budget For Change: may we modify the architecture, or only fine-tune an off-the-shelf long model?
Login to view more content
November 4, 2025
MSD0002 Image to Video Classification

You are given a pretrained image classification network, such as ResNet. How would you adapt it to perform video classification, ensuring that both spatial and temporal information are captured?
Please discuss possible architectural modifications and trade-offs between different approaches.

The Problem: a pretrained 2D CNN understands each frame in isolation. Fill the question mark with an adaptation that adds temporal modeling without throwing away what the image network already knows.
Answer
The missing ingredient is temporal modeling: a 2D CNN sees each frame independently, so any adaptation must add a mechanism that aggregates information across time. Three families cover the design space: inflate the network into 3D convolutions, keep the 2D CNN as a per-frame encoder and add a sequence model on top, or simply pool or attend over per-frame features. The right choice trades temporal modeling power against compute cost and data hunger.
(1) 3D CNNs (C3D / I3D): Extend 2D convolutions into 3D to learn motion directly; I3D inflates pretrained 2D filters into 3D, keeping ImageNet weights useful.
(2) CNN + Sequence Model: Use the 2D CNN as a per-frame feature extractor, then run an LSTM / TCN / Transformer over the feature sequence for temporal modeling.
(3) Temporal Pooling / Attention: Aggregate per-frame features with average/max pooling or attention: cheapest, but frame order and fine motion cues fade.

Figure 1: The CNN + sequence model option: the pretrained 2D CNN is reused per frame, and a temporal model (LSTM/TCN/Transformer) learns how features evolve. This is the balanced default: full ImageNet leverage, moderate cost, natural variable-length handling.
Clarify Before Designing:
(1) Latency: real-time or offline? What is the per-clip inference budget?
(2) Input Statistics: fps, clip duration, resolution: trimmed clips or untrimmed streams?
(3) Data Scale: how much labeled video is available: hundreds or hundreds of thousands of clips?
(4) Compute: can training and inference afford 3D-convolution cost?
(5) Motion Sensitivity: do classes depend on subtle motion (opening vs closing) or mostly on scenes and objects?
Login to view more content
November 2, 2025