Interview for Machine Learning

Category: CV

  • MSD0006 Video Recommendation System

    

    How would you design a scalable and personalized video recommendation system for a platform like YouTube, Netflix, or TikTok that can recommend relevant videos in real time to billions of users?

    Line-art scene: a user with a phone facing a huge wall of video thumbnails labeled billions of videos, with a question mark leading to a clean top twenty feed on the phone

    The Problem: billions of videos, billions of users, one feed request at a time. Pick the top-20 most relevant videos for this user in milliseconds.

    Answer

    A modern recommendation system uses a multi-stage pipeline to narrow billions of videos down to a top-20 list for a user in milliseconds. It typically consists of Candidate Generation (filtering down to hundreds), Ranking (scoring those hundreds with deep learning), and Re-ranking (applying business logic for diversity, freshness, and safety). The full path is: Data Logging → Candidate Generation → Ranking → Re-ranking → Serving.

    (1) Multi-Stage Pipeline: No single model can score a billion items per request. The funnel makes it affordable: retrieve ~500 candidates cheaply, then rank precisely, then polish.
    (2) Data & Features: User signals (watch history, watch time, likes, skips, follows), video features (visual/audio/text embeddings, popularity, freshness), and context (time of day, device, network).
    (3) Candidate Generation: Multi-source retrieval (collaborative filtering / two-tower, content-based, seed-based co-watch, trending), merged and deduplicated to ~100-500.
    (4) Ranking: A deep model scores the candidates: DNNs on hundreds of features, Multi-Task Learning heads (click, watch time, like, share), and sequence models for session context.
    (5) Re-Ranking: Business and quality constraints: remove seen/unsafe items, inject diversity and freshness, fight filter bubbles.

    Funnel diagram: billions of videos narrow through candidate generation to about five hundred, through ranking to a top twenty list, then re-ranking and serving to the user feed

    Figure 1: The recommendation funnel: each stage trades breadth for precision: cheap retrieval cuts billions to hundreds, the ranking model orders them precisely, and re-ranking applies the final business logic before serving.

    Clarify Before Designing:
    (1) Scale: how many videos, daily active users, and feed requests per second?
    (2) Latency Budget: how many milliseconds for a feed request, end to end?
    (3) Objectives: what are we optimizing (clicks, watch time, long-term satisfaction), and are there multiple conflicting goals?
    (4) Cold Start Volume: how many new users and new videos arrive daily?
    (5) Constraints: ads insertion, content safety, regional/legal rules that override relevance?

    Login to view more content

  • MSD0005 Surveillance Video Anomaly Detection

    

    How would you design an end-to-end surveillance system that automatically detects and alerts security personnel to ‘anomalous events’ (e.g., break-ins, fainting, or prohibited movements) in a large shopping mall?

    Line-art scene: a shopping mall corridor watched by a wall-mounted camera, with a walking person and a question mark callout asking normal or anomaly

    The Problem: dozens of cameras stream continuously. Spot the rare anomalous event (break-in, fainting, prohibited movement) in real time, knowing you have no labeled anomalies to train on.

    Answer

    A surveillance anomaly detection system captures video streams, preprocesses them into clips, and uses a deep learning model (typically a pretrained video backbone plus a lightweight anomaly scoring head) to identify unusual behavior. It operates in a semi-supervised setup trained on normal data, runs in real time with sliding windows and temporal smoothing, and wraps the model in alerting, monitoring, and a human-in-the-loop feedback loop for calibration and retraining.

    (1) Ingestion & Preprocessing: Capture real-time streams from many cameras; resize frames and normalize pixels before clipping them into short windows.
    (2) Feature Extraction: A pretrained backbone (2D CNN (EfficientNet), optical flow, 3D CNN (I3D), or a Video Transformer (Swin, TimeSformer)) turns each clip into features.
    (3) Semi-Supervised Normal Model: An autoencoder or GAN trained on months of “normal” mall activity; high reconstruction error on new footage flags an anomaly against a calibrated threshold.
    (4) Alerting & Human-In-The-Loop: Real-time alerts send anomalous frames to operators, whose “Not an Anomaly” clicks feed recalibration and retraining.
    (5) System Design: Edge devices do preliminary processing to save bandwidth, the cloud does heavy computation; evaluation tracks precision, recall, F1 and false-alarm rates.

    Three-panel mechanism diagram: clips flow into a pretrained video backbone for feature extraction, an autoencoder trained on normal activity reconstructs them, and a reconstruction error curve crossing a threshold line flags anomalies

    Figure 1: The detection mechanism: backbone features feed a model trained to reconstruct normal activity; when reconstruction error crosses the calibrated threshold, the clip is flagged anomalous.

    Clarify Before Designing:
    (1) Scale: how many cameras, what fps and resolution, and how many concurrent clips must be scored?
    (2) Alert Latency: how many seconds from event to operator notification?
    (3) Anomaly Definition: a closed list of event types, or anything unusual; and are any labels available (frame-level, video-level, none)?
    (4) False-Alarm Budget: how many false alerts per camera per day can guards tolerate before they start ignoring the system?
    (5) Deployment: on-prem edge hardware available, or cloud-only?

    Login to view more content

  • MSD0002 Image to Video Classification

    

    You are given a pretrained image classification network, such as ResNet. How would you adapt it to perform video classification, ensuring that both spatial and temporal information are captured?

    Please discuss possible architectural modifications and trade-offs between different approaches.

    Problem framing diagram: a pretrained 2D ResNet that sees frames independently flows into a question mark, with the target video classification requiring both spatial and temporal cues

    The Problem: a pretrained 2D CNN understands each frame in isolation. Fill the question mark with an adaptation that adds temporal modeling without throwing away what the image network already knows.

    Answer

    The missing ingredient is temporal modeling: a 2D CNN sees each frame independently, so any adaptation must add a mechanism that aggregates information across time. Three families cover the design space: inflate the network into 3D convolutions, keep the 2D CNN as a per-frame encoder and add a sequence model on top, or simply pool or attend over per-frame features. The right choice trades temporal modeling power against compute cost and data hunger.

    (1) 3D CNNs (C3D / I3D): Extend 2D convolutions into 3D to learn motion directly; I3D inflates pretrained 2D filters into 3D, keeping ImageNet weights useful.
    (2) CNN + Sequence Model: Use the 2D CNN as a per-frame feature extractor, then run an LSTM / TCN / Transformer over the feature sequence for temporal modeling.
    (3) Temporal Pooling / Attention: Aggregate per-frame features with average/max pooling or attention: cheapest, but frame order and fine motion cues fade.

    Mechanism diagram of the CNN plus sequence model option: four video frames each pass through the same pretrained CNN, producing a stack of feature vectors that flows into an LSTM or Transformer temporal model, producing the class prediction

    Figure 1: The CNN + sequence model option: the pretrained 2D CNN is reused per frame, and a temporal model (LSTM/TCN/Transformer) learns how features evolve. This is the balanced default: full ImageNet leverage, moderate cost, natural variable-length handling.

    Clarify Before Designing:
    (1) Latency: real-time or offline? What is the per-clip inference budget?
    (2) Input Statistics: fps, clip duration, resolution: trimmed clips or untrimmed streams?
    (3) Data Scale: how much labeled video is available: hundreds or hundreds of thousands of clips?
    (4) Compute: can training and inference afford 3D-convolution cost?
    (5) Motion Sensitivity: do classes depend on subtle motion (opening vs closing) or mostly on scenes and objects?

    Login to view more content

  • MSD0001 Real-Time Factory Product Inspection

    

    You are tasked with designing and deploying a deep learning-based computer vision system for real-time quality control on a high-speed manufacturing assembly line. The system must classify each product as ‘Pass’ or ‘Fail’ due to surface defects (scratches, cracks, misalignments).

    Describe the complete end-to-end system design, from data acquisition and model selection to deployment and post-deployment maintenance.

    Crucially, how would you address the challenges of real-time inference speed and the severe class imbalance due to the fact that defects are rare?

    Problem framing diagram: a high-speed line producing 600 units per minute with surface defects flows into a question mark, with required outputs Pass or Fail within a 100 millisecond per unit budget at a 0.1 percent defect rate

    The Problem: 600 units per minute stream past a camera. Design the system that fills the question mark: real-time Pass/Fail within 100ms per unit, with defects occurring at only ~0.1%.

    Answer

    The solution is an Edge-AI Computer Vision Pipeline. It starts with a controlled imaging setup to capture consistent, high-quality images. The core is a lightweight CNN (e.g., MobileNetV2) fine-tuned via transfer learning, trained with a specialized loss (Focal Loss) to handle the rare-defect imbalance. Deployment runs on a local Edge GPU to guarantee low-latency inference, and a continuous MLOps loop monitors performance and retrains the model against new or subtle defects (concept drift).

    (1) Data & Setup: Controlled environment (lighting/staging) plus high-resolution cameras; transfer learning reduces the need for large-scale defect collection.
    (2) Imbalance Handling: Focal Loss (or weighted loss) combined with heavy data augmentation and oversampling of the rare ‘Fail’ class.
    (3) Model Architecture: A lightweight CNN (MobileNetV2, EfficientNet-B0) chosen for speed over a large, deep network.
    (4) Real-Time Deployment: Edge deployment on an industrial GPU (e.g., NVIDIA Jetson) with ONNX/TensorRT optimization and INT8 quantization to stay inside a sub-100ms budget.
    (5) Post-Deployment MLOps: Log every classification (especially false negatives) and trigger periodic retraining to combat model drift.

    End-to-end edge-AI quality control pipeline: camera and lighting feed preprocessing, an edge CNN classifier, a pass/fail actuator, and a logging path that loops back to periodic retraining

    Figure 1: The end-to-end pipeline. Solid arrows are the real-time inference path (capture → preprocess → edge CNN → actuator); the dashed loop is the offline MLOps path: logged hard examples and low-confidence frames flow back into retraining.

    Clarify Before Designing:
    (1) Throughput: line speed in units per minute, and the per-unit latency budget?
    (2) Defect Ontology: a closed set of defect types, or can new types appear in production?
    (3) Deployment Constraints: is cloud inference acceptable, or must everything run on-prem at the line?
    (4) Data Availability: how many labeled ‘Fail’ examples exist today, and what is the true defect rate?
    (5) Cost Asymmetry: the business cost of a false accept (defect escapes) versus a false reject (good unit scrapped)?

    Login to view more content