Design a system that automatically tags photos uploaded by hosts (e.g., “kitchen”, “bedroom”, “pool”, “ocean view”) on Airbnb using a deep image classification model. When a host creates or edits a listing, they upload dozens of photos, and Airbnb must automatically label each photo with the room type or amenity it depicts so that guests can filter by feature (“pool”, “mountain view”) and the listing page can organize photos into structured sections (Living Room, Bedroom, Kitchen, Exterior). The system must handle a wide taxonomy of tags (50+ room types and amenities), work across varying photo quality, angles, and lighting from phone cameras worldwide, distinguish similar categories (“living room” vs “family room”, “balcony” vs “patio”), and handle the long tail of rare amenities (“sauna”, “wine cellar”) that have few training examples. Mislabeling a bathroom as a kitchen degrades search relevance and user trust, so the system must flag low-confidence tags for host confirmation rather than auto-assigning them.
How would you design this system? Cover the classification model architecture (single-label vs multi-label vs hierarchical classification), how you handle the long tail of rare tags with few training examples, how you generate training labels from host-provided metadata and guest clicks, how you calibrate confidence for the host-confirmation workflow, the inference pipeline for processing photo uploads in real time, and how you evaluate tagging accuracy across the full tag taxonomy.
The Problem: Thirty phone photos arrive per listing and each must land on one of fifty-plus tags, where the popular tags have millions of examples, the rare ones have a few hundred, and a confident wrong tag costs more trust than no tag at all.
Answer
The design is a shared vision backbone with two heads: a hierarchical softmax head for the mutually exclusive room or scene type, and a multi-label sigmoid head for amenities that co-occur in one photo. Training labels are mined at scale from host captions, existing photo-section placement, and guest filter clicks, then sharpened by the host confirmations the product itself generates. Rare tags are handled by decoupling representation from classifier: the backbone learns from the head-heavy data, the classifier heads are re-balanced, and tags with too few examples are served through a nearest-prototype path until they graduate. Every prediction passes a per-tag calibrated confidence gate with three outcomes: auto-assign, a one-tap host confirmation card, or leave untagged. Inference runs as a batched GPU service triggered by the upload event, so a 30-photo listing is tagged before the host finishes the flow, and every confirmation flows back into the next training round.
(1) Label Structure: a scene tree (indoor, outdoor, view) with roughly 40 leaves under a hierarchical softmax, plus roughly 30 amenity tags under independent sigmoids; sibling pairs humans confuse (“living room” vs “family room”) share one merged leaf until a reliable cue separates them.
(2) Weak Labels at Scale: host captions, section placement, listing-level amenity declarations, and guest clicks after filtering by an amenity give millions of noisy labels; host confirmations and removals form the small gold layer that drives thresholds and evaluation.
(3) Long Tail by Decoupling: train the backbone on all data, then re-fit the classifier heads with class-balanced sampling and logit adjustment; tags below about 5k labels use a prototype classifier on frozen embeddings.
(4) Calibrated Confidence Gate: temperature scaling per head and per-tag thresholds tuned on the confirmed holdout to a 98% precision floor; high confidence auto-assigns, the gray zone asks the host, low confidence stays untagged.
(5) Async GPU Serving at Upload: an upload event triggers fetch, dedupe, batch inference, and a write to a versioned tag store that feeds search filters and section layout; the catalog backfill is a batch job on the same model.
(6) Confirmation Feedback Loop: confirmations, removals, and re-sectioning are logged as labels with model version, and monthly retraining passes per-region and per-tag gates before release.
Figure 1: One backbone pass per photo, two heads, one gate: confident tags ship, doubtful ones become a host card, and every host answer becomes a training label.
Clarify Before Designing:
(1) Taxonomy Ownership: is the 50-tag list fixed or does the product team add tags quarterly, and does a hierarchy (room vs amenity vs view) already exist?
(2) Latency Expectation: must tags appear inside the upload flow within seconds, or only before the listing goes live minutes later?
(3) Label Sources: what fraction of photos carry a host caption or a section placement, and are guest filter-and-click logs joinable to individual photos?
(4) Error Asymmetry: which hurts more, a missing “pool” or a false “pool”, and is there a precision floor the search team already enforces?
(5) Scale: photos uploaded or edited per day, catalog size for a backfill, and the share of phone versus professional photos?
(6) Confirmation Budget: how many confirmation prompts per upload before hosts disengage, and may a listing publish while tags are pending?
Leave a Reply