Design a multimodal search system where a user types a text query and the system returns relevant items from a catalog of billions of images and videos, like Pinterest Visual Search, Google Lens, or TikTok’s visual search. The query may describe objects, scenes, styles, or actions that never appear as literal text anywhere in the catalog (for example “golden retriever catching a frisbee at sunset”), so the system must bridge the modality gap between language and visual content.
Results must return in under 200 ms, the index must stay fresh as new content is uploaded continuously, and the system must handle queries that mix text with an uploaded reference image.
How would you design this system? Cover the shared embedding space and training strategy, the approximate nearest neighbor index at billion scale, the freshness and update strategy, the hybrid text-plus-image query path, and how you evaluate retrieval quality.
The Problem: the words the user types appear nowhere in the catalog, yet the right pixels have to surface out of billions of items inside 200 ms, while new uploads keep arriving.
Answer
The design is a two-tower (dual encoder) retrieval system over one shared embedding space, served as a two-stage retrieve-and-rerank path. A contrastive image-text model of the SigLIP/CLIP family, fine-tuned on in-domain query-engagement pairs, maps text, images, and video clips into the same 512-d space, so “golden retriever catching a frisbee at sunset” becomes a vector whose neighbors are the right pixels even though no matching caption exists. Two decisions carry most of the weight. First, the item side is multi-vector for video (a handful of clip embeddings rather than one averaged vector), because averaging a 60-second video destroys the action the query describes. Second, freshness is solved with a tiered index (hot in-memory buffer, hourly delta shards, nightly rebuilt base) instead of mutating one billion-scale graph in place. The hybrid text-plus-image query runs through a learned combiner that emits a single vector, so the retrieval and ranking path downstream never changes.
(1) Shared Embedding Space: one contrastively trained space for query text, images, and video clips, so retrieval is pure vector search rather than text matching against noisy alt-text.
(2) In-Domain Fine-Tuning: start from a public image-text checkpoint, then fine-tune on click and save pairs from real query logs, with hard negatives mined from the same logs.
(3) Multi-Vector Video Items: sample frames or short clips per video and index several vectors per item, collapsing to one item at merge time; single pooled vectors lose action and scene changes.
(4) Sharded IVF-PQ With Rerank: compress to 64-byte codes so 5 billion vectors fit in roughly 320 GB of RAM across shards, retrieve about 10k candidates, then rescore with full-precision vectors plus business features.
(5) Tiered Freshness: a hot buffer makes new uploads searchable in about a minute, hourly deltas absorb the day, and a nightly base rebuild keeps the big index compact; deletes ride a tombstone list.
(6) Learned Query Combiner: text-plus-image queries are fused into one vector by a small trained combiner, keeping the index path and latency budget identical to text-only search.
Figure 1: One embedding space, two write paths: queries are encoded online, catalog items are encoded on ingest, and the tiered index is the only place they meet.
Clarify Before Designing:
(1) Scale and traffic: how many items (billions of images, or billions of items with heavy video), what peak QPS, and how many uploads per hour?
(2) Latency budget split: is the 200 ms end-to-end including network and thumbnail fetch, or only the retrieval service?
(3) Freshness requirement: must a new upload be findable in seconds (live events, breaking trends) or is an hour acceptable for most content?
(4) Query mix: what fraction is head navigational text, tail descriptive text, action queries over video, and hybrid image-plus-text?
(5) Quality bar and cost asymmetry: is the product optimizing engagement, commercial conversion, or safety-first precision, and what is an acceptable cost per 1k queries?
(6) Available supervision: do we have query-click logs and human relevance ratings, or only alt-text and hashtags?
Leave a Reply