Design a landmark recognition model for a travel app that identifies famous landmarks from user photos and returns relevant historical and visitor information. When a traveler points their phone at the Eiffel Tower, the Colosseum, or a lesser-known temple, the model must identify the landmark in seconds, even from unusual angles, partial occlusion, varying lighting, or when the photo is taken from inside rather than the iconic exterior view.
The system must handle the long tail of tens of thousands of landmarks worldwide, many with only a handful of reference images, and it must gracefully say “I don’t recognize this” rather than confidently misidentifying a similar-looking building.
How would you design this model? Cover the candidate recognition approaches (global descriptor retrieval vs local-feature matching vs deep metric learning with classification), how you handle the long tail of rare landmarks with few training images, how you build the reference index, how you calibrate confidence to enable abstention, and how you evaluate recognition accuracy and false-positive rate.
The Problem: the query is rarely the postcard view, the tail landmark has a handful of reference photos, and a confident wrong name costs more trust than an honest “not sure”.
Answer
Treat this as open-set instance retrieval with abstention, not 100k-way classification. A backbone trained with a margin-based classification loss (ArcFace with sub-centers) produces a compact global descriptor; that descriptor retrieves top candidates from an ANN index over all reference photos; local-feature matching with geometric verification re-ranks those candidates and produces an inlier count. The two pivotal decisions are to keep the final decision in the index rather than in a softmax head, so a new temple is added by indexing seven photos instead of retraining, and to build the confidence signal from geometric inliers plus descriptor margin rather than softmax probability, because that is what makes a calibrated “I don’t recognize this” possible.
(1) Global Descriptor Retrieval: GeM-pooled CNN or ViT embedding, PCA-whitened to 512-d, searched with IVF-PQ over every reference image; fast, memory-light, viewpoint-sensitive.
(2) Local Feature Matching: keypoints and descriptors (DELF, SuperPoint) matched pairwise, then RANSAC for a consistent transform; the inlier count is both the score and the explanation.
(3) Deep Metric Learning With Classification Head: train with ArcFace or sub-center ArcFace over the landmark ID vocabulary, discard the head at inference and keep the embedding for retrieval.
(4) Chosen Hybrid: (3) trains the descriptor, (1) does the cheap recall, (2) verifies the top 100, and a calibrated gate over inliers and margin decides answer versus abstain.
Figure 1: Cheap recall first, expensive verification second, and an explicit gate that is allowed to answer nothing.
Clarify Before Designing:
(1) Inventory And Reference Depth: how many landmarks, and what is the reference-count distribution (median photos per landmark, how many with fewer than 10)?
(2) Compute Budget: server-side with a 1 s budget, or on-device with a fixed memory ceiling and no index round trip?
(3) Metadata: is GPS or EXIF available at query time? A 5 km geo prefilter removes most look-alike confusions for free.
(4) Cost Asymmetry: what precision floor does the product require, and what abstention rate is acceptable to hit it?
(5) Definition Of Recognized: do interiors, statues, details, and reconstructions count as the same landmark, and how are near-identical siblings (replicas, chain temples) labeled?
(6) Data Rights: can web-mined and user-submitted photos enter the index, and can queries be logged for hard-negative mining?
Leave a Reply