On a short-video platform like TikTok or Instagram Reels, creators attach hashtags to reach an audience, and a large share of those tags are clickbait: #fyp or a trending sports tag stapled onto a video that has nothing to do with the topic behind it. The core problem is a multimodal relevance scoring mechanism. Given a video (sampled frames, audio, OCR’d on-screen text, creator caption) and a candidate hashtag, produce a relevance score that separates honest tagging from spam.
How would you design this model? Cover the candidate fusion architectures (late fusion of per-modality encoders, early token-level fusion, cross-attention, contrastive hashtag-video embedding), how you represent the hashtag itself (its text, its trending-topic cluster, its co-occurrence graph), how you generate training labels at scale when human-labeled relevance is scarce, and how you handle cold start for a brand-new hashtag with no co-occurrence history.
The Problem: the same video carries an honest tag and a hijacked trending tag, and nothing in the upload tells you which is which. You need a score for every (video, tag) pair, over a tag vocabulary of tens of millions and with almost no human relevance labels.
Answer
The design is a two-stage multimodal relevance model. A contrastive dual tower is pretrained on billions of weakly labeled creator (video, tag) pairs, which gives every tag in the vocabulary a cheap vector and initializes both encoders. A cross-attention scorer then rescores the handful of tags actually attached to a video, letting the tag’s query tokens read the video’s frame, audio, OCR, and caption tokens so a localized mismatch (a football tag on a pasta video, or a single on-screen word borrowed from a trend) becomes visible. Two decisions are pivotal. First, the hashtag is a sum of three slots (subword text, trending-topic cluster centroid, co-occurrence graph embedding) trained with slot dropout, which is what makes cold start work rather than a special case bolted on later. Second, labels come from creator behavior plus frequency-matched hard negatives drawn from the same topic cluster, anchored by a small human gold set used only for calibration and as the release gate.
(1) Late Fusion: one encoder per modality, each pooled to a vector, concatenated with the tag vector into a shallow MLP; everything cacheable, no token-level interaction.
(2) Early Fusion: all modality tokens plus the tag tokens in one transformer from layer 1; maximally expressive and completely uncacheable across candidate tags.
(3) Cross-Attention Scorer: video tokens encoded once as keys and values, hashtag slots as a small query set, a few cross-attention layers plus a sigmoid head.
(4) Contrastive Embedding: dual-tower video and tag encoders trained with InfoNCE or a sigmoid pairwise loss so relevance is a dot product in a shared space.
Figure 1: The scoring mechanism: the video is encoded once into a token set, the hashtag arrives as three composable slots, and cross-attention decides whether the video contains evidence for the tag.
Clarify Before Designing:
(1) Input budget: how many frames per video, is audio and reliable OCR available, and is the caption trustworthy or itself keyword-stuffed?
(2) Compute budget: how many candidate tags are scored per video, and may the video encoder run once at upload so per-tag cost is the only thing multiplied?
(3) Pretrained assets: is there an in-house video-language encoder to freeze, or must the towers be trained from scratch on platform data?
(4) Vocabulary shape: how large is the tag vocabulary, what fraction of traffic is meta-tags such as #fyp, and how fast do new tags appear?
(5) Label reality: how many human-labeled pairs exist today, and can we buy a gold set for calibration even if we cannot buy a training set?
(6) Output contract: is the consumer a hard threshold, a ranking feature, or a reviewer queue, since only the first two need a calibrated probability?
Leave a Reply