Design an end-to-end ML system to detect ads selling weapons and other policy-violating goods on a large ads platform like Meta or Google Ads. Millions of new ad creatives arrive daily as text, images, and video, and advertisers constantly adapt their creative to evade detection. Google’s Ads Safety Report describes blocking or removing billions of ads per year, so this is an industrial-scale enforcement problem, not a one-off classifier.
The system must block prohibited ads before they serve, route borderline cases to human reviewers, and keep the false-positive rate low enough that legitimate advertisers are not churned.
How would you design this system? Cover the multimodal detection model, the human-review queue prioritization, the adversarial adaptation problem, and how you measure recall and precision at platform scale.
The Problem: millions of new creatives arrive every day, roughly one in ten thousand is a prohibited-goods listing, and the decision must be made before the ad is shown even once.
Answer
The design is a pre-serve multimodal gate inside the ad-ingestion path. A cheap first stage kills known-bad creatives by hash and embedding near-duplicate lookup, then a shared vision-language encoder with per-policy heads scores every creative (ad copy, OCR text, landing-page text, image, video keyframes) before the first impression. Each policy class gets three calibrated bands: auto-block, human review, allow. Two decisions carry the design. First, thresholds are set from the asymmetric cost of a served weapon ad versus a wrongly blocked advertiser, per policy and per locale, rather than from a single global score cut. Second, the whole system is treated as an adversarial loop where the time from a new evasion pattern to a deployed block (hours, not quarters) matters more than static offline AUC.
(1) Staged Funnel: creative-hash cache and embedding kNN against known violations resolve most repeat traffic in milliseconds; only novel creatives pay for the full multimodal pass.
(2) Multimodal Model: a frozen vision tower plus a multilingual text tower over copy, OCR, ASR, and landing-page text, fused into multi-label per-policy heads instead of one binary “violating” output.
(3) Three-Band Decision Policy: per-policy calibrated thresholds split traffic into auto-block, review, and allow, so precision on the auto-block band is protected by construction.
(4) Expected-Harm Review Ranking: the reviewer queue is sorted by violation probability times forecast impressions times policy severity, not by score alone, and clusters of near-duplicate creatives collapse into one decision.
(5) Adversarial Response Loop: evasion signals (re-upload similarity, account bursts, appeal patterns) feed hard-negative mining, an embedding blocklist hotfix within hours, and account-level enforcement that raises the attacker’s cost.
(6) Measurement by Sampling: recall is estimated from stratified random audits of allowed traffic, precision from reviewer audits of blocks and from appeal-overturn rates.
Figure 1: Ingest → asset extraction → encoders → per-policy scores → three-band decision, with reviewer labels and the near-duplicate index closing the loop back into the model.
Clarify Before Designing:
(1) Scale and Asset Mix: how many new creatives per day, how many assets each, and what share is video (video dominates compute cost)?
(2) Enforcement SLA: must an ad go live within minutes of upload, or is holding a creative for a deep asynchronous pass acceptable?
(3) Policy Taxonomy: how many prohibited-goods classes, which are legally mandated, and how much do they vary by jurisdiction?
(4) Cost Asymmetry: what is the tolerated false-block rate before advertiser churn becomes the bigger loss, and what is the appeal SLA?
(5) Review Capacity: how many reviewer hours per day, in which languages, and with what accuracy on gold sets?
(6) Label Availability: do we have historical enforcement decisions, appeal outcomes, and account-level history, or are we starting from policy text only?
Leave a Reply