Design a system to detect fake accounts and bots on a social network like Facebook, Twitter/X, or Instagram. Fake accounts power spam, coordinated inauthentic behavior, follower farming, and political manipulation, and their operators evolve constantly: they buy aged accounts with real history, use LLMs to write bios and mimic human posting cadence, and route traffic through residential proxies so IP reputation says nothing.
The system must score accounts at creation time and keep re-evaluating them as behavior develops, while holding the false-positive rate low enough that real users are not wrongly suspended, and routing the highest-risk cases to human investigators.
How would you design this system? Cover the account-level feature signals (registration, graph, content, behavior), the detection model architecture, how you handle adversarial evolution, the human-in-the-loop review queue, and how you measure precision and recall when ground truth is scarce.
The Problem: millions of new accounts a day, a farm that can forge everything cheap to forge, and one real person sitting in the same batch. Decide who to trust without suspending them.
Answer
Treat this as two coupled problems running at two speeds. At signup, a low-latency gradient-boosted scorer over registration, device, and infrastructure features decides whether the account starts with friction, and a continuous re-scoring layer then re-judges every live account as its graph and behavior accumulate. That second layer is where the real detections happen: a relational GNN over the follow and friend graph plus unsupervised synchrony clustering (the SynchroTrap idea of co-clustering accounts that act on the same targets in the same time buckets) catches campaigns whose individual members each look perfectly human. Scores never map directly to bans. They map to a graduated enforcement ladder (allow, challenge, limit reach, human review, disable) so the expensive mistake, suspending a real user, requires either overwhelming cluster-level evidence or an investigator.
(1) Two-Speed Detection: a synchronous per-signup scorer under a 200 ms budget, plus a daily (and event-triggered) batch pass over every active account, since most fake accounts only reveal themselves after they start acting.
(2) Cluster-Level Evidence: the decision unit is the account and the campaign; graph plus action-synchrony clustering finds coordinated groups without needing labels for the new tactic.
(3) Cost-To-Forge Feature Weighting: prefer signals an operator must pay real money or real time to fake (device integrity, aged organic graph, human-only interaction telemetry) over signals a proxy or an LLM erases for free.
(4) Graduated Enforcement: friction first (phone or CAPTCHA challenge, reach limiting), auto-disable only in the top band with corroboration, always with an appeal path that feeds labels back.
(5) Label Machinery: reviewer verdicts, appeal reversals, honeypot and test-buy cohorts, and a random-audit stream, because production has no clean ground truth and the flagged sample alone teaches the model its own blind spots.
Figure 1: Two speeds, one policy: a synchronous scorer at signup, a daily graph and synchrony pass over every live account, and a ladder whose top band ends at a human, not a model.
Clarify Before Designing:
(1) Scale And Prevalence: how many signups per day, how many active accounts to re-score, and what does the current prevalence estimate say (a few percent of MAU)?
(2) False-Positive Budget: how many wrongly restricted real users per day is the product willing to accept, and is auto-suspension allowed at all under the platform’s appeal and regulatory commitments?
(3) Enforcement Surface: can we impose signup friction (phone verification, CAPTCHA), or does growth own that decision and force us to detect after the fact?
(4) Review Capacity: how many investigator-hours per day exist, in which languages and regions, and what is the target queue latency for high-harm cases?
(5) Threat Priority: which abuse type pays first, follower farms, spam, or coordinated political campaigns, since they differ in scale, sophistication, and tolerance for delay?
(6) Signal Access: what device, telemetry, and payment signals are legally usable per region, and how long may we retain them?
Leave a Reply