A large platform runs thousands of ML models in production: ranking models, fraud classifiers, content moderators, recommendation embedders, each with its own feature pipeline, serving path, and delayed-label feedback. A single broken upstream feature, a silent model upgrade, or a gradual distribution shift can degrade any of them for days before anyone notices.
The ML platform team needs one real-time monitoring system that surfaces drift, performance, and data-quality issues across the entire fleet, with alerting that catches real regressions without drowning on-call engineers in false alarms. How would you design this system? Cover the metrics each model emits, the streaming and storage architecture for thousands of models, the drift and anomaly detection methods that scale across heterogeneous model types, the alerting and triage workflow, and how you measure the monitoring system itself (mean time to detect, alert precision).
The Problem: one model in a fleet of thousands is quietly getting worse, and the on-call engineer is already ignoring a pager full of alerts that were never real.
Answer
The design is a fleet-wide observability service with three layers. Every serving path emits a uniform telemetry record through a shared logging SDK, a streaming job compresses those records into per model, feature, slice and window sketches, and a detection layer runs cheap statistical guards on everything plus expensive label-joined performance checks on the models that carry real money. The two pivotal decisions are that detection never scans raw events at query time (it reads compact profiles, which is what makes thousands of models affordable), and that alerting is engineered for precision rather than coverage: a trigger must clear an effect-size floor, persist across several windows, and survive root-cause grouping before it pages anyone. The monitoring system is itself a product with SLOs, scored on mean time to detect and alert precision against continuously injected synthetic regressions.
(1) Uniform Telemetry Contract: one prediction record schema (model id and version, feature values or hashes, score, slice keys, join key, latency) emitted asynchronously by a sidecar or SDK, so the platform never reimplements logging per team.
(2) Sketch-Based Aggregation: a streaming job keeps t-digest quantiles, count-min sketches for categoricals, and null/count/cardinality counters per 5-minute window, turning terabytes of events into tens of gigabytes of profiles.
(3) Tiered Detection: tier-1 data-quality and drift guards run on every model and feature; tier-2 label-joined AUC, calibration, and slice regressions run only on business-critical models.
(4) Delayed-Label Handling: proxy signals (score distribution, positive rate, confidence) fire in minutes, while true performance is computed on maturity-aware cohorts so immature labels never trigger a page.
(5) Precision-Engineered Alerting: adaptive seasonal baselines, an effect-size floor, multi-window persistence, and deduplication by feature lineage turn thousands of triggers into a handful of owned incidents.
(6) Self-Measurement: synthetic drift and corrupted-feature injections give a labeled incident corpus, so MTTD and alert precision are measured, not asserted.
Figure 1: Raw predictions are compressed once into profiles; every detector, dashboard, and alert afterwards reads those profiles instead of the event stream.
Clarify Before Designing:
(1) Fleet Shape: how many models, how many features each, aggregate QPS, and how many are business-critical versus experimental?
(2) Detection Budget: is the target time to detect minutes, an hour, or a day, and is it the same for a fraud classifier and a feed ranker?
(3) Label Latency: per use case, how long until ground truth arrives (clicks in seconds, chargebacks in weeks, moderation appeals in days)?
(4) Cost Ceiling: what fraction of serving cost may monitoring consume, and can we sample rather than log every request?
(5) Remediation Authority: may the system auto-rollback a model version or quarantine a feature, or does it only page a human owner?
(6) Data Constraints: which features are PII or regulated, and what is the retention limit on stored feature values?
Leave a Reply