ML0086 Drift Detection in Production

You monitor in layers, because no single statistic sees everything. Layer one watches the inputs: per-feature distribution statistics (PSI, KS distance) of live traffic against the training baseline. Layer two watches the outputs: the prediction distribution and confidence profile, which catch label shift and model-side anomalies. Layer three estimates or measures performance: delayed ground truth when it arrives, and label-free estimators like NannyML’s CBPE in the gap. Each layer has thresholds with alerting, and alerts route to a decision: investigate the pipeline, retrain, or roll back. The whole pattern is standardized in AWS SageMaker Model Monitor: a baseline job computes statistics and constraints from training data, scheduled jobs compare live captures, and violations fire CloudWatch alarms.

(1) Baseline vs Window: freeze reference statistics from the training distribution, then compute drift metrics over sliding or scheduled production windows; the metric is a distance between two empirical distributions, not a model property.
(2) What Each Layer Sees: input stats see covariate shift, output stats see label shift and confidence collapse, and only ground truth (or calibrated estimation) sees concept drift; NannyML’s CBPE handles covariate shift but explicitly not concept drift.
(3) Production Pattern (AWS): Model Monitor’s built-in container (Deequ on Spark) emits statistics.json and constraints.json from a baseline, then scheduled monitoring jobs check distribution distance (linf_simple / two-sample KS via linf_robust, LInfinity or ChiSquared for categoricals) and write violation reports that drive alarms. AWS has announced that Model Monitor closes to new customers on 30 July 2026, with existing customers unaffected, but this baseline/constraints/violations design remains the reference pattern.

Figure 1: A drift monitor in action: per-window PSI against the training baseline crosses the moderate band (0.1) and then the significant band (0.25), which is where alerting and retraining triggers fire.

Design details separate a working monitor from alert fatigue. Thresholds must account for sample size (small windows inflate every distance metric), per-feature tests need multiplicity control or aggregation, and the window length trades detection latency against statistical power. When labels are delayed, record them when they land and backfill realized metrics so you can audit how well the label-free estimators tracked reality. Finally, every alert needs a playbook: distinguish a broken upstream pipeline (schema violations, sudden null spikes) from genuine distribution drift, because the fix for the former is a data engineer, not a retraining job.

Figure 2: The layered monitoring stack: input-distribution checks, output/confidence checks, and delayed-label evaluation feed one decision layer that separates pipeline breakage from genuine drift and triggers the right response.

Mathematical Formulation:

Where:

• and are the fractions of the current (actual) and reference (expected) samples in bin ; rules of thumb: PSI above 0.1 indicates moderate drift, above 0.25 significant drift.
• are the empirical CDFs of the baseline and current window; is their maximum vertical gap (the statistic behind SageMaker’s linf distance checks).
• Both are computed per feature per window and compared against thresholds tuned for window size and alert budget.