ML0097 Data Leakage

Data leakage is when information from outside the training set enters the model training process, giving the model knowledge it would not have at prediction time. The result is inflated training and validation metrics that collapse when the model meets real test data, because the leaked signal is absent in production. The most common forms are target leakage (a feature that encodes the label, like “post-purchase amount” in a churn model), train-test contamination (fitting a preprocessing step on the full dataset before splitting, so test statistics bleed into training), and temporal leakage (using future data to predict the past, like computing rolling statistics that include the target time step). At production scale, data governance frameworks use Information Flow Control to enforce purpose limitations on data, blocking problematic data transfers before they occur, and require data lineage tracking and policy compliance checks at training job configuration time to prevent training-data contamination.

(1) Target Leakage: a feature is derived from the label or from information unavailable at prediction time (e.g., “days since last purchase” computed after the churn window closes); the model learns a shortcut that disappears in production.
(2) Train-Test Contamination: fitting scalers, imputers, feature selectors, or PCA on the full dataset before the train-test split, so test-set statistics leak into training; the fix is to fit preprocessing only on the training fold and apply it to test, ideally via a scikit-learn Pipeline.
(3) Temporal Leakage: in time-series, using future observations to compute features for past predictions (e.g., a rolling mean that includes the target timestamp, or shuffling time-series data before splitting); the fix is a time-based split and forward-only feature computation, plus production-scale data governance frameworks for purpose limitation and lineage tracking.

Figure 1: The three most common leakage types: target leakage (a feature derived from the label), train-test contamination (preprocessing fitted before the split), and temporal leakage (future data in past feature computation).

Detection starts with a sanity check: if validation or test performance is suspiciously high compared to production, or if a single feature has outsized importance, suspect leakage. A practical diagnostic is permutation importance on a held-out set: a leaked feature will show extreme importance because shuffling it destroys the shortcut. For time-series, always use a time-based split (train on the past, test on the future) and never shuffle before splitting. For preprocessing, use a scikit-learn Pipeline or ColumnTransformer that fits only on the training fold inside cross-validation, so scalers, imputers, and encoders never see test data. For feature engineering, audit each feature for availability at prediction time: if a feature depends on data that arrives after the prediction, it is leaked. At scale, production data governance frameworks propagate privacy annotations across millions of daily data flows and block problematic transfers before they occur, while requiring data lineage tracking and policy compliance checks at training job configuration time.

Mathematical Formulation:

Where:

• is the mutual information between the test labels and the training features; leakage means this is positive, i.e., the training features contain information about the test labels that should not be there.
• is the model trained on the training set; the goal is that depends only on and nothing from the test set, including test-set statistics used in preprocessing.
• Target leakage is the case where itself contains a feature derived from (or ), so the model learns from via a shortcut. Temporal leakage is the case where a feature is computed from data observed at some later time , which is unavailable when the prediction must actually be made.