ML0099 Noisy Labels

Written by

How would you detect and correct inconsistent or noisy labels in training data?

Answer

Noisy labels are training labels that are wrong or inconsistent: a dog image labeled “cat,” a sentiment review labeled “positive” that is clearly negative, or the same entity labeled differently by different annotators. Detection starts with Confident Learning, a model-agnostic framework that uses cross-validated predicted probabilities to estimate the joint distribution between noisy labels and true labels, then flags examples where the model’s confident prediction disagrees with the given label. The open-source Cleanlab library implements this and has surfaced over 100,000 label issues in ImageNet. Correction ranges from simple relabeling (flip the label if the model is very confident) to weak supervision (using foundation models as labeling functions and denoising their outputs via a label model), to noise-robust training (CANOLA, 2026, achieves 19-52% improvement over SOTA label correction via noise-aware learning and iterative soft label refinement). Confident Learning is now taught as a core data-centric AI technique, and the original framework was published in JAIR 2021.

(1) Detection via Confident Learning: cross-validate the model to get out-of-sample predicted probabilities, estimate the joint distribution of noisy vs true labels, and flag examples where the predicted label confidently disagrees with the given label; Cleanlab surfaced 100,000+ label issues in ImageNet this way.
(2) Correction via Weak Supervision: foundation models (Llama 3.1, GPT-4, CLIP) serve as labeling functions that produce noisy labels, then a label model denoises them by learning each function’s precision and correlations, achieving 19.5% error reduction over zero-shot on the WRENCH benchmark.
(3) Correction via Noise-Aware Training: instead of hard relabeling, use soft labels (probability distributions over classes) refined iteratively during training; CANOLA (2026) achieves 19-52% relative improvement over SOTA, and the Relabeler framework (2026) achieves 58% improvement in label correction precision by jointly leveraging local and global data relationships.

Pipeline: cross-validated model produces out-of-sample probabilities, Confident Learning estimates the noisy-vs-true joint distribution and flags disagreements, then correction routes to relabel, weak supervision, or noise-aware soft-label training

Figure 1: The noisy-label pipeline: cross-validated predictions feed Confident Learning to detect label errors via the noisy-vs-true joint distribution, then correction routes to manual relabeling, weak supervision, or noise-aware soft-label training.

The detection step relies on a key insight: if a well-trained model confidently predicts “dog” for an image labeled “cat,” the label is more likely wrong than the model. Confident Learning formalizes this by estimating the joint distribution $P(\tilde{y}, y^*)$ of observed (noisy) labels and true labels using the cross-validated probability matrix, then identifying the most likely mislabeled examples via pruning, counting, and ranking. This requires no hyperparameters and works with any classifier. For correction, the simplest approach is to remove or relabel the flagged examples if you have access to a human annotator. If human annotation is expensive, weak supervision lets you define labeling functions as natural-language prompts to foundation models, and the label model combines their outputs by learning each function’s accuracy and correlations. For training-time correction, noise-aware methods like CANOLA replace hard labels with soft labels (class probability distributions) that are refined iteratively, avoiding the hard commitment of relabeling while reducing the impact of noise. A 2026 paper on spectral signatures showed that the tail index of eigenvalue distributions at network bottleneck layers predicts test accuracy under label noise with R-squared of 0.984, providing a diagnostic that identifies 9% noise in CIFAR-10N with 3% error.

A three-by-three matrix for classes cat, dog and bird: rows are the given noisy labels and columns the model's confident predictions, each row summing to 1, with green diagonal cells for agreement and red off-diagonal cells marking likely mislabeled examples

Figure 2: Row-normalized rates of confident predictions given each noisy label: off-diagonal mass is the estimated per-class mislabeling rate, and Confident Learning turns these confident counts into the noisy-versus-true joint distribution used to prune, count, and rank label errors.

Mathematical Formulation:
$\hat{P}(\tilde{y}=i,\, y^*=j) = \frac{1}{n}\sum_{k=1}^{n} \mathbb{1}[\tilde{y}_k = i] \cdot \mathbb{1}[\hat{p}_j(x_k) \geq t_j]$
$\hat{y}^*_k = \arg\max_j \hat{p}_j(x_k)$

Where:

$\hat{P}(\tilde{y}=i,\, y^*=j)$ is the estimated joint distribution of the noisy label $\tilde{y}$ and the true label $y^*$ ; $\hat{p}_j(x_k)$ is the cross-validated predicted probability of class $j$ for example $k$ .
$t_j$ is the average per-class confidence threshold (the average self-confidence for class $j$ ); examples where $\hat{p}_j(x_k) \geq t_j$ but $\tilde{y}_k \neq j$ are flagged as likely mislabeled.
$\hat{y}^*_k$ is the Confident Learning estimate of the true label: the argmax of the cross-validated probabilities, used for relabeling or soft-label assignment in noise-aware training.

Method	Approach	Requires Labels?
Confident Learning (Cleanlab)	Cross-validated probabilities estimate noisy-vs-true joint distribution	Noisy labels only
Weak Supervision	Foundation models as labeling functions; label model denoises	No labels needed; optional ground truth
Noise-Aware Training (CANOLA)	Iterative soft-label refinement during training	Noisy labels only
Spectral Signatures (2026)	Eigenvalue tail index at bottleneck layers predicts noise level	Diagnostic; no labels needed

Data

ML0099 Noisy Labels

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation