DL0181 Label Smoothing and Model Calibration

Label smoothing replaces the one-hot target with a mixture of the one-hot vector and the uniform distribution, so the true class receives and every other class receives . Written this way it is exactly standard cross-entropy plus an -weighted KL term pulling the prediction toward uniform, which gives the loss a finite minimizer instead of one that is reached only as the true-class probability approaches 1. That single change removes the pressure that makes logit gaps grow without bound after the argmax is already correct, so the model stops becoming more confident with more training and the classic overconfidence of modern networks largely disappears. The consequences are asymmetric: Expected Calibration Error (ECE) usually drops sharply and top-1 accuracy is flat or slightly better, but the hard-label negative log-likelihood gets worse, because a converged smoothed model deliberately withholds probability mass from the correct class. The Transformer paper states this trade-off explicitly, using and noting that it hurts perplexity while improving accuracy and BLEU.

(1) Target Mixing: the label becomes , which is equivalent to cross-entropy plus up to an additive constant.
(2) Bounded Logit Gap: the optimum sits at a finite logit difference, roughly 9.1 nats for and , so logit norms stop inflating.
(3) Confidence Ceiling: a well-fit smoothed model cannot report top-class confidence above , which is 0.900 at .
(4) Calibration Improves, Then Overshoots: ECE is U-shaped in , moving the model from overconfident through calibrated to systematically underconfident.
(5) Log-Likelihood Penalty: the confidence ceiling implies a hard-label NLL floor of , about 0.105 nats per example at .
(6) Information Erasure: smoothing equalizes the wrong-class logits, which tightens penultimate-layer clusters and damages knowledge distillation and feature transfer.

Figure 1: Reliability curves for the same architecture under three targets. Hard labels fall below the diagonal (confidence exceeds accuracy), tracks the diagonal, and lands above it. Note the truncated horizontal extent: each smoothed curve stops at its confidence ceiling , 0.900 and 0.700 for . Curves are illustrative of published trends rather than a single benchmark run.

The mechanism is easiest to read off the gradient with respect to the true-class logit, which is under hard labels and under smoothing. The hard-label gradient is strictly negative for every finite logit, so optimization keeps pushing the correct logit away from the others long after the prediction is right, and confidence keeps drifting upward while accuracy has already plateaued. The smoothed gradient changes sign once the model reaches the target probability, which pins confidence and caps the logit gap. The wrong-class gradients change too: each incorrect logit is pulled toward the same small target , so the model is discouraged from expressing that “husky” is much closer to “wolf” than to “airliner”. That equalization is exactly the property Müller and colleagues identified as the reason a label-smoothed teacher makes a worse distillation teacher than a hard-label teacher of identical accuracy.

Mathematical Formulation:

Where:

• is the one-hot label and the smoothed target over classes.
• is the smoothing strength, recovering ordinary cross-entropy; is the near-universal default.
• is the predicted probability, the logits, the index of the correct class and any incorrect class.
• is the uniform distribution, and is a constant independent of the parameters, which is why the objective is cross-entropy plus a uniform-KL penalty.
• is the per-example minimizer, giving the finite logit gap in the fifth line; the gap grows only logarithmically as shrinks.
• is the set of the validation examples whose top confidence falls in bin of (typically ), and ECE is the confidence-weighted gap plotted in Figure 1.

Log-Likelihood Floor At , :

The third line is the practical reading for sequence models: a converged smoothed model pays roughly 11% higher perplexity than its own hard-label counterpart would at the same accuracy, purely from the withheld mass, which is why translation systems that smooth at training time are usually evaluated with BLEU or COMET rather than perplexity. Note also that this floor is a property of the optimum, not a hard constraint on the network: a single global temperature fitted on held-out data can sharpen the smoothed logits back and recover most of the lost likelihood. That observation is the reason smoothing is not the right tool if calibrated probabilities are the goal, since post-hoc temperature scaling reaches comparable ECE at zero training cost, one scalar parameter, and no distortion of the wrong-class ranking.

Figure 2: The three metrics disagree about the best . ECE is U-shaped with a minimum near to , hard-label NLL dips only briefly before the analytic floor dominates, and accuracy is flat across a broad plateau. Choosing therefore means choosing which metric you are optimizing. Values are illustrative of published sweeps.

Property	Label smoothing	Temperature scaling	Confidence penalty / focal loss
When applied	Training time, changes the targets	Post-hoc, one scalar fitted on a held-out split	Training time, changes the loss on the prediction side
Effect on accuracy	Flat to slightly better; degrades past roughly 0.2	Exactly zero, the argmax is invariant to a positive temperature	Task dependent; focal loss helps mainly under heavy class imbalance
Effect on hard-label NLL	Worse at convergence, floored at	Directly minimized by the fitting objective, so it improves	Usually worse, for the same mass-withholding reason
Logit ranking preserved	No, wrong-class logits are equalized	Yes, it is a monotone rescaling of all logits	Partially, the entropy term flattens the tail
Main failure mode	Underconfidence at large ; weaker distillation teacher and weaker transfer features	One global scalar cannot fix per-class or per-slice miscalibration, and it needs a clean validation split	Extra hyperparameter with little calibration gain over a fitted temperature