ML0003 Overfitting

Overfitting happens when a model learns the training data too well (including its noise and outliers) and as a result performs poorly on new, unseen data. The model becomes too specialized to the training set and fails to generalize. The telltale sign is a growing generalization gap: training loss keeps falling while validation loss turns back up. To avoid overfitting: simplify the model, get more data or use data augmentation, apply regularization (L1/L2), validate frequently with early stopping, and for neural networks use dropout.

(1) Definition: The model memorizes noise as if it were signal, so training error keeps dropping while test error rises.
(2) Detection: Watch the train/validation loss gap and use cross-validation: wildly varying performance across folds indicates overfitting.
(3) Remedies: More or augmented data, L1/L2 regularization, dropout, early stopping, or a smaller model; all reduce effective capacity or expose the model to more variation.

Figure 1: The validation loss minimum marks the ideal stopping point; training past it widens the generalization gap; that widening is overfitting.

Mathematical Formulation:

Where:

• is the original training loss over parameters .
• is the regularization strength; larger values shrink the weights toward zero, trading training fit for generalization.
• is the squared L2 norm of the weights (weight decay); an L1 penalty instead drives weights to exactly zero.
• and are validation and training loss; a small, stable gap indicates good generalization.

Figure 2: Why the remedies work: they move the model left along the complexity axis, out of the high-variance region and back toward the total-error minimum.