ML0042 Early Stopping

Written by

What is Early Stopping? How is it implemented?

Answer

Early stopping is a regularization technique that halts training when the model’s performance on a validation set stops improving, thus avoiding overfitting. It monitors a metric such as validation loss or validation accuracy and stops after a defined number of stagnant epochs (the patience). This ensures efficient training and better generalization.

(1) Split Data: Reserve a validation set separate from the training set.
(2) Evaluate Each Epoch: After every training epoch, measure performance on the validation set.
(3) Track Improvement: If performance improves, save the model and reset the patience counter; if not, increment the counter; when it reaches the patience, stop training.
(4) Restore Best Weights: After stopping, reload the weights from the epoch that yielded the best validation performance, not the final epoch.

Training loss keeps decreasing while validation loss bottoms out at epoch 60 and rises again, with the actual stop at epoch 70 under patience 10

Figure 1: Early stopping in action: training loss falls monotonically, but validation loss bottoms out at epoch 60 (ideal stop) and then rises as the model overfits. With patience 10, training actually halts at epoch 70 and the weights from epoch 60 are restored.

Mathematical Formulation:
$t^* = \arg\min_{t} \; \mathcal{L}_{\text{val}}\big(\theta_t\big)$
$\text{stop at } t^* + p \text{ if no epoch in } (t^*,\, t^* + p] \text{ beats } \mathcal{L}_{\text{val}}(\theta_{t^*})$

Where:

$\theta_t$ are the model weights after epoch $t$ ; $\mathcal{L}_{\text{val}}$ is the validation loss.
$t^*$ is the epoch with the best validation loss, the checkpoint whose weights are restored at the end.
$p$ is the patience: how many consecutive non-improving epochs are tolerated before stopping.