ML0004 Underfitting

Underfitting occurs when a model is too simple to capture the underlying patterns in the data, so it performs poorly on both the training data and new, unseen data: it has not even learned the training set. An underfit model exhibits high bias and low variance: its predictions are consistently wrong in the same way, regardless of the particular training sample. Common causes are an overly simple model, inadequate training (stopped too early), over-regularization, and poor feature selection. The fixes mirror the causes: increase model complexity, train longer, reduce regularization, and engineer more informative features.

(1) Definition: The model lacks the capacity to fit the signal, so error stays high on training and test data alike.
(2) Recognition: High training error is the key signature; contrast with overfitting, where training error is low and only validation error suffers.
(3) Fixes: Add capacity (more layers, higher-degree features), train longer, weaken regularization, and improve the feature set.

Figure 1: The same data fitted three ways. The underfit line misses the pattern entirely; the good fit follows the trend; the overfit curve chases every noisy point.

Mathematical Formulation:

Where:

• is the expected prediction error of the model at a new point .
• measures how far the average prediction sits from the truth: the term that dominates when a model underfits.
• measures how much the prediction swings across different training sets: the term that dominates in overfitting.
• is the irreducible noise in the data, which no model can remove.