ML0038 Validation and Test

Written by

What are the key purposes of using both a validation and a test set when building machine learning models?

Answer

The validation set and the test set play two different roles that must not be merged: validation guides every development decision, the test set is touched exactly once for the final unbiased verdict. During development you use the validation set to tune hyperparameters (learning rate, architecture, regularization), select among candidate models, and monitor overfitting (e.g., for early stopping): it is the “unseen” data you are allowed to peek at repeatedly. But because those repeated peeks gradually fit your decisions to the validation set, its score becomes optimistically biased. The test set therefore stays locked away until the model and all its settings are frozen: evaluating on it once simulates real-world performance on genuinely unseen data and guarantees no information from it leaked into any modeling choice. Using the validation set as the test set destroys that guarantee; with very scarce data, rigorous cross-validation during development is the acceptable compromise.

(1) Validation Set: Tunes hyperparameters, selects models, watches for overfitting: the decision-making dataset.
(2) Test Set: One-shot final evaluation of the frozen model: the unbiased estimate of real-world performance.
(3) Separation Why: Repeated validation peeking biases its score; only an untouched set can certify generalization.

Workflow of training set fitting candidates, validation set selecting the best, and test set used once for the final score

Figure 1: The role of each split: the training set fits many candidates; the validation set is queried repeatedly to tune and pick the winner (feedback loop); the test set is used exactly once, after everything is frozen; no arrow leads back from it.

Mathematical Formulation:
$\hat{\lambda} = \arg\min_{\lambda} \; \mathcal{L}_{\text{val}}\big(\hat{\theta}(\lambda)\big)$
$\hat{\theta}(\lambda) = \arg\min_{\theta} \; \mathcal{L}_{\text{train}}(\theta; \lambda)$
$\text{Final estimate:} \quad \mathcal{L}_{\text{test}}\big(\hat{\theta}(\hat{\lambda})\big) \quad \text{(computed once, model frozen)}$

Where:

$\theta$ are model parameters fitted on the training set; $\lambda$ the hyperparameters chosen on the validation set.
$\mathcal{L}_{\text{val}}$ is optimized indirectly through many modeling decisions, so it underestimates true error; $\mathcal{L}_{\text{test}}$ enters no optimization and stays unbiased.