ML0006 Cross-Validation

Written by

What are the common cross-validation techniques?

Answer

Cross-validation is a statistical method used to evaluate the performance and generalizability of a model by rotating which part of the data serves as the validation set, so every sample is used for both training and validation. The most common technique is k-Fold Cross-Validation: the data is divided into $k$ equal folds, the model is trained $k$ times, each time on $k-1$ folds with the remaining fold held out, and the final score is the average over all $k$ runs. Leave-One-Out Cross-Validation (LOOCV) is the special case where $k$ equals the number of data points. Stratified k-Fold preserves the class distribution inside every fold, which matters for imbalanced datasets. Time Series Cross-Validation builds folds that respect temporal order, preventing future data from leaking into training.

(1) Core Idea: Rotate the validation fold and average the scores, giving a more reliable estimate than a single train/validation split.
(2) Choosing The Variant: $k=5$ or $10$ is the default, LOOCV suits tiny datasets, stratified folds suit class imbalance, and TSCV is mandatory for sequential data.
(3) Why It Matters: Every sample gets validated on, so the estimate has lower variance and uses all the data, at the cost of training $k$ times.

Four cross-validation schemes: k-fold, LOOCV, stratified k-fold, and time series CV

Figure 1: Four cross-validation schemes. Orange blocks are validation folds: they rotate (k-fold), shrink to one sample (LOOCV), keep class ratios (stratified), or move forward in time (TSCV).

Mathematical Formulation:
$\mathrm{CV}_k = \frac{1}{k}\sum_{i=1}^{k}\mathcal{E}_i$
$\mathcal{E}_i = \mathcal{L}\big(\hat{f}_{-i}, D_i\big)$

Where:

$\mathrm{CV}_k$ is the cross-validation score, the average error across all $k$ folds.
$\mathcal{E}_i$ is the validation error on fold $i$ , and $i\in\{1,\ldots,k\}$ indexes the folds.
$\hat{f}_{-i}$ is the model trained on all folds except fold $i$ , and $D_i$ is the held-out fold.
$\mathcal{L}$ is the evaluation loss or metric (e.g., error rate, log-loss).