ML0020 Data Split

Written by

How to split the dataset?

Answer

A dataset is typically split into three parts. The training set is used to fit the model: it learns the patterns and relationships from this data. The validation set is used during development to tune hyperparameters and compare model configurations, which prevents overfitting the training data. The test set is touched only once, for a final unbiased evaluation on completely unseen data: an estimate of real-world generalization. Typical ratios scale with dataset size: for small datasets (fewer than ~1,000 samples), 60–70% training / 10–15% validation / 15–25% test, with k-fold cross-validation strongly recommended because small validation estimates are noisy; for medium datasets (1,000–100,000), a common starting point is 70–80% / 10–15% / 10–15%; for large datasets (over ~100,000), even 98% / 1% / 1% leaves plenty of validation and test samples. For imbalanced data, use stratified splits so every part keeps the original class proportions.

(1) Three Roles: Train fits, validation tunes, test judges; each set answers a different question.
(2) Size-Dependent Ratios: Small data needs more training share plus cross-validation; big data can spare 1–2% for evaluation.
(3) Stratification: Preserve class ratios in every split when classes are imbalanced.

Train validation test split ratios for small, medium, and large datasets

Figure 1: Split ratios by dataset size: the smaller the data, the larger the training share (and the more you need cross-validation); big data can evaluate on 1–2%.

Mathematical Formulation:
$D = D_{\mathrm{train}} \cup D_{\mathrm{val}} \cup D_{\mathrm{test}}$
$D_{\mathrm{train}} \cap D_{\mathrm{val}} = D_{\mathrm{train}} \cap D_{\mathrm{test}} = D_{\mathrm{val}} \cap D_{\mathrm{test}} = \emptyset$

Where:

$D$ is the full dataset, partitioned into three disjoint subsets: no sample may appear in two roles.
$D_{\mathrm{train}}$ is the training set used to fit the model parameters.
$D_{\mathrm{val}}$ is the validation set used to tune hyperparameters and trigger early stopping.
$D_{\mathrm{test}}$ is the test set, used exactly once for the final unbiased estimate; reusing it for tuning silently turns it into validation data.