ML0035 Model Comparison

Written by

How to compare different machine learning models?

Answer

Comparing machine learning models rigorously means more than reading one accuracy number off one test run. A sound comparison fixes the evaluation protocol first: choose metrics that match the task and its costs (accuracy or F1/ROC-AUC for classification, RMSE/MAE for regression), then evaluate every model on the same train/validation/test splits (or better, the same cross-validation folds), so differences are attributable to the models, not the data lottery. Because training has randomness (shuffles, weight initialization), each model should be run multiple times with different seeds and its mean and spread reported; when two models look close, a paired statistical test (e.g., a paired t-test or Wilcoxon test over per-fold scores) tells whether the gap is significant or noise. Finally, break ties and inform deployment with secondary criteria: training/inference cost, robustness to perturbations, and interpretability.

(1) Right Metrics: Task- and cost-appropriate metrics (F1/ROC-AUC, RMSE, …), never a single default number.
(2) Controlled Comparison: Identical splits/folds, multiple seeds, cross-validation; statistical tests for close calls.
(3) Secondary Criteria: Latency, memory, robustness, interpretability decide between statistical ties.

ROC curves comparing logistic regression and random forest with AUC values on the same test set

Figure 1: Comparing two classifiers on the same test set with ROC curves: the random forest’s curve dominates logistic regression’s at nearly every threshold (AUC 0.98 vs 0.90), a richer comparison than any single-threshold metric.

Mathematical Formulation:
$\bar{s}_m = \frac{1}{K} \sum_{k=1}^{K} s_{m,k}$
$\mathrm{Var}(s_m) = \frac{1}{K-1} \sum_{k=1}^{K} \big(s_{m,k} - \bar{s}_m\big)^2$
$t = \frac{\bar{d}}{\mathrm{std}(d) / \sqrt{K}}$
$d_k = s_{1,k} - s_{2,k} \quad \text{(paired test over folds)}$

Where:

$s_{m,k}$ is the score of model $m$ on fold (or seed) $k$ ; $\bar{s}_m$ and $\mathrm{Var}(s_m)$ summarize central performance and stability.
$d_k$ is the per-fold score difference between two models: pairing removes fold-to-fold variance, so the $t$ -statistic tests whether the mean gap is distinguishable from zero.