ML0018 Data Normalization

Written by

Why is data normalization used in Machine Learning?

Answer

Data normalization scales features into a specific range or distribution: typically [0, 1] with min-max scaling, or mean 0 and standard deviation 1 with standardization, so that every feature contributes comparably to learning. It is used for three reasons. First, better model performance: algorithms built on distance metrics, such as k-nearest neighbors and SVMs, are skewed when one feature spans thousands and another spans decimals, normalization puts them on the same scale. Second, training stability and speed: normalized inputs keep neural network gradients well-conditioned, so training converges faster and more smoothly. Third, fair feature contribution: without scaling, large-magnitude features dominate the objective while small-magnitude ones are ignored, regardless of how informative they actually are.

(1) What It Does: Rescales each feature to a common range (min-max) or distribution (z-score standardization).
(2) Who Needs It: Distance-based models (kNN, SVM), gradient-based learners (neural networks, linear/logistic regression); tree ensembles largely do not.
(3) The Payoff: Faster convergence, stabler training, and features competing on information rather than magnitude.

Two-feature scatter before and after normalization

Figure 1: Before: one axis spans hundreds while the other spans decimals, so distance is decided by the big axis alone. After standardization, both features contribute equally.

Mathematical Formulation:
$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$
$z = \frac{x - \mu}{\sigma}$

Where:

$x'$ is the min-max normalized value of feature value $x$ , squashed into [0, 1].
$x_{\min}$ and $x_{\max}$ are the feature’s minimum and maximum computed on the training set only.
$z$ is the standardized (z-score) value with mean 0 and standard deviation 1.
$\mu$ and $\sigma$ are the feature’s training-set mean and standard deviation; fitting them on all data would leak test information.