DL0007 Batch Norm

Batch normalization stabilizes and accelerates training by normalizing each layer’s inputs across the mini-batch: subtract the batch mean and divide by the batch standard deviation. After normalization, a learnable scale and shift let the network recover the identity transformation if needed. BN is applied after the linear transform (e.g., the convolution) and before the activation (e.g., ReLU).

(1) Stabilizes Learning: Reduces internal covariate shift, making training less sensitive to initialization and hyperparameter choices.
(2) Enables Higher Learning Rates: Larger learning rates can be used without instability, leading to faster convergence.
(3) Improves Generalization: Normalizing per mini-batch introduces noise into activations, preventing over-reliance on specific batches and acting as a mild regularizer.

Figure 1: Each feature channel is normalized using its mini-batch statistics, then rescaled by learnable and .

Mathematical Formulation:

Where:

• is an individual feature value in the batch.
• and are the mean and variance of that feature across the current mini-batch.
• is a small constant (e.g., ) added for numerical stability.
• and are learnable scaling and shifting parameters.

Figure 2: Training computes per-batch statistics and updates running averages; inference uses the frozen running averages, producing deterministic outputs.

Placement in a CNN Block: BN sits between the linear transform and the non-linearity: Conv2D → BatchNorm → ReLU.

Figure 3: In a standard CNN block, BatchNorm normalizes the convolution output before the activation function.