DL0065 Group Normalization

Written by

What is Group Normalization?

Answer

Group Normalization normalizes each sample independently by dividing its channels into groups and computing a mean and variance over the channels and spatial positions within each group. The normalized activations then receive learned per-channel scale and shift parameters. Because its statistics do not depend on other examples, GroupNorm behaves consistently with very small or variable batch sizes and uses the same computation during training and inference. The group count must be compatible with the channel count, and its inductive bias can be less suitable than BatchNorm when large, stable batches provide useful batch statistics.

Group Normalization mechanism partitioning channels and normalizing each sample group.

Figure 1: A convolutional tensor is split into channel groups; each sample-group gets independent statistics before per-channel affine scaling.

(1) Per-Sample Statistics: Each example has its own group means and variances, so no running batch averages are required.
(2) Channel Grouping: For $C$ channels and $G$ groups, each group normalizes $(C/G)\times H\times W$ values; channels within a group share statistics.
(3) Boundary Cases: $G=1$ normalizes all channels and spatial positions together, while $G=C$ gives one channel per group and resembles InstanceNorm.

Comparison of Batch, Layer, Instance, and Group Normalization reduction dimensions.

Figure 2: Normalization families differ mainly in which batch, channel, and spatial axes share statistics and whether inference needs running estimates.

Mathematical Formulation:
$\mu_{n,g}=\frac{1}{m}\sum_{i\in S_{n,g}}x_i$
$\sigma^2_{n,g}=\frac{1}{m}\sum_{i\in S_{n,g}}(x_i-\mu_{n,g})^2$
$y_{n,c,h,w}=\gamma_c\frac{x_{n,c,h,w}-\mu_{n,g(c)}}{\sqrt{\sigma^2_{n,g(c)}+\epsilon}}+\beta_c$

Where:

$n$ indexes samples, $g$ indexes channel groups, and $c,h,w$ index channel and spatial position.
$S_{n,g}$ is the set of activations in group $g$ of sample $n$ ; $i$ indexes elements in that set.
$m=(C/G)HW$ is the number of normalized values when $C$ channels are divided into $G$ groups over height $H$ and width $W$ .
$\mu_{n,g}$ and $\sigma^2_{n,g}$ are the group mean and variance; $x$ and $y$ are input and normalized output activations.
$g(c)$ maps channel $c$ to its group, $\gamma_c,\beta_c$ are learned per-channel scale and shift, and $\epsilon$ stabilizes division.