Why is “weight initialization” important in deep neural networks?
Answer
Initialization decides whether signals survive a deep network. If weights are too small, activations and gradients shrink toward zero layer by layer (vanishing); too large, and they blow up (exploding) or saturate sigmoid/tanh into zero-gradient plateaus. Proper schemes like Xavier/Glorot and He scale the variance to the layer’s fan-in/fan-out so activations keep unit-scale statistics, while randomness breaks symmetry so neurons learn different features.
(1) Prevents Vanishing/Exploding Signals: Keeping activation variance constant across layers keeps gradients at a usable scale during backprop.
(2) Breaks Symmetry: Identical initial weights make neurons identical forever; random init gives each a distinct feature to learn.
(3) Matches the Activation: Xavier suits symmetric activations (tanh/sigmoid); He doubles the variance for ReLU, which zeroes half its inputs.
Mathematical Formulation:
Where:
and are the layer’s input and output unit counts (fan-in / fan-out).- Xavier balances both directions for symmetric activations; He drops and doubles variance because ReLU discards half the signal.
and 
are the layer’s input and output unit counts (fan-in / fan-out).
Figure 1: Post-ReLU activations by depth: plain random init collapses and Xavier fades, while He init keeps a healthy spread even at layer 6.
| Activation / Architecture | Recommended Init | Why |
|---|---|---|
| ReLU (CNNs, ResNet) | He / Kaiming | Doubles variance to compensate for ~50% zeros from ReLU |
| Tanh / Sigmoid (MLPs) | Xavier / Glorot | Balances fan-in and fan-out for symmetric activations |
| GELU (BERT-scale Transformers) | Truncated normal (σ ≈ 0.02) | LayerNorm + residuals already stabilize; gentle init suffices |
| Very deep LLMs (GPT, LLaMA) | Scaled normal (DeepNorm-style) | Residual-branch scaling stops signal growth across 100+ layers |
Table 1: Init choice is activation- and architecture-dependent: there is no single best scheme.
Leave a Reply