ML0033 All Zeros Init

Written by

How does initializing all weights and biases to zero affect a neural network’s training?

Answer

Initializing every weight and bias to zero creates a fatal symmetry problem: all neurons in a layer compute the same output, receive the same gradient, and update identically, so they remain identical forever. The layer effectively behaves as a single neuron no matter how wide it is, the network cannot learn diverse features, and its representational capacity collapses. The pathology goes further: with zero weights, activations propagate as zero through the layers (for ReLU the forward signal is exactly 0), and gradients backpropagating through zero weights vanish entirely, so training can stall from the very first step. Proper initialization is therefore not a cosmetic choice: random schemes like Xavier/He break the symmetry and set the activation scale so signals flow through the full depth of the network.

(1) Symmetry: Identical weights → identical outputs → identical gradients → identical neurons forever; width is wasted.
(2) No Learning Signal: Zero weights kill forward activations and backward gradients (especially with ReLU), stalling training.
(3) Fix: Small random initialization (Xavier/He) breaks symmetry and keeps activation/gradient variance stable across layers.

Training and validation accuracy for zero initialization stuck at chance level versus random initialization learning successfully

Figure 1: Zero versus random initialization on a binary task: with all-zero weights, train and validation accuracy stay pinned at the 0.5 chance level: the network learns nothing; random initialization breaks symmetry and both curves climb above 0.9.

Mathematical Formulation:
$z_j = \sum_{i} w_{ji} x_i + b_j = 0 \quad \forall j \;\; (\text{all-zero init})$
$\frac{\partial L}{\partial w_{ji}} = \delta_j \, x_i$
$\delta_j = f'(z_j) \sum_k \delta_k w_{kj} = 0$

Where:

$w_{ji}$ is the weight from input $i$ to neuron $j$ , $b_j$ its bias, $z_j$ its pre-activation.
$\delta_j$ is the backpropagated error signal of neuron $j$ ; with downstream weights $w_{kj} = 0$ it collapses to 0, so every weight gradient is 0.
Even where a gradient exists (first layer before the collapse), every neuron in a layer gets the same gradient and stays identical: the symmetry itself is the deeper failure.