DL0002 All Ones Init

Initializing all weights to one (or any constant non-zero value) creates a symmetry problem: every neuron in the same layer receives identical gradients, so they learn the same features and cannot develop diverse representations. This limits the network to a single effective feature per layer, severely reducing its representational capacity. Training becomes slow or fails to converge because the optimizer cannot differentiate between neurons. Additionally, constant positive weights can push activations into saturated regions of sigmoid or tanh, worsening vanishing gradients.

(1) Symmetry Problem: All neurons in a layer receive the same input and the same gradient, so they evolve identically throughout training.
(2) Limited Representational Capacity: The network behaves as if each layer has only one neuron, making it impossible to capture complex, varied patterns in the data.
(3) Slow or Failed Convergence: Because neurons cannot differentiate, the loss landscape lacks the diversity needed for gradient descent to find useful minima.
(4) Activation Saturation: Uniform positive weights shift pre-activation values into the flat tails of sigmoid and tanh, producing vanishing gradients that stall learning.

Figure 1: The symmetry problem: with all-ones weights every neuron computes the same function and receives the same gradient, while random initialization lets each neuron learn a distinct feature.

Mathematical Formulation:

Where:

• is the weight from neuron in layer to neuron in layer .
• is the error signal (delta) for neuron in layer .
• is the activation of neuron in the previous layer.
• is the loss function; when all weights are equal, all delta signals are identical, so every weight updates by the same amount.

Figure 2: Training loss comparison: ones initialization converges slowly and plateaus at a high loss, while random initialization descends rapidly to a low minimum.