ML0024 Vanishing Gradient

The vanishing gradient problem occurs when gradients shrink exponentially as they are backpropagated from the output layer toward the early layers of a deep network, so early layers receive almost no learning signal and train extremely slowly. The root cause is the chain rule: the gradient at an early layer is a product of many per-layer factors (weight matrices and activation derivatives), and if those factors are consistently smaller than 1 in magnitude, the product collapses toward zero as depth grows. The classic driver is saturating activation functions (sigmoid’s derivative peaks at only 0.25 and tanh’s at 1, and both are near zero for most of their input range), compounding with poor weight initialization that pushes pre-activations into the saturated tails. Recurrent networks suffer the same effect across time steps.

(1) Saturating Activations: Sigmoid/tanh derivatives are small almost everywhere (sigmoid ); multiplying them across layers shrinks gradients exponentially.
(2) Depth / Chain Rule: The gradient at layer 1 is a product of per-layer factors; each factor below 1 makes the product vanish as grows.
(3) Poor Initialization: Too-large or too-small initial weights push activations into saturation, shrinking derivatives from the start.

Figure 1: Gradient magnitude at each layer during backpropagation (log scale): with sigmoid activations the signal decays roughly geometrically: after 20 layers the earliest layers receive gradients orders of magnitude smaller than the output layer.

Mathematical Formulation:

Where:

• is the pre-activation of layer , and the total number of layers.
• is the weight matrix of layer and the activation derivative, the two per-layer factors in the chain product.
• is the sigmoid; its derivative peaks at 0.25 near and approaches 0 in the saturated tails.

Figure 2: Why sigmoid kills gradients: outside the narrow active region around zero the derivative is essentially zero, so any neuron operating in the flat tails passes almost no gradient backward, and these small factors multiply down the chain.