ML0025 Exploding Gradient

Written by

What are the typical reasons for exploding gradient?

Answer

Exploding gradients occur when gradients grow exponentially during backpropagation, producing huge weight updates that make training unstable: the loss oscillates, spikes, or diverges to NaN. The mechanism mirrors the vanishing problem: the gradient at an early layer is a chain-rule product of per-layer factors, and when those factors are consistently larger than 1 in magnitude (from poorly scaled weight initialization, deep unnormalized architectures, or activation regimes with derivatives above 1), the product blows up with depth. Recurrent networks are especially vulnerable because the same weight matrix is multiplied once per time step, so long sequences amplify the effect. A learning rate set too high then converts already-large gradients into catastrophic updates.

(1) Deep Chains of Large Factors: Products of weight matrices with spectral norm $> 1$ grow exponentially with depth (or RNN time steps).
(2) Bad Initialization: Weights initialized too large produce outsized activations and derivatives from the start.
(3) Compounding Learning Rate: A high learning rate turns large gradients into weight updates that overshoot and destabilize training.

Gradient norm exploding exponentially with depth and the same norm capped by gradient clipping

Figure 1: Gradient norm flowing backward through a deep network: without control it grows geometrically layer by layer (note the log scale); gradient clipping caps the norm at a fixed threshold, keeping updates bounded regardless of depth.

Mathematical Formulation:
$\frac{\partial L}{\partial z_l} = \frac{\partial L}{\partial z_n} \prod_{i=l}^{n-1} W_{i+1} \, f'(z_{i+1})$
$\left\| \prod_i W_i \right\| \sim \prod_i \|W_i\|$
$g \leftarrow g \cdot \min\!\left(1,\; \frac{\tau}{\|g\|}\right) \quad \text{(gradient clipping to threshold } \tau \text{)}$

Where:

$z_l$ is the pre-activation of layer $l$ ; $W_i$ and $f'(z_i)$ are the per-layer weight and activation-derivative factors.
$\|W_i\|$ is the operator (spectral) norm of the weight matrix; when the typical product exceeds 1, the gradient norm grows geometrically with depth.
$g$ is the full gradient vector and $\tau$ the clipping threshold: if $\|g\|$ exceeds $\tau$ , the gradient is rescaled down to norm $\tau$ without changing its direction.