ML0027 Leaky ReLU

Leaky ReLU modifies standard ReLU by replacing the hard zero on the negative side with a small linear slope: negative inputs pass through scaled by a small constant (typically 0.01). This one change directly attacks ReLU’s main weakness: the dying ReLU problem. Because the negative region now carries a small but non-zero gradient, a neuron whose pre-activation goes negative for all inputs still receives learning signal and can be pulled back into the active regime, instead of being frozen at zero output forever. At the same time Leaky ReLU retains everything that made ReLU attractive: the positive side stays the identity with gradient 1 (no vanishing gradients), the computation is still a trivial piecewise-linear threshold, and the output remains unbounded above, preserving ReLU’s scale behavior. In practice the accuracy gain over ReLU is often modest, but it costs nothing and removes a permanent failure mode.

(1) Fixes Dying ReLU: Negative inputs get slope instead of 0, so gradient always flows and “dead” neurons can recover.
(2) Keeps ReLU’s Strengths: Identity on the positive side (gradient 1, no saturation) and near-identical computational cost.
(3) Costs: Introduces the hyperparameter (or learns it, as in PReLU); practical accuracy gains over ReLU are often small.

Figure 1: ReLU versus Leaky ReLU: identical on the positive side, but Leaky ReLU keeps a small slope in the negative region, enough gradient for a stuck neuron to recover instead of dying.

Mathematical Formulation:

Where:

• is the neuron’s pre-activation (weighted sum plus bias).
• is the negative-side slope, a fixed small constant (0.01 by default) or a learned parameter in PReLU.

Table 1: ReLU versus Leaky ReLU: the functions differ only in the negative region, but that small slope is what keeps neurons alive and gradients flowing.