ML0026 ReLU

ReLU (Rectified Linear Unit) is the piecewise-linear activation : it passes positive inputs unchanged and blocks negative ones to zero. Its benefits made it the default hidden-layer activation of deep learning: in the positive region the gradient is a constant 1, which largely eliminates the vanishing gradient problem that saturating sigmoid/tanh units suffer; negative outputs produce sparse activations, so only a subset of neurons is active at any time, yielding efficient and often more robust representations; and the function is a trivial threshold operation, far cheaper to compute than exponentials. Its main limitation is the dying ReLU problem: a neuron that falls into a regime of consistently negative pre-activations outputs zero and has zero gradient, so it can never recover, permanently shrinking model capacity. ReLU is also unbounded on the positive side (large activations can destabilize training if unmanaged) and non-differentiable at zero, a theoretical wrinkle handled in practice by defining a subgradient of 0 there.

(1) Gradient Health: Derivative is 1 for : no saturation, strong gradient flow in deep networks.
(2) Efficiency & Sparsity: A single comparison computes it; zero outputs give sparse, efficient representations.
(3) Dying ReLU: Neurons stuck in the negative region output 0 with 0 gradient and may never recover; output is also unbounded above.

Figure 1: ReLU and its derivative: the identity ramp on the positive side keeps gradients at 1 (no vanishing), while the flat zero region on the left kills both signal and gradient, the source of the dying-ReLU failure mode.

Mathematical Formulation:

Where:

• is the neuron’s pre-activation (weighted sum plus bias).
• is the derivative used in backpropagation; at the function is non-differentiable and implementations assign subgradient 0.