ML0030 Sigmoid

The sigmoid activation squashes any real input into the range with a smooth S-curve. Its advantages: it is smooth and differentiable everywhere, and its bounded output has a natural probability interpretation, which keeps it the standard output activation for binary classification and for independent per-label probabilities in multi-label tasks (and for gates in LSTM/GRU cells, which need values in ). Its disadvantages drove it out of hidden layers: in the tails the function saturates and its derivative (at most 0.25, near zero for most inputs) makes stacked sigmoid layers a prime cause of the vanishing gradient problem; its outputs are not zero-centered, so downstream weight updates are biased in one direction and convergence slows; and the exponential computation is more expensive than ReLU’s simple threshold.

(1) Probability Output: Range , ideal for binary classification output layers and gating mechanisms.
(2) Vanishing Gradient: Derivative peaks at 0.25 and saturates in the tails; deep stacks lose gradient fast.
(3) Not Zero-Centered: Always-positive outputs bias weight updates and slow convergence; also costlier to compute than ReLU.

Figure 1: Sigmoid and its derivative : a useful probability-shaped output, but the derivative only reaches 0.25 at best and is near zero in both tails: the mechanism behind vanishing gradients.

Mathematical Formulation:

Where:

• is the neuron’s pre-activation (weighted sum plus bias).
• is the derivative, computed in one line from the output itself, maximal (0.25) at , approaching 0 as grows.