ML0029 Tanh

The tanh (hyperbolic tangent) activation squashes its input into the range with a smooth S-shape. Its headline advantage over sigmoid is that its output is zero-centered: activations average near zero, so the gradients flowing into the next layer do not systematically push all weights in one direction, which makes optimization easier and convergence faster. It is also smooth and infinitely differentiable, and its maximum derivative of 1 (versus sigmoid’s 0.25) gives stronger gradients near the active region. The disadvantages are the flip side of saturation: for large positive or negative inputs the function flattens and its derivative approaches zero, so deep stacks of tanh units suffer the vanishing gradient problem; and like sigmoid it requires exponentials, making it computationally more expensive than ReLU. Because its output is not in , it is also unsuitable as a probability output; its modern niche is mainly inside RNN/LSTM cell dynamics rather than feedforward hidden layers.

(1) Zero-Centered: Output range centers activations, easing optimization versus sigmoid’s all-positive outputs.
(2) Stronger Gradient: Maximum derivative 1 at the origin (4× sigmoid’s), but it still saturates in the tails.
(3) Costs: Vanishing gradients in deep stacks, exponential computation, and no probability interpretation of the output.

Figure 1: Tanh and its derivative : the zero-centered S-curve has a healthy gradient (peak 1) only in the narrow central region; in both tails the derivative collapses toward zero.

Mathematical Formulation:

Where:

• is the neuron’s pre-activation; is the logistic sigmoid, showing tanh is just a rescaled sigmoid.
• is the derivative: maximal (1) at , decaying toward 0 as grows, the saturation that drives vanishing gradients.