DL0038 Transformer Activation

Transformers use activations in two places. Inside the FFN, the hidden layer applies ReLU (original paper) or, in most modern models, GELU. BERT, GPT, and ViT all standardized on GELU, and newer LLMs adopt gated variants like SwiGLU. Inside attention, softmax normalizes the score matrix into attention weights. The GELU/ReLU choice controls gradient health; softmax controls score interpretability.

(1) GELU (FFN, modern default): Smooth, probabilistic gating: small negative inputs survive with tiny gradients, avoiding dead neurons and smoothing optimization.
(2) ReLU (FFN, original): , cheap and effective, but its hard zero can kill neurons permanently.
(3) Softmax (attention): Converts raw scores into a normalized distribution over keys, giving every query a convex combination of value vectors.

Figure 1: ReLU clamps negatives to a hard zero; GELU curves smoothly through them: small negative activations keep a non-zero gradient.

Mathematical Formulation:

Where:

• is the standard Gaussian CDF: GELU gates the input by its own probability of being positive, a smooth stochastic-regularization view.
• is one raw attention score and the number of scored keys; softmax outputs sum to 1.

Why GELU Won: Its smoothness yields non-zero gradients for negative inputs, reducing the “Dying ReLU” failure; the data-dependent gating acts like a soft, learned threshold. Empirically this means more stable training and better final loss in large language and vision models, which is why BERT/GPT-era models abandoned ReLU in the FFN.

Figure 2: The two activation sites in every block: softmax in attention, GELU/ReLU between the FFN’s two linear layers.

Beyond GELU: Recent LLMs (PaLM, LLaMA) replace the plain activation with gated linear units (SwiGLU/GEGLU), where one projection’s activation multiplies another linear path elementwise; this gating adds quality per parameter and has become the default in state-of-the-art FFN design.