DL0129 SwiGLU, GELU, and ReLU FFNs

All three sit in the same place, namely the two-layer FFN that follows attention in every transformer block, but they differ in smoothness and in whether the nonlinearity is elementwise or multiplicative. ReLU is : cheapest, produces exact zeros (roughly 90-95% of FFN units are zero on a typical token), but has a kink at the origin and zero gradient for negative inputs, which gives dead units. GELU is : smooth, non-monotonic, keeps a small negative response, and was the default through BERT, GPT-2, and GPT-3. SwiGLU is not an elementwise function at all but a gated layer: the up-projection is split into a gate branch passed through SiLU () and a value branch left linear, and the two are multiplied elementwise. That third weight matrix is why practitioners shrink the hidden width by to stay parameter- and FLOP-neutral, and the payoff reported by Shazeer’s GLU-variants ablation is a consistent but modest gain, about 0.05 nats of pretraining log-perplexity on T5’s span-corruption objective at matched budget, which is enough that PaLM, Llama, Mistral, and Qwen2 all adopted it. The cost is one extra matmul, one extra saved activation for the backward pass, loss of exploitable sparsity, and a wider activation range that makes low-bit quantization harder.

(1) Two Matrices Versus Three: a dense FFN is then ; a gated FFN adds , so parameters go from to at the same width.
(2) The 2/3 Width Rule: to keep the block budget fixed you set instead of , which is exactly why Llama 2 7B uses 11008 with .
(3) Smoothness Changes Optimization: ReLU’s derivative is a step with no gradient for negative inputs, while GELU and SiLU are and pass a small negative signal, so units recover instead of dying.
(4) Gating Is A Multiplicative Interaction: the output is a product of two learned projections of the same input, giving a data-dependent, per-channel scaling that no elementwise function can express.
(5) Sparsity Is A ReLU-Only Asset: exact zeros let inference systems skip entire rows of ; GELU and SwiGLU produce values that are small but almost never exactly zero.
(6) SwiGLU Costs Memory Traffic And Range: three matmuls plus an extra stored tensor per layer, and the product of two unbounded projections widens the dynamic range that quantization must cover.

Figure 1: Shape (left) and derivative (right). ReLU’s derivative is a step function with a kink at the origin and no gradient for negative inputs; GELU and SiLU are smooth, non-monotonic, and dip below zero (SiLU’s minimum is at ), so a unit pushed into the negative region still receives gradient and can come back.

The historical progression is ReLU → GELU → gated GLU variants, and it is worth being honest that only the first step has a clean story. Replacing ReLU with GELU removes the dead-unit problem and gives a slightly better-conditioned loss surface. Replacing an elementwise nonlinearity with a gate is different in kind: the value branch stays linear, and the gate branch modulates it, so the layer computes a second-order interaction of the input with itself. Shazeer’s paper explicitly declines to explain why this helps, attributing the gain to luck rather than theory, and later analyses mostly note that gating gives the block a cheap multiplicative pathway similar in spirit to the gates in an LSTM. What made SwiGLU the production default is that the gain survives at scale, needs no extra hyperparameters, and is free once you apply the width rule. Llama 3 8B keeps the gated form but widens back out to with , a deliberate choice to spend more parameters in the FFN rather than a parameter-neutral swap.

Mathematical Formulation:

Where:

• is the post-norm hidden state for one token, and the FFN output has the same dimension so it can be added back through the residual connection.
• are up-projections and the down-projection; the dense block uses only, the gated block uses both (gate) and (value).
• is the standard normal CDF and the logistic sigmoid; GELU is often shipped as the tanh approximation , which is why two frameworks can disagree in the last few digits.
• is the elementwise product over the hidden channels, the operation that turns a gate into a multiplicative interaction rather than a pointwise map.
• is the FFN hidden width; the last line is the parameter-neutral setting, since , and real implementations round the result to a hardware-friendly multiple such as 256.

Figure 2: The structural difference is a branch, not a curve. A dense FFN applies one elementwise function between two matrices; a gated FFN splits the up-projection into a SiLU gate path and a linear value path whose elementwise product feeds . The third matrix is what forces the width if the block budget must stay fixed.