DL0031 FFN in Transformer

The feed-forward network (FFN) inside each Transformer block processes every token independently after attention: it expands the token’s features into a higher-dimensional space, applies a non-linearity, and projects back to the model dimension. Attention mixes information across tokens; the FFN then deepens each token’s representation individually. It is where most of the block’s parameters and feature transformations actually happen.

(1) Non-Linear Transformation: Adds the only per-token non-linearity in the block, letting the model capture complex patterns that attention’s linear weighting cannot express.
(2) Token-Wise Processing: The same MLP is applied to each position separately: no cross-position mixing, so it parallelizes trivially over the sequence.
(3) Dimensional Expansion: The hidden layer typically expands by a factor of 4 (e.g., 512 → 2048), giving the network capacity to re-encode features before compressing them back.

Mathematical Formulation:

Where:

• is one token’s vector after the attention sub-layer; the same function is applied to every position.
• , are trainable weights; typically .
• is the ReLU activation of the original paper; modern models swap in GELU or gated variants like SwiGLU.

Figure 1: : expand, activate, compress. The same two-layer MLP for every token position.

Why the Expansion Matters: Projecting up to gives the network a wide workspace to recombine and re-weight the features attention produced; the second matrix then distills the result back to . Roughly two-thirds of a vanilla Transformer’s parameters live in these FFN matrices.

Complement to Attention: Multi-head attention is a weighted average of value vectors, a linear operation per head. Without the FFN, stacking attention layers would compose mostly linear maps; the FFN’s expansion + non-linearity supplies the representational depth that makes deep stacks worthwhile.