DL0045 Dimension in FFN

The FFN expands each token’s representation from to to create a wide nonlinear workspace: in the higher-dimensional space the activation function can carve out far richer feature combinations than the residual stream’s width allows. It then projects back down so the result matches the residual stream for the skip connection and the next sub-layer. It is a bottleneck design that buys expressiveness where it is cheap while keeping the inter-layer interface narrow.

(1) Extra Capacity: The wider hidden layer holds more features; the FFN is where most of a block’s parameters (, about two-thirds) live.
(2) Nonlinear Mixing: The activation (ReLU/GELU/SwiGLU) acts in the expanded space, letting the network represent sparse, high-order interactions that a -wide layer cannot.
(3) Projection Back: Returning to keeps residual compatibility and a uniform interface across all layers.

Mathematical Formulation:

Where:

• is the per-token input from the residual stream.
• expands the dimension, is the nonlinearity, and projects back down.

Figure 1: The FFN hourglass: expand , activate in the wide space, reduce to rejoin the residual stream.

Empirical Note: The 4x factor dates to the original Transformer and was kept by BERT/GPT because it reliably improves loss; modern SwiGLU models (LLaMA) use ~2.7x with three matrices so the parameter count stays comparable.