ML0066 Model Capacity

Without activation functions, a neural network, regardless of depth, collapses to a single affine transformation (strictly linear if biases are omitted), so the 2-layer and 20-layer networks have the same representational capacity provided no hidden layer is narrower than : both can only express affine mappings. Extra depth adds parameters but zero expressiveness: the collapsed map has at most effective degrees of freedom, and its rank is capped by the narrowest layer (the bottleneck) and by the input/output dimensions, never raised by depth or by parameter count. Depth alone provides no additional power to capture non-linear relationships; it changes only the optimization dynamics and implicit bias, never the representable function class.

(1) Layers Collapse: A composition of linear maps is itself linear: stacking 20 of them adds nothing a single layer cannot express.
(2) Only Bottlenecks Limit Capacity: Only a bottleneck layer (narrower than both input and output) lowers the achievable rank; otherwise a wide 2-layer and a wide 20-layer network express exactly the same affine family. Extra parameters from width or depth are redundant knobs, not more capacity.
(3) Neither Fits Non-Linear Data: Both are restricted to affine functions of the input (a hyperplane decision boundary for classification), so without non-linearities the extra depth buys no expressiveness for its compute.

Mathematical Formulation:

Where:

• is the effective weight matrix, the product of all layers’ matrices, itself just one matrix; the effective bias.
• and are the network’s input and output dimensions, while and are the input and output widths of layer ; the per-layer parameter count is weights plus biases.
• : even with every hidden layer arbitrarily wide, the rank is still capped at , and any hidden bottleneck only lowers it further, the only sense in which architecture limits capacity here.

Figure 1: The collapse in practice: on sine-wave data, a network without activations (blue) fits nothing but a straight line regardless of depth, while the same architecture with ReLU activations (orange) tracks the true curve (black dots) almost perfectly.