Why use non-linear activation functions in neural networks in machine learning, and what limitations would a network face if only linear activation functions were used?
Answer
Non-linear activations are what make a deep network more than a single linear map. Without them, depth is an illusion: composing two linear transformations yields another linear transformation, so a 100-layer network with linear activations collapses algebraically into an equivalent one-layer linear model: it can only ever learn linearly separable relationships, no matter how many parameters it has. Inserting a non-linearity (ReLU, sigmoid, tanh, GELU) after each layer breaks that collapse: the stack can now carve input space into complex regions, build hierarchical representations, and, by the universal approximation theorem, approximate any continuous function to arbitrary accuracy given enough units. The benefits therefore compound: non-linearity introduces the ability to model complex patterns, and it is what lets additional layers add genuine representational power rather than redundant reparameterization.
(1) Introduce Non-Linearity: Enable learning curved decision boundaries and complex input-output patterns.
(2) Universal Approximation: A network with non-linear activations can approximate any continuous function; a linear one cannot.
(3) Depth Matters: With only linear activations, any multilayer network equals a single linear map: layers add nothing.
Figure 1: The limitation in one picture: on V-shaped data, a network with only linear activations can only draw a straight line (red) and misses the structure entirely, while the same-depth network with ReLU activations tracks the true curve (green).
Mathematical Formulation:
Where:
are the weight matrix and bias of layer 
; 
its output.- The first line shows the collapse: two linear layers reduce to a single effective
, the core argument against linear-only depth. 
is the non-linear activation (ReLU, sigmoid, …); once it sits between layers, the composition no longer simplifies to one affine map.
are the weight matrix and bias of layer 
; 
its output.
, the core argument against linear-only depth.
is the non-linear activation (ReLU, sigmoid, …); once it sits between layers, the composition no longer simplifies to one affine map.
Figure 2: Training loss on the same task: the linear-only network plateaus at high error no matter how long it trains (the function class cannot fit the data), while the non-linear network keeps descending to a much lower loss.
Leave a Reply