DL0019 Go Deep

Written by

How does increasing network depth impact the learning process?

Answer

Increasing network depth enhances feature learning and model capacity, but brings training instability, higher computational cost, and design complexity. Deeper stacks learn hierarchical representations (edges become textures, shapes, then objects) and can express certain complex functions far more efficiently than shallow networks, yet without modern techniques like residual connections and normalization, very deep networks are difficult or impossible to train.

Benefits of Depth:

(1) Improved Feature Hierarchy: Each layer composes the previous layer’s features, building increasingly abstract, high-level representations.
(2) Increased Model Capacity: More layers let the network model more complex functions and patterns.
(3) Exponential Efficiency for Complex Functions: Deep networks can represent some functions with vastly fewer units. For example, the $n$ -input parity function needs roughly $2^{n-1}$ neurons in a single hidden layer but only about $n$ small layers in a deep network.

Challenges of Depth:

(1) Vanishing/Exploding Gradients: Gradients shrink or grow multiplicatively through layers; without skip connections a 100-layer network may fail to train because gradients vanish before reaching early layers.
(2) Increased Computational Cost: Deeper networks need significantly more compute and training time.
(3) Higher Data Requirements: More parameters mean more overfitting risk unless the dataset grows correspondingly.

Curve comparison showing a deep network tracking a complex high-frequency target function closely while a shallow network with a single hidden layer smooths over fine structure.

Figure 1: A 6-layer network tracks the target’s high-frequency detail that a single-hidden-layer network smooths over. Depth buys compositional expressiveness.

Where Depth Hurts Training: Each layer multiplies gradients by its Jacobian, so gradient magnitude behaves like a product of many factors; it collapses exponentially when those factors are consistently small.

Log-scale chart of gradient norm per layer showing a plain deep network's gradients decaying exponentially into the vanishing zone while a residual network maintains healthy gradient magnitudes.

Figure 2: In a plain deep network gradient norms decay exponentially toward the vanishing floor; skip connections keep gradients healthy across all 50 layers.

Mathematical Formulation:
$\frac{\partial \mathcal{L}}{\partial x_1} = \prod_{l=1}^{L} J_l \cdot \nabla_{x_L}\mathcal{L}$
$\lVert g_1 \rVert \approx \lVert \nabla_{x_L}\mathcal{L} \rVert \cdot \prod_{l=1}^{L} \sigma_l$

Where:

$J_l$ is the Jacobian of layer $l$ ; gradients reaching the first layer are a product of $L$ Jacobians.
$\sigma_l$ is the dominant singular value of $J_l$ ; if most $\sigma_l$ are below 1, the product, and hence $\lVert g_1 \rVert$ , decays exponentially in $L$ .
Residual blocks replace each factor with $I + J_{F_l}$ , keeping the identity path alive so gradient norms stay near 1 even at 50+ layers.

DL0019 Go Deep

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation