DL0019 Go Deep

Increasing network depth enhances feature learning and model power, but brings training instability, higher cost, and design complexity.

Increasing network depth can bring benefits:
(1) Improved Feature Hierarchy: Deeper layers can learn more abstract, high-level features. In image classification, early layers learn edges, deeper ones learn shapes and objects.
(2) Increased Model Capacity: More layers allow the network to model more complex functions and patterns.
(3) Improved Efficiency for Complex Functions: For certain complex functions, deep networks can represent them more efficiently with fewer neurons compared to shallow ones.

Increasing network depth can bring challenges:
(1) Vanishing/Exploding Gradients: Gradients can become extremely small or large as they propagate through many layers, hindering effective training, e.g., “Without techniques like skip connections, a 100-layer network might struggle to learn because gradients vanish before reaching early layers.“
(2) Increased Computational Cost (Challenge): Training deeper networks requires significantly more computational resources and time.
(3) Higher Data Requirements (Challenge): Deeper models have more parameters and are more prone to overfitting if not trained on large datasets.

The following example visually compares a shallow and a deep neural network on learning a complex function.