DL0071 VGG vs GoogLeNet vs ResNet

The three architectures answer three different questions that the ImageNet era asked in sequence. VGG (2014) asked whether depth alone helps and showed that stacking small 3×3 convolutions to 16 or 19 weight layers beats wide shallow filters, but it paid for that with 138M parameters and 15.3 GFLOPs, roughly 90% of the parameters sitting in three fully connected layers. GoogLeNet/Inception (2014) asked how to get depth and multi-scale receptive fields cheaply: parallel 1×1, 3×3, 5×5, and pooling branches inside one Inception module, with 1×1 bottleneck convolutions cutting channel depth before the expensive kernels, plus global average pooling replacing the fully connected head, giving better accuracy than VGG with about 6.8M parameters and 1.5 GFLOPs. ResNet (2015) asked why depth stopped paying off past roughly 20 to 30 layers and identified the degradation problem: a 56-layer plain net reaches higher training error than a 20-layer one, so this is an optimization failure, not overfitting. Its fix, the identity shortcut , makes each block learn a residual and gives gradients a direct path, which made 152-layer and even 1000-layer networks trainable.

(1) VGG, Depth Through Uniformity: three stacked 3×3 layers match the 7×7 receptive field with fewer parameters and two extra nonlinearities, proving that a simple repeated motif scales; the cost is a parameter-heavy fully connected head and no answer to what happens beyond ~19 layers.
(2) Inception, Depth Under a Compute Budget: the module runs several kernel sizes in parallel so the network does not have to choose one scale per layer, and 1×1 reduce → 5×5 conv keeps the cost of the wide branches roughly an order of magnitude lower than the naive version.
(3) ResNet, Depth Without Degradation: residual blocks reframe the layer’s job as learning a correction to identity, so adding layers can never be worse than copying the input; combined with batch normalization and bottleneck blocks (1×1, 3×3, 1×1), it made very deep training routine.

Parameter Arithmetic:

Where:

• is the channel count held constant across the stack, so the first two lines compare three 3×3 layers against one 7×7 layer of identical receptive field: about 45% fewer weights and two extra ReLUs.
• The third line is the naive Inception 5×5 branch: 192 input channels, 32 output channels, spatial taps.
• The fourth line inserts a 1×1 reduce to 16 channels first, so the same branch costs 15,872 weights, roughly cheaper; this bottleneck is what let GoogLeNet run 22 layers at 1.5 GFLOPs.

Figure 1: Accuracy did not come from parameter count: GoogLeNet beats VGG-16 with about fewer parameters, and ResNet improves again at moderate size. Evaluation protocols differ slightly across the original papers, so read the trend rather than the decimals.

The residual idea is the one that generalized furthest. Writing a block as means the optimizer only has to push toward zero to recover the identity mapping, which is exactly the solution the deeper plain net failed to find. The gradient view is just as important: the identity term contributes an ungated path so the signal reaching early layers cannot vanish through repeated multiplication by small Jacobians. In practice the projection shortcut ( convolution with stride 2) handles the stage boundaries where spatial size and channel count change.

Residual Formulation:

Where:

• is the block input, its output, and the two or three stacked convolutions with weights .
• is the identity Jacobian of the shortcut; it keeps the gradient norm from collapsing even when is tiny, which is why depth beyond 100 layers becomes trainable.
• is the training loss; when shapes differ across a stage, is replaced by with a projection shortcut.

Figure 2: Schematic of the degradation problem: the deeper plain network settles at higher training error than the shallower one, ruling out overfitting; the residual version of the same depth trains to the lowest error.