DL0072 Inception Module

The Inception module is the building block of GoogLeNet: instead of committing a layer to one kernel size, it runs several convolutional branches in parallel over the same input (1×1, 3×3, 5×5, plus a 3×3 max-pool path) and concatenates their outputs along the channel axis. Every branch uses stride 1 with padding chosen so all branches emit the same spatial size, which is what makes the concatenation legal. The naive form is prohibitively expensive because the 5×5 branch convolves over the full input depth, so the practical module inserts 1×1 convolutions as channel bottlenecks before the 3×3 and 5×5 convolutions and after the pooling path. The next layer therefore sees features at several receptive-field sizes at once, and the learned weights decide how much of each scale to use rather than the architect fixing one kernel per layer.

(1) Parallel Multi-Scale Branches: the 1×1, 3×3, 5×5, and pooling paths all read the identical input tensor, so one module covers several receptive fields; a face at 20 px and one at 60 px are both matched inside the same layer.
(2) Channel-Wise Concatenation: branches keep and identical (stride 1, pad 1 for 3×3, pad 2 for 5×5), and outputs are stacked on the channel dimension, so module output depth is the sum of branch depths, not their average.
(3) 1×1 Bottlenecks: convolution cost is linear in , so projecting 192 channels down to 16 before the 5×5 conv cuts that branch roughly 10x with a small accuracy cost.
(4) Dense Ops, Sparse Intent: the design approximates a sparse, locally-optimal connectivity structure using dense operations that BLAS and cuDNN execute efficiently, which is why GoogLeNet reached top ILSVRC-2014 accuracy at about 5M parameters, roughly 12x fewer than AlexNet.

Figure 1: Naive versus dimension-reduced Inception module. The 1×1 convolutions shrink channel depth before the expensive kernels and project the pooled path, so concatenation does not grow depth without bound.

The pooling branch is the reason the naive module is unstable when stacked: max pooling preserves its input depth, so every module would add at least the full input depth back into the concatenated output and channel count would grow monotonically with depth. The 1×1 projection after the pool fixes this, and the same trick applied before the 3×3 and 5×5 convolutions is where nearly all the savings come from. Take the classic Inception (3a) block with a input and a 5×5 branch producing 32 maps.

Mathematical Formulation:

Where:

• are the outputs of the 1×1, 3×3, 5×5, and pooling branches, and stacks them on the channel axis, which requires all four to share the same .
• counts multiply-adds for one convolution, with the output spatial size, and the input and output channels, and the kernel width.
• is the 5×5 branch applied to all 192 input channels; is the 1×1 projection to 16 channels and the 5×5 conv on those 16, summing to about , roughly a tenth of the naive branch.

Figure 2: Multiply-adds per branch for Inception (3a) on a input. The bottlenecks cut the 5×5 branch from about 120M to 12M and the module total from about 304M to 128M, while also shrinking output depth from 416 to 256 channels.