DL0020 CNN Parameter Sharing

Written by

How do Convolutional Neural Networks achieve parameter sharing? Why is it beneficial?

Answer

CNNs share parameters by applying the same convolutional filter at every spatial location: a small kernel of learnable weights slides across the input, and its weights are reused to compute each output activation. This weight reuse means the network learns location-independent features with far fewer parameters than a fully connected layer, which improves generalization and compute efficiency.

How Sharing Works:

(1) Convolutional Filter: A small matrix of learnable weights (e.g., $3 \times 3$ ) is defined per channel pair.
(2) Sliding Window: The filter moves across the whole input feature map, position by position.
(3) Weight Reuse: The identical weights are used at every position, so one filter detects its feature anywhere in the image.

Diagram of one shared 3x3 filter with the same nine weights applied at two different positions of an input grid producing two output activations.

Figure 1: The same 9 weights produce activations at position A and position B. No new parameters are added as the input grows.

Why It Is Beneficial:

(1) Reduced Parameters: A filter has $k \times k \times C_{in}$ weights regardless of input resolution, versus one weight per (input, output) pair in an FC layer.
(2) Translation Equivariance: A feature detected at one location is detected at any location; shifting the input shifts the output correspondingly.
(3) Improved Generalization: Fewer parameters means less overfitting, especially on limited data.
(4) Computational Efficiency: Fewer parameters mean fewer multiply-accumulates in both forward and backward passes, enabling deployment on resource-limited devices.

Four panels showing an input image with a square, its convolution output, a shifted input, and the identically shifted convolution output demonstrating translation equivariance.

Figure 2: Translation equivariance in action: shifting the input shifts the convolution output by the same amount, because the same shared filter processes every position.

How Big Is the Saving? Sharing decouples the parameter count from the input resolution: growing the image grows the compute, not the weights. The contrast with a fully connected layer on the same input is dramatic.

Log-scale bar chart comparing about 307 thousand parameters for a fully connected layer on a 32x32x3 input versus about 1.8 thousand for a shared 3x3 convolution.

Figure 3: Flattening a $32 \times 32 \times 3$ image into 100 FC units needs ~307k parameters; a shared $3 \times 3$ convolution with 64 filters needs ~1.8k, roughly 170x fewer.

Mathematical Formulation:
$y_{ijc} = \sum_{a=1}^{k}\sum_{b=1}^{k}\sum_{d=1}^{C_{in}} w_{abdc}\, x_{i+a,\, j+b,\, d} + b_c$
$P_{conv} = k^2 \cdot C_{in} \cdot C_{out} + C_{out}$

Where:

$y_{ijc}$ is the output activation at spatial position $(i, j)$ of channel $c$ ; the same $w$ is used for every $(i, j)$ : that reuse is parameter sharing.
$w_{abdc}$ is the filter weight at kernel offset $(a, b)$ connecting input channel $d$ to output channel $c$ ; $b_c$ is the per-output-channel bias.
$P_{conv}$ is the layer’s parameter count, independent of the input’s spatial size $H \times W$ .