DL0058 Feature Pyramid Network

Written by

What is a pyramid network in the context of CNNs, and why is it useful for dense prediction?

Answer

A feature pyramid network (FPN) builds a hierarchy of semantically strong feature maps at multiple spatial resolutions. A bottom-up backbone produces progressively smaller maps with richer semantics; a top-down pathway upsamples deep features and merges them with same-resolution lateral projections from earlier stages. The resulting $P_2\text{--}P_5$ maps combine localization detail with high-level context. Detection, instance segmentation, and keypoint heads can then select a pyramid level appropriate to each object or region scale.

Feature Pyramid Network architecture with bottom-up, lateral, and top-down paths.

Figure 1: FPN construction showing tensor resolution, lateral 1×1 projection, top-down upsampling, fusion, and 3×3 smoothing.

(1) Bottom-up Hierarchy: Backbone stages $C_2\text{--}C_5$ reduce spatial resolution while increasing receptive field and semantic abstraction.
(2) Top-down Fusion: Nearest-neighbor upsampling and 1×1 lateral projections align channel widths before element-wise addition; a 3×3 convolution commonly smooths each merged map.
(3) Scale Assignment: Small objects use high-resolution levels such as $P_2$ or $P_3$ , while large objects use coarser levels such as $P_4$ or $P_5$ .

Feature pyramid scale-selection flowchart for small, medium, and large objects.

Figure 2: How object size directs prediction heads toward fine, medium, or coarse pyramid levels.

Mathematical Formulation:
$P_l=\mathrm{Conv}_{3\times3}\!\left(\mathrm{Conv}_{1\times1}(C_l)+\mathrm{Up}_2(P_{l+1})\right)$
$k=\left\lfloor k_0+\log_2\!\left(\frac{\sqrt{wh}}{224}\right)\right\rfloor$

Where:

$C_l$ is the bottom-up backbone feature and $P_l$ is the fused pyramid output at level $l$ .
$\mathrm{Conv}_{1\times1}$ aligns channel width, $\mathrm{Up}_2$ doubles spatial resolution, and $\mathrm{Conv}_{3\times3}$ smooths the merged feature.
$k$ is the assigned pyramid level, $k_0$ is the reference level, and $w,h$ are the region width and height.
$224$ is the canonical reference scale and $\lfloor\cdot\rfloor$ maps the continuous log-scale value to a discrete level.