DL0001 Residual Connection

A residual connection (skip connection) adds a block input to a learned residual transformation, so the block learns an update rather than an entirely new mapping. For a shape-preserving block, the local Jacobian becomes , which supplies a direct identity component and creates shorter paths through the computational graph. This usually improves gradient propagation and optimization, but it does not guarantee nonzero or constant gradients: products of residual-block Jacobians can still shrink, grow, or cancel. Residual connections also address the degradation problem, in which adding layers to a plain network can increase training error because the deeper model is difficult to optimize even though an identity extension exists. These properties enabled effective training of ResNet architectures with hundreds of layers.

Figure 1: A shape-preserving block uses an identity shortcut, while a block that changes resolution or channel width uses a learned projection so the two paths have compatible shapes.

(1) Shorter Gradient Paths: An identity shortcut changes the local block Jacobian from to . The identity component gives backpropagation additional routes, although the product across many blocks can still vanish or explode.
(2) Identity Mapping Fallback: When an additional shape-preserving block is not useful, its residual branch can approach zero, making easier to represent than in a plain nonlinear stack.
(3) Easier Deep Optimization: Residual parameterization helps deeper models avoid the degradation problem, where added layers increase training error because optimization fails to recover a useful identity extension.

Mathematical Formulation:

Where:

• is the input to residual block , and is its output.
• is the learned residual branch parameterized by weights ; it learns an update to the shortcut representation.
• is the residual-branch Jacobian, and is the identity operator with the same feature dimension as .
• is an identity operator when input and output shapes match; otherwise it can be a learned projection, such as a strided convolution, that aligns spatial and channel dimensions.
• Across residual blocks, backpropagation contains products of factors . These factors often improve conditioning relative to plain Jacobian products, but they do not impose a nonzero lower bound on gradient magnitude.

Figure 2: A plain stack multiplies full layer Jacobians, whereas a residual stack multiplies factors . The identity components provide additional gradient routes but do not guarantee stable magnitude.