ML0034 Backpropagation

Written by

What is backpropagation?

Answer

Backpropagation (backward propagation of errors) is the algorithm by which neural networks learn: it efficiently computes how much every weight and bias contributed to the prediction error, so an optimizer can adjust each parameter in the direction that reduces the loss. At its core it is the chain rule of calculus applied systematically: after a forward pass computes the output and the loss, a backward pass starts at the output layer, computes the error signal $\delta = \partial L / \partial z$ there, and recursively propagates it backward: each layer’s error is a weighted sum of the next layer’s errors times the local activation derivative, and each weight’s gradient is simply its neuron’s error signal times the input it received. This reuses intermediate results so the cost of computing all gradients is roughly one extra forward pass, which is what makes training deep networks tractable. Gradients then feed an optimizer (SGD, Adam) that performs the actual update $w \leftarrow w - \eta \, \partial L / \partial w$ .

(1) Forward Pass: Inputs flow through the network producing a prediction; all intermediate activations are cached.
(2) Backward Pass: The loss error propagates backward via the chain rule: each layer’s $\delta$ is built from the next layer’s.
(3) Gradient & Update: $\partial L / \partial w_i = \delta \cdot x_i$ ; the optimizer subtracts a learning-rate-scaled step.

Three layer network with forward activations flowing left to right and gradient deltas flowing right to left

Figure 1: The two passes: activations flow forward (blue) through each layer to the loss; error signals $\delta$ flow backward (orange) along the same edges, and every weight’s gradient combines the forward activation with the backward error.

Mathematical Formulation:
$z = \sum_{i} w_i x_i + b$
$a = f(z)$
$\delta = \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a}\, f'(z)$
$\delta_j = \Big( \sum_{k} \delta_k \, w_{kj} \Big) f'(z_j)$
$\frac{\partial L}{\partial w_i} = \delta \cdot x_i$
$\frac{\partial L}{\partial b} = \delta$

Where:

$z$ is the pre-activation and $a = f(z)$ the activation of a neuron; $L$ the loss.
$\delta$ is the error signal: how much the loss changes per unit change of $z$ ; at hidden neuron $j$ it sums contributions $\delta_k w_{kj}$ from all downstream neurons $k$ .
$x_i$ is the input feeding weight $w_i$ : the gradient is proportional to it; the bias gradient equals $\delta$ itself.

Single neuron computation graph with numeric forward values and backward gradients

Figure 2: One numeric step through a neuron: forward values (blue) compute $z = wx + b$ and $a = \sigma(z)$ ; backward gradients (orange) multiply local derivatives down the chain: $\partial L/\partial w = 0.16 \times 0.5 = 0.08$ , so the weight updates as $w \leftarrow w - \eta \cdot 0.08$ .