ML0023 Gradient Descent

Written by

What is Gradient Descent in machine learning?

Answer

Gradient descent is the iterative first-order optimization algorithm used to minimize a loss function by repeatedly stepping in the direction of steepest descent, opposite to the gradient. In each iteration the algorithm computes the gradient of the loss with respect to every parameter, then updates the parameters by subtracting a fraction of that gradient, where the fraction is the learning rate. The procedure repeats until the updates become negligibly small; for convex losses this converges to the global minimum, and for the non-convex losses of deep networks it reliably finds good local minima. In practice three variants trade gradient accuracy against computation: batch gradient descent uses the full dataset per step (stable but expensive), stochastic gradient descent uses a single sample (fast but noisy), and mini-batch gradient descent uses small subsets and is the default in modern training.

(1) Update Rule: $\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta)$ : move against the gradient, scaled by the learning rate.
(2) Learning Rate: Too large diverges or oscillates; too small converges slowly; it is the single most important hyperparameter.
(3) Variants: Batch (full data, stable), stochastic (one sample, noisy), mini-batch (compromise used in practice).

Gradient descent steps converging down a one-dimensional parabola toward the minimum

Figure 1: Gradient descent on a 1-D quadratic loss: each step moves opposite to the local slope, with step size proportional to the gradient: large steps far from the minimum, automatically shrinking steps near it.

Mathematical Formulation:
$\theta_{t+1} = \theta_t - \alpha \, \nabla_\theta J(\theta_t)$
$\nabla_\theta J(\theta) = \begin{bmatrix} \frac{\partial J}{\partial \theta_1} & \frac{\partial J}{\partial \theta_2} & \cdots & \frac{\partial J}{\partial \theta_p} \end{bmatrix}^{\top}$

Where:

$\theta$ is the parameter vector being optimized (e.g., all network weights and biases).
$\alpha$ is the learning rate, the step-size fraction applied to the gradient at each iteration.
$J(\theta)$ is the loss (cost) function, and $\nabla_\theta J$ its gradient: the vector of partial derivatives pointing in the direction of steepest ascent.