ML0008 Learning Rate Selection

Written by

What are the best practices for selecting an optimal learning rate?

Answer

Selecting an appropriate learning rate is one of the most important choices in training a neural network: it largely determines how quickly and how well the model learns. Four practices cover most situations. First, grid or random search over a range (e.g., 0.0001, 0.001, 0.01) while watching training performance, to narrow down an effective value. Second, use adaptive optimizers such as Adam, RMSProp, or Adagrad, which adjust the effective rate per parameter from gradient history and need less manual tuning. Third, apply learning rate schedules (step decay, exponential decay, or cosine annealing) that shrink the rate as training approaches convergence. Fourth, monitor the training loss: if it stops decreasing or oscillates, adjust the rate. Too high a rate overshoots the optimum and oscillates or diverges; too low a rate converges slowly or stalls; the right rate converges efficiently to a good solution.

(1) Why It Matters: The learning rate scales every update step; it is the single hyperparameter that most often decides whether training works at all.
(2) Four Practices: Search a range, prefer adaptive optimizers, decay the rate over time, and watch the loss curve.
(3) Diagnose From The Curve: Oscillating or rising loss means too high; an almost flat, slowly creeping loss means too low.

Training loss for too low, good, and too high learning rates

Figure 1: Reading the loss curve: too low creeps down slowly, a good rate drops fast and plateaus, too high oscillates and can diverge.

Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta_t \, \nabla_\theta \mathcal{L}(\theta_t)$
$\eta_t = \eta_0 \, \gamma^{\lfloor t / s \rfloor} \quad \text{(step decay)}$

Where:

$\theta_t$ denotes the model parameters at step $t$ .
$\eta_t$ is the learning rate at step $t$ , now time-dependent because of the schedule.
$\eta_0$ is the initial learning rate, $\gamma \in (0,1)$ is the decay factor, and $s$ is the number of steps between decays.
$\nabla_\theta \mathcal{L}(\theta_t)$ is the gradient of the training loss $\mathcal{L}$ , scaled by $\eta_t$ at every update.