What is learning rate warmup, and why does it help stabilize the early steps of deep network training?
Answer
Learning rate warmup starts training with a very small learning rate and increases it gradually (usually linearly) to the target peak over the first few hundred or thousand steps, after which the normal schedule (e.g., cosine decay) takes over. Its purpose is to stabilize early training: at initialization the model’s gradients are noisy and poorly conditioned, so immediately applying the full learning rate can cause loss spikes, divergence, or permanent damage to early layers.
(1) Stabilizes Early Updates: Random initializations produce unreliable gradient estimates; small early steps prevent destructive weight changes before the model finds its footing.
(2) Protects Adaptive Optimizers: With Adam, second-moment estimates are cold at step zero and can amplify the first updates; warmup bridges this biased-estimate phase.
(3) Enables Higher Peak Rates: Deep networks and Transformers tolerate (and benefit from) a higher peak learning rate once representations have settled, which warmup makes reachable without instability.
Figure 1: A typical schedule: linear warmup for 200 steps to the 0.10 peak, then cosine decay to zero, the standard recipe behind models like BERT and GPT.
Mathematical Formulation:
Where:
is the learning rate at step 
; 
is the target peak learning rate.
is the warmup length in steps (e.g., 200–10,000 depending on model and batch size).- is the total number of training steps; after warmup, the schedule shown is cosine decay.
![p \in [0, 1]](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2MCIgaGVpZ2h0PSIxNyIgdmlld0JveD0iMCAwIDYwIDE3Ij48cmVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBzdHlsZT0iZmlsbDojY2ZkNGRiO2ZpbGwtb3BhY2l0eTogMC4xOyIvPjwvc3ZnPg==)
is the post-warmup progress used by the third line, which applies once steps have elapsed: 
at the end of warmup and 
at the final step, so decays from the peak to zero.
is the learning rate at step 
; 
is the target peak learning rate.
is the warmup length in steps (e.g., 200–10,000 depending on model and batch size).
is the total number of training steps; after warmup, the schedule shown is cosine decay.![p \in [0, 1]](https://s0.wp.com/latex.php?latex=p+%5Cin+%5B0%2C+1%5D&bg=ffffff&fg=000&s=2&c=20201002)
is the post-warmup progress used by the third line, which applies once 
at the end of warmup and 
at the final step, so Why It Matters in Practice: Transformers are famously sensitive: without warmup their early loss can spike or diverge outright, while warmup yields a smooth descent; the effect is smaller but still useful in CNN training at very large batch sizes.
Figure 2: Without warmup the early loss can spike and diverge; warmup keeps the fragile first steps small until gradients become trustworthy.
Leave a Reply