ML0083 Weight Decay

Weight decay multiplies every weight by a factor slightly below one at each step, so parameters continuously shrink toward zero unless the data gradient pushes them back up. The equilibrium effect: only directions where the loss gradient persistently outweighs the decay survive with large magnitude, while noise-fitting directions get eroded, which is exactly the capacity control we want. For plain SGD, weight decay is mathematically identical to adding an L2 penalty to the loss. For adaptive optimizers like Adam it is not, because the L2 gradient gets rescaled per-parameter along with everything else; AdamW fixes this by decoupling the decay from the adaptive update, and is now the default recipe for training transformers.

(1) Mechanics: each step shrinks weights multiplicatively before (or alongside) the gradient step, so unneeded complexity decays exponentially while useful weights are constantly re-earned from the data gradient.
(2) SGD Equivalence and Its Caveat: for SGD, weight decay with rate lambda equals L2 regularization with coefficient lambda/alpha, so the two are coupled through the learning rate; retuning alpha silently retunes your regularization.
(3) AdamW Fix: Loshchilov and Hutter showed Adam’s adaptive scaling under-regularizes high-gradient weights when L2 is folded into the loss; AdamW applies decay outside the adaptive step, and PyTorch’s AdamW defaults to a weight decay of 0.01.

Figure 1: The geometric view: the unregularized optimum sits far from the origin; weight decay pulls the solution to where the smallest L2 ball first touches the loss contours, trading a little fit for a lot of magnitude.

The Adam subtlety is worth understanding precisely, because it is a favorite interview trap. In Adam, every component of the gradient, including the penalty’s contribution, is divided by the square root of its running second moment. Parameters with historically large data gradients therefore experience a heavily damped regularization force, which is backwards: the parameters the optimizer is already moving hardest are the ones the penalty reaches least. Note the denominator tracks the second moment of the gradient, not the weight’s own magnitude, so the effect is uneven regularization across parameters rather than a simple “big weights decay less” rule. AdamW removes the penalty from the adaptive gradient entirely and applies it as a direct multiplicative shrink, restoring uniform, controllable decay, which is why nearly every modern pretraining run (GPT-style models included) specifies AdamW.

Figure 2: The decoupling effect: with no decay the weight norm grows unchecked; Adam with L2 in the loss barely contains it because the penalty is adaptively rescaled; AdamW’s decoupled decay settles at a controlled plateau.

Mathematical Formulation:

Where:

• collects all parameters at step , is the learning rate, and is the per-step decay rate.
• is the minibatch loss; the factor is the multiplicative shrink applied every step.
• is the L2 coefficient that makes the two formulations equivalent for SGD, exposing the coupling: change the learning rate and the effective regularization changes too (this equivalence fails for adaptive optimizers, hence AdamW).