DL0048 Adam Optimizer

Adam (Adaptive Moment Estimation) combines momentum and RMSprop: it keeps an exponentially decaying average of the gradient (first moment, the direction) and of the squared gradient (second moment, the scale), then divides the former by the square root of the latter. The result is a per-parameter adaptive learning rate: large steps for parameters with small, consistent gradients, small steps for noisy or steep ones, plus bias correction that fixes the zero-initialization of both averages in early steps.

(1) First Moment (Momentum): smooths the gradient direction over time.
(2) Second Moment (RMSprop): tracks the gradient magnitude for per-parameter scaling.
(3) Bias Correction + Update: , remove the zero-init bias before the normalized step.

Mathematical Formulation:

Where:

• are the model parameters at step , and is the gradient of the loss at that step.
• is the learning rate (default ); prevents division by zero.
• and control the decay of the first and second moment averages (paper defaults).

Figure 1: Adam on a quadratic bowl: adaptive per-parameter steps converge smoothly to the minimum without the zig-zag of plain SGD.

Intuition for the Division: is roughly a signal-to-noise ratio: parameters whose gradients are large but inconsistent (high ) get small updates, while parameters with small but persistent gradients get amplified. This is what makes Adam robust to ill-scaled features and sparse gradients.