DL0069 RMSProp Adaptive Learning Rates

Written by

How does RMSProp adapt the learning rate per parameter?

Answer

RMSProp keeps a per-parameter exponential moving average of squared gradients and divides each gradient component by the square root of that average before stepping, so the effective learning rate shrinks for parameters with historically large gradients and grows for those with small or sparse ones. The running estimate $v_t$ acts as a per-coordinate normalizer: two parameters share one global learning rate yet move by very different amounts. On ill-conditioned surfaces, where curvature differs wildly across directions, this damps oscillation along steep directions while sustaining progress along shallow ones, something a single global learning rate cannot do. Tieleman and Hinton introduced the method in their 2012 Coursera lecture series to cope with non-stationary objectives such as mini-batch and recurrent training; Adam is essentially RMSProp plus a first-moment momentum term and bias correction.

(1) Per-Coordinate Normalization: each parameter steps by $\alpha\, g_t / (\sqrt{v_t} + \epsilon)$ ; a parameter whose gradients run 10x larger builds a 10x larger $\sqrt{v_t}$ , so its effective learning rate shrinks 10x and the two coordinates end up moving by comparable amounts instead of one dwarfing the other.
(2) Memory With a Window: the decay $\rho$ (0.9 in the original lecture; PyTorch’s RMSprop defaults to 0.99) makes $v_t$ an average over recent history, so the normalization adapts as the landscape changes instead of stalling like AdaGrad’s monotonically growing accumulator.
(3) No Bias Correction: $v_t$ starts at zero and is never corrected, so early steps are oversized; Adam fixes this with $\hat{v}_t = v_t / (1 - \rho^t)$ , one of the two additions (the other is momentum) that turn RMSProp into Adam.

Contour plot of the ravine f(x,y) = 0.05x^2 + 2y^2 with two 42-step trajectories from (-10, 4): the red SGD path zigzags across the steep y direction with decaying overshoots while creeping along x to about -1.5, and the blue RMSProp path settles into the valley without oscillating and travels along it to the minimum

Figure 1: On the ravine $f(x,y) = 0.05x^2 + 2y^2$ , SGD spends its budget oscillating across the steep $y$ direction and after 42 steps is still at $x \approx -1.5$ ; RMSProp normalizes both coordinates, never overshoots, and reaches $x \approx -0.3$ in the same 42 steps.

The same normalization explains RMSProp’s strength on sparse features. For an embedding row that receives a gradient only occasionally, $v_t$ decays toward zero between updates, so when a gradient finally arrives its effective learning rate $\alpha / \sqrt{v_t}$ is several times larger than a densely-updated parameter’s, and the rare signal is not drowned out by a global rate tuned for frequent gradients. The flip side appears at the start of every run: with $\rho = 0.99$ the first estimate is $v_1 = 0.01\, g_1^2$ , so the first step has magnitude $10\,\alpha$ whatever the gradient scale, a warmup-like quirk that Adam’s bias correction removes. In practice RMSProp remains a solid choice for RNNs and other non-stationary objectives, while vision recipes still often prefer well-tuned SGD with momentum for final generalization.

Two stacked panels over 60 training steps: top shows sqrt of v_t rising from 0.2 toward its asymptote of 2 for a parameter with large steady gradient while staying near 0.3 for a sparse-gradient parameter; bottom shows the effective learning rate alpha over sqrt(v_t) falling from 5 to 0.74 for the first parameter while the second stays near 2.9 in a sawtooth pattern

Figure 2: Two parameters over 60 steps with $\rho = 0.99$ : the one with a large steady gradient sees its effective learning rate $\alpha / \sqrt{v_t}$ fall 6.7x by step 60 (heading for $\alpha/2$ as $\sqrt{v_t} \to |g|$ ), while the sparse-gradient parameter still enjoys a roughly 4x larger rate.

Mathematical Formulation:
$v_t = \rho\, v_{t-1} + (1 - \rho)\, g_t^2$
$\theta_{t+1} = \theta_t - \frac{\alpha\, g_t}{\sqrt{v_t} + \epsilon}$

Where:

$g_t = \nabla f_t(\theta_t)$ is the minibatch gradient at step $t$ ; the squaring in the accumulator is element-wise.
$v_t$ is the per-parameter running average of squared gradients, with the same shape as $\theta$ , initialized to zero.
$\rho$ is the decay (0.9 in Hinton’s lecture, 0.99 by default in PyTorch), $\alpha$ the global learning rate, and $\epsilon \approx 10^{-8}$ a numerical floor guarding the division.
All operations are element-wise, so each parameter gets its own effective rate $\alpha / (\sqrt{v_t} + \epsilon)$ under one shared $\alpha$ .

Aspect	SGD + Momentum	RMSProp	Adam
State per Parameter	Velocity m (first moment)	Squared-gradient average v (second moment)	Both m and v
Normalization	None; one global rate for all coordinates	Gradient divided by sqrt(v) per coordinate	Bias-corrected m divided by sqrt(v-hat)
Early Steps	Stable from step one	Oversized; v underestimated with no correction	Bias correction keeps steps near alpha scale
Typical Strength	Final test accuracy on tuned vision recipes	RNNs and non-stationary objectives	Default for transformers and general use

Fit

DL0069 RMSProp Adaptive Learning Rates

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation