RMSprop: Addressing AdaGrad's Limitations

AdaGrad introduces the powerful idea of adapting the learning rate for each parameter based on its gradient history. However, a primary drawback of AdaGrad is its persistent accumulation of squared gradients in the denominator. Over time, this sum grows continuously, causing the effective learning rate to shrink monotonically and eventually become infinitesimally small. This aggressive decay can prematurely stop the learning process, especially in non-convex optimization common in deep learning where progress might be needed even after many iterations.

RMSprop (Root Mean Square Propagation) was developed independently around the same time as AdaGrad, specifically to address this rapid decay issue. The central innovation in RMSprop is the use of an exponentially decaying moving average of the squared gradients, rather than summing up all past squared gradients. This approach gives more weight to recent gradient information and effectively "forgets" the distant past.

The RMSprop Update Mechanism

Instead of maintaining a sum $G_t$ of all past squared gradients for each parameter, RMSprop maintains an estimate of the mean square of the gradients, denoted as $E[g^2]_t$ . This is updated at each time step $t$ using a decay rate hyperparameter, $\beta$ (often between 0.9 and 0.99):

E[g^2]_t = \beta E[g^2]_{t-1} + (1 - \beta) g_t^2

Here:

$g_t$ is the gradient of the cost function with respect to the parameter at time step $t$ .
$E[g^2]_t$ is the moving average of the squared gradients at time step $t$ .
$\beta$ is the decay rate, controlling the memory of the moving average. A value closer to 1 means the past is remembered longer.

This calculation ensures that $E[g^2]_t$ is dominated by more recent squared gradients. If gradients have been large recently, $E[g^2]_t$ will be large, and if they have been small, it will decrease.

The parameter update rule for RMSprop then uses the square root of this moving average in the denominator, similar to AdaGrad, but crucially, using the adaptive $E[g^2]_t$ instead of the ever-growing sum:

\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_t

Where:

$\theta_t$ represents a model parameter at time step $t$ .
$\eta$ is the global learning rate.
$\epsilon$ is a small constant (e.g., $10^{-8}$ ) added for numerical stability to prevent division by zero.

Why RMSprop Works Better Than AdaGrad

By using the exponentially decaying average $E[g^2]_t$ , RMSprop prevents the denominator term $\sqrt{E[g^2]_t + \epsilon}$ from growing monotonically throughout training. It adapts to the recent magnitude of gradients for each parameter. If a parameter's gradients become small after initially being large, $E[g^2]_t$ will decrease over time, allowing the effective learning rate for that parameter to potentially increase again, unlike in AdaGrad where it would remain suppressed. This adaptability makes RMSprop significantly more effective for training deep neural networks over extended periods.

Consider how the denominators might evolve:

The AdaGrad denominator term tends to grow continuously as it sums all past squared gradients. The RMSprop denominator adapts based on a moving average, allowing it to decrease if recent gradients become smaller, preventing aggressive learning rate decay. (Note: This is an illustration assuming a gradient sequence that is large initially then becomes small).

Practical Usage

RMSprop is a widely used optimizer available in all major deep learning frameworks. When using it, you typically need to specify the learning rate $\eta$ and often the decay factor $\beta$ (commonly named alpha or rho in libraries, with a default around 0.9 or 0.99) and epsilon $\epsilon$ . While it generally requires less tuning of the learning rate compared to SGD, finding good values for $\eta$ and $\beta$ can still influence performance.

import torch
import torch.optim as optim
import torch.nn as nn

# Assume 'model' is your defined neural network (nn.Module)
# Example: model = nn.Linear(10, 2)

# Initialize RMSprop optimizer
# Common parameters: parameters to optimize, learning rate (lr), alpha (decay factor), epsilon (eps)
optimizer = optim.RMSprop(model.parameters(), lr=0.001, alpha=0.99, eps=1e-08)

# --- Inside your training loop ---
# loss.backward() # Compute gradients
# optimizer.step() # Update parameters using RMSprop
# optimizer.zero_grad() # Reset gradients for the next iteration
# --- End training loop ---

RMSprop provides an effective alternative to AdaGrad, laying the groundwork for further improvements seen in optimizers like Adam, which we will discuss next.

Was this section helpful?

References

Neural Networks for Machine Learning - Lecture 6a: RMSProp, Geoffrey Hinton, Tijmen Tieleman, 2012 Coursera: Neural Networks for Machine Learning (University of Toronto) - This lecture slide deck is the original, informal introduction of the RMSprop algorithm by its creator, Geoffrey Hinton.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Provides a thorough academic treatment of adaptive learning rate methods, including RMSprop, within the context of deep learning optimization.
An Overview of Gradient Descent Optimization Algorithms, Sebastian Ruder, 2016 arXiv preprint arXiv:1609.04747 DOI: 10.48550/arXiv.1609.04747 - A widely cited and accessible survey paper explaining various gradient descent optimization algorithms, including RMSprop, their formulations, and benefits.
torch.optim.RMSprop, PyTorch Documentation, 2024 PyTorch Documentation - Official documentation for using the RMSprop optimizer in PyTorch, covering its parameters and usage, which complements the practical usage section.