RMSprop: Addressing AdaGrad's Diminishing Rates

AdaGrad adapts the learning rate for each parameter by accumulating the squares of past gradients. While this approach helps address varying curvatures, it introduces a potential issue: the accumulated sum in the denominator, $\sum g_i^2$, grows monotonically. Over many iterations, especially in deep learning scenarios with non-convex objectives, this denominator can become very large, causing the effective learning rate to shrink towards zero prematurely. This can effectively halt the learning process long before convergence is reached.

RMSprop (Root Mean Square Propagation), an unpublished adaptive learning rate method proposed by Geoff Hinton, directly addresses this issue. Instead of letting the sum of squared gradients accumulate indefinitely, RMSprop uses an exponentially decaying average of squared gradients. This means older gradients gradually contribute less to the average, preventing the denominator from growing unboundedly large and stopping learning.

Let's look at the mechanism. At each time step $t$, RMSprop first computes the gradient $g_t = \nabla_{\theta} J(\theta_t)$ for the current parameters $\theta_t$. It then maintains a moving average $E[g^2]_t$ of the squared gradients, updated as follows:

$$E[g^2]_t = \gamma E[g^2]_{t-1} + (1 - \gamma) g_t^2$$

Here:

• $g_t^2$ denotes the element-wise square of the gradient vector $g_t$.
• $E[g^2]_t$ is the estimate of the moving average of squared gradients at time step $t$.
• $\gamma$ is the decay rate (or momentum term), a hyperparameter typically set to values like 0.9, 0.99, or 0.999. It controls how much weight is given to past squared gradients versus the current one. A higher $\gamma$ means the average incorporates a longer history of gradients.

Compare this to AdaGrad's denominator, which effectively has $\gamma = 1$ (no decay) and keeps adding the current squared gradient. By using $\gamma < 1$, RMSprop ensures that the influence of very old gradients diminishes over time.

The parameter update rule for RMSprop then uses this moving average:

$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_t$$

Where:

• $\eta$ is the global learning rate, another hyperparameter.
• $\epsilon$ is a small constant (e.g., $10^{-8}$) added for numerical stability to prevent division by zero, especially during the initial steps when $E[g^2]_t$ might be very small.
• The division and square root are performed element-wise.

The core idea is intuitive: if recent gradients for a parameter have been large, $E[g^2]_t$ will be large, and the effective learning rate for that parameter ($\eta / \sqrt{E[g^2]_t + \epsilon}$) will decrease. Conversely, if recent gradients have been small, the effective learning rate will increase. Crucially, because $E[g^2]_t$ is a moving average, it can also decrease if recent gradients become small again, allowing the learning rate to recover, unlike AdaGrad.

Hyperparameters

RMSprop introduces the decay rate $\gamma$ alongside the global learning rate $\eta$ and the stability constant $\epsilon$.

• Learning Rate ($\eta$): Still needs careful selection. Typical starting points might be $0.001$.
• Decay Rate ($\gamma$): Controls the memory of the squared gradient average. Common values are $0.9$ or $0.99$. A value closer to 1 means slower decay and considering a longer history.
• Epsilon ($\epsilon$): Usually set to a small value like $10^{-8}$ or $10^{-6}$ and generally doesn't require extensive tuning.

Advantages

• Addresses AdaGrad's vanishing rates: By using a moving average, RMSprop avoids the monotonically decreasing learning rates of AdaGrad, making it more suitable for longer training runs and non-convex problems.
• Good empirical performance: RMSprop often works well in practice, particularly for training recurrent neural networks, although it has been somewhat superseded by Adam (which we discuss next) as a general default.
• Hyperparameter tuning: Requires tuning $\eta$ and $\gamma$, though default values often provide a reasonable starting point.

In essence, RMSprop modifies AdaGrad's denominator to use a leaky average instead of a cumulative sum. This simple change prevents the aggressive learning rate decay that can stall AdaGrad, leading to a more effective and commonly used adaptive optimization algorithm. It forms a stepping stone towards Adam, which further refines this idea by incorporating momentum.