AdaGrad Limitations: Diminishing Learning Rates

While AdaGrad introduces the valuable concept of per-parameter adaptive learning rates, its specific mechanism for accumulating gradient history leads to a significant practical issue. The AdaGrad update rule scales the effective learning rate for a parameter $\theta_i$ at timestep $t$ by the inverse square root of the sum of squared historical gradients for that parameter:

$$\theta_{t+1, i} = \theta_{t, i} - \frac{\eta}{\sqrt{G_{t, ii} + \epsilon}} g_{t, i}$$

Here, $g_{t, i}$ is the gradient of the loss with respect to parameter $\theta_i$ at timestep $t$, $\eta$ is the global learning rate, $\epsilon$ is a small constant for numerical stability, and $G_{t, ii}$ is the sum of the squares of the gradients with respect to $\theta_i$ from timestep 1 to $t$:

$$G_{t, ii} = \sum_{\tau=1}^{t} g_{\tau, i}^2$$

The core limitation stems from the accumulation term $G_{t, ii}$. Because $g_{\tau, i}^2$ is always non-negative (being a squared value), the sum $G_{t, ii}$ will monotonically increase throughout training, assuming the gradient is non-zero. It never decreases.

As training progresses, $G_{t, ii}$ continuously grows larger. Consequently, the denominator term $\sqrt{G_{t, ii} + \epsilon}$ also grows larger. This causes the effective learning rate, $\frac{\eta}{\sqrt{G_{t, ii} + \epsilon}}$, to shrink monotonically towards zero.

Example illustration of how the effective learning rate for two different parameters might decrease over training iterations using AdaGrad. Parameters experiencing larger gradients (Parameter 1) see a much faster decay compared to those with smaller gradients (Parameter 2), but both trend towards zero. Initial global learning rate $\eta=0.01$.

This overly aggressive decay can be problematic. In deep learning, where optimization landscapes are complex and non-convex, training often requires exploration even late in the process. If the learning rate becomes vanishingly small too early, the optimizer might effectively stop making progress long before reaching a satisfactory minimum. The model's ability to learn can be prematurely halted.

While AdaGrad was an important step towards adaptive learning rates, this limitation prompted the development of algorithms that could adapt learning rates without causing them to decay quite so aggressively. Methods like RMSprop, which we will explore next, modify the accumulation mechanism to prevent this unbounded growth.