The Need for Adaptive Learning Rates

Optimizers such as Stochastic Gradient Descent (SGD), Momentum, and Nesterov Accelerated Gradient (NAG) are workhorses for training deep learning models. A common element among these methods is the use of a single, global learning rate, often denoted as $\alpha$. This learning rate determines the step size for updating all the model's parameters based on the computed gradients. While techniques like learning rate scheduling (which we'll cover later) can adjust this global rate over time, it remains the same for every parameter within a single update step.

This uniform approach, however, can be inefficient or even problematic when training complex deep neural networks. Why? Because the "ideal" step size might not be the same for all parameters. Let's consider a few scenarios where a one-size-fits-all learning rate falls short.

Challenges with a Single Learning Rate

1. Differing Feature Importance and Frequency: Imagine training a model on data where some input features are very common, while others appear rarely. The parameters associated with rare features might only receive informative gradients occasionally. A small global learning rate, suitable for the frequent features, could mean these parameters update incredibly slowly, hindering the model's ability to learn from those rare but potentially important signals. Conversely, a large learning rate might cause parameters linked to frequent features to overshoot their optimal values.
2. Varied Gradient Scales Across Layers: Deep networks often have very different gradient magnitudes in different layers. Gradients might be much larger in later layers compared to earlier ones (or vice-versa, depending on the architecture and activation functions). A learning rate that works well for one layer might be too large (causing divergence) or too small (causing slow learning) for another. Applying the same rate everywhere forces a compromise that might not be optimal for any part of the network.
3. Complex Error Shapes: The loss function of a deep network rarely forms a simple, symmetrical bowl shape. Often, it resembles a complex shape with long, narrow valleys (ravines), flat plateaus, and saddle points.

Consider a simple visualization of an elongated loss surface:

This contour plot shows a loss function where progress is much easier along the horizontal axis (w1) than the steep vertical axis (w2).

In such a scenario, the gradient will be much steeper along the $w_2$ direction than the $w_1$ direction. If we use a global learning rate $\alpha$:

• If $\alpha$ is large enough to make good progress in the shallow $w_1$ direction, it might be too large for the steep $w_2$ direction, causing the updates to oscillate wildly back and forth across the valley, potentially diverging.
• If $\alpha$ is small enough to avoid oscillations in the $w_2$ direction, progress along the $w_1$ direction will be painfully slow.

SGD with Momentum or NAG can help smooth out oscillations and accelerate progress along shallow directions, but they still operate with that single learning rate, limiting their effectiveness on highly non-spherical surfaces.

The Adaptive Approach

Ideally, we want an optimization algorithm that can adapt its step size for each parameter independently. It should:

• Take larger steps for parameters associated with gentle slopes or infrequent features.
• Take smaller steps for parameters associated with steep slopes or very frequent features.

This is precisely the motivation behind adaptive learning rate algorithms. Methods like AdaGrad, RMSprop, and Adam maintain information about the past gradients for each parameter and use this history to scale the learning rate individually. They effectively give each parameter its own learning rate that changes dynamically during training.

By adjusting the learning rate on a per-parameter basis, these algorithms can often navigate complex loss surfaces more effectively, leading to faster convergence and sometimes finding better solutions compared to standard SGD or its momentum variants, especially with default hyperparameter settings.

In the following sections, we will look into the specific mechanisms of AdaGrad, RMSprop, and Adam to understand how they achieve this adaptive behavior.