Learning Rate Schedules and Cyclical Learning Rates

Setting the learning rate is one of the most significant hyperparameter choices when training deep neural networks. Finding a single, optimal, fixed learning rate $\eta$ that works well throughout the entire training process is often difficult. A learning rate that is too large can cause the optimization process to oscillate or even diverge, preventing convergence. Conversely, a learning rate that is too small can lead to excessively slow training or cause the optimizer to become permanently stuck in suboptimal local minima or saddle points.

Learning rate scheduling provides a strategy to adjust the learning rate dynamically during training. The general idea is to start with a relatively higher learning rate to make rapid progress initially and then gradually decrease it as training progresses. This allows the optimizer to settle into deeper, more optimal minima during the later stages of training.

Common Learning Rate Schedules

Several heuristics and predefined functions are commonly used to schedule the learning rate. Let's examine a few popular ones:

Step Decay

This is perhaps the simplest scheduling strategy. The learning rate is kept constant for a fixed number of epochs (or iterations) and then reduced by a specific factor.

For example, you might start with $\eta = 0.01$, train for 30 epochs, then reduce it to $\eta = 0.001$ for another 30 epochs, and finally reduce it to $\eta = 0.0001$ for the remainder of the training.

Mathematically, if $\eta_0$ is the initial learning rate, $\gamma$ is the decay factor (e.g., 0.1), and $S$ is the step size in epochs, the learning rate $\eta_e$ at epoch $e$ can be defined as:

$$\eta_e = \eta_0 \times \gamma^{\lfloor e / S \rfloor}$$

While simple to implement, step decay requires careful manual tuning of the initial rate, the decay factor, and the step size, which can be sensitive to the dataset and model architecture.

Exponential Decay

Instead of discrete steps, exponential decay reduces the learning rate continuously after each epoch (or even each iteration). The learning rate $\eta_e$ at epoch $e$ is given by:

$$\eta_e = \eta_0 \times e^{-k \times e}$$

Here, $\eta_0$ is the initial learning rate, and $k$ is a decay rate hyperparameter. This provides a smoother decrease compared to step decay but still requires tuning $\eta_0$ and $k$. A related approach uses a multiplicative factor $\gamma < 1$ applied every epoch: $\eta_e = \eta_0 \times \gamma^e$.

Cosine Annealing

Cosine annealing offers a more sophisticated decay pattern. The learning rate decreases from an initial value $\eta_{max}$ to a minimum value $\eta_{min}$ (often 0) following a cosine curve over a specified number of epochs, $T_{max}$. The learning rate $\eta_t$ at epoch $t$ (where $t$ ranges from 0 to $T_{max}$) is calculated as:

$$\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{t \pi}{T_{max}}))$$

This schedule starts by decreasing the learning rate slowly, then accelerates the decrease in the middle, and slows down again towards the end. It's often used with "warm restarts," where the schedule is reset periodically (e.g., setting $T_{max}$ to a fraction of the total epochs and repeating the cycle). This allows the optimizer to potentially escape poor local minima by temporarily increasing the learning rate again.

Cyclical Learning Rates (CLR)

An alternative approach, introduced by Leslie N. Smith, involves cyclically varying the learning rate between a lower bound ($\text{base\_lr}$) and an upper bound ($\text{max\_lr}$). Instead of monotonically decreasing the learning rate, CLR suggests that periodically increasing it can have benefits, such as helping the optimizer traverse saddle points more quickly and explore the loss more effectively.

How CLR Works

During training, the learning rate oscillates between $\text{base\_lr}$ and $\text{max\_lr}$. Several functional forms can define this oscillation, with the "triangular" policy being common:

1. A cycle consists of two steps: one where the learning rate increases linearly from $\text{base\_lr}$ to $\text{max\_lr}$, and one where it decreases linearly back to $\text{base\_lr}$.
2. Each step typically takes an equal number of iterations, defined by the stepsize hyperparameter. One full cycle takes $2 \times \text{stepsize}$ iterations.

Other cycle shapes exist, such as triangular2 (where the maximum learning rate is halved after each cycle) or using an exponential decay within the cycle.

Finding Optimal Bounds: The LR Range Test

A significant advantage of CLR is that it provides a systematic way to find reasonable values for $\text{base\_lr}$ and $\text{max\_lr}$ using an "LR Range Test." This involves running the model for a few epochs while linearly increasing the learning rate from a very small value to a large one and recording the loss at each step.

You typically plot the loss against the learning rate (on a log scale). The optimal $\text{base\_lr}$ is often the value where the loss starts to decrease, and the optimal $\text{max\_lr}$ is the value just before the loss starts to explode or oscillate wildly.

The "1cycle" Policy

A popular variant is the "1cycle" policy, also proposed by Smith. It uses a single cycle spanning the entire training duration (or slightly less).

1. It increases the learning rate from $\text{base\_lr}$ (often $\text{max\_lr}/10$) to $\text{max\_lr}$ over the first phase (e.g., 45% of iterations).
2. It decreases the learning rate from $\text{max\_lr}$ back down to $\text{base\_lr}$ over the second phase (e.g., another 45% of iterations).
3. In the final phase (e.g., the last 10% of iterations), it continues to decay the learning rate well below $\text{base\_lr}$ (sometimes by several orders of magnitude) to allow the model to settle deeply into a minimum.

This policy is often combined with a cyclical momentum schedule, where momentum decreases as the learning rate increases, and vice versa. The 1cycle policy has been shown to achieve good results quickly, sometimes achieving super-convergence (reaching high accuracy with fewer epochs than traditional schedules).

Visualization of Schedules

The following chart illustrates the behavior of Step Decay, Cosine Annealing, and a Triangular CLR policy over 100 epochs.

Different learning rate schedules exhibit distinct patterns over training epochs. Step decay shows abrupt changes, cosine annealing provides a smooth curve, and cyclical rates oscillate between bounds. (Note: Y-axis is log scale).

Practical Notes

• Warmup: Especially when using large batch sizes or certain architectures like Transformers, starting training directly with a high learning rate can lead to instability. A common practice is to employ a "warmup" phase for the first few epochs or iterations, where the learning rate is gradually increased from a very small value (or zero) up to the target initial learning rate ($\eta_0$ or $\text{max\_lr}$). This allows the model's parameters to adapt gently at the beginning.
• Choice of Schedule: There's no single best schedule for all tasks. Cosine annealing and CLR (especially the 1cycle policy) are often effective starting points for many computer vision tasks. Step decay remains a viable option but might require more manual tuning. The optimal choice depends on the dataset complexity, model architecture, batch size, and total training budget (epochs).
• Integration: Most deep learning frameworks (TensorFlow/Keras, PyTorch) provide built-in support for various learning rate schedulers, making them easy to integrate into your training loop alongside optimizers like Adam, AdamW, or SGD. You typically define the scheduler after defining the optimizer and call its step() method at the end of each epoch (or sometimes each iteration, depending on the scheduler).

Effectively scheduling the learning rate is a powerful technique for improving convergence speed and the final performance of your deep learning models. Experimenting with different schedules and their hyperparameters, guided by techniques like the LR Range Test for CLR, is an important part of the optimization process.