ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced CNNs for Computer Vision Applications
Chapter 1: Revisiting CNN Foundations and Modern Architectures
Brief Review of CNN Building Blocks
Evolution of CNN Architectures: AlexNet to ResNet
Understanding Residual Connections and Skip Architectures
Inception Modules and Network-in-Network Concepts
DenseNet: Architecture and Connectivity Patterns
EfficientNet: Compound Scaling for Models
Architectural Design Choices and Trade-offs
Implementing Modern Architectures Practice
Chapter 2: Advanced Training and Optimization Techniques
Advanced Optimization Algorithms
Learning Rate Schedules and Cyclical Learning Rates
Regularization Revisited: Advanced Techniques
Batch Normalization Internals and Alternatives
Weight Initialization Strategies for Deep Networks
Gradient Clipping and Gradient Flow Mitigation
Mixed Precision Training Fundamentals
Debugging and Monitoring Deep CNN Training
Hands-on Practical: Implementing Advanced Training Loops
Chapter 3: Object Detection Algorithms
Two-Stage Detectors: R-CNN Family
Region Proposal Networks Explained
Single-Stage Detectors: YOLO Family
Single-Stage Detectors: SSD and RetinaNet
Anchor Boxes: Design and Refinement
Non-Maximum Suppression Variants
Evaluation Metrics for Object Detection
Implementing an Object Detector Practice
Chapter 4: Image Segmentation Techniques
Semantic Segmentation vs. Instance Segmentation
Fully Convolutional Networks for Segmentation
Encoder-Decoder Architectures: U-Net and SegNet
Dilated (Atrous) Convolutions for Segmentation
DeepLab Family: Atrous Spatial Pyramid Pooling
Instance Segmentation Approaches (Mask R-CNN)
Evaluation Metrics for Segmentation
Hands-on Practical: Building a Semantic Segmentation Model
Chapter 5: Attention Mechanisms and Transformers in Vision
Self-Attention Mechanisms in CNNs
Non-local Neural Networks
Introduction to Vision Transformers
ViT Architecture: Patches, Embeddings, Transformer Encoder
Hybrid CNN-Transformer Models
Comparing CNNs and Transformers for Vision Tasks
Implementing Attention Blocks in CNNs Practice
Chapter 6: Advanced Transfer Learning and Domain Adaptation
Revisiting Transfer Learning Strategies
Fine-tuning vs. Feature Extraction: Advanced Considerations
Adapting Models to Different Data Distributions
Domain Generalization Concepts
Few-Shot Learning with CNNs
Self-Supervised Learning Pre-training for Vision
Hands-on Practical: Fine-tuning Models on Specialized Datasets
Chapter 7: Generative Adversarial Networks for Image Synthesis
GAN Fundamentals Revisited
Challenges in Training GANs
Deep Convolutional GANs (DCGANs)
Conditional GANs for Controlled Generation
StyleGAN Architecture and Style-Based Generation
Evaluation Metrics for GANs
Implementing a DCGAN for Image Generation Practice
Chapter 8: Model Compression and Efficient Deep Learning
Motivation for Efficient Models
Network Pruning Techniques
Knowledge Distillation Methods
Quantization: Reducing Model Precision
Designing Efficient Architectures
Neural Architecture Search Overview
Hands-on Practical: Applying Pruning and Quantization

Learning Rate Schedules and Cyclical Learning Rates

Setting the learning rate is one of the most significant hyperparameter choices when training deep neural networks. Finding a single, optimal, fixed learning rate η\etaη that works well throughout the entire training process is often difficult. A learning rate that is too large can cause the optimization process to oscillate or even diverge, preventing convergence. Conversely, a learning rate that is too small can lead to excessively slow training or cause the optimizer to become permanently stuck in suboptimal local minima or saddle points.

Learning rate scheduling provides a strategy to adjust the learning rate dynamically during training. The general idea is to start with a relatively higher learning rate to make rapid progress initially and then gradually decrease it as training progresses. This allows the optimizer to settle into deeper, more optimal minima during the later stages of training.

Common Learning Rate Schedules

Several heuristics and predefined functions are commonly used to schedule the learning rate. Let's examine a few popular ones:

Step Decay

This is perhaps the simplest scheduling strategy. The learning rate is kept constant for a fixed number of epochs (or iterations) and then reduced by a specific factor.

For example, you might start with η=0.01\eta = 0.01η=0.01, train for 30 epochs, then reduce it to η=0.001\eta = 0.001η=0.001 for another 30 epochs, and finally reduce it to η=0.0001\eta = 0.0001η=0.0001 for the remainder of the training.

Mathematically, if η0\eta_0η0​ is the initial learning rate, γ\gammaγ is the decay factor (e.g., 0.1), and SSS is the step size in epochs, the learning rate ηe\eta_eηe​ at epoch eee can be defined as:

ηe=η0×γ⌊e/S⌋\eta_e = \eta_0 \times \gamma^{\lfloor e / S \rfloor}ηe​=η0​×γ⌊e/S⌋

While simple to implement, step decay requires careful manual tuning of the initial rate, the decay factor, and the step size, which can be sensitive to the dataset and model architecture.

Exponential Decay

Instead of discrete steps, exponential decay reduces the learning rate continuously after each epoch (or even each iteration). The learning rate ηe\eta_eηe​ at epoch eee is given by:

ηe=η0×e−k×e\eta_e = \eta_0 \times e^{-k \times e}ηe​=η0​×e−k×e

Here, η0\eta_0η0​ is the initial learning rate, and kkk is a decay rate hyperparameter. This provides a smoother decrease compared to step decay but still requires tuning η0\eta_0η0​ and kkk. A related approach uses a multiplicative factor γ<1\gamma < 1γ<1 applied every epoch: ηe=η0×γe\eta_e = \eta_0 \times \gamma^eηe​=η0​×γe.

Cosine Annealing

Cosine annealing offers a more sophisticated decay pattern. The learning rate decreases from an initial value ηmax\eta_{max}ηmax​ to a minimum value ηmin\eta_{min}ηmin​ (often 0) following a cosine curve over a specified number of epochs, TmaxT_{max}Tmax​. The learning rate ηt\eta_tηt​ at epoch ttt (where ttt ranges from 0 to TmaxT_{max}Tmax​) is calculated as:

ηt=ηmin+12(ηmax−ηmin)(1+cos⁡(tπTmax))\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{t \pi}{T_{max}}))ηt​=ηmin​+21​(ηmax​−ηmin​)(1+cos(Tmax​tπ​))

This schedule starts by decreasing the learning rate slowly, then accelerates the decrease in the middle, and slows down again towards the end. It's often used with "warm restarts," where the schedule is reset periodically (e.g., setting TmaxT_{max}Tmax​ to a fraction of the total epochs and repeating the cycle). This allows the optimizer to potentially escape poor local minima by temporarily increasing the learning rate again.

Cyclical Learning Rates (CLR)

An alternative approach, introduced by Leslie N. Smith, involves cyclically varying the learning rate between a lower bound (base_lr\text{base\_lr}base_lr) and an upper bound (max_lr\text{max\_lr}max_lr). Instead of monotonically decreasing the learning rate, CLR suggests that periodically increasing it can have benefits, such as helping the optimizer traverse saddle points more quickly and explore the loss more effectively.

How CLR Works

During training, the learning rate oscillates between base_lr\text{base\_lr}base_lr and max_lr\text{max\_lr}max_lr. Several functional forms can define this oscillation, with the "triangular" policy being common:

  1. A cycle consists of two steps: one where the learning rate increases linearly from base_lr\text{base\_lr}base_lr to max_lr\text{max\_lr}max_lr, and one where it decreases linearly back to base_lr\text{base\_lr}base_lr.
  2. Each step typically takes an equal number of iterations, defined by the stepsize hyperparameter. One full cycle takes 2×stepsize2 \times \text{stepsize}2×stepsize iterations.

Other cycle shapes exist, such as triangular2 (where the maximum learning rate is halved after each cycle) or using an exponential decay within the cycle.

Finding Optimal Bounds: The LR Range Test

A significant advantage of CLR is that it provides a systematic way to find reasonable values for base_lr\text{base\_lr}base_lr and max_lr\text{max\_lr}max_lr using an "LR Range Test." This involves running the model for a few epochs while linearly increasing the learning rate from a very small value to a large one and recording the loss at each step.

You typically plot the loss against the learning rate (on a log scale). The optimal base_lr\text{base\_lr}base_lr is often the value where the loss starts to decrease, and the optimal max_lr\text{max\_lr}max_lr is the value just before the loss starts to explode or oscillate wildly.

The "1cycle" Policy

A popular variant is the "1cycle" policy, also proposed by Smith. It uses a single cycle spanning the entire training duration (or slightly less).

  1. It increases the learning rate from base_lr\text{base\_lr}base_lr (often max_lr/10\text{max\_lr}/10max_lr/10) to max_lr\text{max\_lr}max_lr over the first phase (e.g., 45% of iterations).
  2. It decreases the learning rate from max_lr\text{max\_lr}max_lr back down to base_lr\text{base\_lr}base_lr over the second phase (e.g., another 45% of iterations).
  3. In the final phase (e.g., the last 10% of iterations), it continues to decay the learning rate well below base_lr\text{base\_lr}base_lr (sometimes by several orders of magnitude) to allow the model to settle deeply into a minimum.

This policy is often combined with a cyclical momentum schedule, where momentum decreases as the learning rate increases, and vice versa. The 1cycle policy has been shown to achieve good results quickly, sometimes achieving super-convergence (reaching high accuracy with fewer epochs than traditional schedules).

Visualization of Schedules

The following chart illustrates the behavior of Step Decay, Cosine Annealing, and a Triangular CLR policy over 100 epochs.

Different learning rate schedules exhibit distinct patterns over training epochs. Step decay shows abrupt changes, cosine annealing provides a smooth curve, and cyclical rates oscillate between bounds. (Note: Y-axis is log scale).

Practical Notes

  • Warmup: Especially when using large batch sizes or certain architectures like Transformers, starting training directly with a high learning rate can lead to instability. A common practice is to employ a "warmup" phase for the first few epochs or iterations, where the learning rate is gradually increased from a very small value (or zero) up to the target initial learning rate (η0\eta_0η0​ or max_lr\text{max\_lr}max_lr). This allows the model's parameters to adapt gently at the beginning.
  • Choice of Schedule: There's no single best schedule for all tasks. Cosine annealing and CLR (especially the 1cycle policy) are often effective starting points for many computer vision tasks. Step decay remains a viable option but might require more manual tuning. The optimal choice depends on the dataset complexity, model architecture, batch size, and total training budget (epochs).
  • Integration: Most deep learning frameworks (TensorFlow/Keras, PyTorch) provide built-in support for various learning rate schedulers, making them easy to integrate into your training loop alongside optimizers like Adam, AdamW, or SGD. You typically define the scheduler after defining the optimizer and call its step() method at the end of each epoch (or sometimes each iteration, depending on the scheduler).

Effectively scheduling the learning rate is a powerful technique for improving convergence speed and the final performance of your deep learning models. Experimenting with different schedules and their hyperparameters, guided by techniques like the LR Range Test for CLR, is an important part of the optimization process.

  • Cyclical Learning Rates for Training Neural Networks, Leslie N. Smith, 2017 Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE) DOI: 10.1109/WACV.2017.58 - Introduces the concept of Cyclical Learning Rates (CLR) and the LR Range Test for determining learning rate bounds.
  • SGDR: Stochastic Gradient Descent with Warm Restarts, Ilya Loshchilov, Frank Hutter, 2017 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1608.03983 - Proposes the cosine annealing learning rate schedule, often combined with warm restarts.
  • CS231n: Deep Learning for Computer Vision. Lecture Notes on Training Neural Networks, Part 2: Setting up the Learning Process, Stanford CS231n Course Staff, 2024 (Stanford University) - Covers various learning rate schedules and optimization techniques in deep learning contexts.

© 2026 ApX Machine Learning