Standard gradient descent methods often rely on a fixed learning rate, $\eta$. Finding an effective $\eta$ requires careful tuning; a rate too small slows convergence, while one too large can hinder it. This chapter introduces algorithms that address this by automatically adjusting the learning rate during training.

We will examine several widely used adaptive learning rate methods. You will learn the mechanisms behind AdaGrad, which scales learning rates based on past gradients, and RMSprop, which modifies AdaGrad's approach to prevent excessive decay. We will then study Adam (Adaptive Moment Estimation), an optimizer combining adaptive rates with momentum estimation. We will also analyze variants such as Adamax, Nadam, and AMSGrad, understanding their specific modifications. The chapter also covers integrating adaptive methods with learning rate schedules.

By the end of this chapter, you will understand the theory and practical application of these adaptive techniques, allowing you to choose and utilize them for more efficient model training.