While optimizers like Stochastic Gradient Descent (SGD) and its momentum variants offer significant improvements over basic gradient descent, they typically rely on a learning rate that applies uniformly to all parameters or decays according to a predefined schedule. However, different parameters in a deep network might benefit from different learning rate adjustments based on the history of their gradients.

This chapter introduces adaptive optimization algorithms designed to automatically adjust the learning rate for each parameter individually. We will examine several popular methods:

• AdaGrad: Learns by adapting the learning rate based on the historical sum of squared gradients for each parameter.
• RMSprop: Addresses AdaGrad's tendency for the learning rate to decrease too rapidly by using a moving average of squared gradients.
• Adam (Adaptive Moment Estimation): Combines the ideas of momentum and RMSprop, storing moving averages of both the gradient and its square.

You will learn the motivation behind adaptive methods, the specific update mechanisms of AdaGrad, RMSprop, and Adam, including their mathematical foundations and bias correction techniques. We will discuss their strengths, weaknesses, and implementation details within standard deep learning frameworks. Finally, we will provide practical considerations for choosing an appropriate optimizer for your models.