Having explored ways to control model complexity through regularization, we now turn our attention to the process of finding the optimal parameters for our deep learning models. This process, known as optimization, is central to training neural networks effectively. Standard gradient descent provides the theoretical foundation, but its application to large datasets presents practical challenges.

This chapter introduces the foundational optimization algorithms used in deep learning. We will begin by reviewing standard gradient descent and discussing its limitations. You will then learn about:

• Stochastic Gradient Descent (SGD): A computationally efficient approximation that updates parameters using single data points.
• Mini-batch Gradient Descent: A widely used approach that balances the benefits of SGD and batch gradient descent.
• Momentum: A technique to accelerate SGD's convergence, particularly in directions of consistent gradient, and dampen oscillations.
• Nesterov Accelerated Gradient (NAG): A refinement of Momentum that often leads to faster convergence.

By the end of this chapter, you will understand the mechanics behind these core algorithms, their respective advantages and disadvantages, and how they address the challenges of navigating complex loss surfaces during model training. We will also implement and compare these optimizers in practice.