Optimization algorithms form the computational core of training machine learning models. Before addressing sophisticated methods, this chapter revisits the essential principles upon which they are built. We will start by reviewing standard first-order optimization algorithms like Stochastic Gradient Descent (SGD), Momentum, and Nesterov Accelerated Gradient (NAG), establishing a baseline for comparison.

Understanding the mathematical properties of the objective function is fundamental. We will examine the concept of convexity, where for a function $f$ and any $x, y$ in its domain and $\lambda \in [0, 1]$, the condition $f(\lambda x + (1-\lambda)y) \le \lambda f(x) + (1-\lambda)f(y)$ holds, and discuss why this property simplifies optimization. We will also explore the typical geometry of loss surfaces encountered in machine learning, especially in high-dimensional spaces.

Key metrics for evaluating optimizers will be introduced through the basics of convergence analysis, looking at different rates of convergence. We will identify common difficulties that arise, particularly in the non-convex optimization problems typical of deep learning, such as local minima and saddle points. Finally, practical considerations regarding numerical stability and the potential impact of floating-point arithmetic will be discussed. The chapter concludes with exercises focused on analyzing the behavior of these foundational algorithms.