Multivariable Optimization Concepts

Single-variable calculus involves identifying critical points where the derivative $f'(x) = 0$ to find potential minima and maxima of a function $f(x)$. At these points, the tangent line is horizontal, indicating that the function is momentarily flat. The second derivative, $f''(x)$, helps classify these points: $f''(x) > 0$ suggests a local minimum (concave up), while $f''(x) < 0$ suggests a local maximum (concave down).

We can extend this idea to functions of multiple variables, like $f(x, y)$ or more generally $f(\mathbf{x})$ where $\mathbf{x}$ is a vector of variables $(x_1, x_2, ..., x_n)$.

Finding Critical Points with the Gradient

For a multivariable function, the analogue of the derivative being zero is the gradient vector being the zero vector. A critical point (or stationary point) $\mathbf{x}_0$ of a function $f(\mathbf{x})$ is a point where the gradient is zero:

$$\nabla f(\mathbf{x}_0) = \mathbf{0}$$

Remember that the gradient $\nabla f$ is a vector containing all the partial derivatives: $\nabla f = \left( \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, ..., \frac{\partial f}{\partial x_n} \right)$. So, the condition $\nabla f(\mathbf{x}_0) = \mathbf{0}$ means that all partial derivatives must be simultaneously zero at $\mathbf{x}_0$:

$$\frac{\partial f}{\partial x_1}(\mathbf{x}_0) = 0, \quad \frac{\partial f}{\partial x_2}(\mathbf{x}_0) = 0, \quad ..., \quad \frac{\partial f}{\partial x_n}(\mathbf{x}_0) = 0$$

What does this mean intuitively? The gradient points in the direction of the steepest ascent. If the gradient is the zero vector, it means there is no direction of ascent (or descent) from that point. The function is locally "flat" in all directions parallel to the input axes. Just like in the single-variable case, these critical points are candidates for being local minima, local maxima, or something else.

Classifying Critical Points: The Role of the Hessian

In the single-variable case, a zero derivative alone wasn't enough; we needed the second derivative test. For multivariable functions, the Hessian matrix, $\mathbf{H}$, which contains all the second-order partial derivatives, plays the role of the second derivative.

$$\mathbf{H} = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{pmatrix}$$

Evaluating the Hessian matrix at a critical point $\mathbf{x}_0$ (where $\nabla f(\mathbf{x}_0) = \mathbf{0}$) helps us understand the function's curvature around that point. The Second Derivative Test for multivariable functions uses properties of the Hessian matrix $\mathbf{H}(\mathbf{x}_0)$:

1. Local Minimum: If $\mathbf{H}(\mathbf{x}_0)$ is positive definite, then $f$ has a local minimum at $\mathbf{x}_0$. Intuitively, this means the function is "concave up" in all directions radiating from $\mathbf{x}_0$. (A matrix is positive definite if $\mathbf{v}^T \mathbf{H} \mathbf{v} > 0$ for all non-zero vectors $\mathbf{v}$. This often corresponds to all its eigenvalues being positive).
2. Local Maximum: If $\mathbf{H}(\mathbf{x}_0)$ is negative definite, then $f$ has a local maximum at $\mathbf{x}_0$. This means the function is "concave down" in all directions. (Negative definite often corresponds to all eigenvalues being negative).
3. Saddle Point: If $\mathbf{H}(\mathbf{x}_0)$ is indefinite (meaning it's neither positive nor negative definite, typically having both positive and negative eigenvalues), then $f$ has a saddle point at $\mathbf{x}_0$. This is a point where the function curves up in some directions and down in others, like a horse saddle. See the visualization below.
4. Inconclusive: If the Hessian matrix test is inconclusive (e.g., if it's positive or negative semi-definite, meaning some eigenvalues might be zero), other methods are needed.

For most optimization problems in machine learning, we are primarily concerned with finding local (or ideally global) minima. Saddle points represent a challenge for some optimization algorithms because the gradient is zero, potentially halting progress, even though it's not a minimum.

At the origin (0,0) of the function $f(x, y) = x^2 - y^2$, the gradient $\nabla f = (2x, -2y)$ is $(0, 0)$. The Hessian is $\begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix}$, which is indefinite. This indicates a saddle point: the function increases along the x-axis but decreases along the y-axis.

Relevance to Machine Learning

In machine learning, the function $f$ we want to optimize is typically a cost function or loss function, which measures how poorly our model performs. The variables $\mathbf{x}$ are the model's parameters (weights and biases). Our goal is almost always to minimize this cost function.

Therefore, finding points where the gradient of the cost function with respect to the parameters is zero is a fundamental step. While the second derivative test using the Hessian provides a way to classify these points, calculating and analyzing the Hessian can be computationally expensive, especially for models with millions of parameters (like deep neural networks).

Furthermore, finding the points where $\nabla f = \mathbf{0}$ analytically by solving the system of partial derivative equations is usually intractable for complex ML models. This motivates the use of iterative optimization algorithms, like gradient descent (which we will explore in the next chapter), that use the gradient $\nabla f$ to navigate the cost surface and progressively move towards a minimum, without necessarily needing to compute the Hessian or explicitly solve $\nabla f = \mathbf{0}$. Understanding the concepts of gradients, critical points, and curvature, however, remains essential for understanding how and why these algorithms work, and what challenges (like saddle points) they might encounter.