Finding Minima and Maxima using Derivatives

The derivative, $f'(x)$, indicates the slope of a function $f(x)$ at any given point $x$. Consider what it means if the slope is zero. Visually, on a graph, a zero slope corresponds to a flat tangent line. Where do these usually occur? They often happen at the very bottom of a valley (a minimum) or the very peak of a hill (a maximum) on the function's graph. This understanding is fundamental to optimization.

Critical Points: Where the Action Happens

Points where the derivative $f'(x)$ is equal to zero, or where $f'(x)$ is undefined, are called critical points. These points are the primary candidates for local minima and maxima. A local minimum is a point lower than all its immediate neighbors, while a local maximum is a point higher than its immediate neighbors.

Why focus on $f'(x) = 0$? Because if a function is smooth and differentiable, the only way it can transition from increasing (positive slope) to decreasing (negative slope), forming a peak, or from decreasing to increasing, forming a valley, is by passing through a point where the slope is momentarily zero.

Consider the function $f(x) = x^2$. Its derivative is $f'(x) = 2x$. Setting $f'(x) = 0$ gives $2x = 0$, so $x = 0$ is the only critical point. We know intuitively that $x=0$ is the minimum of the parabola $y=x^2$.

The function $f(x) = x^2$ has a minimum at $x=0$. At this point, the tangent line is horizontal, indicating its slope, $f'(0)$, is 0.

Using the First Derivative to Classify Critical Points

Knowing a point is critical isn't enough. Is it a minimum, a maximum, or something else (like a flat spot in an otherwise increasing function, e.g., $f(x)=x^3$ at $x=0$)? The First Derivative Test helps us classify these points by looking at how the slope changes around the critical point.

The logic is straightforward:

1. Find the critical points by solving $f'(x) = 0$ (and noting where $f'(x)$ is undefined, although this is less common in typical ML cost functions).
2. Pick test values slightly to the left and right of each critical point $c$.
3. Evaluate the sign of the derivative $f'(x)$ at these test points:
   • If $f'(x)$ changes from negative (function was decreasing) to positive (function is now increasing) as we pass through $c$, then $f$ has a local minimum at $c$. Think of skiing down into a valley ($f'<0$) and then up the other side ($f'>0$).
   • If $f'(x)$ changes from positive (function was increasing) to negative (function is now decreasing) at $c$, then $f$ has a local maximum at $c$. Think of climbing up a hill ($f'>0$) and then down the other side ($f'<0$).
   • If $f'(x)$ does not change sign at $c$ (e.g., positive on both sides, or negative on both sides), then $c$ is neither a local minimum nor a local maximum. It might be an inflection point with a horizontal tangent.

Example: Let's analyze $f(x) = x^3 - 3x$. The derivative is $f'(x) = 3x^2 - 3$. Set $f'(x) = 0$: $3x^2 - 3 = 0$ $3(x^2 - 1) = 0$ $3(x - 1)(x + 1) = 0$ The critical points are $x = 1$ and $x = -1$.

Now, let's test the sign of $f'(x) = 3(x-1)(x+1)$ in the intervals defined by these points: $(-\infty, -1)$, $(-1, 1)$, and $(1, \infty)$.

• Choose $x = -2$ (in $(-\infty, -1)$): $f'(-2) = 3(-2-1)(-2+1) = 3(-3)(-1) = 9$. Since $f'(-2) > 0$, the function is increasing on $(-\infty, -1)$.
• Choose $x = 0$ (in $(-1, 1)$): $f'(0) = 3(0-1)(0+1) = 3(-1)(1) = -3$. Since $f'(0) < 0$, the function is decreasing on $(-1, 1)$.
• Choose $x = 2$ (in $(1, \infty)$): $f'(2) = 3(2-1)(2+1) = 3(1)(3) = 9$. Since $f'(2) > 0$, the function is increasing on $(1, \infty)$.

Analysis based on sign changes at the critical points:

• At $x = -1$, the derivative changes from positive to negative ($+ \rightarrow -$). This indicates a local maximum. The function value is $f(-1) = (-1)^3 - 3(-1) = -1 + 3 = 2$.
• At $x = 1$, the derivative changes from negative to positive ($- \rightarrow +$). This indicates a local minimum. The function value is $f(1) = (1)^3 - 3(1) = 1 - 3 = -2$.

The sign of the first derivative $f'(x)$ tells us if the function is increasing or decreasing. Changes in sign at critical points ($x=-1, x=1$) reveal local maxima and minima.

A More Direct Approach: The Second Derivative Test

Calculating derivatives can sometimes be easier than testing intervals around critical points. The Second Derivative Test provides an alternative way to classify critical points, leveraging the function's concavity at those points.

Recall that the second derivative, $f''(x)$, is the derivative of the first derivative $f'(x)$. It tells us the rate of change of the slope.

• If $f''(x) > 0$ at a point, the slope $f'(x)$ is increasing. This means the function's graph is bending upwards, like a bowl holding water. We say the function is concave up at that point.
• If $f''(x) < 0$ at a point, the slope $f'(x)$ is decreasing. The graph is bending downwards, like an upside-down bowl. We say the function is concave down at that point.

How does this relate to minima and maxima? Consider a critical point $c$ where the tangent line is horizontal, meaning $f'(c) = 0$.

1. Calculate the second derivative, $f''(x)$.
2. Evaluate $f''(c)$, the concavity at the critical point $c$.
   • If $f''(c) > 0$: The function has a horizontal tangent and is concave up. This geometry only fits a local minimum.
   • If $f''(c) < 0$: The function has a horizontal tangent and is concave down. This geometry only fits a local maximum.
   • If $f''(c) = 0$: The test provides no information. The point could be a local minimum, a local maximum, or an inflection point (a point where concavity changes). You would need to use the First Derivative Test in this situation.

Example: Let's revisit $f(x) = x^3 - 3x$. We previously found $f'(x) = 3x^2 - 3$ and critical points $x = 1, x = -1$. Now, find the second derivative by differentiating $f'(x)$: $f''(x) = \frac{d}{dx}(3x^2 - 3) = 6x$

Let's evaluate $f''(x)$ at the critical points:

• At $x = -1$: $f''(-1) = 6(-1) = -6$. Since $f''(-1)$ is negative, the function is concave down at $x=-1$. Therefore, $x = -1$ corresponds to a local maximum.
• At $x = 1$: $f''(1) = 6(1) = 6$. Since $f''(1)$ is positive, the function is concave up at $x=1$. Therefore, $x = 1$ corresponds to a local minimum.

These conclusions match those from the First Derivative Test, and in this case, the second derivative was very simple to compute and evaluate.

The sign of the second derivative $f''(x)$ indicates concavity. At critical points where $f'(x)=0$, positive concavity ($f''>0$) implies a local minimum, while negative concavity ($f''<0$) implies a local maximum. Where $f''(x)=0$ (at $x=0$ here), the concavity changes, defining an inflection point.

Optimization Connection

Why have we spent this time finding minima and maxima? Because this is the core mathematical idea behind training many machine learning models. Models learn by trying to minimize a cost function (also called a loss function or objective function). This function quantifies how well the model's predictions match the actual target values in the training data. A high cost means poor performance. A low cost means the model is doing well.

The process of training involves adjusting the model's internal parameters (like the slope and intercept in linear regression, or the weights and biases in neural networks) to find the set of parameter values that results in the minimum possible value of the cost function.

In simple cases, like some forms of linear regression, we might be able to find this minimum analytically by setting the derivative (or gradient, in higher dimensions) of the cost function to zero and solving, just like we found critical points here. However, for most complex models, especially in deep learning, the cost function is incredibly complicated. We can't just solve $f'(x)=0$. Instead, we use iterative algorithms like gradient descent (which we will cover in detail in Chapter 4). These algorithms use the derivative (gradient) at the current position to figure out which direction is "downhill" towards a minimum and take a small step in that direction, repeating the process many times.

Understanding how the first and second derivatives identify minima and maxima for single-variable functions is the essential building block for understanding these powerful optimization techniques used throughout machine learning. You now have the calculus tools to analyze the shape of simple functions and pinpoint their low points, directly mirroring the goal of minimizing a cost function. Next, we'll put this into practice by optimizing a basic cost function and using Python tools to help with the calculations.