How Derivatives Guide Gradient Descent

The objective is to find the minimum point of a function, particularly a cost function in machine learning, because this minimum represents the point of lowest error. The derivative, $f'(x)$, indicates the slope or the instantaneous rate of change of the function $f(x)$ at any given point $x$. These ideas are fundamental to understanding how the derivative acts as a guide for the gradient descent algorithm.

Imagine you're standing on the side of a hill represented by the graph of your cost function, and you want to get to the bottom (the minimum). It's foggy, so you can only feel the ground right where you are. How do you decide which way to step?

The derivative gives you exactly the information you need.

• If the derivative $f'(x)$ at your current position $x$ is positive, it means the slope is positive. The function is increasing at this point. To go downhill towards the minimum, you need to move in the direction where $x$ is smaller, meaning you need to move to the left (decrease $x$).
• If the derivative $f'(x)$ at your current position $x$ is negative, it means the slope is negative. The function is decreasing at this point. To continue going downhill towards the minimum, you need to move in the direction where $x$ is larger, meaning you need to move to the right (increase $x$).
• If the derivative $f'(x)$ is zero, the slope is flat. This suggests you might be at the bottom (a minimum), or potentially at a peak or a flat plateau. Gradient descent typically stops here.

Notice the pattern: if the derivative is positive, we decrease $x$; if the derivative is negative, we increase $x$. In both cases, we are essentially moving in the direction opposite to the sign of the derivative.

This is the core idea behind how gradient descent uses the derivative. It calculates the derivative at the current point and takes a small step in the opposite direction. The size of the step is also usually proportional to the magnitude of the derivative (steeper slope means a bigger step, gentle slope means a smaller step), adjusted by a factor called the learning rate (which we'll discuss soon).

Mathematically, a single step of gradient descent can be thought of like this:

$$x_{\text{new}} = x_{\text{current}} - \alpha \cdot f'(x_{\text{current}})$$

Here:

• $x_{\text{current}}$ is your current position.
• $f'(x_{\text{current}})$ is the derivative (slope) at your current position.
• $\alpha$ (alpha) is the learning rate, a small positive number that controls how big a step you take. We'll explore this parameter more in the section "The Learning Rate Parameter".
• $x_{\text{new}}$ is your updated position after taking the step.

The main part is the minus sign. It ensures you move against the gradient:

• If $f'(x_{\text{current}})$ is positive (slope points uphill to the right), subtracting $\alpha \cdot f'(x_{\text{current}})$ moves $x$ to the left (decreases $x$).
• If $f'(x_{\text{current}})$ is negative (slope points downhill to the right), subtracting $\alpha \cdot f'(x_{\text{current}})$ actually adds a small value (since subtracting a negative is adding), moving $x$ to the right (increases $x$).

In essence, the derivative acts like a compass pointing uphill (direction of steepest increase), and gradient descent simply takes a step in the opposite direction to go downhill. By repeating this process iteratively, calculating the derivative, and taking a small step in the opposite direction, gradient descent gradually walks down the slope of the cost function towards a minimum.

Let's visualize this. Consider the simple function $f(x) = x^2$. Its minimum is clearly at $x=0$. The derivative is $f'(x) = 2x$.

If we are at $x=2$, the function value is $f(2) = 2^2 = 4$. The derivative is $f'(2) = 2 \times 2 = 4$. Since the derivative is positive, it tells us the function is increasing here (the red dotted line shows the positive tangent slope). Gradient descent uses this positive derivative to decide to move $x$ to the left (decrease $x$), as indicated by the blue arrow, taking a step towards the minimum at $x=0$.

If we were instead at $x = -1.5$, the derivative would be $f'(-1.5) = 2 \times (-1.5) = -3$. A negative derivative tells gradient descent to increase $x$ (move right), again stepping towards the minimum.

This process repeats, with the derivative at each new point guiding the next step, until we ideally reach a point where the derivative is very close to zero, indicating we've found a minimum.