Visualizing Gradient Descent

Visualizing gradient descent clarifies how this optimization algorithm operates. By taking steps based on the derivative, we can efficiently locate the minimum point of a function.

Imagine our cost function is like a valley or a bowl shape. Our goal is to find the very bottom of this valley. Think of yourself standing somewhere on the slope of this valley. How do you get to the bottom? You'd look at the ground beneath your feet to see which way is downhill and take a step in that direction.

The derivative, $f'(x)$, gives us exactly this information: the slope of the function at our current position, $x$.

• If the slope $f'(x)$ is negative, the function is decreasing at that point. To go further downhill, we need to move in the positive $x$ direction (increase $x$).
• If the slope $f'(x)$ is positive, the function is increasing. To go downhill, we need to move in the negative $x$ direction (decrease $x$).
• If the slope $f'(x)$ is zero, we are at a flat point, potentially the minimum (or maximum, or a saddle point, but let's focus on minima for now).

Gradient descent uses this slope information to take iterative steps. At each step, it calculates the derivative at the current point $x_{old}$ and updates the position using the rule:

$$x_{new} = x_{old} - \eta \times f'(x_{old})$$

Here, $\eta$ (eta) is the learning rate, a small positive number we discussed earlier that controls how big each step is. Notice the minus sign:

• If $f'(x_{old})$ is positive (slope is uphill), we subtract a positive value, moving $x$ to the left (decreasing $x$).
• If $f'(x_{old})$ is negative (slope is downhill), we subtract a negative value (which means adding), moving $x$ to the right (increasing $x$).

In both cases, the update moves $x$ in the downhill direction.

Let's visualize this with a simple quadratic function, like $f(x) = (x-3)^2 + 2$. We know its minimum occurs at $x=3$. The derivative is $f'(x) = 2(x-3)$. Let's start at $x=0$ and use a learning rate $\eta = 0.2$.

The blue line shows the function $f(x)=(x-3)^2+2$. The orange points and dotted line show the path taken by gradient descent starting at $x=0$. Each step moves closer to the minimum at $x=3$.

In the chart above:

1. We start at $x=0$. The function value $f(0)=11$. The slope $f'(0) = 2(0-3) = -6$ (steeply downhill).
2. The first update is $x_{new} = 0 - 0.2 \times (-6) = 0 + 1.2 = 1.2$. We move to $x=1.2$.
3. At $x=1.2$, the function value $f(1.2) \approx 5.24$. The slope $f'(1.2) = 2(1.2-3) = -3.6$ (less steep, but still downhill).
4. The next update is $x_{new} = 1.2 - 0.2 \times (-3.6) = 1.2 + 0.72 = 1.92$. We move to $x=1.92$.
5. This process continues. Notice how the steps get smaller as we approach the minimum. This happens because the slope $f'(x)$ gets closer to zero near the bottom of the valley.

This visualization shows the core idea: gradient descent repeatedly uses the derivative (the slope) to determine the direction of the next step, iteratively moving towards a minimum point of the function. In machine learning, the function being minimized is the cost function, and the position $x$ represents the parameters of the model. By finding the parameters that minimize the cost, we find the model that best fits the data.