Performing a Gradient Descent Step

A simple linear regression model, $y = mx + b$, uses a cost function, $J(m, b)$, to measure how well the line fits the data. The gradient of this cost function is calculated as a vector containing the partial derivatives $\frac{\partial J}{\partial m}$ and $\frac{\partial J}{\partial b}$. These derivatives indicate how the cost changes with slight adjustments to $m$ or $b$.

Now, how do we actually use these gradients to improve our model parameters and minimize the cost? This is where the gradient descent algorithm comes into play.

Think of the cost function $J(m, b)$ as defining a surface, perhaps like a terrain with hills and valleys. Our goal is to find the lowest point in this terrain, the minimum cost. The parameters $m$ and $b$ define our current location on this surface.

The gradient, $\nabla J = \left[ \frac{\partial J}{\partial m}, \frac{\partial J}{\partial b} \right]$, at our current location points in the direction of the steepest ascent, the quickest way uphill. Since we want to minimize the cost, we need to go downhill. Therefore, we take a step in the direction opposite to the gradient.

This leads us to the core update rule for gradient descent. For each parameter, we adjust its current value by subtracting a small amount proportional to its partial derivative:

• To update the slope $m$: $m_{\text{new}} = m_{\text{old}} - \alpha \frac{\partial J}{\partial m}$
• To update the y-intercept $b$: $b_{\text{new}} = b_{\text{old}} - \alpha \frac{\partial J}{\partial b}$

Here, $m_{\text{old}}$ and $b_{\text{old}}$ are the parameter values before the update step, and $m_{\text{new}}$ and $b_{\text{new}}$ are the values after the update step. The gradients $\frac{\partial J}{\partial m}$ and $\frac{\partial J}{\partial b}$ are calculated using the current values ($m_{\text{old}}$, $b_{\text{old}}$).

Notice the symbol $\alpha$ (alpha) in the update rules. This is the learning rate, which we'll discuss in the next section. For now, think of it as controlling the size of the step we take downhill. It's a small positive number (e.g., 0.01, 0.1).

• If the gradient component (say, $\frac{\partial J}{\partial m}$) is positive, it means increasing $m$ increases the cost. So, the update rule subtracts a positive value ($\alpha \times \text{positive gradient}$), effectively decreasing $m$.
• If the gradient component is negative, increasing $m$ decreases the cost. The update rule subtracts a negative value ($\alpha \times \text{negative gradient}$), which means we add to $m$, moving it in the direction that reduces the cost.

This single calculation and update for both $m$ and $b$ constitutes one step of gradient descent.

The diagram shows the flow for a single step in gradient descent. Start with the current model parameters, calculate the cost function's gradient using those parameters, and then update the parameters by moving slightly in the opposite direction of the gradient. This process is typically repeated many times.

It's important to understand that gradient descent is an iterative algorithm. One step usually isn't enough to reach the minimum cost. We repeat this process calculating the gradient and updating the parameters many times. With each step, we (hopefully) move closer to the values of $m$ and $b$ that minimize the cost function, resulting in a better-fitting linear regression model.

This step, guided by the derivatives we calculated, is the fundamental mechanism by which many machine learning models learn from data. By repeatedly adjusting parameters in the direction that reduces error (cost), the model gradually improves its predictions.