Calculating Gradients for the Cost Function

Calculating gradients for the cost function is a primary step in optimization. A simple linear model, $y = mx + b$, represents data. To measure how well this line fits the data, a cost function, typically the Mean Squared Error (MSE), quantifies the error. The objective is to find the values of $m$ (slope) and $b$ (y-intercept) that minimize this cost.

Gradient descent needs directions. It needs to know: "If I change $m$ slightly, how does the cost change?" and "If I change $b$ slightly, how does the cost change?" This is precisely what partial derivatives tell us. We need to calculate the gradient of the cost function, $\nabla C$, which consists of the partial derivatives with respect to each parameter: $m$ and $b$.

The Cost Function: Mean Squared Error (MSE)

Let's assume we have $N$ data points, $(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)$. For any given $m$ and $b$, our model predicts $y_{pred, i} = mx_i + b$ for each input $x_i$. The actual value is $y_i$.

The Mean Squared Error cost function $C(m, b)$ is defined as the average of the squared differences between the predicted and actual values:

$$C(m, b) = \frac{1}{N} \sum_{i=1}^{N} (y_{pred, i} - y_{actual, i})^2$$

Substituting our model's prediction:

$$C(m, b) = \frac{1}{N} \sum_{i=1}^{N} ( (mx_i + b) - y_i )^2$$

This function $C(m,b)$ takes $m$ and $b$ as inputs and outputs a single number representing the average error. Our task now is to find $\frac{\partial C}{\partial m}$ and $\frac{\partial C}{\partial b}$.

Calculating the Partial Derivative with Respect to $m$ ($\frac{\partial C}{\partial m}$)

To find $\frac{\partial C}{\partial m}$, we differentiate the cost function $C(m, b)$ with respect to $m$, treating $b$ (and all $x_i$, $y_i$, and $N$) as constants.

Let's look at the expression term by term:

1. Constant Factor: The $\frac{1}{N}$ is a constant multiplier, so it stays put.
2. Sum Rule: The derivative of a sum is the sum of the derivatives. We can move the derivative inside the summation: $$\frac{\partial C}{\partial m} = \frac{1}{N} \sum_{i=1}^{N} \frac{\partial}{\partial m} (mx_i + b - y_i)^2$$
3. Chain Rule: We need to differentiate the term $(mx_i + b - y_i)^2$ with respect to $m$. Let $u = mx_i + b - y_i$. We are differentiating $u^2$ with respect to $m$. The chain rule states $\frac{d}{dm}(u^2) = 2u \cdot \frac{\partial u}{\partial m}$.
   • First part: $2u = 2(mx_i + b - y_i)$.
   • Second part: We need $\frac{\partial u}{\partial m} = \frac{\partial}{\partial m} (mx_i + b - y_i)$. Since $b$ and $y_i$ are treated as constants, their derivatives are zero. The derivative of $mx_i$ with respect to $m$ is just $x_i$ (because $x_i$ is treated as a constant coefficient of $m$). So, $\frac{\partial u}{\partial m} = x_i$.
4. Putting it Together: Substituting back into the chain rule formula: $\frac{\partial}{\partial m} (mx_i + b - y_i)^2 = 2(mx_i + b - y_i) \cdot x_i$.
5. Final Result for $\frac{\partial C}{\partial m}$: Now substitute this back into the summation: $$\frac{\partial C}{\partial m} = \frac{1}{N} \sum_{i=1}^{N} 2(mx_i + b - y_i) x_i$$ We can pull the constant 2 out of the sum: $$\frac{\partial C}{\partial m} = \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) x_i$$

This expression tells us how the cost changes as we slightly change the slope $m$. Notice it depends on the error term $(mx_i + b - y_i)$ and the input value $x_i$ for each data point.

Calculating the Partial Derivative with Respect to $b$ ($\frac{\partial C}{\partial b}$)

Now we repeat the process, but this time we differentiate $C(m, b)$ with respect to $b$, treating $m$ as a constant.

1. Constant Factor and Sum Rule: Same as before, we get: $$\frac{\partial C}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} \frac{\partial}{\partial b} (mx_i + b - y_i)^2$$
2. Chain Rule: Again, let $u = mx_i + b - y_i$. We need $\frac{d}{db}(u^2) = 2u \cdot \frac{\partial u}{\partial b}$.
   • First part: $2u = 2(mx_i + b - y_i)$.
   • Second part: We need $\frac{\partial u}{\partial b} = \frac{\partial}{\partial b} (mx_i + b - y_i)$. This time, $m$, $x_i$, and $y_i$ are treated as constants. The derivative of $mx_i$ with respect to $b$ is 0. The derivative of $b$ with respect to $b$ is 1. The derivative of $y_i$ with respect to $b$ is 0. So, $\frac{\partial u}{\partial b} = 1$.
3. Putting it Together: $\frac{\partial}{\partial b} (mx_i + b - y_i)^2 = 2(mx_i + b - y_i) \cdot 1$.
4. Final Result for $\frac{\partial C}{\partial b}$: Substitute back into the summation: $$\frac{\partial C}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} 2(mx_i + b - y_i) (1)$$ Pulling the constant 2 out: $$\frac{\partial C}{\partial b} = \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i)$$

This expression tells us how the cost changes as we slightly change the y-intercept $b$. It depends only on the error term $(mx_i + b - y_i)$ for each data point.

The Gradient Vector $\nabla C$

We've successfully calculated the partial derivatives! We can now assemble them into the gradient vector for our cost function $C(m, b)$:

$$\nabla C(m, b) = \left[ \frac{\partial C}{\partial m}, \frac{\partial C}{\partial b} \right]$$

Substituting the expressions we found:

$$\nabla C(m, b) = \left[ \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) x_i, \quad \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) \right]$$

This vector, $\nabla C$, is fundamental. For any given values of $m$ and $b$:

• The first component ($\frac{\partial C}{\partial m}$) tells us the rate of change of the cost function if we move purely in the $m$ direction. Its sign tells us if increasing $m$ increases or decreases the cost.
• The second component ($\frac{\partial C}{\partial b}$) tells us the rate of change of the cost function if we move purely in the $b$ direction. Its sign tells us if increasing $b$ increases or decreases the cost.

Crucially, the gradient vector $\nabla C$ points in the direction of the steepest increase of the cost function $C(m,b)$ at the current point $(m,b)$. Since we want to minimize the cost, gradient descent will involve taking steps in the direction opposite to the gradient.

With these calculated gradients, we now have the precise information needed to iteratively update $m$ and $b$ to find the line that best fits our data. The next step is to see how these gradients are used within the gradient descent algorithm itself.