Hands-on Practical: Manual Gradient Calculation

Cost functions, gradients, and the basic idea of gradient descent are essential concepts in machine learning optimization. A manual calculation of the gradient for a simple linear regression model, using a tiny dataset, provides practical insight. This process solidifies understanding of how derivatives drive optimization.

Setting the Scene: A Simple Problem

Imagine we have a very small dataset with just three points $(x, y)$: $(1, 2)$, $(2, 3)$, and $(3, 5)$. Our goal is to find the best-fitting line of the form $y = mx + b$ for these points.

Our tiny dataset containing three points, along with an initial guess for our line: $y = 0x + 0$.

We need a way to measure how "good" our line is. We'll use the Mean Squared Error (MSE) as our cost function, which we discussed earlier. For our three data points $(x_1, y_1), (x_2, y_2), (x_3, y_3)$, the MSE is:

$$J(m, b) = \frac{1}{3} \sum_{i=1}^{3} (y_i - \hat{y}_i)^2 = \frac{1}{3} \sum_{i=1}^{3} (y_i - (mx_i + b))^2$$

Let's start with an initial guess for our parameters: $m = 0$ and $b = 0$. Our line is initially $y = 0$.

Calculating the Initial Cost

First, let's calculate the cost for our initial guess ($m=0, b=0$). The predicted values ($\hat{y}_i = mx_i + b$) are:

• $\hat{y}_1 = 0(1) + 0 = 0$
• $\hat{y}_2 = 0(2) + 0 = 0$
• $\hat{y}_3 = 0(3) + 0 = 0$

The squared errors are:

• $(y_1 - \hat{y}_1)^2 = (2 - 0)^2 = 4$
• $(y_2 - \hat{y}_2)^2 = (3 - 0)^2 = 9$
• $(y_3 - \hat{y}_3)^2 = (5 - 0)^2 = 25$

The Mean Squared Error is:

$$J(0, 0) = \frac{1}{3} (4 + 9 + 25) = \frac{38}{3} \approx 12.67$$

This is our starting cost. Our goal is to reduce this value by adjusting $m$ and $b$.

Calculating the Gradients

Now for the core calculus part: finding the gradient of the cost function $J(m, b)$. The gradient is a vector containing the partial derivatives with respect to each parameter: $\nabla J = \left[ \frac{\partial J}{\partial m}, \frac{\partial J}{\partial b} \right]$. These derivatives tell us how the cost changes as we slightly change $m$ or $b$.

Let's find $\frac{\partial J}{\partial m}$ and $\frac{\partial J}{\partial b}$. Remember, when taking a partial derivative with respect to one variable (like $m$), we treat other variables (like $b$) as constants.

Partial Derivative with respect to $m$ ($\frac{\partial J}{\partial m}$)

We start with the cost function:

$$J(m, b) = \frac{1}{3} \left[ (y_1 - (mx_1 + b))^2 + (y_2 - (mx_2 + b))^2 + (y_3 - (mx_3 + b))^2 \right]$$

We differentiate term by term with respect to $m$. Let's focus on one term: $(y_i - (mx_i + b))^2$. We use the chain rule. Let $u = y_i - mx_i - b$. Then the term is $u^2$. The derivative of $u^2$ with respect to $m$ is $2u \cdot \frac{\partial u}{\partial m}$. Now, we find $\frac{\partial u}{\partial m} = \frac{\partial}{\partial m} (y_i - mx_i - b)$. Since $y_i$, $x_i$, and $b$ are treated as constants when differentiating with respect to $m$, this simplifies to $\frac{\partial}{\partial m} (-mx_i) = -x_i$.

So, the derivative of $(y_i - (mx_i + b))^2$ with respect to $m$ is $2(y_i - mx_i - b)(-x_i)$.

Applying this to our cost function $J(m, b)$:

$$\frac{\partial J}{\partial m} = \frac{1}{3} \sum_{i=1}^{3} 2(y_i - (mx_i + b))(-x_i)$$ $$\frac{\partial J}{\partial m} = -\frac{2}{3} \sum_{i=1}^{3} x_i(y_i - (mx_i + b))$$

Partial Derivative with respect to $b$ ($\frac{\partial J}{\partial b}$)

Similarly, we differentiate $J(m, b)$ with respect to $b$. Again, consider one term $(y_i - (mx_i + b))^2$. Let $u = y_i - mx_i - b$. The derivative with respect to $b$ is $2u \cdot \frac{\partial u}{\partial b}$. Now, $\frac{\partial u}{\partial b} = \frac{\partial}{\partial b} (y_i - mx_i - b)$. Treating $y_i$, $m$, and $x_i$ as constants, this is $\frac{\partial}{\partial b} (-b) = -1$.

So, the derivative of $(y_i - (mx_i + b))^2$ with respect to $b$ is $2(y_i - mx_i - b)(-1)$.

Applying this to the cost function:

$$\frac{\partial J}{\partial b} = \frac{1}{3} \sum_{i=1}^{3} 2(y_i - (mx_i + b))(-1)$$ $$\frac{\partial J}{\partial b} = -\frac{2}{3} \sum_{i=1}^{3} (y_i - (mx_i + b))$$

Numerical Gradient Calculation

Now we plug our data points $(1, 2), (2, 3), (3, 5)$ and our current parameters $m=0, b=0$ into these formulas. Remember our predictions were $\hat{y}_1=0, \hat{y}_2=0, \hat{y}_3=0$.

Calculating $\frac{\partial J}{\partial m}$:

$$\frac{\partial J}{\partial m} = -\frac{2}{3} \left[ x_1(y_1 - \hat{y}_1) + x_2(y_2 - \hat{y}_2) + x_3(y_3 - \hat{y}_3) \right]$$ $$\frac{\partial J}{\partial m} = -\frac{2}{3} \left[ 1(2 - 0) + 2(3 - 0) + 3(5 - 0) \right]$$ $$\frac{\partial J}{\partial m} = -\frac{2}{3} \left[ 1(2) + 2(3) + 3(5) \right]$$ $$\frac{\partial J}{\partial m} = -\frac{2}{3} \left[ 2 + 6 + 15 \right] = -\frac{2}{3} [23] = -\frac{46}{3} \approx -15.33$$

Calculating $\frac{\partial J}{\partial b}$:

$$\frac{\partial J}{\partial b} = -\frac{2}{3} \left[ (y_1 - \hat{y}_1) + (y_2 - \hat{y}_2) + (y_3 - \hat{y}_3) \right]$$ $$\frac{\partial J}{\partial b} = -\frac{2}{3} \left[ (2 - 0) + (3 - 0) + (5 - 0) \right]$$ $$\frac{\partial J}{\partial b} = -\frac{2}{3} \left[ 2 + 3 + 5 \right] = -\frac{2}{3} [10] = -\frac{20}{3} \approx -6.67$$

So, at $m=0, b=0$, the gradient is $\nabla J \approx [-15.33, -6.67]$.

Interpreting the Gradient

What do these numbers tell us?

• $\frac{\partial J}{\partial m} \approx -15.33$: This is negative, meaning if we increase $m$ slightly, the cost $J$ will decrease. The magnitude (15.33) indicates the cost is quite sensitive to changes in $m$.
• $\frac{\partial J}{\partial b} \approx -6.67$: This is also negative. Increasing $b$ slightly will also decrease the cost $J$. The cost is less sensitive to changes in $b$ compared to $m$ at this point.

The gradient $[-15.33, -6.67]$ points in the direction of the steepest increase in cost. To decrease the cost (which is our goal in optimization), we need to move in the opposite direction of the gradient. This means we should increase both $m$ and $b$.

Taking a Small Step (Gradient Descent)

Let's perform one step of gradient descent. We need a learning rate, $\alpha$. Let's choose a small value, say $\alpha = 0.01$.

The update rules are:

• $m_{new} = m_{old} - \alpha \frac{\partial J}{\partial m}$
• $b_{new} = b_{old} - \alpha \frac{\partial J}{\partial b}$

Plugging in our values ($m_{old}=0, b_{old}=0$, $\alpha=0.01$, $\frac{\partial J}{\partial m} = -46/3$, $\frac{\partial J}{\partial b} = -20/3$):

• $m_{new} = 0 - (0.01) \left(-\frac{46}{3}\right) = 0.01 \times \frac{46}{3} = \frac{0.46}{3} \approx 0.153$
• $b_{new} = 0 - (0.01) \left(-\frac{20}{3}\right) = 0.01 \times \frac{20}{3} = \frac{0.20}{3} \approx 0.067$

Our new parameters are approximately $m \approx 0.153$ and $b \approx 0.067$.

Let's visualize the new line $y \approx 0.153x + 0.067$:

Our data points along with the initial line (dashed red) and the line after one gradient descent step (solid green). The new line is slightly closer to the data points.

If we were to calculate the cost $J(0.153, 0.067)$, we would find it's lower than our initial cost of $12.67$ (as calculated in the thought process, it's around 10.03). By repeating this process - calculating gradients and updating parameters - gradient descent iteratively finds better values for $m$ and $b$, minimizing the cost function.

This manual calculation shows the mechanism: derivatives (gradients) tell us the direction to adjust parameters ($m$ and $b$) to improve our model by reducing the cost function ($MSE$). While libraries automate this, understanding the underlying calculation is fundamental to grasping how machine learning models learn.