Defining a Cost Function for Linear Regression

A simple linear model, often expressed as $y = mx + b$, is fundamental in many analyses. When working with data points, such as pairs of (study hours, exam score), the goal is often to find a line that best represents the relationship between these pairs. Defining what "best" means in mathematical terms is essential for this process.

Imagine you have a handful of data points plotted on a graph. You can draw many different lines through these points. Some lines will pass very close to the points, while others will be quite far off. We need a way to measure how well a specific line, defined by a particular slope $m$ and intercept $b$, fits our data. This measure is what we call a cost function (or sometimes a loss function or objective function).

The core idea is to quantify the "error" or "mistake" our line makes for each data point. Let's say we have $n$ data points: $(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)$. For any given input $x_i$, our current line $y = mx + b$ predicts an output value. Let's call this prediction $\hat{y}_i$ (pronounced "y-hat"):

$$\hat{y}_i = mx_i + b$$

The actual output value for this data point is $y_i$. The difference between the actual value and the predicted value, $y_i - \hat{y}_i$, tells us how far off our prediction was for that specific point. This difference is often called the residual.

If the actual value $y_i$ is larger than the prediction $\hat{y}_i$, the residual is positive. If the prediction is larger than the actual value, the residual is negative. If we just add up these residuals across all data points, the positive and negative errors might cancel each other out, giving us a misleadingly small total error even if the line is a poor fit.

To avoid this cancellation and ensure that larger errors contribute more to our total measure of "badness", we typically square each residual: $(y_i - \hat{y}_i)^2$. Squaring makes all errors positive and penalizes larger deviations more significantly than smaller ones (e.g., an error of 2 becomes 4, while an error of 3 becomes 9).

Now that we have a measure of error for each individual point, we need a single number to represent the overall error of our line across all $n$ data points. A very common approach is to calculate the average of these squared errors. This gives us the Mean Squared Error (MSE) cost function.

Mathematically, we define the Mean Squared Error cost function, which we'll call $J$, as a function of our model parameters $m$ and $b$:

$$J(m, b) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Let's break this down:

• $J(m, b)$: This shows that the cost $J$ depends on the chosen values for the slope $m$ and the intercept $b$. Our goal is to find the $m$ and $b$ that make $J$ as small as possible.
• $n$: The total number of data points we have.
• $\sum_{i=1}^{n}$: This is summation notation. It means "add up the following expression for each data point, starting from $i=1$ and going up to $i=n$."
• $(x_i, y_i)$: The input ($x_i$) and the actual output ($y_i$) for the $i$-th data point.
• $\hat{y}_i$: The output predicted by our model ($mx_i + b$) for the input $x_i$.
• $(y_i - \hat{y}_i)^2$: The squared error for the $i$-th data point.
• $\frac{1}{n}$: Dividing the total sum of squared errors by $n$ gives us the mean or average squared error.

We can substitute the definition of $\hat{y}_i$ directly into the cost function formula:

$$J(m, b) = \frac{1}{n} \sum_{i=1}^{n} (y_i - (mx_i + b))^2$$

This formula gives us a single number, $J$, that tells us how badly our line $y = mx + b$ fits the data on average. A smaller value of $J$ means the line's predictions are, on average, closer to the actual data points, indicating a better fit. A larger value means a poorer fit.

This plot shows sample data points (blue dots), a possible linear model (red line), and the vertical errors (residuals, dotted gray lines) between the actual data points and the model's predictions. The MSE cost function calculates the average of the squares of the lengths of these dotted lines.

Our optimization goal is now concrete: find the specific values of $m$ and $b$ that result in the lowest possible value for $J(m, b)$. Minimizing this cost function means finding the line that minimizes the average squared vertical distance to our data points.

Why choose Mean Squared Error? Besides penalizing larger errors more, it turns out that MSE has favorable mathematical properties. Specifically, it's a smooth, continuous function, and importantly, we can easily calculate its derivatives with respect to $m$ and $b$. As we saw in previous chapters, these derivatives (the gradient) are exactly what we need to guide the gradient descent algorithm towards the minimum cost.

In the next section, we'll take this cost function $J(m, b)$ and apply the calculus we've learned to calculate its gradient. This will be the engine that drives our optimization process.