Cost Functions: Measuring Error

Linear regression aims to find the straight line that best represents the relationship within a dataset. But what criterion defines a "best" line? How can we objectively quantify the fit of a particular line to the data points? A method is required to measure the error, often referred to as "cost," associated with any given line. This measurement enables comparison between different potential lines and guides the learning algorithm in adjusting the line for a better fit.

Think about it this way: for any line we draw through our data, some points will be close to the line, and others might be further away. The distance between an actual data point and the point predicted by our line represents an error for that specific prediction.

Calculating the Error for a Single Point

Let's consider a single data point in our training set, represented by its feature(s) $x_i$ and its actual target value $y_i$. If our current linear regression model predicts a value $\hat{y}_i$ (pronounced "y-hat") for this input $x_i$, the error for this single prediction is simply the difference between the actual value and the predicted value:

Error = Actual Value - Predicted Value Error = $y_i - \hat{y}_i$

This difference is often called the residual. A positive residual means the prediction was too low, and a negative residual means the prediction was too high. A residual of zero means the prediction was perfect for that data point.

The vertical dashed line shows the error (residual) for one data point: the difference between the actual value (blue dot) and the value predicted by the line (point on the gray line).

Aggregating Errors: The Cost Function

We need a single number that summarizes the total error across all the data points in our training set. Simply summing the individual errors ($y_i - \hat{y}_i$) isn't very useful, because positive and negative errors could cancel each other out, giving us a misleadingly small total error even if the line is a poor fit.

A common approach is to:

1. Square each individual error: $(y_i - \hat{y}_i)^2$. This solves the cancellation problem (squares are always non-negative) and also penalizes larger errors more heavily than smaller ones. An error of 2 becomes 4, while an error of 10 becomes 100.
2. Calculate the average of these squared errors: Sum up all the squared errors and divide by the number of data points.

This gives us the Mean Squared Error (MSE), a very popular cost function for regression problems.

The formula for MSE is:

$J(\theta_0, \theta_1) = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2$

Let's break this down:

• $N$: The total number of data points in our training set.
• $i=1$: We start summing from the first data point...
• $... N$: ...up to the last data point.
• $\sum$: The summation symbol, meaning we add up the results for each point.
• $y_i$: The actual target value for the $i$-th data point.
• $\hat{y}_i$: The value predicted by our linear model for the $i$-th data point. Remember, for simple linear regression, $\hat{y}_i = \theta_0 + \theta_1 x_i$ (where $\theta_0$ is the intercept and $\theta_1$ is the slope, the parameters our model needs to learn).
• $(\hat{y}_i - y_i)^2$: The squared error for the $i$-th data point. (Note: Sometimes you'll see $(y_i - \hat{y}_i)^2$, which gives the same result when squared).
• $\frac{1}{N}$: We divide the total sum of squared errors by $N$ to get the mean (average).
• $J(\theta_0, \theta_1)$: This shows that the MSE is a function of the model parameters ($\theta_0$ and $\theta_1$). Different values for the slope and intercept will result in a different line, different predictions ($\hat{y}_i$), and therefore a different MSE value.

Sometimes, particularly in statistical contexts or other courses, you might see the formula with $1/(2N)$ instead of $1/N$. The factor of 2 is added for mathematical convenience when calculating derivatives later (specifically for Gradient Descent), but it doesn't change the location of the minimum error. For understanding the concept, $1/N$ representing the mean is often clearer.

The Goal: Minimizing the Cost

"The Mean Squared Error gives us a single, positive value representing how well our line, defined by specific parameters $\theta_0$ and $\theta_1$, fits the overall data. A perfect fit would have an MSE of 0 (though this rarely happens with data). A line that fits poorly will have a large MSE."

Therefore, the goal of our learning algorithm is to find the values of $\theta_0$ and $\theta_1$ that result in the lowest possible MSE.

Minimizing this cost function means finding the line that, on average, makes the smallest squared errors when predicting the target values in our training data. In the next section on Gradient Descent, we'll see how the algorithm systematically adjusts $\theta_0$ and $\theta_1$ to reduce the value of this cost function, effectively finding the best-fitting line.