Practice: Calculating Regression Metrics

Okay, let's put the concepts of MAE, MSE, RMSE, and R-squared into practice. Seeing how these metrics are calculated with actual numbers helps solidify understanding and prepares you to evaluate your own regression models.

Example Scenario: Predicting House Prices

Imagine we've built a simple regression model to predict house prices based on their size (in square feet). We trained this model on some data, and now we want to evaluate its performance on a separate set of 5 houses it hasn't seen before (our test set).

Here are the actual prices and the prices predicted by our model (in thousands of dollars):

House ID	Size (sq ft)	Actual Price ($1000s)$(y_i)$	Predicted Price ($1000s)$(\hat{y}_i)$
1	1500	300	310
2	1200	250	240
3	1800	380	350
4	1400	290	300
5	1600	330	340

Our goal is to use the metrics we've learned to quantify how well our model's predictions match the actual prices. We have $n=5$ data points in our test set.

Step 1: Calculate Individual Errors

The first step is always to find the difference between the actual value ($y_i$) and the predicted value ($\hat{y}_i$) for each data point. This difference is called the error or residual.

$$Error_i = y_i - \hat{y}_i$$

Let's add an 'Error' column to our table:

House ID	Actual Price $(y_i)$	Predicted Price $(\hat{y}_i)$	Error $(y_i - \hat{y}_i)$
1	300	310	-10
2	250	240	10
3	380	350	30
4	290	300	-10
5	330	340	-10

Step 2: Calculate Mean Absolute Error (MAE)

MAE gives us the average size of the errors, ignoring whether they are positive or negative.

The formula is:

$$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

1. Calculate Absolute Errors: Take the absolute value of each error.
2. Sum Absolute Errors: Add them up.
3. Calculate Average: Divide the sum by the number of data points ($n=5$).

| House ID | Error $(y_i - \hat{y}_i)$ | Absolute Error $|y_i - \hat{y}_i|$ | | :------- | :---------------------- | :-------------------------------- | | 1 | -10 | 10 | | 2 | 10 | 10 | | 3 | 30 | 30 | | 4 | -10 | 10 | | 5 | -10 | 10 | | Sum | | 70 |

Now, calculate the average:

$$MAE = \frac{70}{5} = 14$$

Interpretation: On average, our model's price prediction is off by $14,000. The units of MAE are the same as the target variable (thousands of dollars in this case).

Step 3: Calculate Mean Squared Error (MSE)

MSE calculates the average of the squared errors. Squaring the errors makes them all positive and gives much more weight to larger errors.

The formula is:

$$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

1. Calculate Squared Errors: Square each individual error.
2. Sum Squared Errors: Add them up.
3. Calculate Average: Divide the sum by the number of data points ($n=5$).

House ID	Error $(y_i - \hat{y}_i)$	Squared Error $(y_i - \hat{y}_i)^2$
1	-10	100
2	10	100
3	30	900
4	-10	100
5	-10	100
Sum		1300

Now, calculate the average:

$$MSE = \frac{1300}{5} = 260$$

Interpretation: The MSE is 260. Notice how the single large error of 30 for House 3 contributed 900 to the sum, significantly impacting the MSE. The units here are (thousands of dollars) squared, which isn't very intuitive. This leads us to RMSE.

Step 4: Calculate Root Mean Squared Error (RMSE)

RMSE is simply the square root of the MSE. This brings the metric back into the original units of the target variable, making it easier to interpret.

The formula is:

$$RMSE = \sqrt{MSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$$

1. Calculate MSE: We already did this, MSE = 260.
2. Take the Square Root: Calculate $\sqrt{260}$.

$$RMSE = \sqrt{260} \approx 16.12$$

Interpretation: The RMSE is approximately $16,120. Like MAE, it represents a typical magnitude of error in our predictions, measured in thousands of dollars. Because it's derived from MSE, RMSE is more sensitive to large errors than MAE (notice RMSE 16.12 > MAE 14).

Step 5: Calculate Coefficient of Determination (R-squared, $R^2$)

$R^2$ tells us the proportion of the variance in the actual prices that our model is able to explain. It compares our model's errors to the errors of a very simple baseline model that just predicts the average price for all houses.

The formula is:

$$R^2 = 1 - \frac{SSR}{SST} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$$

Where:

• $SSR$ (Sum of Squared Residuals) is the sum of our model's squared errors. We calculated this for MSE: $SSR = 1300$.
• $SST$ (Total Sum of Squares) is the sum of squared differences between the actual prices and the average actual price ($\bar{y}$). This represents the total variance in the actual prices.

Let's calculate $SST$:

1. Calculate the Mean Actual Price ($\bar{y}$):

   $$\bar{y} = \frac{300 + 250 + 380 + 290 + 330}{5} = \frac{1550}{5} = 310$$

   The average actual price is $310,000.

2. Calculate Deviations from the Mean: Find $(y_i - \bar{y})$ for each house.

3. Calculate Squared Deviations: Square these deviations.

4. Sum Squared Deviations: Add them up to get $SST$.

House ID	Actual Price $(y_i)$	Mean Price $(\bar{y})$	Deviation $(y_i - \bar{y})$	Squared Deviation $(y_i - \bar{y})^2$
1	300	310	-10	100
2	250	310	-60	3600
3	380	310	70	4900
4	290	310	-20	400
5	330	310	20	400
Sum				9400 ($SST$)

So, $SST = 9400$.

Now we can calculate $R^2$:

$$R^2 = 1 - \frac{SSR}{SST} = 1 - \frac{1300}{9400}$$ $$R^2 = 1 - 0.1383 \approx 0.8617$$

Interpretation: Our model's $R^2$ is approximately 0.86, or 86%. This means that our model (using house size) explains about 86% of the variability observed in the actual house prices in our test set. This is generally considered a good fit for this simple example. An $R^2$ of 1 would mean a perfect fit, while an $R^2$ of 0 would mean our model is no better than just predicting the average price for every house.

Summary of Results

Here's a summary of the calculated metrics for our house price prediction model:

Metric	Value	Interpretation
MAE	14.00	Average absolute prediction error is $14,000.
MSE	260.00	Average squared prediction error (units are $ squared).
RMSE	16.12	Typical prediction error magnitude is $16,120 (sensitive to large errors).
$R^2$	0.8617	Model explains ~86% of the variance in house prices.

Visualizing Predictions vs Actuals

A scatter plot is often helpful for visualizing regression performance. We plot actual values on one axis and predicted values on the other. Points falling close to the diagonal line ($y=x$) indicate accurate predictions.

Scatter plot showing actual house prices against predicted prices. The dashed gray line represents a perfect prediction ($Predicted = Actual$). Points close to this line indicate good predictions by the model.

This practice exercise demonstrates how to compute the standard regression metrics from a set of actual and predicted values. Calculating these yourself builds intuition about what each metric represents and how they respond differently to prediction errors. When evaluating your own models, you'll typically use libraries that compute these for you, but understanding the underlying calculations is fundamental.