Understanding Regression Predictions

When a regression model makes a prediction, it outputs a continuous numerical value. Regression tasks are defined by predicting such values. For example, predictions for:

• The price of a house based on its features (size, location, number of bedrooms).
• The temperature tomorrow based on historical weather data.
• The expected sales revenue for next quarter based on past performance and marketing spend.

In each case, the model takes some input information and produces a single number as its output, its best guess for the quantity we're interested in.

Predictions vs. Actual Values

For any given input instance (like a specific house or a particular day's weather pattern), there are two values we care about:

1. The Predicted Value ($\hat{y}$): This is the output generated by our trained regression model. It's the model's estimate of the target variable. We often use a "hat" symbol ($\hat{}$) over the variable ($y$) to denote a predicted value.
2. The Actual Value ($y$): This is the true, observed value for that instance. It's the correct answer we were hoping the model would predict (e.g., the actual price the house sold for, the actual temperature recorded).

Ideally, the predicted value $\hat{y}$ would be exactly equal to the actual value $y$. However, in practice, models are rarely perfect. There will almost always be some difference between the prediction and the reality.

Understanding Prediction Error

This difference between the actual value and the predicted value is called the error or sometimes the residual. We calculate it simply as:

$$\text{Error} = \text{Actual Value} - \text{Predicted Value}$$ $$e = y - \hat{y}$$

A positive error means the model's prediction was too low. A negative error means the prediction was too high. An error of zero means the prediction was perfect for that specific instance.

Let's look at a simple example: predicting apartment rent based on square footage.

• Instance 1:
  • Size: 500 sq ft
  • Actual Rent ($y$): $1200
  • Model Prediction ($\hat{y}$): $1150
  • Error: $1200 -$1150 = $50 (Model predicted too low)
• Instance 2:
  • Size: 800 sq ft
  • Actual Rent ($y$): $1800
  • Model Prediction ($\hat{y}$): $1850
  • Error: $1800 -$1850 = -$50 (Model predicted too high)
• Instance 3:
  • Size: 650 sq ft
  • Actual Rent ($y$): $1400
  • Model Prediction ($\hat{y}$): $1400
  • Error: $1400 -$1400 = $0 (Model predicted perfectly)

"Models make errors because data is complex and often contains inherent randomness or "noise". Also, models are usually simplifications of reality. They learn patterns from the training data, but these patterns might not perfectly capture every detail needed for exact predictions on new, unseen data."

The goal of evaluating a regression model isn't necessarily to achieve zero error on every single prediction (which is often impossible), but rather to understand the typical magnitude and characteristics of these errors across a whole dataset.

The plot below illustrates this concept. The blue dots represent actual data points (e.g., actual rent vs. square footage). The orange line represents the predictions made by a simple linear regression model. The vertical gray lines show the error (or residual) for each point – the distance between the actual value (dot) and the predicted value (point on the line).

A simple linear regression model attempting to predict a target value based on a single feature. The blue dots are the actual data points, the orange line shows the model's predictions, and the dotted gray lines represent the prediction errors (residuals) for each point.

When we evaluate a model using a test set (data the model hasn't seen during training), we calculate the error for each prediction. Since we'll have many errors (one for each instance in the test set), we need ways to summarize them into meaningful performance metrics. This is where metrics like Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R-squared ($R^2$) come in. They provide different ways to quantify the overall accuracy and effectiveness of our regression model, which we will explore in the following sections.