Simple Evaluation Workflow Example

Let's walk through a simplified, concrete example to see the basic evaluation workflow in action. Imagine we want to build a model to predict house prices based on their size in square feet. This is a regression problem because the price is a continuous numerical value.

Scenario: Predicting House Prices

We have collected data on 10 houses, noting their size and sale price:

Size (sq ft)	Price ($1000s)
1500	300
1600	320
1700	350
1800	380
1900	400
2000	410
2100	430
2200	450
1400	280
2300	460

Our goal is to train a model on some of this data and then evaluate how well it predicts prices on data it hasn't seen before.

The Evaluation Workflow Steps

Here's how we apply the standard workflow:

A visual representation of the evaluation workflow for our house price prediction example.

1. Choose Metrics: Since this is a regression problem, we'll use metrics suited for continuous values. Let's choose Mean Absolute Error (MAE), Mean Squared Error (MSE), and the Coefficient of Determination (R-squared or $R^2$). These will tell us the average error, penalize larger errors, and indicate the proportion of price variance explained by our model, respectively.

2. Split Data: We need to separate our data into a training set and a test set. A common split is 80% for training and 20% for testing. We'll randomly select 8 houses for training and reserve the remaining 2 for testing. It's important that the model never sees the test data during training.

   • Training Set (8 houses): Let's say these are randomly selected: (1500, 300), (1700, 350), (1800, 380), (1900, 400), (2000, 410), (2100, 430), (1400, 280), (2300, 460).
   • Test Set (2 houses): The remaining houses: (1600, 320), (2200, 450).

3. Train Model: We use the training set (the 8 houses) to train our machine learning model. Let's imagine we use a simple linear regression model. The training process finds the best line that fits the training data points (size vs. price). For this example, let's assume the trained model learns the relationship:

   $$\text{Predicted Price} = 0.2 \times \text{Size} + 50$$

   (Note: This is a simplified model equation for illustration purposes).

4. Generate Predictions: Now, we use our trained model to predict the prices for the houses in the test set. We only give the model the sizes from the test set (1600 sq ft and 2200 sq ft) and see what prices it predicts.

   • For Size = 1600 sq ft: Predicted Price = $0.2 \times 1600 + 50 = 320 + 50 = 370$ ($370,000)
   • For Size = 2200 sq ft: Predicted Price = $0.2 \times 2200 + 50 = 440 + 50 = 490$ ($490,000)

5. Calculate Performance Metrics: We compare the model's predictions ($370k,$490k) with the actual prices in the test set ($320k,$450k).

   • Errors:
     • House 1: Error = Predicted - Actual = $370 - 320 = 50$
     • House 2: Error = Predicted - Actual = $490 - 450 = 40$
   • MAE (Mean Absolute Error): Average of the absolute errors. $$MAE = \frac{|50| + |40|}{2} = \frac{50 + 40}{2} = \frac{90}{2} = 45$$
   • MSE (Mean Squared Error): Average of the squared errors. $$MSE = \frac{50^2 + 40^2}{2} = \frac{2500 + 1600}{2} = \frac{4100}{2} = 2050$$
   • RMSE (Root Mean Squared Error): Square root of MSE. $$RMSE = \sqrt{2050} \approx 45.28$$
   • R-squared ($R^2$): Measures the proportion of variance explained. Requires comparing the model's errors to the variance of the actual target values in the test set.
     • Mean of actual test prices: $(320 + 450) / 2 = 385$
     • Total Sum of Squares (SST): Sum of squared differences from the mean actual price. $SST = (320 - 385)^2 + (450 - 385)^2 = (-65)^2 + (65)^2 = 4225 + 4225 = 8450$
     • Residual Sum of Squares (SSE): Sum of squared errors (already calculated for MSE numerator). $SSE = 50^2 + 40^2 = 2500 + 1600 = 4100$
     • Calculate $R^2$: $R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{4100}{8450} \approx 1 - 0.485 = 0.515$

6. Interpret the Results:

   • The MAE is 45. This means, on average, our model's price predictions on the test set were off by $45,000.
   • The RMSE is approximately 45.28. This also gives an idea of the typical error magnitude in the original units ($ thousands), penalizing larger errors slightly more than MAE does. Here, it's quite close to the MAE because the errors (40 and 50) are similar.
   • The $R^2$ is 0.515. This suggests that our simple model (based only on size) explains about 51.5% of the variation in house prices within our small test set. The remaining 48.5% is unexplained by this model (perhaps due to factors like location, number of bedrooms, age, etc., or model inaccuracies).

Comparison of predicted prices versus actual prices for the two houses in the test set. Points on the dashed line represent perfect predictions. Our model predicted higher than the actual prices for both test houses.

This example, though simplified with a tiny dataset and a model, demonstrates the fundamental steps: split data, train on one part, predict on the other, and calculate metrics to assess performance on unseen data. This structured process helps you understand how well your model is likely to perform when faced with new, real-world examples.