Comparing MAE, MSE, and RMSE

Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) are three common metrics for evaluating regression models. All three metrics give you a measure of the average error your regression model makes, but they do so in different ways and have distinct properties. Understanding these differences is important for choosing the right metric for your specific problem and for correctly interpreting your model's performance.

Let's compare these three metrics side-by-side.

Units of Measurement

One of the most practical differences lies in the units of the resulting error value:

• MAE: The error is measured in the original units of your target variable. If you are predicting house prices in dollars, the MAE will also be in dollars. This makes it quite intuitive to understand. An MAE of $10,000 means the predictions are, on average, off by$10,000.
• MSE: The error is measured in the squared units of your target variable. If you are predicting prices in dollars, the MSE will be in dollars squared ($dollars^2$). This is mathematically convenient in some contexts, but it's not easy to interpret directly in relation to the target variable's scale. What does $100,000,000 dollars squared actually mean in practical terms? It's harder to grasp.
• RMSE: By taking the square root of MSE, RMSE returns the error measure back to the original units of the target variable. Like MAE, if your target is in dollars, RMSE is also in dollars. This makes it easier to interpret than MSE, representing a sort of typical magnitude for the error.

Interpretability: MAE and RMSE are generally easier to interpret than MSE because their units match the target variable.

Sensitivity to Errors (Especially Large Errors)

This is where the core difference between MAE and MSE/RMSE lies: how they treat errors of different sizes.

• MAE: Calculates the average of the absolute errors:

  $$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

  Because it uses the absolute value, MAE treats every error linearly. A prediction that is off by 10 contributes exactly twice as much to the total error as a prediction that is off by 5. It doesn't give extra weight to larger errors. This means MAE is less sensitive to outliers (predictions that are wildly incorrect).

• MSE: Calculates the average of the squared errors:

  $$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

  By squaring the error term $(y_i - \hat{y}_i)$, MSE penalizes larger errors much more heavily than smaller ones. An error of 10 contributes $10^2 = 100$ to the sum, while an error of 5 contributes only $5^2 = 25$. The error of 10 contributes four times as much, not just twice as much. This makes MSE very sensitive to outliers. A few predictions with large errors can dramatically inflate the MSE score.

• RMSE: As the square root of MSE, RMSE shares MSE's sensitivity to large errors, though the final value is scaled back to the original units.

  $$\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$$

  Like MSE, RMSE will be significantly affected by outliers because the squaring operation happens before averaging and the square root.

Sensitivity Example:

Imagine two sets of prediction errors for a model:

1. Normal Errors: [2, -1, 3, -2, 1] (Errors are relatively small)
2. Errors with an Outlier: [2, -1, 3, -15, 1] (One error is much larger)

Let's calculate MAE and RMSE for both (we'll skip MSE for direct comparison as its units are different):

• Set 1 (Normal Errors):

  • MAE = (|2|+|-1|+|3|+|-2|+|1|) / 5 = (2+1+3+2+1) / 5 = 9 / 5 = 1.8
  • MSE = (2^2+(-1)^2+3^2+(-2)^2+1^2) / 5 = (4+1+9+4+1) / 5 = 19 / 5 = 3.8
  • RMSE = sqrt(3.8) ≈ 1.95

• Set 2 (With Outlier):

  • MAE = (|2|+|-1|+|3|+|-15|+|1|) / 5 = (2+1+3+15+1) / 5 = 22 / 5 = 4.4
  • RMSE = sqrt((2^2+(-1)^2+3^2+(-15)^2+1^2) / 5) = sqrt((4+1+9+225+1) / 5) = sqrt(240 / 5) = sqrt(48) ≈ 6.93

Notice how the single large error (-15) affected the metrics:

• MAE increased from 1.8 to 4.4 (an increase of 2.6).
• RMSE increased from 1.95 to 6.93 (an increase of almost 5.0).

The RMSE was much more drastically pulled upwards by the single outlier compared to the MAE.

MAE and RMSE calculated for two sets of errors: one with typical errors and one containing a single large outlier. RMSE shows a much larger relative increase when the outlier is present, highlighting its sensitivity to large errors.

Which Metric Should You Choose?

The choice between MAE, MSE, and RMSE depends on your specific goals and how you want to treat errors:

• Choose MAE if:

  • You want a metric that is easy to interpret directly in the original units.
  • You don't want outliers to dominate the metric. You view an error of 10 as twice as bad as an error of 5.
  • You want a straightforward measure of the average error magnitude.

• Choose RMSE (or MSE) if:

  • Large errors are particularly undesirable in your application, and you want the metric to reflect this strongly. An error of 10 is much worse than twice as bad as an error of 5.
  • You want a metric that is sensitive to outliers, potentially highlighting models that produce occasional very bad predictions.
  • RMSE is generally preferred over MSE for reporting and interpretation because it is in the original units. MSE might be used more often internally within learning algorithms due to its mathematical properties (like being easily differentiable).

• Look at Both: It's often useful to examine multiple metrics. If your RMSE is significantly higher than your MAE, it could indicate the presence of large errors (outliers) that are inflating the RMSE value. Examining both can give you a more complete picture of your model's error distribution.

There isn't a single "best" error metric for all regression problems. Understanding their characteristics helps you select and interpret the metrics most relevant to assessing your model's performance for your specific needs.