Metrics for Sequence Prediction

When your sequence model aims to predict future values, like forecasting stock prices, predicting energy consumption, or estimating sensor readings over time, you're dealing with a sequence prediction task. Unlike classification, where you predict discrete categories, here the goal is typically to predict continuous values. Evaluating these models requires metrics that quantify the difference between the predicted sequence and the actual sequence.

Standard regression metrics are commonly adapted for this purpose. They measure the average error across all the time steps in your prediction horizon. Let's look at the most frequently used ones.

Mean Absolute Error (MAE)

The Mean Absolute Error measures the average magnitude of the errors between predicted and actual values, regardless of their direction. It's the average of the absolute differences over the entire sequence (or set of sequences).

For a single sequence with $T$ time steps, where $y_t$ is the actual value and $\hat{y}_t$ is the predicted value at time step $t$, the MAE is calculated as:

$$MAE = \frac{1}{T} \sum_{t=1}^{T} |y_t - \hat{y}_t|$$

Interpretation: MAE gives you the average error in the original units of your data. If you're predicting temperature in Celsius, the MAE tells you the average absolute temperature difference between your predictions and the actual readings.

Properties:

• Intuitive: Easy to understand and explain. The error is directly interpretable in the units of the target variable.
• Reliable to Outliers: Since it uses the absolute difference, it doesn't heavily penalize large individual errors (outliers) as much as squared-error metrics do. This can be beneficial if occasional large errors are expected or acceptable.

Mean Squared Error (MSE)

The Mean Squared Error measures the average of the squares of the errors. It places a higher penalty on larger errors due to the squaring operation.

The formula for MSE is:

$$MSE = \frac{1}{T} \sum_{t=1}^{T} (y_t - \hat{y}_t)^2$$

Interpretation: MSE is measured in the square of the original units (e.g., degrees Celsius squared), which makes it less intuitive to interpret directly compared to MAE or RMSE. However, it's widely used, partly because the squared term makes it mathematically convenient for optimization algorithms (like gradient descent) as it results in a smooth, differentiable loss function.

Properties:

• Penalizes Large Errors: Squaring the error term means that larger errors contribute disproportionately more to the total MSE. This makes it suitable when large deviations from the actual values are particularly undesirable.
• Mathematically Convenient: The function is continuously differentiable, simplifying its use in gradient-based optimization.
• Sensitive to Outliers: The squaring effect also makes MSE more sensitive to outliers than MAE. A single large error can significantly inflate the MSE.

Root Mean Squared Error (RMSE)

The Root Mean Squared Error is simply the square root of the MSE. Taking the square root brings the error metric back into the original units of the target variable, making it more interpretable than MSE.

The formula for RMSE is:

$$RMSE = \sqrt{MSE} = \sqrt{\frac{1}{T} \sum_{t=1}^{T} (y_t - \hat{y}_t)^2}$$

Interpretation: Like MAE, RMSE is expressed in the same units as the target variable (e.g., degrees Celsius). It represents a sort of "typical" magnitude of the error.

Properties:

• Interpretable Units: Shares the interpretability advantage of MAE.
• Penalizes Large Errors (like MSE): Retains the characteristic of penalizing larger errors more heavily than MAE because it's derived from the squared errors.
• Commonly Used: Often preferred over MSE for reporting performance because its units are more intuitive, while still reflecting the sensitivity to large errors inherent in MSE.

Here's a simple visualization comparing actual values with predictions from two models.

Comparison of actual time series data with predictions from two models. Model A generally tracks the actual values closely, resulting in lower MAE and RMSE. Model B exhibits larger deviations, leading to higher error metrics.

Mean Absolute Percentage Error (MAPE)

Sometimes, you might be interested in the error relative to the actual value. The Mean Absolute Percentage Error expresses the average error as a percentage of the actual values.

$$MAPE = \frac{100\%}{T} \sum_{t=1}^{T} \left| \frac{y_t - \hat{y}_t}{y_t} \right|$$

Interpretation: MAPE gives a sense of the average error percentage. A MAPE of 10% means the predictions are, on average, about 10% off from the actual values.

Properties:

• Relative Error: Useful when the scale of the data varies significantly or when you want a scale-independent measure of error.
• Potential Issues:
  • It's undefined or infinitely large if any actual value $y_t$ is zero.
  • It can yield skewed results if actual values are very close to zero.
  • It puts a heavier penalty on predictions that are higher than the actual value compared to predictions that are lower by the same absolute amount (asymmetric).

Choosing the Right Metric

The choice between these metrics depends on the specific goals and characteristics of your prediction problem:

• Use MAE if you want an easily interpretable error metric in the original units and if you want to treat all errors linearly, without giving extra weight to outliers.
• Use MSE primarily as a loss function during training because of its mathematical properties. Be aware of its sensitivity to outliers and its units (squared).
• Use RMSE if you want an error metric in the original units (like MAE) but still want to penalize larger errors more significantly (like MSE). It's often a good balance for reporting performance.
• Use MAPE when relative error is more significant than absolute error and when you are confident that the actual values will not be zero or very close to it.

In practice, you'll often monitor several of these metrics. For instance, you might optimize your model using MSE as the loss function during training but report both RMSE and MAE to provide a more complete picture of the model's performance. When dealing with sequences, these metrics are typically averaged across all time steps in the evaluation sequences and potentially across all sequences in the evaluation dataset.