Need for Model Evaluation

Time series models like ARIMA and SARIMA are constructed to understand patterns such as trend, seasonality, and autocorrelation by fitting them to historical data. These models can be powerful tools for forecasting. However, merely building a model doesn't guarantee its usefulness. Determining if a complex SARIMA(1,1,1)(1,1,0,12) model performs better at predicting next month's sales than a simpler ARIMA(2,1,0) model, or even a naive forecast, requires careful assessment. This is where model evaluation becomes essential.

Think about it: the ultimate goal is usually to predict future values, values the model hasn't seen during its training process. A model might capture the historical data extremely well, fitting the training points almost perfectly, but fail miserably when asked to extrapolate into the future. This phenomenon, known as overfitting, occurs when a model learns the noise and specific quirks of the training data rather than the underlying signal. Evaluating a model only on the data it was trained on can be misleadingly optimistic.

Therefore, we need a structured way to assess how well our models are likely to perform on new, unseen data. Model evaluation serves several critical purposes:

1. Quantifying Predictive Accuracy: It provides objective measures of how close a model's forecasts are to the actual outcomes. Instead of relying on visual inspection alone, metrics give us concrete numbers to understand the magnitude and nature of forecast errors. Is the model typically off by 10 units or 100 units? Does it tend to over-predict or under-predict?
2. Comparing Different Models: You will often develop multiple candidate models. Perhaps different orders for ARIMA/SARIMA, or even entirely different modeling approaches. Evaluation metrics provide a basis for comparing these models head-to-head on a level playing field, helping you select the one that performs best for your specific forecasting task on unseen data.
3. Building Confidence and Informing Decisions: Forecasts often drive important decisions, such as inventory management, resource allocation, or financial planning. Rigorous evaluation helps build confidence in the chosen model's predictions. Understanding the model's expected error range allows decision-makers to account for uncertainty. A model with poorly understood or unevaluated performance introduces significant risk.
4. Identifying Model Deficiencies: Evaluation on unseen data can highlight problems that weren't apparent during model fitting. Consistently large errors, biased forecasts, or a degradation in performance over time can indicate issues like model misspecification, changing underlying patterns in the data, or the limitations of the chosen approach.

Without a formal evaluation process, selecting and deploying a forecasting model becomes guesswork. You might choose a complex model that performs worse than a simple one, or deploy a model that is unreliable for making future predictions. The subsequent sections in this chapter will equip you with the standard techniques and metrics used in time series forecasting to perform this essential evaluation, starting with how to properly split your data and then calculating metrics like $MAE$, $RMSE$, and using criteria like $AIC$ to guide your model selection process.