Before we can measure how well our forecasting models perform, we need a reliable way to separate the data used for building the model (training) from the data used for testing its predictive accuracy. For many machine learning tasks, you might randomly shuffle your data and split it into training and testing sets. However, this approach is fundamentally flawed for time series analysis.

The Problem with Random Splitting

Time series data has an inherent temporal order. Each observation depends on previous observations. Randomly shuffling the data breaks this crucial sequence, leading to several problems:

Data Leakage: If you randomly assign data points, your training set might contain observations from the future relative to points in your test set. The model could inadvertently learn from future information it shouldn't have access to during a real forecasting scenario, leading to overly optimistic performance estimates. Imagine training a stock price predictor using data from next week to predict this week. It might look great in testing, but it's useless in practice.
Loss of Temporal Structure: Models like ARIMA and SARIMA explicitly rely on the sequence and time dependencies (like autocorrelation) within the data. Shuffling destroys this structure, making it impossible for these models to learn meaningful patterns.

Therefore, evaluating time series models requires a different approach that respects the chronological order of the data.

Temporal Splitting: Maintaining Order

The standard method for splitting time series data is to use a cutoff point in time. All data before this point becomes the training set, and all data after this point forms the test set (also sometimes called the hold-out set).

Training Set: Used to fit the parameters of your forecasting model (e.g., determining the $p, d, q$ orders for ARIMA).
Test Set: Used to evaluate the fitted model's performance on unseen future data. You compare the forecasts generated by the model (trained only on the training data) against the actual values in the test set.

This chronological split mimics a real-world forecasting situation where you build a model based on past data and use it to predict future outcomes.

Implementing a Temporal Split in Pandas

Let's assume you have your time series data loaded into a Pandas DataFrame or Series with a DatetimeIndex. Splitting it chronologically is straightforward. You can either choose a specific date or use an integer index location based on the desired split percentage.

import pandas as pd

# Assume 'series' is your pandas Series with a DatetimeIndex
# Example:
# index = pd.date_range(start='2020-01-01', periods=100, freq='D')
# data = range(100)
# series = pd.Series(data, index=index)

# Method 1: Splitting by Date
cutoff_date = '2020-03-11' # Example cutoff date
train_data = series.loc[series.index < cutoff_date]
test_data = series.loc[series.index >= cutoff_date]

# Method 2: Splitting by Position (e.g., 80% train, 20% test)
split_point = int(len(series) * 0.8)
train_data_pos = series.iloc[:split_point]
test_data_pos = series.iloc[split_point:]

print(f"Original series length: {len(series)}")
print(f"Train set length (Date split): {len(train_data)}")
print(f"Test set length (Date split): {len(test_data)}")
print(f"Train set length (Position split): {len(train_data_pos)}")
print(f"Test set length (Position split): {len(test_data_pos)}")

Here's a visual representation of this split:

Example of splitting a time series into training (green) and testing (orange) sets using a cutoff date. The model is trained on data before the vertical dashed line and evaluated on data after it.

Choosing the Split Point and Test Set Size

How much data should you reserve for the test set? There's no single perfect answer, but here are some guidelines:

Forecast Horizon: The test set should be at least as long as the maximum period you intend to forecast. If you need to predict 12 months ahead, your test set should span at least 12 months.
Seasonality: If your data has seasonality (e.g., yearly patterns), the test set should ideally cover at least one full seasonal cycle to ensure the model's ability to capture it is properly tested.
Data Availability: Common splits range from 70/30 (train/test) to 90/10. With very large datasets, a smaller percentage for the test set might still provide a sufficient number of observations. For smaller datasets, you need to ensure the training set is large enough to reliably estimate model parameters.
Stability: Ensure the patterns (trend, seasonality) observed in the test set are reasonably representative of what was seen in the training set, unless you are specifically testing the model's robustness to changing patterns.

Walk-Forward Validation (Rolling Forecast Origin)

A simple train-test split evaluates the model on only one specific period. A more rigorous approach is walk-forward validation, also known as evaluation on a rolling forecast origin. This method provides a more robust estimate of how the model is likely to perform over time in a real deployment scenario.

The process works iteratively:

Select an initial training period (e.g., the first 70% of the data).
Train the model on this initial period.
Forecast the next single time step (or multiple steps, depending on your needs).
Record the forecast and the actual value for that time step.
Expand the training data to include the actual value from the step you just predicted.
Re-train the model (or efficiently update it, if the model allows) using the expanded training set.
Repeat steps 3-6, moving the forecast origin forward one step at a time until you have forecasts covering the desired evaluation period (the equivalent of the test set in the simple split).

Flow of walk-forward validation. The model is repeatedly trained on expanding data and tested on the immediately following time step.

Benefits:

Provides multiple forecast error measurements across different time points, giving a better sense of the model's consistency.
More closely simulates how a model would be used in production, where it's periodically retrained with new data.

Drawbacks:

Computationally more expensive, as the model needs to be re-estimated multiple times.

Libraries like scikit-learn offer tools like TimeSeriesSplit that can help automate the generation of indices for walk-forward validation, although you still need to implement the model fitting and prediction loop yourself.

Properly splitting your time series data is a non-negotiable first step in reliable model evaluation. By respecting the temporal order, either through a simple chronological split or more advanced walk-forward validation, you ensure that your evaluation metrics reflect the model's true forecasting capability on unseen future data. This sets the stage for calculating and interpreting metrics like MAE, RMSE, and others, which we will cover next.