Time Series Forecasting Models

Recurrent Neural Networks, particularly LSTMs and GRUs, are well-suited for time series forecasting because they can capture temporal dependencies within the data. Unlike static regression models, RNNs process sequences step-by-step, maintaining an internal state (memory) that incorporates information from past observations. Building an effective forecasting model involves choosing the right input-output structure based on the specific prediction task.

Before feeding data into an RNN, time series data typically requires preprocessing, including normalization (scaling values to a specific range, like [0, 1] or standardizing to have zero mean and unit variance) and windowing. Windowing transforms the continuous time series into input-output pairs suitable for supervised learning. An input window consists of a fixed number of past time steps (the lookback period), and the corresponding output is the value(s) we want to predict.

Input Windowing for RNNs

Imagine a time series $x_1, x_2, x_3, \ldots, x_T$. To train an RNN, we create overlapping windows. If we choose a lookback window size of $W$, the first input sequence might be $(x_1, x_2, \ldots, x_W)$, the second $(x_2, x_3, \ldots, x_{W+1})$, and so on. The target for each input sequence depends on the forecasting task.

Creating input windows (size W=3) and corresponding single-step targets from a time series.

The input shape for RNN layers in most frameworks is (batch_size, time_steps, features). For a univariate time series (one value per time step), features is 1. The time_steps dimension corresponds to the window size $W$.

Many-to-One Architecture

This is the most common setup for predicting a single future value. The RNN (e.g., LSTM or GRU) processes the input sequence of length $W$. We typically only need the output or hidden state from the last time step of the RNN, as this state summarizes the information from the entire input window. A Dense layer is then added on top of this final RNN output to produce the single predicted value.

• Input: Sequence of length $W$, e.g., $(x_t, x_{t+1}, \ldots, x_{t+W-1})$.
• Target: Single value at the next time step, e.g., $x_{t+W}$.
• RNN Configuration: The RNN layer (LSTM/GRU) should have return_sequences=False (in Keras/TensorFlow) or only use the final hidden state (in PyTorch). This ensures only the output from the last time step is passed forward.
• Output Layer: A Dense layer with one unit (and typically a linear activation function for regression).

Many-to-One architecture for single-step time series forecasting. The RNN processes the input window, and only the final hidden state is used by the Dense layer to predict the next value.

Many-to-Many Architectures

Sometimes, we need to predict multiple future time steps. Let's say we want to predict the next $H$ steps (the prediction horizon).

• Input: Sequence of length $W$, e.g., $(x_t, x_{t+1}, \ldots, x_{t+W-1})$.
• Target: Sequence of length $H$, e.g., $(x_{t+W}, x_{t+W+1}, \ldots, x_{t+W+H-1})$.

There are a few ways to structure the model for this:

1. Vector Output / Single-Shot Forecasting:

   • Similar to Many-to-One, the RNN processes the entire input sequence of length $W$.
   • Only the final hidden state (or output) from the last time step is used.
   • The final Dense layer has $H$ output units, predicting all $H$ future steps simultaneously.
   • RNN Configuration: return_sequences=False (Keras/TF) or use only the final hidden state (PyTorch).
   • Output Layer: Dense(H).

   Many-to-Many (Vector Output / Single-Shot) architecture. The final RNN state feeds into a Dense layer that outputs the entire forecast horizon $H$ at once.

2. Sequence Output / Autoregressive Forecasting:

   • This approach often involves a Many-to-One model trained to predict just one step ahead ($H=1$).
   • To predict multiple steps, it operates iteratively:
     • Predict $x_{t+W}$ using $(x_t, \ldots, x_{t+W-1})$.
     • Use the predicted $\hat{x}_{t+W}$ as input for the next step. Predict $x_{t+W+1}$ using $(x_{t+1}, \ldots, x_{t+W-1}, \hat{x}_{t+W})$.
     • Continue this process $H$ times, feeding the previous prediction back into the input window.
   • Pros: Requires only a single-step prediction model.
   • Cons: Errors can accumulate over the prediction horizon, as predictions are based on previous predictions.

3. Sequence-to-Sequence (Encoder-Decoder):

   • Uses two RNNs: an encoder processes the input sequence $(x_t, \ldots, x_{t+W-1})$ and summarizes it into a context vector (usually the final hidden state). A decoder RNN takes this context vector and generates the output sequence $(\hat{x}_{t+W}, \ldots, \hat{x}_{t+W+H-1})$ step-by-step.
   • This is a more advanced architecture, often used when the input and output sequences have complex relationships or different lengths. It's introduced later in the context of sequence-to-sequence models. For standard forecasting, the first two methods are more common starting points.

Choosing the Right Architecture

• Single-step ahead prediction ($H=1$): Use the Many-to-One architecture. It's simpler and directly optimized for this task.
• Multi-step prediction ($H > 1$):
  • The Vector Output (Single-Shot) approach is often easier to implement and train initially. It directly optimizes for predicting the entire horizon.
  • The Autoregressive approach can sometimes perform well, especially if the single-step dynamics are well-captured, but be mindful of error accumulation. It might require careful tuning.
  • Use Sequence-to-Sequence models for more complex multi-step forecasting problems, especially if long horizons are needed or if auxiliary input features are involved at each prediction step.

Regardless of the architecture, remember that the input data needs to be appropriately windowed and normalized. The choice of lookback window $W$ and prediction horizon $H$ are hyperparameters that significantly impact performance and should be chosen based on the problem and potentially tuned using validation data.