Recurrent VAEs (RVAEs) for Time Series Modeling

Standard VAEs excel when data samples are independent. However, time series data, with its inherent temporal dependencies, requires a different approach. Recurrent Variational Autoencoders (RVAEs) are an adaptation of the VAE framework specifically designed to model and generate sequential data like time series. By integrating Recurrent Neural Networks (RNNs) into their architecture, RVAEs can capture the dynamics and ordering important for understanding series data.

The Role of Recurrence in VAEs for Time Series

Vanilla VAEs typically process fixed-size inputs and assume independence between data points. This is a significant limitation when dealing with sequences, where the order of elements and their relationships over time are fundamental. Time series data, for example, often exhibits:

• Variable Lengths: Different time series instances can have different durations.
• Temporal Dependencies: The value at a given time step often depends on previous values.
• Long-Range Patterns: Important correlations might exist between distant points in a sequence.

Recurrent Neural Networks (RNNs), such as LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units), are purpose-built to handle such sequential characteristics. They maintain an internal state that evolves over time, allowing them to "remember" information from past elements in the sequence when processing the current one. By incorporating RNNs into the encoder and decoder of a VAE, we create an RVAE capable of learning meaningful representations from, and generating, entire sequences.

RVAE Architecture: Encoding and Decoding Sequences

An RVAE typically employs RNNs for both its encoder and decoder components. Let's break down the typical data flow:

1. Encoder: An RNN (e.g., LSTM or GRU) processes the input time series $x = (x_1, x_2, \ldots, x_T)$ step-by-step. At each time step $t$, the RNN takes $x_t$ and its previous hidden state $h_{t-1}^{enc}$ to produce a new hidden state $h_t^{enc}$. The final hidden state $h_T^{enc}$ (or a function of all hidden states) is then used to parameterize the approximate posterior distribution $q_\phi(z|x)$. This distribution is usually a Gaussian, so the encoder outputs the mean $\mu_z$ and log-variance $\log \sigma_z^2$ of the latent vector $z$.
2. Latent Space: A latent vector $z$ is sampled from $q_\phi(z|x)$ using the reparameterization trick: $z = \mu_z + \sigma_z \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$. This $z$ aims to capture a compressed, holistic representation of the entire input sequence.
3. Decoder: Another RNN takes the latent vector $z$ as input (often as its initial hidden state or concatenated to inputs at each step). It then generates the output sequence $\hat{x} = (\hat{x}_1, \hat{x}_2, \ldots, \hat{x}_T)$ one step at a time. At each generation step $t$, the decoder RNN produces $\hat{x}_t$ based on its current hidden state $h_t^{dec}$ (and potentially the previously generated element $\hat{x}_{t-1}$).

The following diagram illustrates this general RVAE architecture:

An RVAE processes an input sequence with an RNN encoder to produce latent parameters, samples a latent vector $z$, and then uses an RNN decoder conditioned on $z$ to reconstruct or generate a sequence.

The objective function for training an RVAE remains the Evidence Lower Bound (ELBO), as derived in Chapter 2:

$$\mathcal{L}(x, \theta, \phi) = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) || p(z))$$

The main difference lies in how $p_\theta(x|z)$ (the reconstruction term) and $q_\phi(z|x)$ (the approximate posterior) are defined.

• The decoder defines the likelihood $p_\theta(x|z)$. Since it's an RNN generating a sequence, this is typically factorized autoregressively: $p_\theta(x|z) = \prod_{t=1}^T p_\theta(x_t | x_{<t}, z)$. Each $p_\theta(x_t | x_{<t}, z)$ is the output of the decoder RNN at step $t$.
• The encoder defines $q_\phi(z|x)$. As mentioned, this is often $q_\phi(z | h_T^{enc})$, where $h_T^{enc}$ is the final hidden state of the encoder RNN after processing the entire input sequence $x$. The prior $p(z)$ is usually a standard multivariate Gaussian, $p(z) = \mathcal{N}(0, I)$.

Important Design Choices for RVAEs

When implementing RVAEs for time series, several design choices can significantly impact performance:

• RNN Cell Type: LSTMs are generally preferred for their ability to capture longer-term dependencies, though GRUs offer a simpler architecture with fewer parameters and can be effective for many tasks.
• Encoder Representation: While using the last hidden state $h_T^{enc}$ of the encoder RNN is common, alternative strategies include using an attention mechanism over all encoder hidden states $\{h_1^{enc}, \ldots, h_T^{enc}\}$ to form a context vector that then parameterizes $z$. This can help if important information is distributed throughout the sequence rather than concentrated at the end.
• Feeding $z$ to the Decoder: The latent vector $z$ can be used to initialize the decoder RNN's hidden state. Alternatively, $z$ can be concatenated to the input of the decoder at every time step, providing constant conditioning information throughout the generation process.
• Teacher Forcing: During training, it's common to use "teacher forcing" for the decoder. This means that at each step $t$, the decoder receives the true ground truth $x_t$ as input (or $x_{t-1}$ to predict $x_t$), rather than its own previously generated sample $\hat{x}_{t-1}$. This stabilizes training but can lead to a discrepancy between training and inference (when ground truth is unavailable). Scheduled sampling can help bridge this gap.
• Handling Variable Lengths: Padding sequences to a maximum length or using bucketing techniques are common ways to handle variable-length sequences in mini-batches. Masking should be applied to the loss function to ignore contributions from padded time steps.

Training Dynamics and Challenges

Training RVAEs involves backpropagation through time (BPTT) for the RNN components, combined with the reparameterization trick for the latent variable $z$. Some common challenges include:

• Vanishing/Exploding Gradients: LSTMs and GRUs are designed to mitigate these issues, but they can still occur, especially with very long sequences. Gradient clipping is a standard countermeasure.
• Posterior Collapse: This is a known issue in VAEs where the latent variable $z$ is ignored by the decoder ($KL(q_\phi(z|x) || p(z)) \approx 0$), and the decoder generates outputs primarily based on its autoregressive properties. This can happen if the decoder is too powerful (e.g., a very deep LSTM) or if the KL-divergence term in the ELBO is weighted too heavily early in training. Techniques like KL annealing (gradually increasing the weight of the KL term) can help.
• Computational Cost: Training RNNs on long sequences can be computationally intensive and slow due to the sequential nature of BPTT.

Applications of RVAEs in Time Series Analysis

RVAEs offer a versatile framework for various time series tasks:

• Generative Modeling and Forecasting: After training, an RVAE can generate new, synthetic time series by sampling $z$ from the prior $p(z)$ and decoding it. For forecasting, given an initial segment of a time series, one can encode it, potentially sample multiple $z$'s, and then decode to get probabilistic forecasts.
• Anomaly Detection: Time series that are difficult for the RVAE to reconstruct (i.e., have a high reconstruction error) or map to an unusual region in the latent space (low probability under the prior $p(z)$) can be flagged as anomalous.
• Learning Smooth Representations: The latent space $z$ provides a compressed, often smoother, representation of the input time series. These representations can be useful for downstream tasks like clustering or classifying time series.
• Data Imputation/Denoising: RVAEs can be trained to reconstruct clean sequences from noisy or incomplete ones, effectively filling in gaps or removing noise.

Example: Modeling Sinusoidal Patterns

Consider an RVAE trained on a dataset of various sinusoidal waves, each with potentially different frequencies, amplitudes, and phases. The RVAE would learn to:

1. Encode: The RNN encoder would process an input sine wave and map it to a latent vector $z$. Different properties of the sine wave (like frequency and phase) might be captured along different dimensions of $z$.
2. Decode: The RNN decoder, given a $z$, would generate a sequence of points that form a sine wave corresponding to the properties encoded in $z$.

The chart below shows an example of an original signal, its reconstruction by an RVAE, and a new signal generated by sampling from the RVAE's latent space.

Comparison of an original time series signal, its reconstruction by an RVAE, and a novel sample generated from the RVAE's learned latent space.

Strengths and Limitations

RVAEs bring significant advantages to time series modeling:

• Probabilistic Framework: They provide a principled way to model uncertainty and generate diverse samples.
• Temporal Dynamics: RNNs allow them to capture complex dependencies over time.
• Generative Power: They can synthesize realistic-looking time series data.

However, they also have limitations:

• Training Complexity: As mentioned, training can be slow and prone to issues like posterior collapse.
• Long-Range Dependencies: While LSTMs/GRUs help, exceptionally long-range dependencies (spanning hundreds or thousands of time steps) can still be challenging to capture perfectly. Attention mechanisms, which we will discuss next, can further improve this.
• Interpretability: While $z$ provides a representation, directly interpreting what each latent dimension has learned about the time series can be difficult.

RVAEs represent a powerful extension of VAEs for sequential data. By understanding their architecture, training considerations, and application areas, you can effectively leverage them for a variety of time series modeling problems. Later in this chapter, we will also explore how RVAEs relate to classical state-space models, providing another perspective on their capabilities for dynamic systems.