The Vanishing and Exploding Gradient Problems

While the sequential nature of RNNs allows them to process sequences step-by-step, it creates significant hurdles during training, particularly for deep networks or long sequences. The primary mechanism for training RNNs is Backpropagation Through Time (BPTT), which essentially unfolds the recurrent network across the sequence length and applies the standard backpropagation algorithm. This process involves calculating gradients of the loss function with respect to the network's parameters by repeatedly applying the chain rule backward through each time step.

Consider the calculation of the gradient of the loss $L$ at the final time step $T$ with respect to the hidden state $h_k$ at some earlier time step $k$. Using the chain rule, this involves multiplying Jacobians across intermediate time steps:

$$\frac{\partial L}{\partial h_k} = \frac{\partial L}{\partial h_T} \frac{\partial h_T}{\partial h_{T-1}} \frac{\partial h_{T-1}}{\partial h_{T-2}} \dots \frac{\partial h_{k+1}}{\partial h_k}$$

This equation reveals the core issue. The term $\frac{\partial h_t}{\partial h_{t-1}}$ represents how the hidden state at time $t$ changes with respect to the hidden state at time $t-1$. This Jacobian depends on the recurrent weight matrix $W_{hh}$ and the derivative of the activation function used in the recurrent transition. The gradient signal from the final loss must propagate backward through a product of these Jacobian matrices.

Vanishing Gradients

If the norms (or more formally, singular values) of these Jacobian matrices $\frac{\partial h_t}{\partial h_{t-1}}$ are consistently less than 1, their product will shrink exponentially as it propagates backward through time ($T-k$ steps).

$$\frac{\partial L}{\partial h_k} \approx \frac{\partial L}{\partial h_T} \prod_{t=k+1}^{T} J_t \quad \text{where } J_t = \frac{\partial h_t}{\partial h_{t-1}}$$

If $||J_t|| < 1$ on average, then $|| \prod_{t=k+1}^{T} J_t ||$ approaches zero very quickly as $T-k$ increases. This means the gradient signal effectively vanishes before reaching the earlier time steps.

Consequences:

• Difficulty Learning Long-Range Dependencies: The network struggles to learn correlations between inputs and outputs that are far apart in the sequence. The weights responsible for capturing these long-term effects (primarily influenced by gradients from early time steps like $k$) receive negligible updates.
• Slow Training: Parameters associated with the beginning of the sequence learn much slower than those near the end.

The choice of activation functions like hyperbolic tangent (tanh) or sigmoid, commonly used in older RNNs, exacerbates this problem because their derivatives are strictly less than 1 (except at a single point for tanh).

Exploding Gradients

Conversely, if the norms of the Jacobian matrices $\frac{\partial h_t}{\partial h_{t-1}}$ are consistently greater than 1, their product can grow exponentially large as it propagates backward.

Consequences:

• Training Instability: The gradients become excessively large, leading to massive updates to the weights. This can cause the optimization process to diverge, often resulting in numerical overflow (NaNs, or "Not a Number" values) in the computations.
• Destructive Updates: Even if the training doesn't completely diverge, large gradient updates can erase previously learned knowledge.

While exploding gradients are often easier to detect and mitigate (e.g., using gradient clipping, where gradients exceeding a certain threshold are scaled down), they still pose a significant challenge to stable training.

The chart shows how repeated multiplication of values slightly less than 1 (blue line, norm 0.8) leads to exponential decay (vanishing), while values slightly greater than 1 (red line, norm 1.2) lead to exponential growth (exploding) as gradients propagate backward through time steps. The y-axis is clipped to visualize both trends.

These gradient problems fundamentally limit the ability of simple RNN architectures to model sequences effectively, especially when long-term patterns are present. This limitation was a major driving force behind the development of more sophisticated recurrent units like LSTMs and GRUs, which we examine next.