The Problem of Vanishing Gradients

Training a Recurrent Neural Network (RNN) involves calculating the gradient of the loss function with respect to the network's weights. This gradient indicates how to adjust the weights to improve performance. Backpropagation Through Time (BPTT) is a method that achieves this by unrolling the network through time and applying the chain rule backward from the final time step to the first.

The vanishing gradient problem arises directly from the mechanics of this backward pass, particularly due to the repeated application of the chain rule over many time steps. To understand why, let's consider how the gradient information flows. The gradient of the loss $L$ with respect to the hidden state at an early time step $k$, denoted $\frac{\partial L}{\partial h_k}$, depends on the gradient at a later time step $t$, $\frac{\partial L}{\partial h_t}$. The connection is established through the chain rule, involving the derivatives of the hidden state transitions between steps $k$ and $t$:

$$\frac{\partial L}{\partial h_k} = \frac{\partial L}{\partial h_t} \frac{\partial h_t}{\partial h_k}$$

The term $\frac{\partial h_t}{\partial h_k}$ represents how a change in the hidden state at step $k$ affects the hidden state at step $t$. This itself is a product of intermediate Jacobians (matrices of partial derivatives) for each step from $k+1$ to $t$:

$$\frac{\partial h_t}{\partial h_k} = \frac{\partial h_t}{\partial h_{t-1}} \frac{\partial h_{t-1}}{\partial h_{t-2}} \dots \frac{\partial h_{k+1}}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}$$

Each intermediate Jacobian $\frac{\partial h_i}{\partial h_{i-1}}$ depends on the recurrent weight matrix $W_{hh}$ and the derivative of the activation function used in the RNN cell. For a simple RNN with activation function $f$, the hidden state update is $h_i = f(W_{hh}h_{i-1} + W_{xh}x_i + b_h)$. The derivative $\frac{\partial h_i}{\partial h_{i-1}}$ involves $W_{hh}^T$ multiplied by the element-wise derivative of the activation function $f'$.

Now, consider what happens if the "magnitude" (more formally, the largest singular value or spectral radius) of these Jacobian matrices is consistently less than 1. When we multiply these matrices together many times, as required when the time gap $t-k$ is large, the overall magnitude of the resulting product $\frac{\partial h_t}{\partial h_k}$ shrinks exponentially.

Think of it like repeatedly multiplying a number by 0.9.

• $0.9^1 = 0.9$
• $0.9^{10} \approx 0.35$
• $0.9^{50} \approx 0.005$
• $0.9^{100} \approx 0.000027$

The value rapidly approaches zero.

Activation functions like the hyperbolic tangent (tanh) or the sigmoid function, commonly used in early RNNs, have derivatives that are strictly less than 1 for most inputs (tanh's derivative is always $\leq 1$, and sigmoid's derivative is always $\leq 0.25$). When these small derivatives are multiplied repeatedly during the backward pass through many time steps, potentially combined with recurrent weight matrices $W_{hh}$ whose norms might also be less than 1 (perhaps due to initialization or regularization), the Jacobians $\frac{\partial h_i}{\partial h_{i-1}}$ often have magnitudes less than 1.

Illustration of gradient flow during BPTT. The gradient signal (red arrows, thickness indicating magnitude) diminishes as it propagates backward through time steps (from right to left). This decay is caused by repeated multiplication by Jacobian terms, often having magnitudes less than 1 due to activation function derivatives ($f'$) and weight matrices ($W_{hh}$).

The practical consequence is severe: the gradient signal originating from errors at late time steps becomes incredibly small, or "vanishes", by the time it reaches the network layers corresponding to early time steps. These near-zero gradients mean that the weights ($W_{hh}$, $W_{xh}$, $b_h$) responsible for processing the sequence at those early steps receive almost no update signal based on long-term outcomes.

Essentially, the network becomes unable to learn correlations between events separated by long intervals in the sequence. If correctly predicting the sentiment at the end of a long paragraph depends critically on a word used near the beginning, a simple RNN suffering from vanishing gradients will likely fail to capture this dependency. The initial word's influence on the final prediction error gets lost as the gradient signal fizzles out on its way back. This significantly undermines the primary motivation for using recurrent networks: to model temporal dependencies, including long-range ones.

Understanding this vanishing gradient phenomenon is fundamental because it directly motivated the development of more sophisticated recurrent architectures, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which we will explore in subsequent chapters. These architectures incorporate specific mechanisms, often called "gates", designed to regulate information flow and allow gradients to propagate more effectively over long durations, mitigating the vanishing gradient problem.