Challenges with Long-Range Dependencies

While LSTMs and GRUs introduced gating mechanisms to combat the severe vanishing gradient problem found in simple RNNs, they still face inherent difficulties in capturing relationships between elements that are very far apart in a sequence. This limitation stems directly from the sequential nature of processing information.

Think of the hidden state $h_t$ in an RNN as a running summary or memory of the sequence seen up to time step $t$. To influence the output or state at a much later time step $N$, information from an early time step $t$ must be successfully propagated through all intermediate steps $t+1, t+2, \dots, N-1$. This sequential path acts as a bottleneck.

Mathematically, this relates back to the propagation of gradients during backpropagation through time. The gradient of the loss $L$ with respect to an early hidden state $h_t$ depends on a product of Jacobian matrices representing the state transitions at each step:

$$\frac{\partial L}{\partial h_t} = \frac{\partial L}{\partial h_N} \left( \prod_{k=t+1}^{N} \frac{\partial h_k}{\partial h_{k-1}} \right)$$

Even with LSTMs and GRUs designed to keep these Jacobian norms closer to 1 on average, propagating information over a very large number of steps ($N-t$ being large) remains challenging. The gates provide control over information flow, allowing the network to potentially preserve important information over longer durations than a simple RNN. However, this control isn't perfect.

• Information Decay: Over many steps, even small amounts of information loss or transformation at each step can accumulate, making it difficult for the network to precisely recall or utilize information from the distant past. The state vector $h_t$ has a fixed size, forcing the network to compress an arbitrarily long history into a fixed-dimensional representation. This compression inevitably leads to information loss, particularly for details from much earlier in the sequence.
• Path Length: The number of computational steps required for information to travel between two points in the sequence is proportional to the distance between them. This means that learning dependencies between distant elements requires successful gradient propagation over correspondingly long paths, which remains difficult even with gating. LSTMs might forget information, even if it proves relevant later, or struggle to route specific pieces of information across thousands of steps without degradation.

Consider tasks where this limitation becomes apparent:

• Document Analysis: In processing a long article, connecting a concluding statement back to a premise introduced in the first paragraph requires maintaining that initial context over thousands of words.
• Machine Translation: Translating sentences where grammatical agreement or lexical choice depends on words far apart in the source sentence (e.g., subject-verb agreement across multiple clauses).
• Long-Form Question Answering: Answering a question whose answer requires synthesizing information scattered across a lengthy document.
• Code Generation: Maintaining context about variable declarations or function definitions made much earlier in a large code file.

The following diagram illustrates the sequential dependency path inherent in recurrent models. Information from Input 1 must pass through every intermediate hidden state to influence Output N.

The sequential path length in RNNs for information flow between time steps t and N is proportional to N-t.

While LSTMs and GRUs were significant improvements, this persistent difficulty in modeling very long-range dependencies directly motivated the development of architectures that could create shorter paths between distant sequence elements. The Transformer model, primarily through its self-attention mechanism, provides a way to directly model relationships between any two positions in the sequence, regardless of their distance, overcoming this fundamental limitation of recurrent processing. We will examine this mechanism in detail in the next chapter.