To understand the motivation behind the Transformer architecture, we first revisit the primary methods previously used for sequence modeling. Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs) represented significant steps in handling sequential data. However, their inherent structure presents several limitations that hindered progress, especially with increasingly long sequences.

This chapter examines these specific challenges. We will discuss:

• The sequential processing nature inherent in recurrent models and its implications for computational efficiency.
• The mathematical difficulties associated with training deep recurrent networks, specifically the vanishing and exploding gradient problems (e.g., vanishing gradients where $\frac{\partial L}{\partial \theta_t} \approx 0$ for early timesteps $t$).
• How gating mechanisms in LSTMs and GRUs attempt to mitigate gradient issues.
• The persistent problems recurrent models face in effectively modeling dependencies across very long distances in the input sequence.

Recognizing these constraints provides the necessary background for appreciating the architectural innovations introduced by the Transformer model in subsequent chapters.