The Need for Sequence Models

Sequential data carries information not just in individual elements, but significantly in their order and relationships over time. Consider tasks like predicting the next word in a sentence, forecasting future stock prices based on past performance, or classifying the sentiment of a movie review. Understanding the context provided by preceding elements is essential in all these cases.

Now, let's think about standard feedforward neural networks, such as Multi-Layer Perceptrons (MLPs). These networks are powerful tools for many machine learning problems, like image classification or tabular data regression. They operate by taking a fixed-size input vector, passing it through one or more layers of interconnected neurons, and producing an output. Each input presented to the network is processed independently of any other input. If you feed an MLP two different data points, its processing of the second data point has no memory or influence from the first one.

This independent processing presents significant challenges when dealing with sequential data:

Handling Variable Lengths: Sentences can have different numbers of words, time series can span different durations, and audio clips vary in length. Standard feedforward networks require inputs to be of a fixed, predefined size. While techniques like padding or truncation can be used to force sequences into a fixed length, they are often suboptimal. Padding can introduce noise or require models to learn to ignore large portions of the input, while truncation discards potentially valuable information.
Capturing Temporal Dependencies: The core limitation is the lack of memory. A feedforward network processing the word "cloud" in the sentence "The cloud stores data" has no inherent mechanism to know that the preceding word was "The". It processes "cloud" in isolation. This makes it incredibly difficult, if not impossible, to capture relationships between elements that are distant in the sequence, known as long-range dependencies. Predicting the stock price for tomorrow requires looking at trends over the past days, weeks, or even months, not just yesterday's price in isolation.
Parameter Sharing Across Time: Imagine applying a standard MLP separately to each word in a sentence. The network would need separate parameters to learn features related to the first word, the second word, and so on. This is inefficient and doesn't allow the model to generalize patterns. We'd want a model to recognize a pattern or feature (like identifying a noun) regardless of whether it appears at the beginning, middle, or end of the sequence. Standard feedforward networks don't inherently share parameters across different positions in a sequence.

Comparison of processing sequences $x_1, x_2, x_3$ . A feedforward network processes each input independently. A sequence model maintains an internal state (memory, $s_t$ ) that influences the processing of the next input.

To effectively model sequences, we need architectures specifically designed to address these shortcomings. Sequence models are built with the following capabilities in mind:

Processing elements step-by-step: They consume input sequences one element (or chunk) at a time.
Maintaining an internal state (memory): They possess a mechanism to keep track of information from previous steps. This state acts as a summary of the sequence seen so far, influencing how future elements are processed.
Sharing parameters across time steps: The same set of weights is typically used to process the input at each step. This allows the model to learn patterns that are applicable anywhere in the sequence and significantly reduces the number of parameters compared to assigning unique weights for each time step.

These characteristics allow sequence models to handle variable-length inputs more naturally and, importantly, to capture the temporal dependencies that are fundamental to understanding sequential data.

Recurrent Neural Networks (RNNs), which we will begin studying in detail in the next chapter, are a foundational class of neural networks explicitly designed with these principles. They incorporate loops and maintain a hidden state, allowing information to persist across time steps, thereby directly addressing the limitations of standard feedforward models for sequential tasks.

Was this section helpful?

References

Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016 (MIT Press) - A comprehensive textbook providing a foundational understanding of deep learning, with a dedicated chapter that thoroughly explains the need for sequence models and the principles of recurrent neural networks.
Finding structure in time, Jeffrey L. Elman, 1990 Cognitive Science, Vol. 14 (Wiley) DOI: 10.1207/S15516709COG1402_1 - A foundational paper introducing recurrent neural networks (often called Elman networks), demonstrating their ability to learn and represent temporal dependencies in sequential data through an internal state mechanism.
CS224n: Natural Language Processing with Deep Learning, Diyi Yang, Tatsunori Hashimoto, 2025 (Stanford University) - Online course materials from a leading university, offering lectures and readings that clearly explain the limitations of feedforward networks for natural language processing and introduce the architecture and motivation behind sequence models like RNNs.
Learning long-term dependencies with gradient descent is difficult, Yoshua Bengio, Patrice Simard, Paul Rosen, 1993 Advances in Neural Information Processing Systems, Vol. 6 DOI: 10.5591/978-1-57766-004-8.735 - This paper identifies the vanishing and exploding gradient problems in recurrent neural networks, highlighting a challenge in learning long-term dependencies and influencing the development of more robust sequence models.