The Need for Sequence Awareness

Powerful techniques like TF-IDF represent text numerically. These methods often rely on the "bag-of-words" assumption, treating documents as unordered collections of words. While effective for certain tasks like document classification based on topic, this approach overlooks a fundamental property of language: its sequential nature. The order in which words appear is often critical to understanding the intended meaning.

Consider these simple sentences:

• "Dog bites man."
• "Man bites dog."

Using a basic bag-of-words representation, both sentences might look very similar. They contain the exact same words: {dog, bites, man}. However, the meaning is drastically different, and this difference depends entirely on the sequence of the words. A model that ignores order cannot distinguish between these two scenarios.

This isn't an isolated issue. Many important NLP tasks depend heavily on understanding the sequential structure of text:

• Machine Translation: Translating "Ich bin müde" (German) to "I am tired" (English) requires mapping sequences, as word order and structure differ between languages. Simply translating word-by-word often results in grammatically incorrect or nonsensical output.
• Sentiment Analysis: The sentiment expressed can change based on word order and context. Compare "This movie was good" versus "This movie was not good." The word "not" modifies the sentiment of "good," but its position is significant. A model needs to process the sequence to understand this negation.
• Part-of-Speech (POS) Tagging: Determining whether "book" is a noun or a verb often depends on the surrounding words. In "Please book the flight," "book" is a verb. In "I read the book," it's a noun. Sequence provides the necessary context.
• Named Entity Recognition (NER): Identifying entities like "New York City" requires recognizing that these three words together form a single entity, distinct from just "New" or "York" or "City" appearing separately.
• Language Modeling: The goal here is often to predict the next word in a sequence, given the preceding words. This inherently requires understanding the dependencies between words in order.

Frequency-based methods like TF-IDF excel at capturing what words are present and how important they are within a document or corpus, but they discard the where and when information encoded in the sequence. They essentially put all the words into a bag, shake it up, and count them. This process loses the grammatical structure and contextual relationships that sequence provides.

To handle tasks where order matters, we need models that can process input step-by-step, maintaining some form of memory or internal state that captures information about the preceding elements in the sequence. This ability to "remember" past information allows the model to make sense of the current input in the context of what came before.

This chapter introduces a class of models specifically designed for sequential data: Recurrent Neural Networks (RNNs) and their more advanced variants like LSTMs and GRUs. These models address the limitations of order-agnostic representations and provide the foundation for tackling complex language understanding tasks where sequence is indispensable. We will begin by examining the basic architecture of RNNs.