Common Sequence Modeling Tasks

Sequence models are designed to solve specific kinds of problems. Analyzing these common tasks provides context for the architectures and techniques studied later. Sequence modeling problems generally fall into a few primary categories, differing primarily in their input and output structures.

Sequence Prediction and Forecasting

Perhaps the most intuitive application is predicting what comes next in a sequence. Given a series of observations $x_1, x_2, ..., x_t$, the goal is to predict the value of $x_{t+1}$, or perhaps a series of future values $x_{t+1}, ..., x_{t+k}$.

"This category encompasses many applications:"

• Time Series Forecasting: Predicting future stock prices based on historical trends, forecasting weather patterns based on past meteorological data, or estimating energy demand based on previous consumption.
• Next-Step Prediction: In natural language processing (NLP), this could involve predicting the next word in a sentence for autocompletion or suggesting the next item a user might interact with in a recommendation system.

The core idea is learning the temporal dependencies within the sequence to make informed predictions about the future. Depending on whether you predict a single next step or multiple steps, this can be viewed as a "many-to-one" (predicting one step from many) or "many-to-many" (predicting multiple steps from many) problem structure.

Flow for predicting the next element ($x_4$) based on the preceding sequence ($x_1, x_2, x_3$). Each step processes an input and updates an internal state representing the sequence history.

Sequence Classification

In sequence classification, the goal is to assign a single categorical label to an entire input sequence. Instead of predicting the next element, we want to understand the overall meaning or property represented by the sequence as a whole.

Common examples include:

• Sentiment Analysis: Classifying a text review (sequence of words) as positive, negative, or neutral.
• Topic Identification: Determining the topic of an article or document based on its text.
• Activity Recognition: Identifying an activity (e.g., walking, running, sitting) based on a sequence of sensor readings from a wearable device.
• Spam Detection: Classifying an email based on the sequence of words or characters it contains.

Typically, the model processes the entire sequence $x_1, ..., x_T$ and then produces a single output classification $y$. This fits the "many-to-one" structure, where many inputs map to a single categorical output.

Flow for classifying an entire sequence ($x_1$ to $x_T$). Information is processed sequentially, and a final classification decision is made based on the aggregated information from the whole sequence.

Sequence Generation

Sequence generation involves creating new sequences that follow certain learned patterns. Unlike prediction, where the goal is usually the next step, generation often aims to produce longer, coherent sequences.

Applications include:

• Text Generation: Creating human-like text, such as writing assistance, generating dialogue for chatbots, or composing poetry or code.
• Music Generation: Composing novel musical pieces in a specific style.
• Image Captioning: Generating a textual description (sequence of words) for a given image (often treated as a single input context).

Generation models learn the probability distribution of sequences in the training data. They can then sample from this distribution to create new sequences. This can sometimes start from a "seed" input (like the start of a sentence or a musical phrase) and then continue generating elements one after another. This task can involve "one-to-many" (generating a sequence from a single starting point or context) or "many-to-many" structures (where generation proceeds iteratively, potentially based on previous inputs or outputs).

Flow for generating a sequence ($x_1, x_2, x_3, ...$). Starting from an initial signal, the model generates the first element ($x_1$). This output can then be fed back as input (dashed lines) to generate the next element ($x_2$), and so on.

Sequence-to-Sequence (Seq2Seq) Tasks

A more complex category involves transforming an input sequence into a different output sequence. Critically, the input and output sequences in these tasks often have different lengths and structures.

Examples are:

• Machine Translation: Translating a sentence from one language (input sequence) to another (output sequence).
• Summarization: Condensing a long document (input sequence) into a shorter summary (output sequence).
• Question Answering: Generating an answer (output sequence) based on a given context document and question (input sequences).

These tasks typically require more sophisticated architectures, often involving two main components: an "encoder" that processes the input sequence into a fixed-size representation, and a "decoder" that generates the output sequence from that representation. We will touch upon these architectures later in the course.

Understanding these different sequence modeling tasks is an important first step. The specific task you aim to solve will heavily influence the design of your model, the way you prepare your data, and how you evaluate performance. In the following chapters, we will begin exploring Recurrent Neural Networks (RNNs), a fundamental class of models specifically designed to handle the sequential nature of data involved in these tasks.