Sequence Generation Methods

Classifying or predicting values within existing sequences, Recurrent Neural Networks offer the capability to generate entirely new sequences that mimic the patterns learned from training data. This generative ability opens up applications ranging from creating plausible text and composing music to synthesizing realistic time series data.

The fundamental idea behind sequence generation using RNNs is surprisingly straightforward: we train the model to predict the next element in a sequence, given the elements that came before it. Once trained, we can use this predictive capability iteratively to construct new sequences.

The Core Mechanism: Predicting the Next Step

Imagine you have a sequence $x_1, x_2, ..., x_N$ . During training, we present the model with subsequences and ask it to predict the element immediately following that subsequence. For example, given $x_1, ..., x_t$ , the model learns to predict $x_{t+1}$ .

For discrete sequences (like text): This task is typically framed as a classification problem. If we are working at the character level, the model predicts the probability distribution over all possible characters in the vocabulary for the next time step. If working at the word level, it predicts probabilities over all words in the vocabulary. A softmax activation function is commonly used on the output layer for this purpose.
For continuous sequences (like time series): This is often framed as a regression problem. Given the previous values $x_1, ..., x_t$ , the model predicts the continuous value $\hat{x}_{t+1}$ . A linear activation function is typically used on the output layer.

The RNN's hidden state plays a critical role here, summarizing the information from the preceding elements ( $x_1, ..., x_t$ ) to make the prediction for $x_{t+1}$ . LSTMs and GRUs are particularly effective because their gating mechanisms allow them to capture longer-range dependencies, which are often essential for generating coherent sequences.

Generating Sequences Iteratively

Once the model is trained to predict the next element, we can generate new sequences step by step:

Provide a Seed: Start with an initial "seed" sequence. This could be a single element, a short sequence from the training data, or a custom prompt. For text generation, this might be the beginning of a sentence like "The weather today is".
Predict the Next Element: Feed the seed sequence into the trained RNN. The model outputs a prediction for the element that should follow. For classification (e.g., text), this output is a probability distribution over the vocabulary. For regression, it's a predicted value.
Select the Next Element:
- Regression: Use the predicted value directly.
- Classification: Choose the next element based on the probability distribution. We rarely just pick the single most likely element (greedy search), as this often leads to repetitive and predictable results. Instead, we use sampling strategies (discussed below).
Append and Repeat: Append the selected element to the current sequence. This new, longer sequence becomes the input for the next time step. Repeat steps 2 and 3 to generate the sequence element by element until a desired length is reached or a special end-of-sequence token is generated.

This iterative process allows the model to build sequences incrementally, with each new element conditioned on the sequence generated so far.

Iterative sequence generation process using an RNN. A seed sequence initializes the process, the RNN predicts the next element's probabilities, an element is sampled, and the sequence is extended to become the input for the next step.

Sampling Strategies for Discrete Sequences

When generating discrete sequences like text, choosing the next element from the predicted probability distribution is not trivial. Different strategies offer trade-offs between coherence, diversity, and predictability:

Greedy Search: Simply select the element with the highest probability at each step. This is deterministic and often results in repetitive or uninteresting sequences. It might get stuck in loops like "is is is is...".
Sampling with Temperature: This is a popular technique to control the randomness of the selection. Before applying the softmax function to the network's raw output scores (logits, $z_i$ $z_{i}$ ), we divide the logits by a temperature value ( $T$ $T$ ): $P_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$ $P_{i} = \frac{e x p ( z _{i} / T )}{\sum _{j} e x p ( z _{j} / T )}$
- $T = 1$ : Standard sampling according to the learned probabilities.
- $T < 1$ (e.g., 0.5): Makes the distribution "sharper". High-probability elements become even more likely, leading to more focused and predictable, potentially more coherent output, closer to greedy search.
- $T > 1$ (e.g., 1.2): Makes the distribution "flatter". Lower-probability elements become more likely, increasing randomness, novelty, and diversity, but potentially reducing coherence or grammatical correctness. Finding the right temperature often requires experimentation.
Top-k Sampling: Instead of considering all elements in the vocabulary, consider only the $k$ elements with the highest probability. Renormalize the probabilities among these top $k$ elements and sample from this reduced set. This prevents low-probability (often nonsensical) choices while still allowing for some variation.
Top-p (Nucleus) Sampling: A refinement of top-k. Instead of selecting a fixed number $k$ , select the smallest set of elements whose cumulative probability is greater than or equal to a threshold $p$ (e.g., $p=0.9$ ). Sample only from this "nucleus" of probable elements. This adapts the size of the sampling pool based on the model's certainty at each step. If the model is very confident (one word has very high probability), the nucleus is small; if the model is uncertain (probabilities are spread out), the nucleus is larger.

Character-Level vs. Word-Level Generation

For text generation, a significant choice is the level at which you operate:

Character-Level:
- The vocabulary consists of individual characters (letters, punctuation, spaces).
- Pros: Handles any word (no out-of-vocabulary issues), can generate novel spellings or word forms, smaller vocabulary size.
- Cons: The model needs to learn not only sentence structure but also word structure. Capturing long-range dependencies needed for coherent meaning is harder as the sequence length (in characters) becomes very large. Training can be computationally intensive.
Word-Level:
- The vocabulary consists of unique words encountered in the training data.
- Pros: Models language structure more directly (words are primary units of meaning), often generates more coherent text over longer spans, sequences are shorter (in words) than character sequences for the same text.
- Cons: Vocabulary size can become very large, leading to memory and computational challenges. Struggles with words not seen during training (Out-Of-Vocabulary or OOV problem), although techniques like using a special <UNK> (unknown) token or subword tokenization (like BPE or WordPiece) can mitigate this. Cannot invent entirely new words.

The choice depends on the specific task, dataset size, and desired output characteristics.

Practical Notes

Seed Selection: The initial seed sequence heavily influences the generated output's topic and style. Experimenting with different seeds is common.
Training Data: The model learns patterns from the training data. If you train on Shakespeare, it will generate text sounding like Shakespeare. If you train on code, it will generate code. The quality, size, and domain of the training data are critical.
Evaluation: Evaluating generated sequences can be subjective. While metrics like perplexity (measuring how well the model predicts a test set, covered in the next chapter) exist, human judgment is often needed to assess coherence, creativity, and relevance for tasks like story writing or dialogue generation.

Sequence generation showcases the power of RNNs to learn and reproduce complex temporal patterns. While the methods described here form the foundation, generating truly long-range coherent and contextually appropriate sequences often benefits from more advanced architectures like the Encoder-Decoder framework and Attention mechanisms, which are introduced briefly later in this chapter.

Was this section helpful?

References

Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Provides a comprehensive theoretical and practical introduction to recurrent neural networks, sequence modeling, and generative models, covering the foundational aspects of iterative sequence generation and common architectures like LSTMs and GRUs.
The Curious Case of Neural Text Degeneration, Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi, 2020 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1904.09751 - Introduces and analyzes various sampling strategies, including top-p (nucleus) sampling, for improving the quality and diversity of text generated by neural language models, addressing common issues like repetition in greedy decoding.
Long Short-Term Memory, Sepp Hochreiter, Jürgen Schmidhuber, 1997 Neural Computation, Vol. 9 (The MIT Press) DOI: 10.1162/neco.1997.9.8.1735 - Presents the seminal architecture of Long Short-Term Memory (LSTM) networks, which are core to the success of recurrent neural networks in modeling and generating sequences with long-range dependencies, as highlighted in the section.
CS224N: Natural Language Processing with Deep Learning (Course Materials), Diyi Yang, Tatsunori Hashimoto, 2025 (Stanford University) - Offers comprehensive lecture notes and assignments that cover recurrent neural networks, including LSTMs and GRUs, and practical aspects of sequence generation such as sampling strategies and character-level versus word-level models in the context of natural language processing.