Text Generation Models

Text generation is a popular and intuitive application of recurrent neural networks. This technique trains a model to produce human-like text, character by character or word by word, based on patterns learned from a corpus of example text. Applications range from auto-completing sentences and generating code to creative writing assistance and chatbot responses.

The Core Principle: Predicting the Next Token

At its core, text generation using RNNs operates on a simple principle: predict the next element (a character or a word, often called a "token") in a sequence given the elements that came before it. The RNN's hidden state acts as a memory, summarizing the preceding sequence to inform the prediction for the next step.

Imagine you have the sequence "The quick brown fox jumps". The model's task, when fed this sequence, is to predict the next most likely token, which might be "over". We train the model by showing it enormous amounts of text and asking it, at each position, to predict the very next token that actually appears in the training data.

Model Architecture for Text Generation

A common setup for a text generation model involves these layers:

1. Input Layer: Takes sequences of token IDs (integers representing characters or words).
2. Embedding Layer: Converts these integer IDs into dense vector representations. This layer learns meaningful representations for each token in the vocabulary (as discussed in Chapter 8).
3. Recurrent Layer (LSTM/GRU): Processes the sequence of embeddings one step at a time, updating its hidden state. This layer captures the temporal dependencies in the text. Using LSTMs or GRUs is standard practice here to handle potentially long sequences and mitigate vanishing gradient issues. Set return_sequences=True if stacking recurrent layers or if subsequent layers need the full sequence output, but for basic generation predicting the next token, often only the final output is needed after processing the input sequence (though frameworks often handle this implicitly when predicting step-by-step).
4. Dense Output Layer: A fully connected layer with a number of units equal to the vocabulary size (number of unique characters or words). It takes the output from the RNN layer(s) at each time step.
5. Softmax Activation: Applied to the output layer to produce a probability distribution over the entire vocabulary. Each element in the output vector represents the probability of the corresponding token being the next token in the sequence.

The model is trained to minimize a loss function, typically categorical cross-entropy, which measures the difference between the predicted probability distribution and the actual next token (represented as a one-hot encoded vector) in the training data.

For example, if the input sequence is represented by token IDs [10, 34, 5] ("the cat sat") and the vocabulary size is 1000, the RNN processes these inputs. If the next actual word is "on" (ID 12), the model's output after processing "sat" should ideally be a probability distribution where the 12th element is close to 1, and all others are close to 0. The loss function encourages the model to learn weights that achieve this.

Generating New Text: The Sampling Process

Once the model is trained, we can use it to generate novel text. This is typically an iterative process:

1. Provide a Seed: Start with an initial sequence of text (a "seed" or "prompt"), like "The weather today is".
2. Preprocess: Convert the seed sequence into the numerical format the model expects (token IDs).
3. Predict: Feed the seed sequence into the trained model. The model outputs a probability distribution over the vocabulary for the next token.
4. Sample: Select the next token based on the predicted probabilities. This is a critical step where different strategies can be employed (discussed below).
5. Append: Add the chosen token to the end of the current sequence.
6. Iterate: Use the newly extended sequence as the input for the next prediction step. Repeat steps 3-5 to generate text token by token until a desired length is reached or a special end-of-sequence token is generated.

A view of the iterative text generation process using a trained RNN model.

Sampling Strategies

How we choose the next token (Step 4 above) significantly impacts the generated text's quality and style:

• Greedy Search: Simply choose the token with the highest probability at each step. This is deterministic and often leads to repetitive or predictable text.
• Temperature Sampling: This is a common technique to control the randomness of the output. Before applying softmax or sampling, the logits (the raw outputs of the Dense layer before softmax) are divided by a temperature value ($T$).
  • A low temperature ($T < 1$, e.g., 0.5) makes the distribution "peakier," increasing the probability of the most likely candidates. This approaches greedy search as $T \to 0$, resulting in more predictable, focused text.
  • A temperature of $T = 1$ uses the original probabilities.
  • A high temperature ($T > 1$, e.g., 1.2) flattens the distribution, making less likely tokens more probable. This leads to more surprising, random, and potentially creative (or nonsensical) text. The adjusted probabilities $p_i'$ are calculated from the original probabilities $p_i$ (derived from logits) as:
  $$p_i' = \frac{\exp(\log(p_i) / T)}{\sum_j \exp(\log(p_j) / T)}$$ We then sample from this modified distribution $p'$. Adjusting the temperature is a practical way to tune the trade-off between coherence and creativity.
• Top-k Sampling: Limit the sampling pool to the $k$ most likely tokens at each step, then redistribute the probability mass among them and sample. This avoids picking highly improbable tokens while still allowing for some variation.
• Nucleus (Top-p) Sampling: Select the smallest set of tokens whose cumulative probability exceeds a threshold $p$, then redistribute the probability mass among them and sample. This adapts the size of the sampling pool based on the shape of the probability distribution.

Character-Level vs. Word-Level Models

You can build text generation models at different granularities:

• Character-Level:
  • Vocabulary: Consists of individual characters (letters, punctuation, spaces, etc.). Typically small (e.g., < 100).
  • Pros: Can invent new words, handles typos or rare words naturally, captures fine-grained stylistic elements (like punctuation patterns).
  • Cons: Needs to model much longer sequences to capture meaning, computationally more intensive to generate meaningful text, can sometimes produce syntactically invalid words. Requires the RNN to learn word structure from scratch.
• Word-Level:
  • Vocabulary: Consists of unique words in the training corpus. Can be large (tens or hundreds of thousands). Often requires handling unknown words (using an <UNK> token) or subword units (like BPE or WordPiece).
  • Pros: Models semantics more directly, often requires shorter sequences to generate coherent text, computationally less intensive per semantic unit.
  • Cons: Cannot generate words outside its vocabulary (unless using subword techniques), vocabulary size can become very large, sensitive to how the vocabulary is built.

The choice between character and word-level (or subword-level) generation depends on the specific application, the nature of the training data, and computational resources.

Setting up and training these models involves careful data preparation (tokenization, creating input/target pairs often using sliding windows over the text) and leveraging the RNN implementations (LSTM, GRU layers) provided by deep learning frameworks, as explored in previous chapters. The generation step then uses the trained model in an inference loop with a chosen sampling strategy.