Text Classification Models

Recurrent Neural Networks, particularly LSTMs and GRUs, are architected for text classification tasks. These models take a variable-length sequence of text (like a product review, news headline, or email) and assign it to one or more predefined categories (like positive/negative sentiment, topic label, or spam/not spam).

Recurrent models excel at reading text sequentially, updating their internal state (the hidden state) at each word or token. This hidden state acts as a running summary of the sequence processed so far. For classification, we need a mechanism to condense this evolving summary into a single fixed-size representation that can be fed into a standard classification layer (like a Dense layer with a softmax or sigmoid activation).

Using the Final Hidden State

The most straightforward and common approach is to use the hidden state of the RNN after it has processed the entire sequence. The intuition is that this final hidden state, $h_T$ (where $T$ is the length of the sequence), encapsulates the meaning or essence of the complete text relevant to the classification task.

Imagine reading a movie review. As you read word by word, your understanding evolves. By the time you reach the last word, you generally have a good sense of the overall sentiment. The final hidden state of an RNN aims to capture this final understanding numerically.

To implement this, you configure your RNN layer (LSTM or GRU) to only return the output corresponding to the last time step. In frameworks like TensorFlow/Keras, this is often the default behavior or can be explicitly set (e.g., return_sequences=False). This final hidden state vector is then passed directly into one or more Dense layers. The final Dense layer will have units equal to the number of classes and an appropriate activation function (e.g., sigmoid for binary or multi-label classification, softmax for multi-class classification).

Pooling RNN Outputs

An alternative strategy involves using the hidden states from all time steps, not just the final one. This requires configuring the RNN layer to return the full sequence of hidden states (e.g., return_sequences=True in Keras). Since the subsequent Dense layer expects a single fixed-size vector per input sequence, we need to aggregate these time-step outputs. Common pooling methods include:

1. Average Pooling: Calculate the element-wise average of all the hidden state vectors across the time steps. This gives a summary representation that considers the contribution of every part of the sequence equally. $$h_{avg} = \frac{1}{T} \sum_{t=1}^{T} h_t$$
2. Max Pooling: Take the element-wise maximum across all hidden state vectors. This approach focuses on the most prominent features detected by the RNN at any point in the sequence. It can be effective if the classification depends on specific keywords or phrases that strongly signal the class. $$h_{max} = \max_{t=1..T} (h_t) \quad \text{(element-wise max)}$$

The choice between using the final state or a pooling strategy often depends on the specific nature of the text and the task. For sentiment analysis, the concluding phrases might be very important, favoring the final state. For topic classification, relevant keywords might appear anywhere, potentially favoring pooling. You might need to experiment to find the best approach for your specific problem.

Handling Padding with Masking

Remember from Chapter 8 that we often pad sequences to ensure they have the same length within a batch. When using pooling (or even sometimes implicitly with the final state depending on the framework), it's important that these padding tokens don't influence the final representation. This is where masking comes in. By providing a mask to the RNN and subsequent pooling layers, we instruct them to ignore the outputs corresponding to the padded time steps during computation (like averaging or finding the maximum). Most deep learning frameworks handle masking automatically if you use their standard embedding layers with mask_zero=True and compatible RNN/pooling layers.

The Role of Bidirectional RNNs

For many text classification tasks, understanding the context requires looking both backward and forward in the sequence. For example, in the sentence "The movie was not bad, actually it was great!", the word "not" initially suggests negativity, but the later context reverses this.

A Bidirectional RNN (Bi-LSTM or Bi-GRU) processes the input sequence in two directions: one forward pass from start to end, and one backward pass from end to start. The hidden states from both passes at each time step are typically concatenated (or sometimes summed or averaged).

• When using the final hidden state approach with a Bi-RNN, you usually concatenate the final hidden state from the forward pass (which processed the sequence up to the last token) and the final hidden state from the backward pass (which processed the sequence starting from the last token, so its "final" state corresponds to the first token of the original sequence).
• When using pooling with a Bi-RNN, you apply the pooling operation (average or max) over the concatenated hidden states from all time steps.

Bidirectional processing often leads to better performance in text classification because it allows the model to capture context more effectively.

Typical Model Architecture

A common architecture for text classification using RNNs combines these elements:

1. Input Layer: Receives integer-encoded sequences.
2. Embedding Layer: Maps integers to dense vectors, potentially generating masks (mask_zero=True).
3. RNN Layer(s): One or more layers of LSTM or GRU, often bidirectional. Configured to return either the full sequence or just the final state.
4. Pooling Layer (Optional): If the RNN returns the full sequence, a GlobalAveragePooling1D or GlobalMaxPooling1D layer aggregates the outputs.
5. Dense Layer(s): One or more fully connected layers for classification, potentially with Dropout for regularization.
6. Output Layer: A final Dense layer with the appropriate number of units and activation function (sigmoid/softmax).

A typical architecture for RNN-based text classification. Input sequences are embedded, processed by a recurrent layer (often bidirectional), potentially pooled, and finally classified by dense layers.

By combining embeddings, powerful recurrent cells like LSTMs or GRUs, optional bidirectionality, and appropriate pooling or state selection, you can build effective models for a wide range of text classification problems. The specific choices, like using LSTM vs. GRU, pooling vs. final state, or adding bidirectional processing, are often determined through experimentation based on validation performance.