Handling Variable Length Sequences

You've successfully converted your raw sequence data, like text sentences or time series readings, into numerical formats, perhaps sequences of integers representing word indices or floating-point measurements over time. However, you'll immediately notice a common characteristic: these sequences rarely have the same length. One sentence might have 10 words, another 25. A sensor reading might cover 50 time steps, while another covers 100.

This variability presents a practical challenge for training deep learning models, especially when using mini-batches for efficiency. Neural network frameworks like TensorFlow and PyTorch typically expect data within a batch to be organized into tensors. Tensors are multi-dimensional arrays that require consistent dimensions. If you try to stack sequences of different lengths directly into a batch, you'll encounter errors because the time step dimension wouldn't be uniform.

Imagine trying to create a matrix where each row represents a sequence. If the rows have different numbers of columns (time steps), it's not a valid rectangular matrix.

The Need for Uniform Length: Padding

The standard technique to address variable-length sequences is padding. Padding involves adding a special, pre-defined value (the "padding value") to shorter sequences until all sequences in a batch reach a common, fixed length. This fixed length is often referred to as maxlen.

Think of it like adding blank spaces to shorter lines of text so they all align to the same right margin.

Choosing the Padding Value: The padding value should be distinct from any real data values. For integer-encoded text data, 0 is a common choice, assuming your vocabulary mapping starts assigning indices from 1. If 0 is a valid token index in your vocabulary, you'll need to choose a different value or re-index your vocabulary. For numerical time series data that has been normalized (e.g., to have a mean of 0 and standard deviation of 1), a value far outside the typical data range (like -99) might be used, although masking (discussed next) often makes using 0.0 feasible even here. The important point is that the model should eventually learn to ignore this padding value.

Determining the Maximum Length (maxlen): How long should the padded sequences be? You have a couple of options:

1. Longest Sequence: Pad all sequences to the length of the longest sequence in your entire dataset (or at least the current batch). This ensures no information is lost due to truncation, but it can be computationally expensive and memory-intensive if there are extreme outliers (very long sequences).
2. Fixed Percentile: Choose a length based on the distribution of sequence lengths in your dataset, for instance, the 95th percentile length. This means sequences longer than this length will be truncated (cut short), losing some information, but it provides a balance between computational cost and data retention.
3. Pre-defined maxlen: Select a reasonable fixed length based on domain knowledge or resource constraints (e.g., 512 tokens for many NLP tasks). Sequences shorter than this are padded; longer ones are truncated.

The choice often depends on the specific task, the nature of the data, and available computational resources. Truncating very long sequences might be acceptable if the most important information usually occurs near the beginning or end.

Padding Strategies: Pre vs. Post

Once you've decided on maxlen and the padding value, you need to decide where to add the pads:

• Pre-padding: Add padding values at the beginning of the sequence.
• Post-padding: Add padding values at the end of the sequence.

Here's a visual comparison:

Example of pre-padding (adding zeros before) and post-padding (adding zeros after) applied to three integer-encoded sequences to achieve a uniform length of 5. Gray boxes indicate the added padding value (0).

The choice between pre- and post-padding can sometimes influence model performance, although the effect is often minor, especially when masking is used correctly.

• Post-padding is often the default in libraries like Keras. Intuitively, for standard RNNs, the final hidden state is sometimes used as a summary of the sequence. Post-padding ensures this final state corresponds to the last actual element of the original sequence, not a padding value.
• Pre-padding might be considered if you believe the most recent information (at the end of the original sequence) is most significant, as it ensures this information is processed last by the RNN, potentially having a stronger influence on the final hidden state.

In practice, it's usually best to start with the default (often post-padding) and only experiment if performance is unsatisfactory or if you have a strong theoretical reason to prefer one over the other for your specific task.

Similarly, you'll need to decide whether to truncate overly long sequences from the beginning (truncating='pre') or the end (truncating='post'). Post-truncation (removing elements from the end) is common, but pre-truncation might be better if the most recent data points are considered more important.

Implementation Notes

Most deep learning libraries provide convenient functions for padding and truncation. For example, in TensorFlow/Keras, the tf.keras.preprocessing.sequence.pad_sequences function handles this entire process:

# Example using TensorFlow/Keras
from tensorflow.keras.preprocessing.sequence import pad_sequences

sequences = [
    [12, 5, 23, 8],       # Length 4
    [7, 101, 15],         # Length 3
    [3, 9, 42, 11, 2]     # Length 5
]

# Define max length
maxlen = 5
padding_value = 0

# Post-padding (default) and post-truncation (default)
padded_sequences = pad_sequences(
    sequences,
    maxlen=maxlen,
    padding='post',      # 'pre' or 'post'
    truncating='post',   # 'pre' or 'post'
    value=padding_value
)

# padded_sequences will look like the "Post-Padding" example above
# [[ 12   5  23   8   0]
#  [  7 101  15   0   0]
#  [  3   9  42  11   2]]

PyTorch doesn't have a single exact equivalent built-in for sequences directly, but similar results can be achieved using tensor operations or utilities within libraries like torchtext or by manually padding sequences within a custom DataLoader collate function.

Padding solves the structural problem of fitting variable-length sequences into fixed-size tensors for batch processing. However, it introduces artificial padding values that don't represent real data. Feeding these padding values directly into an RNN as if they were genuine inputs can negatively impact learning. The next step is to tell the model to ignore these padded time steps, which is achieved through masking.