Understanding Bidirectional RNNs

Standard Recurrent Neural Networks, such as LSTMs and GRUs, process sequences in a single direction, typically chronologically from the beginning to the end. At any given time step $t$, the hidden state $h_t$ summarizes information from the past inputs $x_1, x_2, ..., x_t$. While this reflects how we often experience time-dependent phenomena, it can be limiting for certain tasks.

Consider understanding the meaning of a word in a sentence. Sometimes, the context needed to interpret a word correctly comes after it appears. For instance, in the sentence "He ate a bat with his baseball team," knowing the words "baseball team" helps disambiguate "bat" (likely a piece of sporting equipment, not the animal). A standard RNN processing this left-to-right would have already processed "bat" before seeing the clarifying context.

This is where Bidirectional Recurrent Neural Networks (BiRNNs) offer an advantage. The core idea is straightforward: process the sequence in both directions simultaneously using two separate recurrent layers.

1. Forward Layer: This layer processes the input sequence from the first element to the last ($t=1$ to $T$). Its hidden state at time $t$, denoted $\overrightarrow{h_t}$, captures information from the past context ($x_1, ..., x_t$).
2. Backward Layer: This layer processes the input sequence in reverse, from the last element to the first ($t=T$ down to $1$). Its hidden state at time $t$, denoted $\overleftarrow{h_t}$, captures information from the future context ($x_t, ..., x_T$).

These two layers operate independently, each maintaining its own set of weights and hidden states. They can be composed of simple RNN, LSTM, or GRU cells.

Architecture and Information Flow

At each time step $t$, the BiRNN produces an output that incorporates information from both the forward and backward passes up to that point. The most common way to combine the information from the two layers is by concatenating their respective hidden states at that time step.

The overall hidden state or output representation $y_t$ at time step $t$ can be formed as:

$$y_t = g([\overrightarrow{h_t} ; \overleftarrow{h_t}])$$

Here, $[\overrightarrow{h_t} ; \overleftarrow{h_t}]$ represents the concatenation of the forward hidden state $\overrightarrow{h_t}$ and the backward hidden state $\overleftarrow{h_t}$. The function $g$ could be an identity function (simply using the concatenated state directly), or it might involve further processing, like passing the concatenated vector through a dense layer, depending on the specific model architecture and task. Other combination methods like summation or averaging exist but are less common than concatenation.

The diagram below illustrates this structure:

A Bidirectional RNN processes the input sequence $x_1, ..., x_T$ using two independent recurrent layers. The forward layer computes hidden states $\overrightarrow{h_t}$ based on past information, while the backward layer computes $\overleftarrow{h_t}$ based on future information. The final output $y_t$ at each step combines both $\overrightarrow{h_t}$ and $\overleftarrow{h_t}$, often through concatenation.

Advantages and Disadvantages

The primary advantage of BiRNNs is their ability to incorporate context from both directions. This often leads to improved performance on tasks where the understanding of an element depends on its surrounding context. Examples include:

• Natural Language Processing: Sentiment analysis, named entity recognition (NER), part-of-speech tagging, and machine translation frequently benefit from bidirectional context.
• Speech Recognition: Understanding phonemes can depend on surrounding sounds.
• Bioinformatics: Analyzing sequences like DNA or protein structures.

However, BiRNNs also come with considerations:

• Dependency on Full Sequence: The backward pass requires processing the sequence from end to beginning. This means the entire input sequence must be available before computation can be completed. Consequently, BiRNNs are generally not suitable for real-time applications where predictions must be made incrementally as data arrives, without access to future inputs (e.g., real-time stock price prediction based only on past prices).
• Increased Computational Cost: Using two recurrent layers instead of one approximately doubles the number of parameters and the computational effort required during training and inference compared to a unidirectional RNN with the same hidden layer size and type.

When to Use Bidirectional Models

Choose a bidirectional architecture when:

1. The task involves processing entire sequences where context from both past and future elements is beneficial for making predictions or classifications at each step (e.g., sequence labeling like NER).
2. The task requires classifying or summarizing an entire sequence based on all its elements (e.g., sentiment analysis of a complete review).
3. The constraint of needing the complete sequence beforehand is acceptable for the application.

Avoid bidirectional architectures for tasks requiring true online processing or forecasting where future inputs are inherently unavailable at the time of prediction. In such cases, a standard unidirectional RNN is the appropriate choice.

Having understood the concept and utility of bidirectional processing, we will now look at how to implement both standard and bidirectional LSTM and GRU layers using popular deep learning frameworks.