Having explored simple Recurrent Neural Networks (RNNs) and the vanishing gradient problem they often encounter, we saw how Long Short-Term Memory (LSTM) networks introduce gating mechanisms to manage information flow over long sequences. LSTMs, however, are quite complex, featuring three distinct gates (input, forget, output) and a separate cell state. In 2014, Cho and colleagues introduced the Gated Recurrent Unit (GRU), a simpler alternative that often achieves comparable performance to LSTMs while being computationally more efficient.

GRUs, like LSTMs, use gating mechanisms to control the flow of information, but they achieve this with a more streamlined architecture. Instead of three gates and a separate cell state, a GRU cell employs just two gates: the Update Gate and the Reset Gate. It also merges the cell state and hidden state into a single hidden state vector, $h_t$ .

Let's look at how these components work together at a given time step $t$ , taking the current input $x_t$ and the previous hidden state $h_{t-1}$ as inputs.

The Reset Gate

The reset gate ( $r_t$ ) determines how to combine the new input $x_t$ with the previous hidden state $h_{t-1}$ . Specifically, it controls how much of the previous hidden state should be "forgotten" or ignored when calculating a candidate for the next hidden state. If the reset gate's value is close to 0 for certain dimensions of $h_{t-1}$ , it effectively makes the unit act as if reading the input for the first time, disregarding irrelevant past information for the current context.

The reset gate's activation is calculated using a sigmoid function ( $\sigma$ ), which outputs values between 0 and 1:

r_t = \sigma(W_r [h_{t-1}, x_t] + b_r)

Here, $[h_{t-1}, x_t]$ represents the concatenation of the previous hidden state and the current input vector. $W_r$ is the weight matrix and $b_r$ is the bias term for the reset gate.

The Update Gate

The update gate ( $z_t$ ) performs a role similar to the combination of the forget and input gates in an LSTM. It decides how much information from the previous hidden state $h_{t-1}$ should be carried over to the current hidden state $h_t$ . Simultaneously, it also controls how much of the newly computed candidate hidden state ( $\tilde{h}_t$ , explained next) should be added.

If $z_t$ is close to 1, the previous hidden state $h_{t-1}$ is largely copied to the new hidden state $h_t$ . If $z_t$ is close to 0, the new candidate state $\tilde{h}_t$ predominantly forms the new hidden state $h_t$ . This mechanism allows the GRU to retain information from distant past time steps.

The update gate's activation is also calculated using a sigmoid function:

z_t = \sigma(W_z [h_{t-1}, x_t] + b_z)

Where $W_z$ and $b_z$ are the weight matrix and bias term for the update gate.

Candidate Hidden State

The GRU computes a candidate hidden state $\tilde{h}_t$ that represents the proposed update for the current time step. This calculation is influenced by the reset gate $r_t$ . The reset gate determines which parts of the previous hidden state $h_{t-1}$ are used in this calculation. The candidate state is typically computed using a hyperbolic tangent ( $\tanh$ ) activation function, which outputs values between -1 and 1.

\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t] + b_h)

Notice the element-wise multiplication ( $\odot$ ) between the reset gate $r_t$ and the previous hidden state $h_{t-1}$ . This is where the reset gate selectively filters the information from $h_{t-1}$ before combining it with the current input $x_t$ . $W_h$ and $b_h$ are the corresponding weight matrix and bias.

Final Hidden State

Finally, the actual hidden state $h_t$ for the current time step is computed by linearly interpolating between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$ . The update gate $z_t$ controls this interpolation:

h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

This equation shows how the update gate acts:

When $z_t$ is close to 1, $h_t$ is primarily composed of the candidate state $\tilde{h}_t$ .
When $z_t$ is close to 0, $h_t$ is primarily composed of the previous state $h_{t-1}$ .

This allows the network to maintain long-term information by keeping $z_t$ close to 0 for multiple time steps, or update quickly based on new inputs by setting $z_t$ close to 1.

The following diagram illustrates the structure of a GRU cell:

Internal structure of a Gated Recurrent Unit (GRU) cell, showing the flow of information through the reset gate ( $r_t$ ) and update gate ( $z_t$ ) to compute the next hidden state ( $h_t$ ).

GRUs vs. LSTMs

While both GRUs and LSTMs effectively address the vanishing gradient problem and capture long-range dependencies, they differ in their internal structure:

Complexity: GRUs have two gates and merge the cell and hidden states. LSTMs have three gates and maintain a separate cell state. This makes GRUs structurally simpler.
Parameters: Due to fewer gates, GRUs generally have fewer trainable parameters than LSTMs for the same hidden state size. This can lead to faster training times and potentially less overfitting, especially on smaller datasets.
Information Flow: LSTMs, with their dedicated cell state and output gate, might offer finer-grained control over memory content and exposure. This could theoretically be advantageous for certain complex tasks, although empirical evidence often shows comparable performance.
Performance: In practice, neither architecture consistently outperforms the other across all tasks. Performance often depends on the specific dataset, task, and hyperparameter tuning. It's common to experiment with both architectures during model development.

Using GRUs in Practice

Similar to LSTMs, GRUs are readily available as layers in standard deep learning frameworks like TensorFlow (via Keras) and PyTorch. You can easily swap an LSTM layer for a GRU layer in your model definitions. Concepts like stacking multiple GRU layers (to create deeper networks) or using Bidirectional GRUs (to process sequences in both forward and backward directions) apply just as they do for LSTMs.

In summary, GRUs present a compelling alternative to LSTMs for sequence modeling. Their simpler architecture often translates to computational efficiency without a significant sacrifice in performance for many NLP tasks. By employing reset and update gates, they effectively manage the flow of information through time, enabling them to learn dependencies across sequences, much like their LSTM counterparts. When building sequence models, considering both LSTMs and GRUs is a standard part of the model selection process.