Introducing GRUs: A Simpler Gated Architecture

Simple Recurrent Neural Networks (RNNs), while elegant in concept, struggle to capture dependencies across long sequences. The vanishing gradient problem often prevents gradients from propagating effectively through many time steps, making it difficult for the network to learn relationships between distant elements.

Long Short-Term Memory (LSTM) networks, detailed in Chapter 5, provide a powerful solution by introducing dedicated memory cells and multiple gating mechanisms (forget, input, and output gates) to meticulously control the flow of information. LSTMs have proven highly effective but come with a relatively complex internal structure and a significant number of parameters.

Around the same time LSTMs were developed, another type of gated recurrent unit emerged: the Gated Recurrent Unit, or GRU. Proposed by Cho et al. in 2014, GRUs aim to achieve similar capabilities in handling long-range dependencies but with a more streamlined architecture. The core idea behind GRUs is to simplify the gating mechanism while retaining its effectiveness in mitigating gradient issues.

Simplifying the Gated Structure

Compared to LSTMs, GRUs introduce two main simplifications:

1. Combined Cell and Hidden State: GRUs do not maintain a separate cell state ($c_t$) like LSTMs. They only have a single hidden state ($h_t$). This hidden state serves the dual purpose of carrying information about the past and being the output for the current time step.
2. Fewer Gates: GRUs use only two gates instead of the three (or four, depending on perspective) used in LSTMs. These are the Reset Gate and the Update Gate.

Let's briefly look at the roles of these two gates:

• Reset Gate ($r_t$): This gate determines how much information from the previous hidden state ($h_{t-1}$) should be disregarded or "reset" when calculating a new candidate hidden state. If the reset gate's value is close to 0 for certain dimensions, it effectively makes the unit act as if it's processing the start of a new sequence for those dimensions, allowing it to forget irrelevant past information.
• Update Gate ($z_t$): This gate plays a role similar to both the forget and input gates in an LSTM, but in a combined fashion. It decides how much information from the previous hidden state ($h_{t-1}$) should be kept and passed directly to the current hidden state ($h_t$). Conversely, it also determines how much of the newly computed candidate hidden state should be incorporated.

The diagram below offers a high-level comparison between the internal structures of LSTM and GRU cells.

High-level view comparing the internal components and flow within LSTM and GRU cells. Note the absence of a separate cell state and fewer gates in the GRU.

Potential Advantages of Simplicity

This reduced complexity offers several potential advantages:

• Fewer Parameters: With only two gates and no separate cell state, a GRU typically has fewer trainable parameters than an LSTM with the same number of hidden units. This can make GRUs computationally less expensive to train and potentially less prone to overfitting, especially on datasets that are not extremely large.
• Faster Training: Fewer parameters often translate to faster computation per training step.

However, the performance difference between LSTMs and GRUs is often task-dependent. Neither architecture is universally superior across all sequence modeling problems. While GRUs offer simplicity, LSTMs, with their distinct cell state and separate gates, might provide finer control over information flow, which could be beneficial for certain complex tasks.

In the following sections, we will examine the GRU architecture in detail, including the specific equations governing its gates and state updates. We will then directly compare its mechanisms and performance characteristics with those of LSTMs to help you decide when each might be the better choice for your application.