Gated Recurrent Units (GRUs) Architecture

Building upon the foundation laid by LSTMs to mitigate the gradient problems inherent in simple RNNs, Gated Recurrent Units (GRUs), introduced by Cho et al. in 2014, offer a variation on the theme of gated architectures. GRUs aim for a similar outcome, controlling information flow through time, but achieve it with a slightly streamlined structure compared to LSTMs. This simplification often leads to fewer parameters and potentially faster computation, while frequently delivering performance comparable to LSTMs.

GRU Gates and State Updates

A GRU cell operates without a separate cell state, modifying its hidden state $h_t$ directly using two primary gating mechanisms: the reset gate and the update gate. Let's examine their roles and computations.

Reset Gate ($r_t$)

The reset gate determines how much of the previous hidden state, $h_{t-1}$, should be effectively "forgotten" or ignored when proposing a new candidate hidden state. If the reset gate outputs values close to 0, it allows the cell to drop information from the past that is deemed irrelevant for the current computation. Conversely, values close to 1 retain most of the previous state's information.

The computation involves the current input $x_t$ and the previous hidden state $h_{t-1}$. A sigmoid function $\sigma$ squashes the output to the range [0, 1]:

$$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$$

Here, $W_r$, $U_r$, and $b_r$ are learned weight matrices and a bias vector specific to the reset gate.

Update Gate ($z_t$)

The update gate plays a role analogous to a combination of the forget and input gates in an LSTM. It decides how much information from the previous hidden state $h_{t-1}$ should be carried forward to the new hidden state $h_t$. Simultaneously, it controls how much of the newly computed candidate hidden state, $\tilde{h}_t$, should be incorporated.

Its computation is similar in structure to the reset gate:

$$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$$

Where $W_z$, $U_z$, and $b_z$ are the learned parameters for the update gate.

Candidate Hidden State ($\tilde{h}_t$)

The candidate hidden state represents a proposal for the new hidden state at time $t$. Its computation is influenced by the reset gate $r_t$. Specifically, the contribution of the previous hidden state $h_{t-1}$ is modulated (element-wise multiplied, denoted by $\odot$) by the reset gate's output before being combined with the processed input $x_t$. A hyperbolic tangent function ($\tanh$) is typically used as the activation:

$$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$

$W_h$, $U_h$, and $b_h$ are the learned parameters for calculating the candidate state. The element-wise multiplication $r_t \odot h_{t-1}$ is the mechanism allowing the GRU to selectively discard parts of the previous state based on $r_t$.

Final Hidden State ($h_t$)

The final hidden state $h_t$ for the current time step is computed by linearly interpolating between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ determines the balance of this interpolation.

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

When $z_t$ is close to 1, the candidate state $\tilde{h}_t$ contributes more, effectively updating the hidden state with new information. When $z_t$ is close to 0, the previous state $h_{t-1}$ is largely preserved, allowing information to carry over long distances.

Data flow within a Gated Recurrent Unit (GRU) cell at time step $t$. $x_t$ is the input, $h_{t-1}$ is the previous hidden state. The reset ($r_t$) and update ($z_t$) gates control the computation of the candidate state ($\tilde{h}_t$) and the final hidden state ($h_t$). Dotted lines indicate usage of a value in a computation.

GRUs vs. LSTMs

The GRU architecture can be seen as a simplification of the LSTM:

1. Gates: GRUs use two gates (reset, update) compared to LSTM's three (input, forget, output).
2. Cell State: GRUs lack the separate cell state ($c_t$) found in LSTMs. The hidden state $h_t$ directly incorporates the mechanisms for controlling long-term dependencies.
3. Parameter Count: Due to the reduced number of gates and absence of a cell state pathway, GRUs generally have fewer trainable parameters than LSTMs with the same hidden state dimension. This can make them computationally slightly cheaper and potentially less prone to overfitting on smaller datasets.

Empirically, neither architecture consistently outperforms the other across all tasks. The choice between LSTM and GRU often comes down to experimental results on the specific problem, though GRUs might be favored when computational resources or parameter efficiency are primary concerns.

Persistent Limitations

Despite their sophisticated gating, GRUs still retain the fundamental characteristic of recurrent models: sequential computation. Information must propagate step-by-step through the sequence length. This inherent sequentiality limits parallelization during training, making them significantly slower to train on very long sequences compared to architectures like the Transformer. Furthermore, while much better than simple RNNs at capturing longer-range dependencies, the reliance on summarizing the past into a fixed-size hidden state can still become a bottleneck for extremely long sequences where intricate dependencies span vast distances. These remaining challenges paved the way for attention-based mechanisms, which we will explore next.