The GRU Cell Architecture

Like LSTMs, Gated Recurrent Units (GRUs) are designed to effectively capture dependencies in sequential data by using gating mechanisms. However, the GRU cell achieves this with a slightly different and often simpler internal structure compared to the LSTM cell. Instead of three gates (forget, input, output) and a separate cell state, the GRU employs two primary gates: the reset gate and the update gate. It also directly combines the functions of the LSTM's cell state and hidden state into a single hidden state vector, $h_t$.

Let's examine the components and the flow of information within a single GRU cell at time step $t$. The cell receives the current input $x_t$ and the hidden state from the previous time step $h_{t-1}$. It then computes the new hidden state $h_t$, which also serves as the output for this time step.

The key components are:

1. Reset Gate ($r_t$): Determines how to combine the new input $x_t$ with the previous hidden state $h_{t-1}$ to compute a candidate hidden state. Specifically, it controls how much of the previous state information should be "forgotten" or ignored when proposing a new state.
2. Update Gate ($z_t$): Determines how much of the previous hidden state $h_{t-1}$ should be retained and how much of the newly computed candidate hidden state $\tilde{h}_t$ should be included in the final hidden state $h_t$.
3. Candidate Hidden State ($\tilde{h}_t$): A "proposed" value for the next hidden state, calculated based on the current input and a potentially reset version of the previous hidden state.
4. Final Hidden State ($h_t$): The output state of the GRU cell for the current time step, computed by interpolating between the previous state $h_{t-1}$ and the candidate state $\tilde{h}_t$, guided by the update gate.

Here is a conceptual view of the GRU cell's architecture:

A simplified view of the GRU cell structure. Inputs $x_t$ and $h_{t-1}$ feed into the reset ($r_t$) and update ($z_t$) gates. The reset gate modulates the influence of $h_{t-1}$ on the candidate state $h̃_t$. The update gate controls the mix between $h_{t-1}$ and $h̃_t$ to produce the final output $h_t$.

Now, let's look at the calculations performed within the cell.

Reset Gate ($r_t$)

The reset gate decides how much information from the previous hidden state $h_{t-1}$ should be disregarded when computing the candidate hidden state $\tilde{h}_t$. It takes the current input $x_t$ and the previous hidden state $h_{t-1}$ as inputs.

The calculation is:

$$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$$

Here, $W_r$ and $U_r$ are weight matrices, $b_r$ is a bias vector, and $\sigma$ is the sigmoid activation function. The sigmoid function outputs values between 0 and 1. A value close to 0 means "reset" or ignore the corresponding element in the previous state, while a value close to 1 means "keep" it when calculating the candidate state.

Update Gate ($z_t$)

The update gate controls the extent to which the hidden state is updated with new information versus retaining old information. It determines how much of the previous hidden state $h_{t-1}$ carries over to the final hidden state $h_t$. Similar to the reset gate, it uses the current input $x_t$ and the previous hidden state $h_{t-1}$.

The calculation is:

$$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$$

Again, $W_z$, $U_z$, and $b_z$ are learned parameters (weights and bias), and $\sigma$ is the sigmoid function. A value of $z_t$ close to 1 indicates that the new hidden state $h_t$ should primarily be based on the candidate state $\tilde{h}_t$, while a value close to 0 suggests retaining most of the previous state $h_{t-1}$.

Candidate Hidden State ($\tilde{h}_t$)

The candidate hidden state is calculated similarly to the hidden state in a simple RNN, but with a modification involving the reset gate. It aims to capture the new information from the current input $x_t$, potentially tempered by the relevant parts of the previous state $h_{t-1}$.

The calculation involves the current input $x_t$ and the previous hidden state $h_{t-1}$, element-wise multiplied ($\odot$) by the reset gate's output $r_t$:

$$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$

Here, $W_h$, $U_h$, and $b_h$ are learned parameters. The tanh activation function squashes the output to be between -1 and 1. The element-wise multiplication $r_t \odot h_{t-1}$ is significant: if an element in $r_t$ is close to 0, the corresponding element from $h_{t-1}$ contributes very little to the calculation of $\tilde{h}_t$, effectively allowing the cell to "forget" irrelevant past information when generating the candidate state.

Final Hidden State ($h_t$)

The final hidden state $h_t$ for the current time step is computed by linearly interpolating between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ controls this interpolation.

The calculation is:

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

This equation shows how the GRU updates its state. The vector $z_t$ acts element-wise:

• If an element of $z_t$ is close to 1, the corresponding element in $h_t$ will be mostly determined by the candidate state $\tilde{h}_t$, incorporating new information.
• If an element of $z_t$ is close to 0, the corresponding element in $h_t$ will be mostly determined by the previous state $h_{t-1}$, retaining past information.

This mechanism allows the GRU to maintain information over long sequences (when $z_t$ is close to 0 for many steps) or update rapidly based on new inputs (when $z_t$ is close to 1). Notably, the GRU does not have a separate cell state like the LSTM; the hidden state $h_t$ carries all the necessary information forward.

This architecture, with its two gates and combined state representation, provides a powerful yet potentially more computationally efficient way to handle sequential data compared to LSTMs, which we will compare more directly later in this chapter.