The GRU Cell Architecture

Gated Recurrent Units (GRUs) are a type of recurrent neural network architecture designed to effectively capture dependencies in sequential data. They achieve this by employing gating mechanisms that regulate information flow. A GRU cell features two primary gates: the reset gate and the update gate. These gates control how new input data and previous hidden state information are used to update the current hidden state. The GRU simplifies its internal structure by directly combining the functions of a cell state and hidden state into a single hidden state vector, $h_t$. This design often results in a simpler model compared to architectures like LSTMs, which typically use three gates and separate cell and hidden states.

Let's examine the components and the flow of information within a single GRU cell at time step $t$. The cell receives the current input $x_t$ and the hidden state from the previous time step $h_{t-1}$. It then computes the new hidden state $h_t$, which also serves as the output for this time step.

The components are:

1. Reset Gate ($r_t$): Determines how to combine the new input $x_t$ with the previous hidden state $h_{t-1}$ to compute a candidate hidden state. Specifically, it controls how much of the previous state information should be "forgotten" or ignored when proposing a new state.
2. Update Gate ($z_t$): Determines how much of the previous hidden state $h_{t-1}$ should be retained and how much of the newly computed candidate hidden state $\tilde{h}_t$ should be included in the final hidden state $h_t$.
3. Candidate Hidden State ($\tilde{h}_t$): A "proposed" value for the next hidden state, calculated based on the current input and a potentially reset version of the previous hidden state.
4. Final Hidden State ($h_t$): The output state of the GRU cell for the current time step, computed by interpolating between the previous state $h_{t-1}$ and the candidate state $\tilde{h}_t$, guided by the update gate.

Here is a view of the GRU cell's architecture:

A simplified view of the GRU cell structure. Inputs $x_t$ and $h_{t-1}$ feed into the reset ($r_t$) and update ($z_t$) gates. The reset gate modulates the influence of $h_{t-1}$ on the candidate state $h̃_t$. The update gate controls the mix between $h_{t-1}$ and $h̃_t$ to produce the final output $h_t$.

Now, let's look at the calculations performed within the cell.

Reset Gate ($r_t$)

The reset gate decides how much information from the previous hidden state $h_{t-1}$ should be disregarded when computing the candidate hidden state $\tilde{h}_t$. It takes the current input $x_t$ and the previous hidden state $h_{t-1}$ as inputs.

The calculation is:

$$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$$

Here, $W_r$ and $U_r$ are weight matrices, $b_r$ is a bias vector, and $\sigma$ is the sigmoid activation function. The sigmoid function outputs values between 0 and 1. A value close to 0 means "reset" or ignore the corresponding element in the previous state, while a value close to 1 means "keep" it when calculating the candidate state.

Update Gate ($z_t$)

The update gate controls the extent to which the hidden state is updated with new information versus retaining old information. It determines how much of the previous hidden state $h_{t-1}$ carries over to the final hidden state $h_t$. Similar to the reset gate, it uses the current input $x_t$ and the previous hidden state $h_{t-1}$.

The calculation is:

$$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$$

Again, $W_z$, $U_z$, and $b_z$ are learned parameters (weights and bias), and $\sigma$ is the sigmoid function. A value of $z_t$ close to 1 indicates that the new hidden state $h_t$ should primarily be based on the candidate state $\tilde{h}_t$, while a value close to 0 suggests retaining most of the previous state $h_{t-1}$.

Candidate Hidden State ($\tilde{h}_t$)

The candidate hidden state is calculated similarly to the hidden state in a simple RNN, but with a modification involving the reset gate. It aims to capture the new information from the current input $x_t$, potentially tempered by the relevant parts of the previous state $h_{t-1}$.

The calculation involves the current input $x_t$ and the previous hidden state $h_{t-1}$, element-wise multiplied ($\odot$) by the reset gate's output $r_t$:

$$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$

Here, $W_h$, $U_h$, and $b_h$ are learned parameters. The tanh activation function squashes the output to be between -1 and 1. The element-wise multiplication $r_t \odot h_{t-1}$ is significant: if an element in $r_t$ is close to 0, the corresponding element from $h_{t-1}$ contributes very little to the calculation of $\tilde{h}_t$, effectively allowing the cell to "forget" irrelevant past information when generating the candidate state.

Final Hidden State ($h_t$)

The final hidden state $h_t$ for the current time step is computed by linearly interpolating between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ controls this interpolation.

The calculation is:

$$h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t$$

This equation shows how the GRU updates its state. The vector $z_t$ acts element-wise:

• If an element of $z_t$ is close to 1, the corresponding element in $h_t$ will be mostly determined by the candidate state $\tilde{h}_t$, incorporating new information.
• If an element of $z_t$ is close to 0, the corresponding element in $h_t$ will be mostly determined by the previous state $h_{t-1}$, retaining past information.

This mechanism allows the GRU to maintain information over long sequences (when $z_t$ is close to 0 for many steps) or update rapidly based on new inputs (when $z_t$ is close to 1). Notably, the GRU does not have a separate cell state like the LSTM; the hidden state $h_t$ carries all the necessary information forward.

This architecture, with its two gates and combined state representation, provides a powerful yet potentially more computationally efficient way to handle sequential data compared to LSTMs, which we will compare more directly later in this chapter.