Calculating the Candidate Hidden State

The Gated Recurrent Unit (GRU) cell utilizes two main gates, the reset gate ($r_t$) and the update gate ($z_t$), to generate a proposal for the new hidden state. This proposal, called the candidate hidden state (often denoted as $ilde{h}_t$), represents the new information the cell considers adding to its memory.

The calculation of the candidate hidden state is where the reset gate ($r_t$) plays its part. Remember, the reset gate determines how much of the previous hidden state ($h_{t-1}$) should influence the current candidate calculation. If the reset gate output is close to 0 for certain dimensions, the corresponding dimensions of the previous hidden state are effectively "forgotten" or ignored when computing the new candidate state. Conversely, if the reset gate output is close to 1, the previous state information is passed through.

The core idea is to combine the current input ($x_t$) with a version of the previous hidden state ($h_{t-1}$) that has been selectively filtered by the reset gate ($r_t$). This combination is then passed through a hyperbolic tangent ($tanh$) activation function, similar to how the hidden state is calculated in a simple RNN.

Mathematically, the candidate hidden state $\tilde{h}_t$ at time step $t$ is computed as follows:

$$\tilde{h}_t = \tanh(W_{\tilde{h}} x_t + U_{\tilde{h}} (r_t \odot h_{t-1}) + b_{\tilde{h}})$$

Let's break down this equation:

• $x_t$: The input vector at the current time step $t$.
• $h_{t-1}$: The hidden state vector from the previous time step $t-1$.
• $r_t$: The output vector of the reset gate at time step $t$.
• $\odot$: This symbol represents the element-wise multiplication (Hadamard product). This is how the reset gate $r_t$ selectively scales the previous hidden state $h_{t-1}$. Each element in $h_{t-1}$ is multiplied by the corresponding element in $r_t$.
• $W_{\tilde{h}}$: A weight matrix that transforms the input $x_t$.
• $U_{\tilde{h}}$: A weight matrix that transforms the reset-gated previous hidden state ($r_t \odot h_{t-1}$).
• $b_{\tilde{h}}$: A bias vector added to the sum.
• $tanh$: The hyperbolic tangent activation function. It squashes the resulting vector's components to be between -1 and 1, helping to regulate the network's activations.

The weight matrices ($W_{\tilde{h}}$, $U_{\tilde{h}}$) and the bias ($b_{\tilde{h}}$) are learned during the training process. They determine how the current input and the relevant parts of the past (as determined by the reset gate) are combined to form the candidate state.

The diagram below illustrates the data flow for calculating the candidate hidden state $\tilde{h}_t$.

Flow diagram showing the computation of the candidate hidden state ($\tilde{h}_t$) using the current input ($x_t$), the previous hidden state ($h_{t-1}$), and the reset gate ($r_t$).

Essentially, $\tilde{h}_t$ represents what the GRU cell could update its state to, based purely on the current input and the selectively remembered parts of the previous state. It's a proposal for the new memory content.

It's important to remember that this candidate state $\tilde{h}_t$ is not the final hidden state $h_t$ for the current time step. The next step, which involves the update gate $z_t$, will determine how much of this new candidate state $\tilde{h}_t$ is actually mixed with the previous hidden state $h_{t-1}$ to produce the final output $h_t$. We will cover that combination process in the section "Calculating the Final Hidden State".