Calculating the Final Hidden State

Within the GRU cell for a given time step $t$, the actual hidden state, $h_t$, is computed as the final step. This computation incorporates the update gate ($z_t$), the reset gate ($r_t$), and the candidate hidden state ($\tilde{h}_t$). The resulting hidden state represents the information the GRU will carry forward to the next time step.

The calculation of the final hidden state $h_t$ is where the update gate $z_t$ plays its central role. Recall that $z_t$ determines how much of the previous hidden state $h_{t-1}$ should be maintained and how much of the new candidate state $\tilde{h}_t$ should be incorporated. The GRU achieves this through a direct interpolation mechanism.

The equation for the final hidden state $h_t$ is:

$$h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t$$

Let's break down this equation:

1.  $z_t \odot h_{t-1}$: This term calculates how much of the previous hidden state $h_{t-1}$ to keep. The update gate $z_t$ contains values between 0 and 1. If a value in $z_t$ is close to 1, the corresponding element from the previous state $h_{t-1}$ is largely preserved. This aligns with $z_t$'s role as the "retention" gate. Conversely, if $z_t$ is close to 0, that part of the previous state is mostly forgotten. The symbol $\odot$ represents element-wise multiplication (Hadamard product).

2.  $(1 - z_t) \odot \tilde{h}_t$: This term calculates how much of the candidate hidden state $\tilde{h}_t$ to incorporate. If a value in $z_t$ is close to 0, the corresponding value in $(1 - z_t)$ is close to 1, meaning the element from the candidate state $\tilde{h}_t$ (which contains the new information) is strongly included. If $z_t$ is close to 1, this term is close to 0, and the new information is mostly ignored.

3.  Addition (+): The two resulting vectors are added together element-wise. This addition completes the interpolation. Each element of the final hidden state $h_t$ is a weighted sum of the corresponding elements from the previous state $h_{t-1}$ and the candidate state $\tilde{h}_t$, with the weights controlled by the update gate $z_t$.

This mechanism allows the GRU cell to dynamically adjust how much information flows from the past versus how much new information is introduced at each time step. If the update gate determines that the previous state is still relevant (high $z_t$ values), it can pass it through largely unchanged. If it decides new information is more important (low $z_t$ values), it incorporates more of the candidate state. This gating mechanism is simpler than the LSTM's separate cell state and output gate but provides a powerful way to manage information flow and combat vanishing gradient issues.

The following diagram illustrates how the previous state, candidate state, and update gate combine to form the final hidden state:

The final hidden state $h_t$ is computed by interpolating between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ controls the balance of this interpolation.

The resulting vector $h_t$ serves two purposes: it is the output of the GRU cell for the current time step $t$ (often passed to subsequent layers or used for prediction), and it becomes the "previous hidden state" $h_{t-1}$ for the next time step $t+1$. This recurrent connection allows the GRU to process sequences step-by-step, maintaining and updating its internal state based on the sequence elements it encounters.