The Update Gate

The update gate within a GRU cell, denoted as $z_t$, plays a significant role in managing the flow of information and determining the composition of the next hidden state, $h_t$. Its primary function is to decide how much of the information from the previous hidden state, $h_{t-1}$, should be retained and carried forward, versus how much influence the newly computed candidate state, $ilde{h}_t$, should have.

Calculating the Update Gate Activation

The update gate computes its activation value, $z_t$, based on the current input vector $x_t$ and the previous hidden state $h_{t-1}$. The calculation involves learned weight matrices ($W_z$ for the input, $U_z$ for the previous hidden state) and a bias vector ($b_z$). These are passed through a sigmoid activation function ($\sigma$) to produce values between 0 and 1.

The formula for the update gate activation at time step $t$ is:

$$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$$

Let's break down the components:

• $x_t$: The input vector at the current time step $t$.
• $h_{t-1}$: The hidden state vector from the previous time step $t-1$.
• $W_z$: The weight matrix associated with the input $x_t$.
• $U_z$: The weight matrix associated with the previous hidden state $h_{t-1}$.
• $b_z$: The bias vector for the update gate.
• $\sigma$: The sigmoid activation function, which squashes the combined, weighted input into the range $(0, 1)$.

The output, $z_t$, is a vector of the same dimension as the hidden state. Each element in $z_t$ corresponds to a dimension in the hidden state vector.

Interpreting the Update Gate Output

The values in the $z_t$ vector act as gates or filters. Because of the sigmoid function, each element is between 0 and 1:

• Value close to 1: This indicates that the information in the corresponding dimension of the previous hidden state ($h_{t-1}$) is important and should be mostly retained. It signals "keep the old memory".
• Value close to 0: This suggests that the information in the corresponding dimension of the previous state is less relevant for the current time step and should be largely replaced by the information from the new candidate hidden state ($\tilde{h}_t$). It signals "update with new information".

This gating mechanism is element-wise, meaning the GRU can decide to retain some features from the past while updating others based on the current input.

Role in Updating the Hidden State

The update gate $z_t$ directly participates in the calculation of the final hidden state $h_t$. The GRU computes $h_t$ by performing an interpolation between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ controls this interpolation.

The formula for the final hidden state $h_t$ is:

$$h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t$$

Here:

• $\tilde{h}_t$: The candidate hidden state, computed using the current input and the reset previous hidden state (we'll discuss the reset gate and candidate state calculation separately). It represents the "new" information proposed for the current time step.
• $\odot$: Denotes element-wise multiplication (Hadamard product).

This equation shows how $z_t$ balances the influence:

• The term $z_t \odot h_{t-1}$ retains parts of the previous state $h_{t-1}$. Where $z_t$ is close to 1, $h_{t-1}$ contributes significantly.
• The term $(1 - z_t) \odot \tilde{h}_t$ incorporates parts of the candidate state $\tilde{h}_t$. Where $z_t$ is close to 0, $(1 - z_t)$ is close to 1, and $\tilde{h}_t$ contributes significantly.

Essentially, $z_t$ dynamically determines, for each dimension of the hidden state, whether to copy the value from the previous time step or to update it with the newly computed candidate value.

Calculation flow for the update gate ($z_t$) and its role in combining the previous hidden state ($h_{t-1}$) and the candidate state ($\tilde{h}_t$) to produce the new hidden state ($h_t$).

Facilitating Long-Term Dependencies

The ability of the update gate to allow information from $h_{t-1}$ to pass through largely unchanged (when $z_t \approx 1$) is fundamental to the GRU's success in capturing longer-range dependencies compared to simple RNNs. If the network learns that certain information is relevant over many steps, it can set the corresponding elements of $z_t$ close to 1 for those steps, effectively creating a shortcut for that information through time.

This mechanism also helps alleviate the vanishing gradient problem. During backpropagation, the gradient can flow back through the $z_t \odot h_{t-1}$ term. If $z_t$ is close to 1, the gradient associated with $h_{t-1}$ can pass backward relatively unimpeded, preventing it from diminishing too quickly across many time steps.

In summary, the update gate $z_t$ provides a flexible and adaptive mechanism within the GRU cell. It learns to control how much of the past context stored in $h_{t-1}$ should be preserved and how much new information from the candidate state $\tilde{h}_t$ should be integrated, enabling the network to effectively model sequential data with varying dependency lengths.