The Reset Gate

The reset gate in a Gated Recurrent Unit (GRU) plays a specific and important role in managing information flow. It determines how much of the past information, carried by the previous hidden state ($h_{t-1}$), should be ignored or "reset" when calculating the new candidate hidden state ($ilde{h}_t$). This gate acts as a filter, deciding the relevance of past context for proposing an updated memory state.

Calculating the Reset Gate Activation

Like the update gate, the reset gate's activation, denoted as $r_t$, is computed based on the current input $x_t$ and the previous hidden state $h_{t-1}$. It uses a sigmoid activation function, ensuring its output values are between 0 and 1.

The calculation involves learning separate weight matrices ($W_{xr}$ and $W_{hr}$) and a bias term ($b_r$):

$$r_t = \sigma(W_{xr} x_t + W_{hr} h_{t-1} + b_r)$$

Here:

• $x_t$ is the input vector at the current time step $t$.
• $h_{t-1}$ is the hidden state vector from the previous time step $t-1$.
• $W_{xr}$ is the weight matrix connecting the input to the reset gate.
• $W_{hr}$ is the weight matrix connecting the previous hidden state to the reset gate.
• $b_r$ is the bias term for the reset gate.
• $\sigma$ represents the sigmoid activation function, $\sigma(z) = 1 / (1 + e^{-z})$.

The output $r_t$ is a vector of the same dimension as the hidden state. Each element in $r_t$ corresponds to a dimension of the hidden state, acting as a gate value for that specific dimension.

How the Reset Gate Modulates Information

The values in the reset gate vector $r_t$ directly control the influence of the previous hidden state $h_{t-1}$ when computing the candidate hidden state $\tilde{h}_t$. A value close to 0 in $r_t$ for a particular dimension effectively "resets" or nullifies the contribution from the corresponding dimension in $h_{t-1}$. Conversely, a value close to 1 allows that part of the previous hidden state to pass through mostly unchanged.

This mechanism is applied via element-wise multiplication ($\odot$) between the reset gate $r_t$ and the previous hidden state $h_{t-1}$. This modulated previous state is then used in the calculation of the candidate hidden state:

$$\tilde{h}_t = \tanh(W_{xh} x_t + W_{hh} (r_t \odot h_{t-1}) + b_h)$$

Notice how $r_t \odot h_{t-1}$ determines exactly which parts of the previous state $h_{t-1}$ are combined with the current input $x_t$ to form the candidate state $\tilde{h}_t$. If an element in $r_t$ is 0, the corresponding element in $h_{t-1}$ is effectively zeroed out before the weighted sum inside the $\tanh$ function.

Flow showing the calculation of the reset gate $r_t$ and its element-wise multiplication ($\odot$) with the previous hidden state $h_{t-1}$ to influence the candidate hidden state $\tilde{h}_t$.

Intuition Behind the Reset Gate

The reset gate gives the GRU unit the ability to dynamically adjust how much the proposed new state ($\tilde{h}_t$) should depend on the immediate past state ($h_{t-1}$). If the current input $x_t$ suggests a significant shift in context or topic compared to what was encoded in $h_{t-1}$, the reset gate can learn to activate close to 0. This effectively allows the unit to "start fresh" in computing the candidate state, focusing more on the current input $x_t$ rather than blending it with potentially irrelevant past information.

For example, in language modeling, if the network encounters the end of a sentence (signaled perhaps by punctuation in $x_t$), the reset gate might activate strongly (values near 0) to diminish the influence of the previous sentence's hidden state when calculating the candidate state for the beginning of the next sentence.

In summary, the reset gate acts as a controller, selectively diminishing parts of the previous hidden state before calculating the candidate hidden state. This allows the GRU to effectively forget information that is deemed irrelevant for the immediate next step, contributing to its ability to handle dependencies over time.