The Forget Gate

Simple RNNs struggle with long-range dependencies because they lack a mechanism to explicitly control their memory. Information from many steps ago could either vanish or overwhelm the network. Long Short-Term Memory (LSTM) networks introduce specialized components, called gates, to manage the information stored in the network's memory, which is known as the cell state ($C$).

The first gate we'll examine is the forget gate. Its function is straightforward but significant: it decides which information should be discarded or kept from the cell state. Think of the cell state as the LSTM's long-term memory. As new input arrives, the forget gate looks at the previous state and the new input to determine which parts of the existing long-term memory are no longer relevant.

How the Forget Gate Operates

The forget gate works by passing the previous hidden state ($h_{t-1}$) and the current input ($x_t$) through a sigmoid activation function ($\sigma$). The sigmoid function is ideal here because it outputs values between 0 and 1.

Inputs ($h_{t-1}$ and $x_t$) are processed by the forget gate's sigmoid layer to produce the forget vector $f_t$.

Mathematically, the calculation is:

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$

Let's break this down:

1. $[h_{t-1}, x_t]$: This represents the concatenation of the previous hidden state vector $h_{t-1}$ and the current input vector $x_t$. If $h_{t-1}$ has dimension $N_h$ and $x_t$ has dimension $N_x$, the concatenated vector has dimension $N_h + N_x$.
2. $W_f$: This is the weight matrix associated with the forget gate. It's a learnable parameter adjusted during training. Its dimensions are typically $(N_h + N_x) \times N_h$, where $N_h$ is the dimension of the cell state (and hidden state).
3. $b_f$: This is the bias vector for the forget gate, also a learnable parameter, with dimension $N_h$.
4. $\sigma$: The sigmoid function is applied element-wise to the result of the matrix multiplication and addition.

Interpreting the Output ($f_t$)

The output $f_t$ is a vector with the same dimension as the cell state $C_{t-1}$. Each element in $f_t$ is a number between 0 and 1, corresponding to an element in the cell state $C_{t-1}$.

• A value close to 0 in $f_t$ means "forget" or "discard" the corresponding information in $C_{t-1}$.
• A value close to 1 in $f_t$ means "keep" or "remember" the corresponding information in $C_{t-1}$.
• Values between 0 and 1 represent partial retention.

This vector $f_t$ acts like a filter. It will be multiplied element-wise with the previous cell state $C_{t-1}$ to decide how much of the old memory should pass through to the next step. We will see this multiplication happen when we discuss updating the cell state later in this chapter.

The ability to selectively forget irrelevant information based on the current input and the past context (via $h_{t-1}$) is a core reason why LSTMs can maintain useful information over much longer sequences compared to simple RNNs. It prevents the cell state from becoming cluttered with outdated or unnecessary details.