The Input Gate

Simple RNNs struggle with long sequences because gradients can vanish or explode during backpropagation through time. LSTMs introduce gating mechanisms precisely to address this. These gates are like regulators, carefully controlling how information flows into, through, and out of the core memory component of the LSTM, the cell state.

One of these critical regulators is the input gate. Its job is to decide what new information from the current input ($x_t$) and the previous hidden state ($h_{t-1}$) should be stored in the cell state ($C_t$). It doesn't operate in isolation; it works alongside the forget gate (which decides what to throw away from the old cell state) to manage the cell's memory effectively.

The input gate's decision process involves two main parts:

1. Deciding Which Values to Update: First, a sigmoid layer determines which parts of the cell state should be updated. The sigmoid function, often denoted as $\sigma$, squashes its input to a range between 0 and 1. A value close to 1 means "let this information through," while a value close to 0 means "block this information." This layer takes the previous hidden state ($h_{t-1}$) and the current input ($x_t$) and produces an output vector, let's call it $i_t$.

   The calculation is:

   $$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$

   Here, $[h_{t-1}, x_t]$ represents the concatenation of the previous hidden state and the current input vector. $W_i$ is the weight matrix and $b_i$ is the bias vector specific to this part of the input gate. The sigmoid function ($\sigma$) is applied element-wise. Each element in $i_t$ corresponds to an element in the cell state, acting as a filter or gate value for that specific element.

2. Creating Candidate Values: Concurrently, a tanh layer creates a vector of new candidate values, denoted as $\tilde{C}_t$ (pronounced "C-tilde sub t"). These are the potential values that could be added to the cell state. Like the sigmoid layer, this layer also uses the previous hidden state ($h_{t-1}$) and the current input ($x_t$). The tanh activation function squashes its input to a range between -1 and 1.

   The calculation is:

   $$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$$

   Again, $W_C$ and $b_C$ are the weight matrix and bias vector for this specific layer. The output $\tilde{C}_t$ represents the new information extracted from the current input and previous context, scaled between -1 and 1.

Think of $i_t$ as the gatekeeper deciding how much of each potential new piece of information ($\tilde{C}_t$) should actually be considered for adding to the memory. $\tilde{C}_t$ holds the potential updates, and $i_t$ holds the filtering values (between 0 and 1) that scale these candidates.

Diagram illustrating the two components of the input gate. It takes the current input ($x_t$) and previous hidden state ($h_{t-1}$), processes them through parallel sigmoid and tanh layers, and produces the gate activation ($i_t$) and candidate values ($\tilde{C}_t$). These are then combined element-wise ($i_t * \tilde{C}_t$) to form the update information for the cell state.

The important step, which connects the input gate to the cell state, involves combining the outputs of these two layers. This is typically done using element-wise multiplication: $i_t * \tilde{C}_t$. This product represents the filtered candidate values, the new information that has been selected and scaled by the input gate.

This resulting vector ($i_t * \tilde{C}_t$) is what gets added to the (appropriately forgotten) previous cell state $C_{t-1}$ to form the new cell state $C_t$. We will examine this addition process in detail when we discuss updating the cell state in the next section. For now, the significant point is that the input gate provides the mechanism for selectively incorporating new information into the LSTM's memory.