Information Flow Through an LSTM Cell

Information moves through a Long Short-Term Memory (LSTM) cell in a specific flow during a single time step. This flow is essential for understanding how LSTMs manage to maintain context over long sequences, effectively addressing the shortcomings of simple Recurrent Neural Networks (RNNs).

At each time step $t$, the LSTM cell receives three inputs:

1. The current input vector $x_t$.
2. The previous hidden state $h_{t-1}$.
3. The previous cell state $c_{t-1}$.

These inputs interact with the gates and the cell state to produce two outputs:

1. The new hidden state $h_t$.
2. The new cell state $c_t$.

Let's follow the data path:

1. The Forget Gate: Deciding What to Throw Away

The first step involves the forget gate ($f_t$). Its job is to decide which parts of the old cell state $c_{t-1}$ are no longer relevant and should be discarded. It looks at the previous hidden state $h_{t-1}$ and the current input $x_t$. A sigmoid activation function ($\sigma$) squashes the output to a value between 0 and 1 for each number in the cell state vector.

$$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$$

Here, $[h_{t-1}, x_t]$ represents the concatenation of the two vectors. $W_f$ and $b_f$ are the weight matrix and bias vector for the forget gate, learned during training.

An output of 1 means "completely keep this information," while an output of 0 means "completely get rid of this information." This gate's output $f_t$ is then multiplied element-wise ($\odot$) with the previous cell state $c_{t-1}$.

2. The Input Gate: Deciding What New Information to Store

Next, the cell needs to determine what new information from the current input $x_t$ and previous hidden state $h_{t-1}$ should be added to the cell state. This involves two parts:

• The Input Gate Layer ($i_t$): Another sigmoid layer decides which values we'll update. $$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$$
• Candidate Values ($\tilde{c}_t$): A $tanh$ layer creates a vector of new candidate values that could be added to the state. $$\tilde{c}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)$$

$W_i, b_i$ and $W_C, b_C$ are the respective weights and biases for these layers. The $tanh$ function outputs values between -1 and 1.

3. Updating the Cell State: Combining Old and New

Now we update the old cell state $c_{t-1}$ to the new cell state $c_t$. We combine the results from the forget gate and the input gate:

• First, we apply the forget gate's decision: $f_t \odot c_{t-1}$. This drops the information marked for forgetting.
• Then, we determine the new information to add: $i_t \odot \tilde{c}_t$. This scales the candidate values by how much we decided to update each state value.
• Finally, we add these two parts together: $$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$

This additive interaction is a significant difference from the repeated matrix multiplications in simple RNNs. It allows gradients to flow more easily through time during backpropagation, reducing the vanishing gradient problem. The cell state acts like a conveyor belt, carrying information along with only minor linear interactions (multiplication by $f_t$ and addition of $i_t \odot \tilde{c}_t$), making it easier to preserve context over many steps.

4. The Output Gate: Deciding What to Output

Finally, we need to decide what the hidden state $h_t$ (and potentially the output for this time step) should be. This output will be a filtered version of the cell state $c_t$.

• The Output Gate Layer ($o_t$): A sigmoid layer determines which parts of the cell state will be output. $$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$$
• Filtering the Cell State: We push the updated cell state $c_t$ through $tanh$ (to squash values between -1 and 1) and then multiply it element-wise by the output of the output gate $o_t$: $$h_t = o_t \odot \tanh(c_t)$$

The resulting $h_t$ is the hidden state passed to the next time step. It also serves as the cell's output at time step $t$ if needed for prediction. $W_o$ and $b_o$ are the weights and bias for the output gate.

Visualizing the Flow

The following diagram illustrates how these components connect and how data flows through an LSTM cell during one time step:

Diagram illustrating the flow of information and calculations within an LSTM cell for a single time step $t$. Inputs $x_t$, $h_{t-1}$, $c_{t-1}$ are processed through forget ($f_t$), input ($i_t$), and output ($o_t$) gates, along with a candidate state ($\tilde{c}_t$), to compute the new cell state $c_t$ and hidden state $h_t$. Sigmoid ($\sigma$) and $tanh$ activations control the gating and state updates. Element-wise multiplication ($\odot$) and addition (+) combine the intermediate results. Dashed red arrows indicate the passing of $c_t$ and $h_t$ to the next time step.

By carefully regulating what information is kept, discarded, added, and output at each step, the LSTM cell creates pathways for gradients to flow more effectively during training. The cell state acts as an explicit memory channel, protected by the gates, allowing the network to learn and remember information over extended periods, which is fundamental for tackling complex sequence modeling tasks.