Updating the Cell State

The LSTM cell updates its cell state, $C_t$, by combining information from its forget gate (which determines what to discard from the previous state) and its input gate (which identifies relevant new information). This update process is central to the LSTM's ability to maintain long-range dependencies.

Recall the previous steps:

1. The forget gate ($f_t$) computed values between 0 and 1 for each number in the previous cell state $C_{t-1}$. A value close to 1 means "keep this information", while a value close to 0 means "forget this information".
2. The input gate decided which values to update ($i_t$, values between 0 and 1) and created a vector of new candidate values ($\tilde{C}_t$, values typically between -1 and 1) to be potentially added to the state.

The cell state update combines these pieces in a straightforward yet effective manner. First, the old cell state $C_{t-1}$ is multiplied element-wise by the output of the forget gate $f_t$. This selectively drops the information marked for forgetting.

$\text{Forgotten State} = f_t \odot C_{t-1}$

Second, the candidate values $\tilde{C}_t$ are multiplied element-wise by the output of the input gate $i_t$. This selects only the relevant parts of the new candidate information.

$\text{Selected New Information} = i_t \odot \tilde{C}_t$

Finally, these two results are added together element-wise to create the updated cell state $C_t$:

$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$

Here, $\odot$ denotes element-wise multiplication (Hadamard product).

Diagram illustrating the update mechanism for the LSTM cell state ($C_t$). The previous state ($C_{t-1}$) is scaled by the forget gate ($f_t$), the candidate state ($\tilde{C}_t$) is scaled by the input gate ($i_t$), and the results are added.

This additive update mechanism is a significant departure from the update rule in simple RNNs, which primarily involves matrix multiplications. The cell state acts like a conveyor belt. Information can travel along it mostly undisturbed if the forget gate is set close to 1 and the input gate is close to 0 for those components. Conversely, old information can be entirely dropped ($f_t \approx 0$) and new information fully integrated ($i_t \approx 1$).

This structure makes it much easier for gradients to flow backward through time without vanishing or exploding as rapidly as they might in a simple RNN. By controlling the flow via additions and element-wise multiplications regulated by gates, LSTMs can preserve error signals over longer durations, enabling the learning of dependencies across extended time intervals. The cell state essentially carries the long-term memory, which is selectively modified at each time step based on the current input and the previous hidden state.