Long Short-Term Memory (LSTM) Networks

As we saw, SimpleRNN layers, while intuitive, struggle to learn patterns that span long intervals in a sequence. This is largely due to the vanishing gradient problem, where the influence of earlier inputs fades rapidly during backpropagation, making it difficult for the network to adjust weights based on long-range dependencies.

To overcome this significant limitation, a more sophisticated recurrent architecture called Long Short-Term Memory (LSTM) was developed by Sepp Hochreiter and Jürgen Schmidhuber in 1997. LSTMs are explicitly designed to remember information for extended periods, making them highly effective for a wide range of sequence modeling tasks.

The core innovation of LSTMs lies in their internal structure. Unlike a SimpleRNN unit which has a simple recurrent connection and transformation, an LSTM unit incorporates a memory cell (or cell state) and several gates. Think of the cell state, often denoted as $C_t$, as the network's long-term memory. It can carry information relatively unchanged across many time steps. The gates are specialized neural network components that act like controllers, regulating the flow of information into and out of this cell state. They learn which information is important to keep, which to discard, and which to output at each time step.

Let's look at the key components within an LSTM unit at a time step $t$:

The Gates

LSTM units typically have three main gates that control the information flow. These gates use sigmoid activation functions, which output values between 0 and 1. A value close to 0 means "let nothing through," while a value close to 1 means "let everything through." They take the current input $x_t$ and the previous hidden state $h_{t-1}$ as inputs.

1. Forget Gate ($f_t$): This gate decides what information should be removed from the cell state. It looks at $h_{t-1}$ and $x_t$ and outputs a number between 0 and 1 for each number in the previous cell state $C_{t-1}$. A 1 represents "completely keep this," while a 0 represents "completely get rid of this."

2. Input Gate ($i_t$): This gate determines what new information should be stored in the cell state. It consists of two parts:

   • A sigmoid layer decides which values to update (outputting $i_t$).
   • A tanh layer creates a vector of new candidate values, $\tilde{C}_t$, that could be added to the state. The tanh function outputs values between -1 and 1.

3. Output Gate ($o_t$): This gate decides what the next hidden state $h_t$ should be. The output will be based on the filtered cell state. First, a sigmoid layer determines which parts of the cell state to output. Then, the current cell state $C_t$ is passed through tanh (to push values between -1 and 1) and multiplied by the output of the sigmoid gate ($o_t$). This filtered version becomes the hidden state $h_t$, which is passed to the next time step and can also be used as the output for the current time step.

The Cell State ($C_t$)

The cell state acts like a conveyor belt for information. It runs straight down the entire chain of time steps, with only minor linear interactions controlled by the gates. The forget gate selectively removes information from the previous cell state $C_{t-1}$, and the input gate selectively adds new candidate information $\tilde{C}_t$. Conceptually, the update looks like this:

$$C_t = (f_t * C_{t-1}) + (i_t * \tilde{C}_t)$$

Here, $*$ denotes element-wise multiplication. The first term ($f_t * C_{t-1}$) represents the information kept from the previous state, and the second term ($i_t * \tilde{C}_t$) represents the new information added. This structure makes it much easier for information to persist over long durations compared to the repeated matrix multiplications and non-linear transformations in SimpleRNN.

The Hidden State ($h_t$)

The hidden state $h_t$ is computed based on the filtered cell state $C_t$, controlled by the output gate $o_t$.

$$h_t = o_t * \tanh(C_t)$$

This $h_t$ is then passed to the LSTM unit at the next time step ($t+1$) and can also serve as the output prediction for the current time step $t$.

Diagram of information flow within a single LSTM unit. Gates (blue) control how the cell state (green cylinder) is updated and what is output as the hidden state (orange parallelogram). Sigmoid ($\sigma$) and tanh activation functions are used within the gates and for state updates.

By using this gating mechanism and the separate cell state pathway, LSTMs can effectively learn which information to store, which to forget, and which to output over long sequences. This addresses the vanishing gradient problem and allows them to capture dependencies that might be hundreds of time steps apart, making them a standard choice for many sequence-related tasks.

In the next section, we will see how to implement LSTMs using the convenient LSTM layer provided by Keras.