Overview: LSTMs and GRUs

Simple Recurrent Neural Networks (RNNs), while elegant in concept, struggle with learning patterns that span long sequences. The primary issue often lies in the backpropagation process through time. Gradients can either shrink exponentially (vanishing gradients), making it impossible for the network to learn connections between distant elements, or grow exponentially (exploding gradients), leading to unstable training. This limitation significantly hinders their effectiveness on tasks requiring long-term memory.

To address these challenges, more sophisticated recurrent architectures were developed, most notably the Long Short-Term Memory (LSTM) network and the Gated Recurrent Unit (GRU). These architectures introduce mechanisms called "gates" that regulate the flow of information within the recurrent unit, allowing them to selectively remember or forget information over long periods.

Long Short-Term Memory (LSTM)

LSTMs tackle the gradient problems by introducing a dedicated cell state alongside the hidden state. Think of the cell state ($c_t$) as an information highway that allows information to flow through the sequence relatively unchanged, unless explicitly modified. Modifications to the cell state are controlled by three main gates:

1. Forget Gate: Decides what information to throw away from the cell state. It looks at the previous hidden state ($h_{t-1}$) and the current input ($x_t$) and outputs a number between 0 and 1 for each piece of information in the previous cell state ($c_{t-1}$). A 1 represents "completely keep this," while a 0 represents "completely get rid of this."
2. Input Gate: Decides which new information to store in the cell state. This gate has two parts: first, a sigmoid layer ($\sigma$) decides which values to update (the "input gate" proper), and second, a tanh layer creates a vector of new candidate values ($\tilde{c}_t$) that could be added to the state.
3. Output Gate: Decides what to output as the hidden state ($h_t$). It first runs a sigmoid layer to decide which parts of the cell state to output. Then, it puts the (now updated) cell state through tanh (to push values between -1 and 1) and multiplies it by the output of the sigmoid gate, so that only the chosen parts are outputted.

These gates use activation functions like sigmoid ($\sigma$), which squashes values between 0 and 1, to control the flow. By learning the parameters for these gates, the LSTM can learn complex dependencies and retain important information across many time steps, mitigating the vanishing gradient problem.

Flow within an LSTM unit, highlighting the roles of the forget, input, and output gates in managing the cell state and hidden state.

Gated Recurrent Unit (GRU)

The Gated Recurrent Unit (GRU) is a newer generation of recurrent architecture, introduced as a simplification of the LSTM. It combines the forget and input gates into a single update gate and merges the cell state and hidden state. It also introduces a reset gate.

1. Reset Gate: Determines how much of the previous hidden state ($h_{t-1}$) should be forgotten when proposing a new candidate hidden state. If the reset gate outputs close to 0, the previous hidden state is largely ignored.
2. Update Gate: Similar to the LSTM's forget and input gates combined. It decides how much of the previous hidden state ($h_{t-1}$) to keep versus how much of the new candidate hidden state (calculated using the reset gate's influence) to incorporate into the final hidden state ($h_t$).

GRUs have fewer parameters than LSTMs (as they lack a separate output gate and cell state) and can sometimes be computationally more efficient. Empirically, their performance is often comparable to LSTMs on many tasks, though there's no universal winner; the best choice often depends on the specific dataset and problem.

Flow within a GRU unit, showing the reset and update gates controlling the information combined into the new hidden state.

Both LSTMs and GRUs represent significant advancements over simple RNNs by incorporating gating mechanisms. These gates allow the networks to learn which information is relevant to keep or discard over long sequences, making them powerful tools for modeling sequential data in natural language processing, time series analysis, and more. While we won't implement them fully in this introductory course, understanding their purpose is important for recognizing when standard feedforward or simple recurrent networks might be insufficient.