Recurrent Neural Networks (RNNs) Overview

While feedforward networks process inputs independently, many problems involve sequences where the order matters and context from previous items influences the current one. Think about understanding a sentence, predicting stock prices, or transcribing speech. Each word, price point, or sound segment depends on what came before it. Standard feedforward networks lack an inherent mechanism to "remember" past information within a sequence.

This is where Recurrent Neural Networks (RNNs) come in. They are specifically designed to handle sequential data by introducing the concept of recurrence.

The Idea of Memory: Hidden State

The defining feature of an RNN is its internal loop. At each step in processing a sequence, the network takes into account not only the current input but also information it has retained from previous steps. This retained information is stored in what's called the hidden state.

Imagine reading a sentence. You don't process each word in isolation. Your understanding of the current word is heavily influenced by the words you've already read. The hidden state in an RNN acts like this running summary or context. It captures information about the preceding elements in the sequence.

Processing Sequences Step-by-Step

An RNN processes a sequence one element (or "time step") at a time. For each time step $t$:

1. It takes the input for that step, let's call it $x_t$.
2. It also takes the hidden state from the previous step, $h_{t-1}$.
3. It combines $x_t$ and $h_{t-1}$ using a set of learned weights to compute the new hidden state, $h_t$. This new hidden state now carries information from all steps up to $t$.
4. Optionally, it can produce an output $y_t$ for the current step, often based on the hidden state $h_t$.

Significantly, the same set of weights (the rules for combining input and previous state, and for producing the output) are used at every time step. This weight sharing makes RNNs efficient and allows them to generalize patterns across different sequence lengths.

Visualizing the Recurrence: Unrolling in Time

It's often helpful to visualize an RNN by "unrolling" it through time. Instead of drawing the loop, we can draw a chain representing the network's state at each time step.

An RNN "unrolled" in time. The same RNN Cell (representing the shared weights) processes input $x_t$ and the previous hidden state $h_{t-1}$ to produce the new hidden state $h_t$ and an optional output $y_t$. The hidden state is passed from one time step to the next.

Mathematically, the core computations inside a simple RNN cell at time step $t$ are often represented as:

Calculate the new hidden state $h_t$:

$$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$

Calculate the output $y_t$:

$$y_t = W_{hy} h_t + b_y$$

Here:

• $x_t$ is the input at time step $t$.
• $h_{t-1}$ is the hidden state from the previous time step.
• $h_t$ is the new hidden state at time step $t$.
• $y_t$ is the output at time step $t$.
• $W_{hh}$, $W_{xh}$, and $W_{hy}$ are weight matrices learned during training. They represent how much influence the previous hidden state, the current input, and the current hidden state have, respectively. These weights are shared across all time steps.
• $b_h$ and $b_y$ are bias vectors, also learned.
• $\tanh$ is the hyperbolic tangent activation function, commonly used in simple RNNs to introduce non-linearity. Another activation function could be used for the output layer depending on the task (e.g., Softmax for classification).

The important part is the recurrent formula for $h_t$, which depends on both the current input $x_t$ and the previous hidden state $h_{t-1}$. This dependence is what gives RNNs their memory.

Where are RNNs Used?

RNNs excel in tasks involving sequential patterns:

• Natural Language Processing (NLP): Language modeling (predicting the next word), machine translation, sentiment analysis, text generation.
• Speech Recognition: Converting spoken audio into text.
• Time Series Analysis: Forecasting stock prices, weather prediction, analyzing sensor data.
• Video Analysis: Understanding actions happening over time in video frames.

Challenges and Next Steps

While powerful, simple RNNs like the one described above can struggle with learning long-range dependencies. Information from early time steps can get diluted or lost as it propagates through many steps, a problem often referred to as the vanishing gradient problem. Conversely, gradients can sometimes grow excessively large, known as the exploding gradient problem.

These challenges led to the development of more sophisticated recurrent architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which use gating mechanisms to better control the flow of information and memory. We will briefly touch upon these later in the chapter.

For now, understanding the fundamental concept of recurrence, the role of the hidden state, and the step-by-step processing is sufficient. In the following sections, we will see how to implement a basic RNN using PyTorch's nn.RNN module.