Recurrent Neural Network Concepts

Many problems involve sequential data where the order of elements is fundamental to understanding the information. For example, tasks like understanding a sentence, predicting the next word someone will type, analyzing stock market trends over time, or interpreting sensor readings from a machine require sequential processing. Standard feedforward neural networks, such as Dense networks or convolutional neural networks (CNNs), process inputs independently. These networks do not inherently maintain information about past inputs when processing the current one, which is a significant limitation for sequential tasks.

Recurrent Neural Networks (RNNs) are designed specifically to address this challenge. The core idea behind an RNN is the concept of recurrence: the network performs the same operation for every element of a sequence, but the output for each element depends not only on the current input but also on the results from previous elements. This is achieved by introducing a "loop" within the network architecture.

The Hidden State: Memory in the Network

Imagine processing a sentence word by word. To understand the meaning of the word "it" in "The cat chased the mouse, and then it ran away," you need to remember whether "it" refers to the cat or the mouse. An RNN accomplishes this kind of context retention through an internal hidden state (often denoted as $h$).

At each step $t$ in the sequence (e.g., processing the $t$-th word or the $t$-th time point), the RNN takes two inputs:

1. The current input element $x_t$.
2. The hidden state from the previous step $h_{t-1}$.

It then computes the new hidden state $h_t$ and, optionally, an output $y_t$. The important part is that the computation of $h_t$ uses $h_{t-1}$. This creates a dependency chain where information from earlier steps can propagate through the sequence via the hidden state. This allows the network to have a form of "memory," retaining context from past elements.

The update rule for the hidden state at time step $t$ can be represented as:

$$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$$

And the output at time step $t$ (if needed at each step) as:

$$y_t = g(W_{hy}h_t + b_y)$$

Here:

• $x_t$ is the input at time step $t$.
• $h_{t-1}$ is the hidden state from the previous time step $t-1$.
• $h_t$ is the new hidden state at time step $t$.
• $y_t$ is the output at time step $t$.
• $W_{xh}$, $W_{hh}$, and $W_{hy}$ are weight matrices (input-to-hidden, hidden-to-hidden, and hidden-to-output, respectively).
• $b_h$ and $b_y$ are bias vectors.
• $f$ and $g$ are activation functions (e.g., tanh or relu for $f$, softmax or sigmoid for $g$ depending on the task).

The same set of weights ($W_{xh}$, $W_{hh}$, $W_{hy}$) and biases ($b_h$, $b_y$) are used across all time steps. This parameter sharing makes RNNs efficient and capable of generalizing patterns across different positions in the sequence.

Unrolling the RNN

To better visualize the flow of information, it's common to "unroll" the RNN loop over the sequence length. Imagine the sequence has $T$ time steps. Unrolling means creating $T$ copies of the network, one for each time step, and connecting the hidden state output of one step to the hidden state input of the next.

An RNN unrolled through time. Each RNN Cell block represents the same set of weights applied at different time steps. The hidden state h carries information from one step to the next. Input x and output y occur at each step.

This unrolled view makes it clearer how gradients flow during backpropagation (backpropagation through time, or BPTT) and why capturing long-range dependencies can sometimes be difficult, leading to issues like the vanishing gradient problem, which we'll discuss later.

Input and Output Scenarios

RNNs are flexible in handling different sequence input/output relationships:

• Many-to-one: Sequence input, single output (e.g., sentiment classification of a sentence).
• One-to-many: Single input, sequence output (e.g., image captioning - input image, output sequence of words).
• Many-to-many (aligned): Sequence input, sequence output where input/output lengths match (e.g., part-of-speech tagging for each word in a sentence).
• Many-to-many (delayed): Sequence input, sequence output where output starts after some delay or processing the entire input (e.g., machine translation).

The core RNN concept provides the foundation for these different patterns. In Keras, layers like SimpleRNN, LSTM, and GRU implement this recurrent behavior. We will explore how to use these layers, starting with the basic SimpleRNN, in the next section.