Basic RNN Architecture

Building upon the idea of recurrence and hidden states, let's examine the structure of a basic Recurrent Neural Network (RNN). Unlike feedforward networks where information flows strictly in one direction, RNNs incorporate a loop, allowing information from previous steps to persist and influence the current step. This internal memory is what makes them suitable for sequential data.

At the heart of an RNN is the RNN cell. Think of this cell as the fundamental processing unit that gets reused for each element in the input sequence. At each time step $t$, the cell takes two inputs:

1. The current input element from the sequence, $x_t$.
2. The hidden state from the previous time step, $h_{t-1}$.

Based on these inputs, the cell performs two main computations:

1. It calculates the new hidden state, $h_t$.
2. It produces an output for the current time step, $y_t$.

The core mechanism involves combining the current input $x_t$ and the previous hidden state $h_{t-1}$ using specific weight matrices and an activation function.

Inside the RNN Cell

Let's break down the calculations happening within a simple RNN cell at a single time step $t$:

1. Calculating the Hidden State ($h_t$): The new hidden state $h_t$ is computed by applying a linear transformation to both the current input $x_t$ and the previous hidden state $h_{t-1}$, adding a bias, and then passing the result through a non-linear activation function (commonly tanh or ReLU).

   The formula is typically:

   $$h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$

   Here:

   • $x_t$ is the input vector at time step $t$.
   • $h_{t-1}$ is the hidden state vector from the previous time step $t-1$. (For the very first time step, $t=0$, $h_{-1}$ is usually initialized as a vector of zeros).
   • $W_{xh}$ is the weight matrix connecting the input $x_t$ to the hidden state.
   • $W_{hh}$ is the weight matrix connecting the previous hidden state $h_{t-1}$ to the current hidden state $h_t$. This is the matrix associated with the "recurrent" connection or loop.
   • $b_h$ is the bias vector for the hidden state calculation.
   • tanh is a common choice for the activation function, squashing the values to be between -1 and 1.

2. Calculating the Output ($y_t$): The output $y_t$ at the current time step is typically computed by applying another linear transformation to the newly calculated hidden state $h_t$, adding a bias, and potentially passing it through another activation function depending on the specific task (e.g., a softmax function if the task is classification at each step).

   The formula is often:

   $$y_t = \text{activation}(W_{hy} h_t + b_y)$$

   Here:

   • $h_t$ is the current hidden state vector.
   • $W_{hy}$ is the weight matrix connecting the hidden state to the output.
   • $b_y$ is the bias vector for the output calculation.
   • activation is an output activation function suitable for the problem (e.g., identity for regression, softmax for multi-class classification).

Unrolling the RNN Through Time

To better visualize how an RNN processes a sequence, we often "unroll" the network through time. This means drawing the network as if it were a deep feedforward network, with one layer corresponding to each time step.

An RNN unrolled through time. Each RNN Cell block represents the processing at one time step, taking the input $x_t$ and the previous hidden state $h_{t-1}$ to produce the output $y_t$ and the next hidden state $h_t$. Notice the hidden state $h$ is passed from one time step to the next (dashed lines). Critically, the same weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) are used across all time steps.

The most important aspect illustrated by unrolling is parameter sharing. The weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) and bias vectors ($b_h$, $b_y$) are the same for every time step. This makes the model efficient, as it doesn't need a separate set of parameters for each input position. It learns a general transformation that can be applied repeatedly across the sequence.

The hidden state $h_t$ acts as the network's memory. It captures information from all the previous time steps ($x_0, x_1, ..., x_{t-1}$) and combines it with the current input $x_t$ to influence the current output $y_t$ and the subsequent hidden state $h_{t+1}$. This simple structure allows RNNs to model dependencies within sequences, forming the basis for processing sequential data. However, as we'll see next, this basic architecture faces certain difficulties when dealing with long sequences.