Simple RNN Architecture

Recurrent Neural Networks process sequential information step-by-step, maintaining an internal memory. A fundamental component enabling this capability is the simple RNN cell.

Think of the RNN cell as the computational engine that operates at each time step within a sequence. For every element $x_t$ in the input sequence (at time step $t$), the cell takes two inputs:

1. The current input element, $x_t$.
2. The hidden state from the previous time step, $h_{t-1}$.

Using these inputs, the cell performs a calculation to produce two outputs:

1. The new hidden state, $h_t$. This state captures information from the current input $x_t$ and relevant context from all previous steps (as summarized in $h_{t-1}$). It serves as the network's memory and is passed forward to the next time step.
2. An optional output, $y_t$. This is the prediction or result generated by the network specifically for the current time step $t$. Whether an output is produced at every step depends on the specific task (e.g., predicting the next word needs an output per step, while sentiment classification might only need one output after the whole sequence).

Inside the RNN Cell

The core computation within a simple RNN cell involves combining the current input and the previous hidden state using learned weights and an activation function. The chapter introduction showed the basic equations:

$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$ $y_t = g(W_{hy}h_t + b_y)$

Let's break this down:

• $W_{xh}$: A weight matrix that transforms the input $x_t$.
• $W_{hh}$: A weight matrix that transforms the previous hidden state $h_{t-1}$. This is the recurrent weight matrix, capturing the influence of past information.
• $b_h$: A bias vector added to the hidden state calculation.
• $f$: The activation function for the hidden state. A common choice here is the hyperbolic tangent (tanh), which squashes the values into the range [-1, 1]. This helps regulate the information flow and mitigate some gradient issues (though not completely, as we'll see later).
• $W_{hy}$: A weight matrix that transforms the current hidden state $h_t$ to produce the output $y_t$.
• $b_y$: A bias vector added to the output calculation.
• $g$: The activation function for the output $y_t$. The choice of $g$ depends on the task (e.g., softmax for classification, linear for regression).

A significant aspect of this architecture is weight sharing. The same weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) and biases ($b_h$, $b_y$) are used at every single time step. This means the network learns a single set of parameters that define how to process an input element and update its memory, regardless of where that element appears in the sequence. This makes RNNs parameter-efficient and capable of generalizing across different sequence lengths.

Visualizing the Cell and Recurrence

We can visualize a single RNN cell and its connections:

A single RNN cell processes input $x_t$ and previous state $h_{t-1}$ to compute the new state $h_t$ and output $y_t$. Weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) are applied during these transformations.

The real power comes from chaining these cells together, creating the recurrent loop. The hidden state $h_t$ computed at time step $t$ becomes the input $h_{t-1}$ for the cell at time step $t+1$. This "unrolling" in time allows information to flow through the sequence:

An RNN unrolled for three time steps. The hidden state ($h_t$) computed by the cell at each step is passed as input to the cell at the next step ($t+1$). The same cell parameters (weights $W$ and biases $b$) are used at each step.

This unrolled view is useful, especially when thinking about how gradients flow backward during training (Backpropagation Through Time), which we'll cover next. However, remember that in practice, it's the same cell structure (with the same weights) being applied repeatedly, not distinct copies for each time step. This architecture allows RNNs to process sequences of arbitrary length using a fixed number of parameters.