Information Flow in RNNs

To truly understand how a Recurrent Neural Network processes sequential information, it's helpful to visualize the flow of data through the network over time. While we often draw an RNN cell with a recurrent loop pointing back to itself, for analysis and understanding training, we can "unroll" this loop across the time steps of the input sequence.

Imagine you have an input sequence $x = (x_1, x_2, ..., x_T)$. When this sequence is fed into an RNN, the network processes it one element ($x_t$) at each time step $t$. The main aspect is the hidden state, $h_t$, which gets updated at each step and carries information from previous steps forward.

Let's visualize this unrolling process:

An RNN unrolled through time. The same RNN cell, with the same weights (W, U, V corresponding to $W_{xh}$, $W_{hh}$, and $W_{hy}$ respectively), is applied at each time step. The hidden state ($h$) acts as the memory, connecting one time step to the next.

In this unrolled view:

1. Input Propagation: At each time step $t$, the corresponding input element $x_t$ is fed into the RNN cell for that step.
2. State Propagation: The hidden state $h_{t-1}$ from the previous time step is also fed into the cell at time step $t$. For the very first time step ($t=1$), an initial hidden state $h_0$ is typically used, often initialized to zeros.
3. Calculation: Inside the cell at time $t$, the input $x_t$ and the previous state $h_{t-1}$ are combined using the weight matrices and activation function $f$ to compute the new hidden state $h_t$: $h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$
4. Output Generation: The new hidden state $h_t$ is then often used to compute the output $y_t$ for the current time step, using another weight matrix and activation function $g$: $y_t = g(W_{hy}h_t + b_y)$ (Note: An output might not be generated or needed at every time step, depending on the specific task.)
5. Iteration: This process repeats for the next time step, $t+1$. The just-computed hidden state $h_t$ becomes the "previous hidden state" for the calculation at $t+1$.

This unrolling clearly shows how the hidden state $h_t$ at any given time step is a function of the current input $x_t$ and all preceding inputs ($x_1, ..., x_{t-1}$) via the chain of hidden states ($h_0, h_1, ..., h_{t-1}$). This chaining of hidden states is how the RNN maintains context or "memory" of the sequence processed so far.

A significant point illustrated by the unrolled view is parameter sharing. Notice that the same weight matrices ($W_{xh}, W_{hh}, W_{hy}$) and biases ($b_h, b_y$) are used within the RNN cell at every single time step. The network learns one set of parameters that is applied repeatedly across the sequence. This makes the model efficient and capable of handling sequences of variable lengths.

Understanding this flow of information and the concept of unrolling is foundational for grasping how RNNs are trained. The dependencies across time steps mean that the error calculated at a later time step needs to be propagated back through this unrolled structure to update the shared weights. This process is known as Backpropagation Through Time (BPTT), which we will examine next.