Training a Recurrent Neural Network presents a unique situation compared to standard feedforward networks. The core difficulty lies in the recurrent connection: the output of a time step feeds back into the network as input for the next time step. This creates cycles in the computational graph, which standard backpropagation isn't directly equipped to handle. How, then, do we calculate gradients when the network's state at time $t$ depends on its state at time $t-1$ , which depends on $t-2$ , and so on?

The solution is a technique called Backpropagation Through Time (BPTT). To understand and implement BPTT, we first need to visualize the flow of computation in a different way. We do this by conceptually unrolling the network through time.

Imagine taking the compact representation of an RNN cell with its loop and unfolding it across the sequence length. For a sequence of length $T$ , we create $T$ copies of the cell, one for each time step. The recurrent connection from a cell at time $t-1$ to the cell at time $t$ is now represented as a directed connection between copy $t-1$ and copy $t$ .

An RNN unrolled for three time steps. The input $x_t$ and previous hidden state $h_{t-1}$ are used to compute the current hidden state $h_t$ . The same weight matrices ( $W_{xh}$ , $W_{hh}$ , $W_{hy}$ ) are applied at each time step.

This unrolled network looks very much like a deep feedforward network. Each time step can be thought of as a layer receiving input from the previous layer (the previous time step's hidden state) and the external input for that specific time step. This transformation is purely conceptual; we don't actually create multiple copies of the weights in memory. It's a way to visualize the dependencies required for gradient calculation.

BPTT on the Unrolled Graph

With the network unrolled, we can now apply the principles of backpropagation. BPTT involves performing a forward pass through the unrolled network, computing the output and loss (typically at the end of the sequence, or at each step depending on the task), and then performing a backward pass to compute gradients.

During the backward pass:

The error gradient is calculated at the output(s).
This gradient is propagated backward, layer by layer (which means time step by time step in the unrolled view).
Importantly, the gradient calculation for the hidden state $h_t$ needs to account for two paths: the influence of $h_t$ on the output $y_t$ (if applicable) and its influence on the next hidden state $h_{t+1}$ .
The gradients for the shared weight matrices ( $W_{xh}$ , $W_{hh}$ , $W_{hy}$ ) and biases ( $b_h$ , $b_y$ ) are calculated at each time step in the unrolled graph. Since these weights are shared, the total gradient for each weight matrix is the sum (or average) of the gradients computed at each individual time step.

For example, the gradient of the loss $L$ with respect to the weight matrix $W_{hh}$ is computed by summing its influence across all time steps where it was used:

\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial W_{hh}}

where $L_t$ represents the portion of the loss dependent on the computation at time step $t$ . This summation reflects the fact that the same $W_{hh}$ affects the hidden state calculation at every step.

Implications and Considerations

Unrolling the network clarifies the computational flow for training but also highlights some practical aspects:

Memory Usage: BPTT requires storing the activations (inputs $x_t$ , hidden states $h_t$ ) from the forward pass for all the time steps involved in the gradient calculation. For sequences with thousands of steps, this can lead to significant memory consumption.
Computational Cost: The backward pass involves computations proportional to the number of time steps the gradient is propagated through.
Truncated BPTT: Because processing very long sequences can be computationally expensive and memory-intensive, a common modification called Truncated Backpropagation Through Time (TBPTT) is often used. Instead of unrolling and backpropagating through the entire sequence length, TBPTT processes the sequence in shorter segments, performing backpropagation only for a fixed number of steps (the truncation length) back in time. This makes training feasible for longer sequences, although it limits the distance over which dependencies can be learned directly through gradients in a single update. We'll discuss the implications of this, particularly regarding long-range dependencies, in Chapter 4.

In essence, unrolling is the conceptual bridge that allows us to apply gradient-based optimization techniques, derived from standard backpropagation, to the recurrent structure of RNNs. It transforms the temporal loop into a sequence of operations, making the calculation of gradients with respect to the shared parameters feasible through BPTT.