The Core Idea: Processing Sequences Iteratively

Standard feedforward neural networks have a fundamental limitation when dealing with sequential data: they lack memory. Each input is processed independently, without regard for the order or context provided by previous inputs. Think about reading this sentence. To understand the meaning of the word "sentence", your brain uses the context provided by the preceding words "reading this". Feedforward networks cannot naturally do this. They are designed for fixed-size inputs where order does not inherently carry meaning in the same way.

This brings us to the central idea behind Recurrent Neural Networks (RNNs). Instead of processing the entire sequence at once, or processing each element in isolation, RNNs process sequences one element at a time, iteratively.

Imagine processing a sequence $x = (x_1, x_2, ..., x_T)$ . An RNN takes the first element $x_1$ , processes it, and produces an internal state (often called the hidden state), let's call it $h_1$ . When it's time to process the second element $x_2$ , the RNN doesn't just look at $x_2$ . It also considers the information it gleaned from the first step, summarized in the hidden state $h_1$ . It combines the new input $x_2$ with the previous state $h_1$ to compute the next hidden state $h_2$ .

This process repeats for every element in the sequence:

At step $t=1$ : Compute $h_1$ using $x_1$ (and often an initial state $h_0$ , usually set to zeros).
At step $t=2$ : Compute $h_2$ using $x_2$ and $h_1$ .
At step $t=3$ : Compute $h_3$ using $x_3$ and $h_2$ .
...
At step $t=T$ : Compute $h_T$ using $x_T$ and $h_{T-1}$ .

The hidden state $h_t$ , which acts as the network's memory. It captures information about all the preceding elements ( $x_1, ..., x_{t-1}$ ) that the network deems relevant for processing the current element $x_t$ and future elements. This state is passed from one time step to the next, creating a loop or recurrence in the network's connections.

A view of an RNN processing one step. The input $x_t$ and the previous hidden state $h_{t-1}$ are combined by the RNN operation to produce the current hidden state $h_t$ . This state $h_t$ is then carried forward to be used in the next time step (processing $x_{t+1}$ ), and an optional output $y_t$ can be generated at this step.

This iterative processing with a persistent state is fundamentally different from feedforward networks. While a feedforward network applies the same transformation to every input independently, an RNN applies the same transformation rule (the same set of weights within the RNN cell) at each step, but the outcome (the hidden state $h_t$ and output $y_t$ ) depends on both the current input $x_t$ and the history summarized in $h_{t-1}$ .

This ability to maintain context makes RNNs naturally suited for tasks where order matters and information needs to be aggregated over time, like understanding language, forecasting time series, or analyzing audio signals. The specific mathematical functions used inside the "RNN Operation" box in the diagram define how the previous state and current input are combined, and we will look into those details in the following sections.

Was this section helpful?

References

Finding Structure in Time, Jeffrey L. Elman, 1990 Cognitive Science, Vol. 14 (Wiley) DOI: 10.1207/s15516709cog1402_1 - A foundational paper introducing Simple Recurrent Networks (SRNs), demonstrating how neural networks can develop an internal representation of sequential information and context over time, thereby laying much of the groundwork for iterative processing.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - This authoritative textbook provides a comprehensive introduction to recurrent neural networks, clearly explaining their fundamental structure, the concept of hidden states, and how they process sequential data iteratively.
Recurrent Neural Networks and Language Models, Christopher Manning, 2023 (Stanford University) - Lecture slides from a highly regarded course, offering clear explanations and visual representations of RNNs, their iterative processing, and the role of the hidden state in maintaining context for natural language processing tasks.

The Core Idea: Processing Sequences Iteratively

This process repeats for every element in the sequence:

At step $t=1$ : Compute $h_1$ using $x_1$ (and often an initial state $h_0$ , usually set to zeros).
At step $t=2$ : Compute $h_2$ using $x_2$ and $h_1$ .
At step $t=3$ : Compute $h_3$ using $x_3$ and $h_2$ .
...
At step $t=T$ : Compute $h_T$ using $x_T$ and $h_{T-1}$ .

A view of an RNN processing one step. The input $x_t$ and the previous hidden state $h_{t-1}$ are combined by the RNN operation to produce the current hidden state $h_t$ . This state $h_t$ is then carried forward to be used in the next time step (processing $x_{t+1}$ ), and an optional output $y_t$ can be generated at this step.

Was this section helpful?

References

Finding Structure in Time, Jeffrey L. Elman, 1990 Cognitive Science, Vol. 14 (Wiley) DOI: 10.1207/s15516709cog1402_1 - A foundational paper introducing Simple Recurrent Networks (SRNs), demonstrating how neural networks can develop an internal representation of sequential information and context over time, thereby laying much of the groundwork for iterative processing.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - This authoritative textbook provides a comprehensive introduction to recurrent neural networks, clearly explaining their fundamental structure, the concept of hidden states, and how they process sequential data iteratively.
Recurrent Neural Networks and Language Models, Christopher Manning, 2023 (Stanford University) - Lecture slides from a highly regarded course, offering clear explanations and visual representations of RNNs, their iterative processing, and the role of the hidden state in maintaining context for natural language processing tasks.