Having established the nature of sequential data and the limitations of standard feedforward networks in the previous chapter, we now turn our attention to models specifically designed for sequences. This chapter introduces the fundamentals of Recurrent Neural Networks (RNNs).

You will learn the core idea behind RNNs: processing sequences element by element while maintaining an internal 'memory' or hidden state. We will examine the architecture of a simple RNN cell, understanding how input $x_t$ at a time step $t$ combines with the previous hidden state $h_{t-1}$ to produce the current hidden state $h_t$ and an optional output $y_t$. The mathematical operations governing this process, often represented as:

$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$ $y_t = g(W_{hy}h_t + b_y)$

(where $f$ and $g$ are activation functions like $\tanh$ or sigmoid) will be detailed. We will visualize how information flows through time and introduce the essential training algorithm for RNNs: Backpropagation Through Time (BPTT), including the concept of unrolling the network.

By the end of this chapter, you will grasp the operational principles of basic RNNs and the mechanics of their training process.