Now that we understand the conceptual flow of information in an RNN – processing sequence elements one at a time while updating a hidden state – let's formalize the computations happening inside a simple RNN cell at each time step $t$ .

Recall that the cell receives two inputs: the current input feature vector $x_t$ from the sequence and the hidden state $h_{t-1}$ computed at the previous time step. Its goal is to compute the new hidden state $h_t$ , which captures information from the past up to the current step, and potentially an output $y_t$ relevant for this specific time step.

The core computations are typically defined by two equations:

Calculating the new hidden state ( $h_t$ ):
$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$
Calculating the output ( $y_t$ ):
$y_t = g(W_{hy}h_t + b_y)$

Let's break down each component of these equations:

$x_t$ : This is the input vector at the current time step $t$ . If you're processing text, this could be an embedding vector for the word at position $t$ . If it's time series data, it might be the sensor readings at time $t$ . Let's say $x_t$ has dimension $D$ (i.e., $x_t \in \mathbb{R}^D$ ).
$h_{t-1}$ : This is the hidden state vector from the previous time step $t-1$ . It serves as the network's memory of the past. We assume the hidden state has a size $N$ (i.e., $h_{t-1} \in \mathbb{R}^N$ ). For the very first time step ( $t=0$ ), $h_{-1}$ is typically initialized, often as a vector of zeros.
$h_t$ : This is the newly computed hidden state vector for the current time step $t$ . It also has size $N$ (i.e., $h_t \in \mathbb{R}^N$ ).
$W_{xh}$ : This is the weight matrix that transforms the current input $x_t$ . It connects the input layer to the hidden layer. Its dimensions must be compatible with matrix multiplication: if $x_t$ is $D$ -dimensional and $h_t$ is $N$ -dimensional, then $W_{xh}$ must have shape $N \times D$ .
$W_{hh}$ : This is the weight matrix that transforms the previous hidden state $h_{t-1}$ . It represents the recurrent connection from the hidden layer at the previous time step to the hidden layer at the current time step. Its dimensions must connect an $N$ -dimensional vector ( $h_{t-1}$ ) to another $N$ -dimensional vector ( $h_t$ ), so $W_{hh}$ has shape $N \times N$ .
$b_h$ : This is the bias vector added to the hidden state calculation. It has the same dimension as the hidden state, $N$ (i.e., $b_h \in \mathbb{R}^N$ ).
$f$ : This is the activation function applied element-wise to compute the hidden state. In simple RNNs, this is often the hyperbolic tangent function (tanh). The tanh function squashes the values into the range [-1, 1], which helps regulate the activations flowing through the network. Other functions like ReLU could potentially be used, but tanh is traditional for basic RNN hidden states.
$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$
$y_t$ : This is the output vector computed at time step $t$ . Its dimension, let's call it $K$ (i.e., $y_t \in \mathbb{R}^K$ ), depends on the specific task. For instance, in language modeling, you might predict the next word, so $K$ would be the vocabulary size. In time series forecasting, $K$ might be 1 if you predict a single value. Note that an output might not be needed at every time step, depending on the application.
$W_{hy}$ : This is the weight matrix that transforms the current hidden state $h_t$ to produce the output $y_t$ . It connects the hidden layer to the output layer. To transform an $N$ -dimensional hidden state to a $K$ -dimensional output, $W_{hy}$ must have shape $K \times N$ .
$b_y$ : This is the bias vector added to the output calculation. It has the same dimension as the output, $K$ (i.e., $b_y \in \mathbb{R}^K$ ).
$g$ : This is the activation function applied element-wise to compute the final output $y_t$ . The choice of $g$ depends heavily on the nature of the task.
- For regression tasks (like forecasting a continuous value), $g$ might simply be the identity function (no activation applied).
- For binary classification at each step, $g$ would typically be the sigmoid function.
- For multi-class classification (like predicting the next word from a vocabulary), $g$ is usually the softmax function.

A significant aspect of RNNs is parameter sharing. The same set of weight matrices ( $W_{xh}$ , $W_{hh}$ , $W_{hy}$ ) and bias vectors ( $b_h$ , $b_y$ ) are used to perform the calculations at every single time step. This makes the model computationally efficient and allows it to generalize patterns learned at one point in a sequence to other points, regardless of the sequence length. The network learns a single set of rules for how to update its state and produce output based on the current input and its memory.

Here's a visual representation of the calculations within a single RNN cell:

Computational flow within a simple RNN cell. Inputs $x_t$ and $h_{t-1}$ are transformed by weight matrices $W_{xh}$ and $W_{hh}$ respectively, summed with bias $b_h$ , passed through activation $f$ to produce $h_t$ . Then $h_t$ is transformed by $W_{hy}$ , summed with bias $b_y$ , and passed through activation $g$ to produce $y_t$ .

These equations define the forward propagation of information through the RNN cell for a single time step. During training, the network's parameters ( $W_{xh}$ , $W_{hh}$ , $W_{hy}$ , $b_h$ , $b_y$ ) are adjusted to minimize a loss function based on the predicted outputs $y_t$ compared to the true target values. This learning process involves calculating gradients, which, in the context of RNNs, requires a technique called Backpropagation Through Time (BPTT), the topic we will address next.