Mathematical Formulation of an RNN Cell

Recurrent Neural Networks process sequence elements one at a time, updating a hidden state. We will now formalize the computations happening inside a simple RNN cell at each time step $t$.

Recall that the cell receives two inputs: the current input feature vector $x_t$ from the sequence and the hidden state $h_{t-1}$ computed at the previous time step. Its goal is to compute the new hidden state $h_t$, which captures information from the past up to the current step, and potentially an output $y_t$ relevant for this specific time step.

The core computations are typically defined by two equations:

1. Calculating the new hidden state ($h_t$):

   $$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$$

2. Calculating the output ($y_t$):

   $$y_t = g(W_{hy}h_t + b_y)$$

Let's break down each component of these equations:

• $x_t$: This is the input vector at the current time step $t$. If you're processing text, this could be an embedding vector for the word at position $t$. If it's time series data, it might be the sensor readings at time $t$. Let's say $x_t$ has dimension $D$ (i.e., $x_t \in \mathbb{R}^D$).

• $h_{t-1}$: This is the hidden state vector from the previous time step $t-1$. It serves as the network's memory of the past. We assume the hidden state has a size $N$ (i.e., $h_{t-1} \in \mathbb{R}^N$). For the very first time step ($t=0$), $h_{-1}$ is typically initialized, often as a vector of zeros.

• $h_t$: This is the newly computed hidden state vector for the current time step $t$. It also has size $N$ (i.e., $h_t \in \mathbb{R}^N$).

• $W_{xh}$: This is the weight matrix that transforms the current input $x_t$. It connects the input layer to the hidden layer. Its dimensions must be compatible with matrix multiplication: if $x_t$ is $D$-dimensional and $h_t$ is $N$-dimensional, then $W_{xh}$ must have shape $N \times D$.

• $W_{hh}$: This is the weight matrix that transforms the previous hidden state $h_{t-1}$. It represents the recurrent connection from the hidden layer at the previous time step to the hidden layer at the current time step. Its dimensions must connect an $N$-dimensional vector ($h_{t-1}$) to another $N$-dimensional vector ($h_t$), so $W_{hh}$ has shape $N \times N$.

• $b_h$: This is the bias vector added to the hidden state calculation. It has the same dimension as the hidden state, $N$ (i.e., $b_h \in \mathbb{R}^N$).

• $f$: This is the activation function applied element-wise to compute the hidden state. In simple RNNs, this is often the hyperbolic tangent function (tanh). The tanh function squashes the values into the range [-1, 1], which helps regulate the activations flowing through the network. Other functions like ReLU could potentially be used, but tanh is traditional for basic RNN hidden states.

  $$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$

• $y_t$: This is the output vector computed at time step $t$. Its dimension, let's call it $K$ (i.e., $y_t \in \mathbb{R}^K$), depends on the specific task. For instance, in language modeling, you might predict the next word, so $K$ would be the vocabulary size. In time series forecasting, $K$ might be 1 if you predict a single value. Note that an output might not be needed at every time step, depending on the application.

• $W_{hy}$: This is the weight matrix that transforms the current hidden state $h_t$ to produce the output $y_t$. It connects the hidden layer to the output layer. To transform an $N$-dimensional hidden state to a $K$-dimensional output, $W_{hy}$ must have shape $K \times N$.

• $b_y$: This is the bias vector added to the output calculation. It has the same dimension as the output, $K$ (i.e., $b_y \in \mathbb{R}^K$).

• $g$: This is the activation function applied element-wise to compute the final output $y_t$. The choice of $g$ depends heavily on the nature of the task.

  • For regression tasks (like forecasting a continuous value), $g$ might simply be the identity function (no activation applied).
  • For binary classification at each step, $g$ would typically be the sigmoid function.
  • For multi-class classification (like predicting the next word from a vocabulary), $g$ is usually the softmax function.

A significant aspect of RNNs is parameter sharing. The same set of weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) and bias vectors ($b_h$, $b_y$) are used to perform the calculations at every single time step. This makes the model computationally efficient and allows it to generalize patterns learned at one point in a sequence to other points, regardless of the sequence length. The network learns a single set of rules for how to update its state and produce output based on the current input and its memory.

Here's a visual representation of the calculations within a single RNN cell:

Computational flow within a simple RNN cell. Inputs $x_t$ and $h_{t-1}$ are transformed by weight matrices $W_{xh}$ and $W_{hh}$ respectively, summed with bias $b_h$, passed through activation $f$ to produce $h_t$. Then $h_t$ is transformed by $W_{hy}$, summed with bias $b_y$, and passed through activation $g$ to produce $y_t$.

These equations define the forward propagation of information through the RNN cell for a single time step. During training, the network's parameters ($W_{xh}$, $W_{hh}$, $W_{hy}$, $b_h$, $b_y$) are adjusted to minimize a loss function based on the predicted outputs $y_t$ compared to the true target values. This learning process involves calculating gradients, which, in the context of RNNs, requires a technique called Backpropagation Through Time (BPTT), the topic we will address next.