The Flow of Information in a Network

Imagine a neural network as a sequence of processing stations, organized into layers. The forward propagation process is essentially the path that your input data takes through these stations, starting from the input layer, moving through any hidden layers, and finally arriving at the output layer to produce a result. In feedforward networks, this is a one-way street: information flows strictly forward, from input to output, without looping back.

The Layer-by-Layer Computation

The fundamental principle is that the output of one layer serves as the input for the very next layer. Let's trace this path:

1. Input Layer: This isn't really a computational layer. It simply receives your initial data, often represented as a vector $x$. This vector $x$ can also be considered the "activation" of the input layer, sometimes denoted as $a^{[0]}$.
2. First Hidden Layer: Each neuron in this layer (let's call it layer 1) receives the activations from the input layer ($a^{[0]}$). Inside each neuron $j$ of layer 1, two things happen:
   • A weighted sum of the inputs plus a bias is calculated: $z_j^{[1]} = \sum_i w_{ji}^{[1]} a_i^{[0]} + b_j^{[1]}$.
   • An activation function $g$ is applied to this sum: $a_j^{[1]} = g(z_j^{[1]})$. The collection of all activations $a_j^{[1]}$ from this layer forms the activation vector $a^{[1]}$, which becomes the input for the next layer.
3. Subsequent Hidden Layers: The process repeats for any additional hidden layers (layer 2, layer 3, etc.). For a generic layer $l$, each neuron computes its weighted sum $z_j^{[l]}$ based on the activations $a^{[l-1]}$ from the previous layer, and then applies the activation function $g$ to get its own activation $a_j^{[l]}$. The vector $a^{[l]}$ is then passed forward. $z_j^{[l]} = \sum_i w_{ji}^{[l]} a_i^{[l-1]} + b_j^{[l]}$ $a_j^{[l]} = g(z_j^{[l]})$
4. Output Layer: The final layer performs the same calculations (weighted sum and activation), but its output, $a^{[L]}$ (where $L$ is the last layer), represents the network's final prediction, often denoted as $\hat{y}$. The choice of activation function in this layer is typically specific to the problem type (e.g., Sigmoid for binary classification, Softmax for multi-class classification, or none for regression).

This sequential processing, where the output of layer $l-1$ becomes the input to layer $l$, continues until the final output is generated.

Visualizing the Flow

Consider a simple network with an input layer, one hidden layer, and an output layer. The information moves strictly from left to right.

A simple feedforward network illustrating the flow of information. Inputs $x_1, x_2$ (activations $a^{[0]}$) are processed by the hidden layer to produce activations $a^{[1]}$, which are then processed by the output layer to produce the final prediction $\hat{y}$ (activation $a^{[2]}$).

Each arrow in the diagram represents a weight connecting the output of a neuron in one layer to the input calculation of a neuron in the next layer. The entire process involves calculating the values $z^{[l]}$ and $a^{[l]}$ for each layer $l$, starting from $l=1$ up to the final layer $L$. The result, $a^{[L]}$, is the prediction $\hat{y}$ generated by the network for the given input $x$.

The next sections will detail the specific calculations involved in the linear transformation ($z$) and the application of activation functions ($a$) within each layer, moving towards using matrix operations for efficiency.