Introduction to Neural Networks as Composite Functions

Let's think about a typical neural network, even a relatively simple one. You might have an input layer, one or more hidden layers, and an output layer. Data flows through this network, undergoing transformations at each step. How can we describe this mathematically?

Imagine a single neuron first. It takes some inputs, computes a weighted sum (plus a bias), and then passes that sum through an activation function. This looks like:

1. Linear Combination: $z = w_1 x_1 + w_2 x_2 + ... + w_n x_n + b = \mathbf{w} \cdot \mathbf{x} + b$
2. Activation: $a = g(z)$

Here, $g$ is the activation function (like Sigmoid, ReLU, etc.). The output $a$ is a function of the inputs $\mathbf{x}$.

Now, consider a network with multiple layers. The output activations from one layer ($a^{(l)}$) become the inputs for the next layer ($x^{(l+1)}$). Let's trace the path for a simple network with one hidden layer:

1. Input: $\mathbf{x}$
2. Hidden Layer Calculation:
   • Linear combination: $\mathbf{z}^{(1)} = \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}$
   • Activation: $\mathbf{a}^{(1)} = g^{(1)}(\mathbf{z}^{(1)})$
3. Output Layer Calculation:
   • Linear combination (using hidden layer's output $\mathbf{a}^{(1)}$ as input): $\mathbf{z}^{(2)} = \mathbf{W}^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)}$
   • Activation (final output): $\hat{y} = a^{(2)} = g^{(2)}(\mathbf{z}^{(2)})$

Look closely at the final output $\hat{y}$. We can write it out by substituting the intermediate steps:

$\hat{y} = g^{(2)}(\mathbf{z}^{(2)})$ $\hat{y} = g^{(2)}(\mathbf{W}^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)})$ $\hat{y} = g^{(2)}(\mathbf{W}^{(2)} g^{(1)}(\mathbf{z}^{(1)}) + \mathbf{b}^{(2)})$ $\hat{y} = g^{(2)}(\mathbf{W}^{(2)} g^{(1)}(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}) + \mathbf{b}^{(2)})$

This equation makes it clear: a neural network is a mathematical representation of a composite function. It's a function ($g^{(2)}$) applied to the result of another calculation ($\mathbf{W}^{(2)} \cdot + \mathbf{b}^{(2)}$), which itself involves the output of another function ($g^{(1)}$), and so on, all the way back to the original input $\mathbf{x}$.

Think of it like nesting Russian dolls. The final output depends on the output layer's calculation, which depends on the hidden layer's output, which depends on the hidden layer's calculation, which depends on the input $\mathbf{x}$ and the parameters $\mathbf{W}^{(1)}, \mathbf{b}^{(1)}$.

We can visualize this flow:

A simple feedforward neural network viewed as a sequence of functional transformations. Each layer applies a linear transformation followed by a non-linear activation. The output of one layer feeds into the next, creating a composite function.

Why is this perspective important? Because when we train a neural network, we usually have a loss function (or cost function), let's call it $L$, that measures how far the network's output $\hat{y}$ is from the true target value $y$. The loss $L$ depends on $\hat{y}$. And $\hat{y}$, as we just saw, is a complex function involving all the weights ($\mathbf{W}^{(1)}, \mathbf{W}^{(2)}, ...$) and biases ($\mathbf{b}^{(1)}, \mathbf{b}^{(2)}, ...$) in the network.

To train the network using gradient descent (which we discussed in the previous chapter), we need to calculate the gradient of the loss $L$ with respect to every single weight and bias in the network. For example, we need $\frac{\partial L}{\partial \mathbf{W}^{(1)}}$.

Because $L$ depends on $\hat{y}$, which depends on $\mathbf{a}^{(1)}$, which depends on $\mathbf{z}^{(1)}$, which finally depends on $\mathbf{W}^{(1)}$, we have a chain of dependencies. Calculating this derivative requires navigating through this nested structure. This is precisely where the chain rule becomes indispensable. It provides the mathematical machinery to compute these gradients efficiently by breaking down the complex derivative into a product of simpler derivatives at each step of the network's computation. This systematic application of the chain rule in neural networks is the core idea behind the backpropagation algorithm, which we will explore next.