Calculating the Final Output Prediction

At the output layer, a neural network produces its ultimate prediction, $\hat{y}$. This is the final stage of processing for input data, which involves linear transformations and non-linear activations. Operations within the output layer are similar to those in hidden layers, but the resulting output is specifically tailored to the problem being addressed.

Let's denote the activations from the last hidden layer (or the input layer, if there are no hidden layers) as $A^{[L-1]}$. To get the output prediction $A^{[L]}$, we perform two final steps:

1. Calculate the Weighted Sum for the Output Layer: Just like in the hidden layers, we compute the linear combination of the inputs to this layer ($A^{[L-1]}$), using the output layer's weights ($W^{[L]}$) and biases ($b^{[L]}$).

   $$Z^{[L]} = W^{[L]} A^{[L-1]} + b^{[L]}$$

   Here, $Z^{[L]}$ represents the pre-activation values for the output layer neurons. The dimensions of $W^{[L]}$, $b^{[L]}$, and consequently $Z^{[L]}$ depend on the number of neurons in the last hidden layer and the number of neurons in the output layer. The number of output neurons is determined by the task: one neuron for binary classification or regression, and $C$ neurons for multi-class classification with $C$ classes.

2. Apply the Output Layer Activation Function: The choice of activation function $g^{[L]}$ for the output layer is significant because it formats the final output $A^{[L]}$ into the desired prediction format.

   $$A^{[L]} = g^{[L]}(Z^{[L]})$$

   This final activation $A^{[L]}$ is the network's prediction, which we often denote as $\hat{y}$.

   The final step in forward propagation involves applying a linear transformation and an activation function specific to the output layer to produce the network's prediction, $\hat{y}$.

Choosing the Right Output Activation

Unlike hidden layers where ReLU is a common choice, the output layer's activation function ($g^{[L]}$) depends directly on the nature of the prediction task:

• Regression: If the network needs to predict a continuous numerical value (like predicting house prices), a linear activation function is typically used. This means the output is simply the weighted sum: $A^{[L]} = Z^{[L]}$. The output can range from $-\infty$ to $+\infty$.
• Binary Classification: For tasks where the output is one of two classes (e.g., spam or not spam, 0 or 1), the Sigmoid function is used. It squashes the output $Z^{[L]}$ into the range (0, 1), which can be interpreted as the probability of the positive class. $$\hat{y} = A^{[L]} = \sigma(Z^{[L]}) = \frac{1}{1 + e^{-Z^{[L]}}}$$
• Multi-class Classification: When classifying among more than two mutually exclusive classes (e.g., identifying digits 0-9), the Softmax function is the standard choice. It takes the vector $Z^{[L]}$ (where each element corresponds to a class) and converts it into a probability distribution. Each element in the output vector $A^{[L]}$ is between 0 and 1, and the sum of all elements is 1. $$\hat{y}_i = A^{[L]}_i = \text{softmax}(Z^{[L]})_i = \frac{e^{Z^{[L]}_i}}{\sum_{j=1}^{C} e^{Z^{[L]}_j}} \quad \text{for } i=1, ..., C$$ Here, $C$ is the number of classes, and $\hat{y}_i$ represents the predicted probability that the input belongs to class $i$.

Understanding the Prediction ($\hat{y}$)

The final output vector $A^{[L]}$, which we call $\hat{y}$, contains the network's prediction(s) based on the input $X$.

• For regression, $\hat{y}$ is a single continuous value (or multiple values if predicting multiple targets).
• For binary classification, $\hat{y}$ is a single value between 0 and 1, representing $P(\text{class}=1 | X)$. You might apply a threshold (e.g., 0.5) to convert this probability into a definite class label (0 or 1).
• For multi-class classification, $\hat{y}$ is a vector of probabilities, one for each class. The class with the highest probability is typically chosen as the final predicted label.

With the calculation of $\hat{y}$, the forward propagation process is complete. The network has taken an input $X$ and produced a prediction $\hat{y}$ by passing information through its layers, applying linear transformations and activation functions along the way. This prediction can now be compared to the true label $y$ using a loss function, which is the first step in the training process (covered in the next chapter).