Let's put the concepts of neurons, weights, biases, and activation functions into practice. Understanding how to calculate the output of a single neuron is fundamental, as this exact process is repeated for every neuron in a network during forward propagation.

Recall the two main steps for a neuron's computation:

Linear Combination: Calculate the weighted sum of the inputs plus the bias. We often denote this intermediate value as $z$ . $z = (\sum_{i} w_i x_i) + b$ Where $w_i$ are the weights, $x_i$ are the inputs, and $b$ is the bias.
Activation: Apply a non-linear activation function $f$ to the result $z$ to get the neuron's final output, denoted as $a$ . $a = f(z)$

Example Calculation

Imagine a single neuron with two inputs. Let's define its parameters and the input values:

Inputs: $x = [x_1, x_2] = [2.0, 3.0]$
Weights: $w = [w_1, w_2] = [0.5, -1.0]$
Bias: $b = 1.0$

Step 1: Calculate the Linear Combination (z)

We multiply each input by its corresponding weight, sum these products, and add the bias:

$z = (w_1 \times x_1) + (w_2 \times x_2) + b$ $z = (0.5 \times 2.0) + (-1.0 \times 3.0) + 1.0$ $z = 1.0 - 3.0 + 1.0$ $z = -1.0$

So, the result of the linear combination is $z = -1.0$ .

Visualizing the inputs, weights, and bias feeding into the calculation of the linear combination ( $z$ ) and the subsequent activation ( $a$ ).

Step 2: Apply an Activation Function (f)

Now, we apply an activation function to $z = -1.0$ . The output $a$ will depend on which function we choose. Let's calculate the output using the three common functions we discussed: Sigmoid, Tanh, and ReLU.

Sigmoid: The Sigmoid function squashes the input into a range between 0 and 1. $a = \sigma(z) = \frac{1}{1 + e^{-z}}$ $a = \frac{1}{1 + e^{-(-1.0)}} = \frac{1}{1 + e^{1.0}}$ Using $e \approx 2.71828$ : $a \approx \frac{1}{1 + 2.71828} = \frac{1}{3.71828} \approx 0.2689$ So, if using Sigmoid, the neuron's output is approximately $0.2689$ .
Tanh (Hyperbolic Tangent): Tanh squashes the input into a range between -1 and 1. $a = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$ $a = \frac{e^{-1.0} - e^{1.0}}{e^{-1.0} + e^{1.0}}$ Using $e^{1.0} \approx 2.71828$ and $e^{-1.0} \approx 0.36788$ : $a \approx \frac{0.36788 - 2.71828}{0.36788 + 2.71828} = \frac{-2.3504}{3.08616} \approx -0.7616$ If using Tanh, the neuron's output is approximately $-0.7616$ .
ReLU (Rectified Linear Unit): ReLU outputs the input directly if it's positive, and 0 otherwise. $a = \text{ReLU}(z) = \max(0, z)$ $a = \max(0, -1.0)$ $a = 0$ If using ReLU, the neuron's output is $0$ .

Summary of Results

For the given inputs $x = [2.0, 3.0]$ , weights $w = [0.5, -1.0]$ , and bias $b = 1.0$ , we first found $z = -1.0$ . The final neuron output $a$ depends on the chosen activation function:

Using Sigmoid: $a \approx 0.2689$
Using Tanh: $a \approx -0.7616$
Using ReLU: $a = 0.0$

This exercise demonstrates the core calculation performed by every artificial neuron. In a real network, the output $a$ from this neuron could then become an input $x$ for neurons in the next layer. Try changing the input values, weights, or bias and recalculate the outputs using different activation functions to solidify your understanding. This simple operation, repeated many times across layers, is what allows neural networks to process information and learn complex patterns.