From Biological to Artificial Neurons

Neural networks draw significant inspiration from the way biological brains process information. While the ultimate goal isn't to perfectly replicate the brain, understanding the basic computational unit of the brain, the neuron, provides a useful starting point for building artificial learning systems.

The Biological Inspiration

A biological neuron is a specialized cell designed for information processing and transmission. At a high level, it operates as follows:

1. Receiving Signals: Dendrites act like input wires, receiving electrochemical signals from other neurons through connections called synapses.
2. Integrating Signals: The cell body, or soma, integrates these incoming signals. The strength of each incoming signal can vary (influenced by the synapse).
3. Generating Output: If the combined signal strength within the soma surpasses a certain threshold, the neuron "fires," sending an electrical pulse down its axon.
4. Transmitting Signals: The axon acts as an output wire, carrying the pulse away from the cell body. At its end, the axon branches out and forms synapses with the dendrites of other neurons, passing the signal along.

A simplified view of information flow in a biological neuron.

This biological process of receiving weighted inputs, integrating them, and producing an output based on a threshold provides the core analogy for the artificial neuron.

The Artificial Neuron: A Mathematical Model

An artificial neuron, sometimes called a perceptron in its earliest form, is a mathematical function conceived as a simplified model of a biological neuron. It takes multiple input signals, processes them, and produces a single output signal. Here are its main components:

1. Inputs ($x_1, x_2, ..., x_n$): These are the numerical values fed into the neuron. They could be pixel values from an image, features from a dataset (like age or income), or outputs from neurons in a previous layer.

2. Weights ($w_1, w_2, ..., w_n$): Each input $x_i$ is associated with a weight $w_i$. The weight represents the strength or importance of that specific input connection. A larger absolute weight means that the corresponding input has a greater influence on the neuron's output. These weights are the primary parameters that the network learns during training.

3. Bias ($b$): This is an additional, learnable parameter associated with the neuron itself, not connected to any specific input. The bias acts like an offset, making it easier or harder for the neuron to activate (produce a non-zero output). Think of it like adjusting the activation threshold. Without a bias, the neuron's weighted sum must pass through the origin, limiting its flexibility.

4. Summation Function: The neuron calculates a weighted sum of its inputs, adding the bias. This is often represented as $z$:

   $$z = (w_1 x_1 + w_2 x_2 + \dots + w_n x_n) + b = \left( \sum_{i=1}^{n} w_i x_i \right) + b$$

   This calculation, $z$, represents the linear combination of the inputs.

5. Activation Function ($f$): The result of the summation, $z$, is then passed through an activation function, $f$. This function introduces non-linearity into the model, allowing neural networks to learn complex relationships in data that simple linear models cannot. The final output of the neuron, often denoted as $a$, is:

   $$a = f(z) = f\left( \left( \sum_{i=1}^{n} w_i x_i \right) + b \right)$$

   We will explore specific activation functions like Sigmoid, Tanh, and ReLU in detail later in this chapter. For now, understand its role is to transform the linear sum $z$ into the neuron's final output $a$.

Structure of a single artificial neuron. Inputs ($x_i$) are multiplied by weights ($w_i$), summed together with a bias ($b$) to produce $z$, which is then passed through an activation function $f$ to generate the output $a$.

While heavily simplified compared to its biological counterpart, this artificial neuron model captures the essential idea of integrating weighted inputs and producing an output based on the result. It's a fundamental computational unit. By connecting many such units together in layers, we can build powerful neural networks capable of learning complex patterns, which is the subject of the rest of this course.