Activation Functions

Activation functions are crucial components in neural networks, governing the output of each neuron and enabling the network to learn and comprehend intricate data patterns. They introduce non-linearity, a vital characteristic that empowers neural networks to model complex relationships beyond the capabilities of linear models.

An activation function is a mathematical operation applied to the input of a neuron, typically the weighted sum of inputs plus a bias term, often denoted as $z = wx + b$. The purpose of applying an activation function is to introduce non-linearity into the system. Without this non-linearity, a neural network, regardless of its depth, would behave like a simple linear model, severely limiting its capability to model complex data.

There are several types of activation functions commonly used in neural networks, each with its own characteristics and use cases. Let's explore some of the most popular ones:

1. Sigmoid Activation Function: The sigmoid function is among the earliest and simplest activation functions used in neural networks. Defined as $\sigma(z) = \frac{1}{1 + e^{-z}}$, it outputs a value between 0 and 1. This makes it particularly useful for binary classification tasks, where the output is often interpreted as a probability. However, sigmoid functions can suffer from vanishing gradient problems, where gradients become too small for effective learning, especially in deep networks.

Sigmoid activation function curve

2. Hyperbolic Tangent (Tanh) Function: The tanh function is a scaled version of the sigmoid function, mapping inputs to a range between -1 and 1. This zero-centered output often results in faster convergence compared to the sigmoid function, as it provides a stronger gradient. The formula for tanh is $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$. Despite its advantages over the sigmoid function, tanh is still susceptible to the vanishing gradient problem.

Tanh activation function curve

3. Rectified Linear Unit (ReLU): ReLU has become the default activation function for many neural network architectures due to its simplicity and effectiveness. It is defined as $f(z) = \max(0, z)$, which means it outputs the input directly if it is positive; otherwise, it outputs zero. This non-linearity helps the network learn faster and more effectively. However, ReLU can suffer from the "dying ReLU" problem, where neurons can sometimes permanently output zero for all inputs if they fall into the negative region during training.

ReLU activation function curve

4. Leaky ReLU: To counteract the dying ReLU issue, Leaky ReLU introduces a small slope for negative inputs instead of a flat zero. This is represented as $f(z) = z$ if $z > 0$, and $f(z) = \alpha z$ if $z \leq 0$, where $\alpha$ is a small constant. This modification allows the network to retain some sensitivity to negative inputs, thus improving learning.

Leaky ReLU activation function curve with α=0.01

5. Softmax Function: While not an activation function in the traditional sense for hidden layers, the softmax function is crucial in the output layer of classification networks. It converts a vector of raw scores (logits) into probabilities by considering the relative scale of each score. The softmax function is defined as $\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$. This ensures that the output values are between 0 and 1 and sum up to 1, making them interpretable as probabilities.

Selecting the appropriate activation function depends on various factors, including the specific problem domain, the architecture of the network, and the nature of the input data. As you experiment with different models, you'll develop an intuition for choosing and tuning activation functions to optimize your network's performance. Understanding these nuances will empower you to build more robust and effective neural networks, paving the way for deeper exploration into network training and optimization techniques in subsequent chapters.