Activation Functions: Introducing Non-Linearity

An artificial neuron combines its inputs using weights and adds a bias to produce a value, often denoted as $z$:

$$z = (\sum_{i} w_i x_i) + b$$

This calculation, $z = \mathbf{w} \cdot \mathbf{x} + b$ in vector notation, is a linear transformation. If we simply stacked layers of neurons that only perform this linear calculation, the entire network would still only be capable of learning linear relationships. Why? Because a sequence of linear transformations can always be mathematically reduced to a single, equivalent linear transformation. If our network function is $f(x) = W_2 (W_1 x + b_1) + b_2$, this simplifies to $f(x) = (W_2 W_1) x + (W_2 b_1 + b_2)$, which is just another linear function $f(x) = W' x + b'$.

To learn complex, non-linear patterns in data (like recognizing images, understanding language, or predicting intricate trends), neural networks need a way to introduce non-linearity. This is precisely the role of activation functions.

The Role of Activation Functions

An activation function, denoted typically as $f(z)$ or $\sigma(z)$ or $g(z)$, is a function applied to the output $z$ of the linear transformation within a neuron. The result, $a = f(z)$, is called the neuron's activation or output.

$$a = f( (\sum_{i} w_i x_i) + b )$$

This function takes the summed, weighted input plus bias ($z$) and transforms it, typically in a non-linear way. By applying this non-linear function after each layer's linear calculations, the network gains the ability to model much more complex relationships between inputs and outputs. It's this introduction of non-linearity that allows deep networks to approximate highly complex functions.

Common Activation Functions

Several activation functions have been developed, each with its own characteristics, advantages, and disadvantages. Let's look at some of the most common ones used in hidden layers and output layers.

Sigmoid

The Sigmoid function, also known as the logistic function, was historically very popular. It squashes its input value into a range between 0 and 1.

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Characteristics:

• Output Range: (0, 1). This makes it suitable for the output layer of a binary classification problem, where the output can be interpreted as a probability.
• Shape: Smooth, 'S'-shaped curve.
• Non-linear: Clearly introduces non-linearity.

Drawbacks:

• Vanishing Gradients: For very high or very low values of $z$, the slope (gradient) of the Sigmoid function becomes extremely close to zero. During backpropagation (which we'll cover later), these small gradients can make it very difficult for the network to update weights effectively, especially in deep networks. This is known as the vanishing gradient problem.
• Not Zero-Centered: The output is always positive (between 0 and 1). This can sometimes slow down learning compared to zero-centered activation functions.

Tanh (Hyperbolic Tangent)

The Hyperbolic Tangent function, or Tanh, is mathematically related to Sigmoid but squashes the input to a range between -1 and 1.

$$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2 \sigma(2z) - 1$$

Characteristics:

• Output Range: (-1, 1).
• Shape: Also an 'S'-shaped curve, similar to Sigmoid.
• Zero-Centered: Its output is centered around zero. This property often helps speed up convergence during training compared to Sigmoid, making Tanh generally preferred over Sigmoid for hidden layers.

Drawbacks:

• Vanishing Gradients: Like Sigmoid, Tanh also suffers from the vanishing gradient problem when the input $z$ becomes very large (positive or negative).

ReLU (Rectified Linear Unit)

ReLU is currently one of the most widely used activation functions in hidden layers, particularly in deep learning. It's computationally simple and effective.

$$ReLU(z) = \max(0, z)$$

Characteristics:

• Output Range: [0, $\infty$).
• Shape: It's zero for all negative inputs and increases linearly for positive inputs.
• Computational Efficiency: Very fast to compute (just a simple threshold check).
• Mitigates Vanishing Gradients: For positive inputs ($z > 0$), the gradient is constant (1), which helps gradients flow better during backpropagation compared to Sigmoid or Tanh.

Drawbacks:

• Dying ReLU Problem: If a neuron's input $z$ is consistently negative during training, it will always output 0. Consequently, the gradient flowing through that neuron will also be zero, and its weights will never be updated. The neuron essentially "dies" and stops contributing to the network's learning.
• Not Zero-Centered: Similar to Sigmoid, the outputs are not zero-centered.

Visual Comparison

The following chart shows the shapes and output ranges of Sigmoid, Tanh, and ReLU functions.

Comparison of Sigmoid (blue, 0 to 1), Tanh (purple, -1 to 1), and ReLU (orange, 0 to $\infty$) activation functions. Notice the saturation points for Sigmoid and Tanh where the gradient approaches zero, versus the constant gradient of ReLU for positive inputs.

Other Variants

Researchers have proposed variants to address the drawbacks of these main functions. For example, Leaky ReLU introduces a small, non-zero slope for negative inputs ($\max(0.01z, z)$) to combat the dying ReLU problem. Other popular choices include ELU (Exponential Linear Unit) and Swish. However, ReLU often remains a strong default choice for hidden layers due to its simplicity and effectiveness in many scenarios.

How Activation Functions Are Used

In a typical neural network layer, the activation function is applied element-wise to the vector or matrix resulting from the linear transformation ($z = Wx + b$). If a layer has 10 neurons, the linear transformation produces 10 values ($z_1, z_2, ..., z_{10}$). The activation function is then applied individually to each of these values to produce the layer's final output activations ($a_1 = f(z_1), a_2 = f(z_2), ..., a_{10} = f(z_{10})$). These activations then serve as the inputs to the next layer.

Choosing an Activation Function

The choice of activation function can significantly impact network performance. Here are some general guidelines, though experimentation is often necessary:

• Hidden Layers: ReLU is a common and often effective starting point. If you encounter issues with dying neurons, try Leaky ReLU or other variants like ELU. Tanh is sometimes used but is less popular than ReLU currently. Sigmoid is generally avoided in hidden layers due to its drawbacks.
• Output Layer: The choice depends heavily on the task:
  • Binary Classification: Sigmoid is typically used to output a probability between 0 and 1.
  • Multi-class Classification: Softmax (a generalization of Sigmoid for multiple classes) is the standard choice. It outputs a probability distribution across the classes.
  • Regression: Often, no activation function (or a linear activation function, $f(z)=z$) is used in the output layer, allowing the network to output values across any range.

In summary, activation functions are a fundamental component of neural networks. By introducing non-linearity after the linear calculations in each neuron, they equip the network with the ability to learn complex patterns and functions from data, moving far past the capabilities of simple linear models.