Activation Functions

When constructing neural networks that use layers like Dense layers, an important consideration is how information flows and transforms between these layers. If these Dense layers were simply stacked without any modification to their outputs, the entire network, regardless of its depth, would behave as a single, large linear transformation. This happens because composing linear functions results in another linear function. To model complex, non-linear relationships present in most data (like images, text, or structured datasets), introducing non-linearity into the network is necessary. This is precisely the role of activation functions.

An activation function takes the output signal from the preceding layer (often a weighted sum of inputs plus a bias) and applies a fixed mathematical operation to it, element-wise for most common activations. This transformed output then becomes the input for the next layer.

Let's look at some of the most frequently used activation functions and how to implement them in Keras.

Why Non-Linearity?

Imagine trying to separate two classes of data points that are not linearly separable (e.g., points forming concentric circles). A simple linear model ($y = wx + b$) can only draw straight lines (or hyperplanes in higher dimensions). No matter how many linear layers you stack, you'll still only be able to produce a linear decision boundary. Activation functions break this linearity, allowing networks to learn complex curves and shapes, effectively warping the data representation space at each layer to make patterns more separable.

Common Activation Functions

Sigmoid

The sigmoid function squashes its input into a range between 0 and 1. Its mathematical form is:

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Historically popular, its output range (0 to 1) makes it suitable for the output layer of a binary classification model, where the output can be interpreted as a probability. However, it has fallen out of favor for hidden layers due to a couple of drawbacks:

1. Vanishing Gradients: For very large positive or negative inputs, the sigmoid function's gradient becomes extremely close to zero. During backpropagation, these small gradients can get multiplied down through layers, effectively stopping the weights in earlier layers from being updated properly.
2. Not Zero-Centered: Its outputs are always positive. This can sometimes slow down the convergence of gradient descent.

In Keras, you can use it directly within a layer:

import keras
from keras import layers

# Example usage in a Dense layer
output_layer = layers.Dense(1, activation='sigmoid')

Tanh (Hyperbolic Tangent)

The hyperbolic tangent function, or tanh, is closely related to the sigmoid but squashes its input into the range between -1 and 1.

$$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = 2 \sigma(2x) - 1$$

Being zero-centered (its outputs range from -1 to 1) often helps with model convergence compared to sigmoid, making it a preferred choice over sigmoid for hidden layers in the past. However, it still suffers from the vanishing gradient problem for large positive or negative inputs.

Usage in Keras:

# Example usage in a Dense layer
hidden_layer = layers.Dense(64, activation='tanh')

ReLU (Rectified Linear Unit)

The Rectified Linear Unit, or ReLU, has become the default activation function for hidden layers in most deep learning applications. It's simple yet remarkably effective.

$$ReLU(x) = \max(0, x)$$

It outputs the input directly if it's positive, and zero otherwise.

Advantages:

1. Computational Efficiency: Very cheap to compute (just a comparison).
2. Mitigates Vanishing Gradients: For positive inputs, the gradient is constant (1), allowing gradients to flow better during backpropagation compared to sigmoid or tanh.

Disadvantage:

1. Dying ReLU Problem: Neurons can sometimes become "stuck" during training if their input consistently falls below zero. In this state, the gradient is zero, and the neuron's weights won't update. Variants like Leaky ReLU or ELU (Exponential Linear Unit) attempt to address this by allowing a small, non-zero gradient for negative inputs.

Usage in Keras:

# Example usage in a Dense layer (most common for hidden layers)
hidden_layer = layers.Dense(128, activation='relu')

Softmax

The Softmax function is typically used in the output layer of a multi-class classification network. Unlike the previous functions that operate element-wise independently, Softmax operates on the entire vector of outputs from the final layer. It converts a vector of raw scores (logits) into a probability distribution, where each element is between 0 and 1, and all elements sum up to 1.

For a vector of inputs $x$, the Softmax output for the $i$-th element is:

$$Softmax(x)_i = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$

Where the sum in the denominator is over all elements in the input vector $x$. This ensures the outputs represent probabilities for each class.

Usage in Keras (typically on the final layer for multi-class classification):

# Example for a 10-class classification problem
output_layer = layers.Dense(10, activation='softmax')

Visualization of Activation Functions

The following chart illustrates the shapes of the Sigmoid, Tanh, and ReLU functions:

Comparison of Sigmoid (blue), Tanh (green), and ReLU (red) activation functions across a range of input values. Note the different output ranges and shapes. Softmax is not shown as it operates on a vector rather than a single scalar input.

Choosing Activation Functions

Here are some general guidelines:

• Hidden Layers: Start with ReLU. It's computationally efficient and generally works well. If you encounter issues with dying neurons, consider Leaky ReLU or other variants. Tanh is sometimes used, especially in recurrent networks, but ReLU is more common in feedforward networks and CNNs.
• Output Layer (Binary Classification): Use Sigmoid (with 1 output unit).
• Output Layer (Multi-Class Classification): Use Softmax (with N output units, where N is the number of classes).
• Output Layer (Regression): Use no activation function (which is equivalent to a linear activation: $f(x)=x$). The output can then take any real value.

Implementing Activation Functions in Keras

As seen in the examples, the most common way to add an activation function is via the activation argument within a layer definition:

model = keras.Sequential([
    layers.Dense(128, activation='relu', input_shape=(784,)), # Input layer with ReLU
    layers.Dense(64, activation='relu'),                      # Hidden layer with ReLU
    layers.Dense(10, activation='softmax')                     # Output layer for 10 classes
])

Alternatively, especially when using the Functional API or if you want to apply an activation after a layer that doesn't have the activation argument (like batch normalization sometimes), you can use the Activation layer:

from keras.layers import Activation

# Functional API example snippet
inputs = keras.Input(shape=(784,))
x = layers.Dense(128)(inputs)
x = Activation('relu')(x) # Apply ReLU using the Activation layer
x = layers.Dense(64)(x)
x = Activation('relu')(x)
outputs = layers.Dense(10, activation='softmax')(x) # Use argument here

model = keras.Model(inputs=inputs, outputs=outputs)

Understanding activation functions is fundamental to building effective neural networks. They are the source of non-linearity that enables networks to learn complex mappings from inputs to outputs. Choosing the right activation function, especially for the output layer, depends heavily on the specific task (classification, regression) you are addressing. In the next sections, we will continue assembling these building blocks to construct and examine complete Keras models.