Hyperbolic Tangent (Tanh) Activation

Having explored the Sigmoid function, we now turn our attention to another popular S-shaped activation function: the Hyperbolic Tangent, commonly known as Tanh. Like Sigmoid, Tanh introduces non-linearity, allowing networks to learn complex patterns beyond simple linear relationships. However, Tanh offers a distinct advantage in certain scenarios due to its output range.

Mathematically, the Tanh function is defined as:

$$tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$

Interestingly, Tanh is closely related to the Sigmoid function. It can be expressed as a scaled and shifted version of the Sigmoid:

$$tanh(x) = 2 \cdot sigmoid(2x) - 1$$

Characteristics of Tanh

Let's examine the key properties of the Tanh function:

1. Output Range: Unlike Sigmoid, which outputs values between 0 and 1, Tanh outputs values in the range $(-1, 1)$. This means the output can be negative, zero, or positive.
2. Zero-Centered: The output of Tanh is centered around zero. This is often considered an advantage over Sigmoid. When activations are centered around zero, the gradients pushed back during backpropagation are more balanced, potentially leading to faster convergence during training because the updates to weights are less likely to consistently push in one direction.
3. Shape: Tanh also has an S-shape, similar to Sigmoid. It is steeper around $x=0$ compared to Sigmoid.
4. Non-linearity: It's a non-linear function, crucial for enabling multi-layer networks to approximate complex functions.

Here's a visualization of the Tanh function and its derivative:

The Tanh function maps inputs to the range (-1, 1) and is zero-centered. Its derivative peaks at 1 when the input is 0 and approaches 0 as the input magnitude increases.

Advantages and Disadvantages

Advantages:

• Zero-Centered Output: As mentioned, the (-1, 1) output range means the average activation value is closer to zero. This helps avoid strong biases in gradient updates during backpropagation, potentially accelerating the training process compared to Sigmoid.
• Steeper Gradient: Around zero, the gradient of Tanh is steeper ($1$) than Sigmoid ($0.25$). This can sometimes lead to stronger gradient flow initially.

Disadvantages:

• Vanishing Gradients: Similar to Sigmoid, Tanh suffers from the vanishing gradient problem. For large positive or negative input values, the function saturates (flattens out), and the derivative becomes extremely close to zero. This can stall learning in deep networks, as gradients struggle to propagate back through layers with saturated Tanh units.
• Computational Cost: While not a major factor in most modern applications, the computation involving exponentials makes Tanh slightly more computationally intensive than functions like ReLU.

Usage in Practice

Historically, Tanh was often preferred over Sigmoid for hidden layers precisely because of its zero-centered output. It was commonly used in feedforward networks and particularly in certain types of Recurrent Neural Networks (RNNs).

However, with the rise of ReLU and its variants (which we will discuss next), Tanh is used less frequently as the default choice for hidden layers in standard feedforward networks and Convolutional Neural Networks (CNNs). It still finds application in specific contexts, particularly within the gating mechanisms of LSTMs and GRUs (advanced types of RNNs), where its (-1, 1) range is beneficial.

Here's how you might use Tanh in a PyTorch layer:

import torch
import torch.nn as nn

# Example input tensor
input_tensor = torch.randn(5, 10) # Batch of 5, 10 features each

# Define a layer with Tanh activation
# Option 1: Using nn.Tanh() as a module
tanh_activation = nn.Tanh()
output_tensor = tanh_activation(input_tensor)

# Option 2: Using torch.tanh directly (functional approach)
output_tensor_functional = torch.tanh(input_tensor)

# Define a linear layer followed by Tanh
linear_layer = nn.Linear(in_features=10, out_features=20)
activated_output = torch.tanh(linear_layer(input_tensor))

print("Input shape:", input_tensor.shape)
print("Output shape (Module):", output_tensor.shape)
print("Output shape (Functional):", output_tensor_functional.shape)
print("Output shape (Linear + Tanh):", activated_output.shape)
print("\nSample Output (first element, first 5 features):\n", activated_output[0, :5])

While Tanh addresses the non-zero-centered issue of Sigmoid, it doesn't solve the vanishing gradient problem inherent in saturating functions. This limitation paved the way for the development and widespread adoption of ReLU and its variants, which we will explore in the next section.