Common Activation Functions in Decoders

The decoder's role in an autoencoder is to reconstruct data. It receives a compressed representation, typically denoted as $z$ and generated by an encoder's bottleneck layer, and transforms it back into a form resembling the original input $X$ . The final layer of the decoder, known as the output layer, plays a crucial part in this reconstruction. The activation function used in this output layer directly determines the nature and range of the reconstructed values, $X'$ . Its choice is therefore guided by the specific characteristics of the data intended for reconstruction.

Activation Functions for the Output Layer

The output layer's activation function needs to ensure that the reconstructed data $X'$ is in the same format and range as the original input data $X$ . If your input data consists of pixel values normalized between 0 and 1, your decoder's output should also fall within this range.

Sigmoid Function

One of the most common activation functions for the output layer of an autoencoder, especially when dealing with input data normalized to a range of [0, 1] (like grayscale image pixel intensities), is the Sigmoid function.

The Sigmoid function is defined as: $\sigma(x) = \frac{1}{1 + e^{-x}}$ It squashes any real-valued input $x$ into an output between 0 and 1. This "S" shape is very useful because it naturally aligns with data that is bounded within this range. For instance, if an input pixel was 0 (black) or 1 (white), or somewhere in between, the Sigmoid function ensures the reconstructed pixel value respects these bounds.

Tanh (Hyperbolic Tangent) Function

Another option for the output layer is the Hyperbolic Tangent function, often abbreviated as tanh. It's similar to Sigmoid but squashes values to a range of [-1, 1].

The tanh function is defined as: $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$ You would typically use tanh if your input data $X$ has been normalized to be between -1 and 1.

The following chart visualizes both the Sigmoid and tanh functions, showing how they map input values to their respective output ranges:

Sigmoid outputs values between 0 and 1, suitable for data normalized to this range. Tanh outputs values between -1 and 1, used when data is normalized accordingly.

Linear Function

What if your input data isn't conveniently bounded between [0, 1] or [-1, 1]? For example, you might be working with raw sensor readings that can take any real value. In such cases, using a Linear activation function (or, equivalently, no activation function) for the output layer is appropriate.

A linear activation function is simply: $f(x) = x$ This means the output of the neuron is just its weighted sum of inputs, without any "squashing." This allows the reconstructed values $X'$ to take on any real number, matching the potential range of the original data $X$ .

Activation Functions for Hidden Layers in the Decoder

While the output layer of the decoder has specific requirements tied to the input data's range, the hidden layers within the decoder have a different role. These layers work to gradually upsample and transform the compressed representation $z$ back towards the original data's structure.

For these intermediate (hidden) layers in the decoder, it's common to use the same types of activation functions you might find in the encoder's hidden layers. The Rectified Linear Unit (ReLU) is a very popular choice.

Recall that ReLU is defined as: $\text{ReLU}(x) = \max(0, x)$ ReLU is favored in hidden layers (both encoder and decoder) because it helps with training deeper networks more effectively (by mitigating issues like vanishing gradients) and is computationally efficient. In the decoder, ReLU allows the network to learn complex, non-linear transformations needed to reconstruct the data from its compressed form. While Sigmoid or tanh might also be used in hidden layers, ReLU is often a strong default choice.

In summary, when designing the decoder:

The output layer's activation function is chosen based on the range of your original data $X$ (e.g., Sigmoid for [0,1], tanh for [-1,1], Linear for unbounded).
The hidden layers' activation functions often use ReLU to help learn the intricate mapping from the bottleneck $z$ back to $X'$ .

Understanding these activation functions and their placement is another step towards grasping how autoencoders effectively learn to reconstruct and represent data.

Was this section helpful?

References

Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016 (MIT Press) - Covers theoretical foundations of deep learning, including detailed explanations of activation functions and autoencoder architectures.
A Comprehensive Survey on Activation Functions in Deep Learning, J. D. Ramachandran, K. Chelliah, D. B. Vijendran, P. H. Reddy, B. Bharath, 2020 Applied Sciences, Vol. 10 (MDPI) DOI: 10.3390/app10217483 - Provides an extensive overview of various activation functions used in deep learning, discussing their characteristics and applications.
Extracting and Composing Robust Features with Denoising Autoencoders, Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol, 2008 Proceedings of the 25th International Conference on Machine Learning (ACM) DOI: 10.1145/1390156.1390294 - A seminal paper introducing denoising autoencoders, demonstrating the decoder's role in reconstructing data and influencing output layer choices.
Neural Networks Part 1: Setting up the Architecture, Justin Johnson, Andrej Karpathy, Fei-Fei Li, and course staff, 2023 (Stanford University) - Provides an accessible overview of neural network architectures and the properties of common activation functions (Sigmoid, Tanh, ReLU) from a leading university course.