The encoder is the heart of an autoencoder's ability to learn useful features. Its job is to take the input data and transform it into a compact, lower-dimensional representation, often called the latent space or bottleneck. The design of this network. how many layers it has, how many neurons are in each layer, and what activation functions are used. directly influences the quality and nature of the features your autoencoder will learn. Getting the encoder architecture right is a significant step towards effective feature extraction.
Let's examine the primary design choices you'll make when constructing an encoder network.
The depth of your encoder, meaning the number of layers it contains, determines its capacity to learn complex patterns and hierarchies in the data.
Shallow Encoders (Few Layers): An encoder with just one or two hidden layers is simpler and trains faster. For relatively simple datasets where the underlying structure isn't excessively complex, a shallow encoder might be sufficient. It can learn a good compressed representation without the risk of overfitting that comes with more complex models.
Deep Encoders (Multiple Layers): As you add more layers, the encoder can learn more abstract representations. Each layer can build upon the features learned by the previous one, creating a hierarchy of features. For instance, in image data, initial layers might learn edges, subsequent layers might combine edges to form simple shapes, and deeper layers could represent more complex object parts. However, deeper networks are more computationally expensive, require more data to train effectively, and can be more prone to overfitting if not regularized. They also present challenges like vanishing gradients, although modern activation functions and initialization techniques help mitigate these.
General Guideline: It's often a good practice to start with a simpler, shallower encoder (e.g., 1-3 hidden layers) and gradually increase the depth if the model isn't capturing the data's complexity adequately. Monitor your reconstruction loss. if it remains high, and you suspect the model lacks capacity, adding layers might help.
For many tabular datasets, an encoder with 2 to 4 layers (including the input-to-first-hidden and last-hidden-to-bottleneck transformations) is a common starting point. For high-dimensional data like images, deeper encoders, especially those using convolutional layers (which we'll discuss in Chapter 5), are standard.
The number of neurons (or units) in each layer of the encoder dictates the "width" of that layer and how much information can pass through it. For an autoencoder designed for dimensionality reduction, the encoder layers typically have a progressively decreasing number of neurons, forming a funnel shape that narrows towards the bottleneck layer.
A diagram showing the typical tapering structure of an encoder network, where the number of neurons decreases with each subsequent layer leading to the bottleneck.
Rate of Reduction: How quickly you reduce the number of neurons is an important design choice.
Information Flow: The width of each layer acts as a constraint on the amount of information that can flow through the network. The encoder learns to preserve the most salient information that is necessary for the decoder to reconstruct the input.
Starting Point: A common approach is to make the first hidden layer smaller than the input dimension but large enough to capture initial patterns (e.g., half or a quarter of the input dimension, but this is highly heuristic). Subsequent layers then continue to reduce this dimensionality. For example, if your input has 784 features, your encoder might look like:
The exact numbers are problem-dependent and often found through experimentation. The goal is to compress the data meaningfully without losing too much information essential for reconstruction and, more importantly, for the features to be useful downstream.
Activation functions determine the output of a neuron given an input or set of inputs. They introduce non-linearities into the network, allowing autoencoders to learn more complex mappings than simple linear transformations (like PCA).
For the hidden layers within the encoder, common choices include:
ReLU (Rectified Linear Unit):
ReLU is currently one of the most popular activation functions. It's computationally efficient and helps mitigate the vanishing gradient problem, which can slow down training in deeper networks. Its non-saturating nature (for positive inputs) means it learns faster. A potential issue is the "dying ReLU" problem, where neurons can become inactive if they consistently get negative input, but variants like Leaky ReLU or Parametric ReLU (PReLU) address this. For most encoder hidden layers, ReLU is a solid default choice.
Leaky ReLU:
where is a small constant (e.g., 0.01). This allows a small, non-zero gradient when the unit is not active, preventing neurons from dying.
Sigmoid:
The sigmoid function squashes its input into the range (0, 1). It was historically popular but is less favored for hidden layers in deep networks today due to issues with vanishing gradients, especially when inputs are very large or very small. If your input data is normalized to be between 0 and 1, and you want the latent features to also reflect probabilities or be in a similar range, sigmoid could be considered for the final encoder layer before the bottleneck, but ReLU is generally preferred for intermediate layers.
Tanh (Hyperbolic Tangent):
Tanh squashes its input into the range (-1, 1). It's zero-centered, which can be an advantage over sigmoid. However, like sigmoid, it can suffer from vanishing gradients. If your input data is normalized to be between -1 and 1, tanh might be a suitable choice.
Activation for the Bottleneck Layer Output? The "bottleneck layer" itself is the representation. The final transformation of the encoder results in this bottleneck vector. Typically, if you think of the bottleneck as the direct output of the last dense layer in the encoder, this layer might have a linear activation (i.e., no non-linear activation, ) or a specific activation depending on the desired properties of the latent space. For a standard autoencoder aiming to learn a continuous, compressed representation, a linear activation for the bottleneck is common, allowing the latent features to take on any real values. If you need the latent features to be bounded (e.g., between 0 and 1), then a sigmoid activation might be applied to the output of the encoder's final layer. However, more often, non-linearities are applied in all hidden layers leading up to the bottleneck.
General Recommendation: For most hidden layers in your encoder, start with ReLU or Leaky ReLU. They generally provide good performance and training stability.
It's important to understand that these design choices. number of layers, neurons per layer, and activation functions. are not independent.
For instance, if you're working with highly structured image data, you'll eventually move to Convolutional Autoencoders (Chapter 5), which have specialized layers (convolutional, pooling) that are very effective at capturing spatial hierarchies. However, the principles of depth, width reduction, and non-linear activations still apply.
While this section focuses on the encoder, its design often informs the decoder's architecture. A common practice, especially for simpler autoencoders, is to make the decoder roughly symmetrical to the encoder. If your encoder has layers with neuron counts [Input_Dim, 128, 64, Latent_Dim], a symmetrical decoder might have layers [Latent_Dim, 64, 128, Input_Dim]. This symmetry is a heuristic that often works well, but it's not a strict requirement. The decoder's task is to reconstruct the original data from the latent representation, so its design should facilitate this up-sampling or de-compression process.
The guidelines provided here offer a starting point. Building effective autoencoders, much like other neural networks, involves an iterative process. You'll propose an initial architecture, train the model, evaluate its performance (both reconstruction quality and the utility of extracted features), and then refine the design.
Monitoring training, visualizing reconstructions, and evaluating extracted features (as discussed later in this chapter and in Chapter 7) will provide valuable feedback for tuning your encoder network design. As you gain experience, you'll develop a better intuition for how these choices affect the autoencoder's behavior and its ability to learn powerful features.
Was this section helpful?
© 2026 ApX Machine LearningEngineered with