Defining Autoencoders: The Basic Structure

Fundamentally, an autoencoder is a type of artificial neural network used for unsupervised learning. Its primary goal sounds deceptively simple: to learn how to copy its input to its output. That is, if you feed it some data $x$, it tries to generate $\hat{x}$ such that $\hat{x}$ is as close to $x$ as possible. While this might seem trivial, the utility of an autoencoder comes from the constraints placed on the network's architecture, specifically a "bottleneck" that forces it to learn a compressed representation of the input.

The basic structure of an autoencoder is typically symmetrical and consists of three main parts, which we'll explore in more detail in subsequent sections:

1. Encoder: This part of the network takes the original, high-dimensional input and transforms it into a lower-dimensional internal representation. It learns to "encode" the input data into a compressed form.
2. Bottleneck Layer (or Code/Latent Space): This is the central layer of the autoencoder and holds the compressed, encoded representation of the input. The dimensionality of this layer is a critical hyperparameter, as it defines the extent of compression. It's this compressed representation that we're often interested in for feature extraction.
3. Decoder: This part of the network takes the compressed representation from the bottleneck layer and tries to reconstruct the original input from it. It learns to "decode" the compressed form back into the original data space.

This structure forces the autoencoder to learn the most salient features of the data that are necessary for reconstruction. Imagine trying to summarize a long document into a few important sentences (the encoding process) and then asking someone else to recreate the original document based only on your summary (the decoding process). To do a good job, your summary must capture the most important information.

Below is a diagram illustrating this fundamental architecture:

An autoencoder processes input data through an encoder to a compressed bottleneck layer, then reconstructs the data using a decoder. The goal is for the reconstructed data to closely match the original input.

Let's briefly describe what happens in each part:

• Input Layer: This is where the raw data (e.g., an image flattened into a vector, a row from a tabular dataset) is fed into the network.
• Encoder Network: Composed of one or more neural network layers, the encoder's job is to map the input data $x$ to a hidden representation $z$ in the bottleneck layer. Each successive layer in the encoder typically has fewer neurons than the previous one, progressively reducing the dimensionality. So, $z = \text{encoder}(x)$.
• Bottleneck Layer (Latent Space): This layer, denoted as $z$, represents the compressed knowledge the autoencoder has learned about the input. If the bottleneck dimension $d$ is smaller than the input dimension $D$, the autoencoder is forced to learn an efficient, compressed representation. These learned representations are often useful as features for other machine learning tasks.
• Decoder Network: Symmetrically to the encoder, the decoder takes the latent representation $z$ and maps it back to a reconstruction $\hat{x}$ in the output layer. Each successive layer in the decoder typically has more neurons than the previous one, expanding the representation back towards the original input's dimensionality. So, $\hat{x} = \text{decoder}(z)$.
• Output Layer: This layer produces the reconstructed data $\hat{x}$. The number of neurons in the output layer must match the number of dimensions in the input layer. The activation function used here depends on the nature of the input data (e.g., sigmoid for pixel values between 0 and 1, or linear for continuous values).

The entire autoencoder is a feed-forward neural network, trained by minimizing a reconstruction loss function. This loss function measures the difference between the original input $x$ and the reconstructed output $\hat{x}$. For example, if your inputs are numerical vectors, the Mean Squared Error (MSE) is a common choice. If your inputs are binary (like black and white images), binary cross-entropy might be more appropriate. We'll discuss loss functions in more detail later in this chapter.

The elegance of this basic structure lies in its ability to learn useful representations without explicit labels, making it a powerful tool for unsupervised feature learning. By training the network to reproduce its input through a constrained bottleneck, we encourage it to discover underlying patterns and structures within the data. These learned patterns, captured in the bottleneck layer, can then be extracted and used as features for various downstream tasks, such as classification, clustering, or anomaly detection. This basic architecture forms the foundation upon which more advanced autoencoder variants, which we'll also cover, are built.