Introduction to Generative Modeling with Autoencoders

Autoencoders are powerful tools for dimensionality reduction and feature learning. They excel at learning a compressed representation of input data and then reconstructing that data from this representation. Variational Autoencoders (VAEs), however, steer the autoencoder framework in a new direction: generative modeling.

What is Generative Modeling?

Essentially, generative modeling is about teaching a machine to create new things. Instead of learning to predict a label from input features (like in classification, a discriminative task), a generative model aims to learn the underlying probability distribution of the data itself. If a model truly understands how your data is distributed, it can then be used to generate new, synthetic data samples that look like they could have come from the original dataset.

Imagine you have a dataset of handwritten digits.

• A discriminative model might learn to tell you if an image is a "7" or not. It learns $P(\text{label} | \text{image})$.
• A generative model would learn the characteristics of what makes an image look like a "7", or a "3", or any digit. It tries to learn $P(\text{image})$. Once trained, you could ask it to "dream up" a new image of a digit it has never seen before.

The applications are wide-ranging: creating realistic images, generating new musical pieces, synthesizing human-like text, or even augmenting existing datasets to improve the performance of other machine learning models.

Autoencoders and the Spark of Generation

You might be wondering how our familiar autoencoders fit into this picture. Recall the basic autoencoder architecture: an encoder maps input data $X$ to a lower-dimensional latent representation $z$, and a decoder attempts to reconstruct the original data $\hat{X}$ from $z$.

Input (X) --> Encoder --> Latent Space (z) --> Decoder --> Reconstructed Output (X_hat)

The decoder part is particularly interesting. If we could somehow feed it meaningful vectors from the latent space $z$, it could, in principle, generate new data. The decoder has learned to map points in the latent space back to the original data space. So, isn't an autoencoder already a generative model?

Not quite, or at least, not a very good one by default. Standard autoencoders are trained to be excellent at reconstruction. Their latent space $z$ learns to capture the necessary information to achieve this goal. However, this latent space isn't necessarily organized in a way that's conducive to generating new, varied, and realistic samples.

If you were to pick a random point in the latent space of a standard autoencoder and pass it through the decoder, the output might be noisy, nonsensical, or not resemble any valid data from your original dataset. The autoencoder has learned to map specific input data to specific regions in the latent space, but the "in-between" areas or regions far from these mappings might be undefined or lead to poor reconstructions. The space might be "clumpy" or have "holes."

A simplified view comparing how input data might be represented in the latent space of a standard autoencoder versus a Variational Autoencoder, and the implications for generating new data using the decoder.

The Need for a Structured Latent Space

To effectively use an autoencoder-like structure for generation, we need the latent space to be more than just a compression chamber. We need it to be structured and continuous. This means:

1. Continuity: Small changes to a vector $z$ in the latent space should result in small, semantically meaningful changes in the generated output $\hat{X}$. If moving a tiny bit in latent space drastically changes the output from a "cat" to a "car," the space isn't very useful for smooth generation or interpolation.
2. Completeness (or density): The regions of the latent space that can be decoded into realistic samples should be reasonably dense. We want to be able to sample points from a chosen distribution (e.g., a Gaussian) in the latent space and have a high probability of generating something sensible.

Standard autoencoders don't explicitly enforce these properties. Their primary objective is minimizing reconstruction error. This is where Variational Autoencoders (VAEs) come in. VAEs are a type of autoencoder specifically designed with generative modeling in mind. They introduce a probabilistic spin to the encoder and the latent space, along with a modified loss function that encourages the latent space to have these desirable properties.

While they still use the encoder-decoder architecture, VAEs don't map an input to a single point in the latent space. Instead, they map it to the parameters of a probability distribution (like the mean and variance of a Gaussian). This probabilistic approach is fundamental to their ability to generate diverse and coherent new samples and is what we will explore in detail throughout this chapter. By learning a smooth and structured latent space, VAEs pave the way for generating novel data by simply sampling from this learned space and passing it through the decoder.