Latent Variable Models: Theory and Formulation

Latent variable models (LVMs) are a prominent approach within probabilistic generative modeling, distinguished by their ability to capture rich, underlying structure in complex datasets. The core idea is elegant: observed data, often denoted as $x$, is assumed to be generated by hidden, or latent, variables, denoted by $z$. These latent variables are not directly observed, but they govern the patterns and variations seen in $x$. For instance, if observed data $x$ consists of images of handwritten digits, the latent variables $z$ might represent underlying factors such as writing style, digit identity, or stroke thickness. By modeling these unobserved factors, LVMs aim to understand data at a more fundamental level.

This approach connects directly to the goals of representation learning. The latent space, $\mathcal{Z}$, where $z$ resides, often serves as a compressed, more meaningful representation of the original data space $\mathcal{X}$. If designed well, this latent representation can disentangle the factors of variation in the data, making it useful for various downstream tasks.

Why Use Latent Variable Models?

You might wonder why we introduce this extra layer of unobserved variables. There are several compelling reasons:

1. Modeling Complex Data Distributions: Directly modeling the probability distribution $p(x)$ for high-dimensional data like images or text is exceedingly difficult. LVMs offer a more manageable approach by decomposing the problem. Instead of modeling $p(x)$ directly, we model a simpler prior distribution $p(z)$ over the latent variables and a conditional distribution $p(x|z)$ that describes how to generate $x$ given $z$.
2. Data Generation: Once an LVM is trained, we can generate new, synthetic data samples that resemble the training data. This is done by first drawing a sample $z_{new}$ from the prior $p(z)$ and then drawing a sample $x_{new}$ from the conditional distribution $p(x|z_{new})$. This generative capability is a hallmark of models like VAEs.
3. Learning Meaningful Representations: The latent variables $z$ can provide a compressed and often interpretable representation of $x$. For instance, if $z$ is lower-dimensional than $x$, the LVM performs a form of non-linear dimensionality reduction. The structure of these representations is a central theme in this course.
4. Handling Missing Data and Uncertainty: LVMs can naturally handle missing data by marginalizing over the unobserved parts of $x$. They also provide a framework for quantifying uncertainty in predictions and representations.

The Mathematical Formulation of LVMs

Let's formalize these ideas. We have:

• Observed variables $x \in \mathcal{X}$: These are the data points we have access to (e.g., images, sentences).
• Latent variables $z \in \mathcal{Z}$: These are the unobserved variables that we assume generate $x$.

The joint probability distribution over observed and latent variables is the foundation of an LVM and is typically defined as:

$$p(x, z) = p(x|z) p(z)$$

Let's break down the components:

• Prior Distribution $p(z)$: This distribution defines our initial beliefs about the latent variables before observing any data. In practice, $p(z)$ is often chosen to be a simple, tractable distribution, such as a standard multivariate Gaussian distribution ($\mathcal{N}(0, I)$). This simplicity aids in sampling and can act as a regularizer.
• Likelihood (or Generative Distribution) $p(x|z)$: This conditional distribution specifies how the observed data $x$ is generated from the latent variables $z$. In modern deep learning, $p(x|z)$ is commonly parameterized by a neural network, often called the decoder or generator network. For example, if $x$ is an image, $p(x|z)$ could be a Gaussian distribution whose mean is the output of a deconvolutional neural network taking $z$ as input, and whose covariance might be fixed (e.g., $\sigma^2 I$). If $x$ is binary, $p(x|z)$ might be a Bernoulli distribution whose parameters are outputs of a neural network.

The primary goal is often to model the marginal distribution of the observed data, $p(x)$, also known as the model evidence:

$$p(x) = \int p(x, z) dz = \int p(x|z)p(z) dz$$

This integral sums (or integrates, for continuous $z$) over all possible configurations of the latent variables. Computing this integral is one of the main challenges in LVMs, as it's often intractable, especially when $z$ is high-dimensional and $p(x|z)$ is complex (like a deep neural network).

A Graphical Viewpoint

We can visualize the generative process of a simple LVM using a probabilistic graphical model (PGM).

A simple directed graphical model representing the generative process in an LVM. Circles denote random variables. The latent variable $z$ (drawn from $p(z)$) generates the observed variable $x$ (via $p(x|z)$).

This graph illustrates that $z$ is sampled first, and then $x$ is sampled conditioned on $z$. The parameters of these distributions (e.g., the weights of the neural networks parameterizing $p(x|z)$) are learned from data.

Inference: Understanding the Latent Space

Beyond generation, we want to perform inference, which in this context typically means computing the posterior distribution over the latent variables given an observed data point $x$:

$$p(z|x) = \frac{p(x|z)p(z)}{p(x)} = \frac{p(x|z)p(z)}{\int p(x|z')p(z') dz'}$$

This posterior distribution $p(z|x)$ tells us what values of $z$ are likely to have generated a specific $x$. In VAEs, a neural network called the encoder (or inference network) is trained to approximate this posterior distribution. Knowing $p(z|x)$ is essential for learning meaningful representations, as it allows us to map observed data $x$ into the latent space $\mathcal{Z}$.

However, the intractability of the marginal likelihood $p(x)$ (the denominator) makes direct computation of $p(z|x)$ just as challenging. This is where approximate inference methods, such as variational inference (the "V" in VAE), become indispensable.

Core Objectives with LVMs

Training and using LVMs generally revolve around a few main objectives:

• Density Estimation: The model should assign high probability $p(x)$ to data points similar to those in the training set and low probability to dissimilar points.
• Data Generation: As discussed, sampling new data points $x_{new}$ that are characteristic of the training distribution.
• Representation Learning: Learning an informative latent space $\mathcal{Z}$ where each $z$ captures significant variations in the data. This involves learning the mapping from $x$ to $z$ (inference) and from $z$ to $x$ (generation).
• Downstream Tasks: Using the learned representations $z$ for tasks like classification, clustering, or regression.

The Computational Hurdles

The power and flexibility of LVMs come with significant computational challenges:

1. Intractable Marginal Likelihood: As we've seen, $p(x) = \int p(x|z)p(z) dz$ is usually intractable. This makes it hard to directly maximize the likelihood of the data during training and difficult to evaluate model performance by comparing $p(x)$ values.
2. Intractable Posterior: The posterior $p(z|x)$ is also typically intractable because its denominator is $p(x)$. This hinders our ability to infer the latent representation for a given data point.

These intractabilities are not just minor inconveniences; they are fundamental obstacles that have driven much of the research in generative models. Variational Autoencoders, which are the focus of this course, provide a clever and effective framework using variational inference and neural networks to navigate these challenges. By approximating the true posterior $p(z|x)$ with a simpler, tractable distribution $q(z|x)$, VAEs manage to learn both the generative model $p(x|z)$ and the inference model $q(z|x)$ simultaneously.

Understanding this foundational theory of LVMs, their promise, and their inherent difficulties is essential as we move towards the specifics of VAEs. In the following sections and chapters, you'll see how VAEs build upon these principles to create powerful generative models capable of learning rich representations from complex data.