Probabilistic Encoders and Decoders

As highlighted in the chapter introduction, standard autoencoders, while proficient at reconstruction, struggle with generating novel data. Their deterministic nature often results in latent spaces that are not continuous or structured enough for meaningful sampling. Generating realistic samples requires moving from mapping an input $x$ to a single point $z$ in the latent space and back to a single reconstruction $x'$.

Variational Autoencoders introduce a probabilistic perspective to both the encoding and decoding processes, laying the groundwork for effective generative modeling. Instead of deterministic mappings, we work with probability distributions.

Probabilistic Encoders: Mapping Inputs to Latent Distributions

In a standard autoencoder, the encoder $f$ maps an input $x$ directly to a latent code $z = f(x)$. In contrast, the VAE encoder doesn't output a single point $z$. Instead, it outputs parameters for a probability distribution over the latent space, conditioned on the input $x$. This distribution, denoted as $q_\phi(z|x)$, represents our belief about the likely latent codes $z$ that could have generated the input $x$. The parameters of the encoder network are denoted by $\phi$.

Typically, $q_\phi(z|x)$ is chosen to be a multivariate Gaussian distribution with a diagonal covariance matrix. This simplifies calculations and works well in practice. Therefore, for a given input $x$, the encoder network (parameterized by $\phi$) outputs two vectors:

1. A vector of means, $\mu_\phi(x)$.
2. A vector of variances, $\sigma^2_\phi(x)$ (often, the network outputs the log-variances, $\log \sigma^2_\phi(x)$, for numerical stability).

The approximate posterior distribution is then:

$$q_\phi(z|x) = \mathcal{N}(z | \mu_\phi(x), \text{diag}(\sigma^2_\phi(x)))$$

Here, $\mathcal{N}(z | \mu, \Sigma)$ denotes a Gaussian distribution over $z$ with mean $\mu$ and covariance $\Sigma$. Using a diagonal covariance matrix means we assume conditional independence between the dimensions of the latent variable $z$, given the input $x$.

Why is this probabilistic encoding significant?

• Capturing Uncertainty: It acknowledges that there might be multiple plausible latent representations $z$ for a given input $x$.
• Regularization: As we will see later when discussing the VAE objective (the ELBO), this probabilistic encoding allows us to regularize the latent space by encouraging $q_\phi(z|x)$ to be close to a chosen prior distribution $p(z)$ (usually a standard Gaussian $\mathcal{N}(0, I)$). This encourages the encoder to utilize the latent space efficiently and continuously, making it suitable for sampling.
• Sampling: During training, instead of passing a fixed $z$ to the decoder, we sample $z$ from the distribution $q_\phi(z|x)$. This requires a special technique (the reparameterization trick, covered later) to allow gradient backpropagation.

Probabilistic Decoders: Generating Data from Latent Codes

Similar to the encoder, the VAE decoder also adopts a probabilistic view. A standard decoder $g$ takes a latent code $z$ and outputs a single reconstruction $x' = g(z)$. The VAE decoder, parameterized by $\theta$, takes a latent variable $z$ (sampled from $q_\phi(z|x)$ during training, or from the prior $p(z)$ for generation) and outputs the parameters of a probability distribution over the data space $x$. This distribution is denoted as $p_\theta(x|z)$. It represents the likelihood of observing data $x$ given the latent code $z$.

The choice of the distribution $p_\theta(x|z)$ depends on the nature of the data $x$:

• Continuous Data (e.g., normalized image pixels): A common choice is a multivariate Gaussian distribution. The decoder network might output the mean $\mu_\theta(z)$, and the covariance could be assumed to be fixed (e.g., $\sigma^2 I$ where $\sigma^2$ is a hyperparameter or learned) or also output by the network.

  $$p_\theta(x|z) = \mathcal{N}(x | \mu_\theta(z), \sigma^2 I)$$

  Maximizing the log-likelihood $\log p_\theta(x|z)$ under this assumption corresponds to minimizing the Mean Squared Error (MSE) between the input $x$ and the generated mean $\mu_\theta(z)$, up to a constant factor.

• Binary Data (e.g., black and white images): A common choice is a product of independent Bernoulli distributions. The decoder network outputs a vector of probabilities $p_\theta(z)$, where each element represents the probability of the corresponding dimension of $x$ being 1.

  $$p_\theta(x|z) = \prod_{i=1}^{D} p_{\theta, i}(z)^{x_i} (1 - p_{\theta, i}(z))^{1-x_i}$$

  where $D$ is the dimensionality of $x$. Maximizing the log-likelihood $\log p_\theta(x|z)$ in this case corresponds to minimizing the Binary Cross-Entropy (BCE) loss between the input $x$ and the output probabilities $p_\theta(z)$.

This probabilistic decoder provides a principled way to define the reconstruction quality. Instead of just minimizing a distance metric, we aim to maximize the likelihood of the original data $x$ under the distribution generated by the decoder, given the latent code $z$. This fits naturally into the overall probabilistic framework of VAEs and the derivation of their objective function.

Simplified flow of a Variational Autoencoder, highlighting the probabilistic nature of the encoder ($q_\phi(z|x)$) and decoder ($p_\theta(x|z)$). The encoder maps input $x$ to parameters of a latent distribution, from which $z$ is sampled. The decoder maps $z$ to parameters of an output distribution, modeling the likelihood of the data.

By defining both the encoder and decoder probabilistically, VAEs establish a formal connection to latent variable models and enable the derivation of a well-principled objective function, the Evidence Lower Bound (ELBO), which balances data reconstruction with latent space regularization. This structure helps VAEs learn latent spaces suitable for generating new data. We will examine the full model perspective and the ELBO derivation in the subsequent sections.