Mathematical Formulation of Basic Autoencoders

Building on the components introduced earlier, let's formalize the basic autoencoder mathematically. An autoencoder consists of two main parts: an encoder and a decoder, which are typically implemented as neural networks.

The Encoder Function

The encoder, denoted by $f$, takes an input vector $x \in \mathbb{R}^d$ and maps it to a lower-dimensional latent representation (or code) $z \in \mathbb{R}^k$, where $k < d$. This mapping is parameterized by the encoder's parameters $\theta_e$, which include weights and biases. We can write this as:

$z = f(x; \theta_e)$

For a simple autoencoder with a single hidden layer, this might look like:

$z = \sigma_e(W_e x + b_e)$

Here, $W_e$ is the weight matrix, $b_e$ is the bias vector, and $\sigma_e$ is an element-wise activation function (like sigmoid, ReLU, or tanh) for the encoder. In practice, the encoder can be a much deeper network with multiple layers.

The Decoder Function

The decoder, denoted by $g$, takes the latent representation $z \in \mathbb{R}^k$ and maps it back to a reconstruction $\hat{x} \in \mathbb{R}^d$. The goal is for $\hat{x}$ to be as close as possible to the original input $x$. This mapping is parameterized by the decoder's parameters $\theta_d$:

$\hat{x} = g(z; \theta_d)$

Similar to the encoder, a simple single-layer decoder might be formulated as:

$\hat{x} = \sigma_d(W_d z + b_d)$

Where $W_d$ and $b_d$ are the decoder's weights and biases, and $\sigma_d$ is the decoder's activation function. The choice of $\sigma_d$ often depends on the nature of the input data $x$. For instance, if $x$ represents pixel values normalized between 0 and 1, a sigmoid activation is common. If $x$ can take any real value (after normalization), a linear activation might be used.

The Overall Autoencoder and Objective Function

The entire autoencoder combines the encoder and decoder. Given an input $x$, the reconstruction $\hat{x}$ is obtained by first encoding $x$ to $z$ and then decoding $z$ back to $\hat{x}$:

$\hat{x} = g(f(x; \theta_e); \theta_d)$

The autoencoder is trained by minimizing the difference between the original input $x$ and its reconstruction $\hat{x}$. This difference is quantified by a reconstruction loss function, $L(x, \hat{x})$. The overall objective function $J(\theta)$ for the autoencoder, where $\theta = \{\theta_e, \theta_d\}$ represents all trainable parameters, is typically the average loss over a dataset $D = \{x^{(1)}, x^{(2)}, ..., x^{(N)}\}$ of $N$ samples:

$J(\theta) = \frac{1}{N} \sum_{i=1}^{N} L(x^{(i)}, \hat{x}^{(i)})$

Substituting the encoder and decoder functions:

$J(\theta) = \frac{1}{N} \sum_{i=1}^{N} L(x^{(i)}, g(f(x^{(i)}; \theta_e); \theta_d))$

Common Reconstruction Loss Functions

The choice of $L(x, \hat{x})$ is important and depends on the characteristics of the input data $x$:

1. Mean Squared Error (MSE): Used when the input data is continuous, often assumed to be Gaussian. It measures the average squared difference between the elements of $x$ and $\hat{x}$. $L_{MSE}(x, \hat{x}) = \frac{1}{d} \sum_{j=1}^{d} (x_j - \hat{x}_j)^2$ Minimizing MSE corresponds to maximizing the log-likelihood under a Gaussian assumption for the reconstruction error. It's suitable for inputs like normalized image pixels or continuous features.

2. Binary Cross-Entropy (BCE): Used when the input data is binary or can be interpreted as probabilities (e.g., values in the range [0, 1]). This often applies to images like MNIST where pixel values are treated as Bernoulli parameters. $L_{BCE}(x, \hat{x}) = - \frac{1}{d} \sum_{j=1}^{d} [x_j \log(\hat{x}_j) + (1 - x_j) \log(1 - \hat{x}_j)]$ Minimizing BCE corresponds to maximizing the log-likelihood under a Bernoulli distribution assumption for each element of the input. When using BCE, the decoder's final activation function $\sigma_d$ should typically be a sigmoid function to ensure outputs $\hat{x}_j$ are in the range (0, 1), interpretable as probabilities.

Optimization Perspective

The goal of training is to find the optimal parameters $\theta^* = \{\theta_e^*, \theta_d^*\}$ that minimize the objective function $J(\theta)$. This is achieved using optimization algorithms based on gradient descent. Backpropagation is used to compute the gradients of the loss $L$ with respect to all parameters in $\theta_e$ and $\theta_d$. These gradients, $\nabla_{\theta_e} J(\theta)$ and $\nabla_{\theta_d} J(\theta)$, are then used by optimizers like Adam or RMSprop to iteratively update the parameters.

In essence, the mathematical formulation defines a specific optimization problem: learn functions $f$ and $g$ such that applying them sequentially ($g \circ f$) reconstructs the input data as accurately as possible, according to the chosen loss metric $L$. The bottleneck $z$ forces the network to learn a compressed representation that captures the most salient information needed for reconstruction.