Bayesian Neural Networks (BNNs): Priors over Weights

Standard neural networks learn a single set of optimal parameters, typically weights and biases, often found by minimizing a loss function using techniques like gradient descent. These represent point estimates in the immense space of possible parameters. While effective for prediction, this approach doesn't inherently capture the uncertainty associated with these parameter values or the resulting predictions. If the training data is limited or noisy, multiple different sets of weights might explain the data almost equally well. A standard network picks just one, potentially becoming overconfident in its predictions.

Bayesian Neural Networks (BNNs) address this by embracing the Bayesian philosophy: treat the network parameters (weights $w$ and biases, collectively denoted as $w$) not as fixed values to be optimized, but as random variables. Instead of finding a single best set of weights $\hat{w}$, our goal in a BNN is to infer the full posterior distribution over the weights given the observed data $\mathcal{D}$: $p(w | \mathcal{D})$. This posterior distribution captures our belief about plausible parameter values after seeing the data, inherently quantifying the uncertainty associated with them.

Defining the Prior: $p(w)$

The foundation of any Bayesian model is the prior distribution, $p(w)$. This distribution encodes our beliefs about the parameters before observing any data. In the context of BNNs, the prior is placed over all the weights and biases in the network.

What does it mean to have a belief about weights? Often, we don't have strong prior information about specific weight values. A common and pragmatic choice is to use a simple, mathematically convenient prior that reflects general assumptions. A very frequent choice is an independent Gaussian prior centered at zero for each weight $w_i$:

$$w_i \sim \mathcal{N}(0, \sigma_p^2)$$

This implies a prior for the entire weight vector $w$:

$$p(w) = \prod_i \mathcal{N}(w_i | 0, \sigma_p^2)$$

Here, $\sigma_p^2$ is the prior variance, a hyperparameter. This prior expresses a belief that weights are likely to be small and centered around zero. Larger values of $\sigma_p^2$ correspond to a "weaker" prior, allowing weights to deviate further from zero, while smaller values impose a stronger constraint, pushing weights towards zero.

Choosing a zero-mean Gaussian prior has a direct connection to L2 regularization (weight decay) used in standard neural network training. Recall that L2 regularization adds a penalty term $\lambda \sum_i w_i^2$ to the loss function. Maximizing the posterior probability (MAP estimation) under a Gaussian prior is equivalent to minimizing a loss function with an L2 penalty, where the regularization strength $\lambda$ is related to the prior variance $\sigma_p^2$. However, the full Bayesian approach goes further; we aim to characterize the entire posterior distribution $p(w|\mathcal{D})$, not just find its mode. The prior influences the shape of this entire distribution.

Other priors are possible:

• Laplace Prior: $p(w_i) \propto \exp(-|w_i|/\beta)$. This encourages sparsity (many weights being exactly zero), analogous to L1 regularization.
• Hierarchical Priors: The prior hyperparameters (like $\sigma_p^2$) can themselves be given prior distributions, allowing the model to learn the appropriate level of regularization from the data.
• Informative Priors: If domain knowledge suggests certain weight structures or values are more plausible, this can be encoded in a more specific prior.

For now, we'll often assume a simple Gaussian prior due to its simplicity and connection to weight decay.

The Likelihood: $p(\mathcal{D}|w)$

The likelihood function, $p(\mathcal{D}|w)$, describes the probability of observing the dataset $\mathcal{D = \{ (x_n, y_n) \}_{n=1}^N}$ given a specific setting of the network parameters $w$. This is the same as in standard neural networks and is determined by the network architecture and the assumed distribution of the output.

• Regression: If we assume the target variable $y$ follows a Gaussian distribution around the network's output $f(x; w)$ with noise variance $\sigma^2$, the likelihood for a single data point $(x_n, y_n)$ is $\mathcal{N}(y_n | f(x_n; w), \sigma^2)$. The likelihood for the entire dataset (assuming independence) is the product:

  $$p(\mathcal{D}|w) = \prod_{n=1}^N \mathcal{N}(y_n | f(x_n; w), \sigma^2)$$

  Minimizing the negative log-likelihood $-\log p(\mathcal{D}|w)$ in this case corresponds to minimizing the Mean Squared Error (MSE) loss, plus a term related to the noise variance $\sigma^2$.

• Classification: For classification, the network output typically represents probabilities (e.g., via a softmax layer). If $y_n$ is a one-hot encoded label, the likelihood for a single data point is often modeled using a Categorical distribution, where the probabilities are given by the network output $f(x_n; w)$.

  $$p(y_n | x_n, w) = \text{Categorical}(y_n | f(x_n; w))$$

  Minimizing the negative log-likelihood $-\log p(\mathcal{D}|w)$ here corresponds to minimizing the cross-entropy loss.

The choice of likelihood links the abstract network parameters $w$ to the actual data $\mathcal{D}$.

The Posterior: $p(w|\mathcal{D})$

Having defined the prior $p(w)$ and the likelihood $p(\mathcal{D}|w)$, we can combine them using Bayes' Theorem to obtain the posterior distribution over the weights:

$$p(w | \mathcal{D}) = \frac{p(\mathcal{D} | w) p(w)}{p(\mathcal{D})}$$

• $p(w | \mathcal{D})$ is the posterior distribution: Our updated belief about the weights after observing the data. This is the central object we want to compute or approximate in a BNN.
• $p(\mathcal{D} | w)$ is the likelihood: The probability of the data given specific weights (related to the standard loss function).
• $p(w)$ is the prior distribution: Our initial belief about the weights before seeing data (acts as a regularizer).
• $p(\mathcal{D})$ is the marginal likelihood or evidence: $p(\mathcal{D}) = \int p(\mathcal{D} | w) p(w) dw$. This term acts as a normalization constant, ensuring the posterior integrates to 1.

The diagram below contrasts the standard and Bayesian approaches:

Comparison between a standard neural network learning point estimates for weights and a Bayesian neural network learning posterior distributions over weights.

The main challenge in BNNs lies in the computation of the posterior $p(w | \mathcal{D})$. For deep neural networks with potentially millions of parameters, the integral required for the marginal likelihood $p(\mathcal{D})$ is intractable. Furthermore, the posterior distribution itself is high-dimensional and complex, making direct calculation impossible.

Therefore, instead of calculating the exact posterior, we must rely on approximation methods. The subsequent sections in this chapter will look at two main families of techniques for addressing this challenge: Markov Chain Monte Carlo (MCMC) methods and Variational Inference (VI). These methods allow us to either draw samples from the posterior distribution or find an approximation to it, enabling us to use the Bayesian framework for deep learning tasks, particularly for uncertainty quantification.