Deep learning models have demonstrated significant success on various complex tasks, yet standard implementations often lack reliable uncertainty estimates. Bayesian methods provide a formal framework for reasoning about uncertainty. This chapter introduces techniques for integrating Bayesian principles into deep learning architectures.

You will learn how to formulate Bayesian Neural Networks (BNNs) by placing prior distributions over network parameters, such as weights $w$, instead of using point estimates. We will address the challenges associated with performing inference in these high-dimensional models, focusing on calculating or approximating the posterior distribution $p(w | \mathcal{D})$.

Key topics include:

• Applying advanced Markov Chain Monte Carlo (MCMC) methods, like Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), to sample from the BNN posterior.
• Using scalable Variational Inference (VI) techniques, such as Bayes by Backprop, to approximate the posterior.
• Quantifying and distinguishing between different types of uncertainty (aleatoric and epistemic) in predictions.
• Understanding Variational Autoencoders (VAEs) from a probabilistic modeling standpoint.
• Analyzing the connection between common regularization techniques like dropout and approximate Bayesian inference.
• Practical considerations for training and evaluating BNNs.

We will cover both the theoretical underpinnings and the practical implementation details needed to build and apply these models.