Before constructing sophisticated Bayesian models, we must solidify our understanding of the underlying principles. This chapter revisits the foundations of Bayesian inference, ensuring we have a shared, advanced perspective necessary for the topics ahead. While you likely have prior exposure to Bayesian concepts, we will re-examine them considering the demands of complex machine learning applications.

We begin with a review of probabilistic modeling principles and analyze Bayes' Theorem, particularly its application and implications in high-dimensional settings. Recall the theorem:

$$P(\theta | \mathcal{D}) = \frac{P(\mathcal{D} | \theta) P(\theta)}{P(\mathcal{D})}$$

Here, $P(\theta | \mathcal{D})$ represents the posterior probability of parameters $\theta$ given data $\mathcal{D}$, $P(\mathcal{D} | \theta)$ is the likelihood, $P(\theta)$ is the prior probability, and $P(\mathcal{D})$ is the model evidence or marginal likelihood.

We will discuss the choices and impacts of prior selection, comparing subjective and objective approaches. The connection between Bayesian inference and information theory will be examined through measures like entropy and Kullback-Leibler (KL) divergence, often written as $D_{KL}(P||Q)$. Furthermore, we identify common computational difficulties faced when applying Bayesian methods, such as intractable integrals, and introduce advanced techniques for checking and criticizing the adequacy of these models. This review prepares the groundwork for the advanced inference algorithms and model types presented later.