Probabilistic Modeling Principles

As outlined in the chapter introduction, our goal here isn't just to recite definitions but to solidify the principles of probabilistic modeling from a perspective suited for advanced Bayesian machine learning. At its heart, probabilistic modeling is about using the language of probability theory to articulate assumptions about how data is generated and to reason about uncertainty. It provides a coherent mathematical framework for representing knowledge, combining evidence, and making predictions.

The Generative Perspective

A core idea in Bayesian modeling is the generative viewpoint. Instead of directly modeling the prediction function $P(y|x)$, we often think about how the entire dataset $\mathcal{D}$, potentially consisting of features $X$ and targets $Y$, could have come into existence. We hypothesize a stochastic process, governed by some parameters $\theta$ and possibly latent (unobserved) variables $z$, that could have generated $\mathcal{D}$.

This generative story involves specifying the joint probability distribution over all relevant quantities, both observed and unobserved: $P(\mathcal{D}, \theta, z)$. Constructing a model means making explicit choices about this distribution.

Components of a Probabilistic Model

Building a probabilistic model requires specifying several key components:

1. Variables: We identify the different types of quantities involved:

   • Observed Variables: These constitute our data $\mathcal{D}$. This could be features, labels, sensor readings, text documents, etc. We treat these as fixed outcomes of the underlying generative process.
   • Parameters ($\theta$): These are the unobserved quantities that govern the relationship between variables in the model. Examples include regression coefficients, the mean and variance of a distribution, or the weights in a neural network. In the Bayesian approach, parameters are treated as random variables themselves, having distributions.
   • Latent Variables ($z$): These are other unobserved variables that are not parameters but part of the model structure, often introduced to capture underlying structure, clusters, or states. Topic models (like LDA discussed in Chapter 5) rely heavily on latent variables representing topic assignments.

2. Structural Assumptions: We define the probabilistic relationships between these variables. Which variables directly influence others? Are certain variables independent given others? These assumptions define the model's structure and are often visualized using Probabilistic Graphical Models (covered later). Conditional independence assumptions are particularly significant for simplifying the model and enabling efficient computation. For example, we often assume data points are independent given the parameters.

3. Probability Distributions: We choose specific mathematical forms for the probability distributions that connect the variables. This involves selecting:

   • The Likelihood $P(\mathcal{D} | \theta, z)$: This distribution quantifies the probability of observing the data $\mathcal{D}$ given specific values for the parameters $\theta$ and latent variables $z$. The choice of likelihood depends heavily on the nature of the data (e.g., Normal distribution for continuous data, Bernoulli for binary outcomes, Poisson for count data).
   • The Prior $P(\theta, z)$ or $P(\theta)P(z)$: This distribution expresses our beliefs about the parameters and latent variables before observing any data. It encodes assumptions or incorporates existing knowledge. We'll discuss prior selection strategies later in this chapter.

The Likelihood Function $P(\mathcal{D} | \theta)$

Let's focus on the likelihood (often written without explicit mention of latent variables $z$ for simplicity, $P(\mathcal{D} | \theta)$). It's derived directly from the generative assumptions about how data points are produced given the parameters. For instance, if we assume $N$ independent and identically distributed (i.i.d.) data points $\mathcal{D} = \{x_1, ..., x_N\}$ drawn from a distribution $f(x | \theta)$, the likelihood is:

$$P(\mathcal{D} | \theta) = \prod_{i=1}^{N} f(x_i | \theta)$$

It's important to remember that the likelihood is viewed as a function of $\theta$ for fixed data \mathcal{D. It tells us how plausible different parameter values are in light of the data we actually observed. It is not a probability distribution over $\theta$. Choosing an appropriate likelihood function reflecting the data's characteristics is a fundamental step in modeling. A common assumption related to the i.i.d. case is exchangeability, which means the order of observations doesn't affect the joint probability.

The Prior Distribution $P(\theta)$

The prior distribution $P(\theta)$ captures our uncertainty or knowledge about the parameters $\theta$ before considering the specific dataset $\mathcal{D}$. This is a defining feature of Bayesian inference. Priors can:

• Incorporate domain knowledge.
• Act as regularizers, preventing parameters from taking extreme values, which is especially important in complex models or high-dimensional settings to avoid overfitting.
• Reflect a state of relative ignorance (using "non-informative" or "objective" priors, though true objectivity is debatable).

The interplay between the prior and the likelihood function, mediated by Bayes' Theorem, allows us to formally update our beliefs.

Putting It Together: The Joint Distribution and Inference

The complete probabilistic model is defined by the joint distribution of data and parameters (and latent variables, if any). For parameters $\theta$ and data $\mathcal{D}$, this is:

$$P(\mathcal{D}, \theta) = P(\mathcal{D} | \theta) P(\theta)$$

This joint distribution encapsulates the entire generative story. Our primary goal in Bayesian inference is then to compute the posterior distribution $P(\theta | \mathcal{D})$, which represents our updated beliefs about the parameters after observing the data. As stated in the chapter introduction, this is achieved via Bayes' Theorem:

$$P(\theta | \mathcal{D}) = \frac{P(\mathcal{D} | \theta) P(\theta)}{P(\mathcal{D})}$$

where $P(\mathcal{D}) = \int P(\mathcal{D} | \theta) P(\theta) d\theta$ is the marginal likelihood or evidence.

A Simple Example: Flipping a Biased Coin

Consider modeling the outcome of $N$ flips of a potentially biased coin.

• Observed Data $\mathcal{D}$: A sequence of Heads (H) and Tails (T), say $N_H$ heads and $N_T$ tails ($N = N_H + N_T$).

• Parameter $\theta$: The unknown probability of getting Heads, $0 \le \theta \le 1$.

• Generative Process:

  1. Choose a value for the bias $\theta$ according to some prior belief.
  2. For each flip $i=1..N$, generate an outcome $x_i \in \{H, T\}$ based on the probability $\theta$.

• Likelihood: Assuming independent flips, the probability of observing $N_H$ heads and $N_T$ tails is given by the Binomial likelihood (up to a constant factor):

  $$P(\mathcal{D} | \theta) \propto \theta^{N_H} (1-\theta)^{N_T}$$

• Prior: We need to specify a prior distribution for $\theta$. A common choice for probabilities is the Beta distribution, $P(\theta) = \text{Beta}(\theta | \alpha, \beta) \propto \theta^{\alpha-1} (1-\theta)^{\beta-1}$, where $\alpha$ and $\beta$ are hyperparameters reflecting prior beliefs (e.g., $\alpha=1, \beta=1$ corresponds to a uniform prior, suggesting no preference).

• Posterior: Using Bayes' Theorem, the posterior distribution for $\theta$ is also a Beta distribution:

  $$P(\theta | \mathcal{D}) \propto P(\mathcal{D} | \theta) P(\theta) \propto \theta^{N_H} (1-\theta)^{N_T} \theta^{\alpha-1} (1-\theta)^{\beta-1} = \theta^{N_H + \alpha - 1} (1-\theta)^{N_T + \beta - 1}$$

  So, $P(\theta | \mathcal{D}) = \text{Beta}(\theta | N_H + \alpha, N_T + \beta)$.

Here's a graphical representation of this simple generative model:

This diagram shows the generative process for the coin flip example. Hyperparameters $\alpha, \beta$ (fixed, represented by points) determine the prior on the coin bias $\theta$ (unobserved parameter, red node). The bias $\theta$ then determines the probability for each observed coin flip $x_i$ (observed variable, double circle, blue node). The plate indicates that this process is repeated $N$ times.

While this example is simple, the underlying principles extend to vastly more complex models encountered in machine learning. Defining a clear generative process, specifying appropriate likelihoods and priors, and then using Bayes' theorem to derive the posterior form the foundation upon which sophisticated techniques like Gaussian Processes and Bayesian Deep Learning are built. Understanding these principles allows us not only to apply existing models but also to construct novel probabilistic models tailored to specific problems, explicitly handling and quantifying uncertainty. The challenge, as we'll see, often lies in computing the posterior distribution, especially when the integral for the evidence $P(\mathcal{D})$ is intractable, motivating the advanced inference methods discussed in subsequent chapters.