Advanced Model Checking and Criticism

Building a Bayesian model involves specifying priors and likelihoods, leading to a posterior distribution $P(\theta | \mathcal{D})$. But how do we know if the model we've constructed is any good? Simply achieving convergence in our MCMC sampler or obtaining a low ELBO in variational inference doesn't guarantee that the model accurately represents the data generation process or provides useful insights. We need systematic ways to check our models and critically evaluate their assumptions and fit to the data. This process is known as model checking.

Unlike frequentist hypothesis testing, Bayesian model checking isn't typically about rejecting a null hypothesis. Instead, it's about understanding the ways in which our model fails to capture aspects of the data. Identifying these failures often points towards specific ways the model can be improved. Remember, as George Box famously said, "All models are wrong, but some are useful." Our goal is to build useful models, and model checking helps us assess that usefulness.

Posterior Predictive Checks (PPCs)

The most common and intuitive approach to Bayesian model checking is through Posterior Predictive Checks (PPCs). The core idea is simple: If our model is a good fit for the data, then data simulated from the model after fitting it should look similar to the actual observed data.

Let $y$ represent our observed data and $\theta$ the model parameters. We've already computed (or approximated) the posterior distribution $p(\theta | y)$. The posterior predictive distribution generates replicated datasets, denoted $y^{rep}$, based on the fitted model:

$$p(y^{rep} | y) = \int p(y^{rep} | \theta) p(\theta | y) d\theta$$

In practice, generating $y^{rep}$ is often straightforward, especially if you have posterior samples of $\theta$ from MCMC:

1. For each posterior sample $\theta^{(s)}$ (where $s=1, ..., S$), draw one replicated dataset $y^{rep(s)}$ from the likelihood $p(y | \theta^{(s)})$.
2. The collection $\{y^{rep(s)}\}_{s=1}^S$ represents samples from the posterior predictive distribution.

Once we have these replicated datasets, we compare them to our observed data $y$.

Graphical Checks

Visual comparisons are often the most effective way to start. We can compare various aspects of $y$ with the distribution of the corresponding aspects in $y^{rep}$. Common graphical checks include:

• Histograms or Density Plots: Compare the distribution of $y$ with the distributions of several $y^{rep}$. Does the shape look similar? Are the modes and spread comparable?
• Summary Statistics: Plot the observed value of a summary statistic (e.g., mean, median, variance, min, max) against the histogram of that statistic calculated for each $y^{rep}$.

Comparing the histogram of observed data (blue) against histograms from multiple datasets simulated from the posterior predictive distribution (gray). Significant discrepancies suggest model misfit.

Test Statistics and Bayesian P-values

We can formalize the comparison using test statistics, $T(y)$, which are functions of the data. These can be standard statistics like the mean or variance, or they can be tailored to probe specific aspects of the model we are concerned about (e.g., the number of zero counts in count data, the maximum value, the autocorrelation).

For a chosen test statistic $T$, we compare its value on the observed data, $T(y_{obs})$, with the distribution of values $T(y^{rep})$ obtained from the replicated datasets. A Bayesian p-value (or posterior predictive p-value) is often calculated as:

$$p_B = P(T(y^{rep}) \ge T(y_{obs}) | y)$$

This is estimated from the posterior samples as the proportion of replicated datasets for which the test statistic is greater than or equal to the observed value:

$$p_B \approx \frac{1}{S} \sum_{s=1}^S \mathbb{I}(T(y^{rep(s)}) \ge T(y_{obs}))$$

where $\mathbb{I}(\cdot)$ is the indicator function.

Interpretation: A Bayesian p-value close to 0 or 1 indicates that the observed data is extreme relative to what the model predicts, suggesting a potential misfit with respect to the chosen test statistic. For example, if $p_B \approx 0$, the observed statistic $T(y_{obs})$ is larger than almost all simulated values $T(y^{rep})$. If $p_B \approx 1$, $T(y_{obs})$ is smaller than almost all simulated values. Values around 0.5 suggest the model captures this particular aspect of the data well.

It is important to remember that this is not a frequentist p-value used for hypothesis testing. It's a measure of surprise. We often use multiple test statistics to probe different potential model failures.

Sensitivity Analysis

Bayesian models depend on the choice of priors $P(\theta)$ and the likelihood $P(\mathcal{D} | \theta)$. An analysis requires understanding how sensitive our conclusions (i.e., the posterior distribution $P(\theta | \mathcal{D})$ and posterior predictive checks) are to these choices.

• Prior Sensitivity: How much does the posterior change if we specify different, plausible prior distributions? This is particularly significant when data is scarce, as the prior has more influence. We might compare results using informative vs. weakly informative priors, or priors with different shapes (e.g., Gaussian vs. Student's t). Significant changes in the posterior suggest that our conclusions are heavily influenced by the prior assumptions, which warrants careful justification or gathering more data.
• Likelihood Sensitivity: Sometimes the choice of likelihood function is an approximation or simplification. For example, assuming Gaussian noise when the actual errors might be heavy-tailed. We can check sensitivity by fitting alternative models with different likelihoods (e.g., using a Student's t distribution instead of a Normal distribution for the likelihood) and comparing the results and model fit (using PPCs or other comparison methods discussed later).

Calibration of Uncertainty

A significant advantage of Bayesian methods is their ability to quantify uncertainty through posterior distributions. Model checking should also assess whether this uncertainty quantification is reliable. We want our model to be calibrated. For example, if we compute many 90% credible intervals for different parameters or predictions, roughly 90% of them should contain the true value (if known, or in simulation studies). Poor calibration might manifest as intervals that are consistently too narrow (overconfident) or too wide (underconfident). Techniques exist to specifically check calibration, often involving simulation studies or examining coverage properties on held-out data.

Model Checking is Iterative

Model checking is not a one-off step performed at the very end. It's an integral part of the modeling workflow. When PPCs or sensitivity analyses reveal problems, they guide us on how to refine the model. Perhaps the prior was poorly chosen, the likelihood assumption was inappropriate, or the model structure itself (e.g., missing predictors, interactions, or latent variables) needs reconsideration. After revising the model, we repeat the inference and checking process. This iterative cycle of formulating, fitting, checking, and refining helps us converge towards models that are not only statistically adequate but also scientifically or practically meaningful. The computational challenges mentioned earlier often intertwine with model checking, as complex models might require advanced MCMC or VI techniques (covered in Chapters 2 and 3) to even allow for effective posterior sampling and subsequent checking.