Latent Dirichlet Allocation (LDA): Bayesian Formulation

Latent Dirichlet Allocation (LDA) stands as a prominent example of a Probabilistic Graphical Model applied to text analysis, specifically for identifying thematic structures, often called "topics," within a collection of documents (a corpus). It operates on the principle of a generative process, meaning it describes a probabilistic mechanism by which the documents could have been created. Understanding this generative story is fundamental to grasping the Bayesian formulation of LDA.

The core idea behind LDA is twofold:

1. Each document is modeled as a mixture of a fixed number of topics. For example, a news article about technology might be 70% "Technology," 20% "Business," and 10% "Politics."
2. Each topic is modeled as a distribution over words in the vocabulary. The "Technology" topic might assign high probabilities to words like "software," "cloud," "AI," and "network," while the "Business" topic favors words like "stock," "market," "profit," and "company."

LDA treats both the topic mixture for each document and the word distribution for each topic as latent (unobserved) random variables. Furthermore, it assigns a specific topic to each word instance within each document. The only observed data are the words themselves. The goal of inference, which we'll discuss in subsequent sections, is to infer these latent structures (topic mixtures, topic-word distributions, and word-topic assignments) given the observed words.

The Generative Process

Let's outline the step-by-step generative process assumed by LDA for a corpus of $M$ documents, with a predefined number of $K$ topics, and a vocabulary of $V$ unique words.

1. Define Priors:

   • Choose a parameter $\alpha$ for the Dirichlet prior over document-topic distributions. Typically, $\alpha$ is a symmetric $K$-dimensional vector, $\alpha = (\alpha_1, ..., \alpha_K)$, often set to $\alpha_i = \alpha_0 / K$ for some scalar $\alpha_0$.
   • Choose a parameter $\beta$ for the Dirichlet prior over topic-word distributions. Typically, $\beta$ is a symmetric $V$-dimensional vector, $\beta = (\beta_1, ..., \beta_V)$, often set to $\beta_j = \beta_0 / V$ for some scalar $\beta_0$.

2. Generate Topic-Word Distributions:

   • For each topic $k \in \{1, ..., K\}$:
     • Draw a word distribution $\phi_k \sim Dir(\beta)$. $\phi_k$ is a $V$-dimensional vector where $\phi_{kv}$ is the probability of word $v$ occurring under topic $k$, and $\sum_{v=1}^V \phi_{kv} = 1$.

3. Generate Document-Specific Variables:

   • For each document $d \in \{1, ..., M\}$:
     • Draw a topic mixture $\theta_d \sim Dir(\alpha)$. $\theta_d$ is a $K$-dimensional vector where $\theta_{dk}$ is the proportion of topic $k$ in document $d$, and $\sum_{k=1}^K \theta_{dk} = 1$.
     • Determine the number of words in the document, $N_d$.
     • For each word position $n \in \{1, ..., N_d\}$:
       • Draw a topic assignment $z_{dn} \sim Cat(\theta_d)$. $z_{dn}$ indicates which topic generated the $n$-th word in document $d$.
       • Draw the observed word $w_{dn} \sim Cat(\phi_{z_{dn}})$. $w_{dn}$ is the actual word observed at position $n$ in document $d$, drawn from the word distribution corresponding to the assigned topic $z_{dn}$.

The hyperparameters $\alpha$ and $\beta$ control the characteristics of the topic mixtures and topic-word distributions, respectively. Lower values generally lead to sparser distributions (documents composed of fewer topics, topics focused on fewer words).

Graphical Model Representation

This generative process corresponds directly to a Bayesian Network structure. We can visualize these dependencies using plate notation, where plates (rectangles) indicate replication of variables.

Plate notation for the Latent Dirichlet Allocation model. Circles represent random variables (shaded are latent, double-circle is observed). Rectangles (plates) denote replication. Arrows indicate conditional dependencies. The dashed line from $\phi$ to $w$ shows that the specific $\phi_k$ used depends on the value of $z$.

Bayesian Formulation Summary

In this PGM, the latent variables are the document-topic distributions $\boldsymbol{\theta} = \{\theta_d\}_{d=1}^M$, the topic-word distributions $\boldsymbol{\phi} = \{\phi_k\}_{k=1}^K$, and the topic assignments for each word $\mathbf{Z} = \{z_{dn}\}_{d=1, n=1}^{M, N_d}$. The observed variables are the words themselves, $\mathbf{W} = \{w_{dn}\}_{d=1, n=1}^{M, N_d}$. The parameters $\alpha$ and $\beta$ are typically treated as fixed hyperparameters, although they can also be learned (e.g., using empirical Bayes or placing hyperpriors).

The complete joint probability distribution over all variables, given the hyperparameters, factorizes according to the graphical model:

$$p(\mathbf{W}, \mathbf{Z}, \boldsymbol{\theta}, \boldsymbol{\phi} | \alpha, \beta) = \left( \prod_{k=1}^K p(\phi_k | \beta) \right) \left( \prod_{d=1}^M p(\theta_d | \alpha) \left( \prod_{n=1}^{N_d} p(z_{dn} | \theta_d) p(w_{dn} | z_{dn}, \boldsymbol{\phi}) \right) \right)$$

Here:

• $p(\phi_k | \beta)$ is the Dirichlet probability density for topic $k$'s word distribution.
• $p(\theta_d | \alpha)$ is the Dirichlet probability density for document $d$'s topic distribution.
• $p(z_{dn} | \theta_d)$ is the Categorical probability mass for assigning topic $z_{dn}$ based on $\theta_d$.
• $p(w_{dn} | z_{dn}, \boldsymbol{\phi})$ is the Categorical probability mass for observing word $w_{dn}$ given its assigned topic $z_{dn}$ and the relevant topic-word distribution $\phi_{z_{dn}}$.

The Bayesian formulation sets the stage for inference. Our objective is typically to compute the posterior distribution of the latent variables given the observed documents: $p(\mathbf{Z}, \boldsymbol{\theta}, \boldsymbol{\phi} | \mathbf{W}, \alpha, \beta)$. This posterior distribution reveals the hidden thematic structure. However, calculating this posterior directly is intractable due to the complex dependencies and high dimensionality. This necessitates the use of approximate inference techniques like Collapsed Gibbs Sampling or Variational Bayes, which are the subjects of the following sections.