Mathematical Formulation of CAI Feedback

The Constitutional AI (CAI) feedback process is formalized, with a particular focus on its supervised learning (SL) phase. This phase aims to refine the LLM's behavior by teaching it to emulate constitutionally-aligned revisions of its own outputs, generated through an AI-driven critique and revision loop.

Let $M_{\theta}$ represent the base large language model parameterized by $\theta$. Given an input prompt $x$, the model generates an initial response $y_{init}$: $y_{init} \sim P(y | x; \theta)$ This notation indicates that $y_{init}$ is sampled from the probability distribution over sequences $y$ defined by the model $M_{\theta}$ conditioned on the input $x$.

The core of the CAI SL phase involves generating a more desirable response, $y_{revised}$, based on a predefined constitution $C$. This is typically achieved in two steps:

1. Critique Generation: An AI system (often the same LLM $M_{\theta}$ prompted appropriately, or a dedicated critique model $M_{critique}$) analyzes the initial response $y_{init}$ in the context of the prompt $x$ and the constitution $C$. It identifies aspects of $y_{init}$ that potentially violate principles in $C$ and generates a critique $c$. We can represent this process functionally: $c = \text{Critique}(x, y_{init}, C)$ This critique might be a textual explanation of the flaws or a specific instruction for improvement.

2. Revision Generation: Another AI system (potentially $M_{\theta}$ again, or a dedicated revision model $M_{revise}$) takes the original prompt $x$, the initial response $y_{init}$, the critique $c$, and the constitution $C$ as input. It then generates a revised response $y_{revised}$ that addresses the critique while adhering to $C$. $y_{revised} = \text{Revise}(x, y_{init}, c, C)$ Like the initial response generation, the critique and revision steps can be viewed probabilistically, involving sampling from distributions conditioned on their respective inputs.

The following diagram illustrates this data generation flow for a single training instance:

Data flow for generating a single supervised fine-tuning example in Constitutional AI. The process uses AI-generated critiques and revisions based on a constitution to refine an initial LLM response.

This entire process is repeated for a large dataset of diverse prompts $\{x_i\}$ to generate a dataset of constitutionally-aligned input-output pairs: $D_{CAI} = \{ (x_i, y_{revised, i}) \}_{i=1}^N$

The primary goal of the CAI SL phase is to train a new model, $M_{aligned}$ (often initialized from $M_{\theta}$), using this dataset $D_{CAI}$. The training objective is typically standard supervised fine-tuning (SFT), minimizing the negative log-likelihood of the revised responses given the prompts: $L_{SFT}(\theta_{aligned}) = - \frac{1}{N} \sum_{i=1}^N \log P(y_{revised, i} | x_i; \theta_{aligned})$ Here, $\theta_{aligned}$ represents the parameters of the model being fine-tuned. By minimizing this loss, the model $M_{aligned}$ learns to generate outputs that resemble the revised, constitution-adhering responses directly, without needing the explicit critique and revision steps during inference.

It's important to recognize that this formulation simplifies a potentially complex process. The critique and revision steps themselves might involve multiple LLM calls, specific prompting strategies, and sampling procedures (e.g., using temperature scaling) to generate diverse and effective feedback.

Furthermore, while the primary output of this phase is the dataset $D_{CAI}$ for SFT, the generated data can sometimes be repurposed. For instance, the pair $(y_{revised, i}, y_{init, i})$ for a given prompt $x_i$ implicitly defines a preference: $y_{revised}$ is preferred over $y_{init}$ according to the constitution. This preference data could potentially be used to train a reward model for subsequent reinforcement learning stages (as explored in RLAIF, discussed later in this course), although the original CAI framework emphasizes the direct SFT step using the revised outputs.

This mathematical framing highlights how CAI translates high-level principles (the constitution) into concrete training data ($D_{CAI}$) suitable for standard machine learning optimization techniques ($L_{SFT}$), thereby guiding the LLM's behavior without requiring human labels for every specific interaction. The effectiveness relies on the quality of the constitution and the capability of the AI systems to generate meaningful critiques and revisions based upon it.