The Supervised Learning Phase (Critique and Revision)

The Supervised Learning (SL) phase is the initial training stage of Constitutional AI, where the principles for designing effective constitutions are practically applied. This phase operationalizes a constitution by using AI to generate examples of desirable behavior, thereby creating a dataset for fine-tuning the language model towards alignment before any reinforcement learning steps.

The core objective here is to translate the abstract rules of the constitution into concrete training signals. Instead of relying solely on human annotators to label preferred outputs, CAI uses an AI system to critique and revise the model's own responses based on the provided constitutional principles. This process forms a scalable mechanism for generating alignment data.

This SL phase typically unfolds in two main stages, often executed iteratively:

1. Critique Generation: Identifying violations of constitutional principles within an initial model response.
2. Revision Generation: Rewriting the initial response to rectify the identified violations and better adhere to the constitution.

Let's analyze each stage in detail.

Critique Generation

The process begins by prompting the initial, pre-aligned Large Language Model (LLM) to generate a response to a given input prompt. This initial response might be helpful but could potentially violate one or more principles outlined in the constitution (e.g., being evasive, generating harmful content, expressing undesirable biases).

• Input: The primary inputs to the critique stage are:
  • The initial response generated by the LLM.
  • The relevant principle(s) from the constitution that the response might violate.
• Mechanism: An AI system, often referred to as the "critiquer," analyzes the initial response specifically through the lens of the provided constitutional principle(s). This critiquer can be the same LLM used for generation but invoked with a carefully crafted meta-prompt, or it could be a separate, dedicated model fine-tuned for this task. The meta-prompt typically instructs the AI to:
  1. Assume the role of an AI assistant evaluating responses against a specific set of rules.
  2. Identify which rule(s), if any, the given response violates.
  3. Explain why the response violates the identified rule(s). Few-shot prompting is often effective here, where the prompt includes examples of responses, relevant principles, and corresponding desired critiques.
• Output: The output is a textual critique. This critique explicitly states the identified constitutional violation and provides reasoning. For example, if a principle states the AI should refuse to generate illegal instructions, and the initial response provided such instructions, the critique might be: "The response provides instructions for an illegal act, violating Principle 7 regarding safety and legality."

Revision Generation

Once a critique is generated, the next step is to revise the initial response to address the identified issues.

• Input: The inputs to the revision stage are:
  • The original, problematic response.
  • The critique generated in the previous step.
• Mechanism: An AI system, the "reviser" (again, potentially the same LLM prompted differently or a separate model), is tasked with rewriting the original response. The meta-prompt for this stage instructs the AI to modify the initial response specifically to address the points raised in the critique, ensuring the output aligns with the constitution while ideally maintaining helpfulness and relevance to the original user prompt.
• Output: The result is a revised response. This revised response represents a constitutionally preferred alternative to the initial output. For instance, following the critique above, the revised response might be: "I cannot provide instructions for illegal activities. However, I can discuss the legal implications or provide information on safer alternatives if applicable."

Creating the Supervised Fine-Tuning Dataset

The power of this critique-and-revision loop lies in its ability to automatically generate training data for supervised fine-tuning (SFT). Each pass through the loop yields a pair: the initial prompt and the revised, constitution-aligned response.

$$\text{SFT Training Pair} = (\text{Input Prompt}, \text{Revised Response})$$

By generating many such pairs across a diverse set of prompts, a substantial dataset is constructed. This dataset embodies the constitutional principles in the form of input-output examples. The original LLM is then fine-tuned on this dataset. The SFT objective teaches the model to directly produce responses similar to the revised outputs when presented with the corresponding prompts, effectively internalizing the constitutional constraints demonstrated during the critique-and-revision process.

Flow of the CAI Supervised Learning phase. An initial response is critiqued based on the constitution, then revised based on the critique, generating a training pair for supervised fine-tuning.

This self-correction mechanism, guided by an explicit constitution, allows for scaling the alignment process past what is feasible with human labeling alone. The resulting model, fine-tuned on this AI-generated data, becomes the foundation for a more constitutionally-aligned system, potentially ready for further refinement through reinforcement learning techniques like RLAIF, which we will discuss in later chapters. The quality of this phase heavily depends on the clarity of the constitution and the sophistication of the prompting strategies used to guide the critique and revision models effectively.