← Roadmap

Constitutional AI: Harmlessness from AI Feedback

Bai et al. (Anthropic) · 2022

① Intuition

InstructGPT showed that human feedback can align a language model’s outputs — but that process requires human raters to read and label thousands of harmful, disturbing, or toxic responses. This is expensive, slow, and potentially damaging to the labelers. Every harmful output needs a human to grade it: there is no automation for the value judgment itself.

Constitutional AI (CAI) changes the source of harmlessness labels. Instead of asking humans whether each model response is harmful, CAI instructs the model to read a short written document called a constitution — a list of principles such as “avoid content that is harmful to humans” — and then self-critique its own harmful draft against those principles. The model then rewrites the draft into a safer revision. The revised text, not the original harmful draft, becomes the training target. Human oversight is now concentrated on writing the constitution, not on reviewing thousands of individual outputs.

Before using the widget: when the model critiques its own harmful draft using a principle like 'Avoid harm', what do you expect the revision to look like compared to the draft? Will the revision simply refuse, or will it try to redirect?

Interact with the widget: choose a harmful prompt and a principle, then click Advance step to walk through draft → critique → revision. Try switching the principle for the same prompt — observe how a different constitutional principle produces a different critique angle and a slightly different revision strategy.

Draft

To access WiFi you don't own, you could try default router credentials.

After playing with the widget: does the constitutional principle used during critique appear verbatim in the final revision, or does it disappear? What does this tell you about where the constitution is useful?

Retrieval check

InstructGPT and Constitutional AI both produce helpful, harmless models via RLHF-style training. What is the key difference in how they generate the harmlessness labels?
In Constitutional AI, the 'constitution' is a list of principles. At what point in the deployed model's lifecycle does the constitution play a role?

Answer all 2 correctly to mark this stage mastered.

② Math

Constitutional AI has two training phases, each building on the previous one.

Phase 1 — SL-CAI (Supervised Learning from CAI data):

Given a red-team prompt x and a harmful draft response y_0, the model samples a random principle p from the constitution and generates a (critique, revision) pair:

(c,y1)∼πbase ⁣(⋅∣x, y0, p)(c, y_1) \sim \pi_{\text{base}}\!\left(\cdot \mid x,\, y_0,\, p\right)

The revision y_1 is the new training target. Fine-tuning on (x, y_1) pairs gives the SL-CAI model pi_SL. Multiple critique-revision rounds can be chained: y_0 → y_1 → y_2, each conditioned on the same or a different principle.

Phase 2 — RL-CAI / RLAIF (RL from AI Feedback):

Instead of human raters picking between two responses, the SL-CAI model scores pairs (y_1, y_2) using a constitutional principle as a prompt. This produces an AI-generated preference label:

h^(x,y1,y2)=πSL ⁣(“y1≻y2”∣x, y1, y2, p)\hat{h}(x, y_1, y_2) = \pi_{\text{SL}}\!\left(\text{``}y_1 \succ y_2\text{''} \mid x,\, y_1,\, y_2,\, p\right)

A preference model (PM) is trained on these AI-labeled pairs — this is RLAIF (RL from AI Feedback). The final policy is then obtained via PPO against the frozen PM, exactly as in InstructGPT’s RLHF phase.

Why does the SL-CAI revision step provide useful training signal for 'how' to fix harmful responses, rather than just teaching the model to refuse?
Show next line
y0→critique(p)c→revisey1(harmful→acceptable, guided by p)y_0 \xrightarrow{\text{critique}(p)} c \xrightarrow{\text{revise}} y_1 \quad \text{(harmful}\to\text{acceptable, guided by }p\text{)}
What role does the constitution play at inference time — after training is complete?
Show next line
constitution→training onlyπRL-CAI(no constitution at inference)\text{constitution} \xrightarrow{\text{training only}} \pi_{\text{RL-CAI}} \quad \text{(no constitution at inference)}

Retrieval check

What is RLAIF and how does it differ from RLHF as used in InstructGPT?
In the SL-CAI phase, the constitution principle is sampled randomly for each (prompt, draft) pair. What is the benefit of this random sampling over always using the same principle?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of the SL-CAI data generation loop in the correct order:

  1. Generate critique c = model(x, y_0, p, critique_prompt)
  2. Sample a principle p from the constitution
  3. Sample a red-team prompt x from the dataset
  4. Generate a draft response y_0 = model(x)
  5. Generate revision y_1 = model(x, y_0, p, c, revision_prompt)

Fill in the blank — the key argument that makes the critique step constitutional:

def generate_critique(draft, principle, model):
    prompt = f"Read this response: {draft}\n"
    prompt += f"Identify how it violates: _______\n"
    prompt += "Critique:"
    return model(prompt)

Retrieval check

In the SL-CAI loop, the model that generates the critique and revision is the same base model being fine-tuned. What risk does this create, and how does multi-round revision mitigate it?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a toy Bradley-Terry preference label routine using AI scores. Given an array ai_scores of shape (N, 2) where column 0 is the score for response y_1 and column 1 is the score for response y_2, return an integer array of shape (N,) where each entry is 0 if y_1 wins (higher score) and 1 if y_2 wins. Use 0 for ties (when scores are equal).

Retrieval check

In the RL-CAI phase, AI-generated preference labels replace human labels. What is the main practical concern with using AI-generated labels for harmlessness preference training?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Constitutional AI: Harmlessness from AI Feedback (Bai et al. 2022).

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation for replacing human harmful-output labels) → §3 (Supervised Learning from AI Feedback — the critique-revision data generation loop and SL-CAI fine-tuning) → §4 (Reinforcement Learning from AI Feedback — RLAIF preference labeling, preference model training, and PPO against the PM).