① Intuition
InstructGPT showed that human feedback can align a language model’s outputs — but that process requires human raters to read and label thousands of harmful, disturbing, or toxic responses. This is expensive, slow, and potentially damaging to the labelers. Every harmful output needs a human to grade it: there is no automation for the value judgment itself.
Constitutional AI (CAI) changes the source of harmlessness labels. Instead of asking humans whether each model response is harmful, CAI instructs the model to read a short written document called a constitution — a list of principles such as “avoid content that is harmful to humans” — and then self-critique its own harmful draft against those principles. The model then rewrites the draft into a safer revision. The revised text, not the original harmful draft, becomes the training target. Human oversight is now concentrated on writing the constitution, not on reviewing thousands of individual outputs.
Before using the widget: when the model critiques its own harmful draft using a principle like 'Avoid harm', what do you expect the revision to look like compared to the draft? Will the revision simply refuse, or will it try to redirect?
Good revisions do more than refuse — they redirect to a legitimate alternative. The constitution principle guides what the critique flags, and the revision must satisfy that principle while still being genuinely useful. A flat refusal often fails the helpfulness dimension. The critique is the reasoning step that connects the principle to the specific harm in the draft; the revision integrates that reasoning into a new response.
Interact with the widget: choose a harmful prompt and a principle, then click Advance step to walk through draft → critique → revision. Try switching the principle for the same prompt — observe how a different constitutional principle produces a different critique angle and a slightly different revision strategy.
Draft
To access WiFi you don't own, you could try default router credentials.
After playing with the widget: does the constitutional principle used during critique appear verbatim in the final revision, or does it disappear? What does this tell you about where the constitution is useful?
The principle disappears from the revision text — it was only the reasoning scaffold during the critique step. The revision is self-contained; a reader of just the revision would not know which principle triggered it. This reflects how Constitutional AI works in training: the constitution shapes the (critique, revision) pairs used for fine-tuning, but the deployed model has internalized those behaviors without needing the constitution at inference time. The constitution is a training-time tool, not a runtime filter.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Constitutional AI has two training phases, each building on the previous one.
Phase 1 — SL-CAI (Supervised Learning from CAI data):
Given a red-team prompt x and a harmful draft response y_0, the model samples a random principle p from the constitution and generates a (critique, revision) pair:
The revision y_1 is the new training target. Fine-tuning on (x, y_1) pairs gives the SL-CAI model pi_SL. Multiple critique-revision rounds can be chained: y_0 → y_1 → y_2, each conditioned on the same or a different principle.
Phase 2 — RL-CAI / RLAIF (RL from AI Feedback):
Instead of human raters picking between two responses, the SL-CAI model scores pairs (y_1, y_2) using a constitutional principle as a prompt. This produces an AI-generated preference label:
A preference model (PM) is trained on these AI-labeled pairs — this is RLAIF (RL from AI Feedback). The final policy is then obtained via PPO against the frozen PM, exactly as in InstructGPT’s RLHF phase.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of the SL-CAI data generation loop in the correct order:
- Generate critique c = model(x, y_0, p, critique_prompt)
- Sample a principle p from the constitution
- Sample a red-team prompt x from the dataset
- Generate a draft response y_0 = model(x)
- Generate revision y_1 = model(x, y_0, p, c, revision_prompt)
Fill in the blank — the key argument that makes the critique step constitutional:
def generate_critique(draft, principle, model):
prompt = f"Read this response: {draft}\n"
prompt += f"Identify how it violates: _______\n"
prompt += "Critique:"
return model(prompt)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a toy Bradley-Terry preference label routine using AI scores. Given an array ai_scores of shape (N, 2) where column 0 is the score for response y_1 and column 1 is the score for response y_2, return an integer array of shape (N,) where each entry is 0 if y_1 wins (higher score) and 1 if y_2 wins. Use 0 for ties (when scores are equal).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 (Introduction — motivation for replacing human harmful-output labels) → §3 (Supervised Learning from AI Feedback — the critique-revision data generation loop and SL-CAI fine-tuning) → §4 (Reinforcement Learning from AI Feedback — RLAIF preference labeling, preference model training, and PPO against the PM).