① Intuition
InstructGPT showed that RLHF can align language models to human preferences — but it requires a multi-stage pipeline: first train a reward model (RM) on preference pairs, then run PPO to optimize the policy against the RM while keeping it close to the reference policy via a KL-divergence penalty. That’s two separate models, two separate training loops, and a dozen sensitive hyperparameters. If any part is misconfigured, the policy either reward-hacks the RM or barely moves from the SFT baseline.
DPO (Rafailov et al., 2023) asks a deceptively simple question: is the reward model even necessary? The key insight is that the KL-regularized RL objective has a closed-form optimal policy: pi*(y|x) ∝ pi_ref(y|x) exp(r(x,y) / beta). Inverting that relationship lets you express the reward implicitly as r(x,y) = beta * log(pi*(y|x) / pi_ref(y|x)) + C(x). Plugging this implicit reward back into the Bradley-Terry preference loss causes the partition function C(x) to cancel — and what remains is a simple supervised cross-entropy loss directly on the preference pairs, parameterized by the policy itself. No explicit RM, no PPO.
Before using the widget: if the policy assigns much higher probability to the winning response than the reference does (large positive log-ratio for y_w) and much lower probability to the losing response (large negative log-ratio for y_l), what do you expect the DPO loss to look like? What happens when the log-ratios are equal?
When log(pi_theta(y_w) / pi_ref(y_w)) is large and positive and log(pi_theta(y_l) / pi_ref(y_l)) is large and negative, the margin is large, sigmoid(beta * margin) approaches 1, and -log(sigmoid(...)) approaches 0 — very low loss. The policy has correctly pushed probability mass toward winners and away from losers. When both log-ratios are equal the margin is zero, sigmoid(0) = 0.5, and the loss is log(2) ≈ 0.693 — the model is maximally uncertain about the preference.
Interact with the widget: set log-ratio winner to 2.0 and log-ratio loser to -2.0, observe the loss drop near zero. Then drag both sliders to the same value — watch the loss jump to log(2). Increase beta — notice that the same margin now produces a steeper preference signal (smaller loss for large margins, but also a sharper cliff near zero).
Blue curve = L_DPO vs margin at current β · Red dot = current state · L_DPO = −log σ(β·(logr_w − logr_l))
After playing with the widget: why does a larger beta sharpen the preference signal? What is the trade-off at very large beta?
Beta scales the implicit reward gap: beta * margin. Larger beta amplifies differences between log-ratios, making the sigmoid more decisive for the same margin — large margins produce near-zero loss much sooner. However, very large beta means even a tiny deviation from the reference policy’s log-ratios triggers a large loss signal. The policy is forced to stay extremely close to pi_ref, limiting how much it can shift probability mass toward winners. This is the same KL-budget trade-off seen in PPO: high beta = conservative alignment, low beta = aggressive alignment with risk of reward hacking.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The derivation starts from the KL-regularized RL objective that InstructGPT’s PPO phase optimizes:
This objective has a closed-form optimal solution. The unique policy that maximizes it is:
where Z(x) is the partition function (normalizing constant). Inverting this gives the implicit reward as a function of the optimal policy:
Substituting into the Bradley-Terry preference loss with C(x) = beta * log Z(x):
The C(x) terms cancel because they appear identically in both the y_w and y_l terms of the preference difference. What remains depends only on log-ratios that are computable from pi_theta and the frozen pi_ref — no explicit reward model, no partition function evaluation.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the four steps of the DPO training loop in the correct order:
- Compute log-ratios: logr_w = log pi_theta(y_w|x) - log pi_ref(y_w|x); logr_l = log pi_theta(y_l|x) - log pi_ref(y_l|x)
- Sample a preference pair (x, y_w, y_l) from the offline preference dataset
- Backpropagate through L into pi_theta only (pi_ref is frozen)
- Form DPO loss: L = -log sigmoid(beta * (logr_w - logr_l))
Fill in the blank — the margin inside the DPO loss sigmoid:
def dpo_loss(logp_theta_w, logp_theta_l, logp_ref_w, logp_ref_l, beta):
logr_w = _______
logr_l = logp_theta_l - logp_ref_l
margin = beta * (logr_w - logr_l)
return -log(sigmoid(margin))Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the DPO loss in NumPy. Given arrays logp_theta_w, logp_theta_l, logp_ref_w, logp_ref_l (log-probabilities of winning and losing responses under the current policy and frozen reference, respectively) and a scalar beta, return the mean DPO loss over the batch.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §4 (Direct Preference Optimization — the full derivation from the KL-regularized RL objective to the DPO loss) → §4.1 (Deriving the DPO Objective — the key algebraic steps showing how the partition function cancels) → §5 (Experiments — comparison with PPO-based RLHF on sentiment control and summarization benchmarks).