← Roadmap

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov, Sharma, Mitchell, Manning, Ermon, Finn · 2023

① Intuition

InstructGPT showed that RLHF can align language models to human preferences — but it requires a multi-stage pipeline: first train a reward model (RM) on preference pairs, then run PPO to optimize the policy against the RM while keeping it close to the reference policy via a KL-divergence penalty. That’s two separate models, two separate training loops, and a dozen sensitive hyperparameters. If any part is misconfigured, the policy either reward-hacks the RM or barely moves from the SFT baseline.

DPO (Rafailov et al., 2023) asks a deceptively simple question: is the reward model even necessary? The key insight is that the KL-regularized RL objective has a closed-form optimal policy: pi*(y|x) ∝ pi_ref(y|x) exp(r(x,y) / beta). Inverting that relationship lets you express the reward implicitly as r(x,y) = beta * log(pi*(y|x) / pi_ref(y|x)) + C(x). Plugging this implicit reward back into the Bradley-Terry preference loss causes the partition function C(x) to cancel — and what remains is a simple supervised cross-entropy loss directly on the preference pairs, parameterized by the policy itself. No explicit RM, no PPO.

Before using the widget: if the policy assigns much higher probability to the winning response than the reference does (large positive log-ratio for y_w) and much lower probability to the losing response (large negative log-ratio for y_l), what do you expect the DPO loss to look like? What happens when the log-ratios are equal?

Interact with the widget: set log-ratio winner to 2.0 and log-ratio loser to -2.0, observe the loss drop near zero. Then drag both sliders to the same value — watch the loss jump to log(2). Increase beta — notice that the same margin now produces a steeper preference signal (smaller loss for large margins, but also a sharper cliff near zero).

margin
2.0000
logr_w − logr_l
loss
0.3133
L_DPO
reward gap
1.0000
β × margin

Blue curve = L_DPO vs margin at current β · Red dot = current state · L_DPO = −log σ(β·(logr_w − logr_l))

After playing with the widget: why does a larger beta sharpen the preference signal? What is the trade-off at very large beta?

Retrieval check

DPO and InstructGPT both optimize for human preferences. What is the key structural difference in how they do it?
Why does the partition function C(x) cancel in the DPO derivation?

Answer all 2 correctly to mark this stage mastered.

② Math

The derivation starts from the KL-regularized RL objective that InstructGPT’s PPO phase optimizes:

max⁡πθ  Ex∼D, y∼πθ(y∣x) ⁣[r(x,y)−β KL ⁣(πθ(y∣x) ∥ πref(y∣x))]\max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(y|x)}\!\left[r(x,y) - \beta\,\mathrm{KL}\!\left(\pi_\theta(y\mid x)\,\|\,\pi_{\text{ref}}(y\mid x)\right)\right]

This objective has a closed-form optimal solution. The unique policy that maximizes it is:

π∗(y∣x)=1Z(x) πref(y∣x)exp⁡ ⁣(r(x,y)β)\pi^*(y \mid x) = \frac{1}{Z(x)}\,\pi_{\text{ref}}(y \mid x)\exp\!\left(\frac{r(x,y)}{\beta}\right)

where Z(x) is the partition function (normalizing constant). Inverting this gives the implicit reward as a function of the optimal policy:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)r(x,y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

Substituting into the Bradley-Terry preference loss with C(x) = beta * log Z(x):

LDPO(θ)=−E(x,yw,yl)∼D ⁣[log⁡σ ⁣(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]

The C(x) terms cancel because they appear identically in both the y_w and y_l terms of the preference difference. What remains depends only on log-ratios that are computable from pi_theta and the frozen pi_ref — no explicit reward model, no partition function evaluation.

Why does the partition function C(x) = beta * log Z(x) cancel when we substitute the implicit reward into the Bradley-Terry loss?
Show next line
r(x,yw)−r(x,yl)=βlog⁡π∗(yw∣x)πref(yw∣x)−βlog⁡π∗(yl∣x)πref(yl∣x)(C(x) cancels)r(x,y_w) - r(x,y_l) = \beta\log\frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \quad \text{(C(x) cancels)}
What is the role of beta in DPO, and what happens to the loss landscape as beta grows large?
Show next line
β↑  ⇒  KL budget shrinks  ⇒  πθ≈πref  ⇒  alignment gain≈0\beta \uparrow \;\Rightarrow\; \text{KL budget shrinks} \;\Rightarrow\; \pi_\theta \approx \pi_{\text{ref}} \;\Rightarrow\; \text{alignment gain} \approx 0

Retrieval check

DPO optimizes the same underlying objective as PPO-based RLHF. Why then might their trained policies differ in practice?
During DPO training, the reference policy pi_ref is frozen. What would happen if pi_ref were updated jointly with pi_theta?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of the DPO training loop in the correct order:

  1. Compute log-ratios: logr_w = log pi_theta(y_w|x) - log pi_ref(y_w|x); logr_l = log pi_theta(y_l|x) - log pi_ref(y_l|x)
  2. Sample a preference pair (x, y_w, y_l) from the offline preference dataset
  3. Backpropagate through L into pi_theta only (pi_ref is frozen)
  4. Form DPO loss: L = -log sigmoid(beta * (logr_w - logr_l))

Fill in the blank — the margin inside the DPO loss sigmoid:

def dpo_loss(logp_theta_w, logp_theta_l, logp_ref_w, logp_ref_l, beta):
    logr_w = _______
    logr_l = logp_theta_l - logp_ref_l
    margin = beta * (logr_w - logr_l)
    return -log(sigmoid(margin))

Retrieval check

In the DPO training loop, why do we need to run a forward pass through the frozen pi_ref for every batch, rather than precomputing all reference log-probabilities once before training?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the DPO loss in NumPy. Given arrays logp_theta_w, logp_theta_l, logp_ref_w, logp_ref_l (log-probabilities of winning and losing responses under the current policy and frozen reference, respectively) and a scalar beta, return the mean DPO loss over the batch.

Retrieval check

In the NumPy implementation, the log-ratios logr_w = logp_theta_w - logp_ref_w represent what quantity geometrically?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al. 2023).

Open in new tab · Download

Suggested reading order: §4 (Direct Preference Optimization — the full derivation from the KL-regularized RL objective to the DPO loss) → §4.1 (Deriving the DPO Objective — the key algebraic steps showing how the partition function cancels) → §5 (Experiments — comparison with PPO-based RLHF on sentiment control and summarization benchmarks).