← Roadmap

Training Language Models to Follow Instructions with Human Feedback

Ouyang et al. · 2022

① Intuition

GPT-3 was a remarkable achievement — 175 billion parameters trained on vast swaths of the internet. Yet when you prompted it, it would just as readily produce a toxic rant, a confident hallucination, or a completely off-topic response as a useful answer. The model had learned to predict text, not to be helpful. There was a fundamental mismatch between the training objective (next-token prediction on web data) and the actual goal (follow the user’s instruction helpfully and harmlessly).

InstructGPT (Ouyang et al., 2022) introduced the RLHF (Reinforcement Learning from Human Feedback) pipeline to bridge this gap. The three-phase recipe is: (1) Supervised Fine-Tuning (SFT) — fine-tune GPT-3 on a small dataset of human-written demonstrations of good responses; (2) Reward Model (RM) — train a separate model to score responses by learning from human pairwise preference judgments (which of two responses is better?); (3) PPO fine-tuning — use PPO to optimize the SFT model to maximize the RM’s score, with a KL-divergence penalty keeping it close to the SFT policy to prevent “reward hacking.”

The reward model’s core mechanism is the Bradley-Terry preference model: given two responses A and B with scalar rewards r(A) and r(B), the probability that a human prefers A over B is sigmoid(r(A) - r(B)). The larger the margin between the two rewards, the more confident the preference. This is exactly what the widget shows.

Before using the widget: if response A has reward 2.0 and response B has reward 0.0, what is P(A > B) according to the Bradley-Terry model? What happens to the RM log-loss for a label of 'A preferred'?

Interact with the widget: set reward A = 2.0, reward B = 0.0 and observe P(A > B) and the log-loss. Then gradually move reward B up toward reward A — watch the preference bar converge toward 50% and the loss climb toward log(2). Try raising the temperature T — this “softens” preferences, making even large margins less decisive.

margin (r_A − r_B)
1.00
p(A > B)
73.11%
AB
RM log-loss for labelled 'A preferred' = −log p(A>B)
0.3133

After exploring the widget: why would a large temperature T be problematic when training a reward model on human labels? What does it imply about the learned reward signal?

Retrieval check

Why can't you just use raw human preference rankings directly as a training signal for the policy, without training a reward model?
What problem does the KL penalty to the SFT policy solve in the PPO phase?

Answer all 2 correctly to mark this stage mastered.

② Math

The reward model is trained via maximum likelihood on the Bradley-Terry preference model. Given a prompt x, a preferred response y_w (winner) and a rejected response y_l (loser), the probability that a human picks y_w is:

P(yw≻yl∣x)=σ ⁣(rϕ(x,yw)−rϕ(x,yl))P(y_w \succ y_l \mid x) = \sigma\!\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)

where sigma is the sigmoid function and r_phi is the reward model with parameters phi. Training minimizes the negative log-likelihood over all collected preference pairs:

LRM(ϕ)=−E(x, yw, yl)∼D ⁣[log⁡σ ⁣(rϕ(x,yw)−rϕ(x,yl))]\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x,\,y_w,\,y_l)\sim\mathcal{D}}\!\left[\log \sigma\!\bigl(r_\phi(x, y_w) - r_\phi(x, y_l)\bigr)\right]

This is a pairwise log-loss: the model is rewarded for assigning a larger scalar reward to the preferred response and penalized when it cannot distinguish the pair.

The PPO objective adds a KL-divergence penalty against the SFT policy pi_SFT to prevent reward hacking:

LPPO(θ)=E(x,y)∼πθ ⁣[rϕ(x,y)−β KL ⁣(πθ(y∣x) ∥ πSFT(y∣x))]\mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_{(x,y)\sim\pi_\theta}\!\left[r_\phi(x,y) - \beta\,\mathrm{KL}\!\left(\pi_\theta(y\mid x)\,\|\,\pi_{\text{SFT}}(y\mid x)\right)\right]

where beta is a hyperparameter controlling how tightly the policy stays near pi_SFT. A larger beta means more conservative updates; beta = 0 removes the constraint entirely (risk of reward hacking).

Why does the RM loss use the difference r(x, y_w) - r(x, y_l) inside sigmoid rather than directly predicting a score for each response independently?
Show next line
LRM=−log⁡σ ⁣(r(yw)−r(yl))  ⇒  margin encodes relative preference, cancels prompt bias\mathcal{L}_{\text{RM}} = -\log\sigma\!\bigl(r(y_w) - r(y_l)\bigr) \;\Rightarrow\; \text{margin encodes relative preference, cancels prompt bias}
In the PPO objective `L_PPO = E[r(x,y) - beta * KL(pi || pi_SFT)]`, increasing beta makes the policy stay closer to pi_SFT. What is the trade-off as beta grows large?
Show next line
High β⇒πθ≈πSFT⇒reward gain≈0    ∣    Low β⇒reward hacking risk\text{High } \beta \Rightarrow \pi_\theta \approx \pi_{\text{SFT}} \Rightarrow \text{reward gain} \approx 0 \;\;|\;\; \text{Low } \beta \Rightarrow \text{reward hacking risk}

Retrieval check

In the PPO objective for InstructGPT, the KL term is KL(pi_theta || pi_SFT). What does this direction of KL mean, and why is it used instead of KL(pi_SFT || pi_theta)?
InstructGPT found that a 1.3B InstructGPT model was preferred by human raters over a 175B GPT-3 model. What does this imply about the relationship between model size and alignment?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of the RLHF pipeline in the correct order:

  1. Train reward model: minimize pairwise log-loss L_RM = -E[log sigmoid(r(y_w) - r(y_l))] on the preference dataset
  2. Collect preference data: for each prompt, generate K responses from pi_SFT, have humans rank them pairwise
  3. SFT: fine-tune GPT-3 on human-written (prompt, demonstration) pairs to get pi_SFT
  4. PPO fine-tuning: optimize pi_theta with PPO to maximize r(x,y) - beta * KL(pi_theta || pi_SFT)

Fill in the blank — the total reward signal used during PPO rollouts:

def total_reward(prompt, response, reward_model, sft_policy, current_policy, beta):
    r_rm = reward_model(prompt, response)
    kl   = _______
    return r_rm - beta * kl

Retrieval check

During PPO fine-tuning in InstructGPT, the reward model is frozen (not updated). Why?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the reward model pairwise log-loss in NumPy. Given 1-D arrays r_winners and r_losers (scalar reward values for preferred and rejected responses respectively), return the mean RM loss: mean(-log(sigmoid(r_w - r_l))).

Retrieval check

The RM loss formula `-log(sigmoid(r_w - r_l))` has the same mathematical form as binary cross-entropy. What is the 'label' and what is the 'predicted probability' in this correspondence?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Training Language Models to Follow Instructions with Human Feedback (Ouyang et al. 2022).

Open in new tab · Download

Suggested reading order: §2 (Methods — the full SFT + RM + PPO pipeline description) → §3.3 (Results: reward model evaluation) → §3.4 (Results: RL fine-tuning and human preference evaluations).