← Roadmap

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI · 2025

① Intuition

Most reasoning models are trained with instruction-style supervised fine-tuning (SFT) followed by RLHF: first show the model thousands of worked examples, then use human preference signals to refine the outputs. DeepSeek-R1-Zero does something striking: skip SFT entirely and train with reinforcement learning from verifiable rewards. The reward signal is grounded — a correct-answer checker for math problems, unit tests for code — not a learned preference model that can be gamed. No human labels, no curated chain-of-thought demonstrations.

What happens when you train this way? Two things emerge that weren’t explicitly programmed. First, response length roughly doubles over the course of training: the model discovers on its own that longer reasoning chains give more opportunity to catch errors and backtrack on hard problems. Second, “aha moments” appear — places where the model writes something like “wait, I made an error earlier” and then corrects itself. Self-verification emerges from the reward signal alone, not from any imitation.

Before touching the widget: if you slide training step from 0 to 200, you predict accuracy will climb. But what do you predict happens to response length — does it stay flat, grow steadily, or plateau early? Why?

Interact with the widget: slide the training step from 0 to 200 and watch accuracy climb while average response length grows. Toggle “Show supervised baseline” to see the flat 55% ceiling of a model trained only on SFT without RL — pure RL-from-scratch eventually surpasses it. The amber “aha moments” counter tallies emergent self-correction events.

accuracy
21.0%
avg length
400 tok
aha moments
0

Blue curve = GRPO accuracy · Red dot = current step

After playing with the widget: the SFT baseline is flat at 55%. Why can't you just collect more SFT data to exceed that ceiling, rather than using RL?

Retrieval check

Why does using a verifiable reward (e.g., correct-answer checker) prevent reward hacking, while a learned reward model does not?
DeepSeek-R1 adds a small SFT warm-up before RL, on top of R1-Zero's pure-RL approach. What problem does this warm-up solve?

Answer all 2 correctly to mark this stage mastered.

② Math

DeepSeek-R1 trains with GRPO (Group Relative Policy Optimization), a PPO variant that avoids maintaining a separate critic network. Instead, for each prompt the algorithm samples a group of G responses, scores each with the verifiable reward, and computes a group-normalized advantage. The policy-gradient update then uses this advantage with a KL penalty to keep the new policy close to a frozen reference.

The composite reward for a single response is:

r=raccuracy+rformatr = r_{\text{accuracy}} + r_{\text{format}}

where r_accuracy = 1 if the final answer is verified correct (zero otherwise), and r_format = 1 if the response obeys the required tag structure <think>...</think><answer>...</answer> (zero otherwise). Both components are binary and verifiable — no learned proxy.

The group-normalized advantage for response i in a group of G responses sampled for the same prompt is:

A^i=ri−μgroupσgroup\hat{A}_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}}}

where \mu_group = (1/G) \sum_{j=1}^G r_j and \sigma_group = \sqrt{(1/G)\sum_{j=1}^G (r_j - \mu_group)^2}. This replaces the critic’s value estimate entirely: the baseline is the empirical mean reward within the group.

The GRPO objective clips the ratio and adds a KL penalty to the reference policy \pi_ref:

LGRPO=−E ⁣[∑i=1Gmin⁡ ⁣(ρiA^i,  clip(ρi,1−ε,1+ε)A^i)]+β DKL(πθ∥πref)\mathcal{L}_{\text{GRPO}} = -\mathbb{E}\!\left[\sum_{i=1}^G \min\!\left(\rho_i \hat{A}_i,\; \text{clip}(\rho_i, 1-\varepsilon, 1+\varepsilon)\hat{A}_i\right)\right] + \beta\, D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})

where \rho_i = \pi_\theta(o_i | q) / \pi_{\text{ref}}(o_i | q) is the probability ratio and \beta controls the KL penalty strength.

Why is verifiable reward critical — what goes wrong if you use a learned reward model (RLHF-style) instead?
Show next line
raccuracy∈{0,1}:grounded, no proxy to hackr_{\text{accuracy}} \in \{0,1\}: \text{grounded, no proxy to hack}
Why does response length grow during training? The reward only checks final correctness — it doesn't explicitly reward longer responses.
Show next line
longer chain⇒more self-check⇒higher raccuracy\text{longer chain} \Rightarrow \text{more self-check} \Rightarrow \text{higher } r_{\text{accuracy}}

Retrieval check

GRPO samples G responses for the same prompt and normalizes advantages within the group. What would happen if G=1 (only one response per prompt)?

Answer all 1 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of one GRPO iteration in the correct order:

  1. Update pi_theta via gradient descent on the GRPO loss
  2. Score each response: r_i = r_accuracy(o_i, q) + r_format(o_i)
  3. Compute group-normalized advantage: A_hat_i = (r_i - mean(r_group)) / std(r_group)
  4. Compute clipped policy-gradient loss with KL penalty to reference policy pi_ref
  5. Sample a group of G responses o_1...o_G for prompt q from current policy pi_theta

Fill in the blank — the group-normalization line inside the GRPO advantage computation:

def grpo_advantage(rewards):
    # rewards: list of G scalar rewards for same prompt
    mu = sum(rewards) / len(rewards)
    sigma = (sum((r - mu)**2 for r in rewards) / len(rewards)) ** 0.5
    return [_______  for r in rewards]

Retrieval check

In the GRPO loop, the KL penalty uses a frozen reference policy pi_ref. What is pi_ref and why is it frozen?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement group-normalized advantages in NumPy. Given rewards of shape (B, G) where B is batch size and G is group size, return a (B, G) array of advantages: subtract the per-row mean and divide by the per-row standard deviation plus epsilon. Each row is one group (all responses sampled for the same prompt).

Retrieval check

In the group-normalized advantages computation, what is the purpose of adding eps (1e-8) to the standard deviation before dividing?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025).

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation and comparison to SFT-first approaches) → §2 (Approach / GRPO — algorithm details and reward design) → §2.2.2 (Self-Evolution Process — emergent length growth and aha-moment analysis).