① Intuition
Most reasoning models are trained with instruction-style supervised fine-tuning (SFT) followed by RLHF: first show the model thousands of worked examples, then use human preference signals to refine the outputs. DeepSeek-R1-Zero does something striking: skip SFT entirely and train with reinforcement learning from verifiable rewards. The reward signal is grounded — a correct-answer checker for math problems, unit tests for code — not a learned preference model that can be gamed. No human labels, no curated chain-of-thought demonstrations.
What happens when you train this way? Two things emerge that weren’t explicitly programmed. First, response length roughly doubles over the course of training: the model discovers on its own that longer reasoning chains give more opportunity to catch errors and backtrack on hard problems. Second, “aha moments” appear — places where the model writes something like “wait, I made an error earlier” and then corrects itself. Self-verification emerges from the reward signal alone, not from any imitation.
Before touching the widget: if you slide training step from 0 to 200, you predict accuracy will climb. But what do you predict happens to response length — does it stay flat, grow steadily, or plateau early? Why?
Response length grows roughly proportionally with training step and does not plateau early. Longer responses give the model more token budget to reason through sub-problems and to detect and fix errors. Because the verifiable reward only fires for final correctness, the model is under pressure to be right — and it learns that extended scratchpad space helps achieve that. The growth is roughly linear (not a sharp early jump) because the RL signal reinforces each incremental improvement in reasoning depth.
Interact with the widget: slide the training step from 0 to 200 and watch accuracy climb while average response length grows. Toggle “Show supervised baseline” to see the flat 55% ceiling of a model trained only on SFT without RL — pure RL-from-scratch eventually surpasses it. The amber “aha moments” counter tallies emergent self-correction events.
Blue curve = GRPO accuracy · Red dot = current step
After playing with the widget: the SFT baseline is flat at 55%. Why can't you just collect more SFT data to exceed that ceiling, rather than using RL?
SFT is bounded by the quality of the demonstration data. If the reference solutions don’t show backtracking and self-correction, the model won’t learn to do them. RL from verifiable rewards has no such ceiling: it can discover reasoning strategies that never appeared in any human-written chain-of-thought, as long as they produce correct final answers. The signal is outcome-based, not imitation-based, so the model can explore and exploit solution paths that humans never wrote down.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
DeepSeek-R1 trains with GRPO (Group Relative Policy Optimization), a PPO variant that avoids maintaining a separate critic network. Instead, for each prompt the algorithm samples a group of G responses, scores each with the verifiable reward, and computes a group-normalized advantage. The policy-gradient update then uses this advantage with a KL penalty to keep the new policy close to a frozen reference.
The composite reward for a single response is:
where r_accuracy = 1 if the final answer is verified correct (zero otherwise), and r_format = 1 if the response obeys the required tag structure <think>...</think><answer>...</answer> (zero otherwise). Both components are binary and verifiable — no learned proxy.
The group-normalized advantage for response i in a group of G responses sampled for the same prompt is:
where \mu_group = (1/G) \sum_{j=1}^G r_j and \sigma_group = \sqrt{(1/G)\sum_{j=1}^G (r_j - \mu_group)^2}. This replaces the critic’s value estimate entirely: the baseline is the empirical mean reward within the group.
The GRPO objective clips the ratio and adds a KL penalty to the reference policy \pi_ref:
where \rho_i = \pi_\theta(o_i | q) / \pi_{\text{ref}}(o_i | q) is the probability ratio and \beta controls the KL penalty strength.
Show next line
Show next line
Retrieval check
Answer all 1 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of one GRPO iteration in the correct order:
- Update pi_theta via gradient descent on the GRPO loss
- Score each response: r_i = r_accuracy(o_i, q) + r_format(o_i)
- Compute group-normalized advantage: A_hat_i = (r_i - mean(r_group)) / std(r_group)
- Compute clipped policy-gradient loss with KL penalty to reference policy pi_ref
- Sample a group of G responses o_1...o_G for prompt q from current policy pi_theta
Fill in the blank — the group-normalization line inside the GRPO advantage computation:
def grpo_advantage(rewards):
# rewards: list of G scalar rewards for same prompt
mu = sum(rewards) / len(rewards)
sigma = (sum((r - mu)**2 for r in rewards) / len(rewards)) ** 0.5
return [_______ for r in rewards]Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement group-normalized advantages in NumPy. Given rewards of shape (B, G) where B is batch size and G is group size, return a (B, G) array of advantages: subtract the per-row mean and divide by the per-row standard deviation plus epsilon. Each row is one group (all responses sampled for the same prompt).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 (Introduction — motivation and comparison to SFT-first approaches) → §2 (Approach / GRPO — algorithm details and reward design) → §2.2.2 (Self-Evolution Process — emergent length growth and aha-moment analysis).