① Intuition
Policy gradient methods (REINFORCE and its variants) update network parameters in the direction that makes high-reward actions more likely. The problem is step size: a single gradient update can move the policy so far from where the data was collected that the new policy behaves entirely differently — performance collapses, and you can never recover because the old data is no longer representative of the new policy. This is the “destructively large update” problem.
TRPO (Trust Region Policy Optimization, 2015) solved it by adding a hard KL-divergence constraint: the new policy may not be more than a fixed distance from the old one in probability space. The constraint ensures safe, monotonically improving updates. But enforcing it requires computing second-order derivatives (a conjugate-gradient solve on the Fisher information matrix) — expensive, tricky to implement, and hard to combine with architectures that share parameters between the policy and value networks.
PPO (Schulman et al., 2017) achieves the same trust-region effect with a single line of algebra. Define the probability ratio r_t = pi_new(a|s) / pi_old(a|s). If the policy improves an action that already has r > 1 + eps, the additional gradient contribution is simply zeroed out — the clip stops rewarding the policy for straying further. If the policy is making a bad action less likely but r has already dropped below 1 - eps, the same clipping applies. Result: a first-order optimizer (Adam) with trust-region-like safety.
Before using the widget: if A > 0 (action was better than expected) and r is already 1.5 with eps = 0.2, what does L_CLIP equal — and crucially, what is its gradient with respect to r?
With eps = 0.2 the clip range is [0.8, 1.2]. r = 1.5 exceeds the upper bound, so the min selects the clipped term: clip(r, 0.8, 1.2) * A = 1.2 * A. This value is constant in r — its gradient with respect to r is zero. No matter how much further the policy moves, it receives no additional gradient signal for that action. The trust region is enforced without any constraint solver.
Interact with the widget: drag the ratio r slider past the upper clip boundary 1 + eps while keeping A > 0. Watch the clipped objective flatline while the unclipped objective keeps growing. Then flip A negative and drag r below 1 - eps — observe the symmetric flatline on the other side.
Green zone = clip range [1−ε, 1+ε] · Blue dot = current r · L_CLIP = min(r·A, clip(r, 1−ε, 1+ε)·A)
After exploring the widget: when A < 0 and r = 0.5 (eps = 0.2), the clip range is [0.8, 1.2]. The unclipped term is 0.5 * A and the clipped term is 0.8 * A. Since A < 0, which is larger (less negative) and which does min pick?
When A < 0, both terms are negative. 0.5 * A is more negative than 0.8 * A (because 0.5 < 0.8 and A < 0 flips the inequality). So min(0.5*A, 0.8*A) = 0.5 * A — the unclipped term. The policy is not penalized for reducing a bad action’s probability below the trust region because reducing bad-action probability is beneficial. Clipping is asymmetric: it only removes the incentive to go further when doing so cannot be justified by current data.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The vanilla policy gradient surrogate — sometimes called the CPI (conservative policy iteration) objective — is:
where r_t(theta) = pi_new(a|s) / pi_old(a|s) is the probability ratio and A_hat_t is the estimated advantage. Maximizing L_CPI without constraint would move the policy arbitrarily far for high-advantage actions.
PPO introduces the clipped surrogate:
The typical value is eps = 0.2. The min of the two terms makes L_CLIP a lower bound on L_CPI (for positive advantages): the clipped version is always as small or smaller than the unclipped version when r exceeds the boundary.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of one PPO update iteration in the correct order:
- Compute advantages A_hat_t using GAE (Generalized Advantage Estimation) from rewards and value estimates
- Form L_CLIP = mean(min(ratio * A_hat, clip(ratio, 1-eps, 1+eps) * A_hat))
- Update theta via Adam on -L_CLIP (+ value loss + entropy bonus) for K epochs over minibatches
- Collect T timestep rollout with current policy pi_old; store (s_t, a_t, r_t, s_{t+1})
- Compute probability ratios: ratio = pi_theta(a_t|s_t) / pi_old(a_t|s_t)
Fill in the blank — the clipped term inside the min:
def ppo_loss(ratio, A_hat, eps=0.2):
unclipped = ratio * A_hat
clipped = _______
return -torch.mean(torch.min(unclipped, clipped))Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the clipped PPO surrogate objective in NumPy. Given arrays ratios (shape N) and advantages (shape N), and scalar eps = 0.2, return the scalar mean clipped objective (not negated — return the positive quantity to maximize).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Clipped Surrogate Objective — the core algorithm, two pages) → §5 (Experiments — Mujoco and Atari comparisons against TRPO, A2C, and ACER) → §6 (Conclusion).