← Roadmap

Proximal Policy Optimization Algorithms

Schulman, Wolski, Dhariwal, Radford, Klimov · 2017

① Intuition

Policy gradient methods (REINFORCE and its variants) update network parameters in the direction that makes high-reward actions more likely. The problem is step size: a single gradient update can move the policy so far from where the data was collected that the new policy behaves entirely differently — performance collapses, and you can never recover because the old data is no longer representative of the new policy. This is the “destructively large update” problem.

TRPO (Trust Region Policy Optimization, 2015) solved it by adding a hard KL-divergence constraint: the new policy may not be more than a fixed distance from the old one in probability space. The constraint ensures safe, monotonically improving updates. But enforcing it requires computing second-order derivatives (a conjugate-gradient solve on the Fisher information matrix) — expensive, tricky to implement, and hard to combine with architectures that share parameters between the policy and value networks.

PPO (Schulman et al., 2017) achieves the same trust-region effect with a single line of algebra. Define the probability ratio r_t = pi_new(a|s) / pi_old(a|s). If the policy improves an action that already has r > 1 + eps, the additional gradient contribution is simply zeroed out — the clip stops rewarding the policy for straying further. If the policy is making a bad action less likely but r has already dropped below 1 - eps, the same clipping applies. Result: a first-order optimizer (Adam) with trust-region-like safety.

Before using the widget: if A > 0 (action was better than expected) and r is already 1.5 with eps = 0.2, what does L_CLIP equal — and crucially, what is its gradient with respect to r?

Interact with the widget: drag the ratio r slider past the upper clip boundary 1 + eps while keeping A > 0. Watch the clipped objective flatline while the unclipped objective keeps growing. Then flip A negative and drag r below 1 - eps — observe the symmetric flatline on the other side.

unclipped
1.0000
r × A
clipped
1.0000
L_CLIP
clip range
[0.80, 1.20]
Regime: no clip
0.80
1.20

Green zone = clip range [1−ε, 1+ε] · Blue dot = current r · L_CLIP = min(r·A, clip(r, 1−ε, 1+ε)·A)

After exploring the widget: when A < 0 and r = 0.5 (eps = 0.2), the clip range is [0.8, 1.2]. The unclipped term is 0.5 * A and the clipped term is 0.8 * A. Since A < 0, which is larger (less negative) and which does min pick?

Retrieval check

Why does vanilla policy gradient suffer from destructively large updates?
Why is TRPO hard to use in practice compared to PPO?

Answer all 2 correctly to mark this stage mastered.

② Math

The vanilla policy gradient surrogate — sometimes called the CPI (conservative policy iteration) objective — is:

LCPI(θ)=E^t[πθ(at∣st)πθold(at∣st)A^t]=E^t[rt(θ) A^t]L^{\text{CPI}}(\theta) = \hat{\mathbb{E}}_t \left[ \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_\text{old}}(a_t | s_t)} \hat{A}_t \right] = \hat{\mathbb{E}}_t \left[ r_t(\theta)\, \hat{A}_t \right]

where r_t(theta) = pi_new(a|s) / pi_old(a|s) is the probability ratio and A_hat_t is the estimated advantage. Maximizing L_CPI without constraint would move the policy arbitrarily far for high-advantage actions.

PPO introduces the clipped surrogate:

LCLIP(θ)=E^t[min⁡ ⁣(rt(θ) A^t,  clip⁡(rt(θ), 1−ε, 1+ε) A^t)]L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \left[ \min\!\left( r_t(\theta)\,\hat{A}_t,\; \operatorname{clip}(r_t(\theta),\, 1-\varepsilon,\, 1+\varepsilon)\,\hat{A}_t \right) \right]

The typical value is eps = 0.2. The min of the two terms makes L_CLIP a lower bound on L_CPI (for positive advantages): the clipped version is always as small or smaller than the unclipped version when r exceeds the boundary.

If A > 0 and r just barely exceeds 1 + eps, which term does the min pick, and what does that imply for the gradient?
Show next line
∇θLCLIP=0 when r>1+ε and A>0  ⇒  trust-region effect without constraint\nabla_\theta L^{\text{CLIP}} = 0 \text{ when } r > 1+\varepsilon \text{ and } A > 0 \;\Rightarrow\; \text{trust-region effect without constraint}
If A < 0 and r just barely drops below 1 - eps, which term does the min pick?
Show next line
min⁡(rA^, (1−ε)A^)=(1−ε)A^ when r<1−ε,  A<0  ⇒  gradient zero\min(r\hat{A},\, (1{-}\varepsilon)\hat{A}) = (1{-}\varepsilon)\hat{A} \text{ when } r < 1-\varepsilon,\; A < 0 \;\Rightarrow\; \text{gradient zero}

Retrieval check

What is the probability ratio r_t in PPO, and what does r_t = 1 mean?
PPO typically performs multiple epochs of minibatch updates on each collected rollout. Why is this safe — isn't it exactly the kind of repeated reuse of old data that causes policy gradient to diverge?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of one PPO update iteration in the correct order:

  1. Compute advantages A_hat_t using GAE (Generalized Advantage Estimation) from rewards and value estimates
  2. Form L_CLIP = mean(min(ratio * A_hat, clip(ratio, 1-eps, 1+eps) * A_hat))
  3. Update theta via Adam on -L_CLIP (+ value loss + entropy bonus) for K epochs over minibatches
  4. Collect T timestep rollout with current policy pi_old; store (s_t, a_t, r_t, s_{t+1})
  5. Compute probability ratios: ratio = pi_theta(a_t|s_t) / pi_old(a_t|s_t)

Fill in the blank — the clipped term inside the min:

def ppo_loss(ratio, A_hat, eps=0.2):
    unclipped = ratio * A_hat
    clipped   = _______
    return -torch.mean(torch.min(unclipped, clipped))

Retrieval check

In the PPO training loop, pi_old is fixed for the entire rollout batch. What is pi_old, and when is it updated?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the clipped PPO surrogate objective in NumPy. Given arrays ratios (shape N) and advantages (shape N), and scalar eps = 0.2, return the scalar mean clipped objective (not negated — return the positive quantity to maximize).

Retrieval check

Why does the PPO code use np.minimum (element-wise) rather than np.min (global) when forming the clipped objective?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Proximal Policy Optimization Algorithms (Schulman et al. 2017).

Open in new tab · Download

Suggested reading order: §3 (Clipped Surrogate Objective — the core algorithm, two pages) → §5 (Experiments — Mujoco and Atari comparisons against TRPO, A2C, and ACER) → §6 (Conclusion).