Proximal Policy Optimization

Teaching a machine to act — to play chess, walk a robot, or phrase a sentence — is at heart a trial-and-error process. The agent tries something, receives a reward, and nudges its behavior to do more of what worked. This is reinforcement learning, and the core idea is nearly as old as the field itself.

What is not easy is how much to nudge. Update the policy too timidly and learning crawls; update it too aggressively and the agent overshoots, forgets everything it knew, and collapses. For years this instability was the main obstacle to practical RL.

Proximal Policy Optimization — PPO, introduced by John Schulman and colleagues at OpenAI in 2017 — solved the problem with a single elegant constraint: never let the new policy stray too far from the old one. The constraint is enforced not by complex second-order math but by a clipped ratio: if the policy changes too much in any direction, the gradient simply stops flowing. The result is an algorithm that is stable, scalable, and astonishingly easy to implement — which is why PPO became the default RL algorithm for everything from neural-network training to RLHF fine-tuning of large language models.

The classic test for any RL algorithm is CartPole: balance a pole on a moving cart by pushing left or right. The agent receives +1 each time step the pole stays upright and the episode ends when it falls.

<!-- {{c_html_intro}} -->
<div class="ppo-wrap">
  <div class="top-row">
    <div class="canvas-box">
      <canvas id="cart-canvas" width="340" height="160" aria-label="{{aria_canvas}}"></canvas>
    </div>
    <div class="stats-box">
      <div class="stat-label">{{label_step}}</div>
      <div class="stat-val" id="step-val">0</div>
      <div class="stat-label">{{label_reward}}</div>
      <div class="stat-val" id="reward-val">0</div>
      <div class="stat-label">{{label_epsilon}}</div>
      <div class="stat-val" id="eps-val">0.20</div>
    </div>
  </div>
  <div class="ratio-section">
    <div class="ratio-label">{{label_ratio}}: <span id="ratio-val">1.00</span></div>
    <div class="bar-track">
      <div class="clip-zone" id="clip-zone"></div>
      <div class="ratio-bar" id="ratio-bar"></div>
      <div class="center-line"></div>
      <div class="clip-left" id="clip-left-line"></div>
      <div class="clip-right" id="clip-right-line"></div>
    </div>
    <div class="bar-axis">
      <span>0</span><span id="lbl-lo"></span><span>1</span><span id="lbl-hi"></span><span>2</span>
    </div>
  </div>
  <div class="clip-row">
    <span class="clip-tag" id="clip-indicator">{{msg_within}}</span>
    <span class="adv-tag">{{label_advantage}}: <span id="adv-val">+0.00</span></span>
  </div>
  <div class="btns">
    <button id="btn-step" type="button">{{btn_step}}</button>
    <button id="btn-run" type="button">{{btn_run}}</button>
    <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
  </div>
  <div id="msg" class="msg"></div>
</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: 14px; }
.ppo-wrap { max-width: 480px; margin: 0 auto; padding: .6rem; }
.top-row { display: flex; gap: .6rem; align-items: flex-start; margin-bottom: .5rem; }
.canvas-box { flex: 1; }
canvas { background: #f0f4f8; border-radius: 8px; display: block; width: 100%; height: auto; }
.stats-box { display: grid; grid-template-columns: auto auto; gap: .15rem .5rem;
             align-content: start; font-size: .82rem; }
.stat-label { color: #667; font-weight: 500; }
.stat-val { font: 700 1rem ui-monospace, monospace; color: #1d3557; }
.ratio-section { margin: .4rem 0; }
.ratio-label { font-size: .85rem; font-weight: 600; margin-bottom: .25rem; }
.bar-track { position: relative; height: 26px; background: #e4e8ed;
             border-radius: 6px; overflow: hidden; }
.clip-zone { position: absolute; top: 0; height: 100%; background: #c8e6c9; opacity: .7; }
.ratio-bar { position: absolute; top: 4px; height: 18px; border-radius: 4px;
             background: #1d3557; transition: left .15s, width .15s; }
.center-line { position: absolute; top: 0; width: 2px; background: #888; height: 100%; }
.clip-left, .clip-right { position: absolute; top: 0; width: 2px; background: #e63946;
                           height: 100%; }
.bar-axis { display: flex; justify-content: space-between; font-size: .72rem; color: #555;
            margin-top: .15rem; padding: 0 0; }
.clip-row { display: flex; align-items: center; gap: .7rem; margin: .3rem 0 .5rem; }
.clip-tag { font-size: .8rem; font-weight: 600; padding: .15rem .45rem;
            border-radius: 4px; background: #c8e6c9; color: #1b5e20; }
.clip-tag.clipped { background: #ffcdd2; color: #b71c1c; }
.adv-tag { font-size: .8rem; color: #555; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .4; cursor: default; }
.msg { font-size: .85rem; font-weight: 600; margin-top: .4rem; min-height: 1.2em; color: #0a7d33; }
.msg.bad { color: #c92f3c; }

// Code not found

The demo shows a simplified PPO training loop. The bar chart tracks the policy ratio $r(\theta) = \pi_\theta(a|s) / \pi_{\theta_\text{old}}(a|s)$ — how much the new policy differs from the old one on a sampled action. The clipped objective caps the ratio in $[1-\varepsilon,\, 1+\varepsilon]$ (here $\varepsilon = 0.2$), so a gradient that would push the ratio outside that band is silently blocked. Click Step to advance one training step and watch how the clipping engages whenever an update overshoots.

The core of PPO is the clipped surrogate objective. Let $r_t(\theta)$ be the probability ratio:

$$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}$$

The naive policy gradient just maximizes $r_t(\theta) \cdot \hat{A}_t$, where $\hat{A}_t$ is an advantage estimate (roughly: how much better this action was than expected). The problem is that $r_t(\theta)$ can become arbitrarily large, causing a catastrophic update.

PPO instead maximizes:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \bigl[\min\!\bigl(r_t(\theta)\,\hat{A}_t,\; \text{clip}(r_t(\theta),\, 1-\varepsilon,\, 1+\varepsilon)\,\hat{A}_t\bigr)\bigr]$$

The $\min$ makes this a pessimistic bound: when the advantage is positive, capping $r_t$ at $1+\varepsilon$ prevents over-exploitation; when negative, capping at $1-\varepsilon$ prevents over-punishment. The gradient is zero outside $[1-\varepsilon, 1+\varepsilon]$, so the update is automatically blocked the moment it would overshoot.

Why this matters for complexity:

• The predecessor TRPO enforced a trust-region constraint via a costly conjugate-gradient solve on every update.
• PPO replaces that with a single $\min$ and $\text{clip}$ — first-order only, the same cost as a plain gradient step.
• The resulting algorithm is $O(T \cdot |\pi|)$ per epoch (where $T$ is the trajectory length and $|\pi|$ is the parameter count), just like stochastic gradient descent on a supervised problem.

PPO does not come with a proof of global convergence — reinforcement learning objectives are non-convex and this is an open problem (related to broader questions in non-convex optimization). What it offers in practice is monotone empirical improvement: updates that reliably make things better rather than occasionally catastrophic.

PPO's combination of simplicity and stability made it the go-to algorithm across domains:

• Robotics locomotion: DeepMind and OpenAI used PPO to train simulated and physical robots to walk, run, and recover from falls — tasks where a single bad update could break months of training.
• Video games: OpenAI Five used PPO to defeat professional Dota 2 players in 2019. The same architecture powered agents in StarCraft II and many Atari games.
• RLHF (Reinforcement Learning from Human Feedback): the fine-tuning stage that turns a language model's raw predictions into helpful, harmless responses (used in InstructGPT, ChatGPT, Claude, and Gemini) is a PPO loop where the reward comes from a human-preference model. The clipping is essential — without it, the policy would quickly collapse to reward-hacking.
• Chip design: Google used PPO-based RL to place components on silicon chips, outperforming human expert layouts in some benchmarks.

The pattern is always the same: a problem where the reward is delayed, the state space is large, and a single update can destroy fragile learned behavior. PPO's clipping makes those updates safe.

Reinforcement learning had all the right ingredients — exploration, reward signals, gradient updates — but kept tripping over its own steps. PPO fixed that with the simplest possible constraint: clip the policy ratio and block any gradient that would take the policy too far from where it already is.

The result is an algorithm that any practitioner can run on a laptop, that scales to billion-parameter models, and that sits at the core of the systems training today's most capable AI. The next time you interact with a language model that actually follows instructions, there is a good chance a PPO loop shaped its behavior — one clipped gradient at a time.