Introduction

Teaching a machine to act — to play chess, walk a robot, or phrase a sentence — is at heart a trial-and-error process. The agent tries something, receives a reward, and nudges its behavior to do more of what worked. This is reinforcement learning, and the core idea is nearly as old as the field itself.

What is not easy is how much to nudge. Update the policy too timidly and learning crawls; update it too aggressively and the agent overshoots, forgets everything it knew, and collapses. For years this instability was the main obstacle to practical RL.

Proximal Policy Optimization — PPO, introduced by John Schulman and colleagues at OpenAI in 2017 — solved the problem with a single elegant constraint: never let the new policy stray too far from the old one. The constraint is enforced not by complex second-order math but by a clipped ratio: if the policy changes too much in any direction, the gradient simply stops flowing. The result is an algorithm that is stable, scalable, and astonishingly easy to implement — which is why PPO became the default RL algorithm for everything from neural-network training to RLHF fine-tuning of large language models.

Try It

The classic test for any RL algorithm is CartPole: balance a pole on a moving cart by pushing left or right. The agent receives +1 each time step the pole stays upright and the episode ends when it falls.

<!-- PPO CartPole demo — pole-balancing with clipped policy updates -->
<div class="ppo-wrap">
  <div class="top-row">
    <div class="canvas-box">
      <canvas id="cart-canvas" width="340" height="160" aria-label="CartPole physics simulation canvas"></canvas>
    </div>
    <div class="stats-box">
      <div class="stat-label">Step</div>
      <div class="stat-val" id="step-val">0</div>
      <div class="stat-label">Total Reward</div>
      <div class="stat-val" id="reward-val">0</div>
      <div class="stat-label">Clip ε</div>
      <div class="stat-val" id="eps-val">0.20</div>
    </div>
  </div>
  <div class="ratio-section">
    <div class="ratio-label">Policy ratio r(θ): <span id="ratio-val">1.00</span></div>
    <div class="bar-track">
      <div class="clip-zone" id="clip-zone"></div>
      <div class="ratio-bar" id="ratio-bar"></div>
      <div class="center-line"></div>
      <div class="clip-left" id="clip-left-line"></div>
      <div class="clip-right" id="clip-right-line"></div>
    </div>
    <div class="bar-axis">
      <span>0</span><span id="lbl-lo"></span><span>1</span><span id="lbl-hi"></span><span>2</span>
    </div>
  </div>
  <div class="clip-row">
    <span class="clip-tag" id="clip-indicator">Within clip range</span>
    <span class="adv-tag">Advantage: <span id="adv-val">+0.00</span></span>
  </div>
  <div class="btns">
    <button id="btn-step" type="button">Step</button>
    <button id="btn-run" type="button">Run</button>
    <button id="btn-reset" type="button" class="ghost">Reset</button>
  </div>
  <div id="msg" class="msg"></div>
</div>

/* CartPole PPO demo styles */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: 14px; }
.ppo-wrap { max-width: 480px; margin: 0 auto; padding: .6rem; }
.top-row { display: flex; gap: .6rem; align-items: flex-start; margin-bottom: .5rem; }
.canvas-box { flex: 1; }
canvas { background: #f0f4f8; border-radius: 8px; display: block; width: 100%; height: auto; }
.stats-box { display: grid; grid-template-columns: auto auto; gap: .15rem .5rem;
             align-content: start; font-size: .82rem; }
.stat-label { color: #667; font-weight: 500; }
.stat-val { font: 700 1rem ui-monospace, monospace; color: #1d3557; }
.ratio-section { margin: .4rem 0; }
.ratio-label { font-size: .85rem; font-weight: 600; margin-bottom: .25rem; }
.bar-track { position: relative; height: 26px; background: #e4e8ed;
             border-radius: 6px; overflow: hidden; }
.clip-zone { position: absolute; top: 0; height: 100%; background: #c8e6c9; opacity: .7; }
.ratio-bar { position: absolute; top: 4px; height: 18px; border-radius: 4px;
             background: #1d3557; transition: left .15s, width .15s; }
.center-line { position: absolute; top: 0; width: 2px; background: #888; height: 100%; }
.clip-left, .clip-right { position: absolute; top: 0; width: 2px; background: #e63946;
                           height: 100%; }
.bar-axis { display: flex; justify-content: space-between; font-size: .72rem; color: #555;
            margin-top: .15rem; padding: 0 0; }
.clip-row { display: flex; align-items: center; gap: .7rem; margin: .3rem 0 .5rem; }
.clip-tag { font-size: .8rem; font-weight: 600; padding: .15rem .45rem;
            border-radius: 4px; background: #c8e6c9; color: #1b5e20; }
.clip-tag.clipped { background: #ffcdd2; color: #b71c1c; }
.adv-tag { font-size: .8rem; color: #555; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .4; cursor: default; }
.msg { font-size: .85rem; font-weight: 600; margin-top: .4rem; min-height: 1.2em; color: #0a7d33; }
.msg.bad { color: #c92f3c; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// PPO CartPole — simplified clipped policy gradient demo

// CartPole physics constants
const DT = 0.02;
const GRAVITY = 9.8;
const MASS_CART = 1.0;
const MASS_POLE = 0.1;
const POLE_LEN = 0.5;
const FORCE_MAG = 10.0;
const EPS = 0.2; // clip parameter

function cartpoleStep(s, action) {
  // Euler integration of CartPole dynamics
  const [x, xDot, theta, thetaDot] = s;
  const force = (action === 1 ? 1 : -1) * FORCE_MAG;
  const cosT = Math.cos(theta), sinT = Math.sin(theta);
  const total_mass = MASS_CART + MASS_POLE;
  const polemass_length = MASS_POLE * POLE_LEN;
  const temp = (force + polemass_length * thetaDot * thetaDot * sinT) / total_mass;
  const thetaAcc = (GRAVITY * sinT - cosT * temp) /
    (POLE_LEN * (4.0 / 3.0 - MASS_POLE * cosT * cosT / total_mass));
  const xAcc = temp - polemass_length * thetaAcc * cosT / total_mass;
  const newX = x + DT * xDot;
  const newXDot = xDot + DT * xAcc;
  const newTheta = theta + DT * thetaDot;
  const newThetaDot = thetaDot + DT * thetaAcc;
  const done = Math.abs(newX) > 2.4 || Math.abs(newTheta) > 0.2094;
  return { s: [newX, newXDot, newTheta, newThetaDot], reward: done ? 0 : 1, done };
}

// Training state
let state, step, totalReward, running, runTimer;
// log-odds for action=1 (push right)
let policyLogit = 0.0;

function softmax1(logit) {
  // sigmoid — probability of action=1
  const e = Math.exp(logit);
  return e / (1 + e);
}
function sampleAction(logit) {
  return Math.random() < softmax1(logit) ? 1 : 0;
}

function resetSim() {
  // Reset everything to initial state
  state = [(Math.random() - 0.5) * 0.1, 0, (Math.random() - 0.5) * 0.1, 0];
  step = 0; totalReward = 0;
  policyLogit = 0.0;
  running = false;
  if (runTimer) clearInterval(runTimer);
  document.getElementById('btn-run').textContent = 'Run';
  document.getElementById('btn-step').disabled = false;
  document.getElementById('msg').textContent = '';
  document.getElementById('msg').className = 'msg';
  updateUI(null, null, false);
  draw();
}

function trainStep() {
  // One PPO-style training step
  const action = sampleAction(policyLogit);
  const oldProb = softmax1(policyLogit);

  const res = cartpoleStep(state, action);
  const advantage = res.reward - 0.5; // simple baseline

  // Compute policy gradient update
  const lr = 0.3;
  const grad = action === 1 ? advantage : -advantage;
  const rawUpdate = lr * grad;
  policyLogit += rawUpdate;

  // Compute probability ratio for the taken action
  const newProb = softmax1(policyLogit);
  const ratio = action === 1
    ? newProb / oldProb
    : (1 - newProb) / (1 - oldProb);

  const clipped = Math.min(Math.max(ratio, 1 - EPS), 1 + EPS);
  const isClipped = Math.abs(ratio - clipped) > 0.001;

  if (isClipped) {
    // Update was clipped — partially revert to stay in range
    policyLogit -= rawUpdate;
    policyLogit += rawUpdate * (clipped / ratio);
  }

  totalReward += res.reward;
  step++;
  state = res.done ? [(Math.random()-0.5)*0.1, 0, (Math.random()-0.5)*0.1, 0] : res.s;
  updateUI(ratio, advantage, isClipped);
  draw();

  if (res.done && step < 5) {
    document.getElementById('msg').textContent = i18nFmt('Pole fell at step %step%. Keep training!', { step: step });
    document.getElementById('msg').className = 'msg bad';
  } else if (step >= 200) {
    document.getElementById('msg').textContent = i18nFmt('Solved! %reward% reward in 200 steps.', { reward: totalReward });
    document.getElementById('msg').className = 'msg';
    if (running) toggleRun();
    document.getElementById('btn-step').disabled = true;
  }
}

function updateUI(ratio, adv, isClipped) {
  document.getElementById('step-val').textContent = step;
  document.getElementById('reward-val').textContent = totalReward;
  document.getElementById('ratio-val').textContent = ratio !== null ? ratio.toFixed(3) : '1.000';
  document.getElementById('adv-val').textContent = adv !== null
    ? (adv >= 0 ? '+' : '') + adv.toFixed(3) : '+0.000';

  // Update ratio bar visualization
  const track = document.querySelector('.bar-track');
  const W = track.offsetWidth || 340;
  const toX = (r) => (r / 2) * W;
  const lo = 1 - EPS, hi = 1 + EPS;
  const clZ = document.getElementById('clip-zone');
  clZ.style.left = toX(lo) + 'px';
  clZ.style.width = (toX(hi) - toX(lo)) + 'px';
  document.getElementById('clip-left-line').style.left = toX(lo) + 'px';
  document.getElementById('clip-right-line').style.left = toX(hi) + 'px';
  document.getElementById('clip-left-line').style.display = 'block';
  document.getElementById('clip-right-line').style.display = 'block';
  document.querySelector('.center-line').style.left = toX(1) + 'px';
  document.getElementById('lbl-lo').textContent = lo.toFixed(1);
  document.getElementById('lbl-hi').textContent = hi.toFixed(1);

  if (ratio !== null) {
    const barX = toX(Math.min(Math.max(ratio, 0), 2));
    const barW = Math.abs(barX - toX(1));
    const barLeft = Math.min(barX, toX(1));
    document.getElementById('ratio-bar').style.left = barLeft + 'px';
    document.getElementById('ratio-bar').style.width = barW + 'px';
  }

  const ind = document.getElementById('clip-indicator');
  if (isClipped) {
    ind.textContent = 'CLIPPED';
    ind.className = 'clip-tag clipped';
  } else {
    ind.textContent = 'Within clip range';
    ind.className = 'clip-tag';
  }
}

// Draw the CartPole simulation
function draw() {
  const cv = document.getElementById('cart-canvas');
  const ctx = cv.getContext('2d');
  const W = cv.width, H = cv.height;
  ctx.clearRect(0, 0, W, H);

  const [x, , theta] = state;
  const scale = W / 4.8;
  const cx = W / 2 + x * scale;
  const cy = H * 0.72;

  // draw ground track
  ctx.strokeStyle = '#aab'; ctx.lineWidth = 2;
  ctx.beginPath(); ctx.moveTo(20, cy + 18); ctx.lineTo(W - 20, cy + 18); ctx.stroke();

  // draw cart body
  ctx.fillStyle = '#1d3557';
  ctx.beginPath();
  ctx.roundRect(cx - 28, cy - 12, 56, 24, 5);
  ctx.fill();

  // draw wheels
  ctx.fillStyle = '#aab';
  [cx - 16, cx + 16].forEach(wx => {
    ctx.beginPath(); ctx.arc(wx, cy + 14, 6, 0, Math.PI * 2); ctx.fill();
  });

  // draw pole
  const poleLen = POLE_LEN * scale * 3.2;
  const px = cx + Math.sin(theta) * poleLen;
  const py = cy - Math.cos(theta) * poleLen;
  ctx.strokeStyle = '#e63946'; ctx.lineWidth = 5; ctx.lineCap = 'round';
  ctx.beginPath(); ctx.moveTo(cx, cy - 2); ctx.lineTo(px, py); ctx.stroke();

  // draw pole tip
  ctx.fillStyle = '#e63946';
  ctx.beginPath(); ctx.arc(px, py, 6, 0, Math.PI * 2); ctx.fill();
}

function toggleRun() {
  running = !running;
  const btn = document.getElementById('btn-run');
  if (running) {
    btn.textContent = 'Pause';
    runTimer = setInterval(() => {
      if (step < 200) trainStep();
      else toggleRun();
    }, 80);
  } else {
    btn.textContent = 'Run';
    clearInterval(runTimer);
  }
}

document.getElementById('btn-step').addEventListener('click', trainStep);
document.getElementById('btn-run').addEventListener('click', toggleRun);
document.getElementById('btn-reset').addEventListener('click', resetSim);
resetSim();
Result
Console

The demo shows a simplified PPO training loop. The bar chart tracks the policy ratio r(θ)=πθ(a∣s)/πθold(a∣s)r(\theta) = \pi_\theta(a|s) / \pi_{\theta_\text{old}}(a|s) — how much the new policy differs from the old one on a sampled action. The clipped objective caps the ratio in [1−ε, 1+ε][1-\varepsilon,\, 1+\varepsilon] (here ε=0.2\varepsilon = 0.2), so a gradient that would push the ratio outside that band is silently blocked. Click Step to advance one training step and watch how the clipping engages whenever an update overshoots.

The Clipping Objective

The core of PPO is the clipped surrogate objective. Let rt(θ)r_t(\theta) be the probability ratio:

rt(θ)=πθ(at∣st)πθold(at∣st)r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}

The naive policy gradient just maximizes rt(θ)⋅A^tr_t(\theta) \cdot \hat{A}_t, where A^t\hat{A}_t is an advantage estimate (roughly: how much better this action was than expected). The problem is that rt(θ)r_t(\theta) can become arbitrarily large, causing a catastrophic update.

PPO instead maximizes:

LCLIP(θ)=Et[min⁡ ⁣(rt(θ) A^t,  clip(rt(θ), 1−ε, 1+ε) A^t)]L^{\text{CLIP}}(\theta) = \mathbb{E}_t \bigl[\min\!\bigl(r_t(\theta)\,\hat{A}_t,\; \text{clip}(r_t(\theta),\, 1-\varepsilon,\, 1+\varepsilon)\,\hat{A}_t\bigr)\bigr]

The min⁡\min makes this a pessimistic bound: when the advantage is positive, capping rtr_t at 1+ε1+\varepsilon prevents over-exploitation; when negative, capping at 1−ε1-\varepsilon prevents over-punishment. The gradient is zero outside [1−ε,1+ε][1-\varepsilon, 1+\varepsilon], so the update is automatically blocked the moment it would overshoot.

Why this matters for complexity:

  • The predecessor TRPO enforced a trust-region constraint via a costly conjugate-gradient solve on every update.
  • PPO replaces that with a single min⁡\min and clip\text{clip} — first-order only, the same cost as a plain gradient step.
  • The resulting algorithm is O(T⋅∣π∣)O(T \cdot |\pi|) per epoch (where TT is the trajectory length and ∣π∣|\pi| is the parameter count), just like stochastic gradient descent on a supervised problem.

PPO does not come with a proof of global convergence — reinforcement learning objectives are non-convex and this is an open problem (related to broader questions in non-convex optimization). What it offers in practice is monotone empirical improvement: updates that reliably make things better rather than occasionally catastrophic.

Where It Matters

PPO's combination of simplicity and stability made it the go-to algorithm across domains:

  • Robotics locomotion: DeepMind and OpenAI used PPO to train simulated and physical robots to walk, run, and recover from falls — tasks where a single bad update could break months of training.
  • Video games: OpenAI Five used PPO to defeat professional Dota 2 players in 2019. The same architecture powered agents in StarCraft II and many Atari games.
  • RLHF (Reinforcement Learning from Human Feedback): the fine-tuning stage that turns a language model's raw predictions into helpful, harmless responses (used in InstructGPT, ChatGPT, Claude, and Gemini) is a PPO loop where the reward comes from a human-preference model. The clipping is essential — without it, the policy would quickly collapse to reward-hacking.
  • Chip design: Google used PPO-based RL to place components on silicon chips, outperforming human expert layouts in some benchmarks.

The pattern is always the same: a problem where the reward is delayed, the state space is large, and a single update can destroy fragile learned behavior. PPO's clipping makes those updates safe.

Conclusion

Reinforcement learning had all the right ingredients — exploration, reward signals, gradient updates — but kept tripping over its own steps. PPO fixed that with the simplest possible constraint: clip the policy ratio and block any gradient that would take the policy too far from where it already is.

The result is an algorithm that any practitioner can run on a laptop, that scales to billion-parameter models, and that sits at the core of the systems training today's most capable AI. The next time you interact with a language model that actually follows instructions, there is a good chance a PPO loop shaped its behavior — one clipped gradient at a time.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/reinforcement-learning-ppo/Content licensed under CC BY-NC 4.0.