PPO: Why It Powers ChatGPT and Game AI

Q: How does The Problem PPO Solves work?

In Part 3, we saw that policy gradient methods compute: ∇J(θ) = E [ ∇log π_θ(a|s) · A(s,a) ] The issue: step size matters enormously. Too small = slow learning. Too large = the policy changes drastically, performance collapses, and it never recovers. TRPO (Trust Region Policy Optimization) solved

Q: How does The PPO-Clip Objective work?

Instead of the standard policy gradient loss, PPO uses: L_CLIP = E [ min(r_t · A_t, clip(r_t, 1-ε, 1+ε) · A_t) ] where: - r_t = π_θ(a|s) / π_θ_old(a|s) — the probability ratio between new and old policy - ε — clipping range (typically 0.2) - A_t — advantage estimate (using GAE) What this does:

Disclosure: As an Amazon Associate, I earn from qualifying purchases. Some links in this post are affiliate links — they cost you nothing extra.

.rl-series-nav{margin:2rem 0;padding:1.5rem;border:1px solid #e2e8f0;border-radius:10px;background:#f8fafc}.rl-series-nav h4{margin:0 0 .8rem;color:#6c5ce7;font-size:1.1rem}.rl-series-nav ul{list-style:none;padding:0;margin:0}.rl-series-nav li{padding:.3rem 0}.rl-series-nav a{color:#4a5568;text-decoration:none;font-size:.9rem}.rl-series-nav a:hover{color:#6c5ce7}.rl-series-nav .current{font-weight:700;color:#6c5ce7}

Reinforcement Learning Series

Part 1: RL Fundamentals — MDP, Bellman, Value Functions
Part 2: From Q-Learning to DQN
Part 3: Policy Gradient and Actor-Critic
→ Part 4: PPO — Why It Powers ChatGPT and Game AI
Part 5: SAC — Mastering Continuous Control

This is Part 4 of our 5-part Reinforcement Learning series. We’re covering the most widely-used RL algorithm in production today.

Series Overview:
– Part 1: RL Basics — MDP, Bellman Equation, Value Functions
– Part 2: From Q-Learning to DQN
– Part 3: Policy Gradient Methods
– Part 4: PPO — The Industry Standard (You are here)
– Part 5: SAC — Mastering Continuous Control

Why PPO Matters

Proximal Policy Optimization (Schulman et al., 2017) is the default algorithm for:
– RLHF in ChatGPT, Claude, and other LLMs
– Game AI — OpenAI Five (Dota 2), hide-and-seek agents
– Robotics — manipulation, locomotion
– Production RL — anywhere stability matters more than sample efficiency

Why? Because PPO is stable, simple to implement, and works across a wide range of problems with minimal hyperparameter tuning.

The Problem PPO Solves

In Part 3, we saw that policy gradient methods compute:

∇J(θ) = E [ ∇log π_θ(a|s) · A(s,a) ]

The issue: step size matters enormously. Too small = slow learning. Too large = the policy changes drastically, performance collapses, and it never recovers.

TRPO (Trust Region Policy Optimization) solved this with a hard constraint on policy change — but it required second-order optimization (computing the Fisher information matrix), making it complex and expensive.

PPO achieves similar stability with a simple clipping trick that requires zero additional computation beyond standard gradient descent.

Enjoying this article? Get more like it delivered to your inbox. Subscribe to the newsletter

The PPO-Clip Objective

Instead of the standard policy gradient loss, PPO uses:

L_CLIP = E [ min(r_t · A_t, clip(r_t, 1-ε, 1+ε) · A_t) ]

where:
– r_t = π_θ(a|s) / π_θ_old(a|s) — the probability ratio between new and old policy
– ε — clipping range (typically 0.2)
– A_t — advantage estimate (using GAE)

What this does:

Scenario	Effect
Advantage > 0 (good action)	r_t is clipped at 1+ε → limits how much we increase probability
Advantage < 0 (bad action)	r_t is clipped at 1-ε → limits how much we decrease probability

The min ensures we take the more pessimistic bound, preventing the policy from changing too aggressively in either direction.

def ppo_loss(old_log_probs, new_log_probs, advantages, clip_epsilon=0.2):
    """Compute PPO clipped surrogate objective."""
    ratio = torch.exp(new_log_probs - old_log_probs)  # π_new / π_old

    # Unclipped objective
    surr1 = ratio * advantages

    # Clipped objective
    surr2 = torch.clamp(ratio, 1 - clip_epsilon, 1 + clip_epsilon) * advantages

    # Take the minimum (pessimistic bound)
    return -torch.min(surr1, surr2).mean()

Tuning PPO hyperparameters at 3am requires sustained focus — Dark Chocolate Espresso Beans are the non-negotiable fuel for convergence.

Full PPO Implementation

Here’s a complete, production-style PPO implementation:

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import gymnasium as gym


class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.actor = nn.Sequential(
            nn.Linear(state_dim, hidden),
            nn.Tanh(),
            nn.Linear(hidden, hidden),
            nn.Tanh(),
            nn.Linear(hidden, action_dim),
            nn.Softmax(dim=-1),
        )
        self.critic = nn.Sequential(
            nn.Linear(state_dim, hidden),
            nn.Tanh(),
            nn.Linear(hidden, hidden),
            nn.Tanh(),
            nn.Linear(hidden, 1),
        )

    def forward(self, state):
        return self.actor(state), self.critic(state)

    def act(self, state):
        probs, value = self.forward(state)
        dist = torch.distributions.Categorical(probs)
        action = dist.sample()
        return action.item(), dist.log_prob(action), value.squeeze()


class RolloutBuffer:
    """Stores experience for one rollout phase."""
    def __init__(self):
        self.states = []
        self.actions = []
        self.log_probs = []
        self.rewards = []
        self.values = []
        self.dones = []

    def store(self, state, action, log_prob, reward, value, done):
        self.states.append(state)
        self.actions.append(action)
        self.log_probs.append(log_prob)
        self.rewards.append(reward)
        self.values.append(value)
        self.dones.append(done)

    def clear(self):
        self.__init__()

    def compute_gae(self, gamma=0.99, lam=0.95):
        """Compute advantages and returns using GAE."""
        advantages = []
        gae = 0
        values = [v.item() for v in self.values]

        for t in reversed(range(len(self.rewards))):
            next_val = 0 if t == len(self.rewards)-1 else values[t+1]
            if self.dones[t]:
                next_val = 0
                gae = 0
            delta = self.rewards[t] + gamma * next_val - values[t]
            gae = delta + gamma * lam * gae
            advantages.insert(0, gae)

        advantages = torch.FloatTensor(advantages)
        returns = advantages + torch.FloatTensor(values)
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
        return advantages, returns


def train_ppo(env_name="CartPole-v1", total_steps=100000, rollout_steps=2048,
              epochs=10, batch_size=64, clip_eps=0.2, gamma=0.99,
              lam=0.95, lr=3e-4, entropy_coef=0.01, value_coef=0.5):

    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    model = ActorCritic(state_dim, action_dim)
    optimizer = optim.Adam(model.parameters(), lr=lr)
    buffer = RolloutBuffer()

    state, _ = env.reset()
    episode_reward = 0
    rewards_history = []
    step = 0

    while step < total_steps:
        # ── Rollout Phase ──
        buffer.clear()
        for _ in range(rollout_steps):
            state_t = torch.FloatTensor(state)
            action, log_prob, value = model.act(state_t)

            next_state, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated

            buffer.store(state, action, log_prob, reward, value, done)
            state = next_state
            episode_reward += reward
            step += 1

            if done:
                rewards_history.append(episode_reward)
                episode_reward = 0
                state, _ = env.reset()

        # ── Training Phase ──
        advantages, returns = buffer.compute_gae(gamma, lam)
        old_log_probs = torch.stack(buffer.log_probs).detach()
        states = torch.FloatTensor(np.array(buffer.states))
        actions = torch.LongTensor(buffer.actions)

        # Multiple epochs over the same data
        for _ in range(epochs):
            # Mini-batch updates
            indices = np.arange(len(buffer.states))
            np.random.shuffle(indices)

            for start in range(0, len(indices), batch_size):
                idx = indices[start:start+batch_size]

                # Current policy evaluation
                probs, values = model(states[idx])
                dist = torch.distributions.Categorical(probs)
                new_log_probs = dist.log_prob(actions[idx])
                entropy = dist.entropy().mean()

                # PPO clipped loss
                ratio = torch.exp(new_log_probs - old_log_probs[idx])
                surr1 = ratio * advantages[idx]
                surr2 = torch.clamp(ratio, 1-clip_eps, 1+clip_eps) * advantages[idx]
                actor_loss = -torch.min(surr1, surr2).mean()

                # Value loss
                critic_loss = nn.MSELoss()(values.squeeze(), returns[idx])

                # Total loss
                loss = actor_loss + value_coef * critic_loss - entropy_coef * entropy

                optimizer.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(model.parameters(), 0.5)
                optimizer.step()

        if rewards_history:
            recent = rewards_history[-10:]
            print(f"Step {step:>6d} | Episodes: {len(rewards_history)} | Avg(10): {np.mean(recent):.1f}")

    env.close()
    return model, rewards_history

model, rewards = train_ppo()

PPO Hyperparameters Guide

Parameter	Typical Value	Effect
clip_eps (ε)	0.1 — 0.3	Larger = more policy change allowed
epochs	3 — 10	More epochs = better sample use, risk of overfitting
rollout_steps	128 — 2048	Longer = better advantage estimates, slower updates
batch_size	32 — 256	Standard mini-batch tradeoffs
lr	1e-4 — 3e-4	Often annealed linearly to 0
γ (gamma)	0.99 — 0.999	Higher for long-horizon tasks
λ (GAE lambda)	0.9 — 0.99	Higher = less bias, more variance
entropy_coef	0.0 — 0.01	Encourages exploration
value_coef	0.5 — 1.0	Weight of critic loss

Practical tips:
– Start with the defaults above, they work for most problems
– If training is unstable, reduce clip_eps and lr
– If the policy converges too slowly, increase epochs or rollout_steps
– Gradient clipping (max_norm=0.5) is almost always helpful

PPO for Continuous Actions

For continuous control, swap the Categorical distribution for a Gaussian:

class ContinuousActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, hidden), nn.Tanh(),
        )
        self.mean = nn.Linear(hidden, action_dim)
        self.log_std = nn.Parameter(torch.zeros(action_dim))
        self.critic = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, hidden), nn.Tanh(),
            nn.Linear(hidden, 1),
        )

    def forward(self, state):
        features = self.shared(state)
        mean = self.mean(features)
        std = self.log_std.exp()
        dist = torch.distributions.Normal(mean, std)
        value = self.critic(state)
        return dist, value

    def act(self, state):
        dist, value = self.forward(state)
        action = dist.sample()
        log_prob = dist.log_prob(action).sum(dim=-1)
        return action.numpy(), log_prob, value.squeeze()

Everything else stays the same — PPO’s clipping works identically for continuous and discrete actions.

PPO in the Real World: RLHF

PPO’s biggest impact might be in language models. In RLHF (Reinforcement Learning from Human Feedback):

Supervised fine-tuning on high-quality examples
Reward model trained on human preference pairs
PPO optimizes the LM policy to maximize the reward model score

The “proximal” constraint is critical here — without it, the LM could degenerate into gaming the reward model (generating repetitive, high-scoring but nonsensical text). PPO’s clipping keeps the model close to its original behavior while improving on the reward signal.

# Pseudocode for RLHF with PPO
for batch in prompts:
    responses = language_model.generate(batch)
    rewards = reward_model.score(batch, responses)
    ppo_update(language_model, batch, responses, rewards,
               kl_penalty=0.1)  # Extra KL term for stability

Key Takeaways

PPO clips the policy ratio to prevent destructive updates — simple but effective
Multiple epochs on the same data improve sample efficiency
GAE provides flexible advantage estimation (bias-variance tradeoff)
Same algorithm works for discrete and continuous actions
PPO is the backbone of RLHF in modern language models

What’s Next

In Part 5 (final), we’ll cover SAC (Soft Actor-Critic) — an off-policy algorithm that adds entropy maximization for robust exploration. SAC is the go-to choice for continuous control and robotics, and offers a fundamentally different perspective: the optimal policy should be as random as possible while still maximizing reward.

FAQ

How is PPO different from TRPO?

TRPO uses a hard KL divergence constraint, requiring conjugate gradient and Fisher vector products — computationally expensive. PPO replaces this with a clipped objective that achieves similar trust-region behavior using only first-order gradients. PPO is simpler to implement, faster to run, and works just as well in practice. Most researchers have switched from TRPO to PPO.

Why does PPO reuse data for multiple epochs?

Policy gradient methods are typically on-policy — you collect data, update once, and throw it away. PPO’s clipping mechanism makes it safe to reuse the same rollout for several gradient steps because the clipping prevents the policy from straying too far from the data-collecting policy. This dramatically improves sample efficiency compared to vanilla policy gradients.

Is PPO on-policy or off-policy?

PPO is on-policy — it collects fresh rollouts with the current policy, then updates. The multiple-epoch training reuses this data, but it’s not truly off-policy because old data is discarded after each rollout phase. This makes PPO less sample-efficient than off-policy methods like SAC, but more stable and easier to tune. For problems where environment interaction is cheap (simulations), this tradeoff is usually worthwhile.

Did you find this helpful?

Your support keeps this blog running and ad-free content coming.

☕ Buy me a coffee