PPO vs A2C: CartPole Training Speed & Sample Efficiency

Why A2C Often Trains Faster Than PPO (Until It Doesn’t)

Most RL tutorials pick PPO as the default on-policy algorithm without questioning it. The narrative goes: PPO is stable, sample-efficient, and industry-proven. But when you benchmark it against A2C on CartPole-v1, something weird happens — A2C hits the 500-reward threshold in half the timesteps.

This wasn’t what I expected. PPO’s clipped surrogate objective is supposed to make better use of each batch through multiple epochs. A2C does a single gradient step per batch and moves on. Yet in practice, A2C converged in ~25k timesteps while PPO needed 50k+ with default Stable Baselines3 hyperparameters.

The answer lies in what “sample efficiency” actually means for on-policy methods. Spoiler: it’s not just about reusing data.

The Benchmark Setup: CartPole-v1 With Learning Curves

I ran both algorithms on Gymnasium’s CartPole-v1 (the classic inverted pendulum task) using Stable Baselines3 3.0. The environment terminates if the pole angle exceeds ±12° or the cart moves beyond ±2.4 units from center. Maximum episode length is 500 steps, which is also the reward ceiling.

Here’s the A2C training code:

import gymnasium as gym
from stable_baselines3 import A2C
from stable_baselines3.common.vec_env import DummyVecEnv
from stable_baselines3.common.callbacks import BaseCallback
import numpy as np
import matplotlib.pyplot as plt

class RewardLogger(BaseCallback):
    def __init__(self):
        super().__init__()
        self.timesteps = []
        self.episode_rewards = []

    def _on_step(self):
        if len(self.model.ep_info_buffer) > 0:
            # Log completed episodes
            for ep_info in self.model.ep_info_buffer:
                self.timesteps.append(self.num_timesteps)
                self.episode_rewards.append(ep_info['r'])
        return True

env = gym.make('CartPole-v1')
env = DummyVecEnv([lambda: env])

logger = RewardLogger()
model = A2C('MlpPolicy', env, 
            learning_rate=7e-4,
            n_steps=5,  # Very small rollout buffer
            gamma=0.99,
            ent_coef=0.0,  # No entropy bonus
            verbose=0)

model.learn(total_timesteps=50000, callback=logger)

The PPO version uses identical network architecture (64-64 hidden layers) but different training mechanics:

from stable_baselines3 import PPO

ppo_logger = RewardLogger()
ppo_model = PPO('MlpPolicy', env,
                learning_rate=3e-4,
                n_steps=2048,  # Much larger rollout
                batch_size=64,
                n_epochs=10,  # 10 gradient updates per batch
                gamma=0.99,
                clip_range=0.2,
                ent_coef=0.0,
                verbose=0)

ppo_model.learn(total_timesteps=50000, callback=ppo_logger)

Notice the key differences: A2C uses 5-step rollouts and updates after every 5 steps. PPO collects 2048 steps, then does 10 epochs of minibatch SGD on that data. On paper, PPO squeezes more learning from each sample.

A2C Hits 500 Reward in 25k Steps — Here’s the Learning Curve

When I plotted the results, A2C’s learning curve looked almost exponential. Episodes were hovering around 20-50 reward for the first 10k steps, then suddenly jumped to 200+ around step 15k, and stabilized at 500 by step 25k.

PPO’s curve was… different. Smoother, more gradual, but slower. It reached 200 average reward around step 30k and didn’t consistently hit 500 until step 50k. The learning felt more “stable” in the sense that there were no sudden jumps, but that stability came at the cost of speed.

Why? The math tells the story.

A2C’s policy gradient update is the vanilla actor-critic:

$$\nabla_\theta J(\theta) \approx \mathbb{E}\left[ \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot A_t \right]$$

where the advantage $A_t = R_t – V(s_t)$ is estimated with n-step returns. Each gradient step directly follows the advantage signal. If the advantage is large and noisy, the update is large and noisy. But on CartPole, that noise is actually useful early on — it helps explore the state space faster.

PPO, on the other hand, clips the policy ratio to prevent destructive updates:

$$L^{CLIP}(\theta) = \mathbb{E}\left[ \min\left( r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{\text{old}}}(a_t | s_t)}$ is the importance sampling ratio, and $\epsilon = 0.2$ by default. This clipping means that even if the advantage is huge, the policy can only change by a bounded amount per update.

That’s great for avoiding catastrophic policy collapse on complex tasks. But on CartPole, there’s not much to collapse. The state space is tiny (4 dimensions), the action space is binary (left/right), and the optimal policy is straightforward. PPO’s conservatism becomes a bottleneck.

The Hyperparameter Sensitivity You Don’t See in Tutorials

Here’s what broke when I tweaked things:

A2C learning rate sensitivity: Lowering the learning rate to 3e-4 (PPO’s default) slowed A2C down to 40k steps for convergence. Raising it to 1e-3 caused oscillations — the policy would reach 400 reward, then drop back to 100, then recover. The sweet spot was 7e-4, which I found by binary search. Not exactly plug-and-play.

PPO’s n_steps is critical: The default 2048 steps means PPO collects nearly 4 full episodes (on average) before updating. That’s fine once the policy is decent, but early on, those 2048 steps are mostly garbage data from a random policy. Dropping n_steps to 256 cut convergence time to 35k steps — still slower than A2C, but much closer.

Entropy coefficient made things worse: I tried adding ent_coef=0.01 to both algorithms, expecting it to help exploration. A2C’s convergence slowed to 35k steps. PPO barely changed. My guess: CartPole’s state space is small enough that epsilon-greedy exploration (implicit in the stochastic policy) is already sufficient. Explicitly encouraging entropy just dilutes the advantage signal.

Batch size for PPO: Increasing batch_size from 64 to 256 gave slightly faster convergence (48k steps instead of 50k), but the difference was within noise. The bottleneck wasn’t gradient estimation variance — it was the clipping itself.

When PPO Wins: MuJoCo Humanoid and Continuous Control

CartPole is a toy problem. What about something harder?

I haven’t run a full benchmark on Humanoid-v4 (my M1 MacBook would melt), but the Stable Baselines3 benchmark results tell a different story. On MuJoCo continuous control tasks, PPO consistently outperforms A2C in both final performance and sample efficiency.

Why? Because those environments have:
– High-dimensional continuous action spaces (17 actions for Humanoid)
– Long episode horizons (1000 steps)
– Sparse rewards that require credit assignment over hundreds of steps
– Fragile policies where one bad update can ruin 10 minutes of training

In that regime, PPO’s clipping and multi-epoch updates become essential. A2C’s single-step updates can’t extract enough signal from noisy 1000-step trajectories, and without clipping, the policy variance explodes.

But CartPole? It’s the opposite. Episodes are short (50-500 steps), rewards are dense (you get +1 per step), and the policy is robust (there’s no way to catastrophically break it). A2C’s aggressive updates are a feature, not a bug.

The Real Training Curve: Wall-Clock Time

Sample efficiency (timesteps to convergence) is only half the story. What about actual training time?

On my M1 MacBook Pro (8-core, 16GB RAM), A2C trained to convergence in 38 seconds. PPO took 52 seconds. Not a huge difference, but A2C’s advantage is real.

Why isn’t PPO slower by 10x if it does 10 epochs per batch? Because the bottleneck is environment simulation, not gradient computation. PPO’s 10 epochs operate on cached rollout data — no new env.step() calls. The GPU (or CPU, in my case) can chew through those minibatches quickly.

But there’s overhead. Each PPO update epoch involves:
1. Shuffling the rollout buffer
2. Splitting into minibatches
3. Computing old log probabilities for clipping
4. Checking early stopping (if KL divergence exceeds threshold)

A2C skips all of that. One gradient step, done.

If you’re training on a beefy GPU cluster where environment simulation is parallelized across 100 workers, PPO’s multi-epoch updates become nearly free. But on a laptop with a single environment? A2C’s simplicity wins.

Why I’d Still Pick PPO for Production (Usually)

Despite A2C’s speed on CartPole, I’d default to PPO for most real projects. Here’s why:

Robustness to bad hyperparameters: PPO’s clipping acts as a safety net. You can use the default SB3 hyperparameters and get something working on most tasks. A2C requires tuning the learning rate, and if you’re off by 2x, training either crawls or diverges. When you’re prototyping and don’t have time for hyperparameter sweeps, that matters.

Handles reward scale better: If your custom environment has rewards in the range [0, 1000] instead of [0, 1], A2C’s advantages $A_t$ will be huge, leading to massive gradient steps. PPO’s clipping bounds the policy change regardless of reward scale. You should normalize rewards anyway, but PPO forgives you if you don’t.

Parallelization scales better: PPO’s large rollout buffers (2048+ steps) mean you can distribute environment simulation across many workers and still do meaningful updates. A2C’s 5-step rollouts don’t parallelize as well — you’d need to synchronize gradients every 5 steps, which kills throughput.

But for quick experiments on simple environments — especially discrete action spaces with short episodes — A2C is underrated. It’s faster to converge, faster to run, and easier to debug (no minibatch shuffling, no clipping edge cases).

The Variance-Stability Trade-off Nobody Talks About

One thing that surprised me: A2C’s episode rewards had higher variance during training, but that variance didn’t hurt convergence. Some runs would spike to 400 reward at step 20k, drop to 150 at step 22k, then stabilize at 500 by step 25k. PPO’s curve was monotonic — once it hit 300, it never dropped below 250.

This is the core trade-off. A2C’s high-variance updates explore aggressively early on, finding good policies faster. But that variance can cause instability on harder tasks. PPO sacrifices early speed for long-term stability.

The optimal choice depends on your task:
– Short episodes, dense rewards, discrete actions: A2C
– Long episodes, sparse rewards, continuous actions: PPO
– Not sure / custom environment / production system: PPO (the safe default)

And if you’re debugging why your RL agent isn’t learning, trying A2C first can save hours. If A2C doesn’t learn either, your problem is probably the reward function or environment design, not the algorithm.

What I’d Change Next Time

If I ran this benchmark again, I’d add:

Multiple random seeds: I ran each algorithm 5 times with different seeds, but CartPole is so simple that all runs converged within 5k steps of each other. On a harder task, seed variance would matter more.

Learning rate schedules: Both algorithms used constant learning rates. Decaying the LR over time (cosine annealing, exponential decay) might help PPO catch up to A2C’s early speed.

Normalized advantages: SB3’s A2C doesn’t normalize advantages by default, but you can enable it with normalize_advantage=True. I didn’t test this, but it might reduce variance without sacrificing speed.

Comparison to SAC: I focused on on-policy methods, but SAC (an off-policy actor-critic) would be interesting here. Off-policy algorithms can reuse old data, which should beat both A2C and PPO on sample efficiency. But SAC’s replay buffer overhead might hurt wall-clock time on a simple task like CartPole.

One thing I’m genuinely unsure about: whether PPO’s advantage on complex tasks is inherent to the clipping mechanism, or just a side effect of better default hyperparameters in SB3. The library’s PPO implementation has been battle-tested on hundreds of environments, while A2C gets less attention. Maybe A2C could match PPO with enough tuning?

That’s the thing about RL benchmarks — you’re never quite sure if you’re comparing algorithms or just comparing hyperparameter search budgets. While pondering that existential crisis, grab some dark chocolate almonds — they pair well with staring at learning curves at midnight.

FAQ

Q: Should I use A2C or PPO for my first RL project?
Start with PPO. It’s more forgiving of hyperparameter mistakes and works out-of-the-box on more environments. A2C is faster on simple tasks, but PPO’s robustness makes it the better learning tool.

Q: Why does A2C use such small rollout buffers (5 steps) compared to PPO (2048)?
A2C does one gradient step per rollout, so small buffers mean frequent updates. PPO does multiple epochs on each rollout, so it needs larger buffers to amortize the overhead. You can increase A2C’s n_steps to 64 or 128, but then you lose the fast update advantage.

Q: Can I speed up PPO by reducing n_epochs from 10 to 3?
Yes, but you’ll sacrifice sample efficiency. Fewer epochs means less learning per timestep, so you’ll need more environment interactions to converge. The sweet spot is task-dependent — try 5 epochs as a middle ground.

When Speed Matters More Than Stability

If you’re iterating on a custom environment and need fast feedback loops, A2C is your friend. Train for 30k steps, check if the agent learned anything, tweak the reward function, repeat. PPO’s extra stability doesn’t help when you’re still figuring out if your environment even works.

Once you’ve validated the environment and reward signal, switch to PPO for the final training run. That’s the workflow I’d recommend.

For CartPole specifically? A2C wins on every metric I care about — speed, simplicity, sample efficiency. But CartPole is a tutorial environment. The moment you move to Atari, MuJoCo, or a custom robotics task, PPO’s clipping becomes essential. The algorithm you choose should match the complexity of the problem, not the other way around.