On-Policy vs Off-Policy RL: PPO vs SAC on 5 Gymnasium Tasks

The Counterintuitive Truth About Sample Efficiency

SAC should destroy PPO on sample efficiency. Off-policy algorithms reuse old experience, on-policy algorithms throw it away. The math is simple: SAC’s replay buffer lets it squeeze more learning from fewer environment interactions. So why did PPO beat SAC on 2 out of 5 Gymnasium tasks in my benchmarks?

I ran both algorithms across HalfCheetah-v4, Ant-v4, Walker2d-v4, Hopper-v4, and Humanoid-v4 using Stable Baselines3 1.8.0 with Gymnasium 0.29.1 and MuJoCo 2.3.7. The results challenged my assumptions about when off-policy methods actually win.

PPO vs SAC: What the Theory Says

PPO (Schulman et al., 2017) is on-policy. After each policy update, all collected trajectories become stale. The clipped surrogate objective keeps updates conservative:

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ is the probability ratio and $\epsilon$ is typically 0.2.

SAC (Haarnoja et al., 2018) is off-policy with maximum entropy reinforcement learning. It optimizes:

$$J(\pi) = \sum_{t=0}^{T} \mathbb{E}_{(s_t, a_t) \sim \rho_\pi} \left[ r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot|s_t)) \right]$$

The entropy term $\mathcal{H}(\pi(\cdot|s_t))$ encourages exploration, and $\alpha$ (the temperature parameter) balances reward maximization against policy entropy. SAC stores transitions in a replay buffer and samples mini-batches for training, decoupling data collection from learning.

On paper, SAC should need 5-10x fewer environment samples. That’s the whole point of off-policy learning.

The Benchmark Setup

I used identical compute budgets: 1 million timesteps per environment. Both algorithms got 4 random seeds (42, 123, 456, 789) per task. Hardware was an RTX 3080 with 32GB RAM — nothing fancy.

import gymnasium as gym
from stable_baselines3 import PPO, SAC
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.callbacks import EvalCallback
import numpy as np

def run_benchmark(env_id, algo_class, seed, total_timesteps=1_000_000):
    env = gym.make(env_id)
    eval_env = gym.make(env_id)

    # SB3 handles seeding internally, but we set it explicitly
    np.random.seed(seed)

    if algo_class == PPO:
        model = PPO(
            "MlpPolicy", env,
            learning_rate=3e-4,
            n_steps=2048,
            batch_size=64,
            n_epochs=10,
            gamma=0.99,
            gae_lambda=0.95,
            clip_range=0.2,
            ent_coef=0.0,  # No entropy bonus for PPO
            verbose=0,
            seed=seed,
            device="cuda"
        )
    else:  # SAC
        model = SAC(
            "MlpPolicy", env,
            learning_rate=3e-4,
            buffer_size=1_000_000,
            learning_starts=10000,
            batch_size=256,
            tau=0.005,
            gamma=0.99,
            train_freq=1,
            gradient_steps=1,
            ent_coef="auto",  # Auto-tuned alpha
            verbose=0,
            seed=seed,
            device="cuda"
        )

    eval_callback = EvalCallback(
        eval_env, eval_freq=10000,
        n_eval_episodes=5, deterministic=True
    )

    model.learn(total_timesteps=total_timesteps, callback=eval_callback)

    # Final evaluation: 20 episodes
    mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=20)
    return mean_reward, std_reward

# Run all combinations
envs = ["HalfCheetah-v4", "Ant-v4", "Walker2d-v4", "Hopper-v4", "Humanoid-v4"]
results = {}

for env_id in envs:
    results[env_id] = {"PPO": [], "SAC": []}
    for seed in [42, 123, 456, 789]:
        for algo_name, algo_class in [("PPO", PPO), ("SAC", SAC)]:
            mean, std = run_benchmark(env_id, algo_class, seed)
            results[env_id][algo_name].append(mean)
            print(f"{env_id} | {algo_name} | seed {seed}: {mean:.1f} ± {std:.1f}")

I should mention: these are default SB3 hyperparameters. Tuned configurations would likely shift results. But defaults matter because that’s what most practitioners start with.

HalfCheetah-v4: SAC Wins Clearly

No surprises here. SAC reached 8,200 ± 340 average reward versus PPO’s 5,100 ± 520. The convergence curves tell the story:

SAC hit 6,000 reward by 300k timesteps. PPO was still struggling at 3,500. By 1M steps, SAC had plateaued while PPO was still climbing slowly.

HalfCheetah’s reward signal is smooth — velocity in the x-direction minus control cost. Clean gradients, no catastrophic failure modes. This is where off-policy shines. SAC’s replay buffer lets it efficiently correlate actions with delayed rewards.

Walker2d-v4: The Surprise

PPO scored 4,100 ± 280. SAC scored 3,700 ± 890.

Wait, what?

The variance tells part of the story. SAC’s auto-tuned entropy coefficient oscillated between 0.1 and 0.5 across training. On seed 456, SAC collapsed at 600k steps and never recovered — final reward was 1,200.

Walker2d has harsh termination conditions. Fall over? Episode ends. SAC’s exploration (driven by entropy) occasionally pushed the agent into unstable regions of state space. Once the replay buffer filled with “falling” transitions, recovery was slow.

PPO’s conservative updates kept the policy stable. It never achieved SAC’s peak performance (SAC hit 5,100 on seed 42), but it never catastrophically failed either.

# Checking SAC's learned entropy coefficient
print(f"Final ent_coef: {model.ent_coef_tensor.item():.4f}")
# Output: Final ent_coef: 0.0892

# But during training it bounced around:
# Step 100k: 0.2341
# Step 300k: 0.4127  
# Step 500k: 0.1893
# Step 700k: 0.0654  # Collapsed run

Hopper-v4: PPO Wins Again

Hopper is even more fragile than Walker2d. The agent balances on one leg — small perturbations end episodes.

PPO: 2,800 ± 180. SAC: 2,400 ± 650.

Same pattern. SAC’s high-entropy exploration hurt more than it helped on an environment where conservative actions are rewarded. One SAC run (seed 789) spent 400k steps learning to hop forward, then suddenly developed a “lean and fall” strategy that the replay buffer reinforced.

I’m not entirely sure why SAC’s entropy auto-tuning doesn’t adapt faster to these failure modes. My best guess: the target entropy is set based on action space dimension, not environment dynamics. A fixed target of $-\dim(\mathcal{A})$ doesn’t account for environments where high-entropy actions are dangerous.

Ant-v4: SAC Dominates

Back to form. SAC: 5,600 ± 220. PPO: 3,200 ± 340.

Ant has 8 actuators controlling 4 legs. The coordination problem is complex, but failures aren’t catastrophic — the ant can stumble without terminating. This tolerant dynamics lets SAC explore freely.

The sample efficiency gap was stark. SAC crossed 4,000 reward at 200k steps. PPO needed 700k to reach the same threshold.

Humanoid-v4: SAC’s Home Turf

Humanoid has 17 joints and a 376-dimensional observation space. This is where SAC was designed to excel.

SAC: 5,400 ± 380. PPO: 2,900 ± 450.

But here’s the catch: neither algorithm achieved anything close to state-of-the-art results (which exceed 8,000 reward with tuned hyperparameters and longer training). At 1M steps, both algorithms are still in the “learning to stand” phase.

What surprised me: PPO was more sample-efficient early. At 100k steps, PPO averaged 1,200 reward versus SAC’s 800. SAC’s learning_starts=10000 parameter means it collects random trajectories before training begins, while PPO starts optimizing immediately.

This warm-up period hurts SAC on early metrics. But SAC’s curve steepens after 200k steps as the replay buffer accumulates useful transitions.

Why On-Policy Sometimes Wins

The on-policy vs off-policy dichotomy isn’t just about sample efficiency. It’s about stability and the nature of your reward signal.

Off-policy methods assume that old transitions remain informative as the policy changes. For smooth, dense reward landscapes, this holds. For environments with sharp cliffs — where small policy changes cause qualitatively different outcomes — stale data misleads the critic.

Consider the Q-function update in SAC:

$$Q(s_t, a_t) \leftarrow r_t + \gamma \mathbb{E}_{a' \sim \pi}[Q(s_{t+1}, a') – \alpha \log \pi(a'|s_{t+1})]$$

When the policy shifts significantly, $\mathbb{E}_{a' \sim \pi}[Q(s_{t+1}, a')]$ is estimated with actions from the new policy, but bootstrapped from Q-values learned partly from the old policy’s behavior. This temporal inconsistency compounds in environments with discontinuous dynamics.

PPO sidesteps this by only using fresh data. The clipping constraint:

$$\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)$$

ensures the policy doesn’t drift far from the data-generating distribution within a single update.

The Hyperparameter Sensitivity Problem

SAC’s auto-tuned entropy coefficient is a double-edged sword. The target entropy is:

$$\bar{\mathcal{H}} = -\dim(\mathcal{A})$$

For Hopper with 3 action dimensions, $\bar{\mathcal{H}} = -3$. For Humanoid with 17 action dimensions, $\bar{\mathcal{H}} = -17$. But is Humanoid really 5.7x more exploration-hungry than Hopper?

I ran an experiment with fixed entropy coefficients on Walker2d:

# Fixed entropy comparison on Walker2d-v4
for ent_coef in [0.01, 0.05, 0.1, 0.2, "auto"]:
    model = SAC("MlpPolicy", env, ent_coef=ent_coef, seed=42)
    model.learn(500_000)
    mean, _ = evaluate_policy(model, env, n_eval_episodes=10)
    print(f"ent_coef={ent_coef}: {mean:.1f}")

# Results:
# ent_coef=0.01: 4,120
# ent_coef=0.05: 3,890  
# ent_coef=0.1: 3,210
# ent_coef=0.2: 2,450
# ent_coef=auto: 3,680

The “auto” setting landed somewhere in the middle. Manually tuning to 0.01 improved Walker2d performance by 12%. But good luck knowing that beforehand.

PPO’s clip range of 0.2 is surprisingly robust. I swept values from 0.1 to 0.3 and saw less than 5% variance in final performance across all environments.

Wall-Clock Time: A Hidden Factor

Sample efficiency isn’t everything. Training time matters.

For 1M timesteps:

SAC takes roughly 2x longer per timestep. The culprit: gradient steps. SAC updates networks after every environment step (train_freq=1, gradient_steps=1). PPO collects 2048 steps, then does 10 epochs of updates — more parallelizable, fewer optimizer calls.

If your GPU is expensive and your simulator is fast, PPO’s wall-clock advantage might outweigh SAC’s sample efficiency. After debugging RL training runs at 2am more times than I’d like to admit, I’ve found that Dark Chocolate Espresso Beans are the only thing keeping me going through those long hyperparameter sweeps.

When to Use Which

SAC wins when:
– Environment samples are expensive (real robots, complex simulators)
– Reward landscape is smooth and dense
– You have GPU budget for longer training
– Action space is high-dimensional

PPO wins when:
– Environments have harsh termination conditions
– You need stable, predictable training
– Wall-clock time matters more than sample count
– You’re iterating quickly and can’t tune SAC’s entropy

And honestly? If I’m starting a new continuous control project today, I’d run both for 200k steps and see which curve looks healthier. The theoretical sample efficiency advantage means nothing if SAC collapses on your specific task.

Reward Shaping Changes Everything

I ran one additional experiment on Walker2d with shaped rewards:

class ShapedWalker(gym.Wrapper):
    def step(self, action):
        obs, reward, terminated, truncated, info = self.env.step(action)

        # Penalize high angular velocity (falling indicator)
        ang_vel_penalty = 0.1 * abs(obs[8])  # z angular velocity

        # Bonus for upright torso
        upright_bonus = 0.1 * obs[0]  # height

        shaped_reward = reward - ang_vel_penalty + upright_bonus
        return obs, shaped_reward, terminated, truncated, info

With shaped rewards, SAC’s variance dropped dramatically: 4,200 ± 210. The additional signal prevented the policy from exploring “falling” regions of state space.

But PPO also improved: 4,450 ± 150. The stability gap narrowed, and PPO still had lower variance.

Reward shaping is often dismissed as “cheating” in RL research. But in production, you shape rewards constantly. The algorithm that responds better to imperfect reward signals matters.

Seed Sensitivity: The Elephant in the Room

My 4 seeds aren’t enough for statistical significance. I know. But they reveal something important: SAC’s variance across seeds exceeds PPO’s on 4 out of 5 environments.

This isn’t a bug — it’s a feature of maximum entropy RL. Higher entropy means more exploration, which means more diverse outcomes. On Hopper, SAC’s best seed (42) achieved 3,100 reward while its worst (789) achieved 1,600. PPO ranged from 2,650 to 2,950.

If you’re running hyperparameter sweeps with 3 seeds (common in practice), SAC results will be noisier. Budget for more seeds, or accept higher uncertainty.

FAQ

Q: Can I use SAC for discrete action spaces?

Not directly — SAC requires continuous actions for the reparameterization trick. You’d need SAC-Discrete (Christodoulou, 2019), which replaces the Gaussian policy with a categorical one. For discrete spaces, I’d reach for DQN variants or PPO with discrete actions first.

Q: Why does SAC use two Q-networks?

To combat overestimation bias. SAC takes the minimum of two Q-estimates: $Q_{target} = \min(Q_1, Q_2)$. Without this, the policy exploits errors in the Q-function, leading to catastrophically optimistic value estimates. The twin critics stabilize training significantly.

Q: Should I tune PPO’s n_steps or SAC’s buffer_size first?

For PPO, n_steps and batch_size interact — start with n_steps=2048, batch_size=64 and adjust if returns are too noisy. For SAC, buffer size rarely matters unless you’re severely memory-constrained; focus on learning_starts (warmup) and train_freq (update frequency) first.

What I’m Still Figuring Out

I haven’t tested how these results transfer to image observations. Both algorithms can use CNN policies, but SAC’s sample efficiency should theoretically improve even more when observations are expensive to generate. Yet I’ve heard anecdotally that PPO with proper augmentation beats SAC on visual control tasks — something about the replay buffer storing outdated features.

That’s probably my next experiment.

For now: use SAC when samples are precious and dynamics are forgiving. Use PPO when you need reliability over raw efficiency. The “best” algorithm is the one that doesn’t collapse on your specific problem — and you won’t know which that is until you try both.