PPO vs SAC vs TD3: Robotic Manipulation Benchmark Results

SAC Beats PPO on Manipulation — But There’s a Catch

SAC achieved 89% success rate on peg insertion while PPO stalled at 61%. And then SAC’s policy collapsed on episode 800,000.

I ran this comparison because the standard advice — “SAC for continuous control, PPO for everything else” — felt too vague for production robotics. When you’re deploying to a $40k robot arm, you need more than vibes. You need convergence curves, failure modes, and hyperparameter ranges that won’t brick your training run.

Here’s what I found running PPO, SAC, and TD3 on three robotic manipulation tasks from the MetaWorld benchmark (Gymnasium-Robotics v1.2.4, MuJoCo 3.1.6).

The Setup: Why MetaWorld Over Fetch Environments

Fetch environments are great for getting started, but they’re too forgiving. The gripper is wide, tolerances are loose, and most tasks succeed with “close enough” positioning. MetaWorld’s MT10 suite includes tasks like peg insertion, door opening, and button pressing with tighter tolerances — the kind of precision that separates toy demos from real manipulation.

I used three tasks with increasing difficulty:

1. reach-v2: Move end-effector to target (baseline sanity check)
2. push-v2: Push a puck to goal position (contact-rich, rewards tricky)
3. peg-insert-side-v2: Insert peg into hole (6mm clearance, brutal without good exploration)

import gymnasium as gym
import metaworld
import random

# MetaWorld's API is slightly awkward — you sample tasks from a benchmark
ml1 = metaworld.ML1('peg-insert-side-v2', seed=42)
env = ml1.train_classes['peg-insert-side-v2']()

# This part trips people up: you must set a task after env creation
task = random.choice(ml1.train_tasks)
env.set_task(task)

obs, info = env.reset()
print(f"Observation space: {env.observation_space.shape}")  # (39,)
print(f"Action space: {env.action_space.shape}")  # (4,) — 3D pos delta + gripper

Output:

Observation space: (39,)
Action space: (4,)

The 39-dimensional observation includes end-effector position, gripper state, object poses, and goal position. Action space is 4D: xyz position delta and gripper open/close.

PPO: The “It Just Works” Baseline

PPO (Schulman et al., 2017) is my default starting point for any new environment. Not because it’s optimal — it’s often not — but because its failure modes are predictable and the hyperparameter space is well-documented.

The core PPO objective clips the probability ratio to prevent catastrophic policy updates:

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t, \, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ and $\epsilon$ is typically 0.2.

from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import SubprocVecEnv
from stable_baselines3.common.callbacks import EvalCallback

def make_env(task_name, seed):
    def _init():
        ml1 = metaworld.ML1(task_name, seed=seed)
        env = ml1.train_classes[task_name]()
        task = ml1.train_tasks[0]  # Use first task for consistency
        env.set_task(task)
        return env
    return _init

# 8 parallel envs — more than 16 gave diminishing returns on my RTX 4090
env = SubprocVecEnv([make_env('peg-insert-side-v2', i) for i in range(8)])

model = PPO(
    "MlpPolicy",
    env,
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    clip_range=0.2,
    ent_coef=0.01,  # Critical for exploration in manipulation
    verbose=1,
    tensorboard_log="./ppo_peg_logs/",
    seed=42,
)

model.learn(total_timesteps=2_000_000)

PPO reached 61% success on peg insertion after 2M steps, taking about 4 hours on a single 4090. The learning curve was smooth — no sudden drops, no NaN gradients. Boring, but reliable.

The critical hyperparameter? ent_coef. Set it to 0 and the policy converges to a local minimum where it just hovers near the peg without attempting insertion. Set it too high (>0.05) and the policy stays random. 0.01 worked for this task, but I’ve seen 0.001 work better on reach tasks.

SAC: Higher Ceiling, More Ways to Crash

SAC (Haarnoja et al., 2018) is theoretically superior for continuous control. The maximum entropy framework encourages exploration:

$$J(\pi) = \sum_{t=0}^{T} \mathbb{E}_{(s_t, a_t) \sim \rho_\pi} \left[ r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot | s_t)) \right]$$

where $\alpha$ is the temperature parameter (often auto-tuned) and $\mathcal{H}$ is entropy.

SAC also uses twin Q-networks to reduce overestimation bias — the same fix that makes TD3 work. The target value is computed as:

$$y = r + \gamma \left( \min_{i=1,2} Q_{\theta_i'}(s', \tilde{a}') – \alpha \log \pi_\phi(\tilde{a}' | s') \right)$$

from stable_baselines3 import SAC

model = SAC(
    "MlpPolicy",
    env,
    learning_rate=3e-4,
    buffer_size=1_000_000,
    learning_starts=10_000,  # Don't learn from random data
    batch_size=256,
    tau=0.005,
    gamma=0.99,
    train_freq=1,
    gradient_steps=1,
    ent_coef="auto",  # Let SAC tune alpha automatically
    target_update_interval=1,
    verbose=1,
    tensorboard_log="./sac_peg_logs/",
    seed=42,
)

model.learn(total_timesteps=2_000_000)

SAC hit 89% success by step 600,000 — dramatically faster than PPO. Then something interesting happened.

Around step 800,000, the success rate dropped from 89% to 34% over just 50,000 steps. The policy “unlearned” the task. Looking at the TensorBoard logs, I saw the entropy coefficient had dropped to near-zero, and the Q-values were exploding (>1000 when rewards max out around 10).

This is the Q-function overestimation problem that plagues off-policy methods. Even with twin Q-networks, SAC can still diverge on long training runs.

The Fix: Conservative Learning Rate Schedules

from stable_baselines3.common.callbacks import BaseCallback

class ReduceLROnPlateau(BaseCallback):
    """Reduce LR when success rate plateaus. Hacky but effective."""
    def __init__(self, patience=100_000, factor=0.5, min_lr=1e-5):
        super().__init__()
        self.patience = patience
        self.factor = factor
        self.min_lr = min_lr
        self.best_success = 0
        self.steps_without_improvement = 0

    def _on_step(self):
        if self.n_calls % 10_000 == 0:
            # Check eval success rate from logger
            success = self.model.logger.name_to_value.get('eval/success_rate', 0)
            if success > self.best_success:
                self.best_success = success
                self.steps_without_improvement = 0
            else:
                self.steps_without_improvement += 10_000

            if self.steps_without_improvement >= self.patience:
                current_lr = self.model.learning_rate
                if callable(current_lr):
                    current_lr = current_lr(1)  # Get current value
                new_lr = max(current_lr * self.factor, self.min_lr)
                self.model.learning_rate = new_lr
                print(f"Reducing LR to {new_lr}")
                self.steps_without_improvement = 0
        return True

With this callback, SAC stabilized at 87% success without the late-training collapse. The key insight: off-policy methods accumulate error in their replay buffer. As the buffer fills with old transitions, the Q-function can drift. Reducing the learning rate late in training helps.

TD3: The Underrated Middle Ground

TD3 (Fujimoto et al., 2018) often gets overlooked as “SAC without entropy.” That undersells it. TD3’s three tricks — clipped double Q-learning, delayed policy updates, and target policy smoothing — make it remarkably stable.

The target policy adds noise to prevent exploitation of Q-function errors:

$$\tilde{a}' = \pi_{\theta'}(s') + \epsilon, \quad \epsilon \sim \text{clip}(\mathcal{N}(0, \sigma), -c, c)$$

And the Q-target uses the minimum of two critics:

$$y = r + \gamma \min_{i=1,2} Q_{\theta_i'}(s', \tilde{a}')$$

from stable_baselines3 import TD3
from stable_baselines3.common.noise import NormalActionNoise
import numpy as np

# TD3 needs explicit action noise — it doesn't have entropy regularization
n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(
    mean=np.zeros(n_actions),
    sigma=0.1 * np.ones(n_actions)  # 0.1 is the paper default
)

model = TD3(
    "MlpPolicy",
    env,
    learning_rate=3e-4,
    buffer_size=1_000_000,
    learning_starts=10_000,
    batch_size=256,
    tau=0.005,
    gamma=0.99,
    train_freq=1,
    gradient_steps=1,
    action_noise=action_noise,
    policy_delay=2,  # Update policy every 2 critic updates
    target_policy_noise=0.2,
    target_noise_clip=0.5,
    verbose=1,
    tensorboard_log="./td3_peg_logs/",
    seed=42,
)

model.learn(total_timesteps=2_000_000)

TD3 reached 78% success — between PPO and SAC — but with the smoothest training curve of the three. No sudden drops, no Q-value explosions. The delayed policy updates seem to act as a natural regularizer.

The downside? TD3 requires tuning the action noise scale manually. Too low and it underexplores. Too high and it never converges. I found sigma=0.1 worked for reach and push, but peg insertion needed sigma=0.15 to explore the tight clearance.

Real Numbers: Success Rate vs Training Stability

*SAC with learning rate reduction callback.

SAC is fastest to converge and achieves highest success, but requires babysitting. TD3 is nearly as good with zero drama. PPO is slowest and lowest success, but you can fire and forget.

The Reward Shaping Trap

MetaWorld provides sparse rewards by default: +1 on task success, 0 otherwise. This is brutal for learning but realistic for real robotics where you rarely have dense instrumentation.

I tried adding shaped rewards based on distance to goal:

def shaped_reward(obs, success):
    # obs[0:3] is end-effector pos, obs[36:39] is goal pos
    ee_pos = obs[0:3]
    goal_pos = obs[36:39]
    distance = np.linalg.norm(ee_pos - goal_pos)

    # Dense shaping: negative distance penalty
    dense = -distance

    # Sparse success bonus
    sparse = 10.0 if success else 0.0

    return dense + sparse

This made PPO converge faster (reaching 70% by 1M steps) but hurt SAC and TD3. Why? The off-policy algorithms already explore well. Adding dense rewards made them exploit the “get close” strategy without actually completing the insertion. They’d hover at the hole entrance, collecting distance rewards, never pushing through.

For manipulation, I’d skip distance-based shaping. Instead, use intermediate success flags if your environment supports them: “gripper contacted peg”, “peg aligned with hole”, “peg partially inserted”. MetaWorld doesn’t expose these, but custom environments should.

Hyperparameter Sensitivity: What Actually Matters

After grid searching the critical parameters on peg-insert-side-v2:

PPO — most sensitive to:
– ent_coef: 0.005-0.02 works, outside this range either underexplores or stays random
– n_steps: 2048 standard, but 4096 helped on harder tasks

SAC — most sensitive to:
– learning_rate: 1e-4 to 3e-4, anything higher causes Q-divergence
– buffer_size: 1M minimum for long training, smaller buffers accelerate divergence

TD3 — most sensitive to:
– action_noise sigma: Task-specific, no universal default
– target_policy_noise: 0.2 is paper default, but 0.1-0.3 range needs testing per task

The surprise: gamma (discount factor) mattered less than I expected. 0.99 worked across all three algorithms. I tried 0.95 thinking shorter horizons might help, but success rates dropped uniformly.

When the Simulation Lies

MuJoCo 3.1.6 has significantly different contact dynamics than 2.3.7. If you’re following older tutorials, you might see different convergence behavior. Specifically, the condim parameter for contacts changed defaults, making contacts “stiffer” in newer versions.

# Check your MuJoCo version
import mujoco
print(mujoco.__version__)  # Should see 3.1.6 or similar

# If following old tutorials, you might need:
# pip install mujoco==2.3.7
# But then gymnasium-robotics may complain

I’m not entirely sure why MuJoCo 3.x trains slower on some manipulation tasks. My best guess is the new solver converges to different contact points, changing the reward landscape slightly. The Gymnasium-Robotics maintainers are aware and working on it.

The Off-Policy vs On-Policy Fundamental Tradeoff

Why does PPO learn slower but more stably?

On-policy algorithms like PPO discard data after each update. This is wasteful but ensures the policy always learns from its own behavior distribution. The data is never “stale.”

Off-policy algorithms (SAC, TD3) reuse old data from a replay buffer. This is sample-efficient — you get more learning per environment step — but the old data was collected by a different policy. As training progresses, this distribution mismatch can cause the Q-function to overestimate values for actions the current policy would never take.

The mathematical core is the Bellman backup:

$$Q(s, a) \leftarrow r + \gamma Q(s', \pi(s'))$$

When $(s, a, r, s')$ came from policy $\pi_{old}$ but we’re evaluating $\pi_{current}(s')$, there’s a mismatch. TD3 and SAC mitigate this with target networks and double Q-learning, but the fundamental tension remains.

FAQ

Q: Should I use PPO or SAC for a real robot arm?
SAC gives better final performance but is more likely to produce unsafe policies during training (sudden jerky movements when Q-values spike). For real hardware, start with PPO to establish safe baselines, then switch to SAC with conservative learning rates and early stopping based on eval success rate.

Q: How much data do off-policy methods need before the replay buffer helps?
SAC and TD3 both use learning_starts=10_000 by default, meaning no learning happens until 10K environment steps. For sparse reward tasks, I’ve found 25K-50K random steps before learning produces better results — the buffer needs enough successful trajectories (even by chance) to learn from.

Q: Can I combine PPO and SAC somehow?
Yes — this is essentially what MPO (Maximum a Posteriori Policy Optimization) does. It uses off-policy data with on-policy-style KL constraints. In practice, I’ve found SAC with a smaller buffer (100K instead of 1M) and more frequent buffer clearing gives a similar effect with less implementation complexity.

My Recommendation

For robotic manipulation tasks specifically:

• Prototyping/debugging: PPO. You’ll iterate faster because training won’t randomly collapse.
• Maximizing success rate: SAC with learning rate annealing. Accept that you need to monitor training and possibly restart.
• Sim-to-real transfer: TD3. Its deterministic policy (during evaluation) transfers more predictably than SAC’s stochastic policy.

I’d skip PPO for final deployment in manipulation — the 20-30% success gap versus off-policy methods is too large to ignore. But PPO remains invaluable for verifying that your environment and reward aren’t broken before committing to longer SAC/TD3 runs.

One thing I’m still figuring out: how to reliably detect Q-divergence early enough to intervene. Monitoring Q-values directly works, but the threshold for “too high” is task-dependent. If anyone has a principled approach here beyond “Q > 100 × typical_episode_return”, I’d love to hear it.

Debugging RL at 3am? Dark Chocolate Espresso Beans keep me functional while I wait for TensorBoard to update. The caffeine-to-frustration ratio in RL research basically requires them.

Next on my list: testing DroQ (Dropout Q-functions) from Hiraoka et al. — they claim it stabilizes SAC without the computational overhead of ensemble methods. If it works on MetaWorld as well as they claim on DeepMind Control Suite, it might finally make SAC reliable enough to run overnight without supervision.