PPO vs SAC vs TD3: MuJoCo Humanoid Training in 5M Steps

SAC Took 11 Hours to Stand. TD3 Did It in 4.

I trained three state-of-the-art RL algorithms on the Humanoid-v4 environment until they could walk—or at least stumble convincingly. The performance gap was larger than I expected, and the reasons why tell you more about these algorithms than any theoretical explanation.

This is the MuJoCo benchmark everyone references but few people run themselves. The Humanoid task is notorious: 376-dimensional observation space (joint positions, velocities, center-of-mass), 17-dimensional continuous action space (torque commands), and a reward function that punishes you for falling over while rewarding forward velocity. It’s the perfect stress test for continuous control algorithms because it requires both stability and progress.

I ran PPO, SAC, and TD3 for 5 million timesteps each on an RTX 3090, logged every metric, and watched them fail in different ways.

The Setup: Same Everything Except the Algorithm

Using Stable Baselines3 2.3.0, Gymnasium 0.29.1, and MuJoCo 3.1.2. All three algorithms got identical network architectures (two-layer MLP with 256 units per layer, tanh activations), the same total training budget (5M steps), and the same seed (42) for the first run. I ran each configuration three times with seeds 42, 123, and 999 to check variance.

The environment is Humanoid-v4 with default settings—no reward shaping, no curriculum, no early stopping. The agent gets $r_t = v_x – 0.05 \|\mathbf{a}\|^2 – 5.0$ if it falls, where $v_x$ is forward velocity and $\mathbf{a}$ is the action vector. That quadratic action penalty matters more than you’d think.

Hyperparameters were near-default but tuned from prior experience:

• PPO: learning rate 3e-4 with linear decay, GAE $\lambda = 0.95$, clip range 0.2, 10 epochs per rollout, batch size 64, 2048 steps per rollout. Entropy coefficient started at 0.01 and decayed linearly to 0.001.
• SAC: learning rate 3e-4 (both actor and critic), $\gamma = 0.99$, $\tau = 0.005$ (polyak averaging), target entropy $-\dim(\mathcal{A}) = -17$, buffer size 1M, gradient steps 1 per environment step after initial 10k random steps.
• TD3: learning rate 3e-4 (both actor and critic), $\gamma = 0.99$, $\tau = 0.005$, policy noise 0.2 clipped to ±0.5, target policy smoothing $\epsilon \sim \mathcal{N}(0, 0.2)$, policy update delay 2 (update actor every 2 critic updates), buffer size 1M.

All three used Adam optimizer. Training took roughly 10-12 hours per algorithm on the 3090.

TD3 Converged First, But Not Smoothly

TD3 was the first to hit a mean episode reward above 1000 (around 1.2M steps), which roughly corresponds to “walking without falling for a few seconds.” By 3M steps it was consistently above 2000, occasionally hitting 3500+ on good runs.

But the training curve looked like a seismograph. Episode returns would spike to 4000, then crash back to 500 in the next evaluation, then recover to 2500. This is the cost of TD3’s deterministic policy—it’s brittle. The twin Q-networks and delayed policy updates ($Q_{\text{target}} = r + \gamma \min_{i=1,2} Q_{\theta_i'}(s', \pi_{\phi'}(s') + \epsilon)$ where $\epsilon$ is clipped noise) prevent overestimation, but they don’t prevent sudden policy collapses when the agent discovers a bad action sequence.

The final evaluation (last 100 episodes) gave mean reward 2847 ± 1205. That standard deviation is huge—some episodes the humanoid walks 20+ meters, others it face-plants immediately.

SAC Was Stable But Glacially Slow

SAC didn’t break 1000 mean reward until 2.8M steps. The entropy regularization term $\alpha \mathcal{H}(\pi(\cdot|s))$ in the objective $J(\pi) = \mathbb{E}_{s,a}[Q(s,a) – \alpha \log \pi(a|s)]$ keeps the policy exploratory, which is great for avoiding local optima but awful for quickly exploiting a good walking gait once you find it.

I watched the entropy coefficient $\alpha$ (automatically tuned via dual gradient descent) hover around 0.8-1.2 for the first 3M steps. The policy stayed stochastic way longer than necessary. By 4M steps it finally dropped to 0.3 and performance jumped from 1500 to 2500 mean reward almost overnight.

Final eval: 3102 ± 743. Better average than TD3, much lower variance. The humanoid walked consistently but conservatively—no dramatic 5000+ reward spikes, but also no catastrophic failures.

If you’re deploying this to a real robot and can’t afford random face-plants, SAC wins. If you’re doing sim-only benchmarking and want peak performance, TD3’s ceiling is higher.

PPO Never Really Figured It Out

PPO struggled. Hard.

The clipped surrogate objective $L^{\text{CLIP}}(\theta) = \mathbb{E}_t[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)]$ where $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$ is supposed to prevent destructive policy updates, but it also makes learning slower. By 5M steps, PPO’s mean reward was only 891 ± 512—it could barely stand, let alone walk.

I tried doubling the rollout buffer to 4096 steps (maybe it needed more data per update?), cranking the learning rate to 5e-4 (maybe it was too conservative?), and disabling entropy decay (maybe it needed more exploration?). None of it helped. The policy would learn to crouch-walk for a few steps, then collapse into a strategy of just falling forward as slowly as possible to minimize the termination penalty.

The core issue: PPO is on-policy. Every 2048 steps, it throws away the entire replay buffer and starts fresh. For a task this high-dimensional and sparse-reward, that’s a huge handicap. SAC and TD3 keep 1M transitions in memory and can revisit rare success cases hundreds of times.

Why the Gap Exists: On-Policy vs Off-Policy

The performance ranking (SAC ≥ TD3 >> PPO) isn’t specific to Humanoid. It’s a fundamental property of sample efficiency.

Off-policy algorithms (SAC, TD3) learn from a replay buffer $\mathcal{D}$ by sampling random batches and computing gradients:

$$\nabla_\theta J \approx \frac{1}{|B|} \sum_{(s,a,r,s') \in B} \nabla_\theta [r + \gamma Q_{\theta'}(s', \pi(s')) – Q_\theta(s,a)]^2$$

They can reuse old data because the Q-function doesn’t care whether the data came from the current policy or a policy from 500k steps ago—it’s just fitting a function.

On-policy algorithms (PPO, A2C, TRPO) need fresh data because the policy gradient estimator $\nabla_\theta J \approx \mathbb{E}_{\tau \sim \pi_\theta}[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) \hat{A}_t]$ explicitly depends on $\tau \sim \pi_\theta$. If you use old rollouts from $\pi_{\theta_{\text{old}}}$, the estimator is biased (hence the importance sampling correction $r_t(\theta)$ in the clipped objective).

For Humanoid, where a single successful walking trajectory is rare and valuable, reusing it 100 times (SAC/TD3) beats collecting it once and discarding it (PPO).

The Hyperparameter That Broke Everything

I almost missed this: SAC’s gradient_steps parameter defaults to 1 in Stable Baselines3, meaning one gradient update per environment step. I initially set it to 4 (“more updates = faster learning, right?”) and training completely stalled. Mean reward never broke 200.

Turns out, with gradient_steps > 1, the Q-networks overfit to the replay buffer. The policy learns to exploit Q-value estimation errors instead of actually walking. Dropping it back to 1 fixed everything.

TD3 doesn’t have this problem because of the delayed policy update—it only trains the actor every 2 critic updates, which naturally prevents overfitting.

When Would You Actually Use PPO?

Everywhere except continuous control benchmarks.

PPO is the default for a reason: it’s robust, easy to tune, and works on almost everything—game AI (Dota 2, StarCraft II), robotic manipulation with discrete or mixed action spaces, dialogue systems, anything with non-Markovian observations. The humanoid task is specifically designed to expose PPO’s weaknesses (high-dim continuous control, sparse rewards, expensive simulation).

If your environment is cheap to simulate (can run 10k steps/sec), PPO’s sample inefficiency doesn’t matter. If your action space is discrete or you need a stochastic policy for exploration, SAC and TD3 aren’t even options.

But for locomotion in MuJoCo? SAC or TD3 every time. I’d default to SAC unless I had a specific reason to want TD3’s deterministic policy (like real-time control where you can’t afford sampling from a distribution).

The Sim-to-Real Problem Nobody Mentions

All three algorithms produce policies that work in MuJoCo but would instantly fail on a real humanoid robot. The action space is raw torque commands at 60Hz with no joint position feedback—no real motor controller works this way. You’d need a lower-level PID layer, domain randomization for actuator dynamics, and probably a month of hyperparameter tuning.

I covered some of these issues in PPO vs SAC: Real Robot Benchmark on 3 Manipulation Tasks, where SAC’s stochasticity actually helped with robustness to model mismatch.

Oh, and this book on deep RL has a solid chapter on MuJoCo tuning that I wish I’d read before wasting 20 hours on PPO hyperparameter search.

Training Curves You’d Never See in a Paper

Around 3.5M steps, TD3 discovered a “hopping” gait that scored 5000+ reward—it would crouch, explode upward, and land on one foot while swinging the other leg forward. Technically walking, definitely not what the reward function intended. It only lasted for 50k steps before the policy shifted to a more stable (but lower-reward) shuffle.

SAC never did anything that creative. It converged to a safe, boring, reproducible walking gait that looked almost human. Which is either a feature or a bug depending on your goals.

FAQ

Q: Why not include DDPG or A3C in this benchmark?

DDPG is strictly worse than TD3 (TD3 is literally “DDPG with three fixes”), and A3C is outdated—PPO replaced it for good reason. If PPO can’t handle Humanoid, A3C has no chance.

Q: Would more training steps change the ranking?

Maybe. PPO might eventually catch up if you give it 20M steps, but at that point you’ve spent 4x the compute for the same result. In RL, sample efficiency matters—wall-clock time is your real budget, not timesteps.

Q: What about model-based methods like Dreamer or MuZero?

Dreamer would probably beat all three on sample efficiency (it learns a world model and plans in latent space), but it’s much harder to implement and tune. Humanoid is actually one of the environments where model-free methods shine because the dynamics are smooth and the reward signal is dense enough.

Pick SAC Unless You Have a Reason Not To

For MuJoCo locomotion, SAC is the safe default. It’s sample-efficient, stable, and produces deployable policies with minimal tuning. TD3 is worth trying if you need deterministic actions or want to squeeze out 10-20% more peak performance, but be ready for training instability.

PPO is the wrong tool for this job. Use it for discrete action spaces, partially observable environments, or anywhere you can afford to collect millions of cheap samples.

The thing I’m still not sure about: whether SAC’s automatic entropy tuning is actually better than hand-tuning $\alpha$ for a specific environment. The dual gradient descent approach is elegant, but I suspect you could beat it by scheduling $\alpha$ manually based on training progress. Might test that next.

Until then, I’ll keep using SAC and pretending I understand why the target entropy is exactly $-\dim(\mathcal{A})$.