SAC: The Best Algorithm for Continuous Control

Q: Why SAC?

PPO is great, but it has a weakness: sample efficiency. As an on-policy algorithm, PPO throws away data after each update. For robotics and real-world systems where each interaction is expensive, this is a major limitation. Soft Actor-Critic (SAC) (Haarnoja et al., 2018) addresses this with three ke

Q: How does The Maximum Entropy Framework work?

Standard RL maximizes expected reward: π* = argmax E [ Σ γ^t · r_t ] SAC maximizes expected reward plus entropy: π* = argmax E [ Σ γ^t · (r_t + α · H(π(·|s_t))) ] where H(π) = -E[log π(a|s)] is the entropy of the policy, and α (alpha) is the temperature parameter controlling the exploration-expl

Disclosure: As an Amazon Associate, I earn from qualifying purchases. Some links in this post are affiliate links — they cost you nothing extra.

.rl-series-nav{margin:2rem 0;padding:1.5rem;border:1px solid #e2e8f0;border-radius:10px;background:#f8fafc}.rl-series-nav h4{margin:0 0 .8rem;color:#6c5ce7;font-size:1.1rem}.rl-series-nav ul{list-style:none;padding:0;margin:0}.rl-series-nav li{padding:.3rem 0}.rl-series-nav a{color:#4a5568;text-decoration:none;font-size:.9rem}.rl-series-nav a:hover{color:#6c5ce7}.rl-series-nav .current{font-weight:700;color:#6c5ce7}

Reinforcement Learning Series

Part 1: RL Fundamentals — MDP, Bellman, Value Functions
Part 2: From Q-Learning to DQN
Part 3: Policy Gradient and Actor-Critic
Part 4: PPO — Why It Powers ChatGPT and Game AI
→ Part 5: SAC — Mastering Continuous Control

This is Part 5 — the finale of our Reinforcement Learning series. We’re covering the state-of-the-art algorithm for continuous control.

Series Overview:
– Part 1: RL Basics — MDP, Bellman Equation, Value Functions
– Part 2: From Q-Learning to DQN
– Part 3: Policy Gradient Methods
– Part 4: PPO — The Industry Standard
– Part 5: SAC — Mastering Continuous Control (You are here)

Why SAC?

PPO is great, but it has a weakness: sample efficiency. As an on-policy algorithm, PPO throws away data after each update. For robotics and real-world systems where each interaction is expensive, this is a major limitation.

Soft Actor-Critic (SAC) (Haarnoja et al., 2018) addresses this with three key ideas:

Off-policy learning — reuse all past experience via replay buffer
Entropy maximization — explore as much as possible while maximizing reward
Automatic temperature tuning — balance exploration and exploitation automatically

SAC is the go-to algorithm for continuous control, dominating benchmarks in robotic manipulation, locomotion, and dexterous hand tasks.

If manually tuning that temperature parameter α has you debugging at 3am, Dark Chocolate Espresso Beans are non-negotiable fuel for balancing exploration and exploitation in your own neural networks.

The Maximum Entropy Framework

Standard RL maximizes expected reward:

π* = argmax E [ Σ γ^t · r_t ]

SAC maximizes expected reward plus entropy:

π* = argmax E [ Σ γ^t · (r_t + α · H(π(·|s_t))) ]

where H(π) = -E[log π(a|s)] is the entropy of the policy, and α (alpha) is the temperature parameter controlling the exploration-exploitation tradeoff.

Why add entropy?

Benefit	Explanation
Better exploration	High entropy = try diverse actions, avoid premature convergence
Robustness	Multiple near-optimal behaviors → handles perturbations
Faster learning	Naturally avoids local optima by maintaining stochasticity
Composability	Learned behaviors transfer better to new tasks

Enjoying this article? Get more like it delivered to your inbox. Subscribe to the newsletter

SAC Architecture

SAC uses five networks (sounds like a lot, but two are just copies):

┌─────────────────┐
│  Policy (Actor)  │  π_θ(a|s) — outputs mean & std of Gaussian
│  π_θ             │
└────────┬────────┘
         │ actions
         ▼
┌─────────────────┐    ┌─────────────────┐
│  Q-Network 1    │    │  Q-Network 2    │  ← Twin Q to reduce overestimation
│  Q_φ1(s, a)     │    │  Q_φ2(s, a)     │
└────────┬────────┘    └────────┬────────┘
         │                      │
         ▼                      ▼
      min(Q1, Q2)  ← Conservative value estimate
         │
┌─────────────────┐    ┌─────────────────┐
│  Target Q1      │    │  Target Q2      │  ← Soft-updated copies
│  Q_φ1'(s, a)    │    │  Q_φ2'(s, a)    │
└─────────────────┘    └─────────────────┘

Twin Q-networks: Taking the minimum of two Q estimates prevents the overestimation bias that plagued DQN and DDPG.

Implementing SAC Step by Step

1. Networks

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random

LOG_STD_MIN = -20
LOG_STD_MAX = 2


class GaussianPolicy(nn.Module):
    """Squashed Gaussian policy for continuous actions."""
    def __init__(self, state_dim, action_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
        )
        self.mean_head = nn.Linear(hidden, action_dim)
        self.log_std_head = nn.Linear(hidden, action_dim)

    def forward(self, state):
        x = self.net(state)
        mean = self.mean_head(x)
        log_std = self.log_std_head(x).clamp(LOG_STD_MIN, LOG_STD_MAX)
        return mean, log_std

    def sample(self, state):
        mean, log_std = self.forward(state)
        std = log_std.exp()
        dist = torch.distributions.Normal(mean, std)

        # Reparameterization trick
        x = dist.rsample()

        # Squash through tanh to bound actions to [-1, 1]
        action = torch.tanh(x)

        # Correct log_prob for tanh squashing
        log_prob = dist.log_prob(x) - torch.log(1 - action.pow(2) + 1e-6)
        log_prob = log_prob.sum(dim=-1, keepdim=True)

        return action, log_prob, mean


class QNetwork(nn.Module):
    """Q-network that takes (state, action) as input."""
    def __init__(self, state_dim, action_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim + action_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, 1),
        )

    def forward(self, state, action):
        return self.net(torch.cat([state, action], dim=-1))

2. The Squashing Trick

SAC uses tanh squashing to bound actions to [-1, 1]. This is crucial because Gaussian distributions have infinite support, but real action spaces are bounded.

The log-probability correction is:

log π(a|s) = log μ(u|s) - Σ log(1 - tanh²(u_i))

This accounts for the change of variables when passing through tanh.

3. Automatic Temperature Tuning

Instead of manually setting α, SAC learns it by solving:

α* = argmin E [ -α · (log π(a|s) + H_target) ]

where H_target = -dim(A) (negative of action dimension) is a heuristic target entropy.

# Initialize alpha (temperature)
target_entropy = -action_dim  # Heuristic: -dim(A)
log_alpha = torch.zeros(1, requires_grad=True)
alpha_optimizer = optim.Adam([log_alpha], lr=3e-4)

4. Complete SAC Agent

class SACAgent:
    def __init__(self, state_dim, action_dim, hidden=256, lr=3e-4,
                 gamma=0.99, tau=0.005, buffer_size=1000000, batch_size=256):
        self.gamma = gamma
        self.tau = tau
        self.batch_size = batch_size
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # Networks
        self.policy = GaussianPolicy(state_dim, action_dim, hidden).to(self.device)
        self.q1 = QNetwork(state_dim, action_dim, hidden).to(self.device)
        self.q2 = QNetwork(state_dim, action_dim, hidden).to(self.device)
        self.q1_target = QNetwork(state_dim, action_dim, hidden).to(self.device)
        self.q2_target = QNetwork(state_dim, action_dim, hidden).to(self.device)

        # Copy weights to targets
        self.q1_target.load_state_dict(self.q1.state_dict())
        self.q2_target.load_state_dict(self.q2.state_dict())

        # Optimizers
        self.policy_opt = optim.Adam(self.policy.parameters(), lr=lr)
        self.q1_opt = optim.Adam(self.q1.parameters(), lr=lr)
        self.q2_opt = optim.Adam(self.q2.parameters(), lr=lr)

        # Auto-tuned temperature
        self.target_entropy = -action_dim
        self.log_alpha = torch.zeros(1, requires_grad=True, device=self.device)
        self.alpha_opt = optim.Adam([self.log_alpha], lr=lr)

        # Replay buffer
        self.buffer = deque(maxlen=buffer_size)

    @property
    def alpha(self):
        return self.log_alpha.exp()

    def select_action(self, state, evaluate=False):
        state_t = torch.FloatTensor(state).unsqueeze(0).to(self.device)
        if evaluate:
            mean, _ = self.policy(state_t)
            return torch.tanh(mean).cpu().detach().numpy()[0]
        action, _, _ = self.policy.sample(state_t)
        return action.cpu().detach().numpy()[0]

    def store(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def update(self):
        if len(self.buffer) < self.batch_size:
            return {}

        # Sample batch
        batch = random.sample(self.buffer, self.batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)

        states = torch.FloatTensor(np.array(states)).to(self.device)
        actions = torch.FloatTensor(np.array(actions)).to(self.device)
        rewards = torch.FloatTensor(rewards).unsqueeze(1).to(self.device)
        next_states = torch.FloatTensor(np.array(next_states)).to(self.device)
        dones = torch.FloatTensor(dones).unsqueeze(1).to(self.device)

        # ── Update Q-networks ──
        with torch.no_grad():
            next_actions, next_log_probs, _ = self.policy.sample(next_states)
            q1_next = self.q1_target(next_states, next_actions)
            q2_next = self.q2_target(next_states, next_actions)
            q_next = torch.min(q1_next, q2_next) - self.alpha * next_log_probs
            q_target = rewards + self.gamma * (1 - dones) * q_next

        q1_loss = nn.MSELoss()(self.q1(states, actions), q_target)
        q2_loss = nn.MSELoss()(self.q2(states, actions), q_target)

        self.q1_opt.zero_grad()
        q1_loss.backward()
        self.q1_opt.step()

        self.q2_opt.zero_grad()
        q2_loss.backward()
        self.q2_opt.step()

        # ── Update Policy ──
        new_actions, log_probs, _ = self.policy.sample(states)
        q1_new = self.q1(states, new_actions)
        q2_new = self.q2(states, new_actions)
        q_new = torch.min(q1_new, q2_new)

        policy_loss = (self.alpha.detach() * log_probs - q_new).mean()

        self.policy_opt.zero_grad()
        policy_loss.backward()
        self.policy_opt.step()

        # ── Update Temperature ──
        alpha_loss = -(self.log_alpha * (log_probs.detach() + self.target_entropy)).mean()

        self.alpha_opt.zero_grad()
        alpha_loss.backward()
        self.alpha_opt.step()

        # ── Soft-update Target Networks ──
        for param, target_param in zip(self.q1.parameters(), self.q1_target.parameters()):
            target_param.data.copy_(self.tau * param.data + (1 - self.tau) * target_param.data)
        for param, target_param in zip(self.q2.parameters(), self.q2_target.parameters()):
            target_param.data.copy_(self.tau * param.data + (1 - self.tau) * target_param.data)

        return {
            "q1_loss": q1_loss.item(),
            "q2_loss": q2_loss.item(),
            "policy_loss": policy_loss.item(),
            "alpha": self.alpha.item(),
        }

5. Training Loop

import gymnasium as gym

def train_sac(env_name="Pendulum-v1", total_steps=100000, start_steps=1000):
    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.shape[0]
    action_scale = env.action_space.high[0]

    agent = SACAgent(state_dim, action_dim)
    rewards_history = []

    state, _ = env.reset()
    episode_reward = 0

    for step in range(total_steps):
        # Random actions for initial exploration
        if step < start_steps:
            action = env.action_space.sample()
        else:
            action = agent.select_action(state) * action_scale

        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated

        agent.store(state, action, reward, next_state, float(done))

        if step >= start_steps:
            agent.update()

        state = next_state
        episode_reward += reward

        if done:
            rewards_history.append(episode_reward)
            if len(rewards_history) % 10 == 0:
                avg = np.mean(rewards_history[-10:])
                alpha = agent.alpha.item()
                print(f"Episode {len(rewards_history)} | Avg: {avg:.1f} | α: {alpha:.3f}")
            episode_reward = 0
            state, _ = env.reset()

    env.close()
    return agent, rewards_history

agent, rewards = train_sac()

SAC vs PPO: When to Use Which

Factor	PPO	SAC
Action space	Discrete or continuous	Continuous (primarily)
Sample efficiency	Low (on-policy)	High (off-policy, replay buffer)
Stability	Very stable	Stable (twin Q helps)
Exploration	Needs entropy bonus	Built-in (max entropy)
Hyperparameter sensitivity	Low	Medium
Wall-clock speed	Fast (parallel envs)	Slower (sequential updates)
Best for	Games, RLHF, simulations	Robotics, continuous control

Rule of thumb:
– Cheap simulation + discrete actions → PPO
– Expensive interaction + continuous control → SAC
– Not sure → Start with PPO, switch to SAC if sample efficiency matters

Advanced: Discrete SAC

SAC can also handle discrete actions by replacing the Gaussian with a Categorical distribution:

class DiscreteSACPolicy(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, action_dim),
        )

    def forward(self, state):
        logits = self.net(state)
        probs = torch.softmax(logits, dim=-1)
        log_probs = torch.log_softmax(logits, dim=-1)

        # Expected Q and entropy over all actions
        return probs, log_probs

    def get_action(self, state):
        probs, _ = self.forward(state)
        dist = torch.distributions.Categorical(probs)
        return dist.sample().item()

For discrete SAC, the policy loss becomes:

# Instead of sampling one action, take expectation over all actions
q_values = torch.min(q1(states), q2(states))  # [batch, n_actions]
policy_loss = (probs * (alpha * log_probs - q_values)).sum(dim=-1).mean()

Series Recap

Over 5 parts, we’ve built up from fundamentals to state-of-the-art:

Part	Algorithm	Key Insight
1	Value Iteration	Bellman equation makes RL tractable
2	Q-Learning → DQN	Neural networks scale to high dimensions
3	REINFORCE → A2C	Direct policy optimization enables continuous actions
4	PPO	Clipping prevents catastrophic updates
5	SAC	Maximum entropy gives robust, sample-efficient learning

The field keeps evolving — Decision Transformers, offline RL, world models — but these five algorithms form the foundation that everything else builds upon.

FAQ

What does “soft” mean in Soft Actor-Critic?

“Soft” refers to the maximum entropy objective — instead of finding a single optimal action (hard), SAC finds a distribution that’s as random as possible while still being near-optimal (soft). Mathematically, it adds an entropy bonus H(π) to the reward, encouraging stochastic policies. This is the key differentiator from earlier actor-critic methods like DDPG or TD3.

Why does SAC use two Q-networks instead of one?

Twin Q-networks address the overestimation bias inherited from Q-learning. When a single Q-network overestimates the value of some actions (which happens frequently due to function approximation errors), the policy exploits these errors. By taking the minimum of two independent Q estimates, SAC provides a more conservative, accurate value estimate. This idea was first introduced in TD3 and adopted by SAC.

Can I use SAC for problems with image observations?

Yes. Replace the MLP feature extractors with convolutional networks for the Q-networks and policy. The rest of the algorithm stays the same. For pixel-based control, also consider adding data augmentation (DrQ, CURL) — random crops of the input image significantly improve sample efficiency and are nearly free computationally.

Did you find this helpful?

Your support keeps this blog running and ad-free content coming.

☕ Buy me a coffee