- DQN learns Q-values through Bellman backups with a target network for stability; PPO directly optimizes the policy with clipped surrogate objectives.
- GAE (lambda=0.95) dramatically reduces advantage variance, but requires careful handling of episode boundaries to avoid bugs.
- The target network update frequency in DQN has a larger impact on training stability than most tutorials suggest—1000 steps works better than 100.
- Entropy coefficient in PPO is environment-specific: 0.01 for CartPole, 0.001 for LunarLander.
- PPO converges faster on CartPole (120 vs 180 episodes) but DQN is more sample-efficient due to experience replay.
800 Lines Later, I Finally Understood Policy Gradients
Most RL tutorials show you how to use Stable Baselines3. That’s great for getting results fast, but it’s terrible for understanding what’s actually happening when your agent refuses to learn. I spent months using PPO("MlpPolicy", env) like a magic incantation before finally deciding to implement both DQN and PPO from scratch in a single, minimal codebase.
The result: SimpleRL, a ~500-line library that implements both algorithms with enough shared infrastructure to see exactly how they differ. Building it broke nearly every assumption I had about reinforcement learning.
Why Build Another RL Library?
This isn’t about creating something production-ready. Stable Baselines3 exists. CleanRL exists. The point is pedagogical: when you implement the Bellman backup yourself, when you compute the GAE advantage yourself, the equations stop being abstract and become debugging targets.
What I wanted:
– Shared replay buffer and environment wrappers between algorithms
– Identical network architectures where possible
– Side-by-side training on the same seeds
– Under 600 total lines including both algorithms
What I learned: DQN and PPO share almost nothing except the environment interface. They’re philosophically different approaches to the same problem.
DQN: Where Value Functions Meet Neural Networks
DQN (Mnih et al., Nature 2015) approximates the optimal action-value function using a neural network. The core idea is deceptively simple: use the Bellman equation as a training target.
The Q-learning update rule:
With function approximation, this becomes a regression problem. The loss:
Here is the target network—a copy of the Q-network that gets updated slowly. Without it, the targets keep moving and training diverges. I didn’t believe this mattered until I removed it.
import torch
import torch.nn as nn
import numpy as np
from collections import deque
import random
class QNetwork(nn.Module):
def __init__(self, obs_dim, act_dim, hidden=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(obs_dim, hidden),
nn.ReLU(),
nn.Linear(hidden, hidden),
nn.ReLU(),
nn.Linear(hidden, act_dim)
)
def forward(self, x):
return self.net(x)
class ReplayBuffer:
def __init__(self, capacity=100000):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (
torch.FloatTensor(np.array(states)),
torch.LongTensor(actions),
torch.FloatTensor(rewards),
torch.FloatTensor(np.array(next_states)),
torch.FloatTensor(dones)
)
def __len__(self):
return len(self.buffer)
The training loop is where things get interesting:
class DQNAgent:
def __init__(self, obs_dim, act_dim, lr=1e-4, gamma=0.99,
epsilon_start=1.0, epsilon_end=0.01, epsilon_decay=0.995,
target_update_freq=1000):
self.q_net = QNetwork(obs_dim, act_dim)
self.target_net = QNetwork(obs_dim, act_dim)
self.target_net.load_state_dict(self.q_net.state_dict())
self.optimizer = torch.optim.Adam(self.q_net.parameters(), lr=lr)
self.buffer = ReplayBuffer()
self.gamma = gamma
self.epsilon = epsilon_start
self.epsilon_end = epsilon_end
self.epsilon_decay = epsilon_decay
self.target_update_freq = target_update_freq
self.act_dim = act_dim
self.steps = 0
def select_action(self, state, training=True):
if training and random.random() < self.epsilon:
return random.randrange(self.act_dim)
with torch.no_grad():
state_t = torch.FloatTensor(state).unsqueeze(0)
q_values = self.q_net(state_t)
return q_values.argmax(dim=1).item()
def update(self, batch_size=64):
if len(self.buffer) < batch_size:
return None
states, actions, rewards, next_states, dones = self.buffer.sample(batch_size)
# Current Q values
q_values = self.q_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
# Target Q values - this is where the magic happens
with torch.no_grad():
next_q_values = self.target_net(next_states).max(dim=1)[0]
targets = rewards + self.gamma * next_q_values * (1 - dones)
loss = nn.MSELoss()(q_values, targets)
self.optimizer.zero_grad()
loss.backward()
# Gradient clipping helps stability
torch.nn.utils.clip_grad_norm_(self.q_net.parameters(), 10.0)
self.optimizer.step()
# Update target network periodically
self.steps += 1
if self.steps % self.target_update_freq == 0:
self.target_net.load_state_dict(self.q_net.state_dict())
# Decay epsilon
self.epsilon = max(self.epsilon_end, self.epsilon * self.epsilon_decay)
return loss.item()
The Target Network Experiment
I was skeptical about whether the target network really mattered that much. So I ran CartPole-v1 with and without it (Gymnasium 0.29.1, seed 42):
| Configuration | Episodes to Solve | Stability |
|---|---|---|
| With target network (update every 1000 steps) | 180 | Stable |
| Without target network | 450+ | Oscillates wildly |
| Target network (update every 100 steps) | 220 | Minor oscillations |
The version without a target network eventually solves the environment, but the Q-values oscillate between 50 and 200 before settling. With the target network, they climb smoothly to around 150 and stay there.
My best guess for why: without the target network, you’re chasing a moving target. The max Q-value in the next state changes every gradient step, so you’re essentially doing regression against a dataset that keeps getting relabeled.
From Values to Policies: The PPO Paradigm Shift
PPO (Schulman et al., 2017) takes a completely different approach. Instead of learning which actions are valuable and then acting greedily, it directly optimizes the policy .
The policy gradient theorem tells us:
where is the advantage function—how much better action is compared to the average. PPO’s key insight is constraining how much the policy can change in a single update to avoid catastrophic policy collapse.
The clipped surrogate objective:
where is the probability ratio.
This looks more complicated than DQN, but the implementation reveals why PPO is actually more stable:
class ActorCritic(nn.Module):
def __init__(self, obs_dim, act_dim, hidden=64):
super().__init__()
# Shared feature extractor
self.shared = nn.Sequential(
nn.Linear(obs_dim, hidden),
nn.Tanh(), # Tanh works better than ReLU for policy networks
)
# Separate heads
self.actor = nn.Sequential(
nn.Linear(hidden, hidden),
nn.Tanh(),
nn.Linear(hidden, act_dim)
)
self.critic = nn.Sequential(
nn.Linear(hidden, hidden),
nn.Tanh(),
nn.Linear(hidden, 1)
)
def forward(self, x):
features = self.shared(x)
return self.actor(features), self.critic(features)
def get_action(self, state, deterministic=False):
logits, value = self.forward(state)
dist = torch.distributions.Categorical(logits=logits)
if deterministic:
action = logits.argmax(dim=-1)
else:
action = dist.sample()
return action, dist.log_prob(action), value.squeeze(-1)
GAE: The Advantage Estimation That Actually Works
Computing advantages correctly is where I burned the most time. The naive approach—just use returns minus value estimates—has high variance. Generalized Advantage Estimation (GAE) from Schulman et al. (2015) fixes this with an exponentially-weighted average of n-step advantages:
where is the TD residual.
In practice:
def compute_gae(rewards, values, dones, gamma=0.99, gae_lambda=0.95):
"""Compute Generalized Advantage Estimation.
This implementation goes backwards through the trajectory,
which is more numerically stable than the forward pass.
"""
advantages = []
gae = 0
# Need value of terminal state (0 if done, bootstrap otherwise)
# Assuming values includes V(s_T) at the end
next_value = values[-1] if not dones[-1] else 0
for t in reversed(range(len(rewards))):
if t == len(rewards) - 1:
next_val = next_value
else:
next_val = values[t + 1] if not dones[t] else 0
delta = rewards[t] + gamma * next_val - values[t]
gae = delta + gamma * gae_lambda * gae * (1 - dones[t])
advantages.insert(0, gae)
return advantages
The parameter controls the bias-variance tradeoff. With , you get one-step TD (low variance, high bias). With , you get Monte Carlo returns (high variance, low bias). I’ve found works well for most environments, but LunarLander specifically seems to prefer .
The Full PPO Implementation
class PPOAgent:
def __init__(self, obs_dim, act_dim, lr=3e-4, gamma=0.99,
gae_lambda=0.95, clip_epsilon=0.2, epochs=10,
entropy_coef=0.01, value_coef=0.5):
self.network = ActorCritic(obs_dim, act_dim)
self.optimizer = torch.optim.Adam(self.network.parameters(), lr=lr)
self.gamma = gamma
self.gae_lambda = gae_lambda
self.clip_epsilon = clip_epsilon
self.epochs = epochs
self.entropy_coef = entropy_coef
self.value_coef = value_coef
# Trajectory storage
self.states = []
self.actions = []
self.log_probs = []
self.rewards = []
self.values = []
self.dones = []
def select_action(self, state):
state_t = torch.FloatTensor(state).unsqueeze(0)
with torch.no_grad():
action, log_prob, value = self.network.get_action(state_t)
self.states.append(state)
self.actions.append(action.item())
self.log_probs.append(log_prob.item())
self.values.append(value.item())
return action.item()
def store_outcome(self, reward, done):
self.rewards.append(reward)
self.dones.append(done)
def update(self):
# Convert to tensors
states = torch.FloatTensor(np.array(self.states))
actions = torch.LongTensor(self.actions)
old_log_probs = torch.FloatTensor(self.log_probs)
# Compute advantages
advantages = compute_gae(
self.rewards, self.values, self.dones,
self.gamma, self.gae_lambda
)
advantages = torch.FloatTensor(advantages)
# Compute returns (advantages + values)
returns = advantages + torch.FloatTensor(self.values)
# Normalize advantages (this matters a lot!)
advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
# PPO update
total_loss = 0
for _ in range(self.epochs):
logits, values = self.network(states)
dist = torch.distributions.Categorical(logits=logits)
new_log_probs = dist.log_prob(actions)
entropy = dist.entropy().mean()
# Policy loss with clipping
ratio = torch.exp(new_log_probs - old_log_probs)
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1 - self.clip_epsilon,
1 + self.clip_epsilon) * advantages
policy_loss = -torch.min(surr1, surr2).mean()
# Value loss
value_loss = nn.MSELoss()(values.squeeze(-1), returns)
# Combined loss
loss = (policy_loss +
self.value_coef * value_loss -
self.entropy_coef * entropy)
self.optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.network.parameters(), 0.5)
self.optimizer.step()
total_loss += loss.item()
# Clear trajectory
self.states.clear()
self.actions.clear()
self.log_probs.clear()
self.rewards.clear()
self.values.clear()
self.dones.clear()
return total_loss / self.epochs
The Hyperparameters That Destroyed Me
I’ve covered PPO hyperparameter disasters in PPO Hyperparameters That Crash in Production, but building from scratch revealed additional failure modes.
Entropy coefficient is the sneakiest. Too low (0.0) and your policy collapses to always picking the same action. Too high (0.1) and it never commits to anything. The default 0.01 works for CartPole, but LunarLander needs 0.001 or the lander just fires thrusters randomly forever.
Clip epsilon at 0.2 is surprisingly robust. I tried 0.1 and 0.3—both worked, just slower. But 0.4 caused training instability, and 0.05 was too conservative to make progress.
The update epochs caught me off guard. Running 10 epochs per batch works fine when your batch is 2048 timesteps. With 128 timesteps? The policy changes too much and you get the “PPO death spiral” where the ratio explodes and gradients become garbage.
DQN vs PPO: Head-to-Head on CartPole-v1
Here’s the training code to compare both:
import gymnasium as gym
def train_dqn(env_name="CartPole-v1", episodes=500, seed=42):
env = gym.make(env_name)
env.reset(seed=seed)
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
agent = DQNAgent(
obs_dim=env.observation_space.shape[0],
act_dim=env.action_space.n
)
rewards_history = []
for ep in range(episodes):
state, _ = env.reset()
episode_reward = 0
while True:
action = agent.select_action(state)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
agent.buffer.push(state, action, reward, next_state, float(done))
agent.update()
state = next_state
episode_reward += reward
if done:
break
rewards_history.append(episode_reward)
if ep % 50 == 0:
avg = np.mean(rewards_history[-50:])
print(f"DQN Episode {ep}: avg_reward={avg:.1f}, epsilon={agent.epsilon:.3f}")
return rewards_history
def train_ppo(env_name="CartPole-v1", episodes=500, seed=42):
env = gym.make(env_name)
env.reset(seed=seed)
torch.manual_seed(seed)
np.random.seed(seed)
agent = PPOAgent(
obs_dim=env.observation_space.shape[0],
act_dim=env.action_space.n
)
rewards_history = []
steps_since_update = 0
update_interval = 2048 # Collect this many steps before updating
for ep in range(episodes):
state, _ = env.reset()
episode_reward = 0
while True:
action = agent.select_action(state)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
agent.store_outcome(reward, done)
steps_since_update += 1
if steps_since_update >= update_interval:
agent.update()
steps_since_update = 0
state = next_state
episode_reward += reward
if done:
break
rewards_history.append(episode_reward)
if ep % 50 == 0:
avg = np.mean(rewards_history[-50:])
print(f"PPO Episode {ep}: avg_reward={avg:.1f}")
return rewards_history
Results on my M1 MacBook (PyTorch 2.1, Gymnasium 0.29.1):
| Metric | DQN | PPO |
|---|---|---|
| Episodes to 475+ avg reward | 180 | 120 |
| Peak reward stability | Good | Excellent |
| Training time (500 episodes) | 45s | 28s |
| Sensitive hyperparameters | epsilon decay, target freq | batch size, epochs |
PPO wins on CartPole, but the gap narrows on harder environments. On LunarLander-v2, DQN actually converges faster to a decent policy, though PPO eventually achieves higher final performance.
What Broke Along the Way
The bugs I encountered were educational:
Bug 1: Forgetting to detach the target network computation. Without torch.no_grad() around the target calculation, gradients flow through the target network and everything breaks silently. The loss looks reasonable but Q-values diverge.
Bug 2: Advantage normalization. Skipping (advantages - mean) / std in PPO doesn’t immediately break training, but the policy updates become erratic. Sometimes it helps, sometimes it hurts—classic high-variance nightmare.
Bug 3: The GAE done mask. When a trajectory includes multiple episodes (done flags in the middle), you need to reset the GAE accumulator at episode boundaries. I spent an hour wondering why my returns were negative before catching this.
# Wrong - GAE bleeds across episode boundaries
gae = delta + gamma * gae_lambda * gae
# Right - reset at done
gae = delta + gamma * gae_lambda * gae * (1 - dones[t])
Bug 4: Not clipping gradients. Both algorithms benefit from gradient clipping, but for different reasons. DQN can have exploding gradients when Q-values grow large. PPO can have them when the probability ratio spikes.
When to Use Which Algorithm
After building both, my heuristics:
Use DQN when:
– You have discrete actions (it doesn’t naturally extend to continuous)
– Sample efficiency matters (it reuses experience via replay)
– You want simpler debugging (Q-values are interpretable)
Use PPO when:
– You have continuous actions (just change the distribution)
– Stability matters more than sample efficiency
– You’re okay with on-policy data (no replay buffer)
But honestly? For most practical applications, I’d use Stable Baselines3’s implementations and spend my time on reward engineering instead. Building SimpleRL was about understanding, not production use.
FAQ
Q: Can I extend SimpleRL to continuous action spaces?
For PPO, swap the Categorical distribution for a Gaussian: output mean and log_std from the actor, then sample using torch.distributions.Normal. DQN doesn’t naturally support continuous actions—you’d need DDPG or SAC instead, which use different approaches entirely.
Q: Why does PPO need so many more samples per update than DQN?
PPO is on-policy: it can only learn from data collected by the current policy. Once you update, old data becomes stale because the probability ratios diverge. DQN stores everything in a replay buffer and reuses it, making each environment step count multiple times.
Q: Is SimpleRL fast enough for serious experimentation?
No. CleanRL’s implementations are 2-3x faster due to vectorized environments and better batching. SimpleRL is optimized for readability, not throughput. If you’re running hyperparameter sweeps, use a proper implementation.
The Surprising Takeaway
Building both algorithms side by side revealed something I didn’t expect: they’re solving fundamentally different optimization problems. DQN approximates a fixed point of the Bellman operator. PPO performs constrained policy optimization. The fact that both work on the same environments is almost coincidental.
The next rabbit hole I want to explore is offline RL—what happens when you can’t collect new data and have to learn purely from a fixed dataset. CQL and IQL supposedly handle this, but after this experience, I don’t trust that I understand them until I’ve implemented them from scratch.
Start with DQN if you want to understand value functions. Start with PPO if you want to understand policy optimization. Build both if you want to understand why choosing between them isn’t always obvious.
Full source code is available at DrunkJin/simpleRL — DQN, REINFORCE, DDPG, TD3, SAC, PPO with training result plots.
Did you find this helpful?
Your support keeps this blog running and ad-free content coming.
☕ Buy me a coffeeMost Popular Posts
- Custom Metaclass in Python: 43% Faster Validation (12,795 views)
- Python match-case: 7 Patterns That Beat if-elif Chains (947 views)
- YOLOv8 INT8 Quantization: 4x Faster on Jetson Orin (763 views)
- yfinance Alternatives 2026: 7 Free APIs Compared (654 views)
- PaddleOCR vs EasyOCR vs Tesseract: Why PaddleOCR Is Slower (550 views)