.rl-series-nav{margin:2rem 0;padding:1.5rem;border:1px solid #e2e8f0;border-radius:10px;background:#f8fafc}.rl-series-nav h4{margin:0 0 .8rem;color:#6c5ce7;font-size:1.1rem}.rl-series-nav ul{list-style:none;padding:0;margin:0}.rl-series-nav li{padding:.3rem 0}.rl-series-nav a{color:#4a5568;text-decoration:none;font-size:.9rem}.rl-series-nav a:hover{color:#6c5ce7}.rl-series-nav .current{font-weight:700;color:#6c5ce7}
Reinforcement Learning Series
This is Part 2 of our 5-part Reinforcement Learning series. We’re moving from theory to our first real algorithms — Q-Learning and Deep Q-Networks.
Series Overview:
– Part 1: RL Basics — MDP, Bellman Equation, Value Functions
– Part 2: Q-Learning to DQN (You are here)
– Part 3: Policy Gradient Methods
– Part 4: PPO — The Industry Standard
– Part 5: SAC — Mastering Continuous Control
From Values to Actions: Q-Learning
In Part 1, we used value iteration — but that requires knowing the environment’s transition probabilities P(s’|s,a). In most real problems, we don’t have that. We need to learn from experience.
Q-Learning solves this by directly learning the action-value function Q(s,a) through interaction:
Q(s, a) ← Q(s, a) + α [ r + γ · max_a' Q(s', a') - Q(s, a) ]
Breaking this down:
– α (learning rate): How much to update per step (0.01–0.1)
– r + γ · max Q(s’, a’): The TD target — what we think Q should be
– Q(s,a) – target: The TD error — how wrong we were
The beauty of Q-Learning is that it’s off-policy — it learns the optimal policy regardless of what exploration strategy you use.
Tabular Q-Learning Implementation
Let’s implement Q-Learning on our GridWorld from Part 1:
import numpy as np
import random
class GridWorld:
def __init__(self, size=5):
self.size = size
self.goal = (size-1, size-1)
self.traps = [(1, 1), (2, 3)]
self.actions = {0: (-1,0), 1: (1,0), 2: (0,-1), 3: (0,1)}
self.state = (0, 0)
def reset(self):
self.state = (0, 0)
return self.state
def step(self, action):
dr, dc = self.actions[action]
r = max(0, min(self.size-1, self.state[0] + dr))
c = max(0, min(self.size-1, self.state[1] + dc))
self.state = (r, c)
if self.state == self.goal:
return self.state, 10.0, True
if self.state in self.traps:
return self.state, -10.0, True
return self.state, -0.1, False
def q_learning(env, episodes=2000, alpha=0.1, gamma=0.99, epsilon_start=1.0, epsilon_end=0.01):
"""Tabular Q-Learning algorithm."""
Q = np.zeros((env.size, env.size, 4))
rewards_history = []
for ep in range(episodes):
state = env.reset()
total_reward = 0
epsilon = max(epsilon_end, epsilon_start - ep / (episodes * 0.8))
for _ in range(200): # Max steps per episode
# Epsilon-greedy action selection
if random.random() < epsilon:
action = random.randint(0, 3)
else:
action = np.argmax(Q[state[0], state[1]])
next_state, reward, done = env.step(action)
total_reward += reward
# Q-Learning update
r, c = state
nr, nc = next_state
best_next = np.max(Q[nr, nc]) if not done else 0
Q[r, c, action] += alpha * (reward + gamma * best_next - Q[r, c, action])
state = next_state
if done:
break
rewards_history.append(total_reward)
return Q, rewards_history
env = GridWorld(size=5)
Q, rewards = q_learning(env)
# Print learned policy
action_symbols = ['↑', '↓', '←', '→']
print("Learned Policy:")
for r in range(env.size):
row = []
for c in range(env.size):
if (r, c) == env.goal:
row.append(' ★')
elif (r, c) in env.traps:
row.append(' ✕')
else:
row.append(f' {action_symbols[np.argmax(Q[r, c])]}')
print(''.join(row))
# Print average reward over last 100 episodes
print(f"nAvg reward (last 100): {np.mean(rewards[-100:]):.2f}")
Training deep Q-networks at 2am? Dark Chocolate Espresso Beans are your non-negotiable debugging fuel.
The Problem with Tables
Tabular Q-Learning works great for small state spaces. But what happens when states are continuous or high-dimensional? A 210×160 pixel Atari screen has more possible states than atoms in the universe.
Solution: Use a neural network to approximate Q(s, a).
This is the core idea behind DQN.
Deep Q-Network (DQN)
DeepMind’s 2015 DQN paper was a breakthrough — the same algorithm learned to play 49 different Atari games at superhuman level. Two key innovations made it work:
1. Experience Replay
Instead of learning from consecutive experiences (which are correlated), DQN stores transitions in a replay buffer and samples random mini-batches:
from collections import deque
class ReplayBuffer:
def __init__(self, capacity=10000):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (
np.array(states),
np.array(actions),
np.array(rewards, dtype=np.float32),
np.array(next_states),
np.array(dones, dtype=np.float32),
)
def __len__(self):
return len(self.buffer)
Why it helps:
– Breaks correlation between consecutive samples
– Each experience can be reused multiple times
– More sample-efficient learning
2. Target Network
Using the same network to compute both current Q-values and targets creates a moving target problem. DQN uses a separate target network that’s updated periodically:
# Every N steps, copy main network weights to target network
target_net.load_state_dict(main_net.state_dict())
Full DQN Implementation with PyTorch
Here’s a complete DQN implementation for CartPole:
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import random
from collections import deque
import gymnasium as gym
class QNetwork(nn.Module):
"""Simple feedforward Q-network."""
def __init__(self, state_dim, action_dim, hidden=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(state_dim, hidden),
nn.ReLU(),
nn.Linear(hidden, hidden),
nn.ReLU(),
nn.Linear(hidden, action_dim),
)
def forward(self, x):
return self.net(x)
class DQNAgent:
def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99,
buffer_size=10000, batch_size=64, target_update=100):
self.action_dim = action_dim
self.gamma = gamma
self.batch_size = batch_size
self.target_update = target_update
self.step_count = 0
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
self.q_net = QNetwork(state_dim, action_dim).to(self.device)
self.target_net = QNetwork(state_dim, action_dim).to(self.device)
self.target_net.load_state_dict(self.q_net.state_dict())
self.optimizer = optim.Adam(self.q_net.parameters(), lr=lr)
self.buffer = deque(maxlen=buffer_size)
def select_action(self, state, epsilon):
if random.random() < epsilon:
return random.randint(0, self.action_dim - 1)
with torch.no_grad():
state_t = torch.FloatTensor(state).unsqueeze(0).to(self.device)
return self.q_net(state_t).argmax(dim=1).item()
def store(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def train_step(self):
if len(self.buffer) < self.batch_size:
return None
# Sample mini-batch
batch = random.sample(self.buffer, self.batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
states = torch.FloatTensor(np.array(states)).to(self.device)
actions = torch.LongTensor(actions).to(self.device)
rewards = torch.FloatTensor(rewards).to(self.device)
next_states = torch.FloatTensor(np.array(next_states)).to(self.device)
dones = torch.FloatTensor(dones).to(self.device)
# Current Q values
current_q = self.q_net(states).gather(1, actions.unsqueeze(1)).squeeze()
# Target Q values (from target network)
with torch.no_grad():
next_q = self.target_net(next_states).max(dim=1)[0]
target_q = rewards + self.gamma * next_q * (1 - dones)
# Update
loss = nn.MSELoss()(current_q, target_q)
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
# Update target network periodically
self.step_count += 1
if self.step_count % self.target_update == 0:
self.target_net.load_state_dict(self.q_net.state_dict())
return loss.item()
def train_dqn():
env = gym.make("CartPole-v1")
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
agent = DQNAgent(state_dim, action_dim)
episodes = 500
epsilon_start, epsilon_end = 1.0, 0.01
rewards_history = []
for ep in range(episodes):
state, _ = env.reset()
total_reward = 0
epsilon = max(epsilon_end, epsilon_start - ep / (episodes * 0.7))
while True:
action = agent.select_action(state, epsilon)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
agent.store(state, action, reward, next_state, float(done))
agent.train_step()
state = next_state
total_reward += reward
if done:
break
rewards_history.append(total_reward)
if (ep + 1) % 50 == 0:
avg = np.mean(rewards_history[-50:])
print(f"Episode {ep+1} | Avg Reward: {avg:.1f} | ε: {epsilon:.3f}")
env.close()
return agent, rewards_history
agent, rewards = train_dqn()
# Episode 50 | Avg Reward: 22.3 | ε: 0.857
# Episode 100 | Avg Reward: 38.5 | ε: 0.714
# Episode 200 | Avg Reward: 142.7 | ε: 0.429
# Episode 300 | Avg Reward: 389.2 | ε: 0.143
# Episode 500 | Avg Reward: 500.0 | ε: 0.010
DQN Variants: What Came After
The original DQN spawned several improvements:
| Variant | Key Idea | Improvement |
|---|---|---|
| Double DQN | Use main net to select action, target net to evaluate | Reduces overestimation |
| Dueling DQN | Separate value and advantage streams | Better state evaluation |
| Prioritized Replay | Sample important transitions more often | Faster learning |
| Rainbow | Combine all improvements | State-of-the-art for Atari |
Double DQN — A Quick Fix
Standard DQN overestimates Q-values because max is biased. Double DQN decouples selection and evaluation:
# Standard DQN target (overestimates)
next_q = self.target_net(next_states).max(dim=1)[0]
# Double DQN target (more accurate)
next_actions = self.q_net(next_states).argmax(dim=1, keepdim=True)
next_q = self.target_net(next_states).gather(1, next_actions).squeeze()
Just two lines changed, but it significantly reduces overestimation bias.
When to Use DQN (and When Not To)
DQN works well for:
– Discrete action spaces (Atari, board games, routing)
– Problems where you can collect lots of experience
– Environments with clear reward signals
DQN struggles with:
– Continuous actions (robot arm angles, throttle) — can’t do max over infinite actions
– Sparse rewards — needs many samples to propagate reward signal
– Multi-agent settings — non-stationary environment breaks assumptions
For continuous actions, we need policy gradient methods — that’s Part 3.
Key Takeaways
- Q-Learning learns action values directly from experience, no model needed
- Experience Replay breaks sample correlation and improves efficiency
- Target Networks stabilize training by providing consistent targets
- DQN scales Q-Learning to high-dimensional states with neural networks
- Double DQN is a simple fix for Q-value overestimation
What’s Next
In Part 3, we’ll learn Policy Gradient methods — instead of learning values and deriving a policy, we’ll directly optimize the policy itself. This opens the door to continuous action spaces and will lead us to Actor-Critic methods.
FAQ
Why is Q-Learning called “off-policy”?
Q-Learning updates use max Q(s', a') — the value of the best possible action — regardless of what action the agent actually took during exploration. This means the learning target follows the optimal policy even while the behavior policy explores randomly. SARSA, by contrast, is on-policy because it uses the action actually taken.
How large should the replay buffer be?
For most problems, 10K–100K transitions work well. Too small and you lose diversity. Too large and you waste memory while also training on very old (potentially irrelevant) experiences. For Atari-scale problems, DeepMind used 1M transitions. Start with 10K and increase if training is unstable.
Can DQN handle continuous action spaces?
Not directly. DQN requires computing max_a Q(s,a), which is easy for discrete actions (just compare N values) but intractable for continuous spaces. Workarounds include discretizing the action space (loses precision) or using algorithms designed for continuous control like DDPG, TD3, or SAC (covered in Part 5).
Did you find this helpful?
Your support keeps this blog running and ad-free content coming.
☕ Buy me a coffeeMost Popular Posts
- Custom Metaclass in Python: 43% Faster Validation (12,793 views)
- Python match-case: 7 Patterns That Beat if-elif Chains (947 views)
- YOLOv8 INT8 Quantization: 4x Faster on Jetson Orin (759 views)
- yfinance Alternatives 2026: 7 Free APIs Compared (649 views)
- PaddleOCR vs EasyOCR vs Tesseract: Why PaddleOCR Is Slower (549 views)