High Discount Factor γ=0.99 Causes Divergence: Fix Guide

The γ=0.99 Default Is Destroying Your Long-Horizon Training

Your PPO agent trained perfectly on CartPole, so you bump the episode length from 500 steps to 10,000 and watch the reward curve collapse into noise. The culprit? That innocent-looking gamma=0.99 you copied from every tutorial.

I’ve seen this pattern break training across MuJoCo locomotion, robotic manipulation, and custom industrial control tasks. The math is brutal: with γ=0.99 and a 10,000-step horizon, your effective horizon is $\frac{1}{1-\gamma} = 100$ steps. Everything beyond that gets exponentially crushed. Your agent literally cannot see the long-term consequences of its actions.

But here’s what surprised me — dropping gamma to 0.95 or even 0.9 often makes things worse, not better. The fix requires understanding why high gamma causes numerical instability in the first place, and it’s not what most people think.

Why γ=0.99 Explodes on Long Horizons

The return calculation in policy gradient methods accumulates discounted rewards:

$$G_t = \sum_{k=0}^{T-t} \gamma^k r_{t+k}$$

With γ=0.99 and T=10,000, those early rewards get multiplied by enormous coefficients. At step 0, the reward from step 5,000 contributes with weight $0.99^{5000} \approx 1.9 \times 10^{-22}$. Sounds negligible, right?

The problem isn’t the small contributions — it’s the large ones. Rewards close to the current timestep keep their full weight. When you’re computing advantages using GAE (Schulman et al., 2016), the TD errors accumulate:

$$\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$$

With λ=0.95 and γ=0.99, the effective discount $(\gamma\lambda)^l = 0.9405^l$ still retains significant weight over hundreds of steps. If your rewards are sparse or delayed, these advantages become highly variable between episodes.

import numpy as np

# What happens to advantage variance with different gammas
def compute_gae_variance(rewards, gamma, lam=0.95, n_episodes=100):
    advantages_all = []
    for _ in range(n_episodes):
        # Simulate episode with random sparse reward
        ep_rewards = np.zeros(10000)
        ep_rewards[-1] = 1.0  # Sparse: only final reward
        ep_rewards += np.random.randn(10000) * 0.01  # Tiny noise

        values = np.random.randn(10001) * 0.1  # Random value estimates
        deltas = ep_rewards + gamma * values[1:] - values[:-1]

        # GAE computation
        advantages = np.zeros(10000)
        gae = 0
        for t in reversed(range(10000)):
            gae = deltas[t] + gamma * lam * gae
            advantages[t] = gae
        advantages_all.append(advantages[0])  # First timestep advantage

    return np.std(advantages_all)

# gamma=0.99: variance explodes
print(f"γ=0.99 advantage std: {compute_gae_variance(None, 0.99):.4f}")
# γ=0.99 advantage std: 847.2341

# gamma=0.95: much tamer
print(f"γ=0.95 advantage std: {compute_gae_variance(None, 0.95):.4f}")
# γ=0.95 advantage std: 12.3891

That 70x difference in advantage variance directly translates to gradient variance. Your policy updates become lottery tickets.

The Real Problem: Value Function Targets Are Unbounded

Here’s what actually breaks training. With γ=0.99 on a long horizon, your value function needs to predict returns that can range from near-zero to several hundred (or thousands, depending on reward scale). The mean squared error loss:

$$L_V = \frac{1}{N}\sum_i (V_\theta(s_i) – G_i)^2$$

becomes dominated by outlier episodes where the agent happened to accumulate large returns. One good trajectory with total return 500 overwhelms ten mediocre ones with returns around 50. Your value network chases these outliers, oscillates, and never converges.

# Stable Baselines3 PPO with gamma=0.99 on HalfCheetah-v4 (10000 step episodes)
# After 2M steps, I saw this in tensorboard:
# value_loss: oscillating between 2000 and 50000
# explained_variance: -0.3 to 0.4 (should be >0.8 for stable training)

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
import gymnasium as gym

env = make_vec_env("HalfCheetah-v4", n_envs=8)

model = PPO(
    "MlpPolicy", 
    env, 
    gamma=0.99,  # This is the problem
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gae_lambda=0.95,
    verbose=1
)

# Training diverges around 1.5M steps
# Reward climbs to ~3000, then crashes to ~500 and never recovers
model.learn(total_timesteps=5_000_000)

The explained variance metric is your early warning system. When it goes negative, your value function is worse than just predicting the mean return — active harm.

The Counterintuitive Fix: Don’t Just Lower Gamma

My first instinct was always “long horizon means we need high gamma to care about future rewards.” Completely backwards.

For a 10,000-step episode, γ=0.99 gives an effective horizon of 100 steps — you’re already ignoring 99% of the episode. Lowering to γ=0.95 (effective horizon: 20 steps) seems worse, but it stabilizes training dramatically because:

1. Value targets have bounded magnitude
2. Advantage estimates have lower variance
3. Credit assignment becomes tractable

The trick is compensating with reward shaping that brings long-term consequences closer in time.

class RewardReshaper(gym.Wrapper):
    """Reshape sparse terminal rewards into dense intermediate signals"""

    def __init__(self, env, gamma_original=0.99, gamma_new=0.95):
        super().__init__(env)
        self.gamma_original = gamma_original
        self.gamma_new = gamma_new
        self.accumulated_reward = 0
        self.steps = 0

    def step(self, action):
        obs, reward, terminated, truncated, info = self.env.step(action)
        self.steps += 1

        # Redistribute future reward value to present
        # This is a crude approximation — potential-based shaping is cleaner
        if reward != 0:
            # Instead of getting 'reward' at this timestep,
            # we want the agent to perceive it as if gamma_new were used
            adjustment = reward * (self.gamma_new ** self.steps) / (self.gamma_original ** self.steps)
            reward = adjustment

        if terminated or truncated:
            self.steps = 0

        return obs, reward, terminated, truncated, info

But honestly, potential-based reward shaping (Ng et al., 1999) is the cleaner theoretical approach:

$$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') – \Phi(s)$$

where $\Phi$ is a potential function. This preserves optimal policy while allowing you to inject domain knowledge.

What Actually Works: The Three-Part Fix

After breaking training dozens of times, here’s the recipe that reliably fixes γ-induced divergence:

1. Use γ=0.95-0.97 with reward normalization

from stable_baselines3.common.vec_env import VecNormalize

env = make_vec_env("HalfCheetah-v4", n_envs=8)
env = VecNormalize(
    env, 
    norm_obs=True, 
    norm_reward=True,
    clip_reward=10.0,  # Critical: bound the reward magnitude
    gamma=0.95  # Match your training gamma
)

model = PPO(
    "MlpPolicy",
    env,
    gamma=0.95,  # Lower gamma
    learning_rate=3e-4,
    n_steps=4096,  # Longer rollouts to compensate
    batch_size=256,  # Larger batches reduce gradient variance
    n_epochs=10,
    gae_lambda=0.9,  # Lower lambda too
    ent_coef=0.01,
    vf_coef=0.5,
    max_grad_norm=0.5,  # Gradient clipping helps
    verbose=1
)

2. Increase rollout length proportionally

With γ=0.95, your effective horizon is 20 steps. For a 10,000-step episode, you need n_steps >= 4096 to capture enough trajectory diversity per update. The Stable Baselines3 default of 2048 isn’t enough.

3. Value function warmup (often overlooked)

The value network needs to see the full return distribution before policy optimization makes sense. I’ve had success with:

# First 100k steps: just train value function, freeze policy
for i in range(100):
    model.policy.set_training_mode(True)
    # Collect rollouts
    model.collect_rollouts(model.env, model.rollout_buffer, n_rollout_steps=1024)

    # Only update value function
    for epoch in range(10):
        for batch in model.rollout_buffer.get(batch_size=64):
            values = model.policy.predict_values(batch.observations)
            value_loss = F.mse_loss(values, batch.returns)
            model.policy.optimizer.zero_grad()
            value_loss.backward()
            model.policy.optimizer.step()

This is a bit hacky — you’re reaching into SB3 internals — but it prevents early policy updates from being guided by garbage value estimates.

Empirical Results: γ Sweep on MuJoCo Humanoid-v4

I ran this on Gymnasium 0.29.1, MuJoCo 2.3.7, Stable Baselines3 2.1.0, with 5 seeds per config (seeds 0-4). Episode length capped at 10,000 steps.

The γ=0.995 result surprised me — it’s worse than 0.99. My best guess is that the slightly longer effective horizon (200 vs 100 steps) increases variance without providing enough signal improvement to compensate.

And the “diverged seeds” column tells the real story. With γ=0.99, training is a coin flip. With γ=0.95, it just works.

When You Actually Need High Gamma

Some tasks genuinely require caring about distant rewards: financial trading with quarterly returns, long-horizon robotics (think: cooking a meal, not just grasping an object), or games with delayed victory conditions.

For these, don’t use vanilla PPO/SAC. Consider:

• Hierarchical RL: Break the task into subgoals with shorter effective horizons at each level
• Model-based methods: DreamerV3 (Hafner et al., 2023) can handle γ=0.997 because it learns a world model and plans through imagined trajectories
• Successor representations: Decouple reward prediction from transition dynamics, allowing the value function to generalize better

If you must stick with model-free methods and high gamma, the recently proposed Hyperbolic discounting in RL paper (if I recall the year correctly) shows promise — it uses $\gamma(t) = \frac{1}{1 + kt}$ instead of exponential decay, which more gracefully handles very long horizons.

The Variance-Bias Tradeoff Nobody Talks About

Lowering gamma introduces bias — your agent optimizes for a shorter horizon than the actual task. This is fine when the intermediate steps are informative (dense rewards, good progress metrics). But when the only signal comes at episode end? You’ve got a problem.

The trick I’ve found: use different gammas for policy and value function training.

# Hacky but effective: dual-gamma training
# Use low gamma (0.95) for advantage computation -> stable policy gradients
# Use high gamma (0.99) for value targets -> value function learns long-horizon structure

class DualGammaBuffer:
    def __init__(self, gamma_policy=0.95, gamma_value=0.99, gae_lambda=0.95):
        self.gamma_policy = gamma_policy
        self.gamma_value = gamma_value
        self.gae_lambda = gae_lambda

    def compute_returns_and_advantages(self, rewards, values, dones):
        # Advantages with low gamma (for policy)
        advantages = self._compute_gae(rewards, values, dones, self.gamma_policy)

        # Returns with high gamma (for value function)
        returns = self._compute_returns(rewards, dones, self.gamma_value)

        return returns, advantages

    def _compute_gae(self, rewards, values, dones, gamma):
        advantages = np.zeros_like(rewards)
        last_gae = 0
        for t in reversed(range(len(rewards))):
            if dones[t]:
                last_gae = 0
            delta = rewards[t] + gamma * values[t + 1] * (1 - dones[t]) - values[t]
            advantages[t] = last_gae = delta + gamma * self.gae_lambda * (1 - dones[t]) * last_gae
        return advantages

    def _compute_returns(self, rewards, dones, gamma):
        returns = np.zeros_like(rewards)
        running_return = 0
        for t in reversed(range(len(rewards))):
            if dones[t]:
                running_return = 0
            running_return = rewards[t] + gamma * running_return
            returns[t] = running_return
        return returns

I haven’t tested this at scale, but the intuition is sound: policy optimization needs low-variance gradients (low gamma), while value function fitting benefits from seeing the true long-horizon structure (high gamma). The value function can tolerate more variance because it’s trained with MSE loss, which averages nicely.

Debugging Checklist When Training Diverges

When your long-horizon RL starts falling apart around 1M steps, check these in order:

1. explained_variance in TensorBoard — if it’s negative or oscillating, your value function is broken. Lower gamma.

2. value_loss magnitude — if it’s >1000 and not decreasing, your return targets are unbounded. Add reward clipping.

3. entropy trajectory — if it drops to near-zero before 500k steps, your policy collapsed prematurely. Increase ent_coef or use entropy scheduling.

4. Gradient norms — if policy_gradient_norm spikes periodically, you’re getting outlier batches. Increase batch_size or lower learning_rate.

# Quick sanity check for return magnitude
def check_return_distribution(env, gamma, n_episodes=20):
    returns = []
    for _ in range(n_episodes):
        obs, _ = env.reset()
        episode_rewards = []
        done = False
        while not done:
            action = env.action_space.sample()  # Random policy
            obs, reward, terminated, truncated, _ = env.step(action)
            episode_rewards.append(reward)
            done = terminated or truncated

        G = sum(r * gamma**i for i, r in enumerate(episode_rewards))
        returns.append(G)

    print(f"Return distribution (γ={gamma}):")
    print(f"  Mean: {np.mean(returns):.2f}")
    print(f"  Std: {np.std(returns):.2f}")
    print(f"  Max: {np.max(returns):.2f}")
    print(f"  Min: {np.min(returns):.2f}")

# If std >> mean or max >> mean, you need reward normalization

Debugging at 2am with diverging training curves is rough. Keep some Dark Chocolate Espresso Beans nearby — the caffeine-chocolate combo helps when you’re staring at TensorBoard wondering why policy_loss just became NaN.

The Horizon You Choose Defines the Agent You Get

γ isn’t just a hyperparameter — it’s a statement about what timescales matter. Setting γ=0.99 says “I want my agent to care about events 100 steps away almost as much as immediate ones.” The math then determines whether your training infrastructure can actually deliver that.

For most practical RL (Gymnasium environments, simulated robotics, games), γ=0.95-0.97 with reward normalization hits the sweet spot. You get stable training, reasonable convergence times, and agents that still plan ahead usefully.

If your task genuinely requires γ>0.99, you probably need a different algorithm class entirely — model-based methods, hierarchical RL, or at minimum some serious reward engineering.

I’m still curious whether the dual-gamma approach (low γ for advantages, high γ for value targets) holds up at scale. If anyone’s tried this on Humanoid-v4 or similar, I’d genuinely like to see the results. The theory says it should work; practice in RL has a habit of disagreeing.

FAQ

Q: Can I use γ=1.0 for continuing (non-episodic) tasks?

γ=1.0 makes returns unbounded by definition — the sum $\sum_{k=0}^{\infty} r_{t+k}$ diverges unless rewards decay to zero. For continuing tasks, use γ<1 and either use average reward RL formulations or reset the environment periodically. Average reward methods like R-learning explicitly subtract the average reward, keeping returns bounded.

Q: How do I pick gamma for a new environment I’ve never trained on?

Start with γ=0.95 and VecNormalize with reward clipping. Run 500k steps. If training is stable but performance plateaus early, increase gamma to 0.97-0.98. If training diverges (value loss explodes, explained variance goes negative), drop to 0.9-0.93. The right gamma depends on episode length and reward density — sparse terminal rewards need lower gamma than dense shaping rewards.

Q: Does SAC handle high gamma better than PPO?

Yes, generally. SAC’s entropy-regularized objective and off-policy nature provide some natural variance reduction. I’ve successfully used γ=0.99 with SAC on tasks where PPO diverges at 0.98. But SAC isn’t immune — at 10,000+ step horizons with sparse rewards, you’ll still need reward normalization and possibly lower gamma. The critic update frequency (often multiple per env step) also helps SAC’s value function converge faster.