Custom Gymnasium Environment: Portfolio Project Guide

Why Your Generic CartPole Project Won’t Land You an Interview

Every RL portfolio on GitHub looks the same: CartPole, LunarLander, maybe Atari if they’re ambitious. I reviewed 30+ candidate portfolios last year, and exactly zero stood out. The ones that did? They built custom environments that solved actual problems.

Here’s the thing: interviewers don’t care that you can run stable_baselines3.PPO on a pre-built environment. They want to see that you understand the MDP formulation deeply enough to model a new problem from scratch. That’s the signal that separates “followed a tutorial” from “can actually do RL work.”

I’m going to walk through building two different custom environments for the same problem domain—inventory management—and show you exactly where each approach falls apart. One uses a naive reward structure, the other uses shaped rewards. The difference in training behavior is dramatic, and understanding why is what makes this portfolio-worthy.

The Problem: Inventory Management as an MDP

Inventory control is a classic operations research problem that maps cleanly to RL. You have stock levels, incoming demand, ordering costs, and holding costs. The goal is to minimize total cost while avoiding stockouts.

The state space looks like this:

$$s_t = (I_t, D_{t-1}, D_{t-2}, …, D_{t-k})$$

where $I_t$ is current inventory and $D_{t-i}$ represents historical demand. The action $a_t$ is the order quantity. The transition follows:

$$I_{t+1} = \max(0, I_t + a_t – D_t)$$

The reward function is where things get interesting—and where most implementations go wrong.

Approach 1: Naive Sparse Reward

Let’s start with the obvious implementation. Here’s a minimal Gymnasium environment:

import gymnasium as gym
from gymnasium import spaces
import numpy as np

class InventoryEnvSparse(gym.Env):
    """Inventory management with sparse rewards."""

    metadata = {"render_modes": ["human"]}

    def __init__(self, max_inventory=100, max_order=50, 
                 holding_cost=1.0, stockout_cost=10.0,
                 order_cost=0.5, demand_mean=20, demand_std=5):
        super().__init__()

        self.max_inventory = max_inventory
        self.max_order = max_order
        self.holding_cost = holding_cost
        self.stockout_cost = stockout_cost
        self.order_cost = order_cost
        self.demand_mean = demand_mean
        self.demand_std = demand_std

        # Observation: [current_inventory, last_5_demands]
        self.observation_space = spaces.Box(
            low=np.array([0] + [0]*5, dtype=np.float32),
            high=np.array([max_inventory] + [100]*5, dtype=np.float32),
            dtype=np.float32
        )

        # Action: order quantity (discrete for simplicity)
        self.action_space = spaces.Discrete(max_order + 1)

        self.demand_history = []
        self.inventory = 0
        self.step_count = 0

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)

        self.inventory = self.max_inventory // 2
        self.demand_history = [self.demand_mean] * 5
        self.step_count = 0

        return self._get_obs(), {}

    def _get_obs(self):
        return np.array(
            [self.inventory] + self.demand_history[-5:],
            dtype=np.float32
        )

    def step(self, action):
        order_qty = int(action)

        # Sample demand
        demand = max(0, int(self.np_random.normal(
            self.demand_mean, self.demand_std
        )))

        # Calculate costs
        ordering_cost = self.order_cost * order_qty

        # Update inventory
        self.inventory = min(
            self.max_inventory, 
            self.inventory + order_qty
        )

        # Fulfill demand
        fulfilled = min(self.inventory, demand)
        stockout = demand - fulfilled
        self.inventory -= fulfilled

        # Reward: penalize holding cost, ordering cost, and stockouts each step
        holding = self.holding_cost * self.inventory
        stockout_penalty = self.stockout_cost * stockout

        reward = -(ordering_cost + holding + stockout_penalty)

        self.demand_history.append(demand)
        self.step_count += 1

        terminated = self.step_count >= 200
        truncated = False

        info = {
            "demand": demand,
            "stockout": stockout,
            "inventory": self.inventory,
            "order": order_qty
        }

        return self._get_obs(), reward, terminated, truncated, info

Let’s see what happens when we train PPO on this:

import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv
from stable_baselines3.common.monitor import Monitor
import matplotlib.pyplot as plt

# Wrap with Monitor to capture episode stats reliably
env = DummyVecEnv([lambda: Monitor(InventoryEnvSparse())])
model = PPO(
    "MlpPolicy", 
    env, 
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    verbose=0,
    seed=42
)

model.learn(total_timesteps=100_000)

# Read episode rewards from Monitor
episode_rewards = env.envs[0].get_episode_rewards()
print(f"Final avg reward: {np.mean(episode_rewards[-50:]):.2f}")
# Output: Final avg reward: -847.23

That’s not great. The agent learned something, but it’s erratic. Let me show you the training curve:

plt.figure(figsize=(10, 4))
plt.plot(episode_rewards, alpha=0.3)
plt.plot(np.convolve(
    episode_rewards, 
    np.ones(50)/50, 
    mode='valid'
), 'r-', linewidth=2)
plt.xlabel('Episode')
plt.ylabel('Total Reward')
plt.title('Sparse Reward Training - Highly Unstable')
plt.show()

The smoothed line oscillates between -900 and -750 without clear convergence. Why?

The Credit Assignment Problem

With step-level costs, the agent still receives a reward signal at each timestep, but the rewards are noisy and weakly informative—the agent can’t easily infer which earlier ordering decisions caused a stockout several steps later. The temporal credit assignment problem—figuring out which action caused which outcome—becomes difficult.

Mathematically, the gradient of the policy objective is:

$$\nabla_\theta J(\theta) = \mathbb{E}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t\right]$$

where $G_t = \sum_{k=t}^{T} \gamma^{k-t} r_k$. When reward signal is weakly correlated with the actions that caused it, the return $G_t$ provides a noisy gradient—the agent struggles to identify which decisions were actually responsible for penalties incurred several steps later.

Approach 2: Shaped Rewards with Potential-Based Shaping

Potential-based reward shaping (Ng et al., 1999) lets us add intermediate rewards without changing the optimal policy. The key insight:

$$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') – \Phi(s)$$

where $\Phi$ is a potential function. This transformation preserves the set of optimal policies—provided you use the same $\gamma$ in shaping as in training. A result I still find somewhat magical.

Here’s the shaped environment:

class InventoryEnvShaped(gym.Env):
    """Inventory management with potential-based reward shaping."""

    metadata = {"render_modes": ["human"]}

    def __init__(self, max_inventory=100, max_order=50,
                 holding_cost=1.0, stockout_cost=10.0,
                 order_cost=0.5, demand_mean=20, demand_std=5,
                 target_inventory=None):
        super().__init__()

        self.max_inventory = max_inventory
        self.max_order = max_order
        self.holding_cost = holding_cost
        self.stockout_cost = stockout_cost
        self.order_cost = order_cost
        self.demand_mean = demand_mean
        self.demand_std = demand_std

        # Target inventory for shaping (if None, use EOQ approximation)
        if target_inventory is None:
            # Economic Order Quantity heuristic
            self.target = int(demand_mean * 1.5)
        else:
            self.target = target_inventory

        self.observation_space = spaces.Box(
            low=np.array([0] + [0]*5, dtype=np.float32),
            high=np.array([max_inventory] + [100]*5, dtype=np.float32),
            dtype=np.float32
        )
        self.action_space = spaces.Discrete(max_order + 1)

        self.demand_history = []
        self.inventory = 0
        self.prev_potential = 0
        self.step_count = 0

    def _potential(self, inventory):
        """Potential function: negative quadratic distance from target."""
        return -0.1 * (inventory - self.target) ** 2

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)

        self.inventory = self.max_inventory // 2
        self.demand_history = [self.demand_mean] * 5
        self.prev_potential = self._potential(self.inventory)
        self.step_count = 0

        return self._get_obs(), {}

    def _get_obs(self):
        return np.array(
            [self.inventory] + self.demand_history[-5:],
            dtype=np.float32
        )

    def step(self, action):
        order_qty = int(action)

        demand = max(0, int(self.np_random.normal(
            self.demand_mean, self.demand_std
        )))

        ordering_cost = self.order_cost * order_qty

        self.inventory = min(
            self.max_inventory,
            self.inventory + order_qty
        )

        fulfilled = min(self.inventory, demand)
        stockout = demand - fulfilled
        self.inventory -= fulfilled

        holding = self.holding_cost * self.inventory
        stockout_penalty = self.stockout_cost * stockout

        # Base reward (same as unshaped version)
        base_reward = -(ordering_cost + holding + stockout_penalty)

        # Potential-based shaping: compute current potential AFTER state update
        current_potential = self._potential(self.inventory)
        shaping = 0.99 * current_potential - self.prev_potential
        self.prev_potential = current_potential  # Must update AFTER computing shaping

        reward = base_reward + shaping

        self.demand_history.append(demand)
        self.step_count += 1

        terminated = self.step_count >= 200
        truncated = False

        info = {
            "demand": demand,
            "stockout": stockout,
            "inventory": self.inventory,
            "base_reward": base_reward,
            "shaping": shaping
        }

        return self._get_obs(), reward, terminated, truncated, info

Same training setup:

env_shaped = DummyVecEnv([lambda: Monitor(InventoryEnvShaped())])
model_shaped = PPO(
    "MlpPolicy",
    env_shaped,
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    verbose=0,
    seed=42
)

model_shaped.learn(total_timesteps=100_000)
episode_rewards_shaped = env_shaped.envs[0].get_episode_rewards()
print(f"Final avg reward: {np.mean(episode_rewards_shaped[-50:]):.2f}")
# Output: Final avg reward: -612.41

That’s a 28% improvement. But the training curve tells the real story—much smoother convergence, fewer catastrophic policy collapses.

How to Register Your Environment Properly

Most tutorials skip this, but proper registration matters for reproducibility and makes your portfolio look professional:

from gymnasium.envs.registration import register

# In your __init__.py or setup
register(
    id="InventoryControl-v0",
    entry_point="inventory_envs:InventoryEnvSparse",
    max_episode_steps=200,
)

register(
    id="InventoryControl-v1",
    entry_point="inventory_envs:InventoryEnvShaped",
    max_episode_steps=200,
)

# Now you can do this
env = gym.make("InventoryControl-v1")

The max_episode_steps wrapper handles truncation automatically—something I’ve seen broken in countless GitHub repos. Gymnasium 0.29+ changed how truncation works (it now returns truncated=True instead of setting done=True with info["TimeLimit.truncated"]=True), and this trips people up constantly.

The Shaping Function That Actually Works

My first attempt at the potential function was linear: $\Phi(s) = -|I – I^*|$. Terrible idea. The gradient is discontinuous at the target, and the agent kept oscillating.

The quadratic version $\Phi(s) = -\alpha(I – I^*)^2$ works because it provides smooth gradients everywhere. But you need to tune $\alpha$. Too high and shaping dominates base rewards. Too low and you’re back to an unguided signal.

I found $\alpha = 0.1$ works for this problem, but your mileage will vary. There’s no principled way to set this—it’s one of those hyperparameters you have to sweep. I usually try {0.01, 0.1, 1.0} and pick based on training stability.

Where Shaped Rewards Still Fail

Here’s the uncomfortable truth: even with shaping, the agent struggles when demand variance increases. Try demand_std=15 instead of 5:

env_noisy = DummyVecEnv([lambda: Monitor(InventoryEnvShaped(demand_std=15))])
# ... train ...
# Final avg reward: -1043.87

The policy degenerates to “always order a fixed amount” regardless of state. Why? The shaping function assumes a static target, but with high variance demand, the optimal policy needs to be more reactive.

A better approach: use a recurrent policy (LSTM) or add demand forecasting as an auxiliary task. But that’s a topic for another post.

Making It Interview-Ready

What transforms this from “homework assignment” to “impressive portfolio piece”?

1. Baseline comparison: Implement the (s,S) policy analytically and show your RL agent beats it
2. Ablation study: Show what happens without shaping, with wrong $\alpha$, with different observation spaces
3. Generalization: Train on one demand distribution, test on another
4. Visualization: Plot inventory trajectories, not just reward curves

Here’s the baseline comparison that lands interviews:

def ss_policy(inventory, s=15, S=40):
    """Classic (s,S) inventory policy."""
    if inventory <= s:
        return S - inventory  # Order up to S
    return 0

def evaluate_policy(env, policy_fn, episodes=100):
    total_rewards = []
    for _ in range(episodes):
        obs, _ = env.reset()
        episode_reward = 0
        done = False
        while not done:
            action = policy_fn(obs)
            obs, reward, terminated, truncated, _ = env.step(action)
            episode_reward += reward
            done = terminated or truncated
        total_rewards.append(episode_reward)
    return np.mean(total_rewards), np.std(total_rewards)

# Compare
env_eval = InventoryEnvShaped()
rl_mean, rl_std = evaluate_policy(
    env_eval,
    lambda obs: int(model_shaped.predict(obs.reshape(1, -1), deterministic=True)[0])
)
ss_mean, ss_std = evaluate_policy(
    env_eval,
    lambda obs: ss_policy(int(obs[0]))
)

print(f"RL Agent: {rl_mean:.2f} ± {rl_std:.2f}")
print(f"(s,S) Policy: {ss_mean:.2f} ± {ss_std:.2f}")
# RL Agent: -608.34 ± 87.42
# (s,S) Policy: -723.91 ± 112.67

Beating a classical operations research baseline by 16%—that’s what you put in your portfolio README.

But here’s what most people miss: the (s,S) policy assumes stationary demand. Add a trend component to your demand process, and the RL agent’s advantage grows substantially because it can adapt.

Hyperparameters That Actually Matter

After training dozens of these, here’s what moves the needle:

The entropy coefficient is particularly sneaky. Too low and the policy collapses to deterministic too fast. Too high and it never converges. I’ve seen training runs where bumping ent_coef from 0.0 to 0.01 cut training time in half.

Seed Sensitivity: The Hidden Landmine

Running the same code with seed=123 instead of seed=42:

model_123 = PPO(..., seed=123)
# Final avg reward: -658.91 (vs -612.41 with seed 42)

That’s a 7.5% difference from seed alone. If you’re only reporting single-seed results, you’re doing it wrong. Report mean and standard deviation across at least 5 seeds. Interviewers know this, and they’ll ask.

I track multi-seed experiments with W&B Sweeps for exactly this reason—it handles parallel runs and gives you proper confidence intervals automatically.

Common Implementation Bugs

Things I’ve seen break in the wild:

Inconsistent observation history length: Python’s list[-5:] silently returns fewer than 5 elements when the list is short. This causes observation space shape mismatches. Always ensure a fixed-length output:

# Bad: returns variable-length list early in an episode
return self.demand_history[-5:]

# Good: always returns exactly 5 elements
pad_len = max(0, 5 - len(self.demand_history))
history = [self.demand_mean] * pad_len + self.demand_history[-5:]

Forgetting to update previous potential: The shaping term needs the old potential before you update state. Compute shaping = gamma * new_potential - old_potential, then assign self.prev_potential = new_potential. I’ve debugged this exact bug at least three times.

Integer overflow in action space: If your max_order is larger than your observation space can represent, spaces.Box will silently clip. Use dtype=np.int32 or switch to spaces.Discrete.

What Interviewers Actually Ask

From my experience on both sides of the table:

1. “Why did you choose this observation space?” — Have a principled answer
2. “What’s your reward scale?” — If rewards are -1000, explain why not normalized
3. “How does this compare to the optimal policy?” — Have a baseline
4. “What happens if demand doubles?” — Show generalization results
5. “Walk me through the MDP formulation” — Know your state transitions cold

The candidates who can’t answer question 5 don’t move forward. The MDP is the foundation—if you can’t derive it from the problem description, you don’t understand RL deeply enough.

FAQ

Q: Should I use Gymnasium or the older OpenAI Gym for portfolio projects?

Gymnasium is the maintained fork since 2022—use it. Gym is deprecated and won’t receive updates. The API differences are minor (mostly around reset() returning a tuple), but using the old library signals you’re following outdated tutorials.

Q: How important is reward shaping for custom environments?

Very important for anything beyond toy problems. Without shaping, you’ll spend significantly more compute on training and may not converge reliably. The Ng et al. potential-based framework preserves the set of optimal policies (when using the same gamma as training), so there’s no correctness downside—only a tuning cost.

Q: What’s the minimum viable portfolio project for RL interviews?

A custom environment with: (1) non-trivial state space (not just position/velocity), (2) baseline comparison showing your agent outperforms a heuristic, (3) ablation showing which design choices mattered. The inventory control example here hits all three.

For portfolio projects, shaped rewards beat unshaped rewards every time. The implementation overhead is minimal—one extra function and a few lines in step(). The training stability gains are substantial.

If you want to go deeper on reward engineering pitfalls, I wrote about the traps that kill training in production settings—different context but overlapping lessons.

The thing I’m still trying to figure out: how to automatically learn the potential function instead of hand-crafting it. Inverse RL and learned reward models feel promising, but I haven’t gotten them to work reliably on problems this simple yet. If you’ve had success there, I’d love to hear about it.