Backtrader Memory Leak on Multi-Year Tests: 3 Debug Fixes

The 8GB RAM Spike That Shouldn’t Exist

Backtesting a simple moving average crossover strategy on SPY from 2015-2023 shouldn’t consume 8GB of RAM. Yet that’s exactly what happened when I extended a working 1-year backtest to 8 years of daily data.

The strategy itself was trivial — 50-day vs 200-day SMA crossover, maybe 20 trades per year. The memory spike made no sense. Running the same period with 1-minute bars would be understandable, but daily bars? Something was leaking.

This isn’t a Backtrader-specific issue. Any event-driven backtesting framework that accumulates state over time can hit this. But Backtrader’s default settings make it particularly easy to blow up memory on multi-year tests without realizing why.

Why Backtrader Holds Every Single Bar

Backtrader’s cerebro engine stores a full time series for every indicator and data feed by default. That self.sma50 indicator you defined? It’s keeping every single value from day 1 in memory.

The rationale makes sense for small tests. You might want to plot results, access historical indicator values, or debug by inspecting the full series. But on a 2000-day backtest with 5 indicators and 3 data feeds, you’re storing $2000 \times 5 \times 8 \text{ bytes} \times 3 = 240\text{KB}$ just for the indicators. Add order history, trade logs, and internal state — multiply by 100x for realistic strategies with more complexity.

The real killer is that most of this data is never accessed. Your strategy logic only cares about the current bar and maybe the last few. You’re not running machine learning over the full history or doing complex lookback operations. You just need to know: did the fast SMA cross above the slow SMA today?

Fix 1: Set exactbars=True to Enable Garbage Collection

The simplest fix is telling Backtrader to drop old bars once they’re no longer needed:

import backtrader as bt

cerebro = bt.Cerebro(exactbars=True)

This enables automatic memory management. Once an indicator or data line is no longer referenced by active calculations, Backtrader discards the old values.

The catch: your strategy must declare all lookback periods upfront using self.data.buflen() or rely on indicator periods. If you try to access self.data.close[-100] without telling Backtrader you need a 100-bar lookback, it’ll throw an index error.

Here’s a working example:

class SMACross(bt.Strategy):
    params = (('fast', 50), ('slow', 200))

    def __init__(self):
        self.fast_sma = bt.indicators.SMA(self.data.close, period=self.params.fast)
        self.slow_sma = bt.indicators.SMA(self.data.close, period=self.params.slow)
        self.crossover = bt.indicators.CrossOver(self.fast_sma, self.slow_sma)

    def next(self):
        if self.crossover > 0:  # fast crossed above slow
            self.buy(size=100)
        elif self.crossover < 0:
            self.sell(size=100)

With exactbars=True, this strategy only keeps the last 200 bars (the slow_sma period) in memory. On an 8-year SPY backtest, memory usage dropped from 8GB to under 500MB.

But there’s a hidden issue: if you want to plot results or access historical data in a custom analyzer, exactbars breaks that functionality. You’re trading off observability for memory efficiency.

Fix 2: Manually Set Line Buffer Lengths

If you need more control — say, you want to keep indicator history for plotting but drop raw OHLCV data — you can set buffer lengths per line:

class OptimizedSMA(bt.Strategy):
    params = (('fast', 50), ('slow', 200))

    def __init__(self):
        # Keep only 300 bars of raw data (50 extra for safety)
        self.data.close.buflen(self.params.slow + 100)

        self.fast_sma = bt.indicators.SMA(self.data.close, period=self.params.fast)
        self.slow_sma = bt.indicators.SMA(self.data.close, period=self.params.slow)

        # Keep full SMA history for plotting
        self.fast_sma.plotinfo.plot = True
        self.slow_sma.plotinfo.plot = True

This approach gives you granular control. The raw close prices are garbage-collected after 300 bars, but the computed SMAs are retained for visualization.

The tricky part is estimating the right buffer size. Too small, and you get runtime errors when an indicator tries to look back further than available. Too large, and you’re back to memory bloat.

I typically set buffer lengths to $\max(\text{indicator periods}) \times 1.5$ as a safety margin. For a strategy with a 200-day SMA as the longest lookback, I’d use buflen(300). The 50% padding handles edge cases like indicators that internally reference slightly older data.

Fix 3: Process Data in Chunks with Runonce=False

The most aggressive optimization is disabling Backtrader’s vectorized precomputation mode:

cerebro = bt.Cerebro(runonce=False, exactbars=True)

runonce=True (the default) precomputes all indicators across the entire dataset before starting the backtest. It’s faster because it leverages NumPy vectorization, but it requires loading the full dataset into memory.

runonce=False switches to event-driven mode. Indicators are computed bar-by-bar as the backtest progresses. This is slower — maybe 2-3x longer on large datasets — but memory usage stays constant regardless of backtest length.

Here’s a stress test comparing the two modes on SPY daily data (2015-2023, ~2000 bars):

The 3x speed penalty sounds bad, but 35 seconds for an 8-year backtest is still reasonable. And critically, memory usage is now independent of dataset size. Extending the test to 20 years of data barely moved the memory needle.

This mode is essential if you’re running parameter sweeps or walk-forward optimization. With runonce=True, testing 50 parameter combinations would load 50 copies of the dataset into memory. With runonce=False, memory usage stays flat.

Why This Matters More Than You Think

Most Backtrader tutorials test strategies on 1-2 years of data. That’s fine for learning, but production quant work often involves:

• Walk-forward optimization across 10-20 years (retraining every 252 days)
• Multi-asset portfolios with 50-100 tickers loaded simultaneously
• High-frequency minute/tick data where a single day can be 390-23400 bars

Without these fixes, you’ll hit memory limits long before you finish testing. I’ve seen strategies that ran fine on 2 years of daily data completely fail when extended to 5 years — not because of logic errors, but because the process was killed by the OOM reaper.

And if you’re running backtests on a cloud VM or in a Docker container with memory limits, these issues become fatal. A single backtest shouldn’t require 16GB of RAM.

The Hidden Cost: Plotting and Analyzers

Here’s the frustrating part: exactbars=True silently breaks cerebro.plot().

Backtrader’s plotting engine expects full historical data for all indicators and data feeds. If you’ve enabled garbage collection, it can’t render the chart because the old data is gone. You’ll either get a blank plot or a cryptic error about missing line data.

The workaround is running two passes:

# First pass: optimization with memory limits
cerebro_opt = bt.Cerebro(exactbars=True, runonce=False)
cerebro_opt.addstrategy(MyStrategy)
cerebro_opt.adddata(data_feed)
results_opt = cerebro_opt.run()

# Second pass: single run with plotting enabled
cerebro_plot = bt.Cerebro()  # default settings, keep full history
cerebro_plot.addstrategy(MyStrategy)
cerebro_plot.adddata(data_feed)
results_plot = cerebro_plot.run()
cerebro_plot.plot()

This doubles the runtime but lets you keep memory-efficient optimization while still generating visual output for final validation.

I’m not entirely sure why Backtrader doesn’t offer a middle ground — maybe keep the last N bars for plotting while dropping ancient history. The all-or-nothing approach feels like a design oversight.

When Leaks Aren’t Backtrader’s Fault

Sometimes the memory issue is in your strategy code, not the framework. Common culprits:

Accumulating trade history manually:

def __init__(self):
    self.trade_log = []  # This list grows indefinitely

def next(self):
    if self.position:
        self.trade_log.append({
            'date': self.data.datetime.date(0),
            'price': self.data.close[0],
            'size': self.position.size
        })  # 2000 bars = 2000 dict entries in RAM

If you need trade logs, use Backtrader’s built-in observers or write to disk incrementally instead of holding everything in memory.

Storing full pandas DataFrames in strategy state:

def __init__(self):
    self.hist_data = self.data.to_dataframe()  # Defeats the entire point of exactbars

If you need pandas for calculations, convert only the current window:

def next(self):
    recent_bars = pd.DataFrame({
        'close': self.data.close.get(size=100),
        'volume': self.data.volume.get(size=100)
    })
    # Do pandas operations on recent_bars only

Leaky indicator implementations:

Custom indicators that don’t properly call super().__init__() or manually manage line arrays can bypass Backtrader’s garbage collection.

FAQ

Q: Does exactbars=True affect backtest accuracy?

No. It only controls memory management, not calculation logic. Your strategy sees the same data and produces identical trades. The only difference is that old bars are discarded once they’re no longer needed for active calculations.

Q: Can I use runonce=False with live trading?

Yes, and you probably should. Live trading is inherently event-driven (bars arrive one at a time), so runonce=False matches production behavior better. The speed penalty doesn’t matter for real-time execution.

Q: What if my strategy needs to access arbitrary historical data?

Then you can’t use exactbars=True without workarounds. Options: (1) store historical data externally (database, pickle file) and load on demand, (2) use a hybrid approach where critical historical data is cached separately, or (3) accept the memory cost. For strategies that genuinely need full history (e.g., bootstrapping resampling for confidence intervals), memory usage is unavoidable.

The Real Bottleneck: Indicator Recalculation

Once you’ve fixed memory leaks, the next performance wall is redundant indicator recalculation. Backtrader recomputes indicators on every next() call, even if the underlying data hasn’t changed.

For simple indicators like SMA, this is fine — $O(1)$ per bar because it’s a rolling sum. But for expensive operations like ATR (requires multiple passes over high/low/close), FFT-based frequency analysis, or machine learning features, the cost compounds.

Consider a strategy that uses a 50-bar rolling correlation between two assets:

class PairsStrategy(bt.Strategy):
    def next(self):
        # This recalculates correlation on 50 bars EVERY SINGLE TIME
        corr = np.corrcoef(
            self.data0.close.get(size=50),
            self.data1.close.get(size=50)
        )[0, 1]

        if corr > 0.8:
            # Enter pair trade
            pass

On a 2000-bar backtest, that’s 2000 correlation calculations, each iterating over 50 data points. The complexity is $O(N \times W)$ where $N$ is total bars and $W$ is window size.

The fix is caching results or using Backtrader’s indicator framework (which handles incremental updates):

class RollingCorrelation(bt.Indicator):
    lines = ('corr',)
    params = (('period', 50),)

    def __init__(self):
        self.addminperiod(self.params.period)

    def next(self):
        data0 = self.data0.get(size=self.params.period)
        data1 = self.data1.get(size=self.params.period)
        self.lines.corr[0] = np.corrcoef(data0, data1)[0, 1]

class PairsStrategy(bt.Strategy):
    def __init__(self):
        self.corr_indicator = RollingCorrelation(self.data0, self.data1)

    def next(self):
        if self.corr_indicator[0] > 0.8:
            # Enter pair trade
            pass

Wrapping expensive calculations in a custom indicator ensures they’re computed once per bar and cached. On the same 2000-bar test, this reduced runtime from 18 seconds to 4 seconds.

The Verdict: Start with exactbars, Escalate if Needed

For most strategies, exactbars=True is enough. It cuts memory usage by 80-95% with zero code changes beyond a single argument to Cerebro().

If you’re running parameter sweeps, add runonce=False. The 2-3x speed hit is worth it to keep memory constant across hundreds of test runs. Debugging memory issues after a 12-hour optimization run is worse than just accepting longer execution time upfront.

If you need plotting, accept the trade-off: run optimizations with memory limits, then do a final validation pass with full data retention for visualization.

The frustrating part is that none of this is documented prominently in Backtrader’s intro tutorials. You only discover these knobs after hitting OOM errors on real datasets. If I were designing a backtesting framework today, I’d make exactbars=True the default and require users to opt into full history retention.

What I’m still curious about: whether there’s a way to hook into Backtrader’s memory management to selectively persist indicator history to disk (HDF5, Parquet) for post-analysis without keeping it all in RAM. For now, the workaround is exporting results to a file during next() and reconstructing plots offline. Not elegant, but it works.

If you’re doing serious backtesting on a budget laptop, you’ll eventually need 32GB of RAM anyway. But these fixes can delay that upgrade for a while.