gdb + py-spy vs pdb: Production Debug Latency Compared

When pdb Fails in Production

Your Flask API is timing out. Not consistently—just on certain requests. You add breakpoint() and restart the server, but by the time you hit the breakpoint, the problematic request pattern has already passed. You try logging, but the issue vanishes when you add enough logs to catch it. This is where traditional Python debugging tools stop being useful.

I spent months debugging production issues with pdb, logging, and the occasional print() statement before realizing these tools weren’t built for the constraints I was working under. You can’t pause a live server. You can’t restart a process that takes 10 minutes to warm up. And you definitely can’t add logging fast enough when the bug only appears under specific load patterns.

That’s when I started using gdb with Python extensions and py-spy. These aren’t beginner-friendly tools—they require understanding how Python’s interpreter works at the C level. But they let you attach to running processes without modifying code or restarting anything. Here’s what actually works in production.

The Obvious Approach That Doesn’t Work

Most Python developers reach for pdb first. You insert breakpoint(), restart the process, and step through execution line by line. This works perfectly in local development.

In production? It’s a disaster.

import pdb

def process_payment(user_id, amount):
    # This line freezes the entire Flask worker
    pdb.set_trace()
    transaction = create_transaction(user_id, amount)
    return transaction.id

The moment pdb.set_trace() hits, that worker thread blocks. If you’re running gunicorn with 4 workers, you just lost 25% of your capacity. Every request routed to that worker now times out. Your monitoring starts screaming.

And that’s assuming you can even reproduce the issue. The bug I was chasing only appeared when a specific combination of cached data and database state aligned—roughly once every 500 requests. Restarting the server to add debugging code destroyed that state every time.

Logging seemed like the safer option. Add enough log statements to track execution flow, deploy, wait for the bug to appear naturally.

import logging

def process_payment(user_id, amount):
    logging.info(f"Starting payment for user {user_id}")
    transaction = create_transaction(user_id, amount)
    logging.info(f"Transaction created: {transaction.id}")
    logging.info(f"Transaction state: {transaction.state}")
    return transaction.id

But logging changes timing. That logging.info() call acquires a lock, formats the string, writes to disk (or network if you’re shipping logs elsewhere). The bug I was hunting disappeared completely once I added detailed logging—a classic Heisenbug.

Attaching gdb to a Running Python Process

gdb is a debugger for C programs, but CPython is written in C. That means gdb can attach to a running Python interpreter and inspect everything—stack frames, local variables, even the Python heap.

First, you need Python debugging symbols. On Ubuntu:

sudo apt-get install python3-dbg

On production systems, you might not have debugging symbols installed. That’s fine—gdb can still attach, you just lose some variable introspection. Better than nothing.

Find the process ID of your hanging Python worker:

ps aux | grep gunicorn
# ubuntu   12847  2.1  3.2  512944  128556 ?  Sl   14:23   0:15 gunicorn: worker [app]

Attach gdb:

sudo gdb -p 12847

This pauses the process immediately. Every thread freezes. If this is your only worker, your API is now completely down. That’s the trade-off—you get full introspection, but you pay with availability.

Inside gdb, load the Python debugging extensions:

(gdb) source /usr/share/gdb/auto-load/usr/bin/python3.11
(gdb) py-bt

py-bt shows the Python-level backtrace, not the C-level one. Here’s what I saw when debugging that payment timeout:

Traceback (most recent call first):
  File "/app/models.py", line 247, in save
    self.db.commit()
  File "/app/payment.py", line 89, in process_payment
    transaction.save()
  File "/app/api.py", line 34, in handle_request
    result = process_payment(user_id, amount)

The process was stuck in db.commit(). Not an infinite loop, not a deadlock—just a very slow database write. The transaction was waiting on a table lock that another process held.

You can inspect variables too:

(gdb) py-print transaction
local 'transaction' = <Transaction object at 0x7f3c4d2a8f10>
(gdb) py-print transaction.id
local 'transaction.id' = 98234

But gdb has a major problem: it stops the world. The moment you attach, every thread in that process freezes. For a background worker processing a queue, maybe that’s acceptable. For a web server handling live traffic? You just created an outage.

py-spy: Sampling Without Stopping

py-spy is a sampling profiler that doesn’t require attaching a debugger or modifying code. It reads process memory directly (using process_vm_readv on Linux) to reconstruct Python stack frames without stopping the interpreter.

Install it:

pip install py-spy

Attach to the same gunicorn worker:

sudo py-spy top --pid 12847

This shows a live view of what the process is executing, sampled at 100Hz by default:

Total Samples 1840
%Own   %Total  OwnTime  TotalTime  Function (filename:line)
89.00%  89.00%   16.38s    16.38s   commit (sqlalchemy/orm/session.py:1542)
 5.40%   5.40%    0.99s     0.99s   _execute_context (sqlalchemy/engine/base.py:1819)
 2.70%   2.70%    0.50s     0.50s   fetchall (psycopg2/cursor.py:181)

The process spent 89% of its time inside commit(). That matches what gdb showed, but py-spy didn’t freeze the worker. Requests kept processing (slowly, but they processed). The overhead is roughly 1-3% CPU—noticeable in benchmarks, invisible in production.

py-spy can also dump stack traces for all threads:

sudo py-spy dump --pid 12847

Thread 12847 (active): "MainThread"
    commit (sqlalchemy/orm/session.py:1542)
    process_payment (payment.py:89)
    handle_request (api.py:34)
    __call__ (flask/app.py:2464)

This is faster than gdb’s py-bt and doesn’t pause the process. The sampling approach means you might miss very short function calls (sub-millisecond), but for debugging slow production requests, that’s not what you’re hunting.

The flame graph mode is where py-spy really shines:

sudo py-spy record -o profile.svg --pid 12847 --duration 60

This records 60 seconds of execution and generates an interactive flame graph. Open profile.svg in a browser and you get a visual breakdown of where time is actually spent. The database commit issue jumped out immediately—a giant red block dominating the graph.

When gdb Beats py-spy

py-spy is non-invasive, but it can’t do everything gdb can. Specifically:

1. Inspecting complex object state

py-spy shows you the call stack. gdb lets you inspect local variables, traverse object attributes, even modify memory in place.

I hit this limitation debugging a NumPy indexing bug. py-spy told me the process was stuck in __getitem__, but I needed to see the actual array shape and indices being used. With gdb:

(gdb) py-print arr.shape
local 'arr.shape' = (1024, 1024, 3)
(gdb) py-print idx
local 'idx' = (slice(None, None, None), slice(None, None, None), 1048576)

That third index was way out of bounds. This should have raised IndexError, but due to a bug in NumPy 1.23 with boolean indexing, it triggered an infinite loop instead.

py-spy can’t introspect variable values. It only sees function names and line numbers.

2. Deadlock detection

py-spy samples execution. If two threads are deadlocked, py-spy just shows both threads waiting—but it won’t tell you what they’re waiting on.

gdb can inspect thread state and lock ownership:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f3c (LWP 12847)  __GI___lll_lock_wait ()
  2    Thread 0x7f3b (LWP 12848)  __GI___lll_lock_wait ()

Both threads waiting on a lock. Now find out which lock:

(gdb) thread 1
(gdb) frame 5
(gdb) p lock
$1 = (_PyMutex *) 0x7f3c4d2a8f10

Thread 1 holds lock 0x7f3c4d2a8f10. Switch to thread 2:

(gdb) thread 2
(gdb) frame 3
(gdb) p lock
$2 = (_PyMutex *) 0x7f3c4d2a8f10

Thread 2 is trying to acquire the same lock. Classic deadlock—though in this case, not truly a deadlock, just very slow lock contention because the critical section was doing I/O.

3. Modifying execution in place

This is the nuclear option, and I’ve only done it twice in production. gdb lets you change variables or even skip lines of code without restarting:

(gdb) set variable retry_count = 0
(gdb) jump +5  # Skip the next 5 lines

I used this once to bypass a validation check that was incorrectly rejecting requests. The proper fix required a code deploy, but we needed the service back online immediately. Changed the variable in gdb, detached, and the process resumed with the check disabled.

Dangerous? Absolutely. But when the alternative is a full outage, sometimes you take the risk.

Overhead Comparison: What You Pay

I benchmarked the overhead on a Flask API handling 100 req/s (gunicorn with 4 workers, each processing ~25 req/s).

Baseline (no debugging):
– Mean latency: $t_0 = 28\text{ms}$
– P99 latency: $t_{99} = 67\text{ms}$
– CPU usage: 15% per worker

With py-spy attached (100Hz sampling):
– Mean latency: $t_0 = 29\text{ms}$ (relative increase $\Delta t = \frac{29-28}{28} \approx 3.6\%$)
– P99 latency: $t_{99} = 69\text{ms}$
– CPU usage: 16% per worker

The overhead is almost negligible. You could run py-spy continuously in production if you wanted (though I wouldn’t recommend it—sampling generates data you need to store somewhere).

With gdb attached (paused):
– Mean latency: $\infty$ (all requests to that worker timeout)
– Throughput drops to $\frac{3}{4}$ of baseline (3 workers still running)

GDB is an all-or-nothing tool. Use it when you’re already in an incident and availability is already compromised.

With pdb.set_trace() (interactive debugging):
– Mean latency: $\infty$ (worker blocks indefinitely)
– Throughput drops to $\frac{3}{4}$ of baseline
– Risk: if multiple workers hit the same breakpoint, entire API goes down

PDB has the same availability impact as gdb, but without the introspection power. There’s almost no reason to use pdb in production.

The Memory Leak I Couldn’t Find with Logging

One of our background workers had a slow memory leak. Over 48 hours, memory usage would climb from 200MB to 8GB, then the process would OOM and restart. Nothing obvious in the code—no global lists, no cache without eviction.

I added memory profiling with tracemalloc (which I’ve written about before), but the overhead was brutal—15% CPU constantly. The worker fell behind on its queue, and we had to disable it.

py-spy has a --native flag that shows both Python and C/C++ frames. That’s important because memory leaks in Python often come from extension modules (NumPy, pandas, Pillow) that allocate memory outside Python’s heap.

sudo py-spy record --native -o leak.svg --pid 15432 --duration 300

The flame graph showed 40% of time spent inside PIL.Image.resize(). Not leaking during resize—the issue was we were caching resized images in a weakref.WeakValueDictionary, but the values were still referenced elsewhere (circular reference through a closure). The weak references never expired.

py-spy didn’t diagnose the leak directly, but it pointed me to the hot path. Once I knew where to look, the bug was obvious.

Amazon Product Rec

Debugging production at 3am is easier with Blue Light Blocking Glasses—your eyes will thank you after staring at flame graphs for 6 hours straight.

What I Actually Use Now

For most production debugging, I start with py-spy. It’s fast, safe, and usually gives me enough context to identify the problem. If I see a slow function, I know where to add instrumentation (proper profiling, or targeted logging).

I reach for gdb in two scenarios:

1. The process is already hung. py-spy won’t help if the interpreter is completely deadlocked. gdb can attach and at least tell you where it’s stuck.
2. I need to inspect state that py-spy can’t show. Array shapes, dictionary contents, lock ownership.

I never use pdb in production anymore. The availability cost is identical to gdb, but the introspection is worse. If I’m going to pause a worker, I might as well get full C-level debugging.

One pattern that works well: run py-spy continuously with a longer sample interval (10Hz instead of 100Hz) and stream the output to a log aggregator. Overhead drops to <1%, and you get a time-series view of where your application spends time. When an incident happens, you have historical flame graphs showing exactly what changed.

The biggest surprise was how often the problem wasn’t in my code. Database lock contention, slow DNS resolution, a misconfigured connection pool—these only show up when you can see the full stack, including library code and native extensions. Traditional logging doesn’t capture that unless you instrument every dependency (which is obviously impractical).

FAQ

Q: Can py-spy work with Docker containers?

Yes, but you need --cap-add SYS_PTRACE or --privileged when running the container. Without SYS_PTRACE, py-spy can’t read process memory. You can also install py-spy inside the container and run it there, which doesn’t require special capabilities.

Q: Does gdb work with PyPy or other Python implementations?

No. gdb’s Python extensions are specific to CPython’s internal structures. PyPy uses a different interpreter architecture (RPython), so the memory layout is completely different. py-spy also only supports CPython currently (as of version 0.3.x).

Q: What’s the difference between py-spy and austin?

Austin is another sampling profiler similar to py-spy. The main difference: austin can attach to processes without root permissions (on Linux with CAP_SYS_PTRACE), while py-spy usually requires sudo. Austin’s output format is also more flexible (supports both flamegraph and speedscope formats). I prefer py-spy because the flame graph UX is cleaner, but austin is worth trying if you can’t get root access.

When Sampling Isn’t Enough

py-spy samples at fixed intervals. If your bug is in code that runs quickly but incorrectly (e.g., an off-by-one error in a tight loop), sampling might not catch it. You need either:

1. Deterministic profiling (like cProfile), which instruments every function call. Overhead is 10-50%, so usually not viable in production.
2. Dynamic instrumentation with sys.settrace() or sys.monitoring (Python 3.12+), which lets you set hooks for specific functions. Still slower than sampling, but targeted.
3. Old-fashioned debugging: reproduce locally, add tests, bisect the commit history.

The right approach depends on the failure mode. If the problem only appears under production load (race conditions, memory pressure, specific data patterns), you probably can’t reproduce locally. That’s when gdb and py-spy are your only options.

I’m still looking for a good solution for debugging async Python (asyncio, Trio). py-spy shows the event loop, but tracking which coroutine is blocking is painful. The flame graph just shows run_forever() → select.select() with no visibility into what task scheduled the I/O. My best guess is you need asyncio debug mode (PYTHONASYNCIODEBUG=1), but the overhead is severe—I haven’t found a production-safe way to introspect async execution yet.

For synchronous Python, though? py-spy for continuous monitoring, gdb when you need to dig deep. pdb stays in local development where it belongs.