ROS2 Node Crashes After 48h: Memory Leak Fix for Jetson

A ROS2 navigation stack running perfectly for two days, then: segfault.

This happened on a Jetson Xavier NX running a multi-sensor fusion pipeline. The node would start clean, pass all integration tests, run flawlessly through the first 24 hours of continuous operation. Then somewhere around the 47-48 hour mark, it would die with a cryptic std::bad_alloc or just vanish from ros2 node list.

The logs were useless. No warnings. No gradual performance degradation. Just sudden death.

Turns out the culprit wasn’t ROS2 itself — it was the intersection of three things: DDS middleware default settings, unbound message queues, and Jetson’s 8GB RAM constraint. The fix took 15 minutes once I knew where to look. The investigation took three days.

What Actually Kills Long-Running Nodes

ROS2 uses DDS (Data Distribution Service) under the hood. By default, most DDS implementations (Fast-DDS, Cyclone DDS) buffer incoming messages in memory when your callback can’t keep up with the publishing rate. The theory is that temporary slowdowns shouldn’t drop data.

The problem? “Temporary” can mean different things to a robot.

If your camera node publishes at 30 Hz but your processing callback occasionally spikes to 40ms (lidar sync delays, thermal throttling on Jetson, whatever), messages start piling up in the DDS queue. Fast-DDS default queue depth is unlimited for RELIABLE QoS. That’s not a typo. Unlimited.

On a workstation with 64GB RAM, this is annoying. On a Jetson with 8GB shared between CPU and GPU, it’s fatal.

Here’s what the memory growth looked like when I finally instrumented it:

import psutil
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image

class MemoryMonitor(Node):
    def __init__(self):
        super().__init__('memory_monitor')
        self.subscription = self.create_subscription(
            Image, '/camera/image_raw', self.listener_callback, 10)
        self.timer = self.create_timer(5.0, self.log_memory)
        self.msg_count = 0

    def listener_callback(self, msg):
        self.msg_count += 1
        # Simulate slow processing (30Hz pub, 25Hz consume)
        time.sleep(0.04)  

    def log_memory(self):
        process = psutil.Process()
        mem_mb = process.memory_info().rss / 1024 / 1024
        self.get_logger().info(
            f'RSS: {mem_mb:.1f} MB | Messages: {self.msg_count}')

Output after 40 hours:

[memory_monitor]: RSS: 145.2 MB | Messages: 4320000
[memory_monitor]: RSS: 891.3 MB | Messages: 4321500  # 10 hours later
[memory_monitor]: RSS: 2847.1 MB | Messages: 4323000  # 20 hours later
[memory_monitor]: RSS: 6201.8 MB | Messages: 4324500  # 38 hours later
terminate called after throwing an instance of 'std::bad_alloc'

The memory growth wasn’t linear — it accelerated. My best guess is that DDS memory allocators fragment over time, and Jetson’s thermal throttling made callback latency worse as the GPU heated up during the day.

The QoS Profile That Saves You

ROS2 lets you override DDS defaults with QoS (Quality of Service) profiles. Most tutorials show you RELIABLE vs BEST_EFFORT, but the critical parameter is queue depth.

Here’s the fix:

from rclpy.qos import QoSProfile, QoSReliabilityPolicy, QoSHistoryPolicy

# Bad: default behavior
self.subscription = self.create_subscription(
    Image, '/camera/image_raw', self.callback, 10)  
    # This "10" only limits the ROS2 layer, not DDS!

# Good: actually bound the memory
qos_profile = QoSProfile(
    reliability=QoSReliabilityPolicy.RELIABLE,
    history=QoSHistoryPolicy.KEEP_LAST,
    depth=5,  # Maximum 5 messages buffered
    # This is the key part:
    liveliness=QoSLivelinessPolicy.AUTOMATIC,
    deadline=Duration(seconds=1),  # Drop messages older than 1s
)

self.subscription = self.create_subscription(
    Image, '/camera/image_raw', self.callback, qos_profile)

Why depth=5 specifically? For image topics at 30 Hz, 5 frames = 166ms of buffer. If your callback can’t process a frame in 166ms, you’re dropping frames anyway — might as well drop them explicitly instead of crashing 48 hours later.

The deadline parameter is insurance. If a message sits in the queue for >1 second (network hiccup, whatever), DDS discards it. This prevents “zombie” messages from ancient timestamps clogging your pipeline.

When to Use BEST_EFFORT Instead

For high-frequency sensor data (lidar, IMU), BEST_EFFORT is often better:

qos_profile = QoSProfile(
    reliability=QoSReliabilityPolicy.BEST_EFFORT,
    history=QoSHistoryPolicy.KEEP_LAST,
    depth=1,  # Only keep the latest message
)

This tells DDS: “If a message arrives while I’m processing the previous one, throw away the old one.” No buffering, no memory growth. The tradeoff is that you might miss data during CPU spikes, but for 100 Hz lidar scans, missing one frame is usually fine.

The math: a 1080p JPEG (compressed) is ~500 KB. At 30 Hz with RELIABLE and no depth limit, you’re allocating 15 MB/s if your callback lags. Over 48 hours, that’s 2.5 TB of turnover. Modern allocators can handle this on a workstation, but Jetson’s unified memory architecture means you’re also starving the GPU.

The Fast-DDS XML Escape Hatch

Sometimes you inherit code where the publisher (a vendor’s proprietary driver node) uses RELIABLE with no depth limit, and you can’t modify it. In that case, configure Fast-DDS globally via XML:

<!-- ~/.ros/fastdds.xml -->
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
  <subscriber profile_name="default_subscriber">
    <qos>
      <resourceLimitsQos>
        <max_samples>10</max_samples>
        <max_samples_per_instance>5</max_samples_per_instance>
      </resourceLimitsQos>
    </qos>
  </subscriber>
</profiles>

Then set the environment variable:

export FASTRTPS_DEFAULT_PROFILES_FILE=~/.ros/fastdds.xml

This caps all subscribers to 10 total samples across topics. It’s a sledgehammer, but it works when you can’t touch the source code.

The Sneaky Shared Pointer Leak

Even with QoS fixed, I still saw slow memory growth (100 MB over 72 hours, not fatal but annoying). The culprit was a classic C++ trap:

class ImageProcessor(Node):
    def __init__(self):
        super().__init__('processor')
        self.latest_image = None  # Storing raw message

    def callback(self, msg):
        self.latest_image = msg  # Holding a reference!
        process(msg.data)

ROS2 Python messages are backed by C++ shared pointers. Storing msg in self.latest_image prevents the C++ allocator from reclaiming the buffer. Over days, this leaks 10-50 MB.

Fix:

def callback(self, msg):
    # Copy only what you need
    self.latest_timestamp = msg.header.stamp
    image_array = np.array(msg.data, copy=True)  # Explicit copy
    process(image_array)
    # msg goes out of scope here, C++ buffer freed

Or use weak_ptr if you need to keep the reference for async processing (this requires rclpy weak reference support, which is experimental as of Humble).

Monitoring What Actually Matters

I wrote a quick node to track DDS queue depth in real time. This should’ve been my first step:

import rclpy
from rclpy.node import Node
from rcl_interfaces.msg import ParameterEvent

class QueueMonitor(Node):
    def __init__(self):
        super().__init__('queue_monitor')
        # This uses ROS2 introspection APIs (requires Iron+)
        self.timer = self.create_timer(10.0, self.check_queues)

    def check_queues(self):
        # Hacky: parse dds stats from /proc/<pid>/status
        # (Fast-DDS doesn't expose queue depth metrics by default)
        with open(f'/proc/{os.getpid()}/status', 'r') as f:
            for line in f:
                if line.startswith('VmRSS'):
                    rss_kb = int(line.split()[1])
                    self.get_logger().warn(
                        f'RSS: {rss_kb/1024:.1f} MB')

This is crude, but it caught the issue. For production, I’d use Perfetto tracing with ROS2’s built-in tracepoints, but that’s overkill for debugging.

What I’d Change Next Time

If I were doing this again, I’d:

1. Default to BEST_EFFORT for all sensor topics unless I had a specific reason (like mapping algorithms that need every lidar scan). The ROS2 docs push RELIABLE as the “safe” default, but for edge devices it’s a footgun.

2. Set depth=1 globally for any high-frequency topic (>10 Hz). The only exception is control loops where you genuinely need the last N commands buffered (e.g., trajectory following with 5-point lookahead).

3. Add a max_memory watchdog that kills the node if RSS exceeds 1.5 GB. Better to crash early and restart via systemd than to let the OOM killer nuke random processes.

4. Test with artificial slowdowns from day one. I should’ve added a time.sleep(random.uniform(0, 0.05)) in every callback during integration testing to simulate worst-case latency.

5. Use Cyclone DDS instead of Fast-DDS. Cyclone has saner defaults for embedded systems (though I haven’t verified this claim rigorously — take with a grain of salt).

The deeper issue is that ROS2’s abstraction leaks: you can write perfectly correct Python code that respects ROS2 APIs, yet still get bitten by DDS implementation details. The only way to avoid this is to understand the stack all the way down.

If you’re running ROS2 on Jetson for anything longer than a demo, invest in NVIDIA Nsight Systems to profile memory over time. And maybe grab some Blue Light Blocking Glasses — you’ll be staring at memory graphs at 2am.

Thermal Throttling Makes It Worse

One thing I didn’t mention earlier: Jetson thermal throttling compounds the problem. When the SoC hits 80°C (common under continuous GPU load), the CPU clocks drop from 2.2 GHz to 1.4 GHz. Your 35ms callback suddenly takes 50ms.

This means the “safe” queue depth at room temperature becomes unsafe after an hour of operation. I ended up adding a thermal monitor:

def check_thermal(self):
    with open('/sys/devices/virtual/thermal/thermal_zone0/temp') as f:
        temp_millicelsius = int(f.read().strip())
        temp_c = temp_millicelsius / 1000
        if temp_c > 75:
            self.get_logger().warn(
                f'Thermal throttling likely at {temp_c:.1f}°C')

At 75°C, I dynamically switch image topics from depth=5 to depth=2. Hacky, but it kept nodes alive during summer testing (ambient 35°C in the lab).

Why the Docs Don’t Warn You

The ROS2 documentation assumes you’re running on a workstation with 32+ GB RAM. The QoS tuning guide mentions depth as a “performance optimization,” not a “your robot will crash” issue. This is technically correct — on most hardware, unbounded queues just slow things down.

But on Jetson (or Raspberry Pi with ROS2), memory is the hard constraint. The official Nav2 tutorials use depth=10 for everything, which is fine for a 10-minute demo but deadly for 48-hour autonomy.

I’m not entirely sure why Fast-DDS defaults to unlimited queues. The DDS spec allows it, and I assume it’s optimized for enterprise systems with aggressive monitoring. But for robotics, I’d argue depth=10 should be the default.

Practical Benchmarks

I ran a controlled test: publishing 1080p JPEG images at 30 Hz to a subscriber with an artificial 40ms processing delay (so it’s always behind by 10 ms/frame).

The deadline=1s variant had slightly higher memory because RELIABLE mode retains messages for retransmission, but the deadline parameter forced cleanup of stale data.

All tests used Fast-DDS 2.10 on Jetpack 5.1 (Ubuntu 20.04, L4T 35.3.1). Python 3.8, ROS2 Humble.

FAQ

Q: Will switching to Cyclone DDS fix this without QoS changes?

Partially. Cyclone DDS has a default max_samples limit (I think it’s 32, but I haven’t verified), so you’re less likely to hit unbounded growth. But you still need to tune depth based on your callback latency — 32 buffered images is still 16 MB of wasted memory if you only need the latest frame.

Q: Does this apply to ROS1 too?

No. ROS1 uses TCP with a fixed subscriber queue (default 1000 messages), but it blocks the publisher if the queue fills. You get different problems (publisher stalls, deadlocks) but not silent memory leaks. ROS2’s DDS layer decouples publishers and subscribers, which is better for distributed systems but worse for embedded devices without tuning.

Q: How do I know if my node is leaking before it crashes?

Run ros2 topic hz /your/topic and ros2 topic bw /your/topic in parallel with watch -n 1 'ps aux | grep your_node'. If RSS grows while bandwidth is constant, you’re leaking. For deeper analysis, use Valgrind with ROS2 suppressions or Perfetto tracing.

When RELIABLE Is Actually Necessary

I don’t want to imply that BEST_EFFORT is always the answer. For command topics (velocity commands, gripper open/close), you absolutely need RELIABLE. Missing a “stop” command because of a dropped packet is unacceptable.

Rule of thumb:
– Sensor data (images, lidar, IMU): BEST_EFFORT, depth=1
– Commands (twist, joint goals): RELIABLE, depth=3-5
– State (odometry, pose): BEST_EFFORT, depth=1 (you only care about latest)
– Configuration (parameter updates): RELIABLE, depth=10 (but these are infrequent)

The mistake is using RELIABLE everywhere because it sounds safer. In practice, for high-frequency data, dropping old frames is safer than buffering them until your process dies.

What I’m still figuring out: optimal QoS for multi-robot coordination topics (e.g., fleet pose sharing at 1 Hz). RELIABLE with depth=5 seems reasonable, but I haven’t stress-tested it with 10+ robots on a single network. If you’ve done this, I’d be curious to hear how it went.