TFLite GPU Delegate Crashes on Jetson: XNNPACK Fix

The GPU Delegate Isn’t Always Faster

TFLite’s GPU delegate crashes on Jetson devices more often than it accelerates inference. The promise is simple: offload compute to the GPU, get faster inference. The reality? Segfaults, cryptic CUDA errors, and latency that’s worse than CPU.

I’ve seen this pattern across Jetson Nano, Xavier NX, and Orin modules. You enable the GPU delegate, run inference, and get either a crash or performance that makes you wonder why you bothered. The XNNPACK delegate, meanwhile, just works — and often beats GPU latency by 20-40% on common vision models.

This isn’t a hardware limitation. It’s a mismatch between what TFLite’s GPU delegate expects and what Jetson actually provides.

Why the GPU Delegate Fails

Jetson devices run NVIDIA’s Tegra architecture, which combines ARM CPU cores with CUDA-capable Maxwell/Pascal/Ampere GPUs. TFLite’s GPU delegate was primarily designed for mobile GPUs (Mali, Adreno) using OpenGL ES compute shaders. Jetson support exists, but it’s treated as a secondary target.

The delegate tries to use OpenGL ES 3.1+ compute shaders by default. On Jetson, this path is buggy. You’ll see errors like:

Failed to create OpenGL context
Segmentation fault (core dumped)

or more subtle issues where inference runs but produces garbage output because shader precision doesn’t match what the model expects.

Even when it works, the GPU path has overhead. Each inference requires:

1. Copying input tensor from CPU to GPU memory
2. Executing compute shaders (kernel launch overhead)
3. Synchronizing GPU execution
4. Copying output tensor back to CPU

For small models (MobileNet, EfficientNet-Lite), the memory copy overhead dominates. You spend more time on PCIe transfers than actual compute. The GPU delegate only wins when the model is large enough that compute savings outweigh transfer costs — typically models with 20M+ parameters.

XNNPACK: The Boring Solution That Works

XNNPACK is TFLite’s CPU-optimized delegate. It uses hand-written NEON SIMD intrinsics for ARM processors. On Jetson’s ARM Cortex-A57/A78 cores, this means vectorized operations without the GPU transfer tax.

Here’s a minimal benchmark on Jetson Nano (MobileNetV2 1.0, 224×224 input):

import numpy as np
import tensorflow as tf
import time

# Load model
interpreter_cpu = tf.lite.Interpreter(model_path="mobilenet_v2.tflite")
interpreter_cpu.allocate_tensors()

interpreter_xnnpack = tf.lite.Interpreter(
    model_path="mobilenet_v2.tflite",
    experimental_delegates=[tf.lite.experimental.load_delegate('libxnnpack_delegate.so')]
)
interpreter_xnnpack.allocate_tensors()

# Warmup + benchmark
input_data = np.random.randn(1, 224, 224, 3).astype(np.float32)

def bench(interpreter, name, runs=100):
    input_idx = interpreter.get_input_details()[0]['index']
    output_idx = interpreter.get_output_details()[0]['index']

    # Warmup
    for _ in range(10):
        interpreter.set_tensor(input_idx, input_data)
        interpreter.invoke()

    start = time.perf_counter()
    for _ in range(runs):
        interpreter.set_tensor(input_idx, input_data)
        interpreter.invoke()
        _ = interpreter.get_tensor(output_idx)
    elapsed = (time.perf_counter() - start) / runs * 1000
    print(f"{name}: {elapsed:.1f}ms")

bench(interpreter_cpu, "CPU baseline")
bench(interpreter_xnnpack, "XNNPACK")

On Jetson Nano (TFLite 2.12, JetPack 4.6.1):

CPU baseline: 147.3ms
XNNPACK: 89.2ms

That’s a 1.65x speedup with zero tuning. XNNPACK uses FP32 by default, but you can quantize to INT8 and see another 2-3x improvement.

The GPU delegate, when it doesn’t crash, gives around 110-130ms — slower than XNNPACK and far less reliable.

When GPU Delegate Actually Wins

The GPU delegate isn’t useless. It wins on larger models where compute dominates memory transfer.

I tested EfficientDet-Lite3 (8.9M params, 512×512 input) on Jetson Xavier NX:

The CUDA backend for GPU delegate exists but isn’t enabled in prebuilt TFLite wheels. You need to compile TFLite from source with --define=tflite_with_xnnpack=true --define=cuda=true. This takes 2+ hours on Jetson and requires matching CUDA/cuDNN versions to your JetPack.

Even then, the CUDA path only helps for models above ~10M parameters. Below that threshold, XNNPACK’s lower overhead wins.

The INT8 Quantization Advantage

XNNPACK supports INT8 quantized models with zero configuration. Quantization reduces memory footprint and often speeds up inference by 2-3x on ARM.

Here’s the conversion (assumes you have a trained SavedModel):

import tensorflow as tf

def representative_dataset():
    # Use real calibration data, not random noise
    for _ in range(100):
        yield [np.random.randn(1, 224, 224, 3).astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model("model/")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

tflite_quant_model = converter.convert()
with open("model_int8.tflite", "wb") as f:
    f.write(tflite_quant_model)

Running this INT8 model with XNNPACK on Jetson Nano:

XNNPACK FP32: 89.2ms
XNNPACK INT8: 34.7ms

That’s 2.57x faster than FP32 XNNPACK, and 4.2x faster than baseline CPU. The GPU delegate doesn’t support INT8 on Jetson without custom CUDA kernels.

One gotcha: the representative_dataset function must use real data from your domain, not random noise. If you calibrate with random data, the quantization thresholds will be wrong and accuracy drops by 5-15%. I learned this the hard way on a defect detection model where random calibration cut accuracy from 94% to 81%.

Memory Pressure on 1GB Devices

Jetson Nano has 4GB RAM, but Xavier NX base config has only 8GB shared between CPU and GPU. When you enable the GPU delegate, it allocates CUDA memory buffers for intermediate tensors. This eats into your memory budget fast.

On a Nano running headless (no GUI), baseline memory usage is ~600MB. Load a FP32 MobileNetV2 model with GPU delegate:

free -h
# Before inference
Mem: 3.9G total, 0.6G used, 3.3G free

# After 10 inferences with GPU delegate
Mem: 3.9G total, 1.4G used, 2.5G free

The GPU delegate allocated ~800MB for buffers. If you’re running multiple models or have other services (ROS2 nodes, camera pipelines), you’ll hit OOM fast.

XNNPACK doesn’t pre-allocate large buffers. Memory usage grows modestly (~50-100MB for MobileNet) and shrinks after inference completes.

How to Force XNNPACK in Code

TFLite 2.10+ enables XNNPACK by default on ARM, but only if no other delegate is specified. If you’ve been fighting GPU crashes, explicitly request XNNPACK:

import tensorflow as tf

# Method 1: Explicit delegate loading
xnnpack_delegate = tf.lite.experimental.load_delegate(
    'libxnnpack_delegate.so',
    options={'num_threads': '4'}  # Match your CPU core count
)
interpreter = tf.lite.Interpreter(
    model_path="model.tflite",
    experimental_delegates=[xnnpack_delegate]
)

If you get OSError: libxnnpack_delegate.so: cannot open shared object file, your TFLite build doesn’t include XNNPACK. Install the prebuilt wheel:

pip install --upgrade tensorflow==2.12.0  # or later

On JetPack 4.6.x (TensorFlow 2.9), you might need to compile TFLite from source with XNNPACK enabled. The official NVIDIA TensorFlow builds for Jetson often skip XNNPACK to reduce binary size.

Edge Case: Why GPU Sometimes Wins on Orin

Jetson Orin modules (Nano, NX, AGX) use Ampere architecture with Tensor Cores. These are specialized matrix multiply units that can accelerate certain ops dramatically — if the delegate knows how to use them.

TFLite’s GPU delegate doesn’t use Tensor Cores. You need TensorRT for that. But if you’re stuck with TFLite (maybe you’re using a model topology TensorRT doesn’t support), the GPU delegate can win on Orin for models with heavy depthwise convolutions.

I tested MobileNetV3-Large (5.4M params) on Orin Nano:

CPU: 76ms
XNNPACK: 41ms
GPU (OpenGL): 38ms  # Didn't crash, surprisingly

The GPU delegate squeaked out a 7% win. But it’s inconsistent — run the same code with batch size > 1 and you get a crash. I wouldn’t deploy this in production.

The Math Behind Memory Transfer Overhead

Why does GPU delegate lose on small models? It’s the latency tax of copying data across PCIe.

For a 224×224×3 FP32 input tensor:

$$\text{Input size} = 224 \times 224 \times 3 \times 4 \text{ bytes} = 602{,}112 \text{ bytes} \approx 0.57 \text{ MB}$$

Jetson Nano’s PCIe 2.0 x4 link has theoretical bandwidth of 2 GB/s, but real-world sustained throughput is closer to 1.2 GB/s. Copying input to GPU:

$$t_{\text{copy\_in}} = \frac{0.57 \text{ MB}}{1200 \text{ MB/s}} \approx 0.48 \text{ ms}$$

Copying output back (assuming 1000-class softmax):

$$t_{\text{copy\_out}} = \frac{1000 \times 4 \text{ bytes}}{1200 \text{ MB/s}} \approx 0.003 \text{ ms}$$

Total transfer overhead: ~0.5ms. That doesn’t sound like much, but XNNPACK can run MobileNetV2 inference in 89ms. If GPU compute takes 80ms, you’ve added 0.5ms overhead for a 0.6% slowdown. Not a problem.

But now add kernel launch latency (CUDA kernel dispatch takes 5-10µs per layer), synchronization overhead, and OpenGL shader compilation costs. Real-world overhead is closer to 5-10ms, which turns a theoretical 80ms GPU inference into 90ms — slower than XNNPACK’s 89ms.

For larger models, compute time dominates. EfficientDet-Lite3’s 178ms GPU inference versus 289ms XNNPACK means the transfer overhead is negligible relative to compute savings.

What About TensorRT?

If you need GPU acceleration on Jetson, skip TFLite GPU delegate and use TensorRT. It’s NVIDIA’s inference engine designed specifically for Tegra.

TensorRT supports Tensor Cores on Orin, INT8 quantization, layer fusion, and dynamic tensor memory. The latency difference is significant:

(Benchmarks on Orin Nano, TensorRT 8.5.2)

The catch: TensorRT requires converting your model to ONNX, then to TensorRT engine format. Some TFLite ops don’t have ONNX equivalents (looking at you, tf.raw_ops.FusedBatchNormV3). You’ll need to write custom plugins or simplify your model.

If your model converts cleanly, TensorRT is the obvious choice. If you’re stuck with TFLite for compatibility reasons, XNNPACK is your best bet.

Debugging GPU Delegate Crashes

If you’re determined to make GPU delegate work, here’s how to get useful errors instead of segfaults.

Enable TFLite verbose logging:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '0'  # Show all logs

import tensorflow as tf
tf.debugging.set_log_device_placement(True)

interpreter = tf.lite.Interpreter(
    model_path="model.tflite",
    experimental_delegates=[tf.lite.experimental.load_delegate('libtensorflowlite_gpu_delegate.so')]
)

You’ll see output like:

INFO: Created TensorFlow Lite delegate for GPU.
INFO: Replacing 47 node(s) with delegate (TfLiteGpuDelegateV2) node, leaving 3 node(s) to TensorFlow Lite runtime.
ERROR: Following operations are not supported by GPU delegate:
  SPACE_TO_DEPTH: Operation not supported
  DEPTH_TO_SPACE: Operation not supported
ERROR: Failed to apply delegate.
Segmentation fault

That tells you which ops are unsupported. You can either:

1. Rebuild your model without those ops
2. Use partial delegation (let unsupported ops run on CPU)
3. Give up and use XNNPACK

Partial delegation example:

from tensorflow.lite.experimental.delegates import gpu

delegate_options = gpu.DelegateOptions(
    allow_precision_loss=True,
    enable_quantization=False
)
gpu_delegate = gpu.Delegate(options=delegate_options)

interpreter = tf.lite.Interpreter(
    model_path="model.tflite",
    experimental_delegates=[gpu_delegate]
)

This often fixes crashes but gives you the worst of both worlds: GPU overhead without full GPU acceleration.

Real-World Deployment Strategy

Here’s what I’d actually ship:

For models <10M parameters (MobileNet, EfficientNet-Lite0-2, small YOLOs): Use XNNPACK with INT8 quantization. Latency is good enough, memory footprint is low, and it never crashes.

For models 10M-50M parameters (EfficientDet, ResNet50, YOLOv5m): Use TensorRT if conversion works. Fall back to XNNPACK if TensorRT is too much hassle.

For models >50M parameters: You probably shouldn’t be running these on Jetson Nano/Xavier NX. Move to Orin or a discrete GPU.

If you absolutely must use GPU delegate (maybe you’re porting an Android app and want code parity), test thoroughly on the exact JetPack version you’ll deploy. GPU delegate behavior varies between JetPack 4.6, 5.0, and 5.1 because NVIDIA keeps changing OpenGL drivers.

And keep XNNPACK as a fallback. Detecting GPU delegate failure at runtime and switching to XNNPACK is trivial:

try:
    interpreter = tf.lite.Interpreter(
        model_path="model.tflite",
        experimental_delegates=[gpu_delegate]
    )
    interpreter.allocate_tensors()
except Exception as e:
    print(f"GPU delegate failed: {e}, falling back to XNNPACK")
    interpreter = tf.lite.Interpreter(
        model_path="model.tflite",
        experimental_delegates=[xnnpack_delegate]
    )
    interpreter.allocate_tensors()

This saved me on a remote deployment where JetPack auto-updated and broke GPU delegate overnight.

What I Still Don’t Understand

Why does GPU delegate work fine on some models but crash on others with identical ops? I’ve seen two MobileNetV2 models — same architecture, trained on different datasets — where one runs on GPU delegate and the other segfaults. My best guess is it’s related to weight distributions triggering edge cases in shader compilation, but I haven’t confirmed this.

Also, NVIDIA’s documentation claims GPU delegate should work on Jetson. Their sample code uses it. But every NVIDIA employee I’ve asked says “just use TensorRT.” There’s a mismatch between what’s officially supported and what actually works in production.

If you’ve got GPU delegate running reliably on Jetson in production, I’d love to know your config. I suspect it requires a very specific JetPack version and model topology.

FAQ

Q: Can I use GPU delegate and XNNPACK together for different models?

Yes, you can load multiple interpreters with different delegates in the same process. Just don’t try to apply both delegates to the same interpreter — TFLite will pick one and ignore the other (usually GPU if both are specified). If you’re running multiple models, profile each one separately and assign the fastest delegate per model.

Q: Does XNNPACK work on Raspberry Pi?

Yes, XNNPACK was originally developed for ARM mobile/embedded devices. It works great on Pi 4/5 (Cortex-A72/A76 cores). You’ll see similar speedups to Jetson — typically 1.5-2x over baseline CPU, and 2-3x more with INT8 quantization. Pi lacks GPU delegate support entirely (no CUDA, no OpenGL ES compute), so XNNPACK is your only acceleration option.

Q: Why not just use PyTorch Mobile instead of TFLite?

PyTorch Mobile has better GPU support on Jetson via Vulkan backend, but the model ecosystem is smaller. If you’re training in PyTorch, exporting to TorchScript, and deploying PyTorch Mobile, it’s a reasonable path. But most pretrained vision models (especially quantized ones) are distributed as TFLite, not TorchScript. Converting between frameworks adds another failure point. I’d only go PyTorch Mobile if your entire pipeline is already PyTorch. Debugging TFLite’s quirks means you can reuse the massive collection of optimized TFLite models on TensorFlow Hub and elsewhere — sometimes the ecosystem wins over technical elegance. And honestly, hauling around a bag of trail mix during a 6-hour model conversion debugging session is the real optimization.

Pick XNNPACK Unless You Know Why You Need GPU

The GPU delegate is a trap. It promises acceleration, delivers crashes, and eats memory. XNNPACK is boring, reliable, and fast enough for 80% of edge inference workloads.

If you’re deploying to Jetson and need maximum performance, learn TensorRT. If you need TFLite compatibility (for pretrained models, cross-platform code, or because your team already knows it), stick with XNNPACK and quantize to INT8.

The GPU delegate only makes sense if:

1. You’re running models >10M parameters
2. You’ve compiled TFLite with CUDA backend
3. You’ve tested on your exact JetPack version
4. You have a fallback when it inevitably breaks

For everything else, XNNPACK is the right default. Save yourself the debugging time and ship something that works.

I’m still curious whether future JetPack releases will fix GPU delegate reliability, or if NVIDIA will just deprecate it in favor of pushing everyone to TensorRT. The current state feels like abandonware — technically supported, practically unusable.