TFLite & ONNX Mobile Setup: 2x Speed Left on Table

The Default Config Is Lying to You

MobileNetV2 running at 180ms on a Raspberry Pi 4. That’s what I got out of the box with TFLite’s default interpreter. Switch on XNNPack, set four threads, and suddenly it’s 35ms. Same model, same hardware, same framework—5x faster just from configuration.

Most mobile ML tutorials skip this part. They benchmark frameworks, declare winners, and leave you with code that’s criminally slow. The truth? Both TFLite and ONNX Runtime Mobile can hit near-identical latency on most ARM devices when properly configured. The 2x speed gap everyone argues about? It’s usually a setup mistake, not a framework limitation.

I’ve spent weeks profiling inference on everything from Pi Zeros to Snapdragon 888s, and the pattern is consistent: default configurations leave 40-80% of your hardware’s capability untouched. This isn’t about which framework is “better.” It’s about not shooting yourself in the foot.

TFLite Delegates: The 3x Speedup You’re Missing

TFLite’s delegate system is where the real performance lives. Without delegates, you’re running on a generic CPU backend that ignores SIMD instructions, multithreading, and any accelerator your device might have.

XNNPack: The Universal Speed Button

XNNPack is TFLite’s optimized CPU backend, and it should be your default for any ARM deployment. It’s not experimental—Google ships production models with XNNPack in mind. The performance delta is dramatic.

import tensorflow as tf
import numpy as np
import time

# Load model
model_path = 'mobilenet_v2_1.0_224.tflite'

# BAD: Default configuration
interpreter_slow = tf.lite.Interpreter(model_path=model_path)
interpreter_slow.allocate_tensors()

# GOOD: XNNPack enabled
interpreter_fast = tf.lite.Interpreter(
    model_path=model_path,
    experimental_delegates=[tf.lite.experimental.load_delegate('libXNNPackDelegate.so')]
)
interpreter_fast.allocate_tensors()

On a Raspberry Pi 4 with MobileNetV2:
– Default interpreter: 180ms
– XNNPack enabled: 60ms

That’s a 3x improvement from one line of code. But here’s where people mess up: the delegate must be added before allocate_tensors(). If you add it after, TFLite ignores it silently. No error, no warning, just slow inference while you wonder why the benchmark numbers don’t match.

GPU Delegate: When It Works, It’s Magic

The GPU delegate promises 3-5x speedups on Android’s OpenGL ES and iOS Metal backends. In practice, it’s powerful but picky.

# Android GPU delegate setup
gpu_delegate = tf.lite.experimental.load_delegate('libGpuDelegate.so')
interpreter = tf.lite.Interpreter(
    model_path='quantized_model.tflite',
    experimental_delegates=[gpu_delegate]
)

The catch: GPU delegates hate dynamic tensor shapes. If your model accepts variable batch sizes or sequence lengths, the delegate silently falls back to CPU. I’ve seen projects where developers added GPU delegates, benchmarked once, got great numbers, then shipped code that ran 5x slower in production because real-world inputs triggered the fallback.

Always export with fixed shapes:

# Converting from SavedModel with fixed shape
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]

# Force fixed input shape (batch=1 for mobile)
# This enables GPU delegate optimization
converter.inference_input_type = tf.uint8  # INT8 for quantization
converter.inference_output_type = tf.float32
tflite_model = converter.convert()

NNAPI: The Android Wildcard

Android’s Neural Networks API routes inference to vendor-specific accelerators—Qualcomm’s Hexagon DSP, Samsung’s NPU, whatever silicon your device has. When it works, NNAPI delivers 2-8x speedups with zero code changes.

nnapi_delegate = tf.lite.experimental.load_delegate('libNnapiDelegate.so')
interpreter = tf.lite.Interpreter(
    model_path=model_path,
    experimental_delegates=[nnapi_delegate]
)

The problem is fragmentation. NNAPI support varies wildly across Android versions and manufacturers. A model that flies on a Pixel 7 might crawl on a Samsung A52. My recommendation: use NNAPI as an optimization layer, not a requirement. Always test on your target device matrix.

Thread Configuration: The Other 2x

Delegates get all the attention, but thread configuration often delivers equal speedups.

TFLite defaults to single-threaded execution on many ARM devices. On a quad-core Cortex-A72, that’s leaving 75% of your compute on the table.

import os

interpreter = tf.lite.Interpreter(model_path=model_path)
interpreter.set_num_threads(os.cpu_count())  # Use all cores
interpreter.allocate_tensors()

But more threads aren’t always better. On memory-constrained devices like the Pi Zero (512MB RAM), setting threads to 4 can cause memory thrashing. The sweet spot is usually min(cpu_count(), 4) for edge devices.

The TFLite Setup Checklist That Actually Matters

Here’s the order that matters:

import tensorflow as tf
import os

def setup_tflite_properly(model_path):
    # 1. Load interpreter with delegate (BEFORE allocate_tensors)
    interpreter = tf.lite.Interpreter(
        model_path=model_path,
        experimental_delegates=[
            tf.lite.experimental.load_delegate('libXNNPackDelegate.so')
        ]
    )

    # 2. Set threads (BEFORE allocate_tensors)
    interpreter.set_num_threads(min(os.cpu_count(), 4))

    # 3. Allocate tensors ONCE
    interpreter.allocate_tensors()

    # 4. Cache input/output details
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    return interpreter, input_details, output_details

# Setup once at initialization
interpreter, inputs, outputs = setup_tflite_properly('model.tflite')

# Inference loop (reuse everything)
for image in batch:
    interpreter.set_tensor(inputs[0]['index'], image)
    interpreter.invoke()
    result = interpreter.get_tensor(outputs[0]['index'])

The order is critical. I’ve reviewed production code that added delegates after allocation, set threads inside the inference loop, and reallocated tensors per frame. Each mistake costs 20-50% performance.

ONNX Runtime Mobile: The Provider Dance

ONNX Runtime’s architecture is different—execution providers handle hardware acceleration instead of delegates. Same concept, different API.

Execution Provider Priority

ONNX Runtime checks providers in order and uses the first one that supports each operation. If you only specify NNAPI and an op isn’t supported, the whole model fails instead of gracefully falling back.

import onnxruntime as ort
import numpy as np

# BAD: No fallback, will crash on unsupported ops
session_bad = ort.InferenceSession('model.onnx', providers=['NNapiExecutionProvider'])

# GOOD: Provider chain with fallback
session_good = ort.InferenceSession(
    'model.onnx',
    providers=['NNapiExecutionProvider', 'CPUExecutionProvider']
)

The provider list is a priority queue, not a selection. Each operation routes to the highest-priority provider that supports it. A single custom op can force your entire model to CPU if you don’t have a fallback.

Session Options: Hidden Performance Multipliers

ONNX Runtime’s session options are where most people leave performance on the table.

import onnxruntime as ort

def setup_onnx_properly(model_path):
    sess_options = ort.SessionOptions()

    # Thread configuration
    sess_options.intra_op_num_threads = 4  # Threads within each op
    sess_options.inter_op_num_threads = 2  # Threads between ops (pipeline)

    # Graph optimization (critical!)
    sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

    # Memory arena strategy (reduces allocation overhead)
    sess_options.enable_cpu_mem_arena = True

    session = ort.InferenceSession(
        model_path,
        sess_options=sess_options,
        providers=['NNapiExecutionProvider', 'CPUExecutionProvider']
    )

    return session

Graph optimization is the big one. ORT_ENABLE_ALL fuses operations like Conv→BatchNorm→ReLU into single kernels, eliminating intermediate memory allocations. The cost is a slightly longer model load time (usually <100ms), but inference speed improves 30-50%.

The Warm-up Problem

ONNX Runtime compiles optimizations on first inference. If you benchmark without warm-up, you’re including 50-200ms of one-time compilation overhead.

# Warm up with dummy inference
input_name = session.get_inputs()[0].name
input_shape = session.get_inputs()[0].shape
dummy_input = np.zeros(input_shape, dtype=np.float32)

# Run 2-3 warm-up inferences
for _ in range(3):
    session.run(None, {input_name: dummy_input})

# Now benchmark actual performance
start = time.perf_counter()
for _ in range(100):
    session.run(None, {input_name: real_input})
end = time.perf_counter()
print(f"Average latency: {(end - start) / 100 * 1000:.1f}ms")

I’ve seen benchmark posts claiming ONNX is 3x slower than TFLite when the real issue was measuring warm-up time. Both frameworks need warm-up for accurate comparison.

Platform-Specific Provider Selection

Not all providers work everywhere. Here’s what actually runs on each platform:

The mistake I see most often: developers assume NNAPI works everywhere. It doesn’t. On budget Android phones, NNAPI either doesn’t exist or routes to CPU anyway. Profile on your actual target devices.

Quantization: The Prerequisite Everyone Forgets

All these delegate and provider optimizations assume you’ve quantized your model. Running FP32 inference on mobile is like driving with the parking brake on.

INT8 quantization delivers roughly:
– 4x model size reduction (30MB → 7.5MB)
– 2-4x inference speedup
– <1% accuracy loss for most vision tasks

The accuracy loss $\Delta_{acc}$ from post-training quantization can be estimated as:

$$\Delta_{acc} \approx \frac{1}{N}\sum_{i=1}^{N} |f_{fp32}(x_i) – f_{int8}(x_i)|$$

where $f_{fp32}$ and $f_{int8}$ are the floating-point and quantized model outputs respectively. For MobileNetV2 on ImageNet, this typically stays under 0.8% Top-1 accuracy drop.

# TFLite quantization
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]

# Representative dataset for calibration
def representative_dataset():
    for _ in range(100):
        yield [np.random.rand(1, 224, 224, 3).astype(np.float32)]

converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

quantized_model = converter.convert()

For ONNX, use the onnxruntime quantization tools:

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    'model.onnx',
    'model_int8.onnx',
    weight_type=QuantType.QInt8
)

Dynamic quantization is easier (no calibration data needed) but gives smaller speedups than static quantization. Start with dynamic, move to static if you need more performance.

When TFLite Wins vs When ONNX Wins

After all this optimization, which framework should you choose? The honest answer: it depends on your deployment target.

TFLite wins when:
– You’re targeting Raspberry Pi or resource-constrained ARM (TFLite’s binary is smaller)
– Your model was trained in TensorFlow/Keras (direct export, no conversion errors)
– You need GPU delegate on Android (more mature than ONNX’s mobile GPU support)

ONNX Runtime wins when:
– You’re targeting iOS with CoreML (ONNX → CoreML is seamless)
– You trained in PyTorch (direct export via torch.onnx)
– You need Jetson deployment (TensorRT execution provider is excellent)
– Cross-platform consistency matters (same runtime on Android, iOS, Edge, Server)

If you’ve read my earlier piece on ONNX INT8 vs FP16 on Jetson Orin Nano, you know ONNX’s TensorRT provider is hard to beat on NVIDIA hardware. But for pure mobile deployment, the performance gap between properly configured TFLite and ONNX is usually under 15%.

The Full Setup Comparison

Here’s what proper setup buys you on a Snapdragon 888 with MobileNetV2:

8x improvement from configuration alone.

The framework difference? About 15%. The setup difference? 8x. Focus your optimization energy accordingly.

What the Benchmarks Don’t Tell You

Latency benchmarks assume steady-state inference. Real mobile apps have other concerns:

Model load time matters for cold starts. TFLite loads slightly faster because its flatbuffer format doesn’t require parsing. ONNX’s protobuf needs deserialization. On a mid-range phone, this can be 200-400ms difference for larger models.

Memory pressure varies between frameworks. TFLite tends to use 10-20% less peak memory because of its more aggressive buffer reuse. On 1GB RAM devices, this can mean the difference between running and crashing.

Battery drain correlates roughly with compute time, but delegate choice matters. GPU delegates use more power per inference than optimized CPU, but finish faster. For always-on inference (like continuous camera processing), test total energy, not just latency.

I’m honestly not sure why ONNX Runtime’s memory footprint runs higher in my tests—it might be the more aggressive graph optimization holding intermediate tensors, or the execution provider architecture requiring more scaffolding. The docs don’t explain it, and I haven’t dug into the source code yet.

Debugging the Debug Output

When your inference is slower than expected, both frameworks offer profiling:

# TFLite profiling
interpreter = tf.lite.Interpreter(model_path=model_path)
interpreter.allocate_tensors()

# Enable profiler
interpreter.invoke()  # First run

# Get profiling info
for op in interpreter.get_tensor_details():
    print(f"{op['name']}: {op['dtype']}")

# ONNX Runtime profiling
sess_options = ort.SessionOptions()
sess_options.enable_profiling = True

session = ort.InferenceSession('model.onnx', sess_options=sess_options)
session.run(None, inputs)

# Profile JSON saved to working directory
prof_file = session.end_profiling()
print(f"Profile saved to: {prof_file}")

The ONNX profiler outputs JSON that’s easier to parse programmatically. TFLite’s profiling is more manual but shows you exactly which ops are falling back to CPU when delegates fail.

The 5-Minute Checklist Before Shipping

1. Quantize your model (INT8 for maximum speed, FP16 as minimum)
2. Add delegates/providers before tensor allocation
3. Set thread count to min(cpu_count(), 4) for edge devices
4. Enable graph optimization (ORT_ENABLE_ALL for ONNX)
5. Profile on target device (Pi benchmark ≠ phone benchmark)
6. Warm up before benchmarking (2-3 dummy inferences)
7. Test provider fallbacks (NNAPI doesn’t work everywhere)
8. Check memory pressure (mobile has limited RAM)

Skipping any of these leaves 20-50% performance on the table. Skipping multiple compounds the loss.

FAQ

Q: Why is my TFLite GPU delegate slower than CPU?

Data transfer overhead. Copying tensors to GPU memory takes time. For small models (MobileNetV2 and smaller), the transfer cost can exceed the compute savings. GPU delegates shine on larger models (EfficientNet-L, YOLO) where compute time dominates. Also check that your model doesn’t have unsupported ops forcing partial CPU fallback—that’s the silent killer.

Q: Can I use both NNAPI and GPU delegate in TFLite?

No. TFLite only supports one delegate at a time. If you add multiple, the last one wins. The recommended approach is to benchmark each delegate on your target device and pick the fastest. For most modern Android phones (Snapdragon 855+), NNAPI beats GPU delegate because it routes to specialized DSPs rather than the general GPU.

Q: How do I know if my ONNX execution provider is actually being used?

Check the session’s providers after creation: print(session.get_providers()). The first provider in the returned list is your active one. If it shows ‘CPUExecutionProvider’ when you specified NNAPI, your ops aren’t supported. Enable verbose logging with sess_options.log_severity_level = 0 to see exactly which ops fell back.

Late-night debugging session? Mechanical Keyboard Wrist Rest saved my forearms during the week I spent benchmarking across 12 devices.

What’s Next

The setup optimizations I’ve covered get you to 80% of peak performance. The last 20% requires model architecture changes—knowledge distillation, neural architecture search for mobile, or custom op implementations.

I’m curious about the new Qualcomm AI Engine Direct (QNN) provider for ONNX Runtime. The benchmarks look promising on Snapdragon 8 Gen 3, but I haven’t tested it against the TFLite Hexagon delegate on the same hardware. That’s a comparison I want to run but haven’t had the device access to do properly.

For now, my recommendation: use TFLite if you’re TensorFlow-native and targeting Android primarily. Use ONNX Runtime if you’re PyTorch-native, need iOS CoreML, or want one runtime across all platforms. Either way, configure properly before declaring a winner. The default settings are lying to you about what’s possible.