ONNX Runtime vs TFLite Android: 3x Speed Benchmark

The Benchmark That Made Me Question Everything

TFLite was supposed to be the gold standard for Android inference. That’s what the tutorials say, anyway. So when I ran a MobileNetV3 classification model through both runtimes on a Pixel 7, expecting maybe a 10-20% difference, watching ONNX Runtime crush TFLite by 3.2x on the same model wasn’t just surprising — it broke my mental model of how mobile inference works.

The numbers: 4.7ms average on ONNX Runtime, 15.1ms on TFLite. Same model architecture, same input resolution, same device. What’s going on?

Why TFLite Should Have Won (But Didn’t)

TFLite has home-field advantage on Android. Google built it specifically for mobile, the NNAPI delegate talks directly to hardware accelerators, and there are years of Android-specific optimizations baked in. ONNX Runtime, by contrast, started as a cross-platform inference engine that happened to add mobile support later.

So I assumed my benchmark was broken.

First thing I checked: was the ONNX model actually equivalent? I’d converted from the same PyTorch checkpoint using both tf2onnx (via a SavedModel intermediate) and direct ONNX export. The output tensors matched within $\epsilon < 10^{-5}$, so the models were functionally identical.

import numpy as np

# Quick sanity check I ran on desktop first
onnx_output = onnx_session.run(None, {"input": test_input})[0]
tflite_output = tflite_interpreter.get_tensor(output_details[0]['index'])

max_diff = np.max(np.abs(onnx_output - tflite_output))
print(f"Max output difference: {max_diff}")
# Output: Max output difference: 8.344650268554688e-06

So the models matched. The difference had to be runtime behavior.

Profiling the Gap: Where TFLite Loses Time

Android’s systrace told the real story. TFLite was spending significant time in delegate initialization and memory allocation between inference calls. Even with the NNAPI delegate enabled, there was visible overhead on every invoke() call that ONNX Runtime simply didn’t have.

Here’s what the profiling setup looked like:

// Kotlin measurement code - nothing fancy
val warmupRuns = 50
val benchmarkRuns = 200

// Warmup (critical - first runs are always slow)
repeat(warmupRuns) {
    interpreter.run(inputBuffer, outputBuffer)
}

val tfliteTimes = mutableListOf<Long>()
repeat(benchmarkRuns) {
    val start = System.nanoTime()
    interpreter.run(inputBuffer, outputBuffer)
    tfliteTimes.add(System.nanoTime() - start)
}

val onnxTimes = mutableListOf<Long>()
repeat(benchmarkRuns) {
    val start = System.nanoTime()
    ortSession.run(Collections.singletonMap("input", ortInput))
    onnxTimes.add(System.nanoTime() - start)
}

println("TFLite median: ${tfliteTimes.median() / 1_000_000.0}ms")
println("ONNX median: ${onnxTimes.median() / 1_000_000.0}ms")

The variance told an interesting story too. TFLite’s standard deviation was 2.3ms, while ONNX Runtime came in at 0.4ms. ONNX wasn’t just faster on average — it was dramatically more consistent.

The NNAPI Delegate Trap

My first instinct was to blame the NNAPI delegate. Maybe I’d configured it wrong?

// TFLite with NNAPI delegate
val nnapiOptions = NnApiDelegate.Options()
    .setAllowFp16(true)
    .setUseNnapiCpu(false) // Force hardware accelerator
    .setExecutionPreference(NnApiDelegate.Options.EXECUTION_PREFERENCE_FAST_SINGLE_ANSWER)

val nnapiDelegate = NnApiDelegate(nnapiOptions)
val interpreterOptions = Interpreter.Options()
    .addDelegate(nnapiDelegate)
    .setNumThreads(4)

val interpreter = Interpreter(modelBuffer, interpreterOptions)

Turning off NNAPI and running pure CPU inference on TFLite actually made things worse — now it was 4.1x slower than ONNX Runtime. The NNAPI delegate was helping, just not enough.

But here’s what I found when I dug into the ONNX Runtime configuration:

// ONNX Runtime with NNAPI execution provider
val sessionOptions = OrtSession.SessionOptions()
sessionOptions.addNnapi()  // That's it. Really.
sessionOptions.setIntraOpNumThreads(4)

val ortEnvironment = OrtEnvironment.getEnvironment()
val ortSession = ortEnvironment.createSession(modelBytes, sessionOptions)

ONNX Runtime’s NNAPI integration seemed more streamlined. My best guess is that the execution provider model (where providers are mostly stateless) has lower per-inference overhead than TFLite’s delegate system.

Memory Layout and the Hidden Cost of Tensor Copies

The real culprit, I think, is memory management. TFLite’s ByteBuffer API requires specific memory layouts, and there’s copying happening that’s hard to avoid:

// TFLite input preparation - watch for the rewind() trap
val inputBuffer = ByteBuffer.allocateDirect(1 * 224 * 224 * 3 * 4)
    .order(ByteOrder.nativeOrder())

fun prepareInput(bitmap: Bitmap) {
    inputBuffer.rewind()  // Forget this and you get garbage output

    val pixels = IntArray(224 * 224)
    bitmap.getPixels(pixels, 0, 224, 0, 0, 224, 224)

    for (pixel in pixels) {
        // NHWC layout expected by TFLite
        inputBuffer.putFloat(((pixel shr 16) and 0xFF) / 255.0f)
        inputBuffer.putFloat(((pixel shr 8) and 0xFF) / 255.0f)
        inputBuffer.putFloat((pixel and 0xFF) / 255.0f)
    }
}

ONNX Runtime’s OrtValue API felt more direct:

// ONNX Runtime input - slightly cleaner IMO
val floatArray = FloatArray(1 * 3 * 224 * 224)  // NCHW layout

fun prepareOnnxInput(bitmap: Bitmap) {
    val pixels = IntArray(224 * 224)
    bitmap.getPixels(pixels, 0, 224, 0, 0, 224, 224)

    // ONNX often expects NCHW (channel-first)
    for (i in pixels.indices) {
        val pixel = pixels[i]
        floatArray[i] = ((pixel shr 16) and 0xFF) / 255.0f
        floatArray[224*224 + i] = ((pixel shr 8) and 0xFF) / 255.0f
        floatArray[2*224*224 + i] = (pixel and 0xFF) / 255.0f
    }
}

val ortInput = OnnxTensor.createTensor(
    ortEnvironment,
    floatArray,
    longArrayOf(1, 3, 224, 224)
)

The layout difference (NHWC vs NCHW) shouldn’t affect inference speed directly, but I noticed TFLite was doing internal transposes on some operations that the ONNX model avoided. The operator fusion seemed more aggressive in ONNX Runtime’s graph optimization.

Does This Hold for Other Models?

One benchmark doesn’t prove much. I tested three more architectures to see if the pattern held:

The pattern held for off-the-shelf architectures but nearly disappeared for my simple custom model. This suggests ONNX Runtime’s advantages come from better handling of complex operator patterns — depthwise separable convolutions, squeeze-and-excitation blocks, the stuff modern efficient architectures are built from.

For simple sequential CNNs without fancy blocks, TFLite closes the gap significantly.

Quantization: Where Things Get Weird

I’d covered YOLOv8 INT8 quantization on Jetson before, so I expected quantized models to behave similarly here. They didn’t.

With INT8 quantization, the gap shrank:

# Post-training quantization for TFLite
import tensorflow as tf

def representative_dataset():
    for _ in range(100):
        yield [np.random.randn(1, 224, 224, 3).astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

quantized_tflite = converter.convert()

INT8 results on the same Pixel 7:

TFLite’s NNAPI delegate is highly optimized for INT8 operations on Qualcomm DSPs, which explains the dramatic improvement. The gap still exists, but it’s much smaller. If you’re committed to INT8, the runtime choice matters less.

The GPU Delegate Wildcard

I also tested TFLite’s GPU delegate, which I’d previously ignored because NNAPI seemed like the obvious choice:

val gpuOptions = GpuDelegate.Options()
    .setPrecisionLossAllowed(true)
    .setInferencePreference(GpuDelegate.Options.INFERENCE_PREFERENCE_FAST_SINGLE_ANSWER)

val gpuDelegate = GpuDelegate(gpuOptions)

This actually beat NNAPI by about 20% on my test device. GPU delegate gave TFLite 12.2ms on MobileNetV3 — still slower than ONNX Runtime’s 4.7ms, but much better than the 15.1ms NNAPI result.

Why didn’t I start with GPU? Honestly, the documentation pushes NNAPI as the default acceleration path, and I’d internalized that without questioning it. The GPU delegate has some limitations (not all ops supported, potential precision issues), but for inference-focused apps it’s worth benchmarking.

Memory Consumption and Initialization Time

Speed isn’t everything. I measured memory footprint and initialization time because these matter for real apps:

// Rough memory measurement - not perfect but indicative
val runtime = Runtime.getRuntime()
val beforeMem = runtime.totalMemory() - runtime.freeMemory()

// Initialize runtime...

val afterMem = runtime.totalMemory() - runtime.freeMemory()
println("Memory delta: ${(afterMem - beforeMem) / 1024 / 1024}MB")

Results for MobileNetV3-Small:

ONNX Runtime is slower to initialize and uses more memory. For apps where the model stays loaded, this is fine. For apps that load models dynamically or run inference occasionally, TFLite’s lighter footprint might matter more than raw inference speed.

The Build Size Problem

Here’s something nobody talks about: APK size impact.

TFLite’s Android library adds about 3MB to your APK (with NNAPI delegate). ONNX Runtime adds closer to 8MB. For some apps, that’s a dealbreaker. For others, the inference speed gain is worth it.

You can strip ONNX Runtime down using custom builds, but that’s a rabbit hole I haven’t fully explored. The official documentation covers custom builds, and theoretically you can get it under 5MB by excluding execution providers you don’t need.

Real-World Integration: The Gotchas

I hit a few issues integrating ONNX Runtime into an existing TFLite-based app:

1. Gradle dependency conflicts. ONNX Runtime’s protobuf dependency clashed with our existing Firebase setup. Fixed with explicit exclusions:

implementation('com.microsoft.onnxruntime:onnxruntime-android:1.16.3') {
    exclude group: 'com.google.protobuf', module: 'protobuf-javalite'
}

2. Threading model differences. ONNX Runtime’s intra-op parallelism behaves differently from TFLite’s thread pool. I had to tune setIntraOpNumThreads() separately rather than assuming the same value would work.

3. Error messages are cryptic. When something goes wrong in ONNX Runtime, you often get unhelpful exceptions. TFLite’s error reporting is cleaner. Debugging model loading issues took longer with ONNX.

When ONNX Runtime Makes Sense

After all this benchmarking, here’s my decision framework:

Use ONNX Runtime when:
– Your model uses complex modern architectures (EfficientNet, MobileNetV3, transformers)
– Inference latency is your primary bottleneck
– You need cross-platform compatibility (same ONNX model works on iOS, Windows, Linux)
– You’re already in the PyTorch ecosystem and want simpler export paths

Stick with TFLite when:
– You’re using INT8 quantized models heavily
– APK size is constrained
– You need fastest possible cold start times
– You’re already invested in the TensorFlow/Keras ecosystem
– Your model is simple (few operators, no fancy blocks)

The Cross-Platform Angle

One thing I haven’t tested thoroughly: whether the same ONNX model gives consistent performance across iOS and Android. I covered ONNX Runtime on iPhone 13 previously, and the numbers were impressive there too. The promise of “export once, deploy everywhere” is appealing, but I’m not sure the performance characteristics transfer perfectly across platforms.

My suspicion is that iOS’s CoreML execution provider has different optimization paths than Android’s NNAPI provider, so you might see different relative performance depending on the hardware.

FAQ

Q: Can I use ONNX Runtime and TFLite in the same app?

Yes, and it’s not uncommon. Some apps use TFLite for lightweight models (where init time matters) and ONNX Runtime for heavy-lifting inference where latency is critical. The runtimes coexist without issues, though your APK size grows.

Q: Does ONNX Runtime support all TFLite operators?

Not all, but the overlap is large for standard vision models. Custom TFLite ops don’t have automatic equivalents. If your model uses TFLite’s Flex delegate for TensorFlow ops, converting to ONNX requires going back to the original TensorFlow graph. Check the ONNX operator coverage before committing.

Q: How do I convert a TFLite model to ONNX directly?

You can’t convert TFLite to ONNX directly — there’s no reliable converter. You need to go back to the original framework (TensorFlow SavedModel or PyTorch checkpoint) and export to ONNX from there. Tools like tf2onnx work reasonably well for TensorFlow models: python -m tf2onnx.convert --saved-model ./model --output model.onnx.

What I’d Do Differently

If I were starting a new Android ML project today, I’d prototype with ONNX Runtime first. The 3x speed difference on modern architectures is significant enough to change what’s feasible in real-time applications. 15ms inference means ~66 FPS theoretical throughput; 5ms means you can layer additional processing without dropping frames.

But I’m still not entirely sure why the gap is so large. The operator fusion explanation makes sense, but I haven’t verified it by inspecting the optimized graphs side by side. That’s next on my list — actually dumping the ONNX graph after optimization passes and comparing operator counts to see what’s being fused.

The mobile inference landscape keeps shifting. Google’s been pushing LiteRT (the TFLite rebrand), and Microsoft keeps improving ONNX Runtime’s mobile story. Benchmarks from six months ago might not reflect current reality. If you’re making this decision for a production app, run your own benchmarks on your target devices. These numbers are from one Pixel 7 — your Galaxy S23 or budget phone might tell a different story.