ONNX Runtime Mobile: 8ms Inference on iPhone 13

The 200ms Problem

Your edge AI model works great on a dev server. 30ms inference, low memory, rock solid. Then you ship it to a phone and suddenly you’re hitting 200ms per frame.

This happened on a real-time object detection prototype I was testing last month. The model was a MobileNetV3 backbone with a custom detection head — 4.2MB TFLite file, quantized to INT8, supposedly optimized for mobile. On my MacBook it ran at 28ms. On an iPhone 13 it crawled at 180-220ms, making the app unusable.

Turns out TFLite isn’t the only game in town anymore. ONNX Runtime Mobile (ORT Mobile) has quietly become a serious alternative, and in some cases it’s significantly faster.

Why I Switched from TFLite to ONNX Runtime

TFLite is the default choice for mobile AI — Google’s docs are good, the tooling is mature, and it’s built into the TensorFlow ecosystem. But it has limitations.

First, TFLite’s op coverage is incomplete. If your model uses a custom layer or a newer op (like certain attention mechanisms), you’re writing custom delegates or rewriting parts of your model. I hit this with a GroupNorm layer that TFLite didn’t support natively in the version I was using (2.12). The workaround was annoying.

Second, TFLite’s optimization story on iOS is good but not great. It leverages Metal via GPU delegate, but the CPU fallback path is slower than it should be for certain ops. On Android it’s better (NNAPI delegate), but iOS is where I saw the biggest performance gap.

ONNX Runtime Mobile supports the same hardware acceleration (CoreML on iOS, NNAPI on Android) but with better fallback performance. It’s also framework-agnostic — you can export from PyTorch, TensorFlow, or anything else that supports ONNX.

If you want to benchmark ONNX Runtime against TFLite on actual edge hardware instead of just your dev machine, the NVIDIA Jetson Nano Developer Kit lets you test both runtimes on real ARM silicon before deploying to production.

Exporting to ONNX: Not as Clean as the Docs Suggest

Exporting a PyTorch model to ONNX should be straightforward. The official docs make it look like a three-liner:

import torch
import torch.onnx

# Your trained model
model = MobileNetV3Detection()  # custom detection head
model.load_state_dict(torch.load('best.pth'))
model.eval()

# Dummy input matching your inference shape
dummy_input = torch.randn(1, 3, 320, 320)

# Export
torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    export_params=True,
    opset_version=14,  # 14 is widely supported
    do_constant_folding=True,
    input_names=['input'],
    output_names=['boxes', 'scores', 'classes'],
    dynamic_axes={'input': {0: 'batch_size'}}
)

This worked, but the resulting ONNX file was 16.8MB — 4x larger than my TFLite version. Why? The model wasn’t quantized.

ONNX doesn’t have a built-in post-training quantization tool like TFLite’s converter. You need to use onnxruntime.quantization separately:

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    model_input="model.onnx",
    model_output="model_quant.onnx",
    weight_type=QuantType.QUInt8  # unsigned 8-bit weights
)

This brought the file down to 4.4MB, close to the TFLite version. But dynamic quantization only quantizes weights, not activations. For full INT8 (weights + activations), you need static quantization with a calibration dataset:

from onnxruntime.quantization import quantize_static, CalibrationDataReader
import numpy as np

class ImageDataReader(CalibrationDataReader):
    def __init__(self, image_paths):
        self.image_paths = image_paths
        self.index = 0
        self.preprocess = preprocess_fn  # your normalization logic

    def get_next(self):
        if self.index >= len(self.image_paths):
            return None
        img = load_and_preprocess(self.image_paths[self.index])
        self.index += 1
        return {'input': img.numpy()}  # must match input name

# 100-500 calibration images recommended
calib_images = [f'calib/{i:04d}.jpg' for i in range(200)]
dr = ImageDataReader(calib_images)

quantize_static(
    model_input="model.onnx",
    model_output="model_int8.onnx",
    calibration_data_reader=dr,
    weight_type=QuantType.QInt8,  # signed for activations
    activation_type=QuantType.QInt8
)

This got me to 4.1MB with full INT8. The calibration step took about 3 minutes on 200 images.

Profiling 8ms inference times and debugging CoreML fallback paths at 3am? A Monster Energy Drink variety pack will keep you sharp when your optimization rabbit hole runs deeper than expected.

The CoreML Execution Provider on iOS

ORT Mobile on iOS can use three execution providers:
– CPU: Pure CPU inference (surprisingly fast for small models)
– CoreML: Apple’s ML framework, uses ANE (Apple Neural Engine) when available
– XNNPACK: Optimized CPU kernels (Facebook’s library)

CoreML is the big win. The Apple Neural Engine is a dedicated AI accelerator present in A11+ chips (iPhone 8 and later). It’s dramatically faster than CPU for supported ops.

Here’s the integration code for iOS (Swift):

import onnxruntime_objc

class ONNXModel {
    private var session: ORTSession?

    init(modelPath: String) throws {
        let env = try ORTEnv(loggingLevel: .warning)

        let options = try ORTSessionOptions()

        // Enable CoreML execution provider
        try options.appendCoreMLExecutionProvider()

        // Fallback to CPU if CoreML can't handle an op
        // (this is automatic, but making it explicit)

        session = try ORTSession(
            env: env,
            modelPath: modelPath,
            sessionOptions: options
        )
    }

    func predict(image: CVPixelBuffer) throws -> (boxes: [[Float]], scores: [Float]) {
        // Convert CVPixelBuffer to ORTValue (ONNX Runtime tensor)
        let inputTensor = try preprocessImage(image)  // -> ORTValue

        let outputs = try session?.run(
            withInputs: ["input": inputTensor],
            outputNames: ["boxes", "scores"],
            runOptions: nil
        )

        // Extract outputs (simplified - real code needs shape handling)
        let boxes = try outputs?["boxes"]?.tensorData() as? [Float] ?? []
        let scores = try outputs?["scores"]?.tensorData() as? [Float] ?? []

        return (parseBoxes(boxes), scores)
    }
}

The key line is appendCoreMLExecutionProvider(). Without it, inference fell back to CPU and ran at ~90ms. With CoreML enabled, it dropped to 8-12ms.

But there’s a catch: CoreML compilation happens on first run and adds ~2 seconds of latency. The compiled model is cached, so subsequent runs are fast, but that initial delay is noticeable. I haven’t found a good way to pre-compile during app install.

Benchmark: TFLite vs ONNX Runtime on Real Hardware

I tested the same MobileNetV3 detection model (4.2MB, INT8 quantized) on iPhone 13 (A15 Bionic) and a mid-range Android device (Snapdragon 778G).

Setup:
– Input: 320×320 RGB images
– Output: 100 bounding boxes + confidence scores
– Measurement: average over 100 inferences after warmup

The iPhone 13 + CoreML result was shocking. 8ms inference is 23x faster than TFLite GPU. Even on Android, ORT was slightly ahead, though the gap was smaller.

Why is TFLite so much slower on iOS? My best guess is that TFLite’s Metal delegate doesn’t fully utilize the Neural Engine. The A15 chip has a 16-core ANE capable of 15.8 TOPS, but TFLite seems to route most ops through the GPU instead. CoreML routes directly to the ANE for supported layers.

Memory Footprint: The Hidden Cost

Latency isn’t the only metric. Memory usage matters on mobile, especially for background apps or devices with <4GB RAM.

I profiled both frameworks using Xcode Instruments:

• TFLite: 38MB peak memory during inference (GPU delegate)
• ONNX Runtime (CoreML): 52MB peak memory
• ONNX Runtime (CPU): 29MB peak memory

CoreML’s higher memory usage comes from the compiled model cache and ANE buffers. If you’re running other heavy tasks (e.g., ARKit + detection), this could push you into memory pressure.

The TFLite CPU delegate was the most memory-efficient at 22MB, but too slow for real-time use.

Op Support: Where ONNX Runtime Falls Short

Not every ONNX op is supported by CoreML. If your model uses an unsupported op, ORT falls back to CPU for that layer, killing performance.

I hit this with a ReduceSum op in my detection head. CoreML doesn’t support ReduceSum with dynamic axes (as of iOS 16). The workaround was to rewrite that part of the model using GlobalAveragePool, which CoreML does support.

You can check op support with:

import onnx
from onnxruntime.tools.symbolic_shape_infer import SymbolicShapeInference

model = onnx.load("model_int8.onnx")
for node in model.graph.node:
    print(f"{node.name}: {node.op_type}")

CoreML’s supported ops are listed in Apple’s docs, but it’s incomplete and version-dependent. iOS 17 added support for more transformer ops (LayerNorm, GELU), which is promising for future models.

Practical Deployment Tips

1. Model size vs latency tradeoff: INT8 quantization cuts size by ~75% with minimal accuracy loss (<2% mAP drop in my tests). Dynamic quantization is faster to set up but gives worse compression.

2. Batch size = 1 on mobile: Don’t bother optimizing for batch>1. Mobile inference is always single-image.

3. Warmup is mandatory: First inference is always slow (model loading + compilation). Run a dummy inference on app launch:

func warmup() {
    let dummyImage = createBlankImage(width: 320, height: 320)
    _ = try? predict(image: dummyImage)
}

4. Monitor thermal throttling: On prolonged use (e.g., video processing), the ANE throttles. I saw latency creep from 8ms to 18ms after 60 seconds of continuous inference. TFLite throttled less aggressively, staying around 200ms.

5. Test on old devices: The performance gap widens on older hardware. On an iPhone X (A11), ORT CoreML was still 3x faster than TFLite, but absolute latency was 28ms vs 95ms.

The Distribution Problem

ONNX Runtime Mobile is less widely adopted than TFLite, which means:
– Fewer Stack Overflow answers when you hit issues
– Less community tooling (no Netron-style visualizers optimized for ORT Mobile)
– Apple’s CoreML tools (coremltools) are sometimes a better fit if you’re iOS-only

If you’re shipping cross-platform and need one solution, TFLite’s maturity might outweigh ORT’s performance edge. But if you can maintain separate iOS/Android pipelines, ORT + CoreML on iOS is hard to beat.

What I’d Use Today

For iOS-only apps: ONNX Runtime with CoreML, no question. The latency improvement is too large to ignore.

For cross-platform (iOS + Android): It depends. If your model is <10MB and latency-critical, maintain separate ORT builds. If you’re >10MB or latency is less critical (e.g., batch photo processing), TFLite’s simplicity wins.

For Android-only: The gap is smaller. NNAPI works well with both frameworks. I’d default to TFLite unless you’re already in the ONNX ecosystem (e.g., training in PyTorch and want one export path).

FAQ

Q: Can I use ONNX Runtime on devices older than iPhone 8?

Yes, but you lose the Neural Engine. On iPhone 7 and older (A10 chip and below), CoreML falls back to GPU/CPU, and the performance advantage over TFLite disappears. In my tests on an iPhone 7, both frameworks ran at ~150ms.

Q: Does ONNX Runtime support GPU delegate on Android like TFLite?

Sort of. ORT Mobile supports OpenCL and Vulkan execution providers on Android, but they’re less mature than TFLite’s GPU delegate. NNAPI is the recommended path for Android hardware acceleration, and both frameworks support it well.

Q: How do I debug CoreML fallback to CPU?

Enable verbose logging in ORTSessionOptions and check the console output. ORT will print warnings like [W:onnxruntime:CoreML] Unsupported op: ReduceSum, falling back to CPU. You’ll see exactly which ops aren’t running on ANE. From there, either rewrite those ops or accept the performance hit.

The 8ms Ceiling

Even with CoreML, I couldn’t get below 8ms consistently. The variance was 7-10ms, with occasional spikes to 15ms (likely thermal throttling or OS scheduling). For comparison, a native CoreML model (exported directly from PyTorch via coremltools) hit 6ms, suggesting there’s still overhead in ORT’s CoreML EP.

But 8ms is fast enough for 120fps processing, which is more than enough for any real-time mobile app I’ve worked on. The bigger bottleneck is usually camera frame capture and UI rendering, not inference.

I’m curious whether ORT’s CoreML backend will improve as Apple exposes more ANE capabilities in future iOS versions. The gap between ORT+CoreML and native CoreML suggests there’s room for optimization.