TorchServe vs ONNX Runtime: First Inference in 5 Minutes

The 47ms Difference That Made Me Reconsider TorchServe

First inference on a ResNet-50 model: 62ms with ONNX Runtime, 109ms with TorchServe. That’s almost 2x slower for TorchServe — but here’s the thing, those numbers flip completely once you understand what each tool is actually optimizing for.

I ran these tests on an AWS t3.medium (2 vCPUs, 4GB RAM) because that’s what most people actually have access to when prototyping. The gap narrows dramatically on GPU instances, but CPU-only deployments are still the reality for many teams shipping their first model.

Why Setup Complexity Matters More Than You Think

TorchServe requires Java. That’s the first surprise for Python developers expecting a pip-install-and-go experience. The model archiver, the configuration files, the handler classes — there’s real infrastructure thinking baked in. ONNX Runtime? It’s pip install onnxruntime and you’re running inference in three lines.

But that simplicity is also ONNX Runtime’s limitation. It’s a runtime, not a serving framework. No batching, no model versioning, no health checks out of the box. You’re building all that yourself or bolting on FastAPI. TorchServe ships with production features that ONNX Runtime expects you to bring.

# ONNX Runtime: literally this simple
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("resnet50.onnx", providers=['CPUExecutionProvider'])
input_name = sess.get_inputs()[0].name
output = sess.run(None, {input_name: np.random.randn(1, 3, 224, 224).astype(np.float32)})
print(f"Output shape: {output[0].shape}")  # (1, 1000)

Three lines to inference. No server, no config, no Java. For prototyping or embedding inference in an existing service, this wins.

TorchServe Setup: The 15-Minute Reality

The official docs make it look clean. Reality is messier. Here’s what actually happens when you try to serve a ResNet-50:

# Step 1: Install TorchServe and model archiver
pip install torchserve torch-model-archiver torch-workflow-archiver

# Step 2: You need Java 11+
# On Ubuntu: sudo apt install openjdk-11-jdk
java -version  # confirm this works first

Already you’re outside Python’s comfort zone. The model archiver packages your model into a .mar file that TorchServe understands. This is where the handler pattern shows up:

# custom_handler.py
import torch
from torchvision import models, transforms
from ts.torch_handler.base_handler import BaseHandler
import io
from PIL import Image

class ResNetHandler(BaseHandler):
    def initialize(self, context):
        self.model = models.resnet50(pretrained=True)
        self.model.eval()
        self.transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                                 std=[0.229, 0.224, 0.225])
        ])

    def preprocess(self, data):
        images = []
        for row in data:
            image = row.get("data") or row.get("body")
            if isinstance(image, (bytes, bytearray)):
                image = Image.open(io.BytesIO(image))
            images.append(self.transform(image))
        return torch.stack(images)

    def inference(self, data):
        with torch.no_grad():
            return self.model(data)

    def postprocess(self, inference_output):
        # Top-5 predictions
        probs = torch.nn.functional.softmax(inference_output, dim=1)
        top5 = torch.topk(probs, 5)
        return [{
            "predictions": [
                {"class_idx": idx.item(), "probability": prob.item()}
                for idx, prob in zip(top5.indices[i], top5.values[i])
            ]
        } for i in range(len(inference_output))]

That’s ~40 lines just to serve a standard pretrained model. Now package it:

torch-model-archiver --model-name resnet50 \
    --version 1.0 \
    --handler custom_handler.py \
    --extra-files index_to_name.json \
    --export-path model_store

# Start the server
torchserve --start --model-store model_store --models resnet50=resnet50.mar

First request takes ~3 seconds. That’s model loading time, not inference. Subsequent requests hit 109ms average on that t3.medium. The warmup penalty is significant.

ONNX Export: Where 90% of Issues Hide

Before ONNX Runtime can do anything, you need an ONNX file. Export from PyTorch is theoretically simple:

import torch
from torchvision import models

model = models.resnet50(pretrained=True)
model.eval()

dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model,
    dummy_input,
    "resnet50.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch_size"}, "output": {0: "batch_size"}},
    opset_version=17  # Use 17 for best compatibility as of 2024+
)

ResNet exports cleanly. But custom models? I’ve hit operator support issues more times than I can count. The opset_version parameter is sneakily important — too low and you lose operators, too high and older ONNX Runtime versions choke. I covered common export failures in ONNX Export Pitfalls: 7 PyTorch → Production Gotchas if you want the full breakdown.

The dynamic axes argument enables variable batch sizes. Skip this and you’re stuck with whatever batch size you exported with — which is fine until someone sends a request during low traffic and you’re still computing a full batch.

First Inference Benchmark: Raw Numbers

I tested both on identical hardware with the same ResNet-50 weights, 100 inferences each, batch size 1:

ONNX Runtime is faster on CPU inference. The graph optimizations (constant folding, node fusion) that ONNX Runtime applies during session creation pay off. TorchServe runs PyTorch’s eager execution path, which carries interpreter overhead.

But watch what happens when you enable batching on TorchServe with batch_size=8 and max_batch_delay=100ms:

TorchServe’s automatic batching is genuinely useful. ONNX Runtime makes you build that batching logic yourself, and naive implementations rarely match TorchServe’s efficiency.

The Memory Footprint Problem

On a 1GB RAM server (like this Oracle Cloud instance I’m writing from), TorchServe’s 1.1GB footprint is a dealbreaker. You’d need swap, which murders latency. ONNX Runtime at 0.6GB fits with room for the OS and other processes.

The memory difference comes from TorchServe loading the full PyTorch runtime plus a JVM. ONNX Runtime is a leaner C++ inference engine with Python bindings. For memory-constrained deployments, this gap matters.

Configuration Files: TorchServe’s Hidden Complexity

TorchServe has a config.properties file that controls everything from worker counts to inference timeout:

# config.properties
inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
number_of_netty_threads=4
job_queue_size=100
model_store=/home/model-server/model-store
load_models=all

# Model-specific config
models={\
  "resnet50": {\
    "1.0": {\
        "defaultVersion": true,\
        "marName": "resnet50.mar",\
        "minWorkers": 1,\
        "maxWorkers": 4,\
        "batchSize": 8,\
        "maxBatchDelay": 100\
    }\
  }\
}

This is powerful for production. But for a first inference test? Overkill. ONNX Runtime’s session options are simpler:

opts = ort.SessionOptions()
opts.intra_op_num_threads = 4
opts.inter_op_num_threads = 2
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

sess = ort.InferenceSession("resnet50.onnx", opts, providers=['CPUExecutionProvider'])

Four lines versus a properties file and JSON config. For rapid iteration, ONNX Runtime wins on developer experience.

GPU Inference: The Gap Narrows

On an NVIDIA T4, the story changes. TorchServe’s overhead becomes negligible compared to GPU compute time:

ONNX Runtime is still faster, but 2ms difference at 8ms total latency is less dramatic than the 47ms gap on CPU. And TorchServe’s automatic batching pulls ahead on throughput when requests arrive unevenly.

My best guess is that TorchServe’s eager execution path overlaps better with GPU memory transfers than ONNX Runtime’s graph execution. But I’m not entirely sure — CUDA profiling is its own rabbit hole.

When Each Tool Actually Makes Sense

I’d pick ONNX Runtime for:
– Embedding inference in an existing Python service
– Memory-constrained deployments (< 2GB available)
– CPU-only inference where every millisecond counts
– Prototyping before committing to a serving infrastructure

I’d pick TorchServe for:
– Multi-model serving with A/B testing
– When you need automatic batching without building it yourself
– Teams that want model versioning and management APIs
– GPU deployments where raw latency matters less than operational features

The comparison with Triton vs TorchServe vs TFServing: 3 GPU Batch Tests showed similar patterns — TorchServe trades raw speed for operational convenience.

The Quantization Question

Both support INT8 quantization, but ONNX Runtime makes it easier. The onnxruntime.quantization module can quantize a model post-export:

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    "resnet50.onnx",
    "resnet50_int8.onnx",
    weight_type=QuantType.QInt8
)

That drops latency by another 30-40% on CPU. TorchServe relies on PyTorch’s quantization tools, which work but require quantization-aware training for best results. For a quick deployment win, ONNX Runtime’s dynamic quantization is hard to beat.

If you’re running large language models and considering INT8 more seriously, the cost implications get interesting — I wrote about this in INT8 vs FP16 Inference: TCO Cut 54% for 7B Models on AWS.

The Handler Pattern: TorchServe’s Best Feature

Here’s where my opinion might be unpopular: TorchServe’s handler abstraction is actually good software design. The preprocess → inference → postprocess pipeline forces you to think about data transformation explicitly.

ONNX Runtime gives you raw tensor in, raw tensor out. Great for benchmarks. Terrible when someone sends you a JPEG and you’re staring at preprocessing code scattered across three files.

The handler pattern also makes testing easier:

# You can unit test the handler without starting TorchServe
handler = ResNetHandler()
handler.initialize(mock_context)
result = handler.handle([{"data": image_bytes}], mock_context)
assert "predictions" in result[0]

ONNX Runtime inference code tends to accumulate in application code, making it harder to test the inference path in isolation.

Common First-Run Errors

TorchServe likes to fail silently. The logs are verbose but the errors hide:

WARN - Model resnet50 version 1.0 failed to load

That’s all you get. The actual error (maybe a missing dependency in your handler, maybe a CUDA version mismatch) is buried in logs/model_log.log. First thing I do on any TorchServe setup is tail that file.

ONNX Runtime throws Python exceptions like a normal library:

onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: 
[ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error: Duplicate definition of name (_0)

Clear, actionable, fixable. The error handling philosophy differs — TorchServe assumes you’re running as a service with logging infrastructure, ONNX Runtime assumes you’re debugging interactively.

Real Production Considerations

Neither tool handles model drift detection. You’re bolting on Evidently AI or similar regardless. Neither does feature validation — garbage inputs produce garbage outputs with no warning.

TorchServe’s metrics endpoint (/metrics) exports Prometheus-compatible data. ONNX Runtime gives you nothing — you’re instrumenting with OpenTelemetry yourself or hoping your FastAPI wrapper has tracing.

For debugging inference issues at 2am, a good standing desk mat helps more than either framework’s documentation.

FAQ

Q: Can I use ONNX Runtime behind TorchServe?

Yes. TorchServe supports ONNX models through a custom handler that wraps ONNX Runtime. You get TorchServe’s serving features (batching, versioning, metrics) with ONNX Runtime’s inference speed. The handler loads the ONNX session in initialize() and calls sess.run() in inference(). This hybrid approach is surprisingly underused.

Q: Which is easier to containerize?

ONNX Runtime. A minimal Dockerfile is ~10 lines and produces a ~500MB image. TorchServe requires Java in the container, pushing images past 2GB. The NVIDIA maintained PyTorch Inference containers include TorchServe but they’re hefty. For size-constrained deployments (edge, serverless), ONNX Runtime wins decisively.

Q: What about transformer models like BERT?

Both handle transformers fine, but ONNX Runtime often has dedicated optimizations (ORT Transformers, FlashAttention integration) that TorchServe’s eager execution misses. For BERT-family models specifically, I’d start with ONNX Runtime and only move to TorchServe if you need its operational features.

What Comes Next

Start with ONNX Runtime. Get your model running, measure latency, understand your bottlenecks. When you need batching, model versioning, or multi-model management, migrate to TorchServe — and keep ONNX Runtime as the backend for inference speed.

The gap between these tools is shrinking. TorchServe 0.9+ has improved significantly on cold start times, and ONNX Runtime keeps adding execution providers. I’m curious whether the upcoming TorchServe integration with torch.compile (from PyTorch 2.x) closes the CPU inference gap entirely. That would change this recommendation. But for now, for a first inference test on limited hardware? ONNX Runtime gets you there faster with less pain.