Triton vs TorchServe vs TFServing: 3 GPU Batch Tests

TorchServe Failed at Batch Size 8

Tested three inference servers (Triton, TorchServe, TensorFlow Serving) with a ResNet-50 model on an A10G GPU. Sent 1000 requests with batch sizes 1, 8, and 32. TorchServe crashed at batch 32, Triton handled all three, and TF Serving added 40ms overhead even at batch 1.

The goal: find which server gives the best throughput-per-dollar for a side project serving image classification. Running costs matter when you’re paying hourly for GPU instances.

Why Batch Inference Servers Exist

You could just wrap PyTorch in FastAPI and call it a day. For low-traffic services, that works. But once you hit 10+ requests per second, you need dynamic batching — the server waits a few milliseconds to collect multiple requests, runs them as a single batch through the GPU, then fans out the results.

The math: a single ResNet-50 forward pass on batch size 1 takes ~8ms. Batch size 16 takes ~22ms. That’s not 16× the time — GPUs love parallelism. Throughput goes from 125 req/s to ~700 req/s.

Three main contenders:
– NVIDIA Triton — supports PyTorch, TensorFlow, ONNX, TensorRT, custom backends
– TorchServe — PyTorch-only, simpler setup, built by AWS and Meta
– TensorFlow Serving — Google’s solution, TF-only unless you convert to SavedModel

I’ve used FastAPI for model serving before, but none of those handle batching automatically. That’s the whole point of these specialized servers.

Test Setup: ResNet-50 on AWS g5.xlarge

Used a stock ResNet-50 from torchvision.models, ImageNet weights, input shape (batch, 3, 224, 224). Deployed on AWS g5.xlarge (A10G GPU, 24GB VRAM, $1.006/hr spot pricing). Ubuntu 22.04, CUDA 12.1, driver 535.

Why ResNet-50? It’s small enough to avoid OOM errors but large enough to stress the batch scheduler. Total params: 25.6M. FP32 model size: ~98MB.

Client: Python script sending async requests via aiohttp, 100 concurrent workers, 1000 total requests per test. Measured end-to-end latency (client → server → client) and throughput (req/s).

Model Conversion

Triton needs models in specific directory layouts. TorchServe uses .mar archives. TF Serving wants SavedModel format.

Triton (PyTorch backend):

import torch
import torchvision.models as models

model = models.resnet50(pretrained=True).eval().cuda()

# Triton expects model_repository/resnet50/1/model.pt
scripted = torch.jit.trace(model, torch.randn(1, 3, 224, 224).cuda())
scripted.save("model_repository/resnet50/1/model.pt")

config.pbtxt for dynamic batching:

name: "resnet50"
platform: "pytorch_libtorch"
max_batch_size: 32
input [
  { name: "input__0", data_type: TYPE_FP32, dims: [3, 224, 224] }
]
output [
  { name: "output__0", data_type: TYPE_FP32, dims: [1000] }
]
dynamic_batching {
  preferred_batch_size: [8, 16, 32]
  max_queue_delay_microseconds: 5000
}
instance_group [{ kind: KIND_GPU }]

TorchServe:

torch-model-archiver --model-name resnet50 \
  --version 1.0 \
  --serialized-file resnet50.pt \
  --handler image_classifier \
  --extra-files index_to_name.json

config.properties:

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
batch_size=8
max_batch_delay=5000

TF Serving: Converted PyTorch → ONNX → TF SavedModel (painful). Used onnx-tf but hit version mismatches. Eventually just trained a new ResNet-50 in Keras to avoid conversion hell.

Batch Size 1: TF Serving Already 40ms Slower

Single-image requests, no batching. This tests raw model latency + framework overhead.

TF Serving’s 51ms is suspicious. The actual inference is ~8ms (confirmed via TensorBoard profiling). The extra 43ms is gRPC overhead + TF runtime initialization per request. Even with --batching_parameters_file disabled, TF Serving seems to assume you’ll batch and adds marshalling overhead.

Triton and TorchServe are roughly tied here. Triton’s 11ms includes HTTP parsing, tensor copying to GPU, inference, and response serialization. TorchServe is slightly slower, probably due to JVM overhead (it runs on Java with a Python worker pool).

One thing I didn’t expect: Triton’s memory footprint was 800MB higher than TorchServe at idle. Not sure if that’s the C++ backend preloading CUDA kernels or just bloated dependencies.

Batch Size 8: Triton Pulls Ahead

Client sends requests at high concurrency, server collects them into batches up to size 8, waits max 5ms for more requests, then runs inference.

Triton’s dynamic batching actually worked. Checked the logs:

Batch size: 8, queue time: 2.1ms, compute time: 14.3ms

The compute time ($t_{compute}$) scales sublinearly with batch size $b$:

$$t_{compute}(b) \approx t_{base} + \alpha \cdot b$$

where $t_{base}$ is overhead (kernel launch, memory allocation) and $\alpha$ is per-sample inference time. For ResNet-50 on A10G, $t_{base} \approx 6\text{ms}$, $\alpha \approx 1.2\text{ms}$. So batch 8 should take $6 + 8 \times 1.2 = 15.6\text{ms}$, which matches observed 14.3ms.

TorchServe’s 26ms p50 is higher than expected. Dug into metrics endpoint:

curl http://localhost:8082/metrics | grep batch

Found:

ts_queue_latency_microseconds{model_name="resnet50",level="model"} 8200
ts_inference_latency_microseconds{model_name="resnet50",level="model"} 15100

Queue latency is 8.2ms — longer than Triton’s 2.1ms. TorchServe’s batching logic seems more conservative. Even with max_batch_delay=5000 (5ms), it’s waiting longer or not filling batches efficiently.

TF Serving still slow. No idea why Google ships this as the default solution for production TensorFlow models. Maybe it shines on TPUs?

Batch Size 32: TorchServe OOM

Pushed batch size to 32. Triton handled it. TorchServe crashed.

TorchServe error:

RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 22.20 GiB total capacity; 20.89 GiB already allocated)

Checked nvidia-smi — A10G has 24GB VRAM. ResNet-50 FP32 model uses ~400MB, activations for batch 32 should be ~2.8GB. Total: ~3.2GB. TorchServe was using 21GB?

Turns out TorchServe preloads multiple worker processes (default: 4). Each worker loads the model into VRAM independently. So $4 \times 400\text{MB} = 1.6\text{GB}$ just for weights, then each worker holds its own activation buffers. With 4 workers × batch 32 × activations, you hit 24GB fast.

Triton uses a single model instance per GPU by default (unless you specify count in instance_group). More memory-efficient but potentially lower throughput if you have multiple GPUs.

Fix for TorchServe: set minWorkers=1 and maxWorkers=1 in config.properties. Reran test — no crash, but throughput dropped to 612 req/s (vs Triton’s 823). Still slower even with single worker.

Latency Breakdown: Where Time Goes

Instrumented Triton with --log-verbose=1 and parsed execution logs. For batch 32 at p50:

• Queue wait: 4.2ms (time request sits in batch queue)
• Tensor copy H2D: 1.8ms (CPU → GPU)
• Inference: 28.1ms (CUDA kernel execution)
• Tensor copy D2H: 1.3ms (GPU → CPU)
• Response serialization: 2.6ms (protobuf encoding)

Total: 38ms, matches observed p50.

The 28.1ms inference time is pure compute. For ResNet-50 batch 32, theoretical FLOPS:

$$\text{FLOPs} = 2 \times 3.8 \times 10^9 \times 32 = 2.43 \times 10^{11}$$

A10G peak FP32 throughput: 31.2 TFLOPS. Expected time:

$$t = \frac{2.43 \times 10^{11}}{31.2 \times 10^{12}} \approx 7.8\text{ms}$$

Wait, 7.8ms vs observed 28.1ms? Where’s the 3.6× gap?

Memory bandwidth. ResNet-50 is memory-bound, not compute-bound. A10G memory bandwidth: 600 GB/s. Model weights: 98MB, activations: ~90MB per sample. For batch 32: $32 \times 90\text{MB} = 2.88\text{GB}$. Add weight reads across 50 layers → actual memory traffic ~15GB. At 600 GB/s: DOLLAR_AMOUNT_4 / 600 \approx 25\text{ms}$.

That’s closer. The 28.1ms includes kernel launch overhead, memory allocation, cuDNN convolution tuning. Not bad.

Cost Per 1M Requests

Assuming AWS g5.xlarge spot ($1.006/hr), running 24/7 for a month.

Triton wins by a lot. If your traffic is spiky (e.g., B2B SaaS with weekend lulls), spot instances + autoscaling can cut this further. But TorchServe’s 3× higher cost is rough.

TF Serving’s 4× cost makes no sense unless you’re locked into TensorFlow for some reason. Even then, export to ONNX and run on Triton.

When TorchServe Still Makes Sense

Triton requires writing config.pbtxt files, understanding model versioning, and dealing with gRPC. TorchServe has simpler deployment:

torch-model-archiver --model-name mymodel --handler my_handler.py
torchserve --start --model-store ./

If you’re prototyping or running a low-traffic service (<50 req/s), TorchServe’s DX is better. The AWS integration is also tighter — SageMaker uses TorchServe under the hood, so if you’re already on SageMaker, switching to Triton adds operational complexity.

For custom preprocessing (e.g., non-standard tokenization, audio feature extraction), TorchServe handlers are just Python functions. Triton requires writing C++ backends or using the Python backend (which is slower and poorly documented).

But if you’re paying for GPUs and throughput matters, Triton is the answer.

TensorFlow Serving: Just Don’t

I tried. The only scenario where TF Serving beats Triton is if you have a massive TensorFlow 1.x codebase and can’t migrate. Even then, Triton’s TensorFlow backend supports SavedModel format.

TF Serving’s gRPC API is clunky. The REST API exists but is even slower (adds another 20ms). The batching config (--batching_parameters_file) is JSON but takes protobuf field names, so you need to read TensorFlow source to understand it.

Google deprecated TF Serving docs in favor of Vertex AI. If you’re not on GCP, you’re on your own.

FAQ

Q: Can I run Triton without Docker?
Yes, but you’ll need to compile it from source and link against CUDA, cuDNN, TensorRT, and half a dozen other libs. The Docker image is 8GB but includes everything. Not worth the pain unless you’re deploying to embedded devices.

Q: Does Triton support multi-GPU batching?
Sort of. You can specify multiple instance_group entries in config.pbtxt, one per GPU. But Triton doesn’t load-balance across GPUs automatically — you need an external reverse proxy (e.g., Envoy) to distribute requests. For 2-4 GPUs, it works. Beyond that, look at Ray Serve or KServe.

Q: Why not just use vLLM or Text Generation Inference for LLMs?
Because this post is about ResNet-50, not LLMs. But yes, for transformer-based models, vLLM’s continuous batching beats Triton’s static batching. Triton is better for CNNs, classical ML, and non-transformer architectures.

Amazon Rec

If you’re benchmarking inference servers at 2am and your A10G instance is burning $1/hr, you need Liquid I.V. Hydration Packets. Faster than coffee, no jitters, and you won’t feel like death when the AWS bill arrives.

What I’d Pick

Triton for anything production. The setup cost is 2-3 hours of reading docs, but the throughput gain pays for itself in week one.

TorchServe for side projects where you’re doing <100 req/s and don’t want to deal with config files. It’s fine. Not great, but fine.

TF Serving: skip it. Export to ONNX, run on Triton. Life’s too short.

One thing I haven’t tested yet: TensorRT optimization on Triton. You can convert PyTorch → ONNX → TensorRT via trtexec, then load the .plan file in Triton. TensorRT does layer fusion, kernel auto-tuning, and INT8 quantization. Should cut latency by another 30-40%. Next benchmark.