FastAPI vs Flask Async: Cut ML Inference Latency 48%

The Problem: Flask Was Blocking Everything

A simple sentiment analysis API was taking 340ms per request. The model itself ran in 80ms. Where did the other 260ms go?

Turns out Flask’s synchronous request handling was the culprit. Each request blocked the thread while waiting for database lookups, preprocessing, and post-processing I/O. With 10 concurrent users, average latency spiked to 1.2 seconds. The model wasn’t slow — the framework was.

Switching to FastAPI with proper async patterns dropped average latency to 178ms under the same load. That’s 48% faster without touching the model code.

But here’s what most tutorials skip: just switching frameworks doesn’t magically make your code async. You need to refactor synchronous I/O calls, understand when async def actually helps, and know which libraries support true async. Get it wrong and FastAPI performs worse than Flask.

This post walks through the migration, shows real latency benchmarks, and explains the async patterns that actually matter for ML serving.

Why Flask Blocks (Even With Threading)

Flask runs on WSGI (Web Server Gateway Interface), which is fundamentally synchronous. Each request ties up a worker thread until the entire response is ready.

Here’s a typical Flask ML endpoint:

from flask import Flask, request, jsonify
import time
import numpy as np
from transformers import pipeline

app = Flask(__name__)
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

@app.route("/predict", methods=["POST"])
def predict():
    text = request.json["text"]

    # Simulate DB lookup for user config (20ms)
    time.sleep(0.02)

    # Model inference (80ms)
    result = classifier(text)[0]

    # Simulate logging to DB (15ms)
    time.sleep(0.015)

    return jsonify(result)

With gunicorn -w 4 (4 worker processes), this handles about 11 requests/second. Under 10 concurrent requests, latency averages 890ms.

The threading model helps some, but Python’s GIL (Global Interpreter Lock) limits true parallelism. And every I/O wait — database reads, file writes, external API calls — blocks the worker thread. The model runs on CPU/GPU, but the thread sits idle during all that I/O.

FastAPI Async: Where It Actually Helps

FastAPI is built on ASGI (Asynchronous Server Gateway Interface) using Starlette and runs on uvicorn. It supports async def endpoints that can yield control during I/O waits.

Here’s the naive port:

from fastapi import FastAPI
from pydantic import BaseModel
import asyncio
from transformers import pipeline

app = FastAPI()
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

class PredictRequest(BaseModel):
    text: str

@app.post("/predict")
async def predict(req: PredictRequest):
    await asyncio.sleep(0.02)  # DB lookup
    result = classifier(req.text)[0]  # PROBLEM: this is still blocking!
    await asyncio.sleep(0.015)  # DB logging
    return result

This is SLOWER than Flask. Why?

Because classifier(req.text) is a CPU-bound synchronous call. Running it in an async def function doesn’t make it non-blocking — it just blocks the entire event loop. With 10 concurrent requests, they all queue up waiting for the event loop, and latency hits 950ms.

I wasted an hour debugging this before realizing: async/await is only useful for I/O-bound operations.

Pattern 1: Offload CPU Work to Thread Pool

The fix is to run the synchronous model inference in a separate thread using asyncio.to_thread() (Python 3.9+):

import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

@app.post("/predict")
async def predict(req: PredictRequest):
    await asyncio.sleep(0.02)  # Async I/O simulation

    # Run blocking model call in thread pool
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(executor, classifier, req.text)

    await asyncio.sleep(0.015)  # Async I/O simulation
    return result[0]

Now latency drops to 320ms under 10 concurrent requests. Better, but still not great.

The event loop isn’t blocked during inference, but we’re still limited by thread pool size. And Python threads aren’t truly parallel due to the GIL — they just interleave CPU work.

Pattern 2: Replace Sync I/O with Async Libraries

The real win comes from making I/O truly async. If you’re using requests, psycopg2, or pymongo, you’re blocking.

Here’s a realistic example with async database calls using asyncpg:

import asyncpg
from datetime import datetime

db_pool = None

@app.on_event("startup")
async def startup():
    global db_pool
    db_pool = await asyncpg.create_pool(
        "postgresql://user:pass@localhost/mldb",
        min_size=2,
        max_size=10
    )

@app.post("/predict")
async def predict(req: PredictRequest):
    # Async DB lookup (non-blocking)
    async with db_pool.acquire() as conn:
        user_config = await conn.fetchrow(
            "SELECT preprocess_config FROM users WHERE api_key = $1",
            req.api_key
        )

    # Offload model to thread pool
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(executor, classifier, req.text)

    # Async DB logging (non-blocking)
    async with db_pool.acquire() as conn:
        await conn.execute(
            "INSERT INTO predictions (text, result, created_at) VALUES ($1, $2, $3)",
            req.text, result[0]["label"], datetime.utcnow()
        )

    return result[0]

With this setup:
– Database calls don’t block the event loop
– Model inference runs in a thread pool
– Under 10 concurrent requests, latency is now 178ms (vs 890ms in Flask)

That’s a 48% improvement just from async I/O. The model runtime is the same — we’re just not wasting time blocking.

Pattern 3: Batch Inference with Async Queuing

For even better throughput, you can queue requests and batch them:

import asyncio
from collections import deque

inference_queue = deque()
inference_lock = asyncio.Lock()

async def batch_inference_worker():
    while True:
        await asyncio.sleep(0.05)  # Batch every 50ms

        async with inference_lock:
            if not inference_queue:
                continue

            batch = [inference_queue.popleft() for _ in range(min(8, len(inference_queue)))]

        texts = [item["text"] for item in batch]

        # Run batch inference in thread pool
        loop = asyncio.get_event_loop()
        results = await loop.run_in_executor(executor, classifier, texts)

        # Resolve futures
        for item, result in zip(batch, results):
            item["future"].set_result(result)

@app.on_event("startup")
async def startup():
    asyncio.create_task(batch_inference_worker())

@app.post("/predict")
async def predict(req: PredictRequest):
    future = asyncio.Future()

    async with inference_lock:
        inference_queue.append({"text": req.text, "future": future})

    result = await future
    return result

This batches up to 8 requests every 50ms. With the DistilBERT model, batched inference is about 2.3x faster than sequential (tested on an RTX 3060). Under moderate load (5-15 req/s), this setup maintains sub-100ms latency.

But there’s a trade-off: at very low request rates, you add 50ms of queueing delay. For a side project with bursty traffic, I’d skip batching. For a production API with steady load, it’s worth it.

Real Latency Benchmarks

I ran locust tests with 10 concurrent users, 100 requests each, on my M1 MacBook Pro (8-core, 16GB RAM). Model: distilbert-base-uncased-finetuned-sst-2-english, average input length 50 tokens.

The naive async port performed worse than Flask. Only after offloading CPU work and replacing sync I/O did FastAPI win.

When Flask is Actually Fine

If your endpoint is purely CPU-bound (e.g., image classification with no database), async won’t help. Just scale horizontally with more workers.

Flask is also simpler to deploy. gunicorn is battle-tested, and you don’t need to worry about event loop bugs or blocking calls sneaking in. For a weekend project or internal tool with <10 req/s, Flask is less hassle.

FastAPI shines when:
– You have significant I/O (database, external APIs, file uploads)
– You want request batching for inference
– You need WebSockets or SSE (Server-Sent Events)
– You’re serving multiple models and want to maximize concurrency

Common Async Pitfalls

1. Blocking libraries in async functions

This silently tanks performance:

import requests  # SYNC library

@app.get("/external")
async def fetch_external():
    resp = requests.get("https://api.example.com/data")  # Blocks event loop!
    return resp.json()

Use httpx instead:

import httpx

async with httpx.AsyncClient() as client:
    resp = await client.get("https://api.example.com/data")

2. Thread pool size too small

If your thread pool has 4 workers and you get 10 concurrent requests, 6 will queue. Monitor ThreadPoolExecutor size based on your CPU cores and inference time.

3. Not using connection pooling

Creating a new DB connection per request kills async benefits. Always use a connection pool (asyncpg.create_pool(), motor.motor_asyncio.AsyncIOMotorClient()).

4. Debugging is harder

Stack traces in async code are messier. I’ve had mysterious hangs from forgetting await on a coroutine. Tools like aiomonitor help, but I’m not entirely sure they catch everything.

Deployment Differences

Flask:

gunicorn -w 4 -b 0.0.0.0:8000 app:app

FastAPI:

uvicorn app:app --host 0.0.0.0 --port 8000 --workers 4

For production, I use uvicorn with --workers equal to CPU cores. Each worker runs its own event loop.

Docker setup is identical (just swap the command). FastAPI memory usage is slightly higher (~20MB per worker vs ~15MB for Flask with gunicorn), but negligible for most deployments.

Migration Checklist

1. Audit your I/O: If >50% of request time is database/API calls, async is worth it.
2. Replace sync libraries: requests → httpx, psycopg2 → asyncpg, pymongo → motor.
3. Offload CPU work: Use run_in_executor() for model inference.
4. Test under load: locust or hey with realistic concurrency.
5. Monitor event loop lag: If p95 latency is >2x p50, you’re probably blocking somewhere.

If you’re migrating a production API, I’d do it incrementally: start with one endpoint, measure, then expand. Don’t rewrite everything at once.

What About ONNX Runtime or Triton?

Converting the model to ONNX and using onnxruntime shaves another 15-20ms off inference time. I covered some of the export gotchas in a previous post on ONNX pitfalls.

For multi-model serving with GPU batching, Triton Inference Server is overkill for side projects but worth it at scale. FastAPI + async is the sweet spot for 1-3 models on a single server.

Amazon Product Recommendation

Debugging async/await bugs at 1am? Grab some Caffeine Pills 200mg — cheaper than coffee and you won’t spill them on your keyboard.

FAQ

Q: Can I mix sync and async endpoints in the same FastAPI app?

Yes. Use def for sync (FastAPI runs it in a thread pool automatically) and async def for async. Just don’t call sync I/O inside async def.

Q: Does async help with GPU inference?

Not directly. GPU calls are synchronous in PyTorch/TensorFlow. But async I/O (preprocessing, database lookups) still helps. For GPU batching, look into NVIDIA Triton or custom batching logic like Pattern 3.

Q: Is FastAPI production-ready?

Yes. It’s used by Microsoft, Netflix, and Uber. Uvicorn is stable (based on uvloop, which is faster than the default asyncio event loop). Just watch for blocking calls in async functions — that’s the main footgun.

What I’d Do Differently Next Time

I’d start with FastAPI from day one if I knew the API would handle database calls or external APIs. The async patterns aren’t that hard once you internalize “don’t block the event loop.”

For pure CPU-bound workloads (e.g., image segmentation with no I/O), I’d stick with Flask + gunicorn and scale horizontally. Simpler is better when async doesn’t buy you anything.

One thing I haven’t tested yet: how FastAPI handles very large request bodies (e.g., 10MB image uploads). My guess is streaming uploads with async for chunk in request.stream() would help, but I haven’t benchmarked it against Flask’s chunked uploads. That’s next on my list.