FastAPI vs Flask ML Serving: 5 Benchmarks Beginners Miss

The Test That Changed My Mind

Flask served my first production model for 18 months without issue. Then I rewrote it in FastAPI, ran the same benchmark suite, and watched throughput jump 3.2x on identical hardware.

But that’s not the whole story. The tests most beginners run — wrk with a single endpoint, no preprocessing, no real model logic — miss the patterns that actually matter when you’re serving ML models. Load testing an empty route tells you nothing about inference latency under concurrent requests, nothing about CPU-bound preprocessing bottlenecks, and nothing about memory behavior when multiple models share the same process.

This post walks through five benchmarks designed specifically for ML serving scenarios: synchronous inference, async preprocessing pipelines, concurrent model loading, streaming predictions, and multipart file uploads with image models. All tests use real (albeit toy) scikit-learn and PyTorch models, not return {"prediction": 42}.

Why Most Flask vs FastAPI Benchmarks Are Useless

Go Google “fastapi vs flask benchmark” right now. You’ll find a dozen Medium posts showing FastAPI crushing Flask at 50k requests/sec on a hello-world route.

Cool. Completely irrelevant.

ML serving doesn’t look like serving static JSON. Your routes spend 80% of their time in numpy array manipulation, model forward passes, and image decoding — all CPU-bound, synchronous operations. The async event loop helps with I/O (database queries, external API calls, file reads), but model.predict(X) blocks the thread either way.

What you actually need to measure:

1. Throughput under realistic concurrency — not 1000 parallel requests hitting an empty route, but 10-20 clients sending real payloads that trigger actual inference
2. Latency distribution — p50 is useless, you care about p95 and p99 when an API gateway has a 5-second timeout
3. Memory behavior — does concurrent loading of the same model instantiate it N times? How does response time degrade as RAM fills?
4. Async preprocessing wins — can you overlap image decoding, resizing, and normalization while waiting for S3?

And one more thing: most benchmarks ignore the fact that Flask with gunicorn runs multiple worker processes, while FastAPI typically runs one process with an async event loop. Different concurrency models, different resource tradeoffs. Comparing single-threaded Flask dev server to production FastAPI is like benchmarking a bicycle against a car by testing them both in a parking lot.

Test Setup: Real Models, Real Preprocessing

All benchmarks run on an M1 MacBook Pro (8-core, 16GB RAM), Python 3.11. Flask uses gunicorn with 4 workers, FastAPI runs with uvicorn (single process). I’m testing latency and throughput, not absolute speed — the goal is to understand the ratio of performance under different patterns.

Dependencies

# requirements.txt
flask==3.0.0
fastapi==0.109.0
uvicorn[standard]==0.27.0
gunicorn==21.2.0
scikit-learn==1.4.0
torch==2.1.2  # CPU-only for this test
Pillow==10.2.0
numpy==1.26.3

I’m using a simple scikit-learn RandomForestClassifier (100 trees, fit on synthetic data) and a minimal PyTorch CNN (3 conv layers, ~50k parameters). Not production-grade models, but realistic enough to create CPU load.

One thing to note: I’m NOT testing GPU inference here. That’s a whole different problem — you’d batch requests, use ONNX Runtime or Triton, maybe throw NVIDIA’s TensorRT at it if you’re trying to squeeze every millisecond out of edge hardware. This post focuses on CPU serving, which is what you’re doing when you first deploy a model and don’t want to pay for a GPU instance yet.

Benchmark 1: Synchronous Inference (Baseline)

The simplest case: POST a JSON payload, run model.predict(), return the result. No async, no preprocessing, just raw inference.

Flask (4 gunicorn workers)

from flask import Flask, request, jsonify
import joblib
import numpy as np

app = Flask(__name__)
model = joblib.load("random_forest.pkl")  # Loaded once per worker

@app.route("/predict", methods=["POST"])
def predict():
    data = request.get_json()
    X = np.array(data["features"]).reshape(1, -1)
    pred = model.predict(X)
    return jsonify({"prediction": int(pred[0])})

Run with gunicorn -w 4 -b 0.0.0.0:5000 app:app.

FastAPI (uvicorn, single process)

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI()
model = joblib.load("random_forest.pkl")  # Loaded once at startup

class PredictionRequest(BaseModel):
    features: list[float]

@app.post("/predict")
def predict(req: PredictionRequest):
    X = np.array(req.features).reshape(1, -1)
    pred = model.predict(X)
    return {"prediction": int(pred[0])}

Run with uvicorn app:app --host 0.0.0.0 --port 8000.

Load Test (wrk, 10 concurrent connections, 30 seconds)

wrk -t4 -c10 -d30s --latency \
  -s post.lua http://localhost:5000/predict

(where post.lua sends {"features": [0.1, 0.2, ..., 0.9]} — 20 features, matching the model’s input shape)

Results:

FastAPI wins by ~4%, but it’s essentially a tie. For synchronous, CPU-bound inference, the async event loop provides zero advantage. Both are bottlenecked by model.predict() holding the GIL.

One surprise: Flask’s p99 latency is slightly worse despite running 4 workers. My best guess is gunicorn’s worker scheduling introduces occasional jitter when all workers are busy. But the difference is negligible — 1ms at p99 doesn’t matter for most use cases.

Benchmark 2: Async Preprocessing Pipeline

Now let’s add realistic preprocessing: fetch data from a “remote” source (simulated with asyncio.sleep(0.05) to mimic a 50ms database query), decode JSON, normalize features, then predict.

This is where async should shine — you can overlap I/O waits across multiple requests.

Flask (still synchronous)

import time

@app.route("/predict_with_fetch", methods=["POST"])
def predict_with_fetch():
    data = request.get_json()
    user_id = data["user_id"]

    # Simulate fetching user features from DB (blocking)
    time.sleep(0.05)
    features = [0.1] * 20  # Fake features

    X = np.array(features).reshape(1, -1)
    pred = model.predict(X)
    return jsonify({"prediction": int(pred[0])})

With 4 workers, this can handle ~80 requests/sec before saturating (4 workers × 1 request per 50ms = 80 req/s theoretical max).

FastAPI (async endpoint)

import asyncio

class FetchRequest(BaseModel):
    user_id: str

async def fetch_user_features(user_id: str):
    await asyncio.sleep(0.05)  # Simulated DB query
    return [0.1] * 20

@app.post("/predict_with_fetch")
async def predict_with_fetch(req: FetchRequest):
    features = await fetch_user_features(req.user_id)
    X = np.array(features).reshape(1, -1)

    # model.predict() is still synchronous, but we've overlapped I/O
    pred = model.predict(X)
    return {"prediction": int(pred[0])}

Results (10 concurrent clients):

FastAPI crushes Flask here — 2.3x throughput, half the p95 latency. The event loop overlaps the 50ms I/O waits across 10 concurrent requests, while Flask workers block one-at-a-time.

But note: this assumes your preprocessing actually involves async I/O. If you’re just doing numpy operations (resizing images, normalizing arrays), there’s no I/O to overlap. You’d need to run that preprocessing in a thread pool executor:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

@app.post("/predict")
async def predict(req: PredictionRequest):
    loop = asyncio.get_event_loop()
    # Offload CPU-bound preprocessing to thread pool
    X = await loop.run_in_executor(executor, preprocess, req.features)
    pred = model.predict(X)
    return {"prediction": int(pred[0])}

This pattern is crucial for image models where you’re decoding JPEG, resizing, and converting to tensors — all CPU-bound. Without the executor, async doesn’t help.

Benchmark 3: Concurrent Model Loading

What happens when multiple requests trigger model loading at the same time? This is common in serverless environments (AWS Lambda cold starts) or when using lazy loading to save memory.

I’m simulating this by removing the global model variable and loading it inside each request handler. Obviously terrible for production, but useful for testing concurrency behavior.

Flask

@app.route("/predict_load", methods=["POST"])
def predict_load():
    model = joblib.load("random_forest.pkl")  # ~200ms load time
    data = request.get_json()
    X = np.array(data["features"]).reshape(1, -1)
    pred = model.predict(X)
    return jsonify({"prediction": int(pred[0])})

With 4 workers, each worker loads the model independently. Total memory usage: 4× model size (~120MB × 4 = 480MB for this toy model).

FastAPI

Same code, but single-process. Only one model instance in memory (~120MB).

Results (10 concurrent requests, measuring first response time):

Flask wins on latency because 4 workers load the model in parallel (each taking ~200ms). FastAPI serializes the loads on a single event loop, so 10 requests × 200ms = 2 seconds.

But Flask uses 4× the memory. For large models (ResNet-50 is ~100MB, BERT-base is ~500MB), this matters. If you’re running on a 2GB RAM instance, you can’t afford 4 workers with heavy models.

The right solution for FastAPI: use a startup event to preload the model once, or cache it after the first load. The right solution for Flask: also preload at worker startup. Don’t load models per-request in production — this test just illustrates the concurrency difference.

Benchmark 4: Streaming Predictions

Sometimes you want to stream results as they’re computed — think generating text with an LLM, or processing video frames. FastAPI’s StreamingResponse makes this trivial. Flask can do it with generators, but it’s clunkier.

FastAPI

from fastapi.responses import StreamingResponse
import asyncio

async def generate_predictions():
    for i in range(10):
        await asyncio.sleep(0.1)  # Simulate slow prediction
        yield f"{{\"step\": {i}, \"value\": {i * 0.1}}}\n"

@app.get("/stream")
async def stream():
    return StreamingResponse(
        generate_predictions(),
        media_type="application/x-ndjson"
    )

Client receives results incrementally (newline-delimited JSON). Total time: ~1 second (10 steps × 100ms).

Flask

import time

def generate_predictions():
    for i in range(10):
        time.sleep(0.1)
        yield f"{{\"step\": {i}, \"value\": {i * 0.1}}}\n"

@app.route("/stream")
def stream():
    return app.response_class(
        generate_predictions(),
        mimetype="application/x-ndjson"
    )

Same total time, but Flask blocks a worker for the entire 1 second. If you have 4 workers and 5 concurrent streaming requests, one client waits.

FastAPI’s async generators let you handle many concurrent streams without blocking. For batch inference on long videos or real-time model outputs, this is a significant advantage.

Benchmark 5: Multipart File Upload (Image Model)

Finally, the classic ML serving scenario: upload an image, run a PyTorch CNN, return class probabilities.

Here’s where things get tricky. Both Flask and FastAPI block on PIL image decoding and torch inference, but FastAPI’s async file handling can overlap uploads.

Flask

from PIL import Image
import io
import torch

model_cnn = torch.load("simple_cnn.pt")
model_cnn.eval()

@app.route("/classify", methods=["POST"])
def classify():
    file = request.files["image"]
    img = Image.open(file.stream).convert("RGB")
    img = img.resize((224, 224))  # ~20ms on M1

    # Convert to tensor, normalize (skipping torchvision transforms for brevity)
    x = torch.tensor(np.array(img)).permute(2, 0, 1).float() / 255.0
    x = x.unsqueeze(0)

    with torch.no_grad():
        logits = model_cnn(x)  # ~30ms inference

    probs = torch.softmax(logits, dim=1).squeeze().tolist()
    return jsonify({"probabilities": probs})

FastAPI

from fastapi import File, UploadFile
from PIL import Image
import io
import torch
import numpy as np

model_cnn = torch.load("simple_cnn.pt")
model_cnn.eval()

@app.post("/classify")
async def classify(image: UploadFile = File(...)):
    contents = await image.read()  # Async file read
    img = Image.open(io.BytesIO(contents)).convert("RGB")
    img = img.resize((224, 224))

    x = torch.tensor(np.array(img)).permute(2, 0, 1).float() / 255.0
    x = x.unsqueeze(0)

    with torch.no_grad():
        logits = model_cnn(x)

    probs = torch.softmax(logits, dim=1).squeeze().tolist()
    return {"probabilities": probs}

Results (10 concurrent uploads, 1MB JPEG each):

FastAPI wins by 35% — not as dramatic as the async I/O test, but noticeable. The async file read overlaps slightly with other requests, and uvicorn’s HTTP parsing is faster than gunicorn’s.

But here’s the catch: if you offload image decoding to a thread pool (which you should for CPU-heavy preprocessing), the gap narrows. And if you’re batching requests (grouping 10 images into a single model(batch) call), you’d switch to a dedicated batching framework like Triton Inference Server anyway.

When Flask Still Makes Sense

Despite FastAPI’s wins in 4 out of 5 benchmarks, Flask isn’t obsolete for ML serving. Here’s when I’d still pick it:

1. Simple CRUD APIs — if your “ML service” is really just a thin wrapper around a database (store embeddings, fetch nearest neighbors), Flask + SQLAlchemy is simpler than FastAPI + async ORM
2. Existing Flask codebase — rewriting a working service just for 2x throughput rarely pays off unless you’re hitting hard scaling limits
3. Team familiarity — if your team knows Flask and you’re not doing async I/O, the productivity hit of learning FastAPI may outweigh the performance gain
4. Debugging simplicity — synchronous code is easier to step through with pdb. Async stack traces can be a nightmare.

And one more thing: Flask with gevent or eventlet can do async I/O too, though it’s less ergonomic than FastAPI’s native async/await. If you’re stuck on Flask for legacy reasons, that’s an option.

The Real Bottleneck Isn’t the Framework

Here’s the uncomfortable truth: for most ML serving workloads, the framework overhead is <5% of total latency. The actual bottleneck is the model itself.

If your model takes 500ms to run inference, shaving 10ms off FastAPI’s request parsing doesn’t move the needle. You need to optimize the model: quantization (INT8 instead of FP32), pruning, distillation, or switching to ONNX Runtime for 2-3x speedups.

I covered ONNX export pitfalls in this post — the TL;DR is that dynamic shapes and custom ops break easily, but when it works, the speedup is worth it.

That said, FastAPI’s async advantages compound when you have external dependencies: database queries, S3 uploads, Redis caching, downstream API calls. If 30% of your request time is I/O, async can cut latency by 20-25% for free. That’s worth the migration cost.

My Actual Recommendation

For beginners building their first ML API:

• Start with Flask if you’re doing synchronous inference, no preprocessing, just model.predict() → JSON. Get it working, deploy it, learn production lessons (monitoring, error handling, versioning).
• Switch to FastAPI when you add async I/O (database, S3, Redis), file uploads, or streaming. Or when you’re refactoring anyway and want better type safety (Pydantic validation catches bugs Flask’s request.get_json() silently swallows).
• Skip both if you need batching, GPU inference, or >100 req/s. Use Triton, TorchServe, or Ray Serve. FastAPI is great for 10-50 req/s on CPU, terrible at scale.

I’m currently rewriting an old Flask service that does video frame extraction + object detection. The frame extraction is CPU-bound (ffmpeg subprocess), detection is batched via Triton, and the orchestration layer is FastAPI with async S3 uploads. Flask could work, but FastAPI’s async makes the code cleaner — await upload_to_s3(frames) instead of thread pools.

I’m not entirely sure FastAPI will be faster in practice — the ffmpeg bottleneck dominates. But the codebase is easier to reason about, which matters more for a solo project I’ll maintain for years.

FAQ

Q: Can I use FastAPI with multiple worker processes like gunicorn?

Yes — run gunicorn -k uvicorn.workers.UvicornWorker app:app -w 4. This gives you 4 processes, each with an async event loop. Useful when you’re CPU-bound but still want async I/O. Memory usage will be 4× (same as Flask), so you lose the single-process advantage.

Q: Why not just use async Flask with gevent?

You can, but gevent monkey-patches the standard library (including socket, ssl, threading), which breaks libraries that assume synchronous behavior. I’ve seen it clash with PyTorch’s DataLoader and certain database drivers. FastAPI’s native async/await is safer.

Q: What about Starlette vs FastAPI?

FastAPI is built on Starlette (the underlying ASGI framework). If you don’t need Pydantic validation or automatic OpenAPI docs, use Starlette directly — it’s slightly faster and more minimal. But for ML APIs, Pydantic’s type validation is worth the tiny overhead. Catching "features": "not a list" at the request layer saves you from cryptic numpy errors deep in the stack.

The framework matters less than you think. Optimize your model first, add caching second, profile your I/O third. Then, if FastAPI’s async patterns fit your bottleneck, migrate. If not, Flask is fine.

What I’m still figuring out: whether FastAPI’s ecosystem (observability tools, managed hosting, community plugins) will catch up to Flask’s 13-year head start. Right now, Flask has better integrations with legacy monitoring tools (Datadog, Sentry), while FastAPI plays nicer with modern cloud-native stuff (Prometheus, OpenTelemetry). Depends on your stack.