Docker vs Kubernetes for First ML Model: When to Use Each

You Probably Don’t Need Kubernetes Yet

Most ML engineers waste a week setting up Kubernetes for their first production model when a single Docker container would’ve shipped in an afternoon. I’ve seen teams spend more time debugging pod networking than actually improving their model.

Here’s the decision tree that would’ve saved me three failed deployments: if your model serves under 100 requests per second and you’re a solo developer or small team, Docker Compose is enough. If you’re coordinating 10+ microservices across multiple machines with auto-scaling requirements, Kubernetes starts paying for itself. Everything in between is a judgment call based on your ops capacity.

But that’s oversimplified. Let me show you what actually breaks when you pick the wrong tool.

The Docker-Only Deployment That Worked for 6 Months

I deployed a YOLOv8 object detection API using nothing but Docker and an AWS EC2 t3.medium instance. The entire production setup was a docker-compose.yml file, a FastAPI server, and a GitHub Actions workflow:

# app.py
from fastapi import FastAPI, File, UploadFile
from ultralytics import YOLO
import numpy as np
from PIL import Image
import io

app = FastAPI()
model = YOLO('yolov8n.pt')  # Loaded once at startup

@app.post("/detect")
async def detect_objects(file: UploadFile = File(...)):
    contents = await file.read()
    image = Image.open(io.BytesIO(contents))
    results = model(image)

    # Extract bounding boxes and scores
    boxes = results[0].boxes.xyxy.cpu().numpy()
    scores = results[0].boxes.conf.cpu().numpy()
    classes = results[0].boxes.cls.cpu().numpy()

    return {
        "detections": [
            {
                "bbox": box.tolist(),
                "confidence": float(score),
                "class_id": int(cls)
            }
            for box, score, cls in zip(boxes, scores, classes)
        ]
    }

The Dockerfile was straightforward — no multi-stage builds, no optimization tricks:

FROM python:3.11-slim

WORKDIR /app

# Install system deps for OpenCV
RUN apt-get update && apt-get install -y \
    libglib2.0-0 libsm6 libxext6 libxrender-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .
COPY yolov8n.pt .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Deployment was docker build, docker push, then SSH into the EC2 instance and docker pull + docker run. Done. Monitoring was CloudWatch logs. Restarts were handled by Docker’s --restart unless-stopped flag.

This setup handled 40-60 req/s with p95 latency around 180ms. Cost? $25/month for the EC2 instance. Zero Kubernetes overhead.

The breaking point came when the client wanted the same model deployed in three regions with automatic failover. Docker Compose doesn’t do cross-region orchestration. That’s when Kubernetes became worth the complexity tax.

What Kubernetes Actually Adds (and Costs)

Kubernetes solves problems Docker alone can’t:

1. Auto-scaling across machines: Docker Compose can’t spawn containers on multiple nodes based on CPU metrics. Kubernetes Horizontal Pod Autoscaler (HPA) can scale replicas from 2 to 20 when traffic spikes.
2. Self-healing with real orchestration: Docker restarts dead containers on the same host. Kubernetes reschedules them on healthy nodes automatically.
3. Zero-downtime deployments: Rolling updates with readiness probes ensure new model versions don’t drop requests.
4. Service discovery without hardcoding IPs: Internal DNS means your preprocessing service can call http://model-server:8000 instead of tracking IP addresses.

But here’s what the tutorials don’t tell you: Kubernetes has a cognitive load tax. You need to understand Pods, Deployments, Services, ConfigMaps, Secrets, Ingress controllers, PersistentVolumes, and RBAC just to get a basic app running. That’s 8 new concepts versus Docker’s 2 (images and containers).

The minimal viable Kubernetes deployment for that same YOLOv8 API looks like this:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yolo-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: yolo-api
  template:
    metadata:
      labels:
        app: yolo-api
    spec:
      containers:
      - name: api
        image: your-registry/yolo-api:v1.2
        ports:
        - containerPort: 8000
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: yolo-api
spec:
  selector:
    app: yolo-api
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

This YAML creates 3 replicas behind a load balancer. Health checks ensure traffic only goes to ready pods. Resource limits prevent one pod from starving others.

But now you need a Kubernetes cluster. Managed options:
– EKS (AWS): ~$75/month for control plane + $50-200/month for worker nodes
– GKE (Google): Similar pricing, slightly better auto-scaling
– AKS (Azure): Free control plane, pay for VMs only

Self-managed (k3s on EC2): Saves control plane cost but you’re debugging etcd failures at 2am.

The Model Serving Scenarios Where Docker Wins

Stick with plain Docker (or Docker Compose) when:

Single-region, predictable traffic: If your model serves internal requests from a known number of clients, you don’t need dynamic scaling. A fixed number of containers behind an Nginx reverse proxy is simpler:

# docker-compose.yml
version: '3.8'
services:
  model:
    build: .
    deploy:
      replicas: 3
    environment:
      - MODEL_PATH=/models/resnet50.onnx

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - model

This gives you basic load balancing without Kubernetes’ YAML sprawl.

Batch inference jobs: If your model processes uploaded files overnight, a simple cron job that runs docker run is easier to debug than a Kubernetes CronJob. You can monitor it with systemd or supervisord.

Prototyping on a budget: When you’re testing product-market fit, burning dev hours on Kubernetes is premature. Ship the Docker version, measure usage, then graduate to K8s when you have real scaling needs.

I prototyped a sentiment analysis API this way — Docker on a single $5/month VPS. It handled 500 daily users for three months before I needed anything fancier. The time saved on infra went into improving the model itself.

When you’re the only engineer: Kubernetes requires ongoing maintenance — security patches, version upgrades, certificate renewals. If you don’t have a dedicated DevOps person, Docker’s simplicity means less time firefighting and more time building features.

The Breaking Points Where Kubernetes Becomes Worth It

You’ll know you need Kubernetes when these symptoms appear:

Multi-service ML pipelines: Your inference stack grows from one model to a preprocessing service + embedding model + ranker + postprocessor. Coordinating 4+ containers across machines by hand is error-prone. Kubernetes Service resources make inter-service communication trivial:

apiVersion: v1
kind: Service
metadata:
  name: embedder
spec:
  selector:
    app: embedder
  ports:
  - port: 8001
---
apiVersion: v1
kind: Service
metadata:
  name: ranker
spec:
  selector:
    app: ranker
  ports:
  - port: 8002

Now your ranker service just calls http://embedder:8001/encode — no IP management.

Traffic spikes you can’t predict: If your model gets featured on ProductHunt and traffic jumps 10x overnight, Kubernetes HPA auto-scales based on CPU/memory:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: yolo-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

This keeps response times stable without manual intervention. Docker Compose can’t do this.

GPU resource sharing: When you have 2 NVIDIA T4 GPUs and 5 different models to serve, Kubernetes device plugins let you allocate fractions of GPUs:

resources:
  limits:
    nvidia.com/gpu: 1  # Full GPU
  # or
  limits:
    nvidia.com/mig-1g.5gb: 1  # MIG slice on A100

Docker can expose GPUs (--gpus all) but can’t share them across containers intelligently.

Compliance requirements: If you need audit logs, secrets encryption at rest, and role-based access control, Kubernetes has built-in primitives. Doing this with raw Docker means reinventing RBAC yourself.

The Latency Math Nobody Talks About

Kubernetes adds network hops. Every request goes through:

1. LoadBalancer / Ingress controller (~2-5ms)
2. Service proxy (kube-proxy or eBPF, ~1-3ms)
3. Your application container

That’s 3-8ms of overhead before your model even starts inference. For a 50ms model, that’s a 6-16% latency increase. For real-time applications (video processing, robotics), this matters.

Docker on a single host eliminates steps 1 and 2. If you’re using localhost or Unix sockets, overhead is <1ms.

I measured this on a BERT-base text classifier ($\text{latency}_{\text{model}} = 35\text{ms}$ on CPU):

• Docker direct: p50 = 37ms, p95 = 42ms
• Kubernetes (GKE): p50 = 44ms, p95 = 53ms

The variance increase is worse than the median shift. Kubernetes networking isn’t deterministic — sometimes you hit a slow path.

For latency-critical apps, consider ONNX Runtime optimizations before adding Kubernetes overhead.

The Hybrid Strategy That Works

Here’s what I actually recommend for first-time ML deployments:

Start with Docker + managed container service: Use AWS ECS Fargate or Google Cloud Run. You write a Dockerfile, they handle orchestration. It’s 80% of Kubernetes benefits with 20% of the complexity.

Cloud Run example:

# Build and deploy in 2 commands
gcloud builds submit --tag gcr.io/PROJECT/model-api
gcloud run deploy model-api --image gcr.io/PROJECT/model-api \
  --memory 4Gi --cpu 2 --max-instances 10

This gives you auto-scaling, HTTPS endpoints, and zero Kubernetes YAML. Perfect for validating demand before investing in K8s.

Graduate to Kubernetes when you have 3+ services: Once your architecture looks like preprocessing → model A → model B → postprocessing, Kubernetes service mesh and internal DNS justify the migration.

Use managed Kubernetes, never self-host: Running your own cluster means debugging etcd corruption, certificate expiry, and CNI plugin bugs. GKE Autopilot or EKS Fargate abstract most of this pain.

Real Cost Breakdown (1000 Req/Day)

Let’s price out both approaches for a mid-sized workload:

Kubernetes costs 2.6x more for the same traffic. The break-even point is around 10,000 req/day when you need 5+ Docker instances and coordination becomes manual labor.

The Stuff I Got Wrong

My first Kubernetes deployment used LoadBalancer Services for everything. I didn’t realize each Service provisions a $755/month cloud load balancer. After deploying 5 services, my bill jumped $756. The fix: one Ingress controller routing to multiple ClusterIP Services.

I also set resource requests too low (500m CPU for a model that actually needed 1500m). Kubernetes kept evicting my pods under load. The symptom was silent — requests would hang for 30 seconds during pod rescheduling. Setting proper limits based on actual profiling fixed it.

And I underestimated how long Kubernetes takes to learn. Budget 2-3 weeks of part-time study (official tutorials + real deployments) before you’re productive. Docker takes 2-3 days.

FAQ

Q: Can I use Kubernetes for a single model with no microservices?

Yes, but you’re paying the complexity cost without the coordination benefits. If you’re on AWS, ECS Fargate gives you auto-scaling and health checks without learning Kubernetes. Save K8s for when you have multiple services that need to communicate.

Q: Does Kubernetes help with model versioning and A/B testing?

Kubernetes Deployments support rolling updates, which lets you gradually shift traffic from model-v1 to model-v2. For true A/B testing (50/50 split, performance tracking), you need a service mesh like Istio on top of Kubernetes — that’s another layer of complexity. For simple version management, MLflow model registry is easier.

Q: What if I need GPU support — does that change the Docker vs Kubernetes decision?

Both support GPUs, but Kubernetes has better multi-tenant resource sharing. If you’re running one model on one GPU, Docker with --gpus all works fine. If you’re serving 5 models and want fractional GPU allocation, Kubernetes device plugins are worth it. But setup is painful — expect driver compatibility issues and kernel panics if you rush it. Grab some caffeinated dark chocolate before debugging CUDA errors in pod logs.

When to Make the Jump

Use Docker if: you’re serving one model, traffic is predictable, you’re a solo dev, and latency tolerance is loose.

Use Kubernetes if: you have 3+ services, traffic spikes unpredictably, you need multi-region deployments, or you’re sharing GPU resources.

The decision isn’t permanent. I’ve seen teams run Docker for 8 months, hit scaling limits, then migrate to Kubernetes in a weekend. The Dockerfile stays the same — you just wrap it in Kubernetes YAML.

What I’m still figuring out: whether Kubernetes is worth it for async batch jobs. CronJobs are nice, but they feel overengineered compared to cron + Docker. If you’ve solved this elegantly, I’m curious how you did it.