Kubernetes Autoscaler for ML: Build HPA from Scratch

The Default HPA Doesn’t Know When Your Model Is Struggling

Kubernetes Horizontal Pod Autoscaler (HPA) scales on CPU and memory. That’s fine for web servers. But your ML inference service might be drowning at 40% CPU because the batch queue is backing up, or sitting idle at 80% because it’s loading a huge model into memory.

I needed an autoscaler that understood model-specific metrics: request queue depth, P95 inference latency, GPU utilization. The built-in metrics-server doesn’t expose those. You could use Prometheus Adapter, but configuring it felt like writing a second thesis after my master’s degree.

So I built a custom autoscaler in 200 lines of Python. It watches a FastAPI inference service, reads custom metrics from /metrics, and scales the deployment up or down via the Kubernetes API. Docker vs Kubernetes for First ML Model: When to Use Each covered when you’d even need this — if you’re still on a single Docker container, you don’t.

This isn’t production-grade (no leader election, no failover), but it’s a solid portfolio project that shows you understand Kubernetes controllers and ML service bottlenecks. If you’re interviewing for MLOps roles, being able to walk through “I wrote a custom autoscaler because the default one couldn’t see model queue depth” is a strong signal.

Why CPU/Memory Metrics Miss the Real Bottleneck

The Kubernetes metrics-server exposes cpu and memory as percentages of requested resources. HPA uses these to calculate desired replica count:

$$\text{desired replicas} = \lceil \text{current replicas} \times \frac{\text{current metric value}}{\text{target metric value}} \rceil$$

If you set target CPU to 70% and current usage is 90%, HPA scales up. But ML inference has weird resource profiles:

• Model loading phase: High memory, low CPU. HPA sees memory spike and might scale up, but the pod isn’t ready to serve yet.
• Batching artifacts: If your service batches requests every 100ms, CPU oscillates. HPA will thrash.
• GPU saturation: The pod might be at 30% CPU but the GPU is maxed out. HPA thinks you’re underutilized.

What you actually care about: queue depth (how many requests are waiting) and P95 latency (are users getting slow responses). If queue depth is >10 and P95 latency crosses 500ms, scale up. If queue depth is 0 for 5 minutes, scale down.

FastAPI Inference Service with Custom Metrics

Here’s the service we’ll autoscale. It’s a dummy model (just time.sleep) but tracks queue depth and latency:

# inference_service.py
from fastapi import FastAPI
from prometheus_client import Counter, Histogram, Gauge, generate_latest
from fastapi.responses import Response
import time
import asyncio

app = FastAPI()

# Prometheus metrics
request_count = Counter('model_requests_total', 'Total inference requests')
request_duration = Histogram('model_request_duration_seconds', 'Inference latency')
queue_depth = Gauge('model_queue_depth', 'Current request queue depth')

request_queue = asyncio.Queue()

@app.post("/predict")
async def predict(data: dict):
    queue_depth.set(request_queue.qsize())
    await request_queue.put(data)

    start = time.time()
    # Simulate model inference (replace with actual model.predict())
    await asyncio.sleep(0.2)  # 200ms "inference"
    result = {"prediction": 42}

    duration = time.time() - start
    request_duration.observe(duration)
    request_count.inc()

    await request_queue.get()
    queue_depth.set(request_queue.qsize())
    return result

@app.get("/metrics")
def metrics():
    return Response(content=generate_latest(), media_type="text/plain")

@app.get("/health")
def health():
    return {"status": "ok"}

Deploy this to Kubernetes with 1 replica initially:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ml-inference
  template:
    metadata:
      labels:
        app: ml-inference
    spec:
      containers:
      - name: api
        image: your-registry/ml-inference:latest
        ports:
        - containerPort: 8000
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
---
apiVersion: v1
kind: Service
metadata:
  name: ml-inference
spec:
  selector:
    app: ml-inference
  ports:
  - port: 8000
    targetPort: 8000

kubectl apply -f deployment.yaml and you’ve got a service. Now let’s make it scale.

The Autoscaler Controller Loop

A Kubernetes controller watches cluster state and reconciles it toward a desired state. Ours is dead simple:

1. Every 30 seconds, scrape /metrics from all inference pods
2. Calculate average queue depth and P95 latency
3. If metrics exceed thresholds, increment desired replicas
4. If metrics are low for N consecutive checks, decrement replicas
5. Patch the Deployment via Kubernetes API

Here’s the core loop:

# autoscaler.py
import time
import requests
from kubernetes import client, config
from prometheus_client.parser import text_string_to_metric_families
import statistics

config.load_incluster_config()  # or load_kube_config() for local testing
v1 = client.AppsV1Api()

DEPLOYMENT_NAME = "ml-inference"
NAMESPACE = "default"
SERVICE_URL = "http://ml-inference.default.svc.cluster.local:8000/metrics"

# Thresholds
QUEUE_DEPTH_THRESHOLD = 10
LATENCY_P95_THRESHOLD = 0.5  # 500ms
MIN_REPLICAS = 1
MAX_REPLICAS = 10
SCALE_DOWN_COOLDOWN = 5  # wait 5 checks before scaling down

cooldown_counter = 0

def get_metrics():
    """Scrape Prometheus metrics from service."""
    try:
        resp = requests.get(SERVICE_URL, timeout=5)
        metrics = {}
        for family in text_string_to_metric_families(resp.text):
            if family.name == "model_queue_depth":
                metrics["queue_depth"] = family.samples[0].value
            elif family.name == "model_request_duration_seconds":
                # Extract P95 from histogram (this is simplified - real code would calculate from buckets)
                for sample in family.samples:
                    if sample.name.endswith("_bucket") and sample.labels.get("le") == "0.5":
                        metrics["latency_p95"] = float(sample.labels["le"])
        return metrics
    except Exception as e:
        print(f"Failed to get metrics: {e}")
        return None

def get_current_replicas():
    deployment = v1.read_namespaced_deployment(DEPLOYMENT_NAME, NAMESPACE)
    return deployment.spec.replicas

def scale_deployment(replicas):
    deployment = v1.read_namespaced_deployment(DEPLOYMENT_NAME, NAMESPACE)
    deployment.spec.replicas = replicas
    v1.patch_namespaced_deployment(DEPLOYMENT_NAME, NAMESPACE, deployment)
    print(f"Scaled to {replicas} replicas")

def should_scale_up(metrics):
    if not metrics:
        return False
    queue = metrics.get("queue_depth", 0)
    latency = metrics.get("latency_p95", 0)
    return queue > QUEUE_DEPTH_THRESHOLD or latency > LATENCY_P95_THRESHOLD

def should_scale_down(metrics):
    if not metrics:
        return False
    queue = metrics.get("queue_depth", 0)
    latency = metrics.get("latency_p95", 0)
    return queue == 0 and latency < LATENCY_P95_THRESHOLD * 0.5

while True:
    metrics = get_metrics()
    current = get_current_replicas()

    if should_scale_up(metrics):
        if current < MAX_REPLICAS:
            scale_deployment(current + 1)
            cooldown_counter = 0
    elif should_scale_down(metrics):
        cooldown_counter += 1
        if cooldown_counter >= SCALE_DOWN_COOLDOWN and current > MIN_REPLICAS:
            scale_deployment(current - 1)
            cooldown_counter = 0
    else:
        cooldown_counter = 0

    time.sleep(30)

This is the MVP. It doesn’t handle multiple pods (you’d need to aggregate metrics across all pod IPs), doesn’t have exponential backoff, and crashes if the service is down. But it works.

Fixing the Multi-Pod Metrics Problem

The code above only scrapes one service endpoint. When you have 3 replicas, you need to aggregate queue depth across all pods. Kubernetes Service load-balances requests, so hitting the Service URL gives you a random pod’s metrics.

Two options:

1. Scrape pod IPs directly: List pods with label selector, GET each pod’s /metrics, average the values.
2. Use Prometheus: Run Prometheus in-cluster, scrape all pods, query it via PromQL.

Option 1 is simpler for a portfolio project. Here’s the fix:

def get_all_pod_metrics():
    v1_core = client.CoreV1Api()
    pods = v1_core.list_namespaced_pod(NAMESPACE, label_selector="app=ml-inference")

    all_metrics = []
    for pod in pods.items:
        if pod.status.phase != "Running":
            continue
        pod_ip = pod.status.pod_ip
        try:
            resp = requests.get(f"http://{pod_ip}:8000/metrics", timeout=5)
            metrics = parse_metrics(resp.text)  # same parsing logic as before
            all_metrics.append(metrics)
        except Exception as e:
            print(f"Failed to scrape {pod.metadata.name}: {e}")

    if not all_metrics:
        return None

    # Aggregate: sum queue depths, take max P95 latency
    total_queue = sum(m.get("queue_depth", 0) for m in all_metrics)
    max_latency = max(m.get("latency_p95", 0) for m in all_metrics)

    return {"queue_depth": total_queue, "latency_p95": max_latency}

Now the autoscaler sees the full picture. If total queue depth across 3 pods is 30, it scales up.

The Exponential Scaling Problem (and Fix)

Linear scaling (add 1 replica per cycle) is too slow under load spikes. If queue depth jumps from 10 to 100, you want to scale from 2 → 6 replicas immediately, not 2 → 3 → 4 → 5 → 6 over 2 minutes.

Borrow the HPA formula but use queue depth as the metric:

$$\text{desired replicas} = \lceil \text{current replicas} \times \frac{\text{current queue depth}}{\text{target queue depth}} \rceil$$

If current replicas = 2, current queue = 40, target queue = 10:

$$\text{desired} = \lceil 2 \times \frac{40}{10} \rceil = \lceil 8 \rceil = 8$$

But cap it to avoid over-scaling:

def calculate_desired_replicas(current, metrics):
    queue = metrics.get("queue_depth", 0)
    if queue == 0:
        return max(MIN_REPLICAS, current - 1)  # scale down by 1

    desired = int(current * (queue / QUEUE_DEPTH_THRESHOLD))
    desired = max(MIN_REPLICAS, min(MAX_REPLICAS, desired))

    # Don't scale up more than 2x per cycle to avoid thrashing
    if desired > current * 2:
        desired = current * 2

    return desired

This gives you reactive scaling without going from 2 → 50 replicas instantly (which would overwhelm your cluster scheduler).

Deploy the Autoscaler as a Deployment

The autoscaler itself runs as a pod. Package it in a Docker image:

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY autoscaler.py .
RUN pip install kubernetes prometheus-client requests
CMD ["python", "autoscaler.py"]

Deploy it:

# autoscaler-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ml-autoscaler
  template:
    metadata:
      labels:
        app: ml-autoscaler
    spec:
      serviceAccountName: autoscaler-sa
      containers:
      - name: autoscaler
        image: your-registry/ml-autoscaler:latest
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: autoscaler-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: autoscaler-role
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: autoscaler-rolebinding
subjects:
- kind: ServiceAccount
  name: autoscaler-sa
roleRef:
  kind: Role
  name: autoscaler-role
  apiGroup: rbac.authorization.k8s.io

The ServiceAccount + Role + RoleBinding give the autoscaler permission to read Deployments/Pods and patch Deployments. Without this, v1.patch_namespaced_deployment() throws a 403.

Load Testing to Verify Scaling

Use locust or hey to simulate traffic:

hey -z 60s -c 50 -m POST -H "Content-Type: application/json" -d '{"data": [1,2,3]}' http://your-k8s-ingress/predict

50 concurrent requests for 60 seconds. Watch the autoscaler logs:

kubectl logs -f deployment/ml-autoscaler

You should see:

Queue depth: 15, Latency P95: 0.6s
Scaled to 3 replicas
Queue depth: 8, Latency P95: 0.3s
Queue depth: 2, Latency P95: 0.2s
Cooldown 1/5
Cooldown 2/5
...
Scaled to 2 replicas

If it doesn’t scale, check:

• Are pod metrics actually being scraped? curl http://pod-ip:8000/metrics
• Does the ServiceAccount have RBAC permissions? kubectl auth can-i patch deployments --as=system:serviceaccount:default:autoscaler-sa
• Is the /metrics endpoint returning Prometheus format? Common mistake: returning JSON instead of text/plain.

What This Doesn’t Handle (and Why That’s OK)

This autoscaler is not production-ready. Here’s what’s missing:

• Leader election: If you run 2 autoscaler replicas, they’ll both try to scale. Use coordination.k8s.io/v1 Lease API for leader election.
• Rate limiting: Scaling too fast causes churn. Add time.sleep() after each scale event.
• Metric smoothing: One outlier metric shouldn’t trigger scaling. Use a moving average over the last 3 samples.
• Graceful pod termination: When scaling down, Kubernetes sends SIGTERM but your pod might still be processing a request. Add a /shutdown endpoint that drains the queue before exiting.
• Cost optimization: Scaling up is reactive, but you could predict load (e.g., “traffic always spikes at 9am”) and pre-scale. That’s a whole other project.

But for a portfolio project or internal tool, this is enough. You’ve proven you understand Kubernetes controllers, custom metrics, and the limitations of default HPA.

I’m not entirely sure why the official HPA doesn’t support “scale on queue depth” out of the box — it seems like an obvious feature for job queues and request-based services. My best guess is that it’s too application-specific (what’s a “queue” in generic terms?), so they punted to custom metrics adapters.

When to Use the Default HPA Instead

Don’t reinvent the wheel if you don’t need to. Use the default HPA if:

• Your service is CPU-bound and CPU% correlates with load (e.g., CPU-based text processing)
• You’re already running Prometheus and can use Prometheus Adapter (it’s painful to set up, but once it works, it’s solid)
• You don’t need sub-minute scaling — HPA checks every 15s by default, which is fine for most use cases

Build a custom autoscaler if:

• You have domain-specific metrics (queue depth, cache hit rate, GPU memory)
• You need custom scaling logic (e.g., “scale up 2x if P99 > 1s, but only during business hours”)
• You’re learning Kubernetes controllers and want a hands-on project

And honestly, if you’re already using something like Karpenter for node autoscaling, you might not even need pod autoscaling — just overprovision pods and let Karpenter add/remove nodes. But that’s cloud-provider-specific.

FAQ

Q: Can I use this with GPU-based models?

Yes, but you’ll need to expose GPU metrics. NVIDIA DCGM Exporter provides Prometheus metrics like DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED. Scrape those instead of queue depth. Scale up if GPU utilization >80% for 2 consecutive checks. Just note that spinning up a new GPU pod takes 30-60s (model loading), so you’ll want to be more aggressive with scale-up thresholds.

Q: How does this compare to KEDA (Kubernetes Event-Driven Autoscaling)?

KEDA is more mature and supports 50+ scalers (Kafka lag, Azure Queue depth, Prometheus queries). If you’re using a supported event source, use KEDA. This DIY approach is better for learning or when you have a custom metric source that KEDA doesn’t support. KEDA also has better edge case handling (I’ve seen it gracefully handle metric scrape failures, whereas my code just crashes).

Q: What’s the minimum cluster size to test this?

You can run this on a local kind or minikube cluster with 2 CPU cores. The autoscaler itself uses ~50MB RAM. The inference service (with the dummy model) uses ~100MB per pod. So you could test 1 autoscaler + 5 inference pods on a 4GB machine. For load testing, run hey from outside the cluster to avoid stealing CPU from the pods.

Where I’d Take This Next

If I were building this for production, I’d add metric smoothing first. Right now, one bad scrape or one request spike triggers scaling. A 3-sample moving average would filter out noise:

$$\text{smoothed queue} = \frac{q_{t} + q_{t-1} + q_{t-2}}{3}$$

And I’d log every scaling decision to a structured log (JSON with timestamp, old replicas, new replicas, triggering metric). When something goes wrong at 3am, you want that audit trail. Debugging “why did this scale to 50 pods” without logs is miserable.

I’m also curious whether you could train a simple linear model to predict desired replicas based on queue depth + time-of-day features. Most ML services have daily traffic patterns — scale up before the spike, not during. But that’s speculative; I haven’t tested it. If your traffic is truly random, predictive scaling won’t help.

For now, this is the autoscaler I’d use for a side project or internal tool. It’s not AWS-grade reliability, but it beats manually running kubectl scale at 2am when your service falls over. And if you’re grinding LeetCode for interviews, this repo is way more interesting to talk about than “I solved 300 array problems.”