Kubernetes HPA + Triton: Custom Metrics Autoscaling Setup

The Default CPU Metric Doesn’t Scale Inference Pods Right

Kubernetes Horizontal Pod Autoscaler (HPA) ships with CPU and memory metrics out of the box. Sounds great until you realize inference workloads don’t behave like web servers. I’ve seen Triton pods sit at 30% CPU utilization while requests queue for 2+ seconds because the GPU is maxed out. The cluster thinks everything’s fine. It’s not.

Triton Inference Server can batch requests and pipeline stages across CPU/GPU, which means CPU usage becomes a terrible proxy for “is this pod overloaded?” You need to scale on what actually matters: GPU utilization, queue depth, or batch occupancy. This post walks through wiring HPA to Triton’s Prometheus metrics so your cluster scales on signal that reflects reality.

I’ll show the full stack: Prometheus → Prometheus Adapter → HPA custom metrics → autoscaling Triton deployments. The key insight is that HPA only knows about metrics the API server exposes, so you’re building a pipeline from Triton metrics to custom.metrics.k8s.io API.

Why CPU-Based HPA Fails for GPU Inference

Triton runs inference in stages: request preprocessing (CPU), model execution (GPU), response postprocessing (CPU). A single request might use 5% CPU and 90% GPU. If you scale on CPU, HPA won’t add pods until CPU crosses the target — say 70%. By that point, GPU utilization is at 100%, requests are queuing, and P99 latency has spiked to 5 seconds.

The opposite problem also happens. During idle periods, Triton keeps GPU memory allocated and runs periodic health checks. CPU might sit at 15%, so HPA never scales down, and you’re paying for pods you don’t need. Docker vs Kubernetes for First ML Model: When to Use Each covers when this complexity is even worth it — spoiler: not always.

Triton exposes metrics via /metrics endpoint in Prometheus format. Key ones:
– nv_inference_request_success (counter): total successful inferences
– nv_inference_request_duration_us (histogram): latency distribution
– nv_inference_queue_duration_us (histogram): time requests spend queued
– nv_gpu_utilization (gauge): GPU busy percentage
– nv_gpu_memory_used_bytes (gauge): GPU VRAM in use

The queue duration is gold. If requests are waiting in the queue, you need more pods, regardless of CPU. If queue time is near zero and GPU utilization is low, you can scale down.

The Metric Pipeline: Triton → Prometheus → Adapter → HPA

HPA doesn’t scrape Prometheus directly. Kubernetes has three metric APIs:
1. metrics.k8s.io (core metrics: CPU, memory from metrics-server)
2. custom.metrics.k8s.io (per-pod/per-object metrics from Prometheus Adapter)
3. external.metrics.k8s.io (cluster-external metrics, also via adapter)

You’re targeting #2. Prometheus scrapes Triton pods, Prometheus Adapter queries Prometheus and exposes those metrics via the custom metrics API, then HPA reads from that API to make scaling decisions.

The data flow:

Triton pod :8002/metrics → Prometheus scrape → Prometheus Adapter queries → custom.metrics.k8s.io API → HPA reads → kubectl scale deployment

Each link in this chain can fail silently. I’ve debugged HPA showing “” for target metrics three times, each time a different cause: Prometheus not scraping the pod (label selectors wrong), adapter not exposing the metric (query template broken), or HPA not finding the metric (name mismatch).

Step 1: Expose Triton Metrics via Service Monitor

Triton serves metrics on port 8002 by default. Assuming you’re running Prometheus Operator (if not, you need raw Prometheus scrape configs, which is more YAML):

apiVersion: v1
kind: Service
metadata:
  name: triton-inference
  labels:
    app: triton
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  - name: grpc
    port: 8001
    targetPort: 8001
  - name: metrics  # critical: name this "metrics" so ServiceMonitor can find it
    port: 8002
    targetPort: 8002
  selector:
    app: triton
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: triton-metrics
  labels:
    release: prometheus  # must match your Prometheus Operator release label
spec:
  selector:
    matchLabels:
      app: triton
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Deploy this and check Prometheus UI → Status → Targets. You should see triton-inference/0 (3/3 up) or similar. If not, check the ServiceMonitor’s release label matches your Prometheus instance’s serviceMonitorSelector.

Query in Prometheus: nv_gpu_utilization. If you see data, scraping works. If not, kubectl logs -n monitoring prometheus-xyz-0 and grep for “triton” to see scrape errors.

Step 2: Configure Prometheus Adapter Rules

Prometheus Adapter translates PromQL queries into custom metrics. The config is a ConfigMap (or Helm values if you’re using the chart). Here’s the relevant section:

rules:
- seriesQuery: 'nv_inference_queue_duration_us_sum{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^nv_inference_queue_duration_us_sum$"
    as: "triton_queue_duration_us"
  metricsQuery: |
    rate(nv_inference_queue_duration_us_sum{<<.LabelMatchers>>}[2m])
    /
    rate(nv_inference_queue_duration_us_count{<<.LabelMatchers>>}[2m])

Breaking this down:
– seriesQuery: Discovers pods that emit this metric (must include namespace and pod labels)
– resources.overrides: Maps Prometheus labels to Kubernetes resources
– name.as: The metric name HPA will reference
– metricsQuery: PromQL to compute the actual value — here, average queue time per request over 2 minutes

The rate(...)[2m] is critical. Triton’s histogram metrics are cumulative counters. You need rate() to get per-second values, then divide sum by count to get mean. If you omit rate(), you’ll get ever-increasing numbers and HPA will scale infinitely (ask me how I know).

Another useful metric:

- seriesQuery: 'nv_gpu_utilization{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^nv_gpu_utilization$"
    as: "triton_gpu_utilization"
  metricsQuery: 'avg_over_time(nv_gpu_utilization{<<.LabelMatchers>>}[1m])'

GPU utilization is a gauge, so no rate() needed. We average over 1 minute to smooth out spikes.

After updating the adapter ConfigMap, restart it: kubectl rollout restart deployment prometheus-adapter -n monitoring. Then test:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/triton_queue_duration_us" | jq

You should see JSON with "value": "12345" (in microseconds). If you get 404, the adapter isn’t exposing the metric — double-check your seriesQuery matches actual Prometheus label names. Prometheus labels are case-sensitive; Kubernetes resource names are lowercase.

Step 3: Create HPA with Custom Metric Target

Now HPA can read the metric. Here’s a manifest targeting 50ms average queue time (50,000 µs):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: triton-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: triton-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: triton_queue_duration_us
      target:
        type: AverageValue
        averageValue: "50000"  # 50ms in microseconds
  behavior:  # optional: tune scale-up/down velocity
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60

The type: Pods metric computes the average across all pods in the deployment. If average queue time exceeds 50ms, HPA scales up. The behavior section is optional but recommended — it prevents thrashing. Scale-up is aggressive (50% more pods every 60s), scale-down is conservative (1 pod per minute, only after 5 minutes of low utilization).

You can also target GPU utilization:

  - type: Pods
    pods:
      metric:
        name: triton_gpu_utilization
      target:
        type: AverageValue
        averageValue: "75"  # 75% GPU utilization

Or combine both — HPA scales if any metric exceeds target. This is often the right move: scale on queue time (immediate user pain) or GPU utilization (approaching capacity).

Deploy the HPA and watch:

kubectl describe hpa triton-hpa

Look for:

Metrics:                                                      
  "triton_queue_duration_us" on pods:  32000 / 50000 (avg)
Current replicas: 2
Desired replicas: 2

If it says <unknown>, HPA can’t find the metric. Common causes:
– Metric name typo (check exact spelling in adapter config and HPA manifest)
– Pods don’t have the label selector HPA expects (check scaleTargetRef matches deployment)
– Adapter hasn’t scraped data yet (wait 30s after restart)

Testing Autoscaling Under Load

Time to generate traffic. I used hey (HTTP load generator) hitting a BERT model on Triton:

kubectl port-forward svc/triton-inference 8000:8000 &
hey -z 5m -c 50 -m POST -H "Content-Type: application/json" \
  -d '{"inputs":[{"name":"input_ids","shape":[1,128],"datatype":"INT32","data":[[101,2023,2003,1037,3231,...]]}]}' \
  http://localhost:8000/v2/models/bert/infer

50 concurrent requests for 5 minutes. Within 90 seconds, I saw:
– Queue duration climbed from 15ms to 120ms
– HPA detected average 120ms > 50ms target
– Scaled deployment from 2 → 3 pods
– After new pod became ready (~30s), queue duration dropped to 40ms
– HPA held at 3 pods

After stopping hey, queue duration dropped to <5ms within 2 minutes, but HPA waited the full 5-minute stabilization window before scaling down to 2.

One gotcha: if your pods take a long time to become ready (e.g., loading a 5GB model), you need a longer stabilizationWindowSeconds for scale-up. Otherwise HPA will see queue time still high after 60s, scale up again, and you’ll end up with more pods than needed.

Handling Multi-Model Scenarios

Triton can serve multiple models in one instance. Metrics include a model label, e.g., nv_inference_queue_duration_us_sum{model="bert"}. If you’re running BERT and ResNet in the same pods, you might want to scale based on the worst queue time across models.

Prometheus Adapter can do this:

metricsQuery: |
  max(
    rate(nv_inference_queue_duration_us_sum{<<.LabelMatchers>>}[2m])
    /
    rate(nv_inference_queue_duration_us_count{<<.LabelMatchers>>}[2m])
  ) by (pod)

The max(...) by pod gives you the worst queue time among all models on each pod. HPA then scales based on the pod with the longest queue.

Alternatively, if models have different SLAs, deploy separate Triton instances (one deployment per model) and configure separate HPAs. More operational overhead, but you can tune scale targets independently.

The Problem with Batch Inference Jobs

HPA is designed for long-running services, not batch jobs. If you’re running nightly inference over 10 million images, HPA will scale up as the queue fills, but by the time new pods are ready, the job might be half done. You end up paying for pods that process the tail end, then scale down after the job finishes.

For batch workloads, consider Keda (Kubernetes Event-Driven Autoscaling) instead. Keda can scale based on message queue depth (RabbitMQ, Kafka, SQS) or cron schedules. You can scale a Triton deployment from 0 → 10 when a batch job starts, then back to 0 when the queue is empty. HPA can’t scale to zero.

But if you’re serving real-time requests (user-facing API), HPA + custom metrics is the right pattern. I’m not entirely sure why Keda and HPA can’t coexist on the same deployment — the docs say they interfere with each other, but I haven’t tested it thoroughly.

Debugging When Metrics Show

This happens. Here’s the checklist:

1. Prometheus is scraping Triton: Query nv_inference_queue_duration_us_sum in Prometheus UI. If no data, check ServiceMonitor labels and Prometheus serviceMonitorSelector.

2. Adapter can query Prometheus: kubectl logs -n monitoring prometheus-adapter-xyz and look for errors like "unable to update list of all metrics". Usually means adapter can’t reach Prometheus service.

3. Adapter exposes the metric: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq lists all custom metrics. Your metric should be in there. If not, adapter config is wrong.

4. Metric has correct labels: Adapter requires namespace and pod labels. Triton metrics include pod by default, but if you’re aggregating with sum() without by (pod), labels disappear and HPA can’t match them to pods.

5. HPA references correct metric name: Must match adapter’s name.as field exactly. Case-sensitive. No typos.

One time I spent 30 minutes debugging <unknown> only to realize I’d named the metric triton_queue_duration_us in the adapter but triton_queue_time_us in the HPA manifest. Kubernetes doesn’t give you a helpful error — just <unknown>.

Cost Implications

Autoscaling saves money if your traffic is spiky. If you have consistent 24/7 load, HPA just adds complexity — static replicas are simpler. I tested this on GKE with g2-standard-4 instances (1x NVIDIA L4 GPU, ~$0.70/hr spot price). Without HPA, I ran 4 pods continuously: $67/day. With HPA scaling 2-8 pods based on queue time, average utilization dropped to 3.2 pods: $54/day. 20% savings, but only because traffic had a clear daily pattern (high during US business hours, low overnight).

If your cluster already runs Prometheus for monitoring, Prometheus Adapter adds negligible cost (50MB RAM, <0.1 CPU). But if you’re spinning up Prometheus just for HPA, that’s a 1-2 GB RAM overhead for the Prometheus pod, plus persistent storage for metrics. At that point, maybe just use CloudWatch metrics (if on AWS) and the AWS CloudWatch adapter instead.

Debugging this setup while sipping Dark Chocolate Espresso Beans at 2am? Slightly more bearable.

When to Just Use Kubernetes Cluster Autoscaler Instead

HPA scales pods. Cluster Autoscaler scales nodes. If you’re on a cloud provider with node autoscaling enabled, you need both. HPA adds pods, Cluster Autoscaler provisions nodes when pods are unschedulable.

But if you’re running on-prem or in a fixed-size cluster, HPA can’t help if you’re already at node capacity. I’ve seen HPA scale a deployment to 10 replicas, but only 6 pods fit on available nodes, so the extra 4 stay Pending. Kubernetes doesn’t give you a warning — the pods just sit there. Check kubectl get pods for Pending status and kubectl describe pod to see if it’s Insufficient nvidia.com/gpu.

In that case, you either need more nodes (Cluster Autoscaler) or you need to rethink resource requests. If each Triton pod requests 1 full GPU and your nodes have 2 GPUs, max 2 pods per node. If you’re only using 50% of GPU capacity, consider MIG (Multi-Instance GPU) partitioning, but that’s a whole other setup.

The Alternative: Scale on Queue Depth Instead of Latency

Another approach: export Triton’s queue depth as a metric and scale directly on that. Triton doesn’t expose this out of the box, but you can add a sidecar container that queries Triton’s gRPC ModelStatistics API every 10s and exports a Prometheus gauge.

Sidecar pseudocode:

import time
import grpc
from prometheus_client import Gauge, start_http_server
from tritonclient.grpc import service_pb2, service_pb2_grpc

queue_depth = Gauge('triton_queue_depth', 'Current inference queue size', ['model'])

def scrape_triton():
    channel = grpc.insecure_channel('localhost:8001')
    stub = service_pb2_grpc.GRPCInferenceServiceStub(channel)
    while True:
        stats = stub.ModelStatistics(service_pb2.ModelStatisticsRequest(name=""))
        for model_stat in stats.model_stats:
            queue_depth.labels(model=model_stat.name).set(model_stat.inference_queue.size)
        time.sleep(10)

if __name__ == '__main__':
    start_http_server(9090)  # expose metrics on :9090
    scrape_triton()

Deploy this as a sidecar, configure Prometheus to scrape :9090, then use triton_queue_depth in HPA. Advantage: queue depth is more intuitive than queue duration — “scale when queue > 50 requests” vs “scale when queue latency > 50ms”. The math is simpler: no rate() or histogram division.

Disadvantage: you’re maintaining custom code, and if Triton’s gRPC API changes, your sidecar breaks. I’d only do this if queue duration metrics aren’t granular enough (e.g., you need per-model queue depth and Triton’s aggregate histogram doesn’t give you that).

FAQ

**Q: Can I use HPA with Triton’s dynamic batching?

Yes, but be careful. Dynamic batching means a single “inference” might process 1 request or 32, depending on queue depth. If you scale on request count, you’ll over-provision — Triton can handle more requests per pod when batching is effective. Queue duration is a better signal because it reflects actual user-perceived latency, regardless of batch size.

**Q: What if my model load time is 2 minutes?

HPA will scale up, but new pods won’t accept traffic until readiness probe passes. Set initialDelaySeconds on readiness probe to your model load time + 10s buffer, and tune HPA’s stabilizationWindowSeconds to at least 3x model load time. Otherwise HPA will scale up multiple times before the first new pod is ready.

**Q: Does this work with Triton on CPU-only pods?

Yes, same setup. Just use nv_inference_queue_duration_us instead of GPU metrics. CPU inference workloads often benefit even more from autoscaling because CPU is cheaper — you can scale 2-20 pods without worrying about GPU availability. Queue time is still the right metric to track.

Use Queue Duration for User-Facing APIs, GPU Utilization for Batch

If you’re serving real-time requests where latency matters, scale on queue duration. It directly correlates with user experience. If queue time is high, users are waiting — add pods. If it’s near zero, you’re over-provisioned — scale down.

For batch inference (nightly jobs, video processing pipelines), scale on GPU utilization instead. Batch jobs don’t care about latency; they care about throughput. If GPU is at 90%+ for sustained periods, you’re saturating the pod — add replicas. If GPU is at 30%, you’re wasting money.

I haven’t tested scaling on a combination of metrics (queue time and GPU utilization) in production. My intuition is it would work fine — HPA scales if any metric exceeds target — but I’d want to see how it behaves when one metric is high and the other is low. Does it thrash? Does it converge? That’s an experiment for another month.