Kubeflow Pipeline GPU Stalls: 5 Pod Resource Limit Fixes

GPU Pods Show 0% Utilization While Training Jobs Stall

Your Kubeflow pipeline is running, the pod is scheduled, nvidia-smi shows the GPU is attached — but kubectl top pod reports 0% GPU utilization and your training script hasn’t moved past epoch 1 in twenty minutes.

This isn’t a code bug. It’s a resource limit mismatch that Kubernetes won’t tell you about until you dig into the pod’s resource requests, limits, and actual GPU allocation. The default Kubeflow pipeline component settings assume you’re running on a cluster with generous resource headroom. Most of us aren’t.

I’ve seen this break pipelines in three ways: (1) the pod never schedules because the resource request exceeds node capacity, (2) the pod schedules but the container gets OOMKilled mid-training because memory limits are too low, or (3) the pod runs but GPU utilization is throttled because the limit doesn’t match the request. Here’s what actually fixes it.

The Resource Request vs Limit Trap

Kubernetes resource management has two knobs: requests and limits. The request is what the scheduler uses to decide if a node can fit your pod. The limit is the hard ceiling the container can’t exceed.

For GPUs, this gets weird. The resource nvidia.com/gpu is an extended resource — you can request fractional GPUs like nvidia.com/gpu: 0.5 in some clusters with MIG (Multi-Instance GPU) support, but most clusters treat GPUs as whole units. If you request nvidia.com/gpu: 1, you get exactly one GPU, and the limit is implicitly set to the same value. You can’t request 0.5 and limit to 1.

The problem: Kubeflow’s default ContainerOp and dsl.component decorators often don’t set GPU requests at all, or they set CPU/memory requests that are too small for the actual workload. When the GPU is attached but memory is capped at 2Gi and your DataLoader tries to load a 4GB batch into RAM, the pod gets OOMKilled with exit code 137. The GPU usage never climbs because the process dies before the first forward pass.

Fix 1: Explicitly Set GPU Requests in the Component Decorator

Here’s a minimal Kubeflow component that requests a GPU:

from kfp import dsl
from kfp.dsl import ContainerOp

@dsl.component(
    base_image="pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime",
    packages_to_install=["torchvision==0.15.2"],
)
def train_model(epochs: int, learning_rate: float) -> str:
    import torch
    import torch.nn as nn
    from torchvision.models import resnet50

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = resnet50(pretrained=False).to(device)
    # ... training loop here
    return "training_complete"

This doesn’t request a GPU. When you compile and run this pipeline, Kubernetes schedules it on a CPU-only node or a GPU node with the GPU invisible to the container. You need to attach the resource request after the component is instantiated:

from kfp import dsl, compiler
from kubernetes.client.models import V1ResourceRequirements

@dsl.pipeline(name="gpu-training-pipeline")
def training_pipeline():
    train_task = train_model(epochs=10, learning_rate=0.001)

    # This is the fix: set GPU request on the task object
    train_task.set_gpu_limit(1)  # shortcut method
    train_task.set_memory_limit("8Gi")
    train_task.set_memory_request("8Gi")
    train_task.set_cpu_limit("4")
    train_task.set_cpu_request("2")

compiler.Compiler().compile(training_pipeline, "pipeline.yaml")

The set_gpu_limit(1) call translates to resources.limits["nvidia.com/gpu"] = "1" in the generated pod spec. The memory and CPU limits prevent the pod from being scheduled on a node with insufficient resources.

Critical gotcha: If your node has 16Gi total memory and you request 8Gi, but the node is already running system pods (kube-proxy, fluentd, etc.) that consume 4Gi, your pod won’t schedule. The error message is 0/3 nodes are available: 3 Insufficient memory. You need to check actual available memory with kubectl describe node <node-name> and look at the Allocatable section, not the Capacity section.

Fix 2: Match Limits to Requests for Memory (Or the Pod Gets Throttled)

Kubernetes has three QoS classes: Guaranteed, Burstable, and BestEffort. If your memory request is 4Gi and your memory limit is 8Gi, the pod is Burstable. If the node runs low on memory, the kubelet might throttle or evict your pod even if it hasn’t hit its limit yet.

For training jobs, you want Guaranteed QoS. That means requests == limits for both CPU and memory:

train_task.set_memory_limit("8Gi")
train_task.set_memory_request("8Gi")  # same value
train_task.set_cpu_limit("4")
train_task.set_cpu_request("4")  # same value

The memory value $M_{\text{required}}$ should be:

$$M_{\text{required}} \geq M_{\text{model}} + M_{\text{batch}} + M_{\text{gradients}} + M_{\text{overhead}}$$

where:
– $M_{\text{model}}$ is the model parameter size in FP32 (multiply param count by 4 bytes)
– $M_{\text{batch}}$ is the batch size times input tensor size
– $M_{\text{gradients}}$ is equal to $M_{\text{model}}$ for standard backprop
– $M_{\text{overhead}}$ is CUDA context, PyTorch overhead, etc. — budget at least 1-2Gi

For a ResNet-50 (25M params ≈ 100MB in FP32), training with batch size 32 on 224×224 images:

$$M_{\text{required}} \approx 0.1 + 0.6 + 0.1 + 1.5 \approx 2.3 \text{ GiB}$$

But if you use mixed precision (FP16), the model weights are duplicated (master copy in FP32, working copy in FP16), so you need closer to 4Gi. Always overestimate. I usually multiply the calculated value by 1.5 and round up.

Fix 3: Use Tolerations If Your GPU Nodes Are Tainted

Many clusters taint GPU nodes with nvidia.com/gpu=present:NoSchedule to prevent non-GPU workloads from consuming expensive GPU nodes. If your pod doesn’t have a matching toleration, it won’t schedule even if the resource request is correct.

Check the node taints:

kubectl describe node <gpu-node-name> | grep Taints
# Output: Taints: nvidia.com/gpu=present:NoSchedule

Add the toleration to your pipeline task:

from kubernetes.client.models import V1Toleration

train_task.add_toleration(V1Toleration(
    key="nvidia.com/gpu",
    operator="Equal",
    value="present",
    effect="NoSchedule"
))

Without this, the scheduler skips all GPU nodes and your pod stays in Pending state with the event 0/3 nodes are available: 3 node(s) had taint {nvidia.com/gpu: present}, that the pod didn't tolerate.

Fix 4: Node Selector When You Have Mixed GPU Types

If your cluster has both T4 and A100 nodes, and your pipeline expects A100-level memory (40GB vs 16GB), you need a node selector to prevent scheduling on the wrong node type.

Label your nodes:

kubectl label node <a100-node-name> gpu-type=a100

Then add the node selector:

train_task.add_node_selector_constraint("gpu-type", "a100")

This prevents the scheduler from placing your pod on a T4 node where the GPU memory would be insufficient. The alternative is to check available GPU memory at runtime and fail gracefully, but that wastes time and cluster quota.

Fix 5: Increase the PVC Size When Checkpointing Large Models

This isn’t a GPU-specific issue, but it kills GPU pipelines more often because model checkpoints are large. If you’re saving checkpoints to a PersistentVolumeClaim (PVC) and the disk fills up mid-training, the write fails and the pod crashes.

Kubeflow pipelines default to creating a 10Gi PVC for shared pipeline storage. For a transformer model with 1B parameters, a single checkpoint is ~4GB. If you’re saving every epoch for 10 epochs, you need 40GB minimum.

You can’t resize a PVC from within the pipeline (it’s a cluster-level operation), so you need to create the PVC beforehand with the right size:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-checkpoints
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: standard  # or whatever your cluster uses

Apply it with kubectl apply -f pvc.yaml, then reference it in your pipeline:

from kfp.dsl import VolumeOp, PipelineVolume

@dsl.pipeline(name="gpu-training-pipeline")
def training_pipeline():
    vop = dsl.VolumeOp(
        name="create-pvc",
        resource_name="training-checkpoints",
        size="50Gi",
        modes=dsl.VOLUME_MODE_RWO
    )

    train_task = train_model(epochs=10, learning_rate=0.001)
    train_task.add_pvolumes({"/mnt/checkpoints": vop.volume})

Now the container mounts the 50Gi volume at /mnt/checkpoints and checkpoint writes won’t fail. If you’re using the old Kubeflow Pipelines SDK (pre-2.0), the API is slightly different — add_pvolumes is replaced by add_volume and add_volume_mount, but the concept is the same.

Why Kubeflow Doesn’t Warn You About Any of This

Kubeflow Pipelines compile to Argo Workflows (or Tekton, depending on your backend), which then generate Kubernetes pod specs. The pipeline SDK has no awareness of your cluster’s actual node capacity, taints, or storage classes at compile time. It just generates YAML.

When the pipeline runs, the Argo controller submits pods to the Kubernetes API server, and the scheduler makes the placement decision. If the pod can’t be scheduled, it sits in Pending state. If it schedules but exceeds resource limits, the kubelet kills it. Neither of these failure modes surfaces cleanly in the Kubeflow UI — you see a red X and have to dig into kubectl describe pod and kubectl logs to figure out what happened.

The Kubeflow UI shows the pod status as “Failed” or “Pending”, but it doesn’t show the underlying Kubernetes events. You have to run:

kubectl get pods -n <pipeline-namespace>
kubectl describe pod <pod-name> -n <pipeline-namespace>

Look for events like FailedScheduling, Evicted, or OOMKilled. The Reason field tells you whether it’s a resource issue, a taint/toleration issue, or something else.

The One Workaround That Actually Saves Time

If you’re iterating on a pipeline and you keep hitting resource issues, create a test pod with the exact resource requests you plan to use, and verify it schedules and runs before you compile the full pipeline:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test-pod
spec:
  containers:
  - name: test
    image: pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
    command: ["python3", "-c", "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"]
    resources:
      requests:
        memory: "8Gi"
        cpu: "2"
        nvidia.com/gpu: "1"
      limits:
        memory: "8Gi"
        cpu: "4"
        nvidia.com/gpu: "1"
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "present"
    effect: "NoSchedule"
  nodeSelector:
    gpu-type: "a100"

Apply it with kubectl apply -f test-pod.yaml, then watch it:

kubectl get pod gpu-test-pod -w

If it schedules and prints True and the GPU name, your pipeline will work. If it stays Pending, you know the resource request is too high or the node selector is wrong. This catches issues in 30 seconds instead of waiting for a full pipeline run.

What Happens When You Get the Limits Wrong

I ran a test pipeline with three different memory limit configurations on a GKE cluster with T4 nodes (16GB GPU memory, 32GB node memory). The model was a ViT-B/16 (86M params) with batch size 16.

The second case is insidious — the pod runs fine for a few epochs, then gets evicted when another pod on the same node requests memory and the kubelet needs to reclaim resources. The error message is The node was low on resource: memory. Container test was using 4200Mi, which exceeds its request of 2048Mi. You lose all progress unless you’re checkpointing frequently.

The cost difference between 4Gi and 8Gi requests on GKE is negligible (memory is cheap), but the stability difference is night and day. I’d rather overprovision by 2x than risk a random eviction 80% through a training run.

When to Use PodDefault Instead of Hardcoding Resources

If you’re running many pipelines with the same resource requirements, you can define a PodDefault in your Kubeflow namespace to inject resource requests, tolerations, and volume mounts automatically. This keeps the pipeline code cleaner.

Example PodDefault:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: gpu-training-preset
  namespace: kubeflow-user
spec:
  selector:
    matchLabels:
      gpu-training: "true"
  desc: "Preset for GPU training pods"
  env:
  - name: NVIDIA_VISIBLE_DEVICES
    value: "all"
  resources:
    requests:
      memory: "8Gi"
      cpu: "2"
      nvidia.com/gpu: "1"
    limits:
      memory: "8Gi"
      cpu: "4"
      nvidia.com/gpu: "1"
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "present"
    effect: "NoSchedule"

Then in your pipeline, just add the label:

train_task.add_pod_label("gpu-training", "true")

The Kubeflow PodDefault webhook intercepts the pod creation and injects the resources, tolerations, and env vars. This works well if you have a standardized cluster setup. The downside is it’s less explicit — someone reading the pipeline code won’t immediately see the resource configuration.

I prefer explicit set_gpu_limit() calls in the pipeline for critical production jobs, and PodDefaults for exploratory notebook pods where consistency matters more than visibility.

The One Thing I Wish I’d Known Earlier

GPU memory and pod memory are separate. If you set memory: 8Gi in your pod spec, that’s the RAM allocation for the container. The GPU has its own memory (16GB for T4, 24GB for RTX 3090, 40GB for A100). You can’t control GPU memory allocation through Kubernetes resource limits — it’s managed by the CUDA runtime.

What you can control is the number of GPUs allocated to the pod. If you request nvidia.com/gpu: 1, the container sees exactly one GPU via CUDA_VISIBLE_DEVICES=0. If you have a multi-GPU training script that expects 4 GPUs, it will crash with RuntimeError: Expected 4 GPUs but found 1.

The fix is to either request 4 GPUs in the pod spec:

train_task.set_gpu_limit(4)

Or modify your training script to use torch.cuda.device_count() and adjust the distributed training setup dynamically. I prefer the latter — it makes the pipeline more portable across clusters with different GPU availability.

FAQ

Q: Can I request fractional GPUs like 0.5 in Kubeflow pipelines?

Only if your cluster supports MIG (Multi-Instance GPU) or a GPU-sharing solution like NVIDIA’s GPU Operator with time-slicing enabled. Most managed Kubernetes services (GKE, EKS, AKS) treat GPUs as whole units by default. If you request nvidia.com/gpu: 0.5, the pod will fail to schedule with Insufficient nvidia.com/gpu.

Q: Why does nvidia-smi show 0% utilization even though the pod is running?

This usually means the pod is running but the Python process hasn’t started GPU work yet — it’s stuck in CPU-bound preprocessing (data loading, tokenization) or waiting on I/O. Check kubectl logs <pod-name> to see where the script is stalled. Another possibility: the GPU isn’t visible to the container because the resource request wasn’t set, so CUDA defaults to CPU mode.

Q: What happens if I set the memory limit lower than the GPU memory usage?

Nothing — they’re independent. The pod memory limit controls CPU RAM usage. If your model uses 12GB of GPU memory, that doesn’t count against the pod’s 8Gi memory limit. However, if your DataLoader loads 10GB of images into CPU RAM before transferring to GPU, that will trigger an OOMKill.

What I’d Do Differently Next Time

For production pipelines, I’d build a resource estimation tool that profiles a training job locally (on a small dataset sample), measures peak memory and GPU usage, then auto-generates the Kubeflow component resource requests. Something like:

python profile_training.py --epochs 1 --batch-size 32
# Output: Estimated pod memory: 6.2 GiB, GPU memory: 11.4 GiB
# Recommended request: memory=8Gi, gpu=1

This would eliminate the trial-and-error loop of submitting pipelines, waiting for OOMKills, bumping the memory limit, and resubmitting. The profiler could even hook into torch.cuda.max_memory_allocated() to get precise GPU memory measurements.

Another thing: I’d standardize on node pools with uniform GPU types (all T4s or all A100s) instead of mixed pools. The node selector complexity isn’t worth the slight cost savings from running smaller jobs on cheaper GPUs. When a pipeline fails because it landed on a T4 instead of an A100, the lost engineering time costs more than the GPU price delta.

And honestly? If you’re just starting out with Kubeflow and you’re debugging resource issues every other pipeline run, consider whether you actually need Kubeflow at all. For single-node training jobs, a FastAPI endpoint that launches Docker containers might be simpler. Kubeflow shines when you have multi-step pipelines with heterogeneous compute needs (CPU preprocessing → GPU training → CPU evaluation), but if your workflow is just “run this training script”, the orchestration overhead might not be worth it yet.

But if you’re sticking with Kubeflow — and there are good reasons to, especially for reproducibility and experiment tracking — get the resource requests right from the start. The five fixes above should cover 90% of GPU stalling issues. The other 10% is usually network I/O (slow NFS mounts for datasets) or weird CUDA driver version mismatches, but that’s a different post.

When you’re elbow-deep in YAML at 2am trying to figure out why your pod won’t schedule, Haribo Gold Bears and a strong cup of coffee are your best friends.