MLflow → Kubernetes Operator: 4 Migration Pitfalls

The Model Registry That Stopped Working

Migrating from MLflow Model Registry to a Kubernetes Operator isn’t just swapping out one tool for another. It’s changing how your models get versioned, promoted, and deployed — and if you mess up the handoff, you’ll discover your staging environment is serving last week’s model while production points at a registry that no longer exists.

I’ve seen this migration go wrong in predictable ways. The mistakes aren’t dramatic explosions. They’re quiet: a model version mismatch that surfaces three days later, a rollback that doesn’t work because the old artifact storage is gone, or a deployment that succeeds but serves inference from the wrong endpoint.

This post walks through four specific pitfalls I’ve hit (or watched others hit) when moving from MLflow’s model registry to a Kubernetes-native operator like Seldon Core or KServe. These aren’t edge cases — they’re the default failure modes if you don’t actively prevent them.

Pitfall 1: Model Version Drift Between Registries

MLflow’s model registry uses a three-stage lifecycle: None, Staging, Production. When you transition a model to Production, it’s a metadata update in the MLflow backend (usually MySQL or SQLite). The actual model artifact stays in S3, GCS, or whatever blob storage you configured.

Kubernetes operators use a different model. In KServe, for instance, you deploy an InferenceService resource that points to a storage URI. There’s no “transition to production” button — you update the resource YAML and apply it. Version management is implicit: whatever URI is in the spec is what gets served.

Here’s where things break: if you’re running both systems during migration, you can have MLflow showing model version v12 as Production while your Kubernetes cluster is serving v10 because someone forgot to update the InferenceService spec after the last MLflow promotion.

The symptom: your model metrics dashboard (which queries MLflow) says you’re serving the model trained on last week’s data. Your inference logs show predictions from two weeks ago.

import mlflow
from kubernetes import client, config

# MLflow says this is Production
mlflow.set_tracking_uri("http://mlflow.internal:5000")
client = mlflow.tracking.MlflowClient()
versions = client.get_latest_versions("fraud-detector", stages=["Production"])
mlflow_prod_version = versions[0].version  # e.g., "12"
mlflow_model_uri = versions[0].source  # s3://models/fraud-detector/12

# Kubernetes says this is deployed
config.load_kube_config()
v1 = client.CustomObjectsApi()
isvc = v1.get_namespaced_custom_object(
    group="serving.kserve.io",
    version="v1beta1",
    namespace="ml-prod",
    plural="inferenceservices",
    name="fraud-detector"
)
k8s_model_uri = isvc["spec"]["predictor"]["model"]["storageUri"]
# This might still be s3://models/fraud-detector/10

if mlflow_model_uri != k8s_model_uri:
    print(f"VERSION MISMATCH: MLflow={mlflow_model_uri}, K8s={k8s_model_uri}")

The fix: automate version sync. Either write a controller that watches MLflow transitions and updates Kubernetes resources, or go all-in on Kubernetes and stop using MLflow’s promotion workflow entirely. Half-migrating is the worst state.

I’ve seen teams write a cron job that polls MLflow every 5 minutes and reconciles the Kubernetes state. It works, but it’s a 5-minute lag window where your production serving is out of sync with your model registry. For high-stakes applications (fraud detection, medical diagnosis), that’s unacceptable.

Better approach: if you’re committed to Kubernetes, treat MLflow as an experiment tracker only. Use a GitOps workflow where model URIs live in version-controlled YAML files, and promotions happen via pull requests that update the InferenceService spec. Your CI/CD pipeline validates the model exists before merging.

Pitfall 2: Artifact Storage Becomes a Single Point of Failure

MLflow’s artifact store configuration is straightforward: you set MLFLOW_ARTIFACT_ROOT to an S3 bucket, and every logged model ends up there. The MLflow server acts as a proxy — your training jobs don’t hit S3 directly; they upload through the MLflow API.

Kubernetes operators expect direct storage access. KServe’s storageUri field can point to s3://, gs://, or pvc://, but the serving pod needs credentials to fetch the model. If you’re running on AWS, that means an IAM role with s3:GetObject permissions. On GCP, it’s a service account with Storage Object Viewer.

During migration, you might have:
– MLflow still writing to s3://mlflow-artifacts/
– Kubernetes pods trying to read from s3://k8s-models/
– A manual sync script copying artifacts between buckets

This is fragile. If the sync script fails (which it will — S3 eventually consistent, network blip, out of disk), your Kubernetes deployments start pulling stale models or fail entirely with ModelNotFound errors.

Worse: if you decide to shut down the MLflow server mid-migration (to save costs, or because “we’re using Kubernetes now”), any model artifacts that only exist in MLflow’s internal storage are gone. I’ve seen teams lose months of experiment history this way.

# This worked when MLflow was running
model_uri = "models:/fraud-detector/12"
model = mlflow.pyfunc.load_model(model_uri)

# After shutting down MLflow, this fails
# because the URI resolver can't reach the backend

The fix: before you migrate anything, ensure all model artifacts are in a storage layer both systems can access. Copy everything from MLflow’s artifact store to your Kubernetes-accessible bucket. Verify every model version you care about.

Then, update your training pipelines to write models directly to the target storage, bypassing MLflow’s artifact API. You can still log to MLflow for experiment tracking — just use mlflow.log_artifact() for metadata and save the model binary yourself.

import mlflow
import boto3
import joblib

# Train model
model = train_fraud_model(X_train, y_train)

# Save locally first
joblib.dump(model, "model.pkl")

# Upload to Kubernetes-accessible storage
s3 = boto3.client("s3")
version = "13"  # Increment however you like
s3.upload_file("model.pkl", "k8s-models", f"fraud-detector/{version}/model.pkl")

# Log to MLflow for tracking (optional)
with mlflow.start_run():
    mlflow.log_params({"n_estimators": 100, "max_depth": 10})
    mlflow.log_metric("auc", 0.87)
    mlflow.log_artifact("model.pkl")
    mlflow.set_tag("k8s_uri", f"s3://k8s-models/fraud-detector/{version}")

Now MLflow has a record of the experiment, and Kubernetes can fetch the model without depending on MLflow’s uptime.

Pitfall 3: Rollback Semantics Change Completely

MLflow rollback is trivial: you transition a previous model version back to Production. The registry updates, and any system polling the registry (your serving layer, your monitoring dashboards) picks up the change.

Kubernetes rollback is a state reconciliation problem. If you deploy a broken model via kubectl apply -f model-v13.yaml, rolling back means applying model-v12.yaml again. But if you deleted that file, or if your GitOps repo only keeps the latest version, you’re stuck.

I’ve watched an on-call engineer try to roll back a model deployment at 2am, only to discover the previous version’s YAML was gone because someone ran a cleanup script that purged “old configs.” The model artifact still existed in S3, but reconstructing the full InferenceService spec from memory — with the right resource limits, autoscaling config, and transformer sidecar — took 30 minutes. The bad model served traffic the whole time.

The mitigation: treat Kubernetes manifests like database migrations. Every model deployment should be a versioned, immutable file in Git. Use directory structure to encode version history:

models/
  fraud-detector/
    v10/
      inferenceservice.yaml
    v11/
      inferenceservice.yaml
    v12/
      inferenceservice.yaml
    v13/
      inferenceservice.yaml  # current

Your CI/CD applies the latest version. To roll back, you apply v12/inferenceservice.yaml. No guesswork, no lost config.

Another gotcha: Kubernetes doesn’t version your model’s runtime dependencies. If v13 required a new version of scikit-learn and you updated the base container image, rolling back to v12 might still use the new image (because you didn’t pin the tag). The model artifact is correct, but the inference code crashes due to API incompatibility.

# Bad: floating tag
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    sklearn:
      storageUri: s3://k8s-models/fraud-detector/12
      image: myregistry/sklearn-server:latest  # Dangerous

# Good: pinned digest
spec:
  predictor:
    sklearn:
      storageUri: s3://k8s-models/fraud-detector/12
      image: myregistry/sklearn-server@sha256:abc123...  # Immutable

Or better yet, version your container images the same way you version models: sklearn-server:v12 corresponds to the environment that fraud-detector/v12 was trained in.

Pitfall 4: Monitoring Breaks Silently

MLflow has built-in model versioning metadata. You can query the registry API to see which model is in production, when it was promoted, who promoted it, and what metrics it achieved during training. Your monitoring dashboards can fetch this metadata and display it alongside inference logs.

Kubernetes operators don’t have this. An InferenceService knows its storage URI and resource limits, but it doesn’t natively track “this is model version 12, trained on 2026-05-20, with an AUC of 0.87.” That metadata lives elsewhere — maybe in MLflow (if you kept it running), maybe in a separate database, maybe in Git commit messages.

If your monitoring setup assumes MLflow is the source of truth, it breaks the moment you migrate. Your Grafana dashboard shows “Unknown Model Version” because the API endpoint it’s scraping no longer exists. Your alerting rules fire false positives because they can’t correlate inference latency with model metadata.

I’ve seen teams migrate to Kubernetes, watch their inference metrics dashboard go blank, and then spend a week rebuilding it from scratch. The model was serving fine — the observability layer just disappeared.

The fix: inject model metadata into your Kubernetes resources as labels or annotations.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
  labels:
    model-name: fraud-detector
    model-version: "13"
    trained-date: "2026-05-25"
    training-auc: "0.89"
spec:
  predictor:
    sklearn:
      storageUri: s3://k8s-models/fraud-detector/13

Now your monitoring tools can scrape Kubernetes labels instead of hitting the MLflow API. If you’re using Prometheus, you can export these as metrics:

from prometheus_client import Gauge
from kubernetes import client, config

config.load_kube_config()
v1 = client.CustomObjectsApi()

model_version_gauge = Gauge(
    "model_version",
    "Currently deployed model version",
    ["model_name", "namespace"]
)

isvc = v1.get_namespaced_custom_object(
    group="serving.kserve.io",
    version="v1beta1",
    namespace="ml-prod",
    plural="inferenceservices",
    name="fraud-detector"
)

version = isvc["metadata"]["labels"]["model-version"]
model_version_gauge.labels(model_name="fraud-detector", namespace="ml-prod").set(int(version))

This metric can trigger alerts when the version changes unexpectedly, or when a rollback happens.

Another silent failure: if you’re logging predictions to a database with a model_version column, and that column was populated by querying MLflow, you’ll start writing NULL or stale version numbers after migration. Your data science team’s post-deployment analysis will be garbage because they can’t tell which predictions came from which model.

Fix this early: update your inference logging to pull version info from Kubernetes labels, environment variables injected into the pod, or a sidecar that writes metadata to a shared volume.

When Kubernetes Wins Anyway

Despite these pitfalls, I’d still migrate to a Kubernetes operator for most production ML systems. MLflow’s model registry is great for experimentation, but it’s not designed to manage deployments at scale. You can’t do canary rollouts, traffic splitting, or autoscaling without bolting on extra tooling. Kubernetes operators give you that for free (or at least, as free as Kubernetes ever is).

But the migration isn’t a lift-and-shift. You’re changing your deployment model from “update a registry entry” to “reconcile desired state via YAML.” That’s a mental shift, and if your team doesn’t internalize it, they’ll keep treating Kubernetes like a fancy MLflow server — and then wonder why rollbacks are broken, monitoring is offline, and model versions are drifting.

If you’re running fewer than 5 models, or deploying updates less than once a week, stick with MLflow and FastAPI. The Kubernetes complexity isn’t worth it. But if you’re hitting MLflow’s scaling limits — slow registry queries, artifact storage bottlenecks, lack of fine-grained deployment control — then Kubernetes is the right move. Just don’t underestimate the migration effort.

One thing I haven’t figured out yet: how to preserve the full lineage chain when you migrate. MLflow tracks “this model was trained from this dataset, using this code version, and here’s every hyperparameter we tried.” Kubernetes doesn’t care about any of that. You can encode it in labels, but there’s no standard schema. Tools like Kubeflow and MLRun try to bridge this gap, but they introduce their own complexity. I’m still experimenting with embedding lineage metadata as JSON in ConfigMaps and seeing how far that scales.

FAQ

Q: Can I run MLflow and Kubernetes operators in parallel indefinitely?

You can, but it’s painful. Every model update requires syncing two systems, and your team will constantly ask “which one is the source of truth?” Use parallel operation as a migration phase (2-4 weeks max), not a permanent architecture. Pick one system to own the production state and demote the other to read-only experiment tracking.

Q: What happens to model experiment history when I migrate?

MLflow experiment runs (hyperparameters, metrics, logs) stay in MLflow’s backend database. You don’t lose that data by migrating the deployment layer. Keep MLflow running for historical queries and new experiments — just stop using its model registry for production deployments. The tracking server and registry are separate components.

Q: Is KServe better than Seldon Core for this migration?

Both work. KServe has cleaner YAML schemas and better integration with Knative autoscaling, which matters if you have bursty traffic. Seldon Core gives you more control over multi-model pipelines and custom serving runtimes. For a pure MLflow → Kubernetes migration with standard sklearn/PyTorch models, I’d lean KServe. But if you need complex inference graphs (ensemble models, A/B test routing), Seldon Core is more flexible. Try both in a dev cluster with your actual models before committing.

If you’re debugging model deployment issues at 3am and need a focus boost, Dark Chocolate Espresso Beans have saved me more than once.