MLflow Basics: 5 Interview Questions Every Beginner Should Know

MLflow Basics: 5 Interview Questions Every Beginner Should Know

You can run experiments for weeks, train dozens of models, and still blank when an interviewer asks, “How would you compare two runs in MLflow?”

MLflow is everywhere in MLOps roles now — job descriptions list it alongside Docker and Kubernetes — but most tutorials skip the stuff interviewers actually ask. They show you how to log a metric, then jump straight to production deployment. The gap between “hello world” and “explain model registry versioning” is wide, and that’s where interview prep falls apart.

This isn’t a comprehensive guide to MLflow’s API. It’s the five questions I’ve seen come up in screening calls, technical rounds, and take-home assignments for junior-to-mid MLOps and ML engineer roles. Each one tests whether you’ve actually used MLflow beyond copy-pasting mlflow.log_param() from a tutorial.

Question 1: Explain the Difference Between Parameters, Metrics, and Artifacts

Interviewers love this one because it’s basic, but candidates mess it up constantly. The confusion usually starts with “isn’t a metric just a special kind of parameter?”

Parameters are inputs to your model: learning rate, batch size, number of layers, regularization strength. These are things you set before training starts. In MLflow, you log them with mlflow.log_param("learning_rate", 0.001).

Metrics are outputs: accuracy, loss, F1 score, RMSE. Things your model produces during or after training. You log them with mlflow.log_metric("accuracy", 0.92). The key difference: metrics can be logged multiple times per run (e.g., loss at each epoch), while parameters are typically logged once.

Artifacts are files: the trained model itself, confusion matrices, plots, feature importance tables, preprocessor pipelines. Anything that isn’t a single number. You log them with mlflow.log_artifact("confusion_matrix.png") or mlflow.sklearn.log_model(model, "model").

Here’s what it looks like in practice:

import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

mlflow.set_experiment("interview-prep-demo")

with mlflow.start_run():
    # Parameters: inputs you control
    n_estimators = 100
    max_depth = 10
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)

    # Train model (simplified)
    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth, random_state=42)
    model.fit(X_train, y_train)

    # Metrics: outputs from evaluation
    preds = model.predict(X_test)
    acc = accuracy_score(y_test, preds)
    mlflow.log_metric("test_accuracy", acc)

    # Artifacts: files you want to save
    plt.figure(figsize=(8, 6))
    # ... plot confusion matrix ...
    plt.savefig("confusion_matrix.png")
    mlflow.log_artifact("confusion_matrix.png")
    mlflow.sklearn.log_model(model, "random_forest_model")

The interviewer might follow up with: “What if you want to log validation loss at every epoch?” Use mlflow.log_metric("val_loss", loss, step=epoch). The step parameter lets you log the same metric multiple times and see the progression in the MLflow UI.

One edge case that trips people up: large numpy arrays. Don’t log them as metrics (they’re not scalar). Save them as .npy files and log as artifacts. MLflow will complain if you try to log a list or array as a metric.

Question 2: How Do You Compare Two Runs in MLflow?

This tests whether you’ve actually used the MLflow UI or API to make decisions. The answer has two parts: UI and programmatic access.

Via the UI: Open the MLflow tracking server (usually mlflow ui on localhost:5000), go to your experiment, check the boxes next to the runs you want to compare, and click “Compare”. You’ll see a side-by-side table of parameters and metrics, plus overlaid plots for any metrics logged with step (like training loss curves).

But the real answer interviewers want is the programmatic approach, because production pipelines don’t use the UI.

import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient()
experiment = client.get_experiment_by_name("interview-prep-demo")
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.test_accuracy DESC"],
    max_results=5
)

for run in runs:
    print(f"Run ID: {run.info.run_id}")
    print(f"Accuracy: {run.data.metrics['test_accuracy']:.4f}")
    print(f"Params: {run.data.params}")
    print("-" * 40)

This pulls the top 5 runs by accuracy. The search_runs API is powerful — you can filter by parameter values, metric thresholds, or tags. For example, filter_string="params.max_depth = '10'" would only return runs where max_depth was 10.

A follow-up I’ve gotten twice: “How would you automatically select the best model from 100 hyperparameter search runs?” Sort by your target metric (order_by=["metrics.test_f1 DESC"]), grab the top run, and load its model with mlflow.sklearn.load_model(f"runs:/{run_id}/model").

One gotcha: run.data.metrics only gives you the final value of each metric. If you logged val_loss at every epoch and want the full history, you need client.get_metric_history(run_id, "val_loss"). Returns a list of Metric objects with step and value fields.

Question 3: What’s the Difference Between mlflow.log_model() and Model Registry?

This one separates people who’ve only done local experiments from those who’ve thought about model versioning in production.

mlflow.log_model() saves a model as an artifact within a specific run. It’s tied to that run’s lifecycle. If you delete the run, the model goes with it. You load it with the run ID: mlflow.sklearn.load_model(f"runs:/{run_id}/model").

The Model Registry is a separate layer on top. It lets you register models by name (e.g., “customer_churn_classifier”), create versions (v1, v2, v3), and assign stages (None, Staging, Production, Archived). A single registry entry can point to models from different runs, different experiments, even different flavors (sklearn, PyTorch, TensorFlow).

Here’s the workflow:

import mlflow
from mlflow.tracking import MlflowClient

# Step 1: Train and log a model
with mlflow.start_run() as run:
    model = train_some_model()  # your training code
    mlflow.sklearn.log_model(model, "model")
    run_id = run.info.run_id

# Step 2: Register the model
client = MlflowClient()
model_name = "churn_predictor"
model_uri = f"runs:/{run_id}/model"
registered_model = client.create_registered_model(model_name)  # only needed once
model_version = client.create_model_version(
    name=model_name,
    source=model_uri,
    run_id=run_id
)

print(f"Registered model '{model_name}' version {model_version.version}")

# Step 3: Transition to Production
client.transition_model_version_stage(
    name=model_name,
    version=model_version.version,
    stage="Production"
)

Now your inference code can load the model by name and stage:

model = mlflow.pyfunc.load_model(f"models:/{model_name}/Production")
predictions = model.predict(new_data)

No hardcoded run IDs. When you deploy a new model, you just transition it to Production, and all downstream services automatically pick it up. Old versions stay in the registry — you can roll back by transitioning an earlier version to Production.

The interviewer might ask: “What if you want to A/B test two models?” Put one in Production, one in Staging, and route 90% of traffic to Production, 10% to Staging in your serving layer (e.g., FastAPI with conditional logic). MLflow doesn’t handle traffic splitting — that’s your deployment infrastructure’s job.

One thing I’m still not sure about: whether the Model Registry is the right abstraction for teams with <5 models. The overhead of versioning and stages feels heavy when you’re just iterating on a single model. But interviewers ask about it constantly, so it’s clearly important at scale.

Question 4: How Do You Track Experiments Across Multiple Scripts or Machines?

This tests whether you understand MLflow’s client-server architecture. Most beginners start with mlflow ui pointing to a local mlruns/ directory, which breaks as soon as you run experiments on a remote server or split work across a team.

MLflow needs a tracking server and a backend store (database) + artifact store (file system or S3/GCS/Azure Blob).

Local setup (single machine, multiple scripts):

# Terminal 1: Start tracking server with a database backend
mlflow server \
    --backend-store-uri sqlite:///mlflow.db \
    --default-artifact-root ./mlruns \
    --host 0.0.0.0 \
    --port 5000

Now in your Python scripts, point to the server:

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my_experiment")

with mlflow.start_run():
    # ... your training code ...
    mlflow.log_metric("accuracy", 0.89)

All scripts share the same tracking server. Experiments, runs, metrics, and artifacts are centralized in sqlite:///mlflow.db and ./mlruns.

Remote setup (multiple machines):

Replace SQLite with PostgreSQL or MySQL, and ./mlruns with S3:

mlflow server \
    --backend-store-uri postgresql://user:password@db-host:5432/mlflow \
    --default-artifact-root s3://my-bucket/mlflow-artifacts \
    --host 0.0.0.0 \
    --port 5000

Your training scripts on different machines all set mlflow.set_tracking_uri("http://tracking-server-ip:5000"). Now experiments run on your laptop, a cloud VM, and a Kubernetes pod all log to the same central server.

A gotcha: if you’re using S3, your training scripts need AWS credentials (via environment variables or IAM roles). MLflow doesn’t manage authentication — it assumes boto3 is configured.

Another common question: “What if two runs log to the same experiment simultaneously?” No problem. MLflow assigns unique run IDs and handles concurrent writes. The backend database prevents collisions.

Question 5: How Do You Load a Logged Model for Inference?

This sounds trivial, but there are three different ways, and interviewers want to see you pick the right one for the situation.

Option 1: Load by run ID (for ad-hoc inference or debugging)

import mlflow

run_id = "a1b2c3d4e5f6"  # from MLflow UI or search_runs()
model = mlflow.sklearn.load_model(f"runs:/{run_id}/model")
predictions = model.predict(X_test)

This works for any model flavor (sklearn, PyTorch, TensorFlow, custom pyfunc). But hardcoding run IDs is fragile — if you retrain the model, you have to update every inference script.

Option 2: Load by model registry name and stage (for production serving)

model_name = "churn_predictor"
stage = "Production"  # or "Staging", "None", "Archived"
model = mlflow.pyfunc.load_model(f"models:/{model_name}/{stage}")
predictions = model.predict(new_data)

This decouples inference from specific runs. When you promote a new model to Production in the registry, inference code automatically uses it. No code changes.

Option 3: Load by version number (for reproducibility or rollback)

model_name = "churn_predictor"
version = 3
model = mlflow.pyfunc.load_model(f"models:/{model_name}/{version}")
predictions = model.predict(X_test)

Useful for A/B testing (load v3 and v4 side-by-side) or debugging (“did v5 introduce a regression?”).

The key difference: mlflow.sklearn.load_model() returns the native sklearn object. mlflow.pyfunc.load_model() returns a generic MLflow wrapper with a .predict() method that works across all flavors. For sklearn and most frameworks, they behave the same. But if you need flavor-specific methods (e.g., PyTorch’s .eval() or .state_dict()), use the flavor-specific loader.

One edge case that bit me: if you log a model with custom preprocessing (e.g., a scikit-learn Pipeline), make sure all dependencies are in your inference environment. MLflow saves the model and its direct dependencies, but if your pipeline calls a custom function from utils.py, you need to install that too. MLflow doesn’t package arbitrary Python files — only the model object.

What Interviewers Are Really Testing

These five questions aren’t about memorizing API calls. They’re checking:

1. Do you understand the difference between experiment tracking and model versioning? (Questions 1, 3)
2. Can you automate model selection and deployment? (Questions 2, 5)
3. Do you know how to set up MLflow for team collaboration? (Question 4)

If you can answer these with working code — not just definitions — you’re ahead of most candidates who’ve only done Kaggle notebooks with mlflow.autolog().

One thing I’d do differently: actually run a full hyperparameter search (say, 50 runs) and use search_runs() to find the best model, then register it and serve it via FastAPI Model Serving: 5 Steps to 50ms Inference. That end-to-end flow — experiment → register → deploy → serve — is the portfolio project that gets interviews in the first place.

The next skill to add after MLflow basics? Probably Docker vs Kubernetes for First ML Model: When to Use Each, because every MLOps role will ask about containerization. But MLflow is the foundation — you can’t version and deploy models if you don’t track them properly.

If you’re prepping for an MLOps interview and find yourself stuck on deployment patterns, keep a copy of Designing Machine Learning Systems by Chip Huyen nearby — grab it here. It covers the context around MLflow (experiment tracking, model registries, monitoring) better than any docs page.

FAQ

Q: Can I use MLflow without a tracking server for small projects?

Yes. By default, MLflow logs to a local mlruns/ directory. Run mlflow ui in your project folder to view experiments. This works fine for solo projects or early prototyping. You only need a tracking server when multiple people or machines need to share experiments.

Q: What’s the difference between mlflow.autolog() and manual logging?

mlflow.autolog() automatically logs parameters, metrics, and the model for supported frameworks (sklearn, TensorFlow, PyTorch, XGBoost, etc.). It’s great for quick experiments, but you lose control over what gets logged. For production pipelines, manual logging with log_param(), log_metric(), and log_model() gives you fine-grained control and avoids clutter (e.g., autolog captures every hyperparameter, even defaults you don’t care about).

Q: How do I delete old experiments or runs to save space?

Use mlflow.delete_experiment(experiment_id) or mlflow.delete_run(run_id). Note: deleted experiments are marked as “deleted” but not removed from the database immediately — use mlflow gc (garbage collection) to purge them. For artifact storage (e.g., S3), you need to manually delete the corresponding directories — MLflow doesn’t clean up artifacts automatically.