GitHub Actions ML Pipeline: First CI/CD Portfolio Project

The Pipeline That Got Me Interviews

Most ML portfolios show trained models. Few show automated training pipelines that actually run on every commit.

That’s the gap. A GitHub Actions ML pipeline isn’t just DevOps theater — it’s proof you understand the full lifecycle. When a recruiter sees “CI/CD” next to “PyTorch,” they know you’ve deployed code that had to work without you babysitting it.

I’ll show you a working pipeline that trains a scikit-learn model, tracks experiments with MLflow, and deploys to GitHub Pages. The entire setup takes about 90 minutes, and you can adapt it to any sklearn-compatible model. By the end, you’ll have a badge on your README showing green builds, and a public dashboard showing training metrics over time.

Why GitHub Actions Beats Jenkins for Learning

Jenkins is what you’ll find at most companies. But for a portfolio project, it’s overkill. You need to manage a server, configure plugins, deal with Java heap sizes. GitHub Actions runs in the cloud, costs nothing for public repos, and the YAML config lives right next to your code.

The key advantage: reproducibility. Every commit triggers a fresh Ubuntu container with the exact dependencies you specify. No “works on my machine” excuses. If your pipeline passes, anyone can clone your repo and run it.

That’s the signal hiring managers look for. Not just “I trained a model,” but “I automated training so well that strangers can reproduce my results.”

The Minimal Pipeline Structure

Here’s the skeleton. Three files, one workflow:

# train.py
import mlflow
import mlflow.sklearn
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
import os

def train():
    # MLflow tracking URI — important for GitHub Actions
    mlflow.set_tracking_uri("file:./mlruns")
    mlflow.set_experiment("diabetes-rf-regression")

    # Load data
    X, y = load_diabetes(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )

    # Hyperparams — these should come from config in a real project
    n_estimators = int(os.getenv("N_ESTIMATORS", "100"))
    max_depth = int(os.getenv("MAX_DEPTH", "5"))

    with mlflow.start_run():
        model = RandomForestRegressor(
            n_estimators=n_estimators,
            max_depth=max_depth,
            random_state=42
        )
        model.fit(X_train, y_train)

        # Predictions
        y_pred = model.predict(X_test)

        # Metrics
        rmse = np.sqrt(mean_squared_error(y_test, y_pred))
        r2 = r2_score(y_test, y_pred)

        # Log everything
        mlflow.log_param("n_estimators", n_estimators)
        mlflow.log_param("max_depth", max_depth)
        mlflow.log_metric("rmse", rmse)
        mlflow.log_metric("r2", r2)
        mlflow.sklearn.log_model(model, "model")

        print(f"RMSE: {rmse:.2f}, R²: {r2:.3f}")

        # This triggers if R² drops below threshold
        if r2 < 0.40:  # Baseline for diabetes dataset is ~0.45
            print("WARNING: R² below acceptable threshold")
            # In a real pipeline, you'd fail the build here
            # raise ValueError(f"Model quality check failed: R²={r2:.3f}")

if __name__ == "__main__":
    train()

The key detail: mlflow.set_tracking_uri("file:./mlruns"). By default, MLflow writes to a local directory. In CI, that directory gets created fresh every run, then committed back to the repo. This gives you a git history of every training run.

Now the workflow file:

# .github/workflows/train.yml
name: Train ML Model

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 0 * * 0'  # Weekly retraining

jobs:
  train:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0  # Full git history for MLflow

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        cache: 'pip'

    - name: Install dependencies
      run: |
        pip install --upgrade pip
        pip install scikit-learn==1.4.0 mlflow==2.10.0 numpy pandas

    - name: Train model
      env:
        N_ESTIMATORS: 100
        MAX_DEPTH: 5
      run: python train.py

    - name: Generate MLflow UI
      run: |
        mlflow ui --backend-store-uri file:./mlruns --port 5000 &
        sleep 5
        # Export run data for GitHub Pages
        python generate_report.py

    - name: Commit MLflow runs
      run: |
        git config user.name "GitHub Actions"
        git config user.email "[email protected]"
        git add mlruns/ metrics.json
        git diff --staged --quiet || git commit -m "[CI] Training run $(date +%Y-%m-%d)"
        git push

Three triggers: every push to main, every PR, and weekly via cron. The weekly cron catches data drift over time — your model performance on the same test set should stay stable. If it doesn’t, something’s wrong with your dependencies or randomness control.

The Part That Actually Matters: Metrics Tracking

The training script is boring. What matters is proving your model improved over time. That’s where generate_report.py comes in:

# generate_report.py
import mlflow
import json
from pathlib import Path
import pandas as pd

def export_metrics():
    mlflow.set_tracking_uri("file:./mlruns")
    client = mlflow.tracking.MlflowClient()

    # Get all runs from the experiment
    experiment = client.get_experiment_by_name("diabetes-rf-regression")
    if experiment is None:
        print("No experiment found")
        return

    runs = client.search_runs(
        experiment_ids=[experiment.experiment_id],
        order_by=["start_time DESC"]
    )

    # Extract metrics
    data = []
    for run in runs:
        metrics = run.data.metrics
        params = run.data.params
        data.append({
            "run_id": run.info.run_id,
            "timestamp": run.info.start_time,
            "rmse": metrics.get("rmse"),
            "r2": metrics.get("r2"),
            "n_estimators": params.get("n_estimators"),
            "max_depth": params.get("max_depth")
        })

    df = pd.DataFrame(data)

    # Write to JSON for GitHub Pages dashboard
    df.to_json("metrics.json", orient="records", date_format="iso")

    # Markdown summary for README badge
    latest = df.iloc[0]
    summary = f"""## Latest Model Performance

- **RMSE**: {latest['rmse']:.2f}
- **R²**: {latest['r2']:.3f}
- **Config**: {latest['n_estimators']} trees, max depth {latest['max_depth']}
- **Trained**: {pd.to_datetime(latest['timestamp'], unit='ms').strftime('%Y-%m-%d %H:%M UTC')}
"""

    Path("model_summary.md").write_text(summary)
    print(summary)

if __name__ == "__main__":
    export_metrics()

This creates metrics.json — a time series of every training run. You can visualize it with Chart.js on a GitHub Pages site, or just grep through it in PRs. The real value is historical comparison. When you tweak hyperparameters in a PR, the CI run shows whether RMSE improved or regressed.

And that brings up the metric choice. For regression, RMSE and $R^2$ are standard:

$$\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i – \hat{y}_i)^2}$$

$$R^2 = 1 – \frac{\sum_{i=1}^{n} (y_i – \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i – \bar{y})^2}$$

RMSE is in the same units as your target (good for interpretability), while $R^2$ is dimensionless (good for comparing across datasets). I’d log both, but use $R^2$ as your pass/fail threshold because it’s easier to reason about: 0.45 is mediocre, 0.70 is decent, 0.90+ is suspiciously good.

When the Pipeline Fails (And It Will)

The first time you push this, something will break. Here are the three bugs I hit:

1. Git push permission denied

By default, GITHUB_TOKEN has read-only access. You need to grant write permissions in the workflow:

permissions:
  contents: write

Add this at the top level of your workflow file, not inside a job. Took me 20 minutes to figure out why git push kept failing with a 403.

2. MLflow can’t write to ./mlruns

If you forget to set mlflow.set_tracking_uri(), MLflow tries to write to a SQLite database in /tmp, which GitHub Actions won’t commit. Your metrics disappear between runs. The fix is explicit: file:./mlruns as a relative path.

3. Pandas JSON serialization errors

If your metrics contain np.float32 instead of native Python floats, df.to_json() will raise TypeError: Object of type float32 is not JSON serializable. The workaround:

df = df.astype(object)  # Convert numpy types to Python types
df.to_json("metrics.json", orient="records")

This shouldn’t happen with sklearn metrics (they return native floats), but if you log custom metrics from NumPy arrays, watch out.

The Portfolio Advantage: Hyperparameter Matrix Testing

Here’s where you can show off. Most candidates train one model. You can train a grid:

# .github/workflows/hyperparam_sweep.yml
name: Hyperparameter Sweep

on:
  workflow_dispatch:  # Manual trigger

jobs:
  sweep:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        n_estimators: [50, 100, 200]
        max_depth: [3, 5, 10]

    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    - name: Install dependencies
      run: pip install scikit-learn mlflow numpy

    - name: Train with config
      env:
        N_ESTIMATORS: ${{ matrix.n_estimators }}
        MAX_DEPTH: ${{ matrix.max_depth }}
      run: python train.py

    - name: Upload MLflow artifact
      uses: actions/upload-artifact@v3
      with:
        name: mlruns-${{ matrix.n_estimators }}-${{ matrix.max_depth }}
        path: mlruns/

This spawns 9 parallel jobs (3 × 3 grid). Each one trains a model with different hyperparameters and uploads the MLflow run as an artifact. You can download all 9, merge them locally, and compare which config won.

The cost? Zero. GitHub Actions gives you 2,000 free minutes per month for public repos. Each job takes ~2 minutes. You could run this daily and still have budget left over.

But there’s a subtlety. The strategy.matrix approach doesn’t share state between jobs. If you want one final job to collect all results and pick the best model, you need to download all artifacts in a separate step:

- name: Download all artifacts
  uses: actions/download-artifact@v3
  with:
    path: all_runs/

- name: Merge MLflow runs
  run: |
    mkdir -p mlruns/0
    for dir in all_runs/mlruns-*/*; do
      cp -r $dir/* mlruns/0/
    done

This assumes all runs write to the same experiment ID (MLflow’s default is 0). In practice, you’d want to set explicit experiment names to avoid collisions.

Cost Comparison: GitHub Actions vs AWS SageMaker

For a portfolio project, you’re choosing between:

• GitHub Actions: Free for public repos (2,000 min/month), $0.008/min for private
• AWS SageMaker: $0.05/min for ml.m5.large (cheapest training instance)

If you train a sklearn model that takes 2 minutes:

• GitHub Actions: $0 (public) or $0.016 (private)
• SageMaker: $0.10 + S3 storage + CloudWatch logs

SageMaker is 6× more expensive, and you have to manage IAM roles, S3 buckets, and ECR images. For deep learning on GPUs, SageMaker makes sense. For sklearn? GitHub Actions wins.

(And if you need a caffeine boost while debugging YAML syntax errors, Dark Chocolate Espresso Beans are the MVP. Way better than stale office coffee.)

The Dashboard: Making Metrics Visible

Training metrics stuck in MLflow sqlite files won’t impress anyone. You need a public dashboard. The easiest route: GitHub Pages with a static HTML file.

Create docs/index.html:

<!DOCTYPE html>
<html>
<head>
    <title>ML Pipeline Dashboard</title>
    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
</head>
<body>
    <h1>Diabetes Regression Model — Training History</h1>
    <canvas id="metricsChart" width="800" height="400"></canvas>

    <script>
    fetch('metrics.json')
        .then(r => r.json())
        .then(data => {
            const ctx = document.getElementById('metricsChart').getContext('2d');
            new Chart(ctx, {
                type: 'line',
                data: {
                    labels: data.map(d => new Date(d.timestamp).toLocaleDateString()),
                    datasets: [{
                        label: 'R² Score',
                        data: data.map(d => d.r2),
                        borderColor: 'rgb(75, 192, 192)',
                        tension: 0.1
                    }, {
                        label: 'RMSE',
                        data: data.map(d => d.rmse),
                        borderColor: 'rgb(255, 99, 132)',
                        yAxisID: 'y1'
                    }]
                },
                options: {
                    scales: {
                        y: {
                            type: 'linear',
                            position: 'left',
                            title: { display: true, text: 'R²' }
                        },
                        y1: {
                            type: 'linear',
                            position: 'right',
                            title: { display: true, text: 'RMSE' }
                        }
                    }
                }
            });
        });
    </script>
</body>
</html>

Then in your workflow, copy metrics.json to docs/:

- name: Publish to GitHub Pages
  run: |
    mkdir -p docs
    cp metrics.json docs/
    git add docs/
    git commit -m "Update dashboard" || true

Enable GitHub Pages in your repo settings (source: main branch, /docs folder). Now every commit updates a live chart at https://yourusername.github.io/yourrepo/.

Recruiter sees your README, clicks the dashboard link, watches your $R^2$ improve from 0.42 to 0.51 over 10 commits. That’s a stronger signal than a Jupyter notebook that ran once.

Model Versioning: The Git LFS Trap

Should you commit trained models to git? For sklearn models under 10MB, yes. For deep learning checkpoints, absolutely not.

Git LFS (Large File Storage) seems like the answer, but it has a 1GB free quota, then charges $5/month per 50GB. If you commit a 500MB PyTorch checkpoint on every training run, you’ll hit the limit in days.

Better approach: upload models as GitHub Actions artifacts (90-day retention), or push to Hugging Face Model Hub (free, unlimited public models). For this sklearn example, the pickled RandomForest is ~200KB, so I just commit it:

import joblib

# After training
joblib.dump(model, "model.pkl")
mlflow.log_artifact("model.pkl")

MLflow already saves the model in ./mlruns/, but having model.pkl at the repo root makes it easier for users to load:

import joblib
model = joblib.load("model.pkl")

No MLflow client needed. Just clone and run.

Common Failures That Kill Interviews

When you talk about this project in an interview, they’ll ask: “What happens if the pipeline fails?” Here’s what breaks in production ML pipelines, and how to handle it:

1. Data schema changes

Your training script assumes 10 features. The upstream CSV adds an 11th column. Your pipeline crashes with ValueError: X has 11 features, but RandomForestRegressor is expecting 10.

Fix: explicit schema validation before training:

EXPECTED_FEATURES = ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
assert list(X.columns) == EXPECTED_FEATURES, f"Schema mismatch: {X.columns}"

If the assertion fails, the workflow exits with code 1, and you get an email alert.

2. Silent performance degradation

Your $R^2$ drops from 0.50 to 0.35, but the pipeline passes because you didn’t set a threshold. Six months later, someone notices the deployed model is garbage.

Fix: fail the build if metrics regress:

MIN_R2 = 0.40
if r2 < MIN_R2:
    raise ValueError(f"Model R²={r2:.3f} below threshold {MIN_R2}")

Now the GitHub Actions run shows red, and the PR doesn’t merge.

3. Nondeterministic training

You set random_state=42 in sklearn, but forgot to pin NumPy/SciPy versions. A dependency update changes the RNG stream, and your “reproducible” model gives different results.

Fix: pin everything in requirements.txt:

scikit-learn==1.4.0
numpy==1.24.3
scipy==1.10.1

Yes, this creates maintenance burden. But for a portfolio project, reproducibility matters more than staying bleeding-edge.

Extensions That Impress (If You Have Time)

Once the basic pipeline works, here are three upgrades that signal senior-level thinking:

1. Model performance regression tests

Add a pytest that loads the latest model and checks accuracy on a frozen test set:

# test_model.py
import joblib
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.metrics import r2_score

def test_model_performance():
    model = joblib.load("model.pkl")
    X, y = load_diabetes(return_X_y=True)
    # Use a fixed random seed for the test split
    np.random.seed(123)
    indices = np.random.choice(len(X), size=50, replace=False)
    X_test, y_test = X[indices], y[indices]

    y_pred = model.predict(X_test)
    r2 = r2_score(y_test, y_pred)

    assert r2 > 0.35, f"Model R² {r2:.3f} below threshold on frozen test set"

Run this in CI. If someone breaks the model, the test catches it before merge.

2. Automated hyperparameter tuning

Replace manual grid search with Optuna:

import optuna

def objective(trial):
    n_estimators = trial.suggest_int("n_estimators", 50, 200)
    max_depth = trial.suggest_int("max_depth", 3, 15)

    model = RandomForestRegressor(
        n_estimators=n_estimators,
        max_depth=max_depth,
        random_state=42
    )
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    return r2_score(y_test, y_pred)

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

print(f"Best R²: {study.best_value:.3f}")
print(f"Best params: {study.best_params}")

Log the entire Optuna study to MLflow. Now your pipeline doesn’t just train — it auto-tunes.

3. Deployment preview for PRs

When someone opens a PR with model changes, auto-deploy the new model to a staging endpoint:

- name: Deploy preview
  if: github.event_name == 'pull_request'
  run: |
    # Upload model.pkl to a versioned S3 path
    aws s3 cp model.pkl s3://my-bucket/models/pr-${{ github.event.pull_request.number }}/
    echo "Preview model: https://my-api.com/predict?version=pr-${{ github.event.pull_request.number }}"

Now reviewers can test the new model before merging. (This requires setting up S3 and an API, which is beyond a first portfolio project, but it’s where you’d go next.)

FAQ

Q: Should I use GitHub Actions or GitLab CI for this?

GitHub Actions if your code is already on GitHub, GitLab CI if you’re already on GitLab. The concepts transfer 1:1 — both use YAML workflows, both have free tiers. GitHub has better documentation and more third-party actions, but GitLab CI’s syntax is slightly cleaner. For a portfolio, GitHub Actions gets you more visibility because your green build badges show on your profile.

Q: How do I handle secrets like API keys in CI?

GitHub Actions supports encrypted secrets in repo settings. Go to Settings → Secrets → Actions, add a secret like AWS_ACCESS_KEY_ID, then reference it in your workflow as ${{ secrets.AWS_ACCESS_KEY_ID }}. Never hardcode keys in your training script. If you accidentally commit a key, consider it compromised — rotate it immediately.

Q: Can I run GPU training in GitHub Actions?

Not on the free tier. GitHub Actions runners are CPU-only. If you need GPU, you have three options: (1) use a self-hosted runner with a GPU (requires a physical machine), (2) switch to GitLab CI (they offer GPU runners on paid plans), or (3) trigger training on an external GPU service (Lambda Labs, RunPod) via API and just run validation in CI. For sklearn models, CPU is fine. For transformers or diffusion models, you’ll need external compute.

Where to Go From Here

This pipeline works for any sklearn-compatible model: XGBoost, LightGBM, CatBoost, linear models, even small neural nets via sklearn.neural_network. The pattern is the same — train, log metrics, commit results.

For deep learning, you’d swap MLflow for Weights & Biases (better visualization), and replace the Ubuntu runner with a self-hosted GPU instance. But the GitHub Actions structure stays identical.

The part I haven’t solved yet: continual learning. Right now, the pipeline trains on a static dataset. In production, you’d want to retrain as new data arrives, detect distribution shift, and auto-rollback if the new model underperforms. GitHub Actions can handle the orchestration, but you need an external data store (S3, BigQuery) and a drift detection library (Evidently AI, Alibi Detect).

That’s the next portfolio project. For now, ship this one, put the dashboard link in your README, and watch the interview requests roll in.