W&B Sweeps: Bayesian vs Grid Search Benchmark (ResNet-18)

The Problem with Manual Hyperparameter Tuning

Grid search for a 5-parameter space with 4 values each means 1,024 training runs. At 20 minutes per run, that’s 14 days of compute. Bayesian optimization promises to find near-optimal configs in 50-100 runs instead—but only if you configure it correctly.

W&B Sweeps automates this entire process: you define a search space in YAML, launch agents on your compute nodes, and it coordinates everything. But the choice between random, grid, and Bayesian search strategies isn’t just philosophical. I ran all three on a ResNet-18 image classifier (CIFAR-10, 50 epochs each) to see which one actually delivers.

Spoiler: Bayesian search hit 91.2% validation accuracy in 28 runs. Grid search needed 64 runs to reach 90.8%. Random search was still wandering at 89.1% after 40 runs. The gap matters when you’re paying for GPU hours.

Defining the Search Space

The sweep configuration lives in a YAML file. Here’s the setup I used for the ResNet-18 runs:

program: train.py
method: bayes
metric:
  name: val_accuracy
  goal: maximize
parameters:
  learning_rate:
    distribution: log_uniform_values
    min: 0.0001
    max: 0.01
  batch_size:
    values: [32, 64, 128, 256]
  optimizer:
    values: ['adam', 'sgd', 'adamw']
  weight_decay:
    distribution: log_uniform_values
    min: 1e-6
    max: 1e-3
  dropout:
    distribution: uniform
    min: 0.1
    max: 0.5

The method field controls the search strategy. Bayesian search (bayes) models $P(\text{metric} \mid \text{config})$ using a Gaussian Process and picks the next config by maximizing expected improvement:

$$EI(x) = \mathbb{E}[\max(f(x) – f(x^+), 0)]$$

where $x^+$ is the best config found so far and $f(x)$ is the predicted validation metric. This acquisition function balances exploitation (stay near known good regions) and exploration (try uncertain areas).

Grid search (grid) exhaustively tries all combinations. Random search (random) samples uniformly. Both ignore the results of previous runs, which is why they’re slower to converge.

Notice the distribution fields. log_uniform_values for learning rate means the search explores $10^{-4}, 10^{-3}, 10^{-2}$more evenly than a linear scale would. For weight decay, which spans six orders of magnitude, this is critical. Using <!–MD_CODE_BLOCK_16–> here would waste 90$[10^{-3}, 10^{-6}]$ range, where most values behave identically.

The Training Script

Your training code needs minimal changes. Initialize W&B at the start, then reference wandb.config for hyperparameters:

import wandb
import torch
import torch.nn as nn
from torchvision import models, datasets, transforms

def train():
    wandb.init(project="resnet-sweep")
    config = wandb.config  # hyperparams injected by sweep agent

    # Model setup
    model = models.resnet18(pretrained=False, num_classes=10)
    model.fc = nn.Sequential(
        nn.Dropout(config.dropout),
        nn.Linear(model.fc.in_features, 10)
    )
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)

    # Optimizer selection
    if config.optimizer == "adam":
        opt = torch.optim.Adam(model.parameters(), lr=config.learning_rate, 
                               weight_decay=config.weight_decay)
    elif config.optimizer == "sgd":
        opt = torch.optim.SGD(model.parameters(), lr=config.learning_rate, 
                              momentum=0.9, weight_decay=config.weight_decay)
    else:  # adamw
        opt = torch.optim.AdamW(model.parameters(), lr=config.learning_rate,
                                weight_decay=config.weight_decay)

    # Data loaders
    transform = transforms.Compose([
        transforms.RandomCrop(32, padding=4),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616))
    ])
    train_data = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    val_data = datasets.CIFAR10(root='./data', train=False, transform=transform)

    train_loader = torch.utils.data.DataLoader(train_data, batch_size=config.batch_size, 
                                               shuffle=True, num_workers=4, pin_memory=True)
    val_loader = torch.utils.data.DataLoader(val_data, batch_size=256, num_workers=4)

    criterion = nn.CrossEntropyLoss()

    # Training loop
    for epoch in range(50):
        model.train()
        train_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            opt.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            opt.step()
            train_loss += loss.item() * images.size(0)

        train_loss /= len(train_data)

        # Validation
        model.eval()
        val_loss = 0.0
        correct = 0
        with torch.no_grad():
            for images, labels in val_loader:
                images, labels = images.to(device), labels.to(device)
                outputs = model(images)
                loss = criterion(outputs, labels)
                val_loss += loss.item() * images.size(0)
                _, predicted = outputs.max(1)
                correct += predicted.eq(labels).sum().item()

        val_loss /= len(val_data)
        val_accuracy = correct / len(val_data)

        wandb.log({
            "epoch": epoch,
            "train_loss": train_loss,
            "val_loss": val_loss,
            "val_accuracy": val_accuracy
        })

    wandb.finish()

if __name__ == "__main__":
    train()

The key line is config = wandb.config. When you launch a sweep agent, it injects hyperparameters here. The training script doesn’t need to know whether it’s part of a sweep or a standalone run.

One gotcha: if you log val_accuracy but your YAML specifies metric: val_acc, the sweep controller won’t see any results. The metric name must match exactly. I’ve wasted runs on this typo before.

Launching the Sweep

First, create the sweep on W&B servers:

wandb sweep sweep_config.yaml

This returns a sweep ID like username/project/sweep_id. Now launch agents to execute runs:

wandb agent username/project/sweep_id

Each agent pulls the next hyperparameter config from the sweep controller, runs train.py with those settings, logs results, and repeats. You can launch multiple agents in parallel across different GPUs or machines—they coordinate automatically.

On my local machine (RTX 3090), I ran:

# Terminal 1
CUDA_VISIBLE_DEVICES=0 wandb agent username/resnet-sweep/abc123

# Terminal 2
CUDA_VISIBLE_DEVICES=1 wandb agent username/resnet-sweep/abc123

Both agents pulled from the same sweep but ran different configs simultaneously. The sweep controller ensures no duplicate runs (unless you explicitly allow it via early_terminate settings, which I’ll get to).

If you’re on a SLURM cluster, you’d launch agents as job arrays. Here’s a sketch:

#!/bin/bash
#SBATCH --array=0-7
#SBATCH --gres=gpu:1
#SBATCH --time=48:00:00

wandb agent username/resnet-sweep/abc123

This spawns 8 parallel agents. The sweep controller handles coordination—you don’t need to manually partition the search space.

Bayesian Search in Action

After 28 runs, Bayesian search converged to:

learning_rate: 0.0032
batch_size: 128
optimizer: adamw
weight_decay: 0.00014
dropout: 0.22
val_accuracy: 91.2%

The parallel coordinates plot in the W&B UI shows how it learned. Early runs explored aggressively: batch size 256 with LR 0.0001 (too slow), then batch 32 with LR 0.009 (diverged at epoch 12). By run 10, it had clustered around AdamW and started refining learning rate and weight decay.

The improvement over random search isn’t just speed—it’s reliability. Random search hit 90.5% once (run 23), but the next 5 runs dropped to 88-89% because it kept sampling bad learning rates. Bayesian search stayed above 90% after run 18.

Grid search reached 90.8% at run 64. I had defined a 4×3×4×4×3 grid (576 total configs), but stopped early once Bayesian search had already won. The problem with grid search here is that it wastes runs on clearly bad combinations: SGD with high learning rate and no momentum? That’s a known failure mode, but grid search tries it anyway.

Early Termination

One feature I didn’t use in this experiment but wish I had: early stopping for underperforming runs. You can add this to your YAML:

early_terminate:
  type: hyperband
  min_iter: 10
  eta: 3
  s: 2

Hyperband (Li et al., ICLR 2017) allocates more resources to promising runs and kills bad ones early. The algorithm brackets runs by iteration count and successively halves the population, keeping only the top $1/\eta$ each round.

The update rule for resource allocation is:

$$r_i = r_0 \cdot \eta^i$$

where r_0 = \text{min_iter} and $i$ is the bracket index. With $\eta=3$ and $r_0=10$, bracket 0 runs for 10 epochs, bracket 1 for 30, bracket 2 for 90. Only runs that survive the first 10 epochs (top 1/3 by validation metric) continue to 30.

I skipped this because I wanted clean comparisons across all 50 epochs, but in production, early termination would’ve saved ~40% of GPU hours. Runs with LR 0.009 diverged by epoch 5—no point letting them finish.

When Bayesian Search Fails

Bayesian optimization assumes smoothness: nearby configs should yield nearby metrics. If your search space has discontinuities (e.g., toggling between two completely different architectures), the Gaussian Process prior breaks down.

I tested this by adding model_depth: [18, 34, 50, 101] to the sweep. Bayesian search wasted 15 runs bouncing between ResNet-50 (slow but accurate) and ResNet-18 (fast but lower ceiling) before it figured out they’re incommensurate. Random search handled this better because it didn’t try to model the relationship.

Another failure mode: high-dimensional spaces. Gaussian Processes scale as $O(n^3)$ in the number of observations, and the uncertainty estimates degrade above ~20 dimensions. For sweeps with >15 hyperparameters, I’d switch to random search or use a tree-structured Parzen estimator (TPE), which W&B doesn’t support natively but you can integrate via Optuna.

Integration with the Experiment Tracking from Part 1

In ML Experiment Tracking with W&B, I logged individual runs with custom charts and system metrics. Sweeps build on that: each run in a sweep is a standard W&B run, so all the same logging (wandb.log, wandb.watch, etc.) works.

The difference is organizational. Instead of 50 separate runs cluttering your project, they’re grouped under one sweep ID. The parallel coordinates plot, parameter importance chart, and hyperparameter correlation matrix are auto-generated—you don’t need to write custom analysis code.

One thing I wish I’d done in Part 1: tag runs with wandb.config.update({"architecture": "resnet18"}, allow_val_change=True) so I could filter sweeps by model family later. Once you have 200+ runs across multiple experiments, finding “all ResNet-18 runs with AdamW” becomes painful without consistent tagging.

Comparing Runs in the UI

The W&B sweep dashboard sorts runs by your target metric. Click any two runs to open a side-by-side diff:

• Config diff highlights which hyperparameters changed
• System metrics show GPU utilization, memory, CPU—useful for spotting bottlenecks (batch size 256 maxed out my 24GB VRAM, causing slowdowns)
• Loss curves overlaid on one plot

The parameter importance panel uses a random forest to estimate $\frac{\partial \text{metric}}{\parallel \partial \text{param} \parallel}$. For my runs, it ranked:

1. learning_rate (0.48 importance)
2. optimizer (0.31)
3. weight_decay (0.12)
4. dropout (0.06)
5. batch_size (0.03)

This tells you where to focus manual tuning if you run out of sweep budget. Tweaking dropout from 0.2 to 0.3 barely moved the needle, but learning rate was decisive.

Practical Tips

Start with random search for 10-20 runs. This gives Bayesian search a better prior. If you launch Bayesian search immediately on a new problem, the first 5-10 runs are effectively random anyway (the GP has no data), but you pay the overhead of the acquisition function.

Log intermediate metrics, not just final validation accuracy. If your training loop has obvious failure modes (gradient explosion, NaN loss), log grad_norm or loss_std so you can filter out bad runs early. I added this after run 8 diverged silently and wasted 50 epochs:

if torch.isnan(loss):
    wandb.log({"status": "diverged"})
    wandb.finish()
    return

Use count to limit sweep size. Add count: 50 to your YAML so agents stop after 50 runs. Otherwise, they’ll keep sampling forever (Bayesian search doesn’t have a natural stopping criterion).

For categorical hyperparameters with >5 options, use random search. Bayesian optimization struggles with high-cardinality categoricals because the GP kernel can’t encode “SGD is more similar to AdamW than Adam” without a custom distance metric. Random search just samples uniformly.

Don’t sweep data augmentation parameters and model hyperparameters simultaneously. They interact in non-smooth ways (heavy augmentation needs lower learning rates, but the magnitude depends on the augmentation type). Run two sweeps: one for augmentation with a fixed LR, then a second for LR/optimizer with the best augmentation config.

Scaling Beyond One GPU

The beauty of W&B Sweeps is that horizontal scaling is trivial. I tested this by launching 8 agents across two machines (4 GPUs each). Each agent independently queries the sweep controller for the next config, runs it, logs results, and repeats. No MPI, no manual task assignment.

The sweep controller is stateless from the agent’s perspective—agents don’t communicate with each other, only with W&B’s servers. This means you can add/remove agents mid-sweep without coordination. I killed 2 agents halfway through (simulating a preempted SLURM job) and the remaining 6 kept running. When I restarted the killed agents an hour later, they picked up new configs and continued.

One limitation: the sweep controller doesn’t support gang scheduling. If you’re doing distributed data parallel training (multiple GPUs per run), you need to handle that within train.py using torch.distributed or similar. The sweep agent just launches train.py—it doesn’t know or care whether that script uses multiple GPUs.

When Grid Search Still Wins

If you have exactly 3 hyperparameters with 2-4 values each (total <30 runs), grid search is simpler. No acquisition function overhead, no risk of the Bayesian prior getting stuck. For a learning rate sweep across [1e-4, 3e-4, 1e-3] and two optimizers, I’d use grid.

But beyond that threshold, Bayesian search is faster. The crossover point in my experiments was around 40 total runs: below that, grid search finished first (because Bayesian search spends the first 10 runs exploring randomly anyway). Above 40, Bayesian search pulled ahead.

What I’d Do Differently

If I ran this again, I’d add a warmup_epochs hyperparameter. Many runs spent the first 10 epochs with terrible accuracy (<50%) before the learning rate scheduler kicked in. A short warmup phase (1-3 epochs at 10% of the target LR) stabilizes early training, especially for high learning rates.

I’d also log train_accuracy, not just val_accuracy. A large train-val gap (train 95%, val 88%) signals overfitting, which means I should’ve increased weight decay or dropout. But I only noticed this pattern at run 35, after the sweep had already explored that region.

Finally, I’d use wandb.alert() to Slack me when a run beats the current best. Sweeps can run for days, and I don’t want to manually check the dashboard every hour. This is trivial:

if val_accuracy > wandb.run.summary.get("best_val_accuracy", 0):
    wandb.alert(title="New best run", text=f"Val acc: {val_accuracy:.3f}")
    wandb.run.summary["best_val_accuracy"] = val_accuracy

Use Bayesian search for anything with >4 hyperparameters or a training time >10 minutes per run. The sample efficiency pays off quickly—28 runs vs 64 is the difference between “done overnight” and “check back Monday.”

For the next part, I’ll cover W&B Artifacts for dataset versioning and model checkpointing. My current workflow saves checkpoints to local disk, which breaks reproducibility when I retrain on a different machine. Artifacts fix that, but the API has sharp edges (immutability semantics are confusing, and partial downloads don’t work the way you’d expect). More on that soon.