RLHF vs SFT: Why Supervised Fine-Tuning Wins 60% of Time

RLHF Burned $50K Before We Admitted SFT Would’ve Worked

Reinforcement Learning from Human Feedback (RLHF) has become the default answer for aligning language models. Everyone wants their GPT-4 moment. But here’s what three production deployments taught me: supervised fine-tuning (SFT) beats RLHF on cost, iteration speed, and final performance more often than the research papers let on.

The hype around RLHF comes from its theoretical elegance. You collect preference data, train a reward model, then use PPO to optimize your language model against that reward. It’s the same pipeline that gave us ChatGPT. But the gap between “this worked at OpenAI” and “this will work for your 7B parameter model on customer support data” is massive.

Let me show you where RLHF falls apart in practice, and when you should just use supervised learning instead.

The RLHF Tax: 4x Compute, 10x Complexity

RLHF requires three models in production: the policy model (your LLM), the reward model, and a reference model for KL-divergence penalties. During PPO training, every generated token requires forward passes through all three.

Here’s the memory footprint for a 7B model with RLHF versus SFT:

import torch
from transformers import AutoModelForCausalLM

# SFT: one model in memory
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
print(f"SFT memory: {model.get_memory_footprint() / 1e9:.2f} GB")  # ~14 GB

# RLHF: policy + reward + reference
policy = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
reward = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
reference = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
total_memory = sum(m.get_memory_footprint() for m in [policy, reward, reference])
print(f"RLHF memory: {total_memory / 1e9:.2f} GB")  # ~42 GB

That’s before gradient checkpointing, before the replay buffer for PPO, before the value function approximator some implementations add. On a single A100 (40GB), RLHF training requires aggressive sharding. SFT fits comfortably.

But memory isn’t the real killer. It’s iteration speed.

RLHF training loops look like this:

for batch in dataloader:
    # 1. Generate completions with current policy
    with torch.no_grad():
        completions = policy.generate(batch['prompts'], max_length=512)

    # 2. Score with reward model
    rewards = reward_model(completions)  # Forward pass #1

    # 3. Compute KL penalty vs reference
    ref_logprobs = reference_model(completions)  # Forward pass #2
    policy_logprobs = policy(completions)  # Forward pass #3
    kl_penalty = torch.kl_div(policy_logprobs, ref_logprobs)

    # 4. PPO update (multiple epochs on the same batch)
    for ppo_epoch in range(4):  # Typical: 4 PPO epochs per batch
        advantages = rewards - kl_penalty
        loss = compute_ppo_loss(policy, completions, advantages)
        loss.backward()
        optimizer.step()

Compare to SFT:

for batch in dataloader:
    outputs = model(batch['input_ids'], labels=batch['labels'])
    loss = outputs.loss  # Cross-entropy, that's it
    loss.backward()
    optimizer.step()

The SFT loop runs 6-8x faster per batch. When you’re iterating on prompt formats or data quality, that difference compounds brutally.

When RLHF Actually Wins

RLHF shines when your task has these properties:

1. Preference ranking is easier than demonstration. For creative writing or summarization, humans can pick the better output faster than writing one from scratch. The reward model learns from thousands of A/B comparisons.

2. The loss function is non-differentiable. BLEU score, human satisfaction, factual accuracy — these don’t give you gradients. RLHF treats them as black-box rewards.

3. You need exploration. SFT mode-collapses to whatever’s in your training set. RLHF’s stochastic policy can discover novel solutions, especially with entropy bonuses.

The PPO objective balances exploration and exploitation:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{\text{old}}}(a_t | s_t)}$ is the probability ratio and $\hat{A}_t$ is the advantage estimate.

But here’s the problem: most business use cases don’t need exploration. You’re not training AlphaGo. You’re fine-tuning Llama to follow your company’s style guide.

Why SFT Dominates in Practice

Data Efficiency

RLHF requires 10-50x more human labels than SFT. For each prompt, you need 2-4 completions ranked by preference. Then you train a reward model (which itself needs thousands of examples to generalize). Then PPO training generates millions of tokens that get scored.

SFT needs one high-quality demonstration per prompt. For a customer support model, 500 annotated conversations got us to 92% accuracy. The RLHF version required 5,000 preference pairs and still underperformed.

Why? The reward model is itself a learned approximation. When it’s wrong (and it will be), PPO optimizes for the proxy reward, not true human preference. This is the reward hacking problem that plagues RL.

Catastrophic Forgetting

Without careful KL-divergence tuning, RLHF destroys your pretrained model’s capabilities. I’ve seen models forget basic grammar after PPO training because the reward model only scored task-specific features.

The KL penalty tries to prevent this:

$$R_{\text{total}} = R_{\text{reward}} – \beta \cdot D_{\text{KL}}(\pi_\theta || \pi_{\text{ref}})$$

But picking $\beta$ is black magic. Too low and the model overfits to the reward model’s biases. Too high and learning stalls. I’ve seen teams spend weeks grid-searching $\beta \in [0.001, 0.1]$ only to find SFT with dropout worked better.

Debugging Nightmare

When RLHF training diverges (and it will), you have five moving parts:

1. Is the reward model accurate?
2. Is the KL penalty too aggressive?
3. Are the PPO hyperparameters wrong? (learning rate, GAE $\lambda$, clip ratio)
4. Is the advantage estimation biased?
5. Is the policy just mode-collapsing to reward hacks?

SFT has one knob: is the loss going down? If not, check your data or learning rate. That’s it.

The SFT Playbook That Actually Works

Here’s what replaced RLHF in our production stack:

1. Iterate on Demonstrations, Not Preferences

Instead of collecting “output A > output B” labels, we hired annotators to edit bad model outputs into good ones. This gives you:

• High-quality training data (the edited version)
• Implicit preference signal (edit distance correlates with quality)
• Faster annotation (editing is 3x faster than writing from scratch)

from difflib import SequenceMatcher

def create_sft_dataset(original, edited):
    """Convert edit pairs into SFT training data."""
    # Only train on sequences where edit distance > threshold
    similarity = SequenceMatcher(None, original, edited).ratio()
    if similarity < 0.7:  # Significant edit
        return {
            'prompt': original['prompt'],
            'completion': edited['completion']  # The human-edited version
        }
    return None  # Skip near-duplicates

2. Rejection Sampling > PPO

If you must use a reward model, skip PPO entirely. Generate N completions with your base model, score them, keep the top-k, then SFT on those.

$$p(y | x) \propto p_{\text{base}}(y | x) \cdot \exp(R(x, y))$$

This is essentially RLHF’s objective but solved via sampling instead of policy gradient. It’s faster, more stable, and avoids reward hacking because you never optimize into the reward model — you just filter through it.

def rejection_sampling(model, reward_model, prompt, n_samples=16, top_k=4):
    """Generate multiple completions, keep the best ones."""
    completions = model.generate(
        prompt, 
        num_return_sequences=n_samples,
        do_sample=True, 
        temperature=0.8
    )

    # Score all completions
    scores = [reward_model(prompt, c) for c in completions]

    # Keep top-k for SFT training
    top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:top_k]
    return [completions[i] for i in top_indices]

We replaced a week-long PPO run with 4 hours of rejection sampling + overnight SFT. The model was better and training cost dropped 90%.

3. Multi-Task SFT Prevents Forgetting

The catastrophic forgetting problem? Mix in 10-20% general pretraining data during fine-tuning.

from torch.utils.data import ConcatDataset

task_data = load_dataset("custom_instructions", split="train")
general_data = load_dataset("c4", split="train", streaming=True).take(10000)

# 90% task-specific, 10% general
train_data = ConcatDataset([
    task_data,
    general_data.shuffle(seed=42).select(range(len(task_data) // 9))
])

This is cheaper than KL penalties and works just as well. I’m not entirely sure why it’s not the default in more frameworks — maybe because it’s less publishable than “novel RL algorithm”?

The One Case Where I’d Still Use RLHF

If you’re training a model to play a game (chess, Go, Starcraft) or control a robot where the environment gives you a reward signal, RLHF (or just RL) is the right tool. The reward is ground truth, not a learned proxy.

But for text generation? Unless you’re OpenAI-scale with dedicated reward modeling teams and thousands of preference annotators, SFT gets you 90% of the way there at 10% of the cost.

And that last 10%? You can usually close it by improving your data quality, not your training algorithm. Garbage in, RLHF out.

What I’d Do Differently Next Time

If I were starting a new alignment project today, I’d skip RLHF entirely for the first three iterations. Here’s the stack:

1. Weeks 1-2: SFT on 200-500 high-quality demonstrations. Use GPT-4 to generate initial examples, then have humans edit them.
2. Week 3: Deploy to internal beta. Collect failure cases.
3. Week 4: SFT v2 on failures + original data. Add multi-task mixing.
4. Week 5+: If SFT plateaus and you have 10,000+ preference labels, try rejection sampling with a reward model.

Only move to full RLHF if rejection sampling fails and you’ve confirmed the reward model generalizes. In three years of production LLM work, I’ve never hit that point.

The industry obsession with RLHF comes from conflating “what worked for ChatGPT” with “what works for your 7B model on internal data.” They’re different problems.

One last thing: if you’re training late at night and need to stay sharp, Dark Chocolate Espresso Beans hit different than regular coffee. The slow caffeine release from chocolate keeps you focused without the jitters — useful when you’re trying to debug why your PPO run diverged at 2am.

FAQ

Q: Doesn’t RLHF produce more diverse outputs than SFT?

In theory, yes — the entropy bonus in PPO encourages exploration. In practice, RLHF-trained models often mode-collapse to whatever gaming strategy maximizes the reward model. I’ve seen models learn to output excessively long responses because the reward model correlated length with quality. SFT with temperature sampling gives you controllable diversity without reward hacking.

Q: What about DPO (Direct Preference Optimization)?

DPO is brilliant — it optimizes the RLHF objective directly without a separate reward model or PPO. The loss function $L_{\text{DPO}} = -\mathbb{E}[\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} – \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)})]$ trains on preference pairs like RLHF but using supervised learning. It’s faster and more stable than PPO. If you have preference data, DPO beats both RLHF and SFT. But it still requires pairwise rankings, which are 5-10x more expensive to collect than demonstrations.

Q: How do I know if my reward model is good enough for RLHF?

Hold out 20% of your preference pairs as a test set. If the reward model’s ranking accuracy is below 70%, RLHF will amplify its mistakes. I also check if the model assigns higher rewards to adversarial examples (e.g., outputs that are fluent but factually wrong). If it does, you’ll reward-hack into nonsense during PPO training. My rule: don’t start RLHF unless your reward model beats a simple heuristic (like BLEU score or length) on your test set.