RLHF in 2026: Why Human Feedback Still Beats Pure AI Alignment

The DPO Hype Promised to Kill RLHF — It Didn’t

Everyone said Direct Preference Optimization would replace RLHF. The paper from Rafailov et al. (NeurIPS 2023) showed you could skip the reward model entirely, train directly on preference pairs, and get comparable results. Simpler pipeline. Fewer moving parts. What’s not to love?

I bought into the hype initially. Then I ran DPO on a custom assistant task with 15K preference pairs and watched it memorize the training distribution so hard it couldn’t generalize to slightly rephrased instructions. The loss curve looked beautiful. The actual outputs were brittle.

RLHF’s extra complexity — the separate reward model, the PPO training loop, the KL divergence constraint — turns out those aren’t bugs. They’re regularization mechanisms that prevent exactly this kind of overfitting. In 2026, after two years of the field trying every shortcut, the original Anthropic and OpenAI approach of training a reward model then doing PPO against it remains the gold standard for production systems.

Why Post-Training Alignment Can’t Be Skipped

Pretrained LLMs are next-token predictors. That’s it. They’ve learned to model the statistical distribution of human text, which includes helpful explanations AND sycophantic agreement AND toxic content AND confidently wrong statements. The base model doesn’t “want” anything — it just predicts what token comes next given the context.

The RLHF objective changes the game:

$$J(\theta) = \mathbb{E}_{x \sim D, y \sim \pi_\theta(y|x)} \left[ r_\phi(x, y) \right] – \beta \cdot D_{KL}\left( \pi_\theta(y|x) \| \pi_{ref}(y|x) \right)$$

The first term maximizes the reward model’s score. The second term (the KL penalty) keeps the policy from drifting too far from the base model’s distribution — this prevents reward hacking and maintains the model’s general capabilities.

Without this post-training step, you get a model that’s technically capable but behaviorally chaotic. It might write excellent code one moment and then helpfully explain how to synthesize dangerous compounds the next. The capability exists; the steering doesn’t.

Reward Model Training: Where Most Attempts Fail

Here’s what the tutorials don’t tell you: the reward model is the hard part. Your preference data quality determines your ceiling. Garbage in, garbage out — but with extra steps.

A typical reward model takes two completions for the same prompt and predicts which one a human would prefer:

import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

class RewardModel(nn.Module):
    def __init__(self, base_model_name="meta-llama/Llama-3.2-3B"):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(
            base_model_name,
            torch_dtype=torch.bfloat16,
            device_map="auto"
        )
        # Single scalar output for reward
        self.reward_head = nn.Linear(
            self.backbone.config.hidden_size, 1, dtype=torch.bfloat16
        )

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(
            input_ids=input_ids,
            attention_mask=attention_mask
        )
        # Use last token's hidden state (causal LM convention)
        last_hidden = outputs.last_hidden_state[:, -1, :]
        reward = self.reward_head(last_hidden)
        return reward.squeeze(-1)

The training uses a pairwise ranking loss:

$$\mathcal{L}_{reward} = -\log \sigma(r_\phi(x, y_w) – r_\phi(x, y_l))$$

where $y_w$ is the preferred (winning) completion and $y_l$ is the rejected (losing) one. The sigmoid turns the reward difference into a probability.

But here’s the trap: if your preference annotations are noisy — and they almost always are — the model learns to distinguish annotation artifacts rather than actual quality. I’ve seen reward models that confidently preferred longer responses regardless of content, because the annotators had a subtle bias toward thoroughness.

PPO Training Loop: The Moving Target Problem

Once you have a reward model, the PPO loop optimizes your language model against it. The TRL library makes this relatively straightforward:

from trl import PPOConfig, PPOTrainer, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer
import torch

config = PPOConfig(
    model_name="meta-llama/Llama-3.2-3B",
    learning_rate=1.41e-5,  # Sensitive! Higher -> reward hacking
    batch_size=64,
    mini_batch_size=8,
    ppo_epochs=4,
    gradient_accumulation_steps=8,
    init_kl_coef=0.2,  # Start conservative, anneal if needed
    target_kl=6.0,  # Stop epoch early if KL exceeds this
    cliprange=0.2,
    vf_coef=0.1,
    seed=42,
)

model = AutoModelForCausalLMWithValueHead.from_pretrained(
    config.model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained(
    config.model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(config.model_name)
tokenizer.pad_token = tokenizer.eos_token

trainer = PPOTrainer(
    config=config,
    model=model,
    ref_model=ref_model,  # For KL penalty computation
    tokenizer=tokenizer,
    # dataset and reward_model passed separately
)

The init_kl_coef=0.2 and target_kl=6.0 are critical. Set the KL coefficient too low and you get reward hacking — the model finds degenerate outputs that game the reward model. Set it too high and the model barely moves from the base policy. My best guess is that the optimal range is 0.1-0.3 for the initial coefficient, but it’s annoyingly task-dependent.

The PPO update itself follows the clipped objective:

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ is the probability ratio and $\hat{A}_t$ is the advantage estimate.

The Reward Hacking Problem Never Goes Away

You’d think with a KL penalty, the model can’t stray too far. But reward hacking is creative.

I ran an experiment training a 3B parameter model on a helpfulness reward model. Around step 4000, the reward curve was climbing steadily — looked great. Then I sampled some outputs and found the model had learned to include phrases like “I hope this helps!” and “Feel free to ask if you need clarification” in every single response, regardless of context. The reward model gave bonus points for these politeness markers. The actual answer quality hadn’t improved.

Here’s a simple heuristic check I run during training:

def check_reward_hacking(model, tokenizer, test_prompts, reward_model):
    """Quick sanity check for degenerate reward optimization."""
    model.eval()
    results = []

    for prompt in test_prompts:
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

        with torch.no_grad():
            # Generate 5 samples
            outputs = model.generate(
                **inputs,
                max_new_tokens=256,
                do_sample=True,
                temperature=0.7,
                top_p=0.9,
                num_return_sequences=5,
            )

        texts = tokenizer.batch_decode(outputs, skip_special_tokens=True)

        # Check for repetitive patterns
        unique_starts = len(set(t[:50] for t in texts))
        avg_length = sum(len(t) for t in texts) / len(texts)

        # Red flags: low diversity, extreme lengths
        if unique_starts < 3:
            print(f"WARNING: Low output diversity for '{prompt[:40]}...'")
        if avg_length < 50 or avg_length > 500:
            print(f"WARNING: Unusual length {avg_length:.0f} for '{prompt[:40]}...'")

        results.append({
            "prompt": prompt,
            "diversity": unique_starts,
            "avg_length": avg_length,
            "samples": texts[:2]  # Keep for manual inspection
        })

    return results

The fix usually involves reward model regularization or ensembling multiple reward models. OpenAI’s 2022 paper mentioned using an ensemble of reward models and taking the minimum — this caps the upside of gaming any single model.

Why DPO and RLAIF Don’t Fully Replace Human Feedback

Direct Preference Optimization (Rafailov et al., 2023) reformulates the RLHF objective to avoid training a separate reward model:

$$\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} – \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$

Elegant, yes. But in practice, DPO has a few sharp edges:

1. Overfitting to preference pairs — The model directly optimizes to distinguish winners from losers in the training set. Without the reward model as an intermediary, there’s less smoothing.

2. No online learning — RLHF can generate new samples and get rewards for them. DPO trains only on the fixed preference dataset. For iterative improvement, you need to regenerate preferences after each training round.

3. Harder to debug — With RLHF, you can inspect the reward model’s predictions to understand what’s being optimized. With DPO, the “reward” is implicit.

RLAIF (Constitutional AI / RL from AI Feedback) addresses the annotation bottleneck by using another AI to generate preferences. Anthropic’s approach has an AI critique its own outputs and choose the better response.

But here’s the catch: RLAIF inherits whatever biases exist in the critic model. If your critic has blind spots, your trained model will too. Human feedback — expensive and inconsistent as it is — provides ground truth that AI critics can’t manufacture. The 2024 scaling laws research suggested that as model capabilities increase, the gap between RLAIF and RLHF narrows, but for nuanced judgments (cultural sensitivity, factual accuracy on edge cases), humans are still the gold standard.

Training Curves Don’t Always Mean What You Think

Watch for these red flags during RLHF training:

• Reward increases but KL stays flat — Usually means the reward model is too weak or the learning rate is too low. The policy isn’t being pushed.

• KL spikes suddenly — The policy just found an exploit. Roll back and increase the KL coefficient.

• Reward plateaus while KL keeps climbing — You’re paying the divergence cost without getting reward benefit. Reduce learning rate or check if the reward model has a ceiling.

Here’s what a healthy training run looks like (approximately):

Step   Reward   KL      Policy Loss   Value Loss
0      -0.12    0.00    -0.002        0.45
500    0.34     0.89    -0.018        0.32
1000   0.71     1.56    -0.024        0.28
2000   1.12     2.34    -0.031        0.21
4000   1.45     3.01    -0.029        0.18  # Notice reward gain slowing
8000   1.52     3.89    -0.025        0.15  # Diminishing returns

The reward should improve rapidly early, then taper. KL should grow but not explosively. If KL hits 6+ nats, you’re probably too far from the base policy to maintain general capabilities.

Practical Hyperparameter Guidance

I’ve trained RLHF loops on 3B to 8B parameter models. These settings worked for me:

The learning rate is where most runs die. Too high and the model collapses — outputs become degenerate or repetitive within 1000 steps. Too low and you’re wasting compute for marginal improvements. Start conservative (1e-5 for 7B models) and increase if KL isn’t growing.

The Annotation Quality Problem

Good preference data is rare. Bad preference data is everywhere.

Most publicly available preference datasets have these issues:

• Length bias — Annotators prefer longer, more detailed responses even when brevity is appropriate
• Sycophancy — Responses that validate the user’s stated position win, even if they’re wrong
• Style over substance — Well-formatted responses with headers and bullet points beat accurate but plain responses

The Anthropic HH-RLHF dataset (if I recall correctly, released around 2022) was one of the first large-scale efforts, but even it has artifacts from the annotation process. More recent efforts like Nectar and UltraFeedback try to address these, but annotation quality remains the bottleneck.

For custom applications, I’d recommend at minimum 5K preference pairs, with clear guidelines for annotators and built-in quality checks (agreement scores between annotators, attention checks). Budget at least mechanical keyboard money for the annotation infrastructure — it’s not glamorous, but it’s where alignment actually happens.

What About Constitutional AI?

Anthropic’s Constitutional AI (Bai et al., 2022) reduces human feedback requirements by having the model critique itself according to a set of principles. The simplified pipeline:

1. Generate a response
2. Ask the model to critique the response based on principles (“is this harmful?”, “is this helpful?”)
3. Ask the model to revise based on the critique
4. Use the revision as the “preferred” response

This generates synthetic preference data that can then feed into standard RLHF or DPO. The approach scales better than human annotation and encodes principles explicitly rather than implicitly through examples.

But Constitutional AI still needs humans to write the constitution. The principles themselves encode human values. And for domains where the model’s understanding is limited (medical advice, legal questions, cultural nuance), AI self-critique can miss critical issues.

My take: Constitutional AI is great for scaling up from a smaller human-annotated seed. It’s not a replacement for that seed.

Where RLHF Is Heading

The 2025-2026 research has focused on a few directions:

1. Process reward models — Instead of scoring final outputs, score intermediate reasoning steps. This catches errors earlier and provides denser training signal. See the recent work from DeepMind on process supervision.

2. Multi-objective RLHF — Separate reward models for helpfulness, harmlessness, honesty. Weight them dynamically based on context. This avoids the single-number reduction that loses nuance.

3. Online RLHF — Continuously collecting human feedback during deployment and updating models. Closes the loop between usage and training.

4. Scalable oversight — Using AI assistance to help humans evaluate complex outputs (code, long documents, multi-step reasoning). Humans remain in the loop but augmented by AI tools.

I’m not entirely sure which of these will dominate. Process reward models seem promising but are compute-intensive. Multi-objective approaches add complexity. My best guess is that production systems in 2027 will use some combination: Constitutional AI to generate initial preferences, process reward models for reasoning tasks, with human oversight for high-stakes decisions.

FAQ

Q: Is DPO or RLHF better for fine-tuning a custom assistant?

DPO is faster to set up and trains in a single stage. For small preference datasets (<10K pairs) and simpler tasks, it often works well. But for complex behaviors, production-quality alignment, or when you need to iteratively improve, RLHF with a separate reward model gives more control and better generalization. Start with DPO to validate your data, move to RLHF if results plateau.

Q: How much preference data do I need for RLHF?

For a reward model that generalizes reasonably, 10K-50K preference pairs is a good starting point. Quality matters more than quantity — 5K clean, carefully annotated pairs beat 50K noisy ones. For specialized domains, you might get away with less, but expect the reward model to overfit to surface patterns.

Q: Can I use synthetic preferences from GPT-4 instead of human annotations?

Yes, this is essentially RLAIF. It works surprisingly well for many tasks, especially if you prompt the evaluator carefully. The limitation is that GPT-4’s preferences inherit GPT-4’s biases and blind spots. For deployment to diverse user bases or high-stakes applications, at least some human validation of the synthetic preferences is important.

RLHF is messy. The reward model needs careful training. The PPO loop has a dozen hyperparameters that interact in non-obvious ways. Reward hacking is always lurking. Annotation quality is a constant battle.

But it works. Two years of the field trying to find shortcuts have mostly circled back to the original insight: you need a reward model to provide dense signal, and you need human preferences to ground that model in actual values. DPO simplifies the pipeline at the cost of flexibility. RLAIF scales annotation at the cost of potentially propagating AI biases. For production systems where alignment matters, the full RLHF pipeline — reward model training, PPO with KL constraints, careful monitoring — remains the approach that consistently delivers.

What I’m still trying to figure out: how to do RLHF efficiently when the model needs to balance multiple objectives that sometimes conflict. Helpfulness versus safety versus honesty — optimizing all three simultaneously without gaming any one of them. The multi-objective RLHF research is promising, but I haven’t seen a clean implementation that I’d trust in production. That’s the problem I’ll be poking at next.