GRPO vs PPO for LLM Reasoning: DeepSeek-R1 Explained

If you’ve been following AI research in 2025-2026, you know that reinforcement learning for LLM reasoning is the single hottest frontier. DeepSeek-R1 stunned the community by producing chain-of-thought reasoning that rivals OpenAI’s o1 and o3 models — but with a fundamentally different training recipe. The secret ingredient? GRPO (Group Relative Policy Optimization), a method that throws away the critic network entirely and still achieves state-of-the-art reasoning performance.

In this post, I’ll break down how GRPO works, why it solves real problems PPO struggles with for reasoning, and how DeepSeek-R1’s full pipeline comes together — with code examples included.

What Is PPO and How Does It Work for RLHF?

Before we can understand why GRPO exists, we need a solid grasp of Proximal Policy Optimization (PPO) — the workhorse behind most RLHF (Reinforcement Learning from Human Feedback) systems, including the original ChatGPT training pipeline.

PPO is an actor-critic algorithm. In the RLHF context, this means you maintain four separate models simultaneously:

• Policy model (actor) — the LLM you’re training to generate better responses
• Reference model — a frozen copy of the original policy, used to compute KL divergence penalties so the model doesn’t drift too far from its pre-trained behavior
• Reward model — trained on human preference data to score outputs
• Value model (critic) — estimates the expected future reward from any given state, used to compute advantages

The core PPO objective looks like this:

$$L_{text{PPO}} = mathbb{E}left[ minleft( r(theta) hat{A},; text{clip}(r(theta), 1-varepsilon, 1+varepsilon) hat{A} right) right] – beta cdot text{KL}(pi_theta | pi_{text{ref}})$$

Where $r(theta) = pi_theta(a|s) / pi_{text{old}}(a|s)$ is the probability ratio between the new and old policy, $hat{A}$ is the advantage estimate (how much better an action was than expected), and the clipping prevents destructively large policy updates.

The advantage is typically computed using Generalized Advantage Estimation (GAE):

$$hat{A}_t = sum_{l=0}^{infty} (gammalambda)^l delta_{t+l}, quad text{where } delta_t = r_t + gamma V(s_{t+1}) – V(s_t)$$

This is where the value model (critic) earns its keep — it provides the baseline $V(s)$ that lets you estimate whether an action was better or worse than average.

Why PPO Struggles with Long Chain-of-Thought Reasoning

PPO works beautifully for general RLHF — teaching a model to be helpful, harmless, and honest based on human preferences. But when you try to apply it to reasoning tasks with long chain-of-thought outputs, several serious problems emerge:

1. The value function is nearly impossible to train well for reasoning. The value model estimates expected reward at each token position. But reasoning chains can be hundreds or thousands of tokens long, with the reward signal (correct/incorrect) arriving only at the very end. The critic has to predict, from the middle of a partial chain-of-thought, whether the final answer will be right. This credit assignment problem is much harder than estimating reward for a short chatbot reply.

2. Memory cost is brutal. You need four full-sized models in memory simultaneously. For a 7B parameter model, that’s roughly:

Model	Parameters	Memory (FP16)
Policy (actor)	7B	~14 GB
Reference (frozen)	7B	~14 GB
Reward model	7B	~14 GB
Value model (critic)	7B	~14 GB
Total	28B	~56 GB

Now scale that to 67B or 671B parameter models like DeepSeek-R1. The infrastructure cost becomes astronomical, and the value model adds zero value (pun intended) if it can’t actually learn a useful baseline for reasoning chains.

3. Training instability. A poorly trained critic introduces noisy, misleading advantage estimates. Instead of helping the policy improve, it can cause training to diverge or oscillate. For reasoning tasks, where reward is sparse and delayed, this instability is significantly worse than for standard alignment tasks.

What Is GRPO and How Does It Differ from PPO?

Group Relative Policy Optimization (GRPO), introduced by DeepSeek, makes a deceptively simple but powerful change: it removes the critic entirely and replaces learned value estimates with a group-relative baseline computed from sampled outputs.

Here’s the key insight: instead of asking a value model “how good is this state?”, GRPO asks “how good is this output compared to other outputs for the same question?”

The procedure works like this:

1. For each prompt/question q, sample a group of G complete outputs: {o_1, o_2, …, o_G}
2. Score each output using a reward function: {r_1, r_2, …, r_G}
3. Compute the group-relative advantage for each output by normalizing within the group:

   $$hat{A}_i = frac{r_i – text{mean}(r_1, ldots, r_G)}{text{std}(r_1, ldots, r_G)}$$

4. Update the policy using the PPO-style clipped objective, but with these group-relative advantages instead of critic-based advantages

The full GRPO objective is:

$$L_{text{GRPO}} = mathbb{E}_q left[ frac{1}{G} sum_{i=1}^{G} frac{1}{|o_i|} sum_{t=1}^{|o_i|} minleft( r_t(theta) hat{A}_i,; text{clip}(r_t(theta), 1-varepsilon, 1+varepsilon) hat{A}_i right) right] – beta cdot text{KL}(pi_theta | pi_{text{ref}})$$

Where $r_t(theta) = pi_theta(o_{i,t} | q, o_{i,<t}) / pi_{text{old}}(o_{i,t} | q, o_{i,<t})$ and the KL divergence is computed per-token as an approximation:

$$text{KL} approx frac{pi_{text{ref}}(o_{i,t})}{pi_theta(o_{i,t})} – logfrac{pi_{text{ref}}(o_{i,t})}{pi_theta(o_{i,t})} – 1$$

The beauty of this formulation is that the advantage $hat{A}_i$ is constant across all tokens in a single output — it’s an outcome-level advantage, not a token-level one. This completely sidesteps the credit assignment problem that makes the critic so hard to train for reasoning.

Why GRPO Works So Well for Reasoning Tasks

GRPO’s design is almost tailor-made for training reasoning capabilities. Here’s why:

Binary reward signals are natural for reasoning. Math problems have verifiable correct answers. Code problems have test cases. Logic puzzles have ground truth. You don’t need a learned reward model — you can use rule-based verification: +1 for correct answer, -1 for incorrect, with optional bonuses for proper formatting. DeepSeek-R1 uses exactly this approach.

Group sampling captures reasoning diversity. When you sample 16-64 outputs for the same math problem, some will find the right approach and others won’t. The group normalization naturally surfaces which reasoning strategies are working. If only 2 out of 16 samples solve a hard problem, those 2 get strongly positive advantages, providing a clear learning signal.

No credit assignment needed within the chain. Since the advantage is assigned per-output (not per-token), GRPO doesn’t need to figure out which specific step in a 500-token reasoning chain was the critical one. It just says “this entire chain led to a correct answer, so reinforce it; that entire chain led to a wrong answer, so suppress it.” Over many training steps, the policy gradually learns which internal reasoning patterns lead to correct outcomes.

Memory efficiency enables larger models. By dropping the critic, GRPO reduces the model count from 4 to 3 (policy, reference, reward — and for math/code tasks, the reward model is just a verifier script, so effectively 2 neural networks). This freed up the GPU memory that DeepSeek needed to train their massive 671B MoE model with RL.

How DeepSeek-R1 Uses GRPO for Chain-of-Thought Training

DeepSeek-R1’s training pipeline is a carefully orchestrated multi-stage process. Understanding the full pipeline reveals why GRPO was essential at each stage:

Stage 1: Cold Start SFT (Supervised Fine-Tuning)

Before any RL, DeepSeek collects a small set of long chain-of-thought examples (thousands, not millions). These are used to fine-tune the base DeepSeek-V3 model so it can produce the <think>...</think> format that reasoning requires. Without this cold start, the RL phase would need to discover the entire concept of step-by-step reasoning from scratch — possible but extremely sample-inefficient.

Stage 2: GRPO Reinforcement Learning

This is where the magic happens. The cold-started model undergoes GRPO training on math, code, and logic tasks with rule-based rewards:

• Accuracy reward: Does the final answer match the ground truth? (For math: exact match or equivalent expression. For code: passes all test cases.)
• Format reward: Does the output properly use the <think> tags and put the final answer in the expected location?

No learned reward model is needed. The policy generates groups of outputs per question, each output is scored by these deterministic rules, and GRPO computes group-relative advantages to update the policy.

A remarkable phenomenon occurs during this stage: the model spontaneously develops sophisticated reasoning behaviors — self-verification (“Wait, let me check that…”), backtracking (“That approach won’t work because…”), and exploration of multiple solution paths. These emerge purely from the RL optimization, not from the supervised data.

Stage 3: Rejection Sampling + New SFT Data

After the GRPO RL phase, the model is used to generate large volumes of reasoning traces. The correct ones (verified by ground truth) are collected as high-quality SFT data. This data is combined with human-written examples for general tasks (writing, summarization, etc.) to create a comprehensive fine-tuning dataset.

Stage 4: Final SFT + Secondary RL

The base model is fine-tuned on this enriched dataset, then undergoes a second, shorter round of RL that includes both reasoning rewards and a helpfulness/safety reward model. This produces the final DeepSeek-R1 model.

PPO vs GRPO: A Practical Comparison

Here’s a side-by-side comparison for practitioners deciding between the two approaches:

Aspect	PPO	GRPO
Models in memory	4 (policy, ref, reward, value)	2-3 (policy, ref, optional reward)
Memory overhead	~4x model size	~2-3x model size
Advantage estimation	Token-level via critic (GAE)	Output-level via group statistics
Credit assignment	Fine-grained but hard to train	Coarse but robust
Reward type	Learned reward model	Rule-based or learned
Compute per step	1 output per prompt	G outputs per prompt (more generation)
Training stability	Sensitive to critic quality	More stable (no critic to overfit)
Best for	General alignment (helpfulness, safety)	Reasoning (math, code, logic)
Sparse rewards	Struggles (critic can’t learn)	Handles well (group normalization)
Sample efficiency	Higher (1 sample per prompt)	Lower (needs G samples per prompt)

The key tradeoff: GRPO trades sample efficiency (it generates more outputs per prompt) for memory efficiency and training stability. For reasoning tasks where you can verify correctness programmatically, this is almost always a worthwhile trade.

Code Example: Simplified GRPO Training Loop in PyTorch

Let’s implement a simplified GRPO training loop so you can see the mechanics in action. This is not production code — real implementations like TRL or OpenRLHF handle distributed training, memory optimization, and many edge cases — but it captures the essential algorithm.

import torch
import torch.nn.functional as F
from torch.distributions import Categorical
from copy import deepcopy


class GRPOTrainer:
    """Simplified GRPO trainer for language model reasoning tasks."""

    def __init__(
        self,
        policy_model,
        tokenizer,
        reward_fn,
        lr=1e-6,
        group_size=16,
        clip_epsilon=0.2,
        kl_coeff=0.04,
        max_gen_len=512,
    ):
        self.policy = policy_model
        self.ref_policy = deepcopy(policy_model)  # frozen reference
        self.ref_policy.eval()
        for p in self.ref_policy.parameters():
            p.requires_grad = False

        self.tokenizer = tokenizer
        self.reward_fn = reward_fn  # fn(question, answer) -> float
        self.optimizer = torch.optim.AdamW(self.policy.parameters(), lr=lr)

        self.group_size = group_size
        self.clip_epsilon = clip_epsilon
        self.kl_coeff = kl_coeff
        self.max_gen_len = max_gen_len

    @torch.no_grad()
    def generate_group(self, prompt_ids):
        """Sample G completions for a single prompt."""
        self.policy.eval()
        completions = []

        for _ in range(self.group_size):
            output = self.policy.generate(
                prompt_ids.unsqueeze(0),
                max_new_tokens=self.max_gen_len,
                do_sample=True,
                temperature=0.7,
                top_p=0.9,
            )
            # Extract only the generated tokens (exclude prompt)
            completion = output[0, prompt_ids.shape[0]:]
            completions.append(completion)

        self.policy.train()
        return completions

    def compute_group_advantages(self, prompt_text, completions):
        """Score each completion and compute group-relative advantages."""
        rewards = []
        for comp in completions:
            answer_text = self.tokenizer.decode(comp, skip_special_tokens=True)
            reward = self.reward_fn(prompt_text, answer_text)
            rewards.append(reward)

        rewards = torch.tensor(rewards, dtype=torch.float32)

        # Group-relative normalization
        mean_r = rewards.mean()
        std_r = rewards.std() + 1e-8  # avoid division by zero
        advantages = (rewards - mean_r) / std_r

        return advantages, rewards

    def compute_log_probs(self, model, prompt_ids, completion_ids):
        """Compute per-token log probabilities for a completion."""
        input_ids = torch.cat([prompt_ids, completion_ids]).unsqueeze(0)
        with torch.no_grad() if not model.training else torch.enable_grad():
            logits = model(input_ids).logits

        # Shift: predict token t+1 from position t
        prompt_len = prompt_ids.shape[0]
        # We want log probs for completion tokens only
        completion_logits = logits[0, prompt_len - 1: -1, :]  # positions that predict completion tokens
        log_probs = F.log_softmax(completion_logits, dim=-1)

        # Gather log probs for actual completion tokens
        token_log_probs = log_probs.gather(1, completion_ids.unsqueeze(1)).squeeze(1)
        return token_log_probs

    def grpo_step(self, prompt_text, ground_truth=None):
        """Execute one GRPO training step for a single prompt."""
        prompt_ids = self.tokenizer.encode(prompt_text, return_tensors="pt")[0]
        device = next(self.policy.parameters()).device
        prompt_ids = prompt_ids.to(device)

        # Step 1: Generate group of completions
        completions = self.generate_group(prompt_ids)

        # Step 2: Compute group-relative advantages
        advantages, rewards = self.compute_group_advantages(
            prompt_text, completions
        )
        advantages = advantages.to(device)

        # Step 3: Compute policy loss with clipping
        total_loss = torch.tensor(0.0, device=device)
        total_kl = torch.tensor(0.0, device=device)

        # Get old policy log probs (before this update)
        old_log_probs_list = []
        for comp in completions:
            comp = comp.to(device)
            with torch.no_grad():
                old_lp = self.compute_log_probs(self.policy, prompt_ids, comp)
            old_log_probs_list.append(old_lp.detach())

        self.policy.train()
        self.optimizer.zero_grad()

        for i, comp in enumerate(completions):
            comp = comp.to(device)

            # Current policy log probs
            new_log_probs = self.compute_log_probs(self.policy, prompt_ids, comp)

            # Reference policy log probs (for KL penalty)
            with torch.no_grad():
                ref_log_probs = self.compute_log_probs(
                    self.ref_policy, prompt_ids, comp
                )

            # Probability ratio
            old_log_probs = old_log_probs_list[i]
            ratio = torch.exp(new_log_probs - old_log_probs)

            # Clipped surrogate objective
            adv = advantages[i]
            surr1 = ratio * adv
            surr2 = torch.clamp(ratio, 1.0 - self.clip_epsilon,
                                1.0 + self.clip_epsilon) * adv
            policy_loss = -torch.min(surr1, surr2).mean()

            # KL divergence penalty (per-token approximation)
            # KL(ref || policy) approx = exp(ref - policy) - (ref - policy) - 1
            log_ratio_kl = ref_log_probs - new_log_probs
            kl = (torch.exp(log_ratio_kl) - log_ratio_kl - 1.0).mean()

            # Output-level loss (averaged over tokens in this completion)
            total_loss += (policy_loss + self.kl_coeff * kl) / self.group_size
            total_kl += kl.detach() / self.group_size

        total_loss.backward()
        torch.nn.utils.clip_grad_norm_(self.policy.parameters(), max_norm=1.0)
        self.optimizer.step()

        return {
            "loss": total_loss.item(),
            "mean_reward": rewards.mean().item(),
            "max_reward": rewards.max().item(),
            "kl": total_kl.item(),
            "solve_rate": (rewards > 0).float().mean().item(),
        }

And here’s how you’d use it with a math reasoning reward function:

import re

def math_reward_fn(question, answer):
    """Rule-based reward: +1 for correct answer, -1 for incorrect."""
    # Extract the final numerical answer from the model output
    # Assumes format like "The answer is 42" or "boxed{42}"
    match = re.search(r'boxed{([^}]+)}', answer)
    if not match:
        match = re.search(r'answer is[:s]*(-?[d.]+)', answer, re.IGNORECASE)
    if not match:
        return -1.0

    predicted = match.group(1).strip()

    # ground_truth would be passed via closure or looked up
    # For simplicity, this is a placeholder
    expected = get_expected_answer(question)

    try:
        if abs(float(predicted) - float(expected)) < 1e-6:
            return 1.0
    except ValueError:
        if predicted == expected:
            return 1.0

    return -1.0


def format_reward_fn(question, answer):
    """Bonus reward for proper formatting."""
    score = 0.0
    if '<think>' in answer and '</think>' in answer:
        score += 0.5  # Used thinking tags
    if 'boxed{' in answer:
        score += 0.5  # Boxed final answer
    return score


def combined_reward(question, answer):
    """Combined reward: accuracy + formatting."""
    return math_reward_fn(question, answer) + format_reward_fn(question, answer)


# Training loop
trainer = GRPOTrainer(
    policy_model=model,
    tokenizer=tokenizer,
    reward_fn=combined_reward,
    group_size=16,
    kl_coeff=0.04,
    clip_epsilon=0.2,
)

for epoch in range(num_epochs):
    for question in math_dataset:
        stats = trainer.grpo_step(question)
        print(f"Reward: {stats['mean_reward']:.3f} | "
              f"Solve rate: {stats['solve_rate']:.1%} | "
              f"KL: {stats['kl']:.4f}")

When to Use PPO vs GRPO: A Decision Framework

Neither algorithm is strictly better — they excel in different regimes. Here's how I think about choosing between them:

Use GRPO when:

• Your task has verifiable outcomes (math, code, logic puzzles, structured output generation)
• You're working with long output sequences where credit assignment is hard
• You want to maximize model size on limited GPU memory
• Training stability is a priority and you want to avoid tuning a critic
• You can afford the extra generation cost (compute for sampling G outputs per prompt)

Use PPO when:

• Your reward is soft/subjective (human preference, helpfulness, writing quality) where you need a learned reward model
• Outputs are relatively short so the critic can learn a good value function
• You need token-level credit assignment (e.g., steering specific parts of the response)
• Sample efficiency matters more than memory efficiency
• You're doing standard RLHF alignment (not reasoning-specific training)

Consider hybrid approaches when:

• You need both reasoning ability AND general alignment (like DeepSeek-R1's Stage 4, which uses both rule-based and model-based rewards)
• You want GRPO for the reasoning RL phase and PPO (or DPO) for the safety/alignment phase

The field is moving fast — variants like DAPO and Dr. GRPO are already refining the group-relative approach with dynamic sampling and entropy bonuses for exploration.

Key Takeaways

The shift from PPO to GRPO for reasoning reflects a broader trend: simpler algorithms that exploit task structure often beat general-purpose methods. Here's what matters most for practitioners:

1. GRPO makes RL-for-reasoning accessible. You don't need 8xH100 to hold four copies of a large model. Two models plus a verifier script is enough.
2. The training pipeline matters as much as the algorithm. DeepSeek-R1's cold start SFT, GRPO RL, rejection sampling, and final SFT stages are all essential. GRPO alone isn't magic — the careful curriculum is what makes it work.
3. Rule-based rewards unlock RL for reasoning. The fact that math and code have verifiable answers means you can skip the entire reward model training step, which is itself a major source of noise and complexity.
4. Emergent reasoning behaviors are real. The self-verification and backtracking that emerge during GRPO training are not hand-designed — they arise from the optimization pressure. This is one of the most exciting findings in recent AI research.

If you're building systems that need strong reasoning, GRPO should be at the top of your toolkit. DeepSeek-R1 demonstrated that you don't need OpenAI's budget — you need the right algorithm, the right data pipeline, and a healthy dose of compute.

FAQ

Can GRPO completely replace PPO for all LLM training?

No. GRPO is specifically designed for tasks with verifiable or clearly scorable outcomes, like math, code generation, and logic problems. For general alignment tasks where rewards are based on subjective human preferences (helpfulness, writing quality, safety), PPO with a learned reward model is still the standard approach. DeepSeek-R1 itself uses a learned reward model in its final RL stage alongside rule-based rewards. The two methods are complementary, not competing.

How many samples per group does GRPO need to work effectively?

DeepSeek-R1 used group sizes of 16 to 64 samples per prompt. Smaller groups (8-16) can work for easier tasks, but harder problems benefit from larger groups to increase the chance of finding correct solutions. The trade-off is generation compute: 64 samples costs 64x more inference than PPO's single sample. In practice, most implementations use 16-32 as a balance between advantage estimation quality and compute cost.

What hardware do I need to train a reasoning model with GRPO?

Since GRPO eliminates the critic network, you need significantly less memory than PPO. For a 7B parameter model, you can train with GRPO on a single node with 4xA100 (80GB) GPUs using DeepSpeed ZeRO-3 or FSDP. Only the policy and reference models are needed as neural networks, and the reference can be offloaded or shared via parameter-efficient techniques. For generation, use vLLM or similar engines to batch the G samples efficiently. Open-source frameworks like TRL and OpenRLHF already support GRPO out of the box.