Reinforcement Learning with Verifiable Rewards: GRPO Explained

By the end of this lesson you will be able to describe one GRPO step in order and say what each piece costs; explain why sampling a group removes the need for a critic network; say what the KL term is measured against and when it is switched off entirely; state the practical difference between a reward a program computes and one a network predicts; read the DAPO, Dr. GRPO and GSPO refinements as four or five specific changes to one objective; and say what reinforcement learning of this kind cannot do, with the mechanical reason rather than a caution.

The loop

Start from the thing being optimised. The model is a policy: given a prompt, it produces a completion, one token at a time. A rollout is one such completion, sampled rather than greedily decoded, so that the same prompt produces different attempts.

One GRPO step

1. 1Take a promptFrom a prompt-only dataset. There is no reference answer in the training data - only the problem, and whatever extra columns the reward function needs.
2. 2Sample G rolloutsG completions to the same prompt, at a temperature above zero so they differ. TRL calls this num_generations and defaults it to 8. This is where the wall clock goes.
3. 3Score each rolloutOne number per rollout from the reward function or functions. For a maths task this is a comparison against the known answer; nothing is learned and nothing is judged.
4. 4Compute group-relative advantagesSubtract the group mean from each reward, and by default divide by the group standard deviation. A rollout better than its siblings gets a positive advantage; one worse gets a negative one.
5. 5Update the policyA clipped surrogate objective, as in PPO: raise the probability of tokens in above-average rollouts, lower it for below-average ones, with the ratio to the sampling policy clipped so no single step moves too far.
6. 6Subtract the KL penaltyA term pulling the policy back towards a frozen reference model, weighted by beta. In TRL beta defaults to 0.0, which skips the reference model altogether.

Two of these six boxes are new relative to supervised fine-tuning: sampling inside the training loop, and scoring what was sampled. Everything else is machinery you have already met.

The advantage formula is worth writing out, because it is the whole idea:

Pseudocode — not a real command

for each prompt:
    rewards   = [ reward(rollout) for rollout in group ]
    advantage = (reward - mean(rewards)) / std(rewards)      # one per rollout

That is what “group relative” means. The comparison is not against an absolute standard, or against a learned estimate of how good this prompt usually goes. It is against the model’s own other attempts at the same prompt, sampled seconds earlier.

Why there is no critic

PPO in its usual form needs a value network: a second model, the critic, trained alongside the policy to predict the expected return from a partial completion. Its output is subtracted from the observed reward to give the advantage, which is what turns “this got a reward of 1” into “this got a reward of 1 where 0.6 was expected, so reinforce it”.

A critic is a whole extra model to hold, train and get wrong. GRPO’s paper introduces the method as “a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO”, and the memory saving is exactly this: the baseline the critic supplied is estimated from the group instead.

The trade is arithmetic. A critic gives you a baseline from one rollout. A group gives you a baseline from G rollouts, which is G times the generation cost and no extra parameters. On a single machine, where generation is fast enough and a fourth resident model is not, that trade is overwhelmingly worth making. The DeepSeek-R1 abstract describes what the recipe produced: “the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories.”

!Important

A group whose rewards are all identical has a mean equal to every member and a spread of zero. Every advantage is zero and the step teaches nothing. This is not an edge case: on a problem the model always solves, or never solves, that is precisely what happens. Half the refinements later in this lesson exist because of it.

The reference model and the KL term

The KL term measures how far the policy has moved from a frozen reference, usually the model you started from, and subtracts a multiple of it from the objective. TRL estimates it per token with the standard low-variance estimator and logs it as kl, “the average KL divergence between the model and the reference model, calculated over generated completions”, logged “only if beta is nonzero”.

That last clause is the practical fact. In TRL TRL 1.12.0 · verified 2026-09-08, beta defaults to 0.0, documented as “KL divergence coefficient (0.0 = reference model not loaded)”. So the default GRPO run has no reference model in memory and no leash at all, and the only thing keeping the policy sane is the clipping in the surrogate objective and the fact that runs are short.

Whether that is right depends on what your reward measures. Against a verifiable reward on a narrow task, a small beta or none is common, because the reward cannot be satisfied by gibberish: a wrong answer is wrong however fluently it is written. Against a learned reward model, dropping the KL term is how you get a policy that has discovered the reward model’s blind spots.

▤Memory budget

Setting beta above zero is a memory decision as well as a training one: it loads a second copy of the model. On a 16 GB machine training a 1.7 billion parameter model in BF16, that is another 3.4 GB against a budget that also has to hold the rollouts. This part’s toolchain lesson has the diagrams.

Verifiable rewards versus learned ones

A verifiable reward is a program. Did the final number match? Did the unit tests pass? Did the output parse as JSON with the required keys? It returns the same value for the same completion every time, it costs microseconds, and it cannot be flattered.

A learned reward is a network trained on human comparisons. It can express “is this a good explanation”, which no program can, and it is wrong in ways an optimiser will find, which is the reward-hacking result from this part’s first lesson.

The difference decides what the whole run is. With a verifiable reward you need problems and a checker, and no labelling at all. With a learned reward you need comparisons, a reward model trained on them, and a KL term doing real work. This course teaches the verifiable case, because it is the one that fits on one machine and the one whose result you can trust without a second experiment.

✕Common mistake

“Verifiable” is a property of the checker, not of the subject. A maths answer checked by string comparison against 18 is not verifiable if the model writes eighteen, and a code task checked by tests is only as verifiable as the tests are complete. The next lesson is entirely about writing checkers that mean what you think they mean.

Where the time goes

Generation dominates. TRL’s vLLM page says it plainly: “Online methods generate completions during training, and generating them with the model’s own generate is the bottleneck.” One step samples G completions of up to several hundred tokens each, for every prompt in the batch, before a single gradient is computed.

That is why the rollout engine is a component with its own architecture rather than an implementation detail.

One training step, four ways to produce the rollouts

Trainer generate()

Sample G rolloutsScoreForward and backward

vLLM, colocate

Sample G rolloutsScoreForward and backwardWeights refreshed in place

vLLM, server

Sample G rolloutsScoreForward and backwardWeights streamed over NCCL

llama-server

Sample G rolloutsScoreForward and backwardNo documented weight update

A schematic of which phase dominates and what closes the loop, not a measurement: the widths are illustrative and the labs record real wall-clock per track. The last row is the one to read carefully. TRL documents that after each optimiser step the trainer 'streams the updated weights into' the vLLM server; llama.cpp's server README documents no equivalent, so a llama-server rollout path samples from a policy that stops matching the one being trained.

TRL’s server mode is worth one more sentence because it explains a warning you will meet. The trainer “asks for completions on the OpenAI-compatible /v1/completions endpoint, sending the prompt token IDs”, and after each optimiser step it “streams the updated weights into it over NCCL”. The documentation also warns that “the vLLM server and the trainer must run on separate CUDA devices to prevent conflicts”, and lists an ipc weight-transfer backend to “use instead when the trainer and the server share a GPU”. Colocate mode, where vLLM runs inside the trainer process, is the default and the simpler answer on one accelerator.

The refinements, and what each one changes

Three papers refine the objective in ways you will meet as configuration values. Read them as bug reports against the formula above.

Refinement	What it says is wrong with GRPO	What it changes	Where it appears in TRL TRL 1.12.0 · verified 2026-09-08
DAPO, “Decoupled Clip and Dynamic sAmpling Policy Optimization”	Symmetric clipping suppresses low-probability tokens, so the policy loses diversity; batches fill with useless groups; long sequences are under-weighted; truncated answers are punished for being cut off	Four changes: clip-higher decouples the clipping bounds, using a larger upper epsilon; dynamic sampling “filters out prompts with the accuracy equal to 1 and 0” so every group has a gradient; token-level policy gradient loss computes the objective per token rather than per sample; overlong reward shaping replaces a flat punishment for truncation with a length-aware penalty	loss_type="dapo" is the default; epsilon_high is documented as “Paper DAPO recommends 0.28”
Dr. GRPO, from “Understanding R1-Zero-Like Training”	Two normalisation terms bias the objective. Dividing by response length favours short correct answers and under-penalises long wrong ones; dividing by the group’s standard deviation over-weights questions that are nearly always or nearly never solved	Removes both terms, recovering an unbiased objective. The paper reports it as “an unbiased optimization method that improves token efficiency while maintaining reasoning performance”	loss_type="dr_grpo"; the standard-deviation term is separately controlled by scale_rewards, which accepts "group", "batch" or False
GSPO, Group Sequence Policy Optimization	The importance ratio is computed per token, which is noisy, and the noise is worst where routing changes between steps	“Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization”; the paper reports that it “notably stabilizes Mixture-of-Experts (MoE) RL training”	An importance_sampling_level of "sequence" appears in the documented definition of the clip_ratio metrics

The Dr. GRPO finding deserves its own sentence, because it explains something readers see and misread. The paper identifies an “optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training”. So if your completion length climbs during a GRPO run, the first hypothesis is not “the model is learning to think for longer”. It is that the objective rewards length directly, for reasons that have nothing to do with the task.

DAPO’s own headline is in its abstract, and the paper reports 50 points on AIME 2024 from a Qwen2.5-32B base model, along with the full open-sourcing of the system that produced it. That is their measurement on their hardware, quoted as reported rather than reproduced here.

Reward hacking, at this scale

With a verifiable reward the classic failures are not about the model outwitting a network. They are about the checker being narrower than the task.

• Answer spraying. The checker takes the last number in the completion, so the model lists every plausible number and lets the extractor find one.
• Format farming. A format reward is worth almost as much as correctness, so the model produces immaculately formatted wrong answers, which is cheaper than being right.
• Length inflation. Encouraged by the objective’s own bias, and often by a reward that pays for visible working.
• Truncation gaming. If a cut-off answer scores zero and a short wrong answer scores zero, but a short wrong answer costs fewer tokens, the model learns to stop early.

Every one of these is visible in the samples and invisible in the reward curve, which is why both labs in this part make you print completions.

What reinforcement learning changes, and what it cannot

It changes the probability of behaviours the model already produces. That is the whole mechanism: sample, compare, reinforce what was better. Nothing in the loop introduces a behaviour from outside.

The consequence is sharp and mechanical. If the model never samples a correct answer to a problem, every reward in that group is zero, every advantage is zero, and the gradient from that prompt is zero. Reinforcement learning cannot teach a capability that never appears in a rollout; it can only raise the frequency of one that appears sometimes. This is why DAPO filters out groups with an accuracy of exactly 0 or 1, why the choice of training problems matters as much as the reward, and why a 1.7 billion parameter model trained on competition mathematics learns very little.

It also does not add knowledge, for the same reason preference tuning does not. And it is expensive per unit of change: G rollouts per prompt per step against one forward pass in supervised fine-tuning. Part 15 teaches the alternative, which is to have a larger model produce the correct rollouts and train on those directly, and the reality check at the end of this part runs both on the same budget and compares them.

Work a group-relative advantage by hand

Consider four completions for one prompt with rewards 0, 0, 1, 1. Their mean reward is one half. Subtracting that mean yields centred values −0.5, −0.5, 0.5, 0.5; normalisation then depends on the configured standard-deviation convention and stabilising epsilon. The positive completions are reinforced relative to their group, while the negative ones are discouraged.

Now consider rewards 0, 0, 0, 0. There is no within-group reward distinction to teach from under this centred advantage construction. A trainer can still have other loss terms, but the task-reward contrast supplies no direction. The same issue arises when every completion succeeds. Track the distribution of all-fail, mixed and all-pass groups rather than only mean reward.

This explains why task difficulty and the number of sampled completions matter. A verifier that is too permissive or a task the policy never solves can produce uninformative groups. Inspect actual completions and verifier decisions before raising rollout counts. More generation is useful only if it supplies a trustworthy learning signal within your compute budget.

Recap

GRPO samples a group of completions per prompt, scores each with a reward function, and uses the group’s mean and spread as the baseline a critic network would otherwise supply, which is where its memory saving comes from. A KL term to a frozen reference is optional and off by default in TRL, which is defensible against a verifiable reward and dangerous against a learned one. Generation dominates the step, which is why the rollout engine matters and why weight synchronisation between trainer and engine is a real constraint. DAPO decouples the clipping bounds, filters degenerate groups, moves the loss to token level and shapes the penalty for overlong answers; Dr. GRPO removes the length and standard-deviation normalisations that bias the objective towards longer wrong answers and easy questions; GSPO moves the importance ratio from tokens to sequences. Reward hacking here is the checker being narrower than the task. And the method can only amplify behaviour the model already samples, which is the sentence to keep.

Check your understanding

Question 1. Why does sampling a group of completions per prompt remove the need for a critic network?

• Because the group is used as training data instead
• Because the mean reward across the group is the baseline the critic existed to estimate, so the advantage can be computed directly from the samples
• Because the critic is replaced by the reference model
• Because clipping makes a baseline unnecessary

Show the answer and why

Answer: Because the mean reward across the group is the baseline the critic existed to estimate, so the advantage can be computed directly from the samples

A critic predicts "how good should this have been". A group of siblings measures it. That is the trade at the heart of GRPO: G times the generation cost, and one fewer model to hold and train.

Question 2. A GRPO run on 200 problems shows a reward curve that never moves off zero. What should you check first?

• The learning rate
• Whether the model ever produces a correct rollout at all: if every reward in every group is zero, every advantage is zero and there is no gradient
• The KL coefficient
• The number of epochs

Show the answer and why

Answer: Whether the model ever produces a correct rollout at all: if every reward in every group is zero, every advantage is zero and there is no gradient

This is the mechanical limit of the method. Reinforcement learning raises the frequency of behaviour that already appears; a task the model never solves supplies no signal at all. Easier problems, a bigger model, or partial credit in the reward are the three ways out.

Question 3. Completion length climbs steadily through a GRPO run. Which explanation should you consider first?

• The model has learned to reason at greater length
• The GRPO objective has a documented optimisation bias that artificially increases response length, especially for incorrect outputs, which Dr. GRPO identifies and removes
• The temperature is too high
• The reward function is broken

Show the answer and why

Answer: The GRPO objective has a documented optimisation bias that artificially increases response length, especially for incorrect outputs, which Dr. GRPO identifies and removes

Length going up is the expected behaviour of the unmodified objective, not evidence about reasoning. The way to find out which you have is to hold length constant and see whether accuracy on held-out problems moved.

Question 4. In TRL, beta defaults to 0.0 for GRPO. What does that mean in practice? Select all that apply.

• No reference model is loaded, so a copy of the model weights is not held in memory
• The kl metric is not logged
• Nothing constrains how far the policy moves except the clipping in the surrogate objective
• The reward function is disabled

Show the answer and why

Answer: No reference model is loaded, so a copy of the model weights is not held in memory, The kl metric is not logged, Nothing constrains how far the policy moves except the clipping in the surrogate objective

It is a memory saving and a loosened constraint at the same time. Against a verifiable reward that is often fine; against a learned reward model it is how you end up optimising the reward model's mistakes.

Question 5. What does DAPO's dynamic sampling do, and why?

• It samples more completions for harder prompts
• It filters out prompts whose group accuracy is exactly 1 or exactly 0, because those groups have zero spread and therefore contribute no gradient
• It reduces the temperature over time
• It selects prompts at random rather than in order

Show the answer and why

Answer: It filters out prompts whose group accuracy is exactly 1 or exactly 0, because those groups have zero spread and therefore contribute no gradient

A group where every rollout scores the same has identical advantages of zero. Filtering those keeps the batch full of prompts that can still teach something, which matters more as the model gets better and more groups become all-correct.

Check my answersClear