From Imitation to Preferences: Why Ranking Beats Copying

By the end of this lesson you will be able to say exactly what supervised fine-tuning cannot learn and why a comparison fixes it; describe the classical preference pipeline as four named stages with a reward model in the middle; explain what the KL term is holding the model to and what happens at both extremes; recognise the three dataset shapes preference methods consume; and place PPO, DPO and GRPO on one family tree so that a paper’s method section stops being a wall of symbols.

Part 3 told this story once, at the level of “what happened and in what order”. This lesson is the same story with the mechanism visible, because from here on you are the one setting the hyperparameters.

Where imitation stops

Supervised fine-tuning computes one thing: the negative log-likelihood of the tokens you supplied, given the prompt. Every example is a target to be reproduced. The training signal never contains the sentence “this answer is better than that one”, because only one answer was ever present.

Three consequences follow, and all three show up in real fine-tunes.

It cannot exceed its data. If the demonstrations hedge, the fine-tune hedges. If they are merely adequate, the fine-tune is merely adequate. There is no gradient pushing towards anything better than what you wrote down, because nothing in the loss represents “better”.

It cannot learn what to avoid. A demonstration says “do this”. It never says “and not that”. A model that habitually pads its answers with a closing offer of further help will keep doing so after a fine-tune on examples that simply lack the padding, because absence is a weak signal compared with presence.

The data is expensive in the one place it matters. Writing an excellent answer to a hard question needs somebody who could have answered it. Collecting a thousand of those is a project. This is not a machine-learning limitation at all; it is a labour cost, and it is the one that preference methods attack.

!Important

The fix is not “more data”. A hundred thousand mediocre demonstrations produce a confident mediocre model. The fix is a different kind of data, one whose unit is a comparison rather than an example.

What a ranking knows that a demonstration does not

Show two answers to the same prompt and ask which is better. The answer is one bit, it takes a few seconds, and the person giving it does not have to be able to write either answer. That is the whole economic argument, and Part 3 quoted the InstructGPT paper making it at scale: the authors collected demonstrations first, then “a dataset of rankings of model outputs”, and used the second to push the model further than the first could.

The technical argument is different and more interesting. A comparison is a statement about the model’s own output distribution. Both answers came from the model; one of them is a direction it already goes in and should go in more, the other a direction it should go in less. A demonstration is a point you supplied from outside, which the model may or may not be able to reach. A preference pair is a gradient defined on ground the model already occupies.

That is why preference tuning tends to move behaviour that fine-tuning could not budge: verbosity, hedging, register, the shape of a refusal, the habit of answering the question that was asked. All of them are differences between two plausible outputs rather than the difference between an output and an ideal.

The three shapes preference data comes in

TRL’s dataset documentation names the shapes, and they matter because each trainer accepts a different one. A preference dataset carries a prompt with a chosen and a rejected completion. An unpaired preference dataset carries a prompt, a single completion and a boolean label saying whether it was desirable. A prompt-only dataset carries the prompt and nothing else, because the completions will be generated during training.

Those three shapes are the fault line running through the rest of this part. Paired preferences feed DPO. Unpaired ones feed KTO. Prompt-only feeds everything that samples from the model as it trains, which is PPO, GRPO and the online methods. If you find yourself unable to run an algorithm on a dataset, this is almost always why.

✕Common mistake

Converting a paired preference set into an unpaired one by splitting it is documented and comes with a warning worth repeating: TRL notes that the chosen and rejected completions “can be both good or bad”, so before you relabel chosen as desirable you have to be sure it actually was. A pair says “A beat B”. It does not say A was any good.

The classical pipeline

Reinforcement learning from human feedback is four stages, and naming them is most of understanding it.

Reinforcement learning from human feedback, stage by stage

1. 11. Supervised fine-tuningTrain on demonstrations. Produces the policy that everything after this starts from, and the reference the KL term is measured against.
2. 22. Collect comparisonsSample two or more completions per prompt from that policy, and have people rank them. The completions come from the model so that the labels describe the model.
3. 33. Train a reward modelA model that reads a prompt and a completion and returns one number, fitted so that the preferred completion in each pair scores higher. Usually the policy with its output head replaced.
4. 44. Optimise the policy against itSample completions, score them with the reward model, and update the policy towards what scored well - with a KL penalty holding it near the stage-1 model.

Stages 3 and 4 are what DPO removes and what GRPO replaces. Everything in this part is a variation on which of these four boxes you keep.

The reward model is a model, with all that implies

Stage 3 is the part people skip over, and it is where most of the trouble lives. A reward model is trained on comparisons, usually under the Bradley-Terry assumption that the probability of preferring one completion over another is a logistic function of the difference between their scores. Fit that on your pairs and you have a function from text to a number.

It is a neural network trained on a finite sample, so it is wrong in places, and the places it is wrong are exactly where an optimiser will go. This has a formal name. Skalse and colleagues give “the first formal definition of reward hacking, a phenomenon where optimizing an imperfect proxy reward function leads to poor performance according to the true reward function”, and their analysis is not encouraging: they define an “unhackable” proxy as one where “increasing the expected proxy return can never decrease the expected true return”, and find that for all stochastic policies this holds only in degenerate cases, concluding that “the linearity of reward (in state-action visit counts) makes unhackability a very strong condition”.

Read plainly: a learned reward model that you optimise hard enough will be exploited. Not through bad luck, but because the optimiser is doing its job and the proxy is not the thing you wanted.

PPO, in one paragraph

Stage 4 is usually proximal policy optimisation. Its paper describes “a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a ‘surrogate’ objective function using stochastic gradient ascent”, and the reason it displaced what came before is stated just as plainly: “whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates”.

Two words in that quotation carry the practical weight. Alternate means the loop is sample-then-update, so generation happens inside training and dominates the wall clock. Surrogate means the objective is not the reward directly but a clipped ratio between the new policy and the one that did the sampling, which is what makes several updates on the same batch safe.

For language models this costs four models in memory at once: the policy being trained, a frozen reference, the reward model, and a value network (the critic) that estimates how good a partial completion is. That count is the single biggest reason PPO is rare on one machine, and it is the count GRPO attacks.

The leash: what the KL term holds

Every method in this part carries a term that measures how far the policy has moved from a frozen reference model, and penalises it. It is worth being precise about why.

Optimising a score with no constraint finds the maximum of that score, which for any learned or hand-written reward is some degenerate output: a wall of the reward model’s favourite tokens, or an empty answer that trivially satisfies a format check. The reference model is a statement of “stay recognisably like the thing that was already useful”. The KL coefficient, usually written beta, is how tightly.

Both extremes fail in a way you can recognise in a log:

• Too tight (large beta, or a KL penalty that dominates): reward barely moves, the completions stay almost identical to the reference, and you have spent an afternoon of GPU time on nothing.
• Too loose (small or zero beta): reward climbs beautifully and the completions degrade. Repetition, drifting language, answers that satisfy the letter of the reward and nothing else. The reward curve is not evidence that this is not happening, which is why the labs in this part make you look at samples.

∴Research note

TRL logs the implicit reward for chosen and rejected completions, their margin, and rewards/accuracies, “the proportion of examples where the implicit reward for the chosen completion is higher than that for the rejected completion”. Those four numbers, read together with the loss, are how you tell a preference run that is learning from one that is drifting. The DPO lab has you write all of them down.

The family tree

Everything after the classical pipeline is an argument about which of the four stages you can delete.

Method	Keeps	Deletes	Data it needs	Cost on one machine
PPO	All four stages	Nothing	Prompt-only, plus a trained reward model	Four models resident; generation inside the loop
DPO	Stages 1 and 2	The reward model and the whole reinforcement-learning loop	Paired preferences, offline	One trainable model and a reference; no sampling during training
GRPO	Stages 1, 2 and 4	The critic network, and the reward model where the reward is a program	Prompt-only, plus a reward function	Policy, optional reference, and a rollout engine

DPO’s route is the one Part 3 described: its authors introduce “a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss”. The reward model does not vanish so much as get folded into the policy, which is what the paper’s subtitle means by “your language model is secretly a reward model”. The next lesson makes that concrete.

GRPO’s route is different. Its paper introduces it as “a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO”. It keeps the sampling loop and removes the critic, by scoring a group of completions to the same prompt and using their spread as the baseline the critic used to supply. Where the reward is a program rather than a model, the reward model goes too. That is two of the four networks gone, and it is why a reinforcement-learning run fits on a desk.

What preference tuning changes, and what it does not

It moves the things a comparison can express. Tone, length, the decision to answer directly rather than restate the question, the shape of a refusal, which of two correct answers gets produced. On a narrow style you can see the change after a few hundred pairs.

It does not install knowledge, for the same reason supervised fine-tuning does not: a comparison between two answers says which you preferred, not what is true. If both answers are confidently wrong, the pair teaches the model which flavour of wrong you like.

And it has a documented tendency to make answers longer, because longer answers are frequently preferred by both people and judges. That tendency is real enough that two of the methods in the next lesson exist mainly to counter it, and the DPO lab records mean answer length before and after for exactly this reason.

A preference label needs a reason and a comparison unit

Suppose two answers are both factually correct, but one is concise and the other explains a useful caveat. Which is preferred depends on the task and reviewer policy. Label the property being judged and permit a tie or exclusion when the rubric does not distinguish the pair.

Keep the prompt identical within a pair. If the chosen answer had access to a different document or tool result, the pair confounds preference with evidence availability. Inspect length and formatting distributions too: a model can learn “longer is preferred” when reviewers intended “more complete is preferred”.

Split by the underlying prompt or source task before creating alternative answers. A held-out paraphrase of a training prompt may not establish transfer. Evaluate the resulting model on new prompts with both the preference rubric and independent correctness checks. Preference optimisation shifts relative behaviour; it does not make every chosen example objectively true or resolve inconsistent values among reviewers. The dataset’s comparison policy is part of the model’s intended behaviour.

Recap

Supervised fine-tuning can only reproduce its examples, cannot express “not that”, and needs expensive labour precisely where quality matters. A preference pair is cheaper to collect and carries a comparison between two outputs the model already produces. The classical pipeline turns those comparisons into a reward model and optimises the policy against it with PPO, holding it near the supervised model with a KL penalty; the reward model is a fitted network, so optimising it hard finds its errors, which reward hacking has a formal definition for. DPO deletes the reward model and the loop; GRPO keeps the loop, deletes the critic by scoring groups, and deletes the reward model wherever a program can compute the reward. The dataset shape - paired, unpaired or prompt-only - decides which of them you can run at all.

Check your understanding

Question 1. Why does a preference pair carry training signal that a single demonstration cannot?

• Because pairs contain twice as much text
• Because both completions came from the model, so the pair states a direction within the model's own output distribution rather than a target supplied from outside
• Because rankings are more accurate than written answers
• Because the loss function is more advanced

Show the answer and why

Answer: Because both completions came from the model, so the pair states a direction within the model's own output distribution rather than a target supplied from outside

A demonstration is a point you hope the model can reach. A pair is a comparison between two things it already produces, which is why preference tuning moves verbosity, register and hedging that fine-tuning leaves alone.

Question 2. A team optimises a policy against a trained reward model with no KL penalty. Reward climbs steadily for hours. What is the most likely explanation?

• The model is getting better at the task in proportion to the reward
• The optimiser is finding regions where the reward model is wrong, which is what the formal definition of reward hacking describes
• The learning rate is too low
• The reward model needs more parameters

Show the answer and why

Answer: The optimiser is finding regions where the reward model is wrong, which is what the formal definition of reward hacking describes

A reward model is fitted on a finite sample and is wrong in places, and an unconstrained optimiser goes to exactly those places. The reward curve measures the proxy, not the thing you wanted; the KL term and looking at samples are what keep the two connected.

Question 3. Which dataset type does each trainer expect? Select all the correct pairings.

• DPO takes a preference dataset with prompt, chosen and rejected
• KTO takes an unpaired preference dataset with prompt, completion and a boolean label
• GRPO takes a prompt-only dataset and generates the completions during training
• GRPO takes a preference dataset with chosen and rejected

Show the answer and why

Answer: DPO takes a preference dataset with prompt, chosen and rejected, KTO takes an unpaired preference dataset with prompt, completion and a boolean label, GRPO takes a prompt-only dataset and generates the completions during training

The shape of the data decides which algorithm you can run. GRPO never sees a reference completion: it samples its own and scores them, which is why it needs only prompts and a reward function.

Question 4. What does GRPO remove from PPO, and what does it use instead?

• The reference model, replaced by a larger learning rate
• The critic network, replaced by the spread of rewards across a group of completions to the same prompt
• The sampling loop, replaced by an offline dataset
• The KL penalty, replaced by gradient clipping

Show the answer and why

Answer: The critic network, replaced by the spread of rewards across a group of completions to the same prompt

The critic exists to supply a baseline that says how good a completion was relative to expectation. Sampling several completions per prompt gives you that baseline directly, which is where the memory saving in the DeepSeekMath paper comes from.

Question 5. After preference tuning on 400 pairs, the model writes in your house style and confidently states two facts about your product that are wrong. What happened?

• The beta was too high
• Preference tuning moved the behaviour the comparisons expressed; nothing in a pair says which answer is true, so if both candidates were wrong the pair only taught which wrong answer you preferred
• The dataset needed to be unpaired
• The reference model was corrupted

Show the answer and why

Answer: Preference tuning moved the behaviour the comparisons expressed; nothing in a pair says which answer is true, so if both candidates were wrong the pair only taught which wrong answer you preferred

The same boundary as supervised fine-tuning, arriving by a different route. Facts come from retrieval or from pretraining; preference data is a statement about presentation and choice among candidates.

Check my answersClear