DPO and Its Family: IPO, KTO, ORPO and SimPO

By the end of this lesson you will be able to read the direct preference optimisation loss and say what each symbol does; explain what the reference model is for and what beta trades off; pick between DPO, IPO, KTO, ORPO and SimPO from the shape of your data and the failure you are trying to avoid; set the four hyperparameters that matter with a reason for each; and recognise verbosity, drift and over-optimisation in a training log before they reach a model you ship.

Everything here is checked against TRL TRL 1.12.0 · verified 2026-09-08, whose trainer is what the lab runs. Where a version changes a default, the page says so rather than letting you find out.

The objective, read out loud

DPO trains on triples of a prompt, a preferred completion and a dispreferred one. For each triple it computes four numbers: the log-probability the policy assigns to the preferred completion, the same under a frozen reference model, and the same two for the dispreferred one.

Subtract within each completion. That difference, scaled by beta, is the implicit reward: how much more likely this completion has become relative to where it started. TRL logs exactly these, as rewards/chosen and rewards/rejected, each defined as beta times the log ratio of the policy to the reference.

Then subtract across the two completions and push the result through a log-sigmoid. That is the whole loss. TRL’s own description is the clearest one-sentence version: “The objective directly optimizes the model to widen the margin between the log-likelihoods of preferred and dispreferred completions, relative to a reference model, without requiring an explicit reward model.”

One DPO training step

1. 1Take one tripleA prompt, a chosen completion, a rejected completion. Nothing is generated: the completions are already in the dataset.
2. 2Four forward passesChosen and rejected, through the policy and through the reference. When ref_model is None, TRL documents the reference as the initial policy: the model exactly as it was before DPO training started.
3. 3Two implicit rewardsbeta times the log ratio of policy to reference, once for chosen and once for rejected. These are what TRL logs as rewards/chosen and rewards/rejected.
4. 4One marginchosen minus rejected. The loss is minus the log-sigmoid of that margin, so a positive margin is a small loss.
5. 5Backward and stepThe gradient widens the margin. No sampling, no reward model, no reinforcement-learning loop anywhere in the step.

Why DPO trains on a laptop where PPO does not: the expensive parts of the classical pipeline are all absent from this list.

!Important

The loss cares about the margin, not about either side on its own. TRL is explicit about the usual mechanism: “In practice, this is typically achieved by suppressing the likelihood of dispreferred completions rather than by increasing the likelihood of preferred ones.” So it is normal for logps/chosen to fall during a healthy run. What must rise is rewards/margins, and what must not collapse is the model’s ability to produce anything at all.

The reference model and beta

The reference is a frozen snapshot of the model you started from. TRL’s DPOTrainer takes it as an argument, and if you pass None “the trainer will automatically use the initial policy corresponding to model, i.e. the model state before DPO training starts”. Train through a LoRA adapter and the starting weights are recoverable by switching the adapter off, which is what TRL’s compatibility notes point at when they mention “PEFT models that do not keep a standalone ref_model”. Check the peak memory your own run reports rather than assuming either way.

Beta scales the log ratio before the sigmoid. TRL documents it as the “parameter controlling the deviation from the reference model. Higher β means less deviation”, with a default of 0.1.

Reading that as a dial: at a large beta a small change in log-probability already produces a large margin, so the loss is satisfied early and the model barely moves. At a small beta the model must move a long way to register any margin at all, and it will, taking its fluency with it. The lab starts at the default and asks you to try one value either side, because the right beta depends on how different your chosen and rejected completions are from each other.

A DPO run on a 16 GB machine: Qwen3-1.7B with a rank-16 LoRA adapter, pairs up to 1,024 tokens

Base weights, BF16, shared by policy and reference

3.4 GB

Adapter, gradients and Adam states

0.3 GB

Activations and logits, four forward passes

2 GB

Reserved for the operating system

2 GB

Free

8.3 GB

Total

16 GB

Estimate from arithmetic, not a measurement. The base size is the BF16 figure for Qwen3-1.7B in the course model reference; the adapter term is Part 11's 16 bytes per trainable parameter applied to an adapter of about twenty million parameters; the activation figure is an allowance for four forward passes rather than a measured peak. No second copy of the weights is shown, because TRL's compatibility notes speak of 'PEFT models that do not keep a standalone ref_model', which is the adapter-disabled reference this bar assumes; confirm it against the peak your own run reports, and add another 3.4 GB if you pass an explicit reference model or fine-tune fully.

▤Memory budget

If the reference model does not fit, TRL offers precompute_ref_log_probs, documented as scoring “the reference model log probabilities for the entire training dataset before training”, which “allows to save memory during training, as the reference model does not need to be kept in memory”. It costs one pass over the dataset and it is incompatible with sync_ref_model and with iterable datasets. On a tight machine it is the first thing to reach for.

The variants, and the problem each one was written for

Every method below started as an objection to DPO. Read the table as a list of objections.

Method	The objection	What it changes	Data it needs	In TRL TRL 1.12.0 · verified 2026-09-08
DPO	—	The baseline: log-sigmoid of the beta-scaled margin against a reference	Paired preference	DPOTrainer, loss_type="sigmoid"
IPO	DPO still assumes pairwise preferences can stand in for pointwise rewards, and the logit transform can overfit	Replaces the transform with the identity, giving a bounded objective that does not chase certainty	Paired preference	loss_type="ipo"; beta is the paper’s regularisation parameter τ
KTO	Paired data is scarce and expensive; people are loss-averse and the objective should say so	Maximises a Kahneman-Tversky utility from a binary desirable or undesirable signal, weighting the two sides asymmetrically	Unpaired preference (or paired, converted automatically)	KTOTrainer, with desirable_weight and undesirable_weight
ORPO	Preference tuning is a second phase after fine-tuning, and it needs a reference model	Appends a log odds ratio term to the ordinary fine-tuning loss, so one phase does both, with no reference model	Paired preference	trl.experimental.orpo.ORPOTrainer; the paper’s λ is the config’s beta
SimPO	The implicit reward is not the quantity the model generates with, and length leaks into it	Uses the average log-probability of the sequence as the reward, dropping the reference model, and adds a target reward margin	Paired preference	No separate trainer; DPO exposes the length-normalised loss as loss_type="sigmoid_norm"

The papers state their own cases compactly. IPO’s authors set out to bypass two approximations, noting that DPO “bypasses the second approximation and learn[s] directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation”, and derive an objective “expressed in terms of pairwise preferences and therefore bypass[ing] both approximations”.

KTO’s argument is about data supply: it “matches or exceeds the performance of preference-based methods at scales from 1B to 30B, despite only learning from a binary signal of whether an output is desirable”, and the abstract closes with a warning the whole table should be read through: “there is no one HALO that is universally superior; the best loss depends on the inductive biases most appropriate for a given setting”.

ORPO’s contribution is structural: a “reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase”, built on the observation “that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT”.

SimPO’s is about what the reward should be: “using the average log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient”, with a target reward margin added “to encourage a larger margin between the winning and losing responses”.

⌚Version note

TRL TRL 1.12.0 · verified 2026-09-08 ships fifteen documented loss_type values for DPOTrainer and lets you combine several with loss_weights, which is how mixed objectives such as MPO are built. Its ORPO trainer lives under trl.experimental.orpo, and its documentation marks experimental features as liable to change. Read from the TRL documentation on 2026-09-09; check the trainer page for your version before you copy an argument from anywhere, including from here.

Choosing one

Three questions settle it in almost every case.

Do you have pairs? If your data is “this output was fine, this one was not”, with no matched comparison, that is unpaired preference and KTO is the trainer that reads it. TRL notes that KTO will also accept a paired set and split it for you, assigning label = True to the chosen side.

Are you fine-tuning anyway? If you are about to run supervised fine-tuning and then preference tuning on the same data, ORPO does both in one pass and drops the reference model. That is a real saving of an afternoon, at the cost of using a trainer TRL marks experimental.

Is your problem length? If the honest description of your current model is “it is right and it will not shut up”, the length-normalised loss is the targeted fix, and it is one argument away.

For a first preference run on a few hundred pairs of your own, plain DPO at the default beta is the right starting point, because it is the one every other method is described relative to.

The four hyperparameters that matter

Beta, default 0.1. Start there. If rewards/margins barely moves, lower it; if the samples start degrading, raise it.

Learning rate. TRL’s DPO default is 1e-6, which is deliberately far below a fine-tuning rate, because preference tuning moves an already-good model a short distance. For adapters the documentation suggests otherwise: “When training adapters, you typically use a higher learning rate (≈1e-5) than full fine-tuning since only new parameters are being learned.” The lab uses 1e-5 with a LoRA adapter for that reason and asks you to try 1e-6 as well.

Epochs. One, usually. Preference sets are small and DPO over-optimises quickly; the run whose evaluation loss turns upward in the second epoch is the common case, not the unlucky one.

max_length, default 1024, with truncation_mode documented as supporting only "keep_start". A pair whose completions are cut in half is a pair about the first half. If your answers are long, raise it and pay the memory, or shorten the answers.

✕Common mistake

KTO’s hyperparameters are not DPO’s and the documentation says so twice. Learning rate: “for the default setting of beta = 0.1, the learning rate should typically not exceed 1e-6”, and “we strongly recommend keeping the learning rate between 5e-7 and 5e-6”. Batch: “use a per-step batch size that is at least 4, and an effective batch size between 16 and 128”, because KTO estimates its KL term by pairing each prompt with a mismatched completion from the same batch. And if your desirable and undesirable counts are unbalanced, upweight the rarer side so that the weighted ratio lands “in the range 1:1 to 4:3”.

Three ways it goes wrong

Verbosity. The best-documented failure. Longer answers are preferred often enough that the margin can be widened by writing more, and length is easier to change than quality. Measure mean answer length before and after every preference run. If it grew and nothing else did, that is your result.

Drift. The model becomes fluent at the tuned behaviour and worse at things nobody was comparing: arithmetic, instruction following, the format your fine-tune installed. This is the same catastrophic forgetting Part 13 measures, arriving through a different door, and it is invisible unless you run your Part 10 task set afterwards. The lab makes that a required step rather than a suggestion.

Over-optimisation. Training loss falls, rewards/margins climbs, rewards/accuracies approaches one, and the samples are worse. What has happened is that the model has found the cheapest way to widen the margin on this dataset, which need not be the behaviour you were trying to describe. The defence is a held-out split, an evaluation on prompts the pairs never contained, and reading a dozen completions with your own eyes.

=Expected result

A healthy DPO run on a few hundred pairs looks like this: training loss falls from about 0.69, which is minus the log of a half and where a model with no preference starts; rewards/accuracies rises above a half and settles; rewards/margins grows steadily; logps/chosen may fall, and that is fine; evaluation loss falls for the first epoch. If evaluation loss turns up while training loss keeps falling, stop and keep the earlier checkpoint.

Interpret the preference margin relative to the reference

DPO compares the policy’s relative log probability of chosen and rejected answers with the same relative quantity under a reference model. The reference matters because a chosen answer may already be much more likely before training. The objective concerns a change in preference, not simply the chosen answer’s raw likelihood.

For a schematic example, suppose policy and reference give exactly the same chosen-versus-rejected log-probability difference. The relative margin is zero, and the sigmoid comparison is at its midpoint. Increasing the policy’s chosen-versus-rejected difference moves the preference objective in the desired direction for that pair. This does not require the absolute probability of the chosen answer to increase on every update.

Record which checkpoint defines the frozen reference, especially when starting from an SFT adapter. Changing that reference changes the experiment. Inspect held-out preference accuracy, response length, task correctness and regressions together. A larger training margin can reflect overfitting, length bias or an inconsistent dataset. Beta and loss variants are settings to evaluate under the same data contract, not universal quality controls.

Recap

DPO computes an implicit reward for each completion as beta times its log-probability ratio against a frozen reference, and minimises the log-sigmoid of the margin between chosen and rejected; there is no reward model and no sampling in the step. The reference is free when you train an adapter, because switching the adapter off recovers it. Beta trades movement against fidelity, and the usual mechanism of the loss is to suppress the rejected side rather than to raise the chosen one, so falling chosen log-probabilities are not a fault. IPO replaces the logit transform to stop the objective chasing certainty, KTO learns from unpaired binary labels with asymmetric weights, ORPO folds preference into fine-tuning and drops the reference, and SimPO’s length-normalised reward is available in TRL as a DPO loss type. The three failures to watch for are verbosity, drift on everything you did not measure, and over-optimisation of the pairs you have.

Check your understanding

Question 1. During a DPO run, logps/chosen falls steadily while rewards/margins rises. What should you conclude?

• The run is broken and should be restarted
• This is the documented normal mechanism: the loss widens the margin, usually by suppressing the rejected completion, and the margin is what matters
• The learning rate is too high
• The reference model was not loaded

Show the answer and why

Answer: This is the documented normal mechanism: the loss widens the margin, usually by suppressing the rejected completion, and the margin is what matters

TRL states that the objective is "typically achieved by suppressing the likelihood of dispreferred completions rather than by increasing the likelihood of preferred ones". Watch the margin and the samples; a falling chosen log-probability on its own is not a symptom.

Question 2. You raise beta from 0.1 to 0.5 and the run barely changes the model. Why?

• Beta is the learning rate and you lowered it
• Beta scales the log ratio, so a higher value produces a large margin from a small movement; the loss is satisfied before the model has moved far from the reference
• Beta controls the batch size
• Higher beta disables the reference model

Show the answer and why

Answer: Beta scales the log ratio, so a higher value produces a large margin from a small movement; the loss is satisfied before the model has moved far from the reference

TRL documents beta as controlling deviation from the reference, with higher meaning less deviation. It is the leash length, and setting it long or short is the main decision a preference run asks of you.

Question 3. Your feedback data is a log of single answers each marked good or bad, with no matched comparisons. Which trainer reads it directly?

• DPOTrainer
• KTOTrainer, which expects an unpaired preference dataset of prompt, completion and a boolean label
• GRPOTrainer
• SFTTrainer

Show the answer and why

Answer: KTOTrainer, which expects an unpaired preference dataset of prompt, completion and a boolean label

That is exactly the shape KTO was written for, and its paper makes the practical case: a binary signal is far easier to collect than matched pairs. DPO would need you to invent pairings you do not have.

Question 4. Which of these are true of ORPO as its paper and TRL describe it? Select all that apply.

• It needs no reference model
• It combines preference optimisation with the supervised fine-tuning loss in one phase
• It uses an odds ratio term to contrast the favoured and disfavoured styles
• It requires an unpaired preference dataset

Show the answer and why

Answer: It needs no reference model, It combines preference optimisation with the supervised fine-tuning loss in one phase, It uses an odds ratio term to contrast the favoured and disfavoured styles

ORPO is described as a "reference model-free monolithic odds ratio preference optimization algorithm... eliminating the necessity for an additional preference alignment phase". It takes a paired preference dataset, like DPO.

Question 5. After a preference run the model writes in your style, the judge prefers it, and its mean answer length has doubled. What is the responsible next step?

• Ship it; the judge preferred it
• Record the length change, then run the Part 10 task set through both models, because verbosity is the best-documented way for a preference run to win a comparison without being better
• Increase the learning rate and run again
• Delete the reference model

Show the answer and why

Answer: Record the length change, then run the Part 10 task set through both models, because verbosity is the best-documented way for a preference run to win a comparison without being better

Length is the classic confound in both human and model judgements. A preference win with a doubled length is one measurement short of a result, and the missing measurement is what the tuned model now does on everything else.

Check my answersClear