Level 3 · Model BuilderLessonPart 15 · page 2 of 932 min

32Minutes

9Sources

Three Kinds of Distillation: Logit, Sequence and On-Policy

By the end of this lesson you will be able to write down what each of the three families actually optimises, say which one your pair of models can support, explain why forward and reverse divergence produce different students, name the trainer that implements each one and its status on a stated date, and estimate what each route will cost you before you start it. The previous lesson argued that distillation works. This one is about which kind.

One picture first: where the compute goes

The three families differ in an objective, but what you will actually notice is when the generation happens, because generation is where the hours go.

Where the time goes in one training run

Sequence-level

Generate once, offlineFilterTrain (dataset is reusable)

Logit / on-policy

genscore + stepgenscore + stepgenscore + step

GRPO, for comparison

gen x Greward + stepgen x Greward + step

Relative shape, not measured time. Sequence-level pays for generation once, up front, and the dataset it produces can be reused for every later experiment. The logit and on-policy routes generate inside the training loop, so the cost is paid again on every run. Group relative policy optimisation from Part 14 generates several completions per prompt and scores each, which is the same shape again with a wider generation block.

Keep that picture in mind through the rest of the lesson. Every difference below eventually turns into that difference.

Sequence-level: train on what the teacher wrote

The oldest and simplest of the three, and the one the long lab in this part uses. Ask the teacher a large set of questions, keep its answers, throw away the bad ones, and fine-tune the student on what is left. There is no distillation-specific machinery: the trainer is the SFTTrainer from Part 13 and the dataset is an ordinary prompt-completion file.

Kim and Rush introduced the idea for machine translation in 2016 and framed the result carefully: they “demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search”. They report a best student that runs about ten times faster than its teacher with little loss in performance, and that pruning on top of distillation gave a student with thirteen times fewer parameters at a cost of a small amount of translation quality. Those are 2016 figures on a translation task, quoted as reported by the paper; they are not a prediction about your model.

What the objective actually is: the student’s ordinary next-token cross entropy, with the teacher’s text as the target. Every soft target has been collapsed into the single token the teacher happened to sample. All the structure the previous lesson was about is gone.

That sounds like a serious loss, and it is, and the method works anyway. The reason is that the teacher’s choice of text is itself informative: which of many valid answers it produced, in what order, at what length, with which caveats. For format, procedure and refusal behaviour, which is most of what people actually want moved, the text carries enough.

What it needs: an endpoint. Nothing more. The teacher can be on another machine, behind the Part 9 gateway, served by any engine, in any quantisation, from any model family, with any tokeniser. This is the only family with no compatibility condition at all, and that is why it is the default recommendation at home.

What it costs: one generation pass over your prompt set, paid once. The resulting dataset is a file. You can train five students on it, change the learning rate and train again, or come back in three months and reuse it, and the generation is not repeated.

Logit: match the distribution

Now keep the soft targets. Run teacher and student over the same tokens, take both distributions at every position, and train the student so that its distribution matches the teacher’s.

The objective is a divergence between two distributions, and the direction matters. TRL documents the general form its distillation trainers use, the generalised Jensen-Shannon divergence interpolated by a coefficient beta:

Pseudocode — not a real command

p_M = (1 - beta) * p_student + beta * p_teacher

loss = beta * KL[p_teacher || p_M] + (1 - beta) * KL[p_student || p_M]

beta = 0.0  ->  forward KL,  KL[p_teacher || p_student]
beta = 1.0  ->  reverse KL,  KL[p_student || p_teacher]

Read the two endpoints as behaviours rather than as formulae.

Forward KL is averaged over the teacher’s distribution, so it is penalised wherever the teacher puts probability and the student does not. The student is pushed to cover everything the teacher might say. It is mean-seeking: when the student cannot represent the teacher’s full spread, it spreads itself thin and puts probability in places neither model would actually choose.

Reverse KL is averaged over the student’s distribution, so it is penalised wherever the student puts probability and the teacher does not. The student is pushed to only say things the teacher would have said. It is mode-seeking: it picks one of the teacher’s good answers and commits.

That distinction is the whole argument of the MiniLLM paper, which replaces “forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution”. For a generative model, spreading probability over things the teacher considered unlikely is exactly the failure you can hear when you read the output.

What it needs: a shared tokeniser, and both models resident. The previous lesson covered why. It also needs care with memory: naively, the loss wants a tensor of batch by sequence length by vocabulary, twice. TRL’s documentation says the projection to vocabulary logits and the divergence “are computed in chunks, so peak activation memory does not scale with the full vocabulary × sequence-length logits tensor”, which is what makes the 24 GB floor in this part’s second lab possible at all.

On-policy: correct the student’s own samples

Both families above train the student on text somebody else produced. The student at inference time is conditioned on text it produced, which will contain tokens the teacher would not have written. This is the distribution mismatch the generalised-knowledge-distillation paper is about: current methods “suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference”.

The fix is to let the student generate and have the teacher score what it generated. GKD “trains the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences”, and it “also offers the flexibility to employ alternative loss functions between the student and teacher, which can be useful when the student lacks the expressivity to mimic the teacher’s distribution”. Both halves matter: the student sees its own trajectories, and you can choose the divergence that suits how much capacity it has.

The paper’s authors “find that on-policy data (high lmbda) performs better and the optimal beta varied depending on the task and evaluation method”, which TRL repeats in its usage tips. Note what that means for you: the mixing fraction has a recommended direction, the divergence does not, and anyone who tells you a single best beta has not read the same page.

The trainers TRL ships, and what state they are in

TRL TRL 1.12.0 · verified 2026-09-08 groups four trainers under knowledge distillation. Read on 2026-09-09, its documentation index marks experimental trainers with a flask and lists them under trl.experimental, and its release note says the DistillationTrainer “graduates to the stable API — on-policy knowledge distillation that matches a teacher’s full next-token distribution with a memory-efficient chunked JSD loss and vLLM-powered generation”.

Trainer	Import	Status on 2026-09-09	What it does
`DistillationTrainer`	`trl`	Stable; vLLM generation supported	On-policy: the student generates, the teacher scores, loss is generalised JSD. Teacher loaded locally and must share the student’s vocabulary.
`GKDTrainer`	`trl.experimental.gkd`	Experimental	A wrapper around `SFTTrainer` with `lmbda` mixing on-policy student data with the dataset, `seq_kd` switching to teacher-generated text, and `beta` selecting the divergence.
`MiniLLMTrainer`	`trl.experimental.minillm`	Experimental	Reverse-KL distillation in the reinforcement-learning shape, with reward functions alongside the divergence.
`AsyncDistillationTrainer`	`trl.experimental.async_distillation`	Experimental	Teacher never loaded locally: it is scored over HTTP against a vLLM server, so it can be far larger than the trainer’s memory.

Two of those deserve a note each.

The stable one is on-policy, which surprises people who expect “distillation” to mean the 2015 offline recipe. Its documented dataset type is prompt-only: “the student generates its own completions on-policy, so only the prompt is needed”. If you want the classic offline recipe, where the student is scored against a fixed set of sequences, that is GKDTrainer with lmbda=0.0, which TRL documents as reducing “to supervised JSD where the student is trained with the token-level probabilities of the teacher”.

The async trainer solves the home problem elegantly and then asks for three GPUs. Because the teacher is scored over HTTP, “the teacher can run on entirely separate hardware from the student and trainer, or even be a much larger model than would otherwise fit alongside the student”. Its documentation also states that the teacher server, the student’s vLLM server and the trainer “must run on separate GPUs”, which puts it outside every track in this course for now. It is worth knowing about because it is the shape a two-machine cluster would use, and Part 18 onwards is where that becomes possible.

The defaults that will surprise you

Three of DistillationConfig’s documented defaults are worth reading before your first run, because each is different from the trainer you are used to.

beta defaults to 1.0, which is reverse KL, not forward. The trainer’s documentation is explicit that unlike GRPO’s beta, which is a penalty coefficient against a reference model, “here it selects the divergence itself; there is no reference-model KL penalty”.
learning_rate defaults to 1e-6, against the 5e-5 of ordinary training arguments. That is a full-model rate. Training an adapter instead wants a much higher one, as Part 13 explained.
gradient_checkpointing defaults to True and bf16 defaults to true when fp16 is not set. Both are memory decisions already made for you, and both cost speed.

GKDConfig has its own set: temperature 0.9, lmbda 0.5, beta 0.5, max_new_tokens 128. The last one is small enough to truncate a reasoning trace in half, so raise it deliberately rather than discovering it.

What this costs against reinforcement learning

Part 14 spent a whole part on getting a model to improve against a reward. Distillation is the other answer to “make this small model better”, and the comparison is worth making explicitly because the two are usually presented as unrelated.

Both need generation, and generation dominates. The difference is what each generated token is worth.

Reinforcement learning generates a group of completions per prompt, computes a reward for each, and derives a gradient from how the group’s rewards vary. The completions are thrown away. Next epoch, generate again. There is no dataset at the end, only a model.
Sequence-level distillation generates one completion per prompt, once, and keeps it. The dataset survives the run, can be inspected, corrected, shared inside your house and reused.
Logit and on-policy distillation generate inside the loop like reinforcement learning, but every generated token gets a dense signal, a full distribution to match, rather than one scalar reward for the whole sequence.

That last point is the real economic difference. A reward is one number for hundreds of tokens; a teacher distribution is a target for every one of them. When a teacher exists that already does the task, distillation extracts far more signal per generated token than reinforcement learning can, and it needs no reward function to be designed, defended and protected against being gamed, which Part 14’s lesson on reward functions shows is most of the work.

When there is no teacher, none of this is available and reinforcement learning against a verifier is the method that remains. That is the honest division: distillation moves capability that exists somewhere; reinforcement learning creates capability that exists nowhere, slowly.

Choosing, in four questions

Do the teacher and student share a tokeniser? If not, sequence-level is the only option, and that is the end of the decision.
Can both models be resident at once? If not, sequence-level, or the async trainer once you have the hardware for it.
Does the task depend on the model’s second choice? Structured extraction, reranking and anything where you read logprobs argue for keeping distributions, so logit or on-policy.
Will you run this more than once? A reusable dataset is worth a great deal on the third experiment, and nothing on the first.

At home, on one machine, for most tasks, the answer is sequence-level, which is why the long lab in this part is the sequence-level one and the logit lab is the comparison against it.

Choose the method by the signal you can actually obtain

Logit distillation needs a comparable probability distribution over aligned output events. A teacher endpoint returning only text, or a small list of top-token scores, does not automatically provide the complete distribution needed for a full-vocabulary divergence. Tokeniser differences require an explicit alignment method rather than matching token IDs by number.

Sequence distillation uses generated demonstrations and can work across different tokenisers, but the student sees the teacher’s trajectories instead of its own mistakes. On-policy methods address that mismatch by training on states reached by the student, with feedback from the teacher or another objective. They introduce extra generation cost and policy-version bookkeeping.

For a selection exercise, list what access you have: teacher text, token probabilities, hidden states or local weights. Eliminate methods whose required signal is unavailable. Then estimate teacher-generation and training cost and choose a held-out task metric. The method with the richest theoretical signal is not automatically the most useful one if its memory footprint prevents the experiment or its alignment assumptions do not hold.

Recap

Sequence-level distillation trains the student on the teacher’s text with ordinary supervised fine-tuning; it discards the soft targets, needs only an endpoint, works across tokenisers and families, and produces a dataset you can reuse. Logit distillation matches distributions and keeps what the text throws away, at the price of a shared tokeniser and two resident models. On-policy distillation has the student generate and the teacher score, which removes the mismatch between the text a student trains on and the text it will produce. Forward divergence makes a student cover everything the teacher might say; reverse divergence makes it commit to one thing the teacher would have said, which is the MiniLLM argument. TRL’s DistillationTrainer is the stable, on-policy one; GKDTrainer, MiniLLMTrainer and AsyncDistillationTrainer are experimental as of 2026-09-09, and GKD with a zero mixing fraction is the way to get the classic offline recipe. Against reinforcement learning, distillation extracts far more signal per generated token, needs no reward function, and is only available when a teacher already exists.

Sources for this lesson

9 verified · checked 2026-09-09

01Sequence-Level Knowledge Distillation (Kim and Rush, arXiv:1606.07947)§ Abstractarxiv.org/abs/1606.079472026-09-09
02On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al., arXiv:2306.13649)§ Abstractarxiv.org/abs/2306.136492026-09-09
03MiniLLM: Knowledge Distillation of Large Language Models (Gu, Dong, Wei and Huang, arXiv:2306.08543)§ Abstractarxiv.org/abs/2306.085432026-09-09
04Distilling the Knowledge in a Neural Network (Hinton, Vinyals and Dean, arXiv:1503.02531)§ 2 Distillationarxiv.org/abs/1503.025312026-09-09
05TRL documentation — index and trainer taxonomy§ What's New; Taxonomy; Knowledge distillationhuggingface.co/docs/trl/en/index2026-09-09
06TRL documentation — Distillation Trainer§ Overview; Computing the loss; Expected dataset type; DistillationConfighuggingface.co/docs/trl/en/distillation_trainer2026-09-09
07TRL documentation — Generalized Knowledge Distillation Trainer§ Usage tips; GKDConfighuggingface.co/docs/trl/en/gkd_trainer2026-09-09
08TRL documentation — MiniLLM Trainer§ Overview; MiniLLMConfighuggingface.co/docs/trl/en/minillm_trainer2026-09-09
09TRL documentation — Async Distillation Trainer§ Overview; How it differs; AsyncDistillationConfighuggingface.co/docs/trl/en/async_distillation_trainer2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.