Why Distillation Works

By the end of this lesson you will be able to say what a soft target is and why it carries more than a correct answer does; predict which of your own tasks will move from a teacher into a student and which will not, and give the reason rather than the observation; choose a size ratio that is worth attempting; and check in one command whether the pair you have in mind can use the logit methods at all. Part 3 introduced distillation in a paragraph. This lesson is the argument underneath it, and the next one is the machinery.

The 2015 argument, precisely

Hinton, Vinyals and Dean were solving a deployment problem: an ensemble is accurate and “cumbersome and may be too computationally expensive to allow deployment to a large number of users”. Their route out was to train one small model on the big one’s outputs rather than on the original labels.

The reason that helps is the sentence the whole field is built on:

The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize.

Their example is a picture of a BMW. It “may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot”. A hard label says “BMW” and nothing else. The teacher’s full distribution says “BMW, and if not, something vehicle-shaped, and certainly not a vegetable”. That second statement is a description of a learned similarity structure, and it is free: the teacher produced it as a side effect of answering.

The paper calls these soft targets, and it is explicit about what they buy:

When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original cumbersome model and using a much higher learning rate.

Two consequences, both of which show up in the labs. More information per example means fewer examples are needed. Less gradient variance means a larger learning rate is safe. Distillation is not just a way to get a smaller model; it is a way to get one from a smaller dataset.

∴Research note

“Dark knowledge” is the phrase that stuck for the information in those near-zero probabilities. It is not in the paper, which says “soft targets” throughout. Worth knowing because you will meet the phrase in blog posts and it is a nickname rather than a technical term with a definition you can check.

Temperature, and the detail everyone drops

If the teacher is confident, its soft targets are nearly hard: one probability close to one and the rest close to zero, which carries almost nothing extra. The paper’s fix is temperature in the softmax, “where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.” The teacher is run at a high temperature to expose the structure, and “the same high temperature is used when training the distilled model, but after it has been trained it uses a temperature of 1.”

Where the correct labels are also known, the paper uses “a weighted average of two different objective functions”: cross entropy with the soft targets at the high temperature, and cross entropy with the correct labels at temperature 1, with “a considerably lower weight on the second objective function”. And then the detail that a re-implementation usually gets wrong:

Since the magnitudes of the gradients produced by the soft targets scale as 1/T² it is important to multiply them by T² when using both hard and soft targets.

You will not type that scaling yourself, because the trainers in the next lesson do it. You need to know it exists because temperature and loss weighting are two of the four faults in this part’s challenge, and a run that changed the temperature without changing anything else has quietly changed how much the two objectives count.

What a language model changes about this

The 2015 setting is image classification: one decision per example, over a thousand classes. A language model makes one decision per token, over its whole vocabulary, and it makes hundreds of them per answer.

The Qwen3 tokeniser has more than 151,000 entries, so every position in every answer is a distribution over that many options. A two-hundred-token answer therefore carries two hundred distributions of 151,000 numbers each, against the two hundred integers that the same answer carries as text. That is the gap between logit distillation and sequence-level distillation, and it is why the first is more informative per example and much more expensive per example.

It also introduces a problem image classification never had. The student’s answer is generated one token at a time, conditioned on the tokens it has already produced. Train it only on the teacher’s text and it never sees what to do after a token the teacher would not have written, which is exactly the situation it will be in. The generalised-knowledge-distillation paper names this directly: current methods “suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference”. That mismatch is the reason the third family exists, and the next lesson is where it gets solved.

What moves, and what does not

The useful mental model is a ladder. Things near the top move readily, cheaply, and from a few hundred examples. Things near the bottom do not move at all, however much data you generate.

What a student can inherit from a teacher

harder to transfer

1. Format and styleAnswer shape, length, register, whether it explains before or after. Moves fastest of all; Part 11's first lab changed it in minutes.moves easily
2. Task procedureThe steps the teacher takes on a problem: what it checks first, how it decomposes, when it asks for more information.moves well
3. Refusal and hedging behaviourDeclining when the answer is unknowable. Moves well, and moves in the wrong direction just as readily if the teacher answers anyway.moves well
4. CalibrationHow much probability the model puts on its second and third choices. Sequence-level text throws this away; the logit methods keep it.needs logits
5. Broad factual knowledgeThe facts a 30B model can answer about and a 4B model cannot. Parameters are where facts are stored, and the student has fewer of them.does not move

The top three move with sequence-level distillation on a few hundred examples. Calibration is where the logit methods earn their extra cost. The bottom layer does not move: a small model has nowhere to put a large fact base, and the answer to that problem is retrieval, from Part 10, not a training run.

The bottom rung is the one that disappoints people, and it is not a limitation of the method. It is arithmetic. A student with a quarter of the teacher’s parameters has a quarter of the places to put things, and distillation does not add capacity. Part 13’s first lesson made the same point about fine-tuning, and the conclusion is the same: if the problem is that the model does not have some information, give it the information at inference time. Part 10 teaches retrieval for exactly this.

Calibration is the interesting middle. The MiniLLM authors report that their students “generate more precise responses with the higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance”, and calibration is on that list because they trained against distributions rather than text. If your task cares about the model’s second choice, and structured extraction and retrieval reranking often do, that is an argument for the logit route.

!Important

Write down, before you start, which rung of that ladder your task sits on. A team that wants their small model to answer in the house style is on the top rung and will succeed this afternoon. A team that wants it to know their product catalogue is on the bottom rung and will spend a week finding out that it cannot.

How far apart can the pair be?

There is no published constant here, and anybody who gives you one is guessing. What there is, is a set of ratios that worked, in public, with the recipe stated.

NVIDIA’s Minitron work compresses the Nemotron-4 family “by a factor of 2-4x”, and reports that deriving 8B and 4B models from a 15B model “requires up to 40x fewer training tokens per model compared to training from scratch”. TRL’s own distillation example uses a 1.5B teacher and a 0.5B student, a factor of three. DeepSeek’s distilled models go much further: the R1 paper’s claim is that the reasoning patterns of the large model “can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models”, and the DeepSeek-R1-Distill-Qwen-7B card says the model was “finetuned with 800k samples curated with DeepSeek-R1” on a 7B base, from a teacher very much larger than that.

Read those together and the shape is: a factor of two to four is comfortable, a factor of eight is done routinely with enough data, and beyond that the amount of data required grows faster than the gain. The practical constraint at home is usually different and much simpler.

Your teacher has to be servable and your student trainable, at the same time or in sequence, on hardware you own. That is the real filter. On a 128 GB machine the teacher is a 30B-class model and the student is a 4B; at 24 GB the teacher is a 14B-class model; at 12 to 16 GB it is an 8B, and the student drops to 1.7B. Those are the pairs the labs use, and they are chosen by memory rather than by theory.

?Reality check

The claim worth resisting is that a distilled small model “matches” its teacher. What the measurement usually shows is that it closes part of the gap on the tasks the training data covered, and none of it elsewhere. That is a genuinely useful result and it is not the same claim. The sequence-level lab in this part reports the fraction of the gap that closed, per category, because the total on its own hides the categories that got worse.

The tokeniser condition

One property decides which methods are available to you, and it takes thirty seconds to check.

Logit distillation matches the student’s probability for token id n against the teacher’s probability for token id n. That only means anything if id n is the same token in both models. TRL’s documentation states the requirement for its distillation trainer plainly: the teacher “must share the student’s vocabulary”, and warns that a teacher with a different vocabulary trains the student against the wrong tokens, silently, when its vocabulary is no larger than the student’s. No error, no warning in the logs, a loss that falls; and a student learning nonsense.

Models from the same family usually satisfy this because they were built with one tokeniser. The tokenizer_config.json files of Qwen3-0.6B and Qwen3-4B, read on 2026-09-09, both declare the Qwen2Tokenizer class and give the same ids for the same special tokens: 151643 for <|endoftext|>, 151644 and 151645 for the two chat-turn markers, 151667 and 151668 for the opening and closing thinking tags. A 0.6B student and a 4B teacher from that family are talking about the same vocabulary, position by position.

Models from different families do not satisfy it, and no amount of care makes them. A Qwen3 teacher and a Llama student are a sequence-level pair or no pair at all.

✦Tip

Sequence-level distillation has no tokeniser condition, because text is text. That is its main practical advantage and the reason the long lab in this part uses it: it works across families, across engines, and with a teacher you can only reach over HTTP.

The three families, in one paragraph each

Sequence-level. Have the teacher answer a large set of prompts, filter the answers, and train the student on them with ordinary supervised fine-tuning. Needs only text, works across tokenisers and families, and is the cheapest to run because generation is the only expensive part and it happens once. Throws away everything except the token the teacher actually chose.

Logit. Run teacher and student together and match distributions rather than text. Keeps the soft targets the 2015 paper is about, including calibration. Needs a shared tokeniser and enough memory for two models, which is why it has the only 24 GB floor in this part.

On-policy. Let the student generate, and have the teacher grade or correct the student’s own output. Attacks the distribution mismatch directly, at the cost of running generation inside the training loop.

The next lesson takes each one apart: the objective, the data it needs, the trainer that implements it, what it costs, and how to choose.

Distinguish information transfer from answer copying

A teacher can supply more informative targets than a small labelled dataset alone: alternative-token probabilities, carefully chosen demonstrations or feedback on student-generated states. What transfers depends on the method. Saving only the teacher’s final answer discards its distribution over alternatives; requesting a written explanation does not recover that full distribution.

Choose a domain where the teacher is demonstrably useful and where outputs can be checked. Keep a student-without-distillation baseline trained under a comparable budget, alongside the untouched student and teacher. This separates a benefit from teacher data from the benefit of simply doing more supervised training.

Inspect where the student fails after distillation. It may reproduce teacher mistakes, lack capacity for a pattern, or encounter inputs unlike the generated demonstrations. A teacher’s confident style can transfer even when correctness does not. The practical target is a student that meets a defined task requirement at lower serving cost, with a measured regression profile. Matching the teacher’s prose is not a sufficient acceptance test.

Recap

Soft targets are a teacher’s full probability distribution, and they carry the structure of what the teacher considers nearly right, which a correct answer alone does not. That is worth more per example and produces less gradient variance, so a student needs less data. Temperature controls how much of that structure is exposed, and the soft-target gradient has to be scaled by the square of the temperature when hard labels are mixed in. Format, procedure and refusal behaviour transfer easily; calibration transfers only if you keep the distributions; broad factual knowledge does not transfer at all, because the student has fewer parameters to store it in. Ratios of two to four between teacher and student are comfortable and eight is done routinely with enough data, but the binding constraint at home is what your machine can hold. Logit methods need teacher and student to share a tokeniser; sequence-level methods do not, which is why they work across families.

Check your understanding

Question 1. Why does a soft target carry more information than a hard label, in the 2015 paper's own terms?

• Because it is more accurate than the label
• Because the relative probabilities of the incorrect answers describe how the teacher generalises, and high-entropy targets give more information per case and less gradient variance
• Because it is longer
• Because it removes the need for a validation set

Show the answer and why

Answer: Because the relative probabilities of the incorrect answers describe how the teacher generalises, and high-entropy targets give more information per case and less gradient variance

The paper's BMW example is the point: mistaking it for a garbage truck is unlikely but far more likely than mistaking it for a carrot, and that ordering is a description of learned similarity. The practical consequence is stated in the same section: fewer training cases and a higher learning rate.

Question 2. A team wants a 4B student to answer support questions in their house format and to know the contents of their internal wiki. Which parts of that will distillation from their 30B model deliver?

• Both, given enough generated data
• The format, reliably and quickly; the wiki contents, not at all, because parameters are where facts live and the student has fewer of them
• The wiki contents only
• Neither, without reinforcement learning

Show the answer and why

Answer: The format, reliably and quickly; the wiki contents, not at all, because parameters are where facts live and the student has fewer of them

Format sits at the top of the transfer ladder and moves in an afternoon. Factual coverage sits at the bottom and is a capacity problem that no amount of generated data fixes. The answer for the wiki is retrieval, taught in Part 10.

Question 3. You want to distil a Qwen3-8B teacher into a Llama 3.1 8B student using TRL's distillation trainer. What happens?

• It works, because both are transformer models
• The trainer converts between the two vocabularies automatically
• The vocabularies differ, so a teacher probability would be applied to the wrong student token; the documented requirement is that the teacher shares the student's vocabulary, and the failure can be silent
• It fails immediately with a shape error
• It works but trains more slowly

Show the answer and why

Answer: The vocabularies differ, so a teacher probability would be applied to the wrong student token; the documented requirement is that the teacher shares the student's vocabulary, and the failure can be silent

The loss indexes the student's vocabulary with the teacher's token ids. A mismatch is not necessarily a crash, which is what makes it dangerous: the loss falls and the student learns the wrong thing. Across families, use sequence-level distillation, which moves text.

Question 4. Which of these are reasons the labs in this part pick their teacher and student sizes? Select all that apply.

• The teacher has to be servable on the reader's own machine
• The student has to be trainable on the same machine
• A published constant defines the maximum safe ratio
• Ratios of roughly two to four are what published prune-and-distil work reports using

Show the answer and why

Answer: The teacher has to be servable on the reader's own machine, The student has to be trainable on the same machine, Ratios of roughly two to four are what published prune-and-distil work reports using

There is no published constant, and anyone quoting one is guessing. What exists is a set of ratios that worked with the recipe stated, and a hard practical limit set by the memory in front of you.

Question 5. Why does raising the distillation temperature change more than just how soft the targets are?

• It changes the tokeniser
• The soft-target gradients scale as one over temperature squared, so the balance between the soft-target and hard-label objectives moves unless the gradient is rescaled
• It changes the student's parameter count
• It has no other effect

Show the answer and why

Answer: The soft-target gradients scale as one over temperature squared, so the balance between the soft-target and hard-label objectives moves unless the gradient is rescaled

The paper says to multiply the soft-target gradients by the square of the temperature precisely so that the relative contribution of the two objectives stays roughly unchanged when you experiment with temperature. Change temperature alone and you have silently changed the loss weighting too.

Check my answersClear