Skip to content
Local LLM Course

Knowledge Distillation: Big Teacher, Small Student

You have a large model running on your own machine and a small one you would rather serve. The large one answers the way you want. The small one is faster, fits the laptop, leaves room for the context you actually use, and gets most things nearly right. This part is about closing some of the distance between them without sending anything to anyone.

That is distillation: a big teacher, a small student, and a training run that moves behaviour from one to the other. Part 3 introduced the idea in a paragraph and named the three families. This part runs them. The teacher is a model you already have, served through the gateway you built in Part 9. The prompts are yours. The filters are scripts you can read. The evaluation is the harness from Part 10, run three times, on the teacher and on the student before and after, because an impression of improvement is not a result.

There is a reason this part sits after supervised fine-tuning and preference optimisation rather than before them. Sequence-level distillation is supervised fine-tuning, with a dataset a model wrote instead of a dataset a person wrote, and everything Part 13 taught about adapters, chat templates, held-out splits and export applies unchanged. What is new is the dataset: where it came from, what is wrong with it, and how to find out before it trains anything.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain what a student can inherit from a teacher and what it cannot, and predict which of your own tasks will move and which will not;
  • tell the three families apart, choose between them for a given pair of models, and say what each one needs that the others do not;
  • generate a dataset from your own served teacher, at stated sampling settings, resumably, and record what it cost in tokens, hours and, where the machine reports it, watt-hours;
  • filter that dataset properly: length, format and verifier checks, near-duplicate removal, and decontamination against the set you will report your result on;
  • train a student on it, evaluate teacher, base student and distilled student on the same tasks, and state how much of the gap closed and where the student got worse;
  • run logit distillation with the trainer TRL ships for it, check that the two tokenisers agree before you start, and compare the result against the sequence-level student on cost as well as score;
  • generate reasoning traces from a thinking-capable teacher, keep only the ones a verifier accepts, and say honestly what the resulting student can and cannot do;
  • read a pruning paper’s recipe, say when prune-and-distil beats training small, and name the tools and their state on a date;
  • diagnose a distillation that went wrong from evidence rather than from guesswork;
  • run the whole thing again next month from one configuration file.

What you need first

Section titled “What you need first”

Level 2, and specifically the gateway from Part 9’s project, because every generation step in this part talks to an OpenAI-compatible endpoint and every evaluation step talks to the same one. Your own task set from Part 10’s benchmark lab, because that is what tells you whether any of this worked. The training environment from Part 11, with the run log from that part’s lab, because every stage here appends to it.

Part 13 is assumed throughout: adapters, dataset shapes, chat templates and the export path are that part’s material and are not repeated. Part 14 is assumed in one place, the reasoning lesson, which reuses the idea of a reward a program can compute and turns it into a filter.

Memory: the sequence-level lab, the challenge and the project run from 12 GB, because the teacher is served rather than trained and the student is small. The logit lab is the exception. It holds a teacher and a student in memory at the same time, and its floor is 24 GB with a smaller documented pair for machines below that.

How to work through it

Section titled “How to work through it”

Read the first two lessons before touching anything: they are the ones that stop you distilling a pair that cannot work. Then do the sequence-level lab, which is the long one and the one everything else refers back to. The reasoning and synthetic-data lessons deepen what that lab did, and both ship a script you can run against the dataset you have already generated.

The logit lab is next if your machine can hold two models, and is worth reading even if it cannot, because the tokeniser argument in it is the reason half of the failures in the challenge happen. The pruning lesson is the one place this part looks at what publishers do rather than what you can do at home, and says which is which.

Then the challenge, which is a diagnosis rather than a build, and the project, which turns the labs into one configuration file and one runner you can point at a different task next month.

Keep the notebook open. Distillation produces more artefacts than any other part of this course: a seed file, a raw generation file, a filtered dataset, an adapter, a merged model, a quantised export and three evaluation runs. The run log is what stops that becoming a directory nobody can read in six weeks.

Part15 of 27Level3 — Model BuilderPages9Estimated time6.8 hours

0 / 9 lessons in this part completed