Distilling a Big Agent Model into a Small Local One

By the end of this lesson you will be able to use a large local model as a teacher for agent work, generate trajectories from it on your own tasks, filter and decontaminate them with the pipeline you already built in Part 15, choose between sequence-level and on-policy distillation knowing what each costs on one machine, measure the student on the agent suite, and give it a role in the Part 25 workstation.

Why a teacher, when you already have your own transcripts

The previous two lessons trained on trajectories your own model produced. That has a ceiling built into it, and the ceiling is the model. Sampling from a policy and training on the good samples sharpens the policy; it cannot demonstrate a behaviour the policy never produces. If your small model never gets past step three of a task, no amount of rejection sampling from it will show it what step four looks like.

A teacher removes that ceiling, at the cost of needing one. The model in Part 26’s project that you used for the hard role, or the coder model from Part 25’s workstation, will do work the small one cannot, and it does that work on your tasks, in your workspace, with your tools. Its transcripts are demonstrations of a behaviour that is not in the student’s distribution yet.

This is not a new idea and it is worth reading how it was done at the scale that made it famous. The DeepSeek-R1-Distill card, read on 2026-09-09, describes the procedure plainly: the distilled models are fine-tuned from existing open-source base models using samples generated by the larger model, with the card giving 800,000 curated samples and supervised fine-tuning without a reinforcement-learning stage. The 7-billion-parameter variant is built on Qwen2.5-Math-7B, and the card records an MIT licence for the distilled weights while noting the Apache-2.0 licence inherited from the Qwen base. The shape of what you are about to do is the same, three orders of magnitude smaller, with your tasks instead of theirs.

!Important

Check the teacher’s licence before you use its output, and record the answer. Part 15 makes this point once and it applies to every teacher: some model licences constrain what may be done with generated output and some do not, and the place to establish that is the card, before the generation run rather than after it. The model reference carries the licence for every model this course names.

Generating teacher trajectories on your tasks

The generator is not a new script. It is the agent loop, with the teacher behind it.

Part 15's pipeline, with a trajectory as the artefact

1. 1Serve the teacherThe large model from Part 26 or Part 25, behind the Part 9 gateway under its own alias.
2. 2Run the agent on your tasksPart 24’s loop or Part 26’s harness, pointed at the teacher alias, over the collection task set.
3. 3Collect, scrub, filterThis part’s three scripts. The outcome filter is doing the work Part 15 gave to rejection sampling.
4. 4DecontaminateAgainst the agent suite you will measure with, exactly as in Part 15’s filter stage.
5. 5Train the studentThe supervised run from the second lesson of this part, on the teacher’s trajectories.
6. 6Measure the tripletTeacher, base student and distilled student on the same suite at the same settings.

Only the first box changes from Part 15's sequence-level lab. Everything after it is the same pipeline with a different unit: an episode instead of an answer.

Part 15’s generate-teacher-data.py answers a seed file of prompts with a served teacher, concurrently and resumably, recording the sampling settings and token counts on every line. For agent work the equivalent is running the loop rather than a single completion, which is what Part 24’s minimal-agent.py with --tasks and Part 26’s agent-eval.py already do; both write the transcripts this part’s collector reads. Use whichever you already have configured.

Three settings from Part 15 carry over and matter more here.

The sampling settings are part of the data. The Qwen3-4B card, read on 2026-09-09, gives temperature 0.7, top-p 0.8, top-k 20 and min-p 0 for non-thinking mode, and temperature 0.6, top-p 0.95, top-k 20 and min-p 0 for thinking mode, and says not to use greedy decoding in thinking mode. Record what you used. A trajectory set generated at one temperature and a comparison run at another are two different experiments.

Several attempts per task beat one. Part 15 generates more than one sample per prompt for the same reason rejection sampling works: the filter needs something to choose between. On agent tasks this matters more, because the same task can be solved by a good trajectory and a wasteful one, and you want the choice.

Resumability is not a luxury. A teacher of the 30-billion-parameter class running a multi-step agent loop over a few dozen tasks with several attempts each is a run measured in hours. Part 15’s generator skips prompts already answered on a re-run, and any equivalent you build should too.

▤Memory budget

The teacher and the student do not need to be resident at the same time for this route, and that is the main reason it is the one to use at home. Generate with the teacher, stop it, then train. On a 16 GB machine a 30-billion-parameter mixture-of-experts teacher at a 4-bit type and a 4-billion-parameter student in bfloat16 will each fit alone and will not fit together. The hardware reference has the tiers.

Filtering, and the two checks trajectories need that answers do not

Part 15’s filter-and-dedupe.py applies length, format and refusal checks, drops degenerate repetition, removes near-duplicates, decontaminates against the evaluation set and splits what survives, counting every rejection by reason. That is exactly the right list, and this part’s trajectories-to-sft.py is the same idea with an episode as the unit.

Two checks are specific to trajectories.

The teacher’s tools must be the student’s tools. A teacher run with a richer tool list produces trajectories that call tools the student will never be shown. Train on those and the student learns to call something that does not exist, which shows up at serving time as a refused call and a wasted turn. This part’s converter rejects any episode calling a tool that is not in the episode’s declared list; the case it cannot catch is a tool that exists for both but is described differently, so generate with the tool list you intend to serve.

Length is the constraint that bites. A teacher with a long context happily produces a twenty-turn trajectory with whole files in the tool results. The student is being trained at a few thousand tokens. Filtering on step count and message count before training is cheaper than discovering it as silent truncation afterwards.

Sequence-level or on-policy, for tool use

Part 15’s lesson on the three kinds of distillation sets out the distinction. It survives the move to agents with one of the three becoming much more expensive.

Sequence-level distillation trains the student on the teacher’s outputs as text. For agent work the output is a trajectory, which is text, so nothing changes: it is the supervised run from the previous lesson with the teacher’s data instead of yours. Part 15’s sequence-level lab is the procedure, and pipeline.py is the runner that ties generation, filtering, training and evaluation together from one configuration file with a record per stage. This is the route to take.

Logit distillation matches the student’s output distribution to the teacher’s, token by token. It needs both models resident, and it needs them to share a tokeniser. Both conditions are harder here: the teacher is chosen for capability rather than for family, and a 30-billion-parameter teacher plus a 4-billion-parameter student in training does not fit on a 16 GB machine. Where it does fit, nothing about tool calls makes it worse; the trajectory is just a longer sequence.

On-policy distillation has the student generate and the teacher supply the targets, which is attractive precisely because it corrects the student’s own mistakes rather than showing it someone else’s successes. For a multi-turn trajectory it means running the student’s episode and then putting the whole thing through the teacher, per training step, with both models resident. That is the expensive combination of the two constraints above, and on one machine it is the route to know about rather than the route to take.

∴Research note

There is a cheaper approximation worth trying before the real thing, and it is the one this part’s pipeline already supports. Run the student on the tasks, keep the episodes it failed, and have the teacher attempt those same tasks. Train on the teacher’s successes for exactly the tasks the student got wrong. It is not on-policy distillation, because the targets are not conditioned on the student’s own trajectory, but it concentrates the teacher’s demonstrations where the student is weak, which is most of what makes on-policy attractive. Record it as what it is rather than as the thing it approximates.

Measuring the triplet

Part 15’s evaluation asks for three models on the same tasks at the same settings: the teacher, the base student and the distilled student. Keep that here, with Part 26’s suite as the instrument, because the three numbers answer three different questions.

The teacher’s score is the ceiling this route can reach. If the teacher only finishes two thirds of the suite, a student that finishes half is closer to the ceiling than the raw number suggests, and the useful figure is the fraction of the gap that closed.

The base student’s score is the thing you are trying to beat and the thing you would have had for free.

The distilled student’s score is the result. Report all three, at the same quantisation and the same number of repeats, along with the mean steps and mean tokens per task, because a student that matches the teacher’s success rate while taking twice as many steps has not matched it.

✕Common mistake

The teacher generated the training data, so it will do well on tasks that resemble it. Running the teacher and the student on the collection tasks rather than the held-out suite produces a comparison that flatters both and measures neither. The decontamination step exists for exactly this and it is worth re-reading its report before you believe a result.

Serving the student in the workstation

The point of the whole exercise is a model you keep. Part 25’s workstation names three roles rather than three files: an agent role with long context and native tool calling, a completion role that is small and fast, and a judge role that must not share weights with the agent. The student goes in as a candidate for the agent role, next to the model it is replacing rather than on top of it.

Keeping both is not sentimentality. It is what makes the next comparison possible, it is the fallback when the fine-tune turns out to be worse on something you had not thought to measure, and it costs a file on disk. This part’s lab exports the student, quantises it and writes the gateway snippets for a new alias, leaving the old one where it was.

✦Tip

Give the alias a name that says what it is and when: something like the role, the base model and the date it was trained. Six weeks later, local/agent-tuned tells you nothing about which of the three adapters you made is behind it, and the run identifier in the lab notebook is only useful if the alias points at it.

Compare the teacher, untouched student and deployed student

Use the same initial environment, tools, permissions and task budget for all three models. The teacher establishes what the chosen scaffold can accomplish; the untouched student establishes the starting point; the distilled student establishes the effect of training. Keep the teacher’s generation dataset disjoint from this evaluation.

Inspect trajectories as well as final results. A student can copy the teacher’s call format while choosing the wrong tool, or solve the task through many more calls. Count unnecessary actions, retries, latency and tokens. A cheaper token generator can still be more expensive per successful task if its loops are longer.

Validate the actual exported representation, because quantisation and chat-template changes can affect tool syntax. Retain the adapter, exact base, scrubbed training data, filter policy and evaluation outputs. The deployment decision is whether the student reaches the required verified success and permission behaviour at acceptable cost. It need not imitate the teacher’s prose or every intermediate action to be a useful replacement.

Recap

• Your own transcripts have a ceiling: they cannot demonstrate what the model never does. A teacher removes it, and the teacher is the large model already serving the hard role in your workstation.
• The generation stage is the agent loop with the teacher behind it. The sampling settings, several attempts per task and resumability all matter more than in Part 15 because the runs are longer.
• Filter with Part 15’s list plus two agent-specific checks: the teacher’s tool list must be the student’s, and trajectory length must fit the student’s training sequence.
• Sequence-level distillation is the route on one machine. Logit distillation needs both models resident and a shared tokeniser; on-policy needs both resident per step.
• Measure teacher, base student and distilled student on the same held-out suite at the same settings, and report steps and tokens alongside the success rate.

Check your understanding

Question 1. Why can a teacher improve a student in ways that rejection sampling from the student itself cannot?

• The teacher is quantised differently
• Sampling from a policy can only sharpen behaviour that policy already produces, while the teacher demonstrates behaviour outside the student’s distribution
• The teacher has a longer context window
• Rejection sampling does not work on agent tasks

Show the answer and why

Answer: Sampling from a policy can only sharpen behaviour that policy already produces, while the teacher demonstrates behaviour outside the student’s distribution

Rejection sampling is selection, not instruction. If the student never gets past step three, no filter over its own runs can show it step four. That is the whole reason to pay for a teacher.

Question 2. Your teacher was run with a tool list containing three tools the student will not be given. What happens if you train on those trajectories unchanged?

• Nothing, because the student ignores unknown tools
• The student learns to call tools it will never be shown, which costs a refused call and a wasted turn at serving time
• The chat template drops them automatically
• The trainer raises an error

Show the answer and why

Answer: The student learns to call tools it will never be shown, which costs a refused call and a wasted turn at serving time

The tool list is part of the training prompt, so the student learns a mapping from a description it will not see to a call that will not resolve. Generate with the tool list you intend to serve, and let the converter reject episodes that call anything undeclared.

Question 3. Which distillation route is practical for agent trajectories on a single 16 GB machine?

• Logit distillation, because it transfers the most information
• On-policy distillation, because it corrects the student’s own mistakes
• Sequence-level distillation, because the artefact is text and the two models are never resident at once
• None of them

Show the answer and why

Answer: Sequence-level distillation, because the artefact is text and the two models are never resident at once

Sequence-level distillation means generate with the teacher, stop it, then train the student on the text. Logit distillation needs both models in memory and a shared tokeniser; on-policy needs both in memory on every step. On one machine the sequential route is the one that fits.

Question 4. You report that the distilled student reaches the same success rate as the teacher on the agent suite. What else must be reported before that claim means anything?

• The mean steps and mean tokens per task for both
• The number of repeats and the spread across them
• The quantisation and engine both were served at
• The teacher’s parameter count

Show the answer and why

Answer: The mean steps and mean tokens per task for both, The number of repeats and the spread across them, The quantisation and engine both were served at

A student that matches the success rate while taking twice as many steps has not matched the teacher. A difference smaller than the run-to-run spread is not a difference. And a comparison across two quantisations is measuring the quantiser. The parameter count is interesting but is not what makes the claim checkable.

Question 5. True or false: once the distilled student is serving under a new alias, the base model can be removed from the gateway.

• True
• False

Show the answer and why

Answer: False

False. The base is the comparison for every future run, the fallback when the fine-tune turns out worse on something you did not measure, and the thing that makes a regression detectable at all. It costs a file on disk and an entry in the configuration.

Check my answersClear