Project: The Distillation Pipeline

Validated on: written from the documentation cited above; not yet validated on hardware on any track. When the validation pass runs this page, the configuration each track used and the two runs it produced will be recorded here.

Objective

By the end of this project you will have a distillation pipeline you can point at a new task next month without remembering anything, and a written report that lets somebody else decide whether to trust its output.

Concretely: one YAML file that holds every setting, one runner that turns it into commands and records each stage in the lab notebook with the hash of that file, a full run on the task from the sequence-level lab, a second full run on a task the pipeline has never seen, and a one-page report with the numbers and the costs.

The point is not automation for its own sake. It is that the sequence-level lab had nine steps with settings scattered across nine command lines, and a result produced that way cannot be reproduced in six weeks, compared with a second result, or handed to anybody. One file and one command fixes all three.

⌂Single-machine path

Single-machine on every track, with the same two-phase memory profile as the sequence-level lab: the teacher is served during the generate stage and stopped before the train stage. If you have two machines, put the teacher on the larger one and change one line of the configuration.

The pipeline

Five stages, each of which is one command, each of which writes a file the next one reads and a line in the run log.

The five stages, and what each one leaves behind

1. 1generateSeed prompts in, teacher answers out. The expensive stage, resumable, and the only one that needs the teacher.
2. 2filterRaw answers in, a training set and a rejection report out. Cheap, repeatable, and where the quality is decided.
3. 3trainTraining set in, an adapter out. Needs the student and not the teacher.
4. 4evaluateThree models on one task file, out come three scores and the fraction of the gap closed.
5. 5exportAdapter in, a merged model and a quantised GGUF out, ready for the gateway.

Each stage is independently runnable with --only, and the pipeline can be resumed from any of them with --from. That is what makes a changed filter threshold cost two minutes instead of two hours.

Three properties are what make this a pipeline rather than a shell script.

Every stage records itself. The stage scripts write their own detailed record; the runner writes one more saying which stage ran, under which configuration hash, for how long. Reading the notebook back gives the run in order.

The configuration is hashed, not the file. The runner hashes the parsed configuration with its keys sorted, so a comment or a reordering does not look like a different experiment and a changed threshold does.

Nothing starts a model. The runner never launches an engine. A runner that starts models is a runner that hides which model answered, and the one thing this pipeline must never be vague about is which teacher produced the data.

What belongs in the configuration, and what does not

The line is easy to state and easy to get wrong, and getting it wrong is how a pipeline turns back into a shell script over a few months.

In the configuration: anything that changes the result. Model ids, aliases, quantisations, sampling settings, filter thresholds, learning rates, ranks, epochs, seeds, paths and the task file. If two runs could differ in it and produce different numbers, it belongs in the file that gets hashed. The test is simple: if you would have to write it down in the report, it belongs here.

Not in the configuration: anything that is a property of the machine rather than the experiment. Which engine is running, on what port, with how many slots; whether the accelerator is available; how much memory is free. Those change between machines running the same experiment, and putting them in the configuration means the file cannot be shared with somebody on a different track.

Never in the configuration: secrets. The teacher.api_key_env key names an environment variable and the runner reads the value from the environment at run time, which is the same pattern Part 9’s gateway configuration uses for its keys. A configuration file with a key in it is a file that cannot be committed, cannot be pasted into an issue, and will eventually be both.

The awkward middle is concurrency. It changes throughput and not results, so in principle it is a machine property; but a concurrency that exceeds the server’s slot count changes the wall clock you report, and the cost table is one of the deliverables. It lives in the configuration here, and the report records the slot count beside it.

✦Tip

A useful discipline when you are unsure: put it in the configuration, run the pipeline on a second machine, and see which values you had to change. Everything you changed was a machine property, and everything you did not was part of the experiment. Ten minutes of that settles arguments that otherwise run for weeks.

Requirements

Everything from this part’s sequence-level lab, plus PyYAML in the training environment. The logit lab is not required; the pipeline runs the sequence-level route, which is the one that works on every track and at every memory tier.

Ninety minutes: about 20 attended for configuration and reading, about 45 unattended across two runs, and about 25 for the report.

Track S — NVIDIA DGX Spark

Everything runs. With 128 GB you can leave the teacher served throughout both runs, which makes the second run in task 6 much quicker: only the seed prompts and the task file change.

Run the pipeline from inside the NGC PyTorch container from Part 11 so that the train stage finds its libraries, with the course directory mounted so that the artefacts survive the container exiting.

Track X — AMD Ryzen AI Max+ 395Partial

The generate, filter, evaluate and export stages are fully supported. The train stage needs the ROCm build of PyTorch: the ROCm 10.0.0 compatibility matrix dated 2026-08-14 lists gfx1151 without a support-tier qualifier and AMD's PyTorch install page read on 2026-09-09 does not mention the chip. The CPU path completes the train stage.

Set gradient_checkpointing: true and a batch size of 1 in the train block if you are finishing on the CPU, and expect that stage to dominate the wall clock. Record in the report that it was a CPU run; the cost table is one of the deliverables and a CPU figure is not comparable with a GPU one.

Track M — Apple silicon

Set student.trainer: mlx in the configuration. The runner then prints the documented mlx_lm.lora command against the data-mlx/ files rather than running the PyTorch path, because silently training a different thing would be worse than printing a command.

Copy the losses out of the mlx-lm output into a run record by hand, as Part 11’s lab does. The energy column of your report is “not reported on this machine”: distillog.py samples power through nvidia-smi or rocm-smi, and neither exists here.

Track N — NVIDIA desktop or laptop

Set the tier’s pair in the configuration: at 24 GB a 14B-class teacher and a 4B student, at 12 to 16 GB an 8B-class teacher and a 1.7B student. Stop the teacher between the generate and train stages, which the runner does not do for you: it prints the commands it is about to run and leaves engine lifetime to you deliberately.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-15-distillation"
cd "$LAB_DIR"
pwd
test -f "distil-config.yaml"

Expected result: pwd ends in part-15-distillation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Read the configuration before you change it

RunnableAll tracks

distil-config.yaml

# Purpose: the whole distillation run in one file. pipeline.py reads this, executes the
#          stages named in `stages`, and records each one in the lab notebook with the
#          hash of this file's contents, so a result a month old can be traced to the
#          settings that produced it and reproduced from them.
# Platform: all (spark, strix, mac, nvidia). Track M swaps the trainer, as noted below.
# Minimum memory: 12 GB for the sequence-level route with the pairs listed here.
# Assumes: the scripts of Part 15 in the same directory as pipeline.py; a served teacher
#          at `teacher.base_url`; the Part 10 evaluation harness at `evaluate.harness_dir`.
#          No secret appears here: the gateway key is read from the environment variable
#          named in `teacher.api_key_env`.

# ---------------------------------------------------------------------------
# What is being distilled into what. The ids are the course's model ids; the
# `served_as` names are what your gateway or engine answers to, which is a
# different thing and is why both are written down.
# ---------------------------------------------------------------------------
run:
  name: house-style-4b
  # Bump this whenever you change anything below that you want treated as a new
  # experiment rather than a rerun of the old one.
  version: 1
  seed: 0
  labbook: labbook.md

teacher:
  id: qwen3-30b-a3b            # a 30B-class teacher on a 128 GB machine
  served_as: local/chat        # the gateway alias from Part 9
  quant: Q4_K_M
  base_url: http://127.0.0.1:4000/v1
  api_key_env: LOCAL_API_KEY   # the name of the variable, never the key itself

student:
  id: qwen3-4b
  hf_id: Qwen/Qwen3-4B
  # Track M trains with mlx_lm.lora on the data-mlx/ files instead; set this to
  # "mlx" and pipeline.py prints the command rather than running the PyTorch path.
  trainer: trl

# ---------------------------------------------------------------------------
# The stages, in order. pipeline.py runs the ones listed here; --from and --only
# on the command line narrow that further without editing this file.
# ---------------------------------------------------------------------------
stages: [generate, filter, train, evaluate, export]

seed_prompts:
  script: make-seed-prompts.py
  out_dir: seeds
  train: 600
  heldout: 120
  maths: 0                     # raise this for the reasoning route

generate:
  script: generate-teacher-data.py
  seeds: seeds/prompts.jsonl
  out: raw/teacher.jsonl
  # Qwen3's card gives these for non-thinking mode; --thinking switches the script
  # to 0.6 / 0.95, which is what the card gives for thinking mode.
  thinking: false
  temperature: 0.7
  top_p: 0.8
  max_tokens: 768
  samples: 1
  # Match this to the server's slot count: llama-server's --parallel, or vLLM's
  # scheduler. More requests in flight than slots queues rather than parallelises.
  concurrency: 4
  resume: true

filter:
  script: filter-and-dedupe.py
  raw: raw/teacher.jsonl
  out_dir: .
  min_words: 8
  max_words: 400
  near_duplicate: 0.8
  contamination: 0.5
  keep_thinking: false
  valid_fraction: 0.1
  # The evaluation set the result will be reported on. Leaving this empty skips
  # decontamination, which makes the final number unreportable; the script says so.
  tasks: my-tasks.json

train:
  script: train-student.py
  data_dir: data
  output_dir: runs/seq-qwen3-4b
  epochs: 2
  batch_size: 2
  grad_accum: 8
  learning_rate: 0.0001
  max_length: 1024
  rank: 16
  alpha: 32
  gradient_checkpointing: false

evaluate:
  script: evaluate-triplet.py
  harness_dir: ~/eval
  tasks: my-tasks.json
  out_dir: eval-out
  # Three names the endpoint answers to. On a machine that cannot hold all three,
  # leave no_pause false and the script waits between models.
  teacher_served_as: local/chat
  base_student_served_as: local/student-base
  distilled_served_as: local/student-distilled
  judge_model: null
  no_pause: true
  engine: llama.cpp
  engine_version: v0.4.0
  quant: Q4_K_M

export:
  # The export stage runs Part 13's merge and llama.cpp's converter and quantiser.
  # pipeline.py prints these commands and runs them only with --run-export, because
  # they write large files and depend on a llama.cpp checkout it cannot assume.
  merge_script: ~/llm-course/merge-adapter.py    # from Part 13
  llama_cpp: ~/llama.cpp
  quant: Q4_K_M
  out_dir: models/seq-qwen3-4b

Download distil-config.yaml118 lines

Every setting the labs passed on a command line is here, grouped by the stage that consumes it. Four blocks deserve a second look.

run holds the name, a version you bump when you want a change treated as a new experiment rather than a rerun, the seed every stage uses, and the notebook path. teacher holds the course model id and the name your endpoint answers to, which are different things: the API cannot be asked which weights are loaded, so the id is recorded rather than discovered. teacher.api_key_env names an environment variable rather than holding a key, so this file can be committed. stages lists which of the five to run, so removing export from that list is how you skip it without deleting its settings.

2. Dry-run it, and read every command

RunnableAll tracks

pipeline.py

"""Run the whole distillation pipeline from one configuration file, and record every stage.

Purpose: the project's runner. Reads distil-config.yaml, turns each stage into the
    command it stands for, runs the stages you asked for, and writes one pipeline
    record per stage into the lab notebook carrying the hash of the configuration
    that produced it. Stages are resumable and skippable, so a run interrupted in
    the third hour continues rather than restarting, and a changed threshold reruns
    the filter without regenerating a dataset.
Platform: all (pure Python; the stages it launches have their own platform notes,
    and Track M's training stage prints an mlx_lm.lora command instead of running
    the PyTorch one)
Minimum memory: 12 GB, set by the stages rather than by this script
Assumes: Python 3.10 or newer, PyYAML installed, and this part's scripts in the same
    directory. The teacher is already served; this script starts no engine, because
    a runner that starts models is a runner that hides which model answered.

Usage: python3 pipeline.py --config distil-config.yaml --dry-run
       python3 pipeline.py --config distil-config.yaml
       python3 pipeline.py --config distil-config.yaml --from train
       python3 pipeline.py --config distil-config.yaml --only evaluate
       python3 pipeline.py --config distil-config.yaml --run-export
       python3 pipeline.py --config distil-config.yaml --status

--dry-run prints every command and touches nothing, which is the way to read a
configuration change before spending an evening on it.
"""

from __future__ import annotations

import argparse
import json
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path
from typing import Any

import distillog

ORDER = ["seed_prompts", "generate", "filter", "train", "evaluate", "export"]


def load_config(path: Path) -> dict:
    try:
        import yaml  # noqa: PLC0415 - reported clearly rather than crashing on import
    except ImportError:
        raise SystemExit(
            "PyYAML is not installed in this environment. Install it with "
            "`uv pip install pyyaml`, or read the configuration by hand and run the "
            "stage scripts directly; every stage is one command."
        ) from None
    if not path.is_file():
        raise SystemExit(f"{path} not found")
    config = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(config, dict):
        raise SystemExit(f"{path} did not parse as a mapping")
    for required in ("run", "teacher", "student", "stages"):
        if required not in config:
            raise SystemExit(f"{path} has no '{required}' section")
    return config


def expand(value: Any) -> Any:
    """Expand ~ in anything that looks like a path, leaving everything else alone."""
    if isinstance(value, str) and value.startswith("~"):
        return os.path.expanduser(value)
    return value


def flag(name: str, value: Any) -> list[str]:
    """One configuration key as command-line arguments, or nothing when it is unset."""
    if value is None or value == "":
        return []
    option = "--" + name.replace("_", "-")
    if isinstance(value, bool):
        return [option] if value else []
    return [option, str(expand(value))]


def build_commands(config: dict, here: Path) -> dict[str, list[list[str]]]:
    """Every stage as the list of commands it stands for. Nothing is run here."""
    run = config["run"]
    teacher = config["teacher"]
    student = config["student"]
    labbook = run.get("labbook", "labbook.md")
    py = sys.executable
    commands: dict[str, list[list[str]]] = {}

    seeds = config.get("seed_prompts", {})
    if seeds:
        commands["seed_prompts"] = [[
            py, str(here / seeds.get("script", "make-seed-prompts.py")),
            *flag("out_dir", seeds.get("out_dir", "seeds")),
            *flag("train", seeds.get("train")),
            *flag("heldout", seeds.get("heldout")),
            *flag("maths", seeds.get("maths")),
            *flag("seed", run.get("seed", 0)),
        ]]

    gen = config.get("generate", {})
    if gen:
        commands["generate"] = [[
            py, str(here / gen.get("script", "generate-teacher-data.py")),
            *flag("seeds", gen.get("seeds")),
            *flag("out", gen.get("out")),
            *flag("base_url", teacher.get("base_url")),
            *flag("model", teacher.get("served_as")),
            *flag("teacher_id", teacher.get("id")),
            *flag("quant", teacher.get("quant")),
            *flag("thinking", gen.get("thinking", False)),
            *flag("temperature", gen.get("temperature")),
            *flag("top_p", gen.get("top_p")),
            *flag("max_tokens", gen.get("max_tokens")),
            *flag("samples", gen.get("samples")),
            *flag("concurrency", gen.get("concurrency")),
            *flag("resume", gen.get("resume", True)),
            *flag("labbook", labbook),
        ]]

    filt = config.get("filter", {})
    if filt:
        commands["filter"] = [[
            py, str(here / filt.get("script", "filter-and-dedupe.py")),
            *flag("raw", filt.get("raw")),
            *flag("out_dir", filt.get("out_dir", ".")),
            *flag("tasks", filt.get("tasks")),
            *flag("min_words", filt.get("min_words")),
            *flag("max_words", filt.get("max_words")),
            *flag("near_duplicate", filt.get("near_duplicate")),
            *flag("contamination", filt.get("contamination")),
            *flag("keep_thinking", filt.get("keep_thinking", False)),
            *flag("valid_fraction", filt.get("valid_fraction")),
            *flag("seed", run.get("seed", 0)),
            *flag("labbook", labbook),
        ]]

    train = config.get("train", {})
    if train:
        if student.get("trainer") == "mlx":
            # Track M: print the documented mlx-lm command rather than running the
            # PyTorch path, which would silently train a different thing.
            commands["train"] = [[
                "mlx_lm.lora",
                "--model", str(student.get("hf_id")),
                "--train",
                "--data", str(train.get("data_dir", "data")) + "-mlx",
                "--batch-size", str(train.get("batch_size", 2)),
                "--iters", str(train.get("iters", 600)),
                "--adapter-path", str(train.get("output_dir", "runs/seq-student")),
            ]]
        else:
            commands["train"] = [[
                py, str(here / train.get("script", "train-student.py")),
                *flag("model", student.get("hf_id")),
                *flag("teacher_id", teacher.get("id")),
                *flag("data_dir", train.get("data_dir")),
                *flag("output_dir", train.get("output_dir")),
                *flag("epochs", train.get("epochs")),
                *flag("batch_size", train.get("batch_size")),
                *flag("grad_accum", train.get("grad_accum")),
                *flag("lr", train.get("learning_rate")),
                *flag("max_length", train.get("max_length")),
                *flag("rank", train.get("rank")),
                *flag("alpha", train.get("alpha")),
                *flag("gradient_checkpointing", train.get("gradient_checkpointing", False)),
                *flag("seed", run.get("seed", 0)),
                *flag("labbook", labbook),
            ]]

    ev = config.get("evaluate", {})
    if ev:
        commands["evaluate"] = [[
            py, str(here / ev.get("script", "evaluate-triplet.py")),
            *flag("harness_dir", ev.get("harness_dir")),
            *flag("tasks", ev.get("tasks")),
            *flag("base_url", teacher.get("base_url")),
            *flag("teacher", ev.get("teacher_served_as")),
            *flag("base_student", ev.get("base_student_served_as")),
            *flag("distilled", ev.get("distilled_served_as")),
            *flag("teacher_id", teacher.get("id")),
            *flag("student_id", student.get("id")),
            *flag("quant", ev.get("quant")),
            *flag("engine", ev.get("engine")),
            *flag("engine_version", ev.get("engine_version")),
            *flag("judge_model", ev.get("judge_model")),
            *flag("out_dir", ev.get("out_dir")),
            *flag("no_pause", ev.get("no_pause", False)),
            *flag("labbook", labbook),
        ]]

    exp = config.get("export", {})
    if exp:
        merge = expand(exp.get("merge_script", "merge-adapter.py"))
        adapter = (config.get("train", {}) or {}).get("output_dir", "runs/seq-student")
        out_dir = expand(exp.get("out_dir", "models/student"))
        llama_cpp = expand(exp.get("llama_cpp", "~/llama.cpp"))
        quant = exp.get("quant", "Q4_K_M")
        commands["export"] = [
            # Part 13's merge-adapter.py takes --adapter and --merged-dir; the
            # converter and the quantiser are llama.cpp's, as in Part 11's lab.
            [py, str(merge), "--adapter", str(adapter), "--merged-dir", str(out_dir)],
            [py, f"{llama_cpp}/convert_hf_to_gguf.py", str(out_dir),
             "--outfile", f"{out_dir}.gguf", "--outtype", "bf16"],
            [f"{llama_cpp}/build/bin/llama-quantize", f"{out_dir}.gguf",
             f"{out_dir}-{quant}.gguf", quant],
        ]

    return commands


def run_command(command: list[str], dry: bool) -> float:
    print("+ " + " ".join(command))
    if dry:
        return 0.0
    if shutil.which(command[0]) is None and not Path(command[0]).exists():
        raise SystemExit(f"{command[0]} not found on this machine")
    started = time.time()
    subprocess.run(command, check=True)
    return time.time() - started


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--config", default="distil-config.yaml")
    parser.add_argument("--dry-run", action="store_true", help="print the commands and change nothing")
    parser.add_argument("--from", dest="from_stage", default=None, help="start at this stage")
    parser.add_argument("--only", default=None, help="run exactly this stage")
    parser.add_argument("--run-export", action="store_true",
                        help="actually run the export commands rather than printing them")
    parser.add_argument("--status", action="store_true",
                        help="list the stages already recorded in the lab notebook and exit")
    args = parser.parse_args()

    here = Path(__file__).resolve().parent
    config_path = Path(args.config)
    config = load_config(config_path)
    labbook = config["run"].get("labbook", "labbook.md")

    # The hash is of the parsed configuration, not of the file's bytes, so a comment
    # or a reordering does not look like a different experiment while a changed
    # threshold does.
    config_hash = distillog.config_sha256(config)
    print(f"configuration: {config_path}  hash {config_hash[:16]}")
    print(f"run: {config['run'].get('name')} v{config['run'].get('version')}  "
          f"seed {config['run'].get('seed')}")

    if args.status:
        recorded = distillog.read_stages(labbook)
        if not recorded:
            print(f"no Part 15 stages recorded in {labbook} yet")
            return
        print(f"\n{'stage':<12}{'date':<22}{'run id':<26}notes")
        for record in recorded:
            print(f"{record.get('stage', '?'):<12}{record.get('date', '?'):<22}"
                  f"{record.get('run_id', '?'):<26}{record.get('notes') or ''}")
        return

    commands = build_commands(config, here)
    wanted = [s for s in ORDER if s in commands]
    configured = config.get("stages") or []
    wanted = [s for s in wanted if s in configured or s == "seed_prompts"]
    if args.only:
        if args.only not in commands:
            raise SystemExit(f"no stage named {args.only!r}; known: {', '.join(commands)}")
        wanted = [args.only]
    elif args.from_stage:
        if args.from_stage not in wanted:
            raise SystemExit(f"cannot start at {args.from_stage!r}; stages to run: {', '.join(wanted)}")
        wanted = wanted[wanted.index(args.from_stage):]

    print(f"stages: {', '.join(wanted)}")
    print()

    for stage in wanted:
        print(f"--- {stage} " + "-" * (60 - len(stage)))
        dry = args.dry_run or (stage == "export" and not args.run_export)
        if stage == "export" and not args.run_export and not args.dry_run:
            print("(printing only: pass --run-export to run these; they write large files)")
        elapsed = 0.0
        for command in commands[stage]:
            elapsed += run_command(command, dry)
        if dry:
            print()
            continue

        # The stage scripts write their own detailed records. This one is the
        # pipeline's own: which stage ran, under which configuration, for how long.
        distillog.record(
            labbook=labbook,
            lab="part-15/pipeline",
            stage=f"pipeline:{stage}",
            teacher={"id": config["teacher"].get("id"),
                     "served_as": config["teacher"].get("served_as")},
            student={"id": config["student"].get("id"),
                     "trainer": config["student"].get("trainer")},
            dataset={"config": str(config_path), "config_sha256": config_hash},
            hyperparameters={"run_name": config["run"].get("name"),
                             "run_version": config["run"].get("version"),
                             "commands": [" ".join(c) for c in commands[stage]]},
            seed=config["run"].get("seed"),
            cost=distillog.build_cost(seconds=elapsed),
            config_path=str(config_path),
            notes=None,
        )
        print(f"({stage} took {elapsed / 60:.1f} minutes)\n")

    print("done.")
    print(f"Run `python3 pipeline.py --config {config_path} --status` to see every stage "
          "this configuration has produced, and `python3 distillog.py --stages` for the "
          "detail each stage recorded.")
    print(json.dumps({"config_sha256": config_hash,
                      "stages_run": wanted,
                      "dry_run": args.dry_run}, indent=2))


if __name__ == "__main__":
    main()

Download pipeline.py319 lines

RunnableAll tracks

print every command and change nothing

python pipeline.py --config distil-config.yaml --dry-run

Output — what you should see

configuration: distil-config.yaml  hash ...
run: house-style-4b v1  seed 0
stages: seed_prompts, generate, filter, train, evaluate, export

--- seed_prompts ------------------------------------------------
+ /usr/bin/python3 .../make-seed-prompts.py --out-dir seeds --train 600 --heldout 120 ...

--- generate ----------------------------------------------------
+ /usr/bin/python3 .../generate-teacher-data.py --seeds seeds/prompts.jsonl ...
...

Read all of it. Every command should be one you recognise from the two labs, with the arguments the configuration says. This is the step that catches a wrong path, a stale alias or a teacher id you forgot to change, and it costs ten seconds.

✦Tip

--dry-run is also how you review a configuration change. Change one value, dry-run, and read the diff in the printed commands. That is faster and more reliable than reading the YAML twice.

3. Configure it for your machine and your task

Edit the copy you downloaded. At minimum: the teacher’s model id, quantisation, base URL and alias; the student’s Hugging Face id; the evaluation block’s three served names and your task file; and, on Track M, student.trainer: mlx.

RunnableAll tracks

confirm the endpoint the configuration names is actually there

curl -sS http://127.0.0.1:4000/v1/models \
    -H "Authorization: Bearer $LOCAL_API_KEY" \
    | python -m json.tool

If the alias in your configuration is not in that list, fix one of them now. A generate stage that runs for an hour against the wrong model is the most expensive mistake available on this page.

4. Run it

RunnableAll tracks

the whole pipeline

python pipeline.py --config distil-config.yaml

The runner prints each stage’s banner, then each command before it runs it, then how long the stage took, and appends a pipeline record. The export stage is printed rather than run unless you pass --run-export, because those commands write large files and depend on a llama.cpp checkout the runner cannot assume.

At 24 GB and below, stop the teacher between the generate and train stages, then continue:

RunnableAll tracks

continue after stopping the teacher

python pipeline.py --config distil-config.yaml --from train

RunnableAll tracks

what has been done so far

python pipeline.py --config distil-config.yaml --status

Output — what you should see

stage       date                  run id                    notes
generate    2026-...T...Z         ...
filter      2026-...T...Z         ...
train       2026-...T...Z         ...

=Expected result

Expect the generate stage to dominate the wall clock, the filter stage to take under a minute, and the evaluate stage to take a few minutes per model. If the filter stage takes more than a minute or two on a few hundred examples, the near-duplicate check is quadratic in the number kept and you have generated far more than the labs did; that is fine, and raising --near-duplicate towards 1.0 or sampling before the check are the two ways out.

5. Prove the reproducibility, rather than assuming it

Three things make a run reproducible, and each has a check.

The configuration. Rerun the dry run and confirm the hash is the same as the one in the run records. A different hash means something in the file changed since that run, whatever the file name says.

RunnableAll tracks

the configuration hash, and the ones the run log recorded

python pipeline.py --config distil-config.yaml --dry-run | head -n 2
python -c "
import json
for line in open('labbook.md'):
    line = line.strip()
    if line.startswith('{'):
        r = json.loads(line)
        if str(r.get('lab', '')).startswith('part-15/pipeline'):
            print(r['stage'], (r['dataset'] or {}).get('config_sha256', '')[:16])
"

The data. Every stage records the SHA-256 of the file it read. Rerun make-seed-prompts.py with the same seed and confirm the hash matches what the generate record holds.

The versions. Every record carries the versions of torch, transformers, trl, peft, datasets, mlx and distilabel as installed at the time. Compare the versions in the oldest record with the newest; if they differ, the two runs are not directly comparable and the report has to say so.

!Important

Reproducible does not mean identical. Part 11’s reproducibility lesson quotes PyTorch’s own position on this: bit-for-bit identical results across releases, commits and platforms are not something the library promises. What these three checks give you is the ability to say exactly what was the same and what was not, which is what a second person actually needs.

6. Run it on a task it has never seen

This is the test of whether you built a pipeline or a very long shell script.

Pick a genuinely different task: a different domain, a different output shape, or the reasoning route with a maths set. Then change only the configuration.

RunnableAll tracks

a second run, on a new task, from the same pipeline

cp distil-config.yaml distil-config-task2.yaml
# edit: run.name, run.version, seed_prompts.out_dir, generate.out,
#       filter.out_dir, filter.tasks, train.output_dir, evaluate.tasks
python pipeline.py --config distil-config-task2.yaml --dry-run
python pipeline.py --config distil-config-task2.yaml

For the reasoning route, set seed_prompts.maths to a few hundred, point generate.seeds at seeds/maths.jsonl, set generate.thinking: true and generate.samples: 4, and replace the filter stage’s script with rejection-sample.py in the configuration. The runner passes the arguments it knows; anything the reasoning script needs and the sequence-level one does not is a line you add to the block.

RunnableAll tracks

the reasoning route's filter, run directly

python rejection-sample.py \
    --raw raw/maths-traces.jsonl \
    --out-dir . \
    --keep-per-problem 1 \
    --tasks my-tasks.json \
    --labbook labbook.md

▷Exercise

Time the second run’s attended portion, not its wall clock. That number is what the pipeline was built to reduce, and if it is not much smaller than the first run’s, the pipeline has not earned its place and the configuration is missing something you are still typing by hand. Write it down either way.

7. Export and serve the student

RunnableAll tracks

run the export stage for real

python pipeline.py --config distil-config.yaml --only export --run-export

The three commands are the ones from Part 13’s merging lesson: PEFT’s merge_and_unload through merge-adapter.py, llama.cpp’s convert_hf_to_gguf.py, and llama-quantize with a documented type. Add the result to your gateway as a fourth alias and check it answers.

RunnableAll tracks

the student, through the gateway, once

curl -sS http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $LOCAL_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"model":"local/student-distilled","temperature":0,
         "messages":[{"role":"user","content":"In one sentence, what does a KV cache store during decoding?"}]}' \
    | python -m json.tool

8. Write the report

RunnableAll tracks

report-template.md

# Distillation report: <your student name>

<!--
Purpose: the written deliverable of Part 15's project. One page that lets somebody else, or
         you in six months, decide whether this student is worth serving, whether the result
         can be believed, and what it would cost to do again on a different task.
Platform: all
Minimum memory: not applicable; this is a document
Assumes: the stage records that labbook.md accumulated, the filter report, the rejection
         report if you took the reasoning route, and the table evaluate-triplet.py printed.
         Fill in every angle-bracket placeholder and delete every comment block, including
         this one. A field you cannot fill is a field to write "not recorded" in, not one to
         delete: the gap is information, and it is usually the most useful information here.

Keep it to one page plus the tables. Every number carries its context, which is the course's
rule about numbers applied to your own work.
-->

**Author:** &lt;you&gt; &nbsp; **Date:** &lt;YYYY-MM-DD&gt; &nbsp; **Track:** &lt;S, X, M or N&gt;
**Machine:** &lt;chip and memory, from the hardware reference&gt;
**Configuration hash:** &lt;the config_sha256 pipeline.py printed&gt;

## What this student is for

<!-- Two or three sentences. The task, the machine it has to run on, and why a smaller model
     was wanted at all. "It fits the laptop" is a legitimate reason and should be stated. -->

## The claim

<!-- One sentence with a number in it, written before the run and unchanged afterwards. The
     shape: "On <task set>, at temperature <t> with seed <s>, the distilled <student> closes
     <n>% of the gap between <base student> and <teacher> in the deterministic checks, and is
     not worse in any category." -->

## The pair

| Field | Teacher | Student |
| --- | --- | --- |
| Model id | &lt;course id&gt; | &lt;course id&gt; |
| Repository | &lt;from the model card&gt; | &lt;from the model card&gt; |
| Parameters | &lt;total / active&gt; | &lt;total&gt; |
| Licence | &lt;from the model card&gt; | &lt;from the model card&gt; |
| Terms on using outputs to train | &lt;quote the clause, or "the licence says nothing"&gt; | not applicable |
| Quantisation served at | &lt;e.g. Q4_K_M&gt; | &lt;BF16 for training, a quant for serving&gt; |
| Same tokeniser? | &lt;yes or no, and how you checked&gt; | |

<!-- The licence row is not decoration. A student trained on another model's outputs inherits
     that model's terms wherever the terms say so, and the naming conditions of some licences
     apply to the model you produce, not only to the one you used. -->

## Route and settings

| Field | Value |
| --- | --- |
| Route | &lt;sequence-level, logit, or reasoning traces with rejection sampling&gt; |
| Seed prompts | &lt;count, and how they were seeded&gt; |
| Generation settings | &lt;temperature, top-p, samples per prompt, thinking mode&gt; |
| Raw teacher answers | &lt;count&gt; |
| Kept after filtering | &lt;count, and the percentage&gt; |
| Top three rejection reasons | &lt;from filter-report.json&gt; |
| Decontaminated against | &lt;task file, and the containment threshold&gt; |
| Trainer and settings | &lt;SFTTrainer, DistillationTrainer or GKDTrainer; rank, epochs, learning rate&gt; |
| Versions | &lt;transformers, trl, peft, or mlx-lm&gt; |
| Seeds | &lt;the seed every stage used&gt; |

## Result

<!-- The table evaluate-triplet.py wrote to benchmark-rows.json. Three rows, one task set, one
     set of sampling settings, and the settings named above the table. Do not report a judge
     mean without saying which model judged and whether you measured its agreement with you. -->

| Model | Served as | Deterministic checks | Judge mean, 1-5 |
| --- | --- | --- | --- |
| Teacher | &lt;alias&gt; | &lt;n of total&gt; | &lt;mean, or "not judged"&gt; |
| Base student | &lt;alias&gt; | &lt;n of total&gt; | &lt;mean, or "not judged"&gt; |
| Distilled student | &lt;alias&gt; | &lt;n of total&gt; | &lt;mean, or "not judged"&gt; |

**Gap closed:** &lt;percentage, from evaluate-triplet.py&gt;

**Categories where the distilled student fell below the base:** &lt;list them, or "none"&gt;

<!-- A category that got worse is the finding, whatever the total says. Part 15's challenge is
     about exactly this, and a report that hides it is a report that will mislead you later. -->

## Cost

| Stage | Completion tokens | Wall clock | Energy |
| --- | --- | --- | --- |
| Generate | &lt;from the run record&gt; | &lt;minutes&gt; | &lt;Wh, or "not reported here"&gt; |
| Filter | not applicable | &lt;minutes&gt; | negligible |
| Train | not applicable | &lt;minutes&gt; | &lt;Wh, or "not reported"&gt; |
| Evaluate | &lt;from the run record&gt; | &lt;minutes&gt; | &lt;Wh, or "not reported"&gt; |
| **Total** | | | |

<!-- Energy is null on Track M, where an unprivileged process is not told the figure. Write
     "not reported on this machine" rather than zero: a missing measurement is not a small one. -->

## What did not work

<!-- The honest section. Thresholds you had to change and why; prompts the teacher answered
     badly; a category the student never learned; a stage you had to run twice. A report with
     nothing here is a report whose author stopped reading their own output. -->

## Would you do it again this way?

<!-- Two or three sentences. Compare against the alternatives you did not take: a bigger
     quantised model served directly, a fine-tune on human data, prompting the base student
     harder, or retrieval. Name the one you would try next and what would make you change
     your mind. -->

## Reproducing this

```
python3 pipeline.py --config <your config file> --dry-run
python3 pipeline.py --config <your config file>
```

| Field | Value |
| --- | --- |
| Configuration file | &lt;name, and where it is kept&gt; |
| Configuration hash | &lt;config_sha256&gt; |
| Seed prompt file SHA-256 | &lt;from make-seed-prompts.py&gt; |
| Training file SHA-256 | &lt;from the filter run record&gt; |
| Lab notebook lines | &lt;the run ids of every stage&gt; |

Download report-template.md124 lines

Fill in every angle-bracket blank. The sections people leave out are the ones that make the report worth reading: the licence row, the category where the student got worse, the cost table, and “what did not work”.

RunnableAll tracks

the numbers the report needs, from the notebook

python distillog.py --stages --labbook labbook.md
python -c "
import json
rows = [json.loads(l) for l in open('labbook.md') if l.strip().startswith('{')]
for r in rows:
    if r.get('stage') == 'evaluate':
        print(json.dumps(r['scores'], indent=2)[:1200])
"

Review the dry run as a dependency graph

For each printed stage, identify its inputs, outputs, model endpoint and expected exit condition. Check that the generation output feeds the filter, that the accepted dataset feeds training and that evaluation reads the intended exported student. Resolve every path before starting the expensive run.

Give a new run its own output directory and preserve the resolved configuration. If a stage fails, stop there and inspect its log; do not assume downstream files from an earlier run belong to this attempt. Resume only through the pipeline’s documented mechanism, after checking that retained artefacts match the current configuration and hashes.

For reproducibility, repeat a small run and compare dataset identities, counts and task outcomes. Exact sampled text need not match across every stack, so state what repeatability criterion you used. Include teacher failures, filtering yield, training cost and final quality in the report. Test one held-out task family that was not used to configure the pipeline. Archive configuration, stage manifests, accepted data, model lineage and raw evaluation results. Automation is successful when these handoffs remain inspectable and recoverable, rather than when a single command hides all the decisions that made the experiment meaningful.

Validation

You are done when all of the following are true:

• pipeline.py --dry-run prints five stages of commands you recognise, and you have read them all;
• a full run completed and --status lists every stage in the configuration’s stages list;
• the configuration hash printed by the dry run matches the one recorded against each stage;
• the seed file’s SHA-256 in the generate record matches what make-seed-prompts.py prints when rerun with the same seed;
• a second configuration, for a different task, ran through the same pipeline with no edits to any script;
• the export stage produced a quantised GGUF and the gateway answered from it under an alias;
• the report is filled in with no angle-bracket blanks left, including the cost table and the category where the distilled student did worst;
• you can state the attended time for the second run, and whether it was smaller than the first.

Expected outcome

A pipeline, two runs, one report. The table records what to measure across the two runs; the validation pass will add the course’s own machines.

Pending validationTwo runs through one pipeline — what to record

Run	Task	Prompts	Kept after filtering	Attended minutes	Total minutes	Gap closed, per cent
1	the sequence-level lab’s task	to be measured	to be measured	to be measured	to be measured	to be measured
2	a task the pipeline had not seen	to be measured	to be measured	to be measured	to be measured	to be measured

your machine: track, chip and memory · llama.cpp for serving; TRL SFTTrainer with PEFT LoRA, or mlx_lm.lora on Track M from the versions block of your run records · teacher and student from your configuration, teacher as served; student BF16 in training, quantised on export · 1,024 tokens of context · the date you ran it

Empty on purpose. The attended-minutes column is the one the pipeline exists to reduce; the gap-closed column is meaningful only alongside the per-category table evaluate-triplet.py prints.

Troubleshooting

PyYAML is not installed. The runner says so and tells you the command. Every stage is one command that you can also run by hand from the dry-run output, so a missing YAML parser blocks the convenience rather than the work.

no stage named .... --only takes a stage name from the configuration, and the runner lists the ones it found. A stage in the file but missing from stages is configured and not scheduled, which is deliberate.

A stage runs but nothing appears in the run log. The stage scripts write their own record only when --labbook is passed. The runner passes it from run.labbook, so an empty or missing value there is the usual cause.

The configuration hash changes when you did not change a setting. The hash covers the parsed document, so a value that changed type counts: true and "true" are different, and so are 0.0001 and 1e-4. That strictness is the point, and the dry-run diff will show you which command changed.

The generate stage starts from zero after an interruption. Check that generate.resume is true. With it set, the script skips answers already in the output file and appends.

The export stage’s merge script is not found. export.merge_script points at Part 13’s merge-adapter.py, wherever you keep it. The runner expands ~ and reports a missing command rather than failing halfway through.

The second configuration overwrites the first run’s artefacts. Change every path in the copy, not just the run name: seed_prompts.out_dir, generate.out, filter.out_dir, train.output_dir and evaluate.out_dir. The dry run shows all of them on one screen, which is the fastest way to check.

Cleanup

Keep the configurations, the notebook, the reports and the quantised students. Part 16 quantises one of these students five ways and measures each, and Part 17 asks whether a small distilled model makes a usable draft model for speculative decoding.

RunnableAll tracks

reclaim the disk, keeping every record

rm -rf runs/*/checkpoint-*
rm -rf models/*-merged

⚠Warning

Do not delete labbook.md, the configuration files or the filter-report.json files. They are the only record of what produced the models you are keeping, and a quantised GGUF with no provenance is a file you will eventually have to delete because nobody can say what is in it.

What you learned

• A pipeline is a configuration plus a runner, not a longer script. The settings moved out of nine command lines into one file, and the file is hashed into every record.
• Stages are the unit of resumption. Changing a filter threshold costs a filter stage, not a regeneration, and that is what makes experimentation affordable.
• Reproducibility is three checks, not a feeling. The configuration hash, the data hash and the version block, each of which can disagree with your memory.
• The proof is a second task. A pipeline that needed a script edited to run on something new was a shell script with extra steps.
• The report is the deliverable. The model is an artefact; the report is what lets somebody else decide whether to use it, including you in six months.
• The costs belong in the report. Tokens, wall clock and, where the machine reports it, energy. A result whose cost nobody wrote down cannot be compared with the cheaper thing you did not try.

Record in the notebook: the configuration hash of both runs; the attended and total minutes of each; the kept fraction after filtering and the top three rejection reasons; the three evaluation scores and the gap closed; the export sizes; and one sentence on whether you would use this route again for this task.

Check your understanding

Question 1. Why does the runner hash the parsed configuration rather than the file's bytes?

• Because hashing bytes is slower
• So that a comment or a reordering does not look like a different experiment, while a changed value does
• Because YAML files cannot be hashed
• To keep the hash short

Show the answer and why

Answer: So that a comment or a reordering does not look like a different experiment, while a changed value does

The hash exists to answer "were these two runs configured the same way". Whitespace and comment changes are not configuration changes; a threshold change is, and so is a change of type, which is why 0.0001 and "0.0001" hash differently.

Question 2. The runner prints the export commands rather than running them unless you pass --run-export. Why?

• Because they are dangerous
• Because they write large files and depend on a llama.cpp checkout the runner cannot assume is present, so the default is to show you what it would do
• Because export is optional in the curriculum
• Because they need root

Show the answer and why

Answer: Because they write large files and depend on a llama.cpp checkout the runner cannot assume is present, so the default is to show you what it would do

The same principle as the dry run: nothing that writes gigabytes or depends on something outside the configuration runs without you having seen the command. It is also why the runner never starts an engine.

Question 3. What is the real test that you built a pipeline rather than a long script?

• That it has a YAML file
• That a run on a different task needed only a changed configuration, with no edit to any script, and took much less attended time than the first
• That it runs without errors
• That it writes a report

Show the answer and why

Answer: That a run on a different task needed only a changed configuration, with no edit to any script, and took much less attended time than the first

Attended time is the number the pipeline exists to reduce, and a second task is what exposes settings still hard-coded somewhere. If the second run was not much cheaper in attention, something is still being typed by hand and belongs in the configuration.

Question 4. Which of these does the run log let you check after the fact? Select all that apply.

• Which teacher produced the training data, and under which alias
• The SHA-256 of the file each stage read
• The versions of the training libraries at the time of the run
• The exact tokens the accelerator was computing at a given moment

Show the answer and why

Answer: Which teacher produced the training data, and under which alias, The SHA-256 of the file each stage read, The versions of the training libraries at the time of the run

The first three are the reproducibility triple: what produced the data, which bytes were used, and what software ran. The last is not something any run log holds, and it is not what reproducibility means.

Question 5. Your second run reports a smaller gap closed than the first. What belongs in the report?

• The first run's number, since it is the better result
• Both, with the task each refers to, since a gap-closed figure is a property of the task set and not of the pipeline
• Neither, until a third run
• An average

Show the answer and why

Answer: Both, with the task each refers to, since a gap-closed figure is a property of the task set and not of the pipeline

Two tasks, two numbers, and the comparison between them is the interesting part: it tells you which kinds of task this teacher and student pair transfers well on. Reporting only the better one is the failure mode the whole part is written against.

Check my answersClear