Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The per-track versions, wall-clock times and results will be recorded here when the validation pass is done.

Objective

By the end of this lab you will have a small model that answers more like your large one than it did this morning, and a number that says by how much. Concretely: a seed prompt set built from a taxonomy, a JSON Lines file of teacher answers with the settings that produced them, a filtered and decontaminated training set with a report saying what was thrown away and why, a LoRA adapter, a quantised GGUF export, and three evaluation runs on the same task file at the same settings.

The deliverable is the comparison, not the model. Teacher, base student and distilled student, on your own Part 10 task set, with the fraction of the gap that closed and the categories where the student got worse, because that second column is the one this part’s challenge is about.

⌂Single-machine path

Single-machine on every track. The teacher is served rather than trained, and the student is trained after the teacher has been stopped, which is what keeps the memory floor at 12 GB. If you own two machines you can serve the teacher on one and train on the other, and the only thing that changes is the base URL.

Requirements

Every track needs the Part 11 training environment, a llama.cpp build from Part 6 for the export step, your own task file from Part 10’s benchmark lab, and about 75 minutes, of which roughly 30 are unattended generation and 20 are unattended training.

The pair depends on how much memory you have, and the reason is arithmetic rather than preference.

Memory	Teacher, served	Student, trained	Both at once?
128 GB (S, X, M)	Qwen3-30B-A3B at Q4_K_M	Qwen3-4B	Yes, comfortably
24 GB (N)	Qwen3-14B at Q4_K_M	Qwen3-4B	Tight; stop the teacher before training
12 to 16 GB (N)	Qwen3-8B at Q4_K_M	Qwen3-1.7B at 12 GB, Qwen3-4B at 16 GB	No; one at a time

All five models are Apache-2.0 according to their model cards, read on 2026-09-09, and none is gated, so nothing here needs an acceptance step. That matters more than usual in this lab: the synthetic-data lesson set out why the teacher’s licence follows its outputs, and Apache-2.0 imposes no naming or attribution condition on the student you produce.

Track S — NVIDIA DGX Spark

Serve the teacher with llama-server from Part 6, or through the Part 9 gateway, and train inside the NGC PyTorch container from Part 11 with your course directory mounted. The 128 GB pool holds the teacher and the training run together, so you can leave the teacher up throughout and skip the stop-and-restart the smaller tiers need.

Allow about 20 GB of disk for the teacher’s GGUF file if you do not already have it, plus about 12 GB for the student, its merged copy and two GGUF exports.

Track X — AMD Ryzen AI Max+ 395Partial

Serving is fully supported through the Vulkan or ROCm build of llama.cpp. The training step needs the ROCm build of PyTorch: the ROCm 10.0.0 compatibility matrix dated 2026-08-14 lists gfx1151 without a support-tier qualifier, and AMD's PyTorch install page read on 2026-09-09 does not mention the chip. The CPU training path completes this lab.

Generation is the part of this lab that this machine is best at: a 128 GB unified pool serves a 30B-class teacher comfortably and the generation stage is pure inference.

For the training stage, use the ROCm wheels as Part 11’s environment lesson describes and confirm with python -c 'import torch; print(torch.cuda.is_available())' before starting. If that prints False, add --precision fp32 and finish on the CPU. A 1.7B or 4B student with a rank-16 adapter on a few hundred examples is slow on a CPU rather than impossible, and it is a supported way through this lab. Record in the run log that it was a CPU run: a CPU result and a GPU result are not interchangeable.

Track M — Apple silicon

Serve the teacher with llama-server built with Metal, or with the mlx-lm server from Part 8. Train with mlx_lm.lora on the data-mlx/ files, exactly as in Part 11’s lab; the PyTorch MPS path also runs and is what you need for the GGUF export at the end.

The generation stage on a 128 GB Mac holds a 30B-class teacher without difficulty. The one thing this track cannot report is energy: distillog.py samples accelerator power through nvidia-smi or rocm-smi, neither of which exists here, so the cost block records a null for watt-hours and the report template asks you to write “not reported on this machine” rather than a zero.

Track N — NVIDIA desktop or laptop

The tier table above is written for this track, because it is the one where the numbers bind. At 24 GB, serve a 14B-class teacher at Q4_K_M, then stop it before training: the teacher’s weights and the training run do not fit together and the failure is an out-of-memory error part way through the first epoch, which wastes the whole generation stage’s wall clock if you have not saved it. The generation script writes as it goes and resumes, so nothing is lost, but the restart is avoidable.

At 12 to 16 GB, an 8B-class teacher at Q4_K_M and a 1.7B student. On Windows, work inside WSL2 exactly as in Part 11, and remember that the virtual machine’s memory limit, not the card’s, is what the training process sees.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-15-distillation"
cd "$LAB_DIR"
pwd
test -f "distillog.py"

Expected result: pwd ends in part-15-distillation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Do the arithmetic for both phases before starting

This lab has two memory profiles, not one, and knowing that is what lets a 12 GB machine complete it. During generation the teacher is resident and nothing is being trained. During training the student is resident and the teacher is not needed at all.

Phase one, generation: a 14B-class teacher at Q4_K_M with four slots, on a 24 GB machine

Teacher weights, Q4_K_M

9 GB

KV cache, 4 slots x 4,096 tokens

2.7 GB

Reserved for the operating system

2 GB

Free

10.3 GB

Total

24 GB

Estimate from arithmetic, not a measurement. Weights are the Q4_K_M size for Qwen3-14B in src/data/models.json; the cache is that model's published 163,840 bytes per token at FP16 times four slots of 4,096 tokens. Raising the slot count or the per-slot context raises the second bar and nothing else.

Phase two, training: a rank-16 adapter on Qwen3-4B, batch 1 at 1,024 tokens, on a 16 GB machine

Frozen base weights, BF16

8 GB

Adapter weights, gradients and optimiser states

0.5 GB

Activations and logits

2.4 GB

Reserved for the operating system

2 GB

Free

3.1 GB

Total

16 GB

Estimate from arithmetic, not a measurement, using Part 11's method. The frozen base is 2 bytes against 4 billion parameters. The adapter figure is 33.0 million trainable parameters at 16 bytes each, computed from Qwen3-4B's config.json: 36 layers, hidden size 2,560, intermediate size 9,728, and a rank-16 adapter on the seven projections. Activations include the logits tensor over a 151,936-entry vocabulary and a full-precision copy of it for the loss.

At 12 GB the second diagram does not fit, which is why the tier table drops the student to Qwen3-1.7B there: 28 layers, hidden size 2,048, intermediate size 6,144, so a rank-16 adapter is about 17.4 million parameters and the frozen base is about 3.4 GB. Write your own four figures in the notebook now, before anything runs, so the peak the training script reports has something to be compared against.

2. Get the lab files and check the environment

RunnableAll tracks

distillog.py

"""Append one machine-readable record per distillation stage to the lab notebook.

Purpose: Part 15's self-contained copy of the run-log format defined in Part 11 and
         reused in Part 13, extended with the two fields distillation adds: the
         teacher a stage used and what the stage cost in tokens, seconds and
         watt-hours. Every stage of the pipeline appends one line, so the whole
         run reads back as an ordered story rather than a directory of artefacts.
Platform: all (standard library only; torch, transformers, trl, peft and mlx are
          inspected for their version strings only if they happen to be installed)
Minimum memory: 8 GB
Assumes: Python 3.10 or newer. The lab notebook is created if it does not exist.
         git is optional and is used only to record the commit the configuration
         was at. nvidia-smi or rocm-smi are optional and are used only by
         sample_power(), which returns None when neither is present.

Usage: imported by this part's Python scripts:
           import distillog
           distillog.record(labbook="labbook.md", lab="part-15/generate", ...)
       or called from a shell script with the stage's own fields as JSON on stdin:
           python3 distillog.py --record --labbook labbook.md < fields.json
       or run with no arguments to print the field list and exit.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import platform
import re
import secrets
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

# Every field below is required in a record. A stage that cannot fill one in
# writes null rather than omitting the key, so a reader always knows the
# difference between "not recorded" and "not applicable".
FIELDS = (
    "run_id",
    "lab",
    "stage",
    "date",
    "config_commit",
    "teacher",
    "student",
    "dataset",
    "hyperparameters",
    "seed",
    "hardware",
    "versions",
    "cost",
    "losses",
    "scores",
    "notes",
)

# The cost block is the reason this file exists rather than Part 13's sftlog.py.
# A distillation result that does not say what it cost cannot be compared with
# the alternative that would have cost less.
COST_FIELDS = ("prompt_tokens", "completion_tokens", "seconds", "watt_hours")


def new_run_id() -> str:
    """A short identifier that sorts by time and does not collide between runs."""
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


def file_sha256(path: str | os.PathLike[str], chunk: int = 1 << 20) -> str | None:
    """Content hash of a data file, so a stage can be tied to the exact bytes it read."""
    p = Path(path)
    if not p.is_file():
        return None
    digest = hashlib.sha256()
    with p.open("rb") as handle:
        while True:
            block = handle.read(chunk)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def config_sha256(config: dict[str, Any]) -> str:
    """Hash of a configuration object, sorted so that key order cannot change it.

    Two runs of the pipeline with the same configuration hash read the same
    settings; two with different hashes did not, whatever the file names say.
    """
    return hashlib.sha256(json.dumps(config, sort_keys=True).encode("utf-8")).hexdigest()


def git_commit(path: str | os.PathLike[str] = ".") -> str | None:
    """The commit the configuration is at, or None outside a repository.

    Records whether the tree was dirty, because "commit abc1234" is misleading
    if the file on disk had uncommitted edits when the stage started.
    """
    if shutil.which("git") is None:
        return None
    target = Path(path)
    cwd = target if target.is_dir() else target.parent
    try:
        rev = subprocess.run(
            ["git", "rev-parse", "--short", "HEAD"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
        dirty = subprocess.run(
            ["git", "status", "--porcelain"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
    except (subprocess.SubprocessError, OSError):
        return None
    return f"{rev}-dirty" if dirty else rev


def describe_hardware() -> dict[str, Any]:
    """What the stage ran on, as far as it can be established without extra packages."""
    info: dict[str, Any] = {
        "os": f"{platform.system()} {platform.release()}",
        "machine": platform.machine(),
        "python": platform.python_version(),
        "accelerator": "cpu",
        "device_name": None,
    }
    try:
        import torch  # noqa: PLC0415 - optional, and only for reporting
    except ImportError:
        return info
    if torch.cuda.is_available():
        info["accelerator"] = "cuda"
        info["device_name"] = torch.cuda.get_device_name(0)
    else:
        mps = getattr(torch.backends, "mps", None)
        if mps is not None and mps.is_available():
            info["accelerator"] = "mps"
            info["device_name"] = platform.processor() or "Apple silicon"
    return info


def package_versions(
    names: tuple[str, ...] = ("torch", "transformers", "trl", "peft", "datasets", "mlx", "distilabel"),
) -> dict[str, str | None]:
    """Version strings for the packages that decide what a stage actually did."""
    from importlib.metadata import PackageNotFoundError, version  # noqa: PLC0415

    out: dict[str, str | None] = {}
    for name in names:
        try:
            out[name] = version(name)
        except PackageNotFoundError:
            out[name] = None
    return out


def sample_power() -> float | None:
    """Instantaneous accelerator power draw in watts, or None where it is not reported.

    Track S and Track N expose it through nvidia-smi; Track X exposes it through
    rocm-smi. Apple silicon does not report a comparable per-accelerator figure
    to an unprivileged process, so Track M records None and the page says so.
    One sample is not an energy measurement: generate-teacher-data.py averages
    samples over the run and multiplies by the elapsed hours.
    """
    if shutil.which("nvidia-smi"):
        try:
            out = subprocess.run(
                ["nvidia-smi", "--query-gpu=power.draw", "--format=csv,noheader,nounits"],
                capture_output=True, text=True, check=True, timeout=10,
            ).stdout.strip().splitlines()
            values = [float(v) for v in out if re.fullmatch(r"\s*[0-9.]+\s*", v)]
            if values:
                return round(sum(values), 1)
        except (subprocess.SubprocessError, OSError, ValueError):
            return None
    if shutil.which("rocm-smi"):
        try:
            out = subprocess.run(
                ["rocm-smi", "--showpower", "--json"],
                capture_output=True, text=True, check=True, timeout=10,
            ).stdout
            data = json.loads(out)
            values = []
            for card in data.values():
                for key, value in card.items():
                    if "power" in key.lower():
                        try:
                            values.append(float(str(value).split()[0]))
                        except (TypeError, ValueError):
                            continue
            if values:
                return round(sum(values), 1)
        except (subprocess.SubprocessError, OSError, ValueError, json.JSONDecodeError):
            return None
    return None


def build_cost(
    prompt_tokens: int | None = None,
    completion_tokens: int | None = None,
    seconds: float | None = None,
    mean_watts: float | None = None,
) -> dict[str, Any]:
    """The four cost numbers, with watt-hours derived rather than typed.

    mean_watts is the average of sample_power() readings taken during the stage.
    Where no reading was available the energy figure is null, which is the honest
    answer: a missing measurement is not zero.
    """
    watt_hours = None
    if mean_watts is not None and seconds is not None:
        watt_hours = round(mean_watts * (seconds / 3600.0), 2)
    return {
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "seconds": round(seconds, 1) if seconds is not None else None,
        "watt_hours": watt_hours,
    }


def build_record(
    lab: str,
    stage: str,
    teacher: dict[str, Any] | None = None,
    student: dict[str, Any] | None = None,
    dataset: dict[str, Any] | None = None,
    hyperparameters: dict[str, Any] | None = None,
    seed: int | None = None,
    cost: dict[str, Any] | None = None,
    losses: dict[str, Any] | None = None,
    scores: dict[str, Any] | None = None,
    config_path: str | os.PathLike[str] | None = None,
    notes: str | None = None,
) -> dict[str, Any]:
    """Assemble the record. Kept separate from writing so it can be inspected first."""
    filled_cost = dict(cost or {})
    for key in COST_FIELDS:
        filled_cost.setdefault(key, None)
    return {
        "run_id": new_run_id(),
        "lab": lab,
        "stage": stage,
        "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "config_commit": git_commit(config_path if config_path is not None else "."),
        "teacher": teacher,
        "student": student,
        "dataset": dataset or {},
        "hyperparameters": hyperparameters or {},
        "seed": seed,
        "hardware": describe_hardware(),
        "versions": package_versions(),
        "cost": filled_cost,
        "losses": losses or {},
        "scores": scores or {},
        "notes": notes,
    }


def append(record: dict[str, Any], labbook: str | os.PathLike[str] = "labbook.md") -> Path:
    """Append one JSON line. Missing keys are an error: a partial record is worse than none."""
    missing = [f for f in FIELDS if f not in record]
    if missing:
        raise ValueError(f"run record is missing required fields: {', '.join(missing)}")
    path = Path(labbook)
    if not path.exists():
        path.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    return path


def record(labbook: str | os.PathLike[str] = "labbook.md", **kwargs: Any) -> dict[str, Any]:
    """Build and append in one call; returns the record so a caller can print it."""
    rec = build_record(**kwargs)
    append(rec, labbook)
    return rec


def read_stages(labbook: str | os.PathLike[str], lab_prefix: str = "part-15/") -> list[dict[str, Any]]:
    """Every Part 15 record in the notebook, oldest first.

    The project's pipeline runner uses this to print what has already been done
    and what a resumed run still has to do.
    """
    path = Path(labbook)
    if not path.is_file():
        return []
    out = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line.startswith("{"):
            continue
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            continue
        if isinstance(rec, dict) and str(rec.get("lab", "")).startswith(lab_prefix):
            out.append(rec)
    return out


def main() -> None:
    parser = argparse.ArgumentParser(description="Run-log helper for the distillation labs.")
    parser.add_argument("--record", action="store_true", help="read the stage's own fields as JSON on stdin")
    parser.add_argument("--stages", action="store_true", help="list the Part 15 stages already in the notebook")
    parser.add_argument("--power", action="store_true", help="print one accelerator power reading and exit")
    parser.add_argument("--labbook", default="labbook.md")
    args = parser.parse_args()

    if args.power:
        watts = sample_power()
        print("not reported on this machine" if watts is None else f"{watts} W")
        return
    if args.stages:
        for rec in read_stages(args.labbook):
            cost = rec.get("cost") or {}
            print(f"{rec['date']}  {rec['stage']:<10}  {rec['run_id']}  "
                  f"{cost.get('completion_tokens') or '-'} completion tokens  "
                  f"{cost.get('seconds') or '-'} s")
        return
    if args.record:
        fields = json.load(sys.stdin)
        allowed = {"lab", "stage", "teacher", "student", "dataset", "hyperparameters",
                   "seed", "cost", "losses", "scores", "config_path", "notes"}
        unknown = set(fields) - allowed
        if unknown:
            raise SystemExit(f"unknown field(s) on stdin: {', '.join(sorted(unknown))}")
        rec = record(labbook=args.labbook, **fields)
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")
        return

    print(__doc__)
    print("Fields in every record:", ", ".join(FIELDS))
    print("Fields in every cost block:", ", ".join(COST_FIELDS))


if __name__ == "__main__":
    main()

Download distillog.py342 lines

Every script in this part imports distillog.py, so it has to sit beside them. It is Part 11’s run log with two fields added: which teacher a stage used, and what the stage cost.

RunnableAll tracks

activate and check

cd "$LAB_DIR"
source "$HOME/llm-course/.venv/bin/activate"
python -c "import torch, transformers, trl, peft, datasets; print(torch.__version__, transformers.__version__, trl.__version__, peft.__version__)"
python distillog.py --power

The second command prints one accelerator power reading, or says that this machine does not report one. Either answer is fine; knowing which you have before the run starts is what stops you looking for a missing number afterwards.

3. Build the seed prompts

RunnableAll tracks

make-seed-prompts.py

"""Build the seed prompt set the teacher will answer, and a verifiable maths set beside it.

Purpose: prompt seeding, done as a script you can read rather than as a folder of
         text somebody pasted. Prompts are generated from a taxonomy crossed with
         topics and constraints, so diversity is a property of the construction
         instead of a hope. A second file holds arithmetic word problems whose
         answers were computed before the questions were written, which is what
         lets rejection-sample.py verify a teacher trace without a second model.
Platform: all (standard library only; no model, no accelerator, no network)
Minimum memory: 8 GB, and far less in practice
Assumes: Python 3.10 or newer. Nothing is downloaded and nothing is served; this
         runs before the teacher is started.

Usage: python3 make-seed-prompts.py --out-dir seeds --seed 0
       python3 make-seed-prompts.py --out-dir seeds --train 600 --heldout 120 --seed 0
       python3 make-seed-prompts.py --out-dir seeds --maths 300 --seed 0

Three files are written:
  seeds/prompts.jsonl   the prompts the teacher answers, one JSON object per line
  seeds/heldout.jsonl   prompts of the same shape, never generated on, for evaluation
  seeds/maths.jsonl     word problems with a computed answer, for rejection sampling

Every line carries an id, a category and the slots it was built from, so a prompt
that turns out to produce bad teacher output can be traced back to the template
that made it rather than deleted one example at a time.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import random
from pathlib import Path

# ---------------------------------------------------------------------------
# The taxonomy. Categories match Part 10's evaluation set on purpose: a student
# is distilled for the tasks it will be measured on, and a category that appears
# in neither the training prompts nor the evaluation set is a category nobody
# learns anything about.
# ---------------------------------------------------------------------------

TOPICS = [
    "a llama.cpp server that stopped answering",
    "a GGUF file that will not load",
    "a KV cache that grew past the memory budget",
    "a fine-tune whose loss went flat",
    "a model download that failed halfway",
    "an embedding index that returns the wrong chunk",
    "a container that cannot see the accelerator",
    "a batch job that runs slower with more workers",
    "a chat template that was applied twice",
    "a quantised model that answers differently from the original",
    "a gateway alias pointing at the wrong port",
    "a tokeniser that splits an identifier into six pieces",
    "a training run that ran out of disk at the last checkpoint",
    "a dataset split that leaked into the evaluation set",
    "a rollout that timed out under load",
    "a draft model with a low acceptance rate",
]

CONSTRAINTS = [
    "Answer in at most three sentences.",
    "Reply with a numbered list of at most four steps.",
    "Answer in one sentence, then give one command on its own line.",
    "Use British spelling and avoid the word 'simply'.",
    "Give the answer first and the reason second, on two lines.",
    "Reply with a single paragraph under sixty words.",
]

TEMPLATES: dict[str, list[str]] = {
    "diagnose": [
        "I am looking at {topic}. What are the two most likely causes, and what would I check first? {constraint}",
        "Something is wrong: {topic}. Give me a diagnostic order, cheapest check first. {constraint}",
    ],
    "explain": [
        "Explain to a colleague who runs models but has never trained one what is going on when there is {topic}. {constraint}",
        "In plain terms, what does {topic} tell you about how the system is put together? {constraint}",
    ],
    "format": [
        "Summarise the situation '{topic}' as a JSON object with the keys symptom, likely_cause and next_check, and output nothing else.",
        "Turn '{topic}' into a table with the headers Symptom, Evidence and Action, and output nothing else.",
    ],
    "code": [
        "Write a short shell function that checks for the condition behind {topic} and exits non-zero when it holds. Code only, no explanation.",
        "Write a Python function that reads a log file and returns True when it shows {topic}. Code only, no explanation.",
    ],
    "refusal": [
        "Given only the phrase '{topic}', tell me the exact hostname and port of the machine involved.",
        "Given only the phrase '{topic}', tell me how many times this happened last week on my machine.",
    ],
    "instruction": [
        "Write three sentences about {topic}. Each sentence must be under twelve words, the second must contain a number, and the third must be a question.",
        "Describe {topic} without using the words 'model', 'server' or 'error'. {constraint}",
    ],
}

# The two refusal templates are the point of the category: the honest answer is
# that the information is not available, and a teacher that invents a hostname is
# a teacher whose output has to be filtered before a student copies the habit.

NAMES = ["Ada", "Bela", "Chi", "Dara", "Eero", "Fen", "Gita", "Hugo", "Ines", "Jonas",
         "Kira", "Liam", "Mira", "Nils", "Oona", "Pia", "Rafa", "Sena", "Tomas", "Ulla"]
GOODS = [("cable", "cables"), ("filter", "filters"), ("drive", "drives"), ("fan", "fans"),
         ("tile", "tiles"), ("bulb", "bulbs"), ("mount", "mounts"), ("clip", "clips")]

MATHS_INSTRUCTION = (
    "Solve the problem. Put your working inside <think> and </think> tags, then end "
    "with a single line of the form 'Answer: N' where N is the final number and "
    "nothing follows it."
)


def maths_problem(rng: random.Random) -> tuple[str, int]:
    """One word problem and the answer, computed here before the question exists.

    Three shapes, all multi-step, all integer. The answer is arithmetic performed
    by this function, so the label cannot be wrong in the way a scraped label can.
    """
    name = rng.choice(NAMES)
    singular, plural = rng.choice(GOODS)
    shape = rng.randrange(3)

    if shape == 0:
        boxes = rng.randint(3, 12)
        per_box = rng.randint(4, 15)
        broken = rng.randint(1, min(9, boxes * per_box - 1))
        answer = boxes * per_box - broken
        text = (f"{name} unpacks {boxes} boxes of {plural}, each holding {per_box} {plural}. "
                f"{broken} of the {plural} are broken and thrown away. "
                f"How many usable {plural} does {name} have?")
        return text, answer

    if shape == 1:
        start = rng.randint(20, 90)
        bought = rng.randint(5, 40)
        given = rng.randint(1, 15)
        days = rng.randint(2, 6)
        per_day = rng.randint(1, 6)
        answer = start + bought - given - days * per_day
        text = (f"{name} starts the week with {start} {plural}, buys {bought} more, and gives "
                f"{given} to a neighbour. Over the next {days} days {name} uses {per_day} "
                f"{plural} each day. How many {plural} are left?")
        return text, answer

    racks = rng.randint(2, 8)
    shelves = rng.randint(2, 6)
    per_shelf = rng.randint(2, 9)
    spare = rng.randint(0, 20)
    answer = racks * shelves * per_shelf + spare
    text = (f"A store room has {racks} racks. Each rack has {shelves} shelves and each shelf "
            f"holds {per_shelf} {plural}. A drawer holds another {spare} {plural}. "
            f"How many {plural} are in the room altogether?")
    return text, answer


def build_prompts(count: int, rng: random.Random, prefix: str) -> list[dict]:
    """Cross the taxonomy with topics and constraints, without repeating a pair."""
    combinations = []
    for category, templates in TEMPLATES.items():
        for template in templates:
            for topic in TOPICS:
                for constraint in CONSTRAINTS:
                    combinations.append((category, template, topic, constraint))
    rng.shuffle(combinations)

    rows = []
    seen: set[str] = set()
    for category, template, topic, constraint in combinations:
        if len(rows) >= count:
            break
        prompt = template.format(topic=topic, constraint=constraint).strip()
        # A template with no {constraint} slot collapses several combinations onto
        # the same text; keep the first and move on rather than shipping duplicates.
        if prompt in seen:
            continue
        seen.add(prompt)
        rows.append({
            "id": f"{prefix}{len(rows) + 1:04d}",
            "category": category,
            "prompt": prompt,
            "slots": {"topic": topic, "constraint": constraint if "{constraint}" in template else None},
        })
    return rows


def write_jsonl(path: Path, rows: list[dict]) -> str:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--out-dir", default="seeds")
    parser.add_argument("--train", type=int, default=600, help="prompts the teacher will answer")
    parser.add_argument("--heldout", type=int, default=120, help="prompts kept back for evaluation")
    parser.add_argument("--maths", type=int, default=300, help="verifiable word problems")
    parser.add_argument("--seed", type=int, default=0)
    args = parser.parse_args()

    rng = random.Random(args.seed)
    out = Path(args.out_dir)

    total = args.train + args.heldout
    everything = build_prompts(total, rng, prefix="p")
    if len(everything) < total:
        print(f"note: the taxonomy yields {len(everything)} distinct prompts; "
              f"asked for {total}. Add topics or templates rather than sampling with repeats.")
    train_rows = everything[:args.train]
    heldout_rows = everything[args.train:]
    for i, row in enumerate(heldout_rows, start=1):
        row["id"] = f"h{i:04d}"

    maths_rows = []
    seen_maths: set[str] = set()
    attempts = 0
    while len(maths_rows) < args.maths and attempts < args.maths * 50:
        attempts += 1
        text, answer = maths_problem(rng)
        if text in seen_maths:
            continue
        seen_maths.add(text)
        maths_rows.append({
            "id": f"m{len(maths_rows) + 1:04d}",
            "category": "maths",
            "prompt": f"{MATHS_INSTRUCTION}\n\n{text}",
            "answer": answer,
        })

    hashes = {
        "prompts.jsonl": write_jsonl(out / "prompts.jsonl", train_rows),
        "heldout.jsonl": write_jsonl(out / "heldout.jsonl", heldout_rows),
        "maths.jsonl": write_jsonl(out / "maths.jsonl", maths_rows),
    }

    categories: dict[str, int] = {}
    for row in train_rows:
        categories[row["category"]] = categories.get(row["category"], 0) + 1

    print(f"written to {out}/")
    print(f"  prompts.jsonl  {len(train_rows)} prompts")
    for name in sorted(categories):
        print(f"      {name:<12} {categories[name]}")
    print(f"  heldout.jsonl  {len(heldout_rows)} prompts, never generated on")
    print(f"  maths.jsonl    {len(maths_rows)} problems with computed answers")
    print()
    for name, digest in hashes.items():
        print(f"sha256({name}) = {digest[:16]}...")
    print("Record those hashes: they are what tie every later stage to this exact seed set.")


if __name__ == "__main__":
    main()

Download make-seed-prompts.py255 lines

RunnableAll tracks

seed the prompt set

python make-seed-prompts.py --out-dir seeds --train 600 --heldout 120 --seed 0

Output — what you should see

written to seeds/
  prompts.jsonl  600 prompts
      code         ...
      diagnose     ...
      explain      ...
      format       ...
      instruction  ...
      refusal      ...
  heldout.jsonl  120 prompts, never generated on
  maths.jsonl    ... problems with computed answers

sha256(prompts.jsonl) = ...

Read ten of them before going further.

RunnableAll tracks

read what you are about to ask the teacher

head -n 3 seeds/prompts.jsonl | python -m json.tool --json-lines

Each line carries the category and the slots it was built from, which is what lets you find and fix a bad template later instead of deleting bad examples one at a time. The categories match the ones in Part 10’s task template on purpose: a student is distilled for the tasks it will be measured on.

4. Serve the teacher, and check one answer before generating six hundred

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

serve the 30B-class teacher

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf \
    --alias teacher \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 32768 \
    --parallel 8 \
    --n-gpu-layers 999 \
    --flash-attn on \
    --jinja

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

serve the 30B-class teacher

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf \
    --alias teacher \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 32768 \
    --parallel 8 \
    --n-gpu-layers 999 \
    --jinja

Track M — Apple silicon

RunnableTrack M · Apple silicon

serve the 30B-class teacher with the Metal build

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf \
    --alias teacher \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 32768 \
    --parallel 8 \
    --n-gpu-layers 999 \
    --flash-attn on \
    --jinja

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

serve the tier's teacher, four slots

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf \
    --alias teacher \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 16384 \
    --parallel 4 \
    --n-gpu-layers 999 \
    --flash-attn on \
    --jinja

At 12 to 16 GB substitute the Qwen3-8B file and keep --parallel 4. The context size and the slot count together decide the KV cache, and the first diagram above is the arithmetic.

Then one request, by hand, before committing to hundreds.

RunnableAll tracks

one answer, to check the endpoint and the shape

curl -sS http://127.0.0.1:8080/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"teacher","temperature":0.7,"top_p":0.8,"max_tokens":256,
         "messages":[{"role":"user","content":"In one sentence, what does a KV cache store during decoding?"}]}' \
    | python -m json.tool

Three things to check in that reply: there is a content string and it is not empty; usage carries token counts, because the generation script’s cost record depends on them; and the answer is in the mode you expect, with or without a <think> block. If the counts are missing your engine does not report usage and the cost block will be incomplete, which is worth knowing now rather than at the end.

✦Tip

If you built the Part 9 gateway, point the generation script at that instead: --base-url http://127.0.0.1:4000/v1 --model local/chat. The gateway loads the model on demand, gives it a stable name, and means the evaluation step later can reach three models under three aliases without your starting a server three times.

5. Generate

RunnableAll tracks

generate-teacher-data.py

"""Answer a seed prompt file with your local teacher, concurrently, and resumably.

Purpose: the generation stage of sequence-level distillation. Sends every prompt in
    a seed file to an OpenAI-compatible endpoint with stated sampling settings and a
    fixed number of concurrent requests, writes one JSON line per answer as it
    arrives, and records the teacher id, the settings, the token counts and the
    wall-clock time. Re-running it skips prompts already answered, so a run
    interrupted after two hours resumes rather than restarting.
Platform: all (pure Python over HTTP; the teacher may be llama-server from Part 6,
    vLLM from Part 9, the Part 9 gateway, or an mlx-lm server on Track M)
Minimum memory: 12 GB on the machine serving the teacher; this script needs very little
Assumes: Python 3.10 or newer and a reachable OpenAI-compatible endpoint. The seed
    file is JSON Lines with an "id" and a "prompt" on every line, as written by
    make-seed-prompts.py. distillog.py sits next to this file.

Usage: python3 generate-teacher-data.py --seeds seeds/prompts.jsonl \\
           --base-url http://127.0.0.1:4000/v1 --model local/chat \\
           --teacher-id qwen3-30b-a3b --out raw/teacher.jsonl --labbook labbook.md
       python3 generate-teacher-data.py --seeds seeds/prompts.jsonl --out raw/teacher.jsonl \\
           --concurrency 8 --temperature 0.7 --top-p 0.8 --samples 2 --resume
       python3 generate-teacher-data.py --seeds seeds/maths.jsonl --out raw/maths-traces.jsonl \\
           --thinking --temperature 0.6 --top-p 0.95 --samples 4

The settings matter and are recorded on every line. Qwen3's model card gives
Temperature 0.6, TopP 0.95, TopK 20, MinP 0 for thinking mode and Temperature 0.7,
TopP 0.8, TopK 20, MinP 0 for non-thinking mode, and says not to use greedy decoding
in thinking mode; --thinking switches this script's defaults to the first set.
"""

from __future__ import annotations

import argparse
import json
import statistics
import sys
import threading
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from typing import Any, Optional

import distillog

# Sampling defaults, from the Qwen3 model cards read on 2026-09-09. They are
# arguments rather than constants because a different teacher wants different
# numbers, and because a run whose settings are not recorded is not reproducible.
NON_THINKING = {"temperature": 0.7, "top_p": 0.8}
THINKING = {"temperature": 0.6, "top_p": 0.95}


def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict:
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:400]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


def read_jsonl(path: Path) -> list[dict]:
    rows = []
    with path.open(encoding="utf-8") as handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{number}: not valid JSON ({exc})") from exc
    return rows


def already_done(path: Path) -> set[str]:
    """Keys of the (prompt, sample) pairs already in the output file.

    Resumability is the difference between a generation stage you can run on a
    laptop overnight and one you have to babysit. The key includes the sample
    index so that --samples 4 resumes at the right sample, not the right prompt.
    """
    if not path.is_file():
        return set()
    done = set()
    for row in read_jsonl(path):
        if "id" in row and "sample" in row:
            done.add(f"{row['id']}#{row['sample']}")
    return done


class PowerSampler:
    """Samples accelerator power in the background so the run can report energy.

    Track M reports nothing to an unprivileged process, so mean() returns None
    there and the cost block records a null rather than a zero.
    """

    def __init__(self, interval: float = 15.0) -> None:
        self.interval = interval
        self.samples: list[float] = []
        self._stop = threading.Event()
        self._thread: threading.Thread | None = None

    def start(self) -> None:
        if distillog.sample_power() is None:
            return
        self._thread = threading.Thread(target=self._loop, daemon=True)
        self._thread.start()

    def _loop(self) -> None:
        while not self._stop.wait(self.interval):
            value = distillog.sample_power()
            if value is not None:
                self.samples.append(value)

    def stop(self) -> None:
        self._stop.set()
        if self._thread is not None:
            self._thread.join(timeout=2)

    def mean(self) -> float | None:
        return round(statistics.fmean(self.samples), 1) if self.samples else None


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--seeds", required=True, help="JSON Lines file of prompts")
    parser.add_argument("--out", required=True, help="JSON Lines file of teacher answers")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--model", default="local/chat",
                        help="the name the endpoint answers to, e.g. a gateway alias")
    parser.add_argument("--teacher-id", default=None,
                        help="the course model id of the teacher, recorded on every line. "
                             "The API cannot be asked which weights are loaded, so say so here.")
    parser.add_argument("--quant", default="unknown",
                        help="the teacher's quantisation, recorded on every line")
    parser.add_argument("--system", default=None, help="system prompt sent with every request")
    parser.add_argument("--samples", type=int, default=1,
                        help="answers per prompt; more than one feeds rejection sampling")
    parser.add_argument("--thinking", action="store_true",
                        help="use the model card's thinking-mode sampling settings")
    parser.add_argument("--temperature", type=float, default=None)
    parser.add_argument("--top-p", type=float, default=None)
    parser.add_argument("--top-k", type=int, default=20)
    parser.add_argument("--max-tokens", type=int, default=768)
    parser.add_argument("--concurrency", type=int, default=4,
                        help="requests in flight. Match it to the server's slot count: "
                             "llama-server's --parallel, or vLLM's scheduler.")
    parser.add_argument("--limit", type=int, default=None, help="stop after this many prompts")
    parser.add_argument("--timeout", type=int, default=600)
    parser.add_argument("--resume", action="store_true",
                        help="skip prompts already present in --out and append")
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    defaults = THINKING if args.thinking else NON_THINKING
    settings = {
        "temperature": args.temperature if args.temperature is not None else defaults["temperature"],
        "top_p": args.top_p if args.top_p is not None else defaults["top_p"],
        "top_k": args.top_k,
        "max_tokens": args.max_tokens,
        "mode": "thinking" if args.thinking else "non-thinking",
    }
    if args.samples > 1 and settings["temperature"] == 0:
        raise SystemExit("--samples above 1 at temperature 0 produces the same answer every time")

    seeds = read_jsonl(Path(args.seeds))
    if args.limit:
        seeds = seeds[:args.limit]
    out_path = Path(args.out)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    done = already_done(out_path) if args.resume else set()
    if done:
        print(f"resuming: {len(done)} answer(s) already in {out_path}")
    elif out_path.exists() and not args.resume:
        raise SystemExit(f"{out_path} exists; pass --resume to append to it, or choose another name")

    jobs = [(row, s) for row in seeds for s in range(args.samples)
            if f"{row['id']}#{s}" not in done]
    if not jobs:
        print("nothing to do: every prompt already has its answers")
        return

    endpoint = args.base_url.rstrip("/") + "/chat/completions"
    print(f"teacher: {args.model} at {endpoint}")
    print(f"settings: {json.dumps(settings)}")
    print(f"{len(jobs)} request(s), {args.concurrency} in flight")

    write_lock = threading.Lock()
    counters = {"prompt_tokens": 0, "completion_tokens": 0, "ok": 0, "failed": 0}
    power = PowerSampler()
    power.start()
    started = time.time()

    def one(job: tuple[dict, int]) -> None:
        row, sample = job
        messages = []
        if args.system:
            messages.append({"role": "system", "content": args.system})
        messages.append({"role": "user", "content": row["prompt"]})
        payload = {
            "model": args.model,
            "messages": messages,
            "temperature": settings["temperature"],
            "top_p": settings["top_p"],
            "max_tokens": settings["max_tokens"],
        }
        began = time.time()
        try:
            body = post_json(endpoint, payload, args.api_key, args.timeout)
            answer = (body["choices"][0]["message"]["content"] or "").strip()
            usage = body.get("usage", {}) or {}
        except (RuntimeError, KeyError, IndexError) as exc:
            with write_lock:
                counters["failed"] += 1
                print(f"  FAILED {row['id']}#{sample}: {exc}", file=sys.stderr)
            return

        record = {
            "id": row["id"],
            "sample": sample,
            "category": row.get("category"),
            "prompt": row["prompt"],
            "completion": answer,
            "teacher": {"id": args.teacher_id, "served_as": args.model, "quant": args.quant},
            "settings": settings,
            "usage": {
                "prompt_tokens": usage.get("prompt_tokens"),
                "completion_tokens": usage.get("completion_tokens"),
            },
            "seconds": round(time.time() - began, 2),
        }
        if "answer" in row:
            record["answer"] = row["answer"]  # carried through for the verifier

        with write_lock:
            with out_path.open("a", encoding="utf-8") as handle:
                handle.write(json.dumps(record, ensure_ascii=False) + "\n")
                handle.flush()
            counters["ok"] += 1
            counters["prompt_tokens"] += usage.get("prompt_tokens") or 0
            counters["completion_tokens"] += usage.get("completion_tokens") or 0
            if counters["ok"] % 25 == 0:
                rate = counters["ok"] / max(time.time() - started, 1e-6)
                print(f"  {counters['ok']}/{len(jobs)} answers, {rate:.1f} per second")

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        futures = [pool.submit(one, job) for job in jobs]
        for future in as_completed(futures):
            future.result()

    elapsed = time.time() - started
    power.stop()
    mean_watts = power.mean()

    print()
    print(f"{counters['ok']} answered, {counters['failed']} failed, {elapsed / 60:.1f} minutes")
    print(f"prompt tokens {counters['prompt_tokens']}, completion tokens {counters['completion_tokens']}")
    print(f"mean accelerator power: {mean_watts if mean_watts is not None else 'not reported here'}")
    print(f"written to {out_path}")

    if args.labbook:
        cost = distillog.build_cost(
            prompt_tokens=counters["prompt_tokens"],
            completion_tokens=counters["completion_tokens"],
            seconds=elapsed,
            mean_watts=mean_watts,
        )
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/generate-teacher-data",
            stage="generate",
            teacher={"id": args.teacher_id, "served_as": args.model, "quant": args.quant,
                     "base_url": args.base_url},
            student=None,
            dataset={
                "seeds": args.seeds,
                "seeds_sha256": distillog.file_sha256(args.seeds),
                "out": str(out_path),
                "out_sha256": distillog.file_sha256(out_path),
                "prompts": len(seeds),
                "answers": counters["ok"],
                "failed": counters["failed"],
            },
            hyperparameters={**settings, "samples": args.samples, "concurrency": args.concurrency},
            seed=None,
            cost=cost,
            config_path=__file__,
            notes=None,
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download generate-teacher-data.py304 lines

RunnableAll tracks

generate the teacher dataset

python generate-teacher-data.py \
    --seeds seeds/prompts.jsonl \
    --out raw/teacher.jsonl \
    --base-url http://127.0.0.1:8080/v1 \
    --model teacher \
    --teacher-id qwen3-30b-a3b \
    --quant Q4_K_M \
    --temperature 0.7 --top-p 0.8 \
    --max-tokens 768 \
    --concurrency 8 \
    --labbook labbook.md

Set --concurrency to the slot count you gave the server, not higher. Extra concurrency queues rather than parallelises, and the symptom is that throughput stops improving while latency climbs.

Output — what you should see

teacher: teacher at http://127.0.0.1:8080/v1/chat/completions
settings: {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "max_tokens": 768, "mode": "non-thinking"}
600 request(s), 8 in flight
  25/600 answers, ... per second
  ...
600 answered, 0 failed, ... minutes
prompt tokens ...   completion tokens ...

This is the unattended stretch. The script writes each answer as it arrives and --resume skips anything already in the file, so an interruption costs you the requests in flight and nothing else.

=Expected result

Expect every request to succeed and the rate to be roughly steady. A rate that falls steadily usually means the KV cache is filling and slots are being evicted; shorten --max-tokens or lower the concurrency. Failures reported to standard error with an HTTP status are a server problem, not a script problem, and the run log will record how many there were.

6. Filter, decontaminate, and read the report

RunnableAll tracks

filter-and-dedupe.py

"""Turn raw teacher output into a training set, and say what was thrown away and why.

Purpose: the filtering stage, which is the stage that decides whether the student
    inherits the teacher's ability or the teacher's mistakes. Applies length,
    format and refusal checks, drops degenerate repetition, removes exact and
    near-duplicates, decontaminates against the Part 10 evaluation set, splits
    what survives, and writes both training shapes. Every rejected example is
    counted by reason and a sample of each reason is written out, because a
    filter you cannot inspect is a filter you cannot trust.
Platform: all (standard library only; no model, no accelerator, no network)
Minimum memory: 12 GB nominally, and far less in practice: this is text in memory
Assumes: Python 3.10 or newer. The input is the JSON Lines file written by
    generate-teacher-data.py. --tasks, when given, is the Part 10 evaluation set:
    a JSON object with a "tasks" list whose items carry "prompt" and "reference".
    distillog.py sits next to this file.

Usage: python3 filter-and-dedupe.py --raw raw/teacher.jsonl --out-dir . \\
           --tasks my-tasks.json --labbook labbook.md
       python3 filter-and-dedupe.py --raw raw/teacher.jsonl --out-dir . \\
           --min-words 12 --max-words 400 --near-duplicate 0.8 --keep-thinking

Written under --out-dir:
  data/{train,valid}.jsonl       TRL conversational prompt-completion
  data-mlx/{train,valid}.jsonl   mlx-lm completions, two plain strings
  filter-report.json             counts by reason, plus five examples of each
"""

from __future__ import annotations

import argparse
import json
import random
import re
from collections import Counter, defaultdict
from pathlib import Path
from typing import Any

import distillog

WORD_RE = re.compile(r"[a-z0-9]+")
THINK_BLOCK = re.compile(r"<think>.*?</think>\s*", re.DOTALL)
UNCLOSED_THINK = re.compile(r"<think>(?!.*</think>)", re.DOTALL)

# A refusal answer that invents a hostname, a port or a count is the failure the
# refusal category exists to catch. These are deliberately blunt: the point is to
# notice the teacher inventing specifics, and a false positive costs one example.
INVENTED_SPECIFIC = re.compile(
    r"\b(?:\d{1,3}(?:\.\d{1,3}){3}|port\s+\d{2,5}|(?:host(?:name)?|machine)\s+is\s+\S+"
    r"|(?:happened|occurred)\s+\d+\s+times?)\b",
    re.IGNORECASE,
)
REFUSAL_MARKERS = re.compile(
    r"\b(?:cannot|can't|can not|no access|not able|do not have|don't have|unable|no way (?:for me )?to know)\b",
    re.IGNORECASE,
)


def normalise(text: str) -> list[str]:
    """Lowercase, drop punctuation, split on words, so formatting differences do not hide a duplicate."""
    return WORD_RE.findall(text.lower())


def ngrams(words: list[str], n: int) -> set[tuple[str, ...]]:
    if len(words) < n:
        return {tuple(words)} if words else set()
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}


def containment(a: set[tuple[str, ...]], b: set[tuple[str, ...]]) -> float:
    """Fraction of a's n-grams that also appear in b. Asymmetric on purpose."""
    return len(a & b) / len(a) if a else 0.0


def looks_repetitive(words: list[str], n: int = 8) -> bool:
    """True when the same 8-gram appears three or more times: a decoding loop.

    A model that falls into a loop produces long, fluent, useless text, and a
    student trained on it learns to loop. Length filters miss this because the
    output is often within the length budget.
    """
    if len(words) < n * 3:
        return False
    counts = Counter(tuple(words[i:i + n]) for i in range(len(words) - n + 1))
    return counts.most_common(1)[0][1] >= 3


def json_is_valid(text: str) -> bool:
    stripped = text.strip()
    if stripped.startswith("```"):
        return False  # a fenced block is not the object the prompt asked for
    try:
        json.loads(stripped)
    except json.JSONDecodeError:
        return False
    return True


def check(row: dict, args: argparse.Namespace) -> str | None:
    """Return the reason to reject this row, or None to keep it."""
    completion = (row.get("completion") or "").strip()
    if not completion:
        return "empty"

    if UNCLOSED_THINK.search(completion):
        return "unclosed-thinking-block"

    body = completion if args.keep_thinking else THINK_BLOCK.sub("", completion).strip()
    if not body:
        return "thinking-only"

    # The category checks come before the length checks on purpose. A refusal that
    # invents a hostname is usually short, and reporting it as "too short" would
    # hide the fault that matters: the teacher answered a question it could not know.
    category = row.get("category")
    if category == "format":
        wants_json = '"symptom"' in row.get("prompt", "") or "JSON object" in row.get("prompt", "")
        if wants_json and not json_is_valid(body):
            return "format-not-json"
        if not wants_json and "|" not in body:
            return "format-not-a-table"
    if category == "refusal":
        if INVENTED_SPECIFIC.search(body):
            return "refusal-invented-a-specific"
        if not REFUSAL_MARKERS.search(body):
            return "refusal-did-not-refuse"

    words = normalise(body)
    if len(words) < args.min_words:
        return "too-short"
    if len(words) > args.max_words:
        return "too-long"
    if looks_repetitive(words):
        return "repetition-loop"

    return None


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--raw", required=True, help="JSON Lines from generate-teacher-data.py")
    parser.add_argument("--out-dir", default=".")
    parser.add_argument("--tasks", default=None,
                        help="the Part 10 evaluation set, for decontamination")
    parser.add_argument("--min-words", type=int, default=8)
    parser.add_argument("--max-words", type=int, default=400)
    parser.add_argument("--near-duplicate", type=float, default=0.8,
                        help="drop a completion whose 13-gram containment in an earlier one is above this")
    parser.add_argument("--contamination", type=float, default=0.5,
                        help="drop an example whose prompt overlaps an evaluation prompt above this")
    parser.add_argument("--n", type=int, default=13, help="n-gram size for the overlap checks")
    parser.add_argument("--keep-thinking", action="store_true",
                        help="keep <think> blocks in the completion instead of stripping them")
    parser.add_argument("--valid-fraction", type=float, default=0.1)
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    raw_path = Path(args.raw)
    rows = [json.loads(line) for line in raw_path.read_text(encoding="utf-8").splitlines() if line.strip()]
    print(f"read {len(rows)} raw answer(s) from {raw_path}")

    rejected: Counter[str] = Counter()
    samples: dict[str, list[dict]] = defaultdict(list)

    def reject(row: dict, reason: str) -> None:
        rejected[reason] += 1
        if len(samples[reason]) < 5:
            samples[reason].append({
                "id": row.get("id"),
                "category": row.get("category"),
                "prompt": (row.get("prompt") or "")[:200],
                "completion": (row.get("completion") or "")[:300],
            })

    # Stage 1: per-example checks.
    stage1 = []
    for row in rows:
        reason = check(row, args)
        if reason:
            reject(row, reason)
            continue
        completion = (row["completion"] or "").strip()
        if not args.keep_thinking:
            completion = THINK_BLOCK.sub("", completion).strip()
        stage1.append({**row, "completion": completion})

    # Stage 2: exact and near-duplicate completions, and repeated prompts.
    kept: list[dict] = []
    seen_exact: set[str] = set()
    seen_prompt: set[str] = set()
    seen_grams: list[set[tuple[str, ...]]] = []
    for row in stage1:
        key = " ".join(normalise(row["completion"]))
        if key in seen_exact:
            reject(row, "duplicate-completion")
            continue
        if row["prompt"] in seen_prompt:
            # More than one sample per prompt survived the filters; keep the first
            # so a single prompt cannot dominate the training set.
            reject(row, "duplicate-prompt")
            continue
        grams = ngrams(normalise(row["completion"]), args.n)
        if any(containment(grams, earlier) >= args.near_duplicate for earlier in seen_grams):
            reject(row, "near-duplicate")
            continue
        seen_exact.add(key)
        seen_prompt.add(row["prompt"])
        seen_grams.append(grams)
        kept.append(row)

    # Stage 3: decontamination against the evaluation set.
    contaminated = 0
    if args.tasks:
        spec = json.loads(Path(args.tasks).read_text(encoding="utf-8"))
        eval_texts = []
        for task in spec.get("tasks", []):
            eval_texts.append(task.get("prompt", ""))
            if task.get("reference"):
                eval_texts.append(task["reference"])
        eval_grams = [ngrams(normalise(t), args.n) for t in eval_texts if t]
        eval_exact = {" ".join(normalise(t)) for t in eval_texts if t}
        clean = []
        for row in kept:
            text = f"{row['prompt']}\n{row['completion']}"
            if " ".join(normalise(row["prompt"])) in eval_exact:
                reject(row, "contaminated-exact")
                contaminated += 1
                continue
            grams = ngrams(normalise(text), args.n)
            if any(containment(e, grams) >= args.contamination for e in eval_grams if e):
                reject(row, "contaminated-overlap")
                contaminated += 1
                continue
            clean.append(row)
        kept = clean
        print(f"decontamination: checked against {len(eval_texts)} evaluation text(s), "
              f"dropped {contaminated}")
    else:
        print("decontamination: SKIPPED, no --tasks given. A gain measured on an "
              "evaluation set you did not check is not a gain you can report.")

    if not kept:
        raise SystemExit("every example was rejected; look at filter-report.json before changing thresholds")

    rng = random.Random(args.seed)
    rng.shuffle(kept)
    cut = max(1, int(len(kept) * args.valid_fraction))
    valid, train = kept[:cut], kept[cut:]

    out = Path(args.out_dir)
    written = {}
    for name, part in (("train", train), ("valid", valid)):
        trl_path = out / "data" / f"{name}.jsonl"
        trl_path.parent.mkdir(parents=True, exist_ok=True)
        with trl_path.open("w", encoding="utf-8") as handle:
            for row in part:
                handle.write(json.dumps({
                    "prompt": [{"role": "user", "content": row["prompt"]}],
                    "completion": [{"role": "assistant", "content": row["completion"]}],
                }, ensure_ascii=False) + "\n")
        mlx_path = out / "data-mlx" / f"{name}.jsonl"
        mlx_path.parent.mkdir(parents=True, exist_ok=True)
        with mlx_path.open("w", encoding="utf-8") as handle:
            for row in part:
                handle.write(json.dumps({"prompt": row["prompt"], "completion": row["completion"]},
                                        ensure_ascii=False) + "\n")
        written[name] = {"count": len(part), "trl": str(trl_path), "mlx": str(mlx_path),
                         "sha256": distillog.file_sha256(trl_path)}

    report = {
        "raw": str(raw_path),
        "raw_sha256": distillog.file_sha256(raw_path),
        "raw_examples": len(rows),
        "kept": len(kept),
        "rejected_total": sum(rejected.values()),
        "rejected_by_reason": dict(rejected.most_common()),
        "examples_by_reason": {k: v for k, v in samples.items()},
        "thresholds": {
            "min_words": args.min_words, "max_words": args.max_words,
            "near_duplicate": args.near_duplicate, "contamination": args.contamination,
            "n": args.n, "keep_thinking": args.keep_thinking,
        },
        "splits": written,
    }
    report_path = out / "filter-report.json"
    report_path.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")

    print()
    print(f"kept {len(kept)} of {len(rows)} ({100 * len(kept) / len(rows):.1f}%)")
    for reason, count in rejected.most_common():
        print(f"  rejected {count:>5}  {reason}")
    print(f"train {len(train)}  valid {len(valid)}")
    print(f"report written to {report_path}: read the examples before you accept the counts")

    if args.labbook:
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/filter-and-dedupe",
            stage="filter",
            teacher=(rows[0].get("teacher") if rows else None),
            student=None,
            dataset={
                "raw": str(raw_path),
                "raw_sha256": report["raw_sha256"],
                "raw_examples": len(rows),
                "kept": len(kept),
                "train": len(train),
                "valid": len(valid),
                "train_sha256": written["train"]["sha256"],
                "tasks": args.tasks,
                "contaminated_dropped": contaminated,
            },
            hyperparameters=report["thresholds"],
            seed=args.seed,
            scores={"rejected_by_reason": dict(rejected.most_common())},
            config_path=__file__,
            notes=None if args.tasks else "decontamination skipped: no evaluation set supplied",
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download filter-and-dedupe.py323 lines

RunnableAll tracks

filter and decontaminate

python filter-and-dedupe.py \
    --raw raw/teacher.jsonl \
    --out-dir . \
    --tasks my-tasks.json \
    --labbook labbook.md

Output — what you should see

read 600 raw answer(s) from raw/teacher.jsonl
decontamination: checked against ... evaluation text(s), dropped ...

kept ... of 600 (...%)
  rejected   ...  duplicate-completion
  rejected   ...  refusal-invented-a-specific
  rejected   ...  too-short
train ...  valid ...
report written to filter-report.json: read the examples before you accept the counts

Now do what the last line says.

RunnableAll tracks

read five rejected examples of each reason

python -c "
import json
r = json.load(open('filter-report.json'))
for reason, rows in r['examples_by_reason'].items():
    print('==', reason, r['rejected_by_reason'][reason])
    for row in rows[:2]:
        print('  ', row['completion'][:160].replace(chr(10), ' '))
"

This is the step people skip. The rejections tell you things the counts do not: whether the refusal prompts made your teacher invent a hostname, whether a template produced answers that all look the same, whether the length filter is throwing away good short answers. Adjust a threshold and rerun if you need to; the filter is cheap and the generation is already on disk.

!Important

--tasks is not optional in any meaningful sense. Without it the script prints that decontamination was skipped and records that in the run log, and the result you get at the end is not reportable: you will not know whether the student improved or simply memorised the evaluation questions. This is the fault the challenge in this part reproduces deliberately.

7. Train the student

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

train-student.py

"""Train the student on the teacher's filtered output: sequence-level distillation.

Purpose: the training stage of the sequence-level lab. This is Part 11's supervised
    fine-tuning recipe with one thing changed, which is the whole point of the
    lesson: the dataset was written by a model rather than by a person. The
    trainer, the adapter, the loss and the evaluation loop are the same, and the
    run record names the teacher so that the resulting student can never be
    mistaken for a fine-tune on human data.
Platform: spark, strix, nvidia (CUDA or ROCm); it also runs on the CPU, slowly.
    Track M uses mlx_lm.lora on data-mlx/ instead, as in Part 11's lab; PyTorch's
    MPS backend will run this script in float32 if you want the comparison.
Minimum memory: 12 GB
Assumes: torch, transformers, trl, peft and datasets installed in the active
    environment; filter-and-dedupe.py has been run so that data/train.jsonl and
    data/valid.jsonl exist; distillog.py sits next to this file.

Usage: python3 train-student.py --model Qwen/Qwen3-4B --data-dir data \\
           --output-dir runs/seq-qwen3-4b --teacher-id qwen3-30b-a3b --labbook labbook.md
       python3 train-student.py --model Qwen/Qwen3-1.7B --data-dir data \\
           --output-dir runs/seq-qwen3-1.7b --epochs 2 --batch-size 1 --grad-accum 8 \\
           --gradient-checkpointing --teacher-id qwen3-8b
       python3 train-student.py --model Qwen/Qwen3-4B --list-modules
"""

from __future__ import annotations

import argparse
import json
import time
from pathlib import Path

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

import distillog

DEFAULT_TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]


def pick_device() -> str:
    """The same choice Part 11 makes: CUDA (or ROCm, which reports as cuda), then MPS, then CPU."""
    if torch.cuda.is_available():
        return "cuda"
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return "mps"
    return "cpu"


def use_bf16(device: str, requested: str) -> bool:
    if requested == "fp32":
        return False
    if requested == "bf16":
        return True
    return device == "cuda" and torch.cuda.is_bf16_supported()


def list_linear_modules(model_id: str) -> None:
    """Print the names LoRA can target, so target_modules is never guessed."""
    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
    names = sorted({n.split(".")[-1] for n, m in model.named_modules() if isinstance(m, torch.nn.Linear)})
    print(f"linear module names in {model_id}:")
    for name in names:
        print(f"  {name}")


def summarise_history(history: list[dict]) -> dict[str, float | int | None]:
    train_losses = [row["loss"] for row in history if "loss" in row]
    evals = [(row["epoch"], row["eval_loss"]) for row in history if "eval_loss" in row]
    best_epoch, best_eval = min(evals, key=lambda pair: pair[1]) if evals else (None, None)
    return {
        "first_train_loss": round(train_losses[0], 4) if train_losses else None,
        "final_train_loss": round(train_losses[-1], 4) if train_losses else None,
        "final_eval_loss": round(evals[-1][1], 4) if evals else None,
        "best_eval_loss": round(best_eval, 4) if best_eval is not None else None,
        "best_epoch": best_epoch,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-4B", help="the student, as a repository id or path")
    parser.add_argument("--data-dir", default="data")
    parser.add_argument("--output-dir", default="runs/seq-student")
    parser.add_argument("--teacher-id", default=None,
                        help="the course model id of the teacher whose output this is. "
                             "Recorded in the run log; a student without one is untraceable.")
    parser.add_argument("--epochs", type=float, default=2.0)
    parser.add_argument("--batch-size", type=int, default=2)
    parser.add_argument("--grad-accum", type=int, default=8)
    parser.add_argument("--lr", type=float, default=1e-4, help="adapters take a higher rate than a full fine-tune")
    parser.add_argument("--max-length", type=int, default=1024)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=int, default=32)
    parser.add_argument("--dropout", type=float, default=0.05)
    parser.add_argument("--target-modules", nargs="+", default=DEFAULT_TARGETS)
    parser.add_argument("--precision", choices=["auto", "bf16", "fp32"], default="auto")
    parser.add_argument("--gradient-checkpointing", action="store_true",
                        help="recompute activations in the backward pass; saves memory, costs time")
    parser.add_argument("--report-to", default="none", choices=["none", "tensorboard"])
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--list-modules", action="store_true")
    args = parser.parse_args()

    if args.list_modules:
        list_linear_modules(args.model)
        return

    device = pick_device()
    bf16 = use_bf16(device, args.precision)
    dtype = torch.bfloat16 if bf16 else torch.float32
    print(f"device: {device}   precision: {'bfloat16' if bf16 else 'float32'}")
    if args.teacher_id is None:
        print("warning: no --teacher-id given. The run record will not say whose output "
              "this student learned, which is the one field distillation adds.")

    data_dir = Path(args.data_dir)
    files = {"train": str(data_dir / "train.jsonl"), "validation": str(data_dir / "valid.jsonl")}
    for split, path in files.items():
        if not Path(path).is_file():
            raise SystemExit(f"{path} is missing; run filter-and-dedupe.py first ({split} split)")
    dataset = load_dataset("json", data_files=files)
    print(f"train examples: {len(dataset['train'])}   validation examples: {len(dataset['validation'])}")

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    if tokenizer.chat_template is None:
        raise SystemExit(
            f"{args.model} has no chat template; pick an instruction-tuned student or set "
            "chat_template_path in SFTConfig"
        )

    config = SFTConfig(
        output_dir=args.output_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        gradient_accumulation_steps=args.grad_accum,
        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_steps=10,
        max_length=args.max_length,
        completion_only_loss=True,     # loss on the teacher's answer, not on the prompt
        gradient_checkpointing=args.gradient_checkpointing,
        bf16=bf16,
        model_init_kwargs={"dtype": dtype},
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        logging_steps=10,
        report_to=args.report_to,
        seed=args.seed,
        data_seed=args.seed,
    )

    peft_config = LoraConfig(
        r=args.rank,
        lora_alpha=args.alpha,
        lora_dropout=args.dropout,
        target_modules=args.target_modules,
        bias="none",
        task_type="CAUSAL_LM",
    )

    trainer = SFTTrainer(
        model=args.model,
        args=config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        processing_class=tokenizer,
        peft_config=peft_config,
    )
    trainer.model.print_trainable_parameters()

    started = time.time()
    mean_watts_before = distillog.sample_power()
    trainer.train()
    elapsed = time.time() - started
    mean_watts_after = distillog.sample_power()

    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    losses = summarise_history(trainer.state.log_history)
    print(json.dumps(losses, indent=2))
    print(f"adapter saved to {args.output_dir}")

    if args.labbook:
        readings = [w for w in (mean_watts_before, mean_watts_after) if w is not None]
        cost = distillog.build_cost(
            seconds=elapsed,
            mean_watts=(sum(readings) / len(readings)) if readings else None,
        )
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/train-student",
            stage="train",
            teacher={"id": args.teacher_id},
            student={"id": args.model, "adapter": args.output_dir, "method": "lora-sft"},
            dataset={
                "path": files["train"],
                "sha256": distillog.file_sha256(files["train"]),
                "train_examples": len(dataset["train"]),
                "validation_examples": len(dataset["validation"]),
            },
            hyperparameters={
                "method": "sequence-level distillation (SFT on teacher output)",
                "rank": args.rank, "alpha": args.alpha, "dropout": args.dropout,
                "target_modules": args.target_modules,
                "epochs": args.epochs, "batch_size": args.batch_size,
                "grad_accum": args.grad_accum,
                "effective_batch": args.batch_size * args.grad_accum,
                "learning_rate": args.lr, "max_length": args.max_length,
                "gradient_checkpointing": args.gradient_checkpointing,
                "precision": "bfloat16" if bf16 else "float32",
                "completion_only_loss": True,
            },
            seed=args.seed,
            cost=cost,
            losses=losses,
            config_path=__file__,
            notes=None,
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download train-student.py233 lines

RunnableTrack S · DGX Spark

sequence-level distillation: SFT on teacher output

python train-student.py \
    --model Qwen/Qwen3-4B \
    --teacher-id qwen3-30b-a3b \
    --data-dir data \
    --output-dir runs/seq-qwen3-4b \
    --epochs 2 \
    --batch-size 2 \
    --grad-accum 8 \
    --labbook labbook.md

Track X — AMD Ryzen AI Max+ 395Partial

ROCm build required for the GPU path; --precision fp32 finishes the lab on the CPU.

RunnableTrack X · Ryzen AI Max+

train-student.py

"""Train the student on the teacher's filtered output: sequence-level distillation.

Purpose: the training stage of the sequence-level lab. This is Part 11's supervised
    fine-tuning recipe with one thing changed, which is the whole point of the
    lesson: the dataset was written by a model rather than by a person. The
    trainer, the adapter, the loss and the evaluation loop are the same, and the
    run record names the teacher so that the resulting student can never be
    mistaken for a fine-tune on human data.
Platform: spark, strix, nvidia (CUDA or ROCm); it also runs on the CPU, slowly.
    Track M uses mlx_lm.lora on data-mlx/ instead, as in Part 11's lab; PyTorch's
    MPS backend will run this script in float32 if you want the comparison.
Minimum memory: 12 GB
Assumes: torch, transformers, trl, peft and datasets installed in the active
    environment; filter-and-dedupe.py has been run so that data/train.jsonl and
    data/valid.jsonl exist; distillog.py sits next to this file.

Usage: python3 train-student.py --model Qwen/Qwen3-4B --data-dir data \\
           --output-dir runs/seq-qwen3-4b --teacher-id qwen3-30b-a3b --labbook labbook.md
       python3 train-student.py --model Qwen/Qwen3-1.7B --data-dir data \\
           --output-dir runs/seq-qwen3-1.7b --epochs 2 --batch-size 1 --grad-accum 8 \\
           --gradient-checkpointing --teacher-id qwen3-8b
       python3 train-student.py --model Qwen/Qwen3-4B --list-modules
"""

from __future__ import annotations

import argparse
import json
import time
from pathlib import Path

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

import distillog

DEFAULT_TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]


def pick_device() -> str:
    """The same choice Part 11 makes: CUDA (or ROCm, which reports as cuda), then MPS, then CPU."""
    if torch.cuda.is_available():
        return "cuda"
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return "mps"
    return "cpu"


def use_bf16(device: str, requested: str) -> bool:
    if requested == "fp32":
        return False
    if requested == "bf16":
        return True
    return device == "cuda" and torch.cuda.is_bf16_supported()


def list_linear_modules(model_id: str) -> None:
    """Print the names LoRA can target, so target_modules is never guessed."""
    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
    names = sorted({n.split(".")[-1] for n, m in model.named_modules() if isinstance(m, torch.nn.Linear)})
    print(f"linear module names in {model_id}:")
    for name in names:
        print(f"  {name}")


def summarise_history(history: list[dict]) -> dict[str, float | int | None]:
    train_losses = [row["loss"] for row in history if "loss" in row]
    evals = [(row["epoch"], row["eval_loss"]) for row in history if "eval_loss" in row]
    best_epoch, best_eval = min(evals, key=lambda pair: pair[1]) if evals else (None, None)
    return {
        "first_train_loss": round(train_losses[0], 4) if train_losses else None,
        "final_train_loss": round(train_losses[-1], 4) if train_losses else None,
        "final_eval_loss": round(evals[-1][1], 4) if evals else None,
        "best_eval_loss": round(best_eval, 4) if best_eval is not None else None,
        "best_epoch": best_epoch,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-4B", help="the student, as a repository id or path")
    parser.add_argument("--data-dir", default="data")
    parser.add_argument("--output-dir", default="runs/seq-student")
    parser.add_argument("--teacher-id", default=None,
                        help="the course model id of the teacher whose output this is. "
                             "Recorded in the run log; a student without one is untraceable.")
    parser.add_argument("--epochs", type=float, default=2.0)
    parser.add_argument("--batch-size", type=int, default=2)
    parser.add_argument("--grad-accum", type=int, default=8)
    parser.add_argument("--lr", type=float, default=1e-4, help="adapters take a higher rate than a full fine-tune")
    parser.add_argument("--max-length", type=int, default=1024)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=int, default=32)
    parser.add_argument("--dropout", type=float, default=0.05)
    parser.add_argument("--target-modules", nargs="+", default=DEFAULT_TARGETS)
    parser.add_argument("--precision", choices=["auto", "bf16", "fp32"], default="auto")
    parser.add_argument("--gradient-checkpointing", action="store_true",
                        help="recompute activations in the backward pass; saves memory, costs time")
    parser.add_argument("--report-to", default="none", choices=["none", "tensorboard"])
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--list-modules", action="store_true")
    args = parser.parse_args()

    if args.list_modules:
        list_linear_modules(args.model)
        return

    device = pick_device()
    bf16 = use_bf16(device, args.precision)
    dtype = torch.bfloat16 if bf16 else torch.float32
    print(f"device: {device}   precision: {'bfloat16' if bf16 else 'float32'}")
    if args.teacher_id is None:
        print("warning: no --teacher-id given. The run record will not say whose output "
              "this student learned, which is the one field distillation adds.")

    data_dir = Path(args.data_dir)
    files = {"train": str(data_dir / "train.jsonl"), "validation": str(data_dir / "valid.jsonl")}
    for split, path in files.items():
        if not Path(path).is_file():
            raise SystemExit(f"{path} is missing; run filter-and-dedupe.py first ({split} split)")
    dataset = load_dataset("json", data_files=files)
    print(f"train examples: {len(dataset['train'])}   validation examples: {len(dataset['validation'])}")

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    if tokenizer.chat_template is None:
        raise SystemExit(
            f"{args.model} has no chat template; pick an instruction-tuned student or set "
            "chat_template_path in SFTConfig"
        )

    config = SFTConfig(
        output_dir=args.output_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        gradient_accumulation_steps=args.grad_accum,
        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_steps=10,
        max_length=args.max_length,
        completion_only_loss=True,     # loss on the teacher's answer, not on the prompt
        gradient_checkpointing=args.gradient_checkpointing,
        bf16=bf16,
        model_init_kwargs={"dtype": dtype},
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        logging_steps=10,
        report_to=args.report_to,
        seed=args.seed,
        data_seed=args.seed,
    )

    peft_config = LoraConfig(
        r=args.rank,
        lora_alpha=args.alpha,
        lora_dropout=args.dropout,
        target_modules=args.target_modules,
        bias="none",
        task_type="CAUSAL_LM",
    )

    trainer = SFTTrainer(
        model=args.model,
        args=config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        processing_class=tokenizer,
        peft_config=peft_config,
    )
    trainer.model.print_trainable_parameters()

    started = time.time()
    mean_watts_before = distillog.sample_power()
    trainer.train()
    elapsed = time.time() - started
    mean_watts_after = distillog.sample_power()

    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    losses = summarise_history(trainer.state.log_history)
    print(json.dumps(losses, indent=2))
    print(f"adapter saved to {args.output_dir}")

    if args.labbook:
        readings = [w for w in (mean_watts_before, mean_watts_after) if w is not None]
        cost = distillog.build_cost(
            seconds=elapsed,
            mean_watts=(sum(readings) / len(readings)) if readings else None,
        )
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/train-student",
            stage="train",
            teacher={"id": args.teacher_id},
            student={"id": args.model, "adapter": args.output_dir, "method": "lora-sft"},
            dataset={
                "path": files["train"],
                "sha256": distillog.file_sha256(files["train"]),
                "train_examples": len(dataset["train"]),
                "validation_examples": len(dataset["validation"]),
            },
            hyperparameters={
                "method": "sequence-level distillation (SFT on teacher output)",
                "rank": args.rank, "alpha": args.alpha, "dropout": args.dropout,
                "target_modules": args.target_modules,
                "epochs": args.epochs, "batch_size": args.batch_size,
                "grad_accum": args.grad_accum,
                "effective_batch": args.batch_size * args.grad_accum,
                "learning_rate": args.lr, "max_length": args.max_length,
                "gradient_checkpointing": args.gradient_checkpointing,
                "precision": "bfloat16" if bf16 else "float32",
                "completion_only_loss": True,
            },
            seed=args.seed,
            cost=cost,
            losses=losses,
            config_path=__file__,
            notes=None,
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

RunnableTrack X · Ryzen AI Max+

sequence-level distillation on ROCm, or on the CPU

python train-student.py \
    --model Qwen/Qwen3-4B \
    --teacher-id qwen3-30b-a3b \
    --data-dir data \
    --output-dir runs/seq-qwen3-4b \
    --epochs 2 \
    --batch-size 1 \
    --grad-accum 16 \
    --gradient-checkpointing \
    --labbook labbook.md

Track M — Apple silicon

RunnableTrack M · Apple silicon

sequence-level distillation with mlx-lm

mlx_lm.lora \
    --model mlx-community/Qwen3-4B-bf16 \
    --train \
    --data data-mlx \
    --batch-size 2 \
    --iters 600 \
    --adapter-path runs/seq-qwen3-4b-mlx

filter-and-dedupe.py wrote data-mlx/ alongside data/ for exactly this: the same examples in the completions shape mlx-lm documents. Copy the final training and validation losses out of the mlx-lm output into a run record by hand, as Part 11’s lab does, since this path does not write one itself.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

train-student.py

"""Train the student on the teacher's filtered output: sequence-level distillation.

Purpose: the training stage of the sequence-level lab. This is Part 11's supervised
    fine-tuning recipe with one thing changed, which is the whole point of the
    lesson: the dataset was written by a model rather than by a person. The
    trainer, the adapter, the loss and the evaluation loop are the same, and the
    run record names the teacher so that the resulting student can never be
    mistaken for a fine-tune on human data.
Platform: spark, strix, nvidia (CUDA or ROCm); it also runs on the CPU, slowly.
    Track M uses mlx_lm.lora on data-mlx/ instead, as in Part 11's lab; PyTorch's
    MPS backend will run this script in float32 if you want the comparison.
Minimum memory: 12 GB
Assumes: torch, transformers, trl, peft and datasets installed in the active
    environment; filter-and-dedupe.py has been run so that data/train.jsonl and
    data/valid.jsonl exist; distillog.py sits next to this file.

Usage: python3 train-student.py --model Qwen/Qwen3-4B --data-dir data \\
           --output-dir runs/seq-qwen3-4b --teacher-id qwen3-30b-a3b --labbook labbook.md
       python3 train-student.py --model Qwen/Qwen3-1.7B --data-dir data \\
           --output-dir runs/seq-qwen3-1.7b --epochs 2 --batch-size 1 --grad-accum 8 \\
           --gradient-checkpointing --teacher-id qwen3-8b
       python3 train-student.py --model Qwen/Qwen3-4B --list-modules
"""

from __future__ import annotations

import argparse
import json
import time
from pathlib import Path

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

import distillog

DEFAULT_TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]


def pick_device() -> str:
    """The same choice Part 11 makes: CUDA (or ROCm, which reports as cuda), then MPS, then CPU."""
    if torch.cuda.is_available():
        return "cuda"
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return "mps"
    return "cpu"


def use_bf16(device: str, requested: str) -> bool:
    if requested == "fp32":
        return False
    if requested == "bf16":
        return True
    return device == "cuda" and torch.cuda.is_bf16_supported()


def list_linear_modules(model_id: str) -> None:
    """Print the names LoRA can target, so target_modules is never guessed."""
    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
    names = sorted({n.split(".")[-1] for n, m in model.named_modules() if isinstance(m, torch.nn.Linear)})
    print(f"linear module names in {model_id}:")
    for name in names:
        print(f"  {name}")


def summarise_history(history: list[dict]) -> dict[str, float | int | None]:
    train_losses = [row["loss"] for row in history if "loss" in row]
    evals = [(row["epoch"], row["eval_loss"]) for row in history if "eval_loss" in row]
    best_epoch, best_eval = min(evals, key=lambda pair: pair[1]) if evals else (None, None)
    return {
        "first_train_loss": round(train_losses[0], 4) if train_losses else None,
        "final_train_loss": round(train_losses[-1], 4) if train_losses else None,
        "final_eval_loss": round(evals[-1][1], 4) if evals else None,
        "best_eval_loss": round(best_eval, 4) if best_eval is not None else None,
        "best_epoch": best_epoch,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-4B", help="the student, as a repository id or path")
    parser.add_argument("--data-dir", default="data")
    parser.add_argument("--output-dir", default="runs/seq-student")
    parser.add_argument("--teacher-id", default=None,
                        help="the course model id of the teacher whose output this is. "
                             "Recorded in the run log; a student without one is untraceable.")
    parser.add_argument("--epochs", type=float, default=2.0)
    parser.add_argument("--batch-size", type=int, default=2)
    parser.add_argument("--grad-accum", type=int, default=8)
    parser.add_argument("--lr", type=float, default=1e-4, help="adapters take a higher rate than a full fine-tune")
    parser.add_argument("--max-length", type=int, default=1024)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=int, default=32)
    parser.add_argument("--dropout", type=float, default=0.05)
    parser.add_argument("--target-modules", nargs="+", default=DEFAULT_TARGETS)
    parser.add_argument("--precision", choices=["auto", "bf16", "fp32"], default="auto")
    parser.add_argument("--gradient-checkpointing", action="store_true",
                        help="recompute activations in the backward pass; saves memory, costs time")
    parser.add_argument("--report-to", default="none", choices=["none", "tensorboard"])
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--list-modules", action="store_true")
    args = parser.parse_args()

    if args.list_modules:
        list_linear_modules(args.model)
        return

    device = pick_device()
    bf16 = use_bf16(device, args.precision)
    dtype = torch.bfloat16 if bf16 else torch.float32
    print(f"device: {device}   precision: {'bfloat16' if bf16 else 'float32'}")
    if args.teacher_id is None:
        print("warning: no --teacher-id given. The run record will not say whose output "
              "this student learned, which is the one field distillation adds.")

    data_dir = Path(args.data_dir)
    files = {"train": str(data_dir / "train.jsonl"), "validation": str(data_dir / "valid.jsonl")}
    for split, path in files.items():
        if not Path(path).is_file():
            raise SystemExit(f"{path} is missing; run filter-and-dedupe.py first ({split} split)")
    dataset = load_dataset("json", data_files=files)
    print(f"train examples: {len(dataset['train'])}   validation examples: {len(dataset['validation'])}")

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    if tokenizer.chat_template is None:
        raise SystemExit(
            f"{args.model} has no chat template; pick an instruction-tuned student or set "
            "chat_template_path in SFTConfig"
        )

    config = SFTConfig(
        output_dir=args.output_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        gradient_accumulation_steps=args.grad_accum,
        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_steps=10,
        max_length=args.max_length,
        completion_only_loss=True,     # loss on the teacher's answer, not on the prompt
        gradient_checkpointing=args.gradient_checkpointing,
        bf16=bf16,
        model_init_kwargs={"dtype": dtype},
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        logging_steps=10,
        report_to=args.report_to,
        seed=args.seed,
        data_seed=args.seed,
    )

    peft_config = LoraConfig(
        r=args.rank,
        lora_alpha=args.alpha,
        lora_dropout=args.dropout,
        target_modules=args.target_modules,
        bias="none",
        task_type="CAUSAL_LM",
    )

    trainer = SFTTrainer(
        model=args.model,
        args=config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        processing_class=tokenizer,
        peft_config=peft_config,
    )
    trainer.model.print_trainable_parameters()

    started = time.time()
    mean_watts_before = distillog.sample_power()
    trainer.train()
    elapsed = time.time() - started
    mean_watts_after = distillog.sample_power()

    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    losses = summarise_history(trainer.state.log_history)
    print(json.dumps(losses, indent=2))
    print(f"adapter saved to {args.output_dir}")

    if args.labbook:
        readings = [w for w in (mean_watts_before, mean_watts_after) if w is not None]
        cost = distillog.build_cost(
            seconds=elapsed,
            mean_watts=(sum(readings) / len(readings)) if readings else None,
        )
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/train-student",
            stage="train",
            teacher={"id": args.teacher_id},
            student={"id": args.model, "adapter": args.output_dir, "method": "lora-sft"},
            dataset={
                "path": files["train"],
                "sha256": distillog.file_sha256(files["train"]),
                "train_examples": len(dataset["train"]),
                "validation_examples": len(dataset["validation"]),
            },
            hyperparameters={
                "method": "sequence-level distillation (SFT on teacher output)",
                "rank": args.rank, "alpha": args.alpha, "dropout": args.dropout,
                "target_modules": args.target_modules,
                "epochs": args.epochs, "batch_size": args.batch_size,
                "grad_accum": args.grad_accum,
                "effective_batch": args.batch_size * args.grad_accum,
                "learning_rate": args.lr, "max_length": args.max_length,
                "gradient_checkpointing": args.gradient_checkpointing,
                "precision": "bfloat16" if bf16 else "float32",
                "completion_only_loss": True,
            },
            seed=args.seed,
            cost=cost,
            losses=losses,
            config_path=__file__,
            notes=None,
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Stop the teacher first at 24 GB and below. It has done its job and its weights are the largest single thing on the machine.

RunnableTrack N · NVIDIA GPU

sequence-level distillation, 24 GB tier

python train-student.py \
    --model Qwen/Qwen3-4B \
    --teacher-id qwen3-14b \
    --data-dir data \
    --output-dir runs/seq-qwen3-4b \
    --epochs 2 \
    --batch-size 1 \
    --grad-accum 16 \
    --labbook labbook.md

At 12 GB substitute --model Qwen/Qwen3-1.7B --teacher-id qwen3-8b and --output-dir runs/seq-qwen3-1.7b, which is the pair the second memory diagram says fits.

Output — what you should see

device: cuda   precision: bfloat16
train examples: ...   validation examples: ...
trainable params: 33,030,144 || all params: 4,0xx,xxx,xxx || trainable%: 0.8xxx

That trainable-parameter count is the arithmetic from task 1 confirmed by the library. If it is not about 33 million for a 4B student at rank 16, either the rank or the target module list is not what you thought it was.

=Expected result

Training loss should fall from the first logged step and evaluation loss should fall with it over two epochs. A dataset of a few hundred machine-written examples on a narrow taxonomy overfits quickly, so an evaluation loss that turns upward in the second epoch is a result rather than a failure, and the script keeps the checkpoint from before the turn.

8. Export the student and serve all three

The evaluation harness talks to an OpenAI-compatible endpoint, so the adapter has to become something an engine can serve. This is Part 13’s export path, unchanged, and Part 11’s lab walked through each command.

RunnableAll tracks

merge, convert and quantise the distilled student

python ~/llm-course/merge-adapter.py \
    --adapter runs/seq-qwen3-4b \
    --merged-dir models/seq-qwen3-4b-merged
python ~/llama.cpp/convert_hf_to_gguf.py models/seq-qwen3-4b-merged \
    --outfile models/seq-qwen3-4b.gguf \
    --outtype bf16
~/llama.cpp/build/bin/llama-quantize \
    models/seq-qwen3-4b.gguf \
    models/seq-qwen3-4b-Q4_K_M.gguf \
    Q4_K_M

You now need three models reachable by name. The tidy route is the Part 9 gateway with three llama-swap entries, local/teacher, local/student-base and local/student-distilled, which loads each on demand and lets the evaluation run unattended. The route with no gateway is one llama-server at a time, and the evaluation script waits for you between models.

9. Evaluate the triplet

RunnableAll tracks

evaluate-triplet.py

"""Score the teacher, the base student and the distilled student on the same tasks.

Purpose: the measurement that decides whether the distillation was worth the hours.
    Three models, one task file, one set of sampling settings, run through Part 10's
    harness so the numbers are comparable with everything else in the course. Prints
    the gap that closed as a fraction of the gap that existed, and emits the rows in
    the shape the course's benchmark tables take, so a result can be pasted into a
    report without being retyped.
Platform: all (pure Python over HTTP; the three models are reached through an
    OpenAI-compatible API, so they may be served by any engine on any track, or by
    the Part 9 gateway one at a time under three aliases)
Minimum memory: 12 GB on the machine serving the largest of the three; this script
    needs very little
Assumes: Python 3.10 or newer; Part 10's run-eval.py and judge.py in --harness-dir;
    the three models reachable at --base-url under the names given, together or one
    at a time; distillog.py next to this file.

Usage: python3 evaluate-triplet.py --harness-dir ~/eval --tasks my-tasks.json \\
           --base-url http://127.0.0.1:4000/v1 \\
           --teacher local/teacher --base-student local/student-base \\
           --distilled local/student-distilled \\
           --quant Q4_K_M --engine llama.cpp --engine-version v0.4.0 \\
           --out-dir eval-out --labbook labbook.md

       Add --judge-model local/chat to grade all three runs with the same judge.
       Leave it out and only the deterministic checks are compared, which is a floor
       rather than a score but is exactly repeatable and needs no fourth model.

The three models must be served one at a time on a machine that cannot hold them
together. The script exits between models with a prompt so you can swap them, unless
--no-pause is given, which is what you want when a gateway loads them on demand.
"""

from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path
from typing import Any

import distillog

ROLES = ("teacher", "base-student", "distilled-student")


def run(script: Path, arguments: list[str]) -> None:
    """Run one harness script, printing the command first so the log says what happened."""
    command = [sys.executable, str(script), *arguments]
    print("+ " + " ".join(command))
    subprocess.run(command, check=True)


def deterministic_rate(results_path: Path) -> tuple[int, int]:
    data = json.loads(results_path.read_text(encoding="utf-8"))
    passed = sum(1 for r in data["results"] if r["checks"]["passed"])
    return passed, len(data["results"])


def judged_mean(judged_path: Path) -> float | None:
    """Mean judge score, or None when a judge was not run or returned nothing usable.

    judge.py already computes this in its summary block; the per-result fallback
    exists so that a partially graded file still yields a number rather than an
    exception.
    """
    if not judged_path.is_file():
        return None
    data = json.loads(judged_path.read_text(encoding="utf-8"))
    summary = data.get("judge") or {}
    if isinstance(summary.get("judge_mean"), (int, float)):
        return summary["judge_mean"]
    scores = [r["judge"]["score"] for r in data.get("results", [])
              if isinstance(r.get("judge"), dict) and isinstance(r["judge"].get("score"), (int, float))]
    return round(sum(scores) / len(scores), 2) if scores else None


def per_category(results_path: Path) -> dict[str, tuple[int, int]]:
    data = json.loads(results_path.read_text(encoding="utf-8"))
    out: dict[str, list[int]] = {}
    for r in data["results"]:
        bucket = out.setdefault(r["category"], [0, 0])
        bucket[0] += int(r["checks"]["passed"])
        bucket[1] += 1
    return {k: (v[0], v[1]) for k, v in out.items()}


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--harness-dir", required=True, help="directory holding Part 10's run-eval.py and judge.py")
    parser.add_argument("--tasks", required=True, help="the Part 10 task file")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--teacher", required=True, help="model name the endpoint serves the teacher under")
    parser.add_argument("--base-student", required=True, help="the student before distillation")
    parser.add_argument("--distilled", required=True, help="the student after distillation")
    parser.add_argument("--teacher-id", default=None, help="course model id of the teacher, for the record")
    parser.add_argument("--student-id", default=None, help="course model id of the student, for the record")
    parser.add_argument("--quant", default="unknown", help="quantisation actually loaded, for the record")
    parser.add_argument("--engine", default="llama.cpp")
    parser.add_argument("--engine-version", default="unknown")
    parser.add_argument("--judge-model", default=None, help="a fourth model that grades all three runs")
    parser.add_argument("--out-dir", default="eval-out")
    parser.add_argument("--no-pause", action="store_true",
                        help="do not stop between models; use when a gateway loads them on demand")
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    harness = Path(args.harness_dir)
    run_eval = harness / "run-eval.py"
    judge = harness / "judge.py"
    if not run_eval.is_file():
        raise SystemExit(f"{run_eval} not found; point --harness-dir at your Part 10 evaluation directory")
    if args.judge_model and not judge.is_file():
        raise SystemExit(f"{judge} not found, but --judge-model was given")

    out = Path(args.out_dir)
    out.mkdir(parents=True, exist_ok=True)
    models = {"teacher": args.teacher, "base-student": args.base_student, "distilled-student": args.distilled}

    summary: dict[str, dict[str, Any]] = {}
    for role in ROLES:
        name = models[role]
        if not args.no_pause and role != ROLES[0]:
            input(f"\nServe {name} now, then press Enter to evaluate it as the {role}. ")
        results = out / f"results-{role}.json"
        run(run_eval, [
            "--base-url", args.base_url,
            *(["--api-key", args.api_key] if args.api_key else []),
            "--model", name,
            "--quant", args.quant,
            "--engine", args.engine,
            "--engine-version", args.engine_version,
            "--tasks", args.tasks,
            "--out", str(results),
            "--notes", f"Part 15 triplet evaluation, {role}",
        ])
        judged = out / f"judged-{role}.json"
        if args.judge_model:
            run(judge, [
                "grade",
                "--base-url", args.base_url,
                *(["--api-key", args.api_key] if args.api_key else []),
                "--judge-model", args.judge_model,
                "--results", str(results),
                "--out", str(judged),
            ])
        passed, total = deterministic_rate(results)
        summary[role] = {
            "model": name,
            "checks_passed": passed,
            "checks_total": total,
            "judge_mean": judged_mean(judged),
            "by_category": per_category(results),
        }

    print("\n" + "=" * 72)
    print(f"{'role':<18}{'model':<26}{'checks':>10}{'judge mean':>14}")
    for role in ROLES:
        s = summary[role]
        judge_cell = "-" if s["judge_mean"] is None else f"{s['judge_mean']:.2f}"
        print(f"{role:<18}{s['model']:<26}{s['checks_passed']:>4}/{s['checks_total']:<5}{judge_cell:>14}")

    # The number the lab exists to produce: how much of the teacher-to-student gap
    # the distillation closed. Negative means the distilled student is worse than
    # the base it started from, which is a result and must be reported as one.
    base = summary["base-student"]["checks_passed"]
    tuned = summary["distilled-student"]["checks_passed"]
    teacher = summary["teacher"]["checks_passed"]
    gap = teacher - base
    closed = None if gap == 0 else round(100 * (tuned - base) / gap, 1)
    print()
    if gap == 0:
        print("The teacher and the base student passed the same number of checks: there was "
              "no gap on this task set, so there is nothing for distillation to close. "
              "Write a harder task set before drawing any conclusion.")
    else:
        print(f"gap teacher minus base student: {gap} check(s)")
        print(f"distilled student moved: {tuned - base} check(s)  ->  {closed}% of the gap closed")

    print("\nPer category, deterministic checks passed:")
    categories = sorted({c for role in ROLES for c in summary[role]["by_category"]})
    print(f"{'category':<14}{'teacher':>10}{'base':>10}{'distilled':>12}")
    for category in categories:
        cells = []
        for role in ROLES:
            got, of = summary[role]["by_category"].get(category, (0, 0))
            cells.append(f"{got}/{of}")
        print(f"{category:<14}{cells[0]:>10}{cells[1]:>10}{cells[2]:>12}")
    print("\nA category where the distilled student fell below the base student is the "
          "finding, whatever the total says. Part 15's challenge is about exactly that.")

    rows = [[role, summary[role]["model"],
             f"{summary[role]['checks_passed']}/{summary[role]['checks_total']}",
             summary[role]["judge_mean"] if summary[role]["judge_mean"] is not None else "not judged"]
            for role in ROLES]
    table_path = out / "benchmark-rows.json"
    table_path.write_text(json.dumps({
        "columns": ["Model", "Served as", "Deterministic checks", "Judge mean, 1-5"],
        "rows": rows,
        "context": {
            "engine": args.engine, "version": args.engine_version,
            "model": f"teacher {args.teacher_id or args.teacher}, student {args.student_id or args.base_student}",
            "quant": args.quant, "task_set": args.tasks,
        },
    }, indent=2), encoding="utf-8")
    print(f"\nbenchmark rows written to {table_path}")

    if args.labbook:
        rec = distillog.record(
            labbook=args.labbook,
            lab="part-15/evaluate-triplet",
            stage="evaluate",
            teacher={"id": args.teacher_id, "served_as": args.teacher},
            student={"id": args.student_id, "base_served_as": args.base_student,
                     "distilled_served_as": args.distilled},
            dataset={"tasks": args.tasks, "tasks_sha256": distillog.file_sha256(args.tasks)},
            hyperparameters={"engine": args.engine, "engine_version": args.engine_version,
                             "quant": args.quant, "judge_model": args.judge_model},
            seed=None,
            scores={
                "teacher": summary["teacher"],
                "base_student": summary["base-student"],
                "distilled_student": summary["distilled-student"],
                "gap": gap,
                "percent_of_gap_closed": closed,
            },
            config_path=__file__,
            notes=None,
        )
        print(f"recorded {rec['stage']} as {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download evaluate-triplet.py236 lines

RunnableAll tracks

teacher, base student, distilled student, same tasks

python evaluate-triplet.py \
    --harness-dir ~/eval \
    --tasks my-tasks.json \
    --base-url http://127.0.0.1:4000/v1 \
    --teacher local/teacher \
    --base-student local/student-base \
    --distilled local/student-distilled \
    --teacher-id qwen3-30b-a3b \
    --student-id qwen3-4b \
    --quant Q4_K_M \
    --engine llama.cpp --engine-version v0.4.0 \
    --out-dir eval-out \
    --no-pause \
    --labbook labbook.md

Drop --no-pause if you are swapping servers by hand; the script then waits for you before each model. Add --judge-model local/chat to grade all three runs with the same judge, remembering Part 10’s rule that the judge must not be a model under test.

Output — what you should see

role              model                        checks    judge mean
teacher           local/teacher                 ../..              -
base-student      local/student-base            ../..              -
distilled-student local/student-distilled       ../..              -

gap teacher minus base student: ... check(s)
distilled student moved: ... check(s)  ->  ...% of the gap closed

Per category, deterministic checks passed:
category         teacher      base   distilled
...

The per-category block is the part to read twice. A total that improved while one category fell is the normal outcome of a first distillation run, and it is what the challenge in this part is built around.

10. Record what it cost

RunnableAll tracks

every stage of this lab, from the notebook

python distillog.py --stages --labbook labbook.md

Output — what you should see

2026-...-...T...Z  generate    ...  ... completion tokens  ... s
2026-...-...T...Z  filter      ...  - completion tokens  ... s
2026-...-...T...Z  train       ...  - completion tokens  ... s
2026-...-...T...Z  evaluate    ...  ... completion tokens  ... s

Add to the notebook, by hand, the two things the scripts cannot see: the peak memory the training run reached, against the four figures you predicted in task 1, and whether power was reported on this machine at all.

Verify each data handoff before moving to the next phase

Use separate files for seed prompts, raw teacher responses, accepted examples and evaluation tasks. Before bulk generation, call the teacher once and inspect both the final content and termination status. Confirm the model alias and settings in the response record.

After generation, reconcile attempted, returned, parsed, verified and deduplicated counts. Read the filter report and a sample of rejected outputs. A high rejection rate can indicate a faulty prompt or verifier; generating more data without diagnosis may repeat the same defect. Confirm held-out source tasks were excluded before training the student.

Stop the teacher service when the student’s memory budget requires it. Train into a new output directory, then export with the exact student base. For the triplet evaluation, identify three distinct artefacts: teacher, untouched student and distilled student. Use sequential serving if they cannot coexist and preserve each model’s results before switching. Report quality, output length, latency and preparation cost. Keep the raw generation provenance, filter policy, accepted dataset, adapter and three result sets. The handoff is complete when another reader can identify which teacher examples produced the student and which unseen tasks support the deployment decision.

Validation

You are done when all of the following are true:

• seeds/prompts.jsonl exists with the number of prompts you asked for, and you have read some;
• raw/teacher.jsonl has one line per prompt, each carrying the teacher id, the sampling settings and a token count;
• filter-report.json exists, you have read at least two examples from each rejection reason, and the report shows a non-zero number of evaluation texts checked for contamination;
• data/train.jsonl and data/valid.jsonl exist, and their SHA-256 appears in the filter run record;
• the trainer reported a trainable-parameter count close to your arithmetic, and training loss fell from the first logged step to the last;
• a quantised GGUF file of the distilled student exists and an engine has generated from it;
• eval-out/ holds three results files and benchmark-rows.json;
• the run log has one line for each of generate, filter, train and evaluate, and the evaluate line carries the per-category scores;
• you can state, in one sentence with a number in it, how much of the teacher-to-student gap closed and in which category the distilled student did worst.

Expected outcome

A student that answers more like the teacher on the categories the seed set covered, a report of what that cost, and a category or two where nothing improved. The table below is what to measure per track; the validation pass will fill it with figures from the course’s own machines.

Pending validationWhat to record from this lab, per track

Track	Teacher and student	Generation wall clock	Completion tokens generated	Training wall clock	Gap closed, per cent
S: DGX Spark, 128 GB	Qwen3-30B-A3B to Qwen3-4B	to be measured	to be measured	to be measured	to be measured
X: Ryzen AI Max+ 395	Qwen3-30B-A3B to Qwen3-4B	to be measured	to be measured	to be measured	to be measured
M: Apple silicon, mlx-lm	Qwen3-30B-A3B to Qwen3-4B	to be measured	to be measured	to be measured	to be measured
N: NVIDIA, 24 GB	Qwen3-14B to Qwen3-4B	to be measured	to be measured	to be measured	to be measured
N: NVIDIA, 12 to 16 GB	Qwen3-8B to Qwen3-1.7B	to be measured	to be measured	to be measured	to be measured

the four platform tracks, one machine each · llama.cpp for serving; TRL SFTTrainer with PEFT LoRA on S, X and N; mlx_lm.lora on M llama.cpp v0.4.0, transformers 5.16.1, trl 1.12.0, peft 0.20.0, mlx-lm 0.31.3 · teacher and student as listed per row, teacher Q4_K_M; student BF16 in training, Q4_K_M after export · 1,024 tokens of context · 2026-09-09

Not yet run on hardware on any track. Until the validation pass fills these in, the table records what to measure rather than what to expect. The gap-closed column comes from evaluate-triplet.py and is meaningful only alongside the per-category table it prints underneath.

Troubleshooting

ModuleNotFoundError: No module named 'distillog'. Every script in this part imports the run log from the directory it sits in. Put distillog.py beside them and run from that directory.

The generation script exits saying the output file exists. That is deliberate: overwriting two hours of generation by rerunning a command is a bad afternoon. Pass --resume to append and skip what is already there, or choose a new name.

Every answer comes back empty, or usage is missing. Check the single curl from task 4 again. An empty content with a populated reasoning_content means the engine is separating the thinking block into its own field, which some servers do; either read that field or turn thinking off for this dataset. Missing usage means the engine does not report token counts and the cost record will have nulls in it.

Throughput stops improving as you raise --concurrency. You have more requests in flight than the server has slots. Check the slots endpoint and set the concurrency to match --parallel.

The filter keeps almost nothing. Read filter-report.json before changing a threshold. The common causes are a length filter set for a different answer style, a teacher answering the format-category prompts in a fenced code block when the prompt asked for bare JSON, and a temperature low enough that many prompts produced near-identical answers.

The filter keeps almost everything, including obvious rubbish. The category checks only run for the categories they know about. If you replaced the seed set with your own prompts, the category field on each line is what selects the check, and a category the script does not recognise gets only the length and duplication filters.

Out of memory during training. In order: lower --max-length, lower --batch-size and raise --grad-accum to keep the effective batch, then add --gradient-checkpointing. If it still does not fit, you are on the wrong row of the tier table; drop the student a size.

Training loss falls but the evaluation shows no change. Check that the model you evaluated is the one you trained. The commonest version of this is evaluating the merged model while the gateway is still serving the previous GGUF file under the same alias, which is a stale-cache problem rather than a training problem.

The distilled student is worse than the base student everywhere. Do not adjust hyperparameters yet. Go to this part’s challenge, which is written for exactly this and starts by collecting evidence.

Cleanup

Keep labbook.md, seeds/, data/, filter-report.json, eval-out/ and the quantised student. The project later in this part reuses all of them, and Part 16 quantises this student five ways.

RunnableAll tracks

reclaim the disk, keeping the evidence

rm -rf runs/seq-qwen3-4b/checkpoint-*
rm -rf models/seq-qwen3-4b-merged models/seq-qwen3-4b.gguf

⚠Warning

That second command deletes the merged model and the full-precision GGUF, which together are the largest artefacts this lab produces. They are safe to delete only because merge-adapter.py recreates them from the adapter, which you are keeping. Never delete the adapter or the training data: the adapter is the training run’s only output and the data is what the run log’s hash refers to.

Raw generation is worth keeping if you have the disk, because it lets you rerun the filter with different thresholds without regenerating. If you delete it, record its hash first; it is already in the run log.

What you learned

• Sequence-level distillation is supervised fine-tuning with a different dataset. Every step after the filter was Part 13’s, unchanged, and the only new machinery was in producing and cleaning the data.
• Two phases, two memory profiles. The teacher is served and the student is trained, and they need not be resident together. That single fact is why a 12 GB machine can distil from an 8B-class teacher at all.
• The filter is where the quality is decided. You read the rejections, and they told you things about your teacher that the totals did not.
• Decontamination is what makes the number reportable. Without it you cannot distinguish a model that improved from a model that saw the answers.
• The gap that closed is the result, and the category that fell is the finding. A single total hides the second, which is why the evaluation prints both.
• Generation is the cost, and it is a one-off. The dataset on disk is reusable; the next experiment on it costs only training time.

Record in the notebook: the four predicted memory figures and the peak reached; the number of prompts, raw answers and kept examples, with the top three rejection reasons; the SHA-256 of the training file; the trainable parameter count and percentage; generation wall clock and completion tokens; training wall clock; the three evaluation scores; the percentage of the gap closed; and the category where the distilled student did worst.

Check your understanding

Question 1. Why does this lab have a 12 GB floor when the logit lab in the same part has 24 GB?

• Because it uses a smaller teacher
• Because the teacher is served and the student is trained in separate phases, so only one large model is resident at a time; the logit lab needs both at once
• Because it trains for fewer epochs
• Because the dataset is smaller

Show the answer and why

Answer: Because the teacher is served and the student is trained in separate phases, so only one large model is resident at a time; the logit lab needs both at once

The two memory diagrams in task 1 are the whole answer. Generation is inference with a resident teacher; training is a resident student with gradients and optimiser state. Sequence-level distillation never needs the two together, which is also why the teacher can be on another machine.

Question 2. The filter report shows 40 examples rejected as "refusal-invented-a-specific". What has it told you?

• The filter is too strict
• That on prompts whose honest answer is "I cannot know that", the teacher invented a hostname, a port or a count instead, and those examples would have taught the student to do the same
• That the refusal prompts are badly written
• That the teacher is broken

Show the answer and why

Answer: That on prompts whose honest answer is "I cannot know that", the teacher invented a hostname, a port or a count instead, and those examples would have taught the student to do the same

It is a property of the teacher, discovered cheaply. Without the check those forty examples go straight into training data, and the student learns confident invention on exactly the questions where it matters most. This is the propagated-teacher-error fault from this part's challenge, caught early.

Question 3. You set --concurrency 16 against a server started with --parallel 4. What happens?

• The server rejects twelve requests
• Generation is four times faster
• Twelve requests queue; throughput stops rising and per-request latency climbs, which looks like the model getting slower
• The KV cache grows to accommodate them

Show the answer and why

Answer: Twelve requests queue; throughput stops rising and per-request latency climbs, which looks like the model getting slower

Slot count is fixed at start-up and the KV cache is sized for it. Extra client concurrency does not create slots. The slots monitoring endpoint, enabled by default, is the way to see what is actually running in parallel.

Question 4. Which of these make the final number reportable rather than merely encouraging? Select all that apply.

• Decontaminating the training data against the evaluation task file
• Evaluating the base student as well as the distilled one
• Evaluating the teacher too, so the gap has a top as well as a bottom
• Recording the sampling settings and the quantisation each model was served at

Show the answer and why

Answer: Decontaminating the training data against the evaluation task file, Evaluating the base student as well as the distilled one, Evaluating the teacher too, so the gap has a top as well as a bottom, Recording the sampling settings and the quantisation each model was served at

Each removes a way of being wrong: contamination invalidates the comparison, no base means no baseline, no teacher means "closed the gap" has no denominator, and unrecorded settings mean nobody can reproduce or compare the run, including you.

Question 5. The trainer reports about 33 million trainable parameters for a rank-16 adapter on Qwen3-4B. Why is that the number to expect?

• It is one per cent of the model by convention
• Because the config gives 36 layers, hidden size 2,560 and intermediate size 9,728, and a rank-16 adapter on the seven projections of each layer comes to about 917,000 parameters per layer
• Because PEFT defaults to that size
• Because the dataset has 33 million tokens

Show the answer and why

Answer: Because the config gives 36 layers, hidden size 2,560 and intermediate size 9,728, and a rank-16 adapter on the seven projections of each layer comes to about 917,000 parameters per layer

It is arithmetic from the architecture, which is why the lab asks you to predict it before the run. A count far from the prediction means the rank or the target module list is not what you thought, and finding that out from a number is cheaper than finding it out from a result.

Check my answersClear