Challenge: The Student That Learned the Teacher's Mistakes

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track reproductions, fixes and the versions they were reproduced with belong here once the validation pass has run this page on real machines.

Objective

Somebody, possibly you last week, says: “I distilled our 30B model into a 4B and the evaluation score went up nicely, but people say the small one is worse.” By the end of this page you will have a procedure that answers that in about ten minutes, and you will have run it on four faults you introduced deliberately, so that you recognise the symptoms when the fault is not yours.

The deliverable is a written diagnosis, not a fixed model: the evidence you collected, the fault it pointed at, the single change you made, and the second measurement that proves it. Evidence, then theory, then proof.

⌂Single-machine path

Single-machine on every track. Three of the four faults need no accelerator at all: they are found by reading files. The fourth, tokeniser mismatch, is reproduced fully only on a machine that can run this part’s logit lab, and the evidence for it is collectable anywhere.

The procedure

A distilled student that got worse: what to do, in order

1. 1Separate the two claimsDid the score go up, and did behaviour get worse? They are different measurements and the second one is usually not written down anywhere yet.
2. 2Check the evaluation before the modelIf the training data overlapped the evaluation set, the score is not evidence of anything and every other investigation is premature.
3. 3Verify a sample of the teacherThe student can only be as right as the data. A few hundred teacher answers, checked by a program where possible and by eye where not.
4. 4Compare the tokenisersOnly for a logit or on-policy run. Thirty seconds, and it either eliminates a fault or explains everything at once.
5. 5Read the settings out of the run logTemperature, sampling mode, samples per prompt, divergence coefficient, filter thresholds. A stage with no record cannot be reproduced.
6. 6Form one hypothesis and change one thingThen re-run the same measurement. Two changes at once means you will not know which worked.
7. 7Write it downBefore and after, the evidence, the fix. Next time this takes two minutes.

The order matters. The second step is first among the real checks because contamination invalidates the number that started the investigation, and there is no point diagnosing a model with a measurement you cannot trust.

The short list of faults

Almost every bad distillation is one of these four. Learn the list and you have learned the diagnosis.

Fault	What the evidence looks like
Teacher errors propagated, because nothing verified the teacher	A sample of the raw teacher file contains confidently wrong answers, invented specifics on unanswerable questions, or empty completions, at a rate above a per cent or two. The student reproduces them faithfully.
Evaluation contaminated by teacher output	Training prompts match evaluation prompts exactly, or a containment check finds high overlap. The gain is concentrated in the contaminated categories and absent elsewhere.
Tokeniser mismatch in a logit run	Teacher and student report different vocabulary sizes, different ids for the same probe string, or different special-token ids. The loss fell anyway.
Temperature and loss weighting chosen without being recorded	The run log has no generate stage, or no temperature in it; or the generation was at temperature 0 and the filter rejected most examples as duplicates; or a divergence coefficient was changed at the same time as something else.

Notice what is not on the list: the learning rate, the rank and the number of epochs. Those change the result by amounts you can argue about. These four change it by amounts you cannot.

Requirements

The artefacts from this part’s sequence-level lab: seeds/, raw/teacher.jsonl, data/, filter-report.json, the adapter, and labbook.md. Your Part 10 task file. Forty-five minutes, all attended. No new downloads except the tokenisers in task 5, which are a few megabytes each.

Track S — NVIDIA DGX Spark

Everything on this page runs. If you also completed the logit lab, do the full tokeniser-mismatch reproduction in task 5 rather than the evidence-only version; a 128 GB pool holds a mismatched pair without difficulty and watching a loss fall while the student learns nothing is the most memorable ten minutes in this part.

Track X — AMD Ryzen AI Max+ 395

Tasks 2, 3, 4 and 6 are pure file reading and need no accelerator. Task 5’s evidence step needs only the tokenisers. The full mismatch reproduction needs the training path from the logit lab, which on this track depends on the ROCm build of PyTorch; where that is unavailable, the evidence-only version is complete for the purposes of this page.

Track M — Apple silicon

All four evidence steps run. distillog.py cannot report power on this track, so the run log’s energy figures are null; that is expected and is not one of the faults. The full mismatch reproduction runs under PyTorch’s MPS backend in float32 with a small pair, and is optional.

Track N — NVIDIA desktop or laptop

At 12 to 16 GB, do the evidence-only version of task 5 and the regeneration in task 6 with the small teacher from the sequence-level lab’s tier table. At 24 GB and above, the full mismatch reproduction is available and worth the ten minutes.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-15-distillation"
cd "$LAB_DIR"
pwd
test -f "diagnose-distillation.py"

Expected result: pwd ends in part-15-distillation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Separate the two claims, and write them down

“The score went up and the model got worse” is two statements. Before touching anything, write both in the notebook in a form that could be false.

• The score went up. Which task file, which settings, which two models compared, and by how many checks. You have this from the sequence-level lab: evaluate-triplet.py wrote it.
• The model got worse. On what? A category in the same task file, a task not in the file at all, or an impression from somebody using it? All three are legitimate starting points, and only the first is already measured.

If the second claim has no measurement behind it, get one now, before diagnosing. Ten prompts of the kind the complainer is running, asked of the base student and the distilled student at the same settings, is enough to turn an impression into a comparison.

2. Collect the evidence with one command

RunnableAll tracks

diagnose-distillation.py

"""Collect the evidence for a distilled student that improved on your set and nothing else.

Purpose: the diagnostic half of Part 15's challenge. Four faults produce that same
    symptom, and each leaves a different trace on disk. This reads the run log, the
    teacher's raw output, the training file, the evaluation set and the two
    tokenisers, then writes one report with a verdict line per fault. It changes
    nothing, needs no accelerator and starts no model, so it is safe to run first
    and cheap to run again after a fix.
Platform: all (standard library, plus transformers only for the tokeniser check,
    which is skipped with a stated reason when transformers is not installed)
Minimum memory: 12 GB nominally; the checks themselves are text in memory
Assumes: Python 3.10 or newer. Whatever exists is read; whatever is missing is
    reported as missing rather than guessed, because "not recorded" is itself
    evidence. distillog.py sits next to this file.

Usage: python3 diagnose-distillation.py --labbook labbook.md --train data/train.jsonl \\
           --tasks my-tasks.json --raw raw/teacher.jsonl --report diagnosis.md
       python3 diagnose-distillation.py --labbook labbook.md --train data/train.jsonl \\
           --tasks my-tasks.json --teacher Qwen/Qwen3-8B --student Qwen/Qwen3-1.7B \\
           --report diagnosis.md
       python3 diagnose-distillation.py --labbook labbook.md --report diagnosis.md
       # the minimum: the run log alone, with everything else reported as absent

The four faults, in the order the challenge page works through them:
  1. teacher errors propagated because nothing verified the teacher
  2. the evaluation set contaminated by teacher output
  3. teacher and student tokenisers differ, in a logit distillation run
  4. sampling temperature and loss weighting chosen without being recorded
"""

from __future__ import annotations

import argparse
import json
import re
from collections import Counter
from pathlib import Path
from typing import Any

import distillog

WORD_RE = re.compile(r"[a-z0-9]+")
UNKNOWN = "not available"

FINAL_PATTERNS = (
    re.compile(r"####\s*(-?[0-9][0-9,]*(?:\.[0-9]+)?)"),
    re.compile(r"\\boxed\{\s*(-?[0-9][0-9,]*(?:\.[0-9]+)?)\s*\}"),
    re.compile(r"(?i)\banswer\s*[:=]\s*\$?(-?[0-9][0-9,]*(?:\.[0-9]+)?)"),
)
INVENTED_SPECIFIC = re.compile(
    r"\b(?:\d{1,3}(?:\.\d{1,3}){3}|port\s+\d{2,5}|(?:host(?:name)?|machine)\s+is\s+\S+)\b",
    re.IGNORECASE,
)
REFUSAL_MARKERS = re.compile(
    r"\b(?:cannot|can't|can not|no access|not able|do not have|don't have|unable)\b", re.IGNORECASE
)

PROBE_STRINGS = (
    "<|im_start|>user\nHello, world!<|im_end|>\n",
    "<think>\n2 + 2 = 4\n</think>\nAnswer: 4",
    "quantisation, tokeniser, générateur, 分词器",
)


def normalise(text: str) -> list[str]:
    return WORD_RE.findall(text.lower())


def ngrams(words: list[str], n: int) -> set[tuple[str, ...]]:
    if len(words) < n:
        return {tuple(words)} if words else set()
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}


def containment(a: set[tuple[str, ...]], b: set[tuple[str, ...]]) -> float:
    return len(a & b) / len(a) if a else 0.0


def read_jsonl(path: Path) -> list[dict]:
    if not path.is_file():
        return []
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError:
                continue
    return rows


def training_texts(rows: list[dict]) -> list[tuple[str, str]]:
    """Accept every dataset shape this course writes, and return (prompt, completion)."""
    out = []
    for row in rows:
        if isinstance(row.get("prompt"), list) and isinstance(row.get("completion"), list):
            prompt = " ".join(m.get("content", "") for m in row["prompt"])
            completion = " ".join(m.get("content", "") for m in row["completion"])
        elif isinstance(row.get("prompt"), str):
            prompt, completion = row["prompt"], row.get("completion", "")
        elif isinstance(row.get("messages"), list):
            prompt = " ".join(m.get("content", "") for m in row["messages"] if m.get("role") != "assistant")
            completion = " ".join(m.get("content", "") for m in row["messages"] if m.get("role") == "assistant")
        elif isinstance(row.get("text"), str):
            prompt, completion = "", row["text"]
        else:
            continue
        out.append((prompt, completion))
    return out


# ---------------------------------------------------------------------------
# Fault 1: teacher errors propagated
# ---------------------------------------------------------------------------

def check_teacher_quality(raw_rows: list[dict], sample: int) -> dict[str, Any]:
    if not raw_rows:
        return {"status": "skipped", "reason": "no --raw teacher file given"}
    checked = raw_rows[:sample] if sample else raw_rows
    verifiable = [r for r in checked if "answer" in r]
    wrong = 0
    unparsed = 0
    for row in verifiable:
        text = row.get("completion") or ""
        predicted = None
        for pattern in FINAL_PATTERNS:
            found = pattern.findall(text)
            if found:
                try:
                    predicted = float(found[-1].replace(",", ""))
                except ValueError:
                    predicted = None
                break
        if predicted is None:
            unparsed += 1
        elif abs(predicted - float(row["answer"])) > 1e-6:
            wrong += 1

    refusal_rows = [r for r in checked if r.get("category") == "refusal"]
    bad_refusals = sum(
        1 for r in refusal_rows
        if INVENTED_SPECIFIC.search(r.get("completion") or "")
        or not REFUSAL_MARKERS.search(r.get("completion") or "")
    )
    empty = sum(1 for r in checked if not (r.get("completion") or "").strip())

    return {
        "status": "checked",
        "sampled": len(checked),
        "verifiable": len(verifiable),
        "verifiably_wrong": wrong,
        "answer_not_found": unparsed,
        "refusal_prompts": len(refusal_rows),
        "refusals_that_invented_or_complied": bad_refusals,
        "empty_completions": empty,
        "error_rate": round((wrong + bad_refusals + empty) / len(checked), 3) if checked else None,
    }


# ---------------------------------------------------------------------------
# Fault 2: contaminated evaluation
# ---------------------------------------------------------------------------

def check_contamination(train_rows: list[dict], tasks_path: Path, n: int, threshold: float) -> dict[str, Any]:
    if not train_rows:
        return {"status": "skipped", "reason": "no --train file given"}
    if not tasks_path or not tasks_path.is_file():
        return {"status": "skipped", "reason": "no --tasks evaluation set given"}
    spec = json.loads(tasks_path.read_text(encoding="utf-8"))
    tasks = spec.get("tasks", [])
    pairs = training_texts(train_rows)
    train_exact = {" ".join(normalise(p)) for p, _ in pairs if p}
    train_grams = [ngrams(normalise(f"{p}\n{c}"), n) for p, c in pairs]

    exact_hits, overlap_hits, worst = [], [], []
    for task in tasks:
        prompt = task.get("prompt", "")
        reference = task.get("reference", "") or ""
        key = " ".join(normalise(prompt))
        if key and key in train_exact:
            exact_hits.append(task.get("id"))
            continue
        target = ngrams(normalise(f"{prompt}\n{reference}"), n)
        best = max((containment(target, g) for g in train_grams if g), default=0.0)
        worst.append((round(best, 3), task.get("id")))
        if best >= threshold:
            overlap_hits.append({"id": task.get("id"), "containment": round(best, 3)})
    worst.sort(reverse=True)
    return {
        "status": "checked",
        "tasks": len(tasks),
        "training_examples": len(pairs),
        "exact_matches": exact_hits,
        "high_overlap": overlap_hits,
        "highest_containment": worst[:5],
        "threshold": threshold,
        "n": n,
    }


# ---------------------------------------------------------------------------
# Fault 3: tokeniser mismatch
# ---------------------------------------------------------------------------

def check_tokenisers(teacher_id: str | None, student_id: str | None) -> dict[str, Any]:
    if not teacher_id or not student_id:
        return {"status": "skipped",
                "reason": "pass --teacher and --student to compare vocabularies"}
    try:
        from transformers import AutoTokenizer  # noqa: PLC0415 - optional here on purpose
    except ImportError:
        return {"status": "skipped", "reason": "transformers is not installed in this environment"}
    try:
        teacher_tok = AutoTokenizer.from_pretrained(teacher_id)
        student_tok = AutoTokenizer.from_pretrained(student_id)
    except (OSError, ValueError) as exc:
        return {"status": "skipped", "reason": f"could not load a tokeniser: {exc}"}

    mismatches = []
    for probe in PROBE_STRINGS:
        a = teacher_tok(probe, add_special_tokens=False)["input_ids"]
        b = student_tok(probe, add_special_tokens=False)["input_ids"]
        if a != b:
            mismatches.append({"probe": probe[:50], "teacher": a[:16], "student": b[:16]})
    teacher_specials = teacher_tok.get_added_vocab()
    student_specials = student_tok.get_added_vocab()
    special_diffs = [
        {"token": t, "teacher_id": i, "student_id": student_specials.get(t)}
        for t, i in sorted(teacher_specials.items()) if student_specials.get(t) != i
    ]
    return {
        "status": "checked",
        "teacher": teacher_id,
        "student": student_id,
        "teacher_class": type(teacher_tok).__name__,
        "student_class": type(student_tok).__name__,
        "teacher_vocab_size": len(teacher_tok),
        "student_vocab_size": len(student_tok),
        "probe_mismatches": mismatches,
        "special_token_differences": special_diffs[:10],
        "compatible": (len(teacher_tok) == len(student_tok) and not mismatches and not special_diffs),
    }


# ---------------------------------------------------------------------------
# Fault 4: settings nobody recorded
# ---------------------------------------------------------------------------

def check_settings(labbook: str) -> dict[str, Any]:
    stages = distillog.read_stages(labbook)
    if not stages:
        return {"status": "skipped", "reason": f"no Part 15 records found in {labbook}"}
    by_stage: dict[str, dict] = {}
    for record in stages:
        by_stage[record.get("stage", "?")] = record

    generate = by_stage.get("generate", {})
    train_logit = by_stage.get("train-logit", {})
    gen_hyper = generate.get("hyperparameters") or {}
    logit_hyper = train_logit.get("hyperparameters") or {}

    missing = []
    for stage in ("generate", "filter", "train"):
        if stage not in by_stage:
            missing.append(stage)
    return {
        "status": "checked",
        "stages_present": sorted(by_stage),
        "stages_missing": missing,
        "generation_temperature": gen_hyper.get("temperature", UNKNOWN),
        "generation_top_p": gen_hyper.get("top_p", UNKNOWN),
        "generation_mode": gen_hyper.get("mode", UNKNOWN),
        "generation_samples_per_prompt": gen_hyper.get("samples", UNKNOWN),
        "distillation_beta": logit_hyper.get("beta", UNKNOWN),
        "distillation_temperature": logit_hyper.get("temperature", UNKNOWN),
        "distillation_trainer": logit_hyper.get("trainer", UNKNOWN),
        "filter_thresholds": (by_stage.get("filter", {}).get("hyperparameters") or {}) or UNKNOWN,
    }


# ---------------------------------------------------------------------------

def verdict(name: str, failed: bool, skipped: bool, detail: str) -> str:
    if skipped:
        return f"- **{name}: not checked.** {detail}"
    return f"- **{name}: {'EVIDENCE FOUND' if failed else 'no evidence'}.** {detail}"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md")
    parser.add_argument("--raw", default=None, help="the teacher's raw output, for the verifier check")
    parser.add_argument("--train", default=None, help="the training file the student learned from")
    parser.add_argument("--tasks", default=None, help="the evaluation set the gain was measured on")
    parser.add_argument("--teacher", default=None, help="teacher model id or path, for the tokeniser check")
    parser.add_argument("--student", default=None, help="student model id or path, for the tokeniser check")
    parser.add_argument("--sample", type=int, default=200, help="teacher answers to verify")
    parser.add_argument("--n", type=int, default=13)
    parser.add_argument("--threshold", type=float, default=0.5)
    parser.add_argument("--report", default="diagnosis.md")
    args = parser.parse_args()

    raw_rows = read_jsonl(Path(args.raw)) if args.raw else []
    train_rows = read_jsonl(Path(args.train)) if args.train else []

    teacher_quality = check_teacher_quality(raw_rows, args.sample)
    contamination = check_contamination(train_rows, Path(args.tasks) if args.tasks else None,
                                        args.n, args.threshold)
    tokenisers = check_tokenisers(args.teacher, args.student)
    settings = check_settings(args.labbook)

    fault1 = (teacher_quality.get("status") == "checked"
              and (teacher_quality.get("error_rate") or 0) > 0.02)
    fault2 = (contamination.get("status") == "checked"
              and (contamination.get("exact_matches") or contamination.get("high_overlap")))
    fault3 = tokenisers.get("status") == "checked" and not tokenisers.get("compatible")
    fault4 = (settings.get("status") == "checked"
              and (settings.get("stages_missing")
                   or settings.get("generation_temperature") == UNKNOWN))

    lines = [
        "# Distillation diagnosis",
        "",
        "Evidence collected without changing anything. Read the verdicts, then read the",
        "sections: a verdict is a pointer to the numbers under it, not a conclusion.",
        "",
        "## Verdicts",
        "",
        verdict("1. Teacher errors propagated",
                bool(fault1), teacher_quality.get("status") != "checked",
                teacher_quality.get("reason", "")
                or f"{teacher_quality.get('verifiably_wrong', 0)} verifiably wrong, "
                   f"{teacher_quality.get('refusals_that_invented_or_complied', 0)} bad refusals, "
                   f"{teacher_quality.get('empty_completions', 0)} empty, out of "
                   f"{teacher_quality.get('sampled', 0)} sampled."),
        verdict("2. Evaluation contaminated by teacher output",
                bool(fault2), contamination.get("status") != "checked",
                contamination.get("reason", "")
                or f"{len(contamination.get('exact_matches', []))} exact match(es) and "
                   f"{len(contamination.get('high_overlap', []))} task(s) above containment "
                   f"{args.threshold} against {contamination.get('training_examples', 0)} "
                   "training examples."),
        verdict("3. Tokeniser mismatch",
                bool(fault3), tokenisers.get("status") != "checked",
                tokenisers.get("reason", "")
                or (f"teacher {tokenisers.get('teacher_vocab_size')} entries, student "
                    f"{tokenisers.get('student_vocab_size')} entries, "
                    f"{len(tokenisers.get('probe_mismatches', []))} probe mismatch(es).")),
        verdict("4. Settings not recorded, or a stage missing",
                bool(fault4), settings.get("status") != "checked",
                settings.get("reason", "")
                or f"stages present: {', '.join(settings.get('stages_present', []))}; "
                   f"missing: {', '.join(settings.get('stages_missing', [])) or 'none'}; "
                   f"generation temperature: {settings.get('generation_temperature')}."),
        "",
        "## 1. Did anything verify the teacher?",
        "",
        "```json",
        json.dumps(teacher_quality, indent=2),
        "```",
        "",
        "A teacher error rate above a couple of per cent, propagated into training data with",
        "no verifier, is enough to teach a student a wrong habit that no evaluation on the",
        "same distribution will catch.",
        "",
        "## 2. Does the training data overlap the evaluation set?",
        "",
        "```json",
        json.dumps(contamination, indent=2),
        "```",
        "",
        "An exact match is contamination and the fix is deletion. A high containment score is",
        "a pair to read yourself: paraphrase or coincidence is a judgement, not a threshold.",
        "",
        "## 3. Do the teacher and the student share a vocabulary?",
        "",
        "```json",
        json.dumps(tokenisers, indent=2),
        "```",
        "",
        "This matters only for logit distillation, where a teacher probability is applied to",
        "the student's vocabulary at the same index. Sequence-level distillation moves text,",
        "so a mismatch here is not a fault for that route.",
        "",
        "## 4. What settings produced this?",
        "",
        "```json",
        json.dumps(settings, indent=2, default=str),
        "```",
        "",
        f"Records read from `{args.labbook}`. A stage with no record did not necessarily not",
        "happen, but it cannot be reproduced, which for this purpose is the same thing.",
        "",
    ]
    Path(args.report).write_text("\n".join(lines), encoding="utf-8")

    verdict_start = lines.index("## Verdicts")
    print("\n".join(lines[verdict_start:verdict_start + 6]))
    print(f"\nfull report written to {args.report}")
    if not any([fault1, fault2, fault3, fault4]):
        print("No fault found from the evidence supplied. Check what you did not pass in: "
              "the checks that were skipped are listed as 'not checked' above.")


if __name__ == "__main__":
    main()

Download diagnose-distillation.py407 lines

The script reads the run log, the raw teacher output, the training file, the evaluation set and the two tokenisers, and writes one report with a verdict per fault. It changes nothing, starts no model and needs no accelerator.

RunnableAll tracks

collect the evidence

python diagnose-distillation.py \
    --labbook labbook.md \
    --raw raw/teacher.jsonl \
    --train data/train.jsonl \
    --tasks my-tasks.json \
    --report diagnosis-healthy.md

Output — what you should see

## Verdicts

- **1. Teacher errors propagated: ...** ... verifiably wrong, ... bad refusals, ... empty, out of ... sampled.
- **2. Evaluation contaminated by teacher output: ...** ... exact match(es) and ... task(s) above containment 0.5 ...
- **3. Tokeniser mismatch: not checked.** pass --teacher and --student to compare vocabularies
- **4. Settings not recorded, or a stage missing: ...** stages present: ...; missing: ...

Read diagnosis-healthy.md end to end now, while everything is as it should be. Knowing what a healthy report looks like is what makes an unhealthy one legible.

✕Common mistake

Reading the score first and theorising from it. A score tells you something is wrong and nothing about what. People who start there change the learning rate, then the rank, then the epochs, and find the contaminated evaluation set by accident two days later. The four checks above are file reads and they answer the question directly.

3. Fault one: the teacher nobody verified

Reproduce it. The sequence-level lab’s filter rejected teacher answers by category, and the refusal category is where a teacher’s confident invention shows up. Turn that check off by removing the category labels, and watch what gets through.

RunnableAll tracks

strip the category labels, so the category checks cannot run

python -c "
import json
rows = [json.loads(l) for l in open('raw/teacher.jsonl') if l.strip()]
with open('raw/teacher-uncategorised.jsonl', 'w') as h:
    for r in rows:
        r.pop('category', None)
        h.write(json.dumps(r) + '\n')
print(len(rows), 'rows written without categories')
"
python filter-and-dedupe.py \
    --raw raw/teacher-uncategorised.jsonl \
    --out-dir broken-1 \
    --tasks my-tasks.json

Compare the two rejection tables. The healthy run rejected a number of examples as refusal-invented-a-specific and format-not-json; this run rejects none of them, because the check that names them had no category to key on. Those examples are now training data.

The evidence.

RunnableAll tracks

what the teacher actually did on the unanswerable prompts

python diagnose-distillation.py \
    --labbook labbook.md \
    --raw raw/teacher.jsonl \
    --train broken-1/data/train.jsonl \
    --tasks my-tasks.json \
    --report diagnosis-fault-1.md

The first verdict block is the one to read: the number of refusal prompts where the teacher invented a hostname, a port or a count instead of declining, and the number of verifiably wrong answers where an answer could be checked at all.

The fix. Restore the category labels so the checks run, and where a category can be verified by a program, verify it. This part’s reasoning lesson is the general form: a verifier is a reward function used as a filter, and Part 14’s lesson on reward functions is where the technique comes from.

The proof. The rejection table names the failures again, and the same prompts asked of a student trained on the cleaned data no longer produce invented specifics. That second half matters: a filter that rejects more examples has not proved anything until the student’s behaviour changes.

?Reality check

The claim behind this fault is that a bigger model’s answers are good enough to train on. The measurable version is that a bigger model’s answers are better on average, and the average is not what a student learns from: it learns every example equally, including the confidently wrong ones, and it has no way to tell them apart.

4. Fault two: the contaminated evaluation

This is the fault that produces the exact symptom in the title, and it is the easiest of the four to create by accident.

Reproduce it. Seed a handful of generation prompts directly from your evaluation set, generate answers for them, and skip decontamination.

RunnableAll tracks

build a deliberately contaminated seed slice

python -c "
import json
spec = json.load(open('my-tasks.json'))
rows = [{'id': 'x%03d' % i, 'category': t.get('category', 'explain'), 'prompt': t['prompt']}
        for i, t in enumerate(spec['tasks'][:10], start=1)]
with open('seeds/contaminated.jsonl', 'w') as h:
    for r in rows:
        h.write(json.dumps(r) + '\n')
print(len(rows), 'evaluation prompts copied into a seed file')
"
python generate-teacher-data.py \
    --seeds seeds/contaminated.jsonl \
    --out raw/contaminated.jsonl \
    --base-url http://127.0.0.1:8080/v1 \
    --model teacher \
    --teacher-id qwen3-8b \
    --temperature 0.7 --top-p 0.8 \
    --concurrency 2

Then filter without --tasks, which is exactly what a hurried run does.

RunnableAll tracks

filter with decontamination skipped

cat raw/teacher.jsonl raw/contaminated.jsonl > raw/mixed.jsonl
python filter-and-dedupe.py --raw raw/mixed.jsonl --out-dir broken-2

The script prints, in capitals, that decontamination was skipped and records that in the run log. That message is the fault announcing itself, and the exercise is to notice how easy it is to scroll past.

The evidence.

RunnableAll tracks

the contamination check, against the same evaluation set

python diagnose-distillation.py \
    --labbook labbook.md \
    --train broken-2/data/train.jsonl \
    --tasks my-tasks.json \
    --report diagnosis-fault-2.md

The second verdict now lists exact matches by task id, and the highest containment scores. A task id in exact_matches means the student was trained on the answer to that question.

The fix. Rerun the filter with --tasks my-tasks.json, which drops those examples and records how many it dropped.

The proof. Retrain on the cleaned data and re-run evaluate-triplet.py. The gain will be smaller than it was, and that smaller number is the real one. A fix that makes your headline result worse is still a fix; reporting the larger number would have been the failure.

5. Fault three: the tokeniser mismatch

Only for a logit or on-policy run. Sequence-level distillation moves text and has no such condition.

The evidence, on any machine.

RunnableAll tracks

compare two vocabularies without training anything

python train-logit-distil.py \
    --teacher meta-llama/Llama-3.1-8B-Instruct \
    --student Qwen/Qwen3-1.7B \
    --check-tokenisers-only

Output — what you should see

teacher tokeniser: ..., ... entries
student tokeniser: Qwen2Tokenizer, 151936 entries
NOT compatible:
  vocabulary sizes differ
  probe '<|im_start|>user\nHello, world!<|im_end|>\n'
    teacher [...]
    student [...]

Two different vocabulary sizes and different id sequences for the same string. A distillation loss would take the teacher’s probability at index n and apply it to whatever the student has at index n, which is a different token. TRL’s own documentation warns that a teacher with a different vocabulary trains the student against the wrong tokens, and that the failure is silent when the teacher’s vocabulary is no larger than the student’s.

Note that Llama 3.1 8B Instruct is gated on the Hub, so this check needs an accepted licence and a logged-in hf session. If you have not accepted it, substitute any non-Qwen instruction model you already have; the point is a different tokeniser, not that particular one.

Reproduce it fully, on 24 GB and above.

RunnableTrack N · NVIDIA GPU

train a mismatched pair on purpose, and watch the loss fall

python train-logit-distil.py \
    --teacher Qwen/Qwen3-4B \
    --student Qwen/Qwen3-0.6B \
    --prompts seeds/prompts.jsonl \
    --output-dir runs/mismatch-demo \
    --limit 40 --epochs 1 \
    --allow-tokeniser-mismatch

That pair is compatible, so it trains normally and gives you the reference shape of a healthy loss curve. Now do the same with a genuinely mismatched teacher and compare: the loss still falls, and the student’s generations are worse than the base model’s. A falling loss is not evidence that a distillation run is working. That sentence is the whole fault.

The fix. Use a same-family pair, or switch to sequence-level distillation, which needs only text.

The proof. The compatibility check reports compatible, and the student’s generations on a handful of prompts are recognisably better than the base model’s rather than worse.

6. Fault four: settings nobody recorded

Reproduce it. Generate a small slice at temperature 0 and compare what the filter does with it.

RunnableAll tracks

generate the same prompts twice, at two temperatures

head -n 60 seeds/prompts.jsonl > seeds/slice.jsonl
python generate-teacher-data.py --seeds seeds/slice.jsonl --out raw/t00.jsonl \
    --base-url http://127.0.0.1:8080/v1 --model teacher --teacher-id qwen3-8b \
    --temperature 0.0 --top-p 1.0 --concurrency 2
python generate-teacher-data.py --seeds seeds/slice.jsonl --out raw/t07.jsonl \
    --base-url http://127.0.0.1:8080/v1 --model teacher --teacher-id qwen3-8b \
    --temperature 0.7 --top-p 0.8 --concurrency 2
python filter-and-dedupe.py --raw raw/t00.jsonl --out-dir broken-4a --tasks my-tasks.json
python filter-and-dedupe.py --raw raw/t07.jsonl --out-dir broken-4b --tasks my-tasks.json

Compare the two filter-report.json files. The greedy run should lose far more examples to near-duplicate and duplicate-completion, because a deterministic teacher answers similar prompts in similar words. A dataset that survived greedy generation is a narrow dataset, and a student trained on it learns one way of answering.

The Qwen3 cards make the same point from the other direction for thinking mode, warning against greedy decoding because it “can lead to performance degradation and endless repetitions”.

The loss-weighting half. In a logit run, temperature and the divergence coefficient interact. The 1980s-era intuition here is exact: the original distillation paper notes that soft-target gradients “scale as 1/T²” and must be multiplied by the square of the temperature when hard and soft targets are mixed, precisely so that changing the temperature does not silently change the balance between the two objectives. Change --temperature and --beta in the same run and you have changed two things, and the log will not tell you which mattered.

The evidence.

RunnableAll tracks

what the run log knows about your settings

python diagnose-distillation.py --labbook labbook.md --report diagnosis-fault-4.md
python distillog.py --stages --labbook labbook.md

The fourth verdict lists the stages present and missing and the generation settings it found. A missing stage or a not available temperature is the fault: the run cannot be reproduced, so it cannot be compared with the next one.

The fix. Pass --labbook to every stage, which the pipeline runner in this part’s project does automatically, and change one setting per experiment.

The proof. A second run of distillog.py --stages shows every stage present, and two runs that differ in one setting can be compared.

7. Find the fault you did not introduce

With the procedure fresh, run the diagnosis once against your real artefacts and read it properly.

RunnableAll tracks

a routine health check on your own distillation

python diagnose-distillation.py \
    --labbook labbook.md \
    --raw raw/teacher.jsonl \
    --train data/train.jsonl \
    --tasks my-tasks.json \
    --teacher Qwen/Qwen3-8B --student Qwen/Qwen3-1.7B \
    --report diagnosis-real.md

This is where people discover that the teacher’s error rate on their own prompts is higher than they assumed, or that two evaluation tasks were paraphrased into the seed set months ago.

Trace a wrong student answer back through the pipeline

Choose a failing held-out task and inspect the untouched student, teacher and distilled student responses. If the teacher also fails, inspect whether similar incorrect demonstrations passed the training filter. If only the distilled student fails, inspect data coverage, template and export before attributing the problem to teacher quality.

Keep one controlled fault per run. For contamination, identify the underlying source task rather than searching only for byte-identical strings. For a tokeniser mismatch, compare the actual mapping required by the method. For settings drift, compare complete request records, including answer limits and reasoning behaviour.

After repair, regenerate or refilter into a new dataset revision and preserve the original rejected examples. Re-evaluate all three models on the frozen task set. Report any regression that remains outside the repaired category. Finish with a pipeline check that would have caught the problem before training, such as verifying teacher labels, grouping source tasks or testing one exported response. The challenge’s output is a causal diagnosis and evidence of the repair, not a new student whose overall average happens to look better.

Validation

You are done when all of the following are true:

• both claims from task 1 are written down in a falsifiable form, with a measurement behind each;
• diagnosis-healthy.md exists and you have read it end to end;
• you reproduced at least three of the four faults and have a diagnosis report for each;
• for each reproduced fault you can name the single line of evidence that identifies it;
• the tokeniser check has printed both compatible and NOT compatible for two different pairs;
• the two temperature runs produced measurably different rejection tables;
• the recording sheet below has its rows filled in;
• one fault has been fixed and the fix proved with a second measurement of the same kind.

Expected outcome

Four faults, four pieces of evidence, and one fixed. The sheet below is the deliverable.

Pending validationOne symptom, four faults, one fix — your recording sheet

State	Evidence line that identifies it	Checks passed, evaluation set	Checks passed, held-out prompts
Healthy baseline	-	-	-
Fault 1: unverified teacher	-	-	-
Fault 2: contaminated evaluation	-	-	-
Fault 3: tokeniser mismatch	-	-	-
Fault 4: unrecorded settings	-	-	-
After the fix	-	-	-

your machine: track, chip and memory, your operating system and version · llama.cpp for serving; TRL for training the versions from your run log, per row · your teacher and student pair, as served · 1,024 tokens of context · the date you ran it

Empty on purpose. The two score columns are the point: the fault in the title is a rise in the first column and a fall in the second, and a fix that lowers the first column while raising the second is a success.

Troubleshooting

The diagnosis reports every check as “not checked”. You passed only the run log. Each check needs its own input: --raw for the teacher check, --train and --tasks for contamination, --teacher and --student for the tokenisers. What is missing is reported rather than guessed, which is deliberate.

The contamination check finds nothing on data you know is contaminated. Look at the n-gram size and the threshold. A short evaluation prompt has few 13-grams, so containment can be low even for a near-copy; try --n 8. Exact matching is unaffected and is the check to trust first.

The tokeniser check fails to download a gated model. Llama models require an accepted licence on the Hub and a logged-in hf session. Substitute any non-Qwen instruction model you already have.

The temperature comparison shows no difference. Check that the greedy run really was greedy: some servers apply a minimum temperature, and top_p below 1 still constrains a temperature-0 sample. Read the settings block written on each line of the raw file rather than trusting the command you typed.

The verdicts all say “no evidence” and the student is still bad. The four faults here are the ones that change a result by a lot. If none is present, you are in the territory of the ordinary training questions from Part 13’s challenge: the learning rate, the number of epochs, the chat template, and whether the model you evaluated is the model you trained.

Cleanup

Keep every diagnosis report and the recording sheet; they are the deliverable. The broken datasets are regenerable and take real disk.

RunnableAll tracks

remove the deliberately broken artefacts

rm -rf broken-1 broken-2 broken-4a broken-4b
rm -f raw/teacher-uncategorised.jsonl raw/contaminated.jsonl raw/mixed.jsonl
rm -f raw/t00.jsonl raw/t07.jsonl seeds/contaminated.jsonl seeds/slice.jsonl
rm -rf runs/mismatch-demo

⚠Warning

Those commands delete directories you created during this page and nothing else. Check the names against what is actually in your working directory before running them: broken-2 is a deliberately contaminated dataset and deleting it is the point, but a typo that removes data deletes the training set your run log’s hash refers to.

What you learned

• Evidence, then theory, then proof. Four file reads answer a question that hyperparameter changes will not.
• Contamination is checked first, because it invalidates the measurement that started the investigation.
• A student cannot be better than its data, and it cannot tell good examples from bad. It learns every one equally, which is why a verifier is not optional wherever one is possible.
• A falling loss is not evidence that a run is working. The tokeniser-mismatch reproduction is the clearest demonstration of that in this course.
• Sampling settings are part of the dataset. A greedy teacher produces a narrow dataset, and the filter’s duplicate count is where you see it.
• A run without a record is a run you cannot compare. The stage list is a check, not bookkeeping.

Record in the notebook: the two claims and their measurements; the healthy diagnosis; one line of identifying evidence per fault; the rejection tables from the two temperature runs; and the before and after of the fix you made, on both the evaluation set and the held-out prompts.

Check your understanding

Question 1. Why does the procedure check the evaluation set for contamination before investigating the model?

• Because it is the fastest check
• Because contamination invalidates the score that started the investigation, so every conclusion drawn from that score is premature until it is ruled out
• Because it is the most common fault
• Because the model cannot be inspected first

Show the answer and why

Answer: Because contamination invalidates the score that started the investigation, so every conclusion drawn from that score is premature until it is ruled out

The symptom is "the score went up and the model got worse". If the training data contained the evaluation questions, the first half of that sentence is not a fact about the model at all, and diagnosing a model with an untrustworthy measurement wastes the whole session.

Question 2. In the tokeniser-mismatch reproduction, the training loss falls steadily. What does that tell you?

• That the distillation is working
• Nothing about whether the run is correct: the loss is well defined for a mismatched pair and falls as the student learns to match probabilities against the wrong tokens
• That the learning rate is too high
• That the teacher is too large

Show the answer and why

Answer: Nothing about whether the run is correct: the loss is well defined for a mismatched pair and falls as the student learns to match probabilities against the wrong tokens

This is the fault's defining property and the reason the check exists. TRL documents that the failure is silent when the teacher's vocabulary is no larger than the student's, and a falling loss is exactly the reassurance that keeps people from looking.

Question 3. You generate your teacher dataset at temperature 0 for reproducibility. What does the filter report look like, and why?

• Cleaner, because greedy answers are more accurate
• Heavier on near-duplicate and duplicate-completion rejections, because a deterministic teacher answers similar prompts in similar words, leaving a narrow dataset
• Identical to a sampled run
• Empty, because temperature 0 is rejected

Show the answer and why

Answer: Heavier on near-duplicate and duplicate-completion rejections, because a deterministic teacher answers similar prompts in similar words, leaving a narrow dataset

Reproducibility of generation is not the same as usefulness of the dataset. The Qwen3 cards separately advise against greedy decoding in thinking mode. Fix the seed and record the settings for reproducibility; do not fix the temperature at zero.

Question 4. Which of these are evidence for "teacher errors propagated"? Select all that apply.

• Refusal-category prompts answered with an invented hostname or port
• Verifiably wrong final answers on problems with a computed answer
• Empty completions in the raw teacher file
• A training loss that plateaus

Show the answer and why

Answer: Refusal-category prompts answered with an invented hostname or port, Verifiably wrong final answers on problems with a computed answer, Empty completions in the raw teacher file

The first three are properties of the data, checkable without running any model, and each teaches the student something specific and wrong. A plateauing loss is an ordinary training observation and says nothing about whether the data was any good.

Question 5. You fix the contamination and the reported gain drops by half. What do you report?

• The original number, since the model did not change
• The smaller number, with a note that the first measurement was contaminated and how it was found
• Neither; the experiment is void
• An average of the two

Show the answer and why

Answer: The smaller number, with a note that the first measurement was contaminated and how it was found

The smaller number is the real one and the note is what makes it trustworthy. A fix that makes your headline result worse is still a fix; publishing the larger number after discovering why it was larger is the actual failure.

Check my answersClear