Challenge: The Fine-Tune That Got Worse

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track reproductions, the versions they were reproduced with and the observed evidence belong here once the validation pass has run this page on real machines.

Objective

Somebody, quite possibly you next week, says: “the training loss went from two point one to nought point one two, the run finished cleanly, and the fine-tuned model is worse than the one I started from.” By the end of this page you will have a procedure that answers that in about five minutes, and you will have run it on four faults you introduced deliberately, so that you recognise the symptoms when the fault is not yours.

The deliverable is not a fixed model. It is a written diagnosis for each fault: the evidence you collected, the fault it pointed at, the single change you made, and the second measurement that proves the change worked. Evidence, then theory, then proof. That order is what separates fixing something from changing things until the symptom moves.

⌂Single-machine path

Single-machine on every track. The diagnostic script reads files and changes nothing; the reproductions re-run the lab’s training script with one setting altered.

The procedure

A fine-tune that scored worse: what to do, in order

1. 1Turn the impression into two numbersScore the base and the fine-tune on the same tasks at the same settings. "Worse" is not a finding; a pair of numbers on a task set is.
2. 2Establish the noise floorRun the base model twice. Any difference smaller than the gap between those two runs is not a difference, and this step ends about a third of these investigations.
3. 3Compare the two chat templatesTraining-side against serving-side, by hash. A mismatch makes everything else moot and it is one command to check.
4. 4Read the two loss curvesWhich epoch was best, and how far the final evaluation loss is from it. This separates overfitting from everything else.
5. 5Check the evaluation set against the training dataExact matches first, near-duplicates second. A contaminated set makes a good number as untrustworthy as a bad one.
6. 6Check what the adapter was merged intoWhich base, at what precision, with which tokeniser. The artefact records all three.
7. 7Change exactly one thing, then measure againSame command, same tasks, same settings. Two changes at once means you will not know which one worked.

Each step is cheap and each one eliminates a large fraction of the possibilities, so by the fourth you are usually looking at one candidate rather than four.

The short list of faults

Almost every fine-tune that scored worse is one of these four, or is no worse at all and inside the noise. Learn the list and you have learned the diagnosis.

Fault	What the evidence looks like
Chat template mismatch	The fine-tune is worse at everything, including tasks unrelated to what you trained, and its answers often start mid-thought or repeat the prompt. The training-side and serving-side template hashes differ.
Learning rate and overfitting	The training loss is very low and the evaluation loss turned upward. The best epoch is the first. The model reproduces training answers verbatim on inputs that resemble training inputs.
Contaminated evaluation	The fine-tune’s score went up implausibly far, or up on a set that should have been unrelated. The overlap check finds exact matches or high n-gram containment.
Wrong base or wrong precision at merge	The adapter behaves correctly when attached at serving time and badly once merged. The merge comparison does not match. The merged model’s dtype or its base is not what the adapter records.

Notice what is not on the list. The rank, the alpha, the number of target modules and the dataset size change results by amounts you have to measure carefully to see. The four above change them by enough that you notice without measuring, which is exactly why they are worth eliminating first.

?Reality check

There is a fifth outcome and it is common: the fine-tune is not worse, and the difference is inside the run-to-run variation. The claim “it got worse” is a claim, and the second step of the procedure tests it. People skip that step because the conclusion feels obvious, and then spend an hour finding a fault that was never there.

Requirements

Forty-five minutes, all of it attended. You need the lab from this part finished: its dataset, its adapter, its exported model and its labbook.md, plus your own Part 10 task file and harness. No new downloads. The memory floor is 12 GB, the same as the lab, because the reproductions re-run the same training configuration with one setting changed.

Track S — NVIDIA DGX Spark

All four faults reproduce here with the lab’s own scripts. With 128 GB you can hold the base model and the fine-tune at once, which makes the before-and-after measurement a matter of two ports rather than two restarts.

Track X — AMD Ryzen AI Max+ 395

All four faults reproduce here. The re-training in faults two and three is the lab’s run again with one changed setting, so budget the same wall-clock you measured there, twice.

If the lab’s run took longer than you want to repeat, lower --epochs to 2 for the healthy baseline and keep the fault runs at the same value: what matters is that the two runs differ in exactly one setting, not what that setting’s absolute value is.

Track M — Apple silicon

All four faults reproduce here, with two changes of tool. Retraining uses train-lora-mlx.sh with LEARNING_RATE and the iteration count as the knobs, and the merge fault is reproduced with mlx_lm.fuse against a different base rather than with merge-adapter.py.

diagnose-fine-tune.py works unchanged on this track for the run record, the overlap check and the sampling settings. The adapter and template sections read PEFT-shaped files, so on an MLX adapter directory they will report what is missing, which is itself accurate: an mlx-lm adapter does not carry a tokeniser, so the training-time template is the base model’s.

Track N — NVIDIA desktop or laptop

All four faults reproduce here. On a 12 GB card use the 1.7-billion-parameter base from the lab for every reproduction; the faults are properties of the procedure rather than of the model size, and a smaller model makes the four re-runs quick.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-13-supervised-fine-tuning"
cd "$LAB_DIR"
pwd
test -f "diagnose-fine-tune.py"

Expected result: pwd ends in part-13-supervised-fine-tuning and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Establish the healthy baseline, and the noise floor

You cannot recognise a fault without knowing what correct looks like on this machine, and you cannot call anything a regression without knowing how much a number moves on its own.

RunnableAll tracks

the base model, twice, and the fine-tune once

python3 evaluate-against-base.py \
    --harness-dir ~/eval \
    --base-url http://127.0.0.1:8080/v1 \
    --base-model qwen3-1.7b \
    --tuned-model qwen3-1.7b \
    --quant Q4_K_M --engine llama.cpp --engine-version v0.4.0 \
    --tasks format-tasks.json --tasks my-tasks.json \
    --out-dir noise-floor --labbook labbook.md

That command deliberately compares the base model with itself. The “change” column is then the run-to-run variation at these settings, and it is the number every later comparison is judged against. Write it down.

Then run the real comparison, with the fine-tune from the lab, and keep the output. That is your healthy report.

2. Collect the evidence with one command

The script gathers everything the procedure asks for into one report: the run record, the adapter’s configuration, the two chat templates and whether they match, the merged model’s precision, the overlap between the training data and the evaluation set, and the sampling settings. It changes nothing.

RunnableAll tracks

diagnose-fine-tune.py

"""Collect the evidence for a fine-tune that scored worse than the model it started from.

Purpose: the diagnostic half of Part 13's challenge. Reads the run record from the lab notebook,
         the adapter configuration, the chat template the training tokeniser carries and the one
         the serving side carries, the merged model's precision, and the overlap between the
         training data and the evaluation set, then writes one report with a verdict line per
         fault. It changes nothing and needs no accelerator.
Platform: all (standard library only; it reads files and writes a report)
Minimum memory: 8 GB, and far less in practice
Assumes: Python 3.10 or newer. Whatever exists is read; whatever is missing is reported as
         missing rather than guessed, because "not recorded" is itself evidence.

Usage: python3 diagnose-fine-tune.py --adapter runs/format-qwen3-1.7b \
           --labbook labbook.md --train data/train.jsonl --tasks my-tasks.json \
           --serving-tokenizer models/format-qwen3-1.7b-merged --report diagnosis.md

       python3 diagnose-fine-tune.py --adapter runs/domain-qwen3-8b --labbook labbook.md
       # the minimum: the run record and the adapter, with everything else reported as absent

The four faults this collects evidence for, in the order the challenge page works through them:
  1. chat template mismatch between training and serving
  2. learning rate and overfitting
  3. evaluation contaminated by training data
  4. adapter merged into the wrong base or the wrong precision
"""
from __future__ import annotations

import argparse
import hashlib
import json
import re
from pathlib import Path
from typing import Any

WORD_RE = re.compile(r"[a-z0-9]+")
UNKNOWN = "not available"


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def normalise(text: str) -> list[str]:
    return WORD_RE.findall(text.lower())


def ngrams(words: list[str], n: int) -> set[tuple[str, ...]]:
    if len(words) < n:
        return {tuple(words)} if words else set()
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}


def containment(a: set[tuple[str, ...]], b: set[tuple[str, ...]]) -> float:
    return len(a & b) / len(a) if a else 0.0


def read_json(path: Path) -> dict[str, Any] | None:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        return None


def find_run_record(labbook: Path, adapter: Path, run_id: str | None) -> dict[str, Any] | None:
    """The last training record for this adapter, or the one named by --run-id.

    The run log is JSON lines mixed into a Markdown file, so anything that does not parse is
    prose and is skipped.
    """
    if not labbook.is_file():
        return None
    matches = []
    for line in labbook.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line.startswith("{"):
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            continue
        if run_id:
            if record.get("run_id") == run_id:
                return record
            continue
        if "hyperparameters" not in record:
            continue
        output_dir = str(record.get("hyperparameters", {}).get("output_dir", ""))
        if output_dir and Path(output_dir).name == adapter.name:
            matches.append(record)
        elif not output_dir and str(record.get("lab", "")).startswith("part-13/train"):
            matches.append(record)
    return matches[-1] if matches else None


def chat_template_of(source: Path) -> tuple[str | None, str]:
    """The chat template a tokeniser directory carries, and where it came from.

    Newer tokenisers keep it in chat_template.jinja; older ones keep it inside
    tokenizer_config.json. Both are checked, because a directory saved by one version and read
    by another is exactly the situation this script exists to diagnose.
    """
    if source.is_file():
        return source.read_text(encoding="utf-8"), str(source)
    jinja = source / "chat_template.jinja"
    if jinja.is_file():
        return jinja.read_text(encoding="utf-8"), str(jinja)
    config = read_json(source / "tokenizer_config.json")
    if config and isinstance(config.get("chat_template"), str):
        return config["chat_template"], str(source / "tokenizer_config.json")
    return None, str(source)


def template_shape(template: str) -> dict[str, Any]:
    """A few properties that differ between templates, so a difference can be described."""
    special = sorted(set(re.findall(r"<\|[a-zA-Z0-9_]+\|>", template)))
    return {
        "sha256": sha256_text(template),
        "characters": len(template),
        "special_tokens": special[:12],
        "mentions_system": "system" in template,
        "mentions_generation_keyword": "generation" in template,
    }


def overlap_report(train_path: Path, tasks_path: Path, n: int, threshold: float) -> dict[str, Any]:
    """Exact and near-duplicate overlap between the training file and the evaluation set."""
    spec = read_json(tasks_path)
    if spec is None:
        return {"status": f"could not read {tasks_path}"}
    tasks = spec.get("tasks") if isinstance(spec, dict) else spec
    if not isinstance(tasks, list):
        return {"status": f"{tasks_path} has no tasks list"}

    task_entries = []
    for index, task in enumerate(tasks):
        text = "\n".join(str(task[k]) for k in ("prompt", "reference") if isinstance(task.get(k), str))
        words = normalise(text)
        task_entries.append({
            "id": str(task.get("id", f"task-{index:03d}")),
            "hash": sha256_text(" ".join(words)),
            "grams": ngrams(words, n),
        })
    task_hashes = {entry["hash"]: entry["id"] for entry in task_entries}

    exact, near, examples = [], [], 0
    try:
        handle = train_path.open("r", encoding="utf-8")
    except OSError:
        return {"status": f"could not read {train_path}"}
    with handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                record = json.loads(line)
            except json.JSONDecodeError:
                continue
            examples += 1
            parts: list[str] = []
            for key in ("prompt", "completion", "messages"):
                value = record.get(key)
                if isinstance(value, str):
                    parts.append(value)
                elif isinstance(value, list):
                    for message in value:
                        if isinstance(message, dict) and isinstance(message.get("content"), str):
                            parts.append(message["content"])
            if isinstance(record.get("text"), str):
                parts.append(record["text"])
            words = normalise("\n".join(parts))
            if not words:
                continue
            digest = sha256_text(" ".join(words))
            if digest in task_hashes:
                exact.append({"line": number, "task": task_hashes[digest]})
                continue
            grams = ngrams(words, n)
            best_id, best = None, 0.0
            for entry in task_entries:
                score = max(containment(grams, entry["grams"]), containment(entry["grams"], grams))
                if score > best:
                    best_id, best = entry["id"], score
            if best >= threshold:
                near.append({"line": number, "task": best_id, "containment": round(best, 3)})
    return {"status": "checked", "training_examples": examples, "evaluation_tasks": len(task_entries),
            "exact_matches": exact, "near_duplicates": near, "n": n, "threshold": threshold}


def section(title: str) -> str:
    return f"\n## {title}\n\n"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--adapter", required=True, help="the adapter directory a run saved")
    parser.add_argument("--labbook", default="labbook.md")
    parser.add_argument("--run-id", default=None, help="a specific run record instead of the last")
    parser.add_argument("--train", default=None, help="the training file, for the overlap check")
    parser.add_argument("--tasks", default=None, help="the evaluation task file it is checked against")
    parser.add_argument("--serving-tokenizer", default=None,
                        help="directory or .jinja file whose chat template the server is using; "
                             "usually the merged model directory")
    parser.add_argument("--merged", default=None,
                        help="the merged model directory, for the precision and base checks")
    parser.add_argument("--n", type=int, default=13)
    parser.add_argument("--threshold", type=float, default=0.5)
    parser.add_argument("--report", default="diagnosis.md")
    args = parser.parse_args()

    adapter = Path(args.adapter)
    lines = [f"# Fine-tune diagnosis: {adapter}\n"]
    verdicts: list[str] = []

    # ---------------------------------------------------------------- the run record
    lines.append(section("1. The run"))
    record = find_run_record(Path(args.labbook), adapter, args.run_id)
    if record is None:
        lines.append(f"No run record found in `{args.labbook}` for this adapter. "
                     "Without it, the hyperparameters and the losses are unknown and every "
                     "conclusion below is weaker.\n")
        verdicts.append("RUN RECORD: missing. Record every run; a result you cannot reproduce "
                        "is not a result.")
    else:
        hyper = record.get("hyperparameters", {})
        losses = record.get("losses", {})
        lines.append("| Field | Value |\n| --- | --- |\n")
        for key in ("run_id", "date", "model", "seed", "config_commit"):
            lines.append(f"| {key} | `{record.get(key, UNKNOWN)}` |\n")
        for key in ("method", "rank", "alpha", "target_modules", "learning_rate", "epochs",
                    "effective_batch", "max_length", "precision", "compute_precision",
                    "load_in_4bit", "completion_only_loss"):
            if key in hyper:
                lines.append(f"| {key} | `{hyper[key]}` |\n")
        dataset = record.get("dataset", {})
        lines.append(f"| dataset | `{dataset.get('path', UNKNOWN)}` |\n")
        lines.append(f"| dataset sha256 | `{dataset.get('sha256', UNKNOWN)}` |\n")
        lines.append(f"| train / validation examples | "
                     f"{dataset.get('train_examples', '?')} / "
                     f"{dataset.get('validation_examples', '?')} |\n")
        lines.append("\n### Losses\n\n```json\n" + json.dumps(losses, indent=2) + "\n```\n")

        # ------------------------------------------------- fault 2: rate and overfitting
        best_epoch = losses.get("best_epoch")
        first, final = losses.get("first_train_loss"), losses.get("final_train_loss")
        final_eval, best_eval = losses.get("final_eval_loss"), losses.get("best_eval_loss")
        if best_epoch is not None and best_epoch <= 1:
            verdicts.append("OVERFITTING: the best epoch was the first, so later epochs made the "
                            "evaluation loss worse. Fewer epochs, a lower learning rate, a lower "
                            "rank or more data, in that order of cost.")
        if final_eval is not None and best_eval is not None and final_eval > best_eval * 1.05:
            verdicts.append("OVERFITTING: the final evaluation loss is meaningfully above the best "
                            "one, so the curves diverged. Confirm load_best_model_at_end was set, "
                            "or the saved adapter is not the best checkpoint.")
        if first is not None and final is not None and abs(first - final) < 0.01:
            verdicts.append("LEARNING RATE: the training loss barely moved. Either the rate is too "
                            "low or the adapter attached to nothing; check the trainable-parameter "
                            "count from the run.")
        rate = hyper.get("learning_rate")
        if isinstance(rate, (int, float)) and rate >= 1e-3:
            verdicts.append(f"LEARNING RATE: {rate} is high for an adapter run, where roughly 1e-4 "
                            "is the documented starting point. A loss that spiked or went to nan "
                            "points here first.")

    # ---------------------------------------------------------------- the adapter
    lines.append(section("2. The adapter"))
    adapter_config = read_json(adapter / "adapter_config.json")
    recorded_base = None
    if adapter_config is None:
        lines.append(f"`{adapter / 'adapter_config.json'}` is missing, so this is not a PEFT "
                     "adapter directory.\n")
        verdicts.append("ADAPTER: no adapter_config.json. Point --adapter at what the run saved.")
    else:
        recorded_base = adapter_config.get("base_model_name_or_path")
        lines.append("| Field | Value |\n| --- | --- |\n")
        for key in ("base_model_name_or_path", "peft_type", "task_type", "r", "lora_alpha",
                    "lora_dropout", "target_modules", "use_dora", "use_rslora"):
            if key in adapter_config:
                lines.append(f"| {key} | `{adapter_config[key]}` |\n")
        if record is not None and record.get("model") and recorded_base:
            if str(record["model"]) != str(recorded_base):
                verdicts.append(f"WRONG BASE: the run record says the base was "
                                f"`{record['model']}` and the adapter records "
                                f"`{recorded_base}`. One of them is not what you think.")

    # ---------------------------------------------------------------- chat templates
    lines.append(section("3. Chat template, training against serving"))
    train_template, train_source = chat_template_of(adapter)
    serve_template, serve_source = (None, UNKNOWN)
    if args.serving_tokenizer:
        serve_template, serve_source = chat_template_of(Path(args.serving_tokenizer))

    if train_template is None:
        lines.append(f"No chat template found under `{adapter}`. The adapter was saved without a "
                     "tokeniser, so the template used in training cannot be recovered from the "
                     "artefact.\n")
        verdicts.append("CHAT TEMPLATE: the adapter carries no tokeniser, so the training-time "
                        "template is unknown. Save the tokeniser with every run.")
    else:
        lines.append(f"Training-side template from `{train_source}`:\n\n```json\n"
                     + json.dumps(template_shape(train_template), indent=2) + "\n```\n")
    if serve_template is None and args.serving_tokenizer:
        lines.append(f"No chat template found under `{serve_source}`.\n")
        verdicts.append("CHAT TEMPLATE: the serving side has no template file where one was "
                        "expected, so the server is using whatever the model file carries.")
    elif serve_template is not None:
        lines.append(f"Serving-side template from `{serve_source}`:\n\n```json\n"
                     + json.dumps(template_shape(serve_template), indent=2) + "\n```\n")

    if train_template and serve_template:
        if sha256_text(train_template) == sha256_text(serve_template):
            lines.append("\n**The two templates are byte-identical.** This fault is eliminated.\n")
        else:
            lines.append("\n**The two templates differ.** Every training example was formatted "
                         "with one and every request is formatted with the other.\n")
            verdicts.append("CHAT TEMPLATE MISMATCH: the training-side and serving-side templates "
                            "have different hashes. This is the first fault to fix, and it usually "
                            "explains a fine-tune that got worse at everything at once.")
    elif not args.serving_tokenizer:
        lines.append("\nNo `--serving-tokenizer` given, so the comparison was not made. Point it "
                     "at the merged model directory the server is loading.\n")

    # ---------------------------------------------------------------- precision and merge
    lines.append(section("4. Precision and the merge"))
    if args.merged:
        merged_config = read_json(Path(args.merged) / "config.json")
        if merged_config is None:
            lines.append(f"`{Path(args.merged) / 'config.json'}` is missing.\n")
        else:
            dtype = merged_config.get("dtype") or merged_config.get("torch_dtype") or UNKNOWN
            quant = merged_config.get("quantization_config")
            lines.append(f"| Field | Value |\n| --- | --- |\n| merged dtype | `{dtype}` |\n")
            lines.append(f"| merged base architecture | `{merged_config.get('model_type', UNKNOWN)}` |\n")
            lines.append(f"| quantization_config present | `{bool(quant)}` |\n")
            if quant:
                verdicts.append("WRONG PRECISION: the merged model still carries a "
                                "quantization_config, so the adapter was merged into a quantised "
                                "base. Merge into the base at bfloat16 and quantise afterwards.")
            if isinstance(dtype, str) and dtype.lower() in {"float16", "torch.float16"}:
                verdicts.append("PRECISION: the merged model is float16 while adapters here are "
                                "trained in bfloat16. The two have different ranges; convert at "
                                "bfloat16 unless you have a reason not to.")
    else:
        lines.append("No `--merged` directory given, so the precision of the exported model was "
                     "not checked.\n")

    if record is not None and record.get("hyperparameters", {}).get("load_in_4bit") and args.merged:
        lines.append("\nThis run trained against a 4-bit base. The merge must still be into a "
                     "bfloat16 copy of that base, never into the quantised one.\n")

    # ---------------------------------------------------------------- contamination
    lines.append(section("5. Overlap between training data and the evaluation set"))
    if args.train and args.tasks:
        overlap = overlap_report(Path(args.train), Path(args.tasks), args.n, args.threshold)
        lines.append("```json\n" + json.dumps(
            {k: v for k, v in overlap.items() if k != "status"} | {"status": overlap["status"]},
            indent=2, default=str) + "\n```\n")
        if overlap.get("exact_matches"):
            verdicts.append(f"CONTAMINATED EVALUATION: {len(overlap['exact_matches'])} training "
                            "example(s) are byte-identical to evaluation tasks after "
                            "normalisation. Any score measured on this set is not a measurement.")
        if overlap.get("near_duplicates"):
            verdicts.append(f"POSSIBLE CONTAMINATION: {len(overlap['near_duplicates'])} "
                            "near-duplicate pair(s) above the threshold. Read them; some will be "
                            "shared stock phrases and some will be paraphrased evaluation tasks.")
        if overlap.get("status") == "checked" and not overlap.get("exact_matches") \
                and not overlap.get("near_duplicates"):
            lines.append("\n**No overlap found at this n-gram size and threshold.** This fault is "
                         "eliminated for this pair of files.\n")
    else:
        lines.append("No `--train` and `--tasks` given, so the overlap check was not run. It is "
                     "the cheapest of the four checks and the one that invalidates the others.\n")

    # ---------------------------------------------------------------- settings
    lines.append(section("6. Sampling settings the evaluation used"))
    if args.tasks:
        spec = read_json(Path(args.tasks))
        settings = (spec or {}).get("settings") if isinstance(spec, dict) else None
        if settings:
            lines.append("```json\n" + json.dumps(settings, indent=2) + "\n```\n")
            lines.append("\nBoth models must be scored at these settings, at the same "
                         "quantisation, with the same system prompt. A comparison across a change "
                         "in any of them is measuring more than the fine-tune.\n")
        else:
            lines.append(f"`{args.tasks}` has no settings block, so the run used whatever the "
                         "harness defaults are. Record them.\n")
    else:
        lines.append("No `--tasks` given.\n")

    # ---------------------------------------------------------------- verdicts
    lines.append(section("Verdict"))
    if verdicts:
        for item in verdicts:
            lines.append(f"- **{item}**\n")
        lines.append("\nFix one thing, then measure again with the same command. Two changes at "
                     "once means you will not know which one worked.\n")
    else:
        lines.append("Nothing in the collected evidence points at one of the four faults. That is "
                     "a real result: the remaining candidates are that the fine-tune genuinely did "
                     "not help on these tasks, that the change is inside the run-to-run noise, or "
                     "that the fault is somewhere this script does not look. Run the base model "
                     "twice to establish the noise floor before concluding anything.\n")

    report = "".join(lines)
    Path(args.report).write_text(report, encoding="utf-8")
    print(report)
    print(f"\nwritten to {args.report}")


if __name__ == "__main__":
    main()

Download diagnose-fine-tune.py411 lines

RunnableAll tracks

collect the evidence for the healthy run

python3 diagnose-fine-tune.py \
    --adapter runs/format-qwen3-1.7b \
    --labbook labbook.md \
    --train data/train.jsonl \
    --tasks my-tasks.json \
    --serving-tokenizer ~/models/format-qwen3-1.7b-merged \
    --merged ~/models/format-qwen3-1.7b-merged \
    --report healthy-diagnosis.md

Output — what you should see

## 3. Chat template, training against serving

**The two templates are byte-identical.** This fault is eliminated.

## 5. Overlap between training data and the evaluation set

**No overlap found at this n-gram size and threshold.** This fault is eliminated for this
pair of files.

## Verdict

Nothing in the collected evidence points at one of the four faults.

Read healthy-diagnosis.md end to end now, while everything is working. Knowing what a healthy report looks like is what makes an unhealthy one legible.

3. Fault one: the chat template mismatch

The claim under test: a model trained with one chat format and served with another will be worse at everything, not just at the thing you trained.

The template lives in the tokeniser and decides how a list of messages becomes a string of tokens. Transformers’ documentation is blunt about what happens when it is wrong: two models fine-tuned from the same base can use entirely different control tokens, and “with the wrong control tokens, these models would have drastically worse performance.”

Track S — NVIDIA DGX Spark

Reproduce it by serving your merged fine-tune with a template from a different family.

RunnableTrack S · DGX Spark

write a template from the wrong family, and serve with it

printf '%s' '{% for m in messages %}[INST] {{ m.content }} [/INST]{% endfor %}' > wrong-template.jinja

llama-server \
    --model ~/models/format-qwen3-1.7b-Q4_K_M.gguf \
    --chat-template-file wrong-template.jinja \
    --alias qwen3-1.7b-wrongtemplate \
    --ctx-size 8192 \
    --host 127.0.0.1 \
    --port 8082

Track X — AMD Ryzen AI Max+ 395

The same reproduction as Track S, on your Vulkan or ROCm build. --chat-template-file is documented as a “custom jinja chat template file (default: template taken from model’s metadata)”, so this replaces the correct template with a deliberately wrong one and changes nothing else.

RunnableTrack X · Ryzen AI Max+

serve the fine-tune with the wrong template

printf '%s' '{% for m in messages %}[INST] {{ m.content }} [/INST]{% endfor %}' > wrong-template.jinja

llama-server \
    --model ~/models/format-qwen3-1.7b-Q4_K_M.gguf \
    --chat-template-file wrong-template.jinja \
    --alias qwen3-1.7b-wrongtemplate \
    --ctx-size 8192 \
    --host 127.0.0.1 \
    --port 8082

Track M — Apple silicon

Two routes. If you exported to GGUF, use the --chat-template-file reproduction from the other tracks. If you are serving the fused MLX model, the equivalent is to generate with a raw prompt that skips the template entirely and compare it with the templated one.

RunnableTrack M · Apple silicon

the same prompt, with and without the model's own format

mlx_lm.generate \
    --model models/format-mlx-fused \
    --max-tokens 96 --temp 0 --seed 0 \
    --prompt "Triage this report. The gateway is unreachable."

The point on this track is the comparison against a request that goes through the chat template, which the harness does. An answer that continues the prompt rather than replying to it is the signature.

Track N — NVIDIA desktop or laptop

The same reproduction as Track S.

RunnableTrack N · NVIDIA GPU

serve the fine-tune with the wrong template

printf '%s' '{% for m in messages %}[INST] {{ m.content }} [/INST]{% endfor %}' > wrong-template.jinja

llama-server \
    --model ~/models/format-qwen3-1.7b-Q4_K_M.gguf \
    --chat-template-file wrong-template.jinja \
    --alias qwen3-1.7b-wrongtemplate \
    --ctx-size 8192 \
    --host 127.0.0.1 \
    --port 8082

Score it against the base with the same command you used in task 1, pointing --tuned-model at the broken alias. Then run the diagnostic with --serving-tokenizer pointing at the wrong template file.

RunnableAll tracks

collect the evidence for the broken case

python3 diagnose-fine-tune.py \
    --adapter runs/format-qwen3-1.7b \
    --labbook labbook.md \
    --serving-tokenizer wrong-template.jinja \
    --report template-diagnosis.md

The evidence: the two template hashes differ, the special-token lists differ, and the score fell on both task files rather than on one. The fix: serve with the template that came with the adapter, which is what --jinja and the model’s own metadata do by default. The proof: the hashes match again and the score returns to the healthy report’s value.

✕Common mistake

Assuming a template mismatch is a merge problem. The symptom, “the merged model is worse than the adapter was”, points at the merge. The cause is usually that the conversion picked up the base model’s tokeniser rather than the one saved with the adapter. Check the hashes before you re-merge anything: the merge arithmetic is rarely the problem when the base and the precision are right.

4. Fault two: the learning rate and overfitting

The claim under test: a lower training loss is a better model.

Reproduce it by training the same data with a learning rate an order of magnitude too high, more epochs than the data supports, and early stopping switched off so that nothing rescues you.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

the same data, deliberately over-trained

python3 train-lora.py \
    --model Qwen/Qwen3-1.7B \
    --data-dir data \
    --output-dir runs/format-overfit \
    --epochs 8 \
    --lr 2e-3 \
    --early-stopping-patience 0 \
    --labbook labbook.md \
    --notes "deliberate overfit: lr 2e-3, 8 epochs, no early stopping"

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

the same data, deliberately over-trained

python3 train-lora.py \
    --model Qwen/Qwen3-1.7B \
    --data-dir data \
    --output-dir runs/format-overfit \
    --epochs 8 \
    --lr 2e-3 \
    --early-stopping-patience 0 \
    --gradient-checkpointing \
    --labbook labbook.md \
    --notes "deliberate overfit: lr 2e-3, 8 epochs, no early stopping"

Track M — Apple silicon

RunnableTrack M · Apple silicon

the same data, deliberately over-trained

LEARNING_RATE=2e-3 \
ADAPTERS=runs/format-mlx-overfit \
SKIP_FUSE=1 \
bash train-lora-mlx.sh mlx-community/Qwen3-1.7B-bf16 2400

Four times the iterations and twenty times the learning rate. Read the validation loss that mlx_lm.lora prints during training rather than only the final test output: the turn upward is the evidence, and it happens well before the run ends.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

the same data, deliberately over-trained

python3 train-lora.py \
    --model Qwen/Qwen3-1.7B \
    --data-dir data \
    --output-dir runs/format-overfit \
    --epochs 8 \
    --lr 2e-3 \
    --early-stopping-patience 0 \
    --gradient-checkpointing \
    --labbook labbook.md \
    --notes "deliberate overfit: lr 2e-3, 8 epochs, no early stopping"

The evidence: the training loss is far lower than the healthy run’s and the evaluation loss is higher; best_epoch is 1; the final evaluation loss is well above the best one. The diagnostic prints all three as verdict lines. Give the model an input close to a training example and it returns that example’s answer, including details from the original input.

The fix: one thing. Put the epochs back and leave the learning rate high, or lower the learning rate and leave the epochs high, but not both, because you want to know which one mattered here.

The proof: the same evaluation command, the same tasks, and a score back at the healthy value, and a best_epoch that is no longer the first.

∴Research note

This is also where catastrophic forgetting shows up most clearly. The study cited in this part’s first lesson reports forgetting “generally observed in LLMs ranging from 1b to 7b parameters”, with severity increasing with size across that range. Score the over-trained model on your own Part 10 set as well as the format set: the format score may still be high while three other categories have collapsed, and that pattern is the one to recognise.

5. Fault three: the contaminated evaluation

The claim under test: the score went up, therefore the model got better.

This is the fault that produces a good number, which is why it is the most dangerous of the four. Reproduce it by moving eight of your evaluation tasks into the training data.

RunnableAll tracks

contaminate the training set on purpose

python3 - <<'PY'
import json, pathlib
tasks = json.loads(pathlib.Path("my-tasks.json").read_text())["tasks"][:8]
with pathlib.Path("data/train-contaminated.jsonl").open("w", encoding="utf-8") as out:
    for line in pathlib.Path("data/train.jsonl").read_text().splitlines():
        out.write(line + "\n")
    for task in tasks:
        out.write(json.dumps({
            "prompt": [{"role": "user", "content": task["prompt"]}],
            "completion": [{"role": "assistant", "content": task["reference"]}],
        }) + "\n")
print("wrote data/train-contaminated.jsonl")
PY

Train on it, score it, and watch your own Part 10 set improve for no honest reason. Then run the check that would have caught it.

RunnableAll tracks

the check that catches it

python3 decontaminate.py \
    --train data/train-contaminated.jsonl \
    --tasks my-tasks.json \
    --report contaminated-report.json \
    --strict

The evidence: exact matches between the training file and the task file, reported by name and line number, and a score rise concentrated entirely on the tasks that appear in both. The fix: remove them from the training data, not from the evaluation set; the evaluation set is the thing you must not edit to suit a result. The proof: the check passes and the score returns to what it was before, which is the uncomfortable part and the whole point.

!Important

This is why the dataset lesson insisted on a held-out split written before training and never edited afterwards. The failure is silent, it produces a number that looks like success, and the only thing standing between it and your notebook is a check that takes seconds.

6. Fault four: merged into the wrong base or the wrong precision

The claim under test: an adapter is a delta, so it can be merged into any nearby checkpoint.

Reproduce it by merging your adapter into a different checkpoint of the same family. The base checkpoint of an instruct model is the perfect candidate: same architecture, same shapes, nothing raised at load time, completely different behaviour.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

merge into the wrong checkpoint, on purpose

python3 merge-adapter.py \
    --adapter runs/format-qwen3-1.7b \
    --merged-dir ~/models/format-wrongbase-merged \
    --base-override Qwen/Qwen3-1.7B-Base \
    --compare 4

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

merge into the wrong checkpoint, on purpose

python3 merge-adapter.py \
    --adapter runs/format-qwen3-1.7b \
    --merged-dir ~/models/format-wrongbase-merged \
    --base-override Qwen/Qwen3-1.7B-Base \
    --compare 4

Track M — Apple silicon

On this track the equivalent is fusing the MLX adapter against a base it was not trained on.

RunnableTrack M · Apple silicon

fuse against a base the adapter never saw

mlx_lm.fuse \
    --model mlx-community/Qwen3-1.7B-bf16 \
    --adapter-path runs/format-mlx-adapters \
    --save-path models/format-wrongbase-fused

Use an adapter you trained against a different MLX model than the one named here, so that the fuse succeeds and the behaviour is wrong. Then generate the same four prompts through both and compare.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

merge into the wrong checkpoint, on purpose

python3 merge-adapter.py \
    --adapter runs/format-qwen3-1.7b \
    --merged-dir ~/models/format-wrongbase-merged \
    --base-override Qwen/Qwen3-1.7B-Base \
    --compare 4

The evidence: the script prints a warning naming both checkpoints, and the comparison shows prompts where the adapter-attached model and the merged model differ. Nothing errors, nothing warns at load time, and the file sizes are identical. The diagnostic’s section 4 shows the merged model’s precision and whether a quantization_config survived, which is the other half of this fault: merging into a base that is still quantised applies your update to weights that were already rounded, which PEFT warns about for the backends it supports and which the export lesson forbids.

The fix: merge into the checkpoint named in adapter_config.json, at bfloat16, and quantise afterwards. The proof: the merge comparison matches on all four prompts, and the score returns to the healthy value.

7. Diagnose from the report, not from memory

Open healthy-diagnosis.md and one of the fault reports side by side and work down the sections. For each, write one line saying what differs.

The report is arranged in the order of the procedure on purpose, so the first section that differs usually names the fault. A template hash mismatch in section 3 ends the investigation there, because nothing below it can be interpreted while the formatting is wrong. A best_epoch of 1 in section 1 points at the run rather than the export. Exact matches in section 5 invalidate every score above them.

✕Common mistake

Reading the score first and theorising from it. A score tells you that something is wrong and almost nothing about what. People who start there change the rank, then the epochs, then the dataset, and find the template mismatch by accident an hour later. Two hashes and a best_epoch answer the question directly and cost one command.

8. Look for the fault you did not introduce

With the procedure fresh, run the diagnostic once against the fine-tune you actually intend to use, with every argument filled in. This is where people find that their evaluation set has three tasks in common with their training data, or that the model they have been serving for a fortnight was merged into the wrong checkpoint.

RunnableAll tracks

a routine health check on the fine-tune you actually use

python3 diagnose-fine-tune.py \
    --adapter runs/format-qwen3-1.7b \
    --labbook labbook.md \
    --train data/train.jsonl \
    --tasks my-tasks.json \
    --serving-tokenizer ~/models/format-qwen3-1.7b-merged \
    --merged ~/models/format-qwen3-1.7b-merged \
    --threshold 0.4 \
    --report routine-diagnosis.md

Diagnose with a transformation-by-transformation comparison

Keep a known base response set and the last healthy adapter before introducing a fault. Name each fault run separately and record the single setting or artefact changed. Use the diagnostic script’s report as evidence to inspect, not as an authoritative verdict on model quality.

For a suspected template fault, compare rendered inputs before retraining. For an optimisation fault, compare loss curves and held-out task results. For contamination, inspect overlapping source tasks. For an export fault, compare the adapter-attached model, merged checkpoint and final quantised file using the same prompt. An error introduced after training cannot be fixed by a longer training run.

After applying one repair, repeat the original evaluation with the original denominator. Preserve the failed result beside the repaired one. Write an incident note containing the symptom, competing hypotheses, decisive observation and rollback or repair. If several variables changed, restore the control and repeat before claiming a cause. Finish by stating what would detect the fault earlier next time, such as a rendered-batch inspection, exact base identity check or an export smoke task.

Validation

You are done when all of the following are true:

• you can state the noise floor for your task set at your settings, from two runs of the base model;
• healthy-diagnosis.md exists and you have read it end to end;
• all four faults have been reproduced, each with its own diagnosis report;
• for each fault you can name, in one sentence each: the evidence that identified it, the single change that fixed it, and the second measurement that proved the fix;
• the recording sheet below has a complete row per fault;
• the routine health check has been run against the fine-tune you actually use, and you have either found nothing or written down what you found;
• labbook.md contains the run records for the deliberate faults, labelled as deliberate.

Expected outcome

A procedure you can run from memory, and a calibrated sense of what each fault costs.

Pending validationFour faults, one fix each — your recording sheet

Fault	Evidence that identified it	Checks passed, format set	Checks passed, own set	After the fix
Healthy baseline	—	—	—	—
Chat template mismatch	—	—	—	—
Learning rate and overfitting	—	—	—	—
Contaminated evaluation	—	—	—	—
Wrong base or precision at merge	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp, or mlx-lm on Track M the engine version, and the package versions from the run record · the base model from the lab, and the adapter run id for each row, the same quantisation for every row · 8,192 tokens of context · the date you ran it

Empty on purpose. Five rows, four deliberate faults, one change between each fault row and its fix. Record the noise floor separately: a row whose change is smaller than the noise floor is not a fault, and knowing that is half of what this page teaches.

The rough shapes, none of which is a measurement and all of which your own table will make concrete. A template mismatch is the largest effect and the least specific: everything gets worse at once. An over-trained adapter is specific in the opposite way: excellent on inputs that resemble the training set and worse elsewhere. A contaminated evaluation looks like the best result you have ever had. And a wrong-base merge produces a model that is fluent, confident and subtly wrong in a way that is hard to see without the reference answers next to it.

=Expected result

After each fix, the score should return to within the noise floor of your healthy report, and the piece of evidence that identified the fault should have changed: the hashes match, the best epoch moved, the overlap check passes, the merge comparison matches. A score that recovered without the evidence changing means something else moved, and you have not finished.

Troubleshooting

The diagnostic reports no run record. The lab’s training script writes one when --labbook is passed. If yours does not have it, that is the first finding: a run you cannot reproduce is not a result. Re-run the lab’s training with --labbook labbook.md before continuing.

Section 3 says the adapter carries no chat template. The adapter was saved without a tokeniser. On Tracks S, X and N the lab’s script saves one; on Track M an mlx-lm adapter genuinely does not carry one, and the training-time template is the base model’s, which is worth writing in the report rather than treating as an error.

The overlap check finds nothing on a set you know overlaps. Lower --threshold, or lower --n. Thirteen-word n-grams do not match a paraphrase that shares no thirteen consecutive words, and a short evaluation prompt has few n-grams to share.

The over-trained run finishes with a sensible best epoch. Your dataset is larger or more varied than the lab’s default, which is a good problem. Push harder: more epochs, or a higher learning rate, or a higher rank, one at a time.

Merging into the wrong base produces identical answers. The two checkpoints are more similar than expected, or the adapter’s effect is small relative to what they share. Try a base from a different size in the same family, and note the result: an adapter whose effect survives a base swap is an adapter that is not doing much.

All four reports look the same. Check you are passing different --report paths. The script overwrites, and four investigations written to diagnosis.md leave you with the last one.

The score changed and none of the four faults is present. Then the honest answer is one of three: the fine-tune genuinely did not help on these tasks, the change is inside the noise floor, or the fault is somewhere this procedure does not look. The diagnostic’s verdict section says exactly that rather than inventing a cause, which is the behaviour you want from an instrument.

Cleanup

Keep every diagnosis report and every run record, including the deliberate faults; they are the before-and-after evidence for the notebook. Delete the deliberately broken artefacts.

RunnableAll tracks

remove the deliberately broken models and data

rm -rf ~/models/format-wrongbase-merged
rm -f data/train-contaminated.jsonl wrong-template.jinja

⚠Warning

That command removes a directory tree and two files. Check the paths before you run it: a mistyped path here deletes the merged model your gateway is serving, and rebuilding it means another merge, conversion and quantisation.

The over-trained adapter is small; keep it. It is the most useful example of a bad model you will have, and comparing its answers with the healthy one is a two-minute exercise worth repeating when somebody new joins.

What you learned

• Evidence before theory. Six cheap observations in a fixed order eliminate almost every candidate. Guessing from the score is slower even when it eventually works.
• “It got worse” is a claim, and the noise floor tests it. Two runs of the base model at the same settings end about a third of these investigations before they start.
• The chat template is the highest-value single check. Two hashes, one command, and it explains the fault that makes a model worse at everything at once.
• A low training loss is not a result. The evaluation loss and the best epoch are what say whether the model learned or memorised, and the logged gradient norm warns before either.
• A good number needs checking as much as a bad one. A contaminated evaluation set produces the best result you have ever had, silently, and the check that catches it takes seconds.
• A merge fails quietly. Wrong base, wrong precision or wrong tokeniser all load, run and produce a fluent model that is not the one you trained. The artefact records what it should have been merged into; read it.
• One change, then measure. The proof of a fix is the same command producing a different number and different evidence.

Record in the notebook: your track and machine; the noise floor for your task set at your settings; one row per fault with the evidence, the fix and the before-and-after numbers; the sentence you would say to a colleague describing each fault in one line; and anything the routine health check found on the fine-tune you actually use.

Check your understanding

Question 1. A fine-tune scores lower than the base model. Which evidence should you look at first?

• The training loss
• The gap between two runs of the base model at the same settings, because a difference smaller than that is not a difference
• The rank and alpha
• The size of the adapter file

Show the answer and why

Answer: The gap between two runs of the base model at the same settings, because a difference smaller than that is not a difference

The noise floor is the cheapest step and it ends a large share of these investigations. Everything else in the procedure assumes there is a real effect to explain.

Question 2. The fine-tune is worse on every category, including ones unrelated to what you trained, and its answers often continue the prompt instead of replying to it. What is the first suspect?

• Overfitting
• A chat template mismatch between training and serving
• A contaminated evaluation set
• Too low a learning rate

Show the answer and why

Answer: A chat template mismatch between training and serving

Overfitting damages what is far from the training distribution and helps what is near it. Damage that is uniform across every category points at formatting, and Transformers' documentation notes that the wrong control tokens produce drastically worse performance.

Question 3. Your fine-tune scores much higher than the base on your own Part 10 task set, which you did not train for. What should you do?

• Record the result and move on
• Check the training data for overlap with that task file, because an implausible gain on an unrelated set is the signature of contamination
• Increase the rank and train again
• Change the judge model

Show the answer and why

Answer: Check the training data for overlap with that task file, because an implausible gain on an unrelated set is the signature of contamination

A good number needs the same scrutiny as a bad one, and contamination is silent. The overlap check reports exact matches and n-gram containment in seconds, and the fix is to remove the examples from the training data rather than from the evaluation set.

Question 4. An adapter behaves correctly when attached to the base at serving time and badly after merging. Which are plausible causes? Select all that apply.

• The merge used a different checkpoint from the one named in adapter_config.json
• The merge was into a base still carrying a quantization_config
• The exported model picked up the base model's tokeniser instead of the adapter's
• The adapter file is corrupt

Show the answer and why

Answer: The merge used a different checkpoint from the one named in adapter_config.json, The merge was into a base still carrying a quantization_config, The exported model picked up the base model's tokeniser instead of the adapter's

All three of the first options load without error and produce a fluent, wrong model. A corrupt adapter fails to load or produces nonsense immediately; it does not degrade behaviour selectively.

Question 5. After a change, the score returned to the healthy value but the template hashes, the best epoch, the overlap report and the merge comparison are all unchanged. What should you conclude?

• The fix worked and the investigation is complete
• The score moved for a reason you have not identified, so the diagnosis is not proven
• The diagnostic script is unreliable
• The baseline was wrong

Show the answer and why

Answer: The score moved for a reason you have not identified, so the diagnosis is not proven

A fix proves itself twice: the number recovers and the evidence that identified the fault changes. A number that recovers on its own usually means the first measurement was inside the noise, or that something else about the serving configuration moved between the two runs.

Check my answersClear