Project: A Specialist Assistant

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The base models, tool versions, wall-clock, peak memory and per-track notes each track was executed with belong here once the validation pass has run this project on real machines.

Objective

By the end of this project you will have a model fine-tuned on a domain you chose, a held-out set it was never trained on, a comparison against the base model on that set and on your own Part 10 tasks, an exported file behind a name in your gateway, and a one-page report that somebody else could act on.

The report is the deliverable. Anyone can produce a fine-tuned model in ninety minutes; the skill this project builds is being able to say what it is for, what it cost, what it improved, what it damaged, and whether you would use it. A model with no report is an artefact nobody can evaluate, including you in six months.

⌂Single-machine path

Single-machine on every track. One training process at a time, and an OpenAI-compatible endpoint for the evaluation, which may be on the same box.

Ninety minutes, in order

1. 1Choose the domain, and write the claimA job you actually have, and one sentence with a number in it that you will be able to check afterwards.
2. 2Choose the base by memory tierPart 11's arithmetic decides this, before any download.
3. 3Build or generate the datasetPairs you wrote, or drafts a served teacher wrote from your documents.
4. 4Review every generated exampleAccept, edit or reject. Nothing unreviewed becomes training data.
5. 5Decontaminate against both task filesThe held-out domain set and your own Part 10 set.
6. 6Train within your tierQLoRA where memory is tight, LoRA where it is not, and record peak memory against your estimate.
7. 7Evaluate: the gain and the costThe domain set shows what you bought. Your own set shows what you paid.
8. 8Export, serve, and write the reportMerged, converted, named in the gateway, and one page a colleague could act on.

The first two boxes are twenty minutes with no model running and they decide whether the rest is worth doing. The review gate in the middle is the one people want to skip and the one that determines whether the number at the end is real.

Requirements

Ninety minutes, of which roughly sixty-five are attended. The unattended parts are the base-model download, the teacher generating drafts if you use one, the training run and the export.

The memory floor is 16 GB, which is an eight-billion-parameter base under QLoRA with a short sequence. You need the training environment from Part 11 for your track, a llama.cpp checkout from Part 6 for the export, your own task file from Part 10, and either fifty to two hundred input and output pairs you can write, or a directory of documents a served model can draft from.

The reference models are Apache-2.0; the model reference records each licence and the Qwen3-4B card states it as apache-2.0. Choose by tier rather than by ambition:

Your memory	Base model	Method	Notes
16 GB	Qwen3-4B	LoRA at BF16, or QLoRA for a longer sequence	The 4B base at BF16 is about 8 GB, so LoRA fits with room for a short sequence.
24 GB	Qwen3-8B	QLoRA	The frozen base drops to roughly a quarter, and the room goes into sequence length.
32 to 64 GB	Qwen3-8B	QLoRA with a longer sequence and a larger batch	Activations become the term that moves, which the diagram below shows.
128 GB	Qwen3-14B	QLoRA	Or Qwen3-8B under LoRA at BF16, which avoids quantisation entirely.

QLoRA on Qwen3-8B, rank 16, batch 2 at 2,048 tokens, on a 16 GB machine

Frozen base weights, 4-bit NF4

4.5 GB

Adapter weights, gradients and Adam states

0.7 GB

Activations and logits, batch 2 at 2,048 tokens

2.5 GB

Reserved for the operating system

2 GB

Free

6.3 GB

Total

16 GB

Estimate from Part 11's arithmetic, not a measurement. The frozen base is about 0.55 bytes per parameter against the published 8.2 billion; the adapter is a rank-16 adapter on every linear layer at 16 bytes per adapter parameter; the activations use the model's layer count and hidden size with Part 11's checkpointed formula, plus a logits tensor of batch by sequence by vocabulary; the reserve is an allowance for the operating system. Compare it with what train-qlora.py reports as peak allocated memory and record both.

Compare that with the same model at batch 1 and 1,024 tokens in the second lesson: the base and the adapter are unchanged and the activations have grown fourfold. Activations are the term you control with settings, and on this tier they are the term that decides what fits.

Track S — NVIDIA DGX Spark

QLoRA with bitsandbytes on CUDA. The 4-bit data types are documented as needing compute capability 6.0 or newer, which the GB10 comfortably has. With 128 GB you have a real choice: Qwen3-14B under QLoRA, or Qwen3-8B under LoRA at BF16 with no quantisation anywhere, which removes one variable from the comparison and is worth doing at least once.

You can also hold the teacher, the training run and a serving process at the same time, which makes the generate-and-review loop in tasks 3 and 4 much quicker.

Track X — AMD Ryzen AI Max+ 395

QLoRA is documented for this chip. bitsandbytes’ installation page lists gfx1151 among its ROCm wheel targets from ROCm 6.4.4 onwards, alongside the other RDNA and CDNA architectures, so pip install bitsandbytes on a ROCm PyTorch build is the documented path. The validation pass has not exercised it, so record what you find with the date.

Two constraints from Part 5 still apply. The GPU-visible share of memory is the budget, not the machine’s total. And if 4-bit loading does not work on your build, --no-4bit with Qwen3-4B at BF16 is the honest alternative rather than a smaller sequence on a model that will not load.

Track M — Apple siliconPartial

bitsandbytes' installation page lists Apple silicon only in its CPU build table, so there is no 4-bit GPU path here. Track M trains adapters with mlx-lm, against an MLX model that is already quantised if memory is tight.

The Mac route to a quantised base is mlx-lm rather than bitsandbytes. mlx_lm.lora trains adapters against MLX models including quantised ones, and --num-layers is the memory knob that --load-in-4bit is elsewhere: adapting fewer layers costs less memory and less capacity.

Use train-lora-mlx.sh from this part’s lab with the dataset this project’s generator writes into data-mlx, and pick a 4-bit MLX conversion of your base if 16 GB is tight. Everything from the evaluation step onwards is identical to the other tracks, because the harness only needs an OpenAI-compatible endpoint.

Track N — NVIDIA desktop or laptop

QLoRA with bitsandbytes on CUDA, which is the configuration the paper describes. On a 16 GB card use Qwen3-8B with batch 1 and a 1,024-token sequence; on 24 GB raise the sequence length first and the batch second.

Inside WSL2, size against the virtual machine’s memory rather than the Windows total, as Part 6’s challenge covers.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-13-supervised-fine-tuning"
cd "$LAB_DIR"
pwd
test -f "make-domain-dataset.py"

Expected result: pwd ends in part-13-supervised-fine-tuning and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Choose the domain, and write the claim first

Fifteen minutes, no model running. This is the part of the project that decides whether the rest produces a result or an anecdote.

Pick a job you actually have. Answering questions about your own runbooks in a fixed form. Classifying incoming messages into your own categories. Rewriting rough notes into the shape your team’s documents use. Extracting the fields your systems consume from the messages your suppliers send. The test is whether you can write ten correct answers by hand in ten minutes: if you can, the knowledge is in your head and fine-tuning can move it into behaviour; if you cannot, you need retrieval and not this project.

!Important

Do not pick “knows our documentation”. The comparison this part’s first lesson cites found retrieval consistently outperforming unsupervised fine-tuning for both new and previously seen knowledge, and that models struggle to learn new facts this way. Pick the behaviour around your documentation instead: the form the answers take, the citation, and the refusal when the material does not contain the answer. That is a fine-tune, and it composes with the retrieval system you built in Part 10.

Then write the claim. One sentence, with a number, in the form you will check later:

Pseudocode — not a real command

On <held-out domain set>, at temperature 0 with seed 7, <fine-tune> passes at least
<n> more deterministic checks than <base>, and no category of my Part 10 task set
falls by more than the difference between two runs of <base> at these settings.

Put it in the report template now, before you have any results. Written afterwards, it becomes a description of what happened rather than a test that could have failed.

2. Establish the baseline and the noise floor

Score the base model on your own Part 10 task file, twice, at the same settings. The first run is the baseline. The difference between the two is your noise floor, and any later change smaller than it is not a change.

RunnableAll tracks

the base model, twice, before anything is trained

python3 run-eval.py \
    --base-url http://127.0.0.1:8080/v1 \
    --model qwen3-8b --quant Q4_K_M \
    --engine llama.cpp --engine-version v0.4.0 \
    --tasks my-tasks.json --out results-base-1.json --labbook labbook.md \
    --notes "part-13 project baseline, run 1 of 2"

python3 run-eval.py \
    --base-url http://127.0.0.1:8080/v1 \
    --model qwen3-8b --quant Q4_K_M \
    --engine llama.cpp --engine-version v0.4.0 \
    --tasks my-tasks.json --out results-base-2.json --labbook labbook.md \
    --notes "part-13 project baseline, run 2 of 2"

3. Build the dataset

RunnableAll tracks

make-domain-dataset.py

"""Build a domain fine-tuning dataset from your own material, with a review gate before training.

Purpose: turn work you already did, or drafts a local teacher wrote from your documents, into the
         train, validation and held-out splits a fine-tune needs. Two modes: --from-pairs takes
         input and output pairs you wrote or reviewed, and --from-docs asks a served model to
         draft candidates from your documents into a review file that is not training data until
         you have marked each one accepted.
Platform: all (standard library only; --from-docs needs a served model over the OpenAI-compatible
          API, which may be on any track or another machine)
Minimum memory: 8 GB on the machine serving the teacher; this script needs very little
Assumes: Python 3.10 or newer. For --from-docs, an OpenAI-compatible endpoint at --teacher-url
         and a directory of .txt or .md files. Everything written here carries whatever licence
         and confidentiality your source material carries, so redact before you generate rather
         than before you publish.

Usage: python3 make-domain-dataset.py --from-pairs my-pairs.jsonl --out-dir . \
           --system "You answer questions about our deployment runbook." \
           --instruction "Answer in at most four sentences and cite the section."

       python3 make-domain-dataset.py --from-docs ~/runbooks --out-dir . \
           --teacher-url http://127.0.0.1:8080/v1 --teacher-model qwen3-8b \
           --per-chunk 2 --max-chunks 60
       # then read drafts.jsonl, set "accepted" on every record, and:
       python3 make-domain-dataset.py --from-pairs drafts.jsonl --out-dir .

Writes, under --out-dir:
  data/train.jsonl, data/valid.jsonl      TRL conversational prompt-completion
  data-mlx/{train,valid,test}.jsonl       mlx-lm completions
  domain-tasks.json                       held-out tasks in the Part 10 task-file shape
  drafts.jsonl                            --from-docs only: candidates awaiting your review
"""
from __future__ import annotations

import argparse
import json
import random
import re
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any

DEFAULT_SYSTEM = "You answer questions about this team's own documents, briefly and exactly."
DEFAULT_INSTRUCTION = (
    "Answer in at most four sentences. If the documents do not contain the answer, "
    "say so in one sentence instead of guessing."
)
DRAFT_SYSTEM = (
    "You write training examples. Given a passage, produce question and answer pairs that a "
    "colleague could plausibly ask and that the passage fully answers. Never use information "
    "that is not in the passage. Reply with one JSON object per line and nothing else, each "
    'of the form {"input": "...", "output": "..."}.'
)


def post_json(url: str, payload: dict, api_key: str | None, timeout: int) -> dict:
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:300]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


def chunk_text(text: str, words_per_chunk: int) -> list[str]:
    """Split on blank lines, then pack paragraphs up to a word budget.

    Paragraph boundaries rather than a fixed window, because a question drafted from half a
    sentence is a question whose answer is not in the passage.
    """
    paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks, current, count = [], [], 0
    for paragraph in paragraphs:
        words = len(paragraph.split())
        if current and count + words > words_per_chunk:
            chunks.append("\n\n".join(current))
            current, count = [], 0
        current.append(paragraph)
        count += words
    if current:
        chunks.append("\n\n".join(current))
    return chunks


def draft_from_docs(args) -> list[dict[str, Any]]:
    """Ask the teacher for candidates. Nothing here is training data yet."""
    docs_dir = Path(args.from_docs).expanduser()
    paths = sorted(p for p in docs_dir.rglob("*") if p.suffix.lower() in {".txt", ".md"})
    if not paths:
        raise SystemExit(f"no .txt or .md files under {docs_dir}")
    print(f"{len(paths)} document(s) under {docs_dir}")

    drafts: list[dict[str, Any]] = []
    chunk_index = 0
    for path in paths:
        for chunk in chunk_text(path.read_text(encoding="utf-8", errors="replace"), args.words_per_chunk):
            if args.max_chunks and chunk_index >= args.max_chunks:
                break
            chunk_index += 1
            payload = {
                "model": args.teacher_model,
                "messages": [
                    {"role": "system", "content": DRAFT_SYSTEM},
                    {"role": "user", "content":
                        f"Write {args.per_chunk} question and answer pair(s) from this passage. "
                        f"Answers must follow this instruction: {args.instruction}\n\n"
                        f"Passage:\n{chunk}"},
                ],
                "temperature": 0.7,
                "max_tokens": 600,
            }
            try:
                body = post_json(f"{args.teacher_url.rstrip('/')}/chat/completions",
                                 payload, args.teacher_key, args.timeout)
            except RuntimeError as exc:
                print(f"teacher call failed on chunk {chunk_index}: {exc}")
                break
            content = (body.get("choices") or [{}])[0].get("message", {}).get("content", "") or ""
            for line in content.splitlines():
                line = line.strip().strip("`")
                if not line.startswith("{"):
                    continue
                try:
                    item = json.loads(line)
                except json.JSONDecodeError:
                    continue
                if not (isinstance(item.get("input"), str) and isinstance(item.get("output"), str)):
                    continue
                drafts.append({
                    "id": f"d{len(drafts) + 1:04d}",
                    "source": str(path.relative_to(docs_dir)),
                    "input": item["input"].strip(),
                    "output": item["output"].strip(),
                    "accepted": None,
                })
            print(f"  chunk {chunk_index} from {path.name}: {len(drafts)} draft(s) so far")
    return drafts


def load_pairs(path: Path, accept_unreviewed: bool) -> list[dict[str, Any]]:
    """Read input/output pairs, honouring the review gate."""
    pairs, unreviewed, rejected = [], 0, 0
    with path.open("r", encoding="utf-8") as handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                item = json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{number}: not valid JSON ({exc.msg})") from exc
            if not (isinstance(item.get("input"), str) and isinstance(item.get("output"), str)):
                raise SystemExit(f"{path}:{number}: every record needs string 'input' and 'output'")
            accepted = item.get("accepted")
            if accepted is False:
                rejected += 1
                continue
            if accepted is None:
                unreviewed += 1
                if not accept_unreviewed:
                    continue
            pairs.append(item)
    if rejected:
        print(f"{rejected} record(s) marked accepted: false, skipped")
    if unreviewed:
        state = "included" if accept_unreviewed else "skipped"
        print(f"{unreviewed} record(s) have no 'accepted' field, {state}")
        if not accept_unreviewed:
            print("  Set \"accepted\": true on the ones you have read, or pass "
                  "--accept-unreviewed if you wrote every pair yourself.")
    return pairs


def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    source = parser.add_mutually_exclusive_group(required=True)
    source.add_argument("--from-pairs", help="JSON Lines with input, output and accepted")
    source.add_argument("--from-docs", help="directory of .txt or .md files for the teacher to read")
    parser.add_argument("--out-dir", default=".")
    parser.add_argument("--system", default=DEFAULT_SYSTEM,
                        help="system prompt; use the same one at serving time or the fine-tune "
                             "is being asked to generalise across a change you never trained")
    parser.add_argument("--instruction", default=DEFAULT_INSTRUCTION,
                        help="appended to every question, and the contract the answers honour")
    parser.add_argument("--valid-fraction", type=float, default=0.1)
    parser.add_argument("--task-count", type=int, default=24,
                        help="held-out examples turned into a Part 10 task file")
    parser.add_argument("--max-words", type=int, default=120,
                        help="deterministic length check written into the task file")
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--accept-unreviewed", action="store_true",
                        help="use records with no 'accepted' field; only for pairs you wrote")
    parser.add_argument("--teacher-url", default=None)
    parser.add_argument("--teacher-model", default=None)
    parser.add_argument("--teacher-key", default=None)
    parser.add_argument("--per-chunk", type=int, default=2)
    parser.add_argument("--words-per-chunk", type=int, default=250)
    parser.add_argument("--max-chunks", type=int, default=0, help="0 means every chunk")
    parser.add_argument("--timeout", type=int, default=300)
    args = parser.parse_args()

    out = Path(args.out_dir)
    out.mkdir(parents=True, exist_ok=True)

    if args.from_docs:
        if not (args.teacher_url and args.teacher_model):
            raise SystemExit("--from-docs needs --teacher-url and --teacher-model")
        drafts = draft_from_docs(args)
        if not drafts:
            raise SystemExit("the teacher produced no usable drafts; check the endpoint and model")
        write_jsonl(out / "drafts.jsonl", drafts)
        print(f"\n{len(drafts)} draft(s) written to {out / 'drafts.jsonl'}")
        print("These are not training data yet. Read every one, set \"accepted\" to true or")
        print("false, and then re-run with --from-pairs drafts.jsonl. The rejections are the")
        print("most informative part: a pattern in them is usually a defect in the prompt to")
        print("the teacher rather than in the teacher.")
        return

    pairs = load_pairs(Path(args.from_pairs), args.accept_unreviewed)
    if len(pairs) < 40:
        print(f"WARNING: only {len(pairs)} usable pair(s). A domain behaviour change usually "
              "wants low thousands; a format change wants a few hundred. Expect a small effect.")
    if not pairs:
        raise SystemExit("no usable pairs; nothing to write")

    rng = random.Random(args.seed)
    rng.shuffle(pairs)

    n_tasks = min(args.task_count, len(pairs) // 4)
    held_out, remaining = pairs[:n_tasks], pairs[n_tasks:]
    n_valid = max(1, int(len(remaining) * args.valid_fraction))
    valid, train = remaining[:n_valid], remaining[n_valid:]

    system_message = {"role": "system", "content": args.system}
    prompt_of = lambda item: f"{item['input']}\n\n{args.instruction}"  # noqa: E731

    write_jsonl(out / "data" / "train.jsonl", [{
        "prompt": [system_message, {"role": "user", "content": prompt_of(p)}],
        "completion": [{"role": "assistant", "content": p["output"]}],
    } for p in train])
    write_jsonl(out / "data" / "valid.jsonl", [{
        "prompt": [system_message, {"role": "user", "content": prompt_of(p)}],
        "completion": [{"role": "assistant", "content": p["output"]}],
    } for p in valid])
    for name, rows in (("train", train), ("valid", valid), ("test", held_out)):
        write_jsonl(out / "data-mlx" / f"{name}.jsonl",
                    [{"prompt": prompt_of(p), "completion": p["output"]} for p in rows])

    tasks = {
        "name": "domain-holdout",
        "version": 1,
        "note": ("Held-out examples from the same source as the training data, never trained on. "
                 "Score the base model and the fine-tune on this file at the same settings, and "
                 "score both on your own Part 10 task file as well: this one shows the gain and "
                 "that one shows the cost."),
        "settings": {"temperature": 0.0, "top_p": 1.0, "seed": 7, "max_tokens": 512,
                     "comment": "Fixed for both models. Changing them makes a new baseline."},
        "categories": {"domain": "Does it answer this domain's questions in the required form?"},
        "tasks": [{
            "id": f"d{index + 1:02d}",
            "category": "domain",
            "prompt": prompt_of(item),
            "reference": item["output"],
            "rubric": (f"Answers the question from the domain material, honouring this "
                       f"instruction: {args.instruction} A confident answer to something the "
                       f"material does not cover scores 1."),
            "max_words": args.max_words,
        } for index, item in enumerate(held_out)],
    }
    (out / "domain-tasks.json").write_text(
        json.dumps(tasks, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")

    print(f"train:           {len(train)} -> {out / 'data' / 'train.jsonl'}")
    print(f"validation:      {len(valid)} -> {out / 'data' / 'valid.jsonl'}")
    print(f"held-out tasks:  {len(held_out)} -> {out / 'domain-tasks.json'}")
    print(f"mlx-lm layout:   {out / 'data-mlx'}")
    print("\nNext: run decontaminate.py against both this task file and your own Part 10 file.")


if __name__ == "__main__":
    main()

Download make-domain-dataset.py295 lines

Two routes, and most projects use both.

Route one: pairs you wrote. A JSON Lines file with input and output on every line. Fifty is enough to see an effect on a narrow behaviour; a few hundred is comfortable. Because you wrote them, pass --accept-unreviewed.

RunnableAll tracks

from pairs you wrote yourself

python3 make-domain-dataset.py \
    --from-pairs my-pairs.jsonl \
    --out-dir . \
    --accept-unreviewed \
    --system "You answer questions about our deployment runbooks." \
    --instruction "Answer in at most four sentences and name the section you used. If the runbooks do not cover it, say so in one sentence."

Route two: a local teacher drafts from your documents. Point it at a directory of text or Markdown, and a served model writes candidate question-and-answer pairs from each passage. This is a preview of Part 15, which teaches distillation properly.

RunnableAll tracks

drafts from your own documents, via a served teacher

python3 make-domain-dataset.py \
    --from-docs ~/runbooks \
    --out-dir . \
    --teacher-url http://127.0.0.1:8080/v1 \
    --teacher-model qwen3-14b \
    --per-chunk 2 \
    --max-chunks 80

⛨Security

Whatever is in those documents ends up in the weights, in a diffuse form you cannot later remove. Redact names, account numbers, credentials and internal hostnames in the source material before the teacher reads it, not before you publish the model. Replace them with stable placeholders so that the file you checksum in the run log is the file that was safe.

4. Review every generated example

The teacher writes drafts.jsonl and stops. Nothing in it is training data until you have opened it and set "accepted" to true or false on every record. The script skips unreviewed records by default, which means an unreviewed file simply produces an empty dataset and tells you why.

Budget fifteen minutes for a hundred drafts. You are checking three things: that the answer is actually supported by the passage it came from, that it honours the instruction you set, and that it is written the way you want every answer written.

∴Research note

The rejections are the most informative output of this step. One bad draft is noise. A pattern in the rejections is a defect in your prompt to the teacher: the answers are too long, or they hedge, or they invent a section number. Fix the instruction, regenerate, and review again. Two rounds of that produce better data than twice as many unreviewed drafts, and it is the same loop Part 15 formalises.

RunnableAll tracks

turn the reviewed drafts into splits and a held-out task file

python3 make-domain-dataset.py --from-pairs drafts.jsonl --out-dir .

5. Decontaminate against both task files

RunnableAll tracks

decontaminate.py

"""Check a supervised fine-tuning dataset for duplicates and for overlap with an evaluation set.

Purpose: the honesty check that has to run before any fine-tune is measured. Reads a
         training file and the Part 10 task file, reports exact duplicates inside the
         training data, exact matches against the evaluation set, and near-duplicates
         found by shared word n-grams, and can write a cleaned training file with the
         contaminated examples removed.
Platform: all (standard library only; no model, no accelerator, no network)
Minimum memory: 8 GB, and far less in practice: the whole check is text in memory
Assumes: Python 3.10 or newer. The training file is JSON Lines in any of the shapes
         Part 11's dataset lesson describes: {"prompt", "completion"}, {"messages": [...]}
         or {"text"}. The task file is the Part 10 evaluation set, a JSON object with a
         "tasks" list whose items carry "prompt" and usually "reference".

Usage: python3 decontaminate.py --train data/train.jsonl --tasks my-tasks.json
       python3 decontaminate.py --train data/train.jsonl --tasks my-tasks.json \
           --write-clean data/train-clean.jsonl --labbook labbook.md
       python3 decontaminate.py --train data/train.jsonl --valid data/valid.jsonl \
           --tasks my-tasks.json --n 13 --threshold 0.4 --strict

Two kinds of contamination are reported separately because they have different fixes.
An exact match means the same text is in both files, and the fix is to delete it from
the training data. A high n-gram containment means one text is largely contained in the
other, which is what happens when an evaluation prompt was paraphrased into a training
example, and the fix is a judgement call you have to make by reading the pair.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

WORD_RE = re.compile(r"[a-z0-9]+")


def normalise(text: str) -> list[str]:
    """Lowercase, drop punctuation, split on words. Two texts that differ only in
    formatting have to compare equal, or the check misses the commonest duplicate."""
    return WORD_RE.findall(text.lower())


def ngrams(words: list[str], n: int) -> set[tuple[str, ...]]:
    """The set of word n-grams. Short texts fall back to one gram of the whole text so
    that a two-word answer is still comparable rather than silently empty."""
    if len(words) < n:
        return {tuple(words)} if words else set()
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}


def containment(a: set[tuple[str, ...]], b: set[tuple[str, ...]]) -> float:
    """How much of a is also in b, between 0 and 1. Containment rather than Jaccard,
    because a short evaluation prompt buried inside a long training example is exactly
    the case that matters and Jaccard would score it low."""
    if not a:
        return 0.0
    return len(a & b) / len(a)


def digest(words: list[str]) -> str:
    return hashlib.sha256(" ".join(words).encode("utf-8")).hexdigest()


def record_text(record: dict[str, Any]) -> str:
    """Flatten one training record to the text a comparison should see, whichever of the
    three dataset shapes it is in."""
    if "messages" in record:
        parts = []
        for message in record["messages"]:
            content = message.get("content")
            if isinstance(content, str):
                parts.append(content)
        return "\n".join(parts)
    if "prompt" in record or "completion" in record:
        parts = []
        for key in ("prompt", "completion"):
            value = record.get(key)
            if isinstance(value, str):
                parts.append(value)
            elif isinstance(value, list):
                for message in value:
                    content = message.get("content") if isinstance(message, dict) else None
                    if isinstance(content, str):
                        parts.append(content)
        return "\n".join(parts)
    if isinstance(record.get("text"), str):
        return record["text"]
    return ""


def load_jsonl(path: Path) -> list[dict[str, Any]]:
    records = []
    with path.open("r", encoding="utf-8") as handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{number}: not valid JSON ({exc.msg})") from exc
    return records


def load_tasks(path: Path) -> list[dict[str, str]]:
    data = json.loads(path.read_text(encoding="utf-8"))
    tasks = data.get("tasks") if isinstance(data, dict) else data
    if not isinstance(tasks, list):
        raise SystemExit(f"{path} does not contain a list of tasks; expected the Part 10 task file")
    out = []
    for index, task in enumerate(tasks):
        text = "\n".join(
            str(task[key]) for key in ("prompt", "reference") if isinstance(task.get(key), str)
        )
        out.append({"id": str(task.get("id", f"task-{index:03d}")), "text": text})
    return out


def summarise(text: str, limit: int = 90) -> str:
    flat = " ".join(text.split())
    return flat if len(flat) <= limit else flat[: limit - 1] + "…"


def check_split(
    name: str,
    records: list[dict[str, Any]],
    task_grams: list[tuple[str, set[tuple[str, ...]]]],
    task_hashes: dict[str, str],
    n: int,
    threshold: float,
) -> dict[str, Any]:
    """One split against the evaluation set, plus its own internal duplicates."""
    seen: dict[str, int] = {}
    internal: list[dict[str, Any]] = []
    exact: list[dict[str, Any]] = []
    near: list[dict[str, Any]] = []
    contaminated: set[int] = set()
    empty = 0

    for index, record in enumerate(records):
        text = record_text(record)
        words = normalise(text)
        if not words:
            empty += 1
            continue
        key = digest(words)
        if key in seen:
            internal.append({"index": index, "duplicate_of": seen[key], "text": summarise(text)})
            contaminated.add(index)
            continue
        seen[key] = index

        if key in task_hashes:
            exact.append({"index": index, "task": task_hashes[key], "text": summarise(text)})
            contaminated.add(index)
            continue

        grams = ngrams(words, n)
        worst_task, worst_score = None, 0.0
        for task_id, gram_set in task_grams:
            score = max(containment(grams, gram_set), containment(gram_set, grams))
            if score > worst_score:
                worst_task, worst_score = task_id, score
        if worst_score >= threshold:
            near.append({
                "index": index, "task": worst_task,
                "containment": round(worst_score, 3), "text": summarise(text),
            })
            contaminated.add(index)

    return {
        "split": name,
        "examples": len(records),
        "empty_examples": empty,
        "internal_duplicates": internal,
        "exact_matches": exact,
        "near_duplicates": near,
        "contaminated_indices": sorted(contaminated),
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--train", required=True, help="training file, JSON Lines")
    parser.add_argument("--valid", default=None, help="validation file, checked the same way")
    parser.add_argument("--tasks", required=True, help="the Part 10 evaluation task file")
    parser.add_argument("--n", type=int, default=13,
                        help="n-gram size for the near-duplicate check; 13 is the size the "
                             "decontamination sections of several pretraining papers use")
    parser.add_argument("--threshold", type=float, default=0.5,
                        help="report a pair when either text's n-grams are this fraction "
                             "contained in the other's")
    parser.add_argument("--report", default=None, help="write the full findings here as JSON")
    parser.add_argument("--write-clean", default=None,
                        help="write the training file with every flagged example removed")
    parser.add_argument("--labbook", default=None, help="append one JSON line recording this check")
    parser.add_argument("--strict", action="store_true",
                        help="exit 1 when anything was flagged, for use in a script")
    args = parser.parse_args()

    train_path = Path(args.train)
    tasks_path = Path(args.tasks)
    for path in (train_path, tasks_path):
        if not path.is_file():
            raise SystemExit(f"{path} does not exist")

    tasks = load_tasks(tasks_path)
    task_grams = [(t["id"], ngrams(normalise(t["text"]), args.n)) for t in tasks]
    task_hashes = {digest(normalise(t["text"])): t["id"] for t in tasks}

    train_records = load_jsonl(train_path)
    results = [check_split("train", train_records, task_grams, task_hashes, args.n, args.threshold)]
    if args.valid:
        valid_path = Path(args.valid)
        if not valid_path.is_file():
            raise SystemExit(f"{valid_path} does not exist")
        results.append(check_split("validation", load_jsonl(valid_path), task_grams,
                                   task_hashes, args.n, args.threshold))

    flagged = 0
    print(f"tasks in evaluation set: {len(tasks)}   n-gram size: {args.n}   "
          f"threshold: {args.threshold}")
    for result in results:
        counts = (len(result["internal_duplicates"]), len(result["exact_matches"]),
                  len(result["near_duplicates"]))
        flagged += sum(counts)
        print(f"\n{result['split']}: {result['examples']} example(s), "
              f"{counts[0]} internal duplicate(s), {counts[1]} exact match(es) against the "
              f"evaluation set, {counts[2]} near-duplicate(s)")
        for row in result["exact_matches"][:5]:
            print(f"  EXACT  line {row['index'] + 1} == task {row['task']}: {row['text']}")
        for row in result["near_duplicates"][:5]:
            print(f"  NEAR   line {row['index'] + 1} ~ task {row['task']} "
                  f"(containment {row['containment']}): {row['text']}")
        for row in result["internal_duplicates"][:5]:
            print(f"  DUP    line {row['index'] + 1} repeats line {row['duplicate_of'] + 1}: "
                  f"{row['text']}")

    if args.report:
        Path(args.report).write_text(
            json.dumps({"tasks_file": str(tasks_path), "n": args.n,
                        "threshold": args.threshold, "results": results}, indent=2),
            encoding="utf-8")
        print(f"\nfull findings written to {args.report}")

    if args.write_clean:
        drop = set(results[0]["contaminated_indices"])
        kept = [r for i, r in enumerate(train_records) if i not in drop]
        with Path(args.write_clean).open("w", encoding="utf-8") as handle:
            for record in kept:
                handle.write(json.dumps(record, ensure_ascii=False) + "\n")
        print(f"clean training file: {args.write_clean} "
              f"({len(kept)} kept, {len(train_records) - len(kept)} removed)")

    if args.labbook:
        line = {
            "check": "part-13/decontaminate",
            "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
            "train": str(train_path),
            "tasks": str(tasks_path),
            "n": args.n,
            "threshold": args.threshold,
            "flagged": flagged,
            "per_split": [
                {"split": r["split"], "examples": r["examples"],
                 "internal_duplicates": len(r["internal_duplicates"]),
                 "exact_matches": len(r["exact_matches"]),
                 "near_duplicates": len(r["near_duplicates"])}
                for r in results
            ],
        }
        book = Path(args.labbook)
        if not book.exists():
            book.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
        with book.open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(line, sort_keys=True) + "\n")
        print(f"recorded the check in {args.labbook}")

    if flagged == 0:
        print("\nNothing flagged. The training data and the evaluation set are disjoint at "
              "this n-gram size and threshold, which is what a comparable measurement needs.")
    else:
        print(f"\n{flagged} item(s) flagged. Read them before you train: a number measured on a "
              "contaminated evaluation set is not a measurement.")
    if args.strict and flagged:
        sys.exit(1)


if __name__ == "__main__":
    main()

Download decontaminate.py294 lines

Two checks, because there are two evaluation sets and both can be contaminated. The domain set was split off from the same pool as the training data, so an internal duplicate straddling the split leaks the answer. Your Part 10 set is independent, unless your domain overlaps the tasks you wrote back then, which is more likely than it sounds.

RunnableAll tracks

check the training data against both sets

python3 decontaminate.py --train data/train.jsonl --tasks domain-tasks.json \
    --report decon-domain.json --labbook labbook.md

python3 decontaminate.py --train data/train.jsonl --tasks my-tasks.json \
    --report decon-mytasks.json --labbook labbook.md --strict

The second uses --strict, so it exits non-zero if anything was flagged. That is deliberate: an overlap with the set you use to detect regressions would hide exactly the damage this project is supposed to measure.

6. Train within your tier

RunnableAll tracks

sftlog.py

"""Append one machine-readable record per fine-tuning run or evaluation to the lab notebook.

Purpose: Part 13's self-contained copy of the run-log format defined in Part 11, so that
         this part's scripts record a run identically without depending on Part 11's files
         being on the path. Every field is filled in or written as null, because a reader a
         month later has to be able to tell "not recorded" from "not applicable".
Platform: all (standard library only; torch, transformers, trl, peft and mlx are inspected
          for their version strings only if they happen to be installed)
Minimum memory: 8 GB
Assumes: Python 3.10 or newer. The lab notebook is created if it does not exist. git is
         optional and is used only to record the commit the configuration was at.

Usage: imported by this part's Python scripts:
           import sftlog; sftlog.record(labbook="labbook.md", lab="part-13/train-lora", ...)
       or called from a shell script with the run's own fields as JSON on stdin:
           python3 sftlog.py --record --labbook labbook.md < fields.json
       or run with no arguments to print the field list and exit.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import platform
import secrets
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

# The same field set Part 11 defines. A record missing any of them is refused, because a
# partial record is harder to interpret than no record at all.
FIELDS = (
    "run_id", "lab", "date", "config_commit", "model", "dataset",
    "hyperparameters", "seed", "hardware", "versions", "losses", "scores", "notes",
)

PACKAGES = ("torch", "transformers", "trl", "peft", "datasets", "bitsandbytes", "mlx", "mlx-lm")


def new_run_id() -> str:
    """Sorts by time and does not collide between two runs started in the same second."""
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


def file_sha256(path: str | os.PathLike[str], chunk: int = 1 << 20) -> str | None:
    """Content hash of a dataset file, so a run is tied to the exact bytes it read.

    This is the field people leave out and then need: a month later the question is not
    which learning rate was used, which is in the script, but whether this run was before
    or after the dataset was fixed.
    """
    p = Path(path)
    if not p.is_file():
        return None
    digest = hashlib.sha256()
    with p.open("rb") as handle:
        while True:
            block = handle.read(chunk)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def git_commit(path: str | os.PathLike[str] = ".") -> str | None:
    """The commit the configuration is at, with -dirty when the tree had uncommitted edits."""
    if shutil.which("git") is None:
        return None
    target = Path(path)
    cwd = target if target.is_dir() else target.parent
    try:
        rev = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=cwd,
                             capture_output=True, text=True, check=True, timeout=10).stdout.strip()
        dirty = subprocess.run(["git", "status", "--porcelain"], cwd=cwd,
                               capture_output=True, text=True, check=True, timeout=10).stdout.strip()
    except (subprocess.SubprocessError, OSError):
        return None
    return f"{rev}-dirty" if dirty else rev


def describe_hardware() -> dict[str, Any]:
    """What the run executed on, as far as it can be established without extra packages."""
    info: dict[str, Any] = {
        "os": f"{platform.system()} {platform.release()}",
        "machine": platform.machine(),
        "python": platform.python_version(),
        "accelerator": "cpu",
        "device_name": None,
    }
    try:
        import torch  # noqa: PLC0415 - optional, and only for reporting
    except ImportError:
        return info
    if torch.cuda.is_available():
        info["accelerator"] = "cuda"
        info["device_name"] = torch.cuda.get_device_name(0)
        return info
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        info["accelerator"] = "mps"
        info["device_name"] = platform.processor() or "Apple silicon"
    return info


def package_versions(names: tuple[str, ...] = PACKAGES) -> dict[str, str | None]:
    """Version strings for the packages that decide what a run actually did."""
    from importlib.metadata import PackageNotFoundError, version  # noqa: PLC0415

    out: dict[str, str | None] = {}
    for name in names:
        try:
            out[name] = version(name)
        except PackageNotFoundError:
            out[name] = None
    return out


def build_record(
    lab: str,
    model: str,
    dataset: dict[str, Any],
    hyperparameters: dict[str, Any],
    seed: int,
    losses: dict[str, Any] | None = None,
    scores: dict[str, Any] | None = None,
    config_path: str | os.PathLike[str] | None = None,
    notes: str | None = None,
) -> dict[str, Any]:
    """Assemble the record. Separate from writing so a caller can inspect it first."""
    return {
        "run_id": new_run_id(),
        "lab": lab,
        "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "config_commit": git_commit(config_path if config_path is not None else "."),
        "model": model,
        "dataset": dataset,
        "hyperparameters": hyperparameters,
        "seed": seed,
        "hardware": describe_hardware(),
        "versions": package_versions(),
        "losses": losses or {},
        "scores": scores or {},
        "notes": notes,
    }


def append(record: dict[str, Any], labbook: str | os.PathLike[str] = "labbook.md") -> Path:
    """Append one JSON line. A missing required field is an error, not a warning."""
    missing = [f for f in FIELDS if f not in record]
    if missing:
        raise ValueError(f"run record is missing required fields: {', '.join(missing)}")
    path = Path(labbook)
    if not path.exists():
        path.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    return path


def record(labbook: str | os.PathLike[str] = "labbook.md", **kwargs: Any) -> dict[str, Any]:
    """Build and append in one call; returns the record so a caller can print its id."""
    rec = build_record(**kwargs)
    append(rec, labbook)
    return rec


def main() -> None:
    parser = argparse.ArgumentParser(description="Run-log helper for Part 13's fine-tuning labs.")
    parser.add_argument("--record", action="store_true",
                        help="read this run's own fields as a JSON object on stdin")
    parser.add_argument("--labbook", default="labbook.md")
    args = parser.parse_args()

    if not args.record:
        print(__doc__)
        print("Fields in every record:", ", ".join(FIELDS))
        return

    fields = json.load(sys.stdin)
    allowed = {"lab", "model", "dataset", "hyperparameters", "seed",
               "losses", "scores", "config_path", "notes"}
    unknown = set(fields) - allowed
    if unknown:
        raise SystemExit(f"unknown field(s) on stdin: {', '.join(sorted(unknown))}")
    rec = record(labbook=args.labbook, **fields)
    print(f"recorded run {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download sftlog.py195 lines

RunnableAll tracks

train-qlora.py

"""Fine-tune a 4B to 8B model with QLoRA: a 4-bit frozen base and trainable LoRA adapters.

Purpose: the project's training run. Loads the base in 4-bit NormalFloat with double
         quantisation so that an eight-billion-parameter model fits a 16 GB machine, attaches
         adapters to every linear layer in QLoRA's own style, trains with an evaluation after
         each epoch and early stopping, reports peak memory, and appends a run record to the
         lab notebook. Passing --no-4bit trains the same configuration with a bfloat16 base,
         which is what Track M and any machine without bitsandbytes use.
Platform: spark, nvidia (CUDA) and strix (ROCm; bitsandbytes lists gfx1151 among its ROCm wheel
          targets from ROCm 6.4.4, read 2026-09-09). Not mac: bitsandbytes' installation page
          shows Apple silicon only in its CPU build table, so Track M runs --no-4bit here, or
          trains against an already-quantised MLX model with mlx_lm.lora instead.
Minimum memory: 16 GB. Part 11's arithmetic: a 4-bit base is about 0.55 bytes per parameter,
          the adapter carries 16 bytes per trainable parameter, and activations depend on your
          batch size and sequence length.
Assumes: torch, transformers, trl, peft and datasets installed, plus bitsandbytes unless
         --no-4bit is used; make-domain-dataset.py has been run so that data/train.jsonl and
         data/valid.jsonl exist; sftlog.py sits next to this file.

Usage: python3 train-qlora.py --model Qwen/Qwen3-8B --data-dir data \
           --output-dir runs/domain-qwen3-8b --labbook labbook.md
       python3 train-qlora.py --model Qwen/Qwen3-4B --no-4bit --rank 32 --alpha 64
"""
from __future__ import annotations

import argparse
import importlib.util
import json
import time
from pathlib import Path

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig, EarlyStoppingCallback
from trl import SFTConfig, SFTTrainer

import sftlog


def pick_device() -> str:
    if torch.cuda.is_available():
        return "cuda"
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return "mps"
    return "cpu"


def check_4bit_available(device: str) -> None:
    """Fail with the reason rather than with a stack trace three minutes into a download."""
    if importlib.util.find_spec("bitsandbytes") is None:
        raise SystemExit(
            "bitsandbytes is not installed, so --load-in-4bit cannot work.\n"
            "  CUDA and ROCm: pip install bitsandbytes, then re-run.\n"
            "  macOS: the installation page lists Apple silicon only under its CPU builds, so\n"
            "         there is no 4-bit GPU path here. Re-run with --no-4bit and a smaller\n"
            "         model, or use mlx_lm.lora against an already-quantised MLX model."
        )
    if device != "cuda":
        raise SystemExit(
            f"the active device is {device!r}, and 4-bit loading with bitsandbytes is documented\n"
            "for CUDA and ROCm devices, both of which PyTorch reports as 'cuda'.\n"
            "Re-run with --no-4bit, and choose a model your memory tier can hold at bfloat16."
        )


def summarise_history(history: list[dict]) -> dict[str, float | int | None]:
    train_losses = [row["loss"] for row in history if "loss" in row]
    evals = [(row.get("epoch"), row["eval_loss"]) for row in history if "eval_loss" in row]
    best_epoch, best_eval = min(evals, key=lambda pair: pair[1]) if evals else (None, None)
    return {
        "first_train_loss": round(train_losses[0], 4) if train_losses else None,
        "final_train_loss": round(train_losses[-1], 4) if train_losses else None,
        "final_eval_loss": round(evals[-1][1], 4) if evals else None,
        "best_eval_loss": round(best_eval, 4) if best_eval is not None else None,
        "best_epoch": best_epoch,
        "evaluations": len(evals),
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-8B",
                        help="base model repository id or local path; an instruct checkpoint")
    parser.add_argument("--data-dir", default="data")
    parser.add_argument("--output-dir", default="runs/domain-qlora")
    parser.add_argument("--no-4bit", dest="load_in_4bit", action="store_false",
                        help="train against a bfloat16 base instead of a 4-bit one")
    parser.set_defaults(load_in_4bit=True)
    parser.add_argument("--epochs", type=float, default=3.0)
    parser.add_argument("--batch-size", type=int, default=1)
    parser.add_argument("--grad-accum", type=int, default=8)
    parser.add_argument("--lr", type=float, default=1e-4)
    parser.add_argument("--max-length", type=int, default=1024)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=int, default=32)
    parser.add_argument("--dropout", type=float, default=0.05)
    parser.add_argument("--target-modules", nargs="+", default=["all-linear"],
                        help="QLoRA's own configuration adapts every linear layer, which PEFT "
                             "expresses as the single value all-linear")
    parser.add_argument("--gradient-checkpointing", action="store_true", default=True)
    parser.add_argument("--no-gradient-checkpointing", dest="gradient_checkpointing",
                        action="store_false")
    parser.add_argument("--early-stopping-patience", type=int, default=2)
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--notes", default=None)
    args = parser.parse_args()

    device = pick_device()
    if args.load_in_4bit:
        check_4bit_available(device)
    bf16 = device == "cuda" and torch.cuda.is_bf16_supported()
    dtype = torch.bfloat16 if bf16 else torch.float32
    method = "qlora" if args.load_in_4bit else "lora"
    print(f"device: {device}   method: {method}   "
          f"compute precision: {'bfloat16' if bf16 else 'float32'}")

    data_dir = Path(args.data_dir)
    files = {"train": str(data_dir / "train.jsonl"), "validation": str(data_dir / "valid.jsonl")}
    for split, path in files.items():
        if not Path(path).is_file():
            raise SystemExit(f"{path} is missing ({split} split); run make-domain-dataset.py first")
    dataset = load_dataset("json", data_files=files)
    print(f"train examples: {len(dataset['train'])}   "
          f"validation examples: {len(dataset['validation'])}")

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    if tokenizer.chat_template is None:
        raise SystemExit(f"{args.model} has no chat template; choose an instruct checkpoint")

    quantization_config = None
    if args.load_in_4bit:
        # The four keys PEFT's quantisation guide names for QLoRA: 4-bit loading, the NF4
        # data type, double quantisation of the quantisation constants, and bfloat16 for
        # the arithmetic. The frozen base is what gets quantised; the adapters do not.
        quantization_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_use_double_quant=True,
            bnb_4bit_compute_dtype=torch.bfloat16,
        )

    targets = args.target_modules[0] if args.target_modules == ["all-linear"] else args.target_modules

    config = SFTConfig(
        output_dir=args.output_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        gradient_accumulation_steps=args.grad_accum,
        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_steps=10,
        max_length=args.max_length,
        packing=False,
        completion_only_loss=True,
        gradient_checkpointing=args.gradient_checkpointing,
        bf16=bf16,
        model_init_kwargs=None if args.load_in_4bit else {"dtype": dtype},
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        logging_steps=5,
        report_to="none",
        seed=args.seed,
        data_seed=args.seed,
    )

    peft_config = LoraConfig(
        r=args.rank,
        lora_alpha=args.alpha,
        lora_dropout=args.dropout,
        target_modules=targets,
        bias="none",
        task_type="CAUSAL_LM",
    )

    callbacks = []
    if args.early_stopping_patience > 0:
        callbacks.append(EarlyStoppingCallback(early_stopping_patience=args.early_stopping_patience))

    if device == "cuda":
        torch.cuda.reset_peak_memory_stats()

    # quantization_config plus peft_config is the documented QLoRA path through SFTTrainer:
    # the trainer loads the base with the quantisation applied and wraps it for adapter
    # training, so the model is never held at full precision.
    trainer = SFTTrainer(
        model=args.model,
        args=config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        processing_class=tokenizer,
        peft_config=peft_config,
        quantization_config=quantization_config,
        callbacks=callbacks or None,
    )
    trainer.model.print_trainable_parameters()

    started = time.time()
    trainer.train()
    elapsed = time.time() - started

    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)

    losses = summarise_history(trainer.state.log_history)
    losses["seconds"] = round(elapsed, 1)
    peak_gb = None
    if device == "cuda":
        peak_gb = round(torch.cuda.max_memory_allocated() / 1e9, 2)
        losses["peak_memory_gb"] = peak_gb
    print(json.dumps(losses, indent=2))
    print(f"adapter saved to {args.output_dir}")
    if peak_gb is not None:
        print(f"peak allocated memory: {peak_gb} GB. Compare it with the estimate you made "
              "from Part 11's arithmetic before the run, and record both.")
    print("\nThe adapter was trained against a 4-bit base." if args.load_in_4bit else
          "\nThe adapter was trained against a bfloat16 base.")
    print("Merge it into the base at bfloat16, never into the quantised copy, and quantise "
          "the merged result afterwards. merge-adapter.py does this.")

    if args.labbook:
        record = sftlog.record(
            labbook=args.labbook,
            lab="part-13/train-qlora",
            model=args.model,
            dataset={
                "path": files["train"],
                "sha256": sftlog.file_sha256(files["train"]),
                "train_examples": len(dataset["train"]),
                "validation_examples": len(dataset["validation"]),
            },
            hyperparameters={
                "method": method,
                "load_in_4bit": args.load_in_4bit,
                "bnb_4bit_quant_type": "nf4" if args.load_in_4bit else None,
                "bnb_4bit_use_double_quant": args.load_in_4bit,
                "rank": args.rank,
                "alpha": args.alpha,
                "dropout": args.dropout,
                "target_modules": targets,
                "epochs": args.epochs,
                "batch_size": args.batch_size,
                "grad_accum": args.grad_accum,
                "effective_batch": args.batch_size * args.grad_accum,
                "learning_rate": args.lr,
                "max_length": args.max_length,
                "gradient_checkpointing": args.gradient_checkpointing,
                "early_stopping_patience": args.early_stopping_patience,
                "compute_precision": "bfloat16" if bf16 else "float32",
                "peak_memory_gb": peak_gb,
                "output_dir": args.output_dir,
            },
            seed=args.seed,
            losses=losses,
            scores={},
            config_path=__file__,
            notes=args.notes,
        )
        print(f"recorded run {record['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download train-qlora.py270 lines

The script loads the base in 4-bit NormalFloat with double quantisation and bfloat16 arithmetic, which are the four settings PEFT’s quantisation guide names, and adapts every linear layer, which is what the QLoRA paper’s own configuration does and PEFT expresses as target_modules="all-linear".

Estimate the peak memory with Part 11’s arithmetic before you start, and write the estimate down. The script prints the measured peak at the end, and the gap between the two is worth more than either number alone.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

QLoRA on a 14-billion-parameter base

python3 train-qlora.py \
    --model Qwen/Qwen3-14B \
    --data-dir data \
    --output-dir runs/domain-qwen3-14b \
    --epochs 3 \
    --batch-size 2 \
    --grad-accum 4 \
    --max-length 2048 \
    --rank 32 --alpha 64 \
    --labbook labbook.md \
    --notes "runbook answering, QLoRA, first run"

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

QLoRA on ROCm, with the BF16 alternative if 4-bit will not load

python3 train-qlora.py \
    --model Qwen/Qwen3-8B \
    --data-dir data \
    --output-dir runs/domain-qwen3-8b \
    --epochs 3 \
    --batch-size 1 \
    --grad-accum 8 \
    --max-length 1024 \
    --labbook labbook.md \
    --notes "runbook answering, QLoRA on ROCm, first run"

If bitsandbytes will not load on your build, the script stops with the reason rather than training something you did not ask for. Re-run with --no-4bit --model Qwen/Qwen3-4B, and record that you did: a comparison between a 4B BF16 fine-tune and an 8B base is measuring two things at once, so change the base model in the comparison as well.

Track M — Apple silicon

RunnableTrack M · Apple silicon

the MLX path, using this project's dataset

MODEL=mlx-community/Qwen3-8B-4bit \
DATA=data-mlx \
ADAPTERS=runs/domain-mlx-adapters \
FUSED=models/domain-mlx-fused \
NUM_LAYERS=16 \
bash train-lora-mlx.sh mlx-community/Qwen3-8B-4bit 800

Training against an already-quantised MLX model is this track’s equivalent of QLoRA, and --num-layers is the knob that keeps it inside 16 GB. Fusing an adapter trained against a quantised base is what mlx_lm.fuse --dequantize exists for; read the export lesson before you convert.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

QLoRA on a 16 GB card

python3 train-qlora.py \
    --model Qwen/Qwen3-8B \
    --data-dir data \
    --output-dir runs/domain-qwen3-8b \
    --epochs 3 \
    --batch-size 1 \
    --grad-accum 8 \
    --max-length 1024 \
    --labbook labbook.md \
    --notes "runbook answering, QLoRA, first run"

On 24 GB, raise --max-length to 2048 before you raise --batch-size: a truncated example teaches a truncated answer.

7. Read the curves, the peak memory and the trainable count

Three numbers before you go near the evaluation.

The trainable-parameter percentage should be well under one per cent. With all-linear it will be higher than the seven-projection default from the lab, and still small.

The best epoch tells you whether the data and the epoch count agree. Best epoch one on three epochs means too many epochs for this much data.

The peak allocated memory against your estimate. On a machine where the run fitted with room to spare, the interesting question is what to spend the room on, and the answer is nearly always sequence length.

8. Evaluate: the gain and the cost

RunnableAll tracks

evaluate-against-base.py

"""Score a fine-tuned model and the base model it came from on the same tasks, and report the gap.

Purpose: the only question this part asks. Runs Part 10's evaluation harness twice over the
         same task files at the same settings, once against the base model and once against
         the fine-tune, optionally grades both with the same judge, and prints the difference
         per category so that a gain on the task you trained for and a loss on everything else
         are both visible. Appends one record to the lab notebook.
Platform: all (pure Python over HTTP; both models are reached through an OpenAI-compatible API,
          so they may be served by any engine on any track, or by the Part 9 gateway)
Minimum memory: 8 GB on the machine serving the models; this script needs very little
Assumes: Python 3.10 or newer; Part 10's run-eval.py and judge.py in --harness-dir; both models
         reachable at --base-url under the names given, one at a time or together; sftlog.py
         next to this file. Pass every task file you care about: the one that tests the
         behaviour you trained, and your own Part 10 set, which is where a regression shows up.

Usage: python3 evaluate-against-base.py --harness-dir ~/eval \
           --base-url http://127.0.0.1:8080/v1 \
           --base-model qwen3-1.7b --tuned-model qwen3-1.7b-format \
           --quant Q4_K_M --engine llama.cpp --engine-version v0.4.0 \
           --tasks format-tasks.json --tasks my-tasks.json \
           --out-dir eval-out --labbook labbook.md

       Add --judge-model qwen3-8b to grade both runs with the same judge. Leave it out and
       only the deterministic checks are compared, which is a floor rather than a score but
       is exactly repeatable and needs no third model.
"""
from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
import shlex
from collections import defaultdict
from pathlib import Path
from typing import Any

import sftlog


def run_harness(script: Path, arguments: list[str]) -> None:
    """Run one harness script, showing the command first so the log says what happened."""
    command = [sys.executable, str(script), *arguments]
    safe_command = list(command)
    for i, value in enumerate(safe_command[:-1]):
        if value == "--api-key":
            safe_command[i + 1] = "[redacted]"
    print("+ " + shlex.join(safe_command))
    subprocess.run(command, check=True)


def category_pass_rates(results: list[dict[str, Any]]) -> dict[str, tuple[int, int]]:
    """Deterministic passes and totals per category. No model involved, exactly repeatable."""
    counts: dict[str, list[int]] = defaultdict(lambda: [0, 0])
    for item in results:
        counts[item["category"]][1] += 1
        counts[item["category"]][0] += int(item["checks"]["passed"])
    return {k: (v[0], v[1]) for k, v in sorted(counts.items())}


def evaluate(args, script_dir: Path, out_dir: Path, tasks_path: Path,
             model: str, label: str) -> dict[str, Any]:
    """One model, one task file: run it, optionally grade it, and return the numbers."""
    stem = f"{label}-{tasks_path.stem}"
    results_path = out_dir / f"results-{stem}.json"
    arguments = [
        "--base-url", (args.tuned_url or args.base_url) if label == "tuned" else args.base_url,
        "--model", model,
        "--quant", args.quant,
        "--engine", args.engine,
        "--engine-version", args.engine_version,
        "--tasks", str(tasks_path),
        "--out", str(results_path),
        "--notes", f"part-13 {label} on {tasks_path.name}",
    ]
    if args.api_key:
        arguments += ["--api-key", args.api_key]
    if args.system:
        arguments += ["--system", args.system]
    if args.labbook:
        arguments += ["--labbook", args.labbook]
    run_harness(script_dir / "run-eval.py", arguments)

    payload = json.loads(results_path.read_text(encoding="utf-8"))
    out: dict[str, Any] = {
        "model": model,
        "results_file": str(results_path),
        "tasks": payload["run"]["tasks"],
        "checks_passed": payload["run"]["checks_passed"],
        "by_category_checks": category_pass_rates(payload["results"]),
        "judge_mean": None,
        "by_category_judge": {},
    }

    if args.judge_model:
        judged_path = out_dir / f"judged-{stem}.json"
        judge_arguments = [
            "grade",
            "--base-url", args.judge_url or args.base_url,
            "--judge-model", args.judge_model,
            "--results", str(results_path),
            "--out", str(judged_path),
        ]
        if args.api_key:
            judge_arguments += ["--api-key", args.api_key]
        if args.labbook:
            judge_arguments += ["--labbook", args.labbook]
        run_harness(script_dir / "judge.py", judge_arguments)
        judged = json.loads(judged_path.read_text(encoding="utf-8"))
        out["judge_mean"] = judged["judge"]["judge_mean"]
        out["by_category_judge"] = judged["judge"]["by_category"]
        out["judged_file"] = str(judged_path)
    return out


def delta(new: float | None, old: float | None) -> str:
    if new is None or old is None:
        return "—"
    difference = new - old
    return f"{difference:+.2f}"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--harness-dir", default=".",
                        help="directory holding Part 10's run-eval.py and judge.py")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--tuned-url", default=None,
                        help="fine-tune endpoint; defaults to --base-url for a shared gateway")
    parser.add_argument("--pause-before-tuned", action="store_true",
                        help="wait for you to stop the base server and start the fine-tune")
    parser.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY"))
    parser.add_argument("--base-model", required=True,
                        help="the name the endpoint answers to for the base model")
    parser.add_argument("--tuned-model", required=True,
                        help="the name the endpoint answers to for the fine-tune")
    parser.add_argument("--quant", required=True,
                        help="the quantisation both models are serving at; a comparison "
                             "across two quantisations measures the quantisation as well")
    parser.add_argument("--engine", default="llama.cpp")
    parser.add_argument("--engine-version", default="unknown")
    parser.add_argument("--system", default=None,
                        help="system prompt sent with every task; use the one you trained with")
    parser.add_argument("--tasks", action="append", required=True,
                        help="a task file; repeat the flag for more than one")
    parser.add_argument("--judge-model", default=None,
                        help="grade both runs with this model; leave out for checks only")
    parser.add_argument("--judge-url", default=None,
                        help="endpoint for the judge, if it is not at --base-url")
    parser.add_argument("--out-dir", default="eval-out")
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--notes", default=None)
    args = parser.parse_args()

    script_dir = Path(args.harness_dir).expanduser()
    for name in ("run-eval.py", "judge.py") if args.judge_model else ("run-eval.py",):
        if not (script_dir / name).is_file():
            raise SystemExit(f"{script_dir / name} not found; --harness-dir must point at the "
                             "directory holding Part 10's harness scripts")
    if args.judge_model in (args.base_model, args.tuned_model):
        print("WARNING: the judge is one of the models under test. Part 10's lab covers "
              "self-preference bias; use a third, larger model if you can.")

    out_dir = Path(args.out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)

    comparison: dict[str, Any] = {}
    for tasks in args.tasks:
        tasks_path = Path(tasks)
        if not tasks_path.is_file():
            raise SystemExit(f"{tasks_path} does not exist")
        print(f"\n=== {tasks_path.name}: base model {args.base_model}")
        base = evaluate(args, script_dir, out_dir, tasks_path, args.base_model, "base")
        if args.pause_before_tuned:
            input("Base results saved. Stop the base server, start the fine-tune, "
                  "verify its readiness, then press Enter: ")
        print(f"\n=== {tasks_path.name}: fine-tune {args.tuned_model}")
        tuned = evaluate(args, script_dir, out_dir, tasks_path, args.tuned_model, "tuned")
        comparison[tasks_path.name] = {"base": base, "tuned": tuned}
        if args.pause_before_tuned and tasks != args.tasks[-1]:
            input("Fine-tune results saved. Restore the base server for the next task file, "
                  "verify readiness, then press Enter: ")

    print("\n" + "=" * 78)
    print("BASE VERSUS FINE-TUNE, same tasks, same settings, same quantisation")
    print("=" * 78)
    for name, pair in comparison.items():
        base, tuned = pair["base"], pair["tuned"]
        print(f"\n{name}   ({base['tasks']} task(s))")
        print(f"  deterministic checks   base {base['checks_passed']:>3}/{base['tasks']}"
              f"   fine-tune {tuned['checks_passed']:>3}/{tuned['tasks']}"
              f"   change {tuned['checks_passed'] - base['checks_passed']:+d}")
        if base["judge_mean"] is not None:
            print(f"  judge mean             base {base['judge_mean']:>5}"
                  f"   fine-tune {tuned['judge_mean']:>5}"
                  f"   change {delta(tuned['judge_mean'], base['judge_mean'])}")
        categories = sorted(set(base["by_category_checks"]) | set(tuned["by_category_checks"]))
        for category in categories:
            b_pass, b_total = base["by_category_checks"].get(category, (0, 0))
            t_pass, t_total = tuned["by_category_checks"].get(category, (0, 0))
            line = (f"    {category:14s} checks {b_pass}/{b_total} -> {t_pass}/{t_total}"
                    f"  ({t_pass - b_pass:+d})")
            if base["by_category_judge"] or tuned["by_category_judge"]:
                b_judge = base["by_category_judge"].get(category)
                t_judge = tuned["by_category_judge"].get(category)
                line += f"   judge {b_judge} -> {t_judge} ({delta(t_judge, b_judge)})"
            print(line)

    print("\nRead the categories, not the total. A fine-tune that gains on the behaviour you")
    print("trained and loses on two others can leave the overall mean unchanged, and that")
    print("trade is the finding. Anything smaller than the difference between two runs of")
    print("the base model at these settings is not a difference.")

    if args.labbook:
        record = sftlog.record(
            labbook=args.labbook,
            lab="part-13/evaluate-against-base",
            model=f"{args.tuned_model} vs {args.base_model}",
            dataset={"task_files": args.tasks,
                     "sha256": {t: sftlog.file_sha256(t) for t in args.tasks}},
            hyperparameters={"base_url": args.base_url,
                             "tuned_url": args.tuned_url or args.base_url, "quant": args.quant,
                             "engine": args.engine, "engine_version": args.engine_version,
                             "judge_model": args.judge_model, "system_prompt": args.system},
            seed=0,
            losses={},
            scores={name: {
                "base_checks_passed": pair["base"]["checks_passed"],
                "tuned_checks_passed": pair["tuned"]["checks_passed"],
                "tasks": pair["base"]["tasks"],
                "base_judge_mean": pair["base"]["judge_mean"],
                "tuned_judge_mean": pair["tuned"]["judge_mean"],
                "base_by_category_judge": pair["base"]["by_category_judge"],
                "tuned_by_category_judge": pair["tuned"]["by_category_judge"],
            } for name, pair in comparison.items()},
            config_path=__file__,
            notes=args.notes,
        )
        print(f"\nrecorded comparison {record['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download evaluate-against-base.py244 lines

Serve the fine-tune alongside the base, as the lab did, and run both task files through both models.

RunnableAll tracks

the comparison that the report is built from

python3 evaluate-against-base.py \
    --harness-dir ~/eval \
    --base-url http://127.0.0.1:8080/v1 \
    --base-model qwen3-8b \
    --tuned-model qwen3-8b-domain \
    --quant Q4_K_M \
    --engine llama.cpp \
    --engine-version v0.4.0 \
    --system "You answer questions about our deployment runbooks." \
    --tasks domain-tasks.json \
    --tasks my-tasks.json \
    --judge-model qwen3-14b \
    --out-dir eval-out \
    --labbook labbook.md

Two things about that command are the whole project. --system passes the same system prompt the training examples used, because serving with a different one is a change you never trained. --judge-model is a third model, larger than both, because Part 10 established that a model grading its own output scores itself generously.

=Expected result

The shape to expect: a clear gain on the domain set, driven mostly by the deterministic checks, and something between “no change” and “a small loss” on your own Part 10 set. A large loss on your own set with a large gain on the domain set is the trade the first lesson described, and it is a finding rather than a failure. A gain on both is worth double-checking for contamination before you celebrate.

9. Export, serve and publish

RunnableAll tracks

merge, convert, quantise, and prove the merge

bash export-gguf.sh runs/domain-qwen3-8b Q4_K_M

Add it to llama-swap.yaml under a name of its own next to the base model, exactly as the lab did, and re-run one task file through the gateway name to confirm the served model is the one you measured.

If you publish it, publish it privately. The Hub documents that a private repository stays out of other users’ search results, returns 404 - Repo not found to other users and cannot be cloned by them; create it private in the web interface before the first upload rather than letting the upload create it for you. Include a model card naming the base model, its licence, the dataset’s provenance and the run identifier.

10. Write the report

Fragment — not complete on its own

report-template.md

# Fine-tune report: <your model name>

<!--
Purpose: the written deliverable of Part 13's project. One page that lets somebody else, or you
         in six months, decide whether to use this model, reproduce this result, or throw it away.
Platform: all
Minimum memory: not applicable; this is a document
Assumes: the run records that labbook.md accumulated, and the comparison that
         evaluate-against-base.py printed. Fill in every angle-bracket placeholder and delete
         every comment block, including this one. A field you cannot fill is a field to write
         "not recorded" in, not one to delete: the gap is information.

Keep it to one page. The numbers live in the tables, and the tables carry their own context,
which is the course's rule about numbers applied to your own work.
-->

**Author:** <you> &nbsp; **Date:** <YYYY-MM-DD> &nbsp; **Track:** <S, X, M or N>
**Machine:** <chip and memory, from the hardware reference>

## What this model is for

<!-- Two or three sentences. The domain, the job, and who would use it. If a reader cannot tell
     from this paragraph whether the model is relevant to them, it is too vague. -->

## The claim

<!-- One sentence with a number in it, written before training and unchanged afterwards. The
     shape: "On <task set>, at temperature <t> with seed <s>, <fine-tune> passes <n> more of the
     deterministic checks than <base> and its judge mean in <categories> is not lower." -->

## Base model and licence

| Field | Value |
| --- | --- |
| Base model | <repository id> |
| Parameters | <from the model card> |
| Licence | <from the model card; a fine-tune is a derivative work> |
| Chat template | <the tokeniser it came from, and whether you changed it> |

## Data

| Field | Value |
| --- | --- |
| Source | <your own work, generated by a teacher and reviewed, transformed from a file> |
| Licence and confidentiality | <what the source material allows> |
| Redaction | <what was replaced before the trainer saw it, or "none needed"> |
| Training examples | <n> |
| Validation examples | <n> |
| Held-out task file | <name and task count> |
| Training file SHA-256 | <from the run record> |
| Decontamination | <what decontaminate.py reported against your Part 10 file, and what you did> |

## Training run

| Field | Value |
| --- | --- |
| Run id | <from labbook.md> |
| Method | <LoRA or QLoRA; 4-bit quantisation type if any> |
| Rank / alpha / dropout | <r> / <a> / <d> |
| Target modules | <the list, or all-linear> |
| Learning rate / schedule | <lr> / <scheduler and warm-up> |
| Epochs / effective batch | <e> / <batch times accumulation> |
| Sequence length | <max_length> |
| Best epoch | <n, and what that says about the data> |
| Wall-clock | <hh:mm> |
| Peak memory | <GB, measured, or "not recorded"> |
| Package versions | <torch, transformers, trl, peft, from the run record> |

## Results

<!-- One row per model per task set. Both models at the same quantisation, the same sampling
     settings and the same system prompt, or the comparison is measuring two things at once. -->

| Task set | Model | Tasks | Checks passed | Judge mean | Change |
| --- | --- | --- | --- | --- | --- |
| <domain-tasks.json> | base | <n> | <n> | <x.xx> | - |
| <domain-tasks.json> | fine-tune | <n> | <n> | <x.xx> | <+/-> |
| <my-tasks.json> | base | <n> | <n> | <x.xx> | - |
| <my-tasks.json> | fine-tune | <n> | <n> | <x.xx> | <+/-> |

**Settings:** temperature <t>, top-p <p>, seed <s>, max tokens <n>.
**Engine and version:** <engine> <version>. **Quantisation served:** <the same for both rows>.
**Judge:** <model, or "deterministic checks only">.
**Judge agreement with you:** <from judge.py human, or "not measured">.
**Noise floor:** <the difference between two runs of the base model at these settings>.

### Per category

| Category | Base | Fine-tune | Change |
| --- | --- | --- | --- |
| <category> | <x.xx> | <x.xx> | <+/-> |

<!-- Every category from your own Part 10 set belongs here, including the ones that did not
     move. A category that got worse and is missing from this table is the one thing that would
     make this report dishonest. -->

## What got worse

<!-- Name it. Every fine-tune trades something, and a report that claims nothing regressed
     either measured too little or is not looking. If genuinely nothing moved outside the noise
     floor, say that and give the noise floor. -->

## Export

| Field | Value |
| --- | --- |
| Merged into | <base model and precision> |
| Merge comparison | <n of n prompts matched the adapter-attached model> |
| Formats produced | <GGUF and type, MLX, safetensors, AWQ, FP8> |
| Served as | <the gateway name> |
| Published to | <private repository, or "kept local"> |

## Verdict

<!-- One of three, and say which: use it, keep iterating, or discard it. Then one sentence of
     reason, and one sentence naming what you would change first if you ran it again. -->

## Reproducing this

<!-- The exact commands, in order, from dataset to served model. Someone with your data and
     your machine should be able to paste these and get a comparable result. -->

```sh
# dataset
# decontamination
# training
# evaluation
# export
```

Download report-template.md129 lines

One page. Fill in every field, including the ones that make the result look worse, and delete the comment blocks. The section that separates a report from an advertisement is What got worse: a fine-tune trades something, and a report claiming nothing regressed has either measured too little or is not looking.

Build the specialist’s evidence package before deployment

Write the domain claim and acceptance threshold before generating training data. Include what the assistant should do when evidence is absent. Bring forward the frozen personal task set from Part 10 and keep the specialist’s held-out task file separate from generated demonstrations.

Run the base evaluation first and retain task-level outputs. During data review, check every generated target against its source and keep rejection reasons. After training, compare the candidate at the same template, precision and sampling settings, inspecting domain gains and general regressions in separate columns.

Use a new export directory for every candidate so a previous merge cannot be reused accidentally. Record the adapter’s base identity and compare the actual served representation before publishing an alias. The report should include dataset provenance, split policy, training configuration, raw results, export commands and the alias mapping. If the specialist needs current documents, document the retrieval update path as well; fine-tuning is not a substitute for keeping evidence current. Keep the base alias available and demonstrate rollback with a representative request. That completes the project’s operational handoff, beyond producing a checkpoint that happens to answer in the domain’s style.

Validation

You are done when all of the following are true:

• the claim was written in the report before training and is unchanged;
• results-base-1.json and results-base-2.json exist, and you can state the noise floor;
• every training example either was written by you or carries "accepted": true after you read it;
• decontaminate.py passed --strict against your Part 10 task file;
• labbook.md holds the training run record with the dataset checksum, every hyperparameter, the losses and the peak memory;
• the trainable-parameter percentage, the best epoch and the peak memory are all explained in the report;
• evaluate-against-base.py has produced a comparison over both task files with a third-model judge, and it is in labbook.md;
• a merged, converted and quantised file exists, the merge comparison matched, and the model answers under its own gateway name;
• the report is complete, one page, with the What got worse section filled in.

Expected outcome

Pending validationSpecialist assistant, base versus fine-tune — your recording sheet

Task set	Model	Tasks	Checks passed	Judge mean	Change
domain-tasks.json	base	—	—	—	—
domain-tasks.json	fine-tune	—	—	—	—
my-tasks.json	base (run 1)	—	—	—	—
my-tasks.json	base (run 2)	—	—	—	—
my-tasks.json	fine-tune	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp, or mlx-lm on Track M the engine version, and the package versions from the run record · your base model and the adapter run id from labbook.md, the same quantisation for every row · 8,192 tokens of context · the date you ran it

Empty on purpose. Two runs of the base model on your own task set give the noise floor; a change in the fine-tune row smaller than the gap between those two rows is not a change. The validation pass replaces this with measured values per track.

Pending validationTraining cost, estimated then measured — your recording sheet

Track	Base model	Method	Estimated peak GB	Measured peak GB	Wall-clock
S	—	—	—	—	—
X	—	—	—	—	—
M	—	—	—	—	—
N	—	—	—	—	—

one row per machine you run this on · PyTorch with Transformers, TRL, PEFT and bitsandbytes; mlx-lm on Track M the package versions sftlog.py recorded · as listed, 4-bit NF4 base with double quantisation, or BF16 where stated · 1,024 tokens of context · the date you ran it

Empty on purpose. The estimate comes from Part 11's arithmetic and is made before the run; the measurement is what train-qlora.py printed. Both belong in the report, because a systematic gap between them is a fact about your configuration worth knowing.

A model you would actually put behind your gateway, and a page that says why. The uncomfortable finding most people get is that the domain gain is smaller than they expected and the format gain is larger, which is the first lesson’s argument arriving as a measurement rather than as advice.

Troubleshooting

The script stops saying bitsandbytes is not available. That is deliberate. On macOS there is no 4-bit GPU path, and the message names the alternatives: --no-4bit with a smaller model, or mlx-lm against an already-quantised MLX model.

The dataset comes out empty after --from-pairs drafts.jsonl. Every record still has "accepted": null, so all of them were skipped. That is the review gate doing its job. Read them and set the field.

The teacher’s drafts are all in the wrong shape. The script only keeps lines that parse as JSON objects with string input and output. If nothing survives, the teacher is wrapping its output in prose or code fences; a smaller --per-chunk, a more explicit instruction, or a model that follows format better all help, and Part 10’s structured-output lesson covers the general fix.

Out of memory at a step some way into the run. The longest example in the dataset arriving in a batch. Lower --max-length, or --batch-size with a matching rise in --grad-accum. Check the length distribution of your data rather than lowering settings until it stops failing.

The fine-tune wins on both task files by a lot. Check for contamination before you believe it. Run decontaminate.py again with a lower --threshold and read what it flags. A large gain on a set that was supposed to be independent is the signature of overlap.

The judge scores the fine-tune lower while the deterministic checks say it improved. Read three answers. Usually the fine-tune has learned a terser style that the rubric does not ask for, and the judge is scoring length. Part 10’s lab covers verbosity bias, and the mean_words_by_score diagnostic is where to look.

The gateway serves an older version. llama-swap caches a running process; stop it, restart with the edited configuration, and re-run one task file to confirm the name maps to the file you think.

Cleanup

Keep the adapter, the dataset, both task files, the report and the notebook.

RunnableAll tracks

archive the project

mkdir -p ~/finetunes/specialist
cp -r runs/domain-* data domain-tasks.json labbook.md report.md ~/finetunes/specialist/
rm -f ~/models/domain-qwen3-8b-bf16.gguf

⚠Warning

The rm removes the intermediate full-precision GGUF, which is several gigabytes and is regenerable from the merged directory. Do not remove the merged directory or the quantised file: one is what you convert from next time and the other is what your gateway is serving.

What you learned

• The claim comes first. Written before training, it is a test that could fail; written afterwards, it is a description of whatever happened.
• Two runs of the base model are the noise floor. Without it, “it improved” is a sentence about a number you cannot interpret.
• A review gate is what separates generated data from training data. The rejections tell you more about your prompt to the teacher than the acceptances tell you about the teacher.
• QLoRA is a memory technique, not a quality technique. It quantises the part that is frozen and changes nothing about the adapter, and the paper reports 16-bit task performance preserved on the configuration it tested.
• Activations are the term you control. On a fixed model and method, sequence length and batch size are what decide whether the run fits, and sequence length is usually what to spend spare memory on.
• Estimate then measure. The gap between Part 11’s arithmetic and the peak memory the run reported is a fact about your configuration that no page can give you.
• The report is the deliverable. A model without one cannot be evaluated, reproduced or responsibly handed to anybody else.

Record in the notebook: the domain and the claim; the base model, licence and method; the dataset’s source, size, checksum and review outcome; the decontamination results against both task files; every hyperparameter and the run id; the estimated and measured peak memory; the full comparison table with its settings, quantisation and judge; the noise floor; what got worse; and your verdict in one word with one sentence of reason.

Check your understanding

Question 1. Why does this project ask you to run the base model twice on your own task set before training anything?

• To warm up the cache
• The difference between the two runs is the noise floor: any later change smaller than it is not a change
• To check the server is stable
• To average the two scores

Show the answer and why

Answer: The difference between the two runs is the noise floor: any later change smaller than it is not a change

Sampling, batching and server restarts all introduce variation. Without a noise floor there is no way to distinguish a real effect from run-to-run wobble, and a small apparent gain is the easiest thing in this subject to over-interpret.

Question 2. The teacher wrote 120 drafts and the dataset builder produced nothing. What happened?

• The teacher output was malformed
• Every draft still has "accepted": null, and unreviewed records are skipped by default
• The output directory is wrong
• The drafts were too long

Show the answer and why

Answer: Every draft still has "accepted": null, and unreviewed records are skipped by default

The review gate is deliberate: generated text is a draft until a person has read it. Set the field to true or false on every record, and treat a pattern in the rejections as a defect in your instruction to the teacher.

Question 3. A reader on a 16 GB Mac wants to run this project with a quantised base. What does the documentation support?

• bitsandbytes 4-bit loading, the same as on CUDA
• Not bitsandbytes: its installation page lists Apple silicon only in the CPU build table, so the Mac path is mlx-lm training adapters against an already-quantised MLX model, with --num-layers as the memory knob
• Nothing; the project cannot be done on a Mac
• FP8 through llm-compressor

Show the answer and why

Answer: Not bitsandbytes: its installation page lists Apple silicon only in the CPU build table, so the Mac path is mlx-lm training adapters against an already-quantised MLX model, with --num-layers as the memory knob

The page names Apple Silicon among officially supported platforms, but the only macOS arm64 entry in its build tables is under CPU. mlx-lm trains against quantised MLX models on the Mac GPU, which is this track's equivalent of QLoRA.

Question 4. Your fine-tune gains on the domain task set and loses two points of judge mean on the reasoning category of your Part 10 set. What should the report say?

• Report only the domain gain, since that was the goal
• Report both, with the noise floor, in the "What got worse" section, and state whether the trade is acceptable for the job
• Retrain until nothing regresses
• Drop the reasoning category from the task set

Show the answer and why

Answer: Report both, with the noise floor, in the "What got worse" section, and state whether the trade is acceptable for the job

The trade is the finding. Catastrophic forgetting on abilities your examples do not exercise is expected, and a report that hides it is the one thing that would make the exercise dishonest. Whether the trade is acceptable depends on the job, which is why the report has a verdict.

Question 5. Which of these are correct about the QLoRA configuration this project uses? Select all that apply.

• The frozen base is quantised to 4-bit NormalFloat with double quantisation
• The adapters remain trainable at higher precision
• target_modules="all-linear" adapts every linear layer, which is the configuration the QLoRA paper describes
• The adapter should be merged into the quantised base to keep the memory saving

Show the answer and why

Answer: The frozen base is quantised to 4-bit NormalFloat with double quantisation, The adapters remain trainable at higher precision, target_modules="all-linear" adapts every linear layer, which is the configuration the QLoRA paper describes

The last is the fault this part's challenge reproduces. Merge into the base at bfloat16 and quantise the merged result afterwards; adding a low-rank update to weights that were already rounded applies it to the wrong numbers.

Check my answersClear