Lab: Fine-Tune a Small Model on Your Own Agent Trajectories and Measure the Gain
Validated on: written from the documentation cited above; not yet validated on hardware on any track. The base model, tool versions, per-track wall-clock and the before-and-after numbers belong here once the validation pass has run this lab on real machines.
Objective
Section titled “Objective”By the end of this lab you will have a small model that has been fine-tuned on trajectories your own agent produced, serving under its own name in the gateway next to the model it came from, and a table that says what improved, what got worse, and whether the difference is bigger than the run-to-run spread.
The deliverable is the table. A fine-tune with no before-and-after is a story; a fine-tune with three instruments run against both models at identical settings is a result, and it is the result the capstone asks you to defend.
The lab, end to end
- Measure the baseReliability test and agent suite, before you change anything. Without this there is no comparison.
- CollectRun the agent over the collection task set and normalise the transcripts into episodes.
- ScrubRedact what has a shape, add your own patterns, verify the output.
- Filter and formatOutcome, step budget, duplicates, decontamination against the suite, then both training layouts.
- InspectRender one row through the model’s chat template and read it. Two minutes.
- TrainLoRA with the loss on the assistant turns. Unattended.
- Export and nameMerge, convert, quantise, and add a gateway alias beside the base one.
- Measure bothThe same three instruments against both aliases, with repeats.
- ReportThe gain, the regressions, and whether the spread allows either claim.
Requirements
Section titled “Requirements”Seventy-five minutes, of which roughly thirty-five are attended. The unattended parts are the trajectory collection, the training run and the export. The memory floor is 16 GB: a 1.7-billion-parameter base with a rank-16 adapter at a 4,096-token sequence, with the gateway stopped during training.
You need, from earlier parts: the Part 9 gateway with at least one model serving under an alias
and configured for tool calling; the training environment from Part 11 for your track; a
llama.cpp checkout built as in Part 6 for the export; Part 13’s export-gguf.sh and
merge-adapter.py; Part 24’s minimal-agent.py, toolbox.py, tool-call-reliability.py and
tool-prompts.json; and Part 26’s agent-eval.py, scaffold-minimal.py and
agent-tasks.json. Copy the three sample documents from Part 10 into a workspace directory, or
point the agent at your own.
If you already have the Part 25 workstation in daily use, the tasks you give its coding agents are the right collection set, because the trajectories worth training on are the ones from work you actually do. Those tools each write their own session format, so this lab runs the tasks through Part 24’s loop instead, which writes the transcript shape the collector reads without needing an adapter per tool. Swap in your own tasks over your own repository at task 2 and the rest of the lab is unchanged.
Four scripts come from the lessons in this part rather than from this page:
collect-trajectories.py and scrub-trajectories.py and trajectories-to-sft.py from
the first lesson,
and train-agent-sft.py from
the second.
Put them all in one directory with the five files below.
About 25 GB of free disk: the base model, the merged model and two GGUF files.
The seventy-five minutes divide differently per track. On Tracks S and N the training run is
the longest unattended block and the collection run is second. On Track X the same is true with
a longer training block, because the ROCm path has not been timed by the validation pass and
the sensible plan is the smaller base. On Track M the training is mlx_lm.lora rather than the
PyTorch script, and the export gains a fusing step, so the attended share is a little higher.
Write your own wall-clock per stage into the notebook as you go: the per-track table in the
report is one of the few numbers in this course that nobody else can supply for you.
The base models are Apache-2.0 according to their cards, and the model reference records the licence for every model this course names. Pick by memory tier:
| Your memory | Base model | Sequence length | Why |
|---|---|---|---|
| 16 GB | Qwen3-1.7B | 4,096 | The floor. Trains in minutes and tool-call discipline still moves visibly. |
| 24 GB | Qwen3-4B | 4,096 | Enough capacity that the habits hold on tasks unlike the training ones. |
| 32 GB and above | Qwen3-4B | 6,144 or 8,192 | Spend the room on sequence length before you spend it on rank: truncation costs you the answer. |
Track S — NVIDIA DGX Spark
The primary path. TRL and PEFT on CUDA with transformers 5.16.1 · verified 2026-09-08, TRL 1.12.0 · verified 2026-09-08 and PEFT 0.20.0 · verified 2026-09-08 from Part 11’s environment lesson. With 128 GB of unified memory you can leave the gateway running during training, which makes task 8 a matter of adding an alias rather than juggling processes.
The 4-billion-parameter base is comfortable here, and so is a longer sequence. If your
trajectories are long, raise --max-length before you raise --rank.
Track X — AMD Ryzen AI Max+ 395Partial
TRL and PEFT on the ROCm PyTorch build is the primary path here and is expected to work; it has not been exercised on this chip by the validation pass, and the GPU-visible share of memory is smaller than the machine total.
Use TRL and PEFT on the ROCm PyTorch build from Part 11. PyTorch reports a ROCm device as
cuda, so train-agent-sft.py needs no changes and its device line will say cuda.
The memory point from Part 5 applies to training as it does to inference: the budget is the GPU’s share, not the machine’s total. Stop the gateway before you train, and start with the 1.7-billion-parameter base at 4,096 tokens even if the machine has 64 GB or more.
Track M — Apple silicon
Training is mlx_lm.lora on the data-mlx layout that trajectories-to-sft.py writes
alongside the TRL one, using mlx-lm 0.31.3 · verified 2026-09-08. Its documentation, read on
2026-09-09, shows a tools-format example whose tool calls carry the arguments as a JSON
string, which is why the converter writes a second layout rather than one.
Two flags matter. --num-layers is the memory knob, documented with a default of 16.
--mask-prompt computes the loss on the completion only; it is the nearest thing on this
track to the assistant-only mask the other tracks use and it is not identical on a
multi-turn trajectory, so record which you used. Export is mlx_lm.fuse rather than a PEFT
merge, and the fused model then goes through the same conversion.
Track N — NVIDIA desktop or laptop
The primary path. TRL and PEFT on CUDA. On a 16 GB card use Qwen3-1.7B at 4,096 tokens with batch 1 and gradient accumulation 8; on 24 GB and above, Qwen3-4B at the same settings.
Inside WSL2, remember Part 6’s warning about the virtual machine’s memory limit: the trainer sees the WSL2 allocation, not the Windows total, and an out-of-memory error that contradicts your arithmetic is usually that.
Working directory and terminal roles
Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:
RunnableAll tracks
export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"export LAB_DIR="$LABS_ROOT/part-27-improving-models-for-agents"cd "$LAB_DIR"pwdtest -f "collection-tasks.json"Expected result: pwd ends in part-27-improving-models-for-agents and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.
Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.
1. Measure the base model, before you change anything
Section titled “1. Measure the base model, before you change anything”Five minutes, and the step people skip. Everything after this is only meaningful relative to these two numbers.
RunnableAll tracks
GATEWAY=http://127.0.0.1:4000/v1P24=../part-24-tools-mcp-and-the-agent-loopP26=../part-26-building-agent-systems
python3 "$P24/tool-call-reliability.py" \ --base-url "$GATEWAY" --model local/agent \ --repeat 5 --temperature 0 --labbook labbook.md
python3 "$P26/agent-eval.py" \ --agent "$P26/scaffold-minimal.py" \ --tasks "$P26/agent-tasks.json" \ --base-url "$GATEWAY" --model local/agent \ --workspace ./agent-workspace --repeats 3 \ --out before-agent.json --labbook labbook.mdOutput — what you should see
call rate 100.0%parse rate 93.3%right tool 86.7%schema valid 80.0%arguments correct 73.3%false calls 20.0%
success rate 0.62 (28 of 45)mean steps 4.31Those figures are an illustration of the shape of the output, not a result: yours will differ and the point of writing them into the notebook is that they are yours. Read the per-case failures from the reliability test before moving on. They tell you what the collection set should exercise.
2. Collect trajectories
Section titled “2. Collect trajectories”Fifteen minutes, mostly unattended while the agent works.
RunnableAll tracks
{ "name": "Part 27 trajectory collection set", "version": "1.0.0", "comment": "Sixteen tasks over the same three course-authored documents Part 26's suite uses (machine-inventory.md, model-policy.md and service-runbook.md from src/labs/part-10-models-at-work/sample-docs/), asking different questions on purpose. This is the set you COLLECT trajectories from; Part 26's agent-tasks.json is the set you MEASURE with, and the two must not overlap or the measurement is of memorisation. trajectories-to-sft.py checks that for you and will tell you if you drift. Replace these with tasks over your own repository as soon as the pipeline runs end to end: a fine-tune on questions about three sample documents improves a model at answering questions about three sample documents, which is not what you want from it.", "workspace": "the three documents from src/labs/part-10-models-at-work/sample-docs/", "tasks": [ { "id": "c-front-end-port", "category": "retrieval", "task": "On which port is the chat front-end published to the house network? Give the number.", "expect_in_answer": ["443"], "max_steps": 6, "notes": "The runbook's service table lists four ports, so a wrong number is a reading failure rather than a refusal." }, { "id": "c-restart-seconds", "category": "retrieval", "task": "About how long does restarting one service take, and what accounts for most of that time?", "expect_regex": "(forty|40)", "expect_in_answer": ["model"], "max_steps": 6, "notes": "Two facts from one sentence. A trajectory that answers half of it is a useful negative example." }, { "id": "c-tern-model-disk", "category": "retrieval", "task": "How much NVMe storage does the always-on inference host set aside for models?", "expect_in_answer": ["2 tb"], "max_steps": 6, "notes": "The agent has to work out which machine is the always-on host before it can answer." }, { "id": "c-petrel-memory", "category": "retrieval", "task": "How much system memory does the workstation have?", "expect_in_answer": ["128"], "max_steps": 6, "notes": "Three memory figures appear in the inventory, so the wrong machine gives the wrong number." }, { "id": "c-skua-accelerator", "category": "retrieval", "task": "What accelerator does the cold-standby machine have?", "expect_regex": "(none|no accelerator)", "max_steps": 6, "notes": "An answer of 'none' is correct and short, which is exactly the shape a small model tends to pad." }, { "id": "c-free-space-floor", "category": "retrieval", "task": "The download rules set a floor on free space. What is the floor, and on which disk?", "expect_in_answer": ["300"], "max_steps": 6, "notes": "One clause of a four-part rule." }, { "id": "c-register-cadence", "category": "reasoning", "task": "How often is the model register reviewed, and what happens at a review to a model nobody has used?", "expect_regex": "(second month|every second)", "expect_in_answer": ["delete"], "max_steps": 8, "notes": "Two sentences that sit together. The exception clause is the trap." }, { "id": "c-remote-code", "category": "reasoning", "task": "A model you want requires custom code to load. Where may it be run, and under what conditions?", "expect_in_answer": ["petrel", "container"], "expect_not_in_answer": ["tern"], "max_steps": 8, "notes": "The forbidden string catches an answer that lists both machines to be safe." }, { "id": "c-standing-exception", "category": "reasoning", "task": "The model policy records exactly one standing exception. What is it, and what has to happen afterwards?", "expect_regex": "(speech|transcription)", "expect_in_answer": ["delete"], "max_steps": 8, "notes": "The word 'exception' appears twice in the document with different meanings." }, { "id": "c-retire-after", "category": "retrieval", "task": "After how long without being powered on is a machine retired?", "expect_regex": "(six months|6 months)", "max_steps": 6, "notes": "A single fact at the end of a document, which is where retrieval tends to miss." }, { "id": "c-oncall", "category": "reasoning", "task": "Who is on call overnight, and what is the stated reason for that arrangement?", "expect_regex": "(no one|nobody|no on-call|there is no)", "max_steps": 8, "notes": "The correct answer is a negative. Models that pattern-match on 'on-call' invent a rota." }, { "id": "c-log-fields", "category": "retrieval", "task": "Name two of the fields the gateway writes for each request, and say what it does not write.", "expect_regex": "(timestamp|token|latency|model)", "expect_in_answer": ["not"], "max_steps": 6, "notes": "The second half matters more than the first: the document is explicit about what is not logged." }, { "id": "c-external-drive", "category": "retrieval", "task": "How often is a copy of the backup taken off the machine it was written on?", "expect_regex": "(first sunday|monthly|each month|every month)", "max_steps": 6, "notes": "Two 'first Sunday' rules exist in different documents; only one is about backups." }, { "id": "c-front-end-stop", "category": "reasoning", "task": "Why is the front-end stopped during the backup window, and for roughly how long?", "expect_regex": "(sqlite|database)", "expect_in_answer": ["ninety"], "max_steps": 8, "notes": "Cause and duration from one paragraph." }, { "id": "c-grep-runlab", "category": "tools", "task": "Search the documents for the command that starts every service, and give the command exactly as written.", "expect_in_answer": ["runlab up"], "max_steps": 6, "notes": "A search task whose answer is a literal string, so a paraphrase fails the check." }, { "id": "c-count-md", "category": "tools", "task": "List the markdown files in your workspace and say how many of them there are.", "expect_regex": "(three|3)", "expect_in_answer": ["service-runbook.md"], "max_steps": 6, "notes": "The shortest successful trajectory is one listing and one finish, which makes it a good check that the collection loop works at all." } ]}Sixteen tasks over the same three documents Part 26’s suite uses, asking different questions on purpose. Replace them with tasks over your own repository as soon as the pipeline runs: a fine-tune on questions about three sample documents improves a model at answering questions about three sample documents.
The collector needs the tool list the agent was given, because the tool descriptions are part of the prompt. Part 24’s toolbox will print it.
RunnableAll tracks
P24=../part-24-tools-mcp-and-the-agent-loop( cd "$P24" && python3 -c "import json, toolbox; print(json.dumps(toolbox.Toolbox(workspace='.').schemas(), indent=2))" ) > tools.json
python3 "$P24/minimal-agent.py" \ --base-url http://127.0.0.1:4000/v1 --model local/agent \ --workspace ./agent-workspace \ --tasks collection-tasks.json \ --transcript-dir transcripts --labbook labbook.mdRun it three times so that each task has several attempts and the filter has something to choose between. Then normalise what it wrote:
RunnableAll tracks
python3 collect-trajectories.py --from part-24 \ --labbook labbook.md --tools tools.json \ --out raw/trajectories.jsonlOutput — what you should see
episodes: 48outcome passed: 31lossy: 0tool calls: 146tools: 6written to raw/trajectories.jsonl3. Scrub the transcripts
Section titled “3. Scrub the transcripts”Three minutes, and do not skip the second command.
RunnableAll tracks
python3 scrub-trajectories.py --in raw/trajectories.jsonl \ --out clean/trajectories.jsonl --report scrub-report.json \ --labbook labbook.md
python3 scrub-trajectories.py --in clean/trajectories.jsonl --checkOutput — what you should see
episodes in: 48episodes kept: 48episodes dropped: 0
redactions by pattern email 2 e.g. ops@example.… home-path 11 e.g. /home/user
clean/trajectories.jsonl: nothing matched. That is a floor, not a proof: read a sample.Now open scrub-report.json and read the samples. Then add your own patterns for the things a
regular expression cannot find on its own: a client name, an internal hostname, a project code
name.
RunnableAll tracks
python3 scrub-trajectories.py --in raw/trajectories.jsonl \ --out clean/trajectories.jsonl --report scrub-report.json \ --extra "client=\bNorthwind\b" \ --extra "internal-host=\b[a-z-]+\.corp\.invalid\b"4. Filter, decontaminate and format
Section titled “4. Filter, decontaminate and format”Three minutes, and the output to read is the report rather than the data files.
RunnableAll tracks
python3 trajectories-to-sft.py \ --in clean/trajectories.jsonl --out-dir . \ --tasks ../part-26-building-agent-systems/agent-tasks.json \ --drop-repeated-calls --max-steps 10 \ --report filter-report.json --labbook labbook.mdOutput — what you should see
episodes in: 48train rows: 23 from 13 task(s)valid rows: 4 from 3 task(s)assistant turns: 71tool calls: 104
rejected by reason outcome-not-passed 17 repeated-identical-call 3 over-step-budget 1
report written to filter-report.jsonThe two settings on that command line are choices, not defaults, and they are worth making
deliberately. --drop-repeated-calls throws away any episode that made the same call with the
same arguments twice, which is the signature of an agent that lost its place; those episodes
finished correctly and teach padding. --max-steps 10 throws away the ones that wandered.
Both make the training set smaller, which is uncomfortable when you have forty-eight episodes
and the temptation is to keep everything. Keep everything once, look at the longest episode
you kept, and the temptation goes away.
Read filter-report.json. The counts by reason are the argument for the settings you chose,
and the sample of each rejected reason is how you find out that a filter is throwing away
something you wanted. If nothing survives, the two usual causes are that no episode passed,
which means the collection tasks are too hard for the base model, and that every collection
task overlaps the evaluation suite, which means you edited one of the two files.
5. Read one row through the model’s own template
Section titled “5. Read one row through the model’s own template”Two minutes, and it is the highest-value two minutes in the lab.
RunnableAll tracks
python3 train-agent-sft.py --model Qwen/Qwen3-1.7B --inspect-onlyOutput — what you should see
<|im_start|>system# Tools
You may call one or more functions to assist with the user query.…--- 1873 token(s) for this example; --max-length is 4096Three things to check, in this order. The tool schemas appear near the top, which means the
tools column reached the template. The tool calls appear in the family’s own markup, which
means the template renders them. And the token count is comfortably under --max-length, which
means nothing is being truncated. If any of the three is wrong, fix it here: a run started now
would train for an hour on a prompt shape that will never occur at serving time.
6. Train
Section titled “6. Train”Twenty-five minutes, unattended, at the floor. Stop the gateway first on any machine where memory is tight.
Track S — NVIDIA DGX Spark
RunnableTrack S · DGX Spark
python3 train-agent-sft.py \ --model Qwen/Qwen3-4B \ --data-dir data --output-dir runs/agent-qwen3-4b \ --max-length 4096 --rank 16 --alpha 32 \ --epochs 3 --batch-size 1 --grad-accum 8 \ --loss assistant --gradient-checkpointing \ --labbook labbook.md --notes "trajectories from collection-tasks, 3 attempts each"Track X — AMD Ryzen AI Max+ 395Partial
The same command on the ROCm PyTorch build; not yet exercised on this chip by the validation pass.
RunnableTrack X · Ryzen AI Max+
python3 train-agent-sft.py \ --model Qwen/Qwen3-1.7B \ --data-dir data --output-dir runs/agent-qwen3-1.7b \ --max-length 4096 --rank 16 --alpha 32 \ --epochs 3 --batch-size 1 --grad-accum 8 \ --loss assistant --gradient-checkpointing \ --labbook labbook.md --notes "ROCm, gfx1151, first run"Track M — Apple silicon
RunnableTrack M · Apple silicon
mlx_lm.lora \ --model Qwen/Qwen3-1.7B \ --train --data data-mlx \ --fine-tune-type lora --num-layers 16 \ --batch-size 1 --iters 600 --learning-rate 1e-4 \ --mask-prompt \ --adapter-path runs/agent-qwen3-1.7b-mlxRecord that you used --mask-prompt rather than an assistant-only mask, and record
--num-layers. Both belong in the run record, because both change what was trained.
Track N — NVIDIA desktop or laptop
RunnableTrack N · NVIDIA GPU
python3 train-agent-sft.py \ --model Qwen/Qwen3-1.7B \ --data-dir data --output-dir runs/agent-qwen3-1.7b \ --max-length 4096 --rank 16 --alpha 32 \ --epochs 3 --batch-size 1 --grad-accum 8 \ --loss assistant --gradient-checkpointing \ --labbook labbook.md --notes "16 GB card, 1.7B base"Output — what you should see
device: cuda precision: bfloat16train rows: 23 validation rows: 4 tools column: tools-as-json-objectstrainable params: 17,432,576 || all params: 1,738,006,528 || trainable%: 1.0030…{ "first_train_loss": 1.2814, "final_train_loss": 0.4417, "best_eval_loss": 0.6902, "best_epoch": 2.0, "seconds": 1180.4}The trainable percentage is the first thing to read. If it is far from what the adapter
arithmetic predicted, the target module names did not match your base model and nothing was
attached; --list-modules prints the names the model actually has.
The second thing to read is best_epoch. If the best epoch is the first, the dataset is small
for three epochs or the learning rate is too high, and the honest response is to look at the
curves before training again rather than to train again immediately.
7. Export and give it a name
Section titled “7. Export and give it a name”Ten minutes, mostly unattended.
RunnableAll tracks
#!/usr/bin/env bash# Purpose: turn the adapter this part trained into a served alias: run Part 13's# export-gguf.sh to merge, convert, quantise and smoke-test it, then write the# llama-swap and LiteLLM snippets that give the result a name next to the base# model, so the comparison in this part's lab can address both at once# Platform: spark, strix, nvidia (and mac, where the adapter comes from mlx_lm.lora and is# fused with mlx_lm.fuse first; see the lab page for that path)# Minimum memory: 16 GB; the merge inside export-gguf.sh needs one bfloat16 copy of the base# Assumes: Part 13's export-gguf.sh and merge-adapter.py in $PART13_DIR; a llama.cpp# checkout holding convert_hf_to_gguf.py at $LLAMA_CPP; llama-quantize and# llama-cli on PATH; and the gateway from Part 9, whose llama-swap.yaml and# litellm-config.yaml this script writes snippets for rather than editing.## Usage: bash export-and-alias.sh ADAPTER_DIR ALIAS [QUANT]# ADAPTER_DIR what train-agent-sft.py saved, e.g. runs/agent-qwen3-4b# ALIAS the gateway name the fine-tune will answer to, e.g. local/agent-tuned# QUANT a type llama-quantize accepts; defaults to Q4_K_M## Environment: PART13_DIR (default ../part-13-supervised-fine-tuning), LLAMA_CPP# (default $HOME/llama.cpp), OUT_DIR (default $HOME/models), SNIPPET_DIR# (default ./gateway-snippets), CTX (default 16384), SKIP_EXPORT=1 to write# only the snippets for a GGUF you already have.set -euo pipefail
ADAPTER="${1:-}"ALIAS="${2:-}"QUANT="${3:-Q4_K_M}"
PART13_DIR="${PART13_DIR:-../part-13-supervised-fine-tuning}"LLAMA_CPP="${LLAMA_CPP:-$HOME/llama.cpp}"OUT_DIR="${OUT_DIR:-$HOME/models}"SNIPPET_DIR="${SNIPPET_DIR:-./gateway-snippets}"CTX="${CTX:-16384}"
die() { echo "export-and-alias: $*" >&2; exit 1; }
[[ -n "$ADAPTER" ]] || die "usage: bash export-and-alias.sh ADAPTER_DIR ALIAS [QUANT]"[[ -n "$ALIAS" ]] || die "give the gateway alias as the second argument, e.g. local/agent-tuned"case "$ALIAS" in */*) : ;; *) die "the alias should look like local/agent-tuned so it reads as a role, not a file" ;;esac
NAME="$(basename "$ADAPTER")"GGUF="$OUT_DIR/$NAME-$QUANT.gguf"
if [[ "${SKIP_EXPORT:-0}" == "1" ]]; then echo "==> 1/3 Skipping the export (SKIP_EXPORT=1); expecting $GGUF"else [[ -f "$ADAPTER/adapter_config.json" ]] || die "$ADAPTER has no adapter_config.json; point at what train-agent-sft.py saved" [[ -f "$PART13_DIR/export-gguf.sh" ]] || die "export-gguf.sh not found under $PART13_DIR; set PART13_DIR to Part 13's lab directory" [[ -f "$LLAMA_CPP/convert_hf_to_gguf.py" ]] || die "convert_hf_to_gguf.py not found under $LLAMA_CPP; set LLAMA_CPP to your llama.cpp checkout" command -v llama-quantize >/dev/null || die "llama-quantize is not on PATH; build it as in Part 6"
echo "==> 1/3 Merging, converting, quantising and smoke-testing with Part 13's script" echo " Nothing about that step is specific to agents: an adapter is an adapter, and" echo " the reason it is one script for both parts is that it should stay one script." LLAMA_CPP="$LLAMA_CPP" OUT_DIR="$OUT_DIR" bash "$PART13_DIR/export-gguf.sh" "$ADAPTER" "$QUANT"fi
[[ -f "$GGUF" ]] || die "expected $GGUF and it is not there; read the export output above"SIZE=$(wc -c < "$GGUF" | tr -d ' ')echo " quantised export: $GGUF ($SIZE bytes)"
echo "==> 2/3 Writing the gateway snippets"mkdir -p "$SNIPPET_DIR"SAFE_NAME="${ALIAS//\//-}"SWAP_SNIPPET="$SNIPPET_DIR/llama-swap-$SAFE_NAME.yaml"LITELLM_SNIPPET="$SNIPPET_DIR/litellm-$SAFE_NAME.yaml"
# The chat template travels inside the GGUF, and --jinja is what makes llama-server use it# rather than a built-in guess. For a tool-calling fine-tune that flag is the whole point:# without it the model is served with a template that does not know what a tool call is.cat > "$SWAP_SNIPPET" <<EOF# Merge this block into the models: section of the llama-swap.yaml you wrote in Part 9.# It sits next to the base model rather than replacing it, because this part's comparison# addresses both aliases in the same run. $ALIAS: name: Agent fine-tune ($NAME) description: $NAME exported at $QUANT by Part 27's export-and-alias.sh cmd: | \${server} --model $GGUF --alias $ALIAS --ctx-size $CTX --jinja checkEndpoint: /healthEOF
cat > "$LITELLM_SNIPPET" <<EOF# Merge this entry into the model_list of the litellm-config.yaml you wrote in Part 9.model_list: - model_name: $ALIAS litellm_params: model: openai/$ALIAS api_base: os.environ/LLAMA_SWAP_BASE_URL api_key: os.environ/LOCAL_API_KEYEOF
echo " $SWAP_SNIPPET"echo " $LITELLM_SNIPPET"
echo "==> 3/3 What to do with them"if [[ -n "${APPEND_TO:-}" ]]; then [[ -f "$APPEND_TO" ]] || die "APPEND_TO=$APPEND_TO does not exist" if grep -qF "$ALIAS:" "$APPEND_TO"; then die "$APPEND_TO already mentions $ALIAS; edit it by hand rather than appending twice" fi cp "$APPEND_TO" "$APPEND_TO.before-$SAFE_NAME" # Only the block itself, without the two comment lines that explain it. tail -n +4 "$SWAP_SNIPPET" >> "$APPEND_TO" echo " appended the llama-swap block to $APPEND_TO" echo " the previous version is $APPEND_TO.before-$SAFE_NAME" echo " the LiteLLM entry is still yours to merge: $LITELLM_SNIPPET"else echo " Merge both snippets into your Part 9 gateway configuration, then reload it." echo " Set APPEND_TO=/path/to/llama-swap.yaml to have this script append the first" echo " one for you; it takes a copy first and refuses if the alias is already there."fi
echoecho "Then check that the alias answers, and that it answers with a tool call:"echo " curl -s \"\$GATEWAY/v1/models\" | grep -o '$ALIAS'"echo " python3 ../part-24-tools-mcp-and-the-agent-loop/tool-call-reliability.py \\"echo " --base-url \"\$GATEWAY/v1\" --model $ALIAS --repeat 1"echo "A fine-tune that serves but does not call tools is almost always the template: check"echo "that --jinja is on and that the tokeniser files were saved with the adapter."The script runs Part 13’s export-gguf.sh to merge the adapter into the base at bfloat16,
convert to GGUF, quantise and put one prompt through the result, then writes the two gateway
snippets so the fine-tune sits beside the base rather than replacing it.
Track S — NVIDIA DGX Spark
RunnableTrack S · DGX Spark
PART13_DIR=../part-13-supervised-fine-tuning \LLAMA_CPP="$HOME/llama.cpp" \bash export-and-alias.sh runs/agent-qwen3-4b local/agent-tuned Q4_K_MTrack X — AMD Ryzen AI Max+ 395
RunnableTrack X · Ryzen AI Max+
PART13_DIR=../part-13-supervised-fine-tuning \LLAMA_CPP="$HOME/llama.cpp" \bash export-and-alias.sh runs/agent-qwen3-1.7b local/agent-tuned Q4_K_MTrack M — Apple silicon
RunnableTrack M · Apple silicon
mlx_lm.fuse \ --model Qwen/Qwen3-1.7B \ --adapter-path runs/agent-qwen3-1.7b-mlx \ --save-path runs/agent-qwen3-1.7b-fused
SKIP_EXPORT=1 bash export-and-alias.sh \ runs/agent-qwen3-1.7b-fused local/agent-tuned Q4_K_Mmlx_lm.fuse produces a merged model rather than an adapter, so the GGUF conversion runs
against it directly with convert_hf_to_gguf.py from your llama.cpp checkout, and the
snippets are written with SKIP_EXPORT=1.
Track N — NVIDIA desktop or laptop
RunnableTrack N · NVIDIA GPU
PART13_DIR=../part-13-supervised-fine-tuning \LLAMA_CPP="$HOME/llama.cpp" \bash export-and-alias.sh runs/agent-qwen3-1.7b local/agent-tuned Q4_K_MMerge the two snippets into the Part 9 gateway’s llama-swap.yaml and litellm-config.yaml,
then reload it. The llama-swap block passes --jinja, which makes llama-server use the chat
template that travelled inside the GGUF rather than a built-in guess. For a tool-calling
fine-tune that flag is the difference between a model that calls tools and one that describes
calling them.
RunnableAll tracks
curl -s http://127.0.0.1:4000/v1/models | grep -o 'local/agent-tuned'
python3 ../part-24-tools-mcp-and-the-agent-loop/tool-call-reliability.py \ --base-url http://127.0.0.1:4000/v1 --model local/agent-tuned --repeat 18. Measure both models, with repeats
Section titled “8. Measure both models, with repeats”Fifteen minutes, unattended.
RunnableAll tracks
"""Score a fine-tuned agent model and the base it came from on the same suites, and report the gap.
Purpose: the only question this part asks. Runs Part 26's agent-eval.py over the same task suite twice, once against the base model and once against the fine-tune, runs Part 24's tool-call-reliability.py against both, optionally runs Part 10's run-eval.py over your general task set to find the regressions, and prints the difference per category so that a gain on the work you trained for and a loss on everything else are both visible at once. Appends one record to the lab notebook.Platform: all (pure Python over HTTP; both models are reached through an OpenAI-compatible API, so they may be served by any engine on any track, or by the Part 9 gateway)Minimum memory: 16 GB on the machine serving the models; this script needs very littleAssumes: Python 3.10 or newer. Part 26's agent-eval.py and agent-tasks.json, and Part 24's tool-call-reliability.py and tool-prompts.json, at the paths you pass. Both models answering under the aliases you give, one at a time or together. agentlog.py sits next to this file. Nothing here starts an engine: a script that starts models is a script that hides which model answered.
Usage: python3 compare-agent-models.py \\ --agent-eval ../part-26-building-agent-systems/agent-eval.py \\ --agent ../part-26-building-agent-systems/scaffold-minimal.py \\ --tasks ../part-26-building-agent-systems/agent-tasks.json \\ --reliability ../part-24-tools-mcp-and-the-agent-loop/tool-call-reliability.py \\ --base-url http://127.0.0.1:4000/v1 \\ --base-model local/agent --tuned-model local/agent-tuned \\ --workspace ./agent-workspace --repeats 3 \\ --out-dir compare-out --labbook labbook.md
Add --harness-dir ~/eval --general-tasks my-tasks.json to run Part 10's harness over your own general set as well. That is where a fine-tune's regressions show up, and leaving it out is how people ship one without noticing."""from __future__ import annotations
import argparseimport jsonimport subprocessimport sysfrom collections import defaultdictfrom pathlib import Pathfrom typing import Any
import agentlog
def run(command: list[str], capture: bool = False) -> str: """Run one child process, printing the command first so the log says what happened.""" print("+ " + " ".join(command)) done = subprocess.run(command, capture_output=capture, text=True, check=False) if capture and done.returncode not in (0, 2): sys.exit(f"{command[1]} exited {done.returncode}:\n{(done.stderr or '')[:800]}") return done.stdout if capture else ""
def agent_suite(args: argparse.Namespace, out_dir: Path, model: str, label: str) -> dict[str, Any]: """One model through Part 26's harness. Exit code 2 means some tasks failed, which is the normal case here and not an error.""" results_path = out_dir / f"agent-{label}.json" command = [ sys.executable, args.agent_eval, "--agent", args.agent, "--tasks", args.tasks, "--base-url", args.base_url, "--model", model, "--repeats", str(args.repeats), "--out", str(results_path), "--trajectory-dir", str(out_dir / f"trajectories-{label}"), "--quant", args.quant, "--engine", args.engine, "--engine-version", args.engine_version, "--notes", f"part-27 comparison, {label}", ] if args.workspace: command += ["--workspace", args.workspace] if args.index: command += ["--index", args.index] if args.cost_per_million is not None: command += ["--cost-per-million", str(args.cost_per_million)] subprocess.run(command, check=False) if not results_path.is_file(): sys.exit(f"{results_path} was not written; read the harness output above") return json.loads(results_path.read_text(encoding="utf-8"))
def reliability(args: argparse.Namespace, out_dir: Path, model: str, label: str) -> dict[str, Any]: """One model through Part 24's tool-calling reliability test, as JSON.""" command = [ sys.executable, args.reliability, "--base-url", args.base_url, "--model", model, "--repeat", str(args.reliability_repeat), "--temperature", str(args.temperature), "--json", ] if args.prompts: command += ["--prompts", args.prompts] output = run(command, capture=True) path = out_dir / f"reliability-{label}.json" path.write_text(output, encoding="utf-8") try: return json.loads(output) except json.JSONDecodeError: sys.exit(f"{args.reliability} did not print JSON; see {path}")
def general_suite(args: argparse.Namespace, out_dir: Path, model: str, label: str) -> dict[str, Any] | None: """Part 10's harness over your own general task set: the regression check.""" if not (args.harness_dir and args.general_tasks): return None script = Path(args.harness_dir) / "run-eval.py" if not script.is_file(): sys.exit(f"{script} does not exist; --harness-dir must hold Part 10's run-eval.py") results_path = out_dir / f"general-{label}.json" run([sys.executable, str(script), "--tasks", args.general_tasks, "--base-url", args.base_url, "--model", model, "--quant", args.quant, "--engine", args.engine, "--engine-version", args.engine_version, "--out", str(results_path)]) if not results_path.is_file(): print(f" {script.name} wrote no results for {label}; skipping the general comparison") return None return json.loads(results_path.read_text(encoding="utf-8"))
def by_category(payload: dict[str, Any]) -> dict[str, tuple[int, int]]: """Passes and attempts per category, from a Part 26 results file.""" counts: dict[str, list[int]] = defaultdict(lambda: [0, 0]) for row in payload.get("results", []): category = row.get("category", "uncategorised") counts[category][1] += 1 counts[category][0] += int(bool((row.get("checks") or {}).get("passed"))) return {k: (v[0], v[1]) for k, v in sorted(counts.items())}
def general_pass_rates(payload: dict[str, Any] | None) -> dict[str, tuple[int, int]]: """Passes and attempts per category from Part 10's harness, whose rows carry the same `category` and `checks.passed` fields.""" if not payload: return {} rows = payload.get("results", payload if isinstance(payload, list) else []) counts: dict[str, list[int]] = defaultdict(lambda: [0, 0]) for row in rows: if not isinstance(row, dict): continue category = row.get("category", "uncategorised") counts[category][1] += 1 counts[category][0] += int(bool((row.get("checks") or {}).get("passed"))) return {k: (v[0], v[1]) for k, v in sorted(counts.items())}
def fraction(pair: tuple[int, int] | None) -> float | None: if not pair or not pair[1]: return None return pair[0] / pair[1]
def show(value: float | None) -> str: return "n/a" if value is None else f"{value * 100:5.1f}%"
def delta(after: float | None, before: float | None) -> str: if after is None or before is None: return " n/a" difference = (after - before) * 100 return f"{difference:+6.1f}"
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--agent-eval", required=True, help="path to Part 26's agent-eval.py") parser.add_argument("--agent", required=True, help="the agent entry point to measure with") parser.add_argument("--tasks", required=True, help="Part 26's agent-tasks.json, or your own") parser.add_argument("--reliability", required=True, help="path to Part 24's tool-call-reliability.py") parser.add_argument("--prompts", default=None, help="Part 24's tool-prompts.json, if not beside it") parser.add_argument("--base-url", default="http://127.0.0.1:4000/v1") parser.add_argument("--base-model", required=True, help="alias the base model answers under") parser.add_argument("--tuned-model", required=True, help="alias the fine-tune answers under") parser.add_argument("--workspace", default=None) parser.add_argument("--index", default=None) parser.add_argument("--repeats", type=int, default=3, help="attempts per task; 3 or more, because one run of a suite this " "size cannot tell a real difference from a sampling one") parser.add_argument("--reliability-repeat", type=int, default=5) parser.add_argument("--temperature", type=float, default=0.0) parser.add_argument("--quant", default="unknown", help="the quantisation actually loaded; not discoverable over the API") parser.add_argument("--engine", default="unknown") parser.add_argument("--engine-version", default="unknown") parser.add_argument("--cost-per-million", type=float, default=None) parser.add_argument("--harness-dir", default=None, help="directory holding Part 10's run-eval.py") parser.add_argument("--general-tasks", default=None, help="your own general task set") parser.add_argument("--out-dir", default="compare-out") parser.add_argument("--labbook", default=None) parser.add_argument("--notes", default=None) args = parser.parse_args()
if args.base_model == args.tuned_model: sys.exit("--base-model and --tuned-model are the same alias; serve them under " "different names or the comparison measures nothing")
out_dir = Path(args.out_dir) out_dir.mkdir(parents=True, exist_ok=True)
print(f"\n=== agent suite: base ({args.base_model}) ===") base_agent = agent_suite(args, out_dir, args.base_model, "base") print(f"\n=== agent suite: fine-tune ({args.tuned_model}) ===") tuned_agent = agent_suite(args, out_dir, args.tuned_model, "tuned")
print("\n=== tool-call reliability: base ===") base_rel = reliability(args, out_dir, args.base_model, "base") print("\n=== tool-call reliability: fine-tune ===") tuned_rel = reliability(args, out_dir, args.tuned_model, "tuned")
base_general = general_suite(args, out_dir, args.base_model, "base") tuned_general = general_suite(args, out_dir, args.tuned_model, "tuned")
# ---------------------------------------------------------------- the agent suite base_cats, tuned_cats = by_category(base_agent), by_category(tuned_agent) print("\n\nAgent suite, success rate per category") print(f"{'category':<18}{'base':>10}{'fine-tune':>12}{'change':>10}") for category in sorted(set(base_cats) | set(tuned_cats)): before, after = fraction(base_cats.get(category)), fraction(tuned_cats.get(category)) print(f"{category:<18}{show(before):>10}{show(after):>12}{delta(after, before):>10}") base_all = base_agent["run"]["summary"]["success_rate"] tuned_all = tuned_agent["run"]["summary"]["success_rate"] print(f"{'all tasks':<18}{show(base_all):>10}{show(tuned_all):>12}{delta(tuned_all, base_all):>10}")
print("\nCost of an attempt") print(f"{'measure':<18}{'base':>10}{'fine-tune':>12}") for key, label in (("mean_steps", "mean steps"), ("mean_tokens", "mean tokens"), ("mean_seconds", "mean seconds")): before = base_agent["run"]["summary"].get(key) after = tuned_agent["run"]["summary"].get(key) print(f"{label:<18}{before!s:>10}{after!s:>12}")
# -------------------------------------------------------------------- reliability print("\nTool-call reliability (Part 24's fixed prompt set)") print(f"{'rate':<20}{'base':>10}{'fine-tune':>12}{'change':>10}") for key, label in (("call_rate", "call rate"), ("parse_rate", "parse rate"), ("right_tool_rate", "right tool"), ("schema_valid_rate", "schema valid"), ("args_correct_rate", "arguments correct"), ("false_call_rate", "false calls")): before = base_rel["rates"].get(key) after = tuned_rel["rates"].get(key) print(f"{label:<20}{show(before):>10}{show(after):>12}{delta(after, before):>10}") print("A rise in the false-call rate is a regression even when everything else improves: " "it means the model learned that calling a tool is always the right move.")
# ------------------------------------------------------------------- the general set regressions: list[str] = [] base_gen, tuned_gen = general_pass_rates(base_general), general_pass_rates(tuned_general) if base_gen or tuned_gen: print("\nGeneral task set (Part 10's harness), pass rate per category") print(f"{'category':<18}{'base':>10}{'fine-tune':>12}{'change':>10}") for category in sorted(set(base_gen) | set(tuned_gen)): before, after = fraction(base_gen.get(category)), fraction(tuned_gen.get(category)) print(f"{category:<18}{show(before):>10}{show(after):>12}{delta(after, before):>10}") if before is not None and after is not None and after < before: regressions.append(f"{category}: {show(before)} to {show(after)}") else: print("\nGeneral task set: not run. Pass --harness-dir and --general-tasks. Without " "it this comparison can only tell you what got better.")
for key in ("false_call_rate",): before, after = base_rel["rates"].get(key), tuned_rel["rates"].get(key) if before is not None and after is not None and after > before: regressions.append(f"tool-call {key}: {show(before)} to {show(after)}")
if regressions: print("\nRegressions worth writing down:") for line in regressions: print(f" {line}")
summary = { "agent_success_base": base_all, "agent_success_tuned": tuned_all, "agent_success_change": None if None in (base_all, tuned_all) else round(tuned_all - base_all, 4), "agent_by_category_base": {k: list(v) for k, v in base_cats.items()}, "agent_by_category_tuned": {k: list(v) for k, v in tuned_cats.items()}, "reliability_base": base_rel["rates"], "reliability_tuned": tuned_rel["rates"], "general_base": {k: list(v) for k, v in base_gen.items()}, "general_tuned": {k: list(v) for k, v in tuned_gen.items()}, "regressions": regressions, } (out_dir / "comparison.json").write_text(json.dumps(summary, indent=2), encoding="utf-8") print(f"\nwritten to {out_dir / 'comparison.json'}")
if args.labbook: record = agentlog.record( labbook=args.labbook, lab="part-27/compare-agent-models", model={"base": args.base_model, "tuned": args.tuned_model}, dataset={"path": args.tasks, "sha256": agentlog.file_sha256(args.tasks), "general_tasks": args.general_tasks}, data_lineage=agentlog.lineage(evaluation_suite=args.tasks, general_tasks=args.general_tasks), hyperparameters={"repeats": args.repeats, "reliability_repeat": args.reliability_repeat, "temperature": args.temperature, "base_url": args.base_url, "quant": args.quant, "engine": args.engine, "engine_version": args.engine_version, "agent_entry_point": args.agent}, seed=None, losses=None, scores=summary, config_path=__file__, notes=args.notes, ) print(f"recorded run {record['run_id']} in {args.labbook}")
if __name__ == "__main__": main()RunnableAll tracks
P24=../part-24-tools-mcp-and-the-agent-loopP26=../part-26-building-agent-systems
python3 compare-agent-models.py \ --agent-eval "$P26/agent-eval.py" \ --agent "$P26/scaffold-minimal.py" \ --tasks "$P26/agent-tasks.json" \ --reliability "$P24/tool-call-reliability.py" \ --base-url http://127.0.0.1:4000/v1 \ --base-model local/agent --tuned-model local/agent-tuned \ --workspace ./agent-workspace --repeats 3 \ --quant Q4_K_M --engine llama.cpp --engine-version v0.4.0 \ --out-dir compare-out --labbook labbook.mdAdd --harness-dir and --general-tasks pointing at Part 10’s run-eval.py and your own task
set. Leaving them out is how people ship a fine-tune that got better at agent tasks and worse
at everything else without noticing.
Output — what you should see
Agent suite, success rate per categorycategory base fine-tune changeretrieval 66.7% 77.8% +11.1reasoning 53.3% 53.3% +0.0tools 58.3% 83.3% +25.0all tasks 60.0% 71.1% +11.1
Tool-call reliability (Part 24's fixed prompt set)rate base fine-tune changeparse rate 93.3% 100.0% +6.7schema valid 80.0% 93.3% +13.3false calls 20.0% 30.0% +10.0
Regressions worth writing down: tool-call false_call_rate: 20.0% to 30.0%That output is an illustration of the shape, not a claim about what you will see. The pattern it illustrates is the one to expect and to look for: the tools category moves most, reasoning moves least, and something got worse.
Read it in that order. First the category breakdown, because a single overall number hides which kind of task changed and the answer is almost always “the ones that were about calling tools correctly”. Second the reliability rates, because they say whether the change is the mechanical one, and a schema-valid rate that rose while the right-tool rate did not means the model learned to fill in arguments rather than to choose better. Third the cost rows, because a model that finishes more tasks by taking more steps has traded latency for success and you should know which you bought. Last the regression list, which the script prints for you and which belongs in the report whether or not it is convenient.
The spread matters as much as the change. Run the comparison twice with different seeds if the
suite is short, or raise --repeats, and put the range next to every figure. A course that
taught you to report a difference of three tasks out of forty-five without saying how much the
number moves between runs would have taught you nothing.
9. Write the report
Section titled “9. Write the report”Ten minutes.
RunnableAll tracks
# Agent fine-tune report — <model>, <date>
Purpose: the write-up this part's lab produces. Fill in every row. A row you cannot fill inis itself a finding: write "not recorded" rather than deleting it, because the next personneeds to know which numbers were never taken. Platform: all. Minimum memory: none.Assumes: the lab-notebook lines this part's scripts appended, and`compare-out/comparison.json` from `compare-agent-models.py`.
Copy this file next to your notebook and edit it. Nothing here is generated: the point ofwriting it by hand is that you have to look at each number and decide whether you believe it.
---
## 1. What was being improved, and why
- The behaviour I wanted more of:- The evidence it was missing before (the base model's line in `comparison.json`, and the per-case failures from Part 24's reliability test):- What I decided not to try to fix with a fine-tune, and why:
## 2. The data
| Field | Value || --- | --- || Collection task set, and how many tasks | || Where the trajectories came from | || Episodes collected | || Episodes dropped by the scrubber, and on which pattern | || Redactions by pattern, from `scrub-report.json` | || Extra patterns I added, and why | || Episodes rejected by each filter, from `filter-report.json` | || Training rows and validation rows, with the task counts behind them | || Contamination found against the evaluation suite | || SHA-256 of `data/train.jsonl` | |
Two questions to answer in prose here, because a table cannot:
1. I read ten training rows. What did the tenth one teach the model that I did not intend?2. Which filter would I loosen next time, and what would I expect to break?
## 3. The run
| Field | Value || --- | --- || Base model, and its licence | || Track and machine | || Adapter rank, alpha and target modules | || Loss mask: assistant-only or full | || Sequence length, and how many rows were truncated | || Epochs, effective batch and learning rate | || Wall-clock, attended and unattended | || Best epoch and best validation loss | || Run id in the lab notebook | |
## 4. The measurement
Every figure below comes from `compare-agent-models.py`, at the same temperature, the samequantisation and the same number of repeats for both models. Say what those were.
| Suite | Base | Fine-tune | Change || --- | --- | --- | --- || Agent suite, all tasks | | | || Agent suite, retrieval | | | || Agent suite, tools | | | || Agent suite, reasoning | | | || Tool-call parse rate | | | || Tool-call right-tool rate | | | || Tool-call schema-valid rate | | | || Tool-call false-call rate | | | || General task set, per category | | | |
| Cost | Base | Fine-tune || --- | --- | --- || Mean steps per task | | || Mean tokens per task | | || Mean seconds per task | | |
Repeats per task: ____. Spread across repeats: ____. State whether the change is larger thanthe spread. If it is not, the honest sentence is "this run cannot tell".
## 5. What got worse
List every regression, including the ones small enough to be tempting to leave out.
--
If the false-call rate rose, give it its own line. A model that learned to always reach fora tool is a worse agent than one that sometimes answers directly, and the overall successrate can hide it.
## 6. What I would do differently
- The change I would make to the collection set:- The change I would make to the filters:- Whether a fine-tune was the right instrument at all, now that I have the numbers:
## 7. Reproduction
- Commands, in order, with the arguments actually used:- Versions of transformers, TRL, PEFT and the serving engine:- Lab-notebook run ids for every stage:Every stage in this lab appended a JSON line to labbook.md, and the lines carry the data
lineage: which trajectory file, which scrub report, which filter report, and the content hash
of each. The run-log module underneath them is this part’s copy of the format Part 11 defines.
RunnableAll tracks
"""Append one machine-readable record per collection, training or evaluation run to the notebook.
Purpose: Part 27's self-contained copy of the run-log format defined in Part 11, so that this part's scripts record a run identically without needing Part 11's or Part 13's files on the path. Every field is filled in or written as null, because a reader a month later has to be able to tell "not recorded" from "not applicable". The extra field this part adds is `data_lineage`: the trajectory file a training set came from, its hash, and the scrub and filter reports that stand between them.Platform: all (standard library only; torch, transformers, trl, peft and mlx are inspected for their version strings only if they happen to be installed)Minimum memory: 8 GBAssumes: Python 3.10 or newer. The lab notebook is created if it does not exist. git is optional and is used only to record the commit the configuration was at.
Usage: imported by this part's Python scripts: import agentlog agentlog.record(labbook="labbook.md", lab="part-27/train-agent-sft", ...) or called from a shell script with the run's own fields as JSON on standard input: python3 agentlog.py --record --labbook labbook.md < fields.json or run with no arguments to print the field list and exit."""from __future__ import annotations
import argparseimport hashlibimport jsonimport osimport platformimport secretsimport subprocessimport sysfrom datetime import datetime, timezonefrom pathlib import Pathfrom typing import Any
# The same field set Part 11 defines, plus data_lineage. A record missing any of them is# refused, because a partial record is harder to interpret than no record at all.FIELDS = ( "run_id", "lab", "date", "config_commit", "model", "dataset", "data_lineage", "hyperparameters", "seed", "hardware", "versions", "losses", "scores", "notes",)
PACKAGES = ("torch", "transformers", "trl", "peft", "datasets", "mlx", "mlx-lm")
def new_run_id() -> str: """Sorts by time and does not collide between two runs started in the same second.""" stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") return f"{stamp}-{secrets.token_hex(3)}"
def file_sha256(path: str | os.PathLike[str], chunk: int = 1 << 20) -> str | None: """Content hash of a file, so a run is tied to the exact bytes it read.
A month later the question is rarely which learning rate was used, which is in the script. It is whether this run was before or after the scrub, and the hash answers it. """ p = Path(path) if not p.is_file(): return None digest = hashlib.sha256() with p.open("rb") as handle: for block in iter(lambda: handle.read(chunk), b""): digest.update(block) return digest.hexdigest()
def package_versions() -> dict[str, str | None]: """Version strings for the packages that decide what a training run actually did.""" out: dict[str, str | None] = {} try: from importlib import metadata except ImportError: # pragma: no cover - Python 3.7 and earlier only return {name: None for name in PACKAGES} for name in PACKAGES: try: out[name] = metadata.version(name) except Exception: out[name] = None return out
def git_commit(path: str | os.PathLike[str] | None) -> str | None: """The commit the configuration was at, when the configuration lives in git.""" if path is None: return None directory = Path(path).resolve() directory = directory if directory.is_dir() else directory.parent try: done = subprocess.run( ["git", "-C", str(directory), "rev-parse", "--short", "HEAD"], capture_output=True, text=True, timeout=10, check=False, ) except (OSError, subprocess.SubprocessError): return None return done.stdout.strip() or None
def hardware() -> dict[str, Any]: """What the run happened on, at the level of detail that changes a conclusion.""" info: dict[str, Any] = { "platform": platform.platform(), "machine": platform.machine(), "python": platform.python_version(), "accelerator": None, "accelerator_count": None, } try: import torch # noqa: PLC0415 - optional and only inspected except Exception: return info try: if torch.cuda.is_available(): info["accelerator"] = torch.cuda.get_device_name(0) info["accelerator_count"] = torch.cuda.device_count() elif getattr(torch.backends, "mps", None) is not None and torch.backends.mps.is_available(): info["accelerator"] = "Apple silicon (Metal, PyTorch MPS)" info["accelerator_count"] = 1 else: info["accelerator"] = "cpu" info["accelerator_count"] = 0 except Exception: pass return info
def lineage(trajectories: str | os.PathLike[str] | None = None, scrub_report: str | os.PathLike[str] | None = None, filter_report: str | os.PathLike[str] | None = None, **extra: Any) -> dict[str, Any]: """Where the training data came from, as paths plus content hashes.
This is the field that makes a trajectory fine-tune auditable. Without it, a data set and a model are two files with no stated relationship, and the question "was this trained on scrubbed data" has no answer that can be checked. """ out: dict[str, Any] = { "trajectories": str(trajectories) if trajectories else None, "trajectories_sha256": file_sha256(trajectories) if trajectories else None, "scrub_report": str(scrub_report) if scrub_report else None, "scrub_report_sha256": file_sha256(scrub_report) if scrub_report else None, "filter_report": str(filter_report) if filter_report else None, "filter_report_sha256": file_sha256(filter_report) if filter_report else None, } out.update(extra) return out
def record(labbook: str | os.PathLike[str], lab: str, **fields: Any) -> dict[str, Any]: """Build one record, append it to the notebook as a JSON line, and return it.""" row: dict[str, Any] = { "run_id": fields.pop("run_id", None) or new_run_id(), "lab": lab, "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), "config_commit": fields.pop("config_commit", None) or git_commit(fields.pop("config_path", None)), "model": fields.pop("model", None), "dataset": fields.pop("dataset", None), "data_lineage": fields.pop("data_lineage", None), "hyperparameters": fields.pop("hyperparameters", None), "seed": fields.pop("seed", None), "hardware": fields.pop("hardware", None) or hardware(), "versions": fields.pop("versions", None) or package_versions(), "losses": fields.pop("losses", None), "scores": fields.pop("scores", None), "notes": fields.pop("notes", None), } # Anything else the caller passed is kept rather than dropped: a script that records # one more number should not have to change this module. row.update(fields) missing = [name for name in FIELDS if name not in row] if missing: raise ValueError(f"run record is missing {missing}; every field is written, even as null")
path = Path(labbook) path.parent.mkdir(parents=True, exist_ok=True) with path.open("a", encoding="utf-8") as handle: handle.write(json.dumps(row, default=str) + "\n") return row
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--record", action="store_true", help="read a JSON object of fields from standard input and append it") parser.add_argument("--labbook", default="labbook.md") parser.add_argument("--lab", default="part-27/unnamed") args = parser.parse_args()
if not args.record: print("Fields written on every record:") for name in FIELDS: print(f" {name}") print("\nPass --record with a JSON object on standard input to append one.") return
payload = json.load(sys.stdin) if not isinstance(payload, dict): sys.exit("standard input must be a JSON object of fields") written = record(labbook=args.labbook, lab=payload.pop("lab", args.lab), **payload) print(f"recorded run {written['run_id']} in {args.labbook}")
if __name__ == "__main__": main()Audit the trajectory-to-training conversion
Section titled “Audit the trajectory-to-training conversion”Start with retained raw trajectories from Parts 24–26, plus the frozen independent task suite. If cleanup removed the trajectories, regenerate them before continuing; a final summary file does not contain the tool interactions needed for training. Record the original task and environment identity.
Run the scrubber and inspect its report and representative transformed records. Confirm secrets are removed without breaking required tool-call IDs or message order. Then apply success and trajectory-quality filters separately. A successful episode can still contain an unsafe attempt or many repeated calls that you do not want the student to imitate.
Split by underlying task or repository before formatting examples. Render one row with the chosen model’s template and inspect assistant/tool boundaries and the loss mask. Train a small diagnostic run before scaling. Export the resulting adapter and compare the deployed file with the base using the same tools, permissions, budgets and held-out tasks. Record syntax validity, argument accuracy, final success and unnecessary actions separately. Keep raw provenance under the appropriate access policy, scrubbed data, filter report, training configuration and task-level evaluation. Promote the new alias only when the measured behaviour and permission boundaries meet the stated contract.
Validation
Section titled “Validation”Five checks. Each one has failed for somebody.
RunnableAll tracks
# 1. Nothing sensitive survived into the training data.python3 scrub-trajectories.py --in clean/trajectories.jsonl --check
# 2. No training row shares a task with the evaluation suite.python3 -c "import json; r=json.load(open('filter-report.json')); print('contaminated:', r['contaminated'])"
# 3. Train and validation come from different tasks.python3 -c "import json; r=json.load(open('filter-report.json')); print(r['train_tasks'], 'train task(s),', r['valid_tasks'], 'validation task(s)')"
# 4. Both aliases are being served, and by different files.curl -s http://127.0.0.1:4000/v1/models | grep -o 'local/agent[a-z-]*'
# 5. Every stage left a run record.grep -c '"lab": "part-27' labbook.mdThe second check should print an empty list. The third should show more than one task on each side; if the validation side is a single task, the validation loss is a measurement of one question. The fifth should be at least four.
Expected outcome
Section titled “Expected outcome”| Instrument | Measure | Base | Fine-tune | Change | Spread over repeats |
|---|---|---|---|---|---|
| Part 24 reliability | parse rate | — | — | — | — |
| Part 24 reliability | right-tool rate | — | — | — | — |
| Part 24 reliability | schema-valid rate | — | — | — | — |
| Part 24 reliability | false-call rate | — | — | — | — |
| Part 26 agent suite | success rate, all tasks | — | — | — | — |
| Part 26 agent suite | success rate, tools | — | — | — | — |
| Part 26 agent suite | mean steps per task | — | — | — | — |
| Part 26 agent suite | mean tokens per task | — | — | — | — |
| Part 10 general set | pass rate per category | — | — | — | — |
your machine: track, chip and memory, your operating system and version · llama.cpp behind the Part 9 gateway, or mlx-lm on Track M the build number from llama-cli --version · the base and the adapter run id from labbook.md, the same type for both rows of every pair · 16,384 tokens of context · the date you ran it
Filled in from compare-out/comparison.json. The spread column is what makes the change column readable: a difference smaller than the spread across repeats is not a result, and saying so is a finding rather than a failure.
You are done when all of these are true:
data/train.jsonlanddata-mlx/train.jsonlexist, the scrub check passes on the file they came from, andfilter-report.jsonshows no contamination against the evaluation suite;- the adapter trained, the trainable-parameter percentage matched the arithmetic, and the run
record in
labbook.mdcarries the dataset hash; - both aliases answer through the gateway and the fine-tune emits parseable tool calls;
compare-out/comparison.jsonholds both models’ numbers on both instruments at the same settings, with at least three repeats;- your report names at least one thing that got worse, or states that the general set was not run, which is itself a finding.
Troubleshooting
Section titled “Troubleshooting”no episodes were built; check the paths inside the log you passed. The transcript paths in
labbook.md are relative to the directory the agent was run from. Run the collector from the
same directory, or edit the paths.
Nothing survives the filters. Read filter-report.json. If outcome-not-passed accounts
for nearly everything, the collection tasks are too hard for the base model; make them easier
or use a teacher, as the fourth lesson describes. If same-task-id-as-evaluation accounts for
it, you are collecting from the suite you intend to measure with.
The rendered example shows no tool schemas. The tools column did not reach the template.
Either the base model’s template ignores it, in which case check the model card for how that
family expects tools to be passed, or the column was lost when the dataset was built; the
trainer prints which encoding it used for the tools column on startup.
assistant_only_loss raises an error about generation markers. The chat template does not
mark the assistant spans. TRL patches the template for known families such as Qwen3; for others
you either supply a template that marks them or run with --loss full and record that you did,
which trains on the tool results as well.
The trainable percentage is tiny or zero. The target module names did not match. Run
python3 train-agent-sft.py --model <your base> --list-modules and pass what it prints.
Out of memory partway through the first epoch. The longest episodes arrive later in the
epoch than the shortest ones. Lower --max-length and re-run the filter with a tighter
--max-steps so the data fits the length rather than the length chasing the data.
The fine-tune serves but never calls a tool. Almost always the template. Check that
--jinja is in the llama-swap command for the new alias, and that the tokeniser files were
saved next to the adapter, which train-agent-sft.py does and a hand-rolled export often does
not.
The comparison script exits before running the second model. Part 26’s harness exits with status 2 when any task fails, which is the normal case. The comparison script expects that; what it does not expect is a missing results file, which means the harness stopped earlier than that. Read its output above the error: a gateway that is not answering and an agent entry point that raised on import both look like this.
The reliability test reports responses containing a tool-call format as visible text. That
is a server-side parser or template mismatch rather than a property of the fine-tune, and it
usually appears after an export. Check --jinja first, then check that the engine’s tool-call
parser matches the model family.
The success rate moved but the spread is as large as the move. Raise --repeats, or accept
that a fifteen-task suite cannot resolve a difference this small and say so. This is the
Part 14 reality check arriving in a new costume.
Cleanup
Section titled “Cleanup”RunnableAll tracks
# The full-precision GGUF is only needed until the quantised one has been scored.rm -f "$HOME/models"/*-bf16.gguf
# The merged safetensors can be rebuilt from the adapter and the base.rm -rf "$HOME/models"/*-merged
# Transcripts and raw trajectories: keep them if they are scrubbed, delete them if not.rm -rf transcripts rawKeep data/, clean/, the adapter directory, the three reports, compare-out/ and
labbook.md. Together they are enough to reproduce the run and to defend the numbers, which is
what the capstone asks for.
Leave both gateway aliases in place. Removing the base is what makes the next comparison impossible. If the fine-tune won, the new alias is a candidate for the agent role in the Part 25 workstation; put it there beside the model it is replacing rather than on top of it, and give it a name that says which base and which run it came from.
What you learned
Section titled “What you learned”- A trajectory becomes training data only if the log carries the assistant turns. You saw this in task 2, when the collector reconstructed episodes from Part 24’s transcripts without loss and would have marked Part 26’s as lossy.
- Transcripts carry your machine’s private details. You saw the count in task 3, by pattern, and you added patterns of your own for the things a regular expression cannot find.
- Decontamination is structural, not a check. You collected from one task set and measured with another, and the report in task 4 confirmed the two did not overlap.
- The chat template is part of the contract. You read one rendered example in task 5 and
passed
--jinjain task 7 so the same template served it. - A fine-tune is a pair of numbers, not a model. You produced them in task 8, on three instruments, at identical settings, with repeats.
- Something got worse. You found it, or you found that you had not measured widely enough to find it, and you wrote down which.
Record in the lab notebook, in the report template: the base model and its licence; the size of the collection set and the number of attempts per task; the redaction counts by pattern; every filter rejection count; the adapter settings and the loss mask; the best epoch and validation loss; the full before-and-after table with the spread across repeats; and, in its own line, the regression you found. The capstone’s fourth project reuses all of it.
Check your understanding
Sources for this lesson
5 verified · checked 2026-09-09
- 01TRL documentation — SFT Trainer§ Train on assistant messages only; Tool Calling with SFT; Train adapters with PEFThuggingface.co/docs/trl/en/sft_trainer2026-09-09
- 02TRL documentation — Dataset formats and types§ Tool Calling; the tools columnhuggingface.co/docs/trl/dataset_formats2026-09-09
- 03Transformers documentation — Tool use§ Passing tools; JSON schemashuggingface.co/docs/transformers/chat_extras2026-09-09
- 04mlx-lm — LoRA and QLoRA fine-tuning§ Data; tools format; mask-prompt; num-layers; fusegithub.com/ml-explore/mlx-lm/blob/main/mlx_lm/LORA.md2026-09-09
- 05Qwen3-4B model card§ Licence; best practiceshuggingface.co/Qwen/Qwen3-4B2026-09-09
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.