Level 5 · Agentic EngineerProjectCapstone · page 5 of 7150 minSXMN 12 GB

150Minutes

3Tools

3Sources

All fourTracks

Tools used on this page3

Capstone 4: Improve a Small Model

Validated on: written from the documentation cited above and from Parts 11 and 13 to 16; not yet validated on hardware on any track. The technique you use here was taught in one of those parts; what this project adds is the discipline around the measurement.

Objective

By the end of this project you will have a small model that behaves differently from the one you started with, in a way you defined in advance, together with the evidence that says whether the difference is real, how large it is compared with the noise, what it cost elsewhere, and whether it survived quantisation.

The technique is not the deliverable. Parts 13, 14 and 15 taught supervised fine-tuning, preference optimisation, reinforcement learning with a verifiable reward and distillation, and any of them is an acceptable answer here. What the capstone adds is the part those labs each practised once and this project insists on all at the same time: an evaluation set built before the training, a measured noise floor, repeats, a report of what got worse, and an honest treatment of the quantised file as the different model it is.

Architecture

Most of the 150 minutes is not spent at the keyboard, and knowing which parts are which is what lets this deliverable overlap with Capstone 3.

Where the time goes, and which parts need you

Evaluation set

attended: write it, count it, date it

Noise floor

mostly waiting

Data

attended: build and filter

Training run

unattended: do Capstone 3 while this runs

Evaluation

attended: three runs each, base and improved

Quantise and export

partly unattended

Measure the export

attended: it is a different model

Blocks marked attended need you at the keyboard; the others are waiting. The last row is the one most often skipped: the file you will serve is not the model you evaluated unless you evaluate the file you will serve.

Requirements

Every track needs the training environment from Part 11, a task set of your own of the kind Part 10’s evaluation lab builds, and the evaluation methods from Part 16. The memory floor of 12 GB follows the smallest path in the course; a larger tier lets you train a larger student or use a more expensive technique. About 150 minutes, of which roughly half is unattended.

Track S — NVIDIA DGX Spark

The full toolchain is available. With a large unified memory you can choose a bigger student or a distillation with a local teacher, which Part 15’s pipeline handles as five stages with the generation stage being the expensive and resumable one. Keep the teacher and the student out of memory at the same time unless your Capstone 1 arithmetic says they both fit.

Track X — AMD Ryzen AI Max+ 395Partial

The four-bit quantised training path on this chip is newer than the rest of the toolchain; Part 13 records the compatibility position and the fallback.

Supervised fine-tuning and preference optimisation both work here, and Part 13’s page covers the compatibility caveat around four-bit quantised training on this chip together with the fallback of training a smaller model without it. Record which path you took, because the memory arithmetic and the peak memory row differ substantially between them.

Track M — Apple silicon

The path here is mlx-lm rather than the PyTorch trainers, and Part 13’s lab has the equivalent commands. mlx-lm documents low-rank and full fine-tuning and its own conversion and quantisation step, so the export at the end of this project produces an MLX-format model rather than a GGUF one unless you convert separately. The number of layers you adapt is the memory knob on this track.

Track N — NVIDIA desktop or laptop

The full toolchain is available and this is the track the tooling is written for first. The constraint is device memory rather than system memory, so Part 11’s memory arithmetic matters more here than anywhere else: weights, gradients, optimiser states and activations all have to fit on the card, and the batch size and sequence length are the terms you control.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/capstone"
cd "$LAB_DIR"
pwd
test -f "model-improvement-template.md"

Expected result: pwd ends in capstone and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Name the behaviour, and write the claim

Open the template and fill in section 1.

Fragment — not complete on its own

# Model improvement — <the behaviour you are trying to change>

<!--
Purpose: the deliverable of Capstone 4. One target behaviour, one evaluation set built
         before any training, one technique chosen with a reason, the run logs, a
         before-and-after comparison with its variance, the quantised export measured as
         the separate model it is, and the regressions stated rather than buried.
Platform: all; the technique is chosen by what your memory tier can train
Minimum memory: 12 GB for the smallest path in the course; more for the larger techniques
Assumes: the training environment from Part 11, the fine-tuning workflow of Part 13, the
         preference and reinforcement-learning labs of Part 14 or the distillation
         pipeline of Part 15, and the evaluation methods of Part 16. Run records come from
         Part 11's runlog and evaluation records from Part 16's scripts.

The rule this template exists to enforce: the evaluation set is written down, dated and
counted before the first training run starts. Everything else here is bookkeeping around
that one discipline. Delete every comment block, including this one.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Track:** <S/X/M/N> ·
**Base model:** <exact repository id> · **Base model licence:** <from the model card>

---

## 1. The target behaviour

**What the model should do differently, in one sentence:** <specific enough to be wrong.
"Answers in our house JSON shape without a preamble" is specific; "is better at coding" is
not.>

**Why a smaller model with this behaviour is worth more to you than a larger one without
it:** <one paragraph. If the honest answer is that it is not, that is a finding and the
project can report it.>

**The claim, written before training and unchanged afterwards:** <one sentence with a
comparison in it: "on <task file>, at temperature <t> with seeds <s>, the improved model
passes more of the deterministic checks than the base model and its judge mean in <these
categories> is not lower.">

## 2. The evaluation set

| Field | Value |
| --- | --- |
| File and format | <name; one task per line> |
| Items | <n> |
| Built on | <YYYY-MM-DD; before the first run record, and here is its date> |
| How items were chosen | <from real use, from your own work, written by hand, generated and reviewed> |
| Deterministic checks | <how many items are checkable without a judge> |
| Judged items | <how many, by which judge model, position-swapped> |
| Categories | <the groups you will report separately> |
| Held out from training | <how you kept them apart> |
| Decontamination | <what you checked the training data against, what overlap was found, what you did> |
| Licence and confidentiality of the material | <what the source allows> |

**The noise floor:** <evaluate the base model twice, unchanged, and record the spread
here.> Any later difference smaller than <the spread> is reported as no difference.

<This row is the one that turns the rest of the document into a measurement. Part 16's
quantisation lab measures one file twice for exactly this reason, and finding that two
things are indistinguishable is an honest and useful result.>

## 3. The technique, and why

| Option | Fits my memory tier? | Would it change the behaviour in section 1? | Chosen? |
| --- | --- | --- | --- |
| Supervised fine-tuning with LoRA or QLoRA | | | |
| Preference optimisation on top of the fine-tune | | | |
| Reinforcement learning with a verifiable reward | | | |
| Distillation from a larger local teacher | | | |

**Chosen:** <technique, or the sequence: fine-tune then preference optimisation.>
**Because:** <two sentences. Memory is a reason; so is "the behaviour is a format, and
imitation is the cheapest way to teach a format".>

**Memory arithmetic for the run, from Part 11:** weights <GB> + gradients <GB> +
optimiser states <GB> + activations <GB> = <GB> against <GB> available. **Peak memory
actually observed:** <GB>. **Gap, and why:** <one sentence.>

## 4. The data

| Field | Value |
| --- | --- |
| Source | <your own work, an agent's trajectories, a teacher's generations, a public dataset> |
| Licence | <exact> |
| Training examples | <n> |
| Validation examples | <n> |
| Kept after filtering | <n, and what the filter rejected> |
| Redaction | <what was removed before the trainer saw it, or "none needed"> |
| Training file hash | <from the run record> |

## 5. The runs

| Run id | Technique | Key settings | Seed | Final loss or reward | Attended minutes | Total minutes | Outcome |
| --- | --- | --- | --- | --- | --- | --- | --- |
| <from labbook.md> | | <rank, alpha, learning rate, epochs, batch, beta> | | | | | <kept / discarded, and why> |

**Runs that were discarded:** <how many, and what went wrong. A project with no discarded
runs either got lucky or is not telling you everything.>

**What the curves said:** <loss, reward, length and any divergence measure, read together.
A reward that rose while the answers got longer and worse is the classic finding and it is
worth reporting when it happens.>

## 6. Before and after

<Same task file, same shots, same chat template, same sampling settings, same seeds, both
models. Three runs of each at minimum, so that a difference can be told from a wobble.>

| Model | Quantisation | Metric | Runs | Mean | Range | Compared with |
| --- | --- | --- | --- | --- | --- | --- |
| Base | <as served> | <deterministic checks passed> | <3> | | | reference |
| Improved | <as served> | <deterministic checks passed> | <3> | | | base |
| Base | | <judge mean, position-swapped> | | | | reference |
| Improved | | <judge mean, position-swapped> | | | | base |

| Category | Base mean | Improved mean | Range of each | Better, worse or indistinguishable |
| --- | --- | --- | --- | --- |
| <category> | | | | |

**Did the claim in section 1 hold?** <yes, no, or partly, in one sentence, against the
noise floor from section 2.>

**On a standard suite:** <the tasks, the harness and its version, the shot count, the
chat-template setting, the number of items scored, the runs and the spread. Or "not run",
with the reason.>

## 7. Regressions and limits

| What got worse | By how much | Against what noise floor | Acceptable for my use? | Why |
| --- | --- | --- | --- | --- |
| <category or behaviour> | | | <yes/no> | |

**What the model still cannot do:** <the things the target behaviour did not touch, stated
plainly, so that nobody deploys it expecting them.>

**What would break it:** <inputs outside the training distribution, a different chat
template, a longer context than you trained for.>

## 8. Quantisation and export

| Field | Value |
| --- | --- |
| Merged or adapter kept separate | <which, and why> |
| Export format | <GGUF, MLX, safetensors> |
| Quantisation | <exact type> |
| File size | <GB> |
| Damage measured how | <divergence against the unquantised model, and your task set> |
| Task set result, quantised | <mean and range over the same number of runs> |
| Difference from the unquantised model | <against the noise floor> |
| Served under which gateway name | <alias> |

<The quantised file is a different model from the adapter you trained, and Part 16's rule
is to measure the file you will actually serve. A report that evaluates the unquantised
model and serves the quantised one has measured something nobody uses.>

## 9. What this does not establish

<What you did not measure, what one run cannot distinguish, which comparisons were not
strictly like for like, and what you took from documentation without verifying. Three
items is a minimum, and "I ran each configuration once" belongs here whenever it is true.>

Download model-improvement-template.md160 lines

RunnableAll tracks

cp model-improvement-template.md model-improvement.md

One sentence saying what the model should do differently, specific enough to be wrong. “Answers in our house JSON shape with no preamble” is specific. “Is better at coding” is not, because nothing could show it to be false.

Then the claim, in the shape Part 13’s project used: on this task file, at these sampling settings with these seeds, the improved model passes more of the deterministic checks than the base model and its judged mean in these categories is not lower. Write it before training and do not edit it afterwards.

2. Build the evaluation set, and date it

Section 2, and this is the discipline the whole project is built around. The set exists before the first training run, it has a file, an item count and a date, and the date is earlier than the first run record in your notebook.

What goes in it: items from your real use, not from a benchmark; a majority that can be checked without a judge, because deterministic checks do not drift; a minority that need a judge, run with the positions swapped; and categories, so that you can report separately on the behaviour you targeted and on everything else.

Then the two rows people skip. Decontamination: check the evaluation items against your training data and record what overlap you found and what you did about it. Held out: say how you kept the two apart, because “I remembered not to include them” is not a procedure.

3. Measure the noise floor before you measure anything else

Evaluate the base model twice, unchanged, with the same settings, and record the spread between the two runs in section 2.

That spread is the resolution of your instrument. Any later difference smaller than it is reported as no difference, however much you want it to be one. Part 16’s quantisation lab builds its whole results table around this, with a literal row recording one file measured twice, and its worked example ends with two files being indistinguishable, which the page calls an honest and useful result.

The two sources of the wobble are worth knowing apart. There is sampling error over items, which shrinks as the set grows and is why a fifty-item run is coarse. And there is run-to-run variation even at temperature zero, because batching and kernel choices change the order in which numbers are summed. Neither is a bug and both are why the answer to “how many runs” is three rather than one.

4. Choose the technique, with the memory arithmetic

Section 3. The trainers available to you are documented in TRL’s taxonomy: supervised fine-tuning offline, the preference family including direct preference optimisation, the online methods including group relative policy optimisation, and the distillation trainers. Part 13, 14 and 15 each taught one branch, and mlx-lm covers the Apple path with its own low-rank fine-tuning.

Choose on two grounds and write both down. What fits: the memory arithmetic from Part 11, with weights, gradients, optimiser states and activations added up against what you have. What could change the behaviour in section 1: a format is a matter of imitation and supervised fine-tuning is the cheapest route to it; a preference between two acceptable answers wants the preference family; a task with a checkable answer is what reinforcement learning with a verifiable reward is for; a capability a larger local model has and yours does not is a distillation problem.

Write the arithmetic and, later, the peak memory you actually observed. The gap between them is a good line for Capstone 6.

5. Build the data, and run the training

Section 4 and section 5. Follow the part that owns your technique; nothing here is new.

Two things this project asks that the labs did not. Record every run, including the ones you threw away, with what went wrong. A project with no discarded runs either got lucky or is not telling you everything, and the discarded runs are frequently where the learning was. And read the curves together rather than one at a time: a reward that rose while the answers grew longer and worse is the classic finding, and it is only visible when you look at the reward, the length and the divergence side by side.

6. Evaluate before and after, three runs each

Section 6. Same task file, same shots, same chat template, same sampling settings, same seeds, both models, three runs each at minimum.

Report the mean and the range, not a single number. Three runs is not statistics; it is enough to tell “these two models differ” from “this measurement moves about”, which is the distinction the whole section exists to make. Then answer the claim from section 1: did it hold, against the noise floor from section 2?

If you also ran a standard suite, record what the harness’s own guidance implies you should carry with the result: the exact task and metric names, the shot count, whether a chat template was applied, the sampling settings including whether a thinking mode was on, how many items were scored and whether a limit was used, how many runs and their spread, and the harness version and date. A suite result quoted without those is not comparable with anybody’s, including your own from last month.

7. Report what got worse

Section 7, and it is the section that makes the rest credible.

Every technique in Parts 13 to 15 trades something. A model tuned to a format may become terser everywhere. A preference-optimised model may lengthen. A distilled student inherits its teacher’s habits, including the wrong ones, which is what Part 15’s challenge is about. A model trained on one domain may answer a general question worse than it did.

For each category that got worse, the numbers, the noise floor it is measured against, and one sentence on whether the trade is acceptable for the use in section 1. Then the two lists that stop somebody deploying it wrongly: what the model still cannot do, and what would break it.

8. Quantise, export, and measure the file you will serve

Section 8. The adapter you trained and the quantised file you will serve are different models, and Part 16’s rule is to measure the one you will serve.

Merge or keep the adapter separate, export to the format your track uses, quantise, and then run the task set again on the quantised file with the same settings and the same number of runs. Record the difference against the unquantised model and against the noise floor. Part 16’s primary quantisation metric is the divergence from the full-precision original rather than perplexity, and its results table pairs the same quantisation with and without an importance matrix precisely because that pairing turns an opinion into a number.

Then serve it behind a name on the gateway from Capstone 2, so that Capstone 5 can drive it.

9. Say what this does not establish

Section 9. What you did not measure, what a small number of runs cannot distinguish, which comparisons were not strictly like for like, and what you took from documentation without verifying. Three items is a minimum, and “I ran the standard suite once” belongs here whenever it is true.

Preserve the baseline through the final deployed comparison

Freeze the target behaviour, held-out task IDs and acceptance rule before building training data. Run the untouched model at the intended deployment precision and keep all repeated results. Choose one intervention and record its dataset, optimiser or conversion configuration.

After training, compare the candidate before export and after the final representation change. Inspect target-task gains, general regressions, output lengths and resource use. If a result changes only after quantisation, attribute it to that stage rather than to the training method. Use the same independent checks and failure denominator throughout.

Keep source groups separated when generating synthetic examples or agent trajectories. Record teacher provenance and verifier/filter versions where applicable. If the measured improvement is smaller than the uncertainty or misses the acceptance rule, retain the base in production and report the negative experiment. The deliverable is an evidence-backed improvement decision, not a promise that every training run wins. Archive the exact base identity, dataset hashes, candidate artefacts, export recipe and task-level outputs so the deployed result can be recreated after a tool upgrade.

Validation

model-improvement.md has every section filled in with no angle-bracket fields left.
The evaluation set’s file has an item count and a date, and that date is earlier than the first training run record in labbook.md.
The noise floor row has a number in it, from the same configuration measured twice.
Section 3 shows the training memory arithmetic and the peak memory actually observed.
Every run, including discarded ones, has a row in section 5 with its run identifier from the notebook.
The before-and-after table has at least three runs per model with a mean and a range.
The claim from section 1 is answered explicitly against the noise floor.
Section 7 lists at least one thing that got worse, or states that nothing measurably did and names what was checked.
The quantised file has its own evaluation rows, with the same settings and the same number of runs as the unquantised model.
The exported model is reachable behind a gateway name.
Section 9 lists at least three things the work does not establish.

Expected outcome

A model, a set of tables and an honest paragraph. The shape below is what the deliverable commits to; your own copy holds the numbers.

Pending validationThe before-and-after this project produces — the shape, to be filled in with your own

Model	Quantisation	Metric	Runs	Mean	Range	Against the noise floor
Base	as served	deterministic checks passed	3	pending	pending	reference
Base, repeated	as served	deterministic checks passed	3	pending	pending	this row is the noise floor
Improved, unquantised	BF16 or the training format	deterministic checks passed	3	pending	pending	pending
Improved, quantised	the file you will serve	deterministic checks passed	3	pending	pending	pending
Base	as served	judged mean, positions swapped	3	pending	pending	reference
Improved, quantised	the file you will serve	judged mean, positions swapped	3	pending	pending	pending
Improved, quantised	the file you will serve	the category you did not target	3	pending	pending	the regression row

the machine that ran the evaluation, not the one that trained · the engine serving the model under evaluation as recorded in your run records · your base model and your improved model, as listed per row · 4,096 tokens of context · 2026-09-09

The second row is the noise floor: the same configuration measured twice. Every comparison in the rows below it is judged against that spread, and a difference smaller than it is reported as no difference.

Three results count as finished.

The improvement. The targeted behaviour improved by more than the noise floor, the untargeted categories did not measurably worsen, and the quantised file kept the gain. Report it with all four numbers.

The trade. The targeted behaviour improved and something else got worse. This is the commonest outcome and the document is stronger for saying which, by how much, and whether you would take the trade again.

The null result. The difference was inside the noise floor. This is a real finding, it is what a measured noise floor is for, and reporting it honestly is worth more than a third training run chosen to produce a better-looking table.

Troubleshooting

The improved model looks much better and you are suspicious. Check decontamination first. An evaluation item that resembles a training item is the usual explanation, and the second usual one is that the two runs used different chat templates, which changes the task rather than the model.

Training runs out of memory. Work Part 11’s arithmetic backwards: sequence length and batch size drive activations, the optimiser state is proportional to the trainable parameters, and a low-rank method exists to make that last term small. On the Apple track the number of adapted layers is the knob; on the NVIDIA track the constraint is device memory rather than system memory.

The loss falls and the behaviour does not change. Usually the data does not contain the behaviour, or the chat template used in training differs from the one used at inference. Print a formatted training example and a formatted inference prompt side by side; the difference is normally visible immediately.

The reward rises and the outputs get worse. The classic reward-hacking result. Test the reward function against deliberately bad completions before optimising against it, which is what Part 14’s lab does before any training, and read the length curve beside the reward curve.

The quantised model scores far below the unquantised one. Compare against the divergence from the original rather than only the task score, check that the export used the chat template you trained with, and try a less aggressive quantisation. A large drop at a moderate quantisation usually means something went wrong in the export rather than in the rounding.

Every difference is inside the noise floor. Report that. Then, if you want a larger effect, the levers are more data, a longer run, a more targeted evaluation set, or a narrower behaviour. Rerunning until a favourable number appears is the one option that is not available.

Cleanup

Keep the adapter, the merged model if you made one, the exported file you serve, the evaluation set, and every run record. Delete the intermediate checkpoints you do not need, which are usually the largest thing on the disk after the models themselves. Note the storage the artefacts occupy in your notebook, because Capstone 6’s maintenance plan needs to say where they live and what backs them up.

If you loaded a large teacher model for a distillation, unload it and confirm the memory came back before you go on to Capstone 5, which needs the memory for the agents.

What you learned

The evaluation set has to exist first, and be dated. Written afterwards it describes the model rather than testing it, and the failure happens by accident.
A noise floor is the resolution of your instrument. Without one, every difference looks real; with one, some of them stop being differences.
Three runs, with the mean and the range. Not statistics, but enough to separate a difference from a wobble, which is the only question the table has to answer.
A fine-tune trades. Reporting what got worse is what makes the report of what got better believable.
The file you serve is not the model you trained. The quantised export is a different model and Part 16’s rule is to measure the one that will answer requests.
A null result is a result. It is the outcome a measured noise floor exists to make reportable.

Record in the notebook: the evaluation set’s name, item count and date; the noise-floor spread; every run record including the discarded ones; the before-and-after table with means and ranges; the quantised file’s own rows; and one sentence on what you expected the improvement to be before you measured it.

Sources for this lesson

3 verified · checked 2026-09-09

01TRL — Transformers Reinforcement Learning§ Trainer taxonomy; offline, online and distillation methodshuggingface.co/docs/trl/en/index2026-09-09
02EleutherAI — lm-evaluation-harness§ README; reproducibility and task implementation guidelinesgithub.com/EleutherAI/lm-evaluation-harness2026-09-09
03mlx-lm — README§ LoRA fine-tuning; conversion and quantisationgithub.com/ml-explore/mlx-lm2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.