# Agent fine-tune report — <model>, <date>

Purpose: the write-up this part's lab produces. Fill in every row. A row you cannot fill in
is itself a finding: write "not recorded" rather than deleting it, because the next person
needs to know which numbers were never taken. Platform: all. Minimum memory: none.
Assumes: the lab-notebook lines this part's scripts appended, and
`compare-out/comparison.json` from `compare-agent-models.py`.

Copy this file next to your notebook and edit it. Nothing here is generated: the point of
writing it by hand is that you have to look at each number and decide whether you believe it.

---

## 1. What was being improved, and why

- The behaviour I wanted more of:
- The evidence it was missing before (the base model's line in `comparison.json`, and the
  per-case failures from Part 24's reliability test):
- What I decided not to try to fix with a fine-tune, and why:

## 2. The data

| Field | Value |
| --- | --- |
| Collection task set, and how many tasks | |
| Where the trajectories came from | |
| Episodes collected | |
| Episodes dropped by the scrubber, and on which pattern | |
| Redactions by pattern, from `scrub-report.json` | |
| Extra patterns I added, and why | |
| Episodes rejected by each filter, from `filter-report.json` | |
| Training rows and validation rows, with the task counts behind them | |
| Contamination found against the evaluation suite | |
| SHA-256 of `data/train.jsonl` | |

Two questions to answer in prose here, because a table cannot:

1. I read ten training rows. What did the tenth one teach the model that I did not intend?
2. Which filter would I loosen next time, and what would I expect to break?

## 3. The run

| Field | Value |
| --- | --- |
| Base model, and its licence | |
| Track and machine | |
| Adapter rank, alpha and target modules | |
| Loss mask: assistant-only or full | |
| Sequence length, and how many rows were truncated | |
| Epochs, effective batch and learning rate | |
| Wall-clock, attended and unattended | |
| Best epoch and best validation loss | |
| Run id in the lab notebook | |

## 4. The measurement

Every figure below comes from `compare-agent-models.py`, at the same temperature, the same
quantisation and the same number of repeats for both models. Say what those were.

| Suite | Base | Fine-tune | Change |
| --- | --- | --- | --- |
| Agent suite, all tasks | | | |
| Agent suite, retrieval | | | |
| Agent suite, tools | | | |
| Agent suite, reasoning | | | |
| Tool-call parse rate | | | |
| Tool-call right-tool rate | | | |
| Tool-call schema-valid rate | | | |
| Tool-call false-call rate | | | |
| General task set, per category | | | |

| Cost | Base | Fine-tune |
| --- | --- | --- |
| Mean steps per task | | |
| Mean tokens per task | | |
| Mean seconds per task | | |

Repeats per task: ____. Spread across repeats: ____. State whether the change is larger than
the spread. If it is not, the honest sentence is "this run cannot tell".

## 5. What got worse

List every regression, including the ones small enough to be tempting to leave out.

-
-

If the false-call rate rose, give it its own line. A model that learned to always reach for
a tool is a worse agent than one that sometimes answers directly, and the overall success
rate can hide it.

## 6. What I would do differently

- The change I would make to the collection set:
- The change I would make to the filters:
- Whether a fine-tune was the right instrument at all, now that I have the numbers:

## 7. Reproduction

- Commands, in order, with the arguments actually used:
- Versions of transformers, TRL, PEFT and the serving engine:
- Lab-notebook run ids for every stage: