Skip to content
Local LLM Course

Improving a Model for Your Agents

By the time you reach this part you have two things you did not have in Level 3. You have agents that produce transcripts: every run of Part 24’s loop, Part 25’s coding tools and Part 26’s harness writes down what the model was asked, what it called, what came back and whether the answer was right. And you have a suite that says whether an agent is any good, with a success rate, a step count and a token count per task.

Those two artefacts are a training set and a reward function wearing different clothes. A transcript that ends in a correct answer is an example of the behaviour you want. A test that passes or fails is a reward that a program can compute. This part turns them into a better small model for your own work, and then measures the result on the suite that produced them.

It is also the part where the course is most likely to tell you that the thing you just built did not help. That is not a failure of the method. A fine-tune on a few hundred of your own trajectories reliably changes format and tool discipline; it does not reliably change whether a four-billion-parameter model can reason its way through a task it could not do before. Knowing which of those you are buying, before you spend an evening on it, is most of the skill.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • log an agent run in a form that can be trained on, and say why a log of tool names and results is not one;
  • scrub credentials, addresses and machine paths out of a transcript, and say what a pattern set cannot find;
  • convert trajectories into the conversational format with a tool list that TRL’s supervised fine-tuning trainer and the Transformers chat templates expect, deduplicate them, and keep them clear of the tasks you will measure with;
  • fine-tune a small model on successful trajectories with an adapter, with the loss on the assistant turns only, and say what that changes and what it does not;
  • write a reward function whose verdict is a test suite rather than an opinion, run it in a bounded sandbox, and budget a small reinforcement-learning run per platform track;
  • generate teacher trajectories from a large model on your own tasks and distil them into a student you can serve;
  • measure a fine-tune against the model it came from on the agent suite, on tool-call reliability and on your general task set, and report the regressions as prominently as the gain.

What you need first

Section titled “What you need first”

Level 3 and the first three parts of Level 5. Concretely: Part 13 for adapters, the export path and the habit of measuring against the base; Part 14 for GRPO and for its reality check on what reinforcement learning does to small models; Part 15 for the distillation pipeline and the filtering that makes it work; Part 24 for the agent loop and the tool-call reliability test; Part 25 for the workstation and the sandbox; and Part 26 for the evaluation suite this part is measured with.

You also need a machine that can both train a small model and serve one, though not necessarily at the same time. The memory floor for the lab is 16 GB. Agent trajectories are long, so the sequence length rather than the parameter count is what makes this part heavier than Part 13.

How to work through it

Section titled “How to work through it”

Read the four lessons in order. The first is about logging and data hygiene and is the one people skip; it is also the one that decides whether anything after it means anything, and it ships the two scripts the rest of the part depends on. The second is the supervised path and the one most readers should actually run. The third is reinforcement learning on agent tasks, taught honestly as something that is within reach on one machine only in a narrow form. The fourth applies Part 15’s distillation pipeline to trajectories.

Then do the lab with the gateway running. It takes seventy-five minutes, of which about forty are unattended, and it ends with a table that says what improved, what got worse and whether the difference is bigger than the run-to-run spread.

Part27 of 27Level5 — Agentic EngineerPages5Estimated time3.1 hours

0 / 5 lessons in this part completed