Lab: One Task, Six Agents

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The tool versions, model files and per-track wall-clock figures each track was run with belong here once the validation pass has run this lab on real machines.

Objective

By the end of this lab you will have run one small, test-defined coding task through six different agents against the same local model, recorded what each one cost and whether it worked, repeated the whole thing with a second model, and produced a table that separates two things people usually confuse: what the tool contributes and what the model contributes.

The deliverable is that table, plus six transcripts you have actually read. The transcripts matter as much as the numbers. A tool that passes in four turns and a tool that passes in forty both show “pass” in the outcome column, and only the transcript tells you which one you want on a Tuesday afternoon.

⌂Single-machine path

Single-machine on every track. The gateway, the engines and all six tools run on the same box. Nothing here needs a second machine.

Requirements

The Part 9 gateway running with at least two model aliases, the task files from this page, and the six tools installed and configured as their lessons in this part described. Seventy-five minutes, almost all of it attended, because reading the transcripts is the lab.

One virtual key per tool. This is the small piece of setup that makes the token column possible. The gateway project’s virtual keys give per-key usage records, so a key named for each tool turns “how many tokens did this cost” from an estimate into a lookup.

Fragment — not complete on its own

curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer ${GATEWAY_MASTER_KEY}" \
  -H 'Content-Type: application/json' \
  -d '{"models": ["local/coder", "local/chat"], "metadata": {"application": "aider"}}'

That is a fragment because it is the gateway project’s own key-generation call with a different metadata label; run it once per tool, with the tool’s name in the metadata, and keep the six keys somewhere you will not commit them.

Track S — NVIDIA DGX Spark

All six tools run natively. With 128 GB of unified memory you can hold both models resident, so the second pass costs no reload time; configure both aliases in llama-swap.yaml in a persistent group before you start, and the whole lab runs without a model swap.

Allow extra time on the first run of each tool for its own installation and first-launch configuration, which is not part of the seventy-five minutes.

Track X — AMD Ryzen AI Max+ 395

All six tools run natively. Remember Part 5’s cap on GPU-visible memory: the machine’s total is not what the engine may use, and a 30B-class coder at a long context is exactly the case where that bites. If the coder alias does not fit, run the lab with the 16 GB-tier model as the primary and a smaller one as the comparison, and say so in the table’s context.

Track M — Apple silicon

All six tools run natively. Two constraints are worth planning around. The models come out of the same unified pool as the editor, the browser and the tools themselves, so close what you do not need before the timed runs. And OpenHands’ container path needs Docker Desktop; if you would rather not install it, use an editor agent by hand as the sixth tool and record it with the script’s manual mode.

Track N — NVIDIA desktop or laptop

All six tools run natively. On a 16 GB card the coder alias at a long context will not fit alongside anything else, so run the models one at a time and let llama-swap handle the swap between passes; the swap time is not part of the wall-clock figure because the script starts its clock after the tool launches, but the first request of a pass will be slow and you should note that in the table rather than treating it as a tool difference.

The task

Small, self-contained, defined by an executable test suite, and genuinely broken in five separate ways. It is deliberately not hard: a task that only the strongest model can finish measures the model, and this lab is trying to measure the tools.

RunnableAll tracks

task-readme.md

# Usage-log summariser

Purpose: the task statement an agent is given in the "one task, six agents" lab.
Platform: all (spark, strix, mac, nvidia).
Minimum memory: none of consequence; this file is the task, not the model.
Assumes: `task-app.py` and `task-tests.py` beside it, Python 3.10 or newer, and pytest.

---

`task-app.py` turns the JSON-lines usage log a model gateway writes into a small table:
one row per model, with the number of calls, the total tokens and the mean tokens per
call.

It does not currently do that correctly. `task-tests.py` describes what correct means.

## What to do

Make every test in `task-tests.py` pass.

```
python3 -m pytest -q task-tests.py
```

## Rules

1. **Do not edit `task-tests.py`.** The tests are the requirement. Changing them is
   failing the task, not passing it.
2. Change the application code only. `task-app.py` is the file under repair.
3. Run the tests before you change anything, so you know the starting state, and after
   every change, so you know what your change did.
4. Change one thing at a time.
5. When the suite passes, say in one or two sentences what was wrong.
6. If several attempts in a row make no progress, stop and say what you tried and what
   you would need in order to continue. Do not keep repeating an edit that failed.

## What the summariser is supposed to do

A usage log is a text file with one JSON object per line, like this:

```
{"model": "local/coder", "prompt_tokens": 100, "completion_tokens": 20}
{"model": "local/chat", "prompt_tokens": 5, "completion_tokens": 6}
```

Real logs are not tidy. They contain blank lines, and they contain the occasional
truncated or corrupt line where a process was killed mid-write. A summariser that stops
at the first bad line is useless on the day you need it.

The report puts the busiest model first, because the question people ask of a usage log is
"what is using all the tokens". A model's total is the sum of its prompt and completion
tokens across every call. The mean column is tokens divided by calls, rounded to the
nearest whole token.

An empty log is an ordinary state on a quiet day and should produce a plain message rather
than an error.

## Scope

This is deliberately a small task. It fits in one file, its requirements are stated by an
executable test suite, and a competent agent should finish it in a handful of turns. That
is what makes it useful for comparing tools: differences in the transcripts are differences
between the tools rather than differences in how hard the problem was.

Download task-readme.md62 lines

RunnableAll tracks

task-app.py

"""Summarise a gateway usage log into a per-model report.

Purpose: the application under repair for the "one task, six agents" lab. It reads the
    JSON-lines usage log the Part 9 gateway writes and turns it into a small table of
    calls, tokens and mean tokens per call. It contains several genuine defects, and the
    accompanying test suite defines what correct behaviour is.
Platform: all (spark, strix, mac, nvidia). Pure standard library; no accelerator needed.
Minimum memory: none of consequence; this file is the task, not the model.
Assumes: Python 3.10 or newer and pytest available for the test suite. Run from the
    directory containing task-tests.py.
Usage:
    python3 task-app.py usage.jsonl
    python3 -m pytest -q task-tests.py

Do not read further than you need to. The point of the lab is to watch an agent find the
defects from the failing tests, so knowing where they are in advance changes what you are
measuring.
"""

from __future__ import annotations

import json
import sys
from pathlib import Path


def parse_usage_lines(text: str) -> list[dict]:
    """Turn the text of a JSON-lines usage log into a list of records."""
    records = []
    for line in text.splitlines():
        records.append(json.loads(line))
    return records


def total_tokens(record: dict) -> int:
    """The number of tokens one logged call consumed."""
    return int(record.get("prompt_tokens", 0))


def summarise(records: list[dict]) -> list[dict]:
    """Aggregate records into one row per model."""
    totals: dict[str, int] = {}
    calls: dict[str, int] = {}

    for record in records:
        model = record.get("model", "unknown")
        totals[model] = totals.get(model, 0) + total_tokens(record)
        calls[model] = calls.get(model, 0) + 1

    rows = []
    for model in sorted(totals):
        rows.append(
            {
                "model": model,
                "calls": calls[model],
                "tokens": totals[model],
                "mean_tokens": totals[model] / calls[model],
            }
        )
    return rows


def format_report(rows: list[dict]) -> str:
    """Render summary rows as a fixed-width table."""
    width = max(len(row["model"]) for row in rows)
    header = f"{'model'.ljust(width)}  calls  tokens  mean"
    lines = [header]
    for row in rows:
        lines.append(
            f"{row['model'].ljust(width)}  "
            f"{row['calls']:5d}  "
            f"{row['tokens']:6d}  "
            f"{int(row['mean_tokens']):4d}"
        )
    return "\n".join(lines)


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: python3 task-app.py <usage.jsonl>", file=sys.stderr)
        return 2
    text = Path(argv[1]).read_text(encoding="utf-8")
    print(format_report(summarise(parse_usage_lines(text))))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Download task-app.py88 lines

RunnableAll tracks

task-tests.py

"""The test suite that defines correct behaviour for task-app.py.

Purpose: six failing tests that specify what the usage-log summariser is supposed to do.
    An agent's job in the lab is to make every one of them pass without editing this file.
Platform: all (spark, strix, mac, nvidia). Pure standard library plus pytest.
Minimum memory: none of consequence.
Assumes: pytest installed, and task-app.py in the same directory. The module name has a
    hyphen in it, which is not importable with a plain import statement, so the loader
    below reads it by path. Leave that loader alone; it is not part of the task.
Usage:
    python3 -m pytest -q task-tests.py
"""

from __future__ import annotations

import importlib.util
from pathlib import Path

_SPEC = importlib.util.spec_from_file_location(
    "task_app", Path(__file__).with_name("task-app.py")
)
assert _SPEC is not None and _SPEC.loader is not None
app = importlib.util.module_from_spec(_SPEC)
_SPEC.loader.exec_module(app)


LOG = "\n".join(
    [
        '{"model": "local/coder", "prompt_tokens": 100, "completion_tokens": 20}',
        "",
        '{"model": "local/chat", "prompt_tokens": 5, "completion_tokens": 6}',
        "this line is not json and a real log will contain one eventually",
        '{"model": "local/chat", "prompt_tokens": 4, "completion_tokens": 6}',
        '{"model": "local/chat", "prompt_tokens": 5, "completion_tokens": 6}',
    ]
)


def test_parse_skips_blank_and_malformed_lines():
    """A log with a blank line and a corrupt line still parses into four records."""
    records = app.parse_usage_lines(LOG)
    assert len(records) == 4
    assert records[0]["model"] == "local/coder"


def test_total_tokens_counts_prompt_and_completion():
    """A call costs its prompt tokens plus its completion tokens."""
    record = {"model": "local/coder", "prompt_tokens": 100, "completion_tokens": 20}
    assert app.total_tokens(record) == 120


def test_total_tokens_tolerates_missing_fields():
    """A record missing a token field counts what it has rather than raising."""
    assert app.total_tokens({"model": "local/chat"}) == 0
    assert app.total_tokens({"model": "local/chat", "completion_tokens": 7}) == 7


def test_summary_is_ordered_by_tokens_descending():
    """The busiest model comes first, so the report answers the question it is asked."""
    rows = app.summarise(app.parse_usage_lines(LOG))
    assert [row["model"] for row in rows] == ["local/coder", "local/chat"]


def test_summary_reports_calls_and_mean_tokens():
    """Each row carries the call count and the mean tokens per call."""
    rows = app.summarise(app.parse_usage_lines(LOG))
    chat = next(row for row in rows if row["model"] == "local/chat")
    assert chat["calls"] == 3
    assert chat["tokens"] == 32
    assert abs(chat["mean_tokens"] - 32 / 3) < 1e-9


def test_format_report_rounds_the_mean():
    """The mean column is rounded to the nearest token, not truncated towards zero."""
    report = app.format_report(app.summarise(app.parse_usage_lines(LOG)))
    rows = [line.split() for line in report.splitlines()]
    assert rows[1] == ["local/coder", "1", "120", "120"]
    assert rows[2] == ["local/chat", "3", "32", "11"]


def test_format_report_handles_an_empty_log():
    """An empty log is a normal state on a quiet day, not an error."""
    assert app.format_report(app.summarise([])) == "no calls recorded"

Download task-tests.py83 lines

!Important

Do not read task-app.py looking for the bugs before you run the lab. Once you know where they are, you cannot read a transcript without knowing the answer, and half of what this lab teaches is how an agent’s transcript looks when it is on the right track versus when it is guessing. Run the tests, see seven failures, and start.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-25-coding-agents"
cd "$LAB_DIR"
pwd
test -f "task-readme.md"

Expected result: pwd ends in part-25-coding-agents and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Put the task in place and confirm it fails

RunnableAll tracks

a fresh copy of the task, and the starting state

mkdir -p ~/agent-lab/task
cd ~/agent-lab/task
python3 -m pytest -q task-tests.py

Output — what you should see

7 failed in 0.05s

Seven failures is the correct starting state. If you see an import error instead, pytest is not installed in the environment you are running from; install it before going further, because five of the six tools will run that same command as their feedback loop.

2. Take the runner

One script, one tool argument. It resets the working copy from the pristine task before every run, so every tool starts from byte-identical files, times the run, applies the verdict itself by running the tests afterwards, checks whether the agent cheated by editing the tests, and appends one JSON line to the notebook.

RunnableAll tracks

run-agent-task.sh

#!/usr/bin/env bash
# Purpose: run one coding task through one agent, from an identical starting state every
#          time, and record wall-clock, outcome, tokens and the size of the change as one
#          JSON line in the lab notebook.
# Platform: all (spark, strix, mac, nvidia). Track M runs it natively or inside the
#          sandbox container from this part's second lab.
# Minimum memory: 16 GB for a 30B-class coder behind the gateway alias
# Assumes: the Part 9 gateway is running and MODEL names one of its aliases; the tool being
#          tested is installed and already configured against that gateway (each tool's
#          lesson in this part shows how); task-app.py, task-tests.py and task-readme.md
#          are in $TASK_SRC; and python3 with pytest is available.
#          Flags used for each tool are the ones documented on that tool's pages as read on
#          2026-09-09. Confirm them against your installed version with --help before the
#          first run; a flag that has been renamed fails inside an unattended run.
#
# Usage:
#   ./run-agent-task.sh aider
#   TOKENS=48210 ./run-agent-task.sh codex
#   USAGE_LOG=~/gateway/usage.jsonl ./run-agent-task.sh opencode

set -euo pipefail

TOOL="${1:-}"
if [ -z "$TOOL" ]; then
    echo "usage: $0 <aider|codex|opencode|claude|goose|openhands|manual>" >&2
    exit 2
fi

TASK_SRC="${TASK_SRC:-$PWD/task}"
WORK="${WORK:-$PWD/run-$TOOL}"
LABBOOK="${LABBOOK:-$PWD/labbook.md}"
MODEL="${MODEL:-local/coder}"
TEST_CMD="${TEST_CMD:-python3 -m pytest -q task-tests.py}"
TIMEOUT_SECONDS="${TIMEOUT_SECONDS:-1800}"
# Tokens for this run. Read it from the gateway's usage records for the virtual key you
# gave this tool, and pass it in; the script fills it in itself only when USAGE_LOG points
# at a JSON-lines file the gateway appends to.
TOKENS="${TOKENS:-}"
USAGE_LOG="${USAGE_LOG:-}"

PROMPT="Read task-readme.md, then make every test in task-tests.py pass by changing only \
task-app.py. Do not edit task-tests.py. Run the tests before and after each change. When \
the suite passes, state in one sentence what was wrong."

for required in python3 git; do
    if ! command -v "$required" >/dev/null 2>&1; then
        echo "$required is required and was not found on PATH." >&2
        exit 1
    fi
done

if [ ! -d "$TASK_SRC" ]; then
    echo "Task directory not found: $TASK_SRC" >&2
    echo "It must contain task-app.py, task-tests.py and task-readme.md." >&2
    exit 1
fi

# ------------------------------------------------------------------ a clean start, always
# Every tool gets a byte-identical starting state. Without this the second tool inherits
# the first one's edits and the comparison measures nothing.
rm -rf "$WORK"
mkdir -p "$WORK"
cp "$TASK_SRC"/task-app.py "$TASK_SRC"/task-tests.py "$TASK_SRC"/task-readme.md "$WORK"/
cp "$WORK"/task-app.py "$WORK"/.task-app-original.py

usage_bytes() {
    if [ -n "$USAGE_LOG" ] && [ -f "$USAGE_LOG" ]; then
        wc -c <"$USAGE_LOG" | tr -d ' '
    else
        echo 0
    fi
}

USAGE_BEFORE="$(usage_bytes)"

# ------------------------------------------------------------------------- the tool itself
run_tool() {
    case "$TOOL" in
        aider)
            aider --model "openai/$MODEL" \
                --message "$PROMPT" \
                --test-cmd "$TEST_CMD" \
                --auto-test \
                --no-auto-commits \
                --yes-always \
                task-app.py
            ;;
        codex)
            codex exec \
                --sandbox workspace-write \
                --ask-for-approval never \
                "$PROMPT"
            ;;
        opencode)
            opencode run --model "gateway/$MODEL" "$PROMPT"
            ;;
        claude)
            claude --print --permission-mode acceptEdits "$PROMPT"
            ;;
        goose)
            goose run --recipe "$TASK_SRC/goose-recipe.yaml" \
                --params "project_dir=$WORK" \
                --no-session \
                --max-turns 40
            ;;
        openhands)
            openhands --headless -t "$PROMPT"
            ;;
        manual)
            echo "Do the task by hand in your editor agent, in $WORK, then press Enter."
            read -r _
            ;;
        *)
            echo "Unknown tool: $TOOL" >&2
            exit 2
            ;;
    esac
}

START="$(date +%s)"
TOOL_STATUS=0
(
    cd "$WORK"
    run_tool
) >"$WORK/transcript.txt" 2>&1 || TOOL_STATUS=$?
END="$(date +%s)"
ELAPSED=$(( END - START ))

if [ "$ELAPSED" -ge "$TIMEOUT_SECONDS" ]; then
    echo "Note: the run took at least the configured timeout of ${TIMEOUT_SECONDS}s." >&2
fi

# ---------------------------------------------------------------------------- the verdict
TESTS_STATUS=0
(
    cd "$WORK"
    eval "$TEST_CMD"
) >"$WORK/test-output.txt" 2>&1 || TESTS_STATUS=$?

TESTS_EDITED=false
if ! diff -q "$TASK_SRC/task-tests.py" "$WORK/task-tests.py" >/dev/null 2>&1; then
    TESTS_EDITED=true
fi

CHANGED_LINES="$(diff -u "$WORK/.task-app-original.py" "$WORK/task-app.py" \
    | grep -c -E '^[+-][^+-]' || true)"

USAGE_AFTER="$(usage_bytes)"
if [ -z "$TOKENS" ] && [ -n "$USAGE_LOG" ] && [ "$USAGE_AFTER" -gt "$USAGE_BEFORE" ]; then
    TOKENS="$(USAGE_LOG="$USAGE_LOG" OFFSET="$USAGE_BEFORE" python3 - <<'PY'
import json
import os

total = 0
with open(os.environ["USAGE_LOG"], "r", encoding="utf-8", errors="replace") as handle:
    handle.seek(int(os.environ["OFFSET"]))
    for line in handle:
        line = line.strip()
        if not line:
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            continue
        total += int(record.get("prompt_tokens", 0)) + int(record.get("completion_tokens", 0))
print(total)
PY
)"
fi

# ------------------------------------------------------------------- one line, one result
TOOL="$TOOL" MODEL="$MODEL" ELAPSED="$ELAPSED" TOOL_STATUS="$TOOL_STATUS" \
TESTS_STATUS="$TESTS_STATUS" TESTS_EDITED="$TESTS_EDITED" \
CHANGED_LINES="$CHANGED_LINES" TOKENS="${TOKENS:-}" LABBOOK="$LABBOOK" \
python3 - <<'PY'
import datetime
import json
import os

tokens = os.environ.get("TOKENS") or ""
record = {
    "lab": "part-25-one-task-six-agents",
    "recorded": datetime.datetime.now(datetime.timezone.utc)
    .replace(microsecond=0)
    .isoformat(),
    "tool": os.environ["TOOL"],
    "model_alias": os.environ["MODEL"],
    "wall_clock_seconds": int(os.environ["ELAPSED"]),
    "tool_exit_status": int(os.environ["TOOL_STATUS"]),
    "tests_pass": os.environ["TESTS_STATUS"] == "0",
    "tests_were_edited": os.environ["TESTS_EDITED"] == "true",
    "changed_lines": int(os.environ["CHANGED_LINES"] or 0),
    "tokens": int(tokens) if tokens.isdigit() else None,
}
with open(os.environ["LABBOOK"], "a", encoding="utf-8") as handle:
    handle.write(json.dumps(record) + "\n")
print(json.dumps(record, indent=2))
PY

echo
echo "transcript:  $WORK/transcript.txt"
echo "test output: $WORK/test-output.txt"
echo "recorded in: $LABBOOK"

if [ "$TESTS_EDITED" = "true" ]; then
    echo
    echo "WARNING: task-tests.py was modified. This run does not count as a pass;" >&2
    echo "record it as a failure and note what the agent changed." >&2
fi

Download run-agent-task.sh209 lines

What one recorded run does

1. 1ResetDelete the working directory and copy the pristine task files in. Without this, tool two inherits tool one's edits.
2. 2Snapshot the usage logRecord its size, so the tokens consumed by this run can be read from the lines appended during it.
3. 3Run the toolOne documented non-interactive invocation per tool, with everything captured to transcript.txt.
4. 4Stop the clockWall clock, not model time. Your test suite is part of the cost and belongs in the number.
5. 5Judge independentlyThe script runs the tests itself. What the agent claimed is not evidence; the exit status is.
6. 6Check for cheatingCompare task-tests.py with the original. An edited test file makes the run invalid, not a pass.
7. 7Record one JSON lineTool, model alias, seconds, tokens, outcome, lines changed, into labbook.md.

The two accented steps are what make this a measurement rather than an impression. Resetting means the tools are comparable; judging independently means the result is not the agent's opinion of its own work.

✕Common mistake

Believing the agent’s summary. Every one of these tools will, sooner or later, end a run by announcing that the tests now pass when they do not. It is the single most common false positive in agent work and it is why this script runs the suite itself afterwards rather than reading the transcript. Never let the thing being measured supply the measurement.

3. Run the first tool, and read the transcript

Start with Aider, because it is the simplest and because a failure here is almost always configuration rather than the model.

RunnableAll tracks

the first recorded run

cd ~/agent-lab
OPENAI_API_BASE=http://127.0.0.1:4000 \
OPENAI_API_KEY="${AIDER_GATEWAY_KEY}" \
MODEL=local/coder \
bash run-agent-task.sh aider

Output — what you should see

{
  "lab": "part-25-one-task-six-agents",
  "tool": "aider",
  "model_alias": "local/coder",
  "wall_clock_seconds": 233,
  "tests_pass": true,
  "tests_were_edited": false,
  "changed_lines": 16,
  "tokens": null
}

Now read run-aider/transcript.txt end to end. Not skim: read. Note four things and write them down, because they are what the numbers cannot tell you.

• How many times it ran the tests. A tool that runs them once at the end is guessing; a tool that runs them after every edit is working.
• Whether it read the readme. Several tools will fix the tests it can see and miss the requirement that only the readme states.
• What it did with the malformed-line failure, which is the one that requires reading the intent rather than pattern-matching the assertion.
• Whether it ever tried to edit task-tests.py. Some models, when a test is hard, delete it.

4. Run the other five

Each tool needs its own key in the environment and its own configuration from the lessons in this part. The invocations are inside the script, so this step is one command per tool.

RunnableAll tracks

the remaining five, one at a time

cd ~/agent-lab
MODEL=local/coder bash run-agent-task.sh codex
MODEL=local/coder bash run-agent-task.sh opencode
MODEL=local/coder bash run-agent-task.sh claude
MODEL=local/coder bash run-agent-task.sh goose
MODEL=local/coder bash run-agent-task.sh openhands

Run them one at a time and read each transcript before starting the next. Running them in parallel would be faster and would ruin the wall-clock column, because six agents sharing one engine queue behind each other.

✱Hardware note

On any track where the model does not fit twice, the tools are contending for one engine anyway. That is fine as long as only one runs at a time. If you have the memory to run two engines and you choose to parallelise, say so beside the table, because the wall-clock column then measures your scheduling rather than the tools.

Two of the six need a note.

Claude Code may refuse, error, or behave oddly against a local endpoint, and if it does, that is a result rather than a failure of the lab. Record it as a tool error with a sentence about what happened. Its own vendor documentation says routing it to non-Claude models through a gateway is not supported, and this lab is where you find out concretely what that means on your machine.

OpenHands starts containers. If you have not read the container note in the Goose and OpenHands lesson, read it before this step. If you would rather not give a tool the Docker socket today, use an editor agent by hand instead and record it with bash run-agent-task.sh manual, which times you instead.

5. Fill in the token column

The script records tokens automatically only when USAGE_LOG points at a JSON-lines file the gateway appends to. Otherwise, read the per-key usage from the gateway for each tool’s virtual key and re-record the run with the figure:

Fragment — not complete on its own

TOKENS=48210 MODEL=local/coder bash run-agent-task.sh aider

If you would rather not re-run, note the tokens beside the table by hand. The important thing is that the number comes from the gateway rather than from the tool’s own report, for the same reason the verdict does.

6. Repeat with a second model

Change one variable. Same six tools, same task, same script, different alias.

RunnableAll tracks

the second pass

cd ~/agent-lab
MODEL=local/chat bash run-agent-task.sh aider
MODEL=local/chat bash run-agent-task.sh codex
MODEL=local/chat bash run-agent-task.sh opencode

Use a model from a different tier: if the first pass used the 24 GB-tier coder, use the 16 GB or 8 GB-tier model here. The point is not to find a winner. The point is to see which column moves when the model changes and which moves when the tool changes, and those are usually different columns.

7. Produce the table

RunnableAll tracks

summarise-agent-runs.py

"""Turn the lab notebook's JSON lines into the lab's results table.

Purpose: read the records run-agent-task.sh appended to labbook.md and print one row per
    tool and model, so the comparison in the "one task, six agents" lab is produced from
    the recorded runs rather than retyped by hand.
Platform: all (spark, strix, mac, nvidia). Standard library only.
Minimum memory: none of consequence.
Assumes: labbook.md contains one JSON object per line for lab
    "part-25-one-task-six-agents", written by run-agent-task.sh. Lines that are not JSON
    are ignored, so a notebook that also contains prose is fine.
Usage:
    python3 summarise-agent-runs.py labbook.md
    python3 summarise-agent-runs.py labbook.md --model local/coder
    python3 summarise-agent-runs.py labbook.md --format markdown
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

LAB = "part-25-one-task-six-agents"
COLUMNS = ("Tool", "Model alias", "Outcome", "Wall clock s", "Tokens", "Lines changed")


def read_records(path: Path, lab: str) -> list[dict]:
    """Every JSON line in the notebook that belongs to this lab."""
    records = []
    for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
        line = line.strip()
        if not line.startswith("{"):
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            continue
        if record.get("lab") == lab:
            records.append(record)
    return records


def outcome(record: dict) -> str:
    """A single word for what happened, which is what the results table wants."""
    if record.get("tests_were_edited"):
        return "invalid (tests edited)"
    if record.get("tests_pass"):
        return "pass"
    if record.get("tool_exit_status", 0) != 0:
        return "tool error"
    return "fail"


def to_rows(records: list[dict]) -> list[list[str]]:
    rows = []
    for record in records:
        tokens = record.get("tokens")
        rows.append(
            [
                str(record.get("tool", "?")),
                str(record.get("model_alias", "?")),
                outcome(record),
                str(record.get("wall_clock_seconds", "")),
                "not recorded" if tokens is None else str(tokens),
                str(record.get("changed_lines", "")),
            ]
        )
    return rows


def render_fixed_width(rows: list[list[str]]) -> str:
    widths = [len(name) for name in COLUMNS]
    for row in rows:
        for index, cell in enumerate(row):
            widths[index] = max(widths[index], len(cell))
    lines = ["  ".join(name.ljust(widths[i]) for i, name in enumerate(COLUMNS))]
    lines.append("  ".join("-" * width for width in widths))
    for row in rows:
        lines.append("  ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)))
    return "\n".join(lines)


def render_markdown(rows: list[list[str]]) -> str:
    lines = ["| " + " | ".join(COLUMNS) + " |"]
    lines.append("| " + " | ".join("---" for _ in COLUMNS) + " |")
    for row in rows:
        lines.append("| " + " | ".join(row) + " |")
    return "\n".join(lines)


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("labbook", help="path to labbook.md")
    parser.add_argument("--model", default=None, help="only rows for this model alias")
    parser.add_argument(
        "--format",
        choices=("fixed", "markdown"),
        default="fixed",
        help="fixed-width for the terminal, markdown to paste into your notes",
    )
    args = parser.parse_args()

    path = Path(args.labbook)
    if not path.exists():
        print(f"No lab notebook at {path}. Run run-agent-task.sh first.")
        return 1

    records = read_records(path, LAB)
    if args.model:
        records = [r for r in records if r.get("model_alias") == args.model]

    if not records:
        print(f"No records for lab {LAB} in {path}.")
        return 1

    records.sort(key=lambda r: (r.get("model_alias", ""), r.get("tool", "")))
    rows = to_rows(records)

    if args.format == "markdown":
        print(render_markdown(rows))
    else:
        print(render_fixed_width(rows))

    passes = sum(1 for r in records if outcome(r) == "pass")
    print()
    print(f"{passes} of {len(records)} recorded runs left the test suite passing.")
    print("Every number above is one run. Repeat each row before drawing a conclusion.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download summarise-agent-runs.py132 lines

RunnableAll tracks

the results table, from the recorded runs

cd ~/agent-lab
python3 summarise-agent-runs.py labbook.md --format markdown

Output — what you should see

| Tool | Model alias | Outcome | Wall clock s | Tokens | Lines changed |
| --- | --- | --- | --- | --- | --- |
| aider | local/chat | fail | 412 | 61044 | 22 |
| aider | local/coder | pass | 233 | 48210 | 16 |

Keep the independent test outside the agents’ control

Record the starting commit and confirm the task’s test fails for the intended reason before any agent runs. Preserve a trusted copy of the test outside each agent’s editable checkout. A candidate that deletes or weakens its local test has not solved the task.

Give every tool a fresh copy of the same starting repository and the same task description. Keep model, budget and permissions equivalent where supported, and record unavoidable client-specific differences. Check endpoint compatibility with a read-only interaction before spending the full task budget; a protocol failure should be recorded separately from a coding failure.

After each run, save the transcript, final diff, independent test result, elapsed time, token usage and manual interventions. Inspect unrelated edits even when tests pass. If usage is unavailable, mark it unknown rather than inferring it from response length. Repeat with the second model using fresh starting states. The final table compares complete workflows under these conditions; it does not isolate a client algorithm when prompts, repository maps or API features differ. Archive each result before resetting any disposable checkout.

Validation

Pending validationOne task, six agents — your recording sheet

Tool	Model alias	Outcome	Wall clock s	Tokens	Lines changed	Test runs in transcript
Aider	—	—	—	—	—	—
Codex CLI	—	—	—	—	—	—
OpenCode	—	—	—	—	—	—
Claude Code	—	—	—	—	—	—
Goose	—	—	—	—	—	—
OpenHands or editor agent	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · the engine behind the gateway alias, with its version each tool's version, recorded per row · the alias, and the model file and quantisation behind it, the quantisation of the model behind the alias · 0 tokens of context · the date you ran it

Empty on purpose, and worth filling twice: once per model. Record the context length you configured behind the alias in place of the zero, and record each tool's version in the version field, because a tool that is updating itself weekly is not the same tool it was last month.

You are done when all of the following are true:

• labbook.md contains at least six records for the first model and three for the second, each one written by the script rather than typed;
• the results table is generated from those records rather than from memory;
• you have read all six transcripts from the first pass, and can say for each tool how many times it ran the tests;
• no row is recorded as a pass on the strength of the agent’s own claim; every pass was verified by the script running the suite afterwards;
• any run in which the agent edited task-tests.py is recorded as invalid rather than as a pass, with a note about what it changed;
• you can name one thing each tool did that the others did not.

Expected outcome

Six rows that differ in ways that surprise you, and a clearer idea of what you are buying from each tool.

The shapes to expect, none of which is a measurement and all of which your own table will make concrete. Wall-clock varies more between tools than the token count does, because much of the difference is how many times a tool runs your test suite and how much it re-reads before each turn. Token counts vary with the size of the system prompt and the number of tool schemas in it, which is why a tool with many MCP servers wired in costs more before it has done anything. The lines-changed column is a proxy for restraint: a tool that changed sixty lines to fix five bugs did something you will want to look at.

=Expected result

The most valuable outcome of this lab is a disagreement between two columns: a tool that passes slowly and a tool that fails quickly, or a tool with the lowest token count and the worst outcome. That is where the trade-off you actually care about lives, and it is invisible in any single number. If every column ranks the tools the same way, either you got lucky or your task was too easy to discriminate.

The second model’s rows should move the outcome column and leave the tool-shaped differences alone. If changing the model changes which tool is fastest, look for contention: something else was using the accelerator.

Troubleshooting

A tool authenticates against the gateway and then reports no such model. The alias in your tool’s configuration and the alias in litellm-config.yaml disagree, or the tool is adding a prefix you did not expect. Ask the gateway what it publishes and compare exactly, including the prefix each tool adds.

Aider runs, edits nothing, and exits successfully. The file was not added to the chat. The script passes task-app.py as a positional argument for exactly this reason; if you have modified the invocation, put it back.

Codex exits immediately with an error about a provider. Its documentation states that with neither --oss nor oss_provider set, codex exec “exits with an error” rather than prompting. Set oss_provider in config.toml, or read the wire-format note in the Codex and OpenCode lesson, because a custom provider block against a chat-completions gateway is the unconfirmed path.

Claude Code connects and then produces no tool calls. If you are using llama.cpp’s Messages endpoint directly, check that the server was started with --jinja. If you are going through the gateway, check that the alias resolves and that ANTHROPIC_AUTH_TOKEN is set rather than ANTHROPIC_API_KEY.

Goose runs and does nothing. Its documentation is explicit that models without tool calling can only do chat completion. Run Part 24’s reliability test against the alias before blaming the recipe.

Every tool fails on the same model, and quickly. That is not six tool failures; it is one model failure. Go back to the second lesson’s checklist and measure tool-call reliability and the context length behind the alias before running anything else.

The wall-clock figures are wildly inconsistent between repeats of the same tool. Something else is using the accelerator, or the model is being swapped in and out between runs. Check the engine log for loads you did not expect, and consider pinning both models resident if you have the memory.

A tool edited the tests. The script catches this and marks the run invalid. Read what it changed; it is usually the malformed-line test, which is the one that cannot be satisfied by a narrow pattern match. This is a genuine and common failure mode, and recording it honestly is worth more than a tidy table.

Cleanup

The task copies and transcripts are small and worth keeping until you have written up the table. When you are done:

RunnableAll tracks

remove the per-tool working copies, keep the notebook

cd ~/agent-lab
rm -rf run-aider run-codex run-opencode run-claude run-goose run-openhands run-manual

⚠Warning

That command removes directories. Check you are in ~/agent-lab before running it, and note that it deliberately does not touch labbook.md, task/ or the transcripts you moved elsewhere. If you want the transcripts, copy them out first; they are inside the directories being deleted.

Revoke the six virtual keys if you are not going to use them again, using the gateway project’s key management. Stop any engines you started for this lab that you do not otherwise run.

What you learned

• A results table needs an independent judge. The agent’s claim of success is not evidence. The script re-runs the suite, and that single design decision is the difference between a measurement and a transcript of six programs being optimistic.
• Identical starting state is what makes tools comparable. Resetting from a pristine copy before every run is not tidiness; without it the second tool is solving a different problem.
• Wall-clock and tokens measure different things. One is dominated by how often a tool runs your tests, the other by how much context it carries. A tool can win one column and lose the other, and that is the trade-off worth knowing about.
• The transcript carries what the table cannot. Test runs per task, whether the readme was read, what happened at the hardest failure, and whether the agent tried to change the requirement rather than meet it.
• Changing the model moves different columns from changing the tool. Doing both passes is what lets you say which is which, and it is why the lab is not finished after one model.
• Editing the tests is a failure, not a pass. It is common enough to need a check in the harness, and it is the clearest example in this course of a metric that can be gamed by the thing being measured.

Record in the notebook: the completed table for both models, the context length and quantisation behind each alias, each tool’s version, the number of test runs per transcript, one sentence per tool about what it did differently, and any run you had to mark invalid and why.

Check your understanding

Question 1. Why does the runner script re-run the test suite itself after the agent finishes?

• To save the agent time
• Because agents routinely announce success without having run the tests, so the only trustworthy verdict is one produced by something other than the agent
• Because the tests are slow
• To generate more tokens for the usage log

Show the answer and why

Answer: Because agents routinely announce success without having run the tests, so the only trustworthy verdict is one produced by something other than the agent

It is the same principle as everywhere else in this course: the thing being measured does not supply the measurement. A false claim of success is the commonest failure mode in agent work and the cheapest one to catch.

Question 2. Why does the script delete and recreate the working directory before every run?

• To free disk space
• So that every tool starts from byte-identical files, otherwise the second tool inherits the first one's edits and the comparison measures nothing
• Because agents cannot overwrite files
• To reset the gateway usage log

Show the answer and why

Answer: So that every tool starts from byte-identical files, otherwise the second tool inherits the first one's edits and the comparison measures nothing

An agent that starts from a half-fixed codebase has an easier task. Without the reset, the ordering of the runs becomes a variable, and the table stops being about the tools.

Question 3. A run shows tests_were_edited as true and tests_pass as true. How should it be recorded?

• As a pass, since the tests pass
• As invalid, with a note about what the agent changed, because the tests are the requirement and altering them is failing the task
• As a tool error
• It should be deleted from the notebook

Show the answer and why

Answer: As invalid, with a note about what the agent changed, because the tests are the requirement and altering them is failing the task

The suite defines correctness. An agent that edits it has changed the goalposts, and the harness checks for this precisely because it happens often enough to matter. Recording it honestly is more informative than a clean-looking table.

Question 4. Which of these are reasons to run the six tools one at a time rather than in parallel? Select all that apply.

• Parallel runs queue behind one engine, so the wall-clock column measures scheduling rather than the tool
• You need to read each transcript before the next run
• Parallel runs would exceed the gateway's key limit
• Contention for the accelerator makes repeats inconsistent

Show the answer and why

Answer: Parallel runs queue behind one engine, so the wall-clock column measures scheduling rather than the tool, You need to read each transcript before the next run, Contention for the accelerator makes repeats inconsistent

The key limit is not the constraint; the accelerator is. Serial runs keep the wall-clock column meaningful and give you time to read the transcript while it is fresh, which is where most of the lab's value is.

Question 5. After the second pass, changing the model changed which tool was fastest. What should you suspect first?

• That some tools are optimised for particular models
• Contention: something else was using the accelerator during one of the passes, or a model swap happened mid-run
• That the task was too easy
• That the gateway mis-routed the requests

Show the answer and why

Answer: Contention: something else was using the accelerator during one of the passes, or a model swap happened mid-run

Tool-shaped differences are mostly about how much a tool reads and how often it runs your tests, and those do not change when the weights change. A reordering usually means the machine, not the software, was different between the passes.

Check my answersClear