Evaluating Agents: Trajectories, Success Rates and Cost
By the end of this lesson you will have a task suite whose answers a program can check, a harness that runs it against any agent you build in this course, trajectory files you can read when something fails, and a report with four numbers per configuration: success rate, steps, tokens and cost. You will also know how many times to repeat a run before believing a difference between two agents.
Everything measured in this course so far has had a comparable shape. Part 10 built a task set for models, Part 16 ran a standard benchmark suite, and Part 23 turned throughput into a cost per million tokens. An agent needs all three ideas at once, plus one that is new: an agent can reach the right answer by a route you would not accept, and only the trajectory shows you that.
A task suite you can check without a judge
Section titled “A task suite you can check without a judge”The first design decision is the one that makes everything else possible: every task has an outcome a program can check.
τ-bench makes the same choice, in a stronger form. It evaluates by comparing “the database state at the end of a conversation with the annotated goal state” — not the wording of the reply, the state of the world. Where you can arrange that, do; a task whose success is “the file now contains X” is beyond argument. Where you cannot, the next best thing is a string, a regular expression, a required tool call or a step limit.
The suite that ships with this part uses all four kinds of check, over the three course-authored documents Part 10 already gave you.
Fragment — not complete on its own
{ "name": "Part 26 agent task suite", "version": "1.0.0", "comment": "Fifteen tasks with checkable outcomes, over the three course-authored documents from Part 10's sample-docs (machine-inventory.md, model-policy.md, service-runbook.md). Copy those three files into the agent's workspace, or point the agent at the Part 10 index built from them, or both. Every expected answer is stated in one of those documents, so nothing here can be answered from a model's own memory, and every check is a string, a regular expression, a tool name or a step count rather than a judgement. The suite is deliberately small: it detects a large difference between two scaffolds and cannot detect a small one.", "workspace": "the three documents from src/labs/part-10-models-at-work/sample-docs/", "tasks": [ { "id": "r-backup-window", "category": "retrieval", "task": "During which hours, and on which day of the week, does the backup window run? Answer with the times and the day.", "expect_in_answer": ["02:00", "04:00", "sunday"], "max_steps": 6, "notes": "One passage in the runbook. The shortest possible successful trajectory: one search, one finish." }, { "id": "r-gateway-port", "category": "retrieval", "task": "Which port does the model gateway listen on? Answer with the number.", "expect_in_answer": ["4000"], "max_steps": 6, "notes": "The runbook's service table. Four ports appear in it, so a wrong answer is a retrieval or reading failure rather than a refusal." }, { "id": "r-log-retention", "category": "retrieval", "task": "How many days are gateway logs kept before they are deleted?", "expect_in_answer": ["28"], "max_steps": 6, "notes": "Tests whether a number survives the trip from a passage into an answer." }, { "id": "r-not-backed-up", "category": "retrieval", "task": "Name one thing the runbook says is deliberately not backed up, and say why.", "expect_regex": "(model weights|certificate authority|logs)", "max_steps": 6, "notes": "Three acceptable answers, so the check is a regular expression rather than a fixed string." }, { "id": "r-client-work", "category": "retrieval", "task": "Which machine is the only one allowed to hold unreleased client work?", "expect_in_answer": ["petrel"], "expect_not_in_answer": ["tern", "skua"], "max_steps": 6, "notes": "The forbidden strings catch an answer that hedges by naming every machine." }, { "id": "r-restore-test", "category": "retrieval", "task": "On what date was the last successful restore test?", "expect_in_answer": ["2026-07-05"], "max_steps": 6, "notes": "A date that appears once. A model answering from its own memory cannot produce it." }, { "id": "t-list-workspace", "category": "tools", "task": "List the file names in your workspace. Give every name in your final answer.", "expect_in_answer": ["machine-inventory.md", "model-policy.md", "service-runbook.md"], "expect_tool": "run_command", "max_steps": 5, "notes": "Checks that the agent reaches for a command rather than guessing at a directory listing." }, { "id": "t-read-review-date", "category": "tools", "task": "Read the file model-policy.md and give the date on which it was last reviewed.", "expect_in_answer": ["2026-09-01"], "expect_tool": "read_file", "max_steps": 5, "notes": "Named file, named field. Failure here is usually a path the file tool rejected." }, { "id": "t-grep-port", "category": "tools", "task": "Search the documents for the port number 8090 and say which service listens on it.", "expect_in_answer": ["document index"], "max_steps": 6, "notes": "Either search_documents or a grep through run_command is a valid route, so no tool is required." }, { "id": "t-count-machines", "category": "tools", "task": "How many machines are described in the machine inventory? Answer with the number and name them.", "expect_in_answer": ["3", "tern", "petrel", "skua"], "max_steps": 7, "notes": "Counting requires reading the whole document rather than the first matching passage." }, { "id": "x-escalation", "category": "reasoning", "task": "If tern becomes unrecoverable, which machine runs the front end and gateway, and which machine serves inference?", "expect_in_answer": ["skua", "petrel"], "max_steps": 8, "notes": "Two hops: the runbook names the escalation, the inventory says why skua cannot serve a model." }, { "id": "x-window-length", "category": "reasoning", "task": "How many hours long is the backup window? Answer with the number of hours.", "expect_regex": "\\b(two|2)\\b", "max_steps": 6, "notes": "Retrieval plus one subtraction. The regular expression accepts the word or the digit." }, { "id": "x-pickled-checkpoint", "category": "reasoning", "task": "A model you want is published only as a pickled checkpoint. Under the model policy, may it be downloaded? Answer yes or no, and give the rule.", "expect_in_answer": ["no"], "expect_regex": "(pickled|safetensors|GGUF)", "max_steps": 7, "notes": "Applies a written rule to a case the document does not state directly." }, { "id": "x-untrusted-machine", "category": "reasoning", "task": "Which machine is treated as physically untrusted, and what is the reason given?", "expect_in_answer": ["skua", "garage"], "max_steps": 7, "notes": "The reason and the conclusion are in different sections of the inventory." }, { "id": "x-refusal", "category": "reasoning", "task": "What is the wireless network password for the house?", "expect_regex": "(cannot|can't|not (?:in|stated|recorded|available|given)|no (?:answer|information|record)|does not (?:say|contain|state))", "expect_not_in_answer": ["password is"], "max_steps": 8, "notes": "The documents do not contain this. A confident invented answer fails; saying so passes. This is the single most informative task in the suite." } ]}Fifteen tasks in three categories: six retrieval tasks whose answers sit in one passage, four tool tasks that need a command or a named file, and five reasoning tasks that need two facts combined or a rule applied. Four properties of that file are worth copying into whatever suite you write for your own work.
The answers cannot be produced from memory. Every fact is invented, in documents written for the course. A task whose answer a model could know is measuring the model’s training data, not your system.
Each check is narrow and stated. expect_in_answer is a list of strings that must all
appear, expect_not_in_answer catches the answer that hedges by naming everything,
expect_regex handles the cases with several right answers, expect_tool requires a
particular tool to have been called, and max_steps fails a task that was answered by
wandering.
One task tests refusal. x-refusal asks for something the documents do not contain. An
agent that invents a plausible answer fails it; an agent that says the documents do not say
passes. On local models this is often the single most informative task in the suite, and it
is the one most people leave out.
The suite is honest about its size. Fifteen tasks detects a large difference between two configurations and cannot detect a small one. That is a property to state in your report, not a disclaimer to bury.
What a trajectory has to contain
Section titled “What a trajectory has to contain”A success rate tells you how often. A trajectory tells you why, and it is the only artefact that can. Log one file per run, as JSON lines, with a metadata line first:
What each line of a trajectory records, and what it lets you answer later
- Line 1: metadataScaffold, model alias, task id, attempt number and the task text. Without this, a directory of trajectories is unsearchable within a week.
- One line per model replyWhich role produced it, which tools it asked for, and the token counts for that call. This is what lets you attribute cost to a role rather than to the system.
- One line per tool callThe tool name, the arguments as sent, whether it errored, and a truncated result. Arguments matter: a wrong answer from a right tool is a different bug from a wrong tool.
- One line per routing decisionThe route chosen and the reason given. A router that sends everything one way is invisible in the success rate and obvious here.
- The stopping reasonFinished, turn limit, token budget, time budget, repeated call, or a failure. "Wrong answer" and "ran out of turns" need different fixes.
The harness
Section titled “The harness”The measurement problem is that you will want to compare things that are not the same kind of object: the loop from Part 24, a framework implementation, and a second framework. So the harness knows nothing about frameworks. It loads a Python file and calls one function.
Fragment — not complete on its own
def run_task(task: str, options: dict) -> dict: """Returns at least {"answer": str, "steps": int, "tokens": int}."""Two optional functions complete the contract: build(options) runs once before the first
task, so four agents are not constructed fifteen times, and close() runs once at the end.
Anything satisfying that contract can be measured, including an entry point that shells out
to a tool written in another language.
RunnableAll tracks
#!/usr/bin/env python3"""Run a task suite against any agent entry point, log the trajectories and report the cost.
Purpose: the measurement half of Part 26. It loads an agent entry point by file path, runs every task in a suite against it (optionally several times), scores each answer with deterministic checks, writes one JSON-lines trajectory file per run, and prints success rate, mean steps, mean tokens and, when you supply a cost per million tokens from Part 23's cost-model.py, a cost per task. It knows nothing about frameworks: anything that satisfies the four-function contract below can be measured with it.Platform: all (pure Python over HTTP; the models may be served on any track or machine)Minimum memory: 16 GB on the machine serving the models; this script needs almost noneAssumes: Python 3.9 or later and no third-party packages for the harness itself. The agent entry point may need whatever its framework needs. A task file in the shape of agent-tasks.json. Nothing here starts a server: the endpoints in --base-url and --option must already be answering.
The entry-point contract, which is the whole interface:
def run_task(task: str, options: dict) -> dict Required. Runs one task and returns at least {"answer": str, "steps": int, "tokens": int} and optionally "prompt_tokens", "completion_tokens", "seconds", "stopped" and "trajectory": a list of JSON-serialisable step records.
def build(options: dict) -> None Optional. Called once, before task one. def close() -> None Optional. Called once, at the end. SCAFFOLD_NAME: str Optional. Used in reports and file names.
Usage: python3 agent-eval.py --agent multi-agent-system.py --tasks agent-tasks.json \\ --model local/chat --workspace ./agent-workspace --out results.json python3 agent-eval.py --agent scaffold-minimal.py --tasks agent-tasks.json \\ --model local/router --repeats 3 --cost-per-million 0.42 --labbook labbook.md"""
from __future__ import annotations
import argparseimport importlib.utilimport jsonimport osimport platformimport reimport statisticsimport sysimport timefrom pathlib import Pathfrom typing import Any, Dict, List, Optional
# --------------------------------------------------------------------------------------# Loading an agent entry point# --------------------------------------------------------------------------------------
def load_agent(path: str): """Import a Python file as a module and check it satisfies the contract.""" file = Path(path) if not file.exists(): sys.exit(f"agent entry point {path} does not exist") spec = importlib.util.spec_from_file_location(file.stem.replace("-", "_"), file) if spec is None or spec.loader is None: sys.exit(f"cannot load {path} as a Python module") module = importlib.util.module_from_spec(spec) # The entry point may import files that sit beside it, so its directory goes on the # path before it is executed. sys.path.insert(0, str(file.resolve().parent)) spec.loader.exec_module(module) if not hasattr(module, "run_task"): sys.exit(f"{path} defines no run_task(task, options); see the contract in this file") return module
# --------------------------------------------------------------------------------------# Scoring# --------------------------------------------------------------------------------------
def check_answer(task: dict, record: dict) -> dict: """Deterministic checks only. No model grades anything here, so a score is repeatable.""" answer = (record.get("answer") or "") lowered = answer.lower() missing = [s for s in task.get("expect_in_answer", []) if s.lower() not in lowered] forbidden = [s for s in task.get("expect_not_in_answer", []) if s.lower() in lowered]
pattern = task.get("expect_regex") regex_ok = True if not pattern else bool(re.search(pattern, answer, re.IGNORECASE))
used = [step.get("tool") for step in record.get("trajectory", []) if step.get("tool")] wanted_tool = task.get("expect_tool") tool_ok = True if not wanted_tool else wanted_tool in used
limit = task.get("max_steps") steps = int(record.get("steps") or 0) within_steps = True if not limit else steps <= limit
return { "missing": missing, "forbidden": forbidden, "regex_ok": regex_ok, "tool_ok": tool_ok, "within_steps": within_steps, "tools_used": used, "passed": (not missing and not forbidden and regex_ok and tool_ok and within_steps), }
# --------------------------------------------------------------------------------------# Reporting# --------------------------------------------------------------------------------------
def summarise(rows: List[dict], cost_per_million: Optional[float]) -> dict: """Aggregate the per-run records. Every figure here is arithmetic over the runs.""" if not rows: return {"runs": 0} passed = [r for r in rows if r["checks"]["passed"]] steps = [r["steps"] for r in rows] tokens = [r["tokens"] for r in rows] seconds = [r["seconds"] for r in rows] summary = { "runs": len(rows), "passed": len(passed), "success_rate": round(len(passed) / len(rows), 3), "mean_steps": round(statistics.fmean(steps), 2), "mean_tokens": round(statistics.fmean(tokens), 1), "mean_seconds": round(statistics.fmean(seconds), 1), "total_tokens": sum(tokens), } if len(rows) > 1: summary["stdev_steps"] = round(statistics.pstdev(steps), 2) summary["stdev_tokens"] = round(statistics.pstdev(tokens), 1) if cost_per_million is not None: summary["cost_per_million_tokens"] = cost_per_million summary["cost_total"] = round(sum(tokens) / 1e6 * cost_per_million, 6) summary["cost_per_task"] = round(summary["cost_total"] / len(rows), 6) return summary
def per_task_view(rows: List[dict]) -> List[dict]: """One line per task id, so a repeated suite shows where the variance is.""" by_id: Dict[str, List[dict]] = {} for row in rows: by_id.setdefault(row["id"], []).append(row) view = [] for task_id, runs in by_id.items(): successes = sum(1 for r in runs if r["checks"]["passed"]) steps = [r["steps"] for r in runs] tokens = [r["tokens"] for r in runs] view.append({ "id": task_id, "category": runs[0]["category"], "attempts": len(runs), "successes": successes, "success_rate": round(successes / len(runs), 3), "steps_min": min(steps), "steps_max": max(steps), "tokens_min": min(tokens), "tokens_max": max(tokens), "first_failure": next((r["checks"] for r in runs if not r["checks"]["passed"]), None), }) return view
# --------------------------------------------------------------------------------------# The run# --------------------------------------------------------------------------------------
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--agent", required=True, help="Python file defining run_task(task, options); see the header") parser.add_argument("--tasks", default="agent-tasks.json") parser.add_argument("--only", default=None, help="run one category only") parser.add_argument("--task-id", default=None, help="run one task by id") parser.add_argument("--repeats", type=int, default=1, help="run every task this many times; 3 or more to see the variance") parser.add_argument("--out", default="agent-results.json") parser.add_argument("--trajectory-dir", default="trajectories")
# Everything below is handed to the agent unchanged, in the options dictionary. parser.add_argument("--base-url", default="http://127.0.0.1:4000/v1", help="OpenAI-compatible endpoint, normally the Part 9 gateway") parser.add_argument("--model", required=True, help="alias the answering model is served as") parser.add_argument("--router-model", default=None, help="alias of the small routing model") parser.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY")) parser.add_argument("--workspace", default=None, help="the only directory tools may read") parser.add_argument("--index", default=None, help="a Part 10 document index, optional") parser.add_argument("--option", action="append", default=[], metavar="KEY=VALUE", help="any further option the entry point understands; repeatable")
# Recording. parser.add_argument("--cost-per-million", type=float, default=None, help="your own cost per million tokens from Part 23's cost-model.py. " "There is no default: this script invents no price.") parser.add_argument("--quant", default="unknown", help="the quantisation actually loaded; not discoverable over the API") parser.add_argument("--engine", default="unknown", help="engine name, for the record") parser.add_argument("--engine-version", default="unknown", help="engine version, for the record") parser.add_argument("--notes", default="", help="one line about what this run is testing") parser.add_argument("--labbook", default=None) parser.add_argument("--verbose", action="store_true") args = parser.parse_args()
options: Dict[str, Any] = { "base_url": args.base_url, "model": args.model, "router_model": args.router_model or args.model, "api_key": args.api_key, "workspace": args.workspace, "index": args.index, } for item in args.option: if "=" not in item: sys.exit(f"--option needs KEY=VALUE, got {item!r}") key, value = item.split("=", 1) options[key.strip()] = value.strip()
spec = json.loads(Path(args.tasks).read_text(encoding="utf-8")) tasks = spec["tasks"] if args.only: tasks = [t for t in tasks if t.get("category") == args.only] if args.task_id: tasks = [t for t in tasks if t["id"] == args.task_id] if not tasks: sys.exit("no tasks selected")
agent = load_agent(args.agent) scaffold = getattr(agent, "SCAFFOLD_NAME", Path(args.agent).stem) if hasattr(agent, "build"): agent.build(options)
out_dir = Path(args.trajectory_dir) out_dir.mkdir(parents=True, exist_ok=True) stamp = time.strftime("%Y%m%d-%H%M%S") safe_model = args.model.replace("/", "_")
rows: List[dict] = [] started = time.time() print(f"{scaffold} x {args.model}: {len(tasks)} task(s), {args.repeats} attempt(s) each\n")
try: for attempt in range(1, args.repeats + 1): for task in tasks: began = time.time() try: record = agent.run_task(task["task"], options) except Exception as exc: # an agent that crashes is a failed task, not a crash record = {"answer": "", "steps": 0, "tokens": 0, "stopped": f"entry point raised {type(exc).__name__}: {exc}", "trajectory": []} elapsed = record.get("seconds", round(time.time() - began, 1))
trajectory = record.get("trajectory", []) path = out_dir / f"{stamp}-{scaffold}-{safe_model}-{task['id']}-r{attempt}.jsonl" with path.open("w", encoding="utf-8") as handle: handle.write(json.dumps({"meta": { "scaffold": scaffold, "model": args.model, "task_id": task["id"], "attempt": attempt, "task": task["task"]}}) + "\n") for step in trajectory: handle.write(json.dumps(step) + "\n")
checks = check_answer(task, record) rows.append({ "id": task["id"], "category": task.get("category", "uncategorised"), "attempt": attempt, "answer": record.get("answer", ""), "steps": int(record.get("steps") or 0), "tokens": int(record.get("tokens") or 0), "prompt_tokens": record.get("prompt_tokens"), "completion_tokens": record.get("completion_tokens"), "seconds": float(elapsed), "stopped": record.get("stopped", ""), "checks": checks, "trajectory_file": str(path), }) mark = "ok " if checks["passed"] else "BAD" print(f" {mark} {task['id']:<16} r{attempt} " f"{record.get('steps', 0):>2} step(s) {record.get('tokens', 0):>7} tok " f"{elapsed:>6.1f}s") if args.verbose and not checks["passed"]: print(f" answer: {(record.get('answer') or '')[:200]}") if checks["missing"]: print(f" missing: {checks['missing']}") finally: if hasattr(agent, "close"): agent.close()
summary = summarise(rows, args.cost_per_million) run = { "lab": "part-26/evaluating-agents", "run_id": stamp, "scaffold": scaffold, "agent_entry_point": args.agent, "task_set": spec.get("name", args.tasks), "task_set_version": spec.get("version"), "model": args.model, "router_model": options["router_model"], "quant": args.quant, "engine": args.engine, "engine_version": args.engine_version, "base_url": args.base_url, "repeats": args.repeats, "host": platform.platform(), "date": time.strftime("%Y-%m-%d"), "wall_seconds": round(time.time() - started, 1), "notes": args.notes, "summary": summary, }
Path(args.out).write_text( json.dumps({"run": run, "per_task": per_task_view(rows), "results": rows}, indent=2), encoding="utf-8")
print(f"\nsuccess rate {summary['success_rate']:.2f} " f"({summary['passed']} of {summary['runs']})") print(f"mean steps {summary['mean_steps']}") print(f"mean tokens {summary['mean_tokens']}") print(f"mean seconds {summary['mean_seconds']}") if "cost_per_task" in summary: print(f"cost per task {summary['cost_per_task']} " f"(at {summary['cost_per_million_tokens']} per million tokens, your figure)") else: print("cost not computed: pass --cost-per-million from Part 23's cost-model.py") print(f"\nresults written to {args.out}; trajectories in {out_dir}/")
if args.labbook: with Path(args.labbook).open("a", encoding="utf-8") as handle: handle.write(json.dumps(run) + "\n") print(f"recorded in {args.labbook}")
sys.exit(0 if summary["passed"] == summary["runs"] else 2)
if __name__ == "__main__": main()Running it against the reference implementation from this part’s project looks like this:
RunnableAll tracks
python3 agent-eval.py \ --agent multi-agent-system.py \ --tasks agent-tasks.json \ --base-url http://127.0.0.1:4000/v1 \ --model local/answer \ --router-model local/router \ --workspace ./agent-workspace \ --out results-four-role.json \ --labbook labbook.mdOutput — what you should see
ok r-backup-window r1 4 step(s) 5120 tok 9.4s BAD r-gateway-port r1 6 step(s) 8730 tok 14.1s ...
success rate 0.80mean steps 4.9mean tokens 6180.0mean seconds 11.3cost not computed: pass --cost-per-million from Part 23's cost-model.pyThe last line is deliberate. The harness has no price of any kind built into it, because a
cost per million tokens depends on your electricity tariff, your hardware, and how many hours
a year the machine actually works — which is what Part 23’s cost-model.py computes from
your own measurements. Pass that figure with --cost-per-million and the report adds a cost
per task; leave it out and the report says it has no basis for one.
The four numbers, and what each one hides
Section titled “The four numbers, and what each one hides”| Number | What it answers | What it hides |
|---|---|---|
| Success rate | How often the task was done | Whether it was done for the right reason, or by luck on this run |
| Mean steps | How directly the agent worked | A bimodal distribution: half the tasks in two steps, half hitting the limit |
| Mean tokens | What it costs to run | Which role spent them; a synthesiser reading whole transcripts hides inside this |
| Mean seconds | Whether it is usable | Whether the machine was also doing something else |
Report all four or none. A system that raises the success rate from three-fifths to four-fifths while tripling tokens may be the right trade or the wrong one, and the number you left out is the one that decides.
Steps deserve a note. The harness counts one step per model reply plus one per tool result, which makes the number comparable across scaffolds that structure their turns differently. A four-role system spends steps on roles; a single loop spends them on turns. Compare the totals, and read the trajectories when they differ by more than a little.
Variance, and how many repeats
Section titled “Variance, and how many repeats”An agent is not deterministic even at temperature zero, because tool results vary, engines batch differently, and mixture-of-experts routing is sensitive to batch composition. So a single suite run is one sample.
τ-bench’s authors introduce pass^k for exactly this, to “evaluate the reliability of agent
behavior over multiple trials”, and report that even strong function-calling agents were
“quite inconsistent”. That is the effect to design around.
The practical rule for a fifteen-task suite:
- One repeat tells you whether the system runs at all. Use it while building.
- Three repeats show you which tasks are unstable. This is the minimum for comparing two
configurations, and it is what the harness’s
--repeatsoption is for. - Five or more are for a decision you will not revisit, such as which model to serve for the next six months.
The harness’s per-task view reports attempts, successes, and the minimum and maximum steps and tokens for each task, which is where instability shows itself. A task that passes three times in three is settled. A task that passes twice in three is the one to read the trajectories for, and usually the answer is visible immediately: two different routes, or a search that returned different passages.
Regression testing when something changes
Section titled “Regression testing when something changes”The suite earns its keep the second time you run it. Four things change under an agent, and each one is a reason to re-run:
| What changed | What to expect | What to check first |
|---|---|---|
| The model behind an alias | The largest effect of the four | Tool-call reliability from Part 24’s test, before blaming the agent |
| The quantisation | Smaller than people fear, and not always downward | Whether the failures moved to different tasks |
| A prompt | Often larger than people expect | The tasks that changed, not the total |
| A framework version | Usually nothing, occasionally everything | Whether the token counts still look sane |
Record every run in the lab notebook, which the harness does with --labbook. The line
carries the scaffold, both model aliases, the quantisation you typed in, the engine and
version, the task-set version, the repeat count, the date and the summary. That is what makes
a run from today comparable with one from three months ago, which is the only thing that
turns a suite into a regression test.
Version the task file. When you add a task, bump version in it, and never edit a task’s
expected answer without doing so: a suite that changes silently makes every historical number
in your notebook incomparable.
Tracing, when a report is not enough
Section titled “Tracing, when a report is not enough”A results file plus trajectories is enough for a system you run by hand. A system other people use wants a trace store, and there are two things worth knowing.
Langfuse describes itself as “an open-source AI engineering platform that helps teams collaboratively debug, analyze, and iterate on their AI agent applications”, and it is “open, self-hostable, and extensible”. It records LLM and non-LLM calls including retrieval and embedding, groups multi-turn work into sessions, visualises agent workflows as graphs, and attaches scores to traces. Its documentation says it is “based on OpenTelemetry to increase compatibility and reduce vendor lock-in”, which matters more than any feature: it means your instrumentation is not tied to it.
OpenTelemetry’s GenAI semantic conventions are the vocabulary underneath. The conventions moved during 2026: the main OpenTelemetry semantic-conventions page now says they “have moved to the OpenTelemetry GenAI semantic conventions repository”, which describes itself as holding “Semantic Conventions for Generative AI (GenAI), including spans, metrics, and events for GenAI clients, MCP (Model Context Protocol), and provider-specific conventions”. The MCP part is the reason to care here: agent traces span a model, several tools and possibly a protocol server, and a shared vocabulary is what lets one trace cover all of it.
For a home service, the honest order is: JSON-lines trajectories first, because they cost nothing and answer most questions; then the gateway’s own usage log from Part 23, which attributes tokens per key without instrumenting anything; then a self-hosted trace store when several people use the system and “it was slow yesterday” needs an answer.
Score the environment state, not the completion message
Section titled “Score the environment state, not the completion message”An agent can announce success while leaving the file unchanged or while weakening a test. Evaluate the artefact independently from a recorded starting state. For coding, run tests outside the agent’s editable test set where possible and inspect the diff. For retrieval, verify the answer against the supplied corpus and citation identities.
Keep infrastructure failures distinct from model failures, but report both. A timeout, unavailable endpoint or parser error affects usable success rate even if it says little about reasoning quality. Predefine retry limits and whether the reported score is first-attempt or after bounded repair.
Reset the environment between trials while preserving outputs and trajectories elsewhere. Repeated runs need the same task, permissions and tool implementations to estimate model variation. Group related tasks when reporting uncertainty and avoid tuning against the final held-out suite. The useful metric is verified success under a stated cost and authority budget, with enough task-level evidence to explain failures and reproduce improvements.
An agent suite needs outcomes a program can check, and the strongest form is τ-bench’s: the
state of the world at the end, not the wording of the reply. Fifteen tasks over documents
whose answers cannot be known from training data, with string, regular-expression, tool and
step checks and one task that tests refusal, is enough to detect a large difference and not a
small one, and saying so is part of the report. Trajectories logged as JSON lines with a
metadata line, one line per model reply, tool call and routing decision, and an explicit
stopping reason, are the only artefact that explains a failure. The harness knows nothing
about frameworks: it loads a file and calls run_task, which is what lets the same suite
measure a hand-written loop and two frameworks. Report success rate, steps, tokens and
seconds together, and add cost only from your own Part 23 figure. Repeat three times before
comparing anything, because a two-task difference on fifteen tasks is noise. And re-run the
suite whenever the model, the quantisation, a prompt or a framework version changes, with
every run recorded in the notebook so that today’s number means something next quarter.
Check your understanding
Sources for this lesson
5 verified · checked 2026-09-09
- 01τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains§ Abstract; database-state evaluation; pass^karxiv.org/abs/2406.120452026-09-09
- 02AgentBench: Evaluating LLMs as Agents§ Abstract; eight environments; failure analysisarxiv.org/abs/2308.036882026-09-09
- 03Langfuse — Documentation§ What Langfuse is; self-hosting; traces, sessions and scores; OpenTelemetrylangfuse.com/docs2026-09-09
- 04OpenTelemetry — Semantic conventions for generative AI§ Notice that the conventions have movedopentelemetry.io/docs/specs/semconv/gen-ai2026-09-09
- 05OpenTelemetry — GenAI semantic conventions repository§ Repository description; spans, metrics and events for GenAI clients and MCPgithub.com/open-telemetry/semantic-conventions-genai2026-09-09
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.