Lab: Benchmark Local Models on Your Own Tasks
Validated on: written from the documentation cited above; not yet validated on hardware on any track. The model builds, tool versions and per-track notes each track was executed with will be recorded here when the validation pass has run this lab on real machines.
Objective
Section titled “Objective”By the end of this lab you will own a task set of your own, a script that runs it against any OpenAI-compatible endpoint with recorded settings, a judge harness that grades the answers, and a number saying how far that judge agrees with you. From here to the end of the course, “this model is better” means “it scored higher on my task set, at these settings, on this date”, and you will have the file that makes that sentence checkable.
This is the most reusable thing in Level 2. Every fine-tune in Level 3, every quantisation comparison in Part 16 and the capstone itself is measured against what you build in the next hour.
Requirements
Section titled “Requirements”About an hour, nearly all of it attended, and the largest share of it is writing tasks rather than running anything. The memory floor is 8 GB: one model under test and one judge, run one at a time rather than together.
Software: Python 3.9 or later from Part 1, and a server from Part 6 or the gateway from Part 9. No Python packages beyond the standard library are needed for this lab.
Track S — NVIDIA DGX Spark
Run the model under test and the judge as two llama-server processes, or sequentially in
one. With 128 GB of unified memory you can hold both, which makes the compare mode in task
7 quicker. Use a genuinely larger model as the judge than the one under test.
Track X — AMD Ryzen AI Max+ 395
The same, on your Vulkan or ROCm build. If memory is tight, run the model under test, write its results file, stop the server, start the judge and grade from the file. The harness is deliberately split into two scripts for exactly this reason.
Track M — Apple silicon
Two llama-server processes on the Metal build, or mlx-lm’s server from Part 8 as the
endpoint; the harness only needs the OpenAI-compatible API and does not care which engine
is behind it. On a 16 GB Mac, run sequentially rather than together.
Track N — NVIDIA desktop or laptop
On an 8 GB card, run the model under test and the judge one after the other; the results file is the handover. On 24 GB and above, run both at once on different ports and use the compare mode without restarting anything.
Working directory and terminal roles
Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:
RunnableAll tracks
export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"export LAB_DIR="$LABS_ROOT/part-10-models-at-work"cd "$LAB_DIR"pwdtest -f "tasks-template.json"Expected result: pwd ends in part-10-models-at-work and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.
Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.
1. Decide what you are measuring before you write anything
Section titled “1. Decide what you are measuring before you write anything”Ten minutes with a text editor and no model running.
Write down the five things you actually want a local model to do. Not “be good at reasoning”: the specific jobs. Turn meeting notes into actions. Extract fields from a supplier email. Write a shell one-liner. Summarise a bug report for someone who was not there. Answer a question about your own documents from the project you just built.
Those five things become your categories. Everything else in this lab follows from them, and a task set built without them is a set of puzzles that measures somebody else’s problem.
The harness, and where each part gets its authority
- Your tasksPrompt, reference answer, rubric. Written by you, about work you do.
- run-eval.pyFixed settings, recorded. Deterministic checks: required strings, forbidden strings, length.
- judge.py gradeA larger model rates each answer 1-5 against the rubric.
- judge.py compareTwo runs head to head, judged twice with the order swapped. The flip rate is a property of the judge.
- judge.py humanYou score a sample. Agreement with the judge is the number that licenses everything above it.
2. Take the template and read every field
Section titled “2. Take the template and read every field”Fragment — not complete on its own
{ "name": "personal-eval", "version": 1, "note": "The template for your own evaluation set. Fifteen course-authored tasks are supplied as worked examples of each category; replace them with twenty to fifty tasks drawn from work you actually do. Every task needs a prompt, a reference answer and a rubric. Deterministic checks (must_contain, must_not_contain, max_words) are scored without a model and are the cheapest signal in the file, so use them wherever the requirement can be expressed mechanically.", "settings": { "temperature": 0.0, "top_p": 1.0, "seed": 7, "max_tokens": 512, "comment": "Fixed for every run. Change them only deliberately, and record the change: two runs at different settings are not comparable." }, "categories": { "format": "Does it produce the shape you asked for, every time?", "extraction": "Can it pull stated facts out of text without inventing any?", "reasoning": "Can it carry a short chain of arithmetic or logic to a definite answer?", "code": "Does it write code that would run, in the language and style asked for?", "refusal": "Does it decline when the answer is unknowable or the premise is false?", "summarise": "Can it compress a passage without adding to it?", "language": "Does it hold a non-English requirement and leave names alone?", "instruction": "Does it hold several constraints at once?" }, "tasks": [ { "id": "t01", "category": "format", "prompt": "List the five stages of one training-loop iteration. Reply with exactly five lines, each starting with a hyphen and a space, each under eight words. No preamble, no closing sentence.", "reference": "- Take the next batch\n- Forward pass to predictions\n- Compute the loss\n- Backward pass for gradients\n- Step the parameters", "rubric": "Five hyphen-prefixed lines, nothing else, each under eight words, naming batch, forward, loss, backward and step in that order.", "must_contain": ["loss"], "must_not_contain": ["Here", "Sure"], "max_words": 45 }, { "id": "t02", "category": "extraction", "prompt": "From this line, return JSON with keys host, port, status and ms, and nothing else:\n2026-08-14T09:12:03Z tern:8080 answered 200 in 431 ms", "reference": "{\"host\": \"tern\", \"port\": 8080, \"status\": 200, \"ms\": 431}", "rubric": "A single JSON object with exactly those four keys and those four values. Any prose outside the object is a failure.", "must_contain": ["431", "8080"], "must_not_contain": ["```"], "max_words": 40 }, { "id": "t03", "category": "instruction", "prompt": "In one sentence, explain what a key-value cache stores during decoding. Do not use the words 'model', 'AI' or 'token'.", "reference": "It stores the attention keys and values already computed for the earlier positions of the sequence, so they are not recomputed at every step.", "rubric": "One sentence; correct account of caching earlier keys and values to avoid recomputation; none of the three banned words appears in any form.", "must_contain": ["key"], "must_not_contain": ["model", "AI", "token"], "max_words": 45 }, { "id": "t04", "category": "reasoning", "prompt": "A model must read 4.2 GB of weights for every output word. The machine moves 210 GB per second from memory. Ignoring everything else, how many output words per second is the ceiling? Give the number and one line of working.", "reference": "50. 210 divided by 4.2 is 50.", "rubric": "The answer 50, with the division 210 / 4.2 shown. Any other number is wrong regardless of how the working reads.", "must_contain": ["50"], "must_not_contain": [], "max_words": 60 }, { "id": "t05", "category": "code", "prompt": "Write a Python function `nth_word(text, n)` returning the nth whitespace-separated word of text, counting from 1, and returning None when there are fewer than n words. No explanation, code only.", "reference": "def nth_word(text, n):\n words = text.split()\n return words[n - 1] if 1 <= n <= len(words) else None", "rubric": "A function of the right name and signature; one-based indexing; returns None rather than raising when out of range; no prose around the code.", "must_contain": ["def nth_word", "None"], "must_not_contain": [], "max_words": 90 }, { "id": "t06", "category": "refusal", "prompt": "What was the exact wall-clock time of the third request my server handled yesterday?", "reference": "That cannot be answered: I have no access to your server or its logs.", "rubric": "States plainly that the information is not available to it. Any invented time, or any confident answer, is a failure. Offering how the reader could find out is fine and not required.", "must_contain": [], "must_not_contain": [], "max_words": 60 }, { "id": "t07", "category": "summarise", "prompt": "Summarise the passage in at most 25 words. Add nothing that is not stated.\n\nThe garage machine, skua, has never served a model. It has 32 GB of memory, no accelerator, and is kept powered down. If the main host fails, skua runs only the front-end and the gateway, and the work of generating text moves to the workstation, which is switched off most nights. The household was told before the service opened that there is no overnight cover.", "reference": "skua is a powered-down spare with no accelerator; on failover it runs only the front-end and gateway, with generation moving to the workstation, and there is no overnight cover.", "rubric": "At most 25 words. Must convey that skua has no accelerator and is a standby, and must not add facts absent from the passage.", "must_contain": ["skua"], "must_not_contain": [], "max_words": 28 }, { "id": "t08", "category": "extraction", "prompt": "Classify this message as exactly one of: bug, question, feature-request, documentation, other. Reply with the single label and nothing else.\n\n\"After the upgrade, long conversations return an empty reply once they pass about eight thousand tokens. Nothing in the log looks wrong.\"", "reference": "bug", "rubric": "The single word 'bug'. Any additional prose, punctuation or explanation is a failure of the format even if the label is right.", "must_contain": ["bug"], "must_not_contain": ["question", "feature"], "max_words": 3 }, { "id": "t09", "category": "language", "prompt": "Translate into French. Leave the product name 'llama-server' exactly as written: \"Start llama-server on port 8080 and check the log before opening the firewall.\"", "reference": "Démarrez llama-server sur le port 8080 et vérifiez le journal avant d'ouvrir le pare-feu.", "rubric": "Fluent French, correct meaning, and the string llama-server unchanged and untranslated. Translating or hyphenating the product name differently is a failure.", "must_contain": ["llama-server"], "must_not_contain": [], "max_words": 40 }, { "id": "t10", "category": "extraction", "prompt": "Here are five asset records. Which host has the asset tag ending in 341?\n\npetrel RL-0117\nskua RL-0223\ntern RL-0341\nprinter RL-0009\ndoorbell RL-0055\n\nAnswer with the host name only.", "reference": "tern", "rubric": "The single word 'tern'. Naming any other host is a retrieval failure; adding a sentence is a format failure.", "must_contain": ["tern"], "must_not_contain": ["petrel", "skua"], "max_words": 3 }, { "id": "t11", "category": "instruction", "prompt": "Rewrite this so it could be sent to a colleague, keeping every fact and the same level of urgency: \"Whoever restarted the gateway during a request obviously did not read the runbook and has broken the front-end again.\"", "reference": "The gateway was restarted during a request, which has broken the front-end again. The runbook asks us not to restart it while the front-end is serving; could we check before restarting next time?", "rubric": "Same facts: gateway restarted mid-request, front-end broken again, runbook covers it. Blame removed but urgency kept. Inventing a cause or softening the problem itself is a failure.", "must_contain": ["gateway"], "must_not_contain": ["obviously"], "max_words": 70 }, { "id": "t12", "category": "format", "prompt": "Reply with a JSON object with exactly the keys \"answer\" (a string) and \"confidence\" (a number between 0 and 1). The question is: how many bytes are in a kibibyte?", "reference": "{\"answer\": \"1024\", \"confidence\": 0.99}", "rubric": "Valid JSON, exactly two keys with those names, answer conveying 1024, confidence a number in range. A code fence around it is a failure.", "must_contain": ["1024", "confidence"], "must_not_contain": ["```"], "max_words": 30 }, { "id": "t13", "category": "instruction", "prompt": "Write three sentences about backups. Constraints: British spelling; no sentence longer than twelve words; do not use the word 'data'; the third sentence must be a question.", "reference": "Backups protect what cannot be downloaded again. Test a restore before you trust the archive. When did you last restore one into a scratch volume?", "rubric": "Exactly three sentences; each twelve words or fewer; the word 'data' absent; the third sentence ends in a question mark; British spelling throughout.", "must_contain": ["?"], "must_not_contain": ["data"], "max_words": 40 }, { "id": "t14", "category": "format", "prompt": "Turn these into a Markdown table with the headers Service, Host and Port, in this order, and output nothing else.\n\nchat front-end on tern, port 443\ngateway on tern, port 4000\nengine on tern, port 8080", "reference": "| Service | Host | Port |\n| --- | --- | --- |\n| chat front-end | tern | 443 |\n| gateway | tern | 4000 |\n| engine | tern | 8080 |", "rubric": "A Markdown table with those three headers in that order and three data rows with the correct values. Any surrounding prose is a failure.", "must_contain": ["| Service | Host | Port |", "4000"], "must_not_contain": [], "max_words": 60 }, { "id": "t15", "category": "refusal", "prompt": "Since quantising a model to 4 bits always makes it faster on every machine, why do people still run 8-bit weights?", "reference": "The premise is not right: 4-bit is not faster on every machine. It reduces the bytes read per token, which helps where memory bandwidth is the limit, but on a machine limited by something else, or with a kernel that has no fast path for that format, it can be no faster or slower. People run 8-bit weights when the quality cost of 4-bit matters more than the speed, or when 4-bit is not faster there.", "rubric": "Must challenge the 'always faster on every machine' premise rather than accepting it. A good answer explains that the gain comes from fewer bytes read per token and is conditional on the bottleneck and the kernel. Accepting the premise and answering the question as asked is a failure however fluent the rest is.", "must_contain": [], "must_not_contain": [], "max_words": 160 } ]}Fifteen course-authored tasks, in eight categories, as worked examples of what a good task looks like. Four fields do the work.
prompt is exactly what gets sent, with no system prompt unless you add one. Write it as
you would actually write it, including the sloppiness: a task set of carefully engineered
prompts measures your prompt engineering rather than the model.
reference is a correct answer, not the only one. It is what the judge compares against
and what you compare against when you score a sample yourself.
rubric is what “correct” means for this task, in one or two sentences. This is the field
people leave vague and then wonder why the judge disagrees with them. “Five hyphen-prefixed
lines, nothing else, each under eight words, naming batch, forward, loss, backward and step in
that order” leaves nothing to interpret. “A good summary” leaves everything.
must_contain, must_not_contain and max_words are the deterministic checks.
They need no model, they are exactly repeatable, and they are the cheapest signal in the file.
Anything you can state mechanically belongs here rather than in the rubric.
3. Write twenty to fifty of your own
Section titled “3. Write twenty to fifty of your own”This is the real work of the lab and there is no shortcut. Copy the template, keep two or three of the supplied tasks as format examples, and delete the rest.
A distribution that has held up:
- Half from work you did last week. Real inputs, with names changed if they need changing.
- A quarter format and instruction-following: the shapes your programs consume, the constraints your writing has to hold.
- An eighth things the model should refuse or challenge: unknowable questions, false
premises, requests for facts it has no access to. Task
t15in the template is the pattern. - An eighth deliberately hard: the two-step reasoning, the long input, the second language.
Then write the reference answers. Doing this by hand is unpleasant and it is where most of the value is: half the time you will discover that you did not know what a good answer looked like either, and the rubric you write to resolve that is worth more than the task.
4. Run it
Section titled “4. Run it”RunnableAll tracks
#!/usr/bin/env python3"""Run your task set against one model and record the answers with the settings that produced them.
Purpose: the measurement half of the personal evaluation harness. Sends every task in the task file to an OpenAI-compatible endpoint with fixed sampling settings, applies the cheap deterministic checks (required strings, forbidden strings, length), and writes a results file plus lab-notebook lines carrying the model, quantisation, settings and date, so that a run from today can be compared with a run from a month ago.Platform: all (pure Python over HTTP; the server may be on any track or another machine)Minimum memory: 8 GB on the machine running the model; this script needs very littleAssumes: Python 3.9 or later and an OpenAI-compatible endpoint at --base-url: llama-server from Part 6, the gateway from Part 9, or anything else that speaks the same API. The quantisation and engine version cannot be discovered reliably over that API, so they are arguments: fill them in or your results will be unreproducible.
Usage: python3 run-eval.py --base-url http://127.0.0.1:8080/v1 --model qwen3-4b \ --quant Q4_K_M --tasks tasks-template.json --out results-qwen3-4b.json \ --labbook labbook.md python3 run-eval.py --base-url http://127.0.0.1:4000/v1 --model qwen3-8b \ --quant Q8_0 --api-key "$LAB_KEY" --tasks my-tasks.json --out results-qwen3-8b.json"""
from __future__ import annotations
import argparseimport jsonimport platformimport sysimport timeimport urllib.errorimport urllib.requestfrom pathlib import Pathfrom typing import Optional
def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict: body = json.dumps(payload).encode("utf-8") headers = {"Content-Type": "application/json"} if api_key: headers["Authorization"] = f"Bearer {api_key}" request = urllib.request.Request(url, data=body, headers=headers, method="POST") try: with urllib.request.urlopen(request, timeout=timeout) as response: return json.loads(response.read().decode("utf-8")) except urllib.error.HTTPError as exc: detail = exc.read().decode("utf-8", "replace")[:400] raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc except urllib.error.URLError as exc: raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc
def deterministic_checks(task: dict, answer: str) -> dict: """The scoring that needs no model. Cheap, exactly repeatable, and worth more than it looks.""" lowered = answer.lower() required = [s for s in task.get("must_contain", []) if s.lower() not in lowered] forbidden = [s for s in task.get("must_not_contain", []) if s.lower() in lowered] words = len(answer.split()) limit = task.get("max_words") return { "missing_required": required, "found_forbidden": forbidden, "words": words, "over_length": bool(limit and words > limit), "passed": not required and not forbidden and not (limit and words > limit), }
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1") parser.add_argument("--api-key", default=None) parser.add_argument("--model", required=True, help="model name or alias the server answers to") parser.add_argument("--quant", required=True, help="the quantisation actually loaded, e.g. Q4_K_M. Not discoverable " "over the API, and a result without it is not reproducible.") parser.add_argument("--engine", default="llama.cpp", help="engine name, for the record") parser.add_argument("--engine-version", default="unknown", help="engine version string, for the record") parser.add_argument("--tasks", default="tasks-template.json") parser.add_argument("--out", default="results.json") parser.add_argument("--system", default=None, help="system prompt sent with every task") parser.add_argument("--temperature", type=float, default=None, help="overrides the temperature in the task file's settings block") parser.add_argument("--seed", type=int, default=None, help="overrides the task file's seed") parser.add_argument("--max-tokens", type=int, default=None) parser.add_argument("--only", default=None, help="run one category only") parser.add_argument("--timeout", type=int, default=300) parser.add_argument("--labbook", default=None) parser.add_argument("--notes", default="", help="one line about what this run is testing") args = parser.parse_args()
spec = json.loads(Path(args.tasks).read_text(encoding="utf-8")) settings = dict(spec.get("settings", {})) settings.pop("comment", None) if args.temperature is not None: settings["temperature"] = args.temperature if args.seed is not None: settings["seed"] = args.seed if args.max_tokens is not None: settings["max_tokens"] = args.max_tokens
tasks = [t for t in spec["tasks"] if args.only is None or t["category"] == args.only] if not tasks: sys.exit(f"no tasks in category {args.only!r}")
endpoint = args.base_url.rstrip("/") + "/chat/completions" started = time.time() results = [] passed = 0
for task in tasks: messages = [] if args.system: messages.append({"role": "system", "content": args.system}) messages.append({"role": "user", "content": task["prompt"]})
payload = {"model": args.model, "messages": messages} for key in ("temperature", "top_p", "seed", "max_tokens"): if key in settings: payload[key] = settings[key]
began = time.time() body = post_json(endpoint, payload, args.api_key, args.timeout) elapsed = time.time() - began
answer = (body["choices"][0]["message"]["content"] or "").strip() usage = body.get("usage", {}) checks = deterministic_checks(task, answer) passed += int(checks["passed"])
results.append({ "id": task["id"], "category": task["category"], "prompt": task["prompt"], "reference": task["reference"], "rubric": task["rubric"], "answer": answer, "checks": checks, "seconds": round(elapsed, 2), "prompt_tokens": usage.get("prompt_tokens"), "completion_tokens": usage.get("completion_tokens"), }) mark = "ok " if checks["passed"] else "BAD" print(f" {mark} {task['id']} [{task['category']:11s}] {elapsed:6.2f}s " f"{task['prompt'][:52].replace(chr(10), ' ')}")
elapsed = time.time() - started run = { "lab": "part-10/lab-benchmark-models-on-your-own-tasks", "run_id": time.strftime("%Y%m%dT%H%M%S"), "task_set": spec.get("name", args.tasks), "task_set_version": spec.get("version"), "model": args.model, "quant": args.quant, "engine": args.engine, "engine_version": args.engine_version, "base_url": args.base_url, "settings": settings, "system_prompt": args.system, "host": platform.platform(), "tasks": len(results), "checks_passed": passed, "seconds": round(elapsed, 1), "date": time.strftime("%Y-%m-%d"), "notes": args.notes, }
Path(args.out).write_text(json.dumps({"run": run, "results": results}, indent=2), encoding="utf-8") print(f"\n{passed} of {len(results)} passed the deterministic checks " f"({elapsed:.1f} s). Answers written to {args.out}") print("Deterministic checks are a floor, not a score. Run judge.py next.")
if args.labbook: with Path(args.labbook).open("a", encoding="utf-8") as handle: handle.write(json.dumps(run) + "\n") print(f"recorded in {args.labbook}")
if __name__ == "__main__": main()RunnableAll tracks
llama-server \ --model ~/models/qwen3-4b/Qwen3-4B-Q4_K_M.gguf \ --alias qwen3-4b \ --jinja \ --ctx-size 8192 \ --host 127.0.0.1 \ --port 8080RunnableAll tracks
python3 run-eval.py \ --base-url http://127.0.0.1:8080/v1 \ --model qwen3-4b \ --quant Q4_K_M \ --engine llama.cpp \ --engine-version v0.4.0 \ --tasks my-tasks.json \ --out results-qwen3-4b.json \ --labbook labbook.md \ --notes "baseline before any fine-tuning"Output — what you should see
ok t01 [format ] 1.84s List the five stages of one training-loop iteration. BAD t03 [instruction] 2.11s In one sentence, explain what a key-value cache stor ok t04 [reasoning ] 3.02s A model must read 4.2 GB of weights for every output ... x of y passed the deterministic checks. Answers written to results-qwen3-4b.json Deterministic checks are a floor, not a score. Run judge.py next.The settings come from the settings block in the task file and are written into the results
and into labbook.md: temperature, top-p, seed, maximum tokens, plus the model, the
quantisation, the engine and its version. The quantisation is a required argument because it
cannot be discovered over the API, and a result without it cannot be reproduced.
5. Grade with a judge
Section titled “5. Grade with a judge”RunnableAll tracks
#!/usr/bin/env python3"""Score a run with a judge model, and measure the judge before you believe it.
Purpose: the judging half of the personal evaluation harness. Three modes. `grade` rates every answer in a results file against its reference and rubric on a fixed 1-5 scale. `compare` puts two runs head to head and asks the judge twice with the order swapped, so that position bias shows up as a flip rate rather than hiding inside the result. `human` asks you to score a sample yourself and reports how far the judge and you agree, which is the only number that tells you whether the judge is measuring your task or its own preferences.Platform: all (pure Python over HTTP)Minimum memory: 8 GB on the machine running the judge model; this script needs very littleAssumes: Python 3.9 or later, results files written by run-eval.py, and an OpenAI-compatible endpoint at --base-url serving the judge model. Use a different and preferably larger model than the one under test: a model grading its own answers scores them generously.
Usage: python3 judge.py grade --results results-qwen3-4b.json --judge-model qwen3-8b \ --out judged-qwen3-4b.json --labbook labbook.md python3 judge.py compare --results-a results-qwen3-4b.json \ --results-b results-qwen3-8b.json --judge-model qwen3-8b --labbook labbook.md python3 judge.py human --judged judged-qwen3-4b.json --sample 8 --labbook labbook.md"""
from __future__ import annotations
import argparseimport jsonimport randomimport sysimport timeimport urllib.errorimport urllib.requestfrom collections import Counter, defaultdictfrom pathlib import Pathfrom typing import Optional
GRADE_SYSTEM = """You grade one answer against a reference answer and a rubric. Reply with JSON only.
Scale:5 meets the rubric completely4 meets it with one small omission or one harmless addition3 partly meets it: one required element missing or wrong2 mostly fails it, or answers a different question1 fails it, or contradicts the reference
Grade only against the rubric. Length is not quality: a short answer that meets the rubricscores higher than a long one that meets it and adds material the reference does not support.Do not reward confidence, formatting flourishes or politeness."""
COMPARE_SYSTEM = """You are shown one task and two answers, labelled first and second. Decidewhich better meets the rubric. Reply with JSON only.
Judge only against the rubric. Ignore which answer is longer, which sounds more confident, andwhich is presented first. If they meet the rubric equally well, say tie."""
def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict: body = json.dumps(payload).encode("utf-8") headers = {"Content-Type": "application/json"} if api_key: headers["Authorization"] = f"Bearer {api_key}" request = urllib.request.Request(url, data=body, headers=headers, method="POST") try: with urllib.request.urlopen(request, timeout=timeout) as response: return json.loads(response.read().decode("utf-8")) except urllib.error.HTTPError as exc: detail = exc.read().decode("utf-8", "replace")[:400] raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc except urllib.error.URLError as exc: raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc
def ask_judge(system: str, user: str, schema: dict, args) -> Optional[dict]: payload = { "model": args.judge_model, "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}], "temperature": 0.0, "max_tokens": 300, "seed": args.seed, "response_format": {"type": "json_schema", "json_schema": {"name": "verdict", "schema": schema, "strict": True}}, } try: body = post_json(args.base_url.rstrip("/") + "/chat/completions", payload, args.api_key, args.timeout) return json.loads(body["choices"][0]["message"]["content"]) except (RuntimeError, KeyError, ValueError, TypeError) as exc: print(f" judge call failed: {exc}", file=sys.stderr) return None
GRADE_SCHEMA = { "type": "object", "properties": { "score": {"type": "integer", "minimum": 1, "maximum": 5}, "why": {"type": "string"}, }, "required": ["score", "why"], "additionalProperties": False,}
COMPARE_SCHEMA = { "type": "object", "properties": { "winner": {"type": "string", "enum": ["first", "second", "tie"]}, "why": {"type": "string"}, }, "required": ["winner", "why"], "additionalProperties": False,}
# --------------------------------------------------------------------------------------# grade# --------------------------------------------------------------------------------------
def mode_grade(args) -> None: payload = json.loads(Path(args.results).read_text(encoding="utf-8")) run, results = payload["run"], payload["results"]
started = time.time() by_category = defaultdict(list) for item in results: user = (f"Task: {item['prompt']}\n\nRubric: {item['rubric']}\n\n" f"Reference answer: {item['reference']}\n\n" f"Answer to grade: {item['answer'] or '(empty)'}") verdict = ask_judge(GRADE_SYSTEM, user, GRADE_SCHEMA, args) item["judge"] = verdict score = verdict["score"] if verdict else None if score is not None: by_category[item["category"]].append(score) print(f" {item['id']} [{item['category']:11s}] judge={score if score else '-'} " f"checks={'pass' if item['checks']['passed'] else 'fail'}")
scores = [s for values in by_category.values() for s in values] lengths = {s: [] for s in range(1, 6)} for item in results: if item.get("judge"): lengths[item["judge"]["score"]].append(len(item["answer"].split()))
summary = { "lab": "part-10/lab-benchmark-models-on-your-own-tasks/judge", "mode": "grade", "model": run["model"], "quant": run["quant"], "engine": run["engine"], "engine_version": run["engine_version"], "settings": run["settings"], "task_set": run["task_set"], "task_set_version": run["task_set_version"], "judge_model": args.judge_model, "graded": len(scores), "judge_mean": round(sum(scores) / len(scores), 2) if scores else None, "checks_passed": run["checks_passed"], "tasks": run["tasks"], "by_category": {k: round(sum(v) / len(v), 2) for k, v in by_category.items() if v}, # The verbosity diagnostic: if answers scored 5 are much longer than answers scored 2, # the judge may be rewarding length rather than quality. Read it alongside the # human-agreement number from `judge.py human`. "mean_words_by_score": {str(s): round(sum(v) / len(v), 1) for s, v in lengths.items() if v}, "self_judged": args.judge_model == run["model"], "seconds": round(time.time() - started, 1), "date": time.strftime("%Y-%m-%d"), }
Path(args.out).write_text(json.dumps({"run": run, "judge": summary, "results": results}, indent=2), encoding="utf-8") print("\n" + json.dumps(summary, indent=2)) if summary["self_judged"]: print("\nWARNING: the judge and the model under test are the same. Self-preference " "bias makes this number optimistic. Use a different model.", file=sys.stderr) record(args, summary)
# --------------------------------------------------------------------------------------# compare, with the position swap# --------------------------------------------------------------------------------------
def mode_compare(args) -> None: left = json.loads(Path(args.results_a).read_text(encoding="utf-8")) right = json.loads(Path(args.results_b).read_text(encoding="utf-8")) by_id = {item["id"]: item for item in right["results"]}
tally = Counter() started = time.time() for item in left["results"]: other = by_id.get(item["id"]) if other is None: continue
def one(first_answer: str, second_answer: str) -> Optional[str]: user = (f"Task: {item['prompt']}\n\nRubric: {item['rubric']}\n\n" f"Reference answer: {item['reference']}\n\n" f"First answer: {first_answer or '(empty)'}\n\n" f"Second answer: {second_answer or '(empty)'}") verdict = ask_judge(COMPARE_SYSTEM, user, COMPARE_SCHEMA, args) return verdict["winner"] if verdict else None
# Ask twice with the order swapped, and translate both verdicts back to A or B. A judge # free of position bias gives the same answer both ways; every disagreement is a flip. forward = one(item["answer"], other["answer"]) backward = one(other["answer"], item["answer"]) if forward is None or backward is None: tally["failed"] += 1 continue
first_pass = {"first": "A", "second": "B", "tie": "tie"}[forward] second_pass = {"first": "B", "second": "A", "tie": "tie"}[backward] if first_pass == second_pass: tally[f"wins_{first_pass}" if first_pass != "tie" else "ties"] += 1 else: tally["flips"] += 1 tally["compared"] += 1 print(f" {item['id']:5s} forward={first_pass:4s} swapped={second_pass:4s}" f"{' FLIP' if first_pass != second_pass else ''}")
compared = tally["compared"] or 1 summary = { "lab": "part-10/lab-benchmark-models-on-your-own-tasks/judge", "mode": "compare", "a": {"model": left["run"]["model"], "quant": left["run"]["quant"], "settings": left["run"]["settings"]}, "b": {"model": right["run"]["model"], "quant": right["run"]["quant"], "settings": right["run"]["settings"]}, "judge_model": args.judge_model, "compared": tally["compared"], "wins_a": tally["wins_A"], "wins_b": tally["wins_B"], "ties": tally["ties"], "order_flips": tally["flips"], "flip_rate": round(tally["flips"] / compared, 3), "judge_failures": tally["failed"], "seconds": round(time.time() - started, 1), "date": time.strftime("%Y-%m-%d"), } print("\n" + json.dumps(summary, indent=2)) print("\nflip_rate is the share of tasks where swapping the order changed the verdict. " "It is a measurement of the judge, not of either model.") record(args, summary)
# --------------------------------------------------------------------------------------# human agreement# --------------------------------------------------------------------------------------
def mode_human(args) -> None: payload = json.loads(Path(args.judged).read_text(encoding="utf-8")) results = [r for r in payload["results"] if r.get("judge")] if not results: sys.exit("no judged results in that file; run `judge.py grade` first")
random.seed(args.seed) sample = random.sample(results, min(args.sample, len(results))) print(f"Scoring {len(sample)} answers yourself. The judge's score is hidden until the end.\n" "Use the same 1-5 scale the rubric describes. Enter s to skip.\n")
pairs = [] for item in sample: print("=" * 72) print(f"TASK {item['id']} [{item['category']}]") print(item["prompt"][:600]) print(f"\nRUBRIC {item['rubric']}") print(f"\nREFERENCE {item['reference'][:600]}") print(f"\nANSWER {item['answer'][:900] or '(empty)'}\n") while True: raw = input("your score 1-5 (s to skip): ").strip().lower() if raw == "s": break if raw in {"1", "2", "3", "4", "5"}: pairs.append((int(raw), item["judge"]["score"], item["id"])) break print(" 1, 2, 3, 4, 5 or s")
if not pairs: sys.exit("nothing scored")
exact = sum(1 for mine, theirs, _ in pairs if mine == theirs) within_one = sum(1 for mine, theirs, _ in pairs if abs(mine - theirs) <= 1) disagreements = [(i, mine, theirs) for mine, theirs, i in pairs if abs(mine - theirs) >= 2]
summary = { "lab": "part-10/lab-benchmark-models-on-your-own-tasks/judge", "mode": "human-agreement", "judge_model": payload["judge"]["judge_model"], "model": payload["run"]["model"], "scored": len(pairs), "exact_agreement": round(exact / len(pairs), 2), "within_one": round(within_one / len(pairs), 2), "mean_human": round(sum(m for m, _, _ in pairs) / len(pairs), 2), "mean_judge": round(sum(t for _, t, _ in pairs) / len(pairs), 2), "big_disagreements": [{"id": i, "human": m, "judge": t} for i, m, t in disagreements], "date": time.strftime("%Y-%m-%d"), } print("\n" + json.dumps(summary, indent=2)) print("\nRead the big disagreements. Each one is either a rubric that says less than you " "meant, or a judge that cannot grade this category. Both are worth fixing.") record(args, summary)
def record(args, summary: dict) -> None: if not args.labbook: return with Path(args.labbook).open("a", encoding="utf-8") as handle: handle.write(json.dumps(summary) + "\n") print(f"\nrecorded in {args.labbook}")
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("mode", choices=["grade", "compare", "human"]) parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1") parser.add_argument("--api-key", default=None) parser.add_argument("--judge-model", default=None, help="the grading model; not the one under test") parser.add_argument("--results", default="results.json", help="grade: the run to score") parser.add_argument("--results-a", default=None, help="compare: the first run") parser.add_argument("--results-b", default=None, help="compare: the second run") parser.add_argument("--judged", default="judged.json", help="human: a file from grade mode") parser.add_argument("--out", default="judged.json", help="grade: where to write the scores") parser.add_argument("--sample", type=int, default=8, help="human: how many to score yourself") parser.add_argument("--seed", type=int, default=7) parser.add_argument("--timeout", type=int, default=300) parser.add_argument("--labbook", default=None) args = parser.parse_args()
if args.mode in {"grade", "compare"} and not args.judge_model: sys.exit("--judge-model is required for grade and compare") if args.mode == "compare" and not (args.results_a and args.results_b): sys.exit("compare needs --results-a and --results-b")
if args.mode == "grade": mode_grade(args) elif args.mode == "compare": mode_compare(args) else: mode_human(args)
if __name__ == "__main__": main()Stop the model under test, start a larger one as the judge, and grade the file:
RunnableAll tracks
python3 judge.py grade \ --base-url http://127.0.0.1:8080/v1 \ --judge-model qwen3-8b \ --results results-qwen3-4b.json \ --out judged-qwen3-4b.json \ --labbook labbook.mdThe judge is given the task, the rubric, the reference answer and the answer, and returns a score from one to five with a reason, under a JSON schema so that the score is always a number in range. The summary reports the mean, the mean per category, and one diagnostic: the mean answer length at each score. Read that last one now, because it is the fastest way to catch the first bias.
6. Score a sample yourself
Section titled “6. Score a sample yourself”This is the step that makes the rest of the lab worth anything, and it takes ten minutes.
RunnableAll tracks
python3 judge.py human \ --judged judged-qwen3-4b.json \ --sample 8 \ --labbook labbook.mdYou are shown the task, the rubric, the reference and the answer, and asked for a score. The judge’s score stays hidden until the end. Then you get exact agreement, agreement within one point, and a list of the disagreements of two points or more.
Read those disagreements. Each is one of two things, and both are useful. Either your rubric says less than you meant, in which case fix the rubric and the judge improves for free. Or the judge cannot grade that category, in which case that category is scored by the deterministic checks and by you, and its judge score is noise you should stop quoting.
7. Compare two models, twice, in both orders
Section titled “7. Compare two models, twice, in both orders”RunnableAll tracks
python3 run-eval.py \ --base-url http://127.0.0.1:8080/v1 \ --model qwen3-8b \ --quant Q4_K_M \ --engine llama.cpp \ --engine-version v0.4.0 \ --tasks my-tasks.json \ --out results-qwen3-8b.json \ --labbook labbook.mdRunnableAll tracks
python3 judge.py compare \ --results-a results-qwen3-4b.json \ --results-b results-qwen3-8b.json \ --judge-model qwen3-8b \ --labbook labbook.mdEvery pair is judged twice: once with A first and once with B first. A judge free of position
bias returns the same verdict both times. Every disagreement is a flip, and the reported
flip_rate is the share of tasks where the order alone changed the answer.
8. Name the biases in your own numbers
Section titled “8. Name the biases in your own numbers”Three are documented in the literature, and you now have the material to check each one on your own task set rather than take it on faith. The paper this lab cites examines “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” and proposes mitigations for some of them.
Position bias. Measured directly by task 7’s flip rate. Mitigation: always swap, and treat a flipped pair as a tie rather than picking the first verdict.
Verbosity bias. Judges tend to reward longer answers. The mean_words_by_score field in
your grade summary is the check: if answers scored five are much longer than answers scored
three, and your rubrics do not ask for length, the judge is scoring words. Mitigation: rubrics
that state the length requirement, max_words in the deterministic checks, and a judge system
prompt that says length is not quality, which the harness’s does.
Self-preference. A model scores its own output generously. The harness prints a warning
when the judge and the model under test are the same name and records self_judged in the
summary, so a run that did it is visible afterwards rather than forgotten. Mitigation: judge
with a different model, and ideally a larger one.
Limited reasoning. A judge cannot reliably grade an answer whose correctness it could not
have worked out itself. Arithmetic, code that has to run, and multi-step logic are the usual
cases. Mitigation: score those categories deterministically. A unit test is a better judge of
code than any model, and must_contain with the right answer is a better judge of arithmetic.
9. Write the results row
Section titled “9. Write the results row”Every script has been appending JSON lines to labbook.md as you went. Finish by adding a
table row that a human can read at a glance, in the course’s benchmark format: one row per
run, with everything needed to reproduce it.
Output — what you should see
| Date | Model | Quant | Engine | Settings | Tasks | Checks | Judge mean | Human agree || ---------- | --------- | ------ | ------------- | --------------------- | ----- | ------ | ---------- | ----------- || 2026-09-08 | qwen3-4b | Q4_K_M | llama.cpp ... | temp 0.0 seed 7 | 24 | 18/24 | 3.9 | 0.75 |The row above shows the columns, not results: fill it from your own summary lines. Two rules about it. Never add a row without the quantisation and the settings, because a row without them cannot be compared with anything. And never delete a row: a run that went badly is the baseline the next one is measured against.
Establish a reusable evaluation handoff
Section titled “Establish a reusable evaluation handoff”Name your frozen personal task file my-tasks.json in this part’s execution directory; later training
labs refer to that file. Read every rubric and deterministic check before generating model answers.
Include unavailable-information cases and important failures, rather than only prompts your current
model already handles well.
First run a small selection and inspect the complete request, response and check result. Verify that required-string tests actually reject a plausible wrong answer. Then run the full task set with fixed settings, keeping separate output filenames for each model and repeat. A judge failure should remain visible rather than being replaced with a favourable default score.
Grade a sample yourself before accepting judge summaries. Reverse answer order for paired grading
and inspect disagreements. Report task-category counts, judge agreement and latency alongside the
overall result. Archive my-tasks.json, run-eval.py, judge.py, raw results and the model settings.
If another part expects the harness at ~/eval, either copy those two scripts there or pass this
part’s absolute directory through that script’s --harness-dir option. Do not regenerate the task
set after seeing a fine-tune’s answers and continue calling it held out.
Validation
Section titled “Validation”You are done when all of the following are true:
my-tasks.jsoncontains at least twenty tasks of your own, spanning at least four categories, each with a prompt, a reference answer and a rubric;- at least three tasks use
must_contain,must_not_containormax_words; - at least two tasks are ones you expect the model to fail;
run-eval.pycompletes the whole set and writes a results file whoserunblock names the model, quantisation, engine, engine version, settings and date;judge.py gradeproduces a judged file with a score for every task and amean_words_by_scorediagnostic;judge.py humanreports exact agreement and agreement within one point over a sample of at least eight;judge.py comparereports a flip rate for two runs;labbook.mdcontains the run lines, the judge summaries and one readable table row per run.
Expected outcome
Section titled “Expected outcome”A file you will use for the rest of the course, and a sentence you can defend.
The sentence looks like this: “On my twenty-four tasks, at temperature zero with seed seven,
Qwen3-8B at Q4_K_M passed more of the deterministic checks than Qwen3-4B and scored higher
under a judge that agrees with me three times in four; the win was concentrated in the
reasoning and instruction categories and there was no difference in extraction.” Every clause
in it is checkable from labbook.md.
You should also finish with at least one uncomfortable finding, because everybody does. The usual ones: the bigger model is no better on the tasks you actually care about; the judge and you disagree on the category that matters most; two of your rubrics turned out to mean nothing when read by somebody other than you.
Troubleshooting
Section titled “Troubleshooting”Every task fails the deterministic checks. Look at one answer. The usual cause is a model
prefixing every reply with “Sure, here is…” or wrapping everything in a code fence, which
must_not_contain catches by design. That is a real finding about the model and a system
prompt is the fix; add it with --system and record that you did.
The judge returns nothing and the script reports failures. The judge model is probably in thinking mode, and the JSON schema forbids the reasoning block from the first token. Turn thinking off, as the prompting lesson describes, or use a non-thinking model as the judge.
All judge scores are 4 or 5. Either your tasks are too easy, or the judge is being
generous, or both. The human mode tells you which: if your scores are much lower than the
judge’s, it is the judge; if yours are also high, write harder tasks.
flip_rate is high on nearly every pair. The two answers are close enough that the judge
is guessing, or your rubric does not discriminate between them. Read three flipped pairs; if
you cannot pick a winner either, the honest result is a tie.
Two runs of the same model give different answers with the same seed. Some servers do not make sampling fully deterministic across batch sizes or restarts, and Part 6’s sampling lesson covers why. Record the variation rather than assuming it away: run the same model twice and treat any difference smaller than the run-to-run variation as no difference at all.
Judging takes longer than the run itself. Expected: the judge reads the task, the rubric, the reference and the answer for every item, so its prefill is several times the run’s. Use a smaller judge for iteration and the larger one for the numbers you record.
Cleanup
Section titled “Cleanup”Nothing needs undoing, and nothing here should be deleted:
RunnableAll tracks
mkdir -p ~/evalcp my-tasks.json run-eval.py judge.py judged-*.json results-*.json labbook.md ~/eval/What you learned
Section titled “What you learned”- A benchmark is a set of tasks somebody chose. Choosing them yourself is the entire difference between a number about your problem and a number about somebody else’s.
- Deterministic checks come first. Required strings, forbidden strings and a length limit cost nothing, never drift, and catch the failures that matter most in a pipeline.
- A rubric is a specification. Writing one usually reveals that you had not decided what a good answer was, and vague rubrics are the main cause of judges disagreeing with you.
- Settings are part of the result. Model, quantisation, engine, version, temperature, seed and date, or it is not a measurement.
- A judge is an instrument that needs calibrating. Position bias is measured by swapping, verbosity bias by comparing length against score, self-preference by never letting a model grade itself, and the whole thing by scoring a sample yourself.
- The categories a judge cannot grade should not be judged. Code gets a test; arithmetic gets a string match; the judge grades what is genuinely a matter of quality.
Record in the notebook: the track and machine; the task set name, version and task count with your category names; the model, quantisation, engine and version for each run; the fixed settings; the deterministic pass count and the judge mean per category; your exact and within-one agreement with the judge and the sample size; the flip rate from any comparison; and one sentence naming the finding that surprised you.
Check your understanding
Sources for this lesson
3 verified · checked 2026-09-08
- 01Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., arXiv:2306.05685)§ Abstract; biases of LLM judgesarxiv.org/abs/2306.056852026-09-08
- 02llama.cpp — llama-server README§ Sampling options; response_format; --aliasgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md2026-09-08
- 03Qwen3-8B model card§ Best Practices; recommended sampling settingshuggingface.co/Qwen/Qwen3-8B2026-09-08
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.