Lab: Run a Standard Benchmark Suite on Your Model

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The wall-clock per track, the harness version each track was executed with and the gaps each track actually observed belong here once the validation pass has run this lab on real machines.

Objective

By the end of this lab you will have produced a public benchmark number yourself, tried to match a published one, and written down exactly why the two differ. Then you will have run the same suite on your own models and reported the result in a form a stranger could repeat.

The reproduction is the point. Anyone can run a benchmark; the skill is accounting for the gap between your number and somebody else’s, because that gap is made of the settings nobody wrote down, and finding them is what makes the rest of your measurements trustworthy.

⌂Single-machine path

Single-machine on every track. The harness is a small Python process; the model is served through the gateway from Part 9 or directly by an engine on the same box.

Requirements

About an hour of attended work, plus a long unattended stretch: a full pass of a benchmark of a few hundred generative items through a locally served model takes considerably longer than the attended hour, and this lab runs it three times. The attended part is the install, the smoke test and the reproduction argument; the full runs are started and left.

You need Python and uv from Part 1, the gateway or a server from Part 9, and a reference model whose publisher put a benchmark table where you can read it. If you have them, your Part 13 fine-tune and your Part 15 student are the models you will evaluate afterwards; if not, two different reference models make the same lab.

Track S — NVIDIA DGX Spark

With 128 GB you can serve the reference model’s released weights at full precision through vLLM, which removes the quantisation from the list of things that could explain your gap. That is the cleanest reproduction attempt available on any track, and it is worth taking. Serve your own fine-tune and student through the gateway as usual.

Track X — AMD Ryzen AI Max+ 395

Serve the reference model as Q8_0 through llama-server on your Vulkan or ROCm build. Eight bits is close enough to the released weights that quantisation is unlikely to be the largest term in your gap, and small enough to leave the machine usable. Record the quantisation regardless; the point of the lab is that it appears in the report.

Track M — Apple silicon

On 32 GB and above, serve the reference model as Q8_0 through the Metal build. On 16 GB, use Q4_K_M and expect the quantisation to be a real term in the gap, which is a legitimate finding and a good reason to run the previous lab’s divergence measurement on the same file. mlx-lm’s server from Part 8 works equally well as the endpoint; the harness does not care which engine answers.

Track N — NVIDIA desktop or laptop

On 24 GB and above, serve Q8_0. On 8 to 16 GB, serve Q4_K_M and note it. Either way the harness runs on the CPU and talks to the server over HTTP, so the card is doing only what it always does.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-16-quantisation-and-evaluation"
cd "$LAB_DIR"
pwd
test -f "run-suite.sh"

Expected result: pwd ends in part-16-quantisation-and-evaluation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Install the harness

lm-evaluation-harness 0.4.13 · verified 2026-09-08 installs from a checkout. Keep it in its own environment; its dependencies are heavier than anything else in this part.

RunnableAll tracks

install lm-evaluation-harness with uv

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness ~/lm-evaluation-harness
uv venv ~/lm-evaluation-harness/.venv
source ~/lm-evaluation-harness/.venv/bin/activate
uv pip install -e ~/lm-evaluation-harness
uv pip install "lm_eval[api]"
lm_eval --tasks list | head -40

The last command is the check that it works and the fastest way to see the task catalogue. Look for ifeval and gsm8k; you will use both.

2. Choose a published claim, and read it properly

You need a model whose publisher states a number, and a task the harness implements. That combination is rarer than it sounds, and finding out how rare is the first lesson of the lab.

The Qwen3-8B card, which this course cites constantly, publishes no benchmark table at all: it gives the architecture, the context lengths and detailed sampling recommendations, and refers the reader to a blog post for evaluation results. That is not a criticism of the card, which is unusually good on the things it does cover. It is a fact about what is available to reproduce, and it is why this lab uses a different model as its target.

Meta’s Llama 3.1 model card does publish a table, with the shot counts and metrics beside the numbers, which is what makes it a usable reproduction target.

Vendor specification, not measuredLlama 3.1 8B Instruct — four rows as published by Meta, not measured here

Benchmark	Shots	Metric	Score
IFEval	—	—	80.4
GSM-8K (CoT)	8	em_maj1@1	84.5
MMLU	5	macro_avg/acc	69.4
GPQA	0	em	30.4

not stated by the source · not stated by the source Llama 3.1 model card in the meta-llama/llama-models repository · Llama 3.1 8B Instruct, not stated; presumed the released weights · 0 tokens of context · 2026-09-09

Reported by Meta in the Llama 3.1 model card, read on 2026-09-09. Context length is recorded as zero because the source states none for these evaluations, which is itself part of what makes them hard to reproduce. Reproduced here so that the IFEval row can be argued with: its shots and metric columns are empty in the source, and the harness's ifeval task reports four different metrics. The licence on these weights is the Llama 3.1 Community licence and the Hugging Face repository is gated; the GGUF conversions the course points at are not.

Read the IFEval row again. The score is stated; the shot count and the metric are not. The harness’s ifeval task is zero-shot and reports four metrics — prompt_level_strict_acc, inst_level_strict_acc, prompt_level_loose_acc and inst_level_loose_acc — so a single published figure is one of four, and which one is a guess until proven. Your job in this lab is to produce all four and see which, if any, lands near the published number.

If you would rather chase a fully specified row instead, take GSM-8K (CoT) from the same table: eight shots, em_maj1@1, and a chain-of-thought prompt. The harness’s gsm8k task is five-shot with its own prompt format and reports one score per extraction filter, so the settings visibly differ before you run anything. That makes it a poorer reproduction and a better illustration: you can name three concrete differences from the published configuration in advance, and predict the direction of at least one of them.

Either target teaches the same thing from a different angle. IFEval gives you a number whose metric is ambiguous; GSM-8K gives you a number whose settings are stated and different from the harness’s. Publishers produce both kinds, and the second is easier to argue with because there is something written down to disagree with.

∴Research note

This is the exercise, not an obstacle to it. A published score with an unstated metric is the normal case, and the useful skill is turning “their number and mine differ” into “their number matches one of my four, at these settings, and here is which”. If none of yours land near it, that is also a result, and the checklist in step five is how you say why.

3. Serve the reference model, and smoke-test the plumbing

Serve the model on your track, at whatever quantisation the requirements above indicate, and write down which one. Then run forty items to prove the harness can reach it.

RunnableAll tracks

forty items, to prove the plumbing works

lm_eval --model local-chat-completions \
    --model_args model=local/chat,base_url=http://127.0.0.1:4000/v1/chat/completions,num_concurrent=1,max_retries=3,tokenized_requests=False \
    --tasks ifeval \
    --apply_chat_template \
    --limit 40 \
    --log_samples \
    --output_path smoke

Output — what you should see

local-chat-completions (model=local/chat,base_url=…), gen_kwargs: (None), limit: 40.0,
num_fewshot: None, batch_size: 1
|Tasks |Version|Filter|n-shot|            Metric            |   |Value |   |Stderr|
|------|------:|------|-----:|------------------------------|---|-----:|---|-----:|
|ifeval|      4|none  |     0|inst_level_loose_acc          |↑  |…     |±  |   N/A|
|ifeval|      4|none  |     0|inst_level_strict_acc         |↑  |…     |±  |   N/A|
|ifeval|      4|none  |     0|prompt_level_loose_acc        |↑  |…     |±  |   N/A|
|ifeval|      4|none  |     0|prompt_level_strict_acc       |↑  |±  |…  |   …  |

Four metrics, as promised. The limit: 40.0 in the header is the harness telling you that this is not a comparable number, and it is right.

Open the logged samples now, while the run is small, and read one prompt as it was actually sent.

RunnableAll tracks

the first prompt, exactly as the model received it

find smoke -name 'samples_*.jsonl' -print -quit | xargs head -c 1200

You are looking for the chat template’s role markers. If they are absent, --apply_chat_template did not take effect and every number from this configuration is a formatting result rather than a model result. This one check costs a minute and prevents the most common wasted afternoon in this subject.

4. Run the suite properly, three times

RunnableAll tracks

run-suite.sh

#!/usr/bin/env bash
# Purpose: run a standard benchmark suite against a served model the same way every time -
#          same tasks, same shot count, same chat-template decision, repeated so the run-to-run
#          spread is measured rather than assumed - and record the settings the harness cannot
#          discover for itself, such as which quantisation is actually behind the alias
# Platform: all (spark, strix, mac, nvidia); the harness runs on the CPU and talks HTTP
# Minimum memory: 8 GB on the machine serving the model; this script needs very little
# Assumes: lm-evaluation-harness installed in a virtual environment (HARNESS), an
#          OpenAI-compatible endpoint at BASE_URL - the Part 9 gateway, llama-server or vLLM -
#          and a tokeniser identifier the harness can load for prompt length accounting
#
# Usage: bash run-suite.sh MODEL_ALIAS [TASKS]
#   MODEL_ALIAS  the name the endpoint answers to, e.g. local/chat
#   TASKS        comma-separated harness task names; defaults to ifeval
#
# Environment: BASE_URL (default http://127.0.0.1:4000/v1), API_KEY, HARNESS (venv directory),
#              BACKEND (local-chat-completions or local-completions), TOKENIZER,
#              NUM_FEWSHOT (unset: use the task file's own), CHAT_TEMPLATE (default yes),
#              RUNS (default 3),
#              LIMIT (leave unset for a full run), CONCURRENCY (default 1),
#              GEN_KWARGS (passed to --gen_kwargs; leave unset to use the task file's own),
#              QUANT (what is actually loaded), ENGINE, ENGINE_VERSION, OUT_DIR
set -euo pipefail

ALIAS="${1:-}"
TASKS="${2:-ifeval}"
BASE_URL="${BASE_URL:-http://127.0.0.1:4000/v1}"
BACKEND="${BACKEND:-local-chat-completions}"
NUM_FEWSHOT="${NUM_FEWSHOT:-}"
CHAT_TEMPLATE="${CHAT_TEMPLATE:-yes}"
RUNS="${RUNS:-3}"
CONCURRENCY="${CONCURRENCY:-1}"
QUANT="${QUANT:-unrecorded}"
ENGINE="${ENGINE:-unrecorded}"
ENGINE_VERSION="${ENGINE_VERSION:-unrecorded}"
OUT_DIR="${OUT_DIR:-suite-results}"

die() { echo "run-suite: $*" >&2; exit 1; }

[[ -n "$ALIAS" ]] || die "usage: bash run-suite.sh MODEL_ALIAS [TASKS]"

if [[ -n "${HARNESS:-}" ]]; then
  # shellcheck source=/dev/null
  source "$HARNESS/bin/activate"
fi
command -v lm_eval >/dev/null || die "lm_eval is not on PATH; set HARNESS to the virtual environment holding it"

if [[ "$QUANT" == "unrecorded" ]]; then
  echo "run-suite: WARNING: QUANT is unrecorded." >&2
  echo "           The endpoint cannot tell you which file is behind the alias, and a benchmark" >&2
  echo "           result without the quantisation is not reproducible. Set QUANT=Q4_K_M or" >&2
  echo "           whatever you actually served." >&2
fi

mkdir -p "$OUT_DIR"

ENDPOINT="$BASE_URL"
if [[ "$BACKEND" == "local-chat-completions" ]]; then
  ENDPOINT="${BASE_URL%/}/chat/completions"
else
  ENDPOINT="${BASE_URL%/}/completions"
fi

MODEL_ARGS="model=$ALIAS,base_url=$ENDPOINT,num_concurrent=$CONCURRENCY,max_retries=3,tokenized_requests=False"
if [[ -n "${TOKENIZER:-}" ]]; then
  MODEL_ARGS="$MODEL_ARGS,tokenizer=$TOKENIZER"
fi
if [[ -n "${API_KEY:-}" ]]; then
  export OPENAI_API_KEY="$API_KEY"
fi

echo "==> Suite: $TASKS"
echo "    alias:         $ALIAS"
echo "    backend:       $BACKEND -> $ENDPOINT"
echo "    quantisation:  $QUANT"
echo "    shots:         ${NUM_FEWSHOT:-the default in the task file}"
echo "    chat template: $CHAT_TEMPLATE"
echo "    gen_kwargs:    ${GEN_KWARGS:-the settings in the task file}"
echo "    runs:          $RUNS"
[[ -n "${LIMIT:-}" ]] && echo "    limit:         $LIMIT  (a limited run is not comparable with a published full run)"
echo

for run in $(seq 1 "$RUNS"); do
  seed=$(( run * 1000 + 7 ))
  echo "==> Run $run of $RUNS, seed $seed"

  ARGS=(
    --model "$BACKEND"
    --model_args "$MODEL_ARGS"
    --tasks "$TASKS"
    --batch_size 1
    --seed "$seed"
    --log_samples
    --output_path "$OUT_DIR/run-$run"
  )
  # Left unset, the shot count in the task file applies. Overriding it is a deliberate act
  # that has to appear in the report, which is why it is a variable rather than a default.
  if [[ -n "$NUM_FEWSHOT" ]]; then
    ARGS+=(--num_fewshot "$NUM_FEWSHOT")
  fi
  if [[ "$CHAT_TEMPLATE" == "yes" ]]; then
    ARGS+=(--apply_chat_template)
    if [[ -n "$NUM_FEWSHOT" && "$NUM_FEWSHOT" -gt 0 ]]; then
      ARGS+=(--fewshot_as_multiturn)
    fi
  fi
  if [[ -n "${LIMIT:-}" ]]; then
    ARGS+=(--limit "$LIMIT")
  fi
  # Generation settings override what the task file asked for. Setting them is how you turn a
  # greedy task into a sampled one, which is the third fault in the challenge.
  if [[ -n "${GEN_KWARGS:-}" ]]; then
    ARGS+=(--gen_kwargs "$GEN_KWARGS")
  fi

  lm_eval "${ARGS[@]}"
done

# The harness records everything it controls. These are the things it cannot see: which file the
# endpoint loaded, which engine is behind it, and which version. Without them the numbers above
# describe an alias rather than a model.
cat >"$OUT_DIR/run-context.json" <<JSON
{
  "alias": "$ALIAS",
  "tasks": "$TASKS",
  "backend": "$BACKEND",
  "endpoint": "$ENDPOINT",
  "quant": "$QUANT",
  "engine": "$ENGINE",
  "engine_version": "$ENGINE_VERSION",
  "num_fewshot": "${NUM_FEWSHOT:-task default}",
  "chat_template": "$CHAT_TEMPLATE",
  "gen_kwargs": "${GEN_KWARGS:-task default}",
  "runs": $RUNS,
  "limit": "${LIMIT:-none}",
  "date": "$(date +%Y-%m-%d)"
}
JSON

echo
echo "==> Done. $RUNS run(s) under $OUT_DIR/, settings in $OUT_DIR/run-context.json"
echo "    Summarise with:"
echo "      python3 summarise-results.py --results $OUT_DIR --labbook labbook.md"

Download run-suite.sh143 lines

The script runs the same suite RUNS times with different seeds, always logs samples, and writes a run-context.json holding the things the harness cannot discover: which file is behind the alias, which engine is serving it and at which version. Without that file the results describe an alias.

RunnableAll tracks

three full runs against the reference model

export HARNESS=~/lm-evaluation-harness/.venv
export BASE_URL=http://127.0.0.1:4000/v1
export API_KEY="${LITELLM_MASTER_KEY}"
QUANT=Q8_0 ENGINE=llama.cpp ENGINE_VERSION="$(llama-cli --version 2>&1 | head -1)" \
OUT_DIR=suite-reference \
    bash run-suite.sh local/chat ifeval

This is the unattended stretch. Start it, note the time, and come back. If your machine is one you also use, run it when you are not using it: a busy machine changes the concurrency the server sees and, through it, the batching, which is one of the ways a greedy run stops being exactly repeatable.

✕Common mistake

Reporting the limited run because the full one takes too long. A forty-item subset of a five-hundred-item benchmark has real sampling error and is not a random sample of it either, since the harness takes the first items. If you genuinely cannot run the full set, report the limit as part of the result and compare only with other runs at the same limit. Do not compare it with a published figure.

5. Account for the gap

RunnableAll tracks

summarise-results.py

#!/usr/bin/env python3
"""Turn several harness runs into one reportable line, with the spread and the settings attached.

Purpose: read the result files lm-evaluation-harness wrote for repeated runs of the same suite,
    report every metric with its mean and its range across runs, attach the settings the harness
    recorded and the ones run-suite.sh recorded for it, and - when you give it a published claim -
    show the gap and the list of settings that could account for it. A single number from a single
    run is not a result; this script is what turns a pile of runs into one.
Platform: all (pure Python, standard library only)
Minimum memory: negligible
Assumes: Python 3.9 or later and a directory written by run-suite.sh, containing one
    subdirectory per run with the harness's results_*.json files, plus run-context.json.

Usage: python3 summarise-results.py --results suite-results --labbook labbook.md
       python3 summarise-results.py --results suite-results \
           --published "ifeval:prompt_level_strict_acc=80.4" \
           --published-by "Meta, Llama 3.1 8B Instruct model card" --labbook labbook.md
"""

from __future__ import annotations

import argparse
import json
import platform
import statistics
import time
from collections import defaultdict
from pathlib import Path
from typing import Optional

# The settings that most often account for a gap between your number and somebody else's, in the
# order it is cheapest to check them. Printed whenever a published claim is supplied.
GAP_CHECKLIST = [
    "Chat template: was it applied on both sides? An instruction-tuned model without it answers "
    "as if its training scaffolding were absent.",
    "Shot count: the task's default, the flag you passed, and whatever the publisher used are "
    "three different numbers until you check.",
    "Metric: one task reports several. ifeval reports four; gsm8k reports one per extraction "
    "filter. A published figure names one of them, sometimes only implicitly.",
    "Quantisation: you almost certainly served a quantised file and the publisher almost "
    "certainly measured the released weights.",
    "Sampling: greedy or sampled, and at what temperature. Some cards advise against greedy "
    "decoding for their own model.",
    "Thinking mode: on or off changes the answer, the length and the cost.",
    "Prompt construction: publishers frequently use their own prompt and their own extraction "
    "code rather than this harness.",
    "Items scored: a limited run is not comparable with a full one, and neither is a different "
    "version of the dataset.",
]


def find_result_files(root: Path) -> list:
    return sorted(root.rglob("results_*.json"))


def metric_rows(payload: dict) -> dict:
    """Flatten {task: {"metric,filter": value}} into {(task, metric, filter): value}."""
    out = {}
    for task, metrics in (payload.get("results") or {}).items():
        if not isinstance(metrics, dict):
            continue
        for key, value in metrics.items():
            if not isinstance(value, (int, float)) or key == "alias":
                continue
            metric, _, filt = key.partition(",")
            if metric.endswith("_stderr"):
                continue
            out[(task, metric, filt or "none")] = float(value)
    return out


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--results", default="suite-results",
                        help="directory written by run-suite.sh")
    parser.add_argument("--published", action="append", default=[],
                        help="a claim to compare against, as task:metric=value; repeatable")
    parser.add_argument("--published-by", default=None,
                        help="who published the claim and where, quoted in the report")
    parser.add_argument("--out", default=None)
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    root = Path(args.results)
    files = find_result_files(root)
    if not files:
        raise SystemExit(f"no results_*.json under {root}; run run-suite.sh first")

    context = {}
    context_file = root / "run-context.json"
    if context_file.exists():
        context = json.loads(context_file.read_text(encoding="utf-8"))

    collected = defaultdict(list)
    harness_config: Optional[dict] = None
    for path in files:
        payload = json.loads(path.read_text(encoding="utf-8"))
        harness_config = payload.get("config") or harness_config
        for key, value in metric_rows(payload).items():
            collected[key].append(value)

    if not collected:
        raise SystemExit("the result files contained no numeric metrics; check the harness output")

    print(f"\n{len(files)} result file(s) under {root}\n")
    header = ["Task", "Metric", "Filter", "Runs", "Mean", "Min", "Max", "Range"]
    print("| " + " | ".join(header) + " |")
    print("| " + " | ".join("---" for _ in header) + " |")

    summary_rows = []
    for (task, metric, filt), values in sorted(collected.items()):
        mean = statistics.fmean(values)
        row = {
            "task": task, "metric": metric, "filter": filt, "runs": len(values),
            "mean": round(mean, 4), "min": round(min(values), 4), "max": round(max(values), 4),
            "range": round(max(values) - min(values), 4), "values": [round(v, 4) for v in values],
        }
        summary_rows.append(row)
        print(f"| {task} | {metric} | {filt} | {len(values)} | {mean:.4f} "
              f"| {min(values):.4f} | {max(values):.4f} | {max(values) - min(values):.4f} |")

    print("\nSettings recorded by the harness:")
    for key in ("model", "model_args", "num_fewshot", "batch_size", "limit", "gen_kwargs",
                "apply_chat_template", "fewshot_as_multiturn", "random_seed"):
        if harness_config and key in harness_config:
            print(f"  {key}: {harness_config[key]}")
    if context:
        print("\nSettings the harness could not discover, recorded by run-suite.sh:")
        for key in ("alias", "quant", "engine", "engine_version", "endpoint", "limit"):
            if key in context:
                print(f"  {key}: {context[key]}")

    # ---- comparison with a published claim -------------------------------
    comparisons = []
    for claim in args.published:
        try:
            target, _, value = claim.partition("=")
            task, _, metric = target.partition(":")
            published = float(value)
        except ValueError:
            raise SystemExit(f"could not read --published {claim!r}; use task:metric=value")

        matches = [r for r in summary_rows if r["task"] == task and r["metric"] == metric]
        if not matches:
            print(f"\nNo measurement of {task}:{metric} in these runs, so nothing to compare.")
            continue
        for row in matches:
            # Published scores are usually percentages and harness metrics are usually fractions.
            mine = row["mean"] * 100 if row["mean"] <= 1.0 else row["mean"]
            gap = mine - published
            comparisons.append({
                "task": task, "metric": metric, "filter": row["filter"],
                "published": published, "measured": round(mine, 2), "gap": round(gap, 2),
                "published_by": args.published_by,
            })
            source = f" reported by {args.published_by}" if args.published_by else ""
            print(f"\n{task} / {metric} [{row['filter']}]")
            print(f"  published{source}: {published}")
            print(f"  measured here:     {mine:.2f}")
            print(f"  gap:               {gap:+.2f}  (spread across your own runs: "
                  f"{row['range'] * (100 if row['mean'] <= 1.0 else 1):.2f})")

    if comparisons:
        print("\nBefore attributing a gap to the model, work down this list:")
        for i, item in enumerate(GAP_CHECKLIST, 1):
            print(f"  {i}. {item}")
        print("\nA gap you can explain is a result. A gap you cannot explain is an open question, "
              "\nand recording it as one is more useful than picking a story for it.")

    record = {
        "lab": "part-16/lab-run-a-standard-benchmark-suite",
        "run_id": time.strftime("%Y%m%dT%H%M%S"),
        "results_dir": str(root),
        "result_files": len(files),
        "rows": summary_rows,
        "harness_config": {k: harness_config.get(k) for k in
                           ("model", "model_args", "num_fewshot", "limit", "gen_kwargs")}
        if harness_config else {},
        "run_context": context,
        "comparisons": comparisons,
        "host": platform.platform(),
        "date": time.strftime("%Y-%m-%d"),
    }

    if args.out:
        Path(args.out).write_text(json.dumps(record, indent=2), encoding="utf-8")
        print(f"\nwritten to {args.out}")
    if args.labbook:
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download summarise-results.py195 lines

RunnableAll tracks

summarise the three runs and compare with the published claim

python3 summarise-results.py \
    --results suite-reference \
    --published "ifeval:prompt_level_strict_acc=80.4" \
    --published-by "Meta, Llama 3.1 model card, read 2026-09-09" \
    --labbook labbook.md

The script reports every metric with its mean and its range across the three runs, prints the settings the harness recorded and the ones you recorded for it, then shows the gap and the checklist of things that could explain it.

Work the checklist in order and write one line per item, even when the answer is “not this one”.

Which metric. Compare the published figure against all four of yours, not against the one you happened to look at first. If one of them is close and the others are not, you have probably identified the publisher’s metric, and you should say “probably” in your notes.

Chat template. You applied it; the publisher almost certainly did too, since the model is instruction-tuned. Confirm from your logged samples rather than from memory.

Shots. The task is zero-shot and the published row is blank. If your number is far below the published one, try the harness’s other likely configurations before concluding anything about the model.

Quantisation. You served a quantised file unless you are on Track S with the released weights. The previous lab’s divergence measurement on this exact file tells you whether that is plausibly worth points; if you did not run it on this model, say the term is unquantified rather than guessing it is small.

Sampling. Confirm what the server did. The harness’s gen_kwargs column and the task file’s own generation settings together decide it, and a server with its own default temperature can override what you thought you asked for.

Prompt construction. Publishers routinely use their own prompt and their own answer extraction. This is the term you usually cannot close, and naming it as unclosable is the honest end of the investigation.

!Important

A gap you can explain is a result. A gap you cannot explain is an open question, and recording it as one is more useful than choosing a story for it. The failure mode this lab exists to prevent is the third option: quietly assuming the difference is the model.

6. Run the same suite on your own models

Now the comparison the lab was actually for. Point the same script at your Part 13 fine-tune and your Part 15 student, at identical settings, through the same gateway.

RunnableAll tracks

the same suite, on your fine-tune

QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=suite-finetune \
    bash run-suite.sh local/finetune ifeval

RunnableAll tracks

the same suite, on your distilled student

QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=suite-student \
    bash run-suite.sh local/student ifeval

If you do not have those models, use two reference models of different sizes from the model library; the mechanics and the report are identical and so is the lesson.

=Expected result

Expect the fine-tune to be worse than its base on a general instruction-following benchmark, and do not be alarmed by it. Part 13 taught you that fine-tuning on a narrow format trades general behaviour for the behaviour you trained; IFEval measures the general behaviour. The interesting number is how much it gave up, set against how much it gained on your own task set, which is the comparison the capstone asks for.

7. Report

Assemble the three summaries into one table with its settings, and put it in the notebook.

RunnableAll tracks

one summary per model, into the notebook

for d in suite-reference suite-finetune suite-student; do
    python3 summarise-results.py --results "$d" --labbook labbook.md
done

The report needs the model, the quantisation and its source, the engine and harness versions, the exact task and metric names, the shot count, whether the chat template was applied, the sampling settings, the number of items scored, the number of runs with their spread, and the date.

Distinguish the smoke result from the reportable result

Use the limited run to inspect one rendered prompt, generated or scored answer, extraction result and task verdict. Confirm the checkpoint, few-shot configuration and chat-template behaviour before running the full suite. The smoke subset is not comparable with a published full-dataset result.

Save the harness revision, task configuration, dataset revision, seeds and limits. When reproducing a published claim, record missing protocol details explicitly. A different quantisation or prompt format can make your result useful while preventing a strict reproduction claim.

Preserve raw outputs from every repeat and keep failed tasks in the documented denominator. Inspect the difference between sampling variation on the same tasks and uncertainty about the wider task population. If the harness executes generated code, use its isolated evaluation path rather than running unreviewed model code in your normal workspace.

Before writing the final table, verify that the summary points to the intended run files and that each row identifies its exact artefact and settings. Include a short explanation of important gaps from the published result and of what the benchmark does not test for your application. Keep the personal task suite alongside the standard suite; they answer complementary questions.

Validation

You are done when all of the following are true.

• lm_eval --tasks list runs, and ifeval is in it.
• The smoke test produced four metrics, and you have read one logged prompt and confirmed the chat template’s role markers are present.
• suite-reference holds three full runs, each with logged samples, plus a run-context.json naming the quantisation and the engine.
• The summariser prints a range across the three runs, and you can state it.
• You have a written line for each item on the gap checklist, including the ones you ruled out.
• You have the same suite run on at least one model of your own at identical settings.
• The notebook holds the table, the settings and the date.

Expected outcome

Three numbers with their spread, one attempted reproduction with its gap accounted for, and a sharper sense of how much of a published benchmark score is the model.

Two shapes are worth predicting. The spread across three runs of the same configuration should be small, and if it is not, something is sampling when you thought it was not. And the four IFEval metrics should not be close together: instruction-level accuracy counts each instruction while prompt-level accuracy demands all of them, so prompt-level is the harder measure by construction.

The gap itself is unlikely to be zero and does not need to be. What the lab asks is that you can put a name to each term in it, and that the terms you cannot close are named as unclosed rather than absorbed into a shrug. A written account that reads “the quantisation is worth an unknown amount, the metric is probably prompt-level strict, and the prompt construction is theirs and unpublished” is a better artefact than a number that happens to match.

Pending validationYour benchmark report — three models, one suite, one settings block

Model	Quantisation	Metric	Mean of 3 runs	Range	Published	Gap
Reference model	—	prompt_level_strict_acc	—	—	—	—
Reference model	—	inst_level_loose_acc	—	—	—	—
Your Part 13 fine-tune	—	prompt_level_strict_acc	—	—	n/a	n/a
Your Part 15 student	—	prompt_level_strict_acc	—	—	n/a	n/a

your machine: track, chip and memory, your operating system and version · the engine behind the alias, per row engine version and harness version · as listed in the first column, as listed in the second column · 4,096 tokens of context · the date you ran it

Empty on purpose. Two metrics for the reference model on purpose: choosing one to quote is a decision that has to be visible. The Published and Gap columns apply only to the reference row, because nobody has published a figure for your fine-tune, which is the whole reason you had to measure it.

Troubleshooting

The harness cannot reach the endpoint. Check the URL includes the full path for the backend: local-chat-completions wants /v1/chat/completions and local-completions wants /v1/completions. The script builds this for you; a hand-written command frequently omits it.

Every request fails with an authorisation error. The gateway from Part 9 requires its key. Export it and let the script pass it through, rather than putting it on a command line where it lands in your shell history.

The harness complains that it cannot load a tokeniser. The HTTP backends want one for prompt accounting. Pass the model’s Hugging Face identifier through TOKENIZER, or set tokenized_requests=False as the script already does and accept the coarser accounting.

A task refuses an explicit shot count. Some tasks fix their shot count and reject an override. Leave NUM_FEWSHOT unset, which is the script’s default, and the task file’s own value applies.

The run is far slower than expected. Generative benchmarks send hundreds of requests one at a time by default. Raise CONCURRENCY if your server batches well, which Part 9’s load test told you. Be aware that changing concurrency changes batching, and therefore is a change of settings that belongs in the report.

Your number is nowhere near the published one on any metric. Read a logged prompt before anything else. In this situation it is almost always the chat template, the wrong model behind the alias, or a task variant that does not match the published one.

Your three runs differ substantially. Something is sampling. Check the task’s generation settings and the server’s own defaults, and remember that a server handling other traffic changes its batching and with it the exact arithmetic.

Cleanup

Keep the results directories and the notebook entry; they are the evidence for every claim you make about these models afterwards, and the challenge on the next page needs two of them.

The harness checkout is a few hundred megabytes and worth keeping. If you need the space back, remove the virtual environment rather than the checkout, and the model files through your usual model-library housekeeping.

RunnableAll tracks

free the environment, keep the results

rm -rf ~/lm-evaluation-harness/.venv

Stop the server or release the gateway alias if you started something specially for this lab.

What you learned

• A published score is a claim with settings attached, or it is not reproducible. The Qwen3-8B card publishes none; Meta’s publishes shot counts and metrics for most rows and leaves them blank for others. Both facts are usable once you have looked.
• One task can report several metrics, and choosing one is a decision. IFEval’s four are not close together, and a published figure names one of them, sometimes only implicitly.
• The chat template is the first thing to verify and the easiest to verify. One logged prompt answers it.
• A limited run is a different measurement. Report the limit or do not report the number.
• Three runs and a range beat one run and a number. The spread is what tells you whether a difference between models is real.
• The gap is made of settings. Metric, template, shots, quantisation, sampling, prompt construction. Work them in order, write a line for each, and name the ones you cannot close.

Record in the notebook: the published claim and where you read it, your four metrics with their spread, the line-by-line account of the gap, the same suite on your own models at identical settings, and the full settings block. Keep two of the result directories intact; the challenge on the next page compares them.

Check your understanding

Question 1. Meta's card reports IFEval 80.4 with the shots and metric columns blank. What does the harness give you?

• The same single number
• Four numbers - prompt-level and instruction-level, each strict and loose - so the published figure corresponds to one of them and which is a matter of argument
• A number only if you pass --num_fewshot
• Nothing, because IFEval is not implemented

Show the answer and why

Answer: Four numbers - prompt-level and instruction-level, each strict and loose - so the published figure corresponds to one of them and which is a matter of argument

The ifeval task configuration defines prompt_level_strict_acc, inst_level_strict_acc, prompt_level_loose_acc and inst_level_loose_acc. Prompt-level demands every instruction in a prompt be satisfied and instruction-level counts each one, so they are not close together.

Question 2. Why does the lab have you read a logged sample after the smoke test?

• To check the model's spelling
• To confirm the chat template was actually applied, since its absence turns every number in the run into a formatting result rather than a model result
• To count the tokens
• Because the harness requires it

Show the answer and why

Answer: To confirm the chat template was actually applied, since its absence turns every number in the run into a formatting result rather than a model result

Chat template applied or not is the largest single lever on an instruction-tuned model and the cheapest to verify: the role markers are either in the prompt as sent or they are not. Without --log_samples the prompt as sent is not recoverable at all.

Question 3. What is run-context.json for?

• It duplicates the harness output
• It records what the harness cannot discover: which file the endpoint actually loaded, which engine served it and at which version
• It stores the API key
• It configures the tasks

Show the answer and why

Answer: It records what the harness cannot discover: which file the endpoint actually loaded, which engine served it and at which version

An endpoint reports an alias, not a quantisation. A benchmark result whose quantisation, engine and engine version are unknown describes a name rather than a model, and the challenge on the next page is built on exactly this gap.

Question 4. You cannot run the full benchmark in the time you have. What is the honest reporting?

• Report the limited run as the benchmark score
• Report the limit as part of the result, compare only with other runs at the same limit, and do not compare with a published full-set figure
• Report the number and mention the limit in a footnote
• Extrapolate to the full set

Show the answer and why

Answer: Report the limit as part of the result, compare only with other runs at the same limit, and do not compare with a published full-set figure

A subset has real sampling error and, since the harness takes the first items, is not even a random subset. It remains useful for comparing two of your own configurations against each other at the same limit, which is a different and legitimate use.

Question 5. Your fine-tune scores lower than its base model on IFEval. Which readings are sound? Select all that apply.

• Fine-tuning on a narrow format trades general instruction-following for the behaviour you trained
• IFEval measures general instruction-following, which is not what the fine-tune was for
• The useful comparison is this loss set against the gain on your own task set
• The fine-tune has failed and should be discarded

Show the answer and why

Answer: Fine-tuning on a narrow format trades general instruction-following for the behaviour you trained, IFEval measures general instruction-following, which is not what the fine-tune was for, The useful comparison is this loss set against the gain on your own task set

The first three. A drop on a general benchmark is the expected cost of specialisation, which Part 13 established; the question is whether the trade was worth it, and that needs both numbers. Discarding the model on one general benchmark discards the thing you actually built.

Check my answersClear