Challenge: The Benchmark That Lied

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The per-track faults, the size of each gap and the tool versions they were reproduced with belong here once the validation pass has run this page on real machines.

Objective

A colleague sends you a benchmark result. You run the same benchmark on the same model and get a number ten points different. One of you is measuring something other than what you think.

By the end of this page you will have a procedure that finds out which, in about five minutes, and you will have run it on five faults you introduced on purpose so that you recognise each one when it is not your fault. The deliverable is not a fixed run; it is five written diagnoses, each with the evidence that identified it, the single change that closed the gap, and the second run that proved it.

⌂Single-machine path

Single-machine on every track. Everything here is two runs of a small benchmark against one served model, and every run uses a limit, because the point is the diagnosis rather than the score.

The procedure

Two evaluation runs disagree: what to do, in order

1. 1Establish the noise floor firstRun one configuration twice. If the two runs differ by as much as the disputed gap, there is no fault to find and the answer is that neither number was a measurement.
2. 2Compare the configurations, not the scoresThe harness records what it was asked to do. Diff the two result files before forming any theory about the model.
3. 3Read one prompt from each runLogged samples show the prompt as actually sent. Role markers present in one and absent in the other ends the investigation immediately.
4. 4Check what the harness could not seeWhich file was behind the alias, which engine served it, at which version. The harness records none of this and it is the fault it cannot help you with.
5. 5Name one fault and change one thingThen re-run. Two changes at once means you will not know which one worked.
6. 6Prove it twice overThe gap closes AND the evidence that identified the fault changes. A gap that closes on its own means something else moved.

Each step is cheap and eliminates a large fraction of the possibilities. By step four you are usually looking at one candidate rather than five.

The short list of faults

Almost every disagreement between two evaluation runs is one of these five. Learn the list and you have learned the diagnosis.

Fault	What the evidence looks like
Chat template applied in one run and not the other	The logged prompts differ structurally: role markers and a system turn in one, bare text in the other. The configuration shows apply_chat_template set in one run only.
A different few-shot count or format	The logged prompts contain a different number of worked examples, or the same examples as one block of text rather than as conversation turns.
Sampling in one run and greedy decoding in the other	gen_kwargs differ, or the server’s own defaults applied. The tell is that the sampled configuration also disagrees with itself between two runs.
Contamination	The scores are fine and the conclusion is not: the model has seen these items. Nothing in the harness output shows it; you have to compare the items with the training data.
A different file behind the same alias	Every recorded setting is identical and the numbers still differ. The quantisation, the engine or its version changed, and only a hand-written record shows it.

Note what is not on that list: the model. Two runs of the same benchmark on the same weights differ because of the five rows above, and treating the difference as a fact about the model is the mistake this page exists to prevent.

Requirements

The harness and the results from the previous lab, a served model, and about forty-five minutes, all of it attended. No new downloads. Every run below uses --limit 40 deliberately: two runs at the same limit are comparable with each other, which is all the diagnosis needs, and a full run would turn a forty-five-minute exercise into an afternoon.

Track S — NVIDIA DGX Spark

Serve the model you used in the previous lab through the gateway. With 128 GB you can also keep a second file loaded under a second alias, which makes the fifth fault - the same alias serving a different file - easier to stage without restarting anything.

Track X — AMD Ryzen AI Max+ 395

Serve through llama-server on your Vulkan or ROCm build, one file at a time. For the fifth fault, stop the server and restart it with a different file under the same --alias, which is exactly how the fault happens in real life.

Track M — Apple silicon

On 16 GB, use a 4B-class model so that swapping between two quantisations is quick. The Metal build and the mlx-lm server from Part 8 both work as the endpoint; if you use both, note that a change of engine is itself an instance of the fifth fault.

Track N — NVIDIA desktop or laptop

Any card that ran the previous lab runs this one. On 8 to 12 GB, use a 4B-class model at Q4_K_M and a second copy at Q8_0 for the fifth fault; both fit comfortably one at a time.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-16-quantisation-and-evaluation"
cd "$LAB_DIR"
pwd
test -f "diff-eval-runs.py"

Expected result: pwd ends in part-16-quantisation-and-evaluation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Establish the noise floor, before anything else

Two runs of an identical configuration. If they disagree by as much as the gap you are chasing, there is nothing to diagnose and the honest answer is that neither number was a measurement.

RunnableAll tracks

the same configuration, twice

export HARNESS=~/lm-evaluation-harness/.venv
export BASE_URL=http://127.0.0.1:4000/v1
LIMIT=40 RUNS=2 QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=noise-floor \
    bash run-suite.sh local/chat ifeval
python3 summarise-results.py --results noise-floor --labbook labbook.md

Read the Range column. That number is your resolution for the rest of this page, and every gap below has to be larger than it before you may call it a fault.

2. Meet the diagnostic tool

RunnableAll tracks

diff-eval-runs.py

#!/usr/bin/env python3
"""Compare two evaluation runs and say, in order of importance, what was different about them.

Purpose: the diagnostic instrument for the challenge. Given two lm-evaluation-harness output
    directories that disagree, it reports the score difference per metric and then every setting
    that differs between the runs, ranked by how much that setting is known to move a score. It
    does not decide which run is right; it removes the guessing from the list of candidates.
Platform: all (pure Python, standard library only)
Minimum memory: negligible
Assumes: Python 3.9 or later, and two directories (or two results_*.json files) written by
    lm-evaluation-harness. Runs made by run-suite.sh also carry a run-context.json holding the
    quantisation and engine, which the harness cannot see and which this script compares too.

Usage: python3 diff-eval-runs.py --a run-alpha --b run-beta
       python3 diff-eval-runs.py --a run-alpha --b run-beta --samples --labbook labbook.md
"""

from __future__ import annotations

import argparse
import json
import platform
import time
from pathlib import Path
from typing import Optional

# Ranked by how far each is known to move a benchmark score, worst first. The order is the
# diagnostic procedure: the first difference found is almost always the explanation, and looking
# further down the list before resolving it wastes the afternoon.
RANKED_SETTINGS = [
    ("apply_chat_template", "Chat template applied or not. The largest single lever on an "
                            "instruction-tuned model, and it changes every prompt in the run."),
    ("fewshot_as_multiturn", "Few-shot examples as conversation turns or as one block of text."),
    ("num_fewshot", "Number of in-context examples. Overrides the task file's own default."),
    ("gen_kwargs", "Generation settings: temperature, top-p, whether sampling happens at all."),
    ("model", "Which backend was used to reach the model."),
    ("model_args", "Which model, at which endpoint, with which tokeniser."),
    ("limit", "How many items were scored. A limited run is a different measurement."),
    ("batch_size", "Batching changes summation order and can flip close decisions."),
    ("random_seed", "Seeds. Only matters when something is sampled."),
    ("numpy_seed", "Seeds. Only matters when something is sampled."),
    ("torch_seed", "Seeds. Only matters when something is sampled."),
    ("fewshot_seed", "Which few-shot examples were drawn."),
    ("device", "Where the model ran, when the harness held the weights itself."),
]

CONTEXT_KEYS = [
    ("quant", "The quantisation actually served. The endpoint cannot report it, so it is "
              "recorded by hand and is therefore the field most often wrong."),
    ("engine", "The engine behind the endpoint."),
    ("engine_version", "The engine's version. Sampler and template handling change between them."),
    ("endpoint", "Which server answered."),
    ("alias", "Which gateway alias was called, which decides which file was loaded."),
]


def newest_results(path: Path) -> Optional[Path]:
    if path.is_file():
        return path
    candidates = sorted(path.rglob("results_*.json"))
    return candidates[-1] if candidates else None


def load_run(path: Path) -> dict:
    results_file = newest_results(path)
    if results_file is None:
        raise SystemExit(f"no results_*.json found under {path}")
    payload = json.loads(results_file.read_text(encoding="utf-8"))

    context = {}
    for parent in [path] + list(path.parents)[:2]:
        candidate = parent / "run-context.json"
        if candidate.is_file():
            context = json.loads(candidate.read_text(encoding="utf-8"))
            break

    return {"file": results_file, "payload": payload, "context": context}


def scores(payload: dict) -> dict:
    out = {}
    for task, metrics in (payload.get("results") or {}).items():
        if not isinstance(metrics, dict):
            continue
        for key, value in metrics.items():
            if isinstance(value, (int, float)) and not key.startswith("alias") \
                    and "_stderr" not in key:
                out[f"{task}/{key}"] = float(value)
    return out


def show(value) -> str:
    text = json.dumps(value) if not isinstance(value, str) else value
    return text if len(text) <= 110 else text[:107] + "..."


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--a", required=True, help="first run: a directory or a results_*.json")
    parser.add_argument("--b", required=True, help="second run: a directory or a results_*.json")
    parser.add_argument("--samples", action="store_true",
                        help="also compare the first logged sample from each run, which shows the "
                             "prompt as actually sent")
    parser.add_argument("--out", default=None)
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    run_a = load_run(Path(args.a))
    run_b = load_run(Path(args.b))
    print(f"A: {run_a['file']}")
    print(f"B: {run_b['file']}")

    # ---- 1. what actually differs in the numbers -------------------------
    score_a, score_b = scores(run_a["payload"]), scores(run_b["payload"])
    shared = sorted(set(score_a) & set(score_b))
    print("\n== Scores ==")
    if not shared:
        print("  The two runs share no metric. They did not measure the same thing, which is "
              "already the answer.")
    differences = []
    for key in shared:
        delta = score_b[key] - score_a[key]
        differences.append({"metric": key, "a": score_a[key], "b": score_b[key],
                            "delta": round(delta, 4)})
        flag = "  <-- " if abs(delta) >= 0.01 else "      "
        print(f"  {key:52s} A={score_a[key]:.4f}  B={score_b[key]:.4f}  Δ={delta:+.4f}{flag}")
    only_a = sorted(set(score_a) - set(score_b))
    only_b = sorted(set(score_b) - set(score_a))
    for key in only_a:
        print(f"  {key:52s} present in A only")
    for key in only_b:
        print(f"  {key:52s} present in B only")

    # ---- 2. the settings, in the order they matter -----------------------
    config_a = run_a["payload"].get("config") or {}
    config_b = run_b["payload"].get("config") or {}
    found = []
    print("\n== Settings that differ, most consequential first ==")
    for key, why in RANKED_SETTINGS:
        left, right = config_a.get(key), config_b.get(key)
        if left != right:
            found.append({"setting": key, "a": left, "b": right, "why": why})
            print(f"  {key}")
            print(f"      A: {show(left)}")
            print(f"      B: {show(right)}")
            print(f"      {why}")

    other_keys = (set(config_a) | set(config_b)) - {k for k, _ in RANKED_SETTINGS}
    for key in sorted(other_keys):
        if config_a.get(key) != config_b.get(key):
            found.append({"setting": key, "a": config_a.get(key), "b": config_b.get(key),
                          "why": "not on the ranked list; check it after the ones above"})
            print(f"  {key} (unranked)")
            print(f"      A: {show(config_a.get(key))}")
            print(f"      B: {show(config_b.get(key))}")

    # ---- 3. the task configuration itself --------------------------------
    task_a = run_a["payload"].get("configs") or {}
    task_b = run_b["payload"].get("configs") or {}
    print("\n== Task configuration ==")
    task_diffs = []
    for task in sorted(set(task_a) | set(task_b)):
        left, right = task_a.get(task, {}), task_b.get(task, {})
        for field in sorted(set(left) | set(right)):
            if left.get(field) != right.get(field):
                task_diffs.append({"task": task, "field": field,
                                   "a": left.get(field), "b": right.get(field)})
                print(f"  {task}.{field}")
                print(f"      A: {show(left.get(field))}")
                print(f"      B: {show(right.get(field))}")
    if not task_diffs:
        print("  Identical. Both runs used the same task definition, so the difference is in the "
              "run settings or outside the harness entirely.")

    # ---- 4. what the harness could not see -------------------------------
    print("\n== Outside the harness ==")
    context_diffs = []
    if not run_a["context"] and not run_b["context"]:
        print("  Neither run has a run-context.json, so the quantisation, the engine and its "
              "version are unrecorded for both. That is itself a finding: the two runs may have "
              "been served different files and nothing here would show it.")
    else:
        for key, why in CONTEXT_KEYS:
            left, right = run_a["context"].get(key), run_b["context"].get(key)
            if left != right:
                context_diffs.append({"setting": key, "a": left, "b": right, "why": why})
                print(f"  {key}")
                print(f"      A: {show(left)}")
                print(f"      B: {show(right)}")
                print(f"      {why}")
        if not context_diffs:
            print("  Identical where recorded.")

    # ---- 5. the prompt as actually sent ----------------------------------
    if args.samples:
        print("\n== First logged sample from each run ==")
        for name, run in (("A", run_a), ("B", run_b)):
            base = run["file"].parent
            sample_files = sorted(base.glob("samples_*.jsonl"))
            if not sample_files:
                print(f"  {name}: no samples_*.jsonl. Re-run with --log_samples; without it the "
                      f"prompt as sent is not recoverable.")
                continue
            with sample_files[0].open(encoding="utf-8") as handle:
                first = handle.readline()
            try:
                record = json.loads(first)
            except ValueError:
                print(f"  {name}: could not parse the first sample line.")
                continue
            shown = False
            for field in ("arguments", "doc", "filtered_resps", "resps"):
                if field in record:
                    print(f"  {name}.{field}: {show(record[field])}")
                    shown = True
            if not shown:
                print(f"  {name}: fields present: {', '.join(sorted(record)[:12])}")
        print("\n  Read the two prompts side by side. Role markers present in one and absent in "
              "the other is a chat-template difference; a different number of worked examples is "
              "a shot-count difference. Both are visible here and invisible in the score alone.")

    # ---- verdict ---------------------------------------------------------
    print("\n== Verdict ==")
    if found:
        first = found[0]
        print(f"  Highest-ranked difference: {first['setting']}.")
        print("  Change that one thing, re-run, and see whether the gap closes. One change at a "
              "time, or you will not know which one worked.")
    elif context_diffs:
        print(f"  The harness settings are identical; the difference is outside it: "
              f"{context_diffs[0]['setting']}.")
    elif task_diffs:
        print("  The run settings are identical and the task definition is not. Check the harness "
              "version and the dataset revision.")
    else:
        print("  No recorded setting differs. Either the difference is genuine run-to-run "
              "variation - establish the noise floor by running one configuration three times - "
              "or something changed that neither the harness nor run-context.json records, such "
              "as which file the endpoint had loaded.")

    record = {
        "lab": "part-16/challenge-the-benchmark-that-lied",
        "run_id": time.strftime("%Y%m%dT%H%M%S"),
        "a": str(run_a["file"]),
        "b": str(run_b["file"]),
        "score_differences": differences,
        "setting_differences": found,
        "task_differences": task_diffs,
        "context_differences": context_diffs,
        "host": platform.platform(),
        "date": time.strftime("%Y-%m-%d"),
    }
    if args.out:
        Path(args.out).write_text(json.dumps(record, indent=2), encoding="utf-8")
        print(f"\nwritten to {args.out}")
    if args.labbook:
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download diff-eval-runs.py263 lines

The script takes two harness output directories, reports the score difference per metric, and then lists every setting that differs between them, ranked by how far that setting is known to move a score. It also compares the run-context.json files, which is where the fault the harness cannot see is recorded. With --samples it prints the first logged sample from each run, which is the prompt as the model actually received it.

It does not tell you which run is right. It removes the guessing from the list of candidates, which is the part of the job that wastes afternoons.

3. Fault one: the chat template

The largest lever, and the one you will meet most often. Produce a run with the template and a run without.

RunnableAll tracks

one run with the chat template, one without

LIMIT=40 RUNS=1 CHAT_TEMPLATE=yes QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=template-on \
    bash run-suite.sh local/chat ifeval
LIMIT=40 RUNS=1 CHAT_TEMPLATE=no QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=template-off \
    bash run-suite.sh local/chat ifeval

RunnableAll tracks

diagnose, with the prompts

python3 diff-eval-runs.py --a template-on --b template-off --samples --labbook labbook.md

Output — what you should see

== Settings that differ, most consequential first ==
  apply_chat_template
      A: true
      B: null
      Chat template applied or not. The largest single lever on an instruction-tuned model,
      and it changes every prompt in the run.

The evidence. The configuration difference, and the logged prompts: one carries role markers and a system turn, the other is bare text. The fix. Apply the template on both sides, because the model under test is instruction-tuned. The proof. The gap closes and the prompts become structurally identical.

✕Common mistake

Assuming the template is always right. It is right for an instruction-tuned model. Many standard tasks were written for base models as sentence-completion problems, and wrapping one of those in a chat template asks an assistant to converse about a fragment. The correct answer depends on the model and the task, and the fault is that the two runs differed, not that one setting is universally correct.

4. Fault two: the few-shot count and format

Switch to gsm8k for this one, because its task file fixes five worked examples and because a publisher quoting the same benchmark at eight shots with chain-of-thought prompting is the everyday version of this disagreement.

RunnableAll tracks

the task default against an explicit override

LIMIT=40 RUNS=1 QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=shots-default \
    bash run-suite.sh local/chat gsm8k
LIMIT=40 RUNS=1 NUM_FEWSHOT=0 QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=shots-zero \
    bash run-suite.sh local/chat gsm8k
python3 diff-eval-runs.py --a shots-default --b shots-zero --samples --labbook labbook.md

The evidence. num_fewshot differs in the configuration, and the logged prompts contain a different number of worked examples before the question. The fix. Match the shot count to whatever you are comparing against, and state it. The proof. The prompts contain the same number of examples and the gap closes.

There is a second half to this fault that costs nothing to see. --fewshot_as_multiturn restructures the same examples as alternating conversation turns rather than one block of text, and the harness records it as a separate setting. For a chat model it is usually the more natural shape, and it can move the score without changing the shot count at all.

Note also that gsm8k reports one score per extraction filter, strict-match and flexible-extract. Two people quoting “GSM8K” while reading different filters have a disagreement that no amount of settings-matching will resolve, because they are quoting different numbers from the same run.

5. Fault three: sampling against greedy decoding

RunnableAll tracks

a sampled run, twice, against a greedy run

LIMIT=40 RUNS=2 QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=sampled \
    GEN_KWARGS="temperature=0.8,do_sample=True" bash run-suite.sh local/chat ifeval
python3 summarise-results.py --results sampled --labbook labbook.md

GEN_KWARGS is passed straight to --gen_kwargs, which takes a key=value list in the same shape as --model_args; if your version of the harness rejects the separator, check lm_eval --help for the form it expects rather than guessing.

If your server applies its own default temperature rather than the task’s, the two runs in sampled/ will differ from each other. That self-disagreement is the diagnostic signature of this fault, and it distinguishes it from every other row in the table: a greedy configuration disagrees with a different configuration, while a sampled configuration disagrees with itself.

The evidence. The two runs of one configuration differ by more than the noise floor from step one, and gen_kwargs or the server’s own defaults show sampling enabled. The fix. Fix the sampling settings explicitly on the server or through --gen_kwargs, and record them. The proof. Two runs of the fixed configuration agree to within the noise floor.

!Important

Greedy is not automatically the right answer. The Qwen3-8B card advises the reader “DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions” and gives recommended temperatures for both of its modes. Evaluating that model greedily is running it at settings its publisher advises against, which is allowed and must be stated. The fault is an unrecorded difference, not a particular setting.

6. Fault four: contamination

This one has no configuration difference to find, which is what makes it dangerous: both runs are correct and the conclusion drawn from them is not.

The check is to compare the benchmark’s items with the data your model was trained on. The harness already logged every prompt it sent; Part 13’s decontaminate.py already knows how to compare two sets of text. All that is needed is to convert one shape into the other.

RunnableAll tracks

turn the logged samples into a task file the decontaminator reads

python3 - <<'PY'
import glob, json, pathlib

FIELD = "arguments"   # check the printed key list below and change this if your harness differs
path = sorted(glob.glob("template-on/**/samples_*.jsonl", recursive=True))[0]
lines = [json.loads(line) for line in pathlib.Path(path).read_text(encoding="utf-8").splitlines()]
print("fields in the first record:", ", ".join(sorted(lines[0])))

def text_of(record):
    value = record.get(FIELD, record.get("doc", record))
    return json.dumps(value) if not isinstance(value, str) else value

tasks = [{"id": f"b{i:03d}", "category": "benchmark", "prompt": text_of(row),
          "reference": "", "rubric": "not graded here; this file exists for the overlap check"}
         for i, row in enumerate(lines)]
pathlib.Path("benchmark-items.json").write_text(
    json.dumps({"name": "benchmark-items", "version": 1, "tasks": tasks}, indent=2),
    encoding="utf-8")
print(f"{len(tasks)} items written to benchmark-items.json")
PY

RunnableAll tracks

does your training data overlap the benchmark

python3 ~/labs/part-13-supervised-fine-tuning/decontaminate.py \
    --train data/train.jsonl \
    --tasks benchmark-items.json \
    --report contamination-report.json \
    --labbook labbook.md

The evidence. Exact matches or a high n-gram containment between your training examples and the benchmark’s prompts. The fix. Remove the overlapping training examples and retrain, or stop quoting that benchmark for this model. The proof. The check comes back clean, and the score either survives or does not — and a score that falls after decontamination is the most informative result on this page.

∴Research note

Exact and near-exact matching is a floor rather than a ceiling. The rephrased-samples research reports that “simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures”, and found substantial hidden overlap with HumanEval in widely used corpora. A clean report means you did not find contamination, not that there is none. The defence that holds is a benchmark whose problems postdate your model’s training.

7. Fault five: a different file behind the same alias

The fault that no amount of reading the harness output can find, because the harness never knew.

RunnableAll tracks

two runs against one alias, two different files

LIMIT=40 RUNS=1 QUANT=Q8_0 ENGINE=llama.cpp OUT_DIR=alias-q8 \
    bash run-suite.sh local/chat ifeval

Now restart the server with a different quantisation under the same alias, and run it again.

RunnableAll tracks

the second run, after the file behind the alias changed

LIMIT=40 RUNS=1 QUANT=Q4_K_M ENGINE=llama.cpp OUT_DIR=alias-q4 \
    bash run-suite.sh local/chat ifeval
python3 diff-eval-runs.py --a alias-q8 --b alias-q4 --labbook labbook.md

Output — what you should see

== Settings that differ, most consequential first ==
  (nothing)

== Outside the harness ==
  quant
      A: "Q8_0"
      B: "Q4_K_M"
      The quantisation actually served. The endpoint cannot report it, so it is recorded by
      hand and is therefore the field most often wrong.

The evidence. Every harness setting is identical and the numbers differ; the difference is in the hand-written context. The fix. Record the quantisation, the engine and its version for every run, which is what run-suite.sh writes for you, and re-run with the file you meant. The proof. The two runs agree once the same file is behind the alias in both.

Then run it once more with QUANT left unset, and read the warning the script prints. A run whose context file says unrecorded is a run you cannot defend, and the diagnostic script will tell you so rather than pretending the two runs were the same.

8. Find the fault you did not introduce

With the procedure fresh, take the two result directories from the previous lab — the reference model and one of your own — and run the diff over them.

RunnableAll tracks

the real comparison, run through the procedure

python3 diff-eval-runs.py --a suite-reference --b suite-finetune --samples --labbook labbook.md

They should differ in the context, because they are different models. What you are looking for is anything differing that should not: a shot count you did not intend to change, a template setting that moved between sessions, an engine version that was upgraded between the two runs. This is where people discover that half their comparison table was produced before an upgrade and half after.

Require a paired reproduction of the misleading score

Keep the healthy benchmark configuration and raw task outputs before introducing a fault. For every fault, change one dimension and give the run a new identifier. Compare rendered prompts, extracted answers and task-level verdicts before interpreting the aggregate score.

If changing the chat template alters the result, inspect the special-token and assistant-prefix differences. If few-shot settings changed, record which examples entered the prompt. If the model alias stayed the same, verify the file hash behind it. An unchanged friendly name does not prove the same checkpoint was evaluated.

Restore the baseline and repeat under the original protocol. The evidence should show both how the misleading number arose and why the corrected comparison answers a different, defensible question. For contamination, demonstrate the split failure with source identities rather than relying solely on an unusually high score. Finish with a reporting rule that prevents recurrence: include prompt configuration, immutable model identity, extraction method and task-level files. A diagnostic tool can highlight configuration differences; it cannot decide which protocol represents the intended application without the requirement you wrote before the experiment.

Validation

You are done when all of the following are true.

• The noise floor is measured and written down, and every gap you called a fault is larger than it.
• Five diagnoses exist in the notebook, one per fault, each naming the evidence, the single change and the second run that proved it.
• For the chat-template fault you can show two logged prompts side by side and point at the difference.
• For the sampling fault you can show a configuration disagreeing with itself across two runs.
• The contamination check has run against real training data and its report is saved, whether or not it found anything.
• For the alias fault, diff-eval-runs.py reported no harness difference and a context difference.
• You have run the procedure over two runs you did not stage, and either found nothing or written down what you found.

Expected outcome

A procedure you can run from memory, and a calibrated sense of which faults are large.

The shapes, none of which is a measurement and all of which your own runs will make concrete: the chat-template fault is the largest by a wide margin on an instruction-tuned model, large enough that it is usually visible without any tooling. Shot count is next and is more variable, because it depends on how much the task’s format needs demonstrating. Sampling produces a smaller shift with a distinctive signature, since it also moves between two runs of one configuration. A quantisation mismatch is usually small on a general benchmark and can be large on a narrow one, which is why the previous lab measured it directly. And contamination does not produce a gap at all: it produces a score that is too good, quietly, with nothing in the output to show for it.

Pending validationFive faults, one benchmark — your recording sheet

Fault	Metric used	Run A	Run B	Gap	After the fix	Evidence that identified it
Noise floor: same configuration twice	—	—	—	—	n/a	n/a
Chat template on or off	—	—	—	—	—	—
Few-shot count	—	—	—	—	—	—
Sampling against greedy	—	—	—	—	—	—
Contamination	—	—	—	no gap by construction	—	—
Different file behind one alias	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · the engine behind the alias, per row engine version and harness version, recorded per row · the model you served for this page, as recorded in each run-context.json · 4,096 tokens of context · the date you ran it

Empty on purpose. The first row is not a fault: it is the resolution of every row beneath it. The contamination row has no gap column by design, which is the point of including it: the failure it describes leaves the numbers looking healthy.

=Expected result

After each fix, the gap should fall to within the noise floor and the evidence that identified the fault should have changed: the configuration field, the logged prompt, or the context file. A gap that closes without the evidence changing means something else moved, and you have not finished.

Troubleshooting

The two runs have no metric in common. They did not measure the same thing. Check the task names in each run-context.json; a typo in a task name produces a run of a different benchmark rather than an error.

diff-eval-runs.py reports no difference at all and the scores still differ. Three candidates remain, in order. The gap is inside the noise floor and is not a difference. Something outside both the harness and the context file changed, most often the engine version or which file the gateway loaded. Or the two runs were made against a server that was busy in one case and idle in the other, which changes batching and therefore arithmetic.

No samples_*.jsonl files. The run was made without --log_samples. Re-run with it; the prompt as sent is not recoverable afterwards, and without it the chat-template and shot-count faults cannot be diagnosed from evidence, only guessed at.

The sample records do not have an arguments field. The field names in the logged samples depend on the harness version. The snippet in task six prints the keys of the first record for exactly this reason; set FIELD to whichever one holds the prompt.

decontaminate.py reports thousands of near-duplicates. The default n-gram threshold is deliberately loose. Raise --threshold, or read a handful of the reported pairs first: a benchmark whose items all share a long instruction preamble will match everything on a short n-gram and mean nothing by it.

The contamination check finds overlap and you cannot retrain. Then the benchmark is no longer evidence for this model, and the correct action is to stop quoting it rather than to adjust the number. Say so in the report; it is a more useful sentence than any score.

Cleanup

Keep every result directory and the notebook entries; they are the worked examples you will compare against the next time somebody sends you a number.

RunnableAll tracks

remove the deliberately broken runs, once they are written up

ls -d template-off shots-zero alias-q4
rm -rf template-off shots-zero alias-q4

⚠Warning

That command removes directories. List them first, as the block does, and keep the ones you have not written up: the diagnoses are worth more than the disk they occupy, and re-staging a fault costs another twenty minutes.

Restart the server with the file you actually intend to serve, and make sure the alias points at it.

What you learned

• Evidence before theory. The harness logs its own configuration and every prompt it sent. Both were available before you had a hypothesis, and reading them is faster than guessing.
• The noise floor comes first. A gap smaller than the spread between two runs of one configuration is not a fault, and chasing it is how afternoons disappear.
• Five faults cover almost everything. Chat template, shot count and format, sampling, contamination, and a different file behind the same name.
• Sampling has a signature. It is the only fault where a configuration disagrees with itself.
• The harness cannot see the model. It sees an alias. The quantisation, the engine and the version are recorded by hand or not at all, which is why an unrecorded run is undefendable.
• Contamination produces no gap. It produces a score that is too good with nothing in the output to show for it, and it is found by comparing items with training data rather than by comparing runs.
• A fix proves itself twice. The gap closes and the evidence changes. Only one of those is not enough.

Record in the notebook: the noise floor, the five diagnoses with their evidence and proofs, the contamination report whatever it said, and anything the procedure found in the two runs you did not stage. Part 17 measures speculative decoding against a baseline, and the baseline is only worth having if you can defend how it was produced.

Check your understanding

Question 1. Two evaluation runs disagree. What should you measure before forming any theory?

• The model's perplexity
• The noise floor: run one configuration twice and see how far it moves on its own
• The server's memory use
• The benchmark's publication date

Show the answer and why

Answer: The noise floor: run one configuration twice and see how far it moves on its own

A gap smaller than the spread between two runs of one configuration is not a fault. Establishing the resolution of the measurement before interpreting differences is the same discipline the quantisation lab applies, and for the same reason.

Question 2. Which fault has the distinctive signature of a configuration disagreeing with itself?

• The chat template
• Sampling instead of greedy decoding
• A different few-shot count
• A different file behind the alias

Show the answer and why

Answer: Sampling instead of greedy decoding

Every other fault makes one configuration disagree with a different configuration. Sampling makes a single configuration produce different numbers on successive runs, which is why two runs of the suspect configuration is the cheapest test for it.

Question 3. diff-eval-runs.py reports that every harness setting is identical, and the scores still differ by more than the noise floor. Where do you look next?

• At the benchmark's dataset
• Outside the harness: which file the endpoint had loaded, which engine served it and at which version, none of which the harness records
• At the random seed
• Nowhere; the difference is the model

Show the answer and why

Answer: Outside the harness: which file the endpoint had loaded, which engine served it and at which version, none of which the harness records

An endpoint reports an alias. run-suite.sh writes run-context.json for exactly this reason, and a run whose context says "unrecorded" cannot be defended against this question at all.

Question 4. Why does contamination not show up as a gap between two runs?

• Because it affects both runs equally, so the numbers agree and are both too high; it is found by comparing the benchmark items with the training data, not by comparing runs
• Because the harness removes contaminated items automatically
• Because contaminated items are skipped
• Because it only affects base models

Show the answer and why

Answer: Because it affects both runs equally, so the numbers agree and are both too high; it is found by comparing the benchmark items with the training data, not by comparing runs

Both runs measure the same contaminated model on the same contaminated benchmark, so they agree. The failure is in the conclusion rather than in the measurement, and exact matching is only a floor: paraphrases survive it, as the rephrased-samples research reports.

Question 5. You fix a fault, the gap closes, and the configuration field you suspected is unchanged. What follows?

• The fix worked and the investigation is complete
• The gap closed for a reason you have not identified, so the diagnosis is unproven and something else moved
• The diff tool is broken
• The noise floor was measured incorrectly

Show the answer and why

Answer: The gap closed for a reason you have not identified, so the diagnosis is unproven and something else moved

A fix proves itself twice: the number recovers and the evidence that identified the fault changes. A number that recovers with the evidence unchanged usually means the machine was busy during one of the runs, or that the fault was inside the noise floor all along.

Check my answersClear