LLM-as-Judge, Contamination and Honest Reporting

Part 10 gave you a judge and told you to measure it before believing it. This lesson explains what you are measuring and why, adds the contamination problem that makes a good score less informative than it looks, and ends with the report format that turns a number into evidence.

By the end you will be able to write a judge prompt that grades against a rubric rather than rewarding length, name the four biases the research identifies and say how to measure two of them on your own setup, explain why string-matching decontamination fails, recognise a benchmark built to resist contamination, and write a result down so that someone else could repeat it.

What a judge is for, and what it is not

A judge model is the answer to a specific problem: most of the work you want a model to do has no single correct string. “Summarise this bug report for someone who was not there” cannot be graded by exact match, and grading it by hand does not scale past a few dozen items.

It is not the answer to problems a program can decide. If the requirement is that the output parses as JSON, that a number is 50, that a function passes its tests, or that a response is under forty words, write the check. Part 10’s run-eval.py scores those with must_contain, must_not_contain and max_words before any model is involved, and those checks are exactly repeatable, free, and immune to every bias in this lesson. Reach for a judge only for what is left.

The judge prompt

A judge prompt has four jobs, and Part 10’s judge.py does each of them explicitly.

It supplies a reference and a rubric. A judge asked “is this a good answer” grades against its own taste. A judge given the rubric and a reference answer grades against yours.

It fixes the scale, with anchors. A bare “score out of ten” produces sevens and eights. A five-point scale with a written description of each point — meets the rubric completely, meets it with one small omission, one required element missing, mostly fails, contradicts the reference — produces a distribution you can read.

It forbids the known failure modes in words. “Length is not quality”; “do not reward confidence, formatting flourishes or politeness”. This does not eliminate the biases, but it reduces them, and it costs one line.

It demands a machine-readable verdict. Ask for JSON against a schema, at temperature zero, with a fixed seed. Part 10’s harness uses the response_format support Part 10’s structured-output lesson introduced, so a judge that rambles produces a parse error rather than a silently wrong score.

The four biases, and which two you can measure

The MT-Bench paper is the standard reference. It examines “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” in LLM judges, and reports that “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans”.

Position bias is a preference for the answer shown first, or second, independent of its content. It is measurable, cheaply, and Part 10’s judge.py compare does it: ask the judge twice with the order swapped, and count how often the verdict flips. The flip rate is a property of the judge and the task, not of either model, and a high one means the comparison is not deciding anything.

Verbosity bias is a preference for longer answers. Also measurable: judge.py grade records the mean answer length at each score. If the answers scored five are systematically much longer than the answers scored two, the judge may be measuring length. Read that alongside the human-agreement number before concluding either way, because on some tasks the better answers genuinely are longer.

Self-enhancement bias is a model preferring its own outputs. The mitigation is structural: never judge with the model under test. Part 10’s harness prints a warning when the judge model and the model being graded are the same, because the resulting number is optimistic in a way no amount of prompting fixes.

Limited reasoning is the one you cannot prompt away. A judge cannot reliably verify arithmetic it cannot do, or a proof it cannot follow, or code it cannot execute. Where a program can decide, the program decides; where it cannot, the judge’s verdict on a hard reasoning item is worth less than its verdict on a summary.

!Important

Agreement with your own labels is the number that licenses everything else. A judge’s scores are a measuring instrument, and an uncalibrated instrument produces confident nonsense. Part 10’s judge.py human mode has you score a sample yourself with the judge’s verdict hidden, then reports exact agreement, agreement within one point, and the items where you and it disagreed by two or more. Read those disagreements: each one is either a rubric that says less than you meant or a category this judge cannot grade, and both are fixable. The published figure of over 80 per cent agreement is for a frontier model on that paper’s tasks; your local judge on your tasks is a separate measurement.

Contamination, and why decontamination is harder than it looks

Part 4 introduced contamination as benchmark questions appearing in training data. This lesson adds the part that makes it a live problem rather than a solved one.

The standard defence is to search the training corpus for exact or near-exact matches of the benchmark’s text and remove them. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples shows that this is not enough: “simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures”. The paper reports that with such variations present, “a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4”, and that on inspection “8-18% of the HumanEval benchmark overlaps” in datasets including RedPajama-Data-1T and StarCoder-Data. Its recommendations are to adopt stronger, model-based decontamination and, more bluntly, for the community “to actively develop fresh one-time exams to evaluate models accurately”.

The structural defence that survives this is time. If a benchmark’s problems carry release dates, a model can be scored only on problems that did not exist when it was trained, and no amount of paraphrasing can put a future problem into a past corpus.

Why a date-stamped benchmark is the defence that holds

Model training corpus

Text scraped up to the cut-offNothing after it

Benchmark problems

Published before the cut-off: possibly memorisedPublished after: genuinely unseen

What you should score

Ignore this windowScore only here

EarlierTraining cut-offLater

LiveCodeBench annotates every problem with its release date so that a model with training cut-off D can be scored only on problems released after D. The site reports models whose scores fall sharply on the later window, which is what contamination looks like when the benchmark is built to reveal it.

Your own contamination

The version that will actually bite you is not a public benchmark’s. It is your own evaluation set, leaking into your own pipeline, in one of four ways.

Your fine-tuning data overlaps your evaluation set, which Part 13’s decontaminate.py exists to catch. Your distillation run generated training data from a teacher using prompts that are also in your evaluation set, which is Part 15’s version of the same mistake. Your importance matrix was calibrated on the text you evaluate on, which the first lesson of this part warned about. Or you have iterated: you looked at the evaluation results, changed the model, looked again, and repeated twenty times, which turns a held-out set into a training set one decision at a time.

The last one has no tool. The defence is to keep a second set that you look at rarely — once at the start and once at the end — and to treat a gap between your working set and your held-out set as evidence that you have been fitting the working set.

Reporting: variance and settings

A single number from a single run is not a result, and there are two different reasons why.

Sampling error over items. A benchmark of two hundred questions resolves to half a percentage point per item, by arithmetic: one item is one two-hundredth. Two models differing by one item are not distinguishable, and neither are two quantisations. A benchmark run with --limit 50 is coarser still. This is why the previous lesson said to report full runs, and why “our quant scored one point higher” is usually a statement about which items happened to be in the set.

Run-to-run variation. Under sampling, two runs of the same model on the same items give different answers, so a reported score should come from repeated runs with different seeds, with the spread reported alongside the mean. Under greedy decoding, variation should be small, and Part 6’s lesson on sampling and determinism explained why it is not zero: batching, backend kernels and server concurrency can change floating-point summation order and flip a close decision.

The practical minimum is three runs, the mean and the range. Three runs is not statistics; it is enough to tell “these two files differ” from “this measurement wobbles by more than the difference I am looking at”.

∴Research note

Establish the noise floor before you interpret anything. Run the same model, at the same settings, three times, and look at the spread. Any difference between two candidates that is smaller than that spread is not a difference you have measured. Part 13’s challenge does this for fine-tuning and the labs in this part do it for quantisation, because it is the single cheapest defence against concluding something from nothing.

The table

Every benchmark result this course produces goes into one shape, and the shape is designed so that the settings are impossible to omit.

Pending validationBenchmark report — the course format, filled in by the second lab of this part

Model	Quantisation	Task	Metric	Shots	Chat template	Score	Range over 3 runs
Reference model	as served	ifeval	prompt_level_strict_acc	0	applied	—	—
Reference model	as served	ifeval	inst_level_loose_acc	0	applied	—	—
Your Part 13 fine-tune	as served	ifeval	prompt_level_strict_acc	0	applied	—	—
Your Part 15 student	as served	ifeval	prompt_level_strict_acc	0	applied	—	—

your machine: track, chip and memory, your operating system and version · the engine behind the gateway alias you evaluated engine version, and the harness version from its output · as listed in the first column, as listed in the second column · 4,096 tokens of context · the date you ran it

Empty on purpose. Two rows for the same task and model on purpose too: one benchmark produces several metrics, and choosing one to quote is a decision that has to be visible. The range column is the noise floor; a difference between rows that is smaller than it has not been measured.

Alongside the table, the report needs the sampling settings, and this is where a model card earns its keep. The Qwen3-8B card recommends temperature 0.6, top-p 0.95, top-k 20 and min-p 0 for thinking mode, and temperature 0.7, top-p 0.8, top-k 20 and min-p 0 for non-thinking mode, with the warning to “DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions”. A run of that model at temperature zero is therefore a run at settings its publisher advises against, which is a legitimate thing to do and an illegitimate thing to leave unstated.

✕Common mistake

Reporting a score without saying whether thinking mode was on. For a model with a thinking mode this is the largest single setting in the report, larger than the quantisation, and it also changes how long the run takes and how many tokens it costs. It belongs in the table’s context, next to the temperature.

The reporting checklist

A result that a stranger could repeat carries all of this, and it fits in a paragraph:

• the model, by its full identifier, and the exact file or gateway alias served;
• the quantisation, by name, and where the file came from;
• the engine and its version, and the harness and its version;
• the task, by its exact name in the harness, and the metric, by its exact name;
• the shot count, whether a chat template was applied, and whether few-shot examples were multi-turn;
• the sampling settings, including whether thinking mode was enabled;
• how many items were scored, and whether any limit was applied;
• how many runs, and the spread across them;
• the date.

If that feels like a lot for one number, it is the correct amount, and it is why the course puts numbers in a component that refuses to render without their context.

Separate judge reliability from model quality

Choose a small calibration set whose answers you grade using the same rubric. Compare the judge’s decisions with yours, inspect disagreements and reverse answer order in paired comparisons. A stable preference that flips when answer order changes reveals a measurement problem before you use the judge to select a model.

Report agreement and task quality as separate quantities. A judge can agree with a human on easy cases while missing a rare but serious factual error. Include those cases deliberately and use deterministic verification where available. Do not select the judge only because it prefers the model you hoped would win.

For uncertainty, distinguish variation across tasks from variation across repeated generations of the same task. Repeating one prompt many times estimates sampling behaviour for that prompt; it does not establish broad domain coverage. Keep related examples grouped when estimating uncertainty. State the evaluation set, repetition policy and scoring failures in the report so a reader can tell whether a small apparent improvement is supported by independent evidence.

Recap

A judge grades what a program cannot; anything mechanically checkable should be checked mechanically first. A good judge prompt carries a reference and a rubric, an anchored scale, an explicit ban on rewarding length and confidence, and a schema for the verdict. The research names position, verbosity, self-enhancement and limited reasoning as the judge’s failure modes; the first two are measurable by swapping the order and by recording length against score, the third is avoided by never judging with the model under test, and the fourth is a boundary on what a judge can be asked. The number that licenses a judge is its agreement with your own labels on a sample. Contamination survives string-matching decontamination because paraphrases do, so the defences that hold are date-stamped problems, execution-graded tasks and a held-out set you rarely look at. And a reported score carries its items, its runs and their spread, its settings, its versions and its date, or it is not a result.

Check your understanding

Question 1. You ask a judge to compare two models' answers and it prefers model A. What should you do before believing it?

• Ask a second judge model
• Ask again with the two answers in the opposite order and count how often the verdict flips, because position bias is a documented failure mode and the flip rate is a property of the judge
• Increase the temperature
• Use a longer rubric

Show the answer and why

Answer: Ask again with the two answers in the opposite order and count how often the verdict flips, because position bias is a documented failure mode and the flip rate is a property of the judge

The MT-Bench paper names position bias explicitly. Part 10's judge.py compare mode asks both ways by construction and reports a flip rate, so the bias appears as a number rather than hiding inside the verdict.

Question 2. Why is it a mistake to grade a model's answers with the same model as the judge?

• It is slower
• Self-enhancement bias: a model scores its own outputs generously, and no amount of prompting removes it
• The judge will run out of context
• It produces invalid JSON

Show the answer and why

Answer: Self-enhancement bias: a model scores its own outputs generously, and no amount of prompting removes it

Self-enhancement is one of the four biases the research identifies, and it is structural rather than promptable. Part 10's harness prints a warning when the judge and the model under test are the same for exactly this reason.

Question 3. A team removes every exact match of a benchmark's questions from its training corpus and declares the benchmark clean. What is the flaw?

• Exact matching is too slow to run at scale
• Paraphrases, translations and other simple variations survive exact matching, and a model trained on them can overfit the benchmark while appearing decontaminated
• Benchmarks cannot be decontaminated at all
• The training corpus is too large to search

Show the answer and why

Answer: Paraphrases, translations and other simple variations survive exact matching, and a model trained on them can overfit the benchmark while appearing decontaminated

This is the finding of the rephrased-samples paper: simple variations bypass such measures, and it reports substantial hidden overlap with HumanEval in widely used corpora. Stronger model-based detection helps; date-stamped problems remove the possibility entirely.

Question 4. Two quantisations of your model score 71 and 72 on a 200-item benchmark, from one run each. What is the sound conclusion?

• The second quantisation is better
• One item is half a percentage point on 200 items, and no noise floor has been established, so nothing has been measured; repeat each run three times and compare the difference with the spread
• The benchmark is broken
• The difference is significant because the benchmark is standard

Show the answer and why

Answer: One item is half a percentage point on 200 items, and no noise floor has been established, so nothing has been measured; repeat each run three times and compare the difference with the spread

A one-point difference on 200 items is two items. Without repeated runs there is no estimate of how much the measurement wobbles on its own, and a difference smaller than that wobble is not evidence. Establishing the noise floor first is the cheapest defence against this error.

Question 5. Which of these belong in a benchmark report that someone else could repeat? Select all that apply.

• The exact harness task name and the exact metric name
• Whether a chat template was applied and how many few-shot examples were used
• Whether thinking mode was enabled, and the sampling settings
• The number of runs and the spread across them

Show the answer and why

Answer: The exact harness task name and the exact metric name, Whether a chat template was applied and how many few-shot examples were used, Whether thinking mode was enabled, and the sampling settings, The number of runs and the spread across them

All four, plus the model identifier, the quantisation and its source, the engine and harness versions, the number of items scored and the date. A benchmark whose task variant and metric are unstated is ambiguous by several points before anything else is considered.

Check my answersClear