Evaluation Harnesses: lm-evaluation-harness, lighteval, EvalPlus and Your Own

Part 10 had you build a task set of your own, and it remains the measurement that decides whether a model is useful to you. This lesson is about the other kind: the standard suites, run properly, so that a number you produce can be set beside a number somebody else produced.

By the end you will be able to install and drive an evaluation harness against a local server, read a task definition and predict what will change the score, name what each of the common benchmarks actually measures, and state the two settings that account for most of the irreproducible benchmark numbers on the internet.

What a harness owns

You could evaluate a benchmark with a for-loop and a string comparison. What you would be quietly deciding, without writing it down, is everything that makes the number what it is.

How the question is turned into a prompt. Whether examples are shown first, and how many, and in what format. Whether the chat template is applied. Whether the model generates text or is scored on the likelihood it assigns to each candidate answer. Where generation is stopped. How the answer is extracted from what came out. How partial credit is handled. And how the per-item results are aggregated.

A harness owns all of those and writes them into a file. That is its entire value: not that it runs the benchmark, but that the way it runs the benchmark is inspectable, versioned and identical for everyone who uses the same version. When two people’s numbers disagree, the harness’s configuration is the first place to look, which is what the challenge at the end of this part is built around.

lm-evaluation-harness

lm-evaluation-harness 0.4.13 · verified 2026-09-08 is the one this course uses. Its README’s install path is a clone and an editable install, with backends installed as extras.

RunnableAll tracks

install the harness with uv, in its own environment

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness ~/lm-evaluation-harness
uv venv ~/lm-evaluation-harness/.venv
source ~/lm-evaluation-harness/.venv/bin/activate
uv pip install -e ~/lm-evaluation-harness
uv pip install "lm_eval[api]"

The [api] extra is the one that matters for this course, because it brings the backends that talk to a server rather than loading the weights themselves.

Backends

The --model flag chooses how the harness reaches the model, and the choice has consequences beyond convenience.

hf loads the checkpoint with transformers in the harness’s own process. The README’s example is lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8. This backend can compute log-probabilities of arbitrary strings, which some task types need, and it needs enough memory to hold the model.

vllm runs the model through vLLM in-process, which is much faster for large task sets on the tracks where vLLM runs.

local-completions and local-chat-completions talk to any OpenAI-compatible server over HTTP. The README’s example is lm_eval --model local-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1/completions. The constructor accepts model, base_url, tokenizer, tokenizer_backend, num_concurrent, max_retries and tokenized_requests among others.

The HTTP backends are what make this part work on all four tracks: the harness runs on your laptop, the model runs behind your Part 9 gateway on whatever engine your track uses, and nothing about the harness needs to know which.

Where a benchmark number comes from

from the task file to the weights

1. Task configurationA YAML file: which dataset, how the prompt is built, how many examples, how the answer is extracted, which metrics. Versioned with the harness.
2. Run settings--num_fewshot, --apply_chat_template, --fewshot_as_multiturn, --gen_kwargs, --limit, --seed. These override the task file and are the usual cause of two runs disagreeing.
3. Backendhf, vllm, or an HTTP client against an OpenAI-compatible server. Decides which request types are possible and how sampling is applied.
4. The serverllama-server, vLLM, mlx-lm or the gateway. Applies its own chat template and its own default sampling unless told otherwise.
5. The engine and its versionSampler implementations and template handling change between releases. Record the version.
6. The weights, at a quantisationThe thing you meant to measure. It is the last layer, not the first, and by itself it explains less of the variation than the two accented layers above.

Every layer can change the number. The top three are recorded in the harness's output; the bottom three are recorded only if you write them down, which is why the lab's report template asks for them.

What a task actually says

The harness’s task guide describes tasks as YAML with a small set of fields: task names it, dataset_path and dataset_name say where the data comes from, doc_to_text is “a Jinja2 template, string, or function to process a sample into the appropriate input”, doc_to_target does the same for the expected output, doc_to_choice produces “a list of possible string choices” for multiple choice, num_fewshot sets the in-context examples, metric_list names the scoring, and filter_list post-processes the model’s output.

The field that changes everything is output_type, which has four values. loglikelihood scores the probability the model assigns to a given continuation. multiple_choice scores each option that way and picks the highest, so the model never generates a single token of free text. generate_until has the model actually write an answer, stopping at a given string. loglikelihood_rolling scores a whole document, which is how perplexity-style tasks are built.

!Important

A multiple_choice task and a generate_until task are different experiments on the same questions. The first asks which of these four strings the model finds most probable, and is unaffected by temperature because nothing is sampled. The second asks the model to answer, and is affected by temperature, by stop strings, by the maximum token count and by whatever regular expression pulls the answer out. Two published “MMLU” numbers measured these two ways are not comparable, and neither one is wrong.

Two real task files make this concrete. The gsm8k configuration sets num_fewshot: 5, an output_type of generate_until, a doc_to_text of "Question: {{question}}\nAnswer:", generation with do_sample: false and stop strings including "Question:", and — this is the part people miss — a filter_list with two entries named strict-match and flexible-extract. The task therefore reports two numbers per run, one for each way of pulling the final figure out of the model’s working. A published “GSM8K score” that does not say which filter it used is ambiguous by several points.

The ifeval configuration uses the google/IFEval dataset, generate_until with a temperature of 0.0 and up to 1280 new tokens, no few-shot examples, and four metrics: prompt_level_strict_acc, inst_level_strict_acc, prompt_level_loose_acc and inst_level_loose_acc. Prompt-level accuracy demands that every instruction in a prompt is satisfied; instruction-level counts each one. Strict and loose differ in how forgiving the checker is about wrappers such as surrounding quotation marks. So an “IFEval” figure is one of four numbers, and they are not close together.

The flags that decide the number

The documented command-line interface is longer than this, but six flags account for most disputes.

Flag	What it does
--num_fewshot	“Number of few-shot examples in context”. Overrides the task file’s default.
--apply_chat_template	“Apply chat template to prompts. Use without argument for default template”.
--fewshot_as_multiturn	“Format few-shot examples as multi-turn conversation”.
--gen_kwargs	“Generation arguments as key=val key2=val2”, including temperature and top-p.
--limit	“Limit examples per task. Integer for count, float (0.0-1.0) for percentage”.
--log_samples	“Save all model inputs/outputs for post-hoc analysis”.

--log_samples is the one to make a habit of. It writes out every prompt as sent and every response as received, which is the difference between “the number is wrong” and “here is the prompt that produced the wrong answer”. The challenge at the end of this part is unsolvable without it.

✦Tip

Use --limit while you are getting a run working and never in a number you report. A hundred-item subset of a five-hundred-item benchmark has real sampling error, and a subset chosen by taking the first hundred is not even a random sample. Report full runs, or report the limit as part of the result.

Chat templates and few-shot: the two settings that break comparability

An instruction-tuned model was trained to see a specific scaffolding around every message: a system turn, role markers, an end-of-turn token. Part 10’s lesson on chat templates established that serving one without its template degrades it. Evaluating one without its template does the same thing, silently, and produces a number that looks like a model result and is a formatting result.

The reverse mistake also exists. Many standard tasks were designed for base models as sentence-completion problems: the prompt ends mid-sentence and the model continues it. Wrap that in a chat template and you have asked an assistant to converse about a fragment, which it will do, politely, and score badly for.

Few-shot count interacts with this. --num_fewshot 5 on a generate_until task puts five worked examples in front of the question, which teaches the model the answer format as much as the task. --fewshot_as_multiturn restructures those examples as alternating conversation turns instead of one block of text, which for a chat model is usually the more natural shape. Both flags change scores substantially and neither is visible in a number quoted without them.

✕Common mistake

Assuming a published number used the harness’s defaults. Model publishers evaluate with their own prompts, their own shot counts and often their own extraction code, and they report the number their setup produced. Reproducing it means matching their settings, which means finding out what they were, and frequently they are not published. That is not a reason to distrust the number; it is a reason to treat it as theirs, and to compare it only with numbers produced the same way.

lighteval

lighteval is Hugging Face’s harness and the second one worth knowing. Its documentation describes it as evaluating “across multiple backends”, and the command names are the backends: lighteval eval using inspect-ai, lighteval accelerate, lighteval vllm, lighteval sglang, lighteval nanotron, lighteval custom, and lighteval endpoint with sub-commands including inference-endpoint, tgi, litellm and inference-providers. Its basic invocation is a model specification and a task:

RunnableAll tracks

lighteval, one task, one model

lighteval accelerate \
    "model_name=openai-community/gpt2" \
    truthfulqa:mc

Three things it does well are worth borrowing even if you stay with the Eleuther harness. It has a task-discovery command — lighteval tasks list, lighteval tasks inspect and lighteval tasks create — so finding out what a task does is a command rather than a repository search. Its endpoint litellm backend reaches anything LiteLLM reaches, which is exactly the gateway you built in Part 9. And it has --remove-reasoning-tags with --reasoning-tags to strip a reasoning model’s thinking trace before the metric is computed, which is a real problem the moment you evaluate a thinking model and its chain of thought lands in the answer field.

The benchmarks, and what each one measures

IFEval tests instruction following with instructions a program can check. Its paper describes “a set of verifiable instructions such as write in more than 400 words and mention the keyword of AI at least 3 times”, and reports identifying “25 types of those verifiable instructions” across “around 500 prompts, with each prompt containing one or more verifiable instructions”. Because the checking is mechanical there is no judge and no ambiguity, which makes it the most reproducible thing on this list and the best first benchmark for a fine-tuned model.

MMLU-Pro is knowledge and reasoning as multiple choice, rebuilt to be harder than MMLU. Its paper describes integrating “more challenging, reasoning-focused questions” and expanding “the choice set from four to ten options”, reports “a significant drop in accuracy by 16% to 33% compared to MMLU”, and reports that prompt sensitivity fell “from 4-5% in MMLU to just 2% in MMLU-Pro”. It also reports that chain-of-thought prompting helps on MMLU-Pro where it did not on MMLU. In the harness it appears as fourteen subject tasks, from mmlu_pro_biology to mmlu_pro_psychology.

GPQA is graduate-level science. Its paper describes 448 multiple-choice questions in biology, physics and chemistry, and gives the baseline that makes it interesting: “experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy”, while “highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web”. The harness ships it as a family of variants — gpqa_diamond_zeroshot, gpqa_diamond_cot_zeroshot, gpqa_diamond_n_shot, gpqa_diamond_cot_n_shot, gpqa_diamond_generative_n_shot and the main and extended equivalents — and the variant you pick changes the result more than most model differences do. Quoting “GPQA” without the variant is the same error as quoting “GSM8K” without the filter.

EvalPlus is code, graded by running it. Its README describes “HumanEval+: 80x more tests than the original HumanEval” and “MBPP+: 35x more tests than the original MBPP”, and gives the interpretation of the gap between the original and the plus version: “less drop means more rigorousness in code generation; while a bigger drop means the generated code tends to be fragile”. That difference, rather than either number alone, is the useful signal.

LiveCodeBench is competitive programming built against contamination. The project annotates every problem with its release date so that “for a newer model with a training-cutoff date D, we can evaluate it on problems released after D to measure its generalization on unseen problems”, and evaluates “code generation, self-repair, test output prediction, and code execution”. Its original set is described as “over three hundred high-quality coding problems published between May 2023 and February 2024”, and the site reports models whose scores drop sharply on problems released after their own cut-off, which is contamination made visible.

Aider polyglot measures editing rather than writing: “225 challenging Exercism coding exercises” across C++, Go, Java, JavaScript, Python and Rust, with two figures per model, the percentage solved and the percentage that used the correct edit format. For a local model driving a real editor, that second figure is often the binding constraint. The leaderboard carries its own last-updated date, which at the time of writing reads 20 November 2025.

⛨Security

EvalPlus and LiveCodeBench grade by executing code a model wrote, and the harness has a flag, --confirm_run_unsafe_code, whose documented purpose is to “confirm understanding of risks for tasks executing arbitrary Python”. Treat that flag as what it is. Run code benchmarks inside a container or a dedicated user with no access to your home directory, your keys or your network, exactly as Part 10’s structured-output work and the agent parts of this course insist on. A generated program that deletes a directory is not a hypothetical failure mode of a coding benchmark; it is the ordinary one.

When to write your own instead

Standard suites answer “how does this model compare with other models on tasks the field agreed to care about”. Your task set answers “does this model do my job”. They are both worth having and they are not interchangeable.

Write a harness task when you want a public number, when you need the same prompt construction and extraction that others use, or when your task genuinely resembles an existing one. The task guide’s YAML is a short file: a dataset path, a template for the prompt, a template for the target, an output type and a metric.

Stay with Part 10’s harness when the thing you are measuring is subjective, when it needs a rubric and a judge, when the tasks come from your own work and cannot be published, or when you want the deterministic string checks that catch a format regression in one line of configuration. The two sit side by side in the lab that follows.

Inspect one rendered example before running a suite

A harness controls prompt construction, answer extraction and scoring. Before committing compute, run a tiny subset and inspect the exact prompt, expected answer, model output and extracted answer. A wrong separator, few-shot setting or chat template can invalidate an otherwise correctly executed benchmark.

Distinguish likelihood-based scoring from generated-answer scoring. A multiple-choice task can compare candidate likelihoods without asking the model to emit a letter; a generation task may require parsing the final answer from text. These are different protocols, and their scores cannot be interchanged merely because the task name is similar.

Record the harness revision, task configuration, dataset revision, few-shot seed, model settings and any limits. A limited subset is a smoke result; label it as such rather than comparing it with a published full-suite score. Preserve task-level outputs so you can investigate extraction failures and recompute summaries. If a benchmark executes generated code, use the isolation required by its evaluation path. The harness is executable measurement software and needs the same scrutiny as the model it evaluates.

Recap

A harness’s value is that it writes down the prompt construction, the shot count, the stopping rule, the extraction and the aggregation, so that a number is inspectable rather than merely produced. lm-evaluation-harness defines tasks in YAML around an output_type that decides whether the model generates or is scored on likelihoods, and drives them through hf, vllm or HTTP backends that speak the OpenAI API, which is how it reaches your gateway. Its task files carry the real settings: gsm8k is five-shot with two extraction filters and therefore two scores, ifeval is zero-shot with four metrics. --apply_chat_template, --fewshot_as_multiturn and --num_fewshot move scores more than most model changes do, and --log_samples is what makes a disagreement diagnosable. lighteval adds task discovery, a LiteLLM backend and reasoning-tag stripping. IFEval checks mechanically verifiable instructions; MMLU-Pro is ten-option knowledge and reasoning; GPQA is graduate science with a published human baseline and a family of harness variants that are not interchangeable; EvalPlus and LiveCodeBench execute code, which means a sandbox; Aider polyglot measures editing and edit-format compliance. And a standard suite never replaces the twenty tasks that are actually yours.

Check your understanding

Question 1. A task has output_type: multiple_choice. What follows about the sampling temperature?

• A lower temperature will produce a higher score
• It has no effect, because the task scores the likelihood the model assigns to each option and never samples any text
• It must be set to 1.0
• The temperature is read from the task file

Show the answer and why

Answer: It has no effect, because the task scores the likelihood the model assigns to each option and never samples any text

multiple_choice tasks compute log-probabilities of the candidate answers and choose the largest. Nothing is generated, so sampling settings are irrelevant. On a generate_until task the same settings matter a great deal, which is why the output type is the first field to read.

Question 2. Someone reports "GSM8K: 78" for a model. What is missing?

• Nothing; GSM8K has one score
• Which extraction filter produced it, since the harness task defines both strict-match and flexible-extract, plus the shot count and whether a chat template was applied
• The model size
• The hardware it ran on

Show the answer and why

Answer: Which extraction filter produced it, since the harness task defines both strict-match and flexible-extract, plus the shot count and whether a chat template was applied

The gsm8k task file defines a filter_list with two entries, so one run reports two numbers that can differ by several points. Its default is five-shot with greedy decoding; a publisher using eight-shot chain-of-thought prompting is measuring something else again.

Question 3. You are evaluating an instruction-tuned model on a generative task and get a very low score. Which setting should you check first?

• The batch size
• Whether the chat template was applied, since an instruction-tuned model shown a raw completion-style prompt answers as if the scaffolding it was trained on were absent
• The random seed
• The number of GPUs

Show the answer and why

Answer: Whether the chat template was applied, since an instruction-tuned model shown a raw completion-style prompt answers as if the scaffolding it was trained on were absent

Chat template applied or not is the single largest source of surprising evaluation results on instruction-tuned models, and the mistake runs both ways: applying a template to a task designed as sentence completion for base models is equally destructive.

Question 4. Which of these benchmarks grade by executing the model's output? Select all that apply.

• EvalPlus (HumanEval+ and MBPP+)
• IFEval
• LiveCodeBench
• GPQA

Show the answer and why

Answer: EvalPlus (HumanEval+ and MBPP+), LiveCodeBench

EvalPlus and LiveCodeBench run generated code against tests, which is why both need a sandbox and why the harness gates such tasks behind a confirmation flag. IFEval checks instructions mechanically without executing anything, and GPQA is multiple choice.

Question 5. Why does LiveCodeBench annotate every problem with its release date?

• To sort the leaderboard chronologically
• So a model can be scored only on problems published after its training cut-off, which measures generalisation to unseen problems rather than recall of memorised ones
• Because the problems expire
• To match the model card publication dates

Show the answer and why

Answer: So a model can be scored only on problems published after its training cut-off, which measures generalisation to unseen problems rather than recall of memorised ones

This is the structural defence against contamination that Part 4 introduced. The project reports models whose scores drop sharply on problems released after their own cut-off, which is what contamination looks like when a benchmark is built to reveal it.

Check my answersClear