Measuring Quantisation Damage: Perplexity, KL Divergence and Task Evaluations

You now have several ways to make a smaller file out of a model. This lesson is about the question that decides which one you keep: how much did it cost, and cost at what?

By the end you will be able to say what perplexity does and does not tell you, compute and read a KL divergence against the full-precision original, choose between the three levels of measurement according to what you are deciding, read a provider’s quality claim for what it actually asserts, and record a comparison in the format the rest of this course expects.

Perplexity, and what it is silent about

Perplexity is the standard first measurement, and the llama.cpp tool that produces it puts the definition plainly: “perplexity measures how well the model can predict the next token with lower values being better”.

Mechanically, you take a held-out text, feed it through the model, and at every position ask what probability the model assigned to the token that actually came next. Average the negative logarithms of those probabilities and exponentiate. A model that assigned probability one to every correct token would score one; a model choosing uniformly among a fifty-thousand-token vocabulary would score fifty thousand. Real models sit in the single digits or low tens on ordinary prose.

It is cheap, it needs no labels, and it is a genuine signal. It also has four blind spots that matter for exactly the question this part asks.

It averages. A quantisation that leaves ninety-nine per cent of positions untouched and destroys the model’s behaviour on the remaining one per cent will move the mean very little. Those are often the positions you care about: the token that opens a JSON object, the one that selects a tool, the digit in the middle of an arithmetic step.

It is measured under teacher forcing. At every position the model is shown the correct prefix and asked about the next token. That is not what generation does. Generation feeds the model its own output, so an error compounds, and a small increase in the chance of a wrong token becomes a large increase in the chance of a wrong paragraph. Perplexity cannot see compounding.

It is measured on somebody’s generic text. The standard corpora are prose. If your workload is tool-calling JSON in a chat template, a perplexity computed on Wikipedia is measuring a different distribution from the one you serve.

It is not portable. The README is explicit that “perplexity is not directly comparable between models, especially if they use different tokenizers”. Two models with different vocabularies are solving different prediction problems, and comparing their perplexities compares nothing. Even within one model, two runs on different texts are not comparable.

✕Common mistake

Quoting a perplexity as a quality score. “This quant has a perplexity of 6.8” is meaningless without the model, the text and the tokeniser, and comparing it with someone else’s 6.5 on a different corpus is comparing nothing at all. Perplexity is only ever a difference, measured between two things that share a tokeniser and a text. The rest of this lesson is built on that observation.

KL divergence: compare the model with itself

If the useful form of perplexity is a difference, then measure the difference directly.

For a given prompt, the full-precision model produces a probability distribution over the next token. So does the quantised model. The Kullback-Leibler divergence between those two distributions is a single number saying how far apart they are, and llama.cpp’s documentation gives the interpretation you need: it is “a measure of how similar the FP16 and the quantized logit distributions are with a value of 0 indicating that the distribution are the same”.

That single property is why this course uses it as the primary quantisation metric. It compares the model with itself, so the tokeniser is identical, the text is identical, the architecture is identical, and the only thing that changed is the rounding. It has a meaningful zero. It is sensitive to changes at positions where perplexity averages them away, because it looks at the whole distribution rather than at the probability of one correct token. And it is cheap: no labels, no judge, no generation.

The tool computes it in two passes. The first runs the full-precision model and records its logits to a file; the second runs the quantised model against that file.

Fragment — not complete on its own

llama-perplexity \
    -m models/gguf/model-BF16.gguf \
    -f wiki.test.raw \
    --kl-divergence-base model-BF16.kld

Fragment — not complete on its own

llama-perplexity \
    -m models/gguf/model-Q4_K_M.gguf \
    -f wiki.test.raw \
    --kl-divergence-base model-BF16.kld \
    --kl-divergence

The second pass prints a block of statistics, and the field names are worth learning because they answer different questions: Mean PPL(Q) and Mean PPL(base) give both perplexities so you can see the classical number too; Mean KLD, Median KLD, Maximum KLD, 99.0% KLD and 99.9% KLD describe the distribution of the divergence rather than only its centre; Mean Δp, Maximum Δp and RMS Δp report how far the probability of the base model’s chosen token moved; and Same top p reports how often the two models would have picked the same token at all.

!Important

Read the tail, not the mean. Mean KLD can look reassuring while 99.0% KLD is enormous, and that shape is precisely the failure that perplexity hides: the model agrees with itself almost everywhere and diverges sharply at a small number of positions. Same top p is the field to check next, because a greedy decode is decided entirely by which token is on top. A quantisation with a tiny mean divergence and a noticeably lower Same top p will produce visibly different text.

When the tool is not available

llama-perplexity needs both models as GGUF files that llama.cpp can load. That covers Tracks S, X and N comfortably and covers Track M for GGUF, but it does not cover an MLX model, an AWQ checkpoint, or a comparison between two different engines.

For those cases the lab in this part ships kl-divergence.py, which computes the same quantity through two OpenAI-compatible servers. It sends identical prompts to both, asks each for the log-probabilities of the top candidate tokens at each position, and accumulates the divergence over the overlapping support.

That approach has an honest limitation and you should know it before you use the number. The API returns only the top few candidates rather than the full distribution, so the result is a divergence computed over a truncated support, not the exact quantity llama-perplexity reports. It is comparable between quantisations measured the same way, on the same prompts, against the same reference, and it is not comparable with a figure from the tool. The script records which method produced each number for exactly this reason.

The three levels, and what each is for

The measurement ladder: cheapest first, most meaningful last

1. 1Does it fit, and does it loadFile size plus KV cache against your memory, then one prompt. Costs seconds. Eliminates most of the candidate list.
2. 2Perplexity on a fixed textOne number per quantisation, comparable only within this model and this text. Useful as a smoke test: a large jump means something is badly wrong.
3. 3KL divergence against the full-precision originalMean, median, tail percentiles and Same top p. The course's primary quantisation metric: sensitive, cheap, unambiguous, and it isolates the rounding from everything else.
4. 4Your own task set, per quantisationPart 10's runner and judge, run once per file. This is where format compliance, tool calling and refusals show up, and none of the levels above can see them.
5. 5A standard suiteThe next lesson. Slowest, and the only level that produces a number other people can compare with. Run it on the quantisation you actually intend to keep.

Climb it until the decision is settled. Most quantisation choices are settled at level three; the ones that are not are the ones where the model is small, the bit width is low, or the task is unusual.

The levels are not substitutes. A quantisation can pass level three and fail level four: a model whose next-token distributions barely moved can still lose the habit of closing a JSON object, because that habit lives in a handful of high-stakes positions per response. And a model can pass level four and fail level five in the other direction, because your task set is twenty things you care about and a standard suite is a thousand things somebody else did.

∴Research note

The reason level three sits between them rather than replacing either is cost. KL divergence over a few hundred kilobytes of text takes minutes and needs no labels, so you can run it on every candidate. Your task set takes longer and needs a judge, so you run it on the two or three survivors. The suite takes an hour, so you run it once, on the file you are going to serve.

The results table

Every comparison in this part, and every quantisation decision you make afterwards, goes into one shape. Here it is, empty, as the recording sheet the lab fills in.

Pending validationPer-quantisation comparison — the course format, filled in by the lab

Quantisation	File GB	Mean KLD	99.0% KLD	Same top p	Task set passed	Judge mean	tg128 tokens/s	Peak memory GB
BF16 (reference)	—	0	0	1.00	—	—	—	—
Q8_0	—	—	—	—	—	—	—	—
Q5_K_M	—	—	—	—	—	—	—	—
Q4_K_M, imatrix	—	—	—	—	—	—	—	—
Q4_K_M, no imatrix	—	—	—	—	—	—	—	—
AWQ 4-bit or MLX 4-bit	—	—	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp for the GGUF rows; the engine that loads the fifth row for that row the build string from llama-cli --version, recorded once per row · your fine-tune from Part 13, or a reference model, as listed in the first column · 4,096 tokens of context · the date you ran it

Empty on purpose. The reference row is BF16 by definition: its divergence from itself is zero and it agrees with itself on every token, which is what makes the other rows readable. Record the calibration text used for the imatrix rows in your notebook, because a quantisation whose calibration text has been lost cannot be reproduced.

Two columns in that table do work that is easy to miss. The two Q4_K_M rows differ only by the importance matrix, which turns “imatrix quants are better” from a slogan into a number on your model. And the speed and memory columns are there because the whole point of the exercise is a trade: a row that costs a little quality and buys a great deal of speed is a different decision from one that costs the same quality and buys nothing.

Reading a provider’s quality claim

Quantisation providers publish quality claims, and the good ones are checkable. Six questions separate the two kinds.

Against what reference? A claim is only meaningful relative to something. “Minimal quality loss” against the BF16 original is a claim; against another quantisation it is a different and weaker one; against nothing it is a mood.

Measured with which metric? Perplexity, KL divergence, a task suite and a judge answer different questions, and a provider quoting perplexity alone has not looked at the tail.

On which text or task? A perplexity claim is inseparable from its corpus, and a task claim from its task.

Mean or tail? As above: this is where the difference between a quantisation you can serve and one you cannot usually lives.

Which exact file? Providers ship dozens of files per model and the claim usually applies to some of them. “Our four-bit quants are near-lossless” says nothing about which of the seven four-bit files in the repository was measured.

On what date, with which version of the tools? Quantisation code changes; a claim from a year ago describes a file produced by different code.

Judged that way, Google’s Gemma 3 QAT claim quoted in the previous lesson is a good one: it names the reference implicitly as the unquantised model, names the metric and the tool, names the format, and reports a relative reduction rather than an absolute equivalence. AWQ’s abstract is another: it states that the method “does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set”, which is a claim about method rather than a number about a file, and is testable as such.

?Reality check

The claim you will meet most is “Q4_K_M is basically lossless”. Held up properly it becomes: on some models, on some tasks, by some metric, the difference is smaller than the noise of the measurement. That is often true and it is worth knowing on which side of it your model falls, because small models and low bit widths are where it stops being true, and instruction following and structured output are where it stops being true first.

The course’s quantisation rule

One sentence, and everything in this part is machinery for applying it.

Measure on your task, and keep the format the measurement supports.

The corollaries are worth spelling out. The reference is always the full-precision model, because a comparison between two quantisations tells you which is closer to the other, not which is closer to the truth. The metric is chosen for the decision: KL divergence to shortlist, your task set to choose, a standard suite to publish. The measurement is recorded with its conditions, so that next month’s comparison is against something rather than against memory. And the answer “the difference is smaller than my measurement can see, so I will take the smaller file” is a perfectly good answer, which is why the lab has you estimate the noise before you interpret the differences.

Use paired failures to locate a regression

Evaluate the original and quantised model on identical task IDs. Mark four outcomes per task: both pass, both fail, only the original passes and only the quantised model passes. The last two categories show the changes that an overall average can hide.

Inspect the changed cases for a pattern: rare identifiers, arithmetic, long context, JSON termination or tool arguments. Keep generation settings and representation metadata with every response. Repeated sampled runs can distinguish a stable regression from ordinary output variation, though they do not create new independent task examples.

Use distributional metrics as diagnostics and application checks as acceptance criteria. Perplexity on one corpus does not cover every task, and divergence between token distributions does not state the cost of an incorrect action. Define a tolerance before inspecting the result: perhaps no regression on critical exact-output cases and a bounded change on the broader set. If the quantised variant fails that contract, choose another representation or a smaller model whose quality is adequate; memory fit alone is insufficient.

Recap

Perplexity measures next-token prediction on a held-out text under teacher forcing; it averages, it cannot see compounding, it is tied to its corpus, and it is not comparable across tokenisers, so it is a smoke test rather than a verdict. KL divergence against the full-precision original compares the model with itself, has a meaningful zero, and exposes the tail behaviour perplexity hides; llama.cpp computes it in two passes with --kl-divergence-base and --kl-divergence, reporting the mean, the median, the percentiles, the movement in the base model’s chosen token and how often the two models agree on the top token. Where that tool cannot reach, the same quantity can be approximated through two servers’ log-probabilities, over a truncated support, and compared only with figures produced the same way. Above KL divergence sit your own task set, where format and tool-calling failures live, and a standard suite, which is the only level that produces a number others can compare with. Provider claims are read by asking what reference, what metric, what text, mean or tail, which file and what date. And the rule is to measure on your task and keep the format the measurement supports.

Check your understanding

Question 1. A four-bit quantisation shows almost no change in perplexity, but the model has started emitting malformed JSON. How is that possible?

• Perplexity was computed incorrectly
• Perplexity averages over every position in a corpus, so damage concentrated on a few high-stakes tokens - such as the ones that open and close a structure - barely moves the mean
• JSON is not text, so perplexity does not apply
• The quantisation changed the tokeniser

Show the answer and why

Answer: Perplexity averages over every position in a corpus, so damage concentrated on a few high-stakes tokens - such as the ones that open and close a structure - barely moves the mean

This is the averaging blind spot, and it is why the course reads the tail percentiles of the KL divergence and the Same top p field rather than the mean alone, and why a task set with deterministic format checks sits above both.

Question 2. Why does this course use KL divergence against the full-precision model rather than perplexity as its primary quantisation metric?

• It is faster to compute
• It compares the model with itself on the same text and tokeniser, so only the rounding differs; it has a meaningful zero; and it exposes the distribution of the change rather than an average of one probability
• It is the only metric llama.cpp supports
• It produces a percentage that can be published

Show the answer and why

Answer: It compares the model with itself on the same text and tokeniser, so only the rounding differs; it has a meaningful zero; and it exposes the distribution of the change rather than an average of one probability

The tool describes a value of zero as meaning the distributions are the same. Because the reference is the same model, every confound - architecture, tokeniser, corpus - cancels, which is what makes the number attributable to the quantisation.

Question 3. Two quantisations of your model report the same Mean KLD, but one has a much lower Same top p. What follows?

• They are equivalent
• The one with the lower Same top p changes which token wins more often, so its greedy output will visibly differ even though the average divergence matches
• The measurement is invalid
• The lower Same top p means a smaller file

Show the answer and why

Answer: The one with the lower Same top p changes which token wins more often, so its greedy output will visibly differ even though the average divergence matches

Greedy decoding is decided entirely by which token is on top. Two distributions can differ by the same average amount while one of them reorders the leaders more often, and that difference is what a reader actually sees in the output.

Question 4. Which of these are sound reasons to run a task evaluation as well as a KL divergence? Select all that apply.

• Format compliance and tool calling live in a few positions per response and can break while the average distribution barely moves
• Generation compounds errors, which a per-position divergence against a fixed prefix cannot see
• A task set measures the work you actually do, which no distributional metric does
• KL divergence cannot be computed for four-bit models

Show the answer and why

Answer: Format compliance and tool calling live in a few positions per response and can break while the average distribution barely moves, Generation compounds errors, which a per-position divergence against a fixed prefix cannot see, A task set measures the work you actually do, which no distributional metric does

The first three are the reasons the ladder has more than one rung. KL divergence is computable for any pair of models you can obtain distributions from; the fourth option is simply false.

Question 5. A repository says "our four-bit quantisations are near-lossless". What is the first question to ask?

• How large are the files
• Near-lossless against what reference, measured with which metric, on what text or task, for which of the several four-bit files, and on what date
• Which hardware was used
• Is the repository popular

Show the answer and why

Answer: Near-lossless against what reference, measured with which metric, on what text or task, for which of the several four-bit files, and on what date

Hardware and file size are inside the larger question. A quality claim with no reference, no metric and no named file is not a claim that can be true or false, which is the same test Part 4 applied to benchmark scores.

Check my answersClear