Skip to content
Local LLM Course

Quantisation and Evaluation, Properly

Every model you have run in this course has been quantised. Part 6 taught you to read the names on the files and pick one that fits; Part 13 exported your own fine-tune to a four-bit GGUF and served it. What none of those pages did was answer the question underneath: what did the rounding cost, on your task, and how would you know?

This part answers it. The first half is quantisation as a set of decisions rather than a set of file names: which weights each method protects, what it needs in order to decide that, and what the current low-precision hardware formats are actually made of. The second half is evaluation done to a standard where a number means something: standard harnesses run with their settings recorded, a published score reproduced and its gap explained, a judge model measured before it is believed, and a report that a stranger could repeat.

The two halves belong together because each is useless alone. A quantisation you cannot measure is a guess. An evaluation you run on the wrong quantisation is a measurement of the wrong thing.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain what GPTQ, AWQ, K-quant mixtures, importance matrices and HQQ each decide and what each one needs from you, and choose between them for a given model, machine and deadline;
  • say what quantisation-aware training changes about a checkpoint, recognise a published QAT checkpoint when you see one, and read FP8, NVFP4 and MXFP4 as concrete layouts of bits rather than as marketing names;
  • measure the damage a quantisation did with perplexity, with KL divergence against the full-precision original, and with a task evaluation, and say which of the three you would trust for which decision;
  • install and drive a standard evaluation harness against a local server, choose a task whose shape matches your question, and state every setting that changed the result;
  • reproduce somebody else’s published score, or explain in specific terms why you could not;
  • use a model as a judge with the biases named and the agreement with your own labels measured;
  • diagnose two evaluation runs that disagree, and prove the fix.

What you need first

Section titled “What you need first”

Part 6, because the whole first lab is llama-quantize and its friends. Part 9’s gateway, because the benchmark lab runs the harness against it rather than against a Python process holding the weights. Part 10’s task set and judge harness, which this part reuses per quantisation instead of per model. And, ideally, an artefact of your own to evaluate: the fine-tune from Part 13’s lab or the student from Part 15’s lab. Every script here also accepts a plain model path or a gateway alias, so a reader who has not done those parts can run all of it against a reference model and lose nothing but the sentimental value.

The memory floor is 12 GB for the quantisation lab, which needs one full-precision copy of a small model on disk and in memory during conversion, and 8 GB for everything else. Disk matters more than memory here: five quantisations of one model plus the full-precision source is a larger download than anything since Part 5.

How to work through it

Section titled “How to work through it”

The five lessons are in dependency order and the labs assume all of them. Read the first two together; they are one subject split at the line between “round a finished model” and “train a model that expects to be rounded”. The third lesson is the one to read slowly, because it defines the metric the rest of the course uses when it says a quantisation is acceptable.

Then do the labs with your notebook open. The first produces a table of five quantisations of one model with the damage and the speed of each, which is the table you will consult every time you choose a file for the rest of your life with this hardware. The second produces something rarer: a public benchmark number you generated yourself, with its settings written down beside it. The challenge at the end is where those settings turn out to matter by ten points.

Part16 of 27Level3 — Model BuilderPages8Estimated time5.1 hours

0 / 8 lessons in this part completed