Lab: Quantise Your Fine-Tune Five Ways and Measure Each

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The per-track model sizes, wall-clock times and the tool versions each track was executed with belong here once the validation pass has run this lab on real machines.

Objective

By the end of this hour you will have one model in six forms — the full-precision original and five quantisations — and one table saying what each of the five cost and what each bought. You will have chosen one of them on the numbers rather than on the file name, and you will have written down the calibration text that produced the importance matrix so that the choice can be reproduced next month.

The model to use is the fine-tune from Part 13’s lab, because a fine-tune is where quantisation damage matters most: the behaviour you trained in lives in a small number of positions and is exactly the kind of thing an averaged metric cannot see. If you have not done Part 13, or you have and would rather not risk the artefact, every script here takes any model path or gateway alias, and a reference model from the model library works identically.

⌂Single-machine path

Single-machine on every track. The reference model and each quantisation are served one after the other, never together, so the memory floor is set by the largest single file rather than by two.

Requirements

An hour, of which roughly twenty-five minutes is attended and the rest is conversion and measurement you can leave running. Disk is the resource to check first: the full-precision file plus five quantisations of an 8B-class model is a substantial directory, and the full-precision file cannot be deleted until every measurement is finished.

You need the llama.cpp build from Part 6 — llama.cpp v0.4.0 · verified 2026-09-08 — with llama-quantize, llama-imatrix, llama-server and llama-bench; Part 10’s run-eval.py and your own task file; and a plain-text calibration file of your own. The memory floor of 12 GB is set by holding one full-precision copy of a small model during conversion and during the importance-matrix pass.

What is resident during the importance-matrix pass, Qwen3-4B at BF16, on a 12 GB machine — an estimate from the course model reference

Weights at BF16

8 GB

KV cache and buffers for the calibration pass

1.2 GB

Free

2.8 GB

Total

12 GB

The reference model reference gives 8.0 GB of weights at BF16 for this model, and the pass allocates a modest context on top. That is the tightest moment in the lab: every quantised file after it is smaller. On a 12 to 16 GB machine, use a 1.7B-class model instead and the same arithmetic gives about 3.4 GB. This is arithmetic, not a measurement.

Track S — NVIDIA DGX Spark

128 GB of unified memory means you can use the 8B-class model and still have room to be careless. Your fifth format is AWQ four-bit through llm-compressor, which is the path this course uses for a non-GGUF quantisation: AutoAWQ’s own README states that it “is officially deprecated and will no longer be maintained”, and llm-compressor is maintained under the vLLM project and targets the engine you already run.

Allow extra time for the AWQ calibration pass, which loads the model at bfloat16 and runs several hundred forward passes over it.

Track X — AMD Ryzen AI Max+ 395

Use the 4B-class or 8B-class model depending on what you fine-tuned. Your fifth format is a second GGUF level, IQ4_XS built with the same importance matrix, which gives you the most interesting comparison available on this track: two four-bit files of different families, both with the same importance signal.

llm-compressor’s documented deployment target is vLLM on CUDA, and this course has not validated its ROCm path; if you have a working PyTorch ROCm environment from Part 11 you may attempt the AWQ row as well, and record it as unvalidated.

Track M — Apple silicon

Use the model size your memory allows; a 32 GB Mac is comfortable with the 4B-class model. Your fifth format is MLX four-bit through mlx_lm.convert, which is the first-party quantiser on this track and produces a file the mlx-lm server from Part 8 loads directly.

The GGUF rows are measured with llama.cpp’s Metal build exactly as on the other tracks; the MLX row is measured through kl-divergence.py against the same reference, because llama-perplexity cannot load an MLX model.

Track N — NVIDIA desktop or laptop

On 24 GB and above, your fifth format is AWQ four-bit through llm-compressor, and the model to use is the 4B-class fine-tune. On 12 to 16 GB, the AWQ calibration pass will not fit alongside a bfloat16 model of a useful size: use a second GGUF level, IQ4_XS with the importance matrix, and use the 1.7B-class fine-tune so that the full-precision reference fits comfortably.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-16-quantisation-and-evaluation"
cd "$LAB_DIR"
pwd
test -f "quantise-five-ways.sh"

Expected result: pwd ends in part-16-quantisation-and-evaluation and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Choose the model, and set the paths

Everything below uses two variables. Set them once, in the shell you will work in.

RunnableAll tracks

the source and the working directory

export SOURCE=~/models/runs/format-qwen3-4b-merged
export WORK=~/models/five-ways
mkdir -p "$WORK"

SOURCE is the merged safetensors directory Part 13’s merge-adapter.py wrote, or any Hugging Face model directory. If you already have a full-precision GGUF, point SOURCE at that file instead and the conversion step is skipped.

2. Write the calibration text

This is the ten minutes that decides whether the importance matrix is worth anything.

The text should resemble the traffic the model will actually see, and it must be disjoint from anything you are going to score on. If you fine-tuned in Part 13, the natural source is the training split of your own dataset, rendered as plain text; the evaluation split is exactly what you must not use.

RunnableAll tracks

a calibration text from the training split of your own dataset

python3 - <<'PY'
import json, pathlib
rows = [json.loads(line) for line in
        pathlib.Path("data/train.jsonl").read_text(encoding="utf-8").splitlines() if line.strip()]
with open("calibration.txt", "w", encoding="utf-8") as out:
    for row in rows:
        for message in row.get("messages", []):
            out.write(message.get("content", "").strip() + "\n\n")
print(f"{len(rows)} rows written to calibration.txt")
PY
wc -c calibration.txt

If you have no dataset of your own, use any prose corpus you already hold that resembles your use — your documentation, your issue tracker exported to text, a book you have the right to keep on disk. A few hundred kilobytes is plenty. What you must not do is reach for the first public text file that comes to hand and then draw conclusions about your own workload from the result.

✕Common mistake

Calibrating on the evaluation set. It is the fastest way to produce a quantisation that scores beautifully and behaves worse than the one it beat. The matrix will protect exactly the weights your test needs, and nothing else. Part 13’s decontaminate.py will tell you whether two files overlap if you are unsure.

3. Convert and quantise three ways

RunnableAll tracks

quantise-five-ways.sh

#!/usr/bin/env bash
# Purpose: produce the GGUF half of the five-way comparison from one source checkpoint - a
#          full-precision reference plus Q8_0, Q5_K_M, Q4_K_M with an importance matrix and
#          Q4_K_M without one - so that every later measurement has the same starting point
# Platform: all (spark, strix, mac, nvidia); uses the llama.cpp build from Part 6
# Minimum memory: 12 GB; conversion holds one full-precision copy, quantisation streams
# Assumes: a llama.cpp checkout holding convert_hf_to_gguf.py (LLAMA_CPP), llama-quantize on
#          PATH or under LLAMA_BIN, an importance matrix from make-imatrix.sh for the fourth
#          file, and disk for roughly three times the full-precision size of the model
#
# Usage: bash quantise-five-ways.sh SOURCE OUTDIR [IMATRIX]
#   SOURCE   a Hugging Face model directory (safetensors) or an existing full-precision GGUF
#   OUTDIR   where the GGUF files are written
#   IMATRIX  the matrix from make-imatrix.sh; omit to skip the imatrix row
#
# Environment: LLAMA_CPP (default ~/llama.cpp), LLAMA_BIN, NAME (output prefix),
#              EXTRA_TYPE (a fifth GGUF type such as IQ4_XS, quantised with the matrix),
#              LABBOOK (default labbook.md)
set -euo pipefail

SOURCE="${1:-}"
OUTDIR="${2:-}"
IMATRIX="${3:-}"
LLAMA_CPP="${LLAMA_CPP:-$HOME/llama.cpp}"
LABBOOK="${LABBOOK:-labbook.md}"

die() { echo "quantise-five-ways: $*" >&2; exit 1; }

[[ -n "$SOURCE" && -n "$OUTDIR" ]] || die "usage: bash quantise-five-ways.sh SOURCE OUTDIR [IMATRIX]"
[[ -e "$SOURCE" ]] || die "$SOURCE does not exist"
command -v python3 >/dev/null || die "python3 is not on PATH"

QUANTIZE_BIN="${LLAMA_BIN:-}/llama-quantize"
if [[ -z "${LLAMA_BIN:-}" ]]; then
  QUANTIZE_BIN="$(command -v llama-quantize || true)"
fi
[[ -x "$QUANTIZE_BIN" ]] || die "llama-quantize not found; set LLAMA_BIN to the directory holding it (built in Part 6)"

mkdir -p "$OUTDIR"
NAME="${NAME:-$(basename "${SOURCE%.gguf}")}"
REFERENCE="$OUTDIR/$NAME-BF16.gguf"

# ---------------------------------------------------------------------------
# 1. The reference. Everything else in this part is measured against this file.
# ---------------------------------------------------------------------------
if [[ -f "$SOURCE" && "$SOURCE" == *.gguf ]]; then
  echo "==> 1/3 Using the supplied GGUF as the full-precision reference"
  REFERENCE="$SOURCE"
elif [[ -d "$SOURCE" ]]; then
  if [[ -f "$REFERENCE" ]]; then
    echo "==> 1/3 $REFERENCE already exists; reusing it"
  else
    [[ -f "$LLAMA_CPP/convert_hf_to_gguf.py" ]] || die "convert_hf_to_gguf.py not found under $LLAMA_CPP; set LLAMA_CPP"
    echo "==> 1/3 Converting $SOURCE to GGUF at bfloat16"
    echo "    Converting at full precision and quantising afterwards keeps information the"
    echo "    quantiser can use. Converting straight to a small type throws it away."
    python3 "$LLAMA_CPP/convert_hf_to_gguf.py" "$SOURCE" \
      --outfile "$REFERENCE" \
      --outtype bf16
  fi
else
  die "$SOURCE is neither a directory nor a .gguf file"
fi

# ---------------------------------------------------------------------------
# 2. The quantisations.
# ---------------------------------------------------------------------------
echo
echo "==> 2/3 Quantising"

quantise() {
  local type="$1" suffix="$2" use_imatrix="$3"
  local out="$OUTDIR/$NAME-$suffix.gguf"
  if [[ -f "$out" ]]; then
    echo "    $suffix already exists; skipping. Delete it to rebuild."
    return 0
  fi
  echo "    $suffix"
  if [[ "$use_imatrix" == "yes" ]]; then
    "$QUANTIZE_BIN" --imatrix "$IMATRIX" "$REFERENCE" "$out" "$type"
  else
    "$QUANTIZE_BIN" "$REFERENCE" "$out" "$type"
  fi
}

quantise q8_0 Q8_0 no
quantise q5_k_m Q5_K_M no
quantise q4_k_m Q4_K_M-noimat no

if [[ -n "$IMATRIX" ]]; then
  [[ -f "$IMATRIX" ]] || die "$IMATRIX does not exist; run make-imatrix.sh first or omit the argument"
  quantise q4_k_m Q4_K_M-imat yes
  if [[ -n "${EXTRA_TYPE:-}" ]]; then
    quantise "$(echo "$EXTRA_TYPE" | tr '[:upper:]' '[:lower:]')" "$EXTRA_TYPE-imat" yes
  fi
else
  echo "    no importance matrix supplied; the imatrix row will be missing from your table"
  if [[ -n "${EXTRA_TYPE:-}" ]]; then
    quantise "$(echo "$EXTRA_TYPE" | tr '[:upper:]' '[:lower:]')" "$EXTRA_TYPE" no
  fi
fi

# ---------------------------------------------------------------------------
# 3. Sizes, and the record.
# ---------------------------------------------------------------------------
echo
echo "==> 3/3 What you now have"
for f in "$OUTDIR/$NAME"-*.gguf "$REFERENCE"; do
  [[ -f "$f" ]] || continue
  printf '    %-44s %s bytes\n' "$(basename "$f")" "$(wc -c < "$f" | tr -d ' ')"
done | sort -u

python3 - "$LABBOOK" "$OUTDIR" "$NAME" "${IMATRIX:-none}" <<'PY'
import glob, json, os, platform, sys, time

labbook, outdir, name, imatrix = sys.argv[1:5]
files = sorted(set(glob.glob(os.path.join(outdir, f"{name}-*.gguf"))))
record = {
    "lab": "part-16/lab-quantise-five-ways-and-measure/quantise",
    "run_id": time.strftime("%Y%m%dT%H%M%S"),
    "name": name,
    "imatrix": os.path.basename(imatrix) if imatrix != "none" else None,
    "files": {os.path.basename(f): os.path.getsize(f) for f in files},
    "host": platform.platform(),
    "date": time.strftime("%Y-%m-%d"),
}
with open(labbook, "a", encoding="utf-8") as handle:
    handle.write(json.dumps(record) + "\n")
print(f"\nrecorded in {labbook}")
PY

echo
echo "    Keep the full-precision file until every measurement is finished. It is the reference"
echo "    every KL divergence in this lab is computed against, and it is not recoverable from"
echo "    the quantised ones."

Download quantise-five-ways.sh135 lines

Run it once with no importance matrix. It converts to a full-precision GGUF if it needs to, then writes Q8_0, Q5_K_M and Q4_K_M-noimat, skipping anything that already exists.

RunnableAll tracks

the first three quantisations, and the reference

bash quantise-five-ways.sh "$SOURCE" "$WORK"

Output — what you should see

==> 1/3 Converting … to GGUF at bfloat16
==> 2/3 Quantising
    Q8_0
    Q5_K_M
    Q4_K_M-noimat
    no importance matrix supplied; the imatrix row will be missing from your table
==> 3/3 What you now have
    format-qwen3-4b-merged-BF16.gguf              8012345678 bytes
    …

The order matters and the script enforces it: converting at full precision first and quantising afterwards keeps information the quantiser can use, which is the same rule Part 13’s export script follows.

4. Build the importance matrix, and the fourth file

RunnableAll tracks

make-imatrix.sh

#!/usr/bin/env bash
# Purpose: build an importance matrix from a calibration text so that llama-quantize can spend
#          its bit budget on the weights that carry the most signal for your workload, and record
#          which text produced it so the quantisation can be reproduced later
# Platform: all (spark, strix, mac, nvidia); uses the llama.cpp build from Part 6
# Minimum memory: 12 GB; the full-precision model is loaded once, and -ngl decides where
# Assumes: llama-imatrix on PATH or under LLAMA_BIN, a full-precision GGUF (made by
#          quantise-five-ways.sh or by convert_hf_to_gguf.py), and a calibration text file of
#          your own that is disjoint from anything you will evaluate on
#
# Usage: bash make-imatrix.sh MODEL_BF16_GGUF CALIBRATION_TXT [OUTPUT_GGUF]
#   MODEL_BF16_GGUF  the full-precision GGUF to collect statistics from
#   CALIBRATION_TXT  plain text resembling your workload; a few hundred kilobytes is plenty
#   OUTPUT_GGUF      where to write the matrix; defaults to imatrix.gguf beside the model
#
# Environment: LLAMA_BIN (directory holding llama-imatrix), NGL (default 99),
#              CHUNKS (limit the number of calibration chunks), LABBOOK (default labbook.md)
set -euo pipefail

MODEL="${1:-}"
CALIB="${2:-}"
OUT="${3:-}"
NGL="${NGL:-99}"
LABBOOK="${LABBOOK:-labbook.md}"

die() { echo "make-imatrix: $*" >&2; exit 1; }

[[ -n "$MODEL" && -n "$CALIB" ]] || die "usage: bash make-imatrix.sh MODEL_BF16_GGUF CALIBRATION_TXT [OUTPUT_GGUF]"
[[ -f "$MODEL" ]] || die "$MODEL does not exist"
[[ -f "$CALIB" ]] || die "$CALIB does not exist"

IMATRIX_BIN="${LLAMA_BIN:-}/llama-imatrix"
if [[ -z "${LLAMA_BIN:-}" ]]; then
  IMATRIX_BIN="$(command -v llama-imatrix || true)"
fi
[[ -x "$IMATRIX_BIN" ]] || die "llama-imatrix not found; set LLAMA_BIN to the directory holding it (built in Part 6)"

[[ -n "$OUT" ]] || OUT="$(dirname "$MODEL")/imatrix.gguf"

CALIB_BYTES=$(wc -c < "$CALIB" | tr -d ' ')
if [[ "$CALIB_BYTES" -lt 20000 ]]; then
  echo "make-imatrix: WARNING: $CALIB is only $CALIB_BYTES bytes." >&2
  echo "              A matrix from a handful of sentences describes a handful of sentences." >&2
  echo "              Aim for a few hundred kilobytes of text that resembles your real traffic." >&2
fi

# The hash is the point of this line: an importance matrix is only reproducible if you can say
# which bytes produced it, and calibration texts get edited.
if command -v sha256sum >/dev/null 2>&1; then
  CALIB_SHA="$(sha256sum "$CALIB" | cut -d' ' -f1)"
elif command -v shasum >/dev/null 2>&1; then
  CALIB_SHA="$(shasum -a 256 "$CALIB" | cut -d' ' -f1)"
else
  CALIB_SHA="unavailable"
fi

echo "==> Collecting activation statistics"
echo "    model:       $MODEL"
echo "    calibration: $CALIB ($CALIB_BYTES bytes, sha256 ${CALIB_SHA:0:16})"
echo "    output:      $OUT"
echo

STARTED=$(date +%s)
IMATRIX_ARGS=(
  -m "$MODEL"
  -f "$CALIB"
  -o "$OUT"
  -ngl "$NGL"
  --output-frequency 20
  --save-frequency 50
  --parse-special
)
if [[ -n "${CHUNKS:-}" ]]; then
  IMATRIX_ARGS+=(--chunks "$CHUNKS")
fi

"$IMATRIX_BIN" "${IMATRIX_ARGS[@]}"
ELAPSED=$(( $(date +%s) - STARTED ))

[[ -f "$OUT" ]] || die "llama-imatrix finished but $OUT was not written"

echo
echo "==> Wrote $OUT in ${ELAPSED}s"
echo "    Read it back with: $IMATRIX_BIN --in-file $OUT --show-statistics"
echo "    Two matrices from different texts can be merged by passing --in-file twice."

python3 - "$LABBOOK" "$MODEL" "$CALIB" "$CALIB_SHA" "$CALIB_BYTES" "$OUT" "$ELAPSED" <<'PY'
import json, os, platform, sys, time

labbook, model, calib, sha, size, out, elapsed = sys.argv[1:8]
record = {
    "lab": "part-16/lab-quantise-five-ways-and-measure/imatrix",
    "run_id": time.strftime("%Y%m%dT%H%M%S"),
    "model": os.path.basename(model),
    "calibration_file": os.path.basename(calib),
    "calibration_sha256": sha,
    "calibration_bytes": int(size),
    "imatrix_file": os.path.basename(out),
    "seconds": int(elapsed),
    "host": platform.platform(),
    "date": time.strftime("%Y-%m-%d"),
}
with open(labbook, "a", encoding="utf-8") as handle:
    handle.write(json.dumps(record) + "\n")
print(f"recorded in {labbook}")
PY

Download make-imatrix.sh106 lines

RunnableAll tracks

collect activation statistics from the full-precision model

bash make-imatrix.sh "$WORK"/*-BF16.gguf calibration.txt "$WORK/imatrix.gguf"

The script records the calibration file’s SHA-256 in your lab notebook. That line is the difference between a quantisation you can rebuild and one you can only re-guess.

Now run the quantiser again, with the matrix. It skips the three files that already exist and adds the one that differs only by the importance signal.

RunnableAll tracks

the same four-bit type, this time with the matrix

bash quantise-five-ways.sh "$SOURCE" "$WORK" "$WORK/imatrix.gguf"

You now hold Q4_K_M-noimat and Q4_K_M-imat: two files of the same nominal width, from the same source, differing in one input. Part 6 told you that imatrix quantisations are supposed to be better. This pair is where that becomes a number on your model.

5. Build the fifth file, on your track

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

quantise-awq.py

#!/usr/bin/env python3
"""Quantise a checkpoint to four-bit AWQ with llm-compressor, for the non-GGUF row of the table.

Purpose: produce the fifth file in the five-way comparison on the tracks that can run it - an
    activation-aware four-bit checkpoint that vLLM and transformers load directly, so the table
    contains one row that was not made by llama.cpp. AWQ is chosen over GPTQ here because its
    paper reports that it does no backpropagation or reconstruction and so generalises without
    overfitting the calibration set, which matters when the calibration data is somebody else's.
    llm-compressor is chosen over AutoAWQ because AutoAWQ's own README states that it "is
    officially deprecated and will no longer be maintained".
Platform: spark, nvidia (llm-compressor targets CUDA and the vLLM deployment path). Track X can
    attempt it where a working PyTorch ROCm stack from Part 11 is present, and Track M should
    use the MLX path in the lab instead; mlx_lm.convert is the first-party quantiser there.
Minimum memory: 24 GB for a 4B-class model at bfloat16 plus the calibration forward passes;
    32 GB and above for an 8B-class model. Reduce --num-calibration-samples and
    --max-seq-length before reducing the model size.
Assumes: a Python environment with torch, transformers and llmcompressor installed
    (`uv pip install llmcompressor`), a local model directory or Hugging Face model id, and
    network access for the calibration dataset unless one is already cached.

Usage: python3 quantise-awq.py --model ~/models/runs/format-qwen3-4b-merged \
           --out ~/models/awq/format-qwen3-4b-awq --labbook labbook.md
       python3 quantise-awq.py --model Qwen/Qwen3-4B --out ~/models/awq/Qwen3-4B-awq \
           --num-calibration-samples 128 --max-seq-length 512
"""

from __future__ import annotations

import argparse
import json
import platform
import sys
import time
from pathlib import Path


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", required=True,
                        help="local model directory or Hugging Face model id to quantise")
    parser.add_argument("--out", required=True, help="directory to write the quantised model to")
    parser.add_argument("--dataset", default="perfectblend",
                        help="calibration dataset, as named in the llm-compressor examples. To "
                             "calibrate on your own text instead, follow llm-compressor's "
                             "custom-dataset example; the recipe below does not change.")
    parser.add_argument("--split", default="train[:512]", help="dataset split expression")
    parser.add_argument("--num-calibration-samples", type=int, default=256)
    parser.add_argument("--max-seq-length", type=int, default=512)
    parser.add_argument("--scheme", default="W4A16_ASYM",
                        help="quantisation scheme; W4A16_ASYM is four-bit weights with sixteen-bit "
                             "activations, which is the configuration that helps single-user decode")
    parser.add_argument("--prompt", default="Hello my name is",
                        help="one prompt run through the quantised model as a smoke test")
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    try:
        from transformers import AutoModelForCausalLM, AutoTokenizer

        from llmcompressor import oneshot
        from llmcompressor.modifiers.quantization import QuantizationModifier
        from llmcompressor.modifiers.transform.awq import AWQModifier
    except ImportError as exc:  # pragma: no cover - depends on the reader's environment
        sys.exit(
            f"missing dependency: {exc}\n"
            "Install with: uv pip install llmcompressor\n"
            "Track M: use the MLX path in the lab instead; llm-compressor targets the CUDA and "
            "vLLM deployment path and Apple silicon is not one of its documented targets."
        )

    print(f"==> Loading {args.model}")
    model = AutoModelForCausalLM.from_pretrained(args.model)
    tokenizer = AutoTokenizer.from_pretrained(args.model)

    # The recipe is two modifiers, in the order llm-compressor's own AWQ example uses. The first
    # computes the per-channel scales from calibration activations; the second does the actual
    # rounding. lm_head is left alone because quantising the output projection costs more than
    # it saves.
    recipe = [
        AWQModifier(duo_scaling="both"),
        QuantizationModifier(
            ignore=["lm_head"],
            scheme=args.scheme,
            targets=["Linear"],
        ),
    ]

    print(f"==> Calibrating on {args.dataset} [{args.split}], "
          f"{args.num_calibration_samples} samples at {args.max_seq_length} tokens")
    print("    This is the step that decides which channels get protected. A calibration set")
    print("    that looks nothing like your traffic protects the wrong channels.")
    started = time.time()
    oneshot(
        model=model,
        dataset=args.dataset,
        splits=args.split,
        recipe=recipe,
        max_seq_length=args.max_seq_length,
        num_calibration_samples=args.num_calibration_samples,
    )
    elapsed = time.time() - started

    print("\n========== SAMPLE GENERATION ==============")
    sample = tokenizer(args.prompt, return_tensors="pt")
    sample = {key: value.to(model.device) for key, value in sample.items()}
    output = model.generate(**sample, max_new_tokens=64)
    print(tokenizer.decode(output[0]))
    print("==========================================\n")
    print("If that is fluent, the quantisation did not break the model outright. It says nothing")
    print("about how much it cost; measure-quants.sh answers that.")

    out = Path(args.out)
    out.parent.mkdir(parents=True, exist_ok=True)
    model.save_pretrained(str(out), save_compressed=True)
    tokenizer.save_pretrained(str(out))
    print(f"\n==> Wrote {out}")

    total_bytes = sum(f.stat().st_size for f in out.rglob("*") if f.is_file())
    record = {
        "lab": "part-16/lab-quantise-five-ways-and-measure/awq",
        "run_id": time.strftime("%Y%m%dT%H%M%S"),
        "source_model": args.model,
        "out": str(out),
        "scheme": args.scheme,
        "calibration_dataset": args.dataset,
        "calibration_split": args.split,
        "num_calibration_samples": args.num_calibration_samples,
        "max_seq_length": args.max_seq_length,
        "bytes": total_bytes,
        "seconds": round(elapsed, 1),
        "host": platform.platform(),
        "date": time.strftime("%Y-%m-%d"),
    }
    print(json.dumps(record, indent=2))

    if args.labbook:
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download quantise-awq.py143 lines

RunnableTrack S · DGX Spark

four-bit AWQ through llm-compressor

uv pip install llmcompressor
python3 quantise-awq.py \
    --model "$SOURCE" \
    --out ~/models/awq/five-ways-awq \
    --labbook labbook.md

Serve it with vLLM to measure it; kl-divergence.py compares it against the same reference through the two servers’ log-probabilities.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

a second four-bit family, with the same importance matrix

EXTRA_TYPE=IQ4_XS bash quantise-five-ways.sh "$SOURCE" "$WORK" "$WORK/imatrix.gguf"

IQ4_XS is a codebook type rather than a K-quant mixture, and it is smaller than Q4_K_M at a nominal width the name calls the same. Whether it is also worse on your model is precisely what the next two steps answer.

Track M — Apple silicon

RunnableTrack M · Apple silicon

four-bit MLX

mlx_lm.convert \
    --hf-path "$SOURCE" \
    --mlx-path ~/models/mlx/five-ways-4bit \
    -q \
    --q-bits 4 \
    --q-group-size 64

Serve it with the mlx-lm server from Part 8 and measure it with kl-divergence.py against the same reference. It is the one row in your table produced by a different engine, which is worth keeping in mind when you read it: the comparison is honest about the file and slightly generous about the engine, because two engines never round identically.

Track N — NVIDIA desktop or laptop

On 24 GB and above, take the AWQ path:

RunnableTrack N · NVIDIA GPU

four-bit AWQ through llm-compressor

uv pip install llmcompressor
python3 quantise-awq.py \
    --model "$SOURCE" \
    --out ~/models/awq/five-ways-awq \
    --num-calibration-samples 128 \
    --labbook labbook.md

On 12 to 16 GB, take the second GGUF level instead:

RunnableTrack N · NVIDIA GPU

a second four-bit family, with the same importance matrix

EXTRA_TYPE=IQ4_XS bash quantise-five-ways.sh "$SOURCE" "$WORK" "$WORK/imatrix.gguf"

6. Establish the noise floor before you measure anything

RunnableAll tracks

kl-divergence.py

#!/usr/bin/env python3
"""Measure how far a quantised model's next-token distributions have moved from the original.

Purpose: the portable version of llama.cpp's KL-divergence mode, for the cases that tool cannot
    reach: an MLX model, an AWQ checkpoint, two different engines, or any pair of models served
    behind the Part 9 gateway. It walks a fixed text, asks both servers for the next-token
    distribution at the same positions, and reports the divergence between them along with how
    often they would have chosen the same token.
Platform: all (pure Python over HTTP; the servers may be on any track or another machine)
Minimum memory: negligible for this script. The servers behind it need whatever their models
    need, and running the reference and the quantised model one after the other on a small
    machine works: pass --save-ref and --load-ref instead of two live endpoints.
Assumes: Python 3.9 or later, and OpenAI-compatible servers exposing /v1/completions with
    log-probabilities (llama-server, vLLM, mlx_lm.server and the Part 9 gateway all do). The two
    models must share a tokeniser, which is guaranteed when one is a quantisation of the other
    and is the reason this comparison is meaningful at all.

    HONEST LIMITATION: the API returns only the top few candidate tokens, so the divergence is
    computed over a truncated support with a floor for the unseen mass. It is comparable between
    quantisations measured this way against the same reference on the same text, and it is NOT
    comparable with a figure from llama-perplexity --kl-divergence, which sees the full
    distribution. The output records which method produced it.

Usage: python3 kl-divergence.py --text calibration.txt \
           --base-url-ref http://127.0.0.1:8080/v1 --model-ref reference \
           --base-url-quant http://127.0.0.1:8081/v1 --model-quant q4-k-m \
           --quant-label Q4_K_M --out kld-Q4_K_M.json --labbook labbook.md
       python3 kl-divergence.py --text calibration.txt --base-url-ref http://127.0.0.1:8080/v1 \
           --model-ref reference --save-ref ref-logprobs.json
       python3 kl-divergence.py --text calibration.txt --load-ref ref-logprobs.json \
           --base-url-quant http://127.0.0.1:8080/v1 --model-quant q4-k-m --quant-label Q4_K_M
"""

from __future__ import annotations

import argparse
import json
import math
import platform
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Optional

# Probability mass the API did not show us has to be given some value. Every token outside a
# model's returned top-k is treated as one natural log unit below the smallest logprob it did
# return, which keeps the divergence finite without pretending the tail is empty. It is an
# assumption, it is applied identically to both models, and it is recorded in the output.
UNSEEN_MARGIN = 1.0


def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict:
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:400]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


def extract_top_logprobs(body: dict) -> dict:
    """Pull {token: logprob} out of either the completions or the chat-completions shape."""
    choice = body["choices"][0]
    logprobs = choice.get("logprobs") or {}

    # Legacy completions: {"top_logprobs": [{token: logprob, ...}]}
    top = logprobs.get("top_logprobs")
    if isinstance(top, list) and top and isinstance(top[0], dict):
        first = top[0]
        if all(isinstance(v, (int, float)) for v in first.values()):
            return {str(k): float(v) for k, v in first.items()}

    # Chat completions: {"content": [{"top_logprobs": [{"token": …, "logprob": …}, …]}]}
    content = logprobs.get("content")
    if isinstance(content, list) and content:
        entries = content[0].get("top_logprobs") or []
        if entries:
            return {str(e["token"]): float(e["logprob"]) for e in entries}

    raise RuntimeError(
        "the server returned no top log-probabilities. Start llama-server without --log-disable "
        "and check that the endpoint is /v1/completions; some servers need the request to ask "
        "for logprobs explicitly, which this script does."
    )


def make_prefixes(text: str, count: int, min_words: int) -> list:
    """Cut the text into increasing prefixes, spread evenly, at whitespace boundaries.

    Positions are chosen deterministically so that two runs of this script on the same text
    measure the same places. Nothing is sampled anywhere in this script.
    """
    words = text.split()
    if len(words) < min_words + count:
        raise SystemExit(
            f"the text has only {len(words)} words; need at least {min_words + count}. "
            "Use a longer calibration text."
        )
    span = len(words) - min_words
    step = max(1, span // count)
    prefixes = []
    for i in range(count):
        end = min(len(words) - 1, min_words + i * step)
        prefixes.append(" ".join(words[:end]))
    return prefixes


def ask(base_url: str, model: str, prompt: str, top_k: int, api_key: Optional[str],
        timeout: int) -> dict:
    payload = {
        "model": model,
        "prompt": prompt,
        "max_tokens": 1,
        "temperature": 0.0,
        "logprobs": top_k,
        "top_logprobs": top_k,
        "seed": 0,
    }
    body = post_json(base_url.rstrip("/") + "/completions", payload, api_key, timeout)
    return extract_top_logprobs(body)


def to_probabilities(logprobs: dict, support: list) -> dict:
    """Renormalise a truncated distribution over a shared support."""
    floor = min(logprobs.values()) - UNSEEN_MARGIN
    raw = {token: math.exp(logprobs.get(token, floor)) for token in support}
    total = sum(raw.values()) or 1.0
    return {token: value / total for token, value in raw.items()}


def divergence(ref: dict, quant: dict) -> tuple:
    support = sorted(set(ref) | set(quant))
    p = to_probabilities(ref, support)
    q = to_probabilities(quant, support)
    kld = sum(p[t] * (math.log(p[t]) - math.log(q[t])) for t in support if p[t] > 0)
    top_ref = max(p, key=p.get)
    top_quant = max(q, key=q.get)
    return kld, top_ref == top_quant, p[top_ref] - q[top_ref]


def percentile(values: list, fraction: float) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    index = min(len(ordered) - 1, int(round(fraction * (len(ordered) - 1))))
    return ordered[index]


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--text", required=True, help="plain text to walk; not your evaluation set")
    parser.add_argument("--positions", type=int, default=64,
                        help="how many positions to measure; each costs one request per model")
    parser.add_argument("--min-words", type=int, default=32,
                        help="shortest prefix, so the first positions are not measured on nothing")
    parser.add_argument("--top-logprobs", type=int, default=20,
                        help="candidates requested per position; servers cap this, often at 20")
    parser.add_argument("--base-url-ref", default=None)
    parser.add_argument("--model-ref", default=None)
    parser.add_argument("--save-ref", default=None,
                        help="write the reference distributions here and stop, so the reference "
                             "and the quantised model can be served one after the other")
    parser.add_argument("--load-ref", default=None, help="read reference distributions from a file")
    parser.add_argument("--base-url-quant", default=None)
    parser.add_argument("--model-quant", default=None)
    parser.add_argument("--quant-label", default="unlabelled",
                        help="the quantisation's name, as it will appear in your results table")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--timeout", type=int, default=300)
    parser.add_argument("--out", default=None)
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    text = Path(args.text).read_text(encoding="utf-8", errors="replace")
    prefixes = make_prefixes(text, args.positions, args.min_words)

    # ---- reference -------------------------------------------------------
    if args.load_ref:
        stored = json.loads(Path(args.load_ref).read_text(encoding="utf-8"))
        if stored["prefix_hashes"] != [hash_prefix(p) for p in prefixes]:
            sys.exit("the stored reference was measured on different positions; regenerate it "
                     "with the same --text, --positions and --min-words")
        reference = stored["distributions"]
        ref_model = stored["model"]
    else:
        if not (args.base_url_ref and args.model_ref):
            sys.exit("give --base-url-ref and --model-ref, or --load-ref")
        print(f"==> Reference: {args.model_ref} at {args.base_url_ref}")
        reference = []
        for i, prefix in enumerate(prefixes, 1):
            reference.append(ask(args.base_url_ref, args.model_ref, prefix,
                                 args.top_logprobs, args.api_key, args.timeout))
            print(f"    {i}/{len(prefixes)}", end="\r", flush=True)
        print()
        ref_model = args.model_ref

    if args.save_ref:
        Path(args.save_ref).write_text(json.dumps({
            "model": ref_model,
            "text": args.text,
            "positions": args.positions,
            "min_words": args.min_words,
            "top_logprobs": args.top_logprobs,
            "prefix_hashes": [hash_prefix(p) for p in prefixes],
            "distributions": reference,
        }), encoding="utf-8")
        print(f"==> Saved the reference distributions to {args.save_ref}")
        if not (args.base_url_quant and args.model_quant):
            return

    # ---- the quantisation ------------------------------------------------
    if not (args.base_url_quant and args.model_quant):
        sys.exit("give --base-url-quant and --model-quant")
    print(f"==> Quantised: {args.model_quant} at {args.base_url_quant}")
    started = time.time()
    klds, agreements, deltas = [], [], []
    for i, prefix in enumerate(prefixes):
        quant = ask(args.base_url_quant, args.model_quant, prefix,
                    args.top_logprobs, args.api_key, args.timeout)
        kld, same_top, delta = divergence(reference[i], quant)
        klds.append(kld)
        agreements.append(same_top)
        deltas.append(delta)
        print(f"    {i + 1}/{len(prefixes)}", end="\r", flush=True)
    print()

    summary = {
        "lab": "part-16/lab-quantise-five-ways-and-measure/kld",
        "run_id": time.strftime("%Y%m%dT%H%M%S"),
        "method": "truncated-support KL over an OpenAI-compatible API; not comparable with "
                  "llama-perplexity --kl-divergence",
        "unseen_margin_nats": UNSEEN_MARGIN,
        "reference_model": ref_model,
        "quant_model": args.model_quant,
        "quant_label": args.quant_label,
        "text": Path(args.text).name,
        "positions": len(klds),
        "top_logprobs_requested": args.top_logprobs,
        "mean_kld": round(sum(klds) / len(klds), 6),
        "median_kld": round(percentile(klds, 0.5), 6),
        "p99_kld": round(percentile(klds, 0.99), 6),
        "max_kld": round(max(klds), 6),
        "same_top_token": round(sum(agreements) / len(agreements), 4),
        "mean_delta_p": round(sum(deltas) / len(deltas), 6),
        "rms_delta_p": round(math.sqrt(sum(d * d for d in deltas) / len(deltas)), 6),
        "max_abs_delta_p": round(max(abs(d) for d in deltas), 6),
        "seconds": round(time.time() - started, 1),
        "host": platform.platform(),
        "date": time.strftime("%Y-%m-%d"),
    }

    print(json.dumps(summary, indent=2))
    print("\nRead p99_kld and same_top_token before mean_kld. A quantisation that agrees with the "
          "\noriginal almost everywhere and diverges sharply at a few positions has a small mean "
          "\nand a large tail, and it is the tail that a reader sees.")

    if args.out:
        Path(args.out).write_text(json.dumps(
            {"summary": summary, "kld_per_position": klds,
             "same_top_per_position": agreements}, indent=2), encoding="utf-8")
        print(f"written to {args.out}")
    if args.labbook:
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(summary) + "\n")
        print(f"recorded in {args.labbook}")


def hash_prefix(prefix: str) -> str:
    """A short, stable fingerprint so a saved reference cannot be paired with different prompts."""
    import hashlib
    return hashlib.sha256(prefix.encode("utf-8")).hexdigest()[:16]


if __name__ == "__main__":
    main()

Download kl-divergence.py284 lines

The script walks a fixed text, asks the reference and the quantised model for the next-token distribution at the same positions, and reports how far apart they are. Nothing in it samples, so two runs against the same pair of servers should agree closely. Confirm that before you believe any difference between two files.

Start the reference and run it twice against one quantisation, then compare the two summaries. The gap between them is your noise floor, and any difference smaller than it in the table below is not a difference you have measured.

∴Research note

Do this even though it looks like a detour. Two quantisations whose mean divergences differ in the fourth decimal place are indistinguishable, and the only way to know that the fourth decimal place is noise is to have looked. Part 13’s challenge makes the same argument about fine-tuning, for the same reason.

7. Measure every file the same way

RunnableAll tracks

measure-quants.sh

#!/usr/bin/env bash
# Purpose: measure every quantisation in a directory the same way - speed and memory with
#          llama-bench, distributional damage with kl-divergence.py against the full-precision
#          reference, and task behaviour with the Part 10 harness - writing one result file per
#          quantisation so that measure-quants.py can assemble the comparison table
# Platform: all (spark, strix, mac, nvidia); GGUF rows only, one server at a time
# Minimum memory: 12 GB. The reference and the quantisations are served one after the other, not
#          together, so the floor is set by the largest single file plus its KV cache
# Assumes: llama-bench and llama-server on PATH or under LLAMA_BIN, curl, python3,
#          kl-divergence.py next to this script, and Part 10's run-eval.py and task file
#          reachable through EVAL_DIR and TASKS
#
# Usage: bash measure-quants.sh QUANT_DIR REFERENCE_GGUF
#   QUANT_DIR       directory of quantised GGUF files from quantise-five-ways.sh
#   REFERENCE_GGUF  the full-precision file every measurement is compared against
#
# Environment: LLAMA_BIN, PORT (default 8099), CTX (default 4096), NGL (default 99),
#              TEXT (calibration text for the divergence walk), POSITIONS (default 64),
#              EVAL_DIR (Part 10 lab directory), TASKS (Part 10 task file),
#              RESULTS (default results/), LABBOOK (default labbook.md), SKIP_EVAL=1
set -euo pipefail

QUANT_DIR="${1:-}"
REFERENCE="${2:-}"
PORT="${PORT:-8099}"
CTX="${CTX:-4096}"
NGL="${NGL:-99}"
POSITIONS="${POSITIONS:-64}"
RESULTS="${RESULTS:-results}"
LABBOOK="${LABBOOK:-labbook.md}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SERVER_PID=""

die() { echo "measure-quants: $*" >&2; exit 1; }

cleanup() {
  if [[ -n "$SERVER_PID" ]] && kill -0 "$SERVER_PID" 2>/dev/null; then
    kill "$SERVER_PID" 2>/dev/null || true
    wait "$SERVER_PID" 2>/dev/null || true
  fi
}
trap cleanup EXIT

[[ -n "$QUANT_DIR" && -n "$REFERENCE" ]] || die "usage: bash measure-quants.sh QUANT_DIR REFERENCE_GGUF"
[[ -d "$QUANT_DIR" ]] || die "$QUANT_DIR is not a directory"
[[ -f "$REFERENCE" ]] || die "$REFERENCE does not exist"
[[ -f "$HERE/kl-divergence.py" ]] || die "kl-divergence.py is not next to this script"
command -v curl >/dev/null || die "curl is not on PATH"
command -v python3 >/dev/null || die "python3 is not on PATH"

BENCH_BIN="${LLAMA_BIN:-}/llama-bench"
SERVER_BIN="${LLAMA_BIN:-}/llama-server"
if [[ -z "${LLAMA_BIN:-}" ]]; then
  BENCH_BIN="$(command -v llama-bench || true)"
  SERVER_BIN="$(command -v llama-server || true)"
fi
[[ -x "$BENCH_BIN" ]] || die "llama-bench not found; set LLAMA_BIN (built in Part 6)"
[[ -x "$SERVER_BIN" ]] || die "llama-server not found; set LLAMA_BIN (built in Part 6)"

TEXT="${TEXT:-}"
[[ -n "$TEXT" && -f "$TEXT" ]] || die "set TEXT to a plain-text file for the divergence walk (not your evaluation set)"

mkdir -p "$RESULTS"
BASE_URL="http://127.0.0.1:$PORT/v1"

start_server() {
  local model="$1" alias="$2"
  echo "    starting llama-server on port $PORT"
  "$SERVER_BIN" \
    --model "$model" \
    --alias "$alias" \
    --host 127.0.0.1 \
    --port "$PORT" \
    --ctx-size "$CTX" \
    --n-gpu-layers "$NGL" \
    --seed 0 \
    >"$RESULTS/server-$alias.log" 2>&1 &
  SERVER_PID=$!
  for _ in $(seq 1 180); do
    if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
      return 0
    fi
    if ! kill -0 "$SERVER_PID" 2>/dev/null; then
      die "llama-server exited while loading $model; see $RESULTS/server-$alias.log"
    fi
    sleep 1
  done
  die "llama-server did not become healthy within 180s; see $RESULTS/server-$alias.log"
}

stop_server() {
  cleanup
  SERVER_PID=""
  sleep 2
}

run_task_set() {
  local alias="$1" label="$2"
  if [[ "${SKIP_EVAL:-0}" == "1" ]]; then
    echo "    task set skipped (SKIP_EVAL=1)"
    return 0
  fi
  if [[ -z "${EVAL_DIR:-}" || ! -f "${EVAL_DIR:-}/run-eval.py" ]]; then
    echo "    task set skipped: set EVAL_DIR to your Part 10 lab directory to include it"
    return 0
  fi
  local tasks="${TASKS:-$EVAL_DIR/tasks-template.json}"
  [[ -f "$tasks" ]] || die "task file $tasks does not exist"
  python3 "$EVAL_DIR/run-eval.py" \
    --base-url "$BASE_URL" \
    --model "$alias" \
    --quant "$label" \
    --engine llama.cpp \
    --tasks "$tasks" \
    --out "$RESULTS/eval-$label.json" \
    --notes "part-16 quantisation comparison"
}

# ---------------------------------------------------------------------------
# Pass one: the reference. Its distributions are saved to disk so that the
# quantisations can be measured against them without both models being resident.
# ---------------------------------------------------------------------------
echo "==> Reference: $(basename "$REFERENCE")"
start_server "$REFERENCE" reference
python3 "$HERE/kl-divergence.py" \
  --text "$TEXT" \
  --positions "$POSITIONS" \
  --base-url-ref "$BASE_URL" \
  --model-ref reference \
  --save-ref "$RESULTS/reference-logprobs.json"
run_task_set reference BF16
stop_server

# ---------------------------------------------------------------------------
# Pass two: every quantisation, one at a time, measured identically.
# ---------------------------------------------------------------------------
shopt -s nullglob
QUANTS=("$QUANT_DIR"/*.gguf)
shopt -u nullglob
[[ ${#QUANTS[@]} -gt 0 ]] || die "no .gguf files in $QUANT_DIR"

for quant in "${QUANTS[@]}"; do
  if [[ "$(cd "$(dirname "$quant")" && pwd)/$(basename "$quant")" == "$(cd "$(dirname "$REFERENCE")" && pwd)/$(basename "$REFERENCE")" ]]; then
    continue
  fi
  label="$(basename "$quant" .gguf)"
  echo
  echo "==> $label"

  echo "    llama-bench: prefill and decode, three repetitions"
  "$BENCH_BIN" \
    --model "$quant" \
    --n-prompt 512 \
    --n-gen 128 \
    --repetitions 3 \
    --n-gpu-layers "$NGL" \
    --output json \
    >"$RESULTS/bench-$label.json"

  start_server "$quant" quant
  echo "    kl-divergence against the saved reference distributions"
  python3 "$HERE/kl-divergence.py" \
    --text "$TEXT" \
    --positions "$POSITIONS" \
    --load-ref "$RESULTS/reference-logprobs.json" \
    --base-url-quant "$BASE_URL" \
    --model-quant quant \
    --quant-label "$label" \
    --out "$RESULTS/kld-$label.json" \
    --labbook "$LABBOOK"
  run_task_set quant "$label"
  stop_server
done

echo
echo "==> Every quantisation measured. Results in $RESULTS/"
echo "    Assemble the table with:"
echo "      python3 $HERE/measure-quants.py --results $RESULTS --labbook $LABBOOK"

Download measure-quants.sh178 lines

The script does three things per file. It runs llama-bench for prefill and decode rates with three repetitions. It serves the file with llama-server, walks the divergence against the saved reference distributions, and runs your Part 10 task set through the same endpoint. Then it stops the server before starting the next one, so the peak memory is one model rather than two.

RunnableAll tracks

measure the whole directory

export EVAL_DIR=~/labs/part-10-models-at-work
export TASKS=~/labs/part-10-models-at-work/my-tasks.json
TEXT=calibration.txt bash measure-quants.sh "$WORK" "$WORK"/*-BF16.gguf

Output — what you should see

==> Reference: format-qwen3-4b-merged-BF16.gguf
    starting llama-server on port 8099
==> Quantised: reference at http://127.0.0.1:8099/v1
==> Saved the reference distributions to results/reference-logprobs.json
…
==> format-qwen3-4b-merged-Q4_K_M-imat
    llama-bench: prefill and decode, three repetitions
    starting llama-server on port 8099
    kl-divergence against the saved reference distributions

The reference model’s distributions are saved to a file and the reference server is then stopped, so the divergence for every quantisation is computed against exactly the same numbers without holding two models in memory. That is also why the walk uses a fixed text and a fixed set of positions: two runs measure the same places, or they measure nothing.

✱Hardware note

On Track M, and for the AWQ row on Tracks S and N, the fifth file is not a GGUF and measure-quants.sh will not find it. Serve it yourself with the engine that loads it, on the same port, and run kl-divergence.py by hand with --load-ref results/reference-logprobs.json and --quant-label set to the name you want in the table. The script writes its result into the same directory and the table picks it up.

8. Assemble the table, and let it choose

RunnableAll tracks

measure-quants.py

#!/usr/bin/env python3
"""Assemble the per-quantisation results into the course comparison table, and pick a file.

Purpose: read everything measure-quants.sh wrote - the divergence summaries, the llama-bench
    output and the Part 10 task-set results - into one table with one row per quantisation, then
    apply the two thresholds you set (a divergence budget and a memory budget) and name the
    fastest file that meets both. The recommendation is arithmetic on your own numbers, not an
    opinion: change the thresholds and it changes.
Platform: all (pure Python, standard library only)
Minimum memory: negligible
Assumes: Python 3.9 or later and a results directory written by measure-quants.sh containing
    kld-*.json files, and optionally bench-*.json and eval-*.json for the speed and task columns.

Usage: python3 measure-quants.py --results results --labbook labbook.md
       python3 measure-quants.py --results results --quant-dir ~/models/gguf/five-ways \
           --max-kld 0.02 --budget-gb 8 --labbook labbook.md
"""

from __future__ import annotations

import argparse
import json
import platform
import time
from pathlib import Path
from typing import Optional

DASH = "—"


def load(path: Path) -> Optional[dict]:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (OSError, ValueError):
        return None


def bench_rates(payload) -> tuple:
    """Pull prefill and decode rates out of llama-bench's JSON, without assuming field names.

    llama-bench reports one record per test. A record with n_gen == 0 is the prefill test and one
    with n_prompt == 0 is the generation test; the throughput field is avg_ts in the versions this
    course was written against. Anything unrecognised comes back as None rather than as a guess.
    """
    if not isinstance(payload, list):
        return None, None
    prefill = decode = None
    for record in payload:
        if not isinstance(record, dict):
            continue
        rate = record.get("avg_ts")
        if rate is None:
            rate = next((v for k, v in record.items()
                         if k.endswith("_ts") and isinstance(v, (int, float))), None)
        if rate is None:
            continue
        if record.get("n_gen") in (0, None) and record.get("n_prompt"):
            prefill = float(rate)
        elif record.get("n_gen"):
            decode = float(rate)
    return prefill, decode


def fmt(value, digits: int = 3) -> str:
    if value is None:
        return DASH
    if isinstance(value, float):
        return f"{value:.{digits}f}"
    return str(value)


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--results", default="results", help="directory written by measure-quants.sh")
    parser.add_argument("--quant-dir", default=None, help="directory of GGUF files, for file sizes")
    parser.add_argument("--reference", default=None,
                        help="the full-precision GGUF, so the reference row carries its size")
    parser.add_argument("--max-kld", type=float, default=None,
                        help="the largest mean divergence you are willing to accept; rows above it "
                             "are excluded from the recommendation")
    parser.add_argument("--min-same-top", type=float, default=None,
                        help="the smallest acceptable share of positions where the quantised model "
                             "still puts the same token on top")
    parser.add_argument("--budget-gb", type=float, default=None,
                        help="memory you can spend on weights; rows larger than this are excluded")
    parser.add_argument("--out", default=None, help="write the assembled table as JSON")
    parser.add_argument("--labbook", default=None)
    args = parser.parse_args()

    results = Path(args.results)
    if not results.is_dir():
        raise SystemExit(f"{results} is not a directory; run measure-quants.sh first")

    sizes = {}
    if args.quant_dir:
        for path in Path(args.quant_dir).glob("*.gguf"):
            sizes[path.stem] = path.stat().st_size / 1e9

    reference_gb = None
    if args.reference:
        reference_path = Path(args.reference)
        if reference_path.is_file():
            reference_gb = round(reference_path.stat().st_size / 1e9, 2)

    reference_checks = reference_tasks = None
    reference_eval = load(results / "eval-BF16.json")
    if reference_eval and "run" in reference_eval:
        reference_checks = reference_eval["run"].get("checks_passed")
        reference_tasks = reference_eval["run"].get("tasks")

    rows = []
    for kld_file in sorted(results.glob("kld-*.json")):
        label = kld_file.stem[len("kld-"):]
        payload = load(kld_file) or {}
        summary = payload.get("summary", {})

        prefill, decode = bench_rates(load(results / f"bench-{label}.json"))

        checks = judged = tasks = None
        evaluation = load(results / f"eval-{label}.json")
        if evaluation and "run" in evaluation:
            checks = evaluation["run"].get("checks_passed")
            tasks = evaluation["run"].get("tasks")
        judged_file = load(results / f"judged-{label}.json")
        if judged_file and "judge" in judged_file:
            judged = judged_file["judge"].get("judge_mean")

        rows.append({
            "quantisation": label,
            "file_gb": round(sizes[label], 2) if label in sizes else None,
            "mean_kld": summary.get("mean_kld"),
            "p99_kld": summary.get("p99_kld"),
            "same_top_token": summary.get("same_top_token"),
            "checks_passed": checks,
            "tasks": tasks,
            "judge_mean": judged,
            "prefill_tokens_per_s": prefill,
            "decode_tokens_per_s": decode,
        })

    if not rows:
        raise SystemExit(f"no kld-*.json files in {results}; measure-quants.sh writes them")

    header = ["Quantisation", "File GB", "Mean KLD", "99% KLD", "Same top",
              "Checks", "Judge", "pp512 t/s", "tg128 t/s"]
    print("\n| " + " | ".join(header) + " |")
    print("| " + " | ".join("---" for _ in header) + " |")
    reference_column = (f"{reference_checks}/{reference_tasks}"
                        if reference_checks is not None and reference_tasks else DASH)
    print(f"| BF16 (reference) | {fmt(reference_gb, 2)} | 0 | 0 | 1.0000 "
          f"| {reference_column} | {DASH} | {DASH} | {DASH} |")
    for row in sorted(rows, key=lambda r: (r["mean_kld"] is None, r["mean_kld"] or 0)):
        checks = (f"{row['checks_passed']}/{row['tasks']}"
                  if row["checks_passed"] is not None and row["tasks"] else DASH)
        print("| " + " | ".join([
            row["quantisation"],
            fmt(row["file_gb"], 2),
            fmt(row["mean_kld"], 5),
            fmt(row["p99_kld"], 5),
            fmt(row["same_top_token"], 4),
            checks,
            fmt(row["judge_mean"], 2),
            fmt(row["prefill_tokens_per_s"], 1),
            fmt(row["decode_tokens_per_s"], 1),
        ]) + " |")

    # ---- the choice ------------------------------------------------------
    eligible = list(rows)
    reasons = []
    if args.max_kld is not None:
        eligible = [r for r in eligible if r["mean_kld"] is not None and r["mean_kld"] <= args.max_kld]
        reasons.append(f"mean KLD at most {args.max_kld}")
    if args.min_same_top is not None:
        eligible = [r for r in eligible
                    if r["same_top_token"] is not None and r["same_top_token"] >= args.min_same_top]
        reasons.append(f"same top token at least {args.min_same_top}")
    if args.budget_gb is not None:
        eligible = [r for r in eligible if r["file_gb"] is not None and r["file_gb"] <= args.budget_gb]
        reasons.append(f"file at most {args.budget_gb} GB")

    choice = None
    if reasons:
        print(f"\nThresholds applied: {'; '.join(reasons)}.")
        with_speed = [r for r in eligible if r["decode_tokens_per_s"] is not None]
        if with_speed:
            choice = max(with_speed, key=lambda r: r["decode_tokens_per_s"])
            print(f"Fastest file meeting all of them: {choice['quantisation']}.")
        elif eligible:
            choice = min(eligible, key=lambda r: r["mean_kld"] or 0)
            print(f"No speed measurements; least divergent file meeting the thresholds: "
                  f"{choice['quantisation']}.")
        else:
            print("No file meets all the thresholds. Either the budget is too tight or the "
                  "quantisations are worse than you assumed; both are results worth recording.")
    else:
        print("\nNo thresholds given, so no recommendation. Pass --max-kld, --min-same-top and "
              "--budget-gb to turn the table into a decision.")

    record = {
        "lab": "part-16/lab-quantise-five-ways-and-measure/table",
        "run_id": time.strftime("%Y%m%dT%H%M%S"),
        "reference": {"file_gb": reference_gb, "checks_passed": reference_checks,
                      "tasks": reference_tasks},
        "rows": rows,
        "thresholds": {"max_kld": args.max_kld, "min_same_top": args.min_same_top,
                       "budget_gb": args.budget_gb},
        "choice": choice["quantisation"] if choice else None,
        "host": platform.platform(),
        "date": time.strftime("%Y-%m-%d"),
    }

    if args.out:
        Path(args.out).write_text(json.dumps(record, indent=2), encoding="utf-8")
        print(f"written to {args.out}")
    if args.labbook:
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print(f"recorded in {args.labbook}")

    print("\nBefore you act on the ordering: run one quantisation twice and compare the two "
          "\nrows. Any difference smaller than that gap is not a difference you have measured.")


if __name__ == "__main__":
    main()

Download measure-quants.py225 lines

RunnableAll tracks

the comparison, and a recommendation from your own thresholds

python3 measure-quants.py \
    --results results \
    --quant-dir "$WORK" \
    --reference "$WORK"/*-BF16.gguf \
    --min-same-top 0.95 \
    --budget-gb 6 \
    --labbook labbook.md

Set --budget-gb from your machine and your context length, using Part 6’s arithmetic: weights plus KV cache plus room for the buffers. Set --min-same-top from what you saw in step six: a threshold below your noise floor is not a threshold. The script prints the table, applies your two constraints and names the fastest file that satisfies both.

It is worth reading the row it did not pick. A file that fails the divergence threshold and is twice as fast is a genuine trade you might want to make for a task where the output is checked anyway; a file that passes everything and is slower than the one above it has no argument for it at all.

Keep every representation tied to the same source

Before conversion, record the high-precision source revision and the calibration-text checksum. Use the same source for every representation. Confirm disk space for intermediates and stop competing servers if the conversion or calibration step needs their memory.

After each quantisation, check that the expected file exists, has a plausible nonzero size and loads in its supported engine. A platform that cannot execute one method should record that method as not run; do not fill its quality or speed cells with numbers from another representation. Keep the matrix/calibration output with the recipe for the methods that require it.

Evaluate each loaded file with identical task IDs, template, prompt lengths and generation settings. Record file size, peak runtime memory, quality and latency separately. If one variant offloads while another fits, label the placement difference. Inspect paired task failures before choosing a format. The decision should satisfy the predeclared task tolerance and memory budget. Retain the chosen artefact, source identity, calibration provenance, conversion commands and all result rows before deleting intermediates. A smallest-file winner is not automatically the best deployment if it breaks a critical output or tool-call contract.

Validation

You are done when all of the following are true.

• Six files exist: the full-precision reference and five quantisations, including the two Q4_K_M files that differ only in the importance matrix.
• labbook.md contains the calibration file’s SHA-256 and its byte count, so the matrix is reproducible.
• results/ contains a kld-*.json for every quantisation and a bench-*.json for every GGUF.
• You have run one quantisation’s divergence measurement twice and can state your noise floor.
• The assembled table has a row per file with divergence, task-set result, speed and size.
• You can name the file you would serve, the two thresholds that chose it, and one row you rejected and why.
• The notebook has the table and the decision.

Expected outcome

A shape, rather than specific numbers, which your own table will make concrete.

Q8_0 should be close to the reference on every column, to the point where the interesting question about it is whether the extra gigabytes buy anything you can measure. Q5_K_M should sit between. The two Q4_K_M files should differ, and the direction is a real result either way: the importance matrix is a bet on your calibration text, and a matrix built from material that does not resemble your traffic can help less than nothing. The fifth file is the one with no prior at all, because it comes from a different family and, on two of the tracks, a different engine.

Pending validationFive quantisations of one model — your recording sheet

Quantisation	File GB	Mean KLD	99% KLD	Same top	Task checks	pp512 tokens/s	tg128 tokens/s
BF16 (reference)	—	0	0	1.0000	—	—	—
Q8_0	—	—	—	—	—	—	—
Q5_K_M	—	—	—	—	—	—	—
Q4_K_M, no imatrix	—	—	—	—	—	—	—
Q4_K_M, with imatrix	—	—	—	—	—	—	—
Fifth format, per track	—	—	—	—	—	—	—
Noise floor: one file, measured twice	—	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp for the GGUF rows; name the engine for the fifth row the build string from llama-cli --version · your Part 13 fine-tune, or the reference model you used instead, as listed in the first column · 4,096 tokens of context · the date you ran it

Empty on purpose. The last row is not a quantisation: it is the same file measured twice, and it is the row that tells you which of the differences above it are real. Record the calibration file's checksum alongside this table; without it the two four-bit rows cannot be reproduced.

=Expected result

If every quantisation shows a divergence of zero and perfect agreement on the top token, the measurement is not working: check that --load-ref is pointing at distributions from the reference model and not from the quantisation. If every quantisation including Q8_0 shows a large divergence, check that the reference file is the full-precision one and not a quantisation you converted by mistake.

Troubleshooting

The server returns no log-probabilities. kl-divergence.py asks for them explicitly and fails loudly when they are absent. Check that the endpoint path is /v1/completions rather than the chat endpoint, and that the server was not started with --log-disable. Some gateways strip the field; measure against the engine directly rather than through the gateway if so.

llama-imatrix is very slow. It is running the full-precision model over your whole calibration text. Pass -ngl 99 through the NGL environment variable so the work happens on the accelerator, and use CHUNKS to cap how much of the text is processed while you are still experimenting.

The importance-matrix quantisation is worse than the one without. That is a result, not a bug, and the usual cause is a calibration text that does not resemble the text you measured on. Rebuild the matrix from more representative material, or merge two matrices from different sources by passing --in-file twice, and record both attempts.

The AWQ step runs out of memory. It loads the model at bfloat16 and runs calibration forward passes. Reduce --num-calibration-samples and --max-seq-length before reducing the model size; both are documented arguments of the example this script follows. On 12 to 16 GB, take the second GGUF level instead, as the requirements say.

The MLX row’s numbers look different in kind from the GGUF rows. They are: a different engine rounded them and a different server produced the log-probabilities. Keep the row, and label it with its engine. Comparing it with the GGUF rows is legitimate as a deployment question and not as a statement about the quantisation method alone.

Two runs of the same measurement disagree more than two different quantisations do. Then you have measured your noise floor and it is larger than the effect. Increase --positions, make sure nothing else is using the machine, and if the floor stays high, report that the quantisations are indistinguishable on this measurement, which is a perfectly good finding.

Cleanup

Keep the results directory, the table and the notebook entry. Keep the importance matrix and the calibration text together: either is useless without the other.

The full-precision GGUF is the largest file and the last one to delete, because it is the reference every measurement in this part is computed against and it cannot be recovered from the quantisations.

⚠Warning

The command below deletes a directory tree. Check the path before running it, and do not run it until the table is filled in and the file you chose has been copied somewhere else. Deleting the wrong directory here costs the whole hour, and on Track S it costs the AWQ calibration pass as well.

RunnableAll tracks

remove the quantisations you rejected, once the table is recorded

ls -la "$WORK"
rm -rf "$WORK"

Stop any server that is still running, and free the port before the next lab.

What you learned

• Five files, one measurement procedure. The value is not any single number; it is that every row was produced the same way against the same reference, which is what makes the column comparable.
• The reference is the full-precision model, always. Comparing two quantisations tells you which is nearer the other. Only the original tells you which is nearer the truth.
• An importance matrix is a bet, and now you know how yours paid. Two files differing in one input, measured identically, is the cleanest experiment in this part.
• The tail matters more than the mean. A file with a small mean divergence and a large ninety-ninth percentile changes behaviour where behaviour is decided.
• The noise floor comes first. A difference smaller than the gap between two runs of the same measurement has not been observed.
• The choice is arithmetic once the thresholds are yours. A divergence budget and a memory budget turn a table into a decision, and changing either changes the answer honestly.

Record in the notebook: the model and where it came from, the calibration file’s checksum and size, the full table including the noise-floor row, the two thresholds you set, the file you chose, and one sentence on the row you rejected and why. The next part serves whichever file you chose, and the capstone quotes this table.

Check your understanding

Question 1. Why does this lab build the full-precision GGUF first and quantise from it, rather than converting straight to Q4_K_M?

• It is faster
• Quantising from full precision keeps information the quantiser can use; converting straight to a small type throws it away, and it also leaves no reference to measure against
• llama-quantize cannot read safetensors
• The file names would be wrong

Show the answer and why

Answer: Quantising from full precision keeps information the quantiser can use; converting straight to a small type throws it away, and it also leaves no reference to measure against

Two reasons at once. Double rounding costs quality, which is why Part 6 warned about --allow-requantize. And every measurement in this lab is a comparison against the unquantised model, which has to exist for that comparison to be possible.

Question 2. The lab builds two Q4_K_M files that differ only in whether an importance matrix was used. What question does that pair answer?

• Which four-bit type is smallest
• What the importance matrix built from your calibration text is worth on your model, measured rather than assumed
• Whether llama-quantize is deterministic
• How fast the machine is

Show the answer and why

Answer: What the importance matrix built from your calibration text is worth on your model, measured rather than assumed

Part 6 introduced "imatrix quants are better" as an untested claim for your workload. Two files from the same source at the same nominal width, differing in one input and measured identically, is the experiment that tests it. The answer can legitimately be that the matrix did not help, when the calibration text does not resemble the measurement text.

Question 3. Why does measure-quants.sh save the reference model's distributions to a file and then stop the reference server?

• To make the script shorter
• So each quantisation is compared against exactly the same numbers without two models being resident at once, which is what keeps the memory floor at one model
• Because llama-server cannot run twice
• To avoid network traffic

Show the answer and why

Answer: So each quantisation is compared against exactly the same numbers without two models being resident at once, which is what keeps the memory floor at one model

Both halves matter. Saving the reference makes every comparison use identical numbers rather than a re-measured approximation, and serving one model at a time is why a 12 GB machine can run this lab at all.

Question 4. Your table shows Q4_K_M with the matrix at a mean divergence of 0.0121 and without it at 0.0125, and two runs of the same file gave 0.0119 and 0.0127. What should you conclude?

• The matrix helped slightly
• Nothing: the gap between the two files is smaller than the gap between two runs of one file, so the difference has not been measured
• The measurement is broken
• The matrix hurt

Show the answer and why

Answer: Nothing: the gap between the two files is smaller than the gap between two runs of one file, so the difference has not been measured

That is what the noise floor is for. The repeat run spans 0.0008 and the difference between the files is 0.0004. Increase the number of positions, quieten the machine, or report the two files as indistinguishable on this measurement, which is an honest and useful result.

Question 5. Which of these belong in the notebook entry for this lab? Select all that apply.

• The SHA-256 and size of the calibration text
• The noise floor from measuring one file twice
• The two thresholds that selected the file you chose
• The engine and version behind each row

Show the answer and why

Answer: The SHA-256 and size of the calibration text, The noise floor from measuring one file twice, The two thresholds that selected the file you chose, The engine and version behind each row

All four. The checksum makes the importance matrix reproducible, the noise floor makes the differences interpretable, the thresholds make the choice reviewable, and the engine and version make the speed column mean something six months from now.

Check my answersClear