Post-Training Quantisation in Depth: GPTQ, AWQ, K-quants, imatrix and HQQ

Part 6 taught you to read Q4_K_M off a file name and decide whether it fits. This lesson is about what produced that file. By the end of it you will be able to say what a quantiser is choosing when it rounds a weight, why two methods at the same bit width can differ in quality, what a calibration set is a bet about, and which of the six or seven tools in current use you should reach for on your track and your deadline.

The reason to know this is not scholarship. It is that “four-bit” names a size, not a quality, and the difference between two four-bit files of the same model can be larger than the difference between two model families. Choosing between them is a measurement, which the third lesson of this part sets up, and an informed measurement needs to know what the candidates are doing differently.

The thing every method has in common

Strip away the names and every post-training quantiser does the same three things.

It groups the weights, usually along a row of a weight matrix, into blocks of some fixed size: 32, 64 or 128 values is typical. It fits a scale to each block, a single higher-precision number that says how big the values in this block are. And it rounds each weight to the nearest point on a small grid, so that the stored value is a small integer or a narrow float, and the original is approximately the grid point times the scale. Affine schemes add a zero-point per block as well, so the grid does not have to be centred on zero.

That is why the bits-per-weight figures in Part 6’s table are always higher than the name suggests. Q4_K_M stores four-bit grid indices, and then it also has to store the scales. Smaller blocks track the data better and cost more scale storage per weight; larger blocks are cheaper and blur more. Group size is the first dial, and 64 or 128 is where most tools sit.

!Important

The naive version of this is round-to-nearest: pick the scale from the block’s largest absolute value, round everything, done. It needs no data and no time, and at eight bits it is close enough that the argument stops. At four bits it is not, and every method in this lesson exists because of what round-to-nearest gives up there.

The error that actually matters

Here is the reframing that the whole field turns on.

Rounding a weight introduces an error in that weight. But nobody cares about the weight; they care about what the layer computes. A layer multiplies its input by the weight matrix, so the error that reaches the next layer is the weight error multiplied by the activation that meets it. A large rounding error on a weight that is always multiplied by something near zero does nothing. A small rounding error on a weight that meets a large activation on every token does a great deal.

So the interesting question is not “which weights are big” but “which weights, times the inputs this model actually sees, contribute most to the layer’s output”. Answering it needs to know what the inputs look like, which is why almost every method beyond round-to-nearest wants a calibration set: a few hundred sequences of text, pushed through the model, with statistics collected on the way.

What a post-training quantiser decides, in order

1. 1Group the weightsBlocks of 32, 64 or 128 along a row. Smaller blocks track the data better and spend more bits on scales.
2. 2Choose a gridInteger levels, a narrow float, or a codebook of vectors. This is what the bit width names.
3. 3Decide what to protectUniform treatment, or a per-channel importance signal computed from calibration activations. This is where GPTQ, AWQ and importance matrices differ.
4. 4Fit the scales, and compensatePick a scale per block; optionally adjust the not-yet-quantised weights to cancel the error already made. GPTQ does the second part; round-to-nearest does not.
5. 5Write the fileIndices, scales, zero-points and a record of which tensor got which type. This is the part the file format standardises.

Methods differ almost entirely in steps three and four. Everything else is bookkeeping that all of them share.

GPTQ: quantise one column, then fix the rest

GPTQ works one linear layer at a time and asks a narrow question: given this layer’s inputs from the calibration set, what quantised weights reproduce this layer’s outputs most closely?

It answers it greedily. Quantise one column of the weight matrix, measure the error that introduces in the layer’s output, and then update the columns not yet quantised so that they absorb as much of that error as they can. The update uses approximate second-order information about the layer’s loss surface, which is where the cost and the accuracy both come from. The paper describes GPTQ as “a new one-shot weight quantization method based on approximate second-order information” that can “quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation”, and reports that it extends to “2-bit or even ternary quantization levels”.

Two consequences are worth carrying away. GPTQ changes the surviving weights, not just their representation, which is why it needs a forward pass over calibration data and why the result is tied to that data. And because it optimises reconstruction of the calibration activations, it can in principle fit them too well: a calibration set that looks nothing like your workload buys you a model tuned for the wrong distribution.

AWQ: scale the important channels up before rounding them

AWQ starts from a different observation. Its paper states that “not all weights in an LLM are equally important” and that “protecting only 1% salient weights can greatly reduce quantization error”, with the crucial detail that “to identify salient weight channels, we should refer to the activation distribution, not weights”.

Having found the salient channels from activation statistics, AWQ does not store them at higher precision, which would break the neat memory layout that makes fast kernels possible. It applies “an equivalent transformation to scale the salient weight channels to protect them”, multiplying those channels up and dividing the corresponding activations down by the same factor, so the layer computes the same function. A weight that has been scaled up occupies more of its block’s grid, so rounding it costs proportionally less.

The property that matters in practice is stated directly in the abstract: “AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set.” Compared with GPTQ, AWQ is using calibration data to find a per-channel scale, not to fit the weights themselves, which makes it noticeably less fussy about what you calibrate on. That is why this part’s lab reaches for it when a non-GGUF four-bit file is needed on Tracks S and N.

Weight-only and weight-and-activation

Everything so far quantises weights and leaves activations at sixteen bits. That configuration is written W4A16: four-bit weights, sixteen-bit activations. It is the right default for local single-user inference, because decode is bandwidth-bound, the weights are what has to be read for every token, and shrinking them is exactly the lever that helps.

W8A8 quantises the activations too, and the reason to want it is different: an eight-bit matrix multiplication runs on hardware paths that a sixteen-bit one does not, so it raises the compute ceiling for prefill and for batched serving. The obstacle is that activations are much harder to quantise than weights, because a handful of channels carry values far larger than the rest and a per-tensor scale sized for them wastes the grid on everything else.

SmoothQuant is the standard answer. Its paper describes it as a “training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution” that enables “8-bit weight, 8-bit activation (W8A8) quantization for LLMs”, built on the observation that “weights are easy to quantize while activations are not”. It “smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation” — the same divide-here-multiply-there trick AWQ uses, aimed at a different problem.

?Reality check

A claim you will meet is that W8A8 is faster than W4A16 and therefore better. Both halves need a condition attached. On a single-user local machine generating one token at a time, the binding constraint is bytes read per token, and four-bit weights win on that regardless of what the arithmetic units could have done. On a server saturated with concurrent requests the arithmetic starts to bind and the ranking can reverse. Part 9 measured which regime you are in; this is where that measurement pays.

llama.cpp: a mixture policy, plus a signal

llama.cpp’s K-quants are often described as if they were an algorithm competing with GPTQ. They are better understood as a policy layered on a simple quantiser.

The quantiser itself is block-wise round-to-nearest with per-block scales, and in the IQ types a codebook rather than a plain grid. What the K-quant mixtures add is a rule for which tensor gets which type: attention and feed-forward tensors are assigned different widths, and the tensors whose damage propagates furthest are kept wider. That is what the _S, _M and _L suffixes select, and it is why --pure exists — it disables the mixtures so you can see what they were worth.

The importance signal comes separately, from llama-imatrix. It runs a calibration text through the unquantised model, collects activation statistics per tensor, and writes a file that llama-quantize reads with --imatrix. The README’s documented options include -m, -f, -o, --output-format, --output-frequency, --save-frequency, --process-output, --in-file, --parse-special, --chunk, --from-chunk, --chunks, --no-ppl, --show-statistics and -ngl; the last of those is the difference between minutes and an afternoon.

Fragment — not complete on its own

llama-imatrix \
    -m models/gguf/model-BF16.gguf \
    -f calibration.txt \
    --output-frequency 20 \
    --save-frequency 50 \
    -ngl 99 \
    -o imatrix.gguf

Three features of that tool are worth knowing before the lab. --in-file can be given more than once to merge matrices built from different calibration texts, which is how you hedge the bet described below. --show-statistics reads a finished matrix back and reports per-tensor activation statistics, so you can see which layers the text actually exercised. And --output-format chooses between the current GGUF form and the legacy dat form that older tooling expects.

✕Common mistake

Calibrating on the material you intend to evaluate on. An importance matrix built from your benchmark’s own prompts produces a model tuned to score well on that benchmark, and a number you can no longer interpret. Part 13’s decontaminate.py exists for the training version of this mistake; the quantisation version is the same mistake with fewer steps. Calibrate on text that resembles your workload and is disjoint from anything you will score on.

Calibration data is an assumption, written down

Every method in this lesson except one takes a calibration set, and the calibration set encodes a belief about what the model will be asked to do. A matrix built from English Wikipedia is a bet on English prose. A matrix built from your own logs is a bet on your own logs, which is usually the better bet and occasionally a much worse one when your logs are narrower than your future.

Practical guidance that follows, none of it exotic: use a few hundred sequences rather than a few, at a length close to what you actually send; mix in some of everything the model will see, including the chat template’s own scaffolding if you serve chat; keep the file, because a quantisation whose calibration text has been lost cannot be reproduced; and where a tool lets you merge matrices, build two from different material and merge them rather than agonising over one.

HQQ: no calibration at all

HQQ takes the opposite position. Its repository describes it as “a fast and accurate model quantizer that skips the need for calibration data”, with the claim that you can “quantize the largest models, without calibration data, in just a few minutes at most”. Rather than reconstructing activations, it treats the choice of scale and zero-point as an optimisation over the weight error itself, solved with a formulation robust to the outliers that make round-to-nearest fail.

The README states that “it supports 8,4,3,2,1 bits”, places no restriction on group size “as long as weight.numel() is divisible by the group_size”, and recommends starting at four bits with a group size of 64.

Read that list of properties as a set of trade-offs rather than a verdict. No calibration means nothing to overfit, no data to source and no licence question about the calibration corpus, and it means minutes instead of hours, which matters when you are quantising a fine-tune you retrain weekly. It also means the method has no information about which weights meet large activations, so where an activation-aware method has real signal to exploit, HQQ is declining to use it. Which of those dominates on your model is, as ever, a measurement.

MLX, on Track M

MLX quantises with an affine group scheme and exposes it through mlx_lm.convert. The script’s arguments include --hf-path for the source, --mlx-path for the destination, -q to quantise at all, --q-bits for bits per weight, --q-group-size for the group size, --q-mode and --quant-predicate.

RunnableTrack M · Apple silicon

a four-bit MLX conversion with an explicit group size

mlx_lm.convert \
    --hf-path ~/models/Qwen/Qwen3-4B \
    --mlx-path ~/models/mlx/Qwen3-4B-4bit-g64 \
    -q \
    --q-bits 4 \
    --q-group-size 64

Two of those arguments are more interesting than they look. --quant-predicate selects a mixed-bit recipe — the script offers mixed_2_6, mixed_3_4, mixed_3_6 and mixed_4_6 — which is MLX’s equivalent of the K-quant mixture idea: different tensors at different widths within one file. --q-mode chooses the grid itself, with affine as the default and mxfp4, nvfp4 and mxfp8 as alternatives, which is the subject of the next lesson.

The tools, and their state on 9 September 2026

Which tool produces which kind of file

from source weights to a served file

1. Source checkpointSafetensors at BF16, from the publisher or from your own fine-tune. The reference every measurement in this part compares against.
2. Transformers-native quantisersllm-compressor, GPTQModel and HQQ produce quantised safetensors that vLLM and transformers load. This is the path for AWQ, GPTQ and W8A8.
3. llama.cppconvert_hf_to_gguf.py then llama-quantize, with llama-imatrix supplying the importance signal. Produces GGUF for every engine built on llama.cpp.
4. MLXmlx_lm.convert with -q, --q-bits and --q-group-size. Produces an MLX model directory for Apple silicon.
5. The served fileWhat your gateway loads, and what every measurement in the next three lessons is made against.

The layer you work at decides the tool. Nothing here converts between the bottom two rows: a GGUF and an AWQ checkpoint are different artefacts made by different programs, and the lab in this part makes both.

Tool	What it does	Status, checked 2026-09-09
llama.cpp v0.4.0 · verified 2026-09-08	llama-quantize, llama-imatrix: GGUF K-quants and IQ types	Current; the option lists above are from the pinned READMEs
mlx-lm 0.31.3 · verified 2026-09-08	mlx_lm.convert -q: affine, mixed-bit and microscaling modes	Current; Track M’s only first-party path
LLM Compressor	GPTQ, AWQ, SmoothQuant, RTN and more, aimed at vLLM	Documents formats including W4A16, W8A16, W8A8-INT8, W8A8-FP8, MXFP8 and NVFP4/MXFP4; its documentation states that 2:4 sparsity “is no longer supported”
GPTQModel	GPTQ, AWQ and several newer methods for transformers, vLLM and SGLang	Its README states it “has fully supplanted AutoGPTQ and AutoAWQ for HF Transformers/Optimum/Peft integration”
AutoAWQ	The original AWQ implementation	Its README states AutoAWQ “is officially deprecated and will no longer be maintained”; use llm-compressor or GPTQModel
HQQ	Calibration-free quantisation at 8, 4, 3, 2 and 1 bits	Current; the fastest route to a quantised model when you have no calibration data

⌚Version note

The two deprecations in that table are the reason to date it. AutoAWQ is the name that appears in almost every tutorial written before 2025 and in a great many still-linked blog posts, and following one of them today leads you to install an unmaintained package. Check a quantisation tool’s own README before you follow anybody’s recipe for it, including this one, and note the date you checked.

Calibration is a dataset design choice

A calibration set estimates which activations or directions matter for preserving the model’s computation. If it contains only short conversational English, it may poorly represent long code, multilingual documents or structured extraction. Keep calibration data separate from the final evaluation set and record its source, length distribution and preprocessing.

Use the same high-precision checkpoint when comparing methods. Preserve group size, scaling choices, excluded modules and the quantiser version. Two outputs described as four-bit can have different metadata overhead, mixed-precision tensors and kernel requirements.

Evaluate the quantised model on both representative and difficult slices. A small average distributional change can conceal a large effect on a rare token or a strict output contract. If a method performs well offline but lacks an efficient kernel on your device, it may not improve the deployed service. The selection therefore joins numerical error, task quality and execution support. Calibration quality cannot compensate for a loader that does not implement the resulting representation.

Recap

Every post-training quantiser groups weights, fits a scale per group and rounds onto a small grid; the differences are in what it protects and how it compensates. The error that matters is the layer’s output error, not the weight’s, which is why activations decide importance and why almost every method wants calibration data. GPTQ reconstructs a layer’s outputs column by column with second-order updates, which is accurate and tied to its calibration set. AWQ finds salient channels from activation statistics and scales them before rounding, without backpropagation or reconstruction, which its authors report generalises better across domains. Weight-only quantisation is the right default for single-user decode; W8A8 with SmoothQuant targets the compute ceiling that batched serving hits instead. llama.cpp’s K-quants are a per-tensor mixture policy plus an optional importance matrix, not a competing optimiser. HQQ declines calibration entirely and buys minutes instead of hours. MLX offers the same knobs under different names, plus mixed-bit recipes and the microscaling grids of the next lesson. And the calibration set is an assumption about your workload that you should choose deliberately, keep, and never draw from the material you intend to score on.

Check your understanding

Question 1. Why do quantisation methods look at activations when deciding which weights to protect?

• Because activations are stored in the same file as the weights
• Because the error that reaches the next layer is the weight error multiplied by the activation it meets, so a weight that always meets a large activation matters more
• Because activations are easier to compress than weights
• Because the calibration set contains the correct answers

Show the answer and why

Answer: Because the error that reaches the next layer is the weight error multiplied by the activation it meets, so a weight that always meets a large activation matters more

A weight error only matters through what it multiplies. AWQ makes this explicit: its paper says that to identify salient weight channels you should refer to the activation distribution, not the weights. This is also what an importance matrix records for llama.cpp.

Question 2. What distinguishes GPTQ from AWQ, in one sentence?

• GPTQ is four-bit and AWQ is eight-bit
• GPTQ reconstructs each layer's outputs by updating the not-yet-quantised weights, while AWQ uses activation statistics to choose a per-channel scale and does no backpropagation or reconstruction
• AWQ needs no calibration data at all
• GPTQ produces GGUF files and AWQ produces safetensors

Show the answer and why

Answer: GPTQ reconstructs each layer's outputs by updating the not-yet-quantised weights, while AWQ uses activation statistics to choose a per-channel scale and does no backpropagation or reconstruction

Both are weight-only and both use calibration data. GPTQ fits the calibration activations, which is powerful and can overfit them; AWQ derives a scale, which its abstract says lets it generalise across domains and modalities without overfitting the calibration set. HQQ is the one that needs no calibration data.

Question 3. You are running one model, one request at a time, on a local machine, and generation speed is what you care about. Which configuration follows from that?

• W8A8, because eight-bit arithmetic is faster
• W4A16, because decode reads every weight for every token and the weights are what shrinking helps
• Sixteen-bit weights, because quality is what matters
• It makes no difference which you choose

Show the answer and why

Answer: W4A16, because decode reads every weight for every token and the weights are what shrinking helps

Decode is bandwidth-bound: the binding constraint is bytes of weights read per token. W8A8 raises the arithmetic ceiling, which is what a batched server hits. Part 9 taught you to identify which regime you are in, and the answer differs between a desktop and a gateway serving twenty users.

Question 4. Which statements about llama.cpp's K-quants are accurate? Select all that apply.

• The _S, _M and _L suffixes select how generous the per-tensor mixture is at the same nominal width
• A K-quant file assigns different types to different tensors
• K-quants are an optimiser that competes with GPTQ for choosing scales
• --pure disables the mixture so a single type is used throughout

Show the answer and why

Answer: The _S, _M and _L suffixes select how generous the per-tensor mixture is at the same nominal width, A K-quant file assigns different types to different tensors, --pure disables the mixture so a single type is used throughout

K-quants are a mixture policy over a simple block quantiser, not a reconstruction algorithm. The importance signal that plays GPTQ's and AWQ's role in llama.cpp is supplied separately by llama-imatrix and consumed with --imatrix.

Question 5. You have a fine-tune that you retrain every week and no calibration corpus you trust. Which tool fits that constraint best, and why?

• GPTQ, because it is the most accurate method published
• HQQ, because its README describes it as skipping the need for calibration data and quantising large models in minutes
• AutoAWQ, because it is the original AWQ implementation
• Round-to-nearest at sixteen bits

Show the answer and why

Answer: HQQ, because its README describes it as skipping the need for calibration data and quantising large models in minutes

A weekly retrain makes quantisation time and calibration sourcing recurring costs, which is exactly what a calibration-free method removes. AutoAWQ is deprecated as of the date on this page. Whether HQQ's output is good enough on your task is still a measurement, which the third lesson of this part sets up.

Check my answersClear