Quantisation-Aware Training and the Blackwell Formats: FP8, NVFP4 and MXFP4
The previous lesson took a finished model and rounded it as carefully as possible. This one changes the order of operations: it trains the model while the rounding is happening, so that the weights it arrives at are weights that survive being rounded. Then it looks at the four-bit and eight-bit formats that current hardware computes with directly, because the names FP8, NVFP4 and MXFP4 are specific layouts of bits, and once you can read them the differences between them stop being mysterious.
By the end you will be able to say what quantisation-aware training does to a training loop, recognise a published QAT checkpoint and what its publisher is claiming, describe FP8, NVFP4 and MXFP4 in terms of element format, block size and scale format, and say which of those your track can compute with natively rather than merely store.
Rounding inside the loop
Section titled “Rounding inside the loop”Post-training quantisation asks: given these weights, what is the least damaging way to round them? Quantisation-aware training asks a better question: given that these weights will be rounded, what weights should we have trained in the first place?
The mechanism is called fake quantisation. During the forward pass, each weight is quantised and immediately dequantised before it is used, so the layer computes with values that are exactly on the quantisation grid, and the loss the model sees is the loss the quantised model will actually suffer. The gradients, however, flow back to a master copy of the weights kept at full precision, and the optimiser updates that copy. Rounding has no useful derivative, so the backward pass simply passes the gradient through the rounding step as if it were the identity, an approximation known as the straight-through estimator.
One step of quantisation-aware training
- Fake-quantise the weightsRound each weight onto the target grid and immediately expand it back. The forward pass now computes with values the deployed model can represent.
- Forward passOrdinary, on the rounded values. The loss measured is the loss the quantised model will have.
- Compute the lossAgainst real labels, or - as LLM-QAT and Gemma 3 both do - against the output distribution of the unquantised model.
- Backward passGradients flow through the rounding as if it were the identity: the straight-through estimator.
- Update the master weightsThe optimiser updates a full-precision copy, which is fake-quantised again on the next step. The master copy is discarded at the end.
The result is a checkpoint whose weights sit comfortably on the grid, so that the final, real quantisation costs much less than it would have on a model that had never seen it.
What the research established
Section titled “What the research established”LLM-QAT is the paper that made the case for language models specifically. Its abstract states that post-training methods “have been shown to perform well down to 8-bits”, that “these methods break down at lower bit precision”, and proposes “a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data”.
Two details from that sentence matter beyond the paper. First, the training signal is the original model’s own output distribution rather than a labelled dataset, which is the distillation idea from Part 15 applied to a student that is the same model in a smaller format. That removes the awkward requirement of having the original training data, which for most published models nobody outside the publisher has. Second, the paper quantises “weights and activations” and also “the KV cache, which is critical for increasing throughput and support long sequence dependencies”, and reports “large improvements over training-free methods, especially in the low-bit settings”.
Published QAT checkpoints, and how to read one
Section titled “Published QAT checkpoints, and how to read one”You are unlikely to run QAT yourself, for reasons the last section of this lesson makes concrete. You are quite likely to download a checkpoint somebody else ran it on, and the Gemma 3 QAT release is the clearest example to learn to read.
Google’s announcement states: “We applied QAT on ~5,000 steps using probabilities from the non-quantized checkpoint as targets.” That is exactly the LLM-QAT recipe: a short fine-tune, on the unquantised model’s own output distribution, with rounding in the loop. The quality claim is narrower and better stated than most: “We reduce the perplexity drop by 54% (using llama.cpp perplexity evaluation) when quantizing down to Q4_0.”
Read that claim carefully, because its precision is what makes it useful. It is a relative
reduction in the perplexity drop, not an absolute quality figure; it names the evaluation tool;
and it names the target format. It does not say the QAT model equals the BF16 model, and the model
card’s own phrasing is the softer “the model is able to preserve similar quality as bfloat16 while
significantly reducing the memory requirements to load the model”. A publisher who tells you which
tool produced the number and which format it applies to is a publisher whose number you can attempt
to reproduce, which is what the second lab of this part is for.
| Model | BF16 memory, GB | int4 memory, GB |
|---|---|---|
| Gemma 3 27B | 54 | 14.1 |
| Gemma 3 12B | 24 | 6.6 |
not stated by the source; figures are the memory needed to load the weights · llama.cpp (the source names llama.cpp perplexity evaluation for its quality claim) not stated by the source · Gemma 3 27B and 12B, instruction-tuned QAT checkpoints, Q4_0 / int4 versus BF16 · 0 tokens of context · 2026-09-09
Reported by Google in the Gemma 3 QAT announcement, read on 2026-09-09. Context length is recorded as zero because these are weight-loading figures only; the KV cache is a separate budget on top, as Part 6 showed. The course has not reproduced either figure.
The licence on the Gemma QAT repositories is Google’s own Gemma licence rather than Apache or MIT, and should be read on the model card before the weights go anywhere near a product.
FP8, NVFP4 and MXFP4, bit by bit
Section titled “FP8, NVFP4 and MXFP4, bit by bit”These three names describe how a group of numbers is stored, and each is fully specified by three things: what each element looks like, how many elements share a scale, and what the scale itself looks like.
FP8 is a single byte per value, in one of two shapes named after how they spend their bits: E4M3 takes four bits of exponent and three of mantissa, E5M2 takes five and two. E4M3 is the usual choice for weights and activations because it keeps more precision; E5M2’s larger range suits gradients. llm-compressor’s documentation dates the hardware side of this by listing FP8 KV-cache quantisation as being for “long context inference on Hopper-class and newer GPUs”, which is the shape of the FP8 story generally: it is a way to raise a server’s compute ceiling on recent NVIDIA silicon rather than a way to make a file small.
MXFP4 is the four-bit member of the OCP microscaling family. Elements are E2M1: one sign bit, two exponent bits, one mantissa bit, giving a grid whose positive values are 0, 0.5, 1, 1.5, 2, 3, 4 and 6. Thirty-two consecutive elements share one scale, and that scale is E8M0 — an eight-bit exponent with no mantissa, so it can only express powers of two. The academic case for the family is made in Microscaling Data Formats for Deep Learning, which describes evaluating “Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements” and reports “the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss”.
NVFP4 keeps the same E2M1 elements and changes both of the other two parameters. Blocks are sixteen elements rather than thirty-two, and the per-block scale is E4M3 rather than E8M0, with “a second-level FP32 scalar applied per tensor” on top. NVIDIA’s explanation of why is worth quoting, because it is the whole argument: E8M0 gives “1 shared power-of-two scale per 32 value block”, while E4M3 “enables non-power-of-two scaling factors with fractional precision”, so each block can be given “one scale factor that makes the block errors collectively as small as possible”.
| Element | Block size | Scale format | Extra | |
|---|---|---|---|---|
| FP8 (E4M3) | 8-bit float, 4 exponent, 3 mantissa | per tensor or per channel | FP32 | — |
| MXFP4 | E2M1, four bits | 32 | E8M0, powers of two only | — |
| NVFP4 | E2M1, four bits | 16 | E4M3, fractional scales | second-level FP32 scale per tensor |
Part 5’s lesson on the DGX Spark already made the hardware half of this point: the machine’s Tensor Cores compute directly with four-bit floating point, so a four-bit model there is not merely smaller but can be multiplied natively. What that lesson deliberately left open is what those four bits are, which is the table above.
Two effects follow from that table without any measurement. Halving the block size doubles the number of scales, so NVFP4 spends more bits per weight on scales than MXFP4 does at the same element width: a four-bit format is again not four bits per weight. And a power-of-two scale can only ever land on certain values, so a block whose true magnitude sits between two powers of two wastes part of its grid; a fractional scale does not. NVIDIA reports, for post-training quantisation of one specific model on seven benchmarks, “1% or less accuracy degradation” against FP8. That is a vendor figure about one model on one date, not a general property, and it belongs in your notes with those conditions attached.
A model that is natively four-bit
Section titled “A model that is natively four-bit”gpt-oss is the course’s example of a model shipped in a low-precision format rather than compressed into one afterwards. Its card describes “MXFP4 quantization of the MoE weights” and states that “all evals were performed with the same MXFP4 quantization”, which makes the published numbers and the downloadable file the same artefact — a rarer thing than it should be.
The practical consequences are that the small model fits where a 21-billion-parameter model has no business fitting, and that quantising it further is quantising something already quantised, with the double-rounding penalty Part 6 warned about.
gpt-oss-20b in its native MXFP4, on a 16 GB machine, with 8k of context — an estimate from the course model reference
- Weights, native MXFP4
- 13 GB
- KV cache, 8k tokens at FP16
- 0.4 GB
- Free
- 2.6 GB
- Total
- 16 GB
What each track computes with natively
Section titled “What each track computes with natively”Storing a format and computing with it are different capabilities. Every track can store a four-bit file; the engine unpacks it and does the arithmetic in whatever precision the hardware prefers. Native support means the multiply itself happens in the narrow format, which is a throughput question and shows up in prefill and in batched serving rather than in single-stream decode.
Track S — NVIDIA DGX Spark
Blackwell-generation silicon, so FP8 and NVFP4 are the formats to care about, and NVIDIA’s Model Optimizer is the first-party route to producing them. Its README describes the library as covering “quantization, pruning, Neural Architecture Search (NAS), distillation, speculative decoding and sparsity”, lists FP8, INT8 with SmoothQuant, INT4 AWQ, W4A8 and NVFP4 among its formats, and offers QAT to “refine accuracy of quantized models even further with a few training steps”. Deployment is through TensorRT-LLM 1.2.1 · verified 2026-09-08 or vLLM. Treat the aarch64 path as the less-travelled one and check the container notes from Part 8 before budgeting time.
Track X — AMD Ryzen AI Max+ 395
The practical four-bit path on this track is GGUF through llama.cpp, which is a storage format the engine unpacks rather than a native compute format. FP8 and NVFP4 tensor-core arithmetic are NVIDIA hardware features and do not apply here. This is not a limitation you will feel on single-user decode, where bytes read per token is the constraint and a four-bit GGUF already solves it; it is a limitation you would feel serving many concurrent users, which is the case Part 9 measured.
Track M — Apple silicon
MLX exposes microscaling grids directly: mlx_lm.convert --q-mode accepts affine, mxfp4,
nvfp4 and mxfp8, so you can write a model in the same layouts this lesson describes and
compare them on your own hardware. Whether the Metal kernels compute in those formats or unpack
them is a question for the MLX release notes on the day you read this; the file layout is
documented, the arithmetic path is not something this course has measured.
Track N — NVIDIA desktop or laptop
Which formats are native depends on the generation of the card, not on the vendor: FP8 arrived with Hopper and Ada, and NVFP4 with Blackwell. A Blackwell consumer card has the same NVFP4 arithmetic the datacentre parts have; an older card will load an NVFP4 file only if the engine unpacks it, and often will not load it at all. Check the compute capability of your card before downloading a format-specific checkpoint, and prefer GGUF or W4A16 when you are not sure.
When QAT is worth it
Section titled “When QAT is worth it”Almost never, for a reader, and almost always, for a publisher. That asymmetry is the useful conclusion.
QAT costs a training run. Not a full pretraining — Gemma 3’s recipe was about five thousand steps — but a real one, with the optimiser state and the activation memory that Part 11’s arithmetic budgets for, on a model you must be able to fit for training rather than merely for inference. Against that you are buying back a fraction of the quality that four-bit rounding costs, on a model you did not train, that somebody else may already have released a QAT version of.
The cases where it is worth doing yourself are specific. You are already retraining the model regularly, so the marginal cost is a configuration change rather than a project. You have a fixed deployment format that you cannot change, so optimising for it has compounding value. Or post-training quantisation has measurably failed on your model at the width you need, which is most likely on small models, where there is less redundancy to spend, and at two and three bits, where LLM-QAT reports that training-free methods break down.
Otherwise the right move is the boring one: look for a published QAT checkpoint of the model you want, read what its publisher measured and with which tool, and then measure it yourself against the unquantised original on your own task. Which is the next lesson.
Ask four questions about every low-precision format
Section titled “Ask four questions about every low-precision format”What values can one element represent? Which elements share a scale? In what dtype are products and sums accumulated? Which kernels execute that representation on the target device? A bit-width label answers only part of the first question.
For a conceptual example, a shared scale large enough to represent an outlier can make small neighbouring values round to zero. Smaller groups may reduce that error while increasing scale metadata and processing overhead. Quantisation-aware training exposes the model to a representation of this distortion during optimisation; it still requires a compatible deployment path.
Record storage, compute and accumulation formats separately. A checkpoint stored at low precision may be expanded during execution, so a smaller file does not prove native low-precision arithmetic or proportional speedup. When choosing between post-training quantisation and QAT, first measure whether a simpler conversion misses your quality target. QAT adds training data, compute and another checkpoint lineage to maintain. Its value is a measured recovery under the actual deployed format, not the presence of a newer format name.
Quantisation-aware training fake-quantises weights in the forward pass, keeps a full-precision master copy for the optimiser, and passes gradients through the rounding with a straight-through estimator, so the model learns weights that tolerate the grid. LLM-QAT established the recipe for language models and made it data-free by training against the original model’s own output distribution; Gemma 3’s published QAT checkpoints are that recipe, applied for about five thousand steps, with a quality claim stated precisely enough to reproduce. FP8 is one byte per value in E4M3 or E5M2. MXFP4 and NVFP4 share E2M1 elements and differ in block size and scale format: thirty-two values with a power-of-two E8M0 scale against sixteen values with a fractional E4M3 scale and a second-level per-tensor scale. gpt-oss ships natively in MXFP4 and was evaluated in it. Native arithmetic support is a Blackwell and Hopper story on the NVIDIA tracks, an unpacking story on the others, and it matters for throughput rather than for single-stream decode. And running QAT yourself is worth it when you already retrain, when the deployment format is fixed, or when post-training quantisation has measurably failed.
Check your understanding
Sources for this lesson
8 verified · checked 2026-09-09
- 01LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (Liu et al., arXiv:2305.17888)§ Abstractarxiv.org/abs/2305.178882026-09-09
- 02Gemma 3 QAT Models: Bringing state-of-the-art AI to consumer GPUs§ QAT training steps; perplexity drop; VRAM figuresdevelopers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus2026-09-09
- 03google/gemma-3-27b-it-qat-q4_0-gguf model card§ Model description; licencehuggingface.co/google/gemma-3-27b-it-qat-q4_0-gguf2026-09-09
- 04Introducing NVFP4 for Efficient and Accurate Low-Precision Inference§ NVFP4 format; comparison with MXFP4developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference2026-09-09
- 05Microscaling Data Formats for Deep Learning (Rouhani et al., arXiv:2310.10537)§ Abstractarxiv.org/abs/2310.105372026-09-09
- 06openai/gpt-oss-20b model card§ MXFP4 quantisation; evaluationhuggingface.co/openai/gpt-oss-20b2026-09-09
- 07NVIDIA Model Optimizer (TensorRT Model Optimizer) repository§ README; quantization formats; QATgithub.com/NVIDIA/TensorRT-Model-Optimizer2026-09-09
- 08mlx-lm — convert.py argument definitions§ setup_arg_parser; --q-moderaw.githubusercontent.com/ml-explore/mlx-lm/main/mlx_lm/convert.py2026-09-09
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.