GGUF and Quantisation Types

By the end of this lesson you will be able to open a model repository, read a list of forty files with names like Qwen3-8B-UD-Q4_K_XL.gguf, and choose the right one for your machine in about ten seconds; explain what the letters mean and roughly what each costs in bytes per weight; say what an importance matrix does and why it appears in some names and not others; check that what you downloaded is what the publisher published; and produce a GGUF file yourself from a safetensors checkpoint.

What is inside the file

GGUF is defined in the ggml repository, and it is a deliberately boring format: a header, a table of metadata, a table of tensor descriptions, then the tensor data.

A GGUF file, from the start of the file downwards

file offset increasing

1. Magic number and versionThe four bytes GGUF, then a version number. Version 3 added big-endian support.
2. CountsHow many tensors and how many metadata entries follow.
3. Metadata key-value pairsTyped, snake_case keys: general.architecture, general.quantization_version, general.alignment, then per-architecture keys such as llama.context_length, llama.block_count and llama.attention.head_count_kv, and the tokeniser under tokenizer.ggml.*.
4. Tensor informationFor each tensor: its name, its dimensions, its ggml type, and its offset into the data section.
5. Padding to general.alignmentDefault 32 bytes, so the data section starts on a boundary the loader can map directly.
6. Tensor dataThe weights. Memory-mapped rather than read, which is why a large model starts answering before the whole file has been pulled from disk.

Everything before the tensor data is small enough to read in milliseconds, which is why a tool can tell you a model's architecture, context length and tokeniser without loading the weights.

Two consequences matter in practice. The metadata is why a GGUF file needs no config.json, tokeniser file or chat template beside it: the specification’s stated goal is that “all information needed to load a model is contained in the model file”. And the alignment and layout are why the file is memory-mapped, which is why --no-mmap is a flag you will meet later, and why a model can appear to load instantly and then be slow on its first few tokens while the pages are actually read.

The specification also defines a file-naming convention, [Sidecar]-BaseName-SizeLabel-FineTune-Version-Encoding-Type-Shard.gguf, with examples such as Grok-100B-v1.0-Q4_0-00003-of-00009.gguf. Publishers follow it loosely, but two parts are always there: the size label and the encoding. The encoding is the thing this lesson is about.

The families, in the order they were invented

The names look arbitrary. They are three generations of the same idea.

The legacy types: Q4_0, Q5_0, Q8_0. The original scheme. Weights are grouped into small blocks; each block stores its values at the stated bit width plus one scale factor for the block. The number is the bits per weight before the scale is counted, and the _0 is the block layout. Q8_0 is still in wide use because at eight bits the quality question effectively disappears; Q4_0 has been superseded.

The K-quants: Q2_K to Q6_K, with _S, _M and _L variants. The insight is that not every tensor deserves the same precision. A K-quant file is a mixture: attention and feed-forward tensors are quantised at different widths, with the tensors that hurt most when damaged kept wider. Q4_K_M means “the four-bit K-quant mixture, medium”, and _S and _M and _L are progressively more generous mixtures at the same nominal width. This is why the file sizes do not line up with the arithmetic: a “four-bit” model is not four bits per weight.

The IQ types: IQ1_S through IQ4_XS and IQ4_NL. These use a codebook rather than a plain scale, which extracts more quality per bit at the low end, and they are normally built with an importance matrix. They are the reason a very large model can be squeezed onto a machine that should not fit it, and they are the types where the quality question is real rather than academic.

MXFP4. Not a llama.cpp invention: a four-bit floating-point format with a shared scale per block, used as a model’s native format rather than as a post-training compression. gpt-oss is the course’s example; its card describes MXFP4 quantisation of the mixture-of-experts weights and says that is how the model was evaluated. Quantising it further would be quantising a quantised model.

What the letters actually cost

The llama-quantize README publishes a table of bits per weight and file size for every type, measured on Llama 3.1 8B. These are the project’s own published figures for that one model, so treat them as a shape rather than as your model’s exact sizes.

Type	Bits per weight	File size, GiB
IQ1_S	2.0042	1.87
IQ2_XXS	2.3824	2.23
Q2_K	3.1593	2.95
IQ3_XXS	3.2548	3.04
Q3_K_M	3.9960	3.74
IQ4_XS	4.4597	4.17
Q4_K_S	4.6672	4.36
Q4_K_M	4.8944	4.58
Q5_K_M	5.7036	5.33
Q6_K	6.5633	6.14
Q8_0	8.5008	7.95
F16	16.0005	14.96

Read the first column twice. IQ1_S, the “one-bit” type, is two bits per weight. Q4_K_M is nearly five. Q8_0 is eight and a half. The overhead is the block scales plus the tensors each mixture keeps at higher precision, and it is why the memory arithmetic from Part 4 uses about 0.55 to 0.6 bytes per parameter for a four-bit model rather than 0.5.

▤Memory budget

The rule of thumb that follows: multiply the parameter count by the bits per weight from this table, divide by eight, and you have gigabytes to within a few per cent. Qwen3-8B at Q4_K_M: 8.2 billion parameters times 4.89 bits, divided by eight, is roughly 5 GB, which is what the course’s model reference records for that file. Then add the KV cache, which is a separate budget and which the next lesson sizes.

Importance matrices

Quantisation rounds every weight. An importance matrix decides which weights can least afford it.

llama-imatrix runs a calibration text through the unquantised model and collects the squared activations for each tensor, producing a score per weight group. llama-quantize then reads that file with --imatrix and spends its bit budget accordingly, protecting the weights that carry the most signal on that text.

RunnableAll tracks

build an importance matrix, then quantise with it

~/llama.cpp/build/bin/llama-imatrix \
    -m ~/models/gguf/Qwen3-8B-BF16.gguf \
    -f calibration.txt \
    -ngl 99

~/llama.cpp/build/bin/llama-quantize \
    --imatrix imatrix.gguf \
    ~/models/gguf/Qwen3-8B-BF16.gguf \
    ~/models/gguf/Qwen3-8B-IQ4_XS.gguf \
    iq4_xs 8

Two honest caveats. The matrix is computed on a particular text, so it encodes an assumption about what the model will be asked to do; a matrix from English prose is a bet that you will use English prose. And the improvement is largest where the bit budget is tightest: at Q6_K or Q8_0 there is little left to protect. The llama-quantize README puts it plainly, saying that quantisation “may introduce some accuracy loss” which “can be minimized by using a suitable imatrix file”.

∴Research note

Whether an imatrix file helps your task is a measurable question, and Part 16 is where the course measures it properly, with perplexity and with task evaluations. Until then, treat “imatrix quants are better” as the untested claim it is for your workload, and prefer them at the low bit widths where the argument is strongest.

Reading a provider’s table

Very few model publishers ship GGUF files themselves. Most GGUF files come from a handful of people who convert and quantise every notable release within a day or two, and their repositories look like a wall of names.

The course’s model reference records one GGUF source per model, and they are of three kinds:

• The publisher’s own, where it exists. ggml-org/gpt-oss-20b-GGUF is maintained by the llama.cpp organisation itself, which is as close to first-party as GGUF gets.
• unsloth, which the course uses for the Qwen3 family. Their repositories carry the standard types plus their own UD- variants, described on the card as “Unsloth Dynamic” quantisation, which are their own mixture choices.
• bartowski, which the course uses for Llama 3.1 8B, the cross-engine comparison model.

Faced with a list, three questions settle it.

Does it fit? Take the file size, add your KV cache, compare with your memory. This eliminates most of the list immediately.

Is it a mixture you recognise? Prefer Q4_K_M, Q5_K_M, Q6_K and Q8_0 for ordinary use. Reach for IQ4_XS or lower only when the model would not otherwise fit, and expect to have to check the quality rather than assume it.

Is it split? Large models are published as shards named -00001-of-00009 and so on. Download all of them into the same directory and point the tool at the first; llama.cpp opens the rest. A common mistake is downloading only the first shard because it is the one named in an example.

✕Common mistake

Picking the biggest quant that fits, exactly. A model that occupies every byte of your memory leaves nothing for the KV cache, and the KV cache grows with the context you ask for. A 24 GB machine running a 19.8 GB model will load and then fail, or crawl, the moment you ask for a long context. Leave room deliberately; the diagram below is what that looks like.

Qwen3-32B at Q4_K_M on a 24 GB machine, with 8k of context — an estimate, not a measurement

Weights, Q4_K_M

19.8 GB

KV cache, 8k tokens at FP16

2.1 GB

Free

2.1 GB

Total

24 GB

Weights and KV cache from the course model reference: about 19.8 GB of weights, and 256 KiB per token of KV cache at FP16, which is about 2.1 GB at 8,192 tokens. The remainder has to hold the compute buffers, the operating system and anything else running. This is why the tier table pairs a 24 GB machine with a 32B model only at four-bit, and why the next lesson's KV-cache quantisation flags matter here.

Downloading, and checking what you got

Part 4 established the model library at ~/models and the habit of verifying a download against the checksum the Hub publishes for the file. GGUF files are where that habit earns its keep, because they are large, because a truncated download often produces a file that loads and then behaves strangely, and because you will collect a lot of them.

RunnableAll tracks

fetch one quantisation, into the shared library

hf download unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf \
    --local-dir ~/models/unsloth/Qwen3-8B-GGUF

To fetch a sharded model, ask for the pattern rather than each file:

RunnableAll tracks

fetch every shard of a split model

hf download unsloth/Qwen3-235B-A22B-GGUF --include "*IQ4_XS*" \
    --local-dir ~/models/unsloth/Qwen3-235B-A22B-GGUF

Part 4’s fetch-model.sh does the same thing and then verifies the SHA-256 against the value the Hub publishes for that file, writing the checksum beside the file. Use it for anything large enough that re-downloading would annoy you.

⛨Security

A GGUF file is data, not code: unlike a pickle checkpoint, loading one does not execute anything the publisher chose. That is a real safety improvement and it is not blanket safety, because any parser can have bugs and a deliberately malformed file is a way to find them. The practical rules are the ordinary ones: download from repositories you can name and would recognise again, prefer the publisher’s own where it exists, verify the checksum, and do not run a model file a stranger sent you on a machine you care about.

Making your own

Sometimes there is no GGUF: a model published this morning, a fine-tune of your own from Part 13, or a quantisation mixture nobody published. Two steps produce one.

Step one: convert the checkpoint to GGUF at full precision. convert_hf_to_gguf.py lives at the root of the llama.cpp checkout and reads a Hugging Face model directory. Its --outtype accepts f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto, where auto is described as “the highest-fidelity 16-bit float type”. The script needs its own Python dependencies, listed in the checkout’s requirements.txt.

RunnableAll tracks

convert a safetensors checkpoint to GGUF at BF16

uv venv ~/llama.cpp/.venv
source ~/llama.cpp/.venv/bin/activate
uv pip install -r ~/llama.cpp/requirements.txt

python3 ~/llama.cpp/convert_hf_to_gguf.py \
    ~/models/Qwen/Qwen3-8B \
    --outfile ~/models/gguf/Qwen3-8B-BF16.gguf \
    --outtype bf16

Step two: quantise it. llama-quantize takes the input file, the output file, a type name and optionally a thread count.

RunnableAll tracks

quantise to a four-bit K-quant mixture

~/llama.cpp/build/bin/llama-quantize \
    ~/models/gguf/Qwen3-8B-BF16.gguf \
    ~/models/gguf/Qwen3-8B-Q4_K_M.gguf \
    q4_k_m 8

The options worth knowing, all from the tool’s README: --imatrix to use an importance matrix, --include-weights and --exclude-weights to limit which tensors it applies to, --output-tensor-type and --token-embedding-type to pin those two tensors to a type of your choosing, --leave-output-tensor to skip quantising the output tensor at all, --pure to disable the K-quant mixtures and use one type everywhere, --allow-requantize to quantise something that is already quantised, and --keep-split to preserve a sharded input’s shard structure.

✦Tip

--pure is the teaching flag. Quantise the same model twice, once normally and once with --pure at the same nominal width, and compare the file sizes and the outputs. The difference is exactly the value the K-quant mixtures add, on your model, measured rather than described.

--allow-requantize deserves a warning rather than a tip: going from Q8_0 to Q4_K_M applies rounding error twice, and the result is worse than quantising once from the full-precision file. It exists for cases where the original is unavailable, not as a shortcut.

Compare representations without changing the experiment

Choose one high-precision source checkpoint and derive the quantised variants from it. Record the converter revision, quantiser settings and calibration data where applicable. Downloading unrelated community conversions with similar filenames may compare conversion choices or different base revisions as well as precision.

Evaluate three properties separately: file size, runtime memory at fixed context and task quality. File size does not include the cache. Runtime memory may change in steps because the engine allocates buffers in blocks. Quality loss can be concentrated in unusual identifiers, arithmetic or structured-output edge cases even when conversational answers remain fluent.

For a worked decision, suppose the smaller representation fits fully on the GPU while the larger one offloads. The smaller model may produce a faster end-to-end answer because placement changed. That is a valid deployment result, but it does not isolate the quantised kernel’s speed. Report both the representation and the residency so readers can tell why the useful outcome changed.

Recap

A GGUF file is a header, a metadata table, a tensor table and the weights, laid out to be memory-mapped and carrying everything needed to load the model, which is why it is one file rather than a directory. Quantisation names come in three generations: the legacy _0 types, the K-quant mixtures that spend different precision on different tensors, and the IQ codebook types built with an importance matrix for the low bit widths. The bits per weight are always higher than the name suggests, by enough to matter to your memory budget: a four-bit K-quant is closer to five bits. Choose by fitting first, leaving room for the KV cache, then by measuring quality if the choice is tight. Download from a repository you can name, verify the checksum, and where no GGUF exists, convert_hf_to_gguf.py and llama-quantize will make one in two commands.

Check your understanding

Question 1. A model is published as Q4_K_M. According to the llama-quantize README's table for Llama 3.1 8B, roughly how many bits per weight does that file actually use?

• Exactly 4
• About 4.9
• About 3.2
• About 8.5

Show the answer and why

Answer: About 4.9

The table gives 4.8944 bits per weight for Q4_K_M. The extra comes from the per-block scales and from the tensors the mixture keeps at higher precision. This is why the course budgets about 0.55 to 0.6 bytes per parameter for a four-bit model.

Question 2. What does an importance matrix change about a quantisation?

• It makes the file smaller at the same type
• It tells llama-quantize which weights matter most on a calibration text, so the bit budget is spent protecting them
• It encrypts the weights
• It converts the model to a different architecture

Show the answer and why

Answer: It tells llama-quantize which weights matter most on a calibration text, so the bit budget is spent protecting them

llama-imatrix collects squared activations over a calibration text; llama-quantize reads the result with --imatrix. The file size is set by the type, not by the matrix. The gain is largest at the low bit widths, and it is conditional on the calibration text resembling your use.

Question 3. A 24 GB machine and a model file of 19.8 GB. Which reasoning is sound?

• It fits with 4.2 GB spare, so use the longest context available
• The KV cache and compute buffers come out of the same 24 GB, so the usable context is limited and a smaller quantisation or a quantised KV cache may be the better choice
• The KV cache lives on the CPU, so it does not count
• File size does not predict memory use at all

Show the answer and why

Answer: The KV cache and compute buffers come out of the same 24 GB, so the usable context is limited and a smaller quantisation or a quantised KV cache may be the better choice

Weights plus KV cache plus compute buffers all share the pool. At 256 KiB per token, an 8k context on a 32B model is about 2.1 GB, and asking for 32k would not fit. The next lesson gives you the flags that trade context length against memory.

Question 4. Which of these are sensible habits when downloading GGUF files? Select all that apply.

• Prefer the publisher's own repository where one exists
• Verify the file against the checksum the Hub publishes
• Download every shard of a split model, not just the first
• Requantise a Q8_0 file down to Q4_K_M rather than downloading the four-bit file

Show the answer and why

Answer: Prefer the publisher's own repository where one exists, Verify the file against the checksum the Hub publishes, Download every shard of a split model, not just the first

The first three are the routine. Requantising applies rounding error twice and gives a worse result than quantising once from the full-precision weights; --allow-requantize exists for when the original is genuinely unavailable.

Question 5. Why does a GGUF file need no config.json or tokeniser file beside it?

• Because llama.cpp downloads them separately
• Because the format's metadata table carries the architecture, the hyperparameters and the tokeniser, by design
• Because the tokeniser is always the same
• Because those files are only needed for training

Show the answer and why

Answer: Because the format's metadata table carries the architecture, the hyperparameters and the tokeniser, by design

Single-file deployment and "full information" are two of the specification's five design goals. The metadata sits between the header and the tensor data, small enough to read without touching the weights.

Check my answersClear