Parameters, Layers and Model Size

By the end of this lesson you will be able to take a model’s config.json and reproduce the parameter count on its card to the last digit, say which of those parameters are read for every token and which merely have to be resident, list what is inside a safetensors file and a GGUF file with a few lines of standard-library Python, tell a context window from a context length, and compute what a context costs in memory before you allocate it. Every number below is arithmetic from a stated input, a field in a configuration file or a file size read from a repository listing on 2026-09-12, and the lab has you check the same arithmetic against a checkpoint on your own disk.

What a parameter count counts

A parameter is one learned number. The count in a model’s name is the total of every learned number in the checkpoint: the embedding matrix, the query, key, value and output projections of every attention sub-layer, the three matrices of every feed-forward network, the scale vector of every normalisation, and the output projection if it is not tied to the embedding.

That list is short enough to add up by hand, and doing so once is how the count stops being a label. The attention lesson drew the block; the Transformers modelling code for the Qwen3 architecture stores each linear layer’s weight as (output width, input width), so every shape follows from config.json. For Qwen3-8B, published under the Apache-2.0 licence, one block holds:

Tensor in one block	Shape, from the config fields	Parameters
self_attn.q_proj.weight	(32 heads × 128, 4096) = (4096, 4096)	16,777,216
self_attn.k_proj.weight	(8 kv heads × 128, 4096) = (1024, 4096)	4,194,304
self_attn.v_proj.weight	(1024, 4096)	4,194,304
self_attn.o_proj.weight	(4096, 32 × 128) = (4096, 4096)	16,777,216
self_attn.q_norm.weight, self_attn.k_norm.weight	(128,) each	256
mlp.gate_proj.weight, mlp.up_proj.weight	(12288, 4096) each	100,663,296
mlp.down_proj.weight	(4096, 12288)	50,331,648
input_layernorm.weight, post_attention_layernorm.weight	(4096,) each	8,192
One block	11 tensors	192,946,432

Then the stack, the final normalisation and the two vocabulary-sized matrices:

non-embedding = 36 blocks × 192,946,432 + model.norm (4,096)        = 6,946,075,648
embedding     = vocab_size × hidden_size = 151,936 × 4,096          =   622,329,856
output head   = a second matrix of the same shape (tie_word_embeddings: false)
total         = 6,946,075,648 + 2 × 622,329,856                     = 8,190,735,360

The card says “8.2B” and, for the non-embedding count, “6.95B”. Those are these two numbers before rounding, and the same arithmetic reproduces Qwen3-1.7B’s (Apache-2.0) 1.4B and 1.7B in the lab. Where the parameters live is worth one look, because it decides what quantisation and fine-tuning act on later:

Where the 8,190,735,360 parameters are	Parameters	Share
Feed-forward networks, 36 × 150,994,944	5,435,817,984	66.4 %
Attention projections and head norms, 36 × 41,943,296	1,509,958,656	18.4 %
Embedding matrix and output head	1,244,659,712	15.2 %
Block and final normalisation scales, 36 × 8,192 + 4,096	299,008	under 0.01 %

The count does not include three things that also occupy memory when the model runs or trains, and confusing them is where most “why did this not fit” surprises come from: activations, the intermediate vectors flowing through the stack, which scale with how many tokens you push through at once; the KV cache, the stored keys and values for the context, which grows as the conversation grows and is the last section of this lesson; and optimiser state, which exists only during training and adds several bytes per parameter on top of the weights, which Part 11’s memory arithmetic counts.

Nor does the count say anything about bytes. Bytes per parameter come from the format, as Part 1’s precision lesson set out, and the honest way to learn the real figure is to divide a file’s size by the count. The course’s GGUF source for Qwen3-8B lists these files (sizes from the repository’s tree listing, 2026-09-12):

File in unsloth/Qwen3-8B-GGUF	Bytes	÷ 8,190,735,360 parameters
Qwen3-8B-BF16.gguf	16,388,044,384	2.00 bytes per parameter
Qwen3-8B-Q8_0.gguf	8,709,519,168	1.06 bytes per parameter
Qwen3-8B-Q4_K_M.gguf	5,027,784,512	0.61 bytes per parameter

A “4-bit” file costs six tenths of a byte per parameter rather than half a byte, because the per-block scales are stored too and because not every tensor in it is four-bit; the GGUF section below shows the mix. Count times bytes per parameter is the weight memory, and that product is the first line of every memory budget in this course.

✕Common mistake

“8B” is a count, not a promise about quality, licence or context length. Two models with the same count can differ by a factor of several on any task you care about, and Part 16’s evaluation lesson is about measuring that rather than inferring it from a name.

Reading config.json

Every Hugging Face model repository has a config.json at its root: the architecture as numbers, everything the modelling code needs to build the empty model before the weights are loaded into it. It is 728 bytes, readable in the browser from the repository’s file list, and once the lab has installed the hf CLI one command fetches it without the weights. The CLI’s documentation gives the form “provide the repo_id and filename”, and the unauthenticated-requests warning the lab explains appears first unless you signed in:

RunnableAll tracks

fetch one file from a repository

hf download Qwen/Qwen3-8B config.json --local-dir ~/llm-course/configs/qwen3-8b

Output — what you should see

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
✓ Downloaded
  path: /home/you/llm-course/configs/qwen3-8b/config.json

Here is the file, cut to the fields this course uses:

Qwen/Qwen3-8B — config.json (extract, retrieved 2026-09-12)

{
  "architectures": ["Qwen3ForCausalLM"],
  "model_type": "qwen3",
  "hidden_size": 4096,
  "intermediate_size": 12288,
  "num_hidden_layers": 36,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "head_dim": 128,
  "vocab_size": 151936,
  "max_position_embeddings": 40960,
  "rope_theta": 1000000,
  "rope_scaling": null,
  "rms_norm_eps": 1e-06,
  "hidden_act": "silu",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16"
}

Field by field, with the memory term each one sizes:

Field	What it decides	Sizes
architectures, model_type	Which modelling class loads the weights; Qwen3ForCausalLM is decoder-only and causal	which formulas apply
hidden_size	Width of the residual stream; appears in every matrix	weights, activations
intermediate_size	Width the feed-forward network expands to inside each block	weights (two thirds of them)
num_hidden_layers	How many blocks are stacked	weights, KV cache
num_attention_heads	Query heads per block	weights only
num_key_value_heads	Key-value heads per block	KV cache
head_dim	Width of each head; times the query-head count gives hidden_size here	weights, KV cache
vocab_size	Rows in the embedding matrix and width of the output distribution	weights
max_position_embeddings	The context length the software reads as the trained one; whether an engine enforces it as a ceiling is the engine’s decision (Part 7)	nothing directly; the context length you allocate does (two sections on)
rope_theta, rope_scaling	The rotary base, and any context-extension method	the window (next section)
tie_word_embeddings	Whether the output head reuses the embedding matrix	weights, one matrix either way
torch_dtype	The precision the weights were saved in	bytes per parameter

Configurations of other reference models, Qwen3-30B-A3B and gpt-oss-20b (both Apache-2.0 according to their cards), add fields you will meet in Part 4, and the next section uses two of them:

Field	Appears in	Meaning
num_experts, num_experts_per_tok, moe_intermediate_size	Qwen3-30B-A3B	experts per block, experts read per token, width of each expert
num_local_experts, experts_per_token	gpt-oss-20b	the same two counts under OpenAI’s names
layer_types, sliding_window	gpt-oss-20b	which layers attend to the whole context and which to a 128-token window
quantization_config	gpt-oss-20b	the weights are stored quantised (quant_method: mxfp4) and which modules are not
transformers_version	all	the library version that saved the file; the model card’s minimum, not the course’s pin

The arithmetic of the first section is short enough to keep as a script. This one handles the dense Qwen3 layout and the Qwen3 mixture-of-experts layout, which is why it has two branches; it needs nothing but the standard library, and the lab’s list-tensors.py later checks its answer against the real tensors. Save each script under the name in its title in ~/llm-course (Track S: /workspace/course), with the lab’s environment active; every python line on this page runs from there.

RunnableAll tracks

count-params.py

import json, sys

c = json.load(open(sys.argv[1]))                       # config.json of a Qwen3 or Qwen3-MoE model
h, layers, vocab = c["hidden_size"], c["num_hidden_layers"], c["vocab_size"]
q_width = c["num_attention_heads"] * c["head_dim"]
kv_width = c["num_key_value_heads"] * c["head_dim"]

attention = 2 * h * q_width + 2 * h * kv_width + 2 * c["head_dim"]   # q,o + k,v + q_norm,k_norm
norms = 2 * h                                                        # input and post-attention RMSNorm
if c["model_type"] == "qwen3_moe":
    per_expert = 3 * h * c["moe_intermediate_size"]                  # gate, up, down, each expert
    router = c["num_experts"] * h                                    # one row of scores per expert
    ffn_total = c["num_experts"] * per_expert + router
    ffn_active = c["num_experts_per_tok"] * per_expert + router
else:
    ffn_total = ffn_active = 3 * h * c["intermediate_size"]          # gate, up, down, once
embedding = vocab * h
head = 0 if c.get("tie_word_embeddings") else embedding              # lm_head, unless tied

per_layer = attention + ffn_total + norms
non_embedding = layers * per_layer + h                               # + the final norm
total = non_embedding + embedding + head
active = layers * (attention + ffn_active + norms) + h + embedding + head

print(f"model_type {c['model_type']}: {layers} layers, hidden {h}, vocab {vocab:,}")
print(f"  attention per layer     {attention:>15,}")
print(f"  feed-forward per layer  {ffn_total:>15,}   read per token {ffn_active:,}")
print(f"  non-embedding           {non_embedding:>15,}")
print(f"  embedding + head        {embedding + head:>15,}")
print(f"  TOTAL                   {total:>15,}   ({total / 1e9:.2f} B)")
print(f"  ACTIVE per token        {active:>15,}   ({active / 1e9:.2f} B)")

RunnableAll tracks

run it on the file you fetched

python count-params.py ~/llm-course/configs/qwen3-8b/config.json

Output — what you should see

model_type qwen3: 36 layers, hidden 4096, vocab 151,936
  attention per layer          41,943,296
  feed-forward per layer      150,994,944   read per token 150,994,944
  non-embedding             6,946,075,648
  embedding + head          1,244,659,712
  TOTAL                     8,190,735,360   (8.19 B)
  ACTIVE per token          8,190,735,360   (8.19 B)

For a dense model the last two lines are equal, which is the definition of dense.

⌚Version note

The configuration and the model card do not always say the same thing. Qwen3-8B’s card gives a native window of 32,768 tokens, while max_position_embeddings in the same repository is 40960 (both checked 2026-09-12). The card explains the gap itself: the default “is set to 40,960. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts”. The card describes what the model was trained to handle; the field is the number engines read: the GGUF specification calls its copy, context_length, “the hard limit on the length of the input” for most architectures, and Part 7’s reality check shows what each engine allocates by default. The course’s rule is to trust the card for what the model can do well and the configuration for what the software will allow.

config.json is one of a handful of small files that travel with the weights, and each answers a different question:

File	Question it answers	Used in
config.json	What shape is the model?	this lesson, every script
generation_config.json	Which sampling defaults and end-of-sequence ids did the publisher ship?	the lab, Part 6
tokenizer.json, tokenizer_config.json	How is text split, and what is the chat template?	the tokens lesson, the lab
model.safetensors, or shards plus model.safetensors.index.json	The weights, and which shard holds which tensor	next section
README.md	The model card: counts, context, licence, evaluation claims	Part 4

What is in a checkpoint

safetensors

The default format in a Hugging Face repository is safetensors, whose documentation describes it as “a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy)”. The “as opposed to pickle” is the point of it: the older PyTorch format is a Python pickle, which executes code when it is loaded, so downloading a checkpoint used to mean running a stranger’s code. A safetensors file contains no code path at all.

The format specification is short enough to hold in your head, and short enough to read with struct and json.

A safetensors file, from the start of the file

byte order

1. 8 bytes: N"an unsigned little-endian 64-bit integer, containing the size of the header".
2. N bytes: the JSON headerOne entry per tensor, each with "dtype", "shape" and "data_offsets"; the specification says the offsets "point to the tensor data relative to the beginning of the byte buffer", BEGIN as the start and END as one past the end.
3. __metadata__An optional key in the same header, "allowed to contain free form string-to-string map". Where a framework records its own notes.
4. Rest of the file: the byte bufferThe tensor data itself, laid out end to end.

The header is JSON, so a tool can list every tensor's name, dtype and shape by reading the first few kilobytes, without touching the weights. That is what the lab's list-tensors.py does, and it is why listing the contents of a 16 GB checkpoint is instant. The specification caps the header at 100 MB.

Two dozen lines are enough to prove it. Point this at any *.safetensors file, for example the one the lab’s reduced path downloads for Qwen3-0.6B (Apache-2.0):

RunnableAll tracks

safetensors-header.py

import json, struct, sys
from pathlib import Path

path = Path(sys.argv[1])                              # any *.safetensors file
with path.open("rb") as fh:
    (n,) = struct.unpack("<Q", fh.read(8))            # 8 bytes: N, little-endian uint64
    header = json.loads(fh.read(n))                   # N bytes: the JSON header
metadata = header.pop("__metadata__", {})
buffer_bytes = path.stat().st_size - 8 - n            # the rest: the byte buffer

params = 0
dtypes = set()
for name, info in header.items():
    count = 1
    for dim in info["shape"]:
        count *= dim
    params += count
    dtypes.add(info["dtype"])

print(f"file            {path.name}  {path.stat().st_size:,} bytes")
print(f"header          {n:,} bytes of JSON, {len(header)} tensors, dtypes {sorted(dtypes)}")
print(f"__metadata__    {metadata}")
print(f"byte buffer     {buffer_bytes:,} bytes")
print(f"parameters      {params:,}  ->  {buffer_bytes / params:.2f} bytes per parameter")
for name in list(header)[:4]:
    info = header[name]
    print(f"  {name:44s} {info['dtype']}  {info['shape']}  offsets {info['data_offsets']}")

The lab’s primary path downloads Qwen3-1.7B as two shards, not this file, so on that path this is one extra download, into the same directory the KV-cache script at the end of the lesson reads (the size is the Hub listing’s, 2026-09-12):

RunnableAll tracks

fetch the 0.6B checkpoint (1.5 GB)

hf download Qwen/Qwen3-0.6B --local-dir ~/llm-course/models/qwen3-0.6b

Output — what you should see

Fetching 10 files: 100%|██████████████████████████| 10/10 [xx:xx<00:00, x.xxs/it]
✓ Downloaded
  path: /home/you/llm-course/models/qwen3-0.6b

The file to point at is its single model.safetensors, 1,503,300,328 bytes in that listing:

RunnableAll tracks

run it on the 0.6B checkpoint

python safetensors-header.py ~/llm-course/models/qwen3-0.6b/model.safetensors

Output — what you should see

file            model.safetensors  1,503,300,328 bytes
header          35,552 bytes of JSON, 311 tensors, dtypes ['BF16']
__metadata__    {'format': 'pt'}
byte buffer     1,503,264,768 bytes
parameters      751,632,384  ->  2.00 bytes per parameter
  lm_head.weight                               BF16  [151936, 1024]  offsets [0, 311164928]
  model.embed_tokens.weight                    BF16  [151936, 1024]  offsets [311164928, 622329856]
  model.layers.0.input_layernorm.weight        BF16  [1024]  offsets [622329856, 622331904]
  model.layers.0.mlp.down_proj.weight          BF16  [1024, 3072]  offsets [622331904, 628623360]

Three things to read off that. The header is 35 KB of a 1.5 GB file, which is why the listing is instant. Two bytes per parameter is BF16, exactly. And the file holds 751,632,384 parameters while count-params.py gives this model 596,049,920: the difference, 155,582,464, is one 151,936 × 1,024 matrix, because the file stores lm_head.weight even though tie_word_embeddings is true and the loader will use the embedding matrix for both jobs. A file can hold more numbers than the model has; the lab reconciles the same discrepancy for Qwen3-1.7B, where the duplicate is an entire second shard.

Two practical consequences. Large models are split across several files, so a repository holds model-00001-of-00005.safetensors and friends plus model.safetensors.index.json, whose weight_map says which tensor is in which file. And because the header is separate from the data, a loader can read a subset of tensors without scanning the whole file, which the documentation describes as lazy loading. On zero-copy the documentation is honest rather than promotional: “No format is really zero-copy in ML, it needs to go from disk to RAM/GPU RAM (that takes time).”

GGUF

The other format you will meet constantly is GGUF, the format llama.cpp and everything built on it use. Its specification states five design goals, and each explains something about how Part 6 behaves:

• “Single-file deployment: they can be easily distributed and loaded, and do not require any external files for additional information.”
• “Extensible: new features can be added to GGML-based executors/new information can be added to GGUF models without breaking compatibility with existing models.”
• “mmap compatibility: models can be loaded using mmap for fast loading and saving.”
• “Easy to use: models can be easily loaded and saved using a small amount of code, with no need for external libraries, regardless of the language used.”
• “Full information: all information needed to load a model is contained in the model file, and no additional information needs to be provided by the user.”

The last one is the real difference from safetensors. A GGUF file carries the architecture numbers, the tokeniser and the chat template inside itself, as typed key-value pairs, so there is no repository of side files to keep together. That is why a GGUF model is one download and one path, and also why a bad conversion is hard to spot: the wrong template travels inside the file. The layout, from the specification:

Part of the file	Contents
Header	magic GGUF, a uint32 version (3), uint64 tensor count, uint64 metadata pair count
Metadata pairs	a string key, a uint32 value type, the value; types 0 to 12 cover integers, floats, bool, string and array
Tensor infos	per tensor: name, dimension count, dimensions, ggml_type, offset into the data section
Padding	to general.alignment; “If the alignment is not specified, assume it is 32”
Tensor data	the weights, each tensor at its offset

“Easy to use” is testable. This reader handles every value type in the specification and stops at the end of the tensor infos, so it never touches the weights:

RunnableAll tracks

gguf-header.py

import struct, sys
from collections import Counter

SCALAR = {0: "<B", 1: "<b", 2: "<H", 3: "<h", 4: "<I", 5: "<i", 6: "<f", 7: "<?", 10: "<Q", 11: "<q", 12: "<d"}
GGML = {0: "F32", 1: "F16", 2: "Q4_0", 3: "Q4_1", 6: "Q5_0", 7: "Q5_1", 8: "Q8_0", 9: "Q8_1", 10: "Q2_K",
        11: "Q3_K", 12: "Q4_K", 13: "Q5_K", 14: "Q6_K", 15: "Q8_K", 16: "IQ2_XXS", 17: "IQ2_XS", 18: "IQ3_XXS",
        19: "IQ1_S", 20: "IQ4_NL", 21: "IQ3_S", 22: "IQ2_S", 23: "IQ4_XS", 24: "I8", 25: "I16", 26: "I32",
        27: "I64", 28: "F64", 29: "IQ1_M", 30: "BF16", 34: "TQ1_0", 35: "TQ2_0", 39: "MXFP4"}

def read(fh, fmt):
    return struct.unpack(fmt, fh.read(struct.calcsize(fmt)))[0]

def read_string(fh):                                   # gguf_string_t: uint64 length, then UTF-8 bytes
    return fh.read(read(fh, "<Q")).decode("utf-8", "replace")

def read_value(fh, value_type):
    if value_type == 8:
        return read_string(fh)
    if value_type == 9:                                # array: element type, count, elements
        element_type, count = read(fh, "<I"), read(fh, "<Q")
        return [read_value(fh, element_type) for _ in range(count)]
    return read(fh, SCALAR[value_type])

with open(sys.argv[1], "rb") as fh:
    magic = fh.read(4)                                 # gguf_header_t
    version, n_tensors, n_kv = read(fh, "<I"), read(fh, "<Q"), read(fh, "<Q")
    print(f"magic {magic!r}  version {version}  tensors {n_tensors}  metadata pairs {n_kv}")
    metadata = {}
    for _ in range(n_kv):
        key = read_string(fh)
        metadata[key] = read_value(fh, read(fh, "<I"))
    params, by_type = 0, Counter()
    for _ in range(n_tensors):                         # gguf_tensor_info_t
        name, n_dims = read_string(fh), read(fh, "<I")
        dims = [read(fh, "<Q") for _ in range(n_dims)]
        ggml_type, offset = read(fh, "<I"), read(fh, "<Q")
        count = 1
        for d in dims:
            count *= d
        params += count
        by_type[GGML.get(ggml_type, str(ggml_type))] += count
    header_end = fh.tell()

for key, value in metadata.items():
    if isinstance(value, list):
        value = f"[array of {len(value)}]"
    elif isinstance(value, str) and len(value) > 48:
        value = value[:45].replace("\n", " ") + "..."
    if not key.startswith("tokenizer.ggml."):          # the vocabulary arrays are not interesting here
        print(f"  {key:40s} {value}")
print(f"header + tensor infos end at byte {header_end:,}; tensor data follows, aligned to {metadata.get('general.alignment', 32)}")
print(f"parameters {params:,}")
for t, n in by_type.most_common():
    print(f"  {t:8s} {n:>15,}  {100 * n / params:5.1f} %")

Run it on the Q4_K_M file from the course’s GGUF source for the same model, unsloth/Qwen3-0.6B-GGUF, which is 396,705,472 bytes at the retrieval date; Part 4 sets up the ~/models library properly, with checksums, and any .gguf file you already have works just as well:

RunnableAll tracks

fetch one GGUF file and read its header

hf download unsloth/Qwen3-0.6B-GGUF Qwen3-0.6B-Q4_K_M.gguf --local-dir ~/models/unsloth/Qwen3-0.6B-GGUF
python gguf-header.py ~/models/unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf

Output — what you should see

✓ Downloaded
  path: /home/you/models/unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf
magic b'GGUF'  version 3  tensors 310  metadata pairs 32
  general.architecture                     qwen3
  general.type                             model
  general.name                             Qwen3-0.6B
  general.basename                         Qwen3-0.6B
  general.quantized_by                     Unsloth
  general.size_label                       0.6B
  general.repo_url                         https://huggingface.co/unsloth
  qwen3.block_count                        28
  qwen3.context_length                     40960
  qwen3.embedding_length                   1024
  qwen3.feed_forward_length                3072
  qwen3.attention.head_count               16
  qwen3.attention.head_count_kv            8
  qwen3.rope.freq_base                     1000000.0
  qwen3.attention.layer_norm_rms_epsilon   9.999999974752427e-07
  qwen3.attention.key_length               128
  qwen3.attention.value_length             128
  tokenizer.chat_template                  {%- if tools %}     {{- '<|im_start|>system\n...
  general.quantization_version             2
  general.file_type                        15
  quantize.imatrix.file                    Qwen3-0.6B-GGUF/imatrix_unsloth.dat
  quantize.imatrix.dataset                 unsloth_calibration_Qwen3-0.6B.txt
  quantize.imatrix.entries_count           196
  quantize.imatrix.chunks_count            688
header + tensor infos end at byte 5,952,180; tensor data follows, aligned to 32
parameters 596,049,920
  Q4_K         381,681,664   64.0 %
  Q6_K         214,302,720   36.0 %
  F32               65,536    0.0 %

This is config.json again under other names, plus everything that was in the side files, plus a record of how the quantisation was made:

config.json field	GGUF key	Value for Qwen3-0.6B
num_hidden_layers	qwen3.block_count	28
hidden_size	qwen3.embedding_length	1024
intermediate_size	qwen3.feed_forward_length	3072
num_attention_heads	qwen3.attention.head_count	16
num_key_value_heads	qwen3.attention.head_count_kv	8
head_dim	qwen3.attention.key_length, qwen3.attention.value_length	128
max_position_embeddings	qwen3.context_length, “the context (in tokens) that the model was trained on”	40960
tokenizer_config.json → chat_template	tokenizer.chat_template, “A Jinja template that specifies the input format expected by the model”	the Qwen3 template
torch_dtype	general.file_type, “the type of the majority of the tensors in the file”	15

The parameter count is 596,049,920, the model’s own count rather than the safetensors file’s, because the converter stored the tied head once. And the last three lines answer the question the file-size table left open: a “Q4_K_M” file is 64 per cent Q4_K tensors and 36 per cent Q6_K, with the normalisation vectors kept in F32, which is why it costs 0.61 to 0.67 bytes per parameter rather than 0.5. Part 6’s quantisation lesson explains which tensors get the larger type and why. Its filenames are structured too: the specification gives the convention as [Sidecar]-BaseName-SizeLabel-FineTune-Version-Encoding-Type-Shard.gguf, with the quantisation as the Encoding component. The unsloth names omit the Version component the specification calls the minimum, so they are read by eye rather than by the specification’s regular expression, which Qwen3-8B-Q4_K_M.gguf does not match; in an unsharded name the quantisation is the last component before .gguf.

Which format you want

The choice follows from what you are doing, not from preference:

You are going to	Format	Because	Where
Run a quantised model on llama.cpp, Ollama or LM Studio	GGUF	one file, mmap, the quantisation types those engines compute with	Parts 6 and 7
Serve with vLLM or SGLang, or run in Transformers	safetensors	those engines load the original tensor layout and quantise their own way	Parts 9 and 11
Run on MLX	safetensors, in an MLX repository or converted	MLX keeps its own layout, converted from the original	Part 8
Fine-tune or train	safetensors	training needs the tensors in their original shapes and precision	Parts 11 to 15
Publish a quantisation you made	GGUF, converted from safetensors	Part 6 converts and shows what is preserved and what is not	Part 6

Dense and mixture-of-experts

Everything so far has assumed a dense model: every parameter participates in every token. Qwen3-8B is dense, and generating one token costs a pass over all 8,190,735,360 parameters.

A mixture-of-experts model replaces the feed-forward sub-layer of each block with many parallel feed-forward networks, called experts, plus a small router that scores them and picks a few per token. The attention part is unchanged. Two of the course’s reference models are built this way:

Model	Card: total	Card: active per token	Card: architecture
Qwen3-30B-A3B	“30.5B in total and 3.3B activated”	3.3B	48 layers, “32 for Q and 4 for KV”, 128 experts with 8 activated
gpt-oss-20b	“21B parameters with 3.6B active parameters”	3.6B	24 layers, 32 experts with 4 per token (config), MXFP4 quantisation of the expert weights

Both numbers on Qwen3-30B-A3B’s card fall out of its config.json, which declares hidden_size 2048, moe_intermediate_size 768, num_experts 128, num_experts_per_tok 8, 48 layers, 32 query heads and 4 key-value heads of dimension 128, and an untied head. Per block:

Part of one block	Arithmetic	Resident	Read per token
Attention: q, k, v, o projections and two head norms	2 × 2048 × 4096 + 2 × 2048 × 512 + 256	18,874,624	18,874,624
One expert: gate, up, down	3 × 2048 × 768	4,718,592	—
128 experts	128 × 4,718,592	603,979,776	8 × 4,718,592 = 37,748,736
Router	128 × 2048	262,144	262,144
Two normalisation scales	2 × 2048	4,096	4,096
Per block		623,120,640	56,889,600
48 blocks + final norm		29,909,792,768	2,730,702,848
Embedding and output head	2 × 151,936 × 2048	622,329,856	622,329,856
Model		30,532,122,624	3,353,032,704

The card’s “29.9B” non-embedding, “30.5B” total and “3.3B” activated are these three numbers rounded. The same script gives them directly, which is what its second branch is for:

RunnableAll tracks

count the mixture-of-experts model

hf download Qwen/Qwen3-30B-A3B config.json --local-dir ~/llm-course/configs/qwen3-30b-a3b
python count-params.py ~/llm-course/configs/qwen3-30b-a3b/config.json

Output — what you should see

✓ Downloaded
  path: /home/you/llm-course/configs/qwen3-30b-a3b/config.json
model_type qwen3_moe: 48 layers, hidden 2048, vocab 151,936
  attention per layer          18,874,624
  feed-forward per layer      604,241,920   read per token 38,010,880
  non-embedding            29,909,792,768
  embedding + head            622,329,856
  TOTAL                    30,532,122,624   (30.53 B)
  ACTIVE per token          3,353,032,704   (3.35 B)

Publishers do not all count “active” the same way. gpt-oss-20b’s config.json gives 24 layers, hidden_size 2880, 32 experts of width 2880 with 4 used per token, 64 query heads and 8 key-value heads of dimension 64, biases on every attention projection and expert, and an untied head. Adding those up the way its modelling code lays them out gives 20,914,757,184 resident, which rounds to the card’s “21B”; the parameters read per token in the blocks come to 3,029,173,824, and the card’s “3.6B” is that plus the unembedding matrix (lm_head, 201,088 × 2,880 = 579,133,440) and not the embedding. The gpt-oss model card paper states the convention in the note to its Table 1, “Unembedding parameters are counted towards active, but not embeddings”, and gives 3.61B active and 20.91B total. Qwen counts both vocabulary matrices in “activated”; OpenAI counts one. Neither is wrong, and it is a reminder that the number on a card is a convention until you have reproduced it.

The consequence for a home machine splits cleanly in two, and Part 3’s inference lesson turns it into a speed ceiling:

At Q4_K_M, arithmetic from the counts above and the bytes-per-parameter table	Qwen3-8B, dense	Qwen3-30B-A3B, mixture of experts
Parameters resident	8,190,735,360	30,532,122,624
Weight bytes resident (models.json file sizes)	5.0 GB	18.6 GB
Parameters read per token	8,190,735,360	3,353,032,704
Weight bytes read per token, at 0.61 bytes per parameter	5.0 GB	about 2.0 GB

Memory follows the total, because the router may pick any expert for the next token and every expert must be resident. Speed follows the active count, because decode is bound by bytes moved per token, as Part 1’s precision lesson established, and only the selected experts are read. That is why a 30-billion-parameter mixture-of-experts model can generate at roughly the pace of a 3-billion-parameter dense one on the same machine, while needing almost four times the memory of the 8-billion-parameter dense one; Part 6’s benchmark lab has you measure both halves. gpt-oss-20b shows the other lever in the same place: its expert weights are published in MXFP4, the four-bit floating-point format Part 16’s QAT lesson covers, and the card states that this lets the model “run within 16GB of memory”. Fewer bytes per parameter and fewer parameters per token are independent savings and they multiply.

Your machine	Which architecture the arithmetic favours
Large unified memory, modest bandwidth (a 128 GB Spark, Ryzen AI Max+ or Mac)	mixture of experts: capacity to hold every expert, and speed set by the small active count
Discrete card with 8 to 24 GB, high bandwidth	dense: the total must fit first, and a dense 8B at Q4_K_M fits where a 30B mixture does not
Either, at long context	check the KV cache too; the last section of this lesson shows that the mixture model’s cache is the smaller one

Part 4’s architecture lesson takes this further, to hybrid models and to what routing does to quantisation.

Context window and context length

Two terms the course keeps apart, because engines do. The context window is what the model was trained to handle; the context length is what you ask an engine to allocate for a run. The first is a property of the checkpoint and lives on the card; the second is a flag, --ctx-size on llama-server, and costs memory whether or not you fill it.

The window is not one number either, and the three places it appears do not agree by design:

Model	Card: native window	Card: extended	max_position_embeddings	rope_scaling in the shipped config
Qwen3-8B	32,768	131,072 “with YaRN”	40,960	null; the card gives the YaRN block to add
Qwen3-30B-A3B	32,768	131,072 “with YaRN”	40,960	null; same
gpt-oss-20b	not stated on the card; 32 × 4,096 = 131,072 from the shipped YaRN block	—	131,072	yarn, factor 32.0 from an original_max_position_embeddings of 4,096

Qwen3’s card gives the extension as a configuration change, "rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768}, and then warns about it in two ways: frameworks “implement static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts”, and “If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance”. gpt-oss ships with the extension already applied: its rope_scaling block is in the file, and its max_position_embeddings is the product of that block’s factor and original_max_position_embeddings.

The decision rule is therefore three checks, in order. Does the conversation you intend fit the native window? If not, does the card document an extension, and at what cost? And does the cache for the length you will allocate fit beside the weights, which is the next section. Part 7’s reality check measures what happens when the allocated length is shorter than the conversation: the engine’s default is not a number you can look up, and a planted fact stops being recoverable at a length you can find.

The KV cache, introduced

The attention lesson said each new token attends to every previous one, using their keys and values. Recomputing those for the whole context at every step would be enormous waste, so they are stored. The Transformers documentation on cache strategies puts the reasoning plainly: for autoregressive models “KV scores are calculated every time because the model predicts one token at a time”, so “a KV cache stores these calculations so they can be reused without recomputing them”.

What is stored is concrete. Every layer keeps two tensors, and their shapes name the four fields that size them:

keys   per layer: [batch, num_key_value_heads, tokens, head_dim]
values per layer: [batch, num_key_value_heads, tokens, head_dim]

bytes per token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes_per_element
                  (2 for one key and one value; bytes_per_element is 2 at FP16 or BF16, 1 at 8-bit)

For Qwen3-8B that is 2 × 36 × 8 × 128 × 2, which is 147,456 bytes for every token in the context. You can watch the cache exist. This runs one forward pass over a twelve-token prompt, reads the cache object the model returns, and compares the bytes it holds with the formula; it needs the Transformers environment the lab installs and the Qwen3-0.6B directory the safetensors section fetched, and it runs on the CPU:

RunnableAll tracks

kv-cache-shapes.py

import sys, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = sys.argv[1]                                  # a downloaded model directory
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir, dtype=torch.bfloat16)
c = model.config
inputs = tokenizer("The KV cache stores one key and one value per token.", return_tensors="pt")
tokens = inputs["input_ids"].shape[1]

with torch.no_grad():
    out = model(**inputs, use_cache=True)                # one forward pass; the cache is built during prefill
cache = out.past_key_values
layer0 = cache.layers[0]
stored = sum(l.keys.numel() * l.keys.element_size() + l.values.numel() * l.values.element_size()
             for l in cache.layers)
predicted = 2 * c.num_hidden_layers * c.num_key_value_heads * c.head_dim * 2 * tokens

print(f"prompt tokens          {tokens}")
print(f"cache layers           {len(cache.layers)}   (config num_hidden_layers {c.num_hidden_layers})")
print(f"layer 0 keys           {tuple(layer0.keys.shape)}  {layer0.keys.dtype}   [batch, kv_heads, tokens, head_dim]")
print(f"layer 0 values         {tuple(layer0.values.shape)}")
print(f"bytes held in cache    {stored:,}")
print(f"formula predicts       {predicted:,}   match: {stored == predicted}")
print(f"per token              {stored // tokens:,} bytes")

RunnableAll tracks

run it on the 0.6B checkpoint

python kv-cache-shapes.py ~/llm-course/models/qwen3-0.6b

Output — what you should see

Loading weights: 100%|██████████| 311/311 [00:00<00:00, xxxx.xxit/s]
prompt tokens          12
cache layers           28   (config num_hidden_layers 28)
layer 0 keys           (1, 8, 12, 128)  torch.bfloat16   [batch, kv_heads, tokens, head_dim]
layer 0 values         (1, 8, 12, 128)
bytes held in cache    1,376,256
formula predicts       1,376,256   match: True
per token              114,688 bytes

Twelve tokens cost 1.4 MB, and the 8 in the shape is num_key_value_heads, not the 16 query heads this model has. Scaling to a real context is the calculator below, which reads the four fields from any config.json and prices three context lengths at two cache precisions:

RunnableAll tracks

kv-cache.py

import json, sys

c = json.load(open(sys.argv[1]))                        # any Hugging Face config.json
layers, kv_heads, head_dim = c["num_hidden_layers"], c["num_key_value_heads"], c["head_dim"]
per_token = {"FP16/BF16": 2, "Q8_0": 1}                 # bytes per element the engine keeps the cache in
contexts = [4096, 32768, 131072]

print(f"{layers} layers x {kv_heads} kv heads x {head_dim} head_dim x 2 (K and V)")
print(f"{'cache dtype':12s} {'bytes/token':>12s}" + "".join(f"{n:>14,}" for n in contexts))
for name, bytes_per_element in per_token.items():
    b = 2 * layers * kv_heads * head_dim * bytes_per_element
    print(f"{name:12s} {b:>12,}" + "".join(f"{b * n / 1e9:>11.2f} GB" for n in contexts))
if c.get("layer_types") or c.get("sliding_window"):
    print("note: this config declares sliding-window layers; the figures above are an upper bound")

RunnableAll tracks

price Qwen3-8B's context

python kv-cache.py ~/llm-course/configs/qwen3-8b/config.json

Output — what you should see

36 layers x 8 kv heads x 128 head_dim x 2 (K and V)
cache dtype   bytes/token         4,096        32,768       131,072
FP16/BF16         147,456       0.60 GB       4.83 GB      19.33 GB
Q8_0               73,728       0.30 GB       2.42 GB       9.66 GB

The same run on the other two configurations, collected (arithmetic, not measurements; an engine adds its own overhead on top):

Model	Layers × KV heads × head_dim	Bytes per token, FP16	4,096 tokens	32,768 tokens	131,072 tokens
Qwen3-1.7B	28 × 8 × 128	114,688	0.47 GB	3.76 GB	15.03 GB
Qwen3-8B	36 × 8 × 128	147,456	0.60 GB	4.83 GB	19.33 GB
Qwen3-30B-A3B	48 × 4 × 128	98,304	0.40 GB	3.22 GB	12.88 GB
gpt-oss-20b, upper bound	24 × 8 × 64	49,152	0.20 GB	1.61 GB	6.44 GB

Read the table against the weights. At the native window, Qwen3-8B’s cache is as large as its Q4_K_M weights; at the extended window it is larger than its BF16 weights. The 30-billion-parameter mixture model has the smaller cache, because it has half the key-value heads. And gpt-oss-20b’s row is a ceiling: its layer_types alternate between full attention and a 128-token sliding window, so half of its layers stop accumulating after 128 tokens, which the calculator’s last line flags.

Qwen3-8B at BF16 with a full 32,768-token context, on a 24 GB machine

Weights, BF16

16.4 GB

KV cache, 32,768 tokens

4.8 GB

Free

2.8 GB

Total

24 GB

Estimates from arithmetic, not measurements: 8,190,735,360 parameters at two bytes each for the weights, and 147,456 bytes per token of context for the cache. Activations and engine overhead are not shown and are not zero. Part 5 measures the real figures on each track.

Three things follow, and each is a later part of the course.

The cache is sized by the length you allocate, not the prompt you happen to send. Asking an engine for a long context reserves or grows toward that budget, which is why Parts 5 to 7 make you choose the context length deliberately and why Part 4’s budget lesson always adds weights and cache before saying a model fits.

Grouped-query attention is a cache decision as much as an attention one. Had Qwen3-8B stored 32 key-value heads instead of 8, every figure in its row would be four times larger and the model would not fit beside its own BF16 weights on a 24 GB machine.

There are ways to shrink it. The Transformers documentation describes offloading, which “saves GPU memory by moving the KV cache for model layers except one to the CPU”, and quantising it, with backends supporting int2, int4 and int8, warning that “Quantizing the cache can harm latency if the context length is short”. llama.cpp exposes the second as --cache-type-k and --cache-type-v, which is the Q8_0 row of the calculator. Part 17’s prefix-caching lesson teaches both properly, along with reusing a cache filled “with kv pairs for a certain prefix prompt”.

✕Common mistake

The symptom is an out-of-memory error at load time, or an engine that starts and then fails on the first long request, from a model whose weights “fit” with room to spare. The cause is a context length chosen without pricing it: a 131,072-token allocation for Qwen3-8B is 19 GB of cache before the first token. The fix is to run kv-cache.py on the configuration, add the result to the weight bytes, and pick the context length that leaves headroom, or an 8-bit cache, before touching the engine.

Reconcile three different model sizes

An operator encounters the parameter count in the model card, the downloaded file size and the process memory after loading. They measure different things. Parameter count describes learned arrays; file size includes their representation and metadata; runtime memory includes caches, temporary buffers and the engine’s allocations as well as weights.

For an illustrative dense checkpoint with a billion parameters, storing every parameter in two bytes gives a weight-only estimate of two billion bytes. This arithmetic excludes tokeniser files, metadata and runtime state. Quantising the main matrices does not imply every tensor receives the same bit width, so multiplying the parameter count by a nominal fraction of a byte is an approximation.

For a mixture-of-experts model, keep total and active parameters in separate columns. Active parameters help estimate per-token work; total parameters remain relevant to storage and residency. If your three sizes disagree, identify which additional term explains the gap. Do not “correct” the model card by substituting a process-memory reading, or conclude that a model fits merely because its compressed download is smaller than available memory.

Recap

A parameter count totals every learned number in the checkpoint and excludes activations, the KV cache and optimiser state; for Qwen3-8B the eleven tensor shapes of one block, times 36, plus two vocabulary matrices, reproduce the card’s 8.2B and 6.95B exactly, and multiplied by the bytes per parameter a file actually has, 2.00 at BF16 and 0.61 at Q4_K_M, the count becomes the weight memory. config.json holds the shape, and count-params.py turns it into the count. Checkpoints ship as safetensors, an eight-byte length, a JSON header and a byte buffer with no executable content, or as GGUF, a single file whose typed metadata carries the same numbers under other names along with the tokeniser and the chat template, and two short readers prove both. Mixture-of-experts models publish two counts: memory follows the total, speed follows the active count, and both fall out of the configuration. The context window is what the model was trained for and the context length is what you allocate; and the cache costs 2 × layers × key-value heads × head dimension × bytes per element for every token of that allocation, 147,456 bytes for Qwen3-8B, which a live forward pass confirms to the byte.

Check your understanding

Question 1. A model card says 30.5B total parameters and 3.3B activated. Which machine constraint does each number speak to?

• Both speak to memory
• The total decides whether the model fits in memory; the active count is what is read per token and therefore drives decode speed
• The total drives speed; the active count drives memory
• Neither; only the quantisation matters

Show the answer and why

Answer: The total decides whether the model fits in memory; the active count is what is read per token and therefore drives decode speed

Every expert must be resident because the router can select any of them, so the total sizes the memory. Only the selected experts are read per token, so bytes moved per token track the active count. For Qwen3-30B-A3B that is 8 of 128 experts per block, and the arithmetic reproduces both card numbers from config.json.

Question 2. Using 36 layers, 8 key-value heads and head_dim 128 at two bytes per element, roughly how much KV cache does Qwen3-8B need for 16,384 tokens?

• About 150 MB
• About 2.4 GB
• About 4.8 GB
• About 16 GB

Show the answer and why

Answer: About 2.4 GB

2 × 36 × 8 × 128 × 2 is 147,456 bytes per token. Times 16,384 tokens is about 2.4 GB, half the figure for the full 32,768-token context. The cache is linear in the allocated length, so halving the context halves the cache and doubling it doubles it.

Question 3. Which line of this KV-cache estimate for Qwen3-8B is the bug? (A) per_token = 2 * 36 * 32 * 128 * 2 (B) total = per_token * 32768 (C) gigabytes = total / 1e9

• Line A: it uses num_attention_heads (32) where the formula needs num_key_value_heads (8), so every figure is four times too large
• Line B: the cache should be multiplied by the number of layers again
• Line C: gigabytes should divide by 1e6
• Nothing is wrong; the estimate is 19.3 GB

Show the answer and why

Answer: Line A: it uses num_attention_heads (32) where the formula needs num_key_value_heads (8), so every figure is four times too large

Grouped-query attention stores keys and values for the 8 key-value heads only; the 32 query heads share them in groups of four. The live forward pass in this lesson shows the cache tensor shape as [1, 8, tokens, 128] on a model with 16 query heads. Line A gives 19.3 GB at 32,768 tokens where the correct figure is 4.8 GB.

Question 4. Which statements about the safetensors format are correct? Select all that apply.

• The file begins with an 8-byte little-endian integer giving the header size
• The JSON header records each tensor’s dtype, shape and data_offsets
• Loading a safetensors file executes code stored in it
• A file can store more parameters than the loaded model has, for example a tied output head saved as a separate tensor

Show the answer and why

Answer: The file begins with an 8-byte little-endian integer giving the header size, The JSON header records each tensor’s dtype, shape and data_offsets, A file can store more parameters than the loaded model has, for example a tied output head saved as a separate tensor

The format exists precisely because the pickle-based alternative could execute code on load. The header is JSON and separate from the byte buffer, which is what makes listing and partial loading cheap. Qwen3-0.6B’s file holds 751,632,384 parameters against the model’s 596,049,920 because lm_head.weight is stored even though tie_word_embeddings is true.

Question 5. A Q4_K_M GGUF file of Qwen3-8B is 5,027,784,512 bytes for 8,190,735,360 parameters. Why is that 0.61 bytes per parameter rather than 0.5?

• The file includes the KV cache
• The header reader shows that a Q4_K_M file mixes Q4_K and Q6_K tensors and keeps some vectors in F32, and every quantised block also stores its scales
• GGUF files are compressed and the size is misleading
• The count on the card is wrong

Show the answer and why

Answer: The header reader shows that a Q4_K_M file mixes Q4_K and Q6_K tensors and keeps some vectors in F32, and every quantised block also stores its scales

The reader in this lesson prints the type mix directly: for Qwen3-0.6B, 64 per cent of the parameters are Q4_K, 36 per cent Q6_K and the normalisation vectors F32. "Q4_K_M" names the majority type and the mix, not a uniform four bits; Part 6 explains which tensors get the larger type and why.

Question 6. Qwen3-8B’s card gives a native context of 32,768 tokens and its config.json gives max_position_embeddings 40960. You start an engine with a 131,072-token context length and no YaRN configuration. What have you done?

• Nothing wrong; the engine will use the model’s full 131,072-token window
• Allocated about 19 GB of cache for a window the model was not trained to handle without the extension the card describes, so both memory and quality beyond 32,768 tokens are at risk
• Made the model faster, because a larger cache means fewer recomputations
• Changed the parameter count

Show the answer and why

Answer: Allocated about 19 GB of cache for a window the model was not trained to handle without the extension the card describes, so both memory and quality beyond 32,768 tokens are at risk

Context length is what you allocate; the window is what the model was trained for. The cache cost is 147,456 bytes per token of the allocation, about 19.3 GB at 131,072, whether or not you fill it. Reaching that length well needs the YaRN rope_scaling block the card gives, and the card warns that enabling it also affects short-text quality, so the extension is a decision, not a default.

Check my answersClear