Choosing a Model for a Memory Budget

By the end of this lesson you will be able to compute, before downloading anything, what a model’s weights cost in any GGUF format down to the per-block scales; what its key-value cache costs at any context length, cache type and number of parallel conversations, the way llama.cpp will allocate it; how much to hold back for everything else; the highest decode speed that configuration can reach on your track; and whether a larger model at lower precision is a better use of the same bytes than a smaller one at higher precision, and which measurement settles it.

The ingredients come from earlier parts: bytes per parameter from Part 1’s precision section, the cache’s shape from Part 2’s KV cache introduction, and the decode ceiling from Part 3’s inference lesson. Later parts spend the budget this lesson draws up: Part 6 picks a quantisation with it, Part 9 sizes a server, Part 13 checks whether a fine-tune fits, and Parts 18 to 22 decide what must be split across machines. Every figure below is arithmetic from stated inputs (a config.json field, a file size from a Hub listing read on 2026-09-12, a line of llama.cpp source at the pinned v0.4.0), not a measurement. Part 6’s benchmark lab measures what the engine really allocates. Every model priced below (Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B, Qwen3-32B and Qwen3-235B-A22B from Qwen, and gpt-oss-20b and gpt-oss-120b from OpenAI) is released under Apache-2.0, and none is gated.

Units, stated once

File sizes and budgets in this course are in gigabytes of 10⁹ bytes, because the Hub’s listings report bytes and publishers quote decimal gigabytes. Operating systems, nvidia-smi and llama.cpp’s load log use binary units. One conversion of each kind, done once:

Quantity	Bytes	GB (109)	GiB (230)	MiB (220)
Qwen3-8B-Q4_K_M.gguf	5,027,784,512	5.03	4.68	4,794.87
Qwen3-8B’s KV cache, 32,768 tokens, f16	4,831,838,208	4.83	4.50	4,608.00
A graphics card sold as 16 GB (16 GiB)	17,179,869,184	17.18	16.00	16,384.00

A GiB is 7.4 per cent larger than a GB. The budget tables below treat a tier label such as “16 GB” as 16 decimal gigabytes, which understates a 16 GiB card by 1.18 GB: a deliberate margin.

What the weights cost

Weights cost parameters times bytes per parameter. The part that needs care is the bytes per parameter of a quantised file, which is neither the nominal bit width nor the same across files that share a name.

Blocks and scales

A quantised tensor is stored as a run of fixed-size blocks. Each block holds a group of small integers plus the higher-precision numbers that turn them back into real values: a scale, sometimes a minimum, and in the K-quants a second level of scales for the scales. Because the block is the unit of storage, the cost of a type is exact:

bytes_per_parameter = bytes_per_block / weights_per_block
bits_per_weight     = 8 × bytes_per_block / weights_per_block

The layouts are C structs in ggml’s ggml-common.h, each guarded by a static_assert on its size. Three of them at llama.cpp v0.4.0:

Type	One block holds	Bytes per block	Weights per block	Bits per weight
Q8_0	one 16-bit scale; 32 signed 8-bit integers	2 + 32 = 34	32	8.5
Q4_K	a 16-bit scale and a 16-bit minimum for the super-block; 12 bytes packing a 6-bit scale and a 6-bit minimum for each of 8 sub-blocks of 32; 256 four-bit values	4 + 12 + 128 = 144	256	4.5
Q6_K	256 six-bit values stored as 128 bytes of low nibbles and 64 bytes of high pairs; an 8-bit scale for each of 16 sub-blocks of 16; a 16-bit super-block scale	128 + 64 + 16 + 2 = 210	256	6.5625

The Hub’s GGUF documentation states the same bits per weight for Q4_K, Q5_K, Q6_K and IQ4_XS. The snippet below runs the arithmetic for every type this course downloads, then does something a table cannot: it rebuilds a real file tensor by tensor. It is standard-library Python, so save it as bits-per-weight.py and run python3 bits-per-weight.py anywhere.

RunnableAll tracks

bits-per-weight.py

"""Bits per weight from ggml's block layouts, then one real file rebuilt tensor by tensor."""

# bytes per block, weights per block: the struct sizes asserted in ggml-common.h
BLOCKS = {
    "F16/BF16": (2, 1),     "Q8_0": (34, 32),       "Q6_K": (210, 256),
    "Q5_K": (176, 256),     "Q4_K": (144, 256),     "Q4_0": (18, 32),
    "IQ4_XS": (136, 256),   "MXFP4": (17, 32),      "Q3_K": (110, 256),
    "Q2_K": (84, 256),
}
for name, (nbytes, nweights) in BLOCKS.items():
    print(f"{name:9s} {nbytes * 8 / nweights:7.4f} bits/weight {nbytes / nweights:7.4f} bytes/param")

def size(params, qtype):                       # bytes a tensor occupies in that type
    nbytes, nweights = BLOCKS[qtype]
    return params * nbytes // nweights

# Qwen3-8B config.json: hidden 4096, 32 query heads and 8 KV heads of 128, FFN 12288, vocab 151936
h, kv, ffn, vocab, layers = 4096, 8 * 128, 12288, 151936, 36
def more_bits(i, n):                           # llama-quant.cpp: which layers Q4_K_M widens
    return i < n // 8 or i >= 7 * n // 8 or (i - n // 8) % 3 == 2

total = size(vocab * h, "Q4_K") + size(vocab * h, "Q6_K")     # embedding table, output head
total += 4 * h                                                  # final norm, F32
for i in range(layers):
    wide = "Q6_K" if more_bits(i, layers) else "Q4_K"
    total += size(h * h, "Q4_K") * 2 + size(h * kv, "Q4_K")     # q and o projections, k projection
    total += size(h * kv, wide) + size(h * ffn, wide)           # v projection, FFN down
    total += size(h * ffn, "Q4_K") * 2                          # FFN gate and up
    total += 4 * (2 * h + 2 * 128)                              # four norm vectors, F32
params, published = 8_190_735_360, 5_027_784_512                # Hub: parameter count, file bytes
print(f"\nQwen3-8B Q4_K_M: {sum(more_bits(i, layers) for i in range(layers))} of {layers} layers widened")
print(f"tensor bytes rebuilt  {total:,}")
print(f"published file bytes  {published:,}  (difference {published - total:,}: header and tokeniser)")
print(f"file-level            {published * 8 / params:.3f} bits/weight, {published / params:.4f} bytes/param")

Output — what you should see

F16/BF16  16.0000 bits/weight  2.0000 bytes/param
Q8_0       8.5000 bits/weight  1.0625 bytes/param
Q6_K       6.5625 bits/weight  0.8203 bytes/param
Q5_K       5.5000 bits/weight  0.6875 bytes/param
Q4_K       4.5000 bits/weight  0.5625 bytes/param
Q4_0       4.5000 bits/weight  0.5625 bytes/param
IQ4_XS     4.2500 bits/weight  0.5312 bytes/param
MXFP4      4.2500 bits/weight  0.5312 bytes/param
Q3_K       3.4375 bits/weight  0.4297 bytes/param
Q2_K       2.6250 bits/weight  0.3281 bytes/param

Qwen3-8B Q4_K_M: 18 of 36 layers widened
tensor bytes rebuilt  5,021,827,072
published file bytes  5,027,784,512  (difference 5,957,440: header and tokeniser)
file-level            4.911 bits/weight, 0.6138 bytes/param

Why the file is heavier than its block type

Q4_K_M is not a block type. It is a recipe that picks a type for each tensor, written in llama-quant.cpp. At v0.4.0 the recipe never quantises one-dimensional tensors, so norm vectors stay F32; it stores the output head in Q6_K; and it widens attn_v and ffn_down to Q6_K in the layers its use_more_bits rule selects, which are the first eighth, the last eighth and every third layer between them, 18 of Qwen3-8B’s 36. Everything else is Q4_K. Applied to the shapes in config.json, that recipe reproduces the tensor bytes to the byte; the 5,957,440 bytes left over are the header, metadata and tokeniser that Part 3’s tensor table lists as a row of its own. An output head of 622 million parameters at 6.56 bits and half the layers’ largest matrices widened is why Q4_K_M costs 4.91 bits per weight rather than 4.5.

The same reasoning covers MXFP4, whose block is one shared 8-bit exponent and 32 four-bit values (17 bytes, 4.25 bits). In the ggml-org gpt-oss files the expert tensors are MXFP4. Attention and the output head are Q8_0, as Part 3’s tensor table shows, and so is the token embedding table; the router and biases stay F32, so the whole file lands at 4.34 bits per weight.

Bytes per parameter, by format

The block figure is the floor; the file figure is what you budget. File figures below divide a published size by the Hub’s parameter count:

Format	Block bits per weight	A real file	File bits per weight	File bytes per parameter
FP32	32	the format	32	4.00
BF16 or FP16	16	Qwen3-8B-BF16.gguf, 16,388,044,384 bytes	16.01	2.001
Q8_0	8.5	Qwen3-8B-Q8_0.gguf, 8,709,519,168 bytes	8.51	1.063
Q6_K	6.5625	Qwen3-8B-Q6_K.gguf, 6,725,900,096 bytes	6.57	0.821
Q5_K_M	5.5, some tensors Q6_K	Qwen3-8B-Q5_K_M.gguf, 5,851,113,280 bytes	5.71	0.714
Q4_K_M	4.5, some tensors Q6_K	Qwen3-8B-Q4_K_M.gguf, 5,027,784,512 bytes	4.91	0.614
IQ4_XS	4.25, some tensors wider	Qwen3-8B-IQ4_XS.gguf, 4,581,287,744 bytes	4.47	0.559
MXFP4 (gpt-oss)	4.25 experts; 8.5 attention, embedding and output head	gpt-oss-120b-MXFP4.gguf, 63,387,346,208 bytes	4.34	0.543
Q3_K_M	3.4375, some tensors wider	Qwen3-8B-Q3_K_M.gguf, 4,124,161,856 bytes	4.03	0.504
Q2_K	2.625, some tensors wider	Qwen3-8B-Q2_K.gguf, 3,281,733,440 bytes	3.21	0.401

Across the unsloth Qwen3 files from 4B to 32B, Q4_K_M lands between 0.603 and 0.621 bytes per parameter and Q8_0 at 1.063 or 1.064. So for arithmetic in your head: about 0.6 GB per billion parameters at Q4_K_M, 1.06 at Q8_0, 2.0 at BF16. A size read from the listing beats all three, and the lab’s calculator uses one wherever it records one. The low types drift furthest from their blocks, because the recipe protects the same sensitive tensors whatever the target: “two-bit” Q2_K is 3.21 bits per weight on this model.

Qwen publishes its own GGUF conversions as well. Its Qwen3-8B repository lists five files, Q4_K_M, Q5_0, Q5_K_M, Q6_K and Q8_0; the four that unsloth also ships are each 1,024 to 1,056 bytes smaller than unsloth’s file of the same type, and unsloth ships no Q5_0 (listings read 2026-09-13). The course standardises on unsloth/Qwen3-*-GGUF because that one namespace ships every tier the course uses, from Q2_K to BF16 and IQ4_XS for the 235B, so paths and sizes stay the same from Part 6 onwards.

✕Common mistake

Estimating a mixture from its nominal bits. Qwen3-235B-A22B has 235,093,634,560 parameters: at a round four bits that is 117.55 GB, at IQ4_XS’s 4.25 it is 124.89 GB, and the published IQ4_XS shards add up to 125,504,687,488 bytes. The first figure appears to leave room on a 128 GB machine; the file does not. Read the listing, and for a sharded model add every shard.

What the context costs

Every token in the context leaves one key vector and one value vector in every attention layer, so that later tokens can attend to it without recomputing it. Part 2 showed the two tensors per layer, [batch, kv_heads, tokens, head_dim], and confirmed their size on a live forward pass. The cost per token of context:

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element

2                  one key and one value
layers             num_hidden_layers whose attention spans the whole context
kv_heads           num_key_value_heads, not num_attention_heads
head_dim           head_dim, or hidden_size / num_attention_heads when the field is absent
bytes_per_element  2 for f16 or bf16; 34/32 for q8_0; 18/32 for q4_0

The last line carries a detail from the weights section: an engine stores a quantised cache in ggml blocks too, so a q8_0 cache costs 1.0625 bytes per element, not 1.

kv_heads is where this arithmetic most often goes wrong. Grouped-query attention lets several query heads share one key-value head; the GQA paper reports quality close to full multi-head attention at a speed comparable to multi-query attention. Qwen3-8B has 32 query heads and 8 key-value heads, so reading the wrong field overstates its cache fourfold.

Reading the numbers out of a config file

Everything the formula needs is in config.json, a few kilobytes you can read in a browser. The relevant fields of two of them:

Qwen/Qwen3-8B            num_hidden_layers 36, num_attention_heads 32, num_key_value_heads 8,
                         head_dim 128, max_position_embeddings 40960, torch_dtype bfloat16
openai/gpt-oss-120b      num_hidden_layers 36, num_attention_heads 64, num_key_value_heads 8,
                         head_dim 64, sliding_window 128,
                         layer_types alternating sliding_attention and full_attention, 18 of each

For Qwen3-8B with an f16 cache, 2 × 36 × 8 × 128 × 2 = 147,456 bytes per token. The course’s reference set, by the plain formula:

Model	Layers	KV heads	Head dim	Bytes per token, f16	8,192 tokens	32,768 tokens
Qwen3-4B	36	8	128	147,456	1.21 GB	4.83 GB
Qwen3-8B	36	8	128	147,456	1.21 GB	4.83 GB
Qwen3-14B	40	8	128	163,840	1.34 GB	5.37 GB
Qwen3-30B-A3B	48	4	128	98,304	0.81 GB	3.22 GB
Qwen3-32B	64	8	128	262,144	2.15 GB	8.59 GB
Qwen3-235B-A22B	94	4	128	192,512	1.58 GB	6.31 GB
gpt-oss-20b	24, half sliding	8	64	49,152, a ceiling	0.40 GB	1.61 GB
gpt-oss-120b	36, half sliding	8	64	73,728, a ceiling	0.60 GB	2.42 GB

The 30-billion-parameter mixture of experts has a smaller cache than the dense 8B because it has half the key-value heads, and Qwen3-32B’s cache at 32,768 tokens is larger than a whole Qwen3-8B at Q4_K_M. The gpt-oss rows are ceilings because half their layers attend only to the last 128 tokens; the architecture lesson extends the formula to sliding-window and hybrid layers. The next section goes the other way, down to what the engine allocates.

What the engine allocates

An engine allocates cells, not formulas. llama.cpp v0.4.0 applies six rules that a plain multiplication misses, each readable in its source or README:

Rule	Where it lives	Consequence
The context is rounded up to a multiple of 256 cells	llama-context.cpp	32,768 is unchanged; 10,000 becomes 10,240
No --ctx-size (the default, 0) means the model’s training context	README, llama-context.cpp	40,960 cells for the Qwen3 GGUF files, 131,072 for gpt-oss, before --fit adjusts it (see the headroom rules)
--cache-type-k and --cache-type-v accept f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, default f16	README	quantised types pay their block scales
A quantised V cache needs Flash Attention	llama-context.cpp	with --flash-attn off the context is refused with “quantized V cache requires flash_attn to be enabled”; the default auto switches Flash Attention on
Sliding-window layers keep the window times the sequences sharing the pool, plus --ubatch-size (default 512), rounded up to 256, capped at the context	llama-kv-cache-iswa.cpp	gpt-oss with one sequence: 768 cells in those 18 layers at any context; --swa-full gives them the full context instead
Parallel slots share or split the cells	llama-context.cpp, server.cpp, common/fit.cpp	the default, automatic --parallel means 4 slots in one unified pool of --ctx-size cells; an explicit --parallel N (or --no-kv-unified) splits the cells, --ctx-size ÷ N per slot, and with no --ctx-size sizes the cache for N training contexts

The calculator encodes those rules for the three models of the worked examples, at three context lengths and three cache types. Save it as kv-allocation.py and run python3 kv-allocation.py; it needs nothing beyond the standard library.

RunnableAll tracks

kv-allocation.py

"""KV cache as llama.cpp allocates it: full and sliding-window layers, cache type, padding."""
MiB, GB = 2**20, 1e9
CACHE = {"f16": (2, 1), "q8_0": (34, 32), "q4_0": (18, 32)}   # bytes per block, elements per block

MODELS = {  # config.json: full-attention layers, sliding layers, KV heads, head_dim, window; GGUF context
    "Qwen3-8B":      (36,  0, 8, 128,   0,  40_960),
    "Qwen3-30B-A3B": (48,  0, 4, 128,   0,  40_960),
    "gpt-oss-120b":  (18, 18, 8,  64, 128, 131_072),
}

def pad(n, to=256):
    return (n + to - 1) // to * to

def kv_bytes(model, ctx, cache="f16", seqs=1, ubatch=512):
    full, sliding, kv_heads, head_dim, window, _ = model
    nbytes, nelem = CACHE[cache]
    per_cell_layer = 2 * kv_heads * head_dim * nbytes // nelem  # K and V, one token, one layer
    cells = pad(ctx)                                          # llama-context.cpp pads n_ctx to 256
    swa_cells = pad(min(cells, window * seqs + ubatch))       # llama-kv-cache-iswa.cpp
    return full * per_cell_layer * cells + sliding * per_cell_layer * swa_cells

for name, m in MODELS.items():
    full, sliding, kv_heads, head_dim, window, trained = m
    naive = 2 * (full + sliding) * kv_heads * head_dim * 2
    print(f"{name}: plain formula {naive:,} bytes/token at f16")
    for ctx in (8_192, 32_768, trained):
        cells = "  ".join(f"{c} {kv_bytes(m, ctx, c) / MiB:9.2f} MiB {kv_bytes(m, ctx, c) / GB:6.2f} GB"
                          for c in CACHE)
        print(f"  {ctx:>7,} tokens  {cells}")

Output — what you should see

Qwen3-8B: plain formula 147,456 bytes/token at f16
    8,192 tokens  f16   1152.00 MiB   1.21 GB  q8_0    612.00 MiB   0.64 GB  q4_0    324.00 MiB   0.34 GB
   32,768 tokens  f16   4608.00 MiB   4.83 GB  q8_0   2448.00 MiB   2.57 GB  q4_0   1296.00 MiB   1.36 GB
   40,960 tokens  f16   5760.00 MiB   6.04 GB  q8_0   3060.00 MiB   3.21 GB  q4_0   1620.00 MiB   1.70 GB
Qwen3-30B-A3B: plain formula 98,304 bytes/token at f16
    8,192 tokens  f16    768.00 MiB   0.81 GB  q8_0    408.00 MiB   0.43 GB  q4_0    216.00 MiB   0.23 GB
   32,768 tokens  f16   3072.00 MiB   3.22 GB  q8_0   1632.00 MiB   1.71 GB  q4_0    864.00 MiB   0.91 GB
   40,960 tokens  f16   3840.00 MiB   4.03 GB  q8_0   2040.00 MiB   2.14 GB  q4_0   1080.00 MiB   1.13 GB
gpt-oss-120b: plain formula 73,728 bytes/token at f16
    8,192 tokens  f16    315.00 MiB   0.33 GB  q8_0    167.34 MiB   0.18 GB  q4_0     88.59 MiB   0.09 GB
   32,768 tokens  f16   1179.00 MiB   1.24 GB  q8_0    626.34 MiB   0.66 GB  q4_0    331.59 MiB   0.35 GB
  131,072 tokens  f16   4635.00 MiB   4.86 GB  q8_0   2462.34 MiB   2.58 GB  q4_0   1303.59 MiB   1.37 GB

At 131,072 tokens gpt-oss-120b’s allocation is 4.86 GB, half the 9.66 GB the plain formula gives. The architecture lesson’s 1.21 GB at 32,768 tokens counts what attention reads; the 1.24 GB here adds the 640 cells of padding and batch room in the sliding layers.

The MiB column is what to compare with the engine. Part 6’s benchmark lab starts the server at three context lengths with the short forms -ngl 99 -c; its 32,768-token launch, written out with long options, is:

Fragment — not complete on its own

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --gpu-layers 99 \
    --ctx-size 32768

Its load log carries one line per cache, and the format string in llama-kv-cache.cpp at v0.4.0 composes it from exactly these numbers. The 4/1 seqs is the four automatic slots sharing one pool. A build other than the pinned one may word it differently, and your logger may add a prefix:

Output — what you should see

llama_kv_cache: size = 4608.00 MiB ( 32768 cells,  36 layers,  4/1 seqs), K (f16): 2304.00 MiB, V (f16): 2304.00 MiB

Three worked examples

Each example states its inputs, prices the launches you might actually use, and ends with the decode ceiling from Part 3, tokens_per_second ≤ bandwidth / (weights read per token + cache read per token), with the weights-read figures Part 3 took from each file’s tensor table and the vendor bandwidth figures in the hardware reference. The reserves are allowances set by the headroom rules further down.

Qwen3-8B at Q4_K_M on a 16 GB card

Qwen3-8B is the first real model on every track. On a Track N card sold as 16 GB the budget is 16 GB less 1.5 GB for the engine’s buffers, 14.5 GB, on a card that is not also drawing a desktop.

Launch	Cells	Cache	KV cache	Weights + cache	Left of 14.5 GB
--ctx-size 8192	8,192	f16	1.21 GB	6.24 GB	8.26 GB
--ctx-size 32768	32,768	f16	4.83 GB	9.86 GB	4.64 GB
no --ctx-size, and --fit finds it fits	40,960	f16	6.04 GB	11.07 GB	3.43 GB
--ctx-size 32768, q8_0 cache	32,768	q8_0	2.57 GB	7.59 GB	6.91 GB
four slots of 32,768: --parallel 4 --ctx-size 131072 --no-kv-unified	4 × 32,768	f16	19.33 GB	24.36 GB	9.86 GB over
the same with a q8_0 cache	4 × 32,768	q8_0	10.27 GB	15.30 GB	0.80 GB over
the same with a q4_0 cache	4 × 32,768	q4_0	5.44 GB	10.46 GB	4.04 GB

Over budget does not mean refused. Under default flags both over-budget launches load: --fit leaves as many layers, with their cache, in system memory as it must, and every layer left there pulls decode towards system-memory speed. With -ngl 99, --fit gives up: the 24.36 GB launch fails with the allocation error the headroom rules quote, and whether the 15.30 GB one loads depends on what its compute buffers and the driver really take on your card.

Qwen3-8B, Q4_K_M, 32,768 tokens at f16, on a 16 GB card

Weights, Q4_K_M

5.0 GB

KV cache, 32,768 tokens, f16

4.8 GB

Reserved for engine buffers

1.5 GB

Free

4.6 GB

Total

16 GB

Estimate. Weights are the published size of unsloth's Qwen3-8B-Q4_K_M.gguf; the cache is 147,456 bytes per token from the model's config; the reserve is an allowance for compute buffers, not a measurement.

Tokens in context	Bytes read per token	Ceiling, Track S (273 GB/s)	Track X (256 GB/s)	Track M, M4 Max (546 GB/s)	Track N, RTX 5090 (1,792 GB/s)
0	4.67 GB	58.4	54.8	116.9	383.6
32,768	9.50 GB	28.7	26.9	57.5	188.6

For one user the card is comfortable at any context the model was trained for. Four simultaneous 32k conversations are where it runs out, and only a 4-bit cache fits them; whether that cache is acceptable is a quality question Part 16’s method measures, not one this arithmetic answers. Ceilings are in tokens per second, and a 16 GB card’s bandwidth is below the RTX 5090 figure, so substitute your own from Part 5. They halve between an empty context and 32,768 tokens, because at that length the cache read per token is as large as the weights.

Qwen3-30B-A3B on a 64 GB unified-memory machine

Qwen3-30B-A3B on a 64 GB Track X or Track M machine. The operating system and your applications share the pool, so hold back 8 GB, a budget of 56 GB. Two quantisations are candidates:

Configuration	Weights	KV cache, f16	Weights + cache	Left of 56 GB
Q4_K_M, 32,768 tokens	18.56 GB	3.22 GB	21.78 GB	34.22 GB
Q4_K_M, four slots of 32,768	18.56 GB	12.88 GB	31.44 GB	24.56 GB
Q8_0, 32,768 tokens	32.48 GB	3.22 GB	35.71 GB	20.29 GB
Q8_0, no --ctx-size (40,960)	32.48 GB	4.03 GB	36.51 GB	19.49 GB
Q8_0, four slots of 32,768	32.48 GB	12.88 GB	45.37 GB	10.63 GB

Qwen3-30B-A3B, Q8_0, 32,768 tokens at f16, on a 64 GB unified-memory machine

Weights, Q8_0

32.5 GB

KV cache, 32,768 tokens, f16

3.2 GB

Reserved for OS and applications

8 GB

Free

20.3 GB

Total

64 GB

Estimate. Weights are the published size of unsloth's Qwen3-30B-A3B-Q8_0.gguf; the cache is 98,304 bytes per token from the config's 48 layers and 4 key-value heads; the reserve is an allowance for the operating system and applications on a shared pool.

Speed is where the two differ. A mixture of experts reads its attention, router, norms and output head for every token but only 8 of each layer’s 128 experts. Part 3 read the Q4_K_M split from the tensor table (0.82 GB shared, 17.55 GB of experts); at Q8_0 the config’s shapes at 1.0625 bytes per parameter, with the router left unquantised as llama-quant.cpp leaves it and counted at 4 bytes, give about 3.27 GB per token:

Weights, tokens in context	Bytes read per token	Ceiling, Track S (273 GB/s)	Track X (256 GB/s)	Track M, M4 Max (546 GB/s)	Track N (1,792 GB/s)
Q4_K_M, 0	1.92 GB	142.2	133.3	284.4	933.3 on a 32 GB card
Q4_K_M, 32,768	5.14 GB	53.1	49.8	106.2	348.6 on a 32 GB card
Q8_0, 0	3.27 GB	83.5	78.3	167.0	548.1 on a 96 GB card
Q8_0, 32,768	6.49 GB	42.1	39.4	84.1	276.1 on a 96 GB card

Both fit with room. Q8_0 costs 13.93 GB more memory and 41 per cent of the empty-context ceiling, 21 per cent at 32,768 tokens where the cache dominates the bytes read. Whether its smaller rounding error is worth that on your task is Part 16’s measurement; the arithmetic’s contribution is that both options exist on this machine and only Q4_K_M exists on a 32 GB card. On a Mac, the wired-memory limit decides whether the GPU may hold 36 GB at all.

gpt-oss-120b in MXFP4 on a 128 GB machine

gpt-oss-120b has 116,829,156,672 parameters, of which about 5.1 billion are active per token according to its card, in the ggml-org MXFP4 file of 63.39 GB. On a 128 GB DGX Spark, reserve 10 GB, a budget of 118 GB. Its context window is 131,072 tokens, and its sliding-window layers make long context cheap:

Launch	Full-attention layers	Sliding-window layers	KV cache, f16	Weights + cache	Left of 118 GB
--parallel 1 --ctx-size 32768	18 × 32,768 cells	18 × 768 cells	1.24 GB	64.62 GB	53.38 GB
--parallel 1 --ctx-size 131072	18 × 131,072	18 × 768	4.86 GB	68.25 GB	49.75 GB
server defaults: no --ctx-size, 4 unified slots	18 × 131,072	18 × 1,024	4.87 GB	68.26 GB	49.74 GB
--parallel 1 --ctx-size 131072 --swa-full	36 × 131,072	none sliding	9.66 GB	73.05 GB	44.95 GB
four slots of 131,072 each: --parallel 4 --ctx-size 524288	4 × 18 × 131,072	4 × 18 × 768	19.44 GB	82.83 GB	35.17 GB

gpt-oss-120b, MXFP4, llama-server defaults (131,072 cells, 4 slots), on a 128 GB unified-memory machine

Weights, MXFP4 experts and Q8_0 attention

63.4 GB

KV cache, 131,072 cells, f16

4.9 GB

Reserved for OS and applications

10 GB

Free

49.7 GB

Total

128 GB

Estimate. Weights are the published size of ggml-org's gpt-oss-120b-MXFP4.gguf; the cache is 18 full-attention layers of 131,072 cells plus 18 sliding-window layers of 1,024 cells at 2,048 bytes per cell per layer, following llama.cpp v0.4.0's allocation rules; the reserve is an allowance.

Tokens in context	Bytes read per token	Ceiling, Track S (273 GB/s)	Track X, 128 GB (256 GB/s)	Track M, M4 Max 128 GB (546 GB/s)	Track N, 96 GB card (1,792 GB/s)
0	3.59 GB	75.9	71.2	151.9	498.5
32,768	4.81 GB	56.8	53.3	113.6	372.8

The ceiling falls by a quarter from an empty context to 32,768 tokens, against a half for Qwen3-8B, because only 18 layers of 8 heads of dimension 64 grow. The weights alone exceed every consumer graphics card, so on Track N this is a 96 GB workstation card or nothing. What the 128 GB machine buys is the 50 GB left over: room for four long conversations, or for a second model, such as the 0.85 GB eagle3-gpt-oss-120b-Q8_0.gguf draft file the same repository ships, for the speculative decoding Part 9 teaches.

What fits where

The budget each track gives you

“Memory” means a different number on each track, and the right one is what the engine can place tensors in:

Track	Memory classes	The budget is	Read it with	Reserve	Vendor bandwidth
S, DGX Spark	128 GB unified	the pool; llama.cpp’s CUDA backend treats MemAvailable in /proc/meminfo as free memory on this machine	grep MemAvailable /proc/meminfo	10 GB	273 GB/s
X, Ryzen AI Max+ 395	64 or 128 GB unified	the part the GPU can address: about 96 GB on Windows at 128 GB; on Linux, the GTT limit the kernel sets	Part 5’s GPU memory section	8 GB at 64, 10 GB at 128	256 GB/s
M, Apple silicon	24 to 512 GB unified	Metal’s recommended working-set size, which llama.cpp reports as the device total and which is below the machine total by default	python3 -c "import torch; print(torch.mps.recommended_max_memory() / 1e9)" in the Part 1 environment	3 GB up to 32 GB, 8 GB at 64, 10 GB above	by chip, 120 (M4) to 1,200 (M5 Ultra) GB/s
N, GeForce RTX or RTX PRO	8 to 96 GB of VRAM	VRAM only; system memory is reachable across PCIe at a fraction of the speed	nvidia-smi --query-gpu=memory.total,memory.used --format=csv	1.5 GB, plus whatever memory.used shows before the engine starts	936 (RTX 3090) to 1,792 (RTX 5090) GB/s

The grid

Each cell gives the longest context at which a configuration fits: full is the training context llama-server allocates with no --ctx-size (40,960 tokens for the Qwen3 files, 131,072 for gpt-oss), then 32k and 8k, all with an f16 cache and one sequence. Budgets are the column’s memory less the reserve above; weights are published file sizes, the gpt-oss caches follow the allocation rules, and the 235B sizes add up the shards.

Configuration	Weights	KV at 32,768	N 8	N 12	N 16	N 24	N 32	M 24	M 32	N 48	X, M 64	N 96	S, X, M 128	M 256	M 512
Qwen3-4B Q4_K_M	2.50 GB	4.83 GB	8k	full	full	full	full	full	full	full	full	full	full	full	full
Qwen3-8B Q4_K_M	5.03 GB	4.83 GB	8k	32k	full	full	full	full	full	full	full	full	full	full	full
Qwen3-8B Q8_0	8.71 GB	4.83 GB	no	8k	32k	full	full	full	full	full	full	full	full	full	full
Qwen3-14B Q4_K_M	9.00 GB	5.37 GB	no	8k	32k	full	full	full	full	full	full	full	full	full	full
Qwen3-14B Q6_K	12.12 GB	5.37 GB	no	no	8k	full	full	full	full	full	full	full	full	full	full
gpt-oss-20b MXFP4	12.11 GB	0.82 GB	no	no	32k	full	full	full	full	full	full	full	full	full	full
Qwen3-30B-A3B Q4_K_M	18.56 GB	3.22 GB	no	no	no	32k	full	8k	full	full	full	full	full	full	full
Qwen3-32B Q4_K_M	19.76 GB	8.59 GB	no	no	no	8k	32k	no	32k	full	full	full	full	full	full
Qwen3-30B-A3B Q8_0	32.48 GB	3.22 GB	no	no	no	no	no	no	no	full	full	full	full	full	full
Qwen3-32B Q8_0	34.82 GB	8.59 GB	no	no	no	no	no	no	no	full	full	full	full	full	full
gpt-oss-120b MXFP4	63.39 GB	1.24 GB	no	no	no	no	no	no	no	no	no	full	full	full	full
Qwen3-235B-A22B Q3_K_M	112.45 GB	6.31 GB	no	no	no	no	no	no	no	no	no	no	8k	full	full
Qwen3-235B-A22B IQ4_XS	125.50 GB	6.31 GB	no	no	no	no	no	no	no	no	no	no	no	full	full
Qwen3-235B-A22B Q6_K	193.02 GB	6.31 GB	no	no	no	no	no	no	no	no	no	no	no	full	full
Qwen3-235B-A22B Q8_0	249.94 GB	6.31 GB	no	no	no	no	no	no	no	no	no	no	no	no	full

Three readings matter. The same 24 GB fits Qwen3-30B-A3B at 32k on a card and only at 8k on a Mac, because the Mac’s reserve covers macOS, and only once the wired limit allows 21 GB. The 235B model’s IQ4_XS files do not fit any 128 GB machine at any context; its Q3_K_M files fit at 8k, 3.83 bits per weight, which is the trade the last section of this lesson examines, and the IQ4_XS row is why Part 19 builds a two-machine cluster. And the Q8_0 235B does not fit a 256 GB Mac: 249.94 GB of weights plus 6.31 GB of cache is past the budget before macOS takes anything.

Every Track M column, and the Track X columns, assume the GPU may address the whole budget. On a Mac that means a raised wired limit (Part 5); otherwise use torch.mps.recommended_max_memory() minus the reserve as your column. On Track X under Windows the 128 GB machine belongs in the N 96 column, whose verdicts are the same with a 10 GB reserve. One cell is marked down from the arithmetic: Qwen3-32B Q4_K_M on N 32 fits its full context with 431,712 bytes to spare, no margin at all, so it reads 32k.

Headroom rules

A budget is the memory the engine may use minus a reserve, and the reserve is several different claims on memory. llama.cpp accounts for three of them per device in a memory breakdown table, whose columns common/fit.cpp defines at v0.4.0:

total = free + self + unaccounted          self = model + context + compute

model is the weights on that device, context the KV cache, compute the scratch buffers for the largest batch, and unaccounted whatever else holds device memory: the driver, a desktop on the same card, other programs. At v0.4.0 llama-server prints this table when it shuts down, at trace level, so start it with --log-verbosity 4 to see it; the prefix is common_memory_breakdown_print:. The fit-params README shows the table in its own example, an RTX 4090 at build 6895 under the older prefix: not the pinned version, not a course measurement. Its context column is exactly Qwen3-30B-A3B’s 98,304 bytes per token × 4,096 tokens, and nearly 4 GB sat in unaccounted:

Output — what you should see

the fit-params README's example, build 6895, RTX 4090

llama_memory_breakdown_print: | memory breakdown [MiB] | total   free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - CUDA0 (RTX 4090)   | 24077 =  945 + (19187 = 17904 +     384 +     898) +        3945 |

Claim on memory	Allowance	Why	Where to see the real figure
Operating system, desktop, browser (unified memory)	3 GB up to 32 GB; 8 GB at 64; 10 GB above	they share the pool; these are the lab calculator’s defaults	MemAvailable before loading on Linux; Activity Monitor’s Memory tab on a Mac
Driver, display and other programs (discrete card)	what memory.used shows before the engine starts, on top of the 1.5 GB	whatever else holds the card lands in unaccounted, 3,945 MiB in the example above	nvidia-smi before the engine starts; the unaccounted column after (with --log-verbosity 4)
Compute buffers	1 to 2 GB for one llama.cpp session	sized for the largest batch the engine processes, --ubatch-size 512 by default	the compute column (with --log-verbosity 4); otherwise device memory in use after loading, less weights and cache
llama-server’s host prompt cache	up to 8,192 MiB	--cache-ram defaults to 8192 at v0.4.0; on unified memory it comes from the same pool	lower --cache-ram, or 0 to disable it
A second model	its file plus its own cache	draft, embedding and reranking models load beside the main one	the second engine’s own memory report

Three rules about context sit outside that table, and they are where budgets that fit on paper fail on the machine.

Budget the context you allocate, not the one you use. llama.cpp allocates every cell at load. With no --ctx-size, llama-server at v0.4.0 starts from the training context (one per slot when the cache is split) and --fit, on by default, shrinks it until each device keeps --fit-target of 1,024 MiB free, never below --fit-ctx of 4,096 (again per slot when split). That margin is measured against what the backend reports free: MemAvailable on a Spark, the Metal working-set size minus current allocations on a Mac. It protects the engine at load time, not the browser you open afterwards. When shrinking the context is not enough, or --ctx-size was given and the launch still does not fit, --fit places layers in system memory instead.

Count conversations. Giving each of N slots its own full context multiplies the cache by N (the slot rows of the first and third worked examples).

A server that pre-allocates takes its fraction up front. vLLM 0.28.0 claims --gpu-memory-utilization of the device, 0.92 by default, loads the weights and turns the rest into KV-cache blocks, so its headroom is whatever the other 8 per cent covers. Part 9’s vLLM lesson sets it for a real model.

✕Common mistake

Expecting an over-budget load to stop. At v0.4.0 a default llama-server launch (--fit on, --gpu-layers auto) does not stop when the budget is exceeded: common/fit.cpp shrinks an unset context, no lower than --fit-ctx, then moves layers (or, for a mixture of experts, expert tensors) into system memory. On a discrete card the model loads and decodes at system-memory speed across PCIe, and the load log’s offloaded N/M layers to GPU line shows N below M. The allocation error “failed to allocate buffer for kv cache” appears when fitting is off (--fit off) or when you set --gpu-layers yourself, as Part 6’s lab does with -ngl 99, and the cache still does not fit. On a unified-memory machine the operating system can also swap or compress memory, with the same symptom: a model that loads, runs and generates far below its ceiling. If a model that should be fast is slow, check that line, free memory and swap before anything else; Part 6’s two-tokens-per-second challenge walks the diagnosis.

Bigger and quantised, or smaller and not?

On a 16 GB card, a 14.5 GB budget, with 32,768 tokens of context, four Qwen3 files of about the same size compete. The table is arithmetic from the Hub’s parameter counts and file sizes; the decode column divides bytes read per token and so needs no bandwidth, and the prefill column is Part 3’s two operations per non-embedding parameter per token:

Option	Parameters	File bits per weight	File	KV at 32,768	Weights + cache	Decode ceiling relative to the 4B, empty / 32k	Prefill arithmetic per token
Qwen3-4B, BF16	4.02 billion	16.01	8.05 GB	4.83 GB	12.88 GB	1.00 / 1.00	7.3 GFLOP
Qwen3-8B, Q8_0	8.19 billion	8.51	8.71 GB	4.83 GB	13.54 GB	1.00 / 1.00	13.9 GFLOP
Qwen3-14B, Q4_K_M	14.77 billion	4.88	9.00 GB	5.37 GB	14.37 GB	0.94 / 0.92	26.4 GFLOP
Qwen3-32B, UD-IQ2_XXS	32.76 billion	2.27	9.28 GB	8.59 GB	17.87 GB, does not fit	0.90 / 0.73	62.4 GFLOP

Two things fall out before any quality question. For dense models an equal file size means an almost equal decode ceiling: the 4B reads its whole file because its output head is its embedding table, while the 8B and 14B skip their 0.66 GB and 0.44 GB embedding tables, so the first three sit within 8 per cent. The 2-bit file keeps its embedding table in Q3_K and its output head in Q5_K; a token skips the 0.33 GB table and reads 8.94 GB, not the file’s 9.28 GB. Prefill does not equalise: the 14B does 3.6 times the 4B’s arithmetic per prompt token, which on a compute-bound long prompt means roughly 3.6 times the wait for the first token. And the 2-bit 32B fails on the cache, not the weights, because 64 layers cost 262,144 bytes per token.

What the evidence says

The arithmetic cannot say which option answers better. Three published results bear on it, each with limits:

• Dettmers and Zettlemoyer (2022) ran more than 35,000 experiments on BLOOM, OPT, Pythia and GPT-2 models from 19 million to 176 billion parameters at 3 to 8 bits, and concluded that “4-bit precision is almost universally optimal for total model bits and zero-shot accuracy”: at a fixed number of bits, more parameters at 4 bits did better than fewer at higher precision. The only reliable improvements they found were small block sizes and the choice of data type, the block scales of this lesson’s first section. The families studied predate Qwen3, and zero-shot accuracy is not your task.
• Kumar and colleagues (2024) report that “the degradation introduced by post-training quantization increases as models are trained on more data”, fitted on more than 465 pretraining runs and validated on models up to 1.7 billion parameters trained on up to 26 billion tokens. The Qwen3 report states that all Qwen3 models were pre-trained on 36 trillion tokens, about 8,950 tokens per parameter for the 4B and 1,100 for the 32B. That is a reason to expect the small model of a family to lose more per bit than the large one, not a measurement: nobody has validated that law at this scale.
• An empirical study of Qwen3 quantisation (2025) finds the family “maintains competitive performance at moderate bit-widths” but “experiences notable degradation in linguistic tasks under ultra-low precision”.

The decision rule

If, for your budget and context	Lean towards	Because
Same family and tokeniser; the larger option is a 4-bit K-quant or wider (Q4_K_M and up; IQ4_XS is the edge) and fits at your context	the larger model, quantised	4 bits was near-optimal per total bit in the k-bit study; the dense decode ceiling barely moves at equal file size; the smaller model saw more tokens per parameter
The larger option needs a 3-bit or lower type: Q3_K, IQ3, Q2_K, IQ2 or a 2-bit dynamic mix	the smaller model, unless your task set says otherwise	the study’s optimum was 4 bits, not below; Qwen3 degrades notably at ultra-low precision
Long prompts dominate: retrieval, whole files, agents re-reading context	the smaller model, or measure time to first token first	prefill arithmetic scales with non-embedding parameters, 3.6 times for the 14B against the 4B
You need long context or several conversations at once	whichever leaves the cache room	larger models usually have more layers, so a bigger cache per token
The candidates come from different families or generations	neither, by arithmetic; measure	the bits-against-parameters trade is a within-family result
The task depends on exact formats: JSON, tool calls, code that must compile	measure at the bit width you intend before choosing	Part 16 explains why format habits live in a few high-stakes tokens that averaged metrics miss

How to settle it on your task

KL divergence, Part 16’s primary quantisation metric, compares a quantised file with its own full-precision original. It ranks Q4_K_M against Q8_0 of one model and cannot rank a 14B at Q4_K_M against an 8B at Q8_0, because the two have different references. Perplexity on one fixed text is comparable across the Qwen3 sizes, which share a tokeniser (the same shared tokeniser is what lets Part 15’s logit distillation pair a larger Qwen3 teacher with a smaller Qwen3 student), so it makes a cheap first cut. The decision belongs to your own task set from Part 10’s lab, run once per candidate file, with the noise estimated as Part 16’s measurement lesson describes. “Indistinguishable on my tasks” is a good result: take the option with the cheaper prefill and the more cache room.

▷Exercise

Do this now, before the lab. Add a row for a model you use, or would like to, to MODELS in kv-allocation.py, with the layer counts, key-value heads, head dimension and window from its config.json. Take the size of the file you would download from its Hub listing. Write four numbers in your lab notebook: weights, cache at the context you want, your track’s reserve, and what is left of your budget. The lab’s calculator automates the f16 case with the plain formula; for a quantised cache or a sliding-window model, trust kv-allocation.py.

Budget for the peak, then choose a fallback

Suppose the weights fit in idle memory but the first long request fails. The missing terms may be KV allocation, temporary prefill buffers, graph capture or a second model loaded for embeddings. A deployment budget must include the peak configuration, not only a process immediately after startup.

Write the sum as weights plus per-request state plus shared runtime overhead plus other resident services plus operating-system headroom. Estimate the parts you can derive, then measure the actual peak at the intended prompt length and concurrency. On unified memory, the operating system and accelerator draw from the same capacity; on a discrete GPU, abundant host RAM does not make device allocations free.

Specify the fallback before loading: shorten the configured context if the task permits, reduce simultaneous requests, choose a smaller representation or use a smaller model. Each changes a different property, so record which you used. Passing a short smoke request establishes only the first rung of the capacity ladder. The useful completion criterion is that the largest permitted request can run under the service’s admitted load with acceptable headroom.

Recap

Bytes per parameter follow from a type’s block layout, and a file’s figure from its recipe, which widens sensitive tensors: Qwen3-8B at Q4_K_M rebuilds to the byte at 4.91 bits per weight. The cache costs 2 × layers × kv_heads × head_dim × bytes_per_element per token, and the engine’s allocation rules decide the real figure. A budget subtracts the reserve for what shares the memory, counts the context allocated and the conversations served, and the bytes read per token divide the bandwidth into a decode ceiling. Within a family, a larger model at a 4-bit type or wider belongs on the shortlist ahead of a smaller one at higher precision; at 3-bit and 2-bit types, and across families, only your task set decides.

Check your understanding

Question 1. Qwen3-14B has 40 layers, 40 query heads, 8 key-value heads and a head dimension of 128. How much does llama.cpp allocate for its KV cache at --ctx-size 32768 with --cache-type-k q8_0 --cache-type-v q8_0?

• About 2.68 GB: 2 × 40 × 8 × 128 × 1 byte × 32,768
• About 2.85 GB: 2 × 40 × 8 × 128 × 34/32 bytes × 32,768
• About 5.37 GB: the f16 figure, since the cache type does not change the allocation
• About 14.3 GB: 2 × 40 × 40 × 128 × 34/32 bytes × 32,768

Show the answer and why

Answer: About 2.85 GB: 2 × 40 × 8 × 128 × 34/32 bytes × 32,768

The cache is a ggml tensor, so q8_0 pays its block scale: 34 bytes for every 32 elements, 1.0625 bytes each. That gives 87,040 bytes per token and 2,852,126,720 bytes at 32,768 tokens. One byte per element understates it by 6 per cent, and the query head count would overstate it fivefold.

Question 2. Which of these llama-server launches fails at load time on llama.cpp v0.4.0?

• llama-server --model qwen3-8b-q4_k_m.gguf --ctx-size 32768 --cache-type-k q8_0
• llama-server --model qwen3-8b-q4_k_m.gguf --ctx-size 32768 --cache-type-v q8_0
• llama-server --model qwen3-8b-q4_k_m.gguf --ctx-size 32768 --flash-attn off --cache-type-v q8_0
• llama-server --model qwen3-8b-q4_k_m.gguf --ctx-size 32768 --flash-attn off --cache-type-k q8_0

Show the answer and why

Answer: llama-server --model qwen3-8b-q4_k_m.gguf --ctx-size 32768 --flash-attn off --cache-type-v q8_0

A quantised V cache requires Flash Attention. With the default --flash-attn auto, llama.cpp switches it on for you; with it forced off, llama_init_from_model refuses the context with "quantized V cache requires flash_attn to be enabled". A quantised K cache alone carries no such requirement in that check.

Question 3. You start llama-server for Qwen3-8B with --parallel 4 --ctx-size 32768 --no-kv-unified. What does each conversation get, and what does the cache cost at f16?

• 32,768 tokens each, and 19.33 GB in total
• 8,192 tokens each, and 4.83 GB in total
• 32,768 tokens shared, and 4.83 GB in total
• 8,192 tokens each, and 1.21 GB in total

Show the answer and why

Answer: 8,192 tokens each, and 4.83 GB in total

With a split cache the context is divided among the slots: 32,768 ÷ 4 = 8,192 cells each, still 32,768 cells of 147,456 bytes in total. To give four people 32,768 tokens at once you allocate 131,072 cells, four times the memory: 19.33 GB, more than a 16 GB card holds beside the weights.

Question 4. gpt-oss-120b has 36 layers of 8 key-value heads of dimension 64, and the plain formula gives 9.66 GB of f16 cache at 131,072 tokens. llama.cpp allocates about 4.86 GB for one sequence. Why?

• MXFP4 weights imply a 4-bit cache
• Half the layers use a 128-token sliding window, so they keep 768 cells (window plus the 512-token micro-batch, rounded up to 256) instead of 131,072
• llama.cpp caps every cache at 64,000 tokens
• Grouped-query attention halves it a second time

Show the answer and why

Answer: Half the layers use a 128-token sliding window, so they keep 768 cells (window plus the 512-token micro-batch, rounded up to 256) instead of 131,072

Eighteen full-attention layers hold all 131,072 cells and eighteen sliding-window layers hold 768, so the total is about half the formula. --swa-full gives the sliding layers the full context and the allocation returns to 9.66 GB. The cache type is chosen with --cache-type-k and --cache-type-v, independently of the weights format.

Question 5. On a 14.5 GB budget at 32,768 tokens you compare Qwen3-4B at BF16, Qwen3-8B at Q8_0, Qwen3-14B at Q4_K_M and Qwen3-32B at UD-IQ2_XXS. Which statements does the arithmetic support? Select all that apply.

• The first three have decode ceilings within about 8 per cent of each other at the same bandwidth
• The 32B option does not fit at 32,768 tokens, because of its cache
• The 14B option needs about 3.6 times the 4B option's arithmetic per prompt token
• The 14B option at Q4_K_M will give better answers on your task than the 8B at Q8_0

Show the answer and why

Answer: The first three have decode ceilings within about 8 per cent of each other at the same bandwidth, The 32B option does not fit at 32,768 tokens, because of its cache, The 14B option needs about 3.6 times the 4B option's arithmetic per prompt token

Equal file sizes mean nearly equal bytes read per token for dense models; the 32B needs 9.28 GB of weights plus 8.59 GB of cache; prefill arithmetic scales with non-embedding parameters. Which answers better is not arithmetic: the published evidence favours the larger model at 4 bits within a family, and your task set decides.

Question 6. Your 128 GB machine runs Qwen3-235B-A22B at Q3_K_M with 8,192 tokens of context. A colleague suggests IQ4_XS instead, "since 235 billion parameters at about 4.25 bits is 125 GB and that is under 128". What is wrong with the suggestion?

• Nothing; it fits with 3 GB to spare
• The published IQ4_XS shards add up to 125.50 GB before any cache, reserve or operating system, so no 128 GB machine holds it
• IQ4_XS cannot be loaded by llama.cpp
• IQ4_XS is smaller than Q3_K_M

Show the answer and why

Answer: The published IQ4_XS shards add up to 125.50 GB before any cache, reserve or operating system, so no 128 GB machine holds it

Weights plus 1.58 GB of cache at 8,192 tokens is already 127 GB, and a 128 GB machine needs about 10 GB for everything else. Q3_K_M at 112.45 GB fits at 8,192 tokens; IQ4_XS needs a 256 GB Mac or two machines, which is Part 19.

Check my answersClear