Dense, Mixture-of-Experts and Hybrid Architectures

By the end of this lesson you will be able to compute what a mixture-of-experts router does to one token, separate the bytes a model keeps resident from the bytes it reads per token, explain what a state-space or linear-attention layer keeps instead of a key-value cache and why hybrids still keep some attention, price the cache of a dense, sliding-window or hybrid model from its config.json, read a GGUF tensor table to see what quantisation touches in a mixture of experts, and decide for your own track whether an architecture relieves the constraint your machine actually has.

The practical question underneath it: Qwen3-32B and Qwen3-30B-A3B, both Apache-2.0, take almost the same memory at Q4_K_M, and the arithmetic below predicts the second decodes about five times faster at a 32k context on any machine that holds both. The “30B” in the name does not tell you that. Part 2 counted Qwen3-30B-A3B’s active parameters and Part 3 turned bytes read into a decode ceiling; this lesson reuses both and goes one level down.

Dense: every parameter, every token

A dense model multiplies each token’s representation through every weight matrix of every layer. The cost model is the one Part 1’s precision lesson and Part 3’s inference lesson set out:

resident bytes        = every tensor in the file
bytes read per token  = every tensor except the embedding table (one row of it is looked up)
decode ceiling        ≈ bandwidth / (bytes read per token + cache bytes read per token)

The course’s Qwen3-32B file, Qwen3-32B-Q4_K_M.gguf from unsloth/Qwen3-32B-GGUF, is 19,762,150,048 bytes. Its tensor table holds 19,756,174,336 bytes of tensors, of which the embedding table is 437,575,680, so 19.32 GB is read for every token out of 19.76 GB resident. Double the parameters and both numbers double. That simplicity is also the problem: every token, easy or hard, pays for all of them.

Mixture of experts: many parameters, few of them per token

A mixture-of-experts model breaks that link. The Hugging Face explainer defines it as replacing the dense feed-forward block in a layer with a set of parallel feed-forward blocks, the experts, plus a small router that decides which experts each token is sent to. Attention is untouched.

The Switch Transformer paper (Fedus, Zoph and Shazeer, 2021) set out the modern form: send each token to few experts, so the computation per token stays roughly constant while the parameter count grows; it reports “up to 7x increases in pre-training speed with the same computational resources” against dense baselines. The DeepSeek-V3 technical report (December 2024) describes the refinements most current models use: many smaller, finely divided experts so the router’s choice is more expressive, shared experts that every token passes through, and what it calls auxiliary-loss-free load balancing, set out below.

What happens to one token in a mixture-of-experts layer

1. 1Attention blockRuns for every token, using the same weights every time. This part is identical to a dense model.
2. 2Router scores the expertsOne matrix multiply gives one score per expert for this token; the highest few are selected. Qwen3-30B-A3B selects 8 of 128; gpt-oss-120b selects 4 of 128.
3. 3Selected experts computeOnly those experts' weights are read and multiplied. A shared expert, where the design has one, runs for every token as well.
4. 4Outputs are combinedThe selected experts' outputs are weighted by their gate values and summed, and the result carries on to the next layer.
5. 5Next token: different expertsThe routing is per token and cannot be known in advance, which is why every expert has to stay resident.

Attention runs for every token as usual. Only the feed-forward part is routed, and only the selected experts' weights take part in the arithmetic.

The router, one level down

For one token with hidden state h in one layer:

scores   = W_router · h                    one number per expert; W_router is num_experts × hidden_size
p        = softmax(scores)
chosen   = the num_experts_per_tok largest entries of p
gate_i   = p_i / (sum of p over chosen)    for each chosen expert i   (norm_topk_prob: true)
output   = sum over chosen of gate_i × expert_i(h)
         + shared_expert(h), where the design has one; Qwen3-Next scales it by sigmoid(w_shared · h)

That is Qwen3MoeTopKRouter in Transformers 5.16.1, and Qwen3-Next’s router is the same. GptOssTopKRouter in the same release adds a learned bias to the scores, takes the top four raw scores first and applies the softmax to those four only. The two orders give identical gates, because a softmax’s ratios depend only on differences between scores, and renormalising over the chosen experts cancels the rest. The fields that configure all of it sit in each config.json:

config.json field	Qwen3-30B-A3B	Qwen3-235B-A22B	gpt-oss-120b	Qwen3-Next-80B-A3B	Nemotron 3 Nano 30B-A3B
Experts per layer	num_experts 128	num_experts 128	num_local_experts 128	num_experts 512	n_routed_experts 128
Experts per token	num_experts_per_tok 8	8	4	10	6
Shared experts	none	none	none	one, shared_expert_intermediate_size 512	n_shared_experts 1
Width of one expert	moe_intermediate_size 768	1,536	intermediate_size 2,880	512	1,856
Score function (Transformers 5.16.1)	softmax	softmax	softmax over the top 4; the router has a bias	softmax	sigmoid; e_score_correction_bias added for selection only; gates × routed_scaling_factor 2.5
Chosen gates renormalised	norm_topk_prob true	true	softmax over the 4	true	true
Balance-loss coefficient	router_aux_loss_coef 0.001	0.001	0.9	0.001	not in the config
Licence	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0	NVIDIA Nemotron Open Model License

Nemotron 3 Nano’s router, NemotronHTopkRouter in the same release, is the exception: it scores each expert with a sigmoid, adds a per-expert selection bias before choosing, takes the gates from the unbiased scores, renormalises them and multiplies by the scaling factor. That is the balancing method of the DeepSeek-V3 technical report, and it is why the Nemotron column has no balance-loss coefficient. Without a softmax, neither the formula nor the equivalence above holds for this model: applied to it, they give the wrong gates.

This routes 512 token states through a router shaped like Qwen3-30B-A3B’s. The weights are random, so it shows the mechanism, not the model’s real routing. Save it as route.py in ~/llm-course with Part 2’s environment active, as every snippet on this page. On Track S, inside the container, type /workspace/course wherever this page writes ~/llm-course, including the config and GGUF paths below:

RunnableAll tracks

route.py

import numpy as np

rng = np.random.default_rng(0)
hidden, n_experts, k = 2048, 128, 8             # sizes from Qwen3-30B-A3B's config.json
W = rng.normal(0, 0.02, (n_experts, hidden))    # the router: one row of weights per expert
h = rng.normal(0, 1, (512, hidden))             # 512 token states arriving at one layer

def softmax(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

logits = h @ W.T                                # one score per token per expert: [512, 128]
p = softmax(logits)
top = np.argsort(-p, axis=1)[:, :k]             # the k experts each token is sent to
gates = np.take_along_axis(p, top, 1)
gates /= gates.sum(axis=1, keepdims=True)       # norm_topk_prob: true
gates_gptoss = softmax(np.take_along_axis(logits, top, 1))   # top-k first, then softmax
print("same gates in both orders:", np.allclose(gates, gates_gptoss))
print("token 0 -> experts", top[0].tolist())
print("       gate weights", np.round(gates[0], 3).tolist())

for t in (1, 8, 64, 512):
    used = len(np.unique(top[:t]))
    print(f"distinct experts used by the first {t:>3} tokens: {used:>3} of {n_experts}")

def balance_loss(p, top):                       # N x sum(f_i x P_i), as Transformers computes it
    f = np.bincount(top.ravel(), minlength=n_experts) / len(p)
    return n_experts * np.sum(f * p.mean(axis=0))

collapsed = np.zeros_like(p)
collapsed[:, :k] = 1 / k                        # every token sends everything to experts 0 to 7
first_k = np.tile(np.arange(k), (len(p), 1))
print(f"balance loss, this router      {balance_loss(p, top):6.2f}")
print(f"balance loss, collapsed router {balance_loss(collapsed, first_k):6.2f}")

RunnableAll tracks

run route.py

python ~/llm-course/route.py

Output — what you should see

same gates in both orders: True
token 0 -> experts [14, 80, 20, 25, 5, 58, 42, 77]
       gate weights [0.176, 0.167, 0.138, 0.13, 0.106, 0.1, 0.096, 0.087]
distinct experts used by the first   1 tokens:   8 of 128
distinct experts used by the first   8 tokens:  47 of 128
distinct experts used by the first  64 tokens: 128 of 128
distinct experts used by the first 512 tokens: 128 of 128
balance loss, this router        8.05
balance loss, collapsed router 128.00

One token reads 8 of the 128 experts; eight tokens have already touched 47 of them. A trained router is less even than a random one, so real counts differ, and the course has no measurement of them. What no router offers is advance notice of which experts the next token needs, and fetching one from storage mid-token would put the slowest link in Part 1’s memory hierarchy into every decode step. Every expert therefore stays resident.

Load balancing, and what it prevents

The last two lines print the balance term Transformers 5.16.1 adds to the training loss, in the form its docstring attributes to the Switch Transformer paper:

balance_loss = num_experts × sum over experts of f_i × P_i
  f_i  = share of the batch's expert assignments that went to expert i    (sums to experts per token)
  P_i  = mean router probability given to expert i                         (sums to 1)
  perfectly even routing  →  experts per token    (8 for Qwen3-30B-A3B)
  complete collapse       →  num_experts          (128)

Training multiplies it by router_aux_loss_coef and adds it to the language-modelling loss. The Hugging Face explainer describes what it prevents: “the gating network converges to mostly activate the same few experts. This self-reinforces as favored experts are trained quicker and hence selected more.” The collapsed router is that end state, with 120 experts resident and idle. The design in the DeepSeek-V3 technical report relies mainly on a different method: it adds a per-expert bias to the scores “to determine the top-K routing”, lowers the bias of an overloaded expert and raises that of an underloaded one after each training step, and notes that “the bias term is only used for routing”; the gate still comes from the original score. The design keeps only a small complementary sequence-wise balance loss, “to prevent extreme imbalance within any single sequence”, and computes its scores with a sigmoid rather than a softmax. You meet balance again at inference in two places: expert parallelism in Part 18, where a popular expert is a busy device, and batching in Part 3, where the spread of routing sets how many experts one step reads.

Two numbers, two different costs

Part 3 put the trade as “capacity is paid in memory, speed is paid in bytes read” and priced it in bytes for both mixtures of experts. The table repeats Part 3’s tensor-table figures for those two, adds the dense Qwen3-32B and one new row, the capacity multiplier; gguf-groups.py, in the quantisation section, reads its tensor byte counts from the files:

resident bytes       = always-read tensors + embedding table + all routed experts
read per token       = always-read tensors + (experts per token ÷ experts per layer) × all routed experts
capacity multiplier  = resident bytes ÷ read per token

At batch 1, from the GGUF tensor tables (retrieved 2026-09-12)	Qwen3-32B Q4_K_M, dense	Qwen3-30B-A3B Q4_K_M	gpt-oss-120b MXFP4
File size, bytes	19,762,150,048	18,556,686,912	63,387,346,208
Routed-expert tensors, bytes	none	17,553,162,240	61,073,326,080
Always read: everything but the embedding table and the routed experts	19,318,598,656	822,523,904	1,685,668,608
Experts read per token, bytes	none	8 of 128: 1,097,072,640	4 of 128: 1,908,541,440
Read per token	19.32 GB	1.92 GB	3.59 GB
Capacity multiplier	1.0	9.7	17.6
Block parameters multiplied per token	31,206,298,624	2,730,702,848	4,553,716,032
Arithmetic per token in the blocks, 2 × that	62.4 GFLOP	5.5 GFLOP	9.1 GFLOP

Read the multiplier as an exchange rate: gpt-oss-120b turns 17.6 GB of memory into the per-token reading cost of 1 GB. The last row matters wherever a step is compute-bound, which is prefill, so a mixture of experts cuts both the bytes and the arithmetic per token by about an order of magnitude against the dense model its memory resembles.

Qwen3-32B (dense) at Q4_K_M, 32k context, on a 64 GB machine

Weights, Q4_K_M

19.8 GB

KV cache at 32k tokens, FP16

8.6 GB

Free

35.6 GB

Total

64 GB

Estimates. Weights are the size of the course's unsloth/Qwen3-32B-GGUF Q4_K_M file; the KV cache is computed from the model's 64 layers, 8 key-value heads and head dimension 128 at FP16. Not measured on hardware.

Qwen3-30B-A3B (mixture of experts) at Q4_K_M, 32k context, on a 64 GB machine

Weights, Q4_K_M

18.6 GB

KV cache at 32k tokens, FP16

3.2 GB

Free

42.2 GB

Total

64 GB

Estimates, computed the same way from the course's unsloth/Qwen3-30B-A3B-GGUF Q4_K_M file and the model's 48 layers, 4 key-value heads and head dimension 128. The weights are almost the same size as the dense model's; the cache is smaller because the attention configuration is smaller, not because the model has experts.

Two things the arithmetic cannot settle. Whether Qwen3-30B-A3B, “30.5B in total and 3.3B activated” on its card, answers your task as well as the 32.8B dense parameters on Qwen3-32B’s card is a measurement: the Qwen3-Next card, for one, claims its 80B-A3B base model “outperforms Qwen3-32B-Base on downstream tasks with 10% of the total training cost”, and that stays a claim with an owner and a date until Part 16’s evaluation methods reproduce it on your work. And the advantage is a batch-1 figure: each sequence in a batch routes on its own, so a step reads the union of their experts, which Part 3 quantifies.

✕Common mistake

The mistake is sizing memory from the active count. “A3B” and the gpt-oss-120b card’s “5.1B active parameters” are speed figures. The symptom is an engine that fails to allocate, or on a unified-memory machine a load that tips the system into swap and generates far below the predicted ceiling, for a model that “only has 3 billion parameters”. The fix is to budget the file, 18.56 GB for Qwen3-30B-A3B at Q4_K_M, plus the cache, as the memory-budget lesson does.

Hybrid: replacing some attention with a recurrent state

Attention reads every cached key and value at every step, which is why the cache and the bytes read per token grow with context. The alternative line of work is state-space models, of which Mamba (Gu and Dao, 2023) is the reference: its abstract describes making the model’s parameters functions of the input so it can selectively propagate or forget information, with linear scaling in sequence length. The property that matters to a memory budget is that such a layer carries a fixed-size state forward instead of a cache.

What a recurrent layer keeps instead of a cache

Take the softmax out of attention and the sum over the cache can be regrouped into one matrix, updated in place:

softmax attention   o_t = sum over j ≤ t of softmax_j(q_t · k_j) v_j      reads t cached (k, v) pairs
linear attention    o_t = sum over j ≤ t of (q_t · k_j) v_j = S_t q_t,    S_t = S_(t-1) + v_t k_tᵀ
Gated DeltaNet      S_t = α_t S_(t-1);   S_t = S_t + β_t (v_t − S_t k_t) k_tᵀ;   o_t = S_t q_t
Mamba-2             h_t = dA_t h_(t-1) + dB_t x_t;   y_t = C_t h_t

S is a value-dimension by key-dimension matrix (Transformers stores the transpose, [key, value]) whatever t is. The Gated DeltaNet line is the recurrent rule in Transformers 5.16.1’s modeling_qwen3_next.py: α_t, between 0 and 1, decays everything stored, and β_t sets how strongly the new value replaces what key k_t currently retrieves. The Gated Delta Networks paper calls the two complementary: “gating enables rapid memory erasure while the delta rule facilitates targeted updates”. The Mamba-2 line is the state update in modeling_nemotron_h.py, where h plays the role of S.

The first half of this script shows the recurrent form is exactly linear attention; the second shows what a fixed state costs, by storing key-to-value facts in a 64 × 64 state and in an attention cache and asking for each fact back:

RunnableAll tracks

state.py

import numpy as np

rng = np.random.default_rng(0)
d, T = 64, 1000
q, k, v = (rng.normal(size=(T, d)) / np.sqrt(d) for _ in range(3))

# Linear attention (no softmax), computed two ways
by_cache = np.stack([v[: t + 1].T @ (k[: t + 1] @ q[t]) for t in range(T)])  # re-read all pairs
S, by_state = np.zeros((d, d)), np.empty((T, d))
for t in range(T):
    S += np.outer(v[t], k[t])                   # S_t = S_(t-1) + v_t k_t^T
    by_state[t] = S @ q[t]                      # o_t = S_t q_t
print(f"largest difference between the two: {np.abs(by_cache - by_state).max():.1e}")
print(f"held after {T} tokens: cache {2 * T * d:,} numbers, state {d * d:,} numbers")

def recall(n):                                  # store n key -> value facts, then ask for each
    keys = rng.normal(size=(n, d))
    keys /= np.linalg.norm(keys, axis=1, keepdims=True)
    vals = rng.normal(size=(n, d))
    S = np.zeros((d, d))
    for kk, vv in zip(keys, vals):              # delta rule: replace what this key retrieves now
        S += np.outer(vv - S @ kk, kk)
    weights = np.exp(20 * keys @ keys.T)        # softmax attention over a cache of all n pairs
    from_attn = (weights / weights.sum(axis=1, keepdims=True)) @ vals
    unit = vals / np.linalg.norm(vals, axis=1, keepdims=True)
    hit = lambda got: np.mean((got @ unit.T).argmax(axis=1) == np.arange(n))
    return hit(keys @ S.T), hit(from_attn)

for n in (16, 64, 256, 1024):
    s, a = recall(n)
    print(f"{n:>5} facts: fixed {d}x{d} state recalls {s:4.0%}   attention over the cache {a:4.0%}")

RunnableAll tracks

run state.py

python ~/llm-course/state.py

Output — what you should see

largest difference between the two: 2.4e-15
held after 1000 tokens: cache 128,000 numbers, state 4,096 numbers
   16 facts: fixed 64x64 state recalls 100%   attention over the cache 100%
   64 facts: fixed 64x64 state recalls  88%   attention over the cache 100%
  256 facts: fixed 64x64 state recalls  25%   attention over the cache 100%
 1024 facts: fixed 64x64 state recalls   6%   attention over the cache 100%

The first difference is rounding (its last digits vary by machine): the recurrent form is the same computation, holding 4,096 numbers where the cache form holds 128,000 and counting. The recall lines are the price. Once the facts outnumber the state’s dimensions, new writes overwrite old ones, while attention over a cache that never overwrote anything recalls every fact. That is the limitation the Gated Delta Networks abstract names, performance “in retrieval and long-context tasks has been limited”, and it is why current hybrids keep a minority of real attention layers for exact recall and give the rest of the depth to recurrent layers.

Two hybrids, read from their configs

	Qwen3-Next-80B-A3B (Apache-2.0)	Nemotron 3 Nano 30B-A3B (NVIDIA Nemotron Open Model License, not gated)
Card: parameters	“80B in total and 3B activated”	“3.5B active parameters and 30B parameters in total”
Card: layout	“12 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE))”	“23 Mamba-2 and MoE layers, along with 6 Attention layers”
Config: how the layout is declared	num_hidden_layers 48 and full_attention_interval 4	hybrid_override_pattern, 52 characters: 23 M, 23 E, 6 *
Attention layers, which keep a growing cache	12, each num_key_value_heads 2 × head_dim 256	6, each num_key_value_heads 2 × head_dim 128
Recurrent layers	36 Gated DeltaNet	23 Mamba-2
State of one recurrent layer	[linear_num_value_heads, linear_key_head_dim, linear_value_head_dim] = 32 × 128 × 128 = 524,288, plus a convolution state of 8,192 × linear_conv_kernel_dim 4	[mamba_num_heads, mamba_head_dim, ssm_state_size] = 64 × 64 × 128 = 524,288, plus a convolution state of 6,144 × conv_kernel 4
Experts	512 routed, 10 per token, 1 shared	128 routed, 6 per token, 1 shared
Context on the card	“262,144 natively”	“up to a 1M context size”; 256k default in the Hugging Face configuration

A “layer” means different things in the two configs. Each of Qwen3-Next’s 48 contains a mixer and an expert block; Nemotron’s pattern gives mixers and expert blocks a character each, so its 52 are 23 recurrent mixers, 23 expert blocks and 6 attention mixers. A calculator that trusts num_hidden_layers gets both wrong. Qwen3-Next’s card also claims “10 times inference throughput for context over 32K tokens” against Qwen3-32B-Base; the next section gives the cache arithmetic behind a claim of that shape.

What changes for the key-value cache

Part 2 derived the cache of a plain transformer, 2 × layers × kv_heads × head_dim × bytes_per_element per token, and the attention lesson showed that grouped-query attention is why kv_heads is 8 rather than 64 for Qwen3-32B. Architecture changes which layers that formula applies to:

cache(t) = 2 × kv_heads × head_dim × bytes_per_element × (full_layers × t + sliding_layers × min(t, window))
         + recurrent_layers × numbers_in_one_state × bytes_per_state_number      (constant in t)

Experts appear nowhere. Routing changes the feed-forward half of a block and the cache belongs to attention, so a mixture of experts has whatever cache its attention configuration gives it.

Fetch the five configurations. Each is a few kilobytes and none of the repositories is gated:

RunnableAll tracks

fetch five configs

hf download Qwen/Qwen3-32B config.json --local-dir ~/llm-course/configs/qwen3-32b
hf download Qwen/Qwen3-30B-A3B config.json --local-dir ~/llm-course/configs/qwen3-30b-a3b
hf download openai/gpt-oss-120b config.json --local-dir ~/llm-course/configs/gpt-oss-120b
hf download Qwen/Qwen3-Next-80B-A3B-Instruct config.json \
  --local-dir ~/llm-course/configs/qwen3-next-80b-a3b
hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 config.json \
  --local-dir ~/llm-course/configs/nemotron-3-nano-30b-a3b

Output — what you should see

✓ Downloaded
  path: /home/you/llm-course/configs/qwen3-32b/config.json
✓ Downloaded
  path: /home/you/llm-course/configs/qwen3-30b-a3b/config.json
✓ Downloaded
  path: /home/you/llm-course/configs/gpt-oss-120b/config.json
✓ Downloaded
  path: /home/you/llm-course/configs/qwen3-next-80b-a3b/config.json
✓ Downloaded
  path: /home/you/llm-course/configs/nemotron-3-nano-30b-a3b/config.json

Grey Hint: lines, and a warning about unauthenticated requests if you have not logged in to the Hub, may appear between these; none of them affects the download. The calculator decodes each of the three layout declarations and prices four context lengths. It counts the recurrent state at 4 bytes per number, the float32 in which Transformers computes the Gated DeltaNet rule and the mamba_ssm_cache_dtype Nemotron’s config declares:

RunnableAll tracks

arch-cache.py

import json, sys

c = json.load(open(sys.argv[1]))
L, kv_heads = c["num_hidden_layers"], c["num_key_value_heads"]
head_dim = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]
window = c.get("sliding_window") or 0

if "hybrid_override_pattern" in c:             # Nemotron-H: M = Mamba-2, E = MoE, * = attention
    kinds = [{"*": "full", "M": "state"}.get(ch, "other") for ch in c["hybrid_override_pattern"]]
elif "layer_types" in c:                       # gpt-oss: sliding_attention or full_attention
    kinds = ["window" if t == "sliding_attention" else "full" for t in c["layer_types"]]
elif "full_attention_interval" in c:           # Qwen3-Next: every n-th layer is full attention
    n = c["full_attention_interval"]
    kinds = ["full" if (i + 1) % n == 0 else "state" for i in range(L)]
else:                                          # a plain transformer, dense or mixture of experts
    kinds = ["full"] * L

if "linear_num_value_heads" in c:              # Gated DeltaNet: [v_heads, k_dim, v_dim] + conv
    conv = 2 * c["linear_num_key_heads"] * c["linear_key_head_dim"]
    conv += c["linear_num_value_heads"] * c["linear_value_head_dim"]
    per_state = c["linear_num_value_heads"] * c["linear_key_head_dim"] * c["linear_value_head_dim"]
    per_state += conv * c["linear_conv_kernel_dim"]
elif "mamba_num_heads" in c:                   # Mamba-2: [heads, head_dim, state_size] + conv
    inner = c["mamba_num_heads"] * c["mamba_head_dim"]
    conv = inner + 2 * c["n_groups"] * c["ssm_state_size"]
    per_state = inner * c["ssm_state_size"] + conv * c["conv_kernel"]
else:
    per_state = 0

full, slid, rec = kinds.count("full"), kinds.count("window"), kinds.count("state")
other = len(kinds) - full - slid - rec
print(f"{len(kinds)} layers: {full} full attention, {slid} sliding window {window}, "
      f"{rec} recurrent, {other} with no cache")
kv_per_token = 2 * kv_heads * head_dim * 2     # key and value, 2 bytes per element (FP16)
state = rec * per_state * 4                    # recurrent states, 4 bytes per element (float32)
for t in (4_096, 32_768, 131_072, 262_144):
    kv = kv_per_token * (full * t + slid * min(t, window))
    print(f"{t:>8,} tokens: KV cache {kv / 1e9:6.2f} GB  + fixed state {state / 1e9:.3f} GB")

RunnableAll tracks

price the two hybrids

python ~/llm-course/arch-cache.py ~/llm-course/configs/qwen3-next-80b-a3b/config.json
python ~/llm-course/arch-cache.py ~/llm-course/configs/nemotron-3-nano-30b-a3b/config.json

Output — what you should see

48 layers: 12 full attention, 0 sliding window 0, 36 recurrent, 0 with no cache
   4,096 tokens: KV cache   0.10 GB  + fixed state 0.080 GB
  32,768 tokens: KV cache   0.81 GB  + fixed state 0.080 GB
 131,072 tokens: KV cache   3.22 GB  + fixed state 0.080 GB
 262,144 tokens: KV cache   6.44 GB  + fixed state 0.080 GB
52 layers: 6 full attention, 0 sliding window 0, 23 recurrent, 23 with no cache
   4,096 tokens: KV cache   0.03 GB  + fixed state 0.050 GB
  32,768 tokens: KV cache   0.20 GB  + fixed state 0.050 GB
 131,072 tokens: KV cache   0.81 GB  + fixed state 0.050 GB
 262,144 tokens: KV cache   1.61 GB  + fixed state 0.050 GB

RunnableAll tracks

price the other three

python ~/llm-course/arch-cache.py ~/llm-course/configs/qwen3-32b/config.json
python ~/llm-course/arch-cache.py ~/llm-course/configs/qwen3-30b-a3b/config.json
python ~/llm-course/arch-cache.py ~/llm-course/configs/gpt-oss-120b/config.json

Output — what you should see

64 layers: 64 full attention, 0 sliding window 0, 0 recurrent, 0 with no cache
   4,096 tokens: KV cache   1.07 GB  + fixed state 0.000 GB
  32,768 tokens: KV cache   8.59 GB  + fixed state 0.000 GB
 131,072 tokens: KV cache  34.36 GB  + fixed state 0.000 GB
 262,144 tokens: KV cache  68.72 GB  + fixed state 0.000 GB
48 layers: 48 full attention, 0 sliding window 0, 0 recurrent, 0 with no cache
   4,096 tokens: KV cache   0.40 GB  + fixed state 0.000 GB
  32,768 tokens: KV cache   3.22 GB  + fixed state 0.000 GB
 131,072 tokens: KV cache  12.88 GB  + fixed state 0.000 GB
 262,144 tokens: KV cache  25.77 GB  + fixed state 0.000 GB
36 layers: 18 full attention, 18 sliding window 128, 0 recurrent, 0 with no cache
   4,096 tokens: KV cache   0.16 GB  + fixed state 0.000 GB
  32,768 tokens: KV cache   1.21 GB  + fixed state 0.000 GB
 131,072 tokens: KV cache   4.84 GB  + fixed state 0.000 GB
 262,144 tokens: KV cache   9.67 GB  + fixed state 0.000 GB

Collected, the five are below. They are arithmetic from the configs, not measurements, and columns past a model’s context window, which Part 2 lists, are for comparison only:

Model	Layers whose cache grows	KV bytes per token, FP16	4,096 tokens	32,768	131,072	262,144	Fixed state
Qwen3-32B, dense	64 full	262,144	1.07 GB	8.59 GB	34.36 GB	68.72 GB	none
Qwen3-30B-A3B, mixture of experts	48 full	98,304	0.40 GB	3.22 GB	12.88 GB	25.77 GB	none
gpt-oss-120b, experts and sliding window	18 full; 18 stop at 128 tokens	36,864 past the window	0.16 GB	1.21 GB	4.84 GB	9.67 GB	none
Qwen3-Next-80B-A3B, hybrid	12 full	24,576	0.10 GB	0.81 GB	3.22 GB	6.44 GB	0.080 GB
Nemotron 3 Nano 30B-A3B, hybrid	6 full	6,144	0.03 GB	0.20 GB	0.81 GB	1.61 GB	0.050 GB

Against Qwen3-32B, whose base model the Qwen3-Next card compares with, Qwen3-Next stores 262,144 ÷ 24,576, about 10.7 times fewer, KV bytes per token, which is the shape of the card’s long-context throughput claim; the claim itself stays a claim until measured.

The hybrid’s saving is in the slope. At 262,144 tokens the 80-billion-parameter Qwen3-Next holds the same 6.44 GB of cache that Qwen3-30B-A3B holds at 65,536, a quarter of the context. The fixed state is small, but it is read and rewritten at every token, so it behaves like an extra 80 MB of weights in the bandwidth arithmetic.

Two engine behaviours follow from these mechanisms. For sliding layers, llama-server at v0.4.0 lists --swa-full, “use full-size SWA cache (default: false)”, so by default the smaller figure applies. The pull request that added checkpoints for that cache states the trade: with --swa-full “we can branch from any past positions of the context (so no need to do checkpoints), but the drawback is that the SWA memory size is much larger”. A recurrent state is the stronger case of the same problem: overwritten at every token, it cannot be rolled back to an earlier position without a saved copy, which matters for the prompt reuse Part 17 teaches. The same README lists --ctx-checkpoints, also spelled --swa-checkpoints, “max number of context checkpoints to create per slot (default: 32)”; how a given engine and version checkpoints a hybrid is a detail to confirm in its documentation before relying on prompt reuse with one.

Where the cache overtakes the weights

A decode step reads the weights it uses plus the cache, and only the cache grows. The context at which the two are equal:

crossover tokens = weight bytes read per token ÷ KV bytes per token of context

Model	Weight bytes read per token	KV bytes per token	Crossover
Qwen3-32B, dense	19,318,598,656	262,144	73,695 tokens
Qwen3-30B-A3B	1,919,596,544	98,304	19,527 tokens
gpt-oss-120b	3,594,210,048	36,864	97,499 tokens

Qwen3-30B-A3B crosses before 20k tokens: past that, most of what a decode step reads is cache, which is why Part 3 found that long context erodes a mixture of experts’ advantage. The decision rule:

Your typical context	What to compare between candidate models
Well below the crossover	Weight bytes read per token, from the tensor table: always-read tensors + (experts per token ÷ experts per layer) × routed-expert bytes (not active parameters × average bytes per parameter, which Part 3 shows undercounts gpt-oss-120b)
Near or above the crossover	KV bytes per token: full-attention layers × kv_heads × head_dim, less any sliding or recurrent layers
Any, on a machine short of capacity	Resident bytes: the file plus the cache at the length you allocate

✕Common mistake

The mistake with a hybrid is the plain formula. 2 × 48 × 2 × 256 × 2 gives Qwen3-Next 98,304 bytes per token, four times the real 24,576, and prices a 262k context at 25.8 GB instead of 6.4 GB; 2 × 52 × 2 × 128 × 2 gives Nemotron 3 Nano 53,248, counting 46 layers that have no cache. The symptom is a context length rejected as “too big” on paper that would have fitted. The opposite error is reading “hybrid” as “context is free”: the full-attention layers still grow linearly. Decode the layer pattern, as arch-cache.py does.

What changes for quantisation

Quantisation is Part 16’s subject; architecture decides where it lands. The tensor tables of the course’s two mixture-of-experts files, grouped:

Tensor group (GGUF names)	Qwen3-30B-A3B Q4_K_M: types	Bytes	Share	gpt-oss-120b MXFP4: types	Bytes	Share
Routed experts: ffn_gate_exps, ffn_up_exps, ffn_down_exps	Q4_K; half of the ffn_down_exps layers Q6_K	17,553,162,240	94.6%	MXFP4; biases F32	61,073,326,080	96.4%
Attention: attn_q, attn_k, attn_v, attn_output	Q4_K; half of the attn_v layers Q6_K	516,096,000	2.8%	Q8_0; biases F32	1,016,386,560	1.6%
Output head: output	Q6_K	255,252,480	1.4%	Q8_0	615,329,280	1.0%
Embedding table: token_embd	Q4_K	175,030,272	0.9%	Q8_0	615,329,280	1.0%
Router: ffn_gate_inp	F32	50,331,648	0.27%	F32	53,102,592	0.08%
Norms and attention sinks	F32	843,776	under 0.01%	F32	850,176	under 0.01%

Every byte count in that table, and the tensor byte counts in the two-numbers section, come from the start of each file: the header and tensor infos end within the first 16 MiB in all three, so a range request stands in for downloads of up to 63 GB. --range asks for those bytes only, --location follows the Hub’s redirect to its storage, and --fail stops on an HTTP error instead of saving the error page as a file. The three requests total 48 MiB:

RunnableAll tracks

fetch the first 16 MiB of three GGUF files

curl --silent --show-error --fail --location --range 0-16777215 --create-dirs \
  --output ~/llm-course/gguf-heads/Qwen3-30B-A3B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF/resolve/main/Qwen3-30B-A3B-Q4_K_M.gguf
curl --silent --show-error --fail --location --range 0-16777215 --create-dirs \
  --output ~/llm-course/gguf-heads/Qwen3-32B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-32B-GGUF/resolve/main/Qwen3-32B-Q4_K_M.gguf
curl --silent --show-error --fail --location --range 0-16777215 --create-dirs \
  --output ~/llm-course/gguf-heads/gpt-oss-120b-MXFP4.gguf \
  https://huggingface.co/ggml-org/gpt-oss-120b-GGUF/resolve/main/gpt-oss-120b-MXFP4.gguf
wc -c ~/llm-course/gguf-heads/Qwen3-30B-A3B-Q4_K_M.gguf \
  ~/llm-course/gguf-heads/Qwen3-32B-Q4_K_M.gguf \
  ~/llm-course/gguf-heads/gpt-oss-120b-MXFP4.gguf

Output — what you should see

16777216 /home/you/llm-course/gguf-heads/Qwen3-30B-A3B-Q4_K_M.gguf
16777216 /home/you/llm-course/gguf-heads/Qwen3-32B-Q4_K_M.gguf
16777216 /home/you/llm-course/gguf-heads/gpt-oss-120b-MXFP4.gguf
50331648 total

The curl commands print nothing when they succeed; macOS’s wc pads the counts with spaces. A line such as curl: (22) The requested URL returned error: 404 means the file name has changed in the repository: check its file listing. The reader below is Part 2’s gguf-header.py cut down to the tensor infos, with ggml’s block sizes to turn each tensor’s parameter count into bytes and a name rule to group the tensors. Save it as gguf-groups.py; it works on a complete .gguf file too:

RunnableAll tracks

gguf-groups.py

import struct, sys
from collections import Counter, defaultdict

# ggml type id: (name, bytes per block, values per block), from ggml-common.h at llama.cpp v0.4.0
BLOCK = {0: ("F32", 4, 1), 1: ("F16", 2, 1), 8: ("Q8_0", 34, 32), 12: ("Q4_K", 144, 256),
         13: ("Q5_K", 176, 256), 14: ("Q6_K", 210, 256), 30: ("BF16", 2, 1), 39: ("MXFP4", 17, 32)}
RULES = [("_exps", "routed experts"), ("ffn_gate_inp", "router"), ("norm", "norms and sinks"),
         ("sinks", "norms and sinks"), ("token_embd", "embedding table"), ("attn_", "attention")]
SCALAR = {0: "<B", 1: "<b", 2: "<H", 3: "<h", 4: "<I", 5: "<i", 6: "<f", 7: "<?", 10: "<Q", 11: "<q", 12: "<d"}
fh = open(sys.argv[1], "rb")

def get(fmt):
    data = fh.read(struct.calcsize(fmt))
    if len(data) < struct.calcsize(fmt):
        sys.exit("the file ends inside the header: download a longer range")
    return struct.unpack(fmt, data)[0]

def text():
    return fh.read(get("<Q")).decode("utf-8", "replace")

def value(kind):                                      # 8 is a string, 9 an array, the rest scalars
    if kind == 9:
        inner, count = get("<I"), get("<Q")
        return [value(inner) for _ in range(count)]
    return text() if kind == 8 else get(SCALAR[kind])

if fh.read(4) != b"GGUF":                              # the magic, then a uint32 version
    sys.exit(f"{sys.argv[1]} is not a GGUF file")
get("<I")
n_tensors, n_kv = get("<Q"), get("<Q")
meta = dict((text(), value(get("<I"))) for _ in range(n_kv))
size, types = defaultdict(int), defaultdict(Counter)
for _ in range(n_tensors):
    name, n_dims = text(), get("<I")
    values = 1
    for _ in range(n_dims):
        values *= get("<Q")
    kind, _offset = get("<I"), get("<Q")
    if kind not in BLOCK:
        sys.exit(f"{name}: ggml type {kind} is not in BLOCK; add its block size first")
    label = "output head" if name.startswith("output.") else next(
        (group for part, group in RULES if part in name), "dense feed-forward")
    type_name, block_bytes, block_values = BLOCK[kind]
    size[label] += values // block_values * block_bytes
    types[label][type_name] += 1
print(f"header and tensor infos end at byte {fh.tell():,}")

total, arch = sum(size.values()), meta["general.architecture"]
for label in sorted(size, key=size.get, reverse=True):
    kinds = "  ".join(f"{t} x{n}" for t, n in types[label].most_common())
    print(f"  {label:18s} {size[label]:>15,}  {100 * size[label] / total:5.2f}%  {kinds}")
used, experts = meta.get(f"{arch}.expert_used_count", 0), meta.get(f"{arch}.expert_count", 0)
always = total - size["routed experts"] - size["embedding table"]
routed = size["routed experts"] * used // experts if experts else 0
print(f"tensor bytes {total:,}; always read {always:,}")
print(f"read per token {always + routed:,}, with {used} of {experts} experts per layer")

RunnableAll tracks

run gguf-groups.py on the three files

python ~/llm-course/gguf-groups.py ~/llm-course/gguf-heads/Qwen3-30B-A3B-Q4_K_M.gguf
python ~/llm-course/gguf-groups.py ~/llm-course/gguf-heads/Qwen3-32B-Q4_K_M.gguf
python ~/llm-course/gguf-groups.py ~/llm-course/gguf-heads/gpt-oss-120b-MXFP4.gguf

Output — what you should see

header and tensor infos end at byte 5,970,481
  routed experts      17,553,162,240  94.62%  Q4_K x120  Q6_K x24
  attention              516,096,000   2.78%  Q4_K x168  Q6_K x24
  output head            255,252,480   1.38%  Q6_K x1
  embedding table        175,030,272   0.94%  Q4_K x1
  router                  50,331,648   0.27%  F32 x48
  norms and sinks            843,776   0.00%  F32 x193
tensor bytes 18,550,716,416; always read 822,523,904
read per token 1,919,596,544, with 8 of 128 experts per layer
header and tensor infos end at byte 5,975,700
  dense feed-forward  15,237,120,000  77.13%  Q4_K x160  Q6_K x32
  attention            3,440,640,000  17.42%  Q4_K x224  Q6_K x32
  output head            638,131,200   3.23%  Q6_K x1
  embedding table        437,575,680   2.21%  Q4_K x1
  norms and sinks          2,707,456   0.01%  F32 x257
tensor bytes 19,756,174,336; always read 19,318,598,656
read per token 19,318,598,656, with 0 of 0 experts per layer
header and tensor infos end at byte 13,022,233
  routed experts      61,073,326,080  96.37%  F32 x108  MXFP4 x108
  attention            1,016,386,560   1.60%  F32 x144  Q8_0 x144
  output head            615,329,280   0.97%  Q8_0 x1
  embedding table        615,329,280   0.97%  Q8_0 x1
  router                  53,102,592   0.08%  F32 x72
  norms and sinks            850,176   0.00%  F32 x109
tensor bytes 63,374,323,968; always read 1,685,668,608
read per token 3,594,210,048, with 4 of 128 experts per layer

A file whose header runs past the range stops with “the file ends inside the header”; a larger --range fixes it. The group bytes and the always-read and read-per-token lines are the two tables’ figures, and the dense file has “0 of 0 experts”, so its last two figures match. The count after each type is how many tensors use it, which is where the table’s “half of the layers Q6_K” comes from.

The experts are the file. Compressing only the experts captures nearly all of the saving, and gpt-oss ships that way: its config’s quantization_config names mxfp4 and lists model.layers.*.self_attn, model.layers.*.mlp.router, model.embed_tokens and lm_head under modules_to_not_convert; the ggml-org conversion stores those at Q8_0, except the router, which is F32. The expert weights cost exactly 4.25 bits each in the file, 60,914,073,600 bytes for 114,661,785,600 weights: four bits per value plus one 8-bit shared scale per block of 32, the format Part 16’s MXFP4 section takes apart.

The router stays at full precision by rule. llama-quant.cpp at v0.4.0 carries the comment “do not quantize expert gating tensors” above the line that skips every ffn_gate_inp.weight, which is why both files hold it as F32. The saving forgone is small: at 4.5 bits Qwen3-30B-A3B’s router would save 50,331,648 × (1 − 4.5 ÷ 32) = 43,253,760 bytes, 0.23 per cent of the file. The risk is not small, because the router’s output is a choice. This gives a router and an expert matrix the same 4-bit rounding and compares the damage:

RunnableAll tracks

router-rounding.py

import numpy as np

rng = np.random.default_rng(0)
hidden, n_experts, k, tokens = 2048, 128, 8, 4096
router = rng.normal(0, 0.02, (n_experts, hidden))
expert = rng.normal(0, 0.02, (768, hidden))     # an expert's up-projection, for comparison
h = rng.normal(0, 1, (tokens, hidden))

def round_4bit(w, block=32):                    # symmetric 4-bit, one scale per block of 32
    x = w.reshape(-1, block)
    scale = np.abs(x).max(axis=1, keepdims=True) / 7
    return (np.clip(np.round(x / scale), -8, 7) * scale).reshape(w.shape)

def rel_err(a, b):
    return np.linalg.norm(a - b) / np.linalg.norm(a)

router_q, expert_q = round_4bit(router), round_4bit(expert)
print(f"weight error:  router {rel_err(router, router_q):.1%}   expert {rel_err(expert, expert_q):.1%}")
print(f"output error:  router scores {rel_err(h @ router.T, h @ router_q.T):.1%}"
      f"   expert output {rel_err(h @ expert.T, h @ expert_q.T):.1%}")

pick = lambda W: np.argsort(-(h @ W.T), axis=1)[:, :k]
changed = np.array([len(set(a) - set(b)) for a, b in zip(pick(router), pick(router_q))])
print(f"tokens sent to a different set of experts: {np.mean(changed > 0):.1%}")
print(f"experts swapped when that happens: {changed[changed > 0].mean():.2f} of {k} on average")

RunnableAll tracks

run router-rounding.py

python ~/llm-course/router-rounding.py

Output — what you should see

weight error:  router 9.7%   expert 9.7%
output error:  router scores 9.7%   expert output 9.7%
tokens sent to a different set of experts: 62.2%
experts swapped when that happens: 1.18 of 8 on average

Both matrices take the same relative error, and for the expert that is the whole story: its output is off by that much. The router’s scores are off by that much too, and for most tokens that is enough to swap an expert, usually one of the eight. Random weights put the scores closer together than a trained router may, so treat the rate as an illustration of the mechanism, not a prediction for any model. The swaps happen where the eighth and ninth scores are nearly tied, which is also where the gate weights are smallest.

Calibration only sees the experts the text routes to. An importance matrix records the activations each expert receives, and an expert receives them only from tokens routed to it. llama-imatrix at v0.4.0 warns that an entry “has partial data” and prints the percentage of experts that received any, and the source names the cause: “this can happen with MoE models where some of the experts end up not being exercised by the provided training data”. Part 16’s calibration section treats the choice of calibration text as a decision to write down.

Hybrids add small tensors that are never rounded, and need engine support first. The same llama-quant.cpp function refuses every tensor with fewer than two dimensions, so norms, biases and per-head scalars keep their precision, and a comment exempts the recurrent layers’ convolution: “do not quantize Mamba/Kimi’s small conv1d weights”. None of that helps until an engine implements the layer type. llama.cpp’s architecture list at v0.4.0 includes qwen3next, nemotron_h and nemotron_h_moe, but the course’s model reference records no GGUF source for either hybrid, so check that a format exists for your engine before planning work around one.

The three choices this section leaves you with, as rules:

What you see or choose	What it means	What to do
llama-imatrix warns “entry ‘…’ has partial data (NN.NN%)” for an ffn_*_exps tensor	the calibration text routed no tokens to some experts	add more, and more varied, calibration text and re-run until no expert tensor is reported with partial data
gguf-groups.py shows router in a type other than F32 in a community file	router rounding can change which experts a token reaches	treat the file as an experiment: evaluate it with Part 16’s methods before use
Choosing a quantisation tier for a mixture of experts	the routed experts are about 95 per cent of the bytes	the expert type sets the file size, so compare tiers by the types gguf-groups.py prints for routed experts

Why mixture of experts suits a unified-memory machine

Two tests per model and machine, both arithmetic:

fits      : file bytes + cache at the context you allocate + reserve  ≤  memory the GPU can use
ceiling   : bandwidth ÷ (weight bytes read per token + cache bytes read per token)

With the three models at a 32,768-token context, the reserves from the memory-budget lesson (10 GB on a 128 GB unified-memory machine, 1.5 GB on a discrete card) and bandwidth from the hardware reference:

Machine (track)	GPU-usable memory	Bandwidth, GB/s	Qwen3-32B dense: 28.35 GB resident, 27.91 GB read per token	Qwen3-30B-A3B: 21.78 GB, 5.14 GB	gpt-oss-120b: 64.60 GB, 4.81 GB
DGX Spark (S)	128 GB	273	fits; ceiling 9.8 tokens/s	fits; 53.1	fits; 56.8
Ryzen AI Max+ 395, 128 GB (X)	below 128 GB: the GTT limit the kernel sets on Linux (check it); about 96 GB on Windows	256	fits; 9.2	fits; 49.8	fits if the GPU can address 74.6 GB, reserve included; 53.3
M4 Max, 128 GB (M)	128 GB once the wired limit is raised	546	fits; 19.6	fits; 106.2	fits; 113.6
RTX 5090 (N)	32 GB	1,792	fits, 2.15 GB left after the reserve; 64.2	fits; 348.6	does not fit
RTX 4090 (N)	24 GB	1,008	does not fit	fits, 0.72 GB left after the reserve; 196.1	does not fit

Every ceiling is arithmetic from the tensor tables and vendor bandwidth, before attention arithmetic, dequantisation and real-world bandwidth, all of which push a measurement lower; Part 5’s bandwidth lab and Part 6’s benchmark lab replace them with your numbers.

Down a column, the architecture’s speed-up is the same on every machine, 27.91 ÷ 5.14 or about 5.4 times at this context, because bandwidth cancels. What differs is where it lands: on the Spark and Ryzen AI Max+ rows the dense model’s ceiling is in single digits and on the M4 Max it is under 20, while the mixture of experts with more than three times the parameters is faster than it on all three. Across a row, capacity decides: gpt-oss-120b exists only on the unified-memory machines. They are short of bandwidth and long on capacity, and a mixture of experts spends capacity to save bandwidth, which is the one trade those machines can afford. The discrete cards are the mirror image.

Experts in system memory on a discrete card

llama-server documents a middle path at v0.4.0: --cpu-moe, “keep all Mixture of Experts (MoE) weights in the CPU”, and --n-cpu-moe N for the experts of the first N layers only. With the layers otherwise offloaded, the always-read tensors and the cache stay on the card and the experts sit in system memory, so every expert byte a token uses is read at system-memory bandwidth. Decode time per token becomes two terms:

time per token = (always-read bytes + cache bytes) ÷ card bandwidth  +  expert bytes read ÷ system memory bandwidth

For gpt-oss-120b at 32k, taking system memory at Part 1’s ceiling for two channels of DDR5-5600:

Card	On the card: resident, read per token	In system memory: resident, read per token	Card term	System-memory term, at 89.6 GB/s	Ceiling
RTX 5090, 1,792 GB/s	3.51 GB, 2.90 GB	61.07 GB, 1.91 GB	1.62 ms	21.30 ms	43.6 tokens/s
RTX 4090, 1,008 GB/s	3.51 GB, 2.90 GB	61.07 GB, 1.91 GB	2.88 ms	21.30 ms	41.4 tokens/s

About nine tenths of the time is the system-memory term, so the card’s speed barely moves the result and the ceiling lands below the Spark’s. The arithmetic also leaves out the processor time for the expert multiplications and the PCIe crossings between the two pools, both of which lower a measurement further, and the machine needs 61 GB of system memory free for the experts alone. It is a legitimate way to run a model the card cannot hold; it is not a way to get the card’s bandwidth for it.

Your machine	Short of	What the arithmetic favours
Unified memory, 96 GB or more	Bandwidth	The largest-total mixture of experts that fits with its cache; a dense model only where its read per token still meets the speed you need
Unified memory, 64 GB	Both	A 30B-class mixture of experts at Q4_K_M; gpt-oss-120b’s 64.6 GB does not fit beside an operating system
Discrete card, 24 to 32 GB	Capacity	Whatever fits entirely in the card, dense or mixture of experts; its bandwidth makes a dense 32B usable where it fits: at Q4_K_M, (card − 1.5 GB reserve − 19.76 GB file) ÷ 262,144 bytes leaves about 10,400 tokens of FP16 cache on a 24 GB card and about 41,000 on a 32 GB card
Discrete card plus 96 GB or more of system memory	Capacity, with a slow second pool	A large mixture of experts with --cpu-moe, at a ceiling set by system memory
Any, with long contexts	Cache	The fewest KV bytes per token: fewer full-attention layers, sliding windows, hybrids

Capacity, active work and attention state are separate axes

Imagine two models with similar active parameter counts. One is dense; the other routes tokens through a subset of experts. Similar active work does not imply similar memory residency, because the expert model may need a much larger collection of weights available for routing. Under concurrent requests, different tokens may select different experts, changing reuse and traffic.

A hybrid architecture adds another distinction: not every layer necessarily stores a conventional full-attention KV history. Use the architecture’s actual layer types and state shapes when calculating memory. Multiplying a dense transformer’s per-layer KV formula by the total layer count can misestimate a hybrid model.

Before choosing an engine, inspect the checkpoint configuration and the engine’s support for that architecture and representation. Loading a dense sibling from the same family does not establish support for its expert or recurrent variant. Your acceptance test should cover loading, representative long prompts, generation and the application feature you need. A successful short completion is evidence of basic execution, not evidence that every architecture-specific optimisation or long-context path is correct.

Recap

A dense model reads nearly its whole file for every token, while a mixture of experts routes each token through a scoring function (softmax in Qwen3 and gpt-oss, sigmoid plus a selection bias in Nemotron 3 Nano), a top-k choice and renormalised gates to a few experts, keeps all of them resident, and so separates the bytes it holds from the bytes it reads. A hybrid’s recurrent layers keep a fixed-size state instead of a growing cache, at the price of exact recall, which is why it keeps a few attention layers. Prefer the architecture that relieves what your machine lacks, capacity or bandwidth. Part 11 returns to experts for training memory and Part 18 splits them across machines.

Check your understanding

Question 1. Qwen3-Next-80B-A3B has 48 layers, a full_attention_interval of 4, 2 key-value heads and a head dimension of 256. A colleague computes its FP16 cache per token. Which calculation is correct?

• 2 × 48 × 2 × 256 × 2 = 98,304 bytes
• 2 × 12 × 2 × 256 × 2 = 24,576 bytes, plus a fixed recurrent state that does not grow
• 2 × 12 × 16 × 256 × 2 = 196,608 bytes, using the 16 query heads
• 2 × 36 × 2 × 256 × 2 = 73,728 bytes, one for each Gated DeltaNet layer

Show the answer and why

Answer: 2 × 12 × 2 × 256 × 2 = 24,576 bytes, plus a fixed recurrent state that does not grow

Only every fourth layer is full attention, so 12 layers keep a growing cache; the other 36 keep a fixed state of 32 × 128 × 128 numbers each plus a small convolution state. The first option is the plain formula and overstates the cache fourfold; the third uses query heads, which grouped-query attention does not cache; the fourth counts exactly the layers that have no key-value cache.

Question 2. You double the allocated context of Qwen3-Next-80B-A3B from 131,072 to 262,144 tokens with an FP16 cache. What happens to its cache memory?

• Nothing: hybrid models do not have a key-value cache
• The key-value part doubles from 3.22 GB to 6.44 GB and the 0.080 GB recurrent state stays the same
• Everything doubles, the recurrent state included
• It quadruples, because attention cost is quadratic

Show the answer and why

Answer: The key-value part doubles from 3.22 GB to 6.44 GB and the 0.080 GB recurrent state stays the same

The 12 full-attention layers grow linearly with tokens; the recurrent state is the same size at any length. Quadratic growth describes the arithmetic of prefill attention, not the cache, which stores one key and one value per token per attention layer.

Question 3. True or false: taking a softmax over all 128 router scores, keeping the top 8 and renormalising them gives the same gate weights as keeping the top 8 raw scores and taking a softmax over those 8.

• True
• False

Show the answer and why

Answer: True

A softmax weight is exp of a score divided by a sum of exps. Renormalising over the chosen eight divides the shared denominator out, leaving exp(score_i) over the sum of exp over the eight, which is the softmax over the eight. Qwen3-MoE and gpt-oss order the steps differently in Transformers 5.16.1 and get the same gates; route.py checks it. The shortcut belongs to the softmax: Nemotron 3 Nano scores experts with a sigmoid, which has no shared denominator to cancel, so its gates must be computed from its own router.

Question 4. In the Qwen3-30B-A3B Q4_K_M file the router is F32 and makes up 0.27 per cent of the bytes. Why do llama-quantize and the gpt-oss configuration both leave the router unquantised?

• Routers cannot be represented in four bits at all
• The saving would be about 43 MB, and rounding changes a discrete choice: a small error in the scores can send a token to a different expert
• The router holds most of the model's knowledge
• Quantising the router would make the key-value cache larger

Show the answer and why

Answer: The saving would be about 43 MB, and rounding changes a discrete choice: a small error in the scores can send a token to a different expert

An expert with a rounding error produces a slightly wrong output; a router with the same relative error produces a different selection whenever two scores near the cut-off are close. The saving is tiny because the experts, not the router, are over 94 per cent of the file.

Question 5. Qwen3-30B-A3B reads 1.92 GB of weights and 98,304 bytes of FP16 cache per token of context. In a conversation at 64,000 tokens, which term dominates the bytes one decode step reads?

• The weights, because the model is 30 billion parameters
• The cache: 64,000 × 98,304 bytes is about 6.3 GB, over three times the weights read
• They are equal
• Neither: a mixture of experts does not read its cache during decode

Show the answer and why

Answer: The cache: 64,000 × 98,304 bytes is about 6.3 GB, over three times the weights read

The crossover is 1.92 GB ÷ 98,304 bytes, about 19,500 tokens. Beyond it the cache dominates each step, so a model with fewer KV bytes per token, such as a hybrid or a model with sliding-window layers, can matter more for speed than the number of active parameters.

Question 6. gpt-oss-120b runs on an RTX 5090 with --cpu-moe and two channels of DDR5-5600 for system memory. Which quantity bounds its decode speed?

• The card's bandwidth, because the attention runs there
• The system memory's bandwidth, because the 1.91 GB of experts read per token cross it, and that term is over nine tenths of the time
• PCIe alone
• The card's 32 GB capacity

Show the answer and why

Answer: The system memory's bandwidth, because the 1.91 GB of experts read per token cross it, and that term is over nine tenths of the time

Time per token is 2.90 GB over the card's bandwidth plus 1.91 GB over the system memory's: about 1.6 ms plus 21.3 ms, before the expert arithmetic and PCIe crossings. A faster card shrinks the small term; the large one is set by system memory, which is why the DGX Spark's unified pool, several times faster than dual-channel DDR5, gives this model the higher ceiling.

Check my answersClear