Inference: Prefill, Decode and Why Memory Bandwidth Rules

By the end of this lesson you will be able to divide generation into its two phases and say which resource limits each; compute a ceiling on decode speed for a model on a machine before running either, and say when the file-size shortcut for that arithmetic misleads; turn a stopwatch reading into time to first token and time per output token and say which fix acts on which; predict what a long context does to memory and to speed; say what batching buys, in numbers, and the three things that stop it; and explain why a thirty-billion-parameter mixture-of-experts model generates faster than an eight-billion dense one at batch one, and why that advantage shrinks under load.

This is the most immediately useful arithmetic in the course. Part 1 built its ingredients: the two clocks, bytes moved and operations done, and Part 2 sized the KV cache and counted active parameters. This lesson assembles them into predictions. Every figure in the tables is arithmetic from stated inputs, not a measurement, except the figures taken from the timed snippets’ outputs, each of which is one run on one CPU and is labelled so.

Two phases, not one

Generating an answer is two different jobs wearing one name.

Prefill reads the prompt. All of its tokens are known, so every layer processes them together: each projection is one matrix multiplication of an n × hidden block of activations by a weight matrix. The pass writes n tokens of keys and values into the cache and, at its last position, produces the logits from which the first output token is sampled.

Decode writes the answer, one token per forward pass. Each pass pushes a single row of activations through every weight the token uses, attends over everything in the cache, appends one token’s keys and values, and emits one token. The passes are strictly sequential: token seventeen cannot be computed before token sixteen exists.

For Qwen3-8B (Apache-2.0), whose config.json gives 36 layers, 8 key-value heads and a head dimension of 128, one pass of each kind moves this much work and data (arithmetic; parameter counts from Part 2, FLOPs by Part 1’s two-per-parameter rule, excluding the embedding lookup):

One forward pass, Qwen3-8B, FP16 cache	Prefill of a 2,048-token prompt	Decode of one token at 2,048 in context
Tokens pushed through the weights	2,048	1
Floating-point operations in the 36 blocks	2 × 6,946,075,648 × 2,048 ≈ 28.5 TFLOP	2 × 6,946,075,648 ≈ 13.9 GFLOP
Weight bytes read at Q4_K_M	about 4.67 GB, once	about 4.67 GB, once
KV cache written	2,048 × 147,456 bytes ≈ 302 MB	147,456 bytes
KV cache read by attention	built as it goes	2,048 × 147,456 bytes ≈ 302 MB
Output	logits at the last position only; first token sampled	logits for one position; next token sampled

The weight bytes are the same in both columns and the operations differ by a factor of 2,048. That single ratio is why the two phases behave differently, and the next section measures it.

Where the time goes, schematically

Short prompt, short answer

PrefillDecode

Long prompt, short answer

PrefillDecode

Short prompt, long answer

PrefillDecode

Schematic, not measured: the point is the shape. Prefill grows with the prompt and is done once; decode grows with the answer and is paid one token at a time. The boundary between the two blocks is when the first token appears on screen.

Compute-bound and bandwidth-bound

Part 1 gave every job two clocks, one for the arithmetic and one for the bytes, with the slower one setting the time. Written for the two phases:

N   parameters multiplied per token (all but the embedding lookup)
W   bytes of weights read per token
B   memory bandwidth, bytes per second
C   floating-point operations per second the device delivers
n   tokens in the prompt
t   tokens already in the cache
k   KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element

prefill seconds        ≈ max( 2 × N × n / C ,  W / B )       + attention's n² term on long prompts
decode seconds / token ≈ max( 2 × N / C ,  (W + k × t) / B )

prompt length where prefill turns compute-bound:  n* ≈ W × C / (2 × N × B)

A job is compute-bound when the first term is the larger one and bandwidth-bound when the second is. C is the peak rate that large multiplies reach; a multiply of one row, or a few, gets much less of it out of the same processor, so the max() is a bound on the time, not a prediction of it, and the batch run later in this lesson shows the difference. Decode’s arithmetic term is 2 × N operations per token. For Qwen3-0.6B (Apache-2.0), whose output head is its embedding matrix, that is about 1.2 GFLOP, under 2 ms at the arithmetic rate the same CPU reaches on a long prompt, inside a step that takes about 87 ms: at batch one the bytes decide. The snippet below counts 0.88 GFLOP per prompt token instead, leaving the head out, because prefill computes logits at one position only (logits_to_keep=1), while every decode token runs the head. Prefill’s arithmetic term grows with n while its bytes term does not, so past n* the arithmetic decides.

You can watch that switch happen. This snippet times prefill and decode separately on Qwen3-0.6B, on the CPU on every track, for prompts from one token to 4,096. Save it, and the batching snippet further down, in ~/llm-course under their titles, then work from there with the Part 1 environment active (cd ~/llm-course, then source .venv/bin/activate) and the libraries Part 2’s lab installed. On Track S, work inside the Part 1 container instead, with /workspace/course in place of ~/llm-course and no activate line. Both snippets need Qwen3-0.6B (not gated, about 1.5 GB on disk); readers who took Part 2’s reduced path already have it, and everyone else downloads it once. Each timing run takes up to about a minute on a 12-thread CPU.

RunnableAll tracks

download Qwen3-0.6B, unless Part 2's reduced path already did

hf download Qwen/Qwen3-0.6B --local-dir ~/llm-course/models/qwen3-0.6b

RunnableAll tracks

prefill-vs-decode.py

"""Time one request in its two phases: prefill follows the prompt, decode barely notices it."""
import sys, time, torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(sys.argv[1], dtype=torch.float32).eval()
vocab = model.config.vocab_size
embedding = model.get_input_embeddings().weight.numel()
flops_per_token = 2 * (sum(p.numel() for p in model.parameters()) - embedding)
torch.manual_seed(0)

def request(prompt_tokens, new_tokens=17):
    ids = torch.randint(0, vocab, (1, prompt_tokens))     # the words do not change the timing
    with torch.inference_mode():
        start = time.perf_counter()
        out = model(input_ids=ids, logits_to_keep=1)      # prefill: the whole prompt in one pass
        token = out.logits[:, -1:].argmax(-1)             # the first output token comes out of it
        ttft = time.perf_counter() - start
        cache, start = out.past_key_values, time.perf_counter()
        for _ in range(new_tokens - 1):                   # decode: one pass per token
            out = model(input_ids=token, past_key_values=cache)
            token = out.logits[:, -1:].argmax(-1)
        tpot = (time.perf_counter() - start) / (new_tokens - 1)
    return ttft, tpot

request(16)                                               # warm-up, not reported
print(f"{'prompt':>6} {'TTFT ms':>8} {'prompt tok/s':>12} {'GFLOP/s':>8} {'TPOT ms':>8} {'decode tok/s':>12}")
for n in (1, 16, 64, 256, 1024, 4096):
    runs = [request(n) for _ in range(3)]                 # best of three, for each phase
    ttft, tpot = min(r[0] for r in runs), min(r[1] for r in runs)
    print(f"{n:>6} {ttft * 1e3:>8.0f} {n / ttft:>12.0f} {n * flops_per_token / ttft / 1e9:>8.0f}"
          f" {tpot * 1e3:>8.1f} {1 / tpot:>12.1f}")

RunnableAll tracks

run it on the 0.6B checkpoint

python prefill-vs-decode.py ~/llm-course/models/qwen3-0.6b

Output — what you should see

prefill-vs-decode.py, torch 2.14.0 and transformers 5.16.1, CPU only (12 threads), one run on the author's test box; your figures will differ, the shape will not

Loading weights: 100%|██████████| 311/311 [00:00<00:00, xxxx.xxit/s]
prompt  TTFT ms prompt tok/s  GFLOP/s  TPOT ms decode tok/s
     1       86           12       10     87.6         11.4
    16      111          144      127     90.0         11.1
    64      172          373      329     84.5         11.8
   256      370          691      609     87.5         11.4
  1024     1217          842      741    102.4          9.8
  4096     6675          614      541    165.7          6.0

Read the output row by row:

Rows	What happens	Which clock
Prompt of 1	A one-token prefill takes as long as a decode step; the CPU delivers 10 GFLOP/s because it spends the pass waiting for 2.4 GB of FP32 weights	bytes: bandwidth-bound
Prompts of 16 to 256	Time rises far more slowly than the tokens, so prompt tokens per second climbs steeply: the same fetch of the weights is doing more arithmetic	moving from bytes to arithmetic
Prompt of 1,024	Operations per second flatten near this CPU’s arithmetic limit	arithmetic: compute-bound
Prompt of 4,096	Prompt tokens per second falls, because attention’s score work grows with n², as Part 2’s attention lesson derived	arithmetic, plus the quadratic term
TPOT column	Flat until the cache is long, then rising; the KV section below explains it	bytes: the weights, then the cache as well
The crossover, n*	W = 2.384 GB; B ≈ 2.384 GB ÷ 0.0876 s ≈ 27 GB/s effective, from the one-token decode step; C ≈ 741 GFLOP/s; 2 × N = 0.88 GFLOP. n* ≈ 2.384 × 741 ÷ (0.88 × 27.2) ≈ 74 tokens	inside the 16-to-256 rows, where the clocks trade places

Part 1 found that an accelerator’s ratio of arithmetic to bandwidth sits well above a CPU’s, so on a GPU the switch to compute-bound comes at a longer prompt, and in decode at a larger batch; the regions are the same. Part 5’s measurement lab measures B and C for your machine, which puts numbers on both clocks and gives you your own n*.

The decode ceiling

Drop the arithmetic term, which the previous section showed is small for decode, and the decode clock becomes the formula this course leans on: bandwidth divided by the bytes read per token, the weights the token uses plus the KV cache attention reads. It is a ceiling, because nothing else has been allowed any time:

tokens_per_second (decode, batch 1)  ≤  B / (W + k × t)

B is a published figure, listed for the four tracks on the hardware reference and measured for yourself in Part 5. The usual shortcut for W is the file size. It is close for a dense model and not exact, because one tensor is not read in full. Reading the tensor table in the header of the course’s Qwen3-8B file (types from ggml’s ggml_type enum) gives the real figure:

Qwen3-8B-Q4_K_M.gguf, tensor group	Type	Parameters	Bytes in the file	Bytes read per decode token
Attention: q, k, v, o projections, 36 layers	Q4_K and Q6_K	1,509,949,440	868,810,752	all
Feed-forward: gate, up, down, 36 layers	Q4_K and Q6_K	5,435,817,984	3,291,217,920	all
Output head	Q6_K	622,329,856	510,504,960	all
Normalisation scales	F32	308,224	1,232,896	all
Token embedding table	Q4_K	622,329,856	350,060,544	one row of 4,096 values: 2,304 bytes
Header: metadata and tokeniser			5,957,440	none
Total		8,190,735,360	5,027,784,512	4,671,768,832, 93 per cent of the file

The embedding table is a lookup: a token reads its own row and nothing else. The output head is a full matrix multiplication and is read in full, in its larger Q6_K type. For a mixture of experts the same table is where most of the file drops out of W, which is the last section’s subject. This calculator carries the tensor-level W for Qwen3-8B, Qwen3-30B-A3B (Apache-2.0) and gpt-oss-120b (Apache-2.0), adds the cache, and prices a context length and a batch; pass your own bandwidth as the argument:

RunnableAll tracks

decode-ceiling.py

"""Decode ceiling = bandwidth / bytes one step reads: shared weights, touched experts, KV cache."""
import sys

GB, WINDOW = 1e9, 128          # gpt-oss sliding-window layers see only the last 128 tokens
bandwidth = float(sys.argv[1]) if len(sys.argv) > 1 else 273   # GB/s; 273 = DGX Spark figure
MODELS = {  # GB read every token, GB of routed experts, experts used, experts per layer,
            # KV bytes per token (FP16) in full-attention layers, in sliding-window layers
    "Qwen3-8B":      (4.672,  0.000, 1,   1, 147_456,      0),
    "Qwen3-30B-A3B": (0.823, 17.553, 8, 128,  98_304,      0),
    "gpt-oss-120b":  (1.686, 61.073, 4, 128,  36_864, 36_864),
}

def step_gb(model, context, batch=1):
    shared, experts, used, per_layer, kv_full, kv_window = model
    touched = 1 - (1 - used / per_layer) ** batch   # share of experts any sequence routes to
    cache = batch * (kv_full * context + kv_window * min(context, WINDOW)) / GB
    return shared + experts * touched + cache

contexts = (0, 4_096, 32_768)
print(f"batch 1 at {bandwidth:g} GB/s: ceiling in tokens/s, and GB read per token")
print(f"{'context':<14}" + "".join(f"{c:>18,}" for c in contexts))
for name, m in MODELS.items():
    cells = (f"{bandwidth / step_gb(m, c):>10.1f} {step_gb(m, c):>4.1f} GB" for c in contexts)
    print(f"{name:<14}" + "".join(cells))

print(f"\n4,096-token sequences: ceiling per sequence / summed over the batch")
print(f"{'batch':<8}" + "".join(f"{name:>18}" for name in MODELS))
for batch in (1, 4, 16, 64):
    each = [bandwidth / step_gb(m, 4_096, batch) for m in MODELS.values()]
    print(f"{batch:<8}" + "".join(f"{e:>10.1f} / {e * batch:>4.0f}" for e in each))

RunnableAll tracks

price the three models at the DGX Spark's vendor bandwidth

python decode-ceiling.py 273

Output — what you should see

batch 1 at 273 GB/s: ceiling in tokens/s, and GB read per token
context                        0             4,096            32,768
Qwen3-8B            58.4  4.7 GB      51.7  5.3 GB      28.7  9.5 GB
Qwen3-30B-A3B      142.2  1.9 GB     117.5  2.3 GB      53.1  5.1 GB
gpt-oss-120b        75.9  3.6 GB      72.8  3.8 GB      56.8  4.8 GB

4,096-token sequences: ceiling per sequence / summed over the batch
batch             Qwen3-8B     Qwen3-30B-A3B      gpt-oss-120b
1             51.7 /   52     117.5 /  118      72.8 /   73
4             38.5 /  154      42.5 /  170      28.5 /  114
16            19.0 /  305      14.7 /  235       9.6 /  153
64             6.3 /  403       6.2 /  398       4.2 /  270

The first block is the batch-one ceiling; the second belongs to the batching section. Run with each track’s figure, at a conversation already 4,096 tokens long, the calculator gives:

Pending validationCeiling on decode speed from bandwidth arithmetic, batch 1, 4,096 tokens in context

Platform	Bandwidth (GB/s)	Qwen3-8B Q4_K_M, 5.28 GB read	Qwen3-30B-A3B Q4_K_M, 2.32 GB read	gpt-oss-120b MXFP4, 3.75 GB read
DGX Spark (Track S)	273	51.7	117.5	72.8
Ryzen AI Max+ 395 (Track X)	256	48.5	110.2	68.3 (128 GB machines only)
Mac Studio, M4 Max (Track M)	546	103.5	235.1	145.6 (with enough memory)
Desktop, RTX 5090 (Track N)	1,792	339.7	771.5	does not fit in 32 GB

The four course tracks; vendor bandwidth figures as recorded in src/data/hardware.json · none - arithmetic only, no engine was run decode-ceiling.py on this page; GGUF tensor tables and config.json files read 2026-09-12 · Qwen3-8B (dense), Qwen3-30B-A3B and gpt-oss-120b (mixture of experts), Q4_K_M for the Qwen3 models, MXFP4 experts with Q8_0 attention for gpt-oss-120b; FP16 KV cache · 4,096 tokens of context · 2026-09-12

Estimates, not measurements: bandwidth divided by the bytes one decode step reads, which is the weights each token uses (from the files' tensor tables) plus 4,096 tokens of FP16 KV cache. Nothing here allows for attention arithmetic, dequantisation, framework overhead, or bandwidth below the specification. gpt-oss-120b's 63.4 GB file needs a 128 GB Track X machine or a Mac with the memory to hold it. Part 5 measures the bandwidth and Part 6 measures the achieved speed; expect the measurement to land below these numbers.

The value of a ceiling is not that you will hit it. It is that you cannot exceed it, so it tells you immediately whether an idea is plausible. If a machine’s arithmetic says forty and you need a hundred, no amount of tuning will get you there: you need fewer bytes per token or more bandwidth. If the arithmetic says four hundred and you are measuring twenty, something is wrong with your setup, and the gap section below and Part 6’s two-tokens-per-second challenge say where to look.

?Reality check

The claim to be sceptical of is any tokens-per-second figure quoted without the model, its quantisation, the context length, the batch size, the engine and the machine. The calculator shows context length alone halving Qwen3-8B’s ceiling and batch size cutting each user’s by a factor of eight. This is why every measurement in this course is shown in a table that carries all six, and why the numbers above are labelled as pending arithmetic rather than results.

Time to first token and time per output token

Users experience the two phases as two separate qualities, and each has a definition you can compute from timestamps:

TTFT        = queueing + tokenisation + prefill(n_prompt)           the wait before anything appears
TPOT        = (end_to_end − TTFT) / (n_output − 1)                  the gap between later tokens
end_to_end  = TTFT + TPOT × (n_output − 1)
decode rate = 1 / TPOT            prompt processing rate = n_prompt / prefill seconds

The − 1 is there because the first output token is produced by prefill, not by a decode step. Part 22’s serving lesson adds goodput, the throughput that stays inside both limits, and names the metrics vLLM exports for each. Worked for an illustrative request, not a measurement of any machine:

An 8,000-token prompt, a 400-token answer, first token at 12.0 s, last at 30.0 s	Arithmetic	Result
Time to first token	read off	12.0 s
Prompt processing rate	8,000 ÷ 12.0, assuming no queueing	667 tokens/s
Time per output token	(30.0 − 12.0) ÷ 399	45.1 ms
Decode rate	1 ÷ 0.0451	22.2 tokens/s
The stopwatch figure	400 ÷ 30.0	13.3 tokens/s, which is neither

Now change one thing at a time and see which clock moves:

Change	TTFT	TPOT	End to end
None	12.0 s	45.1 ms	30.0 s
Prompt cut to 2,000 tokens (prefill roughly proportional at this length)	3.0 s	45.1 ms or slightly less	21.0 s
Decode 1.5 times faster, from fewer bytes per token	12.0 s	30.1 ms	24.0 s
A cached prefix covers 7,500 of the 8,000 prompt tokens	about 0.75 s or more (the 500 new tokens still attend over all 8,000)	45.1 ms	about 18.8 s or more

These are different problems with different fixes, and confusing them wastes days. The decision table, by symptom:

Symptom you measured	Phase	What acts on it	Where the course teaches it
Long wait before the first token, growing with the prompt	prefill	a shorter prompt; reusing the cache for a shared prefix; more arithmetic per second; an attention backend that never builds the n × n matrix	prefix caching, Part 2’s attention lesson
Tokens flow slowly whatever the prompt	decode	fewer bytes per token: coarser quantisation, a smaller or mixture-of-experts model; higher bandwidth; checking several drafted tokens in one pass	Part 16’s quantisation lessons, speculative decoding
Tokens slow down as a conversation grows	decode, cache reads	a shorter context; a quantised cache; a model with fewer KV heads or sliding-window layers	Part 4’s cache section
One user is fine, twenty are slow	batching limits	fewer concurrent sequences, less context each, a continuous-batching server	Part 9

✕Common mistake

The symptom is a benchmark note whose single tokens-per-second figure sits far below the generation rate the engine’s own log reports. The cause is dividing output tokens by total wall-clock time, which charges the prompt’s prefill to decode; with a long prompt and a short answer the figure can be several times too low, and a change that only speeds up prefill appears to speed up “decode”. The fix is to record TTFT and the output token count separately and compute TPOT with the formula above, or read the prompt-processing and generation rates that benchmark tools such as llama-bench report in separate columns, as Part 6 does.

KV cache growth with context

Part 2 derived the cache’s cost, k = 2 × layers × kv_heads × head_dim × bytes_per_element for every token, 147,456 bytes for Qwen3-8B at FP16, and confirmed it on a live forward pass. It costs twice here: once in memory, and once in speed.

Memory follows the context length you allocate, not the conversation you happen to have: llama.cpp reserves the cache when the model loads, which Part 6’s benchmark lab has you watch in the load log, and a server holds one cache per concurrent sequence.

Qwen3-8B at four-bit weights, filling a 12 GB card as the context grows

Weights, Q4_K_M

5 GB

KV cache at the full 32,768-token context, FP16

4.8 GB

Free

2.2 GB

Total

12 GB

Estimates from arithmetic: 147,456 bytes per token times 32,768 tokens for the cache, the file size for the weights; engines add compute buffers and their own overheads. At full context this model's cache approaches the size of its weights, which is why 'it fits' has to be answered for a context length and not just for a model.

Speed follows the tokens actually in the cache, because every decode step’s attention reads all of them in every full-attention layer. That puts k × t in the denominator of the ceiling, next to a weight term that does not change, so the ceiling falls hyperbolically as the conversation grows:

Tokens in context	Qwen3-8B: cache read per token	Share of bytes read that is cache	Ceiling at 273 GB/s	Qwen3-30B-A3B: cache read	Share that is cache	Ceiling at 273 GB/s
0	0	0 per cent	58.4	0	0 per cent	142.2
4,096	0.60 GB	11 per cent	51.7	0.40 GB	17 per cent	117.5
16,384	2.42 GB	34 per cent	38.5	1.61 GB	46 per cent	77.3
32,768	4.83 GB	51 per cent	28.7	3.22 GB	63 per cent	53.1

Two things in that table are not obvious. Doubling the context from 16,384 to 32,768 costs Qwen3-8B a quarter of its ceiling, not half, because the weights are a fixed term. And the mixture-of-experts model loses proportionally more, because its weight term is small: at 32,768 tokens it reads more cache than weights, and its lead over the dense model shrinks from about 2.4 times to 1.9. gpt-oss-120b falls least in the calculator: by a quarter from an empty context to 32,768 tokens (75.9 to 56.8), where Qwen3-8B loses half and Qwen3-30B-A3B nearly two thirds, because its full-attention layers keep 8 heads of dimension 64 and the other half of its layers are sliding-window layers that attend to the last 128 tokens only, which Part 4’s cache section describes.

The CPU run earlier lets you check the byte arithmetic against a clock. Qwen3-0.6B in FP32 reads all 596,049,920 parameters per token, because its output head is the embedding matrix itself (tie_word_embeddings in its config.json), and with that file’s 28 layers, 8 key-value heads and head dimension of 128 its FP32 cache costs 2 × 28 × 8 × 128 × 4 = 229,376 bytes per token:

Qwen3-0.6B, FP32, the snippet’s first and last rows	Bytes moved per token	Predicted TPOT ratio	Measured TPOT
About 1 token in context	2.384 GB	1	87.6 ms
About 4,100 tokens in context, the cache attention reads	2.384 + 0.940 = 3.324 GB	1.39	165.7 ms, a ratio of 1.89
About 4,100 tokens, counting the engine’s copy of the cache	2.384 + 0.940 + 0.940 = 4.264 GB	1.79	165.7 ms, 1.89

The cache attention reads explains under half of the extra time. Attention’s own arithmetic is not the rest: one token’s scores and weighted sum over 4,100 cached tokens come to about 2 × 28 layers × 16 heads × 128 × 4,100 ≈ 0.47 GFLOP, under 1 ms at the 741 GFLOP/s the same CPU reached on a long prompt. The rest is bytes the engine moves beyond what attention reads: Transformers’ DynamicCache appends each step’s keys and values with torch.cat (DynamicLayer.update in cache_utils.py at v5.16.1), which copies the whole cache into a new tensor on every step. Counting that copy as one more pass over the cache gives the third row, close to the clock. The formula counts what attention must read; an engine can move more. Predict, measure, explain the gap: that is the whole method.

This is exactly the problem the PagedAttention paper set out to solve for servers. Its abstract describes the difficulty: “the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size.” Part 9 is where you run the engine that came out of that paper.

Batching and its limits

At batch one the processor fetches every weight to produce a single token and then waits for the next fetch. Put b sequences into the same step and the weights are fetched once for all of them, while the arithmetic and the cache reads scale with b:

bytes read per step   = W + b × k × t            (dense model; every sequence's cache is read)
FLOPs per step        = 2 × N × b
step seconds          ≈ max( (W + b × k × t) / B ,  2 × N × b / C )
each sequence's rate  = 1 / step seconds          all sequences together = b / step seconds
memory resident       = file size + b × k × context_length

That is why a server delivers far more total tokens per second than any one of its users sees. The same abstract puts the requirement first: “High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time”, and reports that “vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems”, as of the paper’s 2023 comparison. That is a throughput claim, not a single-user latency claim; read every serving speed-up that way. The calculator’s second block, for Qwen3-8B at the DGX Spark’s vendor bandwidth with 4,096 tokens in each sequence and compute ignored:

Sequences	GB read per step	Ceiling per sequence (tokens/s)	Ceiling summed (tokens/s)	Memory: weights + caches
1	5.28	51.7	52	5.6 GB
4	7.09	38.5	154	7.4 GB
16	14.34	19.0	305	14.7 GB
64	43.33	6.3	403	43.7 GB

The formulas name the three limits, and each shows up in a column:

Limit	Mechanism	Where you see it
Cache reads	every sequence adds k × t bytes to every step, so the fixed weight term is soon the minority	per-sequence ceiling falls; the sum grows ever more slowly; the 256-token run below
Memory	every sequence holds its own cache for its whole context length	the last column; at some b it exceeds the machine
Arithmetic	FLOPs per step grow with b; once 2 × N × b / C passes the byte term, adding a sequence slows every sequence in proportion	not in the table, which ignores C; the 2-token run below shows it

A CPU reaches both of the step’s limits early, which makes them easy to watch. This snippet decodes one step for 1 to 32 sequences, each already holding the number of tokens given as its second argument (256 when there is none). Run it twice: with 2 tokens of context the cache is negligible and the arithmetic shows alone; with 256 the cache joins it.

RunnableAll tracks

batch-decode.py

"""Decode several sequences in one step: the total rate rises, each sequence's own rate falls."""
import sys, time, torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(sys.argv[1], dtype=torch.float32).eval()
vocab, steps = model.config.vocab_size, 9
context = int(sys.argv[2]) if len(sys.argv) > 2 else 256    # tokens each sequence already holds
torch.manual_seed(0)

def step_time(batch):
    ids = torch.randint(0, vocab, (batch, context))          # every sequence starts with `context` tokens
    with torch.inference_mode():
        out = model(input_ids=ids, logits_to_keep=1)         # prefill all of them first
        cache, token = out.past_key_values, out.logits[:, -1:].argmax(-1)
        times = []
        for _ in range(steps):                               # each step writes one token per sequence
            start = time.perf_counter()
            out = model(input_ids=token, past_key_values=cache)
            token = out.logits[:, -1:].argmax(-1)
            times.append(time.perf_counter() - start)
    return sorted(times)[steps // 2]                         # the median step

step_time(1)                                                 # warm-up, not reported
print(f"{'batch':>5} {'step ms':>8} {'total tok/s':>12} {'each sequence tok/s':>20}")
for batch in (1, 2, 4, 8, 16, 32):
    step = step_time(batch)
    print(f"{batch:>5} {step * 1e3:>8.1f} {batch / step:>12.1f} {1 / step:>20.1f}")

RunnableAll tracks

first run: 2 tokens of context per sequence

python batch-decode.py ~/llm-course/models/qwen3-0.6b 2

Output — what you should see

batch-decode.py with 2 tokens of context, torch 2.14.0 and transformers 5.16.1, CPU only (12 threads), one run on the author's test box; your figures will differ, the shape will not

Loading weights: 100%|██████████| 311/311 [00:00<00:00, xxxx.xxit/s]
batch  step ms  total tok/s  each sequence tok/s
    1     88.8         11.3                 11.3
    2     76.4         26.2                 13.1
    4     85.3         46.9                 11.7
    8    131.6         60.8                  7.6
   16    166.5         96.1                  6.0
   32    235.1        136.1                  4.3

RunnableAll tracks

second run: 256 tokens of context per sequence

python batch-decode.py ~/llm-course/models/qwen3-0.6b 256

Output — what you should see

batch-decode.py with 256 tokens of context, torch 2.14.0 and transformers 5.16.1, CPU only (12 threads), one run on the author's test box; your figures will differ, the shape will not

Loading weights: 100%|██████████| 311/311 [00:00<00:00, xxxx.xxit/s]
batch  step ms  total tok/s  each sequence tok/s
    1     88.5         11.3                 11.3
    2     92.4         21.6                 10.8
    4    104.1         38.4                  9.6
    8    174.8         45.8                  5.7
   16    206.3         77.6                  4.8
   32    433.5         73.8                  2.3

With 2 tokens of context the step stays flat to 4 sequences: it was waiting on the weight fetch, and the extra rows were nearly free. From 8 it lengthens, which is the arithmetic limit, and it arrives sooner than C suggests: 8 sequences need about 9.5 GFLOP in all, 13 ms at the rate a long prompt reached, yet their step is about 43 ms longer than one sequence’s, because a multiply of a few rows gets little of C. Even so, the total keeps growing all the way to 32. With 256 tokens each, the step also reads every sequence’s cache, which Transformers copies again, and the two runs part at 32 sequences:

One decode step for 32 sequences, arithmetic from stated inputs	2 tokens of context each	256 tokens of context each
Weights read, FP32	2.384 GB	2.384 GB
Cache read, 32 × tokens × 229,376 bytes	0.015 GB	1.879 GB
Weight multiplies, 32 × 2 × 596,049,920	38.1 GFLOP	38.1 GFLOP
Attention arithmetic, 32 × 2 × 28 × 16 × 128 × tokens	0.007 GFLOP	0.94 GFLOP
Measured step, from the runs above	235.1 ms	433.5 ms, 1.84 times as long
Measured total	136.1 tokens/s, still rising	73.8 tokens/s, flat since 16 sequences

The arithmetic barely differs between the columns, while the bytes read nearly double before the copy is counted, so the flat total in the second run is the cache-read limit, the one the calculator can see. Part 9’s load test finds both limits on your own machine.

For one person at one machine, which is most of this course, batching gains nothing, because there is one sequence and nothing to share the fetch with. More server slots (llama-server’s --parallel) give that one sequence nothing; the prompt batch (--batch-size and --ubatch-size, the logical and physical maximum batch sizes in llama-server’s README at build b10867) is a different setting, which governs how many prompt tokens prefill processes at once. Batching matters when you serve a household, run an agent that issues many calls at once, or generate a dataset: Part 15’s synthetic-data lesson batches thousands of teacher answers, and does the concurrency arithmetic above before it starts.

What sits between the ceiling and the measurement

A first measurement always comes in below the arithmetic, and the gap is informative. The usual contributors, and how each one shows itself:

Contributor	Mechanism	What it looks like	Checked in
Cache handling in the engine	some engines copy or expand the cache on each step, so more bytes move than attention reads	slowdown with context larger than the byte arithmetic predicts, as in the CPU run	this page’s CPU run
Bandwidth below the specification	the vendor figure is a theoretical peak; real access patterns, controller overhead and heat reduce it	every model on the machine short by a similar fraction	Part 5 measurement lab
Dequantisation and kernels	quantised blocks are unpacked during the multiply; each layer costs framework overhead	some formats slower than their size predicts	Part 6 quantisation sweep
Weights split across two memories	each part of the model is read at its own memory’s bandwidth	a small offloaded fraction costs a large factor (table below)	Part 6 challenge
Other work on the machine	a desktop, a browser, a second model still loaded	run-to-run variation; unified-memory tracks share the pool with everything	Part 5 measurement lab

The split case deserves its arithmetic, because it produces the largest gaps. Layers kept in system memory are either computed by the CPU from system memory or copied across PCIe to the card for every token; either way their bytes move at the slower memory’s or link’s rate, so the time per token is a sum, W_fast / B_fast + W_slow / B_slow. With a fraction f of the bytes on memory r times slower, the slowdown against keeping everything on the fast memory is (1 − f) + f × r. For a mixture of experts, count only the experts a token reads, not the file:

Share of the bytes read per token that sit on the slower memory	Slower by 5 times	Slower by 10 times	Slower by 20 times
5 per cent	1.20 times slower overall	1.45	1.95
10 per cent	1.40	1.90	2.90
25 per cent	2.00	3.25	5.75
50 per cent	3.00	5.50	10.50

A gap of some tens of per cent is ordinary. A gap of a factor is a symptom, and usually a specific one: layers offloaded to the CPU, the wrong build of the engine, a context far larger than the conversation needs, or a model that only just fits. Part 6’s challenge turns each into a check.

Why mixture-of-experts models decode fast

A mixture-of-experts model replaces the feed-forward block of each layer with many smaller ones and a router that picks a few per token. Every expert must be resident, because the next token may need any of them, but only the selected ones are read. Qwen3-30B-A3B’s card states the arrangement: “30.5B in total and 3.3B activated”, with 128 experts of which 8 are activated; gpt-oss-120b’s card gives “117B parameters with 5.1B active parameters”, and its config routes each token to 4 of 128 experts. Reading their files’ tensor tables the same way as Qwen3-8B’s:

At batch 1, bytes	Read every token: everything but the embedding table and the routed experts	Routed experts in the file	Experts read per token	Read per token	Share of the file
Qwen3-8B Q4_K_M, dense	4,671,766,528	none	none	4.67 GB	93 per cent of 5.03 GB
Qwen3-30B-A3B Q4_K_M	822,523,904: attention, router, norms, head	17,553,162,240 (Q4_K and Q6_K)	8 of 128: 1,097,072,640	1.92 GB	10 per cent of 18.56 GB
gpt-oss-120b MXFP4	1,685,668,608: the same groups, attention and head in Q8_0	61,073,326,080 (MXFP4)	4 of 128: 1,908,541,440	3.59 GB	6 per cent of 63.39 GB

That is the whole trade: capacity is paid in memory, speed is paid in bytes read, and a mixture of experts separates the two. The usual shortcut for a mixture of experts, file size times active over total parameters, which Part 5’s predict-decode.py uses, is close for Qwen3 and misleading for gpt-oss, and the reason is in the types:

Model	File × active ÷ total	Tensor by tensor	Why the shortcut is off
Qwen3-8B	5.03 GB (dense: the whole file)	4.67 GB	high: counts the embedding table, of which one row is read
Qwen3-30B-A3B	18.56 × 3.3 ÷ 30.5 = 2.01 GB	1.92 GB	high: Qwen’s “activated” includes the embedding table
gpt-oss-120b	63.39 × 5.1 ÷ 117 = 2.76 GB	3.59 GB	low: the always-read tensors are Q8_0, about twice the bytes per parameter of the MXFP4 experts

Three limits keep the advantage from being free, and all three are already in this lesson’s arithmetic.

Batching spreads the routing. Each sequence picks its own experts, so a step for b sequences reads the union. If every token chose 8 of 128 experts uniformly and independently, a layer would touch 128 × (1 − (120/128)^b) of them per step, the touched line in the calculator:

Sequences in the step	Qwen3-30B-A3B experts touched per layer	Qwen3-30B-A3B expert bytes read per step	gpt-oss-120b experts touched per layer
1	8.0 of 128	1.10 GB	4.0 of 128
4	29.1	3.99 GB	15.3
16	82.4	11.30 GB	51.0
64	125.9	17.27 GB	111.2

For a fixed number of experts per token, uniform independent routing touches the most experts; skewed routing, or sequences in a batch that favour the same experts, touches fewer, so treat the table as the pessimistic end. Even so the calculator’s second block shows the consequence: at 16 sequences Qwen3-30B-A3B’s per-sequence ceiling falls below the dense 8B’s. Its arithmetic stays small: 2 × 2,730,702,848 block parameters is about 5.5 GFLOP per token, 40 per cent of the dense 8B’s, which counts wherever a step is compute-bound.

Long context erodes it, as the KV table showed: the cache term does not care how sparse the feed-forward blocks are.

Memory is the price. The 18.6 GB and 63.4 GB files must be resident in full, and their caches on top. This is why a large mixture of experts suits the unified-memory tracks, which have capacity to spare and bandwidth they cannot raise, and why gpt-oss-120b is a realistic model on a 128 GB desktop machine and not on a 32 GB card however fast its memory; Part 4’s architecture lesson makes that argument per track.

Locate the bottleneck with a controlled perturbation

A bandwidth estimate is a hypothesis about the dominant cost, not a substitute for measurement. Keep the checkpoint and output limit fixed, then change prompt length. If time to first token rises while later token cadence stays similar, prompt processing is a candidate bottleneck. Next hold prompt length fixed and increase simultaneous requests. If aggregate throughput rises while each user’s response slows, batching is amortising some work while adding contention or queueing.

Long-context decode also reads attention state, so a weight-only estimate becomes less informative as the KV cache grows. Host offload introduces another memory tier and transfers; small models can become sensitive to kernel launch and scheduling overhead. The description “decode is bandwidth-bound” is most useful when its assumptions include workload shape and residency.

Record separately request queue time, prefill, decode and end-to-end latency where the engine exposes them. If it exposes only wall time, state that limit. A faster token counter does not imply a faster answer if the new model emits more tokens, waits longer to start or repeatedly retries malformed responses.

Recap

Prefill reads the prompt in one pass per layer, does 2 × N × n operations with one fetch of the weights, and becomes compute-bound as the prompt grows; decode does one pass per token with the same fetch and little arithmetic, and is bandwidth-bound. The decode ceiling is B / (W + k × t), with W read off the file’s tensor table, not its size. TTFT is prefill and TPOT is decode; compute them separately, because a stopwatch average charges one to the other. Batching shares the weight fetch across sequences and is stopped by cache reads, cache memory and arithmetic. A mixture of experts shrinks W to a tenth or less at batch one, and gives some of that back under batching and at long context.

Check your understanding

Question 1. Qwen3-8B at Q4_K_M reads 4.67 GB of weights per token and 147,456 bytes of FP16 cache per token of context. On a machine with 273 gigabytes per second of bandwidth, what happens to the decode ceiling when a conversation grows from 16,384 to 32,768 tokens?

• It halves, because the context doubled
• It falls by about a quarter
• It does not change, because the weights are the same
• It falls by about a tenth

Show the answer and why

Answer: It falls by about a quarter

Bytes read per token go from 4.67 + 2.42 = 7.09 GB to 4.67 + 4.83 = 9.50 GB, so a ceiling of about 38 becomes about 29. Doubling the context doubles only the cache term; the weight term is fixed, which is why the fall is less than half, and why a model with a small weight term, such as a mixture of experts, loses proportionally more.

Question 2. A request with a 6,000-token prompt returns 300 tokens. The first token appears after 8 seconds and the last after 23 seconds. What was the decode rate, in tokens per second?

• About 13
• About 20
• About 37
• About 750

Show the answer and why

Answer: About 20

TPOT is (23 − 8) ÷ 299, about 50 ms, so decode ran near 20 a second. 13 is 300 ÷ 23, which charges the prefill to decode; 37 is 300 ÷ 8, which divides by the wrong phase’s time. 6,000 ÷ 8, about 750, is the prompt processing rate, a different number from a different clock, which the engine reports in its own column.

Question 3. Four readers work on Qwen3-30B-A3B (18.56 GB file, 1.92 GB read per token, 3.22 GB of cache at 32,768 tokens) with 273 gigabytes per second of bandwidth. Which calculation is the bug?

• Ceiling for the first tokens of a new conversation: 273 ÷ 1.92
• Ceiling once 32,768 tokens are in context: 273 ÷ (1.92 + 3.22)
• Ceiling for any conversation: 273 ÷ 18.56, because every expert has to be in memory
• Memory needed at 32,768 tokens: 18.56 + 3.22 GB, before engine overhead

Show the answer and why

Answer: Ceiling for any conversation: 273 ÷ 18.56, because every expert has to be in memory

Resident bytes and bytes read are different quantities. Every expert must be in memory, so the memory line uses the whole file; only 8 of 128 experts are read per token, so the speed line uses 1.92 GB. Dividing by the file size predicts a ceiling of about 15 where the real one is about 142 at an empty context, and would make the model look slower than the dense 8B it outruns.

Question 4. Qwen3-8B serves 16 conversations of 4,096 tokens each on a machine with 273 gigabytes per second of bandwidth, compared with one. Ignoring compute, what does the bandwidth arithmetic say?

• Each conversation keeps its ceiling of about 52 and the total is 16 times higher
• Each conversation falls to about 19 and the total rises about six-fold, to about 305
• Each conversation falls to a sixteenth and the total is unchanged
• Both the per-conversation and the total ceiling fall

Show the answer and why

Answer: Each conversation falls to about 19 and the total rises about six-fold, to about 305

A step reads the weights once (4.67 GB) plus sixteen caches (16 × 0.60 GB), 14.3 GB in all, so each sequence gets 273 ÷ 14.3 and the batch 16 times that. The weights are shared and the caches are not, which is why the total grows sub-linearly; on a real machine the arithmetic limit bends the curve further, as the CPU run showed.

Question 5. Which statements about Qwen3-30B-A3B are supported by this lesson’s arithmetic? Select all that apply.

• At batch 1 it reads about a tenth of its Q4_K_M file per token
• If routing were uniform and independent, a step for 16 sequences would touch about two thirds of the experts in each layer
• Its cache per token is smaller than Qwen3-8B’s, because it has half as many key-value heads per layer
• Because only 3.3B parameters are active, about 2 GB of memory is enough to run it

Show the answer and why

Answer: At batch 1 it reads about a tenth of its Q4_K_M file per token, If routing were uniform and independent, a step for 16 sequences would touch about two thirds of the experts in each layer, Its cache per token is smaller than Qwen3-8B’s, because it has half as many key-value heads per layer

1.92 of 18.56 GB is about 10 per cent; 1 − (120/128)^16 is 64 per cent; and 48 × 4 × 128 × 2 × 2 = 98,304 bytes against Qwen3-8B’s 36 × 8 × 128 × 2 × 2 = 147,456, despite having more layers. Memory follows the total: the whole 18.56 GB file must be resident, because the router may pick any expert for the next token.

Question 6. In the CPU timing run, prompt processing rose from about 12 prompt tokens a second for a one-token prompt to about 840 at 1,024 tokens, then fell at 4,096. Which explanation fits all three regions?

• Bandwidth-bound for tiny prompts, compute-bound once the prompt is long enough, then attention’s n² work growing faster than the prompt
• The CPU warmed up during the run, then throttled
• The KV cache was full at 1,024 tokens
• Compute-bound for tiny prompts, then bandwidth-bound for long ones

Show the answer and why

Answer: Bandwidth-bound for tiny prompts, compute-bound once the prompt is long enough, then attention’s n² work growing faster than the prompt

A one-token pass fetches all the weights to do very little arithmetic, so it waits on memory; longer prompts reuse the same fetch for more work until the processor’s arithmetic is the limit; at 4,096 tokens the score matrices, which grow with the square of the prompt, add work faster than tokens are added. Warming up and throttling would not make a one-token prompt cost exactly a decode step, and the snippet’s cache has no fixed size to fill.

Check my answersClear