Challenge: The Model That Runs at Two Tokens per Second

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track faults, fixes and the versions they were reproduced with belong here once the validation pass has run this page on real machines.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

Somebody, possibly you next month, says: “I installed llama.cpp, loaded an eight-billion-parameter model on a machine that cost as much as a car, and it types slower than I do.” By the end of this page you will have a procedure that answers that in about five minutes, and you will have run it against five faults you introduced deliberately, plus the one your platform is prone to, so that you recognise the evidence when the fault is not yours.

The deliverable is not a fixed machine. It is a written diagnosis per fault: the evidence line that identified it, the size of the slowdown the arithmetic predicted beside the one you measured, the single change that fixed it, and the second measurement that proves the change worked. Evidence, then theory, then proof: that is what separates fixing something from changing things until the symptom moves.

⌂Single-machine path

Single-machine on every track. Nothing here needs a second box.

The procedure

Slow model: what to do, in order

1. 11. Turn the impression into a numberllama-bench with the benchmark lab settings. A generation rate can be set beside your lab row and your Part 5 ceiling; an impression cannot.
2. 22. Ask llama.cpp what it can seellama-bench --list-devices. "(none)" means no GPU backend: stop there.
3. 33. Ask where it put the bytesLoad with --verbose and read the offload line, the model and KV buffer lines, the --fit decisions and the memory breakdown.
4. 44. Watch memory while it generatesvmstat si and so on Linux, vm.swapusage and Memory Pressure on a Mac. Pages moving during decode outrank every other explanation.
5. 55. Check the ceiling your platform imposesThe GTT limit on a Ryzen AI Max+, the wired limit on a Mac, the WSL2 virtual machine, the VRAM on a card.
6. 66. Name one fault and predict its sizeFrom the decision table below, with offload-split.py turning the offload line into a predicted slowdown.
7. 77. Change one thing, measure again, check the evidence movedThe same command; the rate back inside the healthy interval and the evidence line changed.

The order is cost against information. Steps 2 to 4 take seconds and each removes a whole row of the fault list below; a quantisation or context experiment takes minutes and removes nothing until the placement is known. The evidence script in task 2 performs steps 2 to 4 in one command.

The short list of faults

Almost every slow local model is one of these. The model, the quantisation and the prompt change speed by sensible amounts; these change it by factors, because each moves bytes that every token needs onto a slower path.

Fault	Mechanism, one level down	Evidence at v0.4.0	Which rate falls
Layers not offloaded	the first blocks sit in system memory and the CPU computes them for every token	offloaded N/M layers to GPU with N below M; a large host model buffer size	both; decode by (1 − f) + f × r
A CPU build, or a GPU backend present but unused	no GPU device is registered, or --device none selects none	--list-devices prints (none); warning: no usable GPU found; no device buffer lines; dev none in llama-bench	prefill by the arithmetic ratio, decode by the bandwidth ratio
Memory spilling to system RAM or swap	the operating system pages model or cache memory to disk, or managed memory migrates pages across PCIe	si and so above zero during decode; swap used rising; on a Mac a working-set warning	both, and erratically
KV cache too large for the device	the cache for the context you asked for is allocated at load and takes the room the layers needed	--fit lines: context size reduced, or context size set by user … no change followed by a layer count	none, if --fit shrank the context; both, if it moved layers
The wrong backend where two exist	Vulkan and ROCm on a Ryzen AI Max+, or a Vulkan archive on a Spark, are different code paths on the same silicon	the backend column and the device name	a measurement, not a known direction
A ceiling the operating system imposes	the GTT limit, the macOS wired limit or the WSL2 virtual machine caps memory below the machine total	the device total in --list-devices below what you bought; free -m inside WSL2	shows up as one of the rows above

Part 3’s split-memory arithmetic gives the decode slowdown when a fraction f of the bytes read per token sits on a path r times slower: (1 − f) + f × r. Every fault on this page is an instance of it, and task 3 has you predict f and r before you measure.

Where v0.4.0 hides the evidence

Three defaults at the pinned version decide what you can see, and all three are read from its source, not from memory.

The placement lines are below the default verbosity. common/log.cpp maps every message the llama library logs at INFO to verbosity level 4, and the default threshold is 3. So a plain llama-server launch prints neither offloaded N/M layers nor the buffer sizes nor the --fit decisions. Add --verbose (or --log-verbosity 4, which Part 4’s headroom rules use). llama-bench is quieter still: without -v it installs a callback that discards every log line, warnings included.

llama-bench and llama-server disagree about what to do when the model does not fit.

Tool	Default --n-gpu-layers	--fit	Default context	When it does not fit
llama-server, llama-cli, neither -c nor -ngl given	auto	on, keeping --fit-target 1,024 MiB free per device	the model’s training context: 40,960 for Qwen3-8B	shrinks the context, no lower than --fit-ctx 4,096, then moves layers to system memory, and loads
the same, with -c N	auto	on	N, which --fit leaves alone; -c 0 means the training context, also left alone	moves layers to system memory, and loads
the same, with -ngl	as given	on, but it may not change -ngl	as above	logs common_fit_params: failed to fit params to free device memory: …, abort as a warning and tries the load as asked, which fails if it does not fit
llama-bench	-1, every layer	off unless -fitt is given	-p + -n + -d tokens	the allocation fails: failed to load model or failed to create context

That is why a model can benchmark fine and serve slowly: the server quietly placed it differently. Part 4 explains how --fit sizes a load; this page reads what it decided.

llama-cli is a chat interface. At v0.4.0, llama-cli -m model.gguf -p "hi" -n 1 prints a banner, answers, and waits for the next turn (tested on a CPU build: with its input redirected from /dev/null it writes prompts in a loop). An evidence script cannot use it to load and exit, so the one on this page starts llama-server, waits for listening on, and stops it.

✕Common mistake

Trusting the offload line over the buffer lines. In load_tensors, the offloaded N/M layers to GPU line is computed from the layer count requested whenever any GPU backend is registered; it does not check that a device was selected. Reading the source, a GPU build started with --device none still logs offloaded 37/37 layers to GPU for Qwen3-8B while every weight sits in a CPU_Mapped buffer (not observed on hardware for this page). The model buffer size lines and the memory breakdown come from the buffers actually allocated. Read those.

Layers not offloaded, in bytes

llama.cpp’s placement rule, from load_tensors at v0.4.0: the input layer, the token embedding table, always stays on the CPU. With L repeating blocks and N = --n-gpu-layers, block i goes to the accelerator when i ≥ L + 1 − N, and the output layer when N ≥ 1. Qwen3-8B has 36 blocks, so M is 37 and -ngl 18 keeps blocks 0 to 18 on the CPU. Each block’s KV cache sits with its block.

offload-split.py reads a GGUF header, applies that rule, and prints what each -ngl value leaves where, with the host’s share f of the bytes read per generated token. It reads only the header, so it runs in a second on any machine. Save it as ~/llm-course/offload-split.py.

RunnableAll tracks

offload-split.py

#!/usr/bin/env python3
"""Show where llama.cpp v0.4.0 puts a GGUF model's bytes for a given --n-gpu-layers value.

Purpose: read a GGUF file's header (never the weights), apply llama.cpp's layer placement
    rule for one or more -ngl values, and print for each how many weight and KV cache MiB
    sit on the accelerator and on the host, what share of the bytes read per generated
    token is on the host, and, given how many times slower the host path is, the decode
    slowdown Part 3's split-memory arithmetic predicts. It turns an "offloaded N/M layers"
    log line into numbers you can check against the load log's buffer lines.
Platform: all (Python standard library only; no accelerator and no model loading)
Minimum memory: 8 GB (the script itself needs a few megabytes)
Assumes: Python 3.9 or later; a GGUF file, complete or just its first megabytes (the header
    and tensor list come first); for --labbook, a notebook holding a part-05/bandwidth-test
    line written by Part 5's bandwidth-test.py with an accelerator and a cpu figure.

Usage: python3 offload-split.py <model.gguf> [--ngl 37,27,18,0] [--ctx-size 4096]
           [--cache-type f16] [--slow-ratio R | --labbook labbook.md]

Placement, from load_tensors in src/llama-model.cpp at v0.4.0: the input layer
(token_embd) always stays on the CPU; with L repeating blocks and N = --n-gpu-layers,
block i goes to the accelerator when i >= L + 1 - N, and the output layer (output.weight
and output_norm) when N >= 1; a model with tied embeddings has no output.weight, and its
output layer loads a duplicate of token_embd instead. Each block's KV cache sits on the same
device as the block (unless --no-kv-offload). Bytes read per generated token: every block
and the output layer; of token_embd only one row, counted here as nothing. The host column
counts the mapped file once. Slowdown against a full offload:
(1 - f) + f x r, with f the host's share of those bytes and r how many times slower the
host reads them. Tensor bytes use the block sizes asserted in ggml/src/ggml-common.h.
"""

import argparse
import json
import re
import struct
import sys
from collections import defaultdict

MIB = 1024 ** 2
# ggml type id: (name, bytes per block, values per block), ggml-common.h at v0.4.0
BLOCK = {0: ("F32", 4, 1), 1: ("F16", 2, 1), 2: ("Q4_0", 18, 32), 3: ("Q4_1", 20, 32),
         6: ("Q5_0", 22, 32), 7: ("Q5_1", 24, 32), 8: ("Q8_0", 34, 32), 10: ("Q2_K", 84, 256),
         11: ("Q3_K", 110, 256), 12: ("Q4_K", 144, 256), 13: ("Q5_K", 176, 256),
         14: ("Q6_K", 210, 256), 20: ("IQ4_NL", 18, 32), 23: ("IQ4_XS", 136, 256),
         30: ("BF16", 2, 1), 39: ("MXFP4", 17, 32)}
CACHE_BYTES = {"f32": 4.0, "f16": 2.0, "bf16": 2.0, "q8_0": 34 / 32, "q4_0": 18 / 32}
SCALAR = {0: "<B", 1: "<b", 2: "<H", 3: "<h", 4: "<I", 5: "<i", 6: "<f", 7: "<?",
          10: "<Q", 11: "<q", 12: "<d"}


def read_gguf(path):
    """Return (metadata, {tensor name: bytes}) from the header of a GGUF file."""
    fh = open(path, "rb")

    def get(fmt):
        data = fh.read(struct.calcsize(fmt))
        if len(data) < struct.calcsize(fmt):
            sys.exit(f"{path}: the file ends inside the header")
        return struct.unpack(fmt, data)[0]

    def text():
        return fh.read(get("<Q")).decode("utf-8", "replace")

    def value(kind):  # 8 is a string, 9 an array, the rest scalars
        if kind == 9:
            inner, count = get("<I"), get("<Q")
            return [value(inner) for _ in range(count)]
        return text() if kind == 8 else get(SCALAR[kind])

    if fh.read(4) != b"GGUF":
        sys.exit(f"{path} is not a GGUF file")
    get("<I")  # format version
    n_tensors, n_kv = get("<Q"), get("<Q")
    meta = dict((text(), value(get("<I"))) for _ in range(n_kv))
    sizes = {}
    for _ in range(n_tensors):
        name, n_dims = text(), get("<I")
        values = 1
        for _ in range(n_dims):
            values *= get("<Q")
        kind, _offset = get("<I"), get("<Q")
        if kind not in BLOCK:
            sys.exit(f"{name}: ggml type {kind} is not in BLOCK; add its block size first")
        _, block_bytes, block_values = BLOCK[kind]
        sizes[name] = values // block_values * block_bytes
    return meta, sizes


def ratio_from_labbook(path):
    """Accelerator read rate over CPU read rate from the latest part-05/bandwidth-test line."""
    latest = None
    for line in open(path, encoding="utf-8"):
        if '"part-05/bandwidth-test"' in line:
            try:
                latest = json.loads(line)
            except json.JSONDecodeError:
                pass
    rates = (latest or {}).get("gbps", {})
    accel = next((k for k in rates if k != "cpu"), None)
    if accel is None or "cpu" not in rates:
        sys.exit(f"{path}: no part-05/bandwidth-test line with an accelerator and a cpu figure; "
                 "pass --slow-ratio instead")
    return rates[accel]["read"] / rates["cpu"]["read"], accel


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("model")
    ap.add_argument("--ngl", default="",
                    help="comma-separated -ngl values (default: all, 3/4, 1/2, 0)")
    ap.add_argument("--ctx-size", type=int, default=4096, help="context length for the KV cache")
    ap.add_argument("--cache-type", default="f16", choices=sorted(CACHE_BYTES))
    ap.add_argument("--slow-ratio", type=float, help="how many times slower the host path reads")
    ap.add_argument("--labbook", help="take the ratio from Part 5's bandwidth-test line")
    args = ap.parse_args()

    meta, sizes = read_gguf(args.model)
    arch = meta["general.architecture"]
    layers = meta[f"{arch}.block_count"]
    kv_heads = meta[f"{arch}.attention.head_count_kv"]
    kv_heads = max(kv_heads) if isinstance(kv_heads, list) else kv_heads
    head_k = meta.get(f"{arch}.attention.key_length",
                      meta[f"{arch}.embedding_length"] // meta[f"{arch}.attention.head_count"])
    head_v = meta.get(f"{arch}.attention.value_length", head_k)
    kv_layer = kv_heads * (head_k + head_v) * CACHE_BYTES[args.cache_type] * args.ctx_size

    block = defaultdict(int)
    output = embd = 0
    for name, n in sizes.items():
        m = re.match(r"blk\.(\d+)\.", name)
        if m:
            block[int(m.group(1))] += n
        elif name.startswith("token_embd"):
            embd += n
        else:
            output += n  # output.weight and output_norm.weight
    tied = "output.weight" not in sizes
    if tied:  # the output layer loads a duplicate of token_embd on its own device
        output += embd

    ratio, source = args.slow_ratio, "--slow-ratio"
    if ratio is None and args.labbook:
        ratio, accel = ratio_from_labbook(args.labbook)
        source = f"{accel} read / cpu read in {args.labbook}"
    ngls = [int(x) for x in args.ngl.split(",")] if args.ngl else \
        [layers + 1, (layers + 1) * 3 // 4, (layers + 1) // 2, 0]

    total_read = sum(block.values()) + output
    file_total = sum(sizes.values())
    print(f"{args.model}: {arch}, {layers} blocks + output layer = {layers + 1} offloadable")
    print(f"token_embd {embd / MIB:.2f} MiB (always host), output layer {output / MIB:.2f} MiB"
          f"{' (tied: a copy of token_embd)' if tied else ''}, "
          f"blocks {min(block.values()) / MIB:.2f} to {max(block.values()) / MIB:.2f} MiB each")
    print(f"KV cache {kv_layer * layers / MIB:.2f} MiB at {args.ctx_size} tokens, "
          f"{args.cache_type}")
    if ratio:
        print(f"host path {ratio:.2f} times slower ({source})")
    print(f"\n{'ngl':>4} {'log line':>9} {'GPU wts':>9} {'host wts':>9} {'GPU KV':>8} "
          f"{'host KV':>8} {'host read':>9} {'slowdown':>8}")
    for ngl in ngls:
        start = max(layers + 1 - ngl, 0)
        gpu_blocks = [i for i in block if i >= start]
        gpu_w = sum(block[i] for i in gpu_blocks) + (output if ngl >= 1 else 0)
        host_w = file_total - gpu_w + (embd if tied and ngl >= 1 else 0)
        f = 1 - gpu_w / total_read
        slow = f"{(1 - f) + f * ratio:8.2f}" if ratio else f"{'-':>8}"
        print(f"{ngl:>4} {f'{min(ngl, layers + 1)}/{layers + 1}':>9} {gpu_w / MIB:9.2f} "
              f"{host_w / MIB:9.2f} {kv_layer * len(gpu_blocks) / MIB:8.2f} "
              f"{kv_layer * (layers - len(gpu_blocks)) / MIB:8.2f} {100 * f:8.1f}% {slow}")


if __name__ == "__main__":
    main()

Download offload-split.py172 lines

RunnableAll tracks

where four -ngl values put Qwen3-8B at Q4_K_M, and what r = 10 would cost

cd ~/llm-course
python3 offload-split.py ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --ngl 37,27,18,0 --ctx-size 4096 --slow-ratio 10

Output — what you should see

captured on the file's header from the Hub, 2026-09-13; the slowdown column uses an illustrative r of 10

/home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf: qwen3, 36 blocks + output layer = 37 offloadable
token_embd 333.84 MiB (always host), output layer 486.87 MiB, blocks 103.53 to 116.94 MiB each
KV cache 576.00 MiB at 4096 tokens, f16
host path 10.00 times slower (--slow-ratio)

 ngl  log line   GPU wts  host wts   GPU KV  host KV host read slowdown
  37     37/37   4455.34    333.84   576.00     0.00      0.0%     1.00
  27     27/37   3339.58   1449.60   416.00   160.00     25.0%     3.25
  18     18/37   2367.58   2421.61   272.00   304.00     46.9%     5.22
   0      0/37      0.00   4789.19     0.00   576.00    100.0%    10.00

Three readings. A full offload still leaves 333.84 MiB of weights on the host, the embedding table, of which a token reads one row, so a healthy load has a small host model buffer size and that is not a fault. The shares are of bytes, not layers: -ngl 27 leaves 10 of 37 layers behind but 25.0 per cent of the bytes, because the 486.87 MiB output layer stays on the device. And r is the only input that differs by machine. With --labbook labbook.md in place of --slow-ratio, the script takes it from your Part 5 bandwidth-test.py line, as the accelerator’s read rate over the CPU’s; on Track N that ratio is large, on the unified-memory tracks it can be close to one, and the same -ngl 18 predicts a very different loss.

The decision rule: when the offload line shows fewer layers than the model has, run the script with that N and your own ratio. A measured slowdown close to the prediction means the partial offload is the whole story; one far larger means a second fault, usually swap.

The KV cache: allocated at load, placed with its layer

The cache for Qwen3-8B costs 2 × 36 layers × 8 kv_heads × 128 head_dim × 2 bytes = 147,456 bytes per token at f16, which Part 3 derived and the benchmark lab checked against the log. The four inputs are in the GGUF header as qwen3.block_count, qwen3.attention.head_count_kv, qwen3.attention.key_length and qwen3.attention.value_length, and the default context is qwen3.context_length, 40,960. The arithmetic, from those inputs and offload-split.py’s weights on the device, not a measurement; compute is the scratch buffer the breakdown reports, unknown until you load:

Context, cache type	KV cache, MiB	Weights on device + KV, MiB	Fits a device offering 8,000 MiB after the margin?
4,096, f16	576.00	5,031	yes, with 2,969 for compute
16,384, f16	2,304.00	6,759	yes, with 1,241 for compute
32,768, f16	4,608.00	9,063	no
40,960, f16 (no -c)	5,760.00	10,215	no
32,768, q8_0	2,448.00	6,903	yes, with 1,097 for compute
40,960, q8_0	3,060.00	7,515	only if compute fits in 485

What happens in the “no” rows depends on what you set, and the three cases were captured with the pinned version on a CPU-only build with Qwen3-0.6B (Apache-2.0, not gated), asking --fit to keep 44,000 of the machine’s 47,145 MiB free so that its 4,480 MiB default cache would not fit:

Output — what you should see

llama.cpp v0.4.0, CPU-only build, Qwen3-0.6B-Q8_0, --verbose, lines filtered; three launches

# no --ctx-size
common_params_fit_impl: cannot meet free memory target of 44000 MiB, need to reduce device memory by 2004 MiB
common_params_fit_impl: context size reduced from 40960 to 22784 -> need 2005 MiB less memory in total
common_params_fit_impl: entire model can be fit by reducing context
llama_kv_cache: size = 2492.00 MiB ( 22784 cells,  28 layers,  4/1 seqs), K (f16): 1246.00 MiB, V (f16): 1246.00 MiB

# --ctx-size 0
common_params_fit_impl: user has requested full context size of 40960 -> no change
common_fit_params: failed to fit params to free device memory: was unable to fit model into system memory by reducing context, abort

# --ctx-size 4000000
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 458752000000
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

On a machine with an accelerator the first case is the same, and the second continues to a third step: with the context fixed and -ngl unset, --fit moves layers to system memory and logs one line per device, - CUDA0 (…): NN layers ( 0 overflowing), … MiB used, … MiB free. That is the “KV cache too large for the device” fault in its v0.4.0 form: nothing fails, the cache takes the room and the layers move out. --no-kv-offload is the mirror image: layers stay on the device and the whole cache sits in system memory, where every decode step’s attention reads it.

A CPU build, or a GPU that is not used

llama.cpp registers every backend compiled in, and at load it looks for devices of type GPU or integrated GPU. A build without one, or with one whose runtime library failed to load, finds none, and nothing stops: the CPU backend runs the model. Three lines say so, captured from the pinned version on a CPU-only build:

Output — what you should see

llama.cpp v0.4.0, CPU-only build, Qwen3-0.6B-Q8_0: llama-bench --list-devices, llama-server with -ngl, llama-bench with -dev none (lines filtered)

Available devices:
  (none)

warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

| model                          |       size |     params | backend    | threads |  fa | dev          |            test |                  t/s |
| qwen3 0.6B Q8_0                | 604.15 MiB |   596.05 M | CPU        |      12 |   1 | none         |            tg16 |         81.97 ± 0.75 |

The warning appears only when -ngl is on the command line. On a build that has a GPU backend but is told --device none, the backend column still names that backend; the dev column reading none is the evidence.

Which rate falls, and by how much, follows from the two clocks the benchmark lab set out. Decode reads every weight per token, so it falls by the ratio of the bandwidth the accelerator and the CPU can each use, the r your Part 5 bandwidth-test.py line gives. Prefill is set by arithmetic, so it falls by the ratio of the accelerator’s arithmetic rate to the CPU’s, which on a GPU is the larger of the two. On the unified-memory tracks the CPU reads the same memory, so predict a modest decode loss and a large prefill loss; on Track N predict both to collapse. A fault that cuts prefill far more than decode has taken arithmetic away; one that cuts decode as hard has taken bandwidth.

Spilling into system memory or swap

The mechanism depends on the memory architecture.

Where	What spills	How it shows
Discrete card, Linux	layers --fit or -ngl left on the host (above); with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, which the build guide says allows “swapping to system RAM instead of crashing when the GPU VRAM is exhausted”, pages that migrate across PCIe	a load that should have failed succeeds; nvidia-smi memory.used at the card’s total; decode far below the split arithmetic
Discrete card, native Windows	the same, through the driver’s System Memory Fallback, which the build guide names as the Windows equivalent	the same; nothing in llama.cpp’s log
DGX Spark	the pool: NVIDIA’s known-issues page says the CPU “may be able to release additional DRAM pages by moving them to SWAP”, so an allocation can succeed by swapping something else out	free -m available near zero; si and so above zero
Ryzen AI Max+	the GTT domain, and the rest of system memory behind it	as on the Spark
Mac	Metal’s comment in ggml-metal-device.m says “it’s possible to allocate more than recommendedMaxWorkingSetSize”; macOS then compresses and swaps	warning: current allocated size is greater than the recommended max working set size in the verbose log; Swap Used rising; the whole machine slows
Any machine, a long-running server	llama-server’s prompt cache, up to --cache-ram 8,192 MiB of host memory by default	a server fast at start that slows after many conversations

The split formula shows why “a little swap” is never a little. Take Qwen3-8B on a Spark and suppose some of the bytes each token reads come back from a swap device. The swap rate below is an assumption chosen to show the shape, not a measurement of any drive:

W = 4.67 GB read per token        Part 3's tensor count for Qwen3-8B at Q4_K_M
B = 273 GB/s                      the Spark's vendor figure in the course hardware reference
S = 3 GB/s                        illustrative swap read rate, an assumption
r = B / S = 91

Share of bytes read per token coming from swap	r = 273 / 3	Slowdown = (1 − f) + f × r	Share of the healthy rate left
1 per cent	91	1.90	53 per cent
5 per cent	91	5.50	18 per cent
10 per cent	91	10.00	10 per cent

The rule: any swap-out during decode, with the model loaded and nothing else running, is the diagnosis, whatever the other lines say. The honest fix is fewer bytes: a smaller file, a shorter context, a quantised cache, or fewer other programs. No flag creates memory.

Requirements

Every track needs: a llama.cpp build from the install lesson that passed its backend check, pinned here at llama.cpp v0.4.0 · verified 2026-09-08; Qwen3-8B-Q4_K_M.gguf (Apache-2.0, not gated; 5,027,784,512 bytes in unsloth/Qwen3-8B-GGUF on 2026-09-13) in ~/models/unsloth/Qwen3-8B-GGUF/, which the benchmark lab downloaded and verified; that lab’s notebook lines for the file in ~/llm-course/labbook.md; and Part 5’s bandwidth-test.py line in the same notebook. The scripts need python3 3.9 or later, curl and awk. Nothing is downloaded. The reports, logs and JSON take a few megabytes.

Time: about forty-five attended minutes of reading and writing diagnoses. The seven evidence runs are unattended; each is one load and one llama-bench run of three repetitions, and your benchmark lab rates give its length:

Pseudocode — not a real command

seconds per run ≈ load time + 4 × 512 / pp512 rate + (3 × 128 + 1) / tg128 rate
    (llama-bench runs one untimed warm-up of the prompt and one generated token, then three repetitions;
     use the CPU rates for task 3's first fault)

Track S — NVIDIA DGX Spark

The Qwen3-8B file fits many times over. The characteristic fault here is a CPU build: on 2026-09-13 the releases page (build b10936) listed ubuntu-arm64 and ubuntu-vulkan-arm64 archives for this architecture and no CUDA archive, so a reader who downloads instead of building gets a working, silent CPU build, or a Vulkan build on an NVIDIA GPU. Task 4 builds the CPU one on purpose in ~/llama.cpp/build-cpu: allow disk for a second build tree the size of du -sh ~/llama.cpp/build, and the compile time your first build took. Run everything on the DGX OS host, where llama.cpp was built.

Track X — AMD Ryzen AI Max+ 395Partial

The Vulkan against ROCm comparison in task 4 needs the optional HIP build from the install lesson; without it that part is skipped.

Two things are characteristic: the GTT limit Part 5 raised, which caps what the GPU may address below the machine total, and the choice between the Vulkan build in ~/llama.cpp/build and the HIP build in ~/llama.cpp/build-hip. The evidence script starts llama.cpp itself and needs a POSIX shell, so on this machine the page runs on Linux; a native Windows installation can take only task 1’s single measurement.

Track M — Apple silicon

The characteristic fault is the wired limit: iogpu.wired_limit_mb, which Part 5 set and which a restart returns to 0, bounds the Metal working set llama.cpp reports as the MTL0 total. Close the browser and anything else large first; they share the pool. The llama-bench tables show a threads column where the other tracks show ngl, because the backend string MTL,BLAS contains BLAS.

Track N — NVIDIA desktop or laptop

Memory here is the card’s VRAM and it is a hard ceiling. The characteristic faults are a partial offload and, inside WSL2, the virtual machine’s memory limit. The primary path is Linux or WSL2, where every command runs as written; the Windows reader works in WSL2, with one PowerShell command in task 4 run on the Windows side.

Preflight

Open one terminal and keep it for the whole page; the variables live in that shell. If you open a new one later, run the first block again.

RunnableAll tracks

the working directory and the names every command uses

cd ~/llm-course
export LLAMA_BIN=~/llama.cpp/build/bin
MODEL=~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
mkdir -p challenge

Save diagnose-slow-inference.sh from task 2 and offload-split.py from above into ~/llm-course, then check every prerequisite:

RunnableAll tracks

tools, files and notebook lines

command -v python3 curl awk
python3 --version
ls -l "$LLAMA_BIN/llama-server" "$LLAMA_BIN/llama-bench" "$MODEL"
ls -l diagnose-slow-inference.sh offload-split.py labbook.md
grep '"lab": "part-06/lab-benchmark-the-reference-models"' labbook.md | grep -c 'Qwen3-8B-Q4_K_M'
grep -c '"lab": "part-05/bandwidth-test"' labbook.md

Output — what you should see

/usr/bin/python3
/usr/bin/curl
/usr/bin/awk
Python 3.x.x
-rwxr-xr-x 1 you you  xxxxxxxx ... /home/you/llama.cpp/build/bin/llama-server
-rwxr-xr-x 1 you you  xxxxxxxx ... /home/you/llama.cpp/build/bin/llama-bench
-rw-r--r-- 1 you you 5027784512 ... /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
-rw-r--r-- 1 you you     xxxxx ... diagnose-slow-inference.sh
-rw-r--r-- 1 you you     xxxxx ... offload-split.py
-rw-r--r-- 1 you you     xxxxx ... labbook.md
2
1

Pass: three paths, Python 3.9 or later, six files listed with the model at 5,027,784,512 bytes, a first count of at least 2 and a second of at least 1. A smaller model file means an incomplete download: run the benchmark lab’s verify-library.sh. A first count of 0 means the benchmark lab’s task 3 is still to do.

RunnableAll tracks

the build, the device, and the memory it offers

"$LLAMA_BIN/llama-server" --version
"$LLAMA_BIN/llama-bench" --list-devices
FREE="$("$LLAMA_BIN/llama-bench" --list-devices | sed -n 's/.*, \([0-9][0-9]*\) MiB free)$/\1/p' | head -n 1)"
echo "FREE=$FREE"

Output — what you should see

version: 0.4.0-dev (build xxxxx, commit xxxxxxx)
built with <compiler> for <platform>
Available devices:
  CUDA0: <device name> (xxxxx MiB, xxxxx MiB free)
FREE=xxxxx

The device is CUDA0 on Tracks S and N, Vulkan0 on Track X and MTL0 on Track M, where a BLAS: Accelerate (0 MiB, 0 MiB free) line follows it and the first match is still MTL0. FREE must be a number of at least 5,819, the benchmark lab’s figure for this file. FREE= with nothing after it and (none) above it is a CPU-only build: fix it with the install lesson’s backend check before going on. Record the version line and the device line.

RunnableAll tracks

the port the evidence script uses

curl -s --max-time 2 http://127.0.0.1:8089/health || echo "port 8089 is free"

Output — what you should see

port 8089 is free

Anything else means a server is still running there: stop it, or put PORT=8090 in front of every bash diagnose-slow-inference.sh command below.

Tasks

1. Establish the healthy baseline

You cannot recognise a fault without knowing what correct looks like on this machine. Run the benchmark lab’s command:

RunnableAll tracks

the baseline, with the benchmark lab's settings

"$LLAMA_BIN/llama-bench" -m "$MODEL" -p 512 -n 128 -r 5 -ngl 999 -fa on

Output — what you should see

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | CUDA       | 999 |   1 |           pp512 |      xxxx.xx ± xx.xx |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | CUDA       | 999 |   1 |           tg128 |         xx.xx ± x.xx |

build: xxxxxxx (xxxxx)

The backend reads Vulkan on Track X and MTL,BLAS on Track M, with threads in place of ngl. Then print the row the benchmark lab recorded for the same file, with the 95 per cent interval that lab’s bench-stats.py uses:

RunnableAll tracks

the benchmark lab's row for this file, from the notebook

python3 - labbook.md Qwen3-8B-Q4_K_M.gguf <<'PY'
import json, sys
T95 = {2: 12.706, 3: 4.303, 4: 3.182, 5: 2.776, 6: 2.571, 7: 2.447, 8: 2.365, 9: 2.306, 10: 2.262}
latest = {}
for line in open(sys.argv[1], encoding="utf-8"):
    if '"part-06/lab-benchmark-the-reference-models"' in line and sys.argv[2] in line:
        row = json.loads(line)
        if row["test"] in ("pp512", "tg128"):
            latest[row["test"]] = row
for test in ("pp512", "tg128"):
    row = latest.get(test)
    if not row:
        print(f"{test}: no benchmark-lab line for {sys.argv[2]}")
        continue
    mean, sd, n = row["tokens_per_s"], row["tokens_per_s_stddev"], row["reps"]
    half = T95.get(n, 2.0) * sd / n ** 0.5
    print(f"{test}: {mean:.2f} ± {sd:.2f}, 95% interval {mean - half:.2f} to {mean + half:.2f}, "
          f"backend {row['backend']}, ngl {row['n_gpu_layers']}, build {row['build']}, {row['measured_on']}")
PY

Output — what you should see

pp512: xxxx.xx ± xx.xx, 95% interval xxxx.xx to xxxx.xx, backend CUDA, ngl 999, build xxxxxxx, 2026-xx-xxTxx:xx:xxZ
tg128: xx.xx ± x.xx, 95% interval xx.xx to xx.xx, backend CUDA, ngl 999, build xxxxxxx, 2026-xx-xxTxx:xx:xxZ

Compute today’s interval the same way: half-width = 2.776 × the ± ÷ √5. Pass: today’s tg128 interval overlaps the recorded one and the build matches. If the build differs, a rebuild is the likeliest cause of any difference. If the intervals do not overlap on the same build, you have found a real fault before the exercise started, which is a legitimate way to do this page: carry on with task 2 and read it as the broken report.

Record: both rates with their ±, the two intervals, the backend string and the build.

2. Collect the healthy evidence

The script gathers steps 2 to 4 of the procedure into one Markdown report with a Signals block at the top, keeps the full load log and the benchmark’s JSON beside it, and appends one part-06/challenge-two-tokens-per-second line to the notebook. It starts llama-server on 127.0.0.1 only, stops it, and changes no setting on the machine.

RunnableAll tracks

diagnose-slow-inference.sh

#!/usr/bin/env bash
# Purpose: collect, into one Markdown report, the evidence that explains a slow model: the
#          build and the devices llama.cpp can see, what the operating system reports for
#          accelerator memory, memory and swap, where llama-server put the weights and the KV
#          cache when it loaded the model and what --fit changed, and the prompt and
#          generation rates measured with that same placement while swap activity is sampled
# Platform: all (Linux, macOS and WSL2; bash 3.2 or later; uses whichever GPU tool exists)
# Minimum memory: 8 GB
# Assumes: llama.cpp v0.4.0 built as in this part's install lesson (llama-server and
#          llama-bench in $LLAMA_BIN, on PATH, or in ~/llama.cpp/build/bin); a GGUF model
#          file; python3 3.9 or later, curl and awk on PATH; nothing listening on $PORT.
#          It starts llama-server bound to 127.0.0.1 and stops it, then runs llama-bench;
#          it changes no setting on the machine.
#
# Usage: bash diagnose-slow-inference.sh <model.gguf> [report.md] [labbook.md]
#
# Environment. Each of the first six maps to llama.cpp options; unset means llama.cpp's
# own default, which is what a plain llama-server command gets.
#   NGL           --n-gpu-layers for the load     (default: unset, "auto": --fit decides)
#   CTX           --ctx-size for the load         (default: 4096). CTX=default leaves the
#                 option out, as a plain llama-server command does: the model's own context,
#                 which --fit may shrink. CTX=0 asks for the model's own context and forbids
#                 --fit to shrink it (common/arg.cpp at v0.4.0).
#   CACHE_TYPE    --cache-type-k and --cache-type-v, for example q8_0 (default: unset, f16)
#   DEVICE        --device, for example none      (default: unset, every device)
#   FIT_TARGET    --fit-target, MiB left free     (default: unset, 1024)
#   NO_KV_OFFLOAD 1 adds --no-kv-offload          (default: unset)
#   LLAMA_BIN     directory holding the two tools (default: PATH, then ~/llama.cpp/build/bin)
#   PORT          local port for the load probe   (default: 8089)
#   LOAD_TIMEOUT  seconds to wait for the load    (default: 600)
#   PROMPT_LEN, GEN_LEN, REPS   llama-bench --n-prompt, --n-gen, --repetitions
#                                                 (default: 512, 128, 3)
#   LABEL         name for the notebook line      (default: the report name without .md)
#
# How it works. llama-server is started with --verbose, because at v0.4.0 the placement lines
# ("offloaded N/M layers", buffer sizes, the KV cache size, the --fit decisions) are logged
# above the default verbosity. Once it prints "listening on", or exits, it is stopped with
# SIGTERM, which prints the memory breakdown. llama-bench then runs with --n-gpu-layers set to
# the N the server logged (or 0 when the load placed nothing on a device) and the same
# --device and --no-kv-offload values, so the rates describe the placement in the report.
# On Linux vmstat samples swap-in and swap-out every second during the benchmark; on macOS
# vm.swapusage is read before and after. Full outputs stay beside the report:
# <report>.load.log, <report>.bench.json, <report>.bench.err and, on Linux, <report>.vmstat.

set -euo pipefail

MODEL="${1:-}"
REPORT="${2:-slow-inference-report.md}"
LABBOOK="${3:-}"
CTX="${CTX:-4096}"
PORT="${PORT:-8089}"
LOAD_TIMEOUT="${LOAD_TIMEOUT:-600}"
PROMPT_LEN="${PROMPT_LEN:-512}"
GEN_LEN="${GEN_LEN:-128}"
REPS="${REPS:-3}"
LABEL="${LABEL:-$(basename "$REPORT" .md)}"

die() { echo "diagnose-slow-inference: $*" >&2; exit 1; }
have() { command -v "$1" >/dev/null 2>&1; }

[ -n "$MODEL" ] || die "usage: bash diagnose-slow-inference.sh <model.gguf> [report.md] [labbook.md]"
[ -f "$MODEL" ] || die "model file '$MODEL' does not exist (names are case-sensitive)"
case "$REPORT" in *.md) ;; *) die "the report name must end in .md, got '$REPORT'" ;; esac
for tool in python3 curl awk; do
  have "$tool" || die "$tool is not installed or not on PATH"
done
if [ -n "$LABBOOK" ] && [ ! -f "$LABBOOK" ]; then
  die "notebook '$LABBOOK' does not exist; create it or leave the third argument out"
fi

if [ -n "${LLAMA_BIN:-}" ]; then
  SERVER="$LLAMA_BIN/llama-server"; BENCH="$LLAMA_BIN/llama-bench"
elif have llama-server && have llama-bench; then
  SERVER="$(command -v llama-server)"; BENCH="$(command -v llama-bench)"
else
  SERVER="$HOME/llama.cpp/build/bin/llama-server"; BENCH="$HOME/llama.cpp/build/bin/llama-bench"
fi
[ -x "$SERVER" ] || die "llama-server not found at $SERVER; set LLAMA_BIN to the directory holding it"
[ -x "$BENCH" ] || die "llama-bench not found at $BENCH; set LLAMA_BIN to the directory holding it"
if curl -s --max-time 2 "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
  die "something is already answering on 127.0.0.1:$PORT; stop it or set PORT to a free port"
fi

BASE="${REPORT%.md}"
LOAD_LOG="$BASE.load.log"
BENCH_JSON="$BASE.bench.json"
BENCH_ERR="$BASE.bench.err"
VMSTAT_LOG="$BASE.vmstat"
BODY="$(mktemp)"
SWAP_BEFORE="$(mktemp)"
SWAP_AFTER="$(mktemp)"
SERVER_PID=""
VMSTAT_PID=""
cleanup() {
  if [ -n "$SERVER_PID" ]; then kill "$SERVER_PID" 2>/dev/null || true; fi
  if [ -n "$VMSTAT_PID" ]; then kill "$VMSTAT_PID" 2>/dev/null || true; fi
  rm -f "$BODY" "$SWAP_BEFORE" "$SWAP_AFTER"
}
trap cleanup EXIT
trap 'cleanup; exit 130' INT TERM

FENCE="$(printf '\140\140\140')"   # three backticks, kept out of the quoting
section() { printf '\n## %s\n\n' "$1" >> "$BODY"; }
block() { printf '%stext\n%s\n%s\n' "$FENCE" "${1:-(no output)}" "$FENCE" >> "$BODY"; }
note() { printf '%s\n' "$@" >> "$BODY"; }
run() { "$@" 2>&1 || echo "(command failed: $*)"; }
OS="$(uname -s)"

echo "==> Collecting evidence for $(basename "$MODEL") into $REPORT"

# --- Machine ---------------------------------------------------------------------------
section "Machine and operating system"
block "$(run uname -a)"
if [ -r /etc/os-release ]; then block "$(grep -E '^(PRETTY_NAME|VERSION_ID)=' /etc/os-release)"; fi
if [ "$OS" = "Darwin" ]; then block "$(run sw_vers)"; fi
WSL=0
if grep -qi microsoft /proc/version 2>/dev/null; then
  WSL=1
  note "Running inside WSL2. MemTotal below is the virtual machine's memory, set by memory= in" \
       "the [wsl2] section of %UserProfile%\\.wslconfig (default: half of Windows' memory)."
  block "$(grep -E '^(MemTotal|SwapTotal):' /proc/meminfo)"
fi

# --- Build and devices -------------------------------------------------------------------
section "Build, and the devices llama.cpp can see"
block "$(run "$SERVER" --version)"
DEVICES="$(run "$BENCH" --list-devices)"
block "$DEVICES"
note "\`(none)\` under Available devices means no GPU backend is compiled in or its runtime" \
     "did not load. The MiB free figure is what --fit plans with."

# --- Accelerator, as the operating system reports it -----------------------------------------
section "Accelerator memory, as the operating system reports it"
if have nvidia-smi; then
  block "$(run nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free,utilization.gpu --format=csv)"
  block "$(run nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv)"
fi
AMD_FOUND=0
for dir in /sys/class/drm/card*/device; do
  if [ -r "$dir/mem_info_gtt_total" ]; then
    AMD_FOUND=1
    block "$(cd "$dir" && grep -H . mem_info_vram_total mem_info_vram_used mem_info_gtt_total mem_info_gtt_used 2>&1)"
  fi
done
if [ "$AMD_FOUND" = 1 ] && [ -r /sys/module/ttm/parameters/pages_limit ]; then
  block "pages_limit: $(cat /sys/module/ttm/parameters/pages_limit)"
  note "amdgpu files are bytes; pages_limit x 4096 is the GTT ceiling Part 5 set."
fi
if [ "$OS" = "Darwin" ]; then
  block "$(printf 'iogpu.wired_limit_mb: %s\nhw.memsize: %s' "$(sysctl -n iogpu.wired_limit_mb 2>&1)" "$(sysctl -n hw.memsize 2>&1)")"
  note "A wired limit of 0 is the system default; Part 5's value does not survive a restart."
fi
if ! have nvidia-smi && [ "$AMD_FOUND" = 0 ] && [ "$OS" != "Darwin" ]; then
  note "No nvidia-smi, no amdgpu memory files and not macOS: nothing to report here."
fi

# --- Memory and swap, before ----------------------------------------------------------------
section "Memory and swap before the load"
if [ "$OS" = "Darwin" ]; then
  sysctl -n vm.swapusage > "$SWAP_BEFORE" 2>&1 || true
  block "vm.swapusage: $(cat "$SWAP_BEFORE")"
else
  free -m > "$SWAP_BEFORE" 2>&1 || true
  block "$(cat "$SWAP_BEFORE")"
  if have swapon; then block "$(swapon --show 2>&1 || true)"; note "An empty block above means no swap device is configured."; fi
fi

# --- The load probe -----------------------------------------------------------------------
SERVER_ARGS=(--model "$MODEL" --host 127.0.0.1 --port "$PORT" --verbose)
BENCH_EXTRA=(--flash-attn on)
ASKED="(no --ctx-size)"
if [ "$CTX" != "default" ]; then SERVER_ARGS+=(--ctx-size "$CTX"); ASKED="--ctx-size $CTX"; fi
if [ -n "${CACHE_TYPE:-}" ]; then
  SERVER_ARGS+=(--cache-type-k "$CACHE_TYPE" --cache-type-v "$CACHE_TYPE")
  BENCH_EXTRA+=(--cache-type-k "$CACHE_TYPE" --cache-type-v "$CACHE_TYPE")
  ASKED="$ASKED --cache-type-k/v $CACHE_TYPE"
fi
if [ -n "${NGL:-}" ]; then SERVER_ARGS+=(--n-gpu-layers "$NGL"); ASKED="$ASKED --n-gpu-layers $NGL"; fi
if [ -n "${DEVICE:-}" ]; then
  SERVER_ARGS+=(--device "$DEVICE"); BENCH_EXTRA+=(--device "$DEVICE"); ASKED="$ASKED --device $DEVICE"
fi
if [ -n "${FIT_TARGET:-}" ]; then SERVER_ARGS+=(--fit-target "$FIT_TARGET"); ASKED="$ASKED --fit-target $FIT_TARGET"; fi
if [ "${NO_KV_OFFLOAD:-}" = 1 ]; then
  SERVER_ARGS+=(--no-kv-offload); BENCH_EXTRA+=(--no-kv-offload 1); ASKED="$ASKED --no-kv-offload"
fi

echo "==> Loading with llama-server $ASKED (this can take a minute)"
"$SERVER" "${SERVER_ARGS[@]}" > "$LOAD_LOG" 2>&1 < /dev/null &
SERVER_PID=$!
STATUS="timeout"
waited=0
while [ "$waited" -lt "$LOAD_TIMEOUT" ]; do
  if grep -q "listening on" "$LOAD_LOG"; then STATUS="loaded"; break; fi
  if ! kill -0 "$SERVER_PID" 2>/dev/null; then STATUS="failed"; break; fi
  sleep 1
  waited=$((waited + 1))
done
kill -TERM "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
SERVER_PID=""

section "Model load: $STATUS (llama-server $ASKED)"
# The evidence lines; buffers of 0.00 MiB come from --fit's trial pass and are left out.
block "$(grep -E 'no usable GPU|using device|common_params_fit_impl: (projected|will leave|cannot meet|context size|user has requested|entire model|  - )|common_fit_params: (successfully|failed)|offloaded [0-9]+/[0-9]+ layers|model buffer size|KV buffer size|llama_kv_cache: size|llama_context: n_ctx  |recommendedMaxWorkingSetSize|greater than the recommended|cudaMalloc failed|failed to allocate|unable to allocate|error loading model|Device memory allocation|out of memory' "$LOAD_LOG" | grep -v ' D ' | grep -vE 'buffer size = +0\.00 MiB' | head -n 60 || true)"
# The breakdown printed last: the one for the load that actually happened.
block "$(grep 'common_memory_breakdown_print: |' "$LOAD_LOG" | sed 's/.*common_memory_breakdown_print: //' | awk '/memory breakdown \[MiB\]/ { n = 0 } { rows[n++] = $0 } END { for (i = 0; i < n; i++) print rows[i] }' || true)"
note "The breakdown is llama.cpp's own accounting, in MiB: per device, total = free + (self =" \
     "model + context + compute) + unaccounted; the Host row is system memory. Full log: $LOAD_LOG"

# --- The rates, with the placement the load chose ------------------------------------------
OFFLOADED="$(grep -oE 'offloaded [0-9]+/[0-9]+ layers' "$LOAD_LOG" | tail -n 1 | awk '{ split($2, a, "/"); print a[1] }' || true)"
# Weights in buffers whose names are not host buffers (CPU, CPU_Mapped, CPU_REPACK, *_Host).
DEVICE_WEIGHTS="$(grep -E 'model buffer size' "$LOAD_LOG" | grep -v ' D ' | grep -vE ' (CPU[A-Za-z_]*|[A-Za-z0-9]+_Host) +model buffer' | awk -F'= *' '{ s += $2 } END { printf "%.0f", s }' || true)"
if [ -z "$OFFLOADED" ] || [ "${DEVICE_WEIGHTS:-0}" = 0 ]; then BENCH_NGL=0; else BENCH_NGL="$OFFLOADED"; fi
BENCH_ARGS=(--model "$MODEL" --n-prompt "$PROMPT_LEN" --n-gen "$GEN_LEN" --repetitions "$REPS" --n-gpu-layers "$BENCH_NGL" --output json)
rm -f "$BENCH_JSON" "$BENCH_ERR" "$VMSTAT_LOG"

section "Rates measured with that placement"
BENCH_STATUS="not run"
if [ "$STATUS" = "loaded" ]; then
  echo "==> Measuring with llama-bench --n-gpu-layers $BENCH_NGL (repetitions: $REPS)"
  if [ "$OS" = "Linux" ] && have vmstat; then
    vmstat 1 > "$VMSTAT_LOG" 2>&1 &
    VMSTAT_PID=$!
  fi
  BENCH_STATUS=0
  "$BENCH" "${BENCH_ARGS[@]}" "${BENCH_EXTRA[@]}" > "$BENCH_JSON" 2> "$BENCH_ERR" || BENCH_STATUS=$?
  if [ -n "$VMSTAT_PID" ]; then kill "$VMSTAT_PID" 2>/dev/null || true; wait "$VMSTAT_PID" 2>/dev/null || true; VMSTAT_PID=""; fi
  block "llama-bench --n-prompt $PROMPT_LEN --n-gen $GEN_LEN --repetitions $REPS --n-gpu-layers $BENCH_NGL ${BENCH_EXTRA[*]} (exit status $BENCH_STATUS)"
  if [ "$BENCH_STATUS" != 0 ]; then block "$(tail -n 15 "$BENCH_ERR")"; fi
else
  echo "==> The model did not load ($STATUS); skipping the benchmark"
  note "Not run: the load ended with status $STATUS, and the load section above holds the reason."
fi
if [ "$OS" = "Darwin" ]; then sysctl -n vm.swapusage > "$SWAP_AFTER" 2>&1 || true; else free -m > "$SWAP_AFTER" 2>&1 || true; fi
section "Memory and swap during and after the benchmark"
if [ -s "$VMSTAT_LOG" ]; then
  block "$(awk 'NR <= 2 { print; next } { rows[n++] = $0 } END { for (i = (n > 6 ? n - 6 : 1); i < n; i++) print rows[i] }' "$VMSTAT_LOG")"
  note "si and so are pages swapped in and out per second; the first sample after the headers, an average since boot, is left out."
fi
block "$(cat "$SWAP_AFTER")"

# --- Signals: the lines to read first, and the notebook record --------------------------------
SIGNALS="$(python3 - "$LOAD_LOG" "$BENCH_JSON" "$VMSTAT_LOG" "$SWAP_BEFORE" "$SWAP_AFTER" "$DEVICES" \
  "$STATUS" "$ASKED" "$BENCH_NGL" "$MODEL" "$LABEL" "$LABBOOK" "$WSL" <<'PY'
import json, re, statistics, sys
from datetime import datetime, timezone
(load_log, bench_json, vmstat_log, swap_before, swap_after, devices, status, asked,
 bench_ngl, model, label, labbook, wsl) = sys.argv[1:14]
T95 = {2: 12.706, 3: 4.303, 4: 3.182, 5: 2.776, 6: 2.571, 7: 2.447, 8: 2.365, 9: 2.306, 10: 2.262}

def read(path):
    try:
        return open(path, encoding="utf-8", errors="replace").read()
    except OSError:
        return ""

log = "\n".join(l for l in read(load_log).splitlines() if " D " not in l)
last = lambda pattern: (re.findall(pattern, log) or [None])[-1]
dev_lines = re.findall(r"^\s+(\S+): .*\((\d+) MiB, (\d+) MiB free\)$", devices, re.M)
buffers = re.findall(r"load_tensors:\s+(\S+) model buffer size =\s+([\d.]+) MiB", log)
is_host = lambda name: name.startswith("CPU") or name.endswith("_Host")
dev_w = sum(float(v) for n, v in buffers if not is_host(n))
host_w = sum(float(v) for n, v in buffers if is_host(n))
kv_bufs = re.findall(r"llama_kv_cache:\s+(\S+) KV buffer size =\s+([\d.]+) MiB", log)
kv_where = ", ".join(f"{n} {float(v):.0f}" for n, v in kv_bufs if float(v) > 0) or "-"
fit = [m.split("common_params_fit_impl: ", 1)[1] for m in re.findall(
    r".*common_params_fit_impl: (?:will leave|cannot meet|context size|user has requested|  - ).*", log)]
fit_failed = last(r"common_fit_params: failed to fit params[^\n]*")
errors = re.findall(r".*(?:cudaMalloc failed|failed to allocate|unable to allocate|error loading model|"
                    r"Device memory allocation|greater than the recommended max working set).*", log)

def swap_used(text):
    m = re.search(r"^Swap:\s+\d+\s+(\d+)", text, re.M) or re.search(r"used = ([\d.]+)M", text)
    return float(m.group(1)) if m else None

si = so = None
rows = [l.split() for l in read(vmstat_log).splitlines()]
head = next((r for r in rows if "si" in r and "so" in r), None)
if head:
    data = [r for r in rows if len(r) == len(head) and r[0].isdigit()][1:]  # row 1 is since boot
    if data:
        si = max(int(r[head.index("si")]) for r in data)
        so = max(int(r[head.index("so")]) for r in data)

rates = {}
try:
    for row in json.load(open(bench_json, encoding="utf-8")):
        kind = "pp" if row["n_prompt"] and not row["n_gen"] else "tg"
        ts = [(row["n_prompt"] + row["n_gen"]) * 1e9 / ns for ns in row["samples_ns"]]
        sd = statistics.stdev(ts) if len(ts) > 1 else 0.0
        half = T95.get(len(ts), 2.0) * sd / len(ts) ** 0.5
        rates[kind] = {"test": f"{kind}{row['n_prompt'] or row['n_gen']}", "mean": round(row["avg_ts"], 2),
                       "sd": round(row["stddev_ts"], 2), "ci95": [round(row["avg_ts"] - half, 2),
                       round(row["avg_ts"] + half, 2)], "backends": row["backends"],
                       "devices": row["devices"], "ngl": row["n_gpu_layers"]}
except (OSError, ValueError, KeyError, TypeError):
    pass

offload = last(r"offloaded (\d+/\d+) layers")
out = [f"load status        {status}   (asked: {asked})",
       "devices            " + (", ".join(f"{n} {f} MiB free of {t}" for n, t, f in dev_lines) or "(none)"),
       "no-GPU warning     " + ("yes" if "no usable GPU found" in log else "no"),
       "offload line       " + (offload or "not printed"),
       f"weights, MiB       on devices {dev_w:.0f}, on host {host_w:.0f}",
       "context            " + (last(r"llama_context: n_ctx\s+= (\d+)") or "-"),
       "KV cache, MiB      " + (last(r"llama_kv_cache: size =\s+([\d.]+) MiB") or "-") + f"   (buffers: {kv_where})",
       "fit decision       " + ("; ".join(f.strip() for f in fit) if fit else "-")]
if fit_failed:
    out.append("fit failed         " + fit_failed.split(": ", 2)[-1])
for e in errors[:3]:
    out.append("load error         " + e.strip()[-110:])
b, a = swap_used(read(swap_before)), swap_used(read(swap_after))
out.append(f"swap used, MiB     before {b if b is not None else '-'}, after {a if a is not None else '-'}"
           + (f"; max si {si}, max so {so} per s during the benchmark" if si is not None else ""))
if wsl == "1":
    out.append("WSL2               " + (re.search(r"MemTotal:\s+\d+ kB", read("/proc/meminfo")) or [""])[0])
for kind in ("pp", "tg"):
    r = rates.get(kind)
    out.append(f"{kind} rate            " + (f"{r['test']} {r['mean']:.2f} ± {r['sd']:.2f} t/s, 95% interval "
               f"{r['ci95'][0]:.2f} to {r['ci95'][1]:.2f} (backend {r['backends']}, ngl {r['ngl']}, "
               f"dev {r['devices']})" if r else "not measured"))
print("\n".join(out))

if labbook:
    record = {"lab": "part-06/challenge-two-tokens-per-second", "label": label,
              "date": datetime.now(timezone.utc).isoformat(timespec="seconds"), "model_path": model,
              "asked": asked, "load_status": status, "offloaded": offload,
              "device_weights_mib": round(dev_w, 2), "host_weights_mib": round(host_w, 2),
              "kv_mib": float(last(r"llama_kv_cache: size =\s+([\d.]+) MiB") or 0) or None,
              "n_ctx": int(last(r"llama_context: n_ctx\s+= (\d+)") or 0) or None,
              "fit": fit, "fit_failed": fit_failed, "no_gpu_warning": "no usable GPU found" in log,
              "swap_used_mib_before": b, "swap_used_mib_after": a, "max_si": si, "max_so": so,
              "bench_ngl": int(bench_ngl), "rates": rates}
    with open(labbook, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
PY
)"

{
  printf '# Slow inference report: %s\n\n' "$LABEL"
  printf -- '- Collected: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf -- '- Model file: %s (%s bytes)\n' "$MODEL" "$(wc -c < "$MODEL" | tr -d ' ')"
  printf '\n## Signals\n\n%stext\n%s\n%s\n' "$FENCE" "$SIGNALS" "$FENCE"
  cat "$BODY"
} > "$REPORT"

printf '%s\n' "$SIGNALS"
echo "==> Wrote $REPORT${LABBOOK:+ and one part-06/challenge-two-tokens-per-second line to $LABBOOK}"

Download diagnose-slow-inference.sh349 lines

RunnableAll tracks

the healthy report

bash diagnose-slow-inference.sh "$MODEL" challenge/healthy.md labbook.md

Output — what you should see

==> Collecting evidence for Qwen3-8B-Q4_K_M.gguf into challenge/healthy.md
==> Loading with llama-server --ctx-size 4096 (this can take a minute)
==> Measuring with llama-bench --n-gpu-layers 37 (repetitions: 3)
load status        loaded   (asked: --ctx-size 4096)
devices            CUDA0 xxxxx MiB free of xxxxx
no-GPU warning     no
offload line       37/37
weights, MiB       on devices xxxx, on host xxx
context            4096
KV cache, MiB      576.00   (buffers: CUDA0 576)
fit decision       will leave xxxxx >= 1024 MiB of free device memory, no changes needed
swap used, MiB     before x.0, after x.0; max si 0, max so 0 per s during the benchmark
pp rate            pp512 xxxx.xx ± xx.xx t/s, 95% interval xxxx.xx to xxxx.xx (backend CUDA, ngl 37, dev auto)
tg rate            tg128 xx.xx ± x.xx t/s, 95% interval xx.xx to xx.xx (backend CUDA, ngl 37, dev auto)
==> Wrote challenge/healthy.md and one part-06/challenge-two-tokens-per-second line to labbook.md

The layout is the script’s own and was run on a CPU-only build; the llama.cpp lines it quotes are the pinned version’s format strings, so a GPU run has this shape but has not been captured for this page. What healthy looks like, line by line:

Signals line	Healthy	Why
devices	your device, with MiB free near the preflight figure	the backend loaded
offload line	37/37	every block and the output layer requested on the device
weights, MiB	on devices about 4,455, on host about 334 (the embedding table)	offload-split.py’s -ngl 37 row. On Track M the Metal buffer maps a range of the file, so its size can differ; there the offload line and the breakdown are the check
KV cache, MiB	576.00, in a buffer named after the device (CUDA0, Vulkan0, MTL0)	147,456 bytes × 4,096 cells
fit decision	will leave … >= 1024 MiB of free device memory, no changes needed	nothing was moved
swap used	unchanged; max so 0 on Linux	nothing paged
tg rate	interval overlapping task 1’s	the placement is the healthy one

Open challenge/healthy.md and read it end to end, including the memory breakdown under the load section: one device row with model near 4,455 and context 576, and a Host row with model near 334. Knowing a healthy report is what makes a broken one legible. If load status is not loaded, see Troubleshooting before anything else.

Record: the Signals block, and the breakdown’s device and Host rows. Every run below records the same lines, and its notebook entry holds them.

3. Break it five ways, one at a time

Each run changes exactly one llama.cpp setting through the script’s environment variables and leaves everything else as in task 2. Two of the settings simulate a smaller device: --fit-target is the MiB --fit keeps free, so a target close to FREE makes --fit plan as if the device had only the difference. That is how a GTT limit, a wired limit or a smaller card looks to llama.cpp, without changing anything on the machine. Set the three sizes once:

RunnableAll tracks

the simulated device sizes, from your FREE

SMALL=$((FREE - 3000))
ROOM=$(( FREE > 9024 ? FREE - 8000 : 1024 ))
LONG=$(( FREE > 9024 ? 32768 : 16384 ))
echo "SMALL=$SMALL ROOM=$ROOM LONG=$LONG"

Output — what you should see

SMALL=xxxxx ROOM=xxxxx LONG=32768

SMALL leaves --fit 3,000 MiB, less than the 4,455 MiB of weights. ROOM leaves 8,000 MiB, or on a device with 9,024 MiB free or less, the real device with the default margin; LONG is then 16,384, so that the row of the table above that does not fit still does not.

Run	Setting	What it simulates	The real-world cause
A	DEVICE=none	no accelerator in use	a CPU build, a missing runtime, a stray --device
B	NGL=18	layers not offloaded, by request	an old command line, a GUI’s default
C	FIT_TARGET=$SMALL	a device too small for the weights	a GTT or wired limit, another program on the card
D	CTX=default FIT_TARGET=$ROOM	the default context on a device without room for its cache	a plain llama-server -m
E	CTX=$LONG FIT_TARGET=$ROOM	a long context you set, on the same device	-c 32768 copied from a model card

RunnableAll tracks

run A: no accelerator in use

DEVICE=none bash diagnose-slow-inference.sh "$MODEL" challenge/a-no-device.md labbook.md

RunnableAll tracks

run B: 18 of 37 layers requested

NGL=18 bash diagnose-slow-inference.sh "$MODEL" challenge/b-ngl-18.md labbook.md

RunnableAll tracks

run C: a device too small for the weights

FIT_TARGET="$SMALL" bash diagnose-slow-inference.sh "$MODEL" challenge/c-small-device.md labbook.md

RunnableAll tracks

run D: the default context

CTX=default FIT_TARGET="$ROOM" bash diagnose-slow-inference.sh "$MODEL" challenge/d-default-context.md labbook.md

RunnableAll tracks

run E: a long context

CTX="$LONG" FIT_TARGET="$ROOM" bash diagnose-slow-inference.sh "$MODEL" challenge/e-long-context.md labbook.md

Each prints the same shape as task 2. Before reading them, write down your prediction for each column below; this table is what the v0.4.0 source and the arithmetic above predict, not a record of a run:

Signals line	A	B	C	D	E
offload line	37/37, untrue here (the trap above)	18/37	below 37/37, chosen by --fit	37/37	below 37/37
weights, MiB	devices 0, host about 4,789	devices about 2,368, host about 2,422	between	as healthy	between
context	4096	4096	4096	below 40960	LONG
KV cache, MiB	576.00, host buffer only	576.00: device 272, host 304	split	below 5,760.00	4,608.00 or 2,304.00, split
fit decision	projected to use … of host memory …	will leave … no changes needed	cannot meet …, context size set by user to 4096 -> no change, a - <device>: NN layers line	context size reduced from 40960 to …	context size set by user to …, a layers line
pp rate	falls furthest	falls	falls	as healthy	falls
tg rate	healthy ÷ r	healthy ÷ (0.531 + 0.469 × r)	per its own f	as healthy	per its own f

For C and E, read N from the offload line and run python3 offload-split.py "$MODEL" --ngl N --ctx-size 4096 --labbook labbook.md (with --ctx-size "$LONG" for E) to get their f and predicted slowdown. Run D is the instructive surprise: its rates match the healthy ones, because llama-bench sizes its own small context. The fault is capacity, visible only in context, and a server started that way cannot take a conversation longer than the reduced figure.

Record, for each of A to E: the Signals block, N where it applies, the predicted slowdown, and the measured one (healthy tg rate ÷ this run’s). If a run’s prediction and measurement differ by more than their intervals allow, say which input you think is wrong. If a run fails, see Troubleshooting.

4. Your platform’s own ceiling

Runs A to E simulate; this task reads the real limit your track imposes and, where the fault can be produced without touching the machine’s configuration, produces it.

Track S — NVIDIA DGX Spark

The ceiling. There is no operating-system cap: llama.cpp’s CUDA backend reads free memory for this integrated device from MemAvailable, and NVIDIA’s known-issues page says nvidia-smi shows “Memory-Usage: Not Supported” on this platform. Compare the two views:

RunnableTrack S · DGX Spark

the pool as Linux and as llama.cpp see it

free -m
"$LLAMA_BIN/llama-bench" --list-devices

Output — what you should see

               total        used        free      shared  buff/cache   available
Mem:          xxxxxx       xxxxx       xxxxx        xxx       xxxxx      xxxxxx
Swap:          xxxxx           x       xxxxx
Available devices:
  CUDA0: NVIDIA GB10 (xxxxxx MiB, xxxxxx MiB free)

Pass: CUDA0’s free figure is within a few hundred MiB of available (read at slightly different moments). Record both.

The fault: a CPU build. Build one on purpose, in its own directory. This is what the reader who downloaded the ubuntu-arm64 archive has, except that they do not know it.

RunnableTrack S · DGX Spark

a CPU-only build, in its own directory

cmake -S ~/llama.cpp -B ~/llama.cpp/build-cpu -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF
cmake --build ~/llama.cpp/build-cpu --config Release -j "$(nproc)"

The configure output should not mention CUDA. Then collect its evidence, with NGL=999 so that the warning has an option to warn about:

RunnableTrack S · DGX Spark

the CPU build's report

NGL=999 LLAMA_BIN=~/llama.cpp/build-cpu/bin bash diagnose-slow-inference.sh "$MODEL" challenge/spark-cpu-build.md labbook.md

Output — what you should see

load status        loaded   (asked: --ctx-size 4096 --n-gpu-layers 999)
devices            (none)
no-GPU warning     yes
offload line       not printed
weights, MiB       on devices 0, on host xxxx
...
pp rate            pp512 xxx.xx ± x.xx t/s, ... (backend CPU, ngl 0, dev auto)
tg rate            tg128 xx.xx ± x.xx t/s, ... (backend CPU, ngl 0, dev auto)

Record: the Signals block, and the two rates beside run A’s. Run A kept the CUDA build and selected no device; this build has no CUDA at all. The prediction is that they are close, and that both lose far more prefill than decode, because the twenty Arm cores read the same LPDDR5x.

Track X — AMD Ryzen AI Max+ 395Partial

The Vulkan against ROCm comparison needs the HIP build in ~/llama.cpp/build-hip from the install lesson.

The ceiling: GTT. Part 5’s GPU memory section and its preparation lab set the TTM page limit that sizes the GTT domain. Read the driver’s view beside llama.cpp’s:

RunnableTrack X · Ryzen AI Max+

VRAM and GTT from amdgpu, the TTM limit, and Vulkan0

grep -H . /sys/class/drm/card*/device/mem_info_vram_total /sys/class/drm/card*/device/mem_info_gtt_total
cat /sys/module/ttm/parameters/pages_limit
"$LLAMA_BIN/llama-bench" --list-devices

Output — what you should see

/sys/class/drm/card1/device/mem_info_vram_total:xxxxxxxxx
/sys/class/drm/card1/device/mem_info_gtt_total:xxxxxxxxxxx
xxxxxxxx
Available devices:
  Vulkan0: <device name> (xxxxxx MiB, xxxxxx MiB free)

The two sysfs files are bytes; divide by 1,048,576 for MiB. mem_info_gtt_total should equal pages_limit × 4096. For an integrated GPU, ggml_backend_vk_get_device_memory at v0.4.0 adds up every memory heap the Vulkan driver reports, and takes free memory from the driver’s budget where it offers one. Expect Vulkan0’s total to track VRAM plus GTT; the course has not checked that on hardware, so record all three. Decision rule: a GTT total near half the machine’s memory means the limit is at the kernel default, as after a fresh install, and every model near that size will behave like run C. The fix is Part 5’s amd-ttm --set, not a llama.cpp flag.

The wrong backend. Vulkan is this course’s default on this chip; the HIP build is the alternative the install lesson built in ~/llama.cpp/build-hip. Which is faster is a measurement. If the HIP build exists, run the task 1 command with each:

RunnableTrack X · Ryzen AI Max+

the same test on the HIP build, without and with unified memory

ls ~/llama.cpp/build-hip/bin/llama-bench
~/llama.cpp/build-hip/bin/llama-bench --list-devices
~/llama.cpp/build-hip/bin/llama-bench -m "$MODEL" -p 512 -n 128 -r 5 -ngl 999 -fa on
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ~/llama.cpp/build-hip/bin/llama-bench -m "$MODEL" -p 512 -n 128 -r 5 -ngl 999 -fa on

Output — what you should see

/home/you/llama.cpp/build-hip/bin/llama-bench
Available devices:
  ROCm0: <device name> (xxxxx MiB, xxxxx MiB free)
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | ROCm       | 999 |   1 |           pp512 |      xxxx.xx ± xx.xx |
| qwen3 8B Q4_K - Medium         |   4.68 GiB |     8.19 B | ROCm       | 999 |   1 |           tg128 |         xx.xx ± x.xx |
...

Two things in the source explain why the runs can differ. The CUDA code HIP shares takes free memory from MemAvailable for integrated devices only when not built for HIP, so ROCm0’s figures come from ROCm itself and can differ from Vulkan0’s. And the build guide says that on Linux GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 lets an integrated GPU share main memory, which “hurts performance for non-integrated GPUs (but enables working with integrated GPUs)”. Decision rule: the faster backend is the one whose tg128 95 per cent interval lies wholly above the other’s; overlapping intervals mean no difference on this model. Record: the three device lines, the three pairs of rates with their ±, and the build tag of each tree. No such file or directory from the first line means no HIP build: record that and skip this part.

Track M — Apple silicon

The ceiling: the wired limit. Metal’s recommendedMaxWorkingSetSize is MTL0’s total in --list-devices, and the verbose load log prints it. Part 5’s wired-memory limit raised it with iogpu.wired_limit_mb, which does not survive a restart. Compare the three views:

RunnableTrack M · Apple silicon

the wired limit now, the one Part 5 recorded, and MTL0

sysctl -n iogpu.wired_limit_mb
grep -o '"iogpu_wired_limit_mb": [0-9][0-9]*' labbook.md | tail -n 1
"$LLAMA_BIN/llama-bench" --list-devices
grep -E 'recommendedMaxWorkingSetSize|greater than the recommended' challenge/healthy.load.log
sysctl -n vm.swapusage

Output — what you should see

xxxxx
"iogpu_wired_limit_mb": xxxxx
Available devices:
  MTL0: <device name> (xxxxxx MiB, xxxxxx MiB free)
  BLAS: Accelerate (0 MiB, 0 MiB free)
x.xx.xxx.xxx I ggml_metal_device_init: recommendedMaxWorkingSetSize  = xxxxxx.xx MB
total = xxxx.xxM  used = xxx.xxM  free = xxxx.xxM  (encrypted)

The working-set line is in megabytes of 10⁶ bytes, MTL0 in MiB; the prefix before ggml_metal_device_init and the vm.swapusage layout may differ by version. Decision rule: a first line of 0 with a larger recorded value means a restart reset the limit, and every model between the default working set and your raised one will behave like run C, or, if you set -ngl yourself, allocate past the working set and page. Whether the sysctl moves recommendedMaxWorkingSetSize was not verified for this course: the MTL0 total before and after is the check. A greater than the recommended max working set size line in any report is the Metal backend’s own warning that it allocated past that size.

⚠Warning

The block below sets the system-wide limit on memory macOS pins for the GPU back to the figure your notebook holds from Part 5, and changes nothing else. Quit large applications before running it; to undo it, sudo sysctl iogpu.wired_limit_mb=0 restores the default.

RunnableTrack M · Apple silicon

re-apply the recorded wired limit, then list the devices

WIRED_MB="$(grep -o '"iogpu_wired_limit_mb": [0-9][0-9]*' labbook.md | tail -n 1 | awk '{ print $2 }')"
if [ -n "$WIRED_MB" ]; then sudo sysctl iogpu.wired_limit_mb="$WIRED_MB"; else echo "no Part 5 value recorded: leave the limit alone"; fi
"$LLAMA_BIN/llama-bench" --list-devices

Swap, as it looks on this machine. Open Activity Monitor, choose the Memory tab and keep the Memory Pressure graph and Swap Used in view while you re-run run E. Record Swap Used before and after. A graph that turns from green while generation slows, or Swap Used rising by more than the few megabytes of normal activity, is the spill of the table above; stop the run with Ctrl+C.

Record: the three limit figures, whether you re-applied the limit, and Swap Used before and after.

Track N — NVIDIA desktop or laptop

The ceiling: VRAM, and inside WSL2 the virtual machine. The card’s memory is fixed; the part that moves is the system memory the host side of a load needs: the 334 MiB embedding table even at a full offload, every block --fit or -ngl leaves behind, the cache with --no-kv-offload, the page cache for the memory-mapped file, and llama-server’s prompt cache. Microsoft documents the WSL2 virtual machine’s memory as defaulting to “50% of total memory on Windows” and swap to “25% of memory size on Windows rounded up to the nearest GB”.

RunnableTrack N · NVIDIA GPU

the card, what else holds it, and the memory Linux sees

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
free -m

Output — what you should see

name, memory.total [MiB], memory.used [MiB]
NVIDIA GeForce RTX xxxx, xxxxx MiB, xxx MiB
pid, process_name, used_memory [MiB]
               total        used        free      shared  buff/cache   available
Mem:           xxxxx        xxxx       xxxxx          x        xxxx       xxxxx
Swap:          xxxxx           0       xxxxx

An empty process list with memory.used above a few hundred MiB means the desktop holds part of the card; it comes off every budget. Inside WSL2, compare Mem: total with the memory Task Manager’s Performance tab shows on Windows. Decision rule: about half means the default is in force; if run B’s or run C’s host weights plus the reports’ Host rows approach available, raise the limit. The course’s rule from Part 4 is to leave Windows a 10 GB reserve, so on a 64 GB machine memory=54GB. On native Linux there is no such cap, and the matching risk is GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 set in your shell profile, which lets an over-large load succeed and page across PCIe instead of failing: echo "${GGML_CUDA_ENABLE_UNIFIED_MEMORY:-unset}" should print unset.

⚠Warning

The next steps change how much of Windows’ memory WSL2 may take and restart every running WSL2 distribution, stopping anything running inside them. Save your work in every WSL2 terminal first.

On Windows, open the file in PowerShell (it is created if absent):

RunnableTrack N · Windows

Windows: open the global WSL configuration

notepad "$env:USERPROFILE\.wslconfig"

Put your figure under the [wsl2] section, save, and close Notepad:

Fragment — not complete on its own

%UserProfile%.wslconfig (your total minus 10 GB)

[wsl2]
memory=54GB

Then apply it:

RunnableTrack N · Windows

Windows: stop WSL2 so the setting applies

wsl --shutdown

Open a new WSL2 terminal, run free -m again, and check that total moved. Microsoft notes the change needs the virtual machine to stop completely, which wsl --shutdown does at once. Record: memory.total, memory.used at idle, the processes listed, WSL2’s total before and after, and the memory= value, or “native Linux”.

5. Diagnose from the reports, not from memory

Put challenge/healthy.md beside each broken report and read the Signals lines in this order. The first row that matches names the fault; the rows below it are only checked once it is fixed.

Order	Signals line	Reads	Fault	First fix
1	load status	failed, with a load error line	it does not fit as asked	let --fit decide, or fewer bytes (row 6)
2	devices, no-GPU warning	(none), or yes	CPU build or runtime not loaded	rebuild with the backend; the install lesson’s check
3	weights, MiB	on devices 0 while devices lists one	a device present but unused	remove --device; check dev in the rate lines
4	swap used, max so	rose, or so above 0	memory spilling to swap	fewer bytes; close programs; the platform ceiling
5	offload line with no cannot meet in fit decision	N below M	layers not offloaded, by request	raise -ngl, or leave it unset
6	fit decision	context size reduced, or set by user then a layers line	the KV cache took the room	a shorter context, q8_0 cache, a smaller file
7	none of the above, rate still low	everything healthy	not on this list	Troubleshooting’s last rows

Run A stops at row 3 (or row 2 for the Spark CPU build), B at row 5, C at row 6 with a layers line but no context change, D at row 6 with a context change and healthy rates, E at row 6 with both. For each broken report, write one diagnosis in the notebook: the fault, the Signals line that named it, the predicted and measured slowdown, and the single change you would make.

✕Common mistake

Reading the rate first and theorising from it. A generation rate tells you that something is wrong and nothing about what. People who start there change the quantisation, then the context, then the batch size, and find the --device none in a shell alias an hour later. The Signals block is eleven lines and answers the question directly.

6. Fix exactly one thing, and prove it

In runs A to E the fix is to drop the setting you added. The real work is the change you would make when the same evidence appears uninvited:

Evidence	The one change	Command form
devices (none)	the build with the backend compiled in	the install lesson’s cmake for your track
dev none in the rate lines	remove the device selection	delete --device none, or LLAMA_ARG_DEVICE from the environment
offload line below M, -ngl given	ask for every layer, or none at all	-ngl 999, or no -ngl
context size reduced and you need the context	halve the cache, then check again	--cache-type-k q8_0 --cache-type-v q8_0
set by user and a layers line	shorter context first, then a quantised cache	-c 16384, then the cache types above
swap rising	a smaller file or context; close programs	the next quantisation down, from the GGUF lesson
a platform ceiling	raise it where it lives	Part 5’s amd-ttm, the wired-limit block, .wslconfig

Prove one properly, on run E, whose fault needs a real change. Predict first: at the same LONG, a q8_0 cache is 2,448.00 MiB against 4,608.00, and the table in the mechanism section gives room for all 37 layers if the compute buffer fits in what is left (1,097 MiB at 32,768; on the 16,384 row of an 8 GB card, the cache drops from 2,304.00 to 1,224.00 MiB):

RunnableAll tracks

run E again, with the one change

CACHE_TYPE=q8_0 CTX="$LONG" FIT_TARGET="$ROOM" bash diagnose-slow-inference.sh "$MODEL" challenge/e-long-context-q8.md labbook.md

Output — what you should see

load status        loaded   (asked: --ctx-size 32768 --fit-target xxxxx --cache-type-k/v q8_0)
...
offload line       37/37
context            32768
KV cache, MiB      2448.00   (buffers: CUDA0 2448)
fit decision       will leave xxxx >= xxxxx MiB of free device memory, no changes needed
...
tg rate            tg128 xx.xx ± x.xx t/s, 95% interval xx.xx to xx.xx (backend CUDA, ngl 37, dev auto)

The proof has three parts, and all three are needed. The evidence changed: the offload line is back to 37/37 and the fit decision no longer moves layers. The rate recovered: the tg128 interval overlaps task 2’s. And the prediction held: the cache is the size the formula gave. A rate that recovered without the evidence changing means something else moved, usually a quieter machine, and you have not finished. If the offload line is still below 37, the compute buffer took the margin: read compute from the breakdown in challenge/e-long-context-q8.md, redo the sum, and write down which term you had wrong. A quantised cache stores rounded keys and values, so it can cost output quality; that cost is a measurement on your task, not an assumption.

Pending validationOne model, five faults, one fix — your recording sheet

State	Offload line	Weights on device, MiB	KV cache, MiB	pp512 tokens/s	tg128 tokens/s
Healthy (task 2)	—	—	576.00	—	—
A: --device none	—	—	576.00	—	—
B: -ngl 18	18/37	—	576.00	—	—
C: a small device	—	—	—	—	—
D: default context	—	—	—	—	—
E: long context	—	—	—	—	—
E fixed: q8_0 cache	—	—	—	—	—
Your track (task 4)	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp the build number and commit from llama-server --version, for each row · Qwen3-8B, Q4_K_M weights; KV cache type as listed · 32,768 tokens of context · the date you ran it

Empty on purpose. The rates come from each report's Signals block (three repetitions); the healthy row can take task 1's five-repetition rates instead. On the Track S row the build differs from every other row, and that is the point of recording the build per row.

7. Look for the fault you did not introduce

With the procedure fresh, run the script once the way you actually use llama.cpp. MINE is the model you use most; the value below is this page’s model, so edit it before running if yours differs. CTX=default makes the load behave like a plain llama-server -m:

RunnableAll tracks

a routine check on your real setup

MINE=~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
CTX=default bash diagnose-slow-inference.sh "$MINE" challenge/routine.md labbook.md

This is where people find that they have been running with a partial offload for weeks, or that a context they never reach has been taking a third of their memory. Read context and KV cache as carefully as the offload line. Record: the Signals block and one sentence: nothing found, or which row of task 5’s table matched.

Make the diagnosis falsifiable

Save the healthy startup log and benchmark before introducing a fault. Change exactly one setting, write the predicted symptom without looking at the diagnostic summary, then collect the same evidence as the baseline. Keep fault logs under distinct names so the next fault cannot overwrite them.

For each diagnosis, identify the observation that rules out a competing cause. CPU placement in the startup log supports an offload explanation; a long prompt with normal later token cadence points elsewhere. Low throughput alone cannot identify the layer at fault.

Restore the setting and rerun the same workload. The confirmation is a return toward the healthy behaviour under equivalent conditions, not merely that a command exits successfully. If recovery does not occur, record the unresolved difference and inspect background load, thermal state and which binary actually ran. Finish with one short incident note containing symptom, evidence, cause, fix and verification. That note is the reusable output of the challenge; remembering which switch you deliberately broke is not a diagnostic method.

Validation

Run these from ~/llm-course. Each row says what passing prints.

Check	Command	Pass
Every report exists	ls challenge/*.md	healthy, a-no-device, b-ngl-18, c-small-device, d-default-context, e-long-context, e-long-context-q8, routine, and spark-cpu-build on Track S
Every run is in the notebook	grep -c '"lab": "part-06/challenge-two-tokens-per-second"' labbook.md	at least 8 (at least 9 on Track S)
The healthy load was whole	grep 'offload line' challenge/healthy.md	offload line 37/37
Run B shows the request	grep 'offload line' challenge/b-ngl-18.md	offload line 18/37
Run D shows the shrink	grep -c 'context size reduced' challenge/d-default-context.md	1 or more
The fix moved the evidence	grep 'offload line' challenge/e-long-context-q8.md	offload line 37/37
The healthy run did not swap	grep 'swap used' challenge/healthy.md	max so 0 on Linux; before and after equal on a Mac

Beyond the commands: one written diagnosis per broken report, with predicted and measured slowdowns; the recording sheet filled; and the task 4 figures for your track.

Expected outcome

A procedure you can run from memory, and a calibrated sense of what each fault costs on your machine, written in your own numbers rather than anyone’s adjectives. Some shapes are predictable enough to check against. Run A should lose more prefill than decode everywhere, and its decode loss should sit near the r your Part 5 line gives. Run B’s decode loss should sit near 0.531 + 0.469 × r, which is small on the unified-memory tracks and large on a card. Run D should match the healthy rates while its context is smaller. And run E’s fix should restore both the offload line and the rate.

=Expected result

After the fix, the tg128 interval overlaps the healthy one and the evidence line that named the fault has changed. If the rate recovered only partly, you have two faults; go back to task 5’s table and treat the remaining gap as a new investigation rather than as an acceptable loss.

Troubleshooting

Symptom	Cause	Fix
diagnose-slow-inference: something is already answering on 127.0.0.1:8089	a server from another page is still up	stop it with Ctrl+C in its terminal, or put PORT=8090 in front of the command
diagnose-slow-inference: llama-server not found at …	LLAMA_BIN is unset in this terminal	run the preflight’s first block again
load status timeout	the load took longer than 600 seconds, usually a slow disk	put LOAD_TIMEOUT=1800 in front of the command
An older copy of the script stops after Loading model... with a > prompt	the copy before 2026-09-13 loaded the model with llama-cli, which at v0.4.0 is a chat interface waiting for input	press Ctrl+C and save the current file from task 2
FREE= empty in the preflight	--list-devices printed (none)	a CPU-only build: the install lesson’s backend check
load error with cudaMalloc failed: out of memory, failed to allocate, unable to allocate or Device memory allocation of size	the load as asked does not fit, often with fit failed … n_gpu_layers already set by user above it	leave NGL unset so --fit can place layers, or use fewer bytes
Run C or E shows 37/37	the compute buffer was smaller than assumed, or FREE came from a different line	echo "$SMALL $ROOM $LONG"; re-run the preflight’s device block; for C try SMALL=$((FREE - 2000))
pp rate not measured with load status loaded	llama-bench failed after the server loaded	read challenge/<name>.bench.err; a failed to load model there means the benchmark’s placement did not fit
Track M: greater than the recommended max working set size in a report	Metal allocated past the working set	re-apply the wired limit (task 4), or use fewer bytes
Track M: the machine became unresponsive	macOS is paging	stop the process; if the desktop is too slow to do that, wait: it recovers once the process ends
Track N, WSL2: free -m shows about half the machine’s memory	the default memory of the WSL2 virtual machine	task 4’s .wslconfig steps
Track X: the two backends give very different rates	a result, not a fault	record both with their build tags and use the faster one on non-overlapping intervals
The rate is low and every Signals line matches healthy	thermal limiting, another program on the accelerator, or a wrong expectation	a rate that starts fine and degrades over a longer run is heat; memory.used by another process shows in nvidia-smi; check the Part 5 prediction for this file rather than the one you remember

Cleanup

Keep challenge/ and the notebook lines: the reports are the evidence your diagnoses cite. Make sure nothing is listening on the probe port:

RunnableAll tracks

no server left behind

curl -s --max-time 2 http://127.0.0.1:8089/health || echo "port 8089 is free"

Output — what you should see

port 8089 is free

The variables FREE, SMALL, ROOM, LONG and MINE end with the terminal; the settings of runs A to E were given per command and did not persist. Leave Track M’s wired limit and Track N’s .wslconfig at the values task 4 set; later parts assume them. On Track S, delete the deliberately broken build:

⚠Warning

This removes the CPU-only build tree and nothing else. Check the path before you run it: a mistyped build-cpu deletes the CUDA build every later page uses, and getting it back means compiling again.

RunnableTrack S · DGX Spark

remove the CPU-only build tree

ls -d ~/llama.cpp/build-cpu
rm -rf ~/llama.cpp/build-cpu

What you learned

Objective	The observation that proved it	Recorded as
Evidence before theory	the first matching row of task 5’s table named each fault before any rate was compared	one diagnosis per broken report
The placement lines are the highest-value output	offload line, weights, KV cache and fit decision differed between healthy and every broken report	the Signals blocks in challenge/ and the notebook lines
A default can be the fault	run D’s context shrank with healthy rates; run E’s layers moved with nothing failing	runs D and E’s context, fit decision and offload lines
Slowdowns are arithmetic	run B’s measured slowdown beside 0.531 + 0.469 × r, and C’s and E’s beside offload-split.py	predicted and measured slowdown per run
Some ceilings are outside llama.cpp	the GTT, wired-limit or WSL2 figure beside the device total llama.cpp reported	task 4’s figures
A fix proves itself twice	run E with a q8_0 cache: offload line back to 37/37 and the rate interval back over the healthy one	the e-long-context-q8 report and the recording sheet

Check your understanding

Question 1. A launch script runs llama-server -m Qwen3-8B-Q4_K_M.gguf -c 32768 on an 8 GB card and generation is slow, but llama-bench -m Qwen3-8B-Q4_K_M.gguf -ngl 999 on the same card is fast. Which explanation fits v0.4.0's defaults?

• llama-bench uses a faster kernel than llama-server
• The 4,608 MiB cache for the context the script set does not fit beside the weights, so --fit, on by default, left the context alone and moved layers to system memory; llama-bench sizes its context to the test and places every layer
• llama-server does not use the GPU unless --device is given
• The card is thermally limited only during serving

Show the answer and why

Answer: The 4,608 MiB cache for the context the script set does not fit beside the weights, so --fit, on by default, left the context alone and moved layers to system memory; llama-bench sizes its context to the test and places every layer

The two tools have different defaults. With -c given and -ngl unset, the server's --fit keeps the context and moves layers; with no -c it would have shrunk the context first. A --verbose log shows which, in the offload line and the fit decisions. llama-bench has no --fit unless -fitt is given, so it either places every layer or fails.

Question 2. offload-split.py reports that -ngl 18 leaves 46.9 per cent of Qwen3-8B's bytes read per token on the host. Your Part 5 line gives an accelerator-to-CPU read ratio of 10. What decode rate should you predict, relative to the full offload?

• About 49 per cent of the healthy rate, because 18 of 37 layers are on the device
• About 19 per cent: the slowdown is 0.531 + 0.469 × 10 = 5.22
• About 10 per cent, the CPU rate
• Unchanged, because decode is bandwidth-bound

Show the answer and why

Answer: About 19 per cent: the slowdown is 0.531 + 0.469 × 10 = 5.22

Time per token is a sum of the two paths, so the slowdown is (1 − f) + f × r, and 1 ÷ 5.22 is about 0.19. Counting layers instead of bytes gets both f and the shape wrong: the slow path dominates as soon as f × r outgrows 1 − f.

Question 3. Which of these evidence-collection commands is the bug at llama.cpp v0.4.0?

• llama-server -m model.gguf --verbose --host 127.0.0.1 --port 8089, stopped once it logs listening on
• llama-cli -m model.gguf -p "hi" -n 1, captured with 2>&1 into a file
• llama-bench --list-devices
• llama-bench -m model.gguf -p 512 -n 128 -r 3 -ngl 37 -o json

Show the answer and why

Answer: llama-cli -m model.gguf -p "hi" -n 1, captured with 2>&1 into a file

llama-cli is a chat interface at v0.4.0: after answering it waits for the next turn, so a script hangs, or with input from /dev/null loops on empty prompts. It would also hide the placement lines, which are logged at verbosity 4 and need --verbose.

Question 4. A GPU build is started with --device none and -ngl unset. Which evidence is trustworthy?

• The line offloaded 37/37 layers to GPU
• The model buffer size lines and the memory breakdown, which show every weight in a host buffer
• The backend column of llama-bench, which reads CPU
• The absence of any warning

Show the answer and why

Answer: The model buffer size lines and the memory breakdown, which show every weight in a host buffer

The offload line is computed from the requested layer count whenever a GPU backend is registered, so it can claim a full offload with no device selected. The buffer lines come from real allocations. llama-bench's backend column still names the compiled-in backend; its dev column reading none is the tell.

Question 5. Qwen3-8B at 32,768 tokens with an f16 cache does not fit next to its weights, so you switch both cache types to q8_0. By how much does the cache shrink, and what else must you check before calling it fixed?

• By half, to 2,304 MiB; nothing else
• From 4,608.00 to 2,448.00 MiB, since q8_0 stores 32 values in 34 bytes; then check the offload line is back to 37/37 and the tg128 interval overlaps the healthy one
• To zero, because a quantised cache lives in system memory
• From 5,760 to 2,880 MiB; then check the quantisation did not change the weights

Show the answer and why

Answer: From 4,608.00 to 2,448.00 MiB, since q8_0 stores 32 values in 34 bytes; then check the offload line is back to 37/37 and the tg128 interval overlaps the healthy one

4,608 × 34 ÷ 64 = 2,448, slightly more than half because of the per-block scale. A fix proves itself twice: the evidence that named the fault changes, and the rate recovers. The cache type also has a quality cost that has to be measured, not assumed.

Question 6. Inside WSL2 on a 64 GB Windows machine, free -m reports a total of about 32 GB. What is going on, and what is the course's change?

• Windows reserves half its memory for the GPU; nothing can be done
• The WSL2 virtual machine defaults to half of Windows' memory; set memory=54GB under [wsl2] in %UserProfile%\.wslconfig, keeping Part 4's 10 GB reserve, and apply it with wsl --shutdown
• Linux cannot address more than half of the memory
• The model already occupies the other half

Show the answer and why

Answer: The WSL2 virtual machine defaults to half of Windows' memory; set memory=54GB under [wsl2] in %UserProfile%\.wslconfig, keeping Part 4's 10 GB reserve, and apply it with wsl --shutdown

Microsoft documents memory as defaulting to 50 per cent of total memory on Windows and swap to 25 per cent of that. The card's VRAM is unaffected, but everything the host side of a load needs, from the embedding table to layers left behind, must fit in the virtual machine, and nothing in llama.cpp's log mentions Windows.

Check my answersClear