Lab: Run and Benchmark the Course Reference Models

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The build tag, backend and versions each track was run with will be recorded here when the validation pass is done.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this lab you will have a table of numbers that describe your machine: for every reference model that fits it, how fast it reads a prompt, how fast it writes an answer, what the file costs in disk and memory, and how much of your memory a long context takes. You will have produced those numbers with one stated methodology, so that the models can be compared with each other, and recorded them with enough context that a measurement you take in Part 7, Part 9 or the capstone can be compared with them.

You will also set every generation rate beside the ceiling Part 5’s bandwidth arithmetic allows for that exact file, and explain the gap: a measurement next to a prediction is a diagnosis.

⌂Single-machine path

This is a single-machine lab on every track. If you own a Spark pair or several Macs, run it on one machine and record which one; the cluster work starts in Part 18.

What runs where

Everything happens on one machine, in ~/llm-course, the directory Part 1 created. Four scripts from this page do the work, and each writes JSON lines to the same labbook.md, so the last step can read everything back.

From a verified file to a row in the course benchmark format

1. 1fetch-model.sh (Part 4)Downloads one GGUF file into ~/models/<publisher>/<repository>/, checks its size and SHA-256 against the Hub, writes a .sha256 beside it and a part-04/fetch-model line to the notebook.
2. 2bench-reference-models.shRuns llama-bench once per file with the lab methodology; keeps the raw JSON and the error log in bench-results/.
3. 3bench-to-labbook.pyTurns each JSON result into one part-06/lab-benchmark-the-reference-models line per test: build, backend, file, quantisation, settings, mean and standard deviation. A CPU-backend result is refused.
4. 4kv-memory-ladder.shLoads one model in llama-server at four context settings and records the KV cache and device memory llama.cpp reports, beside the formula.
5. 5summarise-benchmarks.pyReads the notebook and models.json, computes each file's decode ceiling from your Part 5 bandwidth, and prints the comparison, or the same rows as a <Benchmark> block.

Requirements

Every track needs: a llama.cpp build from the install lesson that passed its backend check, pinned here at llama.cpp v0.4.0 · verified 2026-09-08; Part 4’s fetch-model.sh and verify-library.sh in ~/llm-course with the hf command-line tool they call (Hugging Face CLI 1.30.0 · verified 2026-09-08 or later); the notebook ~/llm-course/labbook.md; Part 5’s measurement lab done, so the notebook holds a part-05/bandwidth-test line and a part-05/predict-decode line; and the copy of the course’s models.json that predict-decode.py read. The scripts need only python3 3.9 or later with its standard library, curl and awk.

Which models, for your memory

A model belongs in your set when its file fits in the memory your accelerator reports, with a margin. The number to compare against is the MiB free figure llama-bench --list-devices prints in the preflight below, not the memory on the box; each track’s tab says where that figure comes from. The margin is 1,024 MiB, the default --fit-target llama.cpp v0.4.0 keeps free per device when it sizes a load itself, and it covers the compute buffers and the KV cache of a 512-token test.

Sizes are the byte counts in each repository’s file listing on 2026-09-13. They differ from the rounded figures in the model reference for gpt-oss-20b and the 235B row; the listing is what you download.

Tier	Add to the set	Repository and file	File size, GB (10⁹ bytes)	MiB	MiB free needed
8 GB	Qwen3-8B at Q4_K_M	unsloth/Qwen3-8B-GGUF, Qwen3-8B-Q4_K_M.gguf	5.03	4,795	5,819
12 GB	Qwen3-14B at Q4_K_M	unsloth/Qwen3-14B-GGUF, Qwen3-14B-Q4_K_M.gguf	9.00	8,585	9,609
16 GB	gpt-oss-20b at MXFP4	ggml-org/gpt-oss-20b-GGUF, gpt-oss-20b-MXFP4.gguf	12.11	11,549	12,573
24 and 32 GB	Qwen3-30B-A3B at Q4_K_M	unsloth/Qwen3-30B-A3B-GGUF, Qwen3-30B-A3B-Q4_K_M.gguf	18.56	17,697	18,721
24 and 32 GB	Qwen3-32B at Q4_K_M	unsloth/Qwen3-32B-GGUF, Qwen3-32B-Q4_K_M.gguf	19.76	18,847	19,871
48 to 64 GB	Qwen3-30B-A3B at Q8_0	unsloth/Qwen3-30B-A3B-GGUF, Qwen3-30B-A3B-Q8_0.gguf	32.48	30,979	32,003
96 GB	gpt-oss-120b at MXFP4	ggml-org/gpt-oss-120b-GGUF, gpt-oss-120b-MXFP4.gguf	63.39	60,451	61,475
128 GB on Track S; 256 GB and above on Track M	Qwen3-235B-A22B at IQ4_XS	unsloth/Qwen3-235B-A22B-GGUF, three shards in IQ4_XS/	125.50	119,691	120,715

Tasks 6, 7 and 9 use one model throughout, set once in the preflight as QM: Qwen3-8B on 12 GB and above, and Qwen3-4B on the 8 GB tier, because Qwen3-8B at Q8_0 needs 9,330 MiB free and does not fit an 8 GB card. Every model on this page is published under the Apache-2.0 licence and none is gated; the model reference records the licence for each. The GGUF files come from the repository models.json records under sources.gguf: unsloth for every Qwen3 size, because that one namespace ships every tier the course uses including IQ4_XS for the 235B, and ggml-org, the llama.cpp organisation, for gpt-oss. Qwen publishes its own conversions too; the course standardises on one namespace so file names, sizes and checksums in every later part refer to the same bytes.

Downloads, disk and time

Tier	Everything this lab downloads	Hours at 100 Mbit/s	Hours at 1 Gbit/s
8 GB	17.4 GB	0.4	0.04
12 GB	34.0 GB	0.8	0.08
16 GB	46.2 GB	1.0	0.10
24 and 32 GB	84.5 GB	1.9	0.19
48 to 64 GB	117.0 GB	2.6	0.26
96 GB, and a 128 GB Mac	180.3 GB	4.0	0.40
128 GB on Track S; 256 GB and above on Track M	305.9 GB	6.8	0.68

The totals include the three extra quantisations of task 6. The hours are arithmetic, bytes times eight divided by the link rate, not a measurement of the Hub; add the SHA-256 check, which reads each file once more. Free disk must exceed the total, and fetch-model.sh refuses to start a file the disk cannot hold. The manifest allows sixty attended minutes; that figure excludes downloads and sweeps, and the real one will be recorded at validation. Unattended time is the downloads plus the sweep, which for one model is roughly this, in seconds, with the rates from its own table:

Pseudocode — not a real command

sweep seconds per model ≈ load time + 6 × 512 / pp512 rate + (5 × 128 + 1) / tg128 rate
    (prompt test: one untimed warm-up of the whole prompt plus five timed repetitions;
     generation test: a one-token warm-up plus five timed repetitions of 128 tokens)

Run the preflight, then start task 2’s download blocks before reading further.

Track S — NVIDIA DGX Spark

128 GB of unified memory, all of it addressable by the GPU. On the Spark, llama.cpp v0.4.0 does not use cudaMemGetInfo for the free figure. For an integrated CUDA device on Linux it reads MemAvailable from /proc/meminfo (ggml_backend_cuda_device_get_memory in ggml-cuda.cu), so MiB free is the available column of free -m, and every other program running lowers it. NVIDIA’s known-issues page gives the reason the CUDA figure is not used: “the memory size reported by cudaMemGetInfo may be smaller than the actual allocatable memory”. The same page says nvidia-smi “will display ‘Memory-Usage: Not Supported’” on this platform, so the operating-system cross-check in task 7 uses free -m. The backend column must read CUDA. The scripts run on the DGX OS host with its own python3, not inside the Part 1 container, because llama.cpp was built on the host. Run the whole preflight first. If the CUDA0 line shows at least 120,715 MiB free, start the 235B download in a second terminal (after running the preflight’s first block there); otherwise stop at gpt-oss-120b.

Track X — AMD Ryzen AI Max+ 395Partial

The GPU-visible share of memory is capped below the machine total, so the 235B row does not apply, and on a 64 GB machine the gpt-oss-120b row does not either.

64 GB or 128 GB of unified memory, of which the GPU may address the amount the firmware and the amdgpu driver allow, as Part 5’s lesson on this machine explains. llama.cpp’s Vulkan backend sums every memory heap the driver exposes to an integrated GPU, so use the Vulkan0 line. The backend column reads Vulkan, or ROCm for the HIP build in ~/llama.cpp/build-hip. Every task uses the Vulkan build in ~/llama.cpp/build; if you also built HIP, task 5 has the two extra sweeps for it, and each notebook line records the backend, so the builds never mix. The scripts on this page start llama.cpp themselves, so they need Linux on this machine; a native Windows build can take only task 3’s single-model measurement.

Track M — Apple siliconPartial

Task 2's downloads need Part 4's fetch-model.sh corrected for macOS's bash 3.2 (checked 2026-09-13), and the 235B row needs a Mac with 256 GB or more.

Use the MTL0 line of --list-devices. Its total is Metal’s recommendedMaxWorkingSetSize (ggml_metal_device_get_memory in ggml-metal-device.m) and its free figure is that total less what Metal has already allocated; whether raising iogpu.wired_limit_mb moves the total was not verified for this course. Part 5’s wired-memory limit does not survive a restart, so the preflight checks it.

Task 2 needs the corrected fetch-model.sh from Part 4: the copy dated 2026-09-13 expands an empty array under set -u, which macOS’s /bin/bash 3.2 reports as an unbound variable, so every download stops before it starts.

The 235B row needs 120,715 MiB free. On a 128 GB Mac that is more than Part 5’s course limit of 115 GiB (117,760 MiB, from gpu-limit.py), so the course stops the set there at gpt-oss-120b rather than pin more memory for the GPU than Part 5 allows; on 256 GB and 512 GB Macs the course limits are 230 and 460 GiB. The backend column reads MTL,BLAS, because a macOS build compiles Apple’s BLAS in by default, and the markdown table shows a threads column instead of ngl (task 3 explains why). Close the browser and anything else large first: they share the pool you are measuring.

Track N — NVIDIA desktop or laptop

Memory here is VRAM and it is a hard ceiling; the 48 to 64 GB row needs a 48 GB or larger card. The backend column reads CUDA. The primary path is Linux or WSL2, where every command on this page runs as written; inside WSL2 the NVIDIA driver is the Windows one, as Part 1 set up. A native Windows build can take only task 3’s single-model measurement, because the sweep scripts need a POSIX shell. The build guide notes that on Windows the driver setting “System Memory Fallback” lets an over-large load spill into system RAM instead of failing, so on native Windows a model that does not fit may run slowly rather than stop.

Preflight

Open a terminal and keep it for the whole lab; the variables below live in that shell. If you open a new terminal later, run the first block and your tier’s QM block again before anything else.

RunnableAll tracks

the working directory and the names every command uses

cd ~/llm-course
export LLAMA_BIN=~/llama.cpp/build/bin
M=~/models
MODELS_JSON=~/llm-course/models.json
mkdir -p bench-results

Then the one model tasks 6, 7 and 9 use, by tier:

RunnableAll tracks

QM on 12 GB and above

QM=Qwen3-8B

RunnableAll tracks

QM on the 8 GB tier

QM=Qwen3-4B

If your copy of models.json lives elsewhere, put its path in MODELS_JSON. Then check every prerequisite:

RunnableAll tracks

tools and files

command -v hf curl awk python3
python3 --version
ls -l fetch-model.sh verify-library.sh labbook.md "$MODELS_JSON"
ls -l "$LLAMA_BIN/llama-bench" "$LLAMA_BIN/llama-server" "$LLAMA_BIN/llama-cli"
grep -c '"lab": "part-05/bandwidth-test"' labbook.md
grep -c '"lab": "part-05/predict-decode"' labbook.md

Output — what you should see

/home/you/.local/bin/hf
/usr/bin/curl
/usr/bin/awk
/usr/bin/python3
Python 3.x.x
-rw-r--r-- 1 you you  xxxx ... fetch-model.sh
...
1
1

Four paths, a Python of 3.9 or later, seven files listed without No such file or directory, and two counts of at least 1. A missing Part 4 script means saving it from the Part 4 lab into ~/llm-course; a count of 0 means Part 5’s tasks 2 and 6 are still to do.

RunnableAll tracks

the build, and the memory the accelerator offers

"$LLAMA_BIN/llama-cli" --version
"$LLAMA_BIN/llama-bench" --list-devices

Output — what you should see

version: 0.4.0-dev (build xxxxx, commit xxxxxxx)
built with <compiler> for <platform>
Available devices:
  CUDA0: <device name> (xxxxx MiB, xxxxx MiB free)

The version lines have that shape at the pinned tag; a source build says -dev unless it was configured as a release. The device line is named CUDA0 on Tracks S and N, Vulkan0 or ROCm0 on Track X and MTL0 on Track M, and backend start-up messages may print above it. On Track M a second line, BLAS: Accelerate (0 MiB, 0 MiB free), follows MTL0; ignore it and use the MTL0 line. Available devices: followed by (none) means a CPU-only build: stop and go back to the install lesson’s backend check.

On Track M, check that the wired limit Part 5 set survived the last restart, because MTL0 depends on it:

RunnableTrack M · Apple silicon

Track M only: the wired limit now, and the one Part 5 recorded

sysctl -n iogpu.wired_limit_mb
grep -o '"iogpu_wired_limit_mb": [0-9][0-9]*' labbook.md | tail -n 1

Output — what you should see

xxxxx
"iogpu_wired_limit_mb": xxxxx

The two numbers must match. 0 on the first line with a larger number on the second means a restart put the default back; apply the recorded value again, then read MTL0 again.

⚠Warning

This changes how much memory macOS will pin for the GPU, to the value Part 5’s task 4 chose and recorded, and nothing else. Close large applications first. sudo sysctl iogpu.wired_limit_mb=0 returns the system default.

RunnableTrack M · Apple silicon

re-apply the recorded wired limit, then list the devices

WIRED_MB="$(grep -o '"iogpu_wired_limit_mb": [0-9][0-9]*' labbook.md | tail -n 1 | awk '{ print $2 }')"
if [ -n "$WIRED_MB" ]; then sudo sysctl iogpu.wired_limit_mb="$WIRED_MB"; else echo "no Part 5 value recorded: leave the limit alone"; fi
"$LLAMA_BIN/llama-bench" --list-devices

Output — what you should see

iogpu.wired_limit_mb: 0 -> xxxxx       (layout may differ by macOS version)
Available devices:
  MTL0: <device name> (xxxxx MiB, xxxxx MiB free)
  BLAS: Accelerate (0 MiB, 0 MiB free)

RunnableAll tracks

disk, and the port task 7 uses

df -h "$M"
curl -s --max-time 2 http://127.0.0.1:8089/health || echo "port 8089 is free"

Output — what you should see

Filesystem      Size  Used Avail Use% Mounted on
/dev/xxx        xxxG  xxxG  xxxG  xx% /
port 8089 is free

Avail must exceed your tier’s download total; on macOS df adds inode columns after Capacity, and Avail is the column to read. Anything other than port 8089 is free on the last line means a server is already running there; stop it, or set PORT when you run task 7’s script.

Tasks

1. Record what you are measuring with

A benchmark without a build is not reproducible, and llama.cpp merges performance work weekly. Put these fields in the notebook’s ## Environment section now, copied from the preflight output: the date; the full version: line, whose build number and commit identify the source exactly; the backend you built (CUDA, Vulkan, HIP or Metal) and the cmake flags; the device line from --list-devices with its total and free MiB; the track, machine and memory; the operating system version; and, on a laptop, whether it is on mains power. Every line the scripts append later carries the commit, and these fields are what it points back to. The flags are in the build directory’s CMake cache:

RunnableAll tracks

the backend flags this build was configured with

grep -E '^(CMAKE_BUILD_TYPE|GGML_CUDA|GGML_VULKAN|GGML_HIP|GGML_METAL|GGML_BLAS|GGML_NATIVE|GPU_TARGETS):' \
    "$LLAMA_BIN/../CMakeCache.txt"

Output — what you should see

CMAKE_BUILD_TYPE:STRING=Release
GGML_BLAS:BOOL=OFF
GGML_CUDA:BOOL=ON
GGML_HIP:BOOL=OFF
GGML_METAL:BOOL=OFF
GGML_NATIVE:BOOL=ON
GGML_VULKAN:BOOL=OFF

That is the shape on a CUDA build. Copy the build type and every backend line whose value is not OFF: GGML_CUDA on Tracks S and N; on Track X GGML_VULKAN, which reads =1 because the install lesson configured it as -DGGML_VULKAN=1, or GGML_HIP with its GPU_TARGETS line (use ~/llama.cpp/build-hip/CMakeCache.txt for that build); GGML_METAL and GGML_BLAS on Track M.

2. Download the set for your tier

Run the blocks in order and stop after the last one whose files all have a MiB free needed, in the table above, below the free MiB your --list-devices line printed. Each fetch-model.sh line verifies its file, and each block appends the paths to bench-set.txt, the list task 5 benchmarks.

RunnableAll tracks

8 GB and above

bash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf labbook.md
echo "$M/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf" >> bench-set.txt

Output — what you should see

==> Looking up Qwen3-8B-Q4_K_M.gguf in unsloth/Qwen3-8B-GGUF@main
    expected sha256 120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4
    expected size   5027784512 bytes
==> Downloading into /home/you/models/unsloth/Qwen3-8B-GGUF (safe to interrupt and re-run)
...progress...
==> Verifying SHA-256 (this reads the whole file; expect a minute or two on a large one)
    verified, checksum written to /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf.sha256
    recorded in labbook.md
==> Done: /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf

The checksum and size are the ones the Hub publishes for that file; if you already downloaded it in Part 4 or in the install lesson, the download step returns at once and the file is verified again.

RunnableAll tracks

12 GB and above

bash fetch-model.sh unsloth/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf labbook.md
echo "$M/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf" >> bench-set.txt

RunnableAll tracks

16 GB and above

bash fetch-model.sh ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf labbook.md
echo "$M/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf" >> bench-set.txt

RunnableAll tracks

24 GB and above

bash fetch-model.sh unsloth/Qwen3-30B-A3B-GGUF Qwen3-30B-A3B-Q4_K_M.gguf labbook.md
bash fetch-model.sh unsloth/Qwen3-32B-GGUF Qwen3-32B-Q4_K_M.gguf labbook.md
echo "$M/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf" >> bench-set.txt
echo "$M/unsloth/Qwen3-32B-GGUF/Qwen3-32B-Q4_K_M.gguf" >> bench-set.txt

RunnableAll tracks

48 GB and above

bash fetch-model.sh unsloth/Qwen3-30B-A3B-GGUF Qwen3-30B-A3B-Q8_0.gguf labbook.md
echo "$M/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q8_0.gguf" >> bench-set.txt

RunnableAll tracks

96 GB and above

bash fetch-model.sh ggml-org/gpt-oss-120b-GGUF gpt-oss-120b-MXFP4.gguf labbook.md
echo "$M/ggml-org/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4.gguf" >> bench-set.txt

The 235B is split into three shards in the repository’s IQ4_XS/ directory. fetch-model.sh takes the path inside the repository as the file name, keeps the directory, and verifies each shard.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

128 GB: the three 235B shards, then check them

bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf labbook.md
bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00002-of-00003.gguf labbook.md
bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00003-of-00003.gguf labbook.md
echo "$M/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf" >> bench-set.txt
(cd "$M/unsloth/Qwen3-235B-A22B-GGUF" && sha256sum --check IQ4_XS/*.sha256)

Track X — AMD Ryzen AI Max+ 395Not supported

The 235B row needs more memory than the GPU can address on a 128 GB Ryzen AI Max+ machine.

Track M — Apple silicon

Only on a 256 GB or 512 GB Mac whose MTL0 line shows at least 120,715 MiB free; on a 128 GB Mac the row does not pass, as the Requirements tab shows.

RunnableTrack M · Apple silicon

256 GB and above: the three 235B shards, then check them

bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf labbook.md
bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00002-of-00003.gguf labbook.md
bash fetch-model.sh unsloth/Qwen3-235B-A22B-GGUF IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00003-of-00003.gguf labbook.md
echo "$M/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf" >> bench-set.txt
(cd "$M/unsloth/Qwen3-235B-A22B-GGUF" && shasum -a 256 --check IQ4_XS/*.sha256)

Track N — NVIDIA desktop or laptopNot supported

No single consumer or workstation card in this track has the memory for the 235B row.

Output — what you should see

==> Looking up IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf in unsloth/Qwen3-235B-A22B-GGUF@main
    tree listing gave nothing; falling back to the LFS pointer
    expected sha256 e1de53f1ad86ca9b1a010a16d7dabb23a711a08b5ac989869813fbb0eed673ed
    expected size   49985865088 bytes
...
IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf: OK
IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00002-of-00003.gguf: OK
IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00003-of-00003.gguf: OK

The fallback line is expected: the tree listing the script reads first covers the top of the repository only. The last check is run from the repository directory because fetch-model.sh writes each shard’s checksum line as IQ4_XS/…. Part 4’s verify-library.sh checks from the shard’s own directory, where that name does not resolve, so it reports these three as FAILED while the check above says OK; the check above is the one that counts for them.

Finish with an audit of everything, and look at the list:

RunnableAll tracks

audit the library and the benchmark list

bash verify-library.sh labbook.md
sort -u bench-set.txt

Output — what you should see

==> Verifying every *.sha256 under /home/you/models
OK       /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
...
==> x ok, 0 failed, 0 missing, of x recorded file(s); xxxxxxxxxxx bytes verified
    recorded in labbook.md
/home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
...

With the 235B in the library, the same command prints three failures and exits with status 1:

Output — what you should see

==> Verifying every *.sha256 under /home/you/models
...
FAILED   /home/you/models/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf (contents do not match the recorded SHA-256)
FAILED   /home/you/models/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00002-of-00003.gguf (contents do not match the recorded SHA-256)
FAILED   /home/you/models/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00003-of-00003.gguf (contents do not match the recorded SHA-256)
...
==> x ok, 3 failed, 0 missing, of x recorded file(s); xxxxxxxxxxx bytes verified
    recorded in labbook.md

Those three are the path problem above, not damage, so the part-04/verify-library line carries "failed": 3 for them; write beside it in the notebook that the shard check from the repository directory said OK. Any other FAILED line is a damaged file, which Part 4’s audit task treats: delete it and re-run its fetch-model.sh line.

Record the list of files in your set. A fetch-model.sh failure is in the Part 4 troubleshooting table; the most common is a mistyped file name, which is case-sensitive.

3. One model, and what the columns mean

Run the tool once by hand before automating it.

RunnableAll tracks

one model, the lab's settings, markdown output

"$LLAMA_BIN/llama-bench" \
    -m "$M/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf" \
    -p 512 -n 128 -r 5 -ngl 999 -fa on

Output — what you should see

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       | 999 |   1 |           pp512 |         xxxx.xx ± xx.xx |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       | 999 |   1 |           tg128 |           xx.xx ± x.xx |

build: xxxxxxx (xxxxx)

On a native Windows build (Tracks N and X) this single-model measurement is the whole of the lab; the sweep, the ladder and the summary need a POSIX shell. Windows has no checksummed download path of its own, so copy Qwen3-8B-Q4_K_M.gguf and its .sha256 from the library fetch-model.sh verified in WSL2 or Linux into %USERPROFILE%\models\unsloth\Qwen3-8B-GGUF\, save task 5’s bench-to-labbook.py in %USERPROFILE%\llm-course, and check the copy first:

RunnableTrack N · Windows

Tracks N and X, native Windows: the copied file matches its checksum

$Model = "$env:USERPROFILE\models\unsloth\Qwen3-8B-GGUF\Qwen3-8B-Q4_K_M.gguf"
(Get-FileHash -Algorithm SHA256 $Model).Hash -eq (Get-Content "$Model.sha256").Split(" ")[0]

Output — what you should see

True

Then, in the folder holding llama-bench.exe, with Python 3.9 or later on PATH:

RunnableTrack N · Windows

Tracks N and X, native Windows: one model to JSON, then to the notebook

$LabDir = "$env:USERPROFILE\llm-course"
$Model = "$env:USERPROFILE\models\unsloth\Qwen3-8B-GGUF\Qwen3-8B-Q4_K_M.gguf"
New-Item -ItemType Directory -Force "$LabDir\bench-results" | Out-Null
.\llama-bench.exe -m $Model -p 512 -n 128 -r 5 -ngl 999 -fa on -o json |
    Out-File -Encoding utf8 "$LabDir\bench-results\Qwen3-8B-Q4_K_M.json"
python "$LabDir\bench-to-labbook.py" --input "$LabDir\bench-results\Qwen3-8B-Q4_K_M.json" `
    --labbook "$LabDir\labbook.md" --model-path $Model

Output — what you should see

             pp512     xxxx.xx ± xx.xx   tok/s   backend CUDA
             tg128       xx.xx ± x.xx    tok/s   backend CUDA
    recorded 2 line(s) in C:\Users\<you>\llm-course\labbook.md

False from the first block means the copy is incomplete: copy it again. The backend reads Vulkan on Track X. That notebook is a second file beside the WSL2 or Linux one; write its path in the first notebook’s ## Environment section.

Every column, as llama.cpp v0.4.0 fills it:

Column	What it is	Where it comes from
model	architecture, size label and quantisation family	the GGUF metadata; Q4_K - Medium is Q4_K_M
size	bytes of the weight tensors, in GiB (MiB when small)	a few MB below the file size, which also holds the tokeniser and metadata
params	parameters in the file	counted from the tensors
backend	every compiled-in backend except the CPU	CUDA, ROCm (HIP build), Vulkan, or MTL,BLAS on macOS; CPU only when there is nothing else
ngl	layers requested on the accelerator	printed only when the backend string contains neither CPU nor BLAS, which is why a Mac shows threads instead
fa	flash attention: 1 on, 0 off, -1 auto	shown because -fa on differs from the default
test	pp512 is 512 prompt tokens, tg128 is 128 generated tokens, @ d4096 a prefilled context	the -p, -n and -d values
t/s	mean of the per-repetition rates, ± their sample standard deviation	see below
build: line	commit, and in brackets the build number	the same identifier llama-cli --version printed

What each row times. The source at the pinned tag makes the method exact. Before measuring, one untimed warm-up runs the whole prompt test once and one generated token. Then, for each of the five repetitions, the KV cache is cleared and the clock runs around exactly one thing. A pp512 repetition is a single forward pass over 512 random token ids in one batch, because the default physical batch (-ub) is 512. A tg128 repetition is 128 forward passes of one token each, each feeding a random token id to the next. Tokenisation and sampling are outside the clock; the README says so, and the random ids make the words irrelevant. Each repetition’s rate is tokens × 10⁹ divided by its nanoseconds, and t/s is the mean of those five rates.

Those two rows measure different machines inside your machine, which is the most important idea in this lab. A prompt pass reads each weight once and uses it for 512 tokens, so its time is set by arithmetic; a generation step reads every active weight for one token, so its time is set by memory bandwidth. Part 3’s two clocks give the relation between them, with N the parameters multiplied per token, W the bytes read per token, B the bandwidth and C the arithmetic rate:

tg rate  ≈ B / W                                   (bandwidth-bound)
pp rate  ≈ min( C / (2 × N) ,  512 × B / W )       (compute-bound once the prompt passes n*)
pp / tg  ≈ min( n* , 512 )     where  n* = W × C / (2 × N × B)
prefill arithmetic actually delivered  ≈ pp rate × 2 × N

So the ratio between your two rows is, to first order, the prompt length at which prefill turns compute-bound on this machine for this model, capped at the prompt length tested. Here is that arithmetic for Qwen3-8B, not a measurement; replace C with the FP16 figure your Part 5 matmul-test.py recorded, and B with your measured bandwidth:

W = 4.67 GB read per token      Part 3's tensor count for Qwen3-8B at Q4_K_M
B = 273 GB/s                    the Spark's vendor figure in the course hardware reference
N = 8.2 B active parameters     models.json
C = 100 or 1,000 TFLOP/s        illustrative only, chosen to show both regimes

C, TFLOP/s (illustrative)	n* = W × C / (2 × N × B), tokens	tg ≈ B / W, tokens/s	Compute term C / (2 × N), tokens/s	Bandwidth term 512 × B / W, tokens/s	Predicted pp512, tokens/s	Predicted pp512 / tg128
100	104	58.5	6,098	29,931	6,098	104
1,000	1,043	58.5	60,976	29,931	29,931	512

With the slower arithmetic the ratio is n*; with the faster, n* passes 512 and the ratio stops at the prompt length, so a ratio near 512 says prefill was still waiting on bandwidth. The formula’s last line converts the pp512 rate into floating-point operations per second, to set beside that same Part 5 figure; task 8’s script prints it, using these operations per token from the active parameters in models.json:

Model	Active parameters	2 × N, GFLOP per token
Qwen3-4B	4.0 B	8.0
Qwen3-8B	8.2 B	16.4
Qwen3-14B	14.8 B	29.6
gpt-oss-20b	3.6 B	7.2
Qwen3-30B-A3B	3.3 B	6.6
Qwen3-32B	32.8 B	65.6
gpt-oss-120b	5.1 B	10.2
Qwen3-235B-A22B	22 B	44.0

How much to trust the ±. Save this beside the scripts as bench-stats.py. Run without an argument, it recomputes the generation row of the JSON example in the llama-bench README; given a JSON file from bench-results/, which task 5 writes, it does the same for your own run.

RunnableAll tracks

bench-stats.py

"""What llama-bench's t/s and its ± are, recomputed from the per-repetition timings."""
import json, statistics, sys

# No file given: the generation row of the JSON example in the llama-bench README (v0.4.0).
EXAMPLE = [{"n_prompt": 0, "n_gen": 128, "n_depth": 0, "avg_ts": 118.881588, "stddev_ts": 1.041811,
            "samples_ns": [1075361300, 1065089400, 1071761200, 1081934900, 1089692600]}]
T95 = {2: 12.706, 3: 4.303, 4: 3.182, 5: 2.776, 6: 2.571, 7: 2.447, 8: 2.365, 9: 2.306, 10: 2.262}

rows = json.load(open(sys.argv[1], encoding="utf-8-sig")) if len(sys.argv) > 1 else EXAMPLE
for row in rows:
    tokens = row["n_prompt"] + row["n_gen"]          # what one repetition processed
    rates = [tokens * 1e9 / ns for ns in row["samples_ns"]]
    n, mean, sd = len(rates), statistics.mean(rates), statistics.stdev(rates)
    label = f"pp{row['n_prompt']}" if row["n_prompt"] else f"tg{row['n_gen']}"
    label += f"@d{row['n_depth']}" if row["n_depth"] else ""
    print(f"{label}: {n} repetitions, rates " + ", ".join(f"{r:.2f}" for r in rates))
    print(f"  mean of rates {mean:.2f} (reported {row['avg_ts']:.2f}),"
          f" sample sd {sd:.2f} (reported {row['stddev_ts']:.2f}), cv {100 * sd / mean:.2f}%")
    print(f"  tokens / mean time {tokens * 1e9 / statistics.mean(row['samples_ns']):.2f},"
          f" 95% interval for the mean ± {T95.get(n, 2.0) * sd / n ** 0.5:.2f}")

RunnableAll tracks

the README's example

python3 bench-stats.py

Output — what you should see

bench-stats.py with no argument: arithmetic on the README's published timings, Python 3.14

tg128: 5 repetitions, rates 119.03, 120.18, 119.43, 118.31, 117.46
  mean of rates 118.88 (reported 118.88), sample sd 1.04 (reported 1.04), cv 0.88%
  tokens / mean time 118.87, 95% interval for the mean ± 1.29

Three things follow. The reported ± is the sample standard deviation of five rates, not an uncertainty on the mean; the interval on the mean is 2.776 times that divided by √5, because five repetitions leave four degrees of freedom. When one repetition was disturbed, the mean of rates and tokens over mean time drift apart, which is a quick test for a noisy run. And the decision rule this lab uses: a coefficient of variation above 3 per cent means re-run with the machine quiet, and two configurations differ only when their 95 per cent intervals do not overlap.

Record: the pp512 and tg128 rates with their ±, the backend string, ngl, and the ratio pp512 / tg128. On a Mac the table has threads where the others have ngl. If the backend says CPU, stop: see Troubleshooting.

4. The methodology, stated once

Every number in tasks 5 and 6 is produced the same way, and the settings are part of the result.

Setting	Value	Why this value	What changes if you move it
Prompt test	-p 512	llama-bench’s default, so comparable with other published runs; one physical batch at the default -ub 512	above 512 the prompt splits into several batches, and attention’s cost grows with length
Generation test	-n 128	the default; 128 steps swamp the fixed start-up cost of the first step	a much shorter test inflates the ±; a much longer one fills the cache and slows each step
Repetitions	-r 5	the default; four degrees of freedom for the interval	three repetitions widen the 95 per cent multiplier from 2.776 to 4.303
Warm-up	on (the default)	the first pass allocates buffers and compiles kernels	--no-warmup puts that one-off cost into the first repetition
Offload	-ngl 999	more layers than any model here has, so every layer is on the accelerator	a partial offload silently mixes CPU speed into the rate (task 9)
Flash attention	-fa on	the default auto resolves per model and backend, so two runs could differ without the table saying so	off changes the attention path, and a quantised V cache refuses to load without it
KV cache type	f16 (default)	the cache for a 512-token test is small either way	changes both memory (task 7) and the bytes read per step
Build	one commit for the whole sweep	kernels change between commits	a rebuild mid-sweep makes rows incomparable; the build field shows it
The machine	nothing else running; laptops on mains	a benchmark shares memory and clocks with everything else	background work shows as a large ±

∴Research note

This is a comparison methodology, not an absolute one: two models on your machine, or one model on two builds, can be compared honestly. A number from somebody else’s post can be set beside yours only when every row of that table matches, so publish the context block with any number you share.

5. Sweep the whole set

Save both files as ~/llm-course/bench-reference-models.sh and ~/llm-course/bench-to-labbook.py; the first calls the second from its own directory.

RunnableAll tracks

bench-reference-models.sh

#!/usr/bin/env bash
# Purpose: run llama-bench over a list of GGUF models with one fixed methodology, keep the
#          raw JSON and the error log for every model, and append one notebook line per test
# Platform: all (the same command on every track; only the backend differs)
# Minimum memory: 8 GB
# Assumes: llama.cpp built as in this part's install lesson (llama-bench found in $LLAMA_BIN,
#          on PATH, or in ~/llama.cpp/build/bin); models downloaded with Part 4's
#          fetch-model.sh; python3 for bench-to-labbook.py, which sits beside this script;
#          nothing else large running, because a benchmark shares the machine with it
#
# Usage: bash bench-reference-models.sh [labbook.md] [model.gguf ...]
#   With model files: benchmark exactly those, in the order given.
#   Without:          benchmark every *.gguf under $MODELS_DIR, sorted by path.
#   For a split model, give the first shard (...-00001-of-0000N.gguf); llama.cpp opens the
#   others from the same directory, and later shards are skipped if given.
#
# Environment:
#   MODELS_DIR   the model library, used when no files are given (default: $HOME/models)
#   LLAMA_BIN    directory holding llama-bench     (default: PATH, then ~/llama.cpp/build/bin)
#   RESULTS_DIR  where raw JSON and logs are kept  (default: ./bench-results)
#   PROMPT_LEN   prompt-processing test, tokens     (default: 512)
#   GEN_LEN      generation test, tokens            (default: 128)
#   REPS         repetitions per test               (default: 5)
#   NGL          layers to offload                  (default: 999, more than any model has)
#   FLASH_ATTN   on, off or auto                    (default: on)
#   GGML_CUDA_ENABLE_UNIFIED_MEMORY  not read here; when set, llama.cpp's CUDA and HIP
#                backends read it, and bench-to-labbook.py records its value on every line
#
# One llama-bench run per model, two tests per run: prompt processing at PROMPT_LEN tokens
# and generation at GEN_LEN tokens. llama-bench runs one untimed warm-up of each test, then
# REPS timed repetitions, and reports the mean and standard deviation of the per-repetition
# rates. The options are the long forms listed by `llama-bench --help` at llama.cpp v0.4.0.
# A result that bench-to-labbook.py refuses (the CPU backend) is counted as not recorded; its
# raw JSON is kept.

set -euo pipefail

LABBOOK="${1:-labbook.md}"
if [ "$#" -gt 0 ]; then shift; fi
MODELS_DIR="${MODELS_DIR:-$HOME/models}"
RESULTS_DIR="${RESULTS_DIR:-./bench-results}"
PROMPT_LEN="${PROMPT_LEN:-512}"
GEN_LEN="${GEN_LEN:-128}"
REPS="${REPS:-5}"
NGL="${NGL:-999}"
FLASH_ATTN="${FLASH_ATTN:-on}"

die() { echo "bench-reference-models: $*" >&2; exit 1; }

case "$LABBOOK" in
  *.gguf) die "the first argument is the notebook; usage: bash bench-reference-models.sh labbook.md model.gguf ..." ;;
esac

if [ -n "${LLAMA_BIN:-}" ]; then
  BENCH="$LLAMA_BIN/llama-bench"
elif command -v llama-bench >/dev/null 2>&1; then
  BENCH="$(command -v llama-bench)"
else
  BENCH="$HOME/llama.cpp/build/bin/llama-bench"
fi
[ -x "$BENCH" ] || die "llama-bench not found at $BENCH; set LLAMA_BIN to the directory holding it"
command -v python3 >/dev/null || die "python3 is not installed"

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
TO_LABBOOK="$SCRIPT_DIR/bench-to-labbook.py"
[ -f "$TO_LABBOOK" ] || die "bench-to-labbook.py is not beside this script in $SCRIPT_DIR"

# --- 1. Collect the models -------------------------------------------------------------
CANDIDATES=()
if [ "$#" -gt 0 ]; then
  for f in "$@"; do
    [ -f "$f" ] || die "model file '$f' does not exist (names are case-sensitive)"
    CANDIDATES+=("$f")
  done
else
  [ -d "$MODELS_DIR" ] || die "$MODELS_DIR does not exist; download models with fetch-model.sh first"
  while IFS= read -r f; do
    CANDIDATES+=("$f")
  done < <(find "$MODELS_DIR" -type f -name '*.gguf' | sort)
fi

[ "${#CANDIDATES[@]}" -gt 0 ] || die "no .gguf files found under $MODELS_DIR"

# Empty arrays are guarded before every "${array[@]}": macOS ships bash 3.2, where expanding
# an empty array under set -u is an error.
MODELS=()
for f in "${CANDIDATES[@]}"; do
  base="$(basename "$f")"
  case "$base" in
    mmproj*|*mmproj*)
      echo "    skipping $base (a multimodal projector, not a language model)"; continue ;;
    *-0000[2-9]-of-*|*-000[1-9][0-9]-of-*)
      echo "    skipping $base (a later shard; the first shard loads it)"; continue ;;
  esac
  MODELS+=("$f")
done
[ "${#MODELS[@]}" -gt 0 ] || die "no model files to benchmark"

mkdir -p "$RESULTS_DIR"
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
HOSTINFO="$(uname -s)-$(uname -m)"

echo "==> ${#MODELS[@]} model file(s) to benchmark with $BENCH"
echo "    prompt processing ${PROMPT_LEN} tokens, generation ${GEN_LEN} tokens, ${REPS} repetitions"
echo "    --n-gpu-layers $NGL, --flash-attn $FLASH_ATTN, run $RUN_ID"

# --- 2. Benchmark each model -----------------------------------------------------------
OK=0
FAILED_NAMES=()
for model in "${MODELS[@]}"; do
  base="$(basename "$model" .gguf)"
  out="$RESULTS_DIR/${RUN_ID}__${base}.json"
  echo "==> $base"
  if "$BENCH" --model "$model" --n-prompt "$PROMPT_LEN" --n-gen "$GEN_LEN" \
       --repetitions "$REPS" --n-gpu-layers "$NGL" --flash-attn "$FLASH_ATTN" \
       --output json > "$out" 2> "$out.log"; then
    if python3 "$TO_LABBOOK" --input "$out" --labbook "$LABBOOK" \
         --run-id "$RUN_ID" --host "$HOSTINFO" --model-path "$model"; then
      OK=$((OK + 1))
    else
      FAILED_NAMES+=("$base")
      echo "    NOT RECORDED; the raw result stays in $out" >&2
    fi
  else
    FAILED_NAMES+=("$base")
    echo "    FAILED; the last lines of $out.log were:" >&2
    tail -n 5 "$out.log" >&2 || true
    if grep -qiE 'out of memory|OutOfDeviceMemory|failed to allocate|alloc.*failed|unable to allocate' "$out.log"; then
      echo "    That is an allocation failure: the model does not fit. Record it and carry on." >&2
    else
      echo "    No allocation error in the log: check the file with Part 4's verify-library.sh" >&2
      echo "    (for a split model, check its shards from the repository directory, as the page shows)." >&2
    fi
  fi
done

echo "==> Done: $OK recorded, ${#FAILED_NAMES[@]} failed or not recorded. Raw JSON and logs in $RESULTS_DIR."
if [ "${#FAILED_NAMES[@]}" -gt 0 ]; then
  printf '    did not complete: %s\n' "${FAILED_NAMES[@]}" >&2
fi
[ "$OK" -gt 0 ] || exit 1

Download bench-reference-models.sh141 lines

RunnableAll tracks

bench-to-labbook.py

#!/usr/bin/env python3
"""Turn llama-bench JSON output into course lab-notebook lines.

Purpose: read the JSON llama-bench writes with `--output json`, pull out the fields that
    make a measurement meaningful (build, backend, model file, quantisation, offloaded
    layers, batch sizes, KV cache types, test sizes, repetitions, mean and standard
    deviation) and append one JSON line per test to the lab notebook, in the shape every
    other lab in this course uses. A result from the CPU backend is printed and refused, so
    a build without its GPU backend never reaches the notebook, unless --allow-cpu is given.
Platform: all (pure Python standard library; nothing platform-specific)
Minimum memory: 8 GB
Assumes: Python 3.9 or later; a JSON file produced by `llama-bench ... --output json` (the
    field names used here were read from llama-bench's JSON output at llama.cpp v0.4.0);
    the lab notebook started in Part 1, or a path to create.

Usage: python3 bench-to-labbook.py --input bench-results/run.json --labbook labbook.md
       python3 bench-to-labbook.py --input run.json --print-only

Exit status: 0 when the lines were recorded (or printed with --print-only); 1 when the input
is missing or empty, or when llama-bench reports the CPU backend and --allow-cpu was not given,
in which case nothing is written.

Every field is looked up by name and recorded as null when the build did not write it. A
null means "this build did not report it", not "it was zero". The number of repetitions
is counted from the per-repetition samples, because llama-bench's JSON has no field for it.
ctx_size is the context the test asked for, n_prompt + n_gen + n_depth, which is what
llama-bench passes to llama.cpp; llama.cpp v0.4.0 pads it up to a multiple of 256 when it
allocates (512 for pp512, 256 for tg128). ggml_cuda_enable_unified_memory records the value
of that environment variable when it was set (CUDA and HIP builds read it), and null otherwise.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import sys
from pathlib import Path

LAB = "part-06/lab-benchmark-the-reference-models"


def pick(row: dict, *names, default=None):
    """First present, non-empty value among several possible key names."""
    for name in names:
        if name in row and row[name] not in (None, ""):
            return row[name]
    return default


def as_int(value) -> int:
    try:
        return int(value)
    except (TypeError, ValueError):
        return 0


def test_label(row: dict) -> str:
    """llama-bench's own naming: pp512 is prompt processing, tg128 is generation."""
    n_prompt = as_int(pick(row, "n_prompt"))
    n_gen = as_int(pick(row, "n_gen"))
    depth = as_int(pick(row, "n_depth"))
    if n_prompt and n_gen:
        label = f"pp{n_prompt}+tg{n_gen}"
    elif n_gen:
        label = f"tg{n_gen}"
    elif n_prompt:
        label = f"pp{n_prompt}"
    else:
        label = "unknown"
    return f"{label}@d{depth}" if depth else label


def backend_of(row: dict) -> str | None:
    value = pick(row, "backends", "backend")
    if isinstance(value, list):
        return ",".join(str(v) for v in value)
    return str(value) if value is not None else None


# A quantisation name in a file name: Q4_K_M, Q8_0, IQ4_XS, MXFP4, BF16 ...
QUANT_RE = re.compile(r"(?:^|[-_. /])((?:IQ|TQ|Q)\d[A-Z0-9_]*|MXFP4|BF16|F16|F32)(?=$|[-_. /])")
# The same thing in llama-bench's model_type, e.g. "qwen3 8B Q4_K - Medium".
TYPE_RE = re.compile(r"\b((?:IQ|TQ|Q)\d[A-Z0-9_]*|MXFP4|BF16|F16|F32)\b(?: - (SMALL|MEDIUM|LARGE))?")
SUFFIX = {"SMALL": "_S", "MEDIUM": "_M", "LARGE": "_L"}


def quant_of(row: dict, model_path: str) -> str | None:
    """Prefer the file name, which carries the full name (Q4_K_M rather than Q4_K)."""
    for value in (model_path, pick(row, "model_filename")):
        if value:
            found = QUANT_RE.search(Path(str(value)).name.upper())
            if found:
                return found.group(1)
    model_type = pick(row, "model_type")
    if model_type:
        found = TYPE_RE.search(str(model_type).upper())
        if found:
            return found.group(1) + SUFFIX.get(found.group(2) or "", "")
    return None


def read_json(path: Path):
    """PowerShell on Windows can write UTF-16 or UTF-8 with a byte-order mark; accept both."""
    raw = path.read_bytes()
    if raw[:2] in (b"\xff\xfe", b"\xfe\xff"):
        text = raw.decode("utf-16")
    else:
        text = raw.decode("utf-8-sig")
    try:
        return json.loads(text)
    except json.JSONDecodeError as exc:
        raise SystemExit(
            f"{path}: not valid JSON ({exc}). An empty or half-written file means llama-bench "
            "failed; read the .log file beside it."
        )


def rows_from(path: Path) -> list[dict]:
    data = read_json(path)
    if isinstance(data, dict):
        for key in ("results", "data", "rows"):
            if isinstance(data.get(key), list):
                return data[key]
        return [data]
    if isinstance(data, list):
        return data
    raise SystemExit(f"{path}: expected a JSON array of test results, got {type(data).__name__}")


def record(row: dict, args) -> dict:
    size_bytes = as_int(pick(row, "model_size"))
    n_params = as_int(pick(row, "model_n_params"))
    samples = pick(row, "samples_ts", "samples_ns", default=[])
    model_path = args.model_path or pick(row, "model_filename")
    n_prompt, n_gen, n_depth = (as_int(pick(row, name)) for name in ("n_prompt", "n_gen", "n_depth"))
    return {
        "lab": LAB,
        "run": args.run_id,
        "host": args.host,
        "engine": "llama.cpp",
        "build": pick(row, "build_commit"),
        "build_number": pick(row, "build_number"),
        "backend": backend_of(row),
        "gpu_info": pick(row, "gpu_info"),
        "cpu_info": pick(row, "cpu_info"),
        "model": pick(row, "model_type", "model_filename"),
        "model_path": model_path,
        "quant": quant_of(row, args.model_path),
        "params_b": round(n_params / 1e9, 2) if n_params else None,
        "file_gb": round(size_bytes / 1e9, 2) if size_bytes else None,
        "model_size_bytes": size_bytes or None,
        "test": test_label(row),
        "n_prompt": pick(row, "n_prompt"),
        "n_gen": pick(row, "n_gen"),
        "n_depth": pick(row, "n_depth"),
        "ctx_size": n_prompt + n_gen + n_depth,
        "n_gpu_layers": pick(row, "n_gpu_layers"),
        "split_mode": pick(row, "split_mode"),
        "n_batch": pick(row, "n_batch"),
        "n_ubatch": pick(row, "n_ubatch"),
        "n_threads": pick(row, "n_threads"),
        "flash_attn": pick(row, "flash_attn"),
        "type_k": pick(row, "type_k"),
        "type_v": pick(row, "type_v"),
        "ggml_cuda_enable_unified_memory": os.environ.get("GGML_CUDA_ENABLE_UNIFIED_MEMORY"),
        "reps": len(samples) if isinstance(samples, list) and samples else None,
        "tokens_per_s": pick(row, "avg_ts"),
        "tokens_per_s_stddev": pick(row, "stddev_ts"),
        "measured_on": pick(row, "test_time"),
    }


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--input", required=True, help="JSON file written by llama-bench --output json")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--run-id", default="", help="identifier shared by one sweep")
    parser.add_argument("--host", default="", help="short machine description")
    parser.add_argument("--model-path", default="", help="path of the model file benchmarked")
    parser.add_argument("--print-only", action="store_true", help="print the lines, record nothing")
    parser.add_argument("--allow-cpu", action="store_true",
                        help="record a result from the CPU backend (a deliberate CPU run only)")
    args = parser.parse_args()

    path = Path(args.input)
    if not path.exists():
        print(f"{path} does not exist", file=sys.stderr)
        return 1

    rows = rows_from(path)
    if not rows:
        print(f"{path} contained no test results", file=sys.stderr)
        return 1

    entries = [record(row, args) for row in rows]

    for entry in entries:
        rate, spread = entry["tokens_per_s"], entry["tokens_per_s_stddev"]
        rate_text = f"{float(rate):.2f}" if isinstance(rate, (int, float)) else "n/a"
        spread_text = f"{float(spread):.2f}" if isinstance(spread, (int, float)) else "n/a"
        print(f"    {entry['test']:>14}  {rate_text:>10} ± {spread_text:<7} tok/s"
              f"   backend {entry['backend'] or 'unknown'}")

    on_cpu = "CPU" in {entry["backend"] for entry in entries}
    if on_cpu:
        print("    WARNING: llama-bench reports the CPU backend. This build has no GPU backend, or"
              " the GPU runtime did not load (install lesson, 'Proving the backend is in use').",
              file=sys.stderr)

    if args.print_only:
        return 0

    if on_cpu and not args.allow_cpu:
        print("    nothing recorded; pass --allow-cpu only for a deliberate CPU run", file=sys.stderr)
        return 1

    notebook = Path(args.labbook)
    if not notebook.exists():
        print(f"    {notebook} does not exist; creating it", file=sys.stderr)
    with notebook.open("a", encoding="utf-8") as handle:
        for entry in entries:
            handle.write(json.dumps(entry) + "\n")
    print(f"    recorded {len(entries)} line(s) in {notebook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download bench-to-labbook.py230 lines

The script takes the notebook, then the model files. bench-set.txt from task 2 holds your tier’s list, one path per line with no spaces, so the shell can pass it straight through.

RunnableAll tracks

the full sweep for your tier

bash bench-reference-models.sh labbook.md $(sort -u bench-set.txt)

Output — what you should see

==> x model file(s) to benchmark with /home/you/llama.cpp/build/bin/llama-bench
    prompt processing 512 tokens, generation 128 tokens, 5 repetitions
    --n-gpu-layers 999, --flash-attn on, run 2026xxxxTxxxxxxZ
==> gpt-oss-20b-MXFP4
             pp512     xxxx.xx ± xx.xx   tok/s   backend CUDA
             tg128      xxx.xx ± x.xx    tok/s   backend CUDA
    recorded 2 line(s) in labbook.md
==> Qwen3-14B-Q4_K_M
...
==> Done: x recorded, 0 failed or not recorded. Raw JSON and logs in ./bench-results.

What to look for: two lines per model, the backend you built on every one of them, and a ± small against its rate. WARNING: llama-bench reports the CPU backend followed by nothing recorded and NOT RECORDED means bench-to-labbook.py refused the result, as it should: the build has no GPU backend in use, and the fix is in Troubleshooting, not in the notebook. A model that does not fit prints FAILED, the last lines of its log, and either That is an allocation failure or a pointer to verify-library.sh; the sweep carries on, and “does not fit on this machine” is a result.

On Track X, if you also built HIP, run the same sweep with that build. Its lines record ROCm as the backend, so they sit beside the Vulkan ones rather than replacing them. The other tracks have one build and skip to the notebook line below.

RunnableTrack X · Ryzen AI Max+

Track X only: the same sweep with the HIP build

LLAMA_BIN=~/llama.cpp/build-hip/bin bash bench-reference-models.sh labbook.md $(sort -u bench-set.txt)

The output has the shape above with backend ROCm. The build guide’s HIP section documents GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 as the way to share main memory with an integrated GPU on Linux. If a file printed FAILED with an allocation line under HIP but loaded under Vulkan, put its path in HIP_RETRY and run it again with the variable set:

Fragment — not complete on its own

the file that failed under HIP (this path is an example)

HIP_RETRY="$M/ggml-org/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4.gguf"

RunnableTrack X · Ryzen AI Max+

Track X only: that file again, with unified memory enabled

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 LLAMA_BIN=~/llama.cpp/build-hip/bin \
    bash bench-reference-models.sh labbook.md "$HIP_RETRY"

Nothing needs recording by hand: each line of that run carries "ggml_cuda_enable_unified_memory": "1", and task 8’s script labels its rows ROCm+UMA.

Each notebook line looks like this, with your values; the capstone reads these fields by name:

Output — what you should see

{"lab": "part-06/lab-benchmark-the-reference-models", "run": "2026xxxxTxxxxxxZ", "host": "Linux-aarch64", "engine": "llama.cpp", "build": "xxxxxxx", "build_number": xxxxx, "backend": "CUDA", "gpu_info": "...", "cpu_info": "...", "model": "qwen3 8B Q4_K - Medium", "model_path": "/home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf", "quant": "Q4_K_M", "params_b": 8.19, "file_gb": x.xx, "model_size_bytes": xxxxxxxxxx, "test": "tg128", "n_prompt": 0, "n_gen": 128, "n_depth": 0, "ctx_size": 128, "n_gpu_layers": 999, "split_mode": "layer", "n_batch": 2048, "n_ubatch": 512, "n_threads": x, "flash_attn": 1, "type_k": "f16", "type_v": "f16", "ggml_cuda_enable_unified_memory": null, "reps": 5, "tokens_per_s": xx.xxxxxx, "tokens_per_s_stddev": x.xxxxxx, "measured_on": "2026-xx-xxTxx:xx:xxZ"}

Check the spread of one of your own results with task 3’s bench-stats.py, on the newest JSON for Qwen3-8B, and apply task 3’s decision rule to the cv of both rows.

RunnableAll tracks

the spread of your own Qwen3-8B run

python3 bench-stats.py "$(ls bench-results/*__Qwen3-8B-Q4_K_M.json | tail -n 1)"

Output — what you should see

pp512: 5 repetitions, rates xxxx.xx, xxxx.xx, xxxx.xx, xxxx.xx, xxxx.xx
  mean of rates xxxx.xx (reported xxxx.xx), sample sd xx.xx (reported xx.xx), cv x.xx%
  tokens / mean time xxxx.xx, 95% interval for the mean ± xx.xx
tg128: 5 repetitions, rates xx.xx, xx.xx, xx.xx, xx.xx, xx.xx
  mean of rates xx.xx (reported xx.xx), sample sd x.xx (reported x.xx), cv x.xx%
  tokens / mean time xx.xx, 95% interval for the mean ± x.xx

Copy the rates into the recording sheet below as the sweep finishes; task 8 prints the same rows from the notebook, so the sheet is the human-readable copy. Record the models that failed, with the first allocation line from their .log.

Pending validationPrompt processing and generation, one row per model — your recording sheet

Model and quantisation	pp512 tokens/s ± sd	tg128 tokens/s ± sd	pp512 / tg128	File GB
Qwen3-8B Q4_K_M	—	—	—	5.03
Qwen3-14B Q4_K_M	—	—	—	9.00
gpt-oss-20b MXFP4	—	—	—	12.11
Qwen3-30B-A3B Q4_K_M	—	—	—	18.56
Qwen3-32B Q4_K_M	—	—	—	19.76
Qwen3-30B-A3B Q8_0	—	—	—	32.48
gpt-oss-120b MXFP4	—	—	—	63.39
Qwen3-235B-A22B IQ4_XS	—	—	—	125.50

your machine: track, chip and memory, your operating system and version · llama.cpp the build number and commit from llama-cli --version · as listed per row, as listed per row · 512 tokens of context · the date you ran it

Empty on purpose: this is the shape to fill in from your own run, not a result. The file sizes are the Hub byte counts of 2026-09-13. llama-bench creates one context per test, n_prompt + n_gen + n_depth tokens padded by llama.cpp to a multiple of 256: 512 for pp512 and 256 for tg128; the largest is recorded. Replace every context field with what you actually ran; a rate without the build cannot be reproduced.

6. Quantisation against speed and size

Now change one variable: the quantisation of one model, everything else fixed. QM was set in the preflight; check it, then fetch its four quantisations. For Qwen3-8B the Q4_K_M line only re-verifies the file from task 2.

RunnableAll tracks

the four quantisations of that model

echo "$QM"
bash fetch-model.sh "unsloth/${QM}-GGUF" "${QM}-IQ4_XS.gguf" labbook.md
bash fetch-model.sh "unsloth/${QM}-GGUF" "${QM}-Q4_K_M.gguf" labbook.md
bash fetch-model.sh "unsloth/${QM}-GGUF" "${QM}-Q6_K.gguf" labbook.md
bash fetch-model.sh "unsloth/${QM}-GGUF" "${QM}-Q8_0.gguf" labbook.md

The first line must print Qwen3-8B or Qwen3-4B; an empty line means this terminal has not run the preflight’s QM block, and the downloads will fail to find their files. Each download prints the shape task 2 showed.

Predict before you measure. Decode reads the whole file per token for a dense model, so at a fixed bandwidth the generation rate should scale with the inverse of the file size, and prompt processing, which is compute-bound, should move much less. The prediction below is arithmetic from the Hub file sizes, with bits per weight from the parameter counts in models.json:

File	Size, GB	Bits per weight	Size against Q4_K_M	Predicted tg128 against Q4_K_M
Qwen3-8B-IQ4_XS	4.58	4.47	0.911	1.097
Qwen3-8B-Q4_K_M	5.03	4.91	1.000	1.000
Qwen3-8B-Q6_K	6.73	6.56	1.338	0.748
Qwen3-8B-Q8_0	8.71	8.50	1.732	0.577
Qwen3-4B-IQ4_XS	2.27	4.54	0.909	1.100
Qwen3-4B-Q4_K_M	2.50	4.99	1.000	1.000
Qwen3-4B-Q6_K	3.31	6.61	1.324	0.755
Qwen3-4B-Q8_0	4.28	8.56	1.714	0.583

RunnableAll tracks

benchmark the four files

bash bench-reference-models.sh labbook.md \
    "$M/unsloth/${QM}-GGUF/${QM}-IQ4_XS.gguf" \
    "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" \
    "$M/unsloth/${QM}-GGUF/${QM}-Q6_K.gguf" \
    "$M/unsloth/${QM}-GGUF/${QM}-Q8_0.gguf"

The output has the shape of task 5’s. Divide each tg128 by the Q4_K_M one and set the result beside the last column. IQ4_XS is the informative row: its bytes predict the fastest generation of the four, but its weights are decoded through a codebook rather than a K-quant’s block scales, so it is the row where the arithmetic of decoding, not the bytes, can decide on your backend. Whether it did is a measurement to record, not a conclusion to assume. Which quantisation is worth its quality cost is Part 16’s question; this task prices only speed and size.

Pending validationOne model, four quantisations — your recording sheet

File	File GB	pp512 tokens/s	tg128 tokens/s ± sd	tg128 against Q4_K_M, measured	Predicted, Qwen3-8B	Predicted, Qwen3-4B
IQ4_XS	—	—	—	—	1.10	1.10
Q4_K_M	—	—	—	1.00	1.00	1.00
Q6_K	—	—	—	—	0.75	0.76
Q8_0	—	—	—	—	0.58	0.58

your machine: track, chip and memory · llama.cpp the build number and commit from llama-cli --version · Qwen3-8B, or Qwen3-4B on the 8 GB tier, as listed per row · 512 tokens of context · the date you ran it

Empty on purpose. The predicted columns are the table above rounded to two places; use the one for your model. The context length follows the same rule as task 5's sheet. A measured ratio well below the prediction for the largest file calls for checking its MiB free needed against --list-devices before any other explanation.

7. Context length against memory

Every command in this task names the model through QM, so check this terminal has it first:

RunnableAll tracks

the model tasks 7 and 9 use

echo "$QM"

Output — what you should see

Qwen3-8B

Qwen3-4B on the 8 GB tier. An empty line means run the preflight’s first block and your tier’s QM block in this terminal first.

The KV cache is the memory a context costs, and llama.cpp allocates all of it when the model loads, for the context length you ask for, before a single token arrives. Part 3 derived the formula; here is where llama.cpp reads its inputs and what it should print:

KV bytes = layers × kv_heads × (head_dim_k + head_dim_v) × bytes_per_element × context
           f16: 2 bytes per element;  q8_0: 34 bytes per 32 elements = 1.0625

Qwen3-8B and Qwen3-4B, from the load log's print_info lines:
  n_layer = 36   n_head_kv = 8   n_embd_head_k = 128   n_embd_head_v = 128
  f16:  36 × 8 × 256 × 2 = 147,456 bytes per token, the figure in models.json

Context and cache type	KV cache by the formula, MiB	Qwen3-8B Q4_K_M: file + KV, MiB	MiB free it needs, with the 1,024 margin	Qwen3-4B Q4_K_M: file + KV, MiB	MiB free it needs
4,096, f16	576.00	5,371	6,395	2,958	3,982
16,384, f16	2,304.00	7,099	8,123	4,686	5,710
32,768, f16	4,608.00	9,403	10,427	6,990	8,014
32,768, q8_0	2,448.00	7,243	8,267	4,830	5,854

The two models share one KV row because they share layer count, key-value heads and head size; only the weights differ. The course checked the formula against llama.cpp v0.4.0’s own log on a CPU build with Qwen3-0.6B (28 layers, the same heads): it logged 448.00 MiB at 4,096 tokens, 1,792.00 at 16,384 and 952.00 at 16,384 with q8_0, the formula’s figures to the hundredth. Before you run the ladder, write down which rungs your MiB free says will load.

Save the script as ~/llm-course/kv-memory-ladder.sh. It loads the model four times with --verbose, because llama.cpp prints the KV size and the memory breakdown only above the default verbosity, and with --fit off, because llama-server’s default --fit on otherwise adjusts settings you did not set to make a load fit, and a rung that silently became a different rung would measure nothing. Each rung waits for listening on, then stops the server, which prints the breakdown on exit.

RunnableAll tracks

kv-memory-ladder.sh

#!/usr/bin/env bash
# Purpose: load one model in llama-server at several context lengths and KV cache types, one
#          after another, and record for each what llama.cpp allocated: the KV cache size it
#          logged, the per-device memory breakdown it prints on exit, and the KV size the
#          formula predicts from the model's own metadata
# Platform: all (Linux, macOS and WSL2; the backend is whatever llama.cpp was built with)
# Minimum memory: 8 GB
# Assumes: llama.cpp v0.4.0 or later built as in this part's install lesson (llama-server in
#          $LLAMA_BIN, on PATH, or in ~/llama.cpp/build/bin); a GGUF model file; curl, awk
#          and python3 on PATH; nothing listening on $PORT; nothing else large running
#
# Usage: bash kv-memory-ladder.sh <model.gguf> [labbook.md]
#
# Environment:
#   LLAMA_BIN     directory holding llama-server  (default: PATH, then ~/llama.cpp/build/bin)
#   LADDER        context:cache-type rungs, space-separated
#                 (default: "4096:f16 16384:f16 32768:f16 32768:q8_0")
#   NGL           layers to offload                (default: 999, meaning all)
#   PORT          local port for the probe         (default: 8089)
#   LOAD_TIMEOUT  seconds to wait for a load       (default: 600)
#   LOG_DIR       where each rung's full log goes  (default: ./kv-logs)
#
# Each rung starts the server bound to 127.0.0.1 with --verbose (the KV cache and memory
# breakdown lines are not printed at the default verbosity), waits for "listening on" or for
# the process to exit, then stops it with SIGTERM; the memory breakdown is printed on the way
# out. --fit off stops llama.cpp adjusting anything to fit, so a rung that does not fit fails
# instead of silently becoming a different rung. Nothing is sent to the server.

set -euo pipefail

MODEL="${1:-}"
LABBOOK="${2:-}"
LADDER="${LADDER:-4096:f16 16384:f16 32768:f16 32768:q8_0}"
NGL="${NGL:-999}"
PORT="${PORT:-8089}"
LOAD_TIMEOUT="${LOAD_TIMEOUT:-600}"
LOG_DIR="${LOG_DIR:-./kv-logs}"

die() { echo "kv-memory-ladder: $*" >&2; exit 1; }

[ -n "$MODEL" ] || die "usage: bash kv-memory-ladder.sh <model.gguf> [labbook.md]"
[ -f "$MODEL" ] || die "model file '$MODEL' does not exist (names are case-sensitive)"
for tool in curl awk python3; do
  command -v "$tool" >/dev/null || die "$tool is not installed or not on PATH"
done

if [ -n "${LLAMA_BIN:-}" ]; then
  SERVER="$LLAMA_BIN/llama-server"
elif command -v llama-server >/dev/null 2>&1; then
  SERVER="$(command -v llama-server)"
else
  SERVER="$HOME/llama.cpp/build/bin/llama-server"
fi
[ -x "$SERVER" ] || die "llama-server not found at $SERVER; set LLAMA_BIN to the directory holding it"

if curl -s --max-time 2 "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
  die "something is already answering on 127.0.0.1:$PORT; stop it or set PORT to a free port"
fi

mkdir -p "$LOG_DIR"
VERSION="$("$SERVER" --version 2>&1 | head -n 1)"
BASE="$(basename "$MODEL" .gguf)"
echo "==> $BASE with $VERSION"
printf '%-8s %-6s %-9s %12s %12s   %s\n' "context" "cache" "status" "KV MiB" "predicted" "device breakdown (total = free + (self = model + context + compute) + unaccounted)"

SERVER_PID=""
cleanup() { if [ -n "$SERVER_PID" ]; then kill "$SERVER_PID" 2>/dev/null || true; fi; }
trap cleanup EXIT
trap 'cleanup; exit 130' INT TERM

# The value of one "print_info: <key> = <value>" line of the current rung's log.
meta() { awk -v key="$1" '$0 ~ "print_info: " key " +=" { v = $NF } END { print v }' "$log"; }

for rung in $LADDER; do
  ctx="${rung%%:*}"
  cache="${rung##*:}"
  log="$LOG_DIR/${BASE}__c${ctx}__${cache}.log"

  "$SERVER" --model "$MODEL" --n-gpu-layers "$NGL" --ctx-size "$ctx" --parallel 1 \
    --flash-attn on --cache-type-k "$cache" --cache-type-v "$cache" --fit off \
    --host 127.0.0.1 --port "$PORT" --verbose > "$log" 2>&1 &
  SERVER_PID=$!

  status="timeout"
  waited=0
  while [ "$waited" -lt "$LOAD_TIMEOUT" ]; do
    if grep -q "listening on" "$log"; then status="loaded"; break; fi
    if ! kill -0 "$SERVER_PID" 2>/dev/null; then status="failed"; break; fi
    sleep 1
    waited=$((waited + 1))
  done
  kill -TERM "$SERVER_PID" 2>/dev/null || true
  wait "$SERVER_PID" 2>/dev/null || true
  SERVER_PID=""

  # "llama_kv_cache: size = 576.00 MiB (4096 cells, ...)"; a model with sliding-window
  # layers logs two caches, so the sizes are summed.
  kv_mib="$(awk '/llama_kv_cache: size =/ { for (i = 1; i <= NF; i++) if ($i == "=") { s += $(i + 1); break } } END { if (s > 0) printf "%.2f", s }' "$log")"
  predicted="$(python3 -c '
import sys
layers, heads, hk, hv, ctx, cache = sys.argv[1:7]
per_element = {"f32": 4.0, "f16": 2.0, "bf16": 2.0, "q8_0": 34 / 32, "q4_0": 18 / 32}.get(cache)
try:
    print(f"{int(layers) * int(heads) * (int(hk) + int(hv)) * per_element * int(ctx) / 2**20:.2f}")
except (TypeError, ValueError):
    print("")
' "$(meta n_layer)" "$(meta n_head_kv)" "$(meta n_embd_head_k)" "$(meta n_embd_head_v)" "$ctx" "$cache")"
  breakdown="$(grep 'common_memory_breakdown_print: |' "$log" | grep -v 'memory breakdown \[MiB\]' | sed 's/.*common_memory_breakdown_print: //' | tail -n 4 || true)"
  first_row="$(printf '%s\n' "$breakdown" | head -n 1)"
  printf '%-8s %-6s %-9s %12s %12s   %s\n' "$ctx" "$cache" "$status" "${kv_mib:--}" "${predicted:--}" "${first_row:--}"
  if [ "$status" != "loaded" ]; then
    grep -iE 'out of memory|OutOfDeviceMemory|failed to allocate|unable to allocate|error' "$log" | tail -n 3 | sed 's/^/           /' || true
  fi

  if [ -n "$LABBOOK" ]; then
    python3 -c '
import json, sys
from datetime import datetime, timezone
model, ctx, cache, status, kv, predicted, breakdown, version, log = sys.argv[1:10]
print(json.dumps({
    "lab": "part-06/kv-memory-ladder",
    "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    "engine": "llama.cpp", "version": version, "model_path": model,
    "ctx_size": int(ctx), "cache_type": cache, "status": status,
    "kv_mib": float(kv) if kv else None,
    "kv_mib_predicted": float(predicted) if predicted else None,
    "memory_breakdown_mib": [line for line in breakdown.splitlines() if line.strip()],
    "log": log,
}))' "$MODEL" "$ctx" "$cache" "$status" "$kv_mib" "$predicted" "$breakdown" "$VERSION" "$log" >> "$LABBOOK"
  fi
done

echo "==> Full logs in $LOG_DIR${LABBOOK:+; one line per rung appended to $LABBOOK}"

Download kv-memory-ladder.sh133 lines

RunnableAll tracks

four loads, one after another

bash kv-memory-ladder.sh "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" labbook.md

Output — what you should see

==> Qwen3-8B-Q4_K_M with version: 0.4.0-dev (build xxxxx, commit xxxxxxx)
context  cache  status          KV MiB    predicted   device breakdown (total = free + (self = model + context + compute) + unaccounted)
4096     f16    loaded          xxx.xx       576.00   |   - CUDA0 (<device>)   | xxxxx = xxxxx + ( xxxx =  xxxx +   xxx +   xxx) +   xxx |
16384    f16    loaded         xxxx.xx      2304.00   |   - CUDA0 (<device>)   | ...
32768    f16    loaded         xxxx.xx      4608.00   |   - CUDA0 (<device>)   | ...
32768    q8_0   loaded         xxxx.xx      2448.00   |   - CUDA0 (<device>)   | ...
==> Full logs in ./kv-logs; one line per rung appended to labbook.md

How to read a row. KV MiB is parsed from the log line llama_kv_cache: size = … MiB (… cells, … layers, 1/1 seqs), K (f16): … MiB, V (f16): … MiB; it should equal predicted, and a mismatch means the model is not what you think it is. The breakdown row is llama.cpp’s own accounting of the device: model is the weights it placed there, context is the KV cache, compute the scratch buffers for the graph, and unaccounted whatever else holds device memory, the desktop included. On Track M the device is MTL0; on a CPU-only build there is only a Host row. A rung that does not fit shows failed with the log’s allocation line under it: that is the prediction being tested, so record it, and note that the q8_0 rung may load where the f16 one at the same length did not.

Now check the operating system agrees. In a second terminal, take the first reading from the tab for your track below; then start the 16,384-token rung by hand in the first terminal, leave it running, and take the second reading once it is listening.

RunnableAll tracks

terminal 1: one rung, held open

"$LLAMA_BIN/llama-server" -m "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" \
    -ngl 999 -c 16384 -np 1 -fa on --fit off --port 8089

Output — what you should see

...
x.xx.xxx.xxx W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
...
x.xx.xxx.xxx I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 16384, kv_unified = 'false'
x.xx.xxx.xxx I srv  llama_server: listening on http://127.0.0.1:8089

The CORS warning is about browsers reaching this port; the server is bound to 127.0.0.1 by default, so nothing beyond this machine can reach it. n_ctx_slot = 16384 confirms the context you asked for.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

terminal 2: the operating system's view

free -m

Once before terminal 1 starts and once after listening on appears. Record the rise in used and the fall in available on the Mem: row beside the ladder’s self figure for this rung; nvidia-smi cannot show device memory on this platform.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

terminal 2: the operating system's view

free -m

Once before terminal 1 starts and once after listening on appears. Record the rise in used and the fall in available on the Mem: row beside the ladder’s self figure for this rung.

Track M — Apple silicon

Open Activity Monitor, choose the Memory tab, and read Memory Used and Swap Used before starting terminal 1 and after listening on appears, and record the rise in Memory Used beside the ladder’s self figure. Swap Used climbing while the model is loaded means the working set was exceeded, and every number measured in that state describes paging.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

terminal 2: the operating system's view

nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Once before terminal 1 starts and once after listening on appears. The rise in memory.used should be at least the ladder’s self figure for this rung; the rest is the CUDA context and runtime, which the breakdown counts under unaccounted.

Stop terminal 1 with Ctrl+C. Memory is half of the story; the other half is speed, because every generated token’s attention reads the whole cache so far. Generation at depth t reads the weights W plus k × t bytes of cache, so the rate should fall by W / (W + k × t) whatever your bandwidth, a prediction that needs no hardware figure at all:

File	File size used as W (the shortcut), GB	Cache read at 4,096 tokens, GB	Predicted tg at 4,096 against depth 0	Cache read at 16,384, GB	Predicted at 16,384
Qwen3-8B-Q4_K_M	5.03	0.60	0.893	2.42	0.675
Qwen3-4B-Q4_K_M	2.50	0.60	0.805	2.42	0.508

The smaller model loses more, because its fixed weight term is smaller. -d prefills the cache before each timed repetition, outside the clock:

RunnableAll tracks

generation at three depths

"$LLAMA_BIN/llama-bench" -m "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" \
    -p 0 -n 128 -d 0,4096,16384 -r 5 -ngl 999 -fa on -o json \
    > "bench-results/depth-${QM}-Q4_K_M.json"
python3 bench-to-labbook.py --input "bench-results/depth-${QM}-Q4_K_M.json" \
    --labbook labbook.md --model-path "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf"

Output — what you should see

             tg128       xx.xx ± x.xx    tok/s   backend CUDA
       tg128@d4096       xx.xx ± x.xx    tok/s   backend CUDA
      tg128@d16384       xx.xx ± x.xx    tok/s   backend CUDA
    recorded 3 line(s) in labbook.md

The prediction counts bytes only. Attention also does arithmetic over every cached token, and the shorter a machine’s decode step, the larger the share of it that arithmetic takes, so a measured ratio below the prediction on a fast discrete card is that arithmetic showing. Task 8’s script prints both ratios.

Pending validationContext length against memory and speed, one model — your recording sheet

Context and cache type	Loaded?	KV cache logged, MiB	Predicted, MiB	Device self, MiB	tg128 at depth, tokens/s
4,096, f16	—	—	576.00	—	—
16,384, f16	—	—	2,304.00	—	—
32,768, f16	—	—	4,608.00	—	not measured
32,768, q8_0	—	—	2,448.00	—	not measured

your machine: track, chip and memory · llama.cpp the build number and commit from llama-cli --version · Qwen3-8B, or Qwen3-4B on the 8 GB tier, Q4_K_M weights; KV cache type as listed per row · 32,768 tokens of context · the date you ran it

Empty on purpose. The predicted column is the formula above, which is the same for both models. The last column takes the tg128@d4096 and tg128@d16384 rates from the depth run for the first two rows. Also record the operating system's before-and-after reading for the 16,384 rung.

8. Compare with the prediction

Part 5’s predict-decode.py turned your measured bandwidth into a ceiling: decode cannot outrun bandwidth divided by the bytes read per token. Here is that arithmetic for Qwen3-8B at Q4_K_M, using the vendor figures in the course hardware reference so the four tracks can be compared; your own ceiling uses the bandwidth you measured, which Part 5 set beside these figures:

Track, vendor bandwidth	Bytes read per token	Ceiling, tokens per second	Milliseconds per token at the ceiling
S, DGX Spark, 273 GB/s	5.03 GB	54.3	18.4
X, Ryzen AI Max+ 395, 256 GB/s	5.03 GB	50.9	19.6
M, M4 Max, 546 GB/s	5.03 GB	108.6	9.2
N, RTX 5090, 1,792 GB/s	5.03 GB	356.4	2.8

For a mixture of experts the shortcut multiplies the file by active over total parameters. Part 3 counted the tensors and showed where that is close and where it is not:

File	File, GB	Active over total	Shortcut, GB per token	Part 3’s tensor count, GB	Ceiling at 273 GB/s, shortcut
Qwen3-8B Q4_K_M (dense)	5.03	1	5.03	4.67	54.3
Qwen3-30B-A3B Q4_K_M	18.56	3.3 / 30.5	2.01	1.92	136.0
gpt-oss-20b MXFP4	12.11	3.6 / 21	2.08	not counted	131.5
gpt-oss-120b MXFP4	63.39	5.1 / 117	2.76	3.59	98.8
Qwen3-235B-A22B IQ4_XS	125.50	22 / 235	11.75	not counted	23.2

So the file-based ceiling is about 7 per cent low for a dense Qwen3, whose file includes the whole embedding table although a token reads one row of it (the true ceiling is about 8 per cent higher, 5.03 / 4.67 = 1.077), and about 30 per cent high for gpt-oss-120b, whose always-read tensors are Q8_0. Those two corrections bound what a correct measurement can look like.

Save ~/llm-course/summarise-benchmarks.py. It reads the notebook, keeps the latest full-offload measurement of each file, finds the file’s entry in models.json by name, and uses the bandwidth from your latest part-05/predict-decode line unless you pass --bandwidth-gbps.

RunnableAll tracks

summarise-benchmarks.py

#!/usr/bin/env python3
"""Summarise this lab's llama-bench records against the Part 5 decode prediction.

Purpose: read the notebook lines bench-to-labbook.py appended, keep the latest measurement
    of each model file on each backend, and print per model: the prompt-processing and
    generation rates with their spread, the ratio between the two, the decode ceiling Part 5's
    arithmetic gives for the exact file measured, the share of that ceiling reached, and the
    time per token the ceiling does not explain, and the arithmetic rate prefill implies.
    Generation measured at a depth is set beside the slowdown the KV cache formula
    predicts. With --format benchmark the same rows are printed as a <Benchmark> block,
    the course benchmark format.
Platform: all (pure Python standard library; nothing platform-specific)
Minimum memory: 8 GB
Assumes: Python 3.9 or later; labbook.md holding lines written by bench-to-labbook.py; the
    copy of the course's models.json that Part 5's predict-decode.py read; a bandwidth figure,
    passed with --bandwidth-gbps or taken from the latest part-05/predict-decode line.

Usage: python3 summarise-benchmarks.py --labbook labbook.md --models models.json
       python3 summarise-benchmarks.py --labbook labbook.md --models models.json \
           --format benchmark --hardware "Track S, DGX Spark, 128 GB" --os "DGX OS 7"

Rows are grouped by backend and file. A record made with GGML_CUDA_ENABLE_UNIFIED_MEMORY set
(bench-to-labbook.py writes the field) is shown as backend "+UMA", so a HIP run with the
variable never replaces the same file's run without it.

Method: ceiling = bandwidth / active bytes per token, where active bytes = the size
llama-bench reports for the file x active parameters / total parameters (the shortcut
predict-decode.py uses; 1 GB = 1e9 bytes). The unexplained time per token is
1 / measured rate - 1 / ceiling. Prefill TFLOP/s = prompt tokens per second x 2 x active
parameters. Depth prediction: rate(t) / rate(0) = W / (W + k x t), with W the active bytes
and k the KV bytes per token from models.json, scaled for the cache type.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

LAB = "part-06/lab-benchmark-the-reference-models"
# Bytes per cached element, from the ggml block layouts: f16 is 2 bytes; q8_0 stores 32
# values in 34 bytes; q4_0 stores 32 in 18.
BYTES_PER_ELEMENT = {"f32": 4.0, "f16": 2.0, "bf16": 2.0, "q8_0": 34 / 32, "q4_0": 18 / 32}


def read_records(path: Path) -> list[dict]:
    records = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line.startswith("{"):
            continue  # prose, headings and tables in the notebook
        try:
            obj = json.loads(line)
        except json.JSONDecodeError:
            continue
        if isinstance(obj, dict):
            records.append(obj)
    return records


def match_model(catalogue: list[dict], model_path: str) -> dict | None:
    """The catalogue entry whose name starts the file name, longest name first."""
    base = Path(model_path or "").name.lower()
    hits = [m for m in catalogue if base.startswith(str(m.get("name", "")).lower() + "-")]
    return max(hits, key=lambda m: len(m["name"])) if hits else None


def backend_label(record: dict):
    """The backend string, marked when llama.cpp ran with GGML_CUDA_ENABLE_UNIFIED_MEMORY set."""
    backend = record.get("backend")
    if record.get("ggml_cuda_enable_unified_memory") is not None:
        return f"{backend}+UMA"
    return backend


def fmt(value, digits=2, width=8) -> str:
    return f"{value:>{width}.{digits}f}" if isinstance(value, (int, float)) else f"{'-':>{width}}"


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", required=True, help="the notebook bench-to-labbook.py wrote to")
    parser.add_argument("--models", help="the course models.json (needed for ceilings)")
    parser.add_argument("--bandwidth-gbps", type=float, help="measured bandwidth in GB/s (Part 5)")
    parser.add_argument("--format", choices=["table", "benchmark"], default="table")
    parser.add_argument("--hardware", default="FILL-IN: track, machine, memory")
    parser.add_argument("--os", default="FILL-IN: operating system and version")
    parser.add_argument("--record", action="store_true",
                        help="append one part-06/compare-with-prediction line to the notebook")
    args = parser.parse_args()

    notebook = Path(args.labbook)
    if not notebook.exists():
        print(f"{notebook} does not exist", file=sys.stderr)
        return 1
    records = read_records(notebook)
    bench = [r for r in records if r.get("lab") == LAB and isinstance(r.get("tokens_per_s"), (int, float))]
    if not bench:
        print(f"no {LAB} lines with a rate in {notebook}; run the sweep first", file=sys.stderr)
        return 1

    bandwidth, source = args.bandwidth_gbps, "--bandwidth-gbps"
    if bandwidth is None:
        for r in records:
            if r.get("lab") == "part-05/predict-decode" and r.get("bandwidth_gbps"):
                bandwidth, source = float(r["bandwidth_gbps"]), "the latest part-05/predict-decode line"
    catalogue = []
    if args.models:
        catalogue = json.loads(Path(args.models).read_text(encoding="utf-8")).get("models", [])

    # A partial-offload experiment on the same file is a different measurement: keep only the
    # records made with the largest layer count asked for on each file and backend.
    most_layers: dict[tuple, int] = {}
    for r in bench:
        key = (backend_label(r), Path(str(r.get("model_path") or r.get("model"))).name)
        most_layers[key] = max(most_layers.get(key, -1), int(r.get("n_gpu_layers") or 0))
    bench = [r for r in bench if int(r.get("n_gpu_layers") or 0) == most_layers[
        (backend_label(r), Path(str(r.get("model_path") or r.get("model"))).name)]]

    # Latest record per (backend, file, test); later lines in the notebook win.
    latest: dict[tuple, dict] = {}
    for r in bench:
        key = (backend_label(r), Path(str(r.get("model_path") or r.get("model"))).name, r.get("test"))
        latest[key] = r
    groups: dict[tuple, dict] = {}
    for (backend, name, test), r in latest.items():
        groups.setdefault((backend, name), {})[test] = r

    rows, depth_rows = [], []
    for (backend, name), tests in sorted(groups.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
        # At depth 0, the most recently measured prompt-processing and generation tests.
        by_time = sorted(tests.items(), key=lambda kv: str(kv[1].get("measured_on") or ""))
        pp = next((r for t, r in reversed(by_time) if t.startswith("pp") and "@" not in t and "+" not in t), None)
        tg = next((r for t, r in reversed(by_time) if t.startswith("tg") and "@" not in t), None)
        any_row = tg or pp or next(iter(tests.values()))
        entry = match_model(catalogue, any_row.get("model_path") or name)
        size_bytes = any_row.get("model_size_bytes") or (any_row.get("file_gb") or 0) * 1e9
        active_gb = ceiling = unexplained = reached = prefill_tflops = None
        if entry and size_bytes:
            params = entry.get("params", {})
            if pp and params.get("activeB"):
                # Two floating-point operations per active parameter per token, as in Part 3.
                prefill_tflops = pp["tokens_per_s"] * 2 * float(params["activeB"]) * 1e9 / 1e12
            fraction = min(float(params.get("activeB", 0)) / float(params.get("totalB", 1)), 1.0) or 1.0
            active_gb = size_bytes * fraction / 1e9
            if bandwidth:
                ceiling = bandwidth / active_gb
        rate = tg["tokens_per_s"] if tg else None
        if rate and ceiling:
            reached = rate / ceiling
            unexplained = 1000.0 / rate - 1000.0 / ceiling
        spread = tg.get("tokens_per_s_stddev") if tg else None
        rows.append({
            "backend": backend, "file": name, "id": entry.get("id") if entry else None,
            "quant": any_row.get("quant"), "file_gb": round(size_bytes / 1e9, 2) if size_bytes else None,
            "active_gb": round(active_gb, 2) if active_gb else None,
            "pp_test": pp.get("test") if pp else None, "pp": pp["tokens_per_s"] if pp else None,
            "tg_test": tg.get("test") if tg else None, "tg": rate, "tg_stddev": spread,
            "tg_cv_pct": 100.0 * spread / rate if rate and isinstance(spread, (int, float)) else None,
            "pp_over_tg": pp["tokens_per_s"] / rate if pp and rate else None,
            "ceiling": ceiling, "reached": reached, "unexplained_ms": unexplained,
            "prefill_tflops": prefill_tflops,
            "build": any_row.get("build"), "build_number": any_row.get("build_number"),
            "measured_on": any_row.get("measured_on"),
            "context": max(int(r.get("n_prompt") or 0) + int(r.get("n_gen") or 0) + int(r.get("n_depth") or 0)
                           for r in tests.values()),
        })
        for test, r in sorted(tests.items(), key=lambda kv: int(kv[1].get("n_depth") or 0)):
            depth = int(r.get("n_depth") or 0)
            base = tests.get(f"tg{r.get('n_gen')}")  # the same generation test at depth 0
            if not test.startswith("tg") or depth == 0 or not base:
                continue
            predicted = None
            if entry and active_gb and entry.get("kv", {}).get("bytesPerTokenFp16"):
                per_element = BYTES_PER_ELEMENT.get(str(r.get("type_k") or "f16"), 2.0)
                k = entry["kv"]["bytesPerTokenFp16"] / 2.0 * per_element
                predicted = active_gb * 1e9 / (active_gb * 1e9 + k * depth)
            depth_rows.append({"file": name, "backend": backend, "test": test, "rate": r["tokens_per_s"],
                               "measured_ratio": r["tokens_per_s"] / base["tokens_per_s"], "predicted_ratio": predicted,
                               "architecture": entry.get("architecture") if entry else None})

    if args.format == "benchmark":
        context = max(r["context"] for r in rows)
        builds = sorted({f"build {r['build_number']} ({r['build']})" for r in rows})
        dates = sorted(str(r["measured_on"])[:10] for r in rows if r["measured_on"])
        print("<Benchmark")
        print('  title="llama-bench, prompt processing and generation, one row per model file"')
        print("  columns={['Model file', 'Backend', 'File GB', 'pp tokens/s', 'tg tokens/s', "
              "'tg ceiling tokens/s', 'Share of ceiling']}")
        print("  rows={[")
        for r in rows:
            cells = [r["file"], r["backend"], r["file_gb"],
                     round(r["pp"], 1) if r["pp"] else "-", round(r["tg"], 1) if r["tg"] else "-",
                     round(r["ceiling"], 1) if r["ceiling"] else "-",
                     f"{100 * r['reached']:.0f} per cent" if r["reached"] else "-"]
            print("    " + json.dumps(cells).replace('"', "'") + ",")
        print("  ]}")
        print(f"  context={{{{ hardware: '{args.hardware}', os: '{args.os}', engine: 'llama.cpp',")
        print(f"             version: '{', '.join(builds)}', model: 'as listed per row',")
        print(f"             quant: 'as listed per row', contextLength: {context},")
        print(f"             date: '{dates[-1] if dates else 'FILL-IN'}' }}}}")
        print('  status="measured"')
        print(f"  note=\"Ceiling from {bandwidth} GB/s ({source}) and each file's size times its active "
              "fraction.\" />" if bandwidth else '  note="No bandwidth given: ceilings not computed." />')
    else:
        print(f"bandwidth: {bandwidth} GB/s from {source}" if bandwidth
              else "bandwidth: none given, so no ceilings (pass --bandwidth-gbps or run predict-decode.py)")
        print("ceiling = bandwidth / (file bytes x active/total parameters); 1 GB = 1e9 bytes\n")
        header = (f"{'model file':<40} {'backend':<9} {'GB':>6} {'act GB':>6} {'pp t/s':>8} {'tg t/s':>8}"
                  f" {'tg cv%':>6} {'pp/tg':>6} {'ceiling':>8} {'reached':>7} {'gap ms':>8} {'pp TFLOP/s':>10}")
        print(header)
        print("-" * len(header))
        for r in rows:
            print(f"{r['file'][:40]:<40} {str(r['backend'])[:9]:<9} {fmt(r['file_gb'], 2, 6)} {fmt(r['active_gb'], 2, 6)}"
                  f" {fmt(r['pp'], 1)} {fmt(r['tg'], 2)} {fmt(r['tg_cv_pct'], 1, 6)} {fmt(r['pp_over_tg'], 1, 6)}"
                  f" {fmt(r['ceiling'], 1)} {fmt(r['reached'], 2, 7)} {fmt(r['unexplained_ms'], 2)}"
                  f" {fmt(r['prefill_tflops'], 2, 10)}")
            if r["id"] is None and catalogue:
                print(f"    {r['file']}: no models.json entry matches this file name; no ceiling")
            if r["reached"] and r["reached"] > 1.0:
                print("    above the ceiling: an input is wrong (bandwidth figure, file, or active bytes)")
            elif r["reached"] and r["reached"] < 1 / 3:
                print("    below a third of the ceiling: follow the challenge page's procedure")
            if r["tg_cv_pct"] and r["tg_cv_pct"] > 3.0:
                print("    generation spread above 3 per cent: something else was using the machine")
        if depth_rows:
            print(f"\n{'model file':<40} {'test':<14} {'tg t/s':>8} {'measured':>9} {'predicted':>9}")
            for d in depth_rows:
                note = "" if d["architecture"] in ("dense", None) else "  (formula assumes every layer caches every token)"
                print(f"{d['file'][:40]:<40} {d['test']:<14} {fmt(d['rate'], 2)} {fmt(d['measured_ratio'], 3, 9)}"
                      f" {fmt(d['predicted_ratio'], 3, 9)}{note}")

    if args.record:
        with notebook.open("a", encoding="utf-8") as handle:
            handle.write(json.dumps({"lab": "part-06/compare-with-prediction", "bandwidth_gbps": bandwidth,
                                     "bandwidth_source": source if bandwidth else None,
                                     "rows": rows, "depth_rows": depth_rows}) + "\n")
        print(f"\nrecorded in {notebook}", file=sys.stderr)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download summarise-benchmarks.py244 lines

RunnableAll tracks

every model, beside its ceiling

python3 summarise-benchmarks.py --labbook labbook.md --models "$MODELS_JSON"

Output — what you should see

bandwidth: xxx.x GB/s from the latest part-05/predict-decode line
ceiling = bandwidth / (file bytes x active/total parameters); 1 GB = 1e9 bytes

model file                               backend       GB act GB   pp t/s   tg t/s tg cv%  pp/tg  ceiling reached   gap ms pp TFLOP/s
-------------------------------------------------------------------------------------------------------------------------------------
Qwen3-8B-Q4_K_M.gguf                     CUDA        x.xx   x.xx   xxxx.x    xx.xx    x.x   xx.x     xx.x    x.xx     x.xx      xx.xx
...

model file                               test             tg t/s  measured predicted
Qwen3-8B-Q4_K_M.gguf                     tg128@d4096       xx.xx     x.xxx     0.893
Qwen3-8B-Q4_K_M.gguf                     tg128@d16384      xx.xx     x.xxx     0.675

What each computed column means, and what to do with it:

Column	Arithmetic	Reading
act GB	tensor bytes × active ÷ total parameters	the shortcut above; for gpt-oss use Part 3’s count when you judge the gap
ceiling	bandwidth ÷ act GB	the most decode the bytes allow
reached	measured tg ÷ ceiling	above 1.00: an input is wrong. Down to about 0.33: an ordinary gap. Below a third: a fault until proven otherwise
gap ms	1000 ÷ measured − 1000 ÷ ceiling	the time per token left once the bytes are accounted for: arithmetic, kernel launches and synchronisation (llama-bench excludes sampling)
pp/tg	the two rates’ ratio	Part 3’s crossover n*, capped at 512
pp TFLOP/s	pp rate × 2 × active parameters	arithmetic prefill delivered; set it beside Part 5’s matmul-test FP16 figure

The gap ms column explains the pattern across tracks better than the ratio. Take a fixed 2 ms per token beyond the bytes, as arithmetic rather than a measurement: it turns the Spark’s 18.4 ms ceiling step for Qwen3-8B into 20.4 ms, 90 per cent of the ceiling, and the RTX 5090’s 2.8 ms step into 4.8 ms, 58 per cent. The same overhead costs the faster machine a larger share, so a discrete card can reach less of its ceiling on a small model with nothing wrong. A gap growing to tens of milliseconds on any machine is the symptom the challenge page diagnoses.

=Expected result

The mixture-of-experts files should generate far faster than their size suggests: Qwen3-30B-A3B at Q4_K_M is almost the size of Qwen3-32B at Q4_K_M and its ceiling is about ten times higher. Among the dense files, the larger ones should reach a larger share of their ceilings, because a fixed overhead is a smaller part of a longer step. If a mixture-of-experts file reaches a much smaller share than a dense file of similar step time, check that every layer was offloaded before blaming the architecture.

Then write the rows in the course benchmark format and add a summary line to the notebook. First describe your machine in two variables; each tab’s values are an example to edit, not a default:

Track S — NVIDIA DGX Spark

Fragment — not complete on its own

Track S: edit both values

HARDWARE="Track S, DGX Spark (GB10), 128 GB"
OS_NAME="DGX OS 7.x"        # your exact version

Track X — AMD Ryzen AI Max+ 395

Fragment — not complete on its own

Track X: edit both values

HARDWARE="Track X, GMKtec EVO-X2, Ryzen AI Max+ 395, 128 GB"
OS_NAME="Ubuntu 24.04.x"    # grep PRETTY_NAME /etc/os-release prints yours

Track M — Apple silicon

Fragment — not complete on its own

Track M: edit both values

HARDWARE="Track M, Mac Studio, M4 Max, 128 GB"
OS_NAME="macOS 26.x"        # sw_vers -productVersion prints yours

Track N — NVIDIA desktop or laptop

Fragment — not complete on its own

Track N: edit both values

HARDWARE="Track N, RTX 4090 24 GB, Ryzen 9 7950X, 64 GB"
OS_NAME="Ubuntu 24.04.x"    # grep PRETTY_NAME /etc/os-release prints yours

RunnableAll tracks

the same rows as a Benchmark block, and a record

python3 summarise-benchmarks.py --labbook labbook.md --models "$MODELS_JSON" \
    --format benchmark --hardware "${HARDWARE:?set HARDWARE first}" \
    --os "${OS_NAME:?set OS_NAME first}" --record

Output — what you should see

<Benchmark
  title="llama-bench, prompt processing and generation, one row per model file"
  columns={['Model file', 'Backend', 'File GB', 'pp tokens/s', 'tg tokens/s', 'tg ceiling tokens/s', 'Share of ceiling']}
  rows={[
    ['Qwen3-8B-Q4_K_M.gguf', 'CUDA', x.xx, xxxx.x, xx.x, xx.x, 'xx per cent'],
    ...
  ]}
  context={{ hardware: '<the value of HARDWARE>', os: '<the value of OS_NAME>', engine: 'llama.cpp',
             version: 'build xxxxx (xxxxxxx)', model: 'as listed per row',
             quant: 'as listed per row', contextLength: 16512,
             date: '2026-xx-xx' }}
  status="measured"
  note="Ceiling from xxx.x GB/s (the latest part-05/predict-decode line) and each file's size times its active fraction." />

That block is the course benchmark format: every number carries the context that makes it judgeable, hardware, engine, version, model, quantisation, context length and date, which the <Benchmark> component refuses to render without, plus the operating system. contextLength is the largest n_prompt + n_gen + n_depth any row used, so the depth run raises it to 16,512, which llama.cpp allocated as 16,640 after padding. Paste the block into the notebook, and write one sentence per model under it: the share of the ceiling reached, and what you think explains the gap.

9. Optional: what not fitting costs, measured

A model that does not fit is not a slightly slower model. You can measure why without risking swap, by asking for fewer layers on the accelerator than a model that fits has. Qwen3-8B and Qwen3-4B have 36 repeating layers plus the output layer, so -ngl 37 is all of them, 27 keeps the first 10 layers on the CPU, 18 keeps 19, and 0 keeps every layer on the CPU.

RunnableAll tracks

four offload levels, one model

echo "$QM"
"$LLAMA_BIN/llama-bench" -m "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" \
    -p 0 -n 128 -r 3 -ngl 37,27,18,0 -fa on

Output — what you should see

Qwen3-8B
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |  37 |   1 |           tg128 |           xx.xx ± x.xx |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |  27 |   1 |           tg128 |           xx.xx ± x.xx |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |  18 |   1 |           tg128 |            x.xx ± x.xx |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |   0 |   1 |           tg128 |            x.xx ± x.xx |

An empty first line means this terminal has no QM: run the preflight’s QM block. On a Mac the table has a threads column and no ngl column, because the backend string contains BLAS; the rows appear in the order given, 37, 27, 18, 0.

Part 3’s split-memory arithmetic predicts the shape: with a fraction f of the bytes read on a path r times slower, the step takes (1 − f) + f × r times as long. On Track N the slow path is the CPU computing from system memory, far below the card’s bandwidth; on the unified-memory tracks there is no copy but the CPU still does that arithmetic. Rearranged for what your rows imply:

slowdown = tg128 rate at -ngl 37 / tg128 rate at -ngl 27 (or 18)
slowdown ≈ (1 − f) + f × r        so        r = (slowdown − (1 − f)) / f
-ngl 27: f = 10/36 = 0.278        -ngl 18: f = 19/36 = 0.528

The build guide’s general note on GPU-accelerated backends says the GPU may still accelerate parts of the computation with -ngl 0, and that --device none disables it fully, so the -ngl 0 row is mostly-CPU, not a pure CPU baseline. Its Metal section says instead that --n-gpu-layers 0 disables GPU inference; on Track M the two notes disagree, and one more row settles it:

RunnableAll tracks

the same test with no GPU device at all

"$LLAMA_BIN/llama-bench" -m "$M/unsloth/${QM}-GGUF/${QM}-Q4_K_M.gguf" \
    -p 0 -n 128 -r 3 -ngl 0 -dev none -fa on

Output — what you should see

| model                          |       size |     params | backend    | ngl |  fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | --------------: | -------------------: |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |   0 |   1 | none         |           tg128 |            x.xx ± x.xx |

The backend column still names the backend compiled in; the dev column reading none is what says no GPU device was used. Record the five rates and, for the -ngl 27 and -ngl 18 rows, the slowdown and the r it implies. The notebook is not given these rows on purpose: they are an experiment on one file, not its benchmark.

Separate engine speed, memory fit and application latency

First run one benchmark manually and identify the prompt-processing and generation rows. Verify the checkpoint, quantisation and device placement in the log. Only then launch the sweep, with distinct output files for every changed configuration.

Before accepting the summary, check that each expected model produced the intended repeats and that failures were retained. A missing row is not a zero-time run. Compare warm runs with warm runs and cold loading separately. For the context ladder, record actual cache allocation and peak memory, not only the context requested on the command line.

Use the offload experiment to test the memory-budget hypothesis. Keep the same checkpoint when changing placement so you can attribute the slowdown. Then compare with a smaller fully resident model as a separate deployment choice. The benchmark reports engine behaviour under a fixed token workload; the application may add queueing, retrieval and different output lengths. Preserve that distinction when transferring these numbers into your model shortlist and later gateway capacity plan.

Validation

Run these from ~/llm-course. Each line says what passing prints.

Check	Command	Pass
The build is recorded	grep -A 12 '^## Environment' labbook.md	the version: line and the device line are there
Every model in your set has both tests	grep -c '"lab": "part-06/lab-benchmark-the-reference-models"' labbook.md	at least two lines per file in bench-set.txt that loaded, plus eight from task 6 and three from task 7
No CPU rows	grep '"lab": "part-06/lab-benchmark-the-reference-models"' labbook.md | grep -c '"backend": "CPU"'	0
Raw results kept	ls bench-results/*.json | wc -l	one per file in bench-set.txt, four from task 6, and the depth file, plus one per file of each Track X HIP sweep
The ladder ran	grep -c '"lab": "part-06/kv-memory-ladder"' labbook.md	4, and every loaded rung has kv_mib equal to kv_mib_predicted
The comparison ran	python3 summarise-benchmarks.py --labbook labbook.md --models "$MODELS_JSON"	exit status 0; a ceiling for every file whose name is in models.json
The record exists	grep -c '"lab": "part-06/compare-with-prediction"' labbook.md	at least 1
The noise is acceptable	the tg cv% column of the same output	3.0 or less on every row, or a re-run recorded

Beyond the commands: the three recording sheets are filled, every model that failed is listed with its allocation line, and there is one sentence per model on the gap between the measurement and its ceiling.

Expected outcome

A section of your notebook that describes your machine’s behaviour rather than its specification: a prefill rate and a decode rate with their spread for every reference model that fits, the share of the bandwidth ceiling each reached, the exchange rate between quantisation and speed for one model, the KV cache cost of four context settings confirmed against the formula, and the slowdown at depth. Some shapes are common enough to predict, and noticing them is part of the point.

Prompt processing should be much faster than generation on every machine and model.

Generation tracks the bytes read per token within a family. A dense model’s Q8_0 file should generate at close to 0.58 of its Q4_K_M rate, until something else, memory headroom or dequantisation arithmetic, becomes the limit.

Troubleshooting

Symptom	Cause	Fix
Available devices: then (none), or backend reads CPU, or WARNING: llama-bench reports the CPU backend with nothing recorded	a CPU-only build, or the GPU runtime did not load; bench-to-labbook.py refuses such a result and writes nothing	fix the build with the install lesson’s backend check and the challenge page, then re-run the sweep; --allow-cpu is only for a deliberate CPU run you want in the notebook
bench-reference-models: llama-bench not found at …	LLAMA_BIN unset in this terminal, or a different build directory	run the preflight block again; on Track X check build against build-hip
error: invalid parameter for argument: --version from llama-bench	llama-bench has no version option at v0.4.0	take the build from llama-cli --version or the table’s build: line
FAILED with cudaMalloc failed: out of memory, unable to allocate CUDA0 buffer, failed to allocate buffer or OutOfDeviceMemory in the .log	the model does not fit the device	expected past your tier: record it. Otherwise compare the file’s MiB free needed with --list-devices, close other programs, and on Track X check the GPU-visible limit
FAILED with llama_bench: error: failed to load model '…' and no allocation line	a damaged or incomplete file, or a later shard given alone	bash verify-library.sh labbook.md; for the 235B pass the -00001-of-00003 shard
bench-results/….json: not valid JSON (…) from bench-to-labbook.py	llama-bench stopped partway, leaving an empty or half-written file	read the .log beside it; re-run that model once the cause is fixed
tg cv% above 3, or a ± large against its rate	background work, thermal limits, or battery power	close everything, plug in, wait for the machine to cool, re-run the model; record both runs
Large ± on pp512 only	prompt processing is compute-bound and is the first to suffer when clocks drop	check the power source and temperature
reached above 1.00	the bandwidth figure is the vendor peak rather than your measurement, the file differs from the one predicted, or the active-bytes shortcut over-counts (dense Qwen3’s bytes by about 8 per cent, 5.03 / 4.67, which puts the file-based ceiling about 7 per cent low)	pass --bandwidth-gbps with Part 5’s measured figure and re-read task 8’s second table
reached below a third on every model	layers not offloaded, the wrong build, or swapping	the challenge page
no models.json entry matches this file name	a file outside the reference set, or a renamed file	keep the Hub file name; the script matches the name at the start of the file name
kv-memory-ladder: something is already answering on 127.0.0.1:8089	a server from an earlier lesson or from task 7’s manual run is still up	stop it with Ctrl+C in its terminal, or run the script with PORT=8090 in front
A ladder rung shows timeout	a very large model loading from a slow disk takes longer than 600 seconds	re-run with LOAD_TIMEOUT=1800 in front of the command
common_fit_params: failed to fit params to free device memory in a manual server log	--fit on could not change settings you set yourself	expected when -c and -ngl are explicit; pass --fit off as the page does
verify-library.sh lists the 235B shards as FAILED, prints 3 failed and exits with status 1	fetch-model.sh names each shard IQ4_XS/… in its checksum file, and verify-library.sh checks from the shard’s own directory	check them with task 2’s command from the repository directory and note the result beside the part-04/verify-library line
Track M: fetch-model.sh stops at once with a line ending AUTH_ARGS[@]: unbound variable (wording may differ)	macOS’s /bin/bash 3.2 treats an empty array under set -u as unset	use the corrected fetch-model.sh from Part 4; the copy dated 2026-09-13 has this fault
Track N native Windows: a model past your tier runs very slowly instead of failing	the driver’s System Memory Fallback	turn it off in the NVIDIA control panel for benchmarking, or record the result as spilled
Track M: a large model loads and the machine becomes unresponsive	the working set was exceeded and macOS is paging	stop it, drop a row, record the substitution; the challenge page treats it

Cleanup

Keep the notebook, bench-results/, kv-logs/, bench-set.txt and the scripts: Part 7 and Part 9 benchmark the same files with other engines against these rows, and the challenge page starts from this table. Keep the models too if you have the disk; Qwen3-8B at Q4_K_M returns in almost every later part. Make sure no server is left running: every terminal you started llama-server in should be back at a prompt.

If you need the disk back, remove the task 6 quantisations you will not use again, largest first, and write in the notebook which files you removed, so a later comparison does not quietly become a different model set.

⚠Warning

The commands below delete a downloaded file permanently, together with its checksum and the hf record that marks it complete; the only way back is to download it again. They remove the Q8_0 and Q6_K files of the model in QM and nothing else. Check the value first: echo "$QM" must print Qwen3-8B or Qwen3-4B; if it prints nothing, run the preflight’s QM block before this one.

RunnableAll tracks

remove two task 6 quantisations and their companions

echo "$QM"
ls -lh "$M/unsloth/${QM}-GGUF/"
rm -f "$M/unsloth/${QM}-GGUF/${QM}-Q8_0.gguf" "$M/unsloth/${QM}-GGUF/${QM}-Q8_0.gguf.sha256"
rm -f "$M/unsloth/${QM}-GGUF/.cache/huggingface/download/${QM}-Q8_0.gguf.metadata"
rm -f "$M/unsloth/${QM}-GGUF/${QM}-Q6_K.gguf" "$M/unsloth/${QM}-GGUF/${QM}-Q6_K.gguf.sha256"
rm -f "$M/unsloth/${QM}-GGUF/.cache/huggingface/download/${QM}-Q6_K.gguf.metadata"

What you learned

Objective	The observation that proved it	Recorded as
Prefill and decode are different measurements	pp512 many times tg128 on every model, the ratio near Part 3’s crossover	the two rates, their ± and pp/tg per model
A methodology is part of a result	task 3’s recomputation of t/s and ± from the timings, and task 4’s table of fixed settings	build, backend, ngl, fa, repetitions in every notebook line
Quantisation buys speed as well as space	generation against Q4_K_M set beside the inverse size ratio, with IQ4_XS as the test of bytes against arithmetic	the quantisation sheet and eight notebook lines
Context is memory spent at load, and speed spent per token	the logged KV size equal to the formula, the failed or loaded rungs matching your prediction, and the rate at depth beside W / (W + k × t)	four part-06/kv-memory-ladder lines, the context sheet, three depth lines
Sparse models change the arithmetic	Qwen3-30B-A3B’s ceiling about ten times Qwen3-32B’s at nearly the same file size, and its measured rate following	the act GB, ceiling and reached columns
A prediction makes a measurement useful	reached and gap ms for every file, with one sentence each on the cause	the part-06/compare-with-prediction line and the <Benchmark> block

Check your understanding

Question 1. Your table shows pp512 at thirty times the rate of tg128 for Qwen3-8B. What does the ratio tell you, by Part 3's arithmetic?

• The benchmark is misconfigured, because the two rates should be equal
• That prefill on this machine turns compute-bound at a prompt of roughly thirty tokens for this model, well below the 512 tested
• That the model is quantised too heavily
• That the GPU is being shared with another process

Show the answer and why

Answer: That prefill on this machine turns compute-bound at a prompt of roughly thirty tokens for this model, well below the 512 tested

Generation runs at about bandwidth over bytes per token; prefill past the crossover runs at about arithmetic rate over 2N. Their ratio is W × C / (2 × N × B), which is the crossover prompt length n*, capped at the prompt length tested. A ratio near 512 would instead say prefill was still waiting on memory.

Question 2. Qwen3-8B at Q4_K_M loads with a KV cache of 576.00 MiB at -c 4096. What does the same server log at -c 32768 with --cache-type-k q8_0 --cache-type-v q8_0?

• 576.00 MiB, because the cache grows as the conversation fills it
• 4,608.00 MiB
• 2,448.00 MiB
• 1,152.00 MiB

Show the answer and why

Answer: 2,448.00 MiB

llama.cpp allocates the whole cache at load, for the context asked. Eight times the context is 4,608.00 MiB at f16; q8_0 stores 32 values in 34 bytes, 1.0625 bytes per element against 2, so 4,608 × 34 / 64 = 2,448.00 MiB. It is slightly more than half, not half, because of the per-block scale.

Question 3. Which of these commands measures something other than the lab methodology, without saying so in its output?

• llama-bench -m model.gguf -p 512 -n 128 -r 5 -ngl 999 -fa on
• llama-bench -m model.gguf -p 512 -n 128 -r 5 -ngl 20 -fa on, for a 36-layer model that fits
• llama-bench -m model.gguf -p 512 -n 128 -r 5 -ngl 999 -fa on -o json
• llama-bench -m model.gguf -p 0 -n 128 -d 0,4096 -r 5 -ngl 999 -fa on

Show the answer and why

Answer: llama-bench -m model.gguf -p 512 -n 128 -r 5 -ngl 20 -fa on, for a 36-layer model that fits

With 20 of 37 offloadable layers requested, 17 run on the CPU and every token passes through them, so the rate mixes two machines. The ngl column shows 20, but nothing flags it as a problem. The JSON variant is the same measurement in another format, and the depth run is a different test that labels itself tg128 @ d4096.

Question 4. Your Spark's measured bandwidth gives Qwen3-8B Q4_K_M a ceiling from the file size, and the measured tg128 is 5 per cent above it. Which explanations are consistent with the arithmetic? Select all that apply.

• The file-based estimate counts the whole embedding table, of which one row is read per token, so the true bytes read are about 7 per cent lower
• The bandwidth figure used was lower than what decode actually achieves on this machine
• The measurement is impossible and must be discarded
• The prediction was made for a different file from the one benchmarked

Show the answer and why

Answer: The file-based estimate counts the whole embedding table, of which one row is read per token, so the true bytes read are about 7 per cent lower, The bandwidth figure used was lower than what decode actually achieves on this machine, The prediction was made for a different file from the one benchmarked

A ceiling can only be exceeded when an input is wrong. Part 3 counted 4.67 GB read per token against a 5.03 GB file, which alone allows a rate about 8 per cent above the file-based figure. The bandwidth measurement and the file identity are the other two inputs. "Impossible" is not a diagnosis.

Question 5. Two configurations of the same model give tg128 rows of 41.0 ± 0.5 and 41.6 ± 0.6, each from five repetitions. Is the second one faster?

• Yes, it is 0.6 higher
• Not shown: the 95 per cent intervals for the means, about ± 0.62 and ± 0.74, overlap
• Yes, because both standard deviations are below 3 per cent
• No, the first is faster

Show the answer and why

Answer: Not shown: the 95 per cent intervals for the means, about ± 0.62 and ± 0.74, overlap

The ± is a sample standard deviation. The interval for a mean of five repetitions is 2.776 × sd / √5: 0.62 and 0.74 here, so the ranges 40.38 to 41.62 and 40.86 to 42.34 overlap. A low coefficient of variation says each run was quiet, not that the difference between them is real.

Check my answersClear