Lab: Measure Your Memory Bandwidth and Compute

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track versions and measured figures go here once the validation pass has run these scripts.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this lab you will have measured, on your own machine, the two numbers the first lesson of this part said govern everything, and put them to work:

• memory bandwidth on the accelerator and on the CPU, by two methods, at three buffer sizes, each with its run-to-run spread;
• matrix-multiply throughput at FP32, BF16 and FP16, and at two lower precisions: FP8 where the PyTorch build has a kernel for it, and weight-only INT8 everywhere; on Track M, also MLX’s quantised kernels at 8 and 4 bits;
• both set beside the vendor’s published figures, as a ratio, with the cause of the gap you think dominates and the check that supports it;
• six decode ceilings for three models you have not downloaded, each with a verdict on whether it fits your memory budget, written down before Part 6 runs them;
• a dated notebook section and JSON lines that Part 6’s benchmark lab and Part 12’s compute budget read back.

A prediction written before the measurement is a different thing from an explanation offered afterwards, and this lab is where you make the first kind.

What runs where

Task	Script	Runs on	Produces	Notebook line
2	bandwidth-test.py	the accelerator PyTorch sees, then the CPU	copy and read GB/s with spread	part-05/bandwidth-test
3	mlx-bandwidth-test.py	Track M only: the GPU through MLX	the same two figures, and MLX matrix multiplies including quantised ones	part-05/mlx-bandwidth-test
4	matmul-test.py	the accelerator, or the CPU if PyTorch sees none	TFLOPS per format	part-05/matmul-test
6	predict-decode.py	anywhere; it is arithmetic	ceilings at two context depths, and fit	part-05/predict-decode
Validation	check-part05.py	anywhere	pass or fail, the spec ratio and the ridge point	none

⌂Single-machine path

Single machine on every track. Nothing here needs a model, a network or a second computer.

Requirements

Part 1’s lab (the ~/llm-course environment with PyTorch and labbook.md) and this part’s prepare lab, whose task 4 set the GPU memory limit you use in task 6. Pinned versions: PyTorch PyTorch 2.14.0 · verified 2026-09-12 and, on Track M, MLX MLX 0.32.2 · verified 2026-09-12. No model is downloaded: the four scripts and models.json are under 100 KB together, and nothing is written except the notebook.

The memory floor is 8 GB with no reduced path. The largest allocation is task 2’s 1,024 MiB run: two 1,024 MiB buffers on the accelerator, then the same on the CPU, released in between. If an allocation fails, halve --size-mb for that run and record that you did.

Track	Device the scripts use	Extra for this lab	Attended	Unattended (script run time)
S	cuda, NVIDIA GB10	none	60 min	under 5 min
X, ROCm build	cuda, an AMD Radeon name	none	60 min	under 5 min
X, CPU build	cpu	none	60 min	up to 15 min
M	mps, and MLX’s gpu	mlx-bandwidth-test.py	70 min	under 8 min
N	cuda, your card’s name	none	60 min	under 5 min

Both time columns are estimates, not measurements; attended time is mostly reading the mechanism sections and filling in task 7. The unattended column is bounded by the scripts’ own caps: matmul-test.py gives each format about 30 seconds after a warm-up and a sizing multiplication, so even a CPU doing every format at both sizes finishes within the CPU row, and a GPU finishes far inside it.

Track S — NVIDIA DGX Spark

PyTorch from Part 1, either path. Container: from ~/llm-course, run TAG=26.08-py3 bash setup-env-spark.sh and do every task in the shell it opens, where the course directory is /workspace/course, python is the container’s own, and labbook.md is the same file as on the host. Wheel: activate ~/llm-course/.venv as the commands below do. The GPU and the Arm CPU cores read the same LPDDR5x, so both of task 2’s devices measure one pool.

Track X — AMD Ryzen AI Max+ 395Partial

PyTorch reaches the Radeon 8060S only through the ROCm build Part 1 installed, and Part 1 records AMD's documentation as inconsistent about integrated-graphics support (read 2026-09-12). A CPU-only run on this machine measures the same physical memory and is a supported result.

Two legitimate situations. With Part 1’s torch 2.14.0+rocm7.2 and the prepare lab’s ROCm 7.2.1, PyTorch reports the GPU under the cuda device name and you get accelerator figures. With Part 1’s CPU fallback, the scripts measure the CPU, which reads the same LPDDR5x; record that it is a CPU measurement. Neither blocks the course: Part 6’s llama.cpp reaches this GPU through Vulkan, not PyTorch. This lab assumes the Linux install the prepare lab used; on Windows, PyTorch would see a different device set, and the course has not written that path.

Track M — Apple silicon

Part 1’s torch 2.14.0 (MPS built in) and mlx, both in ~/llm-course/.venv. Close large applications first: every figure here comes from the pool they share. If you raised iogpu.wired_limit_mb in the prepare lab and have restarted since, it is back at 0; the measurements do not need it, but task 6’s budget does, so re-apply it first.

Track N — NVIDIA desktop or laptop

Part 1’s torch 2.14.0+cu130 (or +cu126). On Windows every command runs inside the WSL2 Ubuntu terminal, as Part 1 set up. On a laptop, plug it in and record whether it was on mains, because the power source is part of the conditions a figure was measured under.

Preflight

Put the four scripts in ~/llm-course with the download link under each listing below (mlx-bandwidth-test.py on Track M only). predict-decode.py also needs the course’s models.json, which the model reference is built from and which Part 6 expects at ~/llm-course/models.json: copy it there from src/data/models.json in the course repository. Then check everything from the directory you will work in.

Track S — NVIDIA DGX Spark

Container path, inside the shell setup-env-spark.sh opened:

RunnableTrack S · DGX Spark

preflight, container

cd /workspace/course
python - <<'EOF'
import torch
if torch.cuda.is_available():
    runtime = f"ROCm {torch.version.hip}" if torch.version.hip else f"CUDA {torch.version.cuda}"
    print("torch", torch.__version__, "| cuda |", torch.cuda.get_device_name(0), "|", runtime)
elif torch.backends.mps.is_available():
    print("torch", torch.__version__, "| mps")
else:
    print("torch", torch.__version__, "| cpu only")
EOF
ls bandwidth-test.py matmul-test.py predict-decode.py models.json labbook.md

Wheel path: the same, with cd ~/llm-course and source .venv/bin/activate in place of the first line.

Output — what you should see

torch 2.14.0a0+xxxxxxxxxx | cuda | NVIDIA GB10 | CUDA 13.x
bandwidth-test.py  labbook.md  matmul-test.py  models.json  predict-decode.py

The wheel path prints torch 2.14.0+cu130 and CUDA 13.0.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

preflight

cd ~/llm-course
source .venv/bin/activate
python - <<'EOF'
import torch
if torch.cuda.is_available():
    runtime = f"ROCm {torch.version.hip}" if torch.version.hip else f"CUDA {torch.version.cuda}"
    print("torch", torch.__version__, "| cuda |", torch.cuda.get_device_name(0), "|", runtime)
elif torch.backends.mps.is_available():
    print("torch", torch.__version__, "| mps")
else:
    print("torch", torch.__version__, "| cpu only")
EOF
ls bandwidth-test.py matmul-test.py predict-decode.py models.json labbook.md

Output — what you should see

torch 2.14.0+rocm7.2 | cuda | AMD Radeon ... | ROCm 7.2.xxxxx
bandwidth-test.py  labbook.md  matmul-test.py  models.json  predict-decode.py

The device name for this chip is not documented and may differ. torch 2.14.0+cpu | cpu only is Part 1’s CPU fallback: carry on and label every figure as CPU.

Track M — Apple silicon

RunnableTrack M · Apple silicon

preflight

cd ~/llm-course
source .venv/bin/activate
python - <<'EOF'
import torch
if torch.cuda.is_available():
    runtime = f"ROCm {torch.version.hip}" if torch.version.hip else f"CUDA {torch.version.cuda}"
    print("torch", torch.__version__, "| cuda |", torch.cuda.get_device_name(0), "|", runtime)
elif torch.backends.mps.is_available():
    print("torch", torch.__version__, "| mps")
else:
    print("torch", torch.__version__, "| cpu only")
EOF
python -c "import mlx.core as mx; print('mlx', mx.__version__, mx.default_device())"
ls bandwidth-test.py matmul-test.py mlx-bandwidth-test.py predict-decode.py models.json labbook.md

Output — what you should see

torch 2.14.0 | mps
mlx 0.32.2 Device(gpu, 0)
bandwidth-test.py  labbook.md  matmul-test.py  mlx-bandwidth-test.py  models.json  predict-decode.py

cpu only here means an x86 Python under Rosetta; Part 1’s troubleshooting covers it.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

preflight

cd ~/llm-course
source .venv/bin/activate
python - <<'EOF'
import torch
if torch.cuda.is_available():
    runtime = f"ROCm {torch.version.hip}" if torch.version.hip else f"CUDA {torch.version.cuda}"
    print("torch", torch.__version__, "| cuda |", torch.cuda.get_device_name(0), "|", runtime)
elif torch.backends.mps.is_available():
    print("torch", torch.__version__, "| mps")
else:
    print("torch", torch.__version__, "| cpu only")
EOF
ls bandwidth-test.py matmul-test.py predict-decode.py models.json labbook.md
nvidia-smi

Output — what you should see

torch 2.14.0+cu130 | cuda | NVIDIA GeForce RTX xxxx | CUDA 13.0
bandwidth-test.py  labbook.md  matmul-test.py  models.json  predict-decode.py
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 5xx.xx       Driver Version: 5xx.xx       CUDA Version: 13.x                  |
...
| Processes:                                                                              |
...

Read the Processes table at the bottom: anything other than your desktop session is using the card you are about to measure, so close it first.

Every line must print. No such file or directory for a script means it is not in this directory; for labbook.md, you are in the wrong directory or skipped Part 1. A cpu only line on Tracks S or N is a fault to fix before task 2 (see Troubleshooting); on Track X it is a choice to record.

Tasks

1. Set and record the conditions

A measurement taken while a video call is running is a measurement of a machine running a video call. Close browsers, containers other than Track S’s PyTorch one, and anything using the GPU. Leave the machine idle for five minutes if it has just been busy, since a warm chassis sustains less. Write the first lines of the notebook section now (task 7 has the template): the date, the power source, how long the machine was idle, and what was still running. These cannot be recovered later, and the scripts’ spread column is only interpretable against them.

2. Measure memory bandwidth, two ways, at three sizes

RunnableAll tracks

bandwidth-test.py

"""Measure memory bandwidth on the accelerator and on the CPU, two ways, with PyTorch.

Purpose: turn the vendor's published bandwidth figure into a measurement you made
         yourself, with two workloads dominated by moving bytes rather than arithmetic.
Platform: all (cuda on Tracks S and N and on Track X with the ROCm build, mps on
          Track M, cpu everywhere as a comparison and as the fallback)
Minimum memory: 8 GB
Assumes: torch is installed in the active environment; each device has room for two
         buffers of --size-mb (1 GiB at the default), because a copy needs a source
         and a destination

Usage: python3 bandwidth-test.py [--size-mb 512] [--iters 20] [--repeats 3]
                                 [--device auto] [--labbook labbook.md]

Method. Two tests per device, each run --repeats times after a warm-up. The median of
the repeats is reported with the spread, (slowest - fastest) / median. 1 GB = 1e9 bytes.
  copy  dst.copy_(src) on a buffer of N bytes reads N bytes and writes N bytes, so it
        reports (2 x N x iters) / seconds: bus traffic in both directions.
  read  y = W @ x, with W a square float32 matrix of about N bytes and x a vector, is one
        decode step at batch 1 in miniature: it reads every weight once and writes only
        the small output vector, so it reports (bytes of W x iters) / seconds.
The read figure counts bytes the same way predict-decode.py counts bytes per token, so it
is the figure to pass to that script.
"""
import argparse
import json
import statistics
import sys
import time
from datetime import datetime, timezone
from pathlib import Path

import torch

GB = 1_000_000_000
MIB = 1024 * 1024
SPREAD_WARNING = 10.0  # per cent


def pick_device(requested):
    if requested != "auto":
        return torch.device(requested)
    if torch.cuda.is_available():
        return torch.device("cuda")
    if torch.backends.mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")


def synchronize(device):
    """Block until the device has finished, so that timings are not of queueing."""
    if device.type == "cuda":
        torch.cuda.synchronize()
    elif device.type == "mps":
        torch.mps.synchronize()


def release(device):
    if device.type == "cuda":
        torch.cuda.empty_cache()
    elif device.type == "mps":
        torch.mps.empty_cache()


def describe(device):
    if device.type == "cuda":
        if torch.version.hip:
            runtime = f"ROCm (HIP {torch.version.hip})"
        else:
            runtime = f"CUDA {torch.version.cuda}"
        return f"{torch.cuda.get_device_name(0)}, {runtime}"
    if device.type == "mps":
        return "Apple silicon GPU through MPS"
    return f"CPU, {torch.get_num_threads()} threads"


def timed(device, fn, iters):
    synchronize(device)
    start = time.perf_counter()
    for _ in range(iters):
        fn()
    synchronize(device)
    return time.perf_counter() - start


def copy_test(device, size_mb, iters, repeats):
    nbytes = size_mb * MIB
    src = torch.empty(nbytes // 4, dtype=torch.float32, device=device)
    src.uniform_(-1.0, 1.0)
    dst = torch.empty_like(src)
    for _ in range(3):  # the first copies pay for allocation and kernel setup
        dst.copy_(src)
    rates = []
    for _ in range(repeats):
        seconds = timed(device, lambda: dst.copy_(src), iters)
        rates.append(2 * nbytes * iters / seconds / GB)
    del src, dst
    release(device)
    return rates, 2 * nbytes


def read_test(device, size_mb, iters, repeats):
    side = int((size_mb * MIB / 4) ** 0.5)
    w = torch.empty(side, side, dtype=torch.float32, device=device)
    w.uniform_(-1.0, 1.0)
    x = torch.empty(side, dtype=torch.float32, device=device)
    x.uniform_(-1.0, 1.0)
    nbytes = w.numel() * w.element_size()
    for _ in range(3):
        torch.matmul(w, x)
    rates = []
    for _ in range(repeats):
        seconds = timed(device, lambda: torch.matmul(w, x), iters)
        rates.append(nbytes * iters / seconds / GB)
    del w, x
    release(device)
    return rates, side, nbytes


def summarise(rates):
    median = statistics.median(rates)
    spread = (max(rates) - min(rates)) / median * 100.0
    return median, spread


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--size-mb", type=int, default=512,
                        help="bytes per buffer in MiB; the copy test allocates two")
    parser.add_argument("--iters", type=int, default=20,
                        help="operations per timed repeat")
    parser.add_argument("--repeats", type=int, default=3,
                        help="timed repeats per test; the median is reported")
    parser.add_argument("--device", default="auto", help="auto, cuda, mps or cpu")
    parser.add_argument("--labbook", default=None,
                        help="append one JSON line per run to this file")
    args = parser.parse_args()
    if args.size_mb < 1 or args.iters < 1 or args.repeats < 1:
        sys.exit("bandwidth-test: --size-mb, --iters and --repeats must be at least 1")

    device = pick_device(args.device)
    devices = [device] if device.type == "cpu" else [device, torch.device("cpu")]
    print(f"torch {torch.__version__}")
    for d in devices:
        print(f"{d.type}: {describe(d)}")
    print(f"buffer {args.size_mb} MiB; {args.iters} operations x {args.repeats} repeats "
          f"per test; 1 GB = 1e9 bytes\n")
    header = f"{'device':<7}{'test':<6}{'bytes per operation':<32}{'GB/s':>8}{'spread':>9}"
    print(header)
    print("-" * len(header))

    results = {}
    for d in devices:
        try:
            copy_rates, copy_bytes = copy_test(d, args.size_mb, args.iters, args.repeats)
            read_rates, side, read_bytes = read_test(d, args.size_mb, args.iters, args.repeats)
        except RuntimeError as exc:
            reason = str(exc).splitlines()[0][:160]
            sys.exit(f"bandwidth-test: {d.type} failed with --size-mb {args.size_mb}: {reason}\n"
                     f"bandwidth-test: halve --size-mb and run again")
        copy_median, copy_spread = summarise(copy_rates)
        read_median, read_spread = summarise(read_rates)
        print(f"{d.type:<7}{'copy':<6}{f'{copy_bytes:,} read + written':<32}"
              f"{copy_median:>8.1f}{copy_spread:>8.1f}%")
        print(f"{d.type:<7}{'read':<6}{f'{read_bytes:,} read ({side}^2 x 4)':<32}"
              f"{read_median:>8.1f}{read_spread:>8.1f}%")
        results[d.type] = {
            "copy": round(copy_median, 1), "read": round(read_median, 1),
            "copy_runs": [round(r, 1) for r in copy_rates],
            "read_runs": [round(r, 1) for r in read_rates],
            "read_matrix_side": side,
        }
        if max(copy_spread, read_spread) > SPREAD_WARNING:
            print(f"        spread above {SPREAD_WARNING:.0f} per cent on {d.type}: "
                  f"the machine was not steady (see Troubleshooting)")

    if args.labbook:
        record = {
            "lab": "part-05/bandwidth-test",
            "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "tool": "torch", "torch": torch.__version__,
            "device": device.type, "device_name": describe(device),
            "method": "copy: bytes read + written per second; read: W @ x, bytes of W per second",
            "size_mb": args.size_mb, "iters": args.iters, "repeats": args.repeats,
            "gbps": results,
        }
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download bandwidth-test.py193 lines

What the two tests measure. Both are chosen so that almost no arithmetic happens per byte, which leaves the memory path as the slower clock (Part 1’s bytes-moved-versus-operations section has the two-clock model):

copy   dst.copy_(src), N-byte buffers        GB/s = 2 × N × iters / seconds / 1e9   (read + write)
read   y = W @ x, W square float32 ≈ N bytes  GB/s = bytes(W) × iters / seconds / 1e9 (read only)

The read test is one decode step at batch 1 in miniature: every weight is fetched once and multiplied once, and only a vector is written back. It counts bytes exactly as predict-decode.py counts bytes per token, so its figure is the one that goes forward. The copy test counts each byte twice, once read and once written, the convention of many memory benchmarks; keep it because it is the figure to use when you compare with someone else’s copy-based number. Here is what each run in this task moves, from the scripts’ own arithmetic:

Run	Copy: bytes per operation	Read: side of W, bytes per operation	Traffic per timed repeat, copy / read
--size-mb 64, 20 operations	134,217,728	4,096² × 4 = 67,108,864	2.68 GB / 1.34 GB
default, 512 MiB, 20 operations	1,073,741,824	11,585² × 4 = 536,848,900	21.47 GB / 10.74 GB
--size-mb 1024 --iters 50	2,147,483,648	16,384² × 4 = 1,073,741,824	107.37 GB / 53.69 GB

Why three sizes. Every operation also pays a fixed cost t0 that does not scale with bytes: launching a GPU kernel, synchronising, the Python loop. The rate a test reports is then bytes / (bytes / B + t0). Here it is with B at the Spark’s published figure from task 5’s table and an illustrative t0 of 0.1 ms, arithmetic rather than a measurement:

B, GB/s	Bytes read per operation	Time for the bytes, bytes / B	Plus t0	Reported rate, GB/s	Share of B
273	67,108,864	0.246 ms	0.346 ms	194.1	71.1 %
273	536,848,900	1.966 ms	2.066 ms	259.8	95.2 %
273	1,073,741,824	3.933 ms	4.033 ms	266.2	97.5 %

So on an accelerator a small buffer reads low. On a CPU the effect can run the other way: a buffer that partly fits in the processor’s caches is served by memory much faster than the main bus, and the small run reads high. The largest run is also the longest, which exposes any thermal or power limit. Run all three, smallest first, so that the last line in the notebook is the sustained one:

RunnableAll tracks

a small buffer

python bandwidth-test.py --size-mb 64 --labbook labbook.md

RunnableAll tracks

the default, 512 MiB

python bandwidth-test.py --labbook labbook.md

RunnableAll tracks

the sustained run, 1,024 MiB, 50 operations per repeat

python bandwidth-test.py --size-mb 1024 --iters 50 --labbook labbook.md

Each prints the same shape. This is the last run on an accelerator track; a CPU-only run has only the two cpu rows:

Output — what you should see

torch 2.14.0+xxxxx
cuda: <device name>, CUDA 13.x                 (ROCm (HIP 7.2.xxxxx) on Track X; mps on Track M)
cpu: CPU, xx threads
buffer 1024 MiB; 50 operations x 3 repeats per test; 1 GB = 1e9 bytes

device test  bytes per operation                 GB/s   spread
--------------------------------------------------------------
cuda   copy  2,147,483,648 read + written       xxx.x     x.x%
cuda   read  1,073,741,824 read (16384^2 x 4)   xxx.x     x.x%
cpu    copy  2,147,483,648 read + written        xx.x     x.x%
cpu    read  1,073,741,824 read (16384^2 x 4)    xx.x     x.x%

recorded in labbook.md

spread is (fastest − slowest) ÷ median across the three repeats. If a line spread above 10 per cent appears under a device, the machine was not steady during that run: close what is still running, wait, and run that command again before moving on. The previous line stays in the notebook as evidence; the latest one is what task 6 reads.

Record from each run: the size, the accelerator and CPU read and copy figures, and the largest spread. If the 1,024 MiB read figure is lower than the 512 MiB one by more than the spreads explain, you have found a sustained limit rather than an error.

Track S — NVIDIA DGX Spark

Both devices read the same LPDDR5x, so the two read figures describe one pool through two processors: the Blackwell GPU and the Arm CPU cores. Expect them to be far closer together than on Track N, which is unified memory showing up in a measurement.

Track X — AMD Ryzen AI Max+ 395Partial

Without the ROCm build of PyTorch the accelerator rows are absent; the CPU rows still measure the same memory.

With the ROCm build, the two devices share the LPDDR5x as on Track S. With the CPU build there are only cpu rows; they are this machine’s memory read by its Zen 5 cores, and task 6 uses them as the bandwidth.

Track M — Apple silicon

mps and cpu are two paths to one pool. If mps fails to allocate, the 1,024 MiB run exceeded what macOS will keep resident for the GPU, the wired-memory limit from the Apple silicon lesson; run it with --size-mb 512 --iters 50 and record the substitution.

Track N — NVIDIA desktop or laptop

The two rows measure two different memories: GDDR on the card and DDR on the motherboard. Their ratio is the penalty for every weight that does not fit on the card, and task 6 turns it into a prediction for models that must be split.

3. Track M: the same measurements through MLX

Apple’s own framework reaches the same memory by a different route, and it is the framework Part 8 serves models with. Running both separates “what the hardware does” from “what this framework does with it”. The script repeats task 2’s two tests (its pass test is an elementwise multiply, counted read plus write like the copy) and then times matrix multiplies: three float formats through mx.matmul, and four quantised formats, where the second matrix is packed by mx.quantize and multiplied by mx.quantized_matmul, MLX’s documented operation for a weight in that packed form. MLX evaluates lazily, so each timed operation ends in mx.eval, which blocks until the result exists.

RunnableTrack M · Apple silicon

mlx-bandwidth-test.py

"""Measure bandwidth and matrix-multiply throughput with MLX on Apple silicon, beside PyTorch.

Purpose: run the same two measurements through Apple's own array framework, so that the
         Track M reader has two numbers per quantity for one machine and can see how much
         of the gap to the specification is the framework rather than the hardware; and
         time MLX's quantised matrix multiply, the kernel mlx-lm runs for 4-bit and 8-bit
         models, at four lower precisions
Platform: mac (Apple silicon; the lab runs it only on Track M)
Minimum memory: 8 GB
Assumes: mlx is installed in the active environment (Part 1's lab installs it); arrays
         live in unified memory, so the buffer size is bounded by the machine's memory
         and by the wired-memory limit rather than by a separate GPU memory pool

Usage: python3 mlx-bandwidth-test.py [--size-mb 512] [--iters 20] [--repeats 3]
                                     [--matmul-size 4096] [--matmul-iters 30]
                                     [--max-seconds 30] [--labbook labbook.md]

Method. MLX is lazy: nothing is computed until mx.eval asks for it, and mx.eval blocks
until it is done, so every timed operation ends in mx.eval. 1 GB = 1e9 bytes.
  pass    mx.eval(src * 0.5) over an N-byte float32 array reads N bytes and writes N:
          (2 x N x iters) / seconds, the convention of bandwidth-test.py's copy test
  read    mx.eval(mx.matmul(W, x)) with W a square float32 matrix of about N bytes:
          (bytes of W x iters) / seconds, the convention of bandwidth-test.py's read test
  matmul  A @ B for two N x N matrices, counted as 2 x N^3 operations, in float32,
          bfloat16 and float16 through mx.matmul; then with B quantised by mx.quantize and
          multiplied by mx.quantized_matmul (A stays float16) in four modes: affine 8-bit
          and 4-bit with group size 64, mxfp8 and mxfp4 with group size 32
Each matmul format gets one untimed warm-up and one timed multiplication to size the
loop to --max-seconds, never fewer than three multiplications.
"""
import argparse
import json
import statistics
import sys
import time
from datetime import datetime, timezone
from pathlib import Path

import mlx.core as mx

GB = 1_000_000_000
MIB = 1024 * 1024
FLOAT_FORMATS = {"float32": mx.float32, "bfloat16": mx.bfloat16, "float16": mx.float16}
QUANT_FORMATS = {  # name: (mode, bits, group_size), all documented for mx.quantize
    "affine8": ("affine", 8, 64),
    "affine4": ("affine", 4, 64),
    "mxfp8": ("mxfp8", 8, 32),
    "mxfp4": ("mxfp4", 4, 32),
}
INFO_KEYS = ("architecture", "device_name", "memory_size",
             "max_recommended_working_set_size", "max_buffer_length")


def timed(fn, count):
    start = time.perf_counter()
    for _ in range(count):
        fn()
    return time.perf_counter() - start


def bandwidth(size_mb, iters, repeats):
    nbytes = size_mb * MIB
    src = mx.random.uniform(shape=(nbytes // 4,), dtype=mx.float32)
    mx.eval(src)
    for _ in range(3):
        mx.eval(src * 0.5)
    passes = [2 * nbytes * iters / timed(lambda: mx.eval(src * 0.5), iters) / GB
              for _ in range(repeats)]
    del src
    side = int((nbytes / 4) ** 0.5)
    w = mx.random.uniform(shape=(side, side), dtype=mx.float32)
    x = mx.random.uniform(shape=(side,), dtype=mx.float32)
    mx.eval(w, x)
    for _ in range(3):
        mx.eval(mx.matmul(w, x))
    reads = [side * side * 4 * iters / timed(lambda: mx.eval(mx.matmul(w, x)), iters) / GB
             for _ in range(repeats)]
    del w, x
    mx.clear_cache()
    return passes, reads, side


def multiply_fn(name, size):
    a = mx.random.normal(shape=(size, size), dtype=mx.float32)
    b = mx.random.normal(shape=(size, size), dtype=mx.float32)
    if name in FLOAT_FORMATS:
        a, b = a.astype(FLOAT_FORMATS[name]), b.astype(FLOAT_FORMATS[name])
        mx.eval(a, b)
        return lambda: mx.eval(mx.matmul(a, b))
    mode, bits, group = QUANT_FORMATS[name]
    a = a.astype(mx.float16)
    packed = mx.quantize(b.astype(mx.float16), group_size=group, bits=bits, mode=mode)
    wq, scales = packed[0], packed[1]
    biases = packed[2] if len(packed) > 2 else None  # affine returns biases, mxfp does not
    mx.eval(a, *packed)
    return lambda: mx.eval(mx.quantized_matmul(a, wq, scales, biases, transpose=True,
                                               group_size=group, bits=bits, mode=mode))


def matmul(name, size, iters, max_seconds):
    fn = multiply_fn(name, size)
    fn()  # warm-up: Metal kernel compilation and allocation
    one = timed(fn, 1)
    runs = max(min(iters, 3), min(iters, int(max_seconds / max(one, 1e-9))))
    seconds = timed(fn, runs)
    del fn
    mx.clear_cache()
    return 2.0 * size ** 3 * runs / seconds / 1e12, runs, seconds


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--size-mb", type=int, default=512)
    parser.add_argument("--iters", type=int, default=20)
    parser.add_argument("--repeats", type=int, default=3)
    parser.add_argument("--matmul-size", type=int, default=4096,
                        help="square matrix side; must be divisible by 64")
    parser.add_argument("--matmul-iters", type=int, default=30)
    parser.add_argument("--max-seconds", type=float, default=30.0)
    parser.add_argument("--labbook", default=None,
                        help="append one JSON line per run to this file")
    args = parser.parse_args()
    if args.matmul_size % 64 or min(args.size_mb, args.iters, args.repeats,
                                     args.matmul_iters) < 1:
        sys.exit("mlx-bandwidth-test: --matmul-size must be a positive multiple of 64 and "
                 "the counts at least 1")

    print(f"mlx {mx.__version__}")
    print(f"default device: {mx.default_device()}")
    info = mx.device_info()
    shown = {k: info[k] for k in INFO_KEYS if k in info}
    for key, value in shown.items():
        if isinstance(value, int) and value > 1024 ** 3:
            print(f"{key}: {value} bytes ({value / 1024 ** 3:.1f} GiB)")
        else:
            print(f"{key}: {value}")

    passes, reads, side = bandwidth(args.size_mb, args.iters, args.repeats)
    print(f"\nbuffer {args.size_mb} MiB; {args.iters} operations x {args.repeats} repeats; "
          f"1 GB = 1e9 bytes")
    for label, runs in (("pass", passes), (f"read ({side}^2 x 4)", reads)):
        median = statistics.median(runs)
        spread = (max(runs) - min(runs)) / median * 100.0
        print(f"   mlx {label:<22} {median:8.1f} GB/s   spread {spread:.1f}%")

    print(f"\nmatrices: {args.matmul_size} x {args.matmul_size}; up to {args.matmul_iters} "
          f"multiplications or {args.max_seconds:.0f} s per format")
    tflops = {}
    for name in list(FLOAT_FORMATS) + list(QUANT_FORMATS):
        try:
            value, runs, seconds = matmul(name, args.matmul_size, args.matmul_iters,
                                          args.max_seconds)
        except (RuntimeError, ValueError, TypeError) as exc:
            reason = str(exc).splitlines()[0][:100]
            print(f"   {name:<9} unsupported: {reason}")
            tflops[name] = f"unsupported: {reason}"
            continue
        print(f"   {name:<9} {value:8.3f} TFLOPS   ({runs} in {seconds:.2f} s)")
        tflops[name] = round(value, 3)

    if args.labbook:
        record = {
            "lab": "part-05/mlx-bandwidth-test",
            "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "tool": "mlx", "mlx": mx.__version__,
            "device": str(mx.default_device()),
            "method": "pass: bytes read + written per second; read: W @ x, bytes of W per "
                      "second; matmul: 2 x N^3 flops, quantised formats via quantized_matmul",
            "size_mb": args.size_mb, "iters": args.iters, "repeats": args.repeats,
            "gbps": {"pass": round(statistics.median(passes), 1),
                     "read": round(statistics.median(reads), 1),
                     "pass_runs": [round(r, 1) for r in passes],
                     "read_runs": [round(r, 1) for r in reads]},
            "matmul_size": args.matmul_size, "tflops": tflops,
            "device_info": shown,
        }
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True, default=str) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download mlx-bandwidth-test.py183 lines

RunnableTrack M · Apple silicon

the MLX comparison

python mlx-bandwidth-test.py --size-mb 1024 --iters 50 --labbook labbook.md

Output — what you should see

mlx 0.32.2
default device: Device(gpu, 0)
architecture: applegpu_gxxx
device_name: ...
memory_size: xxxxxxxxxxx bytes (xx.x GiB)
max_recommended_working_set_size: xxxxxxxxxxx bytes (xx.x GiB)
max_buffer_length: xxxxxxxxxxx bytes (xx.x GiB)

buffer 1024 MiB; 50 operations x 3 repeats; 1 GB = 1e9 bytes
   mlx pass                     xxx.x GB/s   spread x.x%
   mlx read (16384^2 x 4)       xxx.x GB/s   spread x.x%

matrices: 4096 x 4096; up to 30 multiplications or 30 s per format
   float32      x.xxx TFLOPS   (30 in x.xx s)
   bfloat16     x.xxx TFLOPS   (30 in x.xx s)
   float16      x.xxx TFLOPS   (30 in x.xx s)
   affine8      x.xxx TFLOPS   (30 in x.xx s)
   affine4      x.xxx TFLOPS   (30 in x.xx s)
   mxfp8        x.xxx TFLOPS   (30 in x.xx s)
   mxfp4        x.xxx TFLOPS   (30 in x.xx s)

recorded in labbook.md

device_info keys depend on the backend, so a key may be missing; the architecture and name lines were not captured on hardware for this page. Record the MLX read figure beside PyTorch’s mps read figure from the same size, and max_recommended_working_set_size in GiB. There is no correct ratio between the two frameworks to check against; a difference on identical hardware is information about the frameworks. The quantised rows answer a question the float rows cannot: whether 4-bit and 8-bit arithmetic through MLX’s kernels is faster or slower per operation than float16 on your chip. Record which, because task 4 explains why the answer does not decide decode speed.

4. Measure matrix-multiply throughput at five precisions

RunnableAll tracks

matmul-test.py

"""Measure matrix-multiply throughput in TFLOPS at FP32, BF16, FP16 and two lower precisions.

Purpose: measure the compute half of the pair that predicts everything. Bandwidth sets
         the decode ceiling; matrix-multiply throughput sets what prefill and training
         can do. This is the compute-bound counterpart to bandwidth-test.py.
Platform: all (cuda on Tracks S and N and on Track X with the ROCm build, mps on
          Track M, cpu everywhere as a fallback)
Minimum memory: 8 GB
Assumes: torch is installed in the active environment; three matrices of --size fit on
         the device (at the default 4096, 64 MiB each in FP32, 256 MiB each at 8192)

Usage: python3 matmul-test.py [--size 4096] [--iters 30] [--device auto]
                              [--dtypes fp32,bf16,fp16,fp8,int8w] [--max-seconds 30]
                              [--bandwidth-gbps 250] [--labbook labbook.md]

Method. C = A @ B for two N x N matrices performs N^3 multiplies and about N^3 adds, so
each multiplication counts as 2 x N^3 floating-point operations, and the rate is
2 x N^3 x multiplications / seconds / 1e12 TFLOPS. The formats:
  fp32, bf16, fp16  A and B both in that dtype, one torch.matmul
  fp8               A and B as torch.float8_e4m3fn through the same plain torch.matmul;
                    a build without that kernel raises, recorded as unsupported
  int8w             weight-only INT8: B is stored as int8 with one bf16 scale per row and
                    converted back to bf16 before every multiplication, the way a
                    weight-only format without a native kernel runs; same operation count
Before each format a 512 x 512 probe estimates one full-size multiplication. A format
whose estimate exceeds --max-seconds is recorded as "slow path" with the probe's rate
rather than left running for many minutes, which is what a CPU without a fast kernel for
that dtype would otherwise do; otherwise the loop is shortened to fit --max-seconds, but
never below three multiplications.
With --bandwidth-gbps, the ridge point (TFLOPS x 1e12) / (GB/s x 1e9) is printed in
floating-point operations per byte.
"""
import argparse
import json
import sys
import time
from datetime import datetime, timezone
from pathlib import Path

import torch

FORMATS = ("fp32", "bf16", "fp16", "fp8", "int8w")
DTYPES = {"fp32": "float32", "bf16": "bfloat16", "fp16": "float16", "fp8": "float8_e4m3fn"}
PROBE = 512
FAILURES = (RuntimeError, TypeError, NotImplementedError)


def pick_device(requested):
    if requested != "auto":
        return torch.device(requested)
    if torch.cuda.is_available():
        return torch.device("cuda")
    if torch.backends.mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")


def synchronize(device):
    if device.type == "cuda":
        torch.cuda.synchronize()
    elif device.type == "mps":
        torch.mps.synchronize()


def release(device):
    if device.type == "cuda":
        torch.cuda.empty_cache()
    elif device.type == "mps":
        torch.mps.empty_cache()


def describe(device):
    if device.type == "cuda":
        if torch.version.hip:
            runtime = f"ROCm (HIP {torch.version.hip})"
        else:
            runtime = f"CUDA {torch.version.cuda}"
        return f"{torch.cuda.get_device_name(0)}, {runtime}"
    if device.type == "mps":
        return "Apple silicon GPU through MPS"
    return f"CPU, {torch.get_num_threads()} threads"


def build(fmt, side, device):
    """Return a zero-argument function that performs one multiplication in this format."""
    a32 = torch.randn(side, side, device=device, dtype=torch.float32)
    b32 = torch.randn(side, side, device=device, dtype=torch.float32)
    if fmt == "int8w":
        a = a32.to(torch.bfloat16)
        scale = b32.abs().amax(dim=1, keepdim=True).clamp(min=1e-8) / 127.0
        q = torch.round(b32 / scale).clamp(-127, 127).to(torch.int8)
        s = scale.to(torch.bfloat16)
        del a32, b32
        return lambda: torch.matmul(a, q.to(torch.bfloat16) * s)
    dtype = getattr(torch, DTYPES[fmt])
    a, b = a32.to(dtype), b32.to(dtype)
    del a32, b32
    return lambda: torch.matmul(a, b)


def rate(device, fn, side, count=None, min_seconds=None):
    """Time count multiplications, or as many as fill min_seconds; return (flops/s, seconds)."""
    synchronize(device)
    start = time.perf_counter()
    done = 0
    while True:
        fn()
        done += 1
        if count is not None and done >= count:
            break
        if min_seconds is not None:
            synchronize(device)
            if time.perf_counter() - start >= min_seconds:
                break
    synchronize(device)
    seconds = time.perf_counter() - start
    return 2.0 * side ** 3 * done / seconds, seconds


def measure(device, fmt, size, iters, max_seconds):
    """Return (result, runs, seconds, probe GFLOP/s); result is TFLOPS or a string."""
    probe_side = min(PROBE, size)
    fn = build(fmt, probe_side, device)
    fn()  # warm-up: kernel selection, allocation
    probe_flops, _ = rate(device, fn, probe_side, min_seconds=0.25)
    del fn
    one = 2.0 * size ** 3 / probe_flops  # estimated seconds for one full-size multiply
    if one > max_seconds:
        release(device)
        note = (f"slow path: {probe_side} probe {probe_flops / 1e9:.2f} GFLOP/s, "
                f"one {size} multiply ~{one:.0f} s")
        return note, 0, 0.0, probe_flops / 1e9
    fn = build(fmt, size, device)
    fn()  # warm-up at full size
    _, one = rate(device, fn, size, count=1)  # the loop is sized from a real multiplication
    runs = max(min(iters, 3), min(iters, int(max_seconds / max(one, 1e-9))))
    flops, seconds = rate(device, fn, size, count=runs)
    del fn
    release(device)
    return flops / 1e12, runs, seconds, probe_flops / 1e9


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--size", type=int, default=4096, help="square matrix side")
    parser.add_argument("--iters", type=int, default=30,
                        help="multiplications per format, fewer if --max-seconds binds")
    parser.add_argument("--device", default="auto", help="auto, cuda, mps or cpu")
    parser.add_argument("--dtypes", default=",".join(FORMATS),
                        help=f"comma-separated, from {','.join(FORMATS)}")
    parser.add_argument("--max-seconds", type=float, default=30.0,
                        help="time budget per format")
    parser.add_argument("--bandwidth-gbps", type=float, default=None,
                        help="read figure from bandwidth-test.py, to print the ridge point")
    parser.add_argument("--labbook", default=None,
                        help="append one JSON line per run to this file")
    args = parser.parse_args()
    wanted = [d.strip() for d in args.dtypes.split(",") if d.strip()]
    unknown = [d for d in wanted if d not in FORMATS]
    if unknown or args.size < 2 or args.iters < 1 or args.max_seconds <= 0:
        sys.exit(f"matmul-test: formats must come from {','.join(FORMATS)} "
                 f"(got {','.join(unknown) or 'none unknown'}); --size, --iters and "
                 f"--max-seconds must be positive")

    device = pick_device(args.device)
    print(f"torch {torch.__version__}")
    print(f"device: {device.type} ({describe(device)})")
    print(f"matrices: {args.size} x {args.size}; up to {args.iters} multiplications or "
          f"{args.max_seconds:.0f} s per format; 2 x N^3 = {2 * args.size ** 3 / 1e9:,.1f} "
          f"GFLOP per multiplication\n")
    ridge = args.bandwidth_gbps is not None
    header = f"{'format':<7}{'TFLOPS':>9}{'runs':>6}{'seconds':>9}"
    header += f"{'FLOP/byte':>11}" if ridge else ""
    print(header + "  note")
    print("-" * (len(header) + 6))

    tflops, runs_by_format, probes = {}, {}, {}
    for fmt in wanted:
        try:
            result, runs, seconds, probe = measure(device, fmt, args.size, args.iters,
                                                   args.max_seconds)
        except FAILURES as exc:
            release(device)
            reason = str(exc).splitlines()[0][:100]
            print(f"{fmt:<7}{'-':>9}{'-':>6}{'-':>9}" + (f"{'-':>11}" if ridge else "")
                  + f"  unsupported: {reason}")
            tflops[fmt] = f"unsupported: {reason}"
            continue
        probes[fmt] = round(probe, 2)
        if isinstance(result, str):
            print(f"{fmt:<7}{'-':>9}{'-':>6}{'-':>9}" + (f"{'-':>11}" if ridge else "")
                  + f"  {result}")
            tflops[fmt] = result
            continue
        line = f"{fmt:<7}{result:>9.3f}{runs:>6}{seconds:>9.2f}"
        if ridge:
            line += f"{result * 1e12 / (args.bandwidth_gbps * 1e9):>11.0f}"
        print(line)
        tflops[fmt] = round(result, 3)
        runs_by_format[fmt] = runs

    if args.labbook:
        record = {
            "lab": "part-05/matmul-test",
            "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "tool": "torch", "torch": torch.__version__,
            "device": device.type, "device_name": describe(device),
            "method": "torch.matmul, 2 x N^3 flops per multiplication; int8w dequantises B first",
            "size": args.size, "iters": args.iters, "max_seconds": args.max_seconds,
            "tflops": tflops, "runs": runs_by_format, "probe_gflops": probes,
        }
        if ridge:
            record["bandwidth_gbps"] = args.bandwidth_gbps
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download matmul-test.py220 lines

Why a big matrix multiply measures compute. C = A @ B for two N × N matrices computes N² dot products of length N, each N multiplies and about N adds, so the script counts 2 × N³ operations per multiplication. It reads only the two input matrices to do it, so the operations per byte read are enormous, far above any machine’s ridge point, and the arithmetic is the slower clock:

Size N	Operations per multiplication, 2 × N³	One matrix at FP32 / BF16	Bytes of A and B at BF16	Operations per byte read
4,096	137,438,953,472 = 137.4 GFLOP	64 / 32 MiB	67,108,864	2,048
8,192	1,099,511,627,776 = 1.10 TFLOP	256 / 128 MiB	268,435,456	4,096

Compare that with decode at batch 1, one operation per byte at BF16 in Part 1’s table. The same chip is on opposite sides of its ridge point in the two tests, which is why the course measures both.

The five formats. Bits per element and layout are PyTorch’s own descriptions of each dtype.

Format	Stored as	How the script multiplies	What its figure tells you
fp32	32 bits	torch.matmul	the general-purpose baseline
bf16	16 bits, sign-exponent-mantissa 1-8-7	torch.matmul	the training format; Part 12’s compute budget reads this figure
fp16	16 bits, 1-5-10	torch.matmul	the format Part 6 compares its prompt-processing rate with
fp8	8 bits, float8_e4m3fn, 1-4-3	plain torch.matmul on both matrices	whether this build has a plain FP8 multiply at all
int8w	8-bit integers with one BF16 scale per row	converted back to BF16 before every multiply	what a weight-only low-precision format costs when there is no native kernel

On FP8 the script is deliberately modest. PyTorch 2.14 documents a separate torch.nn.functional.scaled_mm for scaled low-precision multiplies, but its documentation page does not list the values of the scaling-recipe enums it requires, so this lab does not call it; the engines in Part 8 exercise real FP8 kernels. The int8w row is the useful lower-precision result on every track: it performs the same 2 × N³ operations as bf16 plus a conversion, so it cannot legitimately beat bf16, and the size of its shortfall is the arithmetic price of a weight format without its own kernel.

Before each format the script times a 512 × 512 probe for at least a quarter of a second and estimates one full-size multiplication. A format whose estimate exceeds --max-seconds (30 by default) is recorded as a slow path with the probe’s rate, instead of leaving you waiting many minutes for a kernel that does the work element by element.

RunnableAll tracks

all five formats at 4,096

python matmul-test.py --labbook labbook.md

Output — what you should see

torch 2.14.0+xxxxx
device: cuda (<device name>, CUDA 13.x)
matrices: 4096 x 4096; up to 30 multiplications or 30 s per format; 2 x N^3 = 137.4 GFLOP per multiplication

format    TFLOPS  runs  seconds  note
-------------------------------------
fp32      xx.xxx    30     x.xx
bf16      xx.xxx    30     x.xx
fp16      xx.xxx    30     x.xx
fp8            -     -        -  unsupported: <the first line of the error>
int8w     xx.xxx    30     x.xx

recorded in labbook.md

A slow path or unsupported row is a result, not an error. For comparison, this is a real CPU-only run on the test box used to write this page, which is not one of the four tracks; it shows what a slow path looks like and nothing about your machine’s figures:

Output — what you should see

matmul-test.py, torch 2.14.0+cpu, 12 CPU threads, one run on the author's test box; not a track machine

torch 2.14.0+cpu
device: cpu (CPU, 12 threads)
matrices: 4096 x 4096; up to 30 multiplications or 30 s per format; 2 x N^3 = 137.4 GFLOP per multiplication

format    TFLOPS  runs  seconds  note
-------------------------------------
fp32       1.241    30     3.32
bf16       4.300    30     0.96
fp16           -     -        -  slow path: 512 probe 3.06 GFLOP/s, one 4096 multiply ~45 s
fp8            -     -        -  slow path: 512 probe 0.89 GFLOP/s, one 4096 multiply ~155 s
int8w      3.498    30     1.18

Then the larger size, which spreads each multiplication’s fixed costs over eight times the arithmetic. The low-precision rows add nothing new at this size, so only three formats:

RunnableAll tracks

three formats at 8,192

python matmul-test.py --size 8192 --iters 20 --dtypes fp32,bf16,fp16 --labbook labbook.md

Read your ladder with this table, and record each format’s figure or note for both sizes:

What you see	What it means
bf16 or fp16 well above fp32	this device has fast 16-bit kernels
a 16-bit format marked slow path	no fast kernel for that dtype on this device; common for fp16 on a CPU
a format slow path at 8,192 that ran at 4,096	something else took the device during the probe; rerun that command
fp8 unsupported	plain torch.matmul has no FP8 kernel in this build
int8w below bf16	the conversion’s cost; weight-only formats save bytes, which decode needs, and spend arithmetic, which prefill pays
int8w above bf16 by more than a few per cent	the machine was not steady between the two formats; rerun
8,192 above 4,096	fixed per-call costs amortised; use the 8,192 figure as the sustained compute

5. Compare with the specification, and account for the gap

Where a published bandwidth figure comes from. It is the memory interface’s signalling rate multiplied by its width, which is capacity of the wires rather than throughput of a workload:

published GB/s = transfers per second per line (MT/s) × lines (bits) ÷ 8 ÷ 1,000

Track	What the vendor publishes	Arithmetic	Published bandwidth
S, DGX Spark	“LPDDR5X 8533”, “16 channels (256 bit)”	8,533 × 256 ÷ 8 = 273,056 MB/s	273 GB/s
X, EVO-X2	GMKtec: LPDDR5X at “8000MHz”; no width	if 8,000 is the transfer rate, 256 × 8 ÷ 8,000 = 256 bits, the GB10’s width	256 GB/s (course hardware reference)
M	Apple: the bandwidth per chip only	none possible	120 (M4) to 1,200 (M5 Ultra) GB/s
N, RTX 5090	NVIDIA: “512-bit” interface; no rate	1,792 × 8 ÷ 512 = 28 Gbit/s per line	1,792 GB/s (course hardware reference)

The X row shows a trap worth knowing: the same memory speed appears in MHz on one page and MT/s on another, and the course’s Spark lesson quotes the GB10’s memory as 4266 MHz, half of NVIDIA’s 8533. A factor of two that comes from what was counted, like copy against read, is not a fault in the machine; find out what was counted before comparing.

Take your published figure from the hardware reference, or from the vendor page for a chip or card it does not list; the check in Validation prints the ratio for you. Then work down this table to name the cause you think dominates:

Cause	Mechanism	What it does to your figures	How to check it here
Counting convention	a copy moves each byte twice	copy and read differ by up to a factor of two	compare the two rows of one run
Fixed per-call cost	bytes / (bytes / B + t0)	small buffers read low on a GPU	64 MiB against 1,024 MiB rows
Processor caches	a small buffer is partly served from cache	small buffers read high on a CPU	the same rows on the CPU
Interface against workload	the published figure is raw signalling; the controller, refresh and protocol take their share	every figure below the published one	not separable at home: it is what remains
Sharing	the display, the desktop and other programs use the same memory or card	lower accelerator figures, larger spread	close them and rerun; compare spreads
Heat and power	a small enclosure or a laptop sustains less than it peaks	the 1,024 MiB run below the 512 MiB run	the spread line; on Track N, nvidia-smi --query --display=PERFORMANCE straight after the run, read Clocks Event Reasons
Wrong device	the scripts ran on the CPU or another GPU	accelerator rows missing, or below the CPU rows on Track N	the device lines of task 2
Framework	two frameworks drive the same memory differently	MLX and MPS read figures differ	Track M: task 3 against task 2

Then apply the decision rule, which is about your inputs rather than about what a good machine scores:

Result	Verdict	Do
read ÷ published above 1.00	an input is wrong: another part’s figure, MiB taken for GB, or a copy figure against a one-direction specification	recheck the part number and use the read row
any spread above 10 per cent	the machine was not steady	fix the condition and rerun that size
Track N: accelerator read at or below CPU read	not measuring the card	Troubleshooting
none of the above	a valid ratio	record it with its conditions and the cause you named

Compute figures are rarely comparable. Vendors publish compute at their most favourable format:

Machine	Published	What it counts	Against your bf16 figure
DGX Spark	“Up to 1,000 TOPS”; “up to 1 PFLOP (petaFLOP) at FP4 precision with sparsity”	4-bit operations, zeros skipped, peak	not comparable: 1 PFLOP is 1,000 TFLOPS of a different operation
RTX PRO 6000 Blackwell	“4000 TOPS”, “Theoretical FP4 TOPS using sparsity”	theoretical FP4 with sparsity	not comparable
GeForce RTX 5090	“3352 AI TOPS”	method not stated on the page	not comparable
GMKtec EVO-X2	“50 TOPS” for the NPU; “Overall processor performance up to 126 TOPS”	the NPU, and a whole-processor total; no GPU figure	not comparable: PyTorch does not run on the NPU in this lab
Apple silicon	no GPU operations-per-second figure on the Mac Studio page	nothing	your measurement is the only figure

Record “not comparable” with the reason rather than a ratio. Your own bf16 and fp16 figures are the compute numbers the course uses later.

6. Predict decode speed for three models

RunnableAll tracks

predict-decode.py

"""Predict decode-speed ceilings from measured bandwidth, a model's bytes and a memory budget.

Purpose: turn the bandwidth you just measured into a falsifiable prediction. Decode
         reads the active weights and the key-value cache once per token, so the token
         rate cannot exceed bandwidth divided by those bytes. Part 6 runs the models and
         shows how close the engines get.
Platform: all (this is arithmetic; it needs no accelerator and no model download)
Minimum memory: 8 GB
Assumes: a readable copy of the course's models.json, whose path is passed on the
         command line; a bandwidth figure, either typed or read from the latest
         part-05/bandwidth-test line in the lab notebook

Usage: python3 predict-decode.py --models models.json --labbook labbook.md
                                 [--bandwidth-gbps 250] [--budget-gb 22.5]
                                 [--split | --cpu-bandwidth-gbps 60]
                                 [--model-ids qwen3-8b,qwen3-30b-a3b,qwen3-32b]
                                 [--quants q4_k_m,bf16] [--context-tokens 4096]

Method. 1 GB = 1e9 bytes throughout, the convention of the bandwidth scripts.
  bandwidth   --bandwidth-gbps, or else the accelerator's "read" figure from the latest
              part-05/bandwidth-test line in --labbook; --split takes that line's CPU
              "read" figure as --cpu-bandwidth-gbps
  active GB   file size x active / total parameters (the whole file for a dense model);
              Part 3 shows this is a few per cent high for dense Qwen3 files
  KV GB       kv.bytesPerTokenFp16 x context tokens: an f16 cache, one sequence
  ceiling     bandwidth / (active GB + KV GB), at 0 tokens and at --context-tokens
  fit         with --budget-gb: "fits" when file + KV <= budget. Otherwise, with
              --cpu-bandwidth-gbps, "split": the cache and (budget - KV) GB of weights stay
              on the accelerator, the rest is read from system memory, and
              seconds per token = (active on accelerator + KV) / bandwidth
                                + active in system memory / cpu bandwidth
              Without it, "over": the prediction assumes memory the device does not have.
Every figure is a ceiling: sampling, kernel launches and the arithmetic itself are given
no time, and a split also ignores the CPU's arithmetic on the layers it holds.
"""
import argparse
import json
import sys
from datetime import datetime, timezone
from pathlib import Path

DEFAULT_IDS = "qwen3-8b,qwen3-30b-a3b,qwen3-32b"
DEFAULT_QUANTS = "q4_k_m,bf16"
QUANT_LABELS = {"q4_k_m": "Q4_K_M", "q8_0": "Q8_0", "q6_k": "Q6_K", "bf16": "BF16",
                "mxfp4": "MXFP4", "iq4_xs": "IQ4_XS"}


def load_models(path):
    try:
        data = json.loads(Path(path).read_text(encoding="utf-8"))
    except (OSError, ValueError) as exc:
        sys.exit(f"predict-decode: cannot read {path}: {exc}")
    return {m["id"]: m for m in data.get("models", [])}


def active_fraction(model):
    params = model.get("params", {})
    total = float(params.get("totalB") or 0.0)
    active = float(params.get("activeB") or 0.0)
    if total <= 0 or active <= 0:
        return 1.0
    return min(active / total, 1.0)


def ceiling(size_gb, fraction, kv_gb, bandwidth, budget, cpu_bandwidth):
    """Return (fit label, tokens per second or None)."""
    active = size_gb * fraction
    if budget is None or size_gb + kv_gb <= budget:
        return ("fits" if budget is not None else "-"), bandwidth / (active + kv_gb)
    if cpu_bandwidth is None or budget <= kv_gb:
        return "over", None
    on_device = (budget - kv_gb) / size_gb  # share of the file the accelerator keeps
    seconds = (active * on_device + kv_gb) / bandwidth + active * (1 - on_device) / cpu_bandwidth
    return "split", 1.0 / seconds


def from_labbook(path):
    """Return the latest part-05/bandwidth-test record in the notebook, or exit."""
    latest = None
    try:
        lines = Path(path).read_text(encoding="utf-8").splitlines()
    except OSError as exc:
        sys.exit(f"predict-decode: cannot read {path}: {exc}")
    for line in lines:
        if line.startswith("{") and '"part-05/bandwidth-test"' in line:
            try:
                latest = json.loads(line)
            except ValueError:
                continue
    if latest is None or "read" not in latest.get("gbps", {}).get(latest.get("device"), {}):
        sys.exit(f"predict-decode: no part-05/bandwidth-test line with a read figure in {path}; "
                 f"run bandwidth-test.py --labbook {path} first, or pass --bandwidth-gbps")
    return latest


def show(rate):
    return f"{rate:>9.1f}" if rate is not None else f"{'-':>9}"


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--bandwidth-gbps", type=float, default=None,
                        help="the read figure bandwidth-test.py measured, in GB/s; "
                             "default: the latest one in --labbook")
    parser.add_argument("--models", required=True, help="path to the course models.json")
    parser.add_argument("--model-ids", default=DEFAULT_IDS)
    parser.add_argument("--quants", default=DEFAULT_QUANTS)
    parser.add_argument("--context-tokens", type=int, default=4096,
                        help="tokens already in the cache for the second ceiling column")
    parser.add_argument("--budget-gb", type=float, default=None,
                        help="memory the accelerator may use for weights and cache, in GB")
    parser.add_argument("--cpu-bandwidth-gbps", type=float, default=None,
                        help="the CPU read figure, to estimate models that must be split")
    parser.add_argument("--split", action="store_true",
                        help="use the CPU read figure from the same notebook line for "
                             "rows over budget (Track N)")
    parser.add_argument("--labbook", default=None,
                        help="read the bandwidth from, and append one JSON line to, this file")
    args = parser.parse_args()
    source = "--bandwidth-gbps"
    if args.bandwidth_gbps is None or args.split:
        if not args.labbook:
            sys.exit("predict-decode: pass --bandwidth-gbps, or --labbook with a "
                     "part-05/bandwidth-test line in it")
        record = from_labbook(args.labbook)
        if args.bandwidth_gbps is None:
            args.bandwidth_gbps = record["gbps"][record["device"]]["read"]
            source = (f"{record['device']} read figure, {record.get('size_mb')} MiB run of "
                      f"{record.get('date', 'an undated run')}")
        if args.split and args.cpu_bandwidth_gbps is None:
            if record["device"] == "cpu" or "cpu" not in record["gbps"]:
                sys.exit("predict-decode: --split needs an accelerator run with a cpu figure")
            args.cpu_bandwidth_gbps = record["gbps"]["cpu"]["read"]
    if args.bandwidth_gbps <= 0 or args.context_tokens < 0:
        sys.exit("predict-decode: --bandwidth-gbps must be positive and --context-tokens "
                 "not negative")

    catalogue = load_models(args.models)
    wanted = [i.strip() for i in args.model_ids.split(",") if i.strip()]
    quants = [q.strip() for q in args.quants.split(",") if q.strip()]
    ctx = args.context_tokens

    print(f"bandwidth: {args.bandwidth_gbps:.1f} GB/s, from {source}")
    if args.cpu_bandwidth_gbps:
        print(f"cpu bandwidth for split rows: {args.cpu_bandwidth_gbps:.1f} GB/s")
    print(f"budget: {args.budget_gb:.1f} GB" if args.budget_gb is not None
          else "budget: not given, so no fit column")
    print("ceiling = bandwidth / (active GB + KV GB); 1 GB = 1e9 bytes\n")
    header = (f"{'model':<15}{'arch':<6}{'format':<8}{'file GB':>8}{'active GB':>10}"
              f"{f'KV@{ctx}':>9}{'fit':>6}{'tok/s@0':>9}{f'tok/s@{ctx}':>11}")
    print(header)
    print("-" * len(header))

    predictions = []
    for model_id in wanted:
        model = catalogue.get(model_id)
        if model is None:
            print(f"{model_id:<15}not found in {args.models}")
            continue
        fraction = active_fraction(model)
        kv_bytes = float(model.get("kv", {}).get("bytesPerTokenFp16") or 0.0)
        kv_gb = kv_bytes * ctx / 1e9
        sizes = model.get("sizesGB", {})
        for quant in quants:
            size_gb = sizes.get(quant)
            if size_gb is None:
                print(f"{model_id:<15}{model.get('architecture', '?'):<6}{quant:<8}"
                      f"  not listed (available: {', '.join(sorted(sizes)) or 'none'})")
                continue
            size_gb = float(size_gb)
            fit0, rate0 = ceiling(size_gb, fraction, 0.0, args.bandwidth_gbps,
                                  args.budget_gb, args.cpu_bandwidth_gbps)
            fit, rate = ceiling(size_gb, fraction, kv_gb, args.bandwidth_gbps,
                                args.budget_gb, args.cpu_bandwidth_gbps)
            print(f"{model_id:<15}{model.get('architecture', '?'):<6}"
                  f"{QUANT_LABELS.get(quant, quant):<8}{size_gb:>8.1f}"
                  f"{size_gb * fraction:>10.2f}{kv_gb:>9.2f}{fit:>6}{show(rate0)}"
                  f"{show(rate):>11}")
            predictions.append({
                "model": model_id, "quant": quant, "size_gb": size_gb,
                "active_gb": round(size_gb * fraction, 3), "kv_gb": round(kv_gb, 3),
                "fit": fit, "fit_at_zero_context": fit0,
                "predicted_tokens_per_second": round(rate0, 1) if rate0 else None,
                "predicted_tokens_per_second_at_context": round(rate, 1) if rate else None,
            })

    if args.labbook and predictions:
        record = {
            "lab": "part-05/predict-decode",
            "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "bandwidth_gbps": args.bandwidth_gbps, "bandwidth_source": source,
            "cpu_bandwidth_gbps": args.cpu_bandwidth_gbps,
            "budget_gb": args.budget_gb, "context_tokens": ctx,
            "models_json": str(args.models),
            "method": "bandwidth / (file x active/total + kv bytes x context); split rows "
                      "add the system-memory share at the cpu bandwidth",
            "predictions": predictions,
        }
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download predict-decode.py205 lines

The formula. Part 3 derived the decode ceiling, and the script adds the two corrections this machine’s budget needs:

active GB     = file GB × active parameters ÷ total parameters     (the whole file when dense)
KV GB         = kv.bytesPerTokenFp16 × context tokens ÷ 1e9          (from models.json)
ceiling       = B ÷ (active GB + KV GB)                               tokens per second, batch 1
fit           = file GB + KV GB ≤ budget GB
split, when it does not fit and a CPU figure is given:
  on device   = (budget GB − KV GB) ÷ file GB                          share of the file kept
  s per token = (active GB × on device + KV GB) ÷ B  +  active GB × (1 − on device) ÷ B_cpu

B is your accelerator’s read figure; B_cpu the CPU’s. Here is the script run on stated inputs, arithmetic and not a measurement:

Input	Value	Where it comes from
B	1,008 GB/s	the course hardware reference’s figure for a 24 GB RTX 4090, used as if measured
budget	22.5 GB	24 GB less Part 4’s 1.5 GB reserve for a card
B_cpu	60 GB/s	illustrative, not a measurement of any machine

RunnableAll tracks

the worked example, on stated inputs

python predict-decode.py --models models.json --bandwidth-gbps 1008 --cpu-bandwidth-gbps 60 --budget-gb 22.5

Output — what you should see

bandwidth: 1008.0 GB/s, from --bandwidth-gbps
cpu bandwidth for split rows: 60.0 GB/s
budget: 22.5 GB
ceiling = bandwidth / (active GB + KV GB); 1 GB = 1e9 bytes

model          arch  format   file GB active GB  KV@4096   fit  tok/s@0 tok/s@4096
----------------------------------------------------------------------------------
qwen3-8b       dense Q4_K_M       5.0      5.00     0.60  fits    201.6      179.9
qwen3-8b       dense BF16        16.4     16.40     0.60  fits     61.5       59.3
qwen3-30b-a3b  moe   Q4_K_M      18.6      2.01     0.40  fits    500.9      417.4
qwen3-30b-a3b  moe   BF16        61.0      6.60     0.40 split     13.9       13.7
qwen3-32b      dense Q4_K_M      19.8     19.80     1.07  fits     50.9       48.3
qwen3-32b      dense BF16        65.6     65.60     1.07 split      1.4        1.3

Check one row by hand, the Qwen3-32B BF16 split with an empty cache:

on device    = 22.5 ÷ 65.6                      = 0.343 of the file, 22.5 GB
card term    = 22.5 GB ÷ 1,008 GB/s             =  22.3 ms
system term  = (65.6 − 22.5) GB ÷ 60 GB/s       = 718.3 ms
per token    = 22.3 + 718.3                     = 740.6 ms  →  1 ÷ 0.7406 = 1.35 tokens per second

The card’s bandwidth has almost stopped mattering: the system-memory term is 97 per cent of the time. That is the arithmetic behind the offloading section of the Track N lesson.

Read the shape before the numbers. Qwen3-30B-A3B and Qwen3-32B are almost the same size at Q4_K_M and their ceilings differ by about ten times, because the mixture of experts reads 3.3 of its 30.5 billion parameters per token and the dense model reads all of itself. The KV column is why the second ceiling column is lower, and it costs the fast model the largest share. All three models are published under the Apache-2.0 licence and none is gated, as the model reference records.

Your budget. Use the memory your accelerator may place tensors in, less the reserve, following Part 4’s budget table and the limits the prepare lab set. Set it once:

Track S — NVIDIA DGX Spark

The whole pool less Part 4’s 10 GB reserve:

RunnableTrack S · DGX Spark

the budget

BUDGET_GB=$(awk '/^MemTotal:/ {printf "%.1f", $2 * 1024 / 1e9 - 10}' /proc/meminfo)
echo "$BUDGET_GB"

Run it in the same shell as the other tasks, container or host. Do not pass --split: both devices read the same memory.

Track X — AMD Ryzen AI Max+ 395

The GTT size the prepare lab set, which already keeps its reserve back:

RunnableTrack X · Ryzen AI Max+

the budget

BUDGET_GB=$(awk '{printf "%.1f", $1 / 1e9}' /sys/class/drm/card*/device/mem_info_gtt_total | head -n 1)
echo "$BUDGET_GB"

Do not pass --split.

Track M — Apple silicon

Metal’s recommended working set, as the prepare lab left it:

RunnableTrack M · Apple silicon

the budget

BUDGET_GB=$(python -c "import torch; print(round(torch.mps.recommended_max_memory() / 1e9, 1))")
echo "$BUDGET_GB"

Do not pass --split.

Track N — NVIDIA desktop or laptop

Free VRAM less Part 4’s 1.5 GB reserve, with nvidia-smi printing mebibytes:

RunnableTrack N · NVIDIA GPU

the budget

BUDGET_GB=$(nvidia-smi --query-gpu=memory.total,memory.used --format=csv,noheader,nounits | head -n 1 | awk -F', ' '{printf "%.1f", ($1 - $2) * 1048576 / 1e9 - 1.5}')
echo "$BUDGET_GB"

Pass --split in the command below: rows that do not fit on the card then get a split estimate from your CPU read figure.

echo must print one number, in GB. An empty line means the command found nothing to read; fix it before predicting, because a missing budget turns the fit column off. Now predict from your own figure. Without --bandwidth-gbps, the script takes the accelerator read figure from the latest part-05/bandwidth-test line in the notebook, which is task 2’s sustained run:

RunnableAll tracks

predict: Tracks S, X and M

python predict-decode.py --models models.json --budget-gb "$BUDGET_GB" --labbook labbook.md

RunnableTrack N · NVIDIA GPU

predict: Track N

python predict-decode.py --models models.json --budget-gb "$BUDGET_GB" --split --labbook labbook.md

Output — what you should see

bandwidth: xxx.x GB/s, from cuda read figure, 1024 MiB run of 2026-xx-xxTxx:xx:xx+00:00
cpu bandwidth for split rows: xx.x GB/s                 (Track N only)
budget: xx.x GB
ceiling = bandwidth / (active GB + KV GB); 1 GB = 1e9 bytes

model          arch  format   file GB active GB  KV@4096   fit  tok/s@0 tok/s@4096
----------------------------------------------------------------------------------
qwen3-8b       dense Q4_K_M       5.0      5.00     0.60  fits     xx.x       xx.x
...
qwen3-32b      dense BF16        65.6     65.60     1.07  over        -          -

recorded in labbook.md

The first line must name the 1,024 MiB run; if it names another size, the sustained run is not the latest line, so run task 2’s last command again. over means the model does not fit your budget and the script declines to predict a speed for memory you do not have.

▤Memory budget

These are ceilings, and optimistic ones in known directions. Engines spend time on arithmetic, kernel launches and sampling, which the formula gives none. The file-size shortcut is slightly high for dense Qwen3 and for Qwen3-30B-A3B, because the embedding table sits in the file but a token reads one row of it; Part 3 counted the tensors and found 4.67 GB against the shortcut’s 5.03 GB for Qwen3-8B. A split row also ignores the CPU’s own arithmetic on the layers it holds. Part 6’s summarise-benchmarks.py reads the bandwidth from this task’s notebook line and sets each measured speed beside its ceiling.

7. Write the numbers down where Part 6 will find them

The scripts appended JSON lines, which later scripts parse. The notebook also needs the human-readable version, with the judgements no script can make. Add this section to labbook.md, one line per field:

Fragment — not complete on its own

labbook.md, the Part 5 measurement section

## Part 5: bandwidth and compute, YYYY-MM-DD

- Conditions: mains or battery; minutes idle before task 2; what was still running
- Devices: the device lines bandwidth-test.py printed; Track S: container or wheel
- Read GB/s, accelerator, 64 / 512 / 1,024 MiB: ___ / ___ / ___ (largest spread ___ %)
- Read GB/s, CPU, 64 / 512 / 1,024 MiB: ___ / ___ / ___
- Copy GB/s, accelerator and CPU, 1,024 MiB: ___ and ___
- Published bandwidth: ___ GB/s, from ___; read ÷ published: ___
- The cause I think dominates the gap, and the check that supports it: ___
- TFLOPS at 4,096: fp32 ___, bf16 ___, fp16 ___, fp8 ___, int8w ___ (figure or note)
- TFLOPS at 8,192: fp32 ___, bf16 ___, fp16 ___
- Vendor compute figure: ___; comparable or not, and why: ___
- Ridge point at bf16 (the Validation check prints it): ___ FLOP per byte
- Track M: MLX read ___ GB/s against mps read ___ GB/s; float16 ___, affine4 ___, mxfp4 ___ TFLOPS; max_recommended_working_set_size ___ GiB
- Budget: ___ GB, from the task 6 command for my track
- Ceilings, Q4_K_M, at 0 / 4,096 tokens: Qwen3-8B ___ / ___; Qwen3-30B-A3B ___ / ___; Qwen3-32B ___ / ___; fit verdicts ___

The Q4_K_M ceilings are the ones Part 6 tests first: its tg128 runs start with an empty cache, the first column, and its depth runs at 4,096 tokens are the second.

Read the measurement as a controlled experiment

Run the smallest array or matrix size first and inspect the device and dtype reported by the script. Then increase the size using the lesson’s sequence. Record warm-up policy, synchronisation, competing processes and sustained power conditions. A number returned before asynchronous device work finishes does not measure the operation you intended.

Use a separate row for each backend, dtype and size. Do not average CPU and accelerator results or substitute a vendor specification for a missing measurement. A format that is unsupported should have a not-run or failed row with its diagnostic, rather than a speed inferred from another format.

Compare the largest representative result with the specification only after reconciling read/write traffic accounting. Then feed your measured bandwidth into the decode prediction and label it as an estimate with explicit weight-residency assumptions. Keep the raw measurements as well as the prediction: Part 6 needs both to explain why a real engine can fall below the simple ceiling. The lab succeeds when you can explain the units, timing boundary and major gap, even if the device does not achieve the rate you expected.

Validation

Save this as check-part05.py in ~/llm-course. It reads the notebook, checks the three things every track must have, checks the MLX line on a Mac, and prints the two numbers the section above asks for that no script has printed yet: the ratio to the published figure and the ridge point.

RunnableAll tracks

check-part05.py

"""Check the Part 5 notebook lines. Usage: python check-part05.py labbook.md [published GB/s]"""
import json
import sys

path = sys.argv[1]
spec = float(sys.argv[2]) if len(sys.argv) > 2 else None
rows = {}
with open(path, encoding="utf-8") as notebook:
    for line in notebook:
        if line.startswith('{"') and '"part-05/' in line:
            record = json.loads(line)
            rows.setdefault(record["lab"], []).append(record)
bw = rows.get("part-05/bandwidth-test", [])
mm = rows.get("part-05/matmul-test", [])
pd = rows.get("part-05/predict-decode", [])
checks = [
    ("bandwidth-test at three buffer sizes", len({r["size_mb"] for r in bw}) >= 3),
    ("matmul-test with a bf16 entry", any("bf16" in r["tflops"] for r in mm)),
    ("predict-decode: six predictions with a budget",
     any(len(r["predictions"]) >= 6 and r.get("budget_gb") for r in pd)),
]
if sys.platform == "darwin":
    checks.append(("mlx-bandwidth-test", "part-05/mlx-bandwidth-test" in rows))
for label, ok in checks:
    print(f"{'PASS' if ok else 'FAIL'}  {label}")
if bw:
    last = bw[-1]
    for device, fig in last["gbps"].items():
        ratio = f" = {fig['read'] / spec:.2f} of {spec:g}" if spec and device == last["device"] else ""
        print(f"      {device}: read {fig['read']} GB/s{ratio}, copy {fig['copy']} GB/s, "
              f"{last['size_mb']} MiB run")
    bf16 = next((r["tflops"]["bf16"] for r in reversed(mm)
                 if isinstance(r["tflops"].get("bf16"), float)), None)
    if bf16 is not None:
        read = last["gbps"][last["device"]]["read"]
        print(f"      ridge point at bf16: {bf16 * 1e12 / (read * 1e9):,.0f} FLOP per byte")
sys.exit(0 if all(ok for _, ok in checks) else 1)

Run it with your published figure; 273 is the Spark’s, so use your own track’s:

RunnableAll tracks

check the notebook

python check-part05.py labbook.md 273
echo "exit status $?"

Output — what you should see

PASS  bandwidth-test at three buffer sizes
PASS  matmul-test with a bf16 entry
PASS  predict-decode: six predictions with a budget
      cuda: read xxx.x GB/s = x.xx of 273, copy xxx.x GB/s, 1024 MiB run
      cpu: read xx.x GB/s, copy xx.x GB/s, 1024 MiB run
      ridge point at bf16: x,xxx FLOP per byte
exit status 0

The lab passes when every line starts with PASS, the exit status is 0, the ratio line names the 1,024 MiB run, and the notebook section from task 7 has no blank fields other than those for another track. A FAIL names the task to repeat. On a CPU-only run the ratio appears on the cpu line.

Expected outcome

Two measured numbers for your machine with their method and conditions, a ratio against the vendor’s published figure with a named and checked cause, a precision ladder including two lower precisions, and six decode ceilings with fit verdicts written before you have run a single model.

=Expected result

Whatever your figures, the shape should be: every read ratio below 1.00; on Tracks S, X and M the CPU and accelerator read figures much closer to each other than on Track N; int8w no faster than bf16; an fp8 row that is a figure, a slow path or unsupported, recorded as whichever it was; and the Qwen3-30B-A3B ceiling several times the Qwen3-32B one at Q4_K_M.

Troubleshooting

Symptom	Cause	Fix
Preflight prints cpu only on Track S or N	the environment holds a CPU wheel, or the Track S shell is not the container	Part 1’s troubleshooting; on Track S, start the container with setup-env-spark.sh
bandwidth-test: cuda failed with --size-mb 1024: followed by an out-of-memory line	the buffers do not fit in free accelerator memory	run that size with --size-mb 512 --iters 50; on Track N, check the Processes table of nvidia-smi
bandwidth-test: mps failed ... on Track M	the wired-memory limit	as task 2’s Track M tab says
spread above 10 per cent	something else ran, or the machine throttled	close it, wait idle, rerun that size; record both lines
Track N: the accelerator read figure at or below the CPU’s	the scripts measured another device, or a laptop was on battery	check task 2’s device line names the card; plug the laptop in and rerun
Accelerator read figure far lower at 64 MiB than at 1,024 MiB	fixed per-call cost, task 2’s arithmetic	expected; not a fault
slow path on a GPU for bf16 or fp16	the probe shared the device with another program	close it and rerun; --max-seconds 120 lets a slow format run anyway
fp8 ... unsupported: on hardware advertised for FP8	plain torch.matmul has no FP8 kernel; FP8 arithmetic goes through scaled kernels	record it as unsupported by this method; Part 8
MLX: unsupported: on a quantised row	this MLX build or GPU rejects that mode	record the message and the MLX version
predict-decode: no part-05/bandwidth-test line with a read figure in labbook.md	task 2 ran without --labbook, or against another file	rerun task 2’s last command with --labbook labbook.md
predict-decode: cannot read models.json	the file is not in this directory	copy it as the preflight says, or pass its path
Every row fits on a small card	the budget was given in MiB, or is the machine’s memory rather than the card’s	print "$BUDGET_GB"; it must be GB of accelerator memory
predict-decode: --split needs an accelerator run with a cpu figure	the latest bandwidth line is CPU-only	use --split only on Track N with the card visible
check-part05.py shows FAIL bandwidth-test at three buffer sizes	one of task 2’s runs was skipped or not recorded	run the missing size with --labbook labbook.md

Cleanup

Nothing runs after the scripts exit, and they write only to labbook.md. On Track S’s container path, type exit to leave the container; setup-env-spark.sh started it with --rm, so it is removed, and your files stay in ~/llm-course. Keep the scripts, check-part05.py and models.json: Part 6 reads the notebook lines and models.json, and Part 12 reads the bf16 figure. When you add a machine later, run task 2 on it the same way so the figures compare.

What you learned

Objective	The observation that proved it	Recorded as
Bandwidth on accelerator and CPU	read and copy figures at three sizes, with spreads, from a workload with almost no arithmetic per byte	three part-05/bandwidth-test lines; the section’s read and copy fields
Throughput at BF16 and lower precision	a ladder in which int8w trails bf16 by the conversion’s cost and fp8 has no plain kernel (or, on a Mac, MLX’s quantised kernels have their own rates)	two part-05/matmul-test lines; the TFLOPS fields
A specification is a claim	read ÷ published below 1.00, the published figure rebuilt from rate × width, and a cause checked against your own rows	the ratio, the published source and the cause field
Prediction before measurement	six ceilings from your own bandwidth and budget, with the dense and mixture-of-experts pair about ten times apart	the part-05/predict-decode line; the ceilings field
Numbers kept for later	a check script that passes and a dated section	check-part05.py output, the Part 5 section

Check your understanding

Question 1. A colleague's memory-copy benchmark reports about half of your copy figure on the same machine. What is the most likely explanation?

• One of the two machines is faulty
• Their tool counts each copied byte once, while the copy test counts it read and written
• The colleague ran the test on the GPU
• The read test is wrong

Show the answer and why

Answer: Their tool counts each copied byte once, while the copy test counts it read and written

A copy moves each byte twice. Counting both directions or one is a convention, and a factor of two follows from it alone. The read test counts bytes read, which is what decode does, so it is the figure that goes into predictions; the copy figure is for comparing with copy-based benchmarks once you know their convention.

Question 2. Your accelerator read figure is 250 gigabytes per second. Qwen3-8B at Q4_K_M is 5.0 GB and its cache is 147,456 bytes per token. What ceiling, in tokens per second, does the formula give with 4,096 tokens already in the cache?

• About 50
• About 45
• About 41
• About 25

Show the answer and why

Answer: About 45

The cache is 147,456 × 4,096 = 603,979,776 bytes, 0.60 GB, so each token reads 5.60 GB and 250 ÷ 5.60 = 44.6. With an empty cache it would be 50. The cache is read every token like the weights, so a long context lowers the ceiling as well as filling memory.

Question 3. Which of these predict-decode.py invocations, run on a 24 GB card, is the bug?

• python predict-decode.py --models models.json --budget-gb 22.5 --split --labbook labbook.md
• python predict-decode.py --models models.json --budget-gb 24564 --split --labbook labbook.md
• python predict-decode.py --models models.json --bandwidth-gbps 900 --budget-gb 22.5
• python predict-decode.py --models models.json --budget-gb 22.5 --labbook labbook.md

Show the answer and why

Answer: python predict-decode.py --models models.json --budget-gb 24564 --split --labbook labbook.md

24564 is the card's memory.total in MiB from nvidia-smi. Passed as GB it declares a 24.5 TB budget, every row "fits", and the BF16 rows get ceilings for memory the card does not have instead of split estimates. The fourth command is not wrong, only less informative: without --split, rows over budget print "over".

Question 4. On a CPU, matmul-test.py reports bf16 at 4.3 and int8w at 3.5, both in TFLOPS. What does the gap show?

• INT8 arithmetic is slower than BF16 arithmetic on every chip
• The int8w path performs the same BF16 multiplication plus a conversion from INT8 on every call, so beyond run-to-run noise it cannot be faster; a weight-only format saves bytes, not arithmetic, unless a native kernel exists
• The run was unsteady and the two should be equal
• The int8w path reads more bytes than bf16

Show the answer and why

Answer: The int8w path performs the same BF16 multiplication plus a conversion from INT8 on every call, so beyond run-to-run noise it cannot be faster; a weight-only format saves bytes, not arithmetic, unless a native kernel exists

int8w stores half the bytes of bf16 but multiplies in bf16 after converting. Decode, which waits on bytes, benefits from the smaller storage; prefill, which waits on arithmetic, pays the conversion. Native low-precision kernels, such as MLX's quantized_matmul or FP8 kernels in the engines, are what change the arithmetic itself.

Question 5. Why does this lab ask for the predictions to be written down before Part 6 runs the models?

• To save time later
• Because a prediction made in advance can be wrong in a way that teaches something, while an explanation offered after the measurement cannot be
• Because the models are not available yet
• Because Part 6 requires the file format

Show the answer and why

Answer: Because a prediction made in advance can be wrong in a way that teaches something, while an explanation offered after the measurement cannot be

The gap between a prior prediction and a later measurement is where the learning is: it points at the key-value cache, at engine overhead or at an offload you did not know was happening. A number reverse-engineered afterwards always fits.

Check my answersClear