Lab: Same Model, Every Engine

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The engine versions, build tags and container tags each track was run with will be recorded here when the validation pass is done.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this lab you will have served one model, Qwen3-8B (Apache-2.0, not gated, as the model reference records), from llama-server and from the native engines on your track, one after another on the same machine. You will have predicted each engine’s decode ceiling and memory from the bytes in its model file before starting it, measured time to first token, prefill rate, decode rate and memory through each server’s own API, and probed whether each server returns a real tool call and a schema-conforming JSON object.

The feature columns are the ones people leave out. An engine that generates faster but cannot return a tool call is not faster for Part 24 and Part 25; it is unusable there, and one that ignores a JSON schema fails Part 10’s structured output. Every number is taken with the same prompts, context length, generation length and script, and every notebook line carries the engine, version, quantisation and machine, so that Part 9 and Part 17 can lay their measurements beside yours.

Architecture

One machine, one server at a time. Two servers loaded together measure their competition for memory rather than either engine, so each serve script refuses its own busy port, the port loop before each start checks the others, and Task 3 records what the machine looks like with no model loaded.

The measurement loop, repeated once per engine

1. 1Idle snapshotcompare-engines.py --snapshot-only records memory with no model server running. Taken once.
2. 2Start one serverA serve script checks its inputs, states every setting that changes the result, and runs the server in the foreground of terminal 1.
3. 3Read its load logDevice, weights, KV cache and compute buffers, compared with the prediction from Task 1.
4. 4Measurecompare-engines.py in terminal 2: two prompts, a warm-up and three measured runs each, memory as a difference from the idle snapshot.
5. 5Probefeature-probe.py: model listing, tool calling, JSON-schema output.
6. 6Stop it and confirm the port is freeThen the next engine.
7. 7Summarise and checksummarise-engines.py turns the notebook lines into the recording sheets and runs the validation rules.

Engine	Port	Started by	Model it loads
llama-server (every track; HIP build on Track X)	8080	serve-llama-cpp.sh	~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
mlx_lm.server (Track M)	8081	serve-mlx.sh	~/models/mlx-community/Qwen3-8B-4bit
trtllm-serve in the NGC container (Track S)	8082	serve-trtllm.sh	~/models/Qwen/Qwen3-8B (bf16)
vLLM (Track N)	8083	serve-vllm.sh	~/models/Qwen/Qwen3-8B-AWQ
TabbyAPI with ExLlamaV3 (Track N)	5000	serve-exllamav3.sh	~/models/turboderp/Qwen3-8B-exl3-4.0bpw
Lemonade Server (Track X)	13305	its own service; serve-lemonade.sh loads the model	Lemonade’s copy of the same Q4_K_M file

Everything listens on 127.0.0.1 only. Work in ~/llm-course, where Part 1 created labbook.md and Part 4 left fetch-model.sh, and save every lab file on this page there with its download link.

Requirements

Every track needs the llama.cpp build from Part 6’s install lesson (llama.cpp v0.4.0 · verified 2026-09-08), the hf CLI from Part 4 (Hugging Face CLI 1.30.0 · verified 2026-09-08), Python 3.9 or later, and the lab notebook. The GGUF file is common to all tracks: Qwen3-8B-Q4_K_M.gguf, 5,027,784,512 bytes in the repository listing read on 2026-09-13.

Memory floor. 12 GB (video memory on Track N). The largest four-bit allocation in the lab is 6.10 GB of AWQ weights plus a 1.25 GiB KV cache; the bf16 path runs only on Track S.

Track	Portable engine	Native engines measured against it	Second-format download	Disk to free
S	llama-server, CUDA build	trtllm-serve, NGC release container	Qwen/Qwen3-8B bf16, 16.4 GB; the release:1.3.0rc13 image, 19,547,210,461 bytes compressed for arm64 in the NGC catalogue on 2026-09-13	25 GB plus the unpacked image, whose size NGC does not publish: check it with docker image ls after the preflight pull
X	llama-server, Vulkan build	llama-server HIP build; Lemonade Server with its ROCm backend	Lemonade’s own copy of the Q4_K_M file, 5.0 GB	12 GB
M	llama-server, Metal build	mlx_lm.server	mlx-community/Qwen3-8B-4bit, 4.6 GB	11 GB
N	llama-server, CUDA build	TabbyAPI with ExLlamaV3; vLLM	EXL3 4.0 bpw, 5.2 GB; AWQ, 6.1 GB	17 GB plus two Python environments with PyTorch, whose size neither installer states: Task 7 measures them with du -sh

Track S — NVIDIA DGX Spark

Pinned: TensorRT-LLM 1.2.1 · verified 2026-09-08; the container tags carry 1.3 release candidates (Task 7). Attended time about 70 minutes; unattended, the 16.4 GB checkpoint and the container pull, which the DGX Spark playbook budgets within its 45 to 60 minutes of setup. Start both before reading on.

The manifest allows vLLM as the Track S native engine instead. That path belongs to Part 9, whose lab compares vLLM against the llama-server numbers you record here.

Track X — AMD Ryzen AI Max+ 395

Track X measures three engines rather than two: two backends of your own llama.cpp build, and Lemonade Server, which ships its own llama.cpp builds. Pinned: Lemonade Server current · verified 2026-09-09. Attended time about 75 minutes; unattended, Lemonade’s 5.0 GB pull and its first ROCm backend install. You need the Vulkan build in ~/llama.cpp/build from Part 6, the HIP build in ~/llama.cpp/build-hip from the AMD lesson, and Lemonade installed from AMD’s playbook. The GPU-visible memory cap is not the constraint at this model size.

Run this lab on Linux. A Windows installation runs Lemonade natively, but this page’s scripts assume a POSIX shell and /proc/meminfo.

Track M — Apple silicon

Pinned: mlx-lm 0.31.3 · verified 2026-09-08. Attended time about 55 minutes; unattended, the 4.6 GB MLX download. Any Mac with 16 GB or more runs the primary path; close the browser and other large applications first, because on unified memory they are part of the measurement.

Track N — NVIDIA desktop or laptop

Pinned: ExLlamaV3 1.4.8 · verified 2026-09-08, TabbyAPI current · verified 2026-09-09 and vLLM 0.28.0 · verified 2026-09-08. Attended time about 80 minutes; unattended, the 11.3 GB of EXL3 and AWQ files, TabbyAPI’s first start (it installs PyTorch and ExLlamaV3) and the vLLM install. The vLLM row is optional below 16 GB of video memory.

Windows readers work in WSL2, following Part 5’s CUDA-on-WSL setup, with every command on this page run inside the Linux distribution. NVIDIA’s CUDA on WSL guide lists “active compute process” among the NVML queries not yet supported, so under WSL2 the memory column falls back to the whole device’s figure (Task 7).

Preflight

Run the common check, then your track’s.

RunnableAll tracks

preflight: tools, disk, free ports

cd ~/llm-course
python3 --version
hf version
ls fetch-model.sh labbook.md
df -h ~/models
for port in 8080 8081 8082 8083 5000 13305; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"

Output — what you should see

Python 3.12.x
✓ hf version
  version: 1.30.0
fetch-model.sh  labbook.md
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  xxxG  xxxG  xxxG  xx% /
port check finished

Avail must exceed the disk figure for your track. A port … is in use line means a server from an earlier part is still running: stop it now. The one exception is port 13305 on Track X, which is Lemonade’s service and is expected to answer.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

preflight: CUDA build, Docker, container GPU access

~/llama.cpp/build/bin/llama-server --version
~/llama.cpp/build/bin/llama-server --list-devices
docker --version
docker run --rm --gpus all nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 nvidia-smi

Output — what you should see

version: x.x.x (build xxxxx, commit xxxxxxx)
built with ... for aarch64
Available devices:
  CUDA0: NVIDIA GB10 (xxxxxx MiB, xxxxxx MiB free)
Docker version xx.x.x, build xxxxxxx
... the nvidia-smi table, naming the GB10, with Memory-Usage reported as Not Supported ...

The last command is the validation step of NVIDIA’s DGX Spark playbook for TensorRT-LLM, read on 2026-09-13, and its first run pulls the image. Not Supported under memory usage is documented behaviour on this machine (NVIDIA’s DGX Spark known-issues page); Task 3 explains what the lab reads instead. Available devices: (none) means a CPU-only build: rebuild before going on.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

preflight: both builds, ROCm SMI, Lemonade

~/llama.cpp/build/bin/llama-server --list-devices
~/llama.cpp/build-hip/bin/llama-server --list-devices
rocm-smi --showmeminfo vram gtt
lemonade --version
lemonade status

Output — what you should see

Available devices:
  Vulkan0: AMD Radeon ... (xxxxx MiB, xxxxx MiB free)
Available devices:
  ROCm0: AMD Radeon ... (xxxxx MiB, xxxxx MiB free)
... a Memory Usage (Bytes) block with VRAM Total Memory (B), VRAM Total Used Memory (B),
    GTT Total Memory (B) and GTT Total Used Memory (B) ...
x.x.x
... a line saying the server is running on port 13305 ...

Device names come from llama.cpp’s backends: Vulkan0 for the Vulkan build, ROCm0 for the HIP build. If lemonade status cannot reach the server: AMD’s playbook states only that the server runs in the background after installation, and the Lemonade repository’s systemd unit is named lemond (Troubleshooting).

Track M — Apple silicon

RunnableTrack M · Apple silicon

preflight: Metal build and mlx-lm

~/llama.cpp/build/bin/llama-server --list-devices
command -v mlx_lm.server
"$(head -n 1 "$(command -v mlx_lm.server)" | sed 's/^#!//')" -c 'import mlx_lm; print(mlx_lm.__version__)'
sysctl -n hw.memsize

Output — what you should see

Available devices:
  MTL0: Apple M... (xxxxx MiB, xxxxx MiB free)
/Users/you/.../bin/mlx_lm.server
0.31.3
xxxxxxxxxxx

The third line runs the interpreter named on the first line of the mlx_lm.server script, which is the Python whose packages the server uses. A version other than 0.31.3 is worth recording, not fixing.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

preflight: CUDA build, card, uv, git

~/llama.cpp/build/bin/llama-server --list-devices
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
uv --version
git --version

Output — what you should see

Available devices:
  CUDA0: NVIDIA GeForce RTX ... (xxxxx MiB, xxxxx MiB free)
name, memory.total [MiB], driver_version
NVIDIA GeForce RTX ..., xxxxx MiB, xxx.xx
uv 0.12.x
git version 2.xx.x

Now write the per-track variables every later command reads, editing HOST_DESC to describe your machine. Each terminal you open for this lab starts with cd ~/llm-course and source lab.env.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

lab.env for Track S

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export BACKEND=CUDA
export HOST_DESC="Track S, DGX Spark GB10, 128 GB"
export TRTLLM_IMAGE=nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13
EOF

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

lab.env for Track X

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export HIP_BIN=$HOME/llama.cpp/build-hip/bin
export BACKEND=Vulkan
export HOST_DESC="Track X, Ryzen AI Max+ 395, 128 GB"
EOF

Track M — Apple silicon

RunnableTrack M · Apple silicon

lab.env for Track M

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export BACKEND=Metal
export HOST_DESC="Track M, M4 Max, 64 GB"
export MLX_MODEL=$HOME/models/mlx-community/Qwen3-8B-4bit
export MLX_PY="$(head -n 1 "$(command -v mlx_lm.server)" | sed 's/^#!//')"
EOF

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

lab.env for Track N

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export BACKEND=CUDA
export HOST_DESC="Track N, RTX 4090 24 GB, Ubuntu 24.04"
export TABBY_DIR=$HOME/tabbyAPI
EOF

Tasks

1. Fix the variables, then predict the results

A comparison is only as good as its list of things that were not allowed to vary.

Held fixed	Value	Why it matters
Model	Qwen3-8B, one checkpoint family	Different weights are a different experiment
Context allocated	8,192 tokens, one sequence	Sets the KV cache size and, on some engines, the whole memory reservation
Generation length	128 tokens, temperature 0	Decode rate is averaged over the same count
Prompts	A one-line question and a 16-paragraph passage, byte-identical across engines	Their difference isolates prefill
Prefix cache	Defeated by a random first line in every request	A repeated prompt measures the cache, not prefill (Task 5)
Repetitions	One warm-up, three measured runs, medians	First requests pay for caches, compilation and allocation
Load	One server at a time	Two loaded models measure contention

What cannot be held fixed is the number format, because each engine reads its own. The file headers say exactly how each format spends its bits. Qwen3-8B has 8,190,735,360 parameters: 6,945,767,424 in the 36 layers’ weight matrices, two 622,329,856-parameter vocabulary tables (the input embedding and the output head), and 308,224 in norms.

Format	How the bits are spent	Arithmetic	Tensor bytes in the published file
GGUF Q4_K_M	6.59 B parameters in Q4_K blocks (4.5 bits), 1.60 B in Q6_K blocks (6.5625 bits), from the file’s tensor table	3.705 GB + 1.316 GB	5,021,827,072
MLX 4-bit, group 64	Every matrix, both vocabulary tables included, at 4 bits plus a 16-bit scale and bias per 64 weights: 4.5 bits	8.19 B × 4.5 / 8	4,607,731,712
EXL3 4.0 bpw	Layers at 4.0 bits, output head at 6 (head_bits in its config), input embedding kept at 16	3.473 + 0.467 + 1.245 = 5.185 GB, plus per-tensor scales	5,190,675,200
AWQ 4-bit, group 128	Layers at 4 bits plus a 16-bit scale and 4-bit zero per 128 weights, both vocabulary tables at 16	3.609 + 2.489 GB	6,098,479,104
bf16	Everything at 16 bits	8.19 B × 2	16,381,470,720

Decode reads every weight once per token, except the input embedding, which is a lookup of one row. So the bytes read per generated token are the tensor bytes minus the embedding table, plus the KV cache the new token attends over:

Pseudocode — not a real command

KV bytes per token     = 2 (K and V) × 36 layers × 8 KV heads × 128 head_dim × 2 bytes = 147,456
bytes read per token   = tensor bytes − embedding bytes + KV bytes per token × tokens in context
decode ceiling (tok/s) = memory bandwidth (bytes/s) / bytes read per token

Save this as predict.py, set PAIRS to your own machine’s bandwidth (Part 5 measured it; the hardware reference lists vendor figures) and run it.

RunnableAll tracks

predict.py

# Decode ceilings and memory for Qwen3-8B in each format, from stated inputs only.
# Tensor bytes and embedding-table bytes: read from each repository's file headers (2026-09-13).
FORMATS = {  # name: (all tensor bytes, embedding table bytes)
    "GGUF Q4_K_M":     (5_021_827_072,   350_060_544),
    "MLX 4-bit g64":   (4_607_731_712,   350_060_544),
    "EXL3 4.0 bpw":    (5_190_675_200, 1_244_659_712),
    "AWQ 4-bit g128":  (6_098_479_104, 1_244_659_712),
    "bf16":           (16_381_470_720, 1_244_659_712),
}
KV_PER_TOKEN = 2 * 36 * 8 * 128 * 2       # K and V, layers, KV heads, head_dim, bytes (fp16)
PAIRS = {  # your machine's bandwidth in GB/s (hardware.json), and the formats your track runs
    "Spark, 273 GB/s":    (273,  ["GGUF Q4_K_M", "bf16"]),
    "Strix, 256 GB/s":    (256,  ["GGUF Q4_K_M"]),
    "M4 Max, 546 GB/s":   (546,  ["GGUF Q4_K_M", "MLX 4-bit g64"]),
    "RTX 4090, 1008 GB/s": (1008, ["GGUF Q4_K_M", "EXL3 4.0 bpw", "AWQ 4-bit g128"]),
}
print(f"KV cache: {KV_PER_TOKEN:,} bytes per token; 8,192 tokens = "
      f"{KV_PER_TOKEN * 8192:,} bytes = {KV_PER_TOKEN * 8192 / 2**20:,.2f} MiB")
print(f"{'machine':<21}{'format':<16}{'weights GB':>11}{'read/token GB':>14}"
      f"{'ceiling tok/s':>14}{'at 1,500 ctx':>13}")
for machine, (gbps, names) in PAIRS.items():
    for name in names:
        total, embedding = FORMATS[name]
        read = total - embedding              # the embedding is a lookup: one row per token
        read_long = read + KV_PER_TOKEN * 1500  # decode also reads the cache it attends over
        print(f"{machine:<21}{name:<16}{total / 1e9:>11.2f}{read / 1e9:>14.2f}"
              f"{gbps * 1e9 / read:>14.1f}{gbps * 1e9 / read_long:>13.1f}")

Output — what you should see

KV cache: 147,456 bytes per token; 8,192 tokens = 1,207,959,552 bytes = 1,152.00 MiB
machine              format           weights GB read/token GB ceiling tok/s at 1,500 ctx
Spark, 273 GB/s      GGUF Q4_K_M            5.02          4.67          58.4         55.8
Spark, 273 GB/s      bf16                  16.38         15.14          18.0         17.8
Strix, 256 GB/s      GGUF Q4_K_M            5.02          4.67          54.8         52.3
M4 Max, 546 GB/s     GGUF Q4_K_M            5.02          4.67         116.9        111.6
M4 Max, 546 GB/s     MLX 4-bit g64          4.61          4.26         128.2        121.9
RTX 4090, 1008 GB/s  GGUF Q4_K_M            5.02          4.67         215.8        206.0
RTX 4090, 1008 GB/s  EXL3 4.0 bpw           5.19          3.95         255.4        241.9
RTX 4090, 1008 GB/s  AWQ 4-bit g128         6.10          4.85         207.7        198.6

That output is arithmetic from the stated inputs, not a measurement. The ceilings ignore compute entirely, so a fast card can land well below them, and the last column is why the long prompt’s decode rate should come out a few per cent lower than the short one’s on a bandwidth-bound machine. Two consequences to write down before measuring anything: on Track S the bf16 engine starts with a decode handicap of 4.67 / 15.14, about 0.31, that no kernel can remove; on Track X all three rows read the same bytes, so every decode difference there is software.

Memory depends as much on each engine’s allocation policy as on the file:

Engine as this lab configures it	Weights	KV cache	Allocation policy
llama-server	5,021,827,072 bytes, of which the 350,060,544-byte input embedding stays in a CPU_Mapped host buffer	1,152.00 MiB at load	Allocates the whole context when it loads; host-RAM prompt cache disabled with --cache-ram 0 (default 8,192 MiB)
mlx_lm.server	4,607,731,712 bytes	Grows in steps of 256 tokens as a request uses it	Allocates on demand; keeps finished caches for reuse, limited to one here
trtllm-serve	16,381,470,720 bytes	Capped at 8,192 tokens by kv_cache_config.max_tokens	Otherwise a fraction of free GPU memory, 0.9 by default
vLLM	6,098,479,104 bytes	1,342,177,280 bytes set explicitly	Otherwise --gpu-memory-utilization of the whole card
TabbyAPI with ExLlamaV3	5,190,675,200 bytes, of which the 1,244,659,712-byte 16-bit input embedding stays in host RAM (prefer_cpu in ExLlamaV3 1.4.8’s modules/embedding.py)	cache_size 8,192 tokens, FP16	Allocates the configured cache size
Lemonade	Same file as llama-server	--ctx-size 8192	Runs a llama.cpp server underneath

Record: your bandwidth figure, the ceiling for each format your track runs, and the predicted decode ratio between native and portable engines.

2. Get the model in each format

The GGUF file comes through Part 4’s script, which verifies the SHA-256 the Hub publishes:

RunnableAll tracks

the portable format, verified

cd ~/llm-course
bash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf labbook.md

Output — what you should see

==> Looking up Qwen3-8B-Q4_K_M.gguf in unsloth/Qwen3-8B-GGUF@main
    expected sha256 120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4
    expected size   5027784512 bytes
...
==> Done: /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf

If Part 6 already downloaded it, the script re-verifies it and downloads nothing. The other formats are multi-file repositories, so they use hf download directly. Appending --dry-run to any of them prints the file list and total without downloading; the totals quoted in each tab are what hf 1.30.0 printed that way on 2026-09-13.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

the bf16 checkpoint trtllm-serve will load

hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen/Qwen3-8B
ls ~/models/Qwen/Qwen3-8B/*.safetensors | wc -l

Output — what you should see

... progress ...
✓ Downloaded
  path: /home/you/models/Qwen/Qwen3-8B
5

The dry run reports [dry-run] Will download 15 files (out of 15) totalling 16.4G. Five safetensors shards is the pass condition. The serve script mounts ~/models read-only into the container, so nothing is downloaded a second time into a container cache.

Track X — AMD Ryzen AI Max+ 395

Lemonade downloads into its own cache and needs its own registration. Its built-in Qwen3-8B-GGUF entry points at unsloth/Qwen3-8B-GGUF:Q4_1 in the registry file read on 2026-09-13, a different quantisation from the control, so register the exact variant under a user. name, as the CLI reference documents:

RunnableTrack X · Ryzen AI Max+

register and pull the same Q4_K_M file in Lemonade

lemonade pull user.Qwen3-8B-Q4_K_M \
    --checkpoint main unsloth/Qwen3-8B-GGUF:Q4_K_M \
    --recipe llamacpp
lemonade list --downloaded

Output — what you should see

... download progress ...
... user.Qwen3-8B-Q4_K_M listed among the downloaded models ...

The exact wording of both commands’ output is not documented; the pass condition is that user.Qwen3-8B-Q4_K_M appears in the downloaded list.

Track M — Apple silicon

RunnableTrack M · Apple silicon

the MLX four-bit conversion

hf download mlx-community/Qwen3-8B-4bit --local-dir ~/models/mlx-community/Qwen3-8B-4bit
grep -A3 '"quantization"' ~/models/mlx-community/Qwen3-8B-4bit/config.json

Output — what you should see

... progress ...
✓ Downloaded
  path: /Users/you/models/mlx-community/Qwen3-8B-4bit
    "quantization": {
        "group_size": 64,
        "bits": 4
    },

The dry run reports [dry-run] Will download 11 files (out of 11) totalling 4.6G. group_size 64 and bits 4 are the quantisation the table in Task 1 assumes. If you would rather make your own, the MLX lesson’s convert command with -q --q-bits 4 --q-group-size 64 produces the same recipe from a 16.4 GB download; set MLX_MODEL in lab.env to its output directory.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

the EXL3 and AWQ checkpoints

hf download turboderp/Qwen3-8B-exl3 --revision 4.0bpw \
    --local-dir ~/models/turboderp/Qwen3-8B-exl3-4.0bpw
hf download Qwen/Qwen3-8B-AWQ --local-dir ~/models/Qwen/Qwen3-8B-AWQ
grep -A3 '"quant_method": "exl3"' ~/models/turboderp/Qwen3-8B-exl3-4.0bpw/config.json

Output — what you should see

... progress ...
✓ Downloaded
  path: /home/you/models/turboderp/Qwen3-8B-exl3-4.0bpw
... progress ...
✓ Downloaded
  path: /home/you/models/Qwen/Qwen3-8B-AWQ
        "quant_method": "exl3",
        "version": "0.0.1",
        "bits": 4.0,
        "head_bits": 6,

The dry runs report [dry-run] Will download 10 files (out of 10) totalling 5.2G. for the EXL3 branch and [dry-run] Will download 12 files (out of 12) totalling 6.1G. for AWQ. turboderp/Qwen3-8B-exl3 is published by ExLlamaV3’s author, and its card names Qwen/Qwen3-8B as the base model under Apache-2.0; each bit rate is a branch, which is why --revision selects it. Qwen/Qwen3-8B-AWQ is Qwen’s own conversion, also Apache-2.0.

Converting your own EXL3 file is the alternative, and the course has not run it: from an ExLlamaV3 checkout, python convert.py --in_dir ~/models/Qwen/Qwen3-8B --out_dir ~/models/local/Qwen3-8B-exl3-4.0bpw --work_dir ~/exl3-work --bits 4.0, per the argument definitions in 1.4.8, after a 16.4 GB bf16 download.

Record: repository, revision or file, and total size for every format.

3. Take the idle memory snapshot

Stop every model server. On Track X also run lemonade unload. Then, in terminal 2:

RunnableAll tracks

compare-engines.py

#!/usr/bin/env python3
"""Measure prefill, decode and memory for one OpenAI-compatible server.

Purpose: send the same two prompts to whichever engine is listening, time the first token
    and the tokens after it from the streaming response, derive prefill from the difference
    between a short and a long prompt, read what the machine reports about memory while the
    model is loaded (as a difference from an idle snapshot taken before any server started),
    and append one JSON line per test to the lab notebook so that engines on the same machine
    can be compared honestly.
Platform: all (Python standard library only; the memory readers use /proc/meminfo, nvidia-smi,
    rocm-smi or vm_stat, whichever exist)
Minimum memory: 12 GB
Assumes: Python 3.9 or later; a server already running and reachable at --base-url, serving
    the model named by --model; for memory deltas, an idle snapshot written earlier with
    --snapshot-only while no model server was running.

Usage: python3 compare-engines.py --snapshot-only idle-memory.json
       python3 compare-engines.py --engine llama.cpp --base-url http://127.0.0.1:8080/v1 \
           --model Qwen3-8B-Q4_K_M --quant Q4_K_M --context-length 8192 \
           --engine-version "0.4.0 (build 12345)" --backend CUDA --host "DGX Spark" \
           --server-pid "$(cat llama-server.pid)" --idle-snapshot idle-memory.json \
           --labbook labbook.md

Method, in the order the script applies it:
  * Two tests. "short" is a one-line question; "long" repeats a fixed paragraph so that the
    prompt is some 1,500 tokens longer. Both ask for --max-tokens tokens.
  * Every request starts with a random run tag. Engines keep the KV cache of recent prompts
    and skip prefill for a prefix they have seen (llama-server, vLLM, TensorRT-LLM and
    mlx_lm.server all do this by default), so an identical prompt sent twice measures the
    cache, not prefill. A different first line makes every prompt new. The server's own
    count of reused tokens is recorded as max_cached_prompt_tokens so that you can check.
  * One unmeasured warm-up request per test, then --repetitions measured ones; medians.
  * Time to first token (ttft) is from sending the request to the first streamed chunk that
    carries text: answer text, reasoning text (reasoning_content or reasoning) or a tool call.
    Qwen3 thinks before answering, and servers put that text in different fields.
  * Decode rate is (completion tokens - 1) / (time from first to last text chunk), because
    prefill produces the first token. Completion tokens come from the server's usage block
    when it sends one, and from counting chunks otherwise (token_source says which).
  * Prefill rate is taken from the difference between the tests:
        (prompt_tokens_long - prompt_tokens_short) / (ttft_long - ttft_short)
    which cancels the fixed costs (HTTP, scheduling, the first decode step) that a single
    ttft includes. llama-server also reports its own rates in a "timings" block; when present
    they are recorded as server_prefill_tokens_per_s and server_decode_tokens_per_s.
"""

from __future__ import annotations

import argparse
import json
import platform
import shutil
import statistics
import subprocess
import sys
import time
import urllib.error
import urllib.request
import uuid
from pathlib import Path

LAB = "part-08/lab-same-model-every-engine"

SHORT_PROMPT = "In two sentences, say what a KV cache is and why it grows with context length."

LONG_PARAGRAPH = (
    "A language model reads its prompt in one pass and then writes its answer one token at a "
    "time. The first phase is compute bound because every weight that is read is reused across "
    "many tokens of the prompt. The second phase is bandwidth bound because every weight has to "
    "be read again for each single token that is produced. Any measurement that reports one "
    "number for both phases has thrown away the distinction that explains how the machine "
    "behaves. "
)

FOLLOW_UP = "Summarise the passage above in exactly three sentences."

TEXT_FIELDS = ("content", "reasoning_content", "reasoning")


def long_prompt(repeats: int) -> str:
    """A deterministic prompt of `repeats` copies of the paragraph, ending in an instruction."""
    return (LONG_PARAGRAPH * repeats) + "\n\n" + FOLLOW_UP


def headers(api_key: str, accept: str) -> dict:
    out = {"Content-Type": "application/json", "Accept": accept}
    if api_key:
        out["Authorization"] = f"Bearer {api_key}"
    return out


def wait_for_server(args) -> None:
    """Poll GET {base}/models until it answers 200, so a slow load is not measured as a failure."""
    url = args.base_url.rstrip("/") + "/models"
    deadline = time.monotonic() + args.wait_seconds
    last = "no answer yet"
    while True:
        request = urllib.request.Request(url, headers=headers(args.api_key, "application/json"))
        try:
            with urllib.request.urlopen(request, timeout=10) as response:  # noqa: S310 - local server
                if response.status == 200:
                    return
                last = f"status {response.status}"
        except urllib.error.HTTPError as exc:
            last = f"status {exc.code}"
        except (urllib.error.URLError, TimeoutError, OSError) as exc:
            last = str(exc)
        if time.monotonic() >= deadline:
            raise RuntimeError(f"{url} did not answer 200 within {args.wait_seconds} s ({last})")
        time.sleep(2)


def post_stream(base_url: str, api_key: str, body: dict, timeout: float):
    """POST a streaming chat completion and yield (arrival_time, parsed_chunk) pairs."""
    url = base_url.rstrip("/") + "/chat/completions"
    data = json.dumps(body).encode("utf-8")
    request = urllib.request.Request(
        url, data=data, headers=headers(api_key, "text/event-stream"), method="POST"
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - local server
        while True:
            raw = response.readline()
            if not raw:
                return
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[len("data:"):].strip()
            if payload == "[DONE]":
                return
            try:
                yield time.perf_counter(), json.loads(payload)
            except json.JSONDecodeError:
                continue


def one_run(args, prompt: str) -> dict:
    """One streamed completion, timed. Returns raw timings and counts rather than rates."""
    content = prompt if args.reuse_prompt else f"[run {uuid.uuid4().hex[:12]}]\n{prompt}"
    body = {
        "model": args.model,
        "messages": [{"role": "user", "content": content}],
        "max_tokens": args.max_tokens,
        "temperature": 0,
        "stream": True,
        "stream_options": {"include_usage": True},
    }
    started = time.perf_counter()
    first_at = None
    last_at = None
    chunks = 0
    usage = None
    timings = None
    finish_reason = None
    fields_seen = set()

    for arrived, chunk in post_stream(args.base_url, args.api_key, body, args.timeout):
        if chunk.get("usage"):
            usage = chunk["usage"]
        if chunk.get("timings"):
            timings = chunk["timings"]
        for choice in chunk.get("choices") or []:
            delta = choice.get("delta") or {}
            carried = [f for f in TEXT_FIELDS if isinstance(delta.get(f), str) and delta.get(f)]
            if delta.get("tool_calls"):
                carried.append("tool_calls")
            if carried:
                fields_seen.update(carried)
                chunks += 1
                if first_at is None:
                    first_at = arrived
                last_at = arrived
            if choice.get("finish_reason"):
                finish_reason = choice["finish_reason"]

    if first_at is None:
        raise RuntimeError("the server streamed no text; check the model name and the server log")

    usage = usage or {}
    details = usage.get("prompt_tokens_details") or {}
    cached = details.get("cached_tokens")
    if cached is None and timings:
        cached = timings.get("cache_n")
    return {
        "ttft_s": first_at - started,
        "decode_s": max(last_at - first_at, 1e-9),
        "chunks": chunks,
        "prompt_tokens": usage.get("prompt_tokens"),
        "completion_tokens": usage.get("completion_tokens"),
        "cached_tokens": cached,
        "finish_reason": finish_reason,
        "fields": sorted(fields_seen),
        "server_prefill": (timings or {}).get("prompt_per_second"),
        "server_decode": (timings or {}).get("predicted_per_second"),
    }


def rates(run: dict) -> dict:
    """Turn one run's timings into rates, saying where the token counts came from."""
    if run["completion_tokens"]:
        produced, source = run["completion_tokens"], "usage"
    else:
        produced, source = run["chunks"], "chunks"
    after_first = max(produced - 1, 1)
    return {
        "ttft_s": run["ttft_s"],
        "decode_tokens_per_s": after_first / run["decode_s"],
        "prompt_tokens": run["prompt_tokens"],
        "completion_tokens": produced,
        "token_source": source,
        "cached_tokens": run["cached_tokens"],
        "finish_reason": run["finish_reason"],
        "fields": run["fields"],
        "server_prefill": run["server_prefill"],
        "server_decode": run["server_decode"],
    }


def run_command(command: list) -> str | None:
    """Run a short informational command; return its output, or None if it is unavailable."""
    if not shutil.which(command[0]):
        return None
    try:
        out = subprocess.run(command, capture_output=True, text=True, timeout=20, check=False)
    except (OSError, subprocess.SubprocessError):
        return None
    return out.stdout.strip() if out.returncode == 0 else None


def to_float(text) -> float | None:
    try:
        return float(str(text).strip())
    except (TypeError, ValueError):
        return None


def memory_snapshot() -> dict:
    """Every memory figure this machine will report, in MB (1 MB = 1,000,000 bytes)."""
    snap = {"taken_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}

    meminfo = Path("/proc/meminfo")
    if meminfo.exists():
        values = {}
        for line in meminfo.read_text(encoding="utf-8").splitlines():
            key, _, rest = line.partition(":")
            parts = rest.split()
            if parts and parts[0].isdigit():
                values[key] = int(parts[0]) * 1024 / 1e6  # the file reports kB (KiB)
        snap["mem_total_mb"] = values.get("MemTotal")
        snap["mem_available_mb"] = values.get("MemAvailable")
        snap["swap_free_mb"] = values.get("SwapFree")

    text = run_command(["nvidia-smi", "--query-gpu=memory.used,memory.total",
                        "--format=csv,noheader,nounits"])
    if text:
        fields = [f.strip() for f in text.splitlines()[0].split(",")]
        snap["gpu_used_mb"] = to_float(fields[0]) if fields else None  # MiB; null on a GB10
        snap["gpu_total_mb"] = to_float(fields[1]) if len(fields) > 1 else None
        # Per-process figures are recorded only when at least one row parses to a number. Under
        # WSL2, NVIDIA's CUDA on WSL guide lists "active compute process" among the NVML
        # queries not yet supported, so the query returns no usable rows there; recording 0
        # would make every server look as if it added nothing.
        apps = run_command(["nvidia-smi", "--query-compute-apps=pid,used_memory",
                            "--format=csv,noheader,nounits"])
        if apps is not None:
            used = [to_float(line.split(",")[1]) for line in apps.splitlines() if "," in line]
            used = [u for u in used if u is not None]
            if used:
                snap["gpu_process_mb"] = sum(used)

    text = run_command(["rocm-smi", "--showmeminfo", "vram", "gtt", "--json"])
    if text:
        try:
            cards = json.loads(text)
        except json.JSONDecodeError:
            cards = {}
        vram = gtt = 0.0
        for card in cards.values() if isinstance(cards, dict) else []:
            vram += to_float(card.get("VRAM Total Used Memory (B)")) or 0.0
            gtt += to_float(card.get("GTT Total Used Memory (B)")) or 0.0
        snap["rocm_vram_used_mb"] = vram / 1e6
        snap["rocm_gtt_used_mb"] = gtt / 1e6

    text = run_command(["vm_stat"])
    if text:
        page = 4096
        pages = {}
        for line in text.splitlines():
            if "page size of" in line:
                digits = [t for t in line.replace("(", " ").split() if t.isdigit()]
                page = int(digits[0]) if digits else page
            key, _, rest = line.partition(":")
            rest = rest.strip().rstrip(".")
            if rest.isdigit():
                pages[key.strip()] = int(rest)
        mb = lambda k: pages.get(k, 0) * page / 1e6  # noqa: E731 - tiny local helper
        snap["mac_free_mb"] = mb("Pages free")
        snap["mac_wired_mb"] = mb("Pages wired down")
        snap["mac_compressed_mb"] = mb("Pages occupied by compressor")
        snap["mac_used_mb"] = mb("Pages active") + snap["mac_wired_mb"] + snap["mac_compressed_mb"]
        swap = run_command(["sysctl", "-n", "vm.swapusage"])
        if swap and "used =" in swap:
            snap["mac_swap_used_mb"] = to_float(swap.split("used =")[1].split()[0].rstrip("M"))
    return snap


def memory_delta(idle: dict, loaded: dict) -> dict:
    """Memory the server added, per source: loaded minus idle (idle minus loaded for 'available')."""
    out = {}
    if idle.get("mem_available_mb") is not None and loaded.get("mem_available_mb") is not None:
        out["mem_available_mb"] = round(idle["mem_available_mb"] - loaded["mem_available_mb"])
    for key in ("gpu_used_mb", "gpu_process_mb", "rocm_vram_used_mb", "rocm_gtt_used_mb",
                "mac_wired_mb", "mac_used_mb"):
        before = idle.get(key)
        if key == "gpu_process_mb" and before is None and idle.get("gpu_total_mb"):
            before = 0.0  # nvidia-smi answered at idle and listed no compute process
        if before is not None and loaded.get(key) is not None:
            out[key] = round(loaded[key] - before)
    return out


def process_resident_mb(pid: int) -> float | None:
    """Resident set size of the server process from ps, in MB. Only the process itself."""
    if not pid:
        return None
    text = run_command(["ps", "-o", "rss=", "-p", str(pid)])
    try:
        return round(float(text.split()[0]) * 1024 / 1e6) if text else None
    except (IndexError, ValueError):
        return None


def median(runs: list, key: str):
    values = [r[key] for r in runs if r.get(key) is not None]
    return statistics.median(values) if values else None


def measure(args, label: str, prompt: str) -> dict:
    """Warm up once, repeat the test, take medians."""
    if not args.no_warmup:
        one_run(args, prompt)
    runs = [rates(one_run(args, prompt)) for _ in range(args.repetitions)]
    cached = [r["cached_tokens"] for r in runs if r["cached_tokens"] is not None]
    return {
        "lab": LAB,
        "engine": args.engine,
        "engine_version": args.engine_version,
        "backend": args.backend,
        "host": args.host,
        "model": args.model,
        "quant": args.quant,
        "context_length": args.context_length,
        "test": label,
        "repetitions": args.repetitions,
        "max_tokens": args.max_tokens,
        "ttft_s": round(median(runs, "ttft_s"), 4),
        "decode_tokens_per_s": round(median(runs, "decode_tokens_per_s"), 2),
        "prompt_tokens": median(runs, "prompt_tokens"),
        "completion_tokens": median(runs, "completion_tokens"),
        "token_source": runs[-1]["token_source"],
        "finish_reason": runs[-1]["finish_reason"],
        "text_fields": runs[-1]["fields"],
        "max_cached_prompt_tokens": max(cached) if cached else None,
        "prompt_tag": not args.reuse_prompt,
        "server_prefill_tokens_per_s": median(runs, "server_prefill"),
        "server_decode_tokens_per_s": median(runs, "server_decode"),
        "measured_on": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--snapshot-only", metavar="FILE",
                        help="write a memory snapshot to FILE and exit (run with no server up)")
    parser.add_argument("--engine", help="engine name as it should appear in the notebook")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1", help="OpenAI-compatible base URL")
    parser.add_argument("--model", help="model name to send in every request")
    parser.add_argument("--api-key", default="", help="bearer token, when the server wants one")
    parser.add_argument("--engine-version", default="", help="the version string you read from the engine")
    parser.add_argument("--backend", default="", help="CUDA, Metal, Vulkan, ROCm, and so on")
    parser.add_argument("--host", default="", help="short description of the machine")
    parser.add_argument("--quant", default="", help="quantisation of the weights being served")
    parser.add_argument("--context-length", type=int, default=0, help="context the server was started with")
    parser.add_argument("--server-pid", type=int, default=0, help="process id of the server, for ps")
    parser.add_argument("--idle-snapshot", default="", help="snapshot file written by --snapshot-only")
    parser.add_argument("--max-tokens", type=int, default=128, help="tokens to generate per run")
    parser.add_argument("--long-repeats", type=int, default=16, help="paragraph repeats in the long prompt")
    parser.add_argument("--repetitions", type=int, default=3, help="measured runs per test")
    parser.add_argument("--timeout", type=float, default=600.0, help="seconds to wait for one response")
    parser.add_argument("--wait-seconds", type=int, default=120, help="seconds to wait for the server to answer")
    parser.add_argument("--no-warmup", action="store_true", help="skip the unmeasured first run")
    parser.add_argument("--reuse-prompt", action="store_true",
                        help="omit the run tag, so repeats hit the prefix cache (a demonstration, not a measurement)")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--print-only", action="store_true", help="print the results, record nothing")
    args = parser.parse_args()

    if args.snapshot_only:
        snap = memory_snapshot()
        snap["system"] = f"{platform.system()} {platform.machine()}"
        Path(args.snapshot_only).write_text(json.dumps(snap, indent=2) + "\n", encoding="utf-8")
        readings = {k: round(v) for k, v in snap.items() if isinstance(v, float)}
        print(f"    idle snapshot written to {args.snapshot_only}")
        print(f"    {json.dumps(readings)}")
        return 0

    if not args.engine or not args.model:
        parser.error("--engine and --model are required unless --snapshot-only is given")

    idle = {}
    if args.idle_snapshot:
        try:
            idle = json.loads(Path(args.idle_snapshot).read_text(encoding="utf-8"))
        except (OSError, json.JSONDecodeError) as exc:
            print(f"    cannot read {args.idle_snapshot}: {exc}", file=sys.stderr)
            return 1

    tests = [("short", SHORT_PROMPT), ("long", long_prompt(args.long_repeats))]
    records = []
    try:
        wait_for_server(args)
        for label, prompt in tests:
            records.append(measure(args, label, prompt))
    except (urllib.error.URLError, RuntimeError, TimeoutError, OSError) as exc:
        print(f"    FAILED: {exc}", file=sys.stderr)
        print("    Usually: the server is not up yet, the model name differs from GET /v1/models,",
              file=sys.stderr)
        print("    or the server needs --api-key.", file=sys.stderr)
        return 1

    loaded = memory_snapshot()
    delta = memory_delta(idle, loaded) if idle else {}
    rss = process_resident_mb(args.server_pid)
    short, long_ = records
    marginal = None
    if (long_["prompt_tokens"] and short["prompt_tokens"]
            and long_["ttft_s"] > short["ttft_s"]):
        marginal = round((long_["prompt_tokens"] - short["prompt_tokens"])
                         / (long_["ttft_s"] - short["ttft_s"]), 1)
    for record in records:
        record["prefill_tokens_per_s"] = marginal
        record["memory"] = loaded
        record["memory_delta_mb"] = delta
        record["process_resident_mb"] = rss

    for r in records:
        prompt = f"{r['prompt_tokens']:.0f}" if r["prompt_tokens"] is not None else "?"
        print(f"    {args.engine:>14} {r['test']:>5}  prompt {prompt:>5} tok  ttft {r['ttft_s']:.3f} s  "
              f"decode {r['decode_tokens_per_s']:.2f} tok/s  "
              f"{r['completion_tokens']:.0f} tok ({r['finish_reason']}), counts from {r['token_source']}, "
              f"cached {r['max_cached_prompt_tokens']}")
    if marginal is not None:
        extra = long_["prompt_tokens"] - short["prompt_tokens"]
        print(f"    {args.engine:>14} prefill from the difference: {marginal:.1f} tok/s over {extra:.0f} extra prompt tokens")
    else:
        print(f"    {args.engine:>14} prefill from the difference: not computable (no prompt token counts)")
    if long_["server_prefill_tokens_per_s"] is not None:
        print(f"    {args.engine:>14} server's own timings, long test: prefill "
              f"{long_['server_prefill_tokens_per_s']:.1f} tok/s, decode {long_['server_decode_tokens_per_s']:.2f} tok/s")
    print(f"    {args.engine:>14} memory added since the idle snapshot (MB): {json.dumps(delta) if delta else 'no idle snapshot given'}")
    print(f"    {args.engine:>14} server process resident set (MB): {rss}")

    if args.print_only:
        return 0
    notebook = Path(args.labbook)
    if not notebook.exists():
        print(f"    {notebook} does not exist; creating it", file=sys.stderr)
    with notebook.open("a", encoding="utf-8") as handle:
        for record in records:
            handle.write(json.dumps(record) + "\n")
    print(f"    recorded {len(records)} line(s) in {notebook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download compare-engines.py475 lines

RunnableAll tracks

memory with no model loaded

cd ~/llm-course
python3 compare-engines.py --snapshot-only idle-memory.json

Output — what you should see

    idle snapshot written to idle-memory.json
    {"mem_total_mb": xxxxxx, "mem_available_mb": xxxxxx, "swap_free_mb": xxxx}

That is the Linux shape. Track N adds gpu_used_mb, gpu_total_mb and gpu_process_mb from nvidia-smi; Track X adds rocm_vram_used_mb and rocm_gtt_used_mb from rocm-smi --showmeminfo vram gtt; Track M has no /proc/meminfo and prints mac_free_mb, mac_wired_mb, mac_compressed_mb, mac_used_mb and mac_swap_used_mb from vm_stat and sysctl.

After each measurement the script takes the same snapshot again and records the difference, which works the same way for a bare process, a container and a service. Task 8 reports one source per track: nvidia-smi’s per-process figure on a discrete card (its device figure under WSL2), rocm-smi’s VRAM plus GTT on Track X, the macOS pages on Track M, and MemAvailable on a DGX Spark, where nvidia-smi reads Not Supported and NVIDIA’s known-issues page points developers at /proc/meminfo instead. The process resident set from ps is also recorded, but it misses GPU allocations on discrete cards, and for a container or a service there is no single process to ask.

4. Start the portable engine and read its load log

RunnableAll tracks

serve-llama-cpp.sh

#!/usr/bin/env bash
# Purpose: start llama-server as the portable control in this lab's engine comparison, with
#          every setting that changes the result stated on the command line rather than left
#          to a default, the load log copied to a file you can search, and the server's
#          process id written where the measuring script can read it
# Platform: all (CUDA, Metal, Vulkan and HIP builds; set LLAMA_BIN to choose which one)
# Minimum memory: 12 GB
# Assumes: llama.cpp built as in Part 6, a GGUF file in $MODEL, the port free and nothing
#          else large running; the server runs in the foreground so that Ctrl-C stops it
#
# Usage: MODEL=~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf bash serve-llama-cpp.sh
#        LLAMA_BIN=~/llama.cpp/build-hip/bin PIDFILE=./llama-hip.pid LOGFILE=./llama-hip.log \
#            MODEL=... bash serve-llama-cpp.sh
#
# Environment:
#   MODEL       path to the GGUF file                    (required)
#   LLAMA_BIN   directory holding llama-server           (default: $HOME/llama.cpp/build/bin)
#   HOST        address to bind                          (default: 127.0.0.1)
#   PORT        port to listen on                        (default: 8080)
#   CTX         context length, in tokens                (default: 8192)
#   NGL         layers to offload                        (default: 999, meaning all)
#   FLASH_ATTN  on, off or auto                          (default: on)
#   PARALLEL    concurrent slots                         (default: 1)
#   ALIAS       model name reported by the API           (default: the file's base name)
#   PIDFILE     where to write the server's pid          (default: ./llama-server.pid)
#   LOGFILE     copy of the server log                   (default: ./llama-server.log)
#
# One slot and an explicit context mean the KV cache is 8,192 tokens for one sequence, which
# is the allocation every other engine in the lab is configured to match. --jinja (the
# default in current builds, passed anyway) applies the model's own chat template, which is
# what turns a "tools" array into something Qwen3 was trained to answer. --verbosity 4 is the
# level at which the pinned build prints the device, model buffer, KV cache and compute buffer
# sizes during load; at the default level 3 those lines are not shown. --cache-ram 0 disables
# the host-RAM prompt cache: the pinned build otherwise keeps up to 8192 MiB of earlier
# prompts' KV state in host RAM and copies the previous prompt's state into it at the start of
# each new request, which adds to this engine's memory figure and to the timed time to first
# token ("-cram, --cache-ram N ... (default: 8192, -1 - no limit, 0 - disable)" in v0.4.0's
# --help).

set -euo pipefail

MODEL="${MODEL:-}"
LLAMA_BIN="${LLAMA_BIN:-$HOME/llama.cpp/build/bin}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8080}"
CTX="${CTX:-8192}"
NGL="${NGL:-999}"
FLASH_ATTN="${FLASH_ATTN:-on}"
PARALLEL="${PARALLEL:-1}"
PIDFILE="${PIDFILE:-./llama-server.pid}"
LOGFILE="${LOGFILE:-./llama-server.log}"

die() { echo "serve-llama-cpp: $*" >&2; exit 1; }

[ -n "$MODEL" ] || die "set MODEL to the GGUF file to serve"
[ -f "$MODEL" ] || die "$MODEL does not exist"
[ -x "$LLAMA_BIN/llama-server" ] || die "no llama-server in $LLAMA_BIN; set LLAMA_BIN"
if command -v curl >/dev/null && curl --silent --max-time 2 "http://$HOST:$PORT/" >/dev/null 2>&1; then
  die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi

ALIAS="${ALIAS:-$(basename "$MODEL" .gguf)}"

echo "==> llama-server"
"$LLAMA_BIN/llama-server" --version 2>&1 || true
echo "    model   $MODEL"
echo "    alias   $ALIAS"
echo "    listen  http://$HOST:$PORT/v1"
echo "    context $CTX tokens, ngl $NGL, flash attention $FLASH_ATTN, $PARALLEL slot(s)"
echo "    pid written to $PIDFILE, log copied to $LOGFILE"

rm -f "$LOGFILE"
echo $$ > "$PIDFILE"

# exec keeps this shell's pid, so $PIDFILE names the server process itself.
exec "$LLAMA_BIN/llama-server" \
    --model "$MODEL" \
    --alias "$ALIAS" \
    --host "$HOST" \
    --port "$PORT" \
    --ctx-size "$CTX" \
    --n-gpu-layers "$NGL" \
    --flash-attn "$FLASH_ATTN" \
    --parallel "$PARALLEL" \
    --jinja \
    --cache-ram 0 \
    --verbosity 4 \
    --log-file "$LOGFILE"

Download serve-llama-cpp.sh88 lines

RunnableAll tracks

terminal 1: the control

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
MODEL=$GGUF bash serve-llama-cpp.sh

Output — what you should see

port check finished
==> llama-server
version: x.x.x (build xxxxx, commit xxxxxxx)
built with ... for ...
    model   /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
    alias   Qwen3-8B-Q4_K_M
    listen  http://127.0.0.1:8080/v1
    context 8192 tokens, ngl 999, flash attention on, 1 slot(s)
    pid written to ./llama-server.pid, log copied to ./llama-server.log
... load log ...
x.xx.xxx.xxx I srv  llama_server: model loaded
x.xx.xxx.xxx I srv  llama_server: listening on http://127.0.0.1:8080

A port … is in use line before port check finished means another server is still loaded: stop it before going on. The script passes --verbosity 4 because the pinned build prints its device and buffer lines at that level and not at the default of 3, and --cache-ram 0 because v0.4.0 otherwise keeps up to 8,192 MiB of earlier prompts’ KV state in host RAM and copies the previous prompt into it at the start of each request, inside the timed window; both were checked by running the v0.4.0 server. Pull the lines that matter out of the log copy:

RunnableAll tracks

terminal 2: what it allocated

cd ~/llm-course
grep -E "using device|offloaded|model buffer size|KV buffer size|llama_kv_cache: size|compute buffer size|prompt cache is" llama-server.log

Output — what you should see

x.xx.xxx.xxx I llama_prepare_model_devices: using device CUDA0 (NVIDIA ...) (xxxx:xx:xx.x) - xxxxx MiB free
x.xx.xxx.xxx I load_tensors: offloaded 37/37 layers to GPU
x.xx.xxx.xxx I load_tensors:        CUDA0 model buffer size =  xxxx.xx MiB
x.xx.xxx.xxx I load_tensors:   CPU_Mapped model buffer size =   xxx.xx MiB
x.xx.xxx.xxx I llama_kv_cache:      CUDA0 KV buffer size =  1152.00 MiB
x.xx.xxx.xxx I llama_kv_cache: size = 1152.00 MiB (  8192 cells,  36 layers,  1/1 seqs), K (f16):  576.00 MiB, V (f16):  576.00 MiB
x.xx.xxx.xxx I sched_reserve:      CUDA0 compute buffer size =   xxx.xx MiB
x.xx.xxx.xxx I srv    load_model: prompt cache is disabled - use `--cache-ram N` to enable it

The KV lines are exact: the format string in llama-kv-cache.cpp filled with the arithmetic from Task 1; the same format string printed 896.00 MiB ( 8192 cells, 28 layers for a 28-layer model when this script was tested on the pinned build. 37/37 is 36 layers plus the output layer. The device is CUDA0 on Tracks S and N, Vulkan0 on Track X and MTL0 on Track M, whose model buffer is named MTL0_Mapped. The two model buffers should sum to about the 5,021,827,072 tensor bytes, around 4,789 MiB, with roughly the 334 MiB embedding table on the CPU side, because the model loader keeps the input layer on the CPU. If the device line is missing or the layers read 0/37, stop: everything after this would measure the CPU (Troubleshooting). The last line confirms the prompt cache is off; the log also carries a warning, --cache-idle-slots requires --cache-ram, disabling, which is the expected consequence of the same flag.

Record: the version line, device, offloaded layers, both model buffers, KV buffer, compute buffer.

5. Measure it

The script’s docstring states the method; three parts of it deserve a reason.

Time to first token counts reasoning text. Qwen3 thinks before it answers. llama-server, TensorRT-LLM and TabbyAPI stream that text in reasoning_content, while vLLM 0.28.0 and mlx-lm 0.31.3 use reasoning; a client that watches only content sees nothing for the first hundred-odd tokens and, on the long prompt, often nothing at all. The script counts all three fields, so every engine’s 128 tokens include the same thinking.

Prefill comes from the difference between the prompts. A single time to first token includes the HTTP round trip, scheduling and the first decode step. Subtracting the short test removes them:

Illustrative inputs, not a measurement	Short test	Long test	Difference
Prompt tokens (the tokeniser’s count, which the server reports)	41	1,378	1,337
Time to first token	0.20 s	1.60 s	1.40 s
Prefill rate			1,337 / 1.40 = 955 tokens per second

Every prompt starts with a random tag. All four serving stacks keep recent prompts’ KV cache and skip prefill for a prefix they have already seen: llama-server’s --cache-prompt is on by default, TensorRT-LLM’s KV cache page says block reuse is on by default, vLLM 0.28.0 enables prefix caching by default for models that support it, and mlx_lm.server keeps a prompt cache. Without the tag, the warm-up would turn every measured long run into a cache hit. The server’s own count of reused tokens is recorded, so this is checked, not assumed.

RunnableAll tracks

terminal 2: measure the control

cd ~/llm-course
source lab.env
python3 compare-engines.py \
    --engine llama.cpp \
    --base-url http://127.0.0.1:8080/v1 \
    --model Qwen3-8B-Q4_K_M \
    --quant Q4_K_M \
    --context-length 8192 \
    --engine-version "$("$LLAMA_BIN/llama-server" --version 2>&1 | head -n 1)" \
    --backend "$BACKEND" \
    --host "$HOST_DESC" \
    --server-pid "$(cat llama-server.pid)" \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md

Output — what you should see

         llama.cpp short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
         llama.cpp  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
         llama.cpp prefill from the difference: xxxx.x tok/s over xxxx extra prompt tokens
         llama.cpp server's own timings, long test: prefill xxxx.x tok/s, decode xx.xx tok/s
         llama.cpp memory added since the idle snapshot (MB): {"mem_available_mb": xxxx}
         llama.cpp server process resident set (MB): xxxx
    recorded 2 line(s) in labbook.md

Four things to check in that output. 128 tok (length) on both lines: the model was still thinking when the budget ran out, so both tests generated the same count. counts from usage: the server reported token counts, so the rates are exact. cached in single digits: the tag worked and only the chat template’s opening tokens were reused. And the server’s own timings line, which appears only when the server sends llama.cpp’s timings block, should agree with the client-side prefill and decode figures to within a few per cent; a large disagreement means the client is timing something other than generation.

▷Exercise

See the cache for yourself. Add --reuse-prompt --print-only to the command above. cached climbs to within a token of the prompt length, the long test’s time to first token falls to about the short test’s, and the difference-based prefill figure becomes absurdly large. Nothing is recorded.

Record: everything the script appended; the two notebook lines hold it.

6. Probe what it can actually do

RunnableAll tracks

feature-probe.py

#!/usr/bin/env python3
"""Test what an OpenAI-compatible server actually supports, rather than what it claims.

Purpose: send three probes to a server - a model listing, a tool-calling request and a
    JSON-schema structured-output request - and record for each whether the server accepted
    it, whether the answer had the shape the OpenAI Chat Completions API specifies, and the
    exact reason when it did not. One JSON line per server is appended to the lab notebook.
Platform: all (Python standard library only)
Minimum memory: 12 GB
Assumes: Python 3.9 or later; a server already running and reachable at --base-url with the
    model named by --model loaded. Nothing is installed and nothing is written except the
    notebook line.

Usage: python3 feature-probe.py --engine llama.cpp --base-url http://127.0.0.1:8080/v1 \
           --model Qwen3-8B-Q4_K_M --engine-version "0.4.0" --labbook labbook.md
       python3 feature-probe.py --engine exllamav3 --base-url http://127.0.0.1:5000/v1 \
           --model Qwen3-8B-exl3-4.0bpw --api-key "$TABBY_KEY" --print-only

What counts as supported:
  * tool calling: the reply carries message.tool_calls, the first call names
    get_current_weather, and its arguments are a JSON object with a "city" string. A reply
    whose text contains "<tool_call>" is reported separately: the model produced a call in
    its own format and the server did not convert it, which is a server configuration
    finding, not a model failure.
  * structured output: the request uses response_format {"type": "json_schema", ...}, the
    prompt does not mention JSON or the field names, and the reply parses as a JSON object
    with exactly the three required keys and the right types. A server that ignores
    response_format gets a sentence of prose back and fails, which is the point: only a
    server that constrains generation to the schema can pass reliably.
Both probe prompts end with Qwen3's documented "/no_think" switch (change it with
--prompt-suffix) so that thinking does not consume the token budget. A reply that still
starts with an empty <think></think> block has that block removed before parsing, and the
record says so.
"""

from __future__ import annotations

import argparse
import json
import re
import time
import urllib.error
import urllib.request
from pathlib import Path

LAB = "part-08/lab-same-model-every-engine"

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a named city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name, for example Lisbon"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city"],
        },
    },
}

PERSON_SCHEMA = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "born": {"type": "integer"},
        "field": {"type": "string"},
    },
    "required": ["name", "born", "field"],
    "additionalProperties": False,
}

THINK_BLOCK = re.compile(r"^\s*<think>.*?</think>\s*", re.DOTALL)


def call(base_url: str, path: str, api_key: str, body, timeout: float):
    """One request. Returns (status, parsed body or text) and turns errors into the same shape."""
    url = base_url.rstrip("/") + path
    headers = {"Content-Type": "application/json", "Accept": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    data = json.dumps(body).encode("utf-8") if body is not None else None
    method = "POST" if body is not None else "GET"
    request = urllib.request.Request(url, data=data, headers=headers, method=method)
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - local server
            raw = response.read().decode("utf-8", "replace")
            try:
                return response.status, json.loads(raw)
            except json.JSONDecodeError:
                return response.status, raw[:400]
    except urllib.error.HTTPError as exc:
        return exc.code, exc.read().decode("utf-8", "replace")[:400]
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return 0, str(exc)[:400]


def first_message(response: dict) -> tuple:
    choice = (response.get("choices") or [{}])[0] or {}
    return choice.get("message") or {}, choice.get("finish_reason")


def reasoning_field(message: dict):
    for key in ("reasoning_content", "reasoning"):
        if message.get(key):
            return key
    return None


def probe_models(args) -> dict:
    status, body = call(args.base_url, "/models", args.api_key, None, args.timeout)
    ids = []
    if status == 200 and isinstance(body, dict):
        ids = [m.get("id") for m in body.get("data") or [] if isinstance(m, dict)]
    result = {"status": status, "supported": status == 200, "model_ids": ids[:10]}
    if status != 200:
        result["detail"] = f"status {status}: {str(body)[:160]}"
    return result


def probe_tool_calling(args) -> dict:
    body = {
        "model": args.model,
        "messages": [{"role": "user", "content":
                      "What is the weather in Lisbon right now? Use the tool." + args.prompt_suffix}],
        "tools": [WEATHER_TOOL],
        "tool_choice": "auto",
        "max_tokens": args.max_tokens,
        "temperature": 0,
    }
    status, response = call(args.base_url, "/chat/completions", args.api_key, body, args.timeout)
    result = {"status": status, "supported": False, "called": None, "arguments": None}
    if status != 200 or not isinstance(response, dict):
        result["detail"] = f"status {status}: {str(response)[:160]}"
        return result
    message, finish = first_message(response)
    result["finish_reason"] = finish
    result["reasoning_field"] = reasoning_field(message)
    calls = message.get("tool_calls") or []
    if not calls:
        text = message.get("content") or ""
        if "<tool_call>" in text:
            result["detail"] = "the model wrote a <tool_call> block as text; the server did not parse it"
        elif finish == "length":
            result["detail"] = "ran out of tokens before any tool call (finish_reason length)"
        else:
            result["detail"] = "answered without a tool_calls field: " + text.strip()[:80]
        return result
    function = (calls[0] or {}).get("function") or {}
    result["called"] = function.get("name")
    arguments = function.get("arguments")
    try:
        parsed = json.loads(arguments) if isinstance(arguments, str) else arguments
    except json.JSONDecodeError:
        result["detail"] = "tool_calls present but the arguments were not valid JSON"
        return result
    result["arguments"] = parsed
    ok = (result["called"] == "get_current_weather" and isinstance(parsed, dict)
          and isinstance(parsed.get("city"), str))
    result["supported"] = ok
    if not ok:
        result["detail"] = "tool_calls present but not the expected function and city argument"
    return result


def probe_structured_output(args) -> dict:
    body = {
        "model": args.model,
        "messages": [{"role": "user", "content":
                      "Name one physicist who won a Nobel Prize." + args.prompt_suffix}],
        "response_format": {
            "type": "json_schema",
            "json_schema": {"name": "person", "strict": True, "schema": PERSON_SCHEMA},
        },
        "max_tokens": args.max_tokens,
        "temperature": 0,
    }
    status, response = call(args.base_url, "/chat/completions", args.api_key, body, args.timeout)
    result = {"status": status, "supported": False, "value": None, "think_block_removed": False}
    if status != 200 or not isinstance(response, dict):
        result["detail"] = f"status {status}: {str(response)[:160]}"
        return result
    message, finish = first_message(response)
    result["finish_reason"] = finish
    result["reasoning_field"] = reasoning_field(message)
    content = message.get("content") or ""
    stripped = THINK_BLOCK.sub("", content, count=1)
    result["think_block_removed"] = stripped != content
    try:
        value = json.loads(stripped)
    except json.JSONDecodeError:
        result["detail"] = "accepted the request but the content was not JSON: " + stripped.strip()[:80]
        return result
    result["value"] = value
    if not isinstance(value, dict):
        result["detail"] = "valid JSON, but not an object"
        return result
    problems = []
    missing = [k for k in PERSON_SCHEMA["required"] if k not in value]
    extra = [k for k in value if k not in PERSON_SCHEMA["properties"]]
    if missing:
        problems.append(f"missing keys {missing}")
    if extra:
        problems.append(f"extra keys {extra}")
    if "born" in value and not (isinstance(value["born"], int) and not isinstance(value["born"], bool)):
        problems.append("born is not an integer")
    for key in ("name", "field"):
        if key in value and not isinstance(value[key], str):
            problems.append(f"{key} is not a string")
    result["supported"] = not problems
    if problems:
        result["detail"] = "valid JSON that does not match the schema: " + "; ".join(problems)
    return result


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--engine", required=True, help="engine name as it should appear in the notebook")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1", help="OpenAI-compatible base URL")
    parser.add_argument("--model", required=True, help="model name to send in the request")
    parser.add_argument("--api-key", default="", help="bearer token, when the server wants one")
    parser.add_argument("--engine-version", default="", help="the version string you read from the engine")
    parser.add_argument("--host", default="", help="short description of the machine")
    parser.add_argument("--prompt-suffix", default=" /no_think", help="appended to both probe prompts")
    parser.add_argument("--max-tokens", type=int, default=1024, help="tokens to allow per probe")
    parser.add_argument("--timeout", type=float, default=300.0, help="seconds to wait for a response")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--print-only", action="store_true", help="print the result, record nothing")
    args = parser.parse_args()

    record = {
        "lab": LAB,
        "probe": "features",
        "engine": args.engine,
        "engine_version": args.engine_version,
        "host": args.host,
        "model": args.model,
        "base_url": args.base_url,
        "models_endpoint": probe_models(args),
        "tool_calling": probe_tool_calling(args),
        "structured_output": probe_structured_output(args),
        "measured_on": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

    def mark(section: str, shown) -> str:
        entry = record[section]
        if entry.get("supported"):
            return "yes " + json.dumps(shown)
        return "no (" + str(entry.get("detail")) + ")"

    models = record["models_endpoint"]
    tools = record["tool_calling"]
    schema = record["structured_output"]
    print(f"    {args.engine}: /v1/models {mark('models_endpoint', models['model_ids'])}")
    print(f"    {args.engine}: tool calling {mark('tool_calling', {tools['called']: tools['arguments']})}")
    print(f"    {args.engine}: JSON schema {mark('structured_output', schema['value'])}")

    if args.print_only:
        return 0
    notebook = Path(args.labbook)
    if not notebook.exists():
        print(f"    {notebook} does not exist; creating it")
    with notebook.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record) + "\n")
    print(f"    recorded 1 line in {notebook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download feature-probe.py272 lines

RunnableAll tracks

terminal 2: three probes against the same server

cd ~/llm-course
source lab.env
python3 feature-probe.py \
    --engine llama.cpp \
    --base-url http://127.0.0.1:8080/v1 \
    --model Qwen3-8B-Q4_K_M \
    --engine-version "$("$LLAMA_BIN/llama-server" --version 2>&1 | head -n 1)" \
    --host "$HOST_DESC" \
    --labbook labbook.md

Output — what you should see

    llama.cpp: /v1/models yes ["Qwen3-8B-Q4_K_M"]
    llama.cpp: tool calling yes {"get_current_weather": {"city": "Lisbon"}}
    llama.cpp: JSON schema yes {"name": "...", "born": xxxx, "field": "..."}
    recorded 1 line in labbook.md

That shape is what the v0.4.0 server returned when the probe was tested against it. The probe is deliberately strict. Tool calling passes only when message.tool_calls names the function with a city argument; prose about the weather fails, because a program cannot act on prose. The structured-output prompt never mentions JSON or the field names, so a server that ignores response_format gets a sentence back and fails, and only a server that constrains generation can pass reliably. A pass means the shape was honoured, not that the answer is true: a small model can return a well-formed object naming the wrong physicist.

Whether a server passes depends on machinery you can read before running anything:

Engine, version read	Turns the model’s call text into tool_calls when	Honours response_format json_schema when	Prediction
llama-server v0.4.0	--jinja (the default) and a template with a tool section, which Qwen3’s has	Always: the server converts the schema to a grammar	Both pass
mlx_lm.server 0.31.3	The tokeniser’s tool parser recognises the model’s call format	server.py contains no response_format handling	Tool calling may pass; JSON schema is expected to fail
trtllm-serve 1.2.1	--tool_parser qwen3, which the serve script passes	guided_decoding_backend: xgrammar in the YAML, which the script writes	Both should pass; see the note below
vLLM 0.28.0	--enable-auto-tool-choice --tool-call-parser hermes	By default	Both pass
TabbyAPI, commit read 2026-09-13	model.tool_format names a supported format; its table lists none for the original Qwen3 call format	Always, with the schema unwrapped from the OpenAI envelope	JSON schema passes; tool calling is expected to report a <tool_call> block written as text
Lemonade	tools is a documented chat-completions parameter	response_format is not in the documented parameter list	Record what your version does

The TensorRT-LLM note: in the 1.2.1 source, response_format.json_schema is passed to the grammar compiler whole, without lifting out the inner schema object the OpenAI format wraps it in; the main branch read the same day does lift it out. A container built from older source can therefore return valid JSON that misses the required keys, which the probe reports as valid JSON that does not match the schema.

✕Common mistake

Recording a server-configuration failure as an engine limitation. A model that writes <tool_call> into its text has done its part; the server lacked the parser flag or format setting that converts it. The probe says which happened, so put its detail string in the notebook, not a bare “no”.

Stop the server with Ctrl-C in terminal 1, then confirm the port is free:

RunnableAll tracks

terminal 2: nothing left on 8080

curl --silent --max-time 2 http://127.0.0.1:8080/health || echo "port 8080 is free"

Output — what you should see

port 8080 is free

7. Start the native engines and repeat

Each block below is complete; run them in order, one server at a time. Every block that starts a server opens with the port loop from Task 4, and its output must begin with port check finished and no in use line.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

serve-trtllm.sh

#!/usr/bin/env bash
# Purpose: start trtllm-serve on the same model the portable engine is serving, inside the NGC
#          release container with the flags NVIDIA's documentation requires, with the KV cache
#          capped at the lab's context length, the Qwen3 tool and reasoning parsers selected,
#          and guided decoding enabled so that JSON-schema requests can be honoured
# Platform: spark (also valid on nvidia when the card has the memory for bf16 weights)
# Minimum memory: 12 GB
# Assumes: an NVIDIA GPU with a working driver; Docker with the NVIDIA Container Toolkit when
#          trtllm-serve is not on PATH; the checkpoint already downloaded under $MODELS_DIR;
#          network access to nvcr.io the first time the image is pulled; the port free
#
# Usage: TRTLLM_IMAGE=nvcr.io/nvidia/tensorrt-llm/release:<tag> bash serve-trtllm.sh
#        MODEL=$HOME/models/Qwen/Qwen3-8B PORT=8082 TRTLLM_IMAGE=... bash serve-trtllm.sh
#
# Environment:
#   MODEL         checkpoint directory on the host      (default: $HOME/models/Qwen/Qwen3-8B)
#   MODELS_DIR    directory mounted read-only as /models (default: $HOME/models)
#   TRTLLM_IMAGE  NGC image, when trtllm-serve is not on PATH (no default; record the tag)
#   HOST          address to publish on                 (default: 127.0.0.1)
#   PORT          port to listen on                     (default: 8082)
#   MAX_SEQ_LEN   longest request, prompt plus output   (default: 8192)
#   KV_TOKENS     tokens the KV cache may hold          (default: 8192)
#   CONFIG        YAML options file this script writes  (default: ./trtllm-lab.yml)
#   CONTAINER     name given to the container           (default: trtllm-lab)
#
# Why the YAML file: TensorRT-LLM sizes its KV cache as a fraction of free GPU memory (0.9 by
# default, per its KV cache documentation). On a 128 GB unified-memory machine that is tens of
# gigabytes, and the memory column would measure the fraction rather than the engine.
# kv_cache_config.max_tokens caps it: the documentation says the lesser of max_tokens and the
# fraction is allocated. guided_decoding_backend: xgrammar is how the guided-decoding page says
# to enable response_format constraints for trtllm-serve.
# No pid file is written in the container case: the process this shell would record is the
# docker client, not the server, so memory for this engine comes from the idle-snapshot delta.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/Qwen/Qwen3-8B}"
MODELS_DIR="${MODELS_DIR:-$HOME/models}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8082}"
MAX_SEQ_LEN="${MAX_SEQ_LEN:-8192}"
KV_TOKENS="${KV_TOKENS:-8192}"
CONFIG="${CONFIG:-./trtllm-lab.yml}"
CONTAINER="${CONTAINER:-trtllm-lab}"

die() { echo "serve-trtllm: $*" >&2; exit 1; }

command -v nvidia-smi >/dev/null || die "nvidia-smi not found; this script needs an NVIDIA GPU"
[ -f "$MODEL/config.json" ] || die "$MODEL/config.json not found; download the checkpoint first"
case "$MODEL" in
  "$MODELS_DIR"/*) ;;
  *) die "MODEL ($MODEL) must be inside MODELS_DIR ($MODELS_DIR), which is what the container sees" ;;
esac
if command -v curl >/dev/null && curl --silent --max-time 2 "http://$HOST:$PORT/health" >/dev/null 2>&1; then
  die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi

cat > "$CONFIG" <<EOF
guided_decoding_backend: xgrammar
kv_cache_config:
  max_tokens: $KV_TOKENS
EOF
CONFIG_ABS="$(cd "$(dirname "$CONFIG")" && pwd)/$(basename "$CONFIG")"

echo "==> trtllm-serve"
echo "    model       $MODEL"
echo "    listen      http://$HOST:$PORT/v1"
echo "    max_seq_len $MAX_SEQ_LEN, max_batch_size 1, KV cache capped at $KV_TOKENS tokens"
echo "    parsers     tool qwen3, reasoning qwen3; guided decoding xgrammar"
echo "    options     $CONFIG_ABS"

if command -v trtllm-serve >/dev/null; then
  echo "    running trtllm-serve from PATH"
  echo $$ > ./trtllm-serve.pid
  # exec keeps this shell's pid, so the pid file names the server process itself.
  exec trtllm-serve "$MODEL" \
      --host "$HOST" \
      --port "$PORT" \
      --max_batch_size 1 \
      --max_seq_len "$MAX_SEQ_LEN" \
      --tool_parser qwen3 \
      --reasoning_parser qwen3 \
      --extra_llm_api_options "$CONFIG_ABS"
fi

[ -n "${TRTLLM_IMAGE:-}" ] || die "trtllm-serve is not on PATH; set TRTLLM_IMAGE to an nvcr.io/nvidia/tensorrt-llm/release tag"
command -v docker >/dev/null || die "docker is not installed, and TensorRT-LLM is distributed as a container"
if docker ps --all --quiet --filter "name=^${CONTAINER}$" | grep -q .; then
  die "a container named $CONTAINER already exists; docker rm -f $CONTAINER first"
fi

IN_CONTAINER_MODEL="/models/${MODEL#"$MODELS_DIR"/}"
echo "    image       $TRTLLM_IMAGE (container name $CONTAINER)"
echo "    in container $IN_CONTAINER_MODEL"
echo "    stop it with: docker stop $CONTAINER"

# --ipc=host and the two ulimits are the documented docker run flags for these images; the
# server binds 0.0.0.0 inside the container, and Docker publishes it on $HOST only.
exec docker run --rm \
    --name "$CONTAINER" \
    --gpus=all \
    --ipc=host \
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \
    --publish "$HOST:$PORT:$PORT" \
    --volume "$MODELS_DIR:/models:ro" \
    --volume "$CONFIG_ABS:/config/trtllm-lab.yml:ro" \
    "$TRTLLM_IMAGE" \
    trtllm-serve "$IN_CONTAINER_MODEL" \
        --host 0.0.0.0 \
        --port "$PORT" \
        --max_batch_size 1 \
        --max_seq_len "$MAX_SEQ_LEN" \
        --tool_parser qwen3 \
        --reasoning_parser qwen3 \
        --extra_llm_api_options /config/trtllm-lab.yml

Download serve-trtllm.sh116 lines

NVIDIA’s installation page named release:1.3.0rc26 on 2026-09-13 while the DGX Spark playbook still validated release:1.3.0rc13. lab.env uses the playbook’s tag because the playbook is NVIDIA’s statement about this machine; if you choose the other, change lab.env and record it.

RunnableTrack S · DGX Spark

terminal 1: TensorRT-LLM in the NGC container

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
bash serve-trtllm.sh

Output — what you should see

port check finished
==> trtllm-serve
    model       /home/you/models/Qwen/Qwen3-8B
    listen      http://127.0.0.1:8082/v1
    max_seq_len 8192, max_batch_size 1, KV cache capped at 8192 tokens
    parsers     tool qwen3, reasoning qwen3; guided decoding xgrammar
    options     /home/you/llm-course/trtllm-lab.yml
    image       nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 (container name trtllm-lab)
    in container /models/Qwen/Qwen3-8B
    stop it with: docker stop trtllm-lab
...
[TRT-LLM] [I] Guided decoder initialized with backend: GuidedDecodingBackend.XGRAMMAR
...

The guided-decoder line is the one the TensorRT-LLM documentation says confirms the grammar backend; the rest of the log varies by release. Loading takes minutes: the measuring script waits up to --wait-seconds for /v1/models to answer.

RunnableTrack S · DGX Spark

terminal 2: version, measurement, probe

cd ~/llm-course
source lab.env
curl --silent http://127.0.0.1:8082/version; echo
python3 compare-engines.py \
    --engine tensorrt-llm \
    --base-url http://127.0.0.1:8082/v1 \
    --model Qwen3-8B \
    --quant bf16 \
    --context-length 8192 \
    --engine-version "$TRTLLM_IMAGE" \
    --backend CUDA \
    --host "$HOST_DESC" \
    --wait-seconds 900 \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine tensorrt-llm \
    --base-url http://127.0.0.1:8082/v1 \
    --model Qwen3-8B \
    --engine-version "$TRTLLM_IMAGE" \
    --host "$HOST_DESC" \
    --labbook labbook.md
docker stop trtllm-lab
curl --silent --max-time 2 --output /dev/null http://127.0.0.1:8082/health || echo "port 8082 is free"

Output — what you should see

... the /version JSON ...
      tensorrt-llm short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
      tensorrt-llm  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
      tensorrt-llm prefill from the difference: xxxx.x tok/s over xxxx extra prompt tokens
      tensorrt-llm memory added since the idle snapshot (MB): {"mem_available_mb": xxxxx, ...}
      tensorrt-llm server process resident set (MB): None
    recorded 2 line(s) in labbook.md
    tensorrt-llm: /v1/models yes ["Qwen3-8B"]
    tensorrt-llm: tool calling yes ... or no (detail)
    tensorrt-llm: JSON schema yes ... or no (detail)
    recorded 1 line in labbook.md
trtllm-lab
port 8082 is free

The /version endpoint is documented but its fields are not; record whatever it returns. In the 1.2.1 source, trtllm-serve reports a local directory by its base name, so /v1/models lists Qwen3-8B, and nothing in that file compares a request’s model field with it. The container runs a 1.3 release candidate, which may differ: if the measurement fails with a model-name error, use the id from curl --silent http://127.0.0.1:8082/v1/models instead. Expect cached to read a count only if the server reports prompt_tokens_details; a null is a warning, not a failure.

Track X — AMD Ryzen AI Max+ 395

The HIP build. Same engine, same source, same file; only the backend differs.

RunnableTrack X · Ryzen AI Max+

terminal 1: llama-server, HIP build

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
LLAMA_BIN=$HIP_BIN PIDFILE=./llama-hip.pid LOGFILE=./llama-hip.log MODEL=$GGUF \
    bash serve-llama-cpp.sh

Output — what you should see

port check finished
==> llama-server
version: x.x.x (build xxxxx, commit xxxxxxx)
built with ... for Linux x86_64
    model   /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
    alias   Qwen3-8B-Q4_K_M
    listen  http://127.0.0.1:8080/v1
    context 8192 tokens, ngl 999, flash attention on, 1 slot(s)
    pid written to ./llama-hip.pid, log copied to ./llama-hip.log
... load log ...
x.xx.xxx.xxx I srv  llama_server: listening on http://127.0.0.1:8080

RunnableTrack X · Ryzen AI Max+

terminal 2: load log, measurement, probe

cd ~/llm-course
source lab.env
grep -E "using device|offloaded|llama_kv_cache: size" llama-hip.log
python3 compare-engines.py \
    --engine llama.cpp-hip \
    --base-url http://127.0.0.1:8080/v1 \
    --model Qwen3-8B-Q4_K_M \
    --quant Q4_K_M \
    --context-length 8192 \
    --engine-version "$("$HIP_BIN/llama-server" --version 2>&1 | head -n 1)" \
    --backend ROCm \
    --host "$HOST_DESC" \
    --server-pid "$(cat llama-hip.pid)" \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine llama.cpp-hip \
    --base-url http://127.0.0.1:8080/v1 \
    --model Qwen3-8B-Q4_K_M \
    --engine-version "$("$HIP_BIN/llama-server" --version 2>&1 | head -n 1)" \
    --host "$HOST_DESC" \
    --labbook labbook.md

Output — what you should see

x.xx.xxx.xxx I llama_prepare_model_devices: using device ROCm0 (AMD Radeon ...) (...) - xxxxx MiB free
x.xx.xxx.xxx I load_tensors: offloaded 37/37 layers to GPU
x.xx.xxx.xxx I llama_kv_cache: size = 1152.00 MiB (  8192 cells,  36 layers,  1/1 seqs), K (f16):  576.00 MiB, V (f16):  576.00 MiB
     llama.cpp-hip short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
     llama.cpp-hip  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
... prefill, server timings, memory ({"mem_available_mb": ..., "rocm_vram_used_mb": ..., "rocm_gtt_used_mb": ...}) ...
    recorded 2 line(s) in labbook.md
    llama.cpp-hip: /v1/models yes ["Qwen3-8B-Q4_K_M"]
    llama.cpp-hip: tool calling yes {"get_current_weather": {"city": "Lisbon"}}
    llama.cpp-hip: JSON schema yes {"name": "...", "born": xxxx, "field": "..."}
    recorded 1 line in labbook.md

The device line must name ROCm0 and the KV line must read 1152.00 MiB. Stop the server with Ctrl-C in terminal 1, then confirm the port is free:

RunnableTrack X · Ryzen AI Max+

terminal 2: nothing left on 8080

curl --silent --max-time 2 http://127.0.0.1:8080/health || echo "port 8080 is free"

Output — what you should see

port 8080 is free

Lemonade. Lemonade’s llama.cpp backend documentation describes its ROCm backend as custom builds from lemonade-sdk/llama.cpp with its own ROCm runtime, and its Vulkan backend as upstream llama.cpp releases. The Lemonade rows therefore differ from yours in build and runtime as well as in the wrapper, and the two-build comparison above is the clean backend comparison.

RunnableTrack X · Ryzen AI Max+

serve-lemonade.sh

#!/usr/bin/env bash
# Purpose: load the lab's model into the running Lemonade Server with its llama.cpp backend and
#          context length chosen explicitly, so that Lemonade can be measured against the
#          llama-server builds from Part 6 and the AMD lesson through the same API
# Platform: strix (Lemonade runs elsewhere too; the backend choice below is the AMD one)
# Minimum memory: 12 GB
# Assumes: Lemonade Server installed from AMD's playbook and running in the background (it
#          starts as a service after installation), the model registered with "lemonade pull"
#          as the lab page shows, and no other model server holding the memory
#
# Usage: MODEL=user.Qwen3-8B-Q4_K_M BACKEND=rocm bash serve-lemonade.sh
#        MODEL=user.Qwen3-8B-Q4_K_M BACKEND=vulkan bash serve-lemonade.sh
#
# Environment:
#   MODEL    registered Lemonade model name            (required)
#   BACKEND  llama.cpp backend: rocm or vulkan         (default: rocm)
#   CTX      context size, in tokens                   (default: 8192)
#   HOST     server address, exported as LEMONADE_HOST (default: 127.0.0.1)
#   PORT     server port, exported as LEMONADE_PORT    (default: 13305)
#   LLAMACPP_ARGS  arguments passed to llama-server    (default: --flash-attn on --cache-ram 0)
#
# Lemonade is a long-running server, not a foreground process: "lemonade load" asks it to
# start a llama.cpp backend with the given options and returns. There is no pid to record;
# the memory for this engine comes from the idle-snapshot delta. Stop the model afterwards
# with "lemonade unload". The Lemonade CLI documentation read on 2026-09-13 lists load with
# --ctx-size, --llamacpp and --llamacpp-args for the llamacpp recipe. --cache-ram 0 in the
# llama.cpp arguments turns off llama-server's host-RAM prompt cache, as serve-llama-cpp.sh does,
# so that the Lemonade rows and the llama-server rows allocate the same memory.

set -euo pipefail

MODEL="${MODEL:-}"
BACKEND="${BACKEND:-rocm}"
CTX="${CTX:-8192}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-13305}"
LLAMACPP_ARGS="${LLAMACPP_ARGS:---flash-attn on --cache-ram 0}"

die() { echo "serve-lemonade: $*" >&2; exit 1; }

[ -n "$MODEL" ] || die "set MODEL to a registered Lemonade model name; see 'lemonade list'"
command -v lemonade >/dev/null || die "lemonade not found; install it as the AMD lesson describes"
case "$BACKEND" in
  rocm|vulkan) ;;
  *) die "BACKEND must be rocm or vulkan for this lab, not '$BACKEND'" ;;
esac

export LEMONADE_HOST="$HOST"
export LEMONADE_PORT="$PORT"

lemonade status || die "Lemonade Server is not reachable on $HOST:$PORT; start or restart its service"

echo "==> Lemonade Server"
echo "    CLI      $(lemonade --version 2>/dev/null || echo 'version not reported')"
echo "    model    $MODEL"
echo "    backend  llama.cpp / $BACKEND, context $CTX tokens"
echo "    args     $LLAMACPP_ARGS"
echo "    API      http://$HOST:$PORT/v1"

lemonade unload >/dev/null 2>&1 || true
lemonade load "$MODEL" \
    --ctx-size "$CTX" \
    --llamacpp "$BACKEND" \
    --llamacpp-args "$LLAMACPP_ARGS"

echo "    loaded; measure it now, then run: lemonade unload"

Download serve-lemonade.sh66 lines

RunnableTrack X · Ryzen AI Max+

terminal 2: load in Lemonade, measure, probe, unload

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
lemonade backends install llamacpp:rocm
lemonade backends
MODEL=user.Qwen3-8B-Q4_K_M BACKEND=rocm bash serve-lemonade.sh
python3 compare-engines.py \
    --engine lemonade-rocm \
    --base-url http://127.0.0.1:13305/v1 \
    --model user.Qwen3-8B-Q4_K_M \
    --quant Q4_K_M \
    --context-length 8192 \
    --engine-version "lemonade $(lemonade --version)" \
    --backend ROCm \
    --host "$HOST_DESC" \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine lemonade-rocm \
    --base-url http://127.0.0.1:13305/v1 \
    --model user.Qwen3-8B-Q4_K_M \
    --engine-version "lemonade $(lemonade --version)" \
    --host "$HOST_DESC" \
    --labbook labbook.md
lemonade unload

Output — what you should see

port check finished
... backend install output (wording not documented) ...
... the recipes and backends Lemonade has installed, with their versions ...
... lemonade status output ...
==> Lemonade Server
    CLI      x.x.x
    model    user.Qwen3-8B-Q4_K_M
    backend  llama.cpp / rocm, context 8192 tokens
    args     --flash-attn on --cache-ram 0
    API      http://127.0.0.1:13305/v1
... load progress (wording not documented) ...
    loaded; measure it now, then run: lemonade unload
     lemonade-rocm short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
     lemonade-rocm  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
... prefill, memory, recorded 2 line(s) ...
    lemonade-rocm: /v1/models yes [... "user.Qwen3-8B-Q4_K_M" ...]
    lemonade-rocm: tool calling yes ... or no (detail)
    lemonade-rocm: JSON schema yes ... or no (detail)
    recorded 1 line in labbook.md

lemonade backends install llamacpp:rocm is the command Lemonade’s llama.cpp backend page gives; its CLI reference does not say that load installs a missing backend by itself, so the block installs it first. Record the lemonade backends output: it names the backend versions Lemonade installed. The serve script passes --cache-ram 0 through --llamacpp-args, as serve-llama-cpp.sh passes it to llama-server; if the load fails, see Troubleshooting. Whether Lemonade passes llama.cpp’s timings block and cached-token count through is not documented; a missing line or a cached None warning is a finding, not an error.

The fourth row is optional and repeats the load with the Vulkan backend:

RunnableTrack X · Ryzen AI Max+

optional: Lemonade with its Vulkan backend

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
lemonade backends install llamacpp:vulkan
MODEL=user.Qwen3-8B-Q4_K_M BACKEND=vulkan bash serve-lemonade.sh
python3 compare-engines.py \
    --engine lemonade-vulkan \
    --base-url http://127.0.0.1:13305/v1 \
    --model user.Qwen3-8B-Q4_K_M \
    --quant Q4_K_M \
    --context-length 8192 \
    --engine-version "lemonade $(lemonade --version)" \
    --backend Vulkan \
    --host "$HOST_DESC" \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine lemonade-vulkan \
    --base-url http://127.0.0.1:13305/v1 \
    --model user.Qwen3-8B-Q4_K_M \
    --engine-version "lemonade $(lemonade --version)" \
    --host "$HOST_DESC" \
    --labbook labbook.md
lemonade unload

Its output has the Lemonade block’s shape, with vulkan and lemonade-vulkan in place of rocm and lemonade-rocm.

Track M — Apple silicon

RunnableTrack M · Apple silicon

serve-mlx.sh

#!/usr/bin/env bash
# Purpose: start the mlx-lm server on a local MLX model directory, so that the same measuring
#          scripts can be pointed at Apple's native engine and at llama-server without
#          changing anything but the port and the model name
# Platform: mac (mlx-lm runs on Apple silicon)
# Minimum memory: 12 GB
# Assumes: mlx-lm installed (Part 8's MLX lesson), an MLX model directory in $MODEL whose
#          config.json carries a "quantization" block, the port free and nothing else large
#          running; the server runs in the foreground so that Ctrl-C stops it
#
# Usage: MODEL=$HOME/models/mlx-community/Qwen3-8B-4bit bash serve-mlx.sh
#
# Environment:
#   MODEL              absolute path of the MLX model directory   (required)
#   HOST               address to bind                            (default: 127.0.0.1)
#   PORT               port to listen on                          (default: 8081)
#   PROMPT_CACHE_SIZE  distinct prompt KV caches the server keeps (default: 1)
#   LOGLEVEL           server log level                           (default: INFO)
#   PIDFILE            where to write the server's pid            (default: ./mlx-server.pid)
#
# Two things about mlx_lm.server 0.31.3, read from its source, shape this script:
#   * A request's "model" field is resolved as a model to load. Send the exact string passed
#     as --model (the absolute path this script prints), or the server tries to load
#     whatever name you sent.
#   * The server keeps up to --prompt-cache-size distinct KV caches from earlier requests
#     (default 10). This lab sends a different prompt every time, so at the default the
#     memory reading would include several stale caches. 1 keeps it comparable with a
#     single llama-server slot.
# The server's own documentation says it is not recommended for production because it only
# implements basic security checks, so it stays bound to the loopback address.

set -euo pipefail

MODEL="${MODEL:-}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8081}"
PROMPT_CACHE_SIZE="${PROMPT_CACHE_SIZE:-1}"
LOGLEVEL="${LOGLEVEL:-INFO}"
PIDFILE="${PIDFILE:-./mlx-server.pid}"

die() { echo "serve-mlx: $*" >&2; exit 1; }

[ "$(uname -s)" = "Darwin" ] || die "mlx-lm runs on Apple silicon; this machine reports $(uname -s)"
[ -n "$MODEL" ] || die "set MODEL to the absolute path of an MLX model directory"
case "$MODEL" in /*) ;; *) die "MODEL must be an absolute path (it is also the name requests send)" ;; esac
[ -f "$MODEL/config.json" ] || die "$MODEL/config.json not found"
grep -q '"quantization"' "$MODEL/config.json" || echo "serve-mlx: warning: no quantization block in config.json" >&2
command -v mlx_lm.server >/dev/null || die "mlx_lm.server not found; install mlx-lm as the MLX lesson describes"
if command -v curl >/dev/null && curl --silent --max-time 2 "http://$HOST:$PORT/health" >/dev/null 2>&1; then
  die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi

# The interpreter that runs mlx_lm.server is the one whose packages it uses.
PY="$(head -n 1 "$(command -v mlx_lm.server)" | sed 's/^#!//')"

echo "==> mlx_lm.server"
"$PY" -c 'import mlx_lm, mlx.core as mx; print("    mlx-lm", mlx_lm.__version__, "/ mlx", mx.__version__)' \
  || echo "    (could not read the mlx-lm version from $PY)"
echo "    model   $MODEL"
echo "    quant   $(grep -A3 '"quantization"' "$MODEL/config.json" | tr -d ' \n' | cut -c1-60)"
echo "    listen  http://$HOST:$PORT/v1"
echo "    prompt cache size $PROMPT_CACHE_SIZE"
echo "    pid written to $PIDFILE"
echo "    send \"model\": \"$MODEL\" in every request"

echo $$ > "$PIDFILE"

# exec keeps this shell's pid, so $PIDFILE names the server process itself.
exec mlx_lm.server \
    --model "$MODEL" \
    --host "$HOST" \
    --port "$PORT" \
    --prompt-cache-size "$PROMPT_CACHE_SIZE" \
    --log-level "$LOGLEVEL"

Download serve-mlx.sh74 lines

RunnableTrack M · Apple silicon

terminal 1: mlx_lm.server

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
MODEL=$MLX_MODEL bash serve-mlx.sh

Output — what you should see

port check finished
==> mlx_lm.server
    mlx-lm 0.31.3 / mlx x.xx.x
    model   /Users/you/models/mlx-community/Qwen3-8B-4bit
    quant   "quantization":{"group_size":64,"bits":4},
    listen  http://127.0.0.1:8081/v1
    prompt cache size 1
    pid written to ./mlx-server.pid
    send "model": "/Users/you/models/mlx-community/Qwen3-8B-4bit" in every request
...

In mlx-lm 0.31.3 a request’s model field is resolved as a model to load, so the measuring commands send the same absolute path the server was started with; any other name makes the server try to load that name.

RunnableTrack M · Apple silicon

terminal 2: measurement and probe

cd ~/llm-course
source lab.env
python3 compare-engines.py \
    --engine mlx-lm \
    --base-url http://127.0.0.1:8081/v1 \
    --model "$MLX_MODEL" \
    --quant "MLX 4-bit, group size 64" \
    --context-length 8192 \
    --engine-version "mlx-lm $("$MLX_PY" -c 'import mlx_lm; print(mlx_lm.__version__)')" \
    --backend Metal \
    --host "$HOST_DESC" \
    --server-pid "$(cat mlx-server.pid)" \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine mlx-lm \
    --base-url http://127.0.0.1:8081/v1 \
    --model "$MLX_MODEL" \
    --engine-version "mlx-lm $("$MLX_PY" -c 'import mlx_lm; print(mlx_lm.__version__)')" \
    --host "$HOST_DESC" \
    --labbook labbook.md

Output — what you should see

            mlx-lm short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
            mlx-lm  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
            mlx-lm prefill from the difference: xxxx.x tok/s over xxxx extra prompt tokens
            mlx-lm memory added since the idle snapshot (MB): {"mac_wired_mb": xxxx, "mac_used_mb": xxxx}
            mlx-lm server process resident set (MB): xxxx
    recorded 2 line(s) in labbook.md
    mlx-lm: /v1/models yes [... "/Users/you/models/mlx-community/Qwen3-8B-4bit" ...]
    mlx-lm: tool calling yes ... or no (detail)
    mlx-lm: JSON schema no (accepted the request but the content was not JSON: ...)
    recorded 1 line in labbook.md

The JSON-schema line shows the predicted failure; if yours passes, your mlx-lm version handles response_format, and that belongs in the notebook with the version.

--context-length 8192 is recorded for comparability, but mlx_lm.server has no such setting: its cache grows in 256-token steps with the request, so after the long test (about 1,378 prompt tokens plus 128 generated) it holds 1,536 tokens where llama-server holds 8,192. The difference is 6,656 × 147,456 = 981,467,136 bytes, and Task 9’s memory table accounts for it. Stop the server with Ctrl-C in terminal 1, then confirm the port is free:

RunnableTrack M · Apple silicon

terminal 2: nothing left on 8081

curl --silent --max-time 2 --output /dev/null http://127.0.0.1:8081/ || echo "port 8081 is free"

Output — what you should see

port 8081 is free

Track N — NVIDIA desktop or laptop

TabbyAPI with ExLlamaV3. Clone it and write a configuration that loads the EXL3 model with the lab’s context. Keys and defaults are from config_sample.yml; clearing override_preset follows its getting-started note for metrics, so no sampler defaults are filled in behind the request’s own.

RunnableTrack N · NVIDIA GPU

install TabbyAPI and write config.yml

git clone https://github.com/theroyallab/tabbyAPI ~/tabbyAPI
cat > ~/tabbyAPI/config.yml <<EOF
network:
  host: 127.0.0.1
  port: 5000
  disable_auth: false
model:
  model_dir: $HOME/models/turboderp
  model_name: Qwen3-8B-exl3-4.0bpw
  max_seq_len: 8192
  cache_size: 8192
  cache_mode: FP16
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
sampling:
  override_preset:
EOF
git -C ~/tabbyAPI rev-parse --short HEAD

RunnableTrack N · NVIDIA GPU

serve-exllamav3.sh

#!/usr/bin/env bash
# Purpose: start TabbyAPI, the server the ExLlamaV3 project names as its recommended backend,
#          after checking that its config.yml loads the lab's EXL3 model with the lab's
#          context length, so that the model can be measured through the same
#          OpenAI-compatible interface as every other engine here
# Platform: nvidia (ExLlamaV3 requires CUDA; its README lists ROCm support as still to do)
# Minimum memory: 12 GB
# Assumes: a TabbyAPI checkout in $TABBY_DIR, a config.yml written as this lab's Track N tab
#          shows, the EXL3 model directory it names, an NVIDIA GPU with a working driver, the
#          port free; the first start creates a venv and installs PyTorch and ExLlamaV3 (a
#          long, unattended download); the server runs in the foreground
#
# Usage: bash serve-exllamav3.sh
#        TABBY_DIR=~/tabbyAPI GPU_LIB=cu13 bash serve-exllamav3.sh
#
# Environment:
#   TABBY_DIR  the TabbyAPI checkout                        (default: $HOME/tabbyAPI)
#   GPU_LIB    dependency set for the first start: cu12, cu13 (default: let start.py choose)
#   PIDFILE    where to write the process id                (default: ./tabbyapi.pid)
#
# TabbyAPI is configured through config.yml rather than command-line options: model_dir and
# model_name choose the model, max_seq_len and cache_size (in tokens, a multiple of 256) set
# the context, and api_tokens.yml holds the generated API key unless network.disable_auth is
# true. The pid recorded is start.sh's shell; the Python server is its child, so for memory
# use nvidia-smi's per-process figure, which the measuring script reads. TabbyAPI is licensed
# AGPL-3.0; read it before building anything on it.

set -euo pipefail

TABBY_DIR="${TABBY_DIR:-$HOME/tabbyAPI}"
GPU_LIB="${GPU_LIB:-}"
PIDFILE="${PIDFILE:-./tabbyapi.pid}"

die() { echo "serve-exllamav3: $*" >&2; exit 1; }

command -v nvidia-smi >/dev/null || die "nvidia-smi not found; ExLlamaV3 needs an NVIDIA GPU"
[ -d "$TABBY_DIR" ] || die "$TABBY_DIR does not exist; clone TabbyAPI there or set TABBY_DIR"
[ -f "$TABBY_DIR/start.sh" ] || die "no start.sh in $TABBY_DIR"
CONFIG="$TABBY_DIR/config.yml"
[ -f "$CONFIG" ] || die "no config.yml in $TABBY_DIR; write it as the lab page shows"

value() { sed -n "s/^[[:space:]]*$1:[[:space:]]*//p" "$CONFIG" | head -n 1 | tr -d '"'; }
MODEL_DIR="$(value model_dir)"
MODEL_NAME="$(value model_name)"
HOST="$(value host)"
PORT="$(value port)"
[ -n "$MODEL_NAME" ] || die "config.yml has no model_name; TabbyAPI would start with no model loaded"
[ -f "$MODEL_DIR/$MODEL_NAME/config.json" ] || die "$MODEL_DIR/$MODEL_NAME/config.json not found"
grep -q '"quant_method": "exl3"' "$MODEL_DIR/$MODEL_NAME/config.json" \
  || echo "serve-exllamav3: warning: $MODEL_NAME/config.json does not say quant_method exl3" >&2
[ "$(value max_seq_len)" = "8192" ] || echo "serve-exllamav3: warning: max_seq_len is not 8192" >&2
[ "$(value cache_size)" = "8192" ] || echo "serve-exllamav3: warning: cache_size is not 8192" >&2
if command -v curl >/dev/null && curl --silent --max-time 2 "http://${HOST:-127.0.0.1}:${PORT:-5000}/health" >/dev/null 2>&1; then
  die "something is already answering on ${HOST:-127.0.0.1}:${PORT:-5000}; stop it first"
fi

echo "==> TabbyAPI with the ExLlamaV3 backend"
echo "    checkout $TABBY_DIR ($(git -C "$TABBY_DIR" rev-parse --short HEAD 2>/dev/null || echo 'not a git checkout'))"
echo "    model    $MODEL_DIR/$MODEL_NAME"
echo "    context  max_seq_len $(value max_seq_len), cache_size $(value cache_size) tokens"
echo "    listen   http://${HOST:-127.0.0.1}:${PORT:-5000}/v1"
echo "    pid written to $PIDFILE"
if [ -f "$TABBY_DIR/api_tokens.yml" ]; then
  echo "    API key: grep '^api_key:' $TABBY_DIR/api_tokens.yml"
else
  echo "    api_tokens.yml is created on this first start; read the key from it afterwards"
fi

echo $$ > "$PIDFILE"
cd "$TABBY_DIR"
if [ -n "$GPU_LIB" ]; then
  exec ./start.sh --gpu-lib "$GPU_LIB"
fi
exec ./start.sh

Download serve-exllamav3.sh74 lines

RunnableTrack N · NVIDIA GPU

terminal 1: TabbyAPI

cd ~/llm-course
source lab.env
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
bash serve-exllamav3.sh

Output — what you should see

port check finished
==> TabbyAPI with the ExLlamaV3 backend
    checkout /home/you/tabbyAPI (xxxxxxx)
    model    /home/you/models/turboderp/Qwen3-8B-exl3-4.0bpw
    context  max_seq_len 8192, cache_size 8192 tokens
    listen   http://127.0.0.1:5000/v1
    pid written to ./tabbyapi.pid
    api_tokens.yml is created on this first start; read the key from it afterwards
Venv doesn't exist! Creating one for you.
... dependency installation on the first start ...
... Your API key is: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ...

Venv doesn't exist! is start.sh’s own message and the API key line is from TabbyAPI’s auth.py; the log between them varies by commit.

Wait until terminal 1 has printed Your API key is: and the model has finished loading; on a first start this follows the dependency install. The first lines of the next block wait for the key file and stop if the key is empty, so an early start cannot turn into ten minutes of 401 answers.

RunnableTrack N · NVIDIA GPU

terminal 2: key, measurement, probe

cd ~/llm-course
source lab.env
until [ -f "$TABBY_DIR/api_tokens.yml" ]; do echo "waiting for TabbyAPI to write api_tokens.yml"; sleep 10; done
export TABBY_KEY="$(sed -n 's/^api_key: //p' "$TABBY_DIR/api_tokens.yml")"
[ -n "$TABBY_KEY" ] || { echo "TABBY_KEY is empty; check $TABBY_DIR/api_tokens.yml"; exit 1; }
EXL3_VERSION="tabbyAPI $(git -C "$TABBY_DIR" rev-parse --short HEAD), exllamav3 $("$TABBY_DIR/venv/bin/python" -c 'from exllamav3.version import __version__; print(__version__)')"
python3 compare-engines.py \
    --engine exllamav3 \
    --base-url http://127.0.0.1:5000/v1 \
    --model Qwen3-8B-exl3-4.0bpw \
    --api-key "$TABBY_KEY" \
    --quant "EXL3 4.0 bpw, head 6 bpw" \
    --context-length 8192 \
    --engine-version "$EXL3_VERSION" \
    --backend CUDA \
    --host "$HOST_DESC" \
    --wait-seconds 600 \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine exllamav3 \
    --base-url http://127.0.0.1:5000/v1 \
    --model Qwen3-8B-exl3-4.0bpw \
    --api-key "$TABBY_KEY" \
    --engine-version "$EXL3_VERSION" \
    --host "$HOST_DESC" \
    --labbook labbook.md

Output — what you should see

         exllamav3 short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached ...
         exllamav3  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached ...
         exllamav3 prefill from the difference: xxxx.x tok/s over xxxx extra prompt tokens
         exllamav3 memory added since the idle snapshot (MB): {"mem_available_mb": xxxx, "gpu_used_mb": xxxx, "gpu_process_mb": xxxx}
         exllamav3 server process resident set (MB): xx
    recorded 2 line(s) in labbook.md
    exllamav3: /v1/models yes ["Qwen3-8B-exl3-4.0bpw"]
    exllamav3: tool calling no (the model wrote a <tool_call> block as text; the server did not parse it)
    exllamav3: JSON schema yes {"name": "...", "born": xxxx, "field": "..."}
    recorded 1 line in labbook.md

Those are the probe results Task 6’s table predicts. Stop TabbyAPI with Ctrl-C in terminal 1, then confirm the port is free:

RunnableTrack N · NVIDIA GPU

terminal 2: nothing left on 5000

curl --silent --max-time 2 --output /dev/null http://127.0.0.1:5000/health || echo "port 5000 is free"

Output — what you should see

port 5000 is free

vLLM. Install it into its own environment, pinned, with the form Part 9’s install lesson uses:

RunnableTrack N · NVIDIA GPU

install vLLM

uv venv ~/vllm-env --python 3.12 --seed
source ~/vllm-env/bin/activate
uv pip install vllm==0.28.0 --torch-backend=auto
vllm --version

Output — what you should see

... resolution and download ...
0.28.0

Neither installer states how much disk its PyTorch environment takes, so measure both now that TabbyAPI’s first start and the vLLM install have finished:

RunnableTrack N · NVIDIA GPU

disk used by the two environments

du -sh ~/tabbyAPI/venv ~/vllm-env
df -h ~

Output — what you should see

x.xG  /home/you/tabbyAPI/venv
x.xG  /home/you/vllm-env
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  xxxG  xxxG  xxxG  xx% /

Record: both environment sizes and Avail.

RunnableTrack N · NVIDIA GPU

serve-vllm.sh

#!/usr/bin/env bash
# Purpose: start vLLM's OpenAI-compatible server on a four-bit checkpoint of the lab's model,
#          as the third engine on Track N, with one sequence, the lab's context length, a KV
#          cache sized for that context rather than for a fraction of the card, and the Qwen3
#          tool-call and reasoning parsers enabled
# Platform: nvidia (vLLM's CUDA wheels; on Track S use the NGC container from Part 9 instead)
# Minimum memory: 12 GB
# Assumes: vLLM installed into the active Python environment as this lab's Track N tab shows,
#          an NVIDIA GPU with a working driver, the checkpoint downloaded to $MODEL, the port
#          free and nothing else on the GPU; the server runs in the foreground
#
# Usage: MODEL=$HOME/models/Qwen/Qwen3-8B-AWQ bash serve-vllm.sh
#
# Environment:
#   MODEL       checkpoint directory                     (default: $HOME/models/Qwen/Qwen3-8B-AWQ)
#   SERVED_AS   name the API reports for the model       (default: the directory's base name)
#   HOST        address to bind                          (default: 127.0.0.1)
#   PORT        port to listen on                        (default: 8083)
#   MAX_LEN     maximum model length, in tokens          (default: 8192)
#   KV_BYTES    KV cache size in bytes                   (default: 1342177280, 1.25 GiB)
#   PIDFILE     where to write the process id            (default: ./vllm-server.pid)
#
# Sizing: Qwen3-8B's KV cache is 2 x 36 layers x 8 KV heads x 128 x 2 bytes = 147,456 bytes per
# token, so 8,192 tokens need 1,207,959,552 bytes; 1.25 GiB leaves room for whole blocks.
# vLLM's cache configuration documents that --kv-cache-memory-bytes, when set, ignores
# --gpu-memory-utilization, which otherwise reserves a fraction of the whole card.
# Parsers: vLLM's tool-calling page gives --enable-auto-tool-choice with --tool-call-parser
# hermes for Qwen models, and its reasoning page lists the qwen3 reasoning parser. Part 9
# teaches both properly. --enable-prompt-tokens-details adds usage.prompt_tokens_details, the
# cached-token count the measuring script uses to confirm that prefix caching was not hit.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/Qwen/Qwen3-8B-AWQ}"
SERVED_AS="${SERVED_AS:-$(basename "$MODEL")}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8083}"
MAX_LEN="${MAX_LEN:-8192}"
KV_BYTES="${KV_BYTES:-1342177280}"
PIDFILE="${PIDFILE:-./vllm-server.pid}"

die() { echo "serve-vllm: $*" >&2; exit 1; }

command -v vllm >/dev/null || die "vllm not found; activate the environment you installed it into"
command -v nvidia-smi >/dev/null || die "nvidia-smi not found; vLLM here expects an NVIDIA GPU"
[ -f "$MODEL/config.json" ] || die "$MODEL/config.json not found; download the checkpoint first"
if command -v curl >/dev/null && curl --silent --max-time 2 "http://$HOST:$PORT/health" >/dev/null 2>&1; then
  die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi

echo "==> vllm serve"
vllm --version || true
echo "    model    $MODEL"
echo "    served   $SERVED_AS"
echo "    listen   http://$HOST:$PORT/v1"
echo "    context  $MAX_LEN tokens, 1 sequence, KV cache $KV_BYTES bytes"
echo "    pid written to $PIDFILE"
echo "    The first start profiles and captures CUDA graphs; wait for the startup-complete line."

echo $$ > "$PIDFILE"

# exec keeps this shell's pid, so $PIDFILE names the server process itself.
exec vllm serve "$MODEL" \
    --served-model-name "$SERVED_AS" \
    --host "$HOST" \
    --port "$PORT" \
    --max-model-len "$MAX_LEN" \
    --max-num-seqs 1 \
    --kv-cache-memory-bytes "$KV_BYTES" \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --reasoning-parser qwen3 \
    --enable-prompt-tokens-details

Download serve-vllm.sh73 lines

RunnableTrack N · NVIDIA GPU

terminal 1: vLLM

cd ~/llm-course
source ~/vllm-env/bin/activate
for port in 8080 8081 8082 8083 5000; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"
bash serve-vllm.sh

Output — what you should see

port check finished
==> vllm serve
0.28.0
    model    /home/you/models/Qwen/Qwen3-8B-AWQ
    served   Qwen3-8B-AWQ
    listen   http://127.0.0.1:8083/v1
    context  8192 tokens, 1 sequence, KV cache 1342177280 bytes
    pid written to ./vllm-server.pid
    The first start profiles and captures CUDA graphs; wait for the startup-complete line.
... startup log ...
... Application startup complete.

Startup profiles the model and captures CUDA graphs; the server is ready when uvicorn’s Application startup complete. line appears, and the measuring script waits for it anyway.

RunnableTrack N · NVIDIA GPU

terminal 2: measurement and probe

cd ~/llm-course
source lab.env
python3 compare-engines.py \
    --engine vllm \
    --base-url http://127.0.0.1:8083/v1 \
    --model Qwen3-8B-AWQ \
    --quant "AWQ 4-bit, group size 128" \
    --context-length 8192 \
    --engine-version "vllm $(~/vllm-env/bin/vllm --version)" \
    --backend CUDA \
    --host "$HOST_DESC" \
    --server-pid "$(cat vllm-server.pid)" \
    --wait-seconds 900 \
    --idle-snapshot idle-memory.json \
    --labbook labbook.md
python3 feature-probe.py \
    --engine vllm \
    --base-url http://127.0.0.1:8083/v1 \
    --model Qwen3-8B-AWQ \
    --engine-version "vllm $(~/vllm-env/bin/vllm --version)" \
    --host "$HOST_DESC" \
    --labbook labbook.md

Output — what you should see

              vllm short  prompt    xx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
              vllm  long  prompt  xxxx tok  ttft x.xxx s  decode xx.xx tok/s  128 tok (length), counts from usage, cached x
              vllm prefill from the difference: xxxx.x tok/s over xxxx extra prompt tokens
              vllm memory added since the idle snapshot (MB): {"mem_available_mb": xxxx, "gpu_used_mb": xxxx, "gpu_process_mb": xxxx}
              vllm server process resident set (MB): xxxx
    recorded 2 line(s) in labbook.md
    vllm: /v1/models yes ["Qwen3-8B-AWQ"]
    vllm: tool calling yes {"get_current_weather": {"city": "Lisbon"}}
    vllm: JSON schema yes {"name": "...", "born": xxxx, "field": "..."}
    recorded 1 line in labbook.md

Stop vLLM with Ctrl-C in terminal 1, then confirm the port is free:

RunnableTrack N · NVIDIA GPU

terminal 2: nothing left on 8083

curl --silent --max-time 2 http://127.0.0.1:8083/health || echo "port 8083 is free"

Output — what you should see

port 8083 is free

On this track the memory figure to compare is gpu_process_mb: nvidia-smi lists every process holding GPU memory, so it counts vLLM’s engine process and TabbyAPI’s Python server even though neither is the process whose id was recorded. Under WSL2 that list is empty, because NVIDIA’s CUDA on WSL guide lists “active compute process” among the NVML queries not yet supported. compare-engines.py then records no gpu_process_mb, and summarise-engines.py uses gpu_used_mb with the source nvidia-smi device, which also counts Windows applications using the card: close them before the idle snapshot.

8. Record the results in the course benchmark format

RunnableAll tracks

summarise-engines.py

#!/usr/bin/env python3
"""Turn this lab's notebook lines into the course benchmark tables, and check them.

Purpose: read the JSON lines that compare-engines.py and feature-probe.py appended to the lab
    notebook, keep the most recent measurement per engine, and print (1) a Markdown table of
    speed and memory, (2) a Markdown table of API features, and (3) the rows in the form the
    course's <Benchmark> recording sheet takes. With --check, test every line against the
    lab's validation rules and exit non-zero if any rule fails.
Platform: all (Python standard library only)
Minimum memory: 12 GB
Assumes: Python 3.9 or later; a notebook file in which each result is one JSON object per line
    (other lines, such as Markdown notes, are ignored).

Usage: python3 summarise-engines.py --labbook labbook.md
       python3 summarise-engines.py --labbook labbook.md --check
       python3 summarise-engines.py --labbook labbook.md --check --allow-cpu   (script testing only)
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

LAB = "part-08/lab-same-model-every-engine"
CACHE_LIMIT = 64  # more reused prompt tokens than this means the run measured the prefix cache


def load(path: Path) -> list:
    records = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line.startswith("{"):
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            continue
        if record.get("lab") == LAB:
            records.append(record)
    return records


def latest(records: list) -> tuple:
    """Most recent short and long speed line, and most recent feature line, per engine."""
    speed, features, order = {}, {}, []
    for r in sorted(records, key=lambda x: x.get("measured_on", "")):
        engine = r.get("engine", "?")
        if engine not in order:
            order.append(engine)
        if r.get("probe") == "features":
            features[engine] = r
        elif r.get("test") in ("short", "long"):
            speed.setdefault(engine, {})[r["test"]] = r
    return order, speed, features


def memory_added(record: dict) -> tuple:
    """The memory reading that best describes what this server added, and its name.

    A discrete NVIDIA card reports a device total, and nvidia-smi's per-process figure counts
    every CUDA allocation of the server wherever it runs. Under WSL2 NVML does not report
    compute processes (CUDA on WSL guide, features not yet supported), so a per-process delta
    that is missing, zero or negative falls back to the device figure. An AMD GPU reports VRAM
    and GTT use, which together cover the BIOS carve-out and the dynamically mapped share. A
    DGX Spark reports no device total, and NVIDIA's guidance there is MemAvailable. A Mac has
    vm_stat.
    """
    delta = record.get("memory_delta_mb") or {}
    loaded = record.get("memory") or {}
    if loaded.get("gpu_total_mb"):
        if delta.get("gpu_process_mb") is not None and delta["gpu_process_mb"] > 0:
            return delta["gpu_process_mb"], "nvidia-smi per-process"
        if delta.get("gpu_used_mb") is not None:
            return delta["gpu_used_mb"], "nvidia-smi device"
    if "rocm_vram_used_mb" in delta or "rocm_gtt_used_mb" in delta:
        return delta.get("rocm_vram_used_mb", 0) + delta.get("rocm_gtt_used_mb", 0), "rocm-smi VRAM+GTT"
    if delta.get("mem_available_mb") is not None:
        return delta["mem_available_mb"], "MemAvailable"
    if delta.get("mac_used_mb") is not None:
        return delta["mac_used_mb"], "vm_stat active+wired+compressed"
    return None, "no idle snapshot"


def fmt(value, digits=2) -> str:
    if value is None:
        return "—"
    if isinstance(value, float):
        return f"{value:.{digits}f}"
    return str(value)


def yes_no(section: dict) -> str:
    if not section:
        return "—"
    return "yes" if section.get("supported") else "no"


def print_tables(order, speed, features) -> None:
    print("Speed and memory (medians; prefill from the difference between the two prompts)\n")
    print("| Engine | Quant | Backend | TTFT short s | TTFT long s | Prefill tok/s | "
          "Decode short tok/s | Decode long tok/s | Memory added MB | Memory source |")
    print("| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |")
    rows = []
    for engine in order:
        pair = speed.get(engine)
        if not pair:
            continue
        s, l = pair.get("short", {}), pair.get("long", {})
        any_line = l or s
        mem, source = memory_added(any_line)
        print(f"| {engine} | {any_line.get('quant') or '—'} | {any_line.get('backend') or '—'} | "
              f"{fmt(s.get('ttft_s'), 3)} | {fmt(l.get('ttft_s'), 3)} | "
              f"{fmt(any_line.get('prefill_tokens_per_s'), 1)} | {fmt(s.get('decode_tokens_per_s'))} | "
              f"{fmt(l.get('decode_tokens_per_s'))} | {fmt(mem)} | {source} |")
        rows.append([f"{engine}, {any_line.get('quant') or '?'}", s.get("ttft_s"), l.get("ttft_s"),
                     any_line.get("prefill_tokens_per_s"), s.get("decode_tokens_per_s"),
                     l.get("decode_tokens_per_s"), mem])

    print("\nAPI features\n")
    print("| Engine | /v1/models | Tool calling | JSON schema | Detail |")
    print("| --- | --- | --- | --- | --- |")
    feature_rows = []
    for engine in order:
        f = features.get(engine)
        if not f:
            continue
        details = [d for d in (f.get("tool_calling", {}).get("detail"),
                               f.get("structured_output", {}).get("detail")) if d]
        detail = "; ".join(details) or "—"
        cells = [yes_no(f.get("models_endpoint")), yes_no(f.get("tool_calling")),
                 yes_no(f.get("structured_output"))]
        print(f"| {engine} | {cells[0]} | {cells[1]} | {cells[2]} | {detail} |")
        feature_rows.append([engine, *cells, detail])

    def js(value):
        return "'—'" if value is None else (json.dumps(value) if not isinstance(value, str) else repr(value))

    print("\nRows for the speed recording sheet (paste into the Benchmark rows prop)\n")
    for row in rows:
        print("  [" + ", ".join(js(v) for v in row) + "],")
    print("\nRows for the features recording sheet\n")
    for row in feature_rows:
        print("  [" + ", ".join(js(v) for v in row) + "],")


def check(order, speed, features, allow_cpu: bool) -> int:
    failures = 0

    def report(level: str, engine: str, message: str) -> None:
        nonlocal failures
        failures += level == "FAIL"
        print(f"{level:<4}  {engine:<16} {message}")

    if not order:
        report("FAIL", "-", f"no lines with lab = {LAB}")
    for engine in order:
        pair = speed.get(engine, {})
        if set(pair) != {"short", "long"}:
            report("FAIL", engine, f"speed tests recorded: {sorted(pair) or 'none'} (need short and long)")
        for test, r in sorted(pair.items()):
            label = f"{test}:"
            if r.get("ttft_s") is None or r.get("decode_tokens_per_s") is None:
                report("FAIL", engine, f"{label} ttft_s or decode_tokens_per_s is null")
            if not r.get("engine_version") or not r.get("quant"):
                report("FAIL", engine, f"{label} engine_version or quant is empty")
            backend = (r.get("backend") or "").strip()
            if not backend or (backend.upper() == "CPU" and not allow_cpu):
                report("FAIL", engine, f"{label} backend is '{backend}'; it must name the accelerator")
            if r.get("prompt_tag") is False:
                report("FAIL", engine, f"{label} recorded with --reuse-prompt; that measures the cache")
            cached = r.get("max_cached_prompt_tokens")
            if cached is None:
                report("WARN", engine, f"{label} server reported no cached-token count; check its log")
            elif cached > CACHE_LIMIT:
                report("FAIL", engine, f"{label} {cached} prompt tokens came from the cache")
            if r.get("completion_tokens") != r.get("max_tokens"):
                report("WARN", engine, f"{label} {r.get('completion_tokens')} tokens generated, "
                                       f"not {r.get('max_tokens')} (finish_reason {r.get('finish_reason')})")
            if r.get("token_source") != "usage":
                report("WARN", engine, f"{label} token counts estimated from chunks")
        any_line = pair.get("long") or pair.get("short")
        if any_line:
            if any_line.get("prefill_tokens_per_s") is None:
                report("WARN", engine, "prefill from the difference could not be computed")
            if not any_line.get("memory_delta_mb"):
                report("WARN", engine, "no memory delta (was --idle-snapshot given?)")
            else:
                mem, source = memory_added(any_line)
                if mem is not None and mem <= 0:
                    report("WARN", engine, f"memory added is {mem} MB ({source}); a loaded model "
                                           "cannot add nothing, so take a new idle snapshot")
        f = features.get(engine)
        if not f:
            report("FAIL", engine, "no feature-probe line")
        else:
            for section in ("tool_calling", "structured_output"):
                if not isinstance((f.get(section) or {}).get("supported"), bool):
                    report("FAIL", engine, f"feature line has no boolean {section}.supported")
        if pair and f and set(pair) == {"short", "long"}:
            report("PASS", engine, "speed pair and feature line present")
    print(f"\n{failures} failure(s)")
    return 1 if failures else 0


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md", help="notebook to read")
    parser.add_argument("--check", action="store_true", help="validate the lines instead of printing tables")
    parser.add_argument("--allow-cpu", action="store_true", help="accept a CPU backend (for testing the scripts)")
    args = parser.parse_args()

    path = Path(args.labbook)
    if not path.exists():
        print(f"{path} does not exist", file=sys.stderr)
        return 1
    order, speed, features = latest(load(path))
    if args.check:
        return check(order, speed, features, args.allow_cpu)
    print_tables(order, speed, features)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download summarise-engines.py226 lines

RunnableAll tracks

the tables, from the notebook lines

cd ~/llm-course
python3 summarise-engines.py --labbook labbook.md

Output — what you should see

Speed and memory (medians; prefill from the difference between the two prompts)

| Engine | Quant | Backend | TTFT short s | TTFT long s | Prefill tok/s | Decode short tok/s | Decode long tok/s | Memory added MB | Memory source |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| llama.cpp | Q4_K_M | CUDA | x.xxx | x.xxx | xxxx.x | xx.xx | xx.xx | xxxx | MemAvailable |
| ... one row per engine ... |

API features

| Engine | /v1/models | Tool calling | JSON schema | Detail |
| --- | --- | --- | --- | --- |
| llama.cpp | yes | yes | yes | — |
| ... |

Rows for the speed recording sheet (paste into the Benchmark rows prop)

  ['llama.cpp, Q4_K_M', x.xxxx, x.xxx, xxxx.x, xx.xx, xx.xx, xxxx],
...

The script keeps the most recent run per engine, so re-measuring after a mistake replaces the row, and names the memory source it chose for your machine as Task 3 describes. Copy the rows into the two recording sheets, and fill in each context field with what you ran.

Pending validationSpeed and memory, one row per engine — your recording sheet

Engine and quantisation	Time to first token, short prompt (s)	Time to first token, long prompt (s)	Prefill tokens/s, from the difference	Decode tokens/s, short prompt	Decode tokens/s, long prompt	Memory added, MB
llama.cpp, Q4_K_M	—	—	—	—	—	—
Native engine on your track	—	—	—	—	—	—
Second native engine, Tracks X and N	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · as listed per row the version string recorded for each engine · Qwen3-8B, as listed per row; the engines do not serve the same bits · 8,192 tokens of context · the date you ran it

Empty on purpose: fill it from summarise-engines.py. Prefill is (long prompt tokens minus short prompt tokens) divided by (long minus short time to first token). Memory is the difference from the idle snapshot, from the source the script names.

Pending validationAPI features, one row per engine — your recording sheet

Engine	Model listing	Tool calling returned tool_calls	JSON schema produced a matching object	Detail from the probe
llama.cpp	—	—	—	—
Native engine on your track	—	—	—	—
Second native engine, Tracks X and N	—	—	—	—

your machine: track, chip and memory · as listed per row the version string recorded for each engine · Qwen3-8B, as served per engine · 8,192 tokens of context · the date you ran it

Not a speed table, and just as important. Part 10 needs structured output; Parts 24 and 25 need tool calling. Put the probe's detail string in the last column: 'the model wrote a tool_call block as text' and 'accepted the request but the content was not JSON' are different problems with different fixes.

9. Explain the differences

Write one sentence per question in the notebook, each with the arithmetic that supports it.

Decode. Divide each native engine’s short-prompt decode rate by llama-server’s, and put that beside the ratio of bytes read per token from Task 1:

Track	Predicted decode ratio, native / portable	If the measured ratio is lower	If it is higher
S	4.67 / 15.14 = 0.31	TensorRT-LLM is further below its bandwidth ceiling than llama-server	llama-server is further below its ceiling; check its offload lines
X	4.67 / 4.67 = 1.00 for every row	That build’s kernels, or a device line naming the wrong backend	The other build is the inefficient one
M	4.67 / 4.26 = 1.10	MLX’s kernels cost more than the smaller file saves	llama-server’s Metal path is further below its ceiling
N, EXL3	4.67 / 3.95 = 1.18	The ExLlamaV3 README notes its kernel is less efficient on Ampere and at low bit rates	Rare; check that llama-server offloaded 37/37 layers
N, AWQ	4.67 / 4.85 = 0.96	Kernel or scheduling overhead at batch size 1	llama-server is further below its ceiling

Also compare each measured rate with its own ceiling: a rate above the ceiling means the bandwidth figure is wrong, and a rate far below it on a fast discrete card is normal, because compute rather than memory becomes the limit.

Prefill. Prefill is compute-bound, so bandwidth predicts nothing here. State which engine’s difference-based rate is higher and by what factor; that is kernel quality and batching at work, which is where a native engine has the most room to differ from a portable one.

Memory. Subtract from the measurement only the bytes its memory source can see. Two engines keep the input embedding in host memory, where the per-process GPU figure does not count it, and llama-server keeps its copy in memory-mapped file pages, which MemAvailable counts as available. The predictions below are arithmetic from Task 1’s tensor bytes and KV sizes, not measurements:

Engine	Track	Memory source	Predicted visible bytes	Excluded
llama-server	N	nvidia-smi per-process (device under WSL2)	5,021,827,072 − 350,060,544 = 4,671,766,528 weights + 1,207,959,552 KV = 5,879,726,080	The CPU_Mapped input embedding
llama-server	S	MemAvailable	4,671,766,528 + 1,207,959,552 = 5,879,726,080	The CPU_Mapped input embedding: file-backed pages, not reflected in MemAvailable
llama-server, both builds; Lemonade	X	rocm-smi VRAM + GTT	4,671,766,528 + 1,207,959,552 = 5,879,726,080	The CPU_Mapped input embedding, host memory outside the VRAM and GTT counters
TabbyAPI with ExLlamaV3	N	nvidia-smi per-process	5,190,675,200 − 1,244,659,712 = 3,946,015,488 + 1,207,959,552 KV = 5,153,975,040	The 16-bit input embedding, kept on the CPU (prefer_cpu in ExLlamaV3 1.4.8’s modules/embedding.py)
vLLM	N	nvidia-smi per-process	6,098,479,104 + 1,342,177,280 = 7,440,656,384	Nothing: the embedding is on the GPU
trtllm-serve	S	MemAvailable	16,381,470,720 + 1,207,959,552 (8,192 tokens) = 17,589,430,272	Nothing
mlx_lm.server	M	vm_stat active + wired + compressed	4,607,731,712 + 1,536 × 147,456 = 4,834,224,128	Nothing; the cache holds about 1,536 tokens in 256-token steps
llama-server	M	vm_stat active + wired + compressed	Between 1,207,959,552 (KV alone) and 6,229,786,624 (KV plus all 5,021,827,072 mapped weight bytes)	Whatever share of the MTL0_Mapped and CPU_Mapped file pages vm_stat does not count as active or wired

Pseudocode — not a real command

overhead (MB) = memory added (MB) − predicted visible bytes / 1,000,000
illustrative inputs, not a measurement: llama-server on Track N adding 6,400 MB
overhead      = 6,400 − 5,879,726,080 / 1,000,000 = 6,400 − 5,880 = 520 MB

The script’s MB are 1,000,000 bytes. What the overhead holds is the CUDA or Metal context, compute buffers and the server process; a negative overhead means you subtracted something the source cannot see. On Track M, write down the MTL0_Mapped model buffer size line from the load log beside the vm_stat delta, and state that mapped file pages may be counted differently from MLX’s allocations, so the two Track M rows are not measured on the same terms.

Features. Name every probe failure and the setting or version that would change it.

Label what each comparison can establish

Build an identity row before each engine run: source checkpoint, representation, template, prompt, context, answer limit and cache state. Mark every difference from the portable baseline. Where equal representations are unavailable, describe the result as a comparison of deployable stacks.

Use the feature probe before the performance sweep. Check plain completion, structured output and tools only where the checkpoint and engine claim the relevant feature. Preserve the raw response when a probe fails; a framework may otherwise hide a parser mismatch with retries.

Stop the previous engine before starting the next on constrained hardware, confirm memory has been released and check which process owns the requested port. Run warm-up consistently. Summarise useful throughput, first-token latency, peak memory and task failures together. A fast engine that cannot perform the application’s required tool round trip is not an eligible replacement. The report should name the winning configuration for your task and the limitations of the comparison, including any backend or format whose run remains unvalidated.

Validation

Check	Command	Pass
Every rule the notebook can check	python3 summarise-engines.py --labbook labbook.md --check	Every engine has a PASS line and the last line is 0 failure(s); WARN lines are reviewed, not fatal
The control allocated what Task 1 predicted	grep "llama_kv_cache: size" llama-server.log	size = 1152.00 MiB ( 8192 cells, 36 layers, 1/1 seqs)
The control’s prompt cache was off	grep "prompt cache is disabled" llama-server.log	load_model: prompt cache is disabled - use `--cache-ram N` to enable it
The control ran on the accelerator	grep offloaded llama-server.log	offloaded 37/37 layers to GPU
Decode is physically plausible	Compare each short-prompt decode rate with its ceiling from predict.py	Every rate is at or below its ceiling
Only one server was loaded per measurement	The port loop at the top of every block that starts a server	It printed only port check finished before each server started, and every Lemonade block ended with lemonade unload
The recording sheets are complete	Read them	Every row and context field holds your values

The --check rules fail a line whose backend is empty or CPU, whose engine version or quantisation is empty, whose time to first token or decode rate is null, which was recorded with --reuse-prompt, or whose server reported more than 64 cached prompt tokens; they also fail an engine with no feature line. They warn when a server reported no cached-token count, generated fewer than 128 tokens, or when token counts came from chunks.

Expected outcome

A notebook section that states, for your machine on this date, which engine reads a prompt faster, which writes faster and whether the file size explains it, which uses more memory and where it went, and which API features each engine delivered with the settings used.

The native engine is not uniformly ahead. Winning prefill and losing decode, or the reverse, is a common and useful result, and it is why the course keeps llama.cpp as the baseline.

Troubleshooting

Symptom	Cause	Fix
FAILED: ... did not answer 200 within 120 s	The server is still loading, or crashed	Read terminal 1; for slow loaders pass --wait-seconds 900
FAILED: the server streamed no text	Model name not recognised, or a server that streams no text fields the script knows	curl http://127.0.0.1:<port>/v1/models and use the id; on Track M send the exact --model path
mlx_lm.server log shows it downloading or failing to load a model you did not name	The request’s model differed from the served path, so the server tried to load it	Send "$MLX_MODEL", the absolute path the server was started with
cached equals nearly the whole prompt	--reuse-prompt was passed, or a proxy stripped the tag	Re-run without it; --check fails such lines
128 tok is lower, with finish_reason stop	The model finished within the budget, usually on the short prompt with thinking disabled server-side	Record it; the decode rate is still valid, and --check only warns
Device line names no GPU, or offloaded 0/37 layers	A CPU-only build, or the wrong LLAMA_BIN	Run llama-server --list-devices from that directory; rebuild per Part 6
Decode rates vary by more than a few per cent between runs	Another process, thermal limits, battery power	Close applications, plug in, re-run; the most recent run replaces the row
Tool-calling probe reports <tool_call> written as text	The server did not parse the model’s call format	llama-server: check --jinja; vLLM: the two tool flags; TensorRT-LLM: --tool_parser qwen3; TabbyAPI: no listed format for this model, which is the finding
JSON-schema probe: valid JSON that does not match the schema	The server applied a grammar but not this schema, as TensorRT-LLM 1.2.1’s source does with the OpenAI envelope	Record the container tag; compare with a newer tag if you have time
JSON-schema probe: status 400	The server rejects response_format outright	Record the detail string; it is a clear result
Container exits with Bus error (core dumped)	Missing --ipc=host	Use serve-trtllm.sh, which passes it, as NVIDIA’s container documentation requires
serve-trtllm: a container named trtllm-lab already exists	An earlier container is still present	docker rm -f trtllm-lab
waiting for TabbyAPI to write api_tokens.yml repeating, sed: can't read /home/you/tabbyAPI/api_tokens.yml, or did not answer 200 within 600 s (status 401)	Terminal 2 ran before TabbyAPI’s first start finished installing and writing its key	Wait for Your API key is: in terminal 1, then re-run the terminal 2 block
status 401 from TabbyAPI	No or wrong API key	Re-read api_key from ~/tabbyAPI/api_tokens.yml into TABBY_KEY
vLLM exits during startup with CUDA out of memory on a 12 GB card	Weights, 1.25 GiB of cache, CUDA graphs and activations exceed the card	Skip the vLLM row and write why; it is optional below 16 GB
lemonade status cannot reach the server	The Lemonade Server service is not running; the lemonade-sdk/lemonade repository’s data/lemond.service.in names the unit lemond	systemctl status lemond; if it is inactive, sudo systemctl start lemond, or systemctl --user start lemond for the per-user unit the repository also ships; then re-run lemonade status
lemonade load fails after the ==> Lemonade Server header	The backend is not installed, or its llama.cpp build rejected an option passed through --llamacpp-args	Run lemonade backends to confirm the backend is installed; if it is, re-run with LLAMACPP_ARGS="--flash-attn on" MODEL=user.Qwen3-8B-Q4_K_M BACKEND=rocm bash serve-lemonade.sh and record that the prompt cache stayed on
Lemonade requests return 404 on /v1/...	Base path differs in your version; AMD’s playbook gives /api/v1	Re-run with --base-url http://127.0.0.1:13305/api/v1 and record it
memory added is empty	No --idle-snapshot, or the snapshot was taken on another boot	Take a new snapshot with no server running and re-measure
Under WSL2, no gpu_process_mb in the memory line and the source reads nvidia-smi device; or WARN ... memory added is 0 MB	NVML in WSL2 does not report compute processes (CUDA on WSL guide, features not yet supported)	Expected: the script falls back to the device figure, which counts every application on the card, so close other GPU applications, take a new idle snapshot and re-measure
Memory added is much larger than predicted on Track M	Other applications grew during the run	Close them, take a fresh idle snapshot, re-measure

Cleanup

Stop every server: Ctrl-C in each foreground terminal, docker stop trtllm-lab on Track S, lemonade unload on Track X. Then confirm nothing answers:

RunnableAll tracks

confirm every port is free

for port in 8080 8081 8082 8083 5000 13305; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port still answers"
done; echo "check finished"
rm -f ~/llm-course/*.pid

Output — what you should see

check finished

On Track X, 13305 may still answer, because Lemonade’s service stays up with no model loaded; that is expected.

Keep labbook.md, idle-memory.json, lab.env, the recording sheets and the model files. Part 9 serves Qwen3-8B again and compares against these numbers, and Part 17 reuses the measuring shape for speculative decoding.

⚠Warning

The commands below delete things that took a long time to download. Each removes exactly the path shown: check it before pressing return, and note in the notebook what you removed, so that a later comparison does not quietly become a different model.

RunnableAll tracks

reclaim disk, one format at a time

du -sh ~/models/Qwen/Qwen3-8B ~/models/mlx-community/Qwen3-8B-4bit \
    ~/models/turboderp/Qwen3-8B-exl3-4.0bpw ~/models/Qwen/Qwen3-8B-AWQ 2>/dev/null
rm -r ~/models/Qwen/Qwen3-8B

That rm removes the 16.4 GB bf16 checkpoint only; a later part that needs bf16 weights downloads them again. The same pattern with another path removes another format. On Track S, docker image rm nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 removes the image; on Track X, lemonade delete user.Qwen3-8B-Q4_K_M removes Lemonade’s copy; on Track N, rm -r ~/vllm-env ~/tabbyAPI removes both environments and TabbyAPI’s configuration and keys.

What you learned

Objective	The observation that proved it	Fields recorded
An engine comparison is a list of things held fixed	Every notebook line carries context length, generation length, repetitions and the run tag, and --check passed	context_length, max_tokens, repetitions, prompt_tag
File formats spend bits differently, and the headers say how	The five format sizes derived in Task 1 match the published tensor bytes	Repository, revision, size per format
Decode is bandwidth arithmetic	Measured decode ratios beside the ratios of bytes read per token	decode_tokens_per_s per engine, the predicted ratio
Prefill and decode are different measurements	The difference between two prompts gave a prefill rate that llama-server’s own timings confirmed	ttft_s per test, prefill_tokens_per_s, server_prefill_tokens_per_s
A cache can counterfeit a measurement	--reuse-prompt drove cached to the prompt length and the prefill figure to nonsense	max_cached_prompt_tokens
Memory depends on allocation policy, not just weights	llama-server’s 1,152 MiB KV line, TensorRT-LLM and vLLM capped explicitly, MLX growing on demand	memory_delta_mb and its source, the load-log buffer lines
“OpenAI-compatible” is a claim to test	The probe results, and which parser or grammar setting each depended on	tool_calling, structured_output and their detail strings

Check your understanding

Question 1. Why does the lab send two prompts of very different lengths rather than one?

• To test the context window
• Because the difference between the two time-to-first-token figures, divided into the difference in prompt tokens, isolates prefill from the fixed costs every request pays
• To make the model produce longer answers
• To warm the cache

Show the answer and why

Answer: Because the difference between the two time-to-first-token figures, divided into the difference in prompt tokens, isolates prefill from the fixed costs every request pays

A single time to first token includes the HTTP round trip, scheduling and the first decode step. Subtracting the short test cancels them. On llama-server you can check the result against the timings block the server itself reports.

Question 2. You remove the random run tag and re-measure. The long prompt’s time to first token drops to almost the short prompt’s, and the prefill figure is forty times higher. What happened?

• The native kernels warmed up
• The warm-up request left the whole prompt in the prefix cache, so the measured runs reused it and skipped prefill; the server’s cached-token count would read within a token of the prompt length
• The engine switched to a faster backend
• The prompt was truncated

Show the answer and why

Answer: The warm-up request left the whole prompt in the prefix cache, so the measured runs reused it and skipped prefill; the server’s cached-token count would read within a token of the prompt length

llama-server, TensorRT-LLM, vLLM and mlx_lm.server all reuse the KV cache of a prefix they have seen, by default. An identical prompt measures the cache. That is why the script prefixes every request with a random tag and records the cached-token count.

Question 3. On Track S, llama-server reads 4.67 GB per generated token and TensorRT-LLM serving bf16 reads 15.14 GB. TensorRT-LLM decodes at 0.29 times llama-server’s rate. What is the best reading?

• TensorRT-LLM is a slow engine
• Almost all of the gap is bytes read per token, which predicts 0.31; the small remainder says TensorRT-LLM sits slightly further below its bandwidth ceiling than llama-server does
• The measurement is wrong because a native engine cannot be slower
• Decode speed does not depend on the file size

Show the answer and why

Answer: Almost all of the gap is bytes read per token, which predicts 0.31; the small remainder says TensorRT-LLM sits slightly further below its bandwidth ceiling than llama-server does

Decode is bandwidth-bound: the ceiling is bandwidth divided by bytes read per token. Comparing the measured ratio with 4.67 / 15.14 separates what the number format costs from what the engine costs. The prefill column is where kernel differences show more clearly.

Question 4. Which of these commands, run on Track X, breaks the comparison with the llama-server control?

• lemonade pull user.Qwen3-8B-Q4_K_M --checkpoint main unsloth/Qwen3-8B-GGUF:Q4_K_M --recipe llamacpp
• lemonade pull Qwen3-8B-GGUF, then measuring that model as if it were the same file
• LLAMA_BIN=$HIP_BIN MODEL=$GGUF bash serve-llama-cpp.sh
• lemonade unload

Show the answer and why

Answer: lemonade pull Qwen3-8B-GGUF, then measuring that model as if it were the same file

Lemonade’s built-in Qwen3-8B-GGUF entry pointed at the Q4_1 file in the registry read on 2026-09-13, not Q4_K_M. A different quantisation reads different bytes per token, so the row would measure the file as well as the engine. Registering the exact variant under a user. name keeps the bytes identical.

Question 5. The feature probe reports that the native engine answered with a <tool_call> block written as text. Which conclusions are reasonable? Select all that apply.

• The model produced a tool call in its own format
• The server was not configured, or not able, to parse that format into tool_calls
• This server is not usable for Part 24 as configured
• Tool calling is impossible with this engine and model

Show the answer and why

Answer: The model produced a tool call in its own format, The server was not configured, or not able, to parse that format into tool_calls, This server is not usable for Part 24 as configured

A probe measures the configuration you ran. vLLM needs two flags, TensorRT-LLM a --tool_parser and TabbyAPI a tool_format that matches the model. Recording the detail string, not a bare no, is what lets you tell a missing setting from a missing capability.

Question 6. On a 24 GB card, vLLM started with its default memory settings shows about 20 GB in use for a model whose weights are 6.1 GB. What does that figure measure?

• vLLM’s inefficiency
• Mostly the reservation: vLLM sizes its KV cache from --gpu-memory-utilization of the whole card unless --kv-cache-memory-bytes is set, so the memory column would record the setting, not the engine
• The model’s true size
• Memory leaked by the previous server

Show the answer and why

Answer: Mostly the reservation: vLLM sizes its KV cache from --gpu-memory-utilization of the whole card unless --kv-cache-memory-bytes is set, so the memory column would record the setting, not the engine

Engines that preallocate a paged KV pool fill whatever fraction they are given. That is why this lab caps TensorRT-LLM with kv_cache_config.max_tokens and gives vLLM an explicit cache size, then predicts memory as the weight and cache bytes the memory source can see, plus runtime overhead.

Check my answersClear