Lab: Serve a Model to Twenty Concurrent Users

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The engine versions, model file and backend each track was run with will be recorded here when the validation pass is done.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this lab you will have a curve, not a number. For Qwen3-8B on your machine you will have output throughput, request throughput, goodput, time to first token and time per output token at concurrency one, five, ten and twenty, for llama-server’s parallel slots and for a second engine: vLLM on Tracks S and N (and X if it runs), the MLX server on Track M. You will have predicted the KV pool each engine allocates and checked the prediction against its log, found the concurrency at which your machine stops gaining throughput, measured what prefix caching is worth against a control of the same length, and changed slots, context, cache type and memory fraction one at a time.

Part 6 measured one user. This measures a service. The two produce different-shaped answers, and the second is the one every remaining level of this course cares about.

⌂Single-machine path

This is a single-machine lab on every track. The load generator may run on a second machine on the same network if you would rather not have the client competing with the server for CPU; Task 4 shows how to tell whether that matters. Multi-machine serving starts in Part 18.

Architecture

One server at a time, in terminal 1. The load generator and every other command run in terminal 2. Two servers loaded together measure their competition for memory, not either engine.

What one load-test level does

1. 1load-test.py starts c workersClosed loop: each worker sends a streamed chat completion, waits for the last token, and sends the next. c requests are in flight until the queue drains.
2. 2The server admits up to its limitllama-server: one request per slot, extras deferred. vLLM: up to --max-num-seqs, and only while the block pool has room. mlx_lm.server: up to 32 decoding and 8 prefilling by default.
3. 3Continuous batchingEvery step reads the weights once for all running sequences, plus each sequence's KV cache.
4. 4One JSON line per levelThroughput, goodput, TTFT and TPOT percentiles, prompt and cached tokens, finish reasons, appended to labbook.md.
5. 5summarise-load.pyScaling efficiency, knee, TPOT fit, operating point, comparisons and the validation check.

Engine	Port	Started by	Weights it loads	Tracks
llama-server, 20 slots	8080	serve-llama-parallel.sh	~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf	all
vLLM, NGC container	8000	serve-vllm-container.sh	~/models/Qwen/Qwen3-8B-AWQ	S
vLLM, native	8000	serve-vllm-native.sh	~/models/Qwen/Qwen3-8B-AWQ (N), ~/models/Qwen/Qwen3-8B (X)	N, X if it runs
mlx_lm.server	8081	Part 8’s serve-mlx.sh	~/models/mlx-community/Qwen3-8B-4bit	M

Work in ~/llm-course, where Part 1 created labbook.md, and save every lab file on this page there with its download link. Everything listens on 127.0.0.1 only.

Requirements

Qwen3-8B is Apache-2.0 licensed and ungated; the model reference records the licence, and the AWQ and bf16 checkpoints below are Qwen’s own conversions under the same licence. Every track needs llama.cpp built as in Part 6’s install lesson (llama.cpp v0.4.0 · verified 2026-09-08), Python 3.9 or later (both scripts are standard library only), the hf CLI from Part 4 (Hugging Face CLI 1.30.0 · verified 2026-09-08), and the GGUF file. On 2026-09-13 the repository listing gave Qwen3-8B-Q4_K_M.gguf as 5,027,784,512 bytes; Part 6 downloaded it.

Track	Second engine	Second-format download (Hub listing, 2026-09-13)	Attended	Unattended
S	vLLM, NGC container	Qwen/Qwen3-8B-AWQ, 12 files, 6,114,568,235 bytes; the container image if Part 9’s install lesson did not pull it	about 100 minutes	the AWQ download; first vLLM start
X	vLLM from the ROCm wheels, if it starts	Qwen/Qwen3-8B bf16, 15 files, 16,397,461,266 bytes	about 90 minutes, or 60 on llama-server alone	the bf16 download
M	mlx_lm.server	mlx-community/Qwen3-8B-4bit, 4.6 GB, from Part 8	about 80 minutes	none if Part 8 was done
N	vLLM, native	Qwen/Qwen3-8B-AWQ, as for S; Part 8 downloaded it	about 100 minutes	first vLLM start

The attended times include two to five minutes per sweep while the generator runs; you can read ahead during those.

Track S — NVIDIA DGX Spark

Pinned: vLLM 0.28.0 · verified 2026-09-08 for the documentation this page follows; the NGC image nvcr.io/nvidia/vllm:26.08-py3 carries its own vLLM build, which Task 8 records. 128 GB of unified memory holds every configuration on this page with room to spare, so the lab runs the same 20 × 4,096 baseline as a 24 GB card and the extra memory goes into Task 9’s memory-fraction experiment. nvidia-smi reports memory usage as not supported on this machine (NVIDIA’s DGX Spark known issue, as Part 8 showed); read allocations from the engines’ logs instead.

Track X — AMD Ryzen AI Max+ 395Partial

vLLM's GPU installation page names gfx1151 among its ROCm GPUs, but the course has not run it and its quantisation matrix does not list AWQ for AMD GPUs; llama-server is the path this track can count on.

Pinned: llama.cpp v0.4.0 · verified 2026-09-08 with the Vulkan build from Part 6. If you installed vLLM from the ROCm wheels in the install lesson, try it in Task 8 with the bf16 checkpoint; if it does not start in ten minutes, skip Tasks 8 and 9 and record that it did not. The GPU-visible memory cap from Part 5 applies: the KV pool comes out of the visible share, not the machine total. Run this lab on Linux; on a Windows installation run llama-server natively and this page’s scripts from WSL2 against localhost.

Track M — Apple silicon

Pinned: mlx-lm 0.31.3 · verified 2026-09-08. vLLM has no supported path here, for the reasons the install lesson quotes, so the second engine is mlx_lm.server, started with Part 8’s serve-mlx.sh. Its 0.31.3 source batches requests that carry no seed: up to 32 decoding at once (--decode-concurrency) and 8 prefilling (--prompt-concurrency), allocating KV cache as it goes rather than at load. Close the browser and other large applications first; on unified memory they compete for the pool.

At 24 GB or less the baseline is reduced to 10 slots with an eight-bit cache (lab.env sets it), so the concurrency-20 level measures queueing. That is your machine’s result, not a failure.

Track N — NVIDIA desktop or laptop

Pinned: vLLM 0.28.0 · verified 2026-09-08, installed with uv as in the install lesson. Video memory is a hard ceiling. From 24 GB the baseline uses an f16 cache on both engines; a 16 GB card uses an eight-bit cache on both (q8_0 in llama.cpp, fp8 in vLLM), which lab.env sets. Anything else on the card, including the desktop, comes out of the same memory: check it in the preflight. Windows readers work in WSL2 with every command on this page run inside the Linux distribution, following Part 5’s CUDA-on-WSL setup.

Preflight

Download the four lab files embedded on this page into ~/llm-course first (load-test.py, summarise-load.py, serve-llama-parallel.sh and the vLLM script for your track). Then:

RunnableAll tracks

preflight: files, disk, free ports

cd ~/llm-course
python3 --version
ls labbook.md load-test.py summarise-load.py serve-llama-parallel.sh
ls -l ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
df -h ~/models
for port in 8000 8080 8081; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is in use"
done; echo "port check finished"

Output — what you should see

Python 3.x.x
labbook.md  load-test.py  serve-llama-parallel.sh  summarise-load.py
-rw-r--r-- 1 you you 5027784512 ... /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
Filesystem      Size  Used Avail Use% Mounted on
/dev/...        xxxG  xxxG  xxxG  xx% /
port check finished

The Python version must be 3.9 or later. If the GGUF file is missing, fetch it with Part 4’s script: bash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf labbook.md. A port … is in use line means a server from an earlier part is still running: stop it now.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

preflight: CUDA build, container image, AWQ checkpoint

~/llama.cpp/build/bin/llama-server --list-devices
docker image inspect --format '{{.Id}}' nvcr.io/nvidia/vllm:26.08-py3 \
    || docker pull nvcr.io/nvidia/vllm:26.08-py3
ls ~/models/Qwen/Qwen3-8B-AWQ/config.json \
    || hf download Qwen/Qwen3-8B-AWQ --local-dir ~/models/Qwen/Qwen3-8B-AWQ
free -g

Output — what you should see

Available devices:
  CUDA0: NVIDIA GB10 (xxxxxx MiB, xxxxxx MiB free)
sha256:xxxxxxxxxxxx...
/home/you/models/Qwen/Qwen3-8B-AWQ/config.json
               total        used        free      shared  buff/cache   available
Mem:             xxx          xx          xx          xx          xx         xxx

Available devices: (none) means a CPU-only build of llama.cpp: rebuild before going on. If the download ran instead of ls, it ends with ✓ Downloaded and path: /home/you/models/Qwen/Qwen3-8B-AWQ.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

preflight: Vulkan build, and vLLM if you installed it

~/llama.cpp/build/bin/llama-server --list-devices
~/vllm/.venv/bin/vllm --version 2>/dev/null || echo "no vLLM environment: Tasks 8 and 9 are skipped"

Output — what you should see

Available devices:
  Vulkan0: AMD Radeon ... (xxxxx MiB, xxxxx MiB free)
0.28.0

The install lesson creates its environment as .venv in the directory you ran uv venv in; this page assumes ~/vllm/.venv. If yours is elsewhere, use that path here and in lab.env. The free MiB figure on the device line is the GPU-visible memory the KV pool comes out of. If you will run vLLM, download the bf16 checkpoint now: hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen/Qwen3-8B.

Track M — Apple silicon

RunnableTrack M · Apple silicon

preflight: Metal build, mlx-lm, Part 8's files

~/llama.cpp/build/bin/llama-server --list-devices
command -v mlx_lm.server
ls ~/llm-course/serve-mlx.sh ~/models/mlx-community/Qwen3-8B-4bit/config.json
sysctl -n hw.memsize

Output — what you should see

Available devices:
  MTL0: Apple M... (xxxxx MiB, xxxxx MiB free)
/Users/you/.../bin/mlx_lm.server
/Users/you/llm-course/serve-mlx.sh  /Users/you/models/mlx-community/Qwen3-8B-4bit/config.json
xxxxxxxxxxx

hw.memsize is in bytes; it decides which lab.env you write below. If serve-mlx.sh is missing, download it from Part 8’s lab.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

preflight: CUDA build, card, vLLM, AWQ checkpoint

~/llama.cpp/build/bin/llama-server --list-devices
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
~/vllm/.venv/bin/vllm --version
ls ~/models/Qwen/Qwen3-8B-AWQ/config.json \
    || hf download Qwen/Qwen3-8B-AWQ --local-dir ~/models/Qwen/Qwen3-8B-AWQ

Output — what you should see

Available devices:
  CUDA0: NVIDIA GeForce RTX ... (xxxxx MiB, xxxxx MiB free)
name, memory.total [MiB], memory.used [MiB]
NVIDIA GeForce RTX ..., xxxxx MiB, xxx MiB
0.28.0
/home/you/models/Qwen/Qwen3-8B-AWQ/config.json

The install lesson creates its environment as .venv in the directory you ran uv venv in; this page assumes ~/vllm/.venv, so use your own path here and in lab.env if it differs. memory.used above a few hundred MiB with no model loaded is memory the lab cannot have: close what holds it.

Now write the variables every later command reads. Edit HOST_DESC to describe your machine. Each terminal you open for this lab starts with cd ~/llm-course and source lab.env.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

lab.env for Track S

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track S, DGX Spark GB10, 128 GB"
export SLOTS=20 KV_TYPE=f16 ALT_KV_TYPE=q8_0
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.08-py3
export VLLM_MODEL=/models/Qwen/Qwen3-8B-AWQ VLLM_QUANT=AWQ
export VLLM_KV_DTYPE=auto VLLM_KV_BYTES=12079595520
export MEM_FRACTION_A=0.30 MEM_FRACTION_B=0.40
EOF

VLLM_MODEL is the path inside the container, where ~/models is mounted as /models.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

lab.env for Track X

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track X, Ryzen AI Max+ 395, 128 GB"
export SLOTS=20 KV_TYPE=f16 ALT_KV_TYPE=q8_0
export VLLM_MODEL=$HOME/models/Qwen/Qwen3-8B VLLM_QUANT=bf16
export VLLM_KV_DTYPE=auto VLLM_KV_BYTES=12079595520
export MEM_FRACTION_A=0.40 MEM_FRACTION_B=0.50
export VLLM_VENV=$HOME/vllm/.venv
EOF

Track M — Apple silicon

RunnableTrack M · Apple silicon

lab.env for Track M, 32 GB or more

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track M, M4 Max, 64 GB"
export SLOTS=20 KV_TYPE=f16 ALT_KV_TYPE=q8_0
export MLX_MODEL=$HOME/models/mlx-community/Qwen3-8B-4bit
EOF

RunnableTrack M · Apple silicon

lab.env for Track M, 16 or 24 GB

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track M, M4, 16 GB"
export SLOTS=10 KV_TYPE=q8_0 ALT_KV_TYPE=q4_0
export MLX_MODEL=$HOME/models/mlx-community/Qwen3-8B-4bit
EOF

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

lab.env for Track N, 24 GB or more

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track N, RTX 4090 24 GB, Ubuntu 24.04"
export SLOTS=20 KV_TYPE=f16 ALT_KV_TYPE=q8_0
export VLLM_MODEL=$HOME/models/Qwen/Qwen3-8B-AWQ VLLM_QUANT=AWQ
export VLLM_KV_DTYPE=auto VLLM_KV_BYTES=12079595520
export MEM_FRACTION_A=0.90 MEM_FRACTION_B=0.80
export VLLM_VENV=$HOME/vllm/.venv
EOF

RunnableTrack N · NVIDIA GPU

lab.env for Track N, 16 GB

cat > ~/llm-course/lab.env <<'EOF'
export GGUF=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
export LLAMA_BIN=$HOME/llama.cpp/build/bin
export LLAMA_VERSION="$("$LLAMA_BIN/llama-server" --version 2>&1 | grep -m1 '^version:')"
export HOST_DESC="Track N, RTX 4080 16 GB, Ubuntu 24.04"
export SLOTS=20 KV_TYPE=q8_0 ALT_KV_TYPE=q4_0
export VLLM_MODEL=$HOME/models/Qwen/Qwen3-8B-AWQ VLLM_QUANT=AWQ
export VLLM_KV_DTYPE=fp8 VLLM_KV_BYTES=6039797760
export MEM_FRACTION_A=0.90 MEM_FRACTION_B=0.85
export VLLM_VENV=$HOME/vllm/.venv
EOF

Every number in these files is derived in Task 1.

Record: the lab.env you wrote, python3 --version, and the device line from --list-devices.

Tasks

1. Predict the pool, the fit and the ceiling

Two sums decide everything the curve will do. The first is memory: every concurrent sequence needs its own KV cache, and llama-server allocates all of it at load.

Pseudocode — not a real command

KV bytes per token   = 2 (K and V) × layers × KV heads × head_dim × bytes per value
                     = 2 × 36 × 8 × 128 × 2 = 147,456 for Qwen3-8B at f16
KV pool              = KV bytes per token × slots × context per slot
fits                 = weights + KV pool + compute buffers (and, on vLLM, graphs) ≤ usable memory

The second is bandwidth. A batched decode step reads the weights once for every running sequence, plus each sequence’s own cache:

Pseudocode — not a real command

bytes read per step (batch B) = weight bytes read + B × tokens in context × KV bytes per token
per-user ceiling (tokens/s)   = bandwidth (bytes/s) / bytes read per step
aggregate ceiling             = B × per-user ceiling

Save this as serving-arithmetic.py and run it. The inputs are Qwen3-8B’s config.json, the tensor bytes of the two four-bit files (the GGUF tensor table, as Part 8 read it, and the AWQ safetensors listing), and the vendor bandwidth figures in the hardware reference.

RunnableAll tracks

serving-arithmetic.py

# Memory and bandwidth arithmetic for serving Qwen3-8B to many users. Stated inputs only.
LAYERS, KV_HEADS, HEAD_DIM = 36, 8, 128        # Qwen3-8B config.json
SLOTS, CTX = 20, 4096                          # the lab's baseline: 20 sequences of 4,096
ELEMENTS = 2 * LAYERS * KV_HEADS * HEAD_DIM    # K and V values cached per token
KV_BYTES = {"f16": ELEMENTS * 2,              # llama.cpp f16; vLLM auto on an fp16 model
            "q8_0": ELEMENTS // 32 * 34,       # llama.cpp: 34 bytes per block of 32 values
            "fp8": ELEMENTS}                   # vLLM --kv-cache-dtype fp8
WEIGHTS = {"llama-server, Q4_K_M GGUF": (5_021_827_072, ["f16", "q8_0"]),
           "vLLM, AWQ safetensors": (6_098_581_864, ["f16", "fp8"])}
GIB = 2**30

print(f"KV pool for {SLOTS} x {CTX:,} = {SLOTS * CTX:,} tokens")
for name, per_token in KV_BYTES.items():
    pool = per_token * SLOTS * CTX
    print(f"  {name:<5} {per_token:>7,} B/token {pool:>15,} B = {pool / GIB:5.2f} GiB")
print("Weights + pool, before compute buffers and graphs (GiB)")
for engine, (weights, types) in WEIGHTS.items():
    for name in types:
        print(f"  {engine:<26} + {name:<5} = {(weights + KV_BYTES[name] * SLOTS * CTX) / GIB:5.2f}")

# Decode: every step reads the weights once plus each sequence's cache.
READ_PER_STEP = 5_021_827_072 - 350_060_544    # Q4_K_M tensors minus the embedding lookup
print("Bandwidth ceiling, Q4_K_M, f16 cache: aggregate tok/s (per user) at batch B")
for machine, gbps in {"Spark 273 GB/s": 273, "Strix 256 GB/s": 256,
                     "M4 Max 546 GB/s": 546, "RTX 4090 1008 GB/s": 1008}.items():
    for ctx in (100, 1500):                    # tokens each sequence attends over
        cells = []
        for batch in (1, 5, 10, 20):
            step = READ_PER_STEP + batch * ctx * KV_BYTES["f16"]
            per_user = gbps * 1e9 / step
            cells.append(f"B={batch:<2}{batch * per_user:6.0f} ({per_user:5.1f})")
        print(f"  {machine:<19} ctx {ctx:>5}: " + "  ".join(cells))

Output — what you should see

KV pool for 20 x 4,096 = 81,920 tokens
  f16   147,456 B/token  12,079,595,520 B = 11.25 GiB
  q8_0   78,336 B/token   6,417,285,120 B =  5.98 GiB
  fp8    73,728 B/token   6,039,797,760 B =  5.62 GiB
Weights + pool, before compute buffers and graphs (GiB)
  llama-server, Q4_K_M GGUF  + f16   = 15.93
  llama-server, Q4_K_M GGUF  + q8_0  = 10.65
  vLLM, AWQ safetensors      + f16   = 16.93
  vLLM, AWQ safetensors      + fp8   = 11.30
Bandwidth ceiling, Q4_K_M, f16 cache: aggregate tok/s (per user) at batch B
  Spark 273 GB/s      ctx   100: B=1     58 ( 58.3)  B=5    288 ( 57.5)  B=10   566 ( 56.6)  B=20  1099 ( 55.0)
  Spark 273 GB/s      ctx  1500: B=1     56 ( 55.8)  B=5    236 ( 47.3)  B=10   397 ( 39.7)  B=20   600 ( 30.0)
  Strix 256 GB/s      ctx   100: B=1     55 ( 54.6)  B=5    270 ( 53.9)  B=10   531 ( 53.1)  B=20  1031 ( 51.5)
  Strix 256 GB/s      ctx  1500: B=1     52 ( 52.3)  B=5    222 ( 44.3)  B=10   372 ( 37.2)  B=20   563 ( 28.1)
  M4 Max 546 GB/s     ctx   100: B=1    117 (116.5)  B=5    575 (115.1)  B=10  1133 (113.3)  B=20  2199 (109.9)
  M4 Max 546 GB/s     ctx  1500: B=1    112 (111.6)  B=5    473 ( 94.5)  B=10   793 ( 79.3)  B=20  1201 ( 60.0)
  RTX 4090 1008 GB/s  ctx   100: B=1    215 (215.1)  B=5   1062 (212.4)  B=10  2092 (209.2)  B=20  4059 (203.0)
  RTX 4090 1008 GB/s  ctx  1500: B=1    206 (206.0)  B=5    872 (174.5)  B=10  1464 (146.4)  B=20  2216 (110.8)

That output is arithmetic from stated inputs, not a measurement. Three things follow from it.

Why the tiers in lab.env differ. At f16 the llama-server pool plus weights is 15.93 GiB before any compute buffer, so a 16 GB card cannot hold twenty slots of 4,096 tokens; at q8_0 it is 10.65 GiB. vLLM’s AWQ weights are a gigabyte larger because Qwen’s AWQ build keeps both vocabulary tables at 16 bits. Two numbers in lab.env come straight from the first block: VLLM_KV_BYTES is the f16 or fp8 pool in bytes, so vLLM is given exactly the tokens llama-server allocates.

What the logs should say. Each engine prints its own version of this arithmetic at startup, and Tasks 2 and 8 compare it with yours:

Configuration	llama-server load log	vLLM startup log
20 × 4,096, f16 / auto	llama_kv_cache: size = 11520.00 MiB ( 4096 cells, 36 layers, 20/20 seqs), K (f16): 5760.00 MiB, V (f16): 5760.00 MiB	reserved 11.25 GiB memory for KV Cache … then GPU KV cache size: 81,920 tokens, Maximum concurrency for 4,096 tokens per request: 20.00x
20 × 4,096, q8_0 / fp8	llama_kv_cache: size = 6120.00 MiB ( 4096 cells, 36 layers, 20/20 seqs), K (q8_0): 3060.00 MiB, V (q8_0): 3060.00 MiB	reserved 5.62 GiB memory for KV Cache … then the same 81,920 tokens and 20.00x
10 × 4,096, q8_0 (Track M at 24 GB or less)	llama_kv_cache: size = 3060.00 MiB ( 4096 cells, 36 layers, 10/10 seqs), …	not run

The llama-server lines are the format string v0.4.0 printed when this page’s script was tested, filled with the arithmetic. The vLLM tokens follow from its block arithmetic in kv_cache_utils.py: blocks = pool bytes ÷ (16 tokens × bytes per token), 5,120 blocks here, and a 4,096-token request needs 256 of them, if the attention backend keeps the default block size of 16.

Where the curve should bend. With short prompts (context near 100 tokens) the bandwidth ceiling per user barely falls from B=1 to B=20: batching is nearly free as far as memory traffic goes, so any early flattening you measure is compute, scheduling or the client, not bandwidth. With a 1,500-token context the cache traffic at B=20 is comparable to the weights, and the per-user ceiling roughly halves on every track. That is why Task 6’s long prompts decode more slowly at high concurrency even when prefix caching removes their prefill.

Record: the pool for your configuration in GiB, the weights-plus-pool figure against your usable memory, and the four aggregate ceilings for your machine at context 100.

2. Start llama-server with twenty slots and read what it allocated

RunnableAll tracks

serve-llama-parallel.sh

#!/usr/bin/env bash
# Purpose: start llama-server with a fixed number of parallel slots and continuous batching,
#          every setting that changes a load test stated on the command line, the load log
#          copied to a file you can search, and the process id written where you can find it
# Platform: all (spark, strix, mac, nvidia; CUDA, Vulkan and Metal builds); the only served
#           path on Track M and the certain one on Track X
# Minimum memory: 16 GB, because the context is allocated for every slot at load
# Assumes: llama.cpp built as in Part 6 (v0.4.0 or a build with the same options), a GGUF
#          file in $MODEL, nothing else answering on $PORT; the server runs in the foreground
#          so that Ctrl-C stops it
#
# Usage: MODEL=$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf bash serve-llama-parallel.sh
#        SLOTS=8 CACHE_TYPE=q8_0 MODEL=... bash serve-llama-parallel.sh
#
# Environment:
#   MODEL         path to the GGUF file                   (default: the Qwen3-8B Q4_K_M file)
#   LLAMA_BIN     directory holding llama-server, or the binary itself
#                                                         (default: $HOME/llama.cpp/build/bin)
#   ALIAS         model name the API reports              (default: local-chat)
#   HOST          address to bind                         (default: 127.0.0.1)
#   PORT          port to listen on                       (default: 8080)
#   SLOTS         parallel slots, the most sequences decoded at once        (default: 20)
#   CTX_PER_SLOT  context per slot, in tokens              (default: 4096)
#   CACHE_TYPE    KV cache type for keys and values: f16, q8_0, ...        (default: f16)
#   CACHE_RAM     host-RAM prompt cache limit in MiB; 0 disables it         (default: 0)
#   NGL           layers to offload                        (default: 999, meaning all)
#   PIDFILE       where to write the server's pid          (default: ./llama-server.pid)
#   LOGFILE       copy of the server log                   (default: ./llama-server.log)
#
# What the options do, as read from llama-server --help at v0.4.0:
#   --ctx-size is the total across slots. With --parallel given explicitly the KV cache is not
#   unified ("default: enabled if number of slots is auto"), so each slot gets
#   ctx-size / parallel tokens and the load log says so: "n_slots = 20, n_ctx_slot = 4096,
#   kv_unified = 'false'". The whole cache is allocated when the model loads.
#   --cache-type-k/-v choose the bytes per cached token; a quantised value cache needs flash
#   attention, hence --flash-attn on.
#   --cache-prompt is on by default: each slot keeps its last prompt's KV and a new request
#   reuses the longest common prefix of the slot it lands on. That is the prefix caching the
#   lab measures. --cache-ram (default 8192 MiB) adds a second cache in host RAM that saves
#   idle slots and restores them; this script turns it off so that the baseline measures
#   batching rather than copies to and from host memory. Set CACHE_RAM=8192 to measure it.
#   --verbosity 4 is the level at which the load log prints the KV cache and buffer sizes.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf}"
LLAMA_BIN="${LLAMA_BIN:-$HOME/llama.cpp/build/bin}"
ALIAS="${ALIAS:-local-chat}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8080}"
SLOTS="${SLOTS:-20}"
CTX_PER_SLOT="${CTX_PER_SLOT:-4096}"
CACHE_TYPE="${CACHE_TYPE:-f16}"
CACHE_RAM="${CACHE_RAM:-0}"
NGL="${NGL:-999}"
PIDFILE="${PIDFILE:-./llama-server.pid}"
LOGFILE="${LOGFILE:-./llama-server.log}"

die() { echo "serve-llama-parallel: $*" >&2; exit 1; }

if [ -f "$LLAMA_BIN" ] && [ -x "$LLAMA_BIN" ]; then
    SERVER="$LLAMA_BIN"
elif [ -x "$LLAMA_BIN/llama-server" ]; then
    SERVER="$LLAMA_BIN/llama-server"
else
    die "no llama-server at $LLAMA_BIN; set LLAMA_BIN, or build llama.cpp as in Part 6"
fi
[ -f "$MODEL" ] || die "model file not found: $MODEL"
case "$SLOTS$CTX_PER_SLOT" in
    *[!0-9]*) die "SLOTS and CTX_PER_SLOT must be whole numbers" ;;
esac
[ "$SLOTS" -ge 1 ] || die "SLOTS must be at least 1"
if command -v curl >/dev/null 2>&1 \
    && curl --silent --max-time 2 --output /dev/null "http://$HOST:$PORT/"; then
    die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi

TOTAL_CTX=$(( SLOTS * CTX_PER_SLOT ))

echo "==> llama-server, $SLOTS parallel slot(s)"
"$SERVER" --version 2>&1 | head -n 2 || true
cat <<INFO
    model             $MODEL
    served as         $ALIAS
    listening on      http://$HOST:$PORT/v1
    slots             $SLOTS
    context per slot  $CTX_PER_SLOT
    total context     $TOTAL_CTX   (--ctx-size is the total across slots)
    KV cache type     $CACHE_TYPE (keys and values)
    host prompt cache $CACHE_RAM MiB
    pid written to $PIDFILE, log copied to $LOGFILE
INFO

rm -f "$LOGFILE"
echo $$ > "$PIDFILE"

# exec keeps this shell's pid, so $PIDFILE names the server process itself.
exec "$SERVER" \
    --model "$MODEL" \
    --alias "$ALIAS" \
    --host "$HOST" \
    --port "$PORT" \
    --ctx-size "$TOTAL_CTX" \
    --parallel "$SLOTS" \
    --cont-batching \
    --n-gpu-layers "$NGL" \
    --flash-attn on \
    --cache-type-k "$CACHE_TYPE" \
    --cache-type-v "$CACHE_TYPE" \
    --cache-ram "$CACHE_RAM" \
    --metrics \
    --jinja \
    --verbosity 4 \
    --log-file "$LOGFILE"

Download serve-llama-parallel.sh114 lines

The script’s header explains every option it passes. The two that shape this lab: --ctx-size is the total across slots, and because --parallel is given explicitly the v0.4.0 build does not unify the cache, so each slot gets exactly CTX_PER_SLOT tokens of its own. The host-RAM prompt cache is off (CACHE_RAM=0) so the baseline measures batching rather than copies to host memory; per-slot prompt caching stays on, and Task 6 measures it.

RunnableAll tracks

terminal 1: llama-server, baseline configuration

cd ~/llm-course
source lab.env
SLOTS=$SLOTS CACHE_TYPE=$KV_TYPE MODEL=$GGUF LLAMA_BIN=$LLAMA_BIN bash serve-llama-parallel.sh

Output — what you should see

==> llama-server, 20 parallel slot(s)
version: x.x.x (build xxxxx, commit xxxxxxx)
built with ... for ...
    model             /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
    served as         local-chat
    listening on      http://127.0.0.1:8080/v1
    slots             20
    context per slot  4096
    total context     81920   (--ctx-size is the total across slots)
    KV cache type     f16 (keys and values)
    host prompt cache 0 MiB
    pid written to ./llama-server.pid, log copied to ./llama-server.log
... load log ...
x.xx.xxx.xxx I srv  llama_server: listening on http://127.0.0.1:8080

Wait for the listening on line, then pull the allocation out of the log copy in terminal 2:

RunnableAll tracks

terminal 2: what it allocated

cd ~/llm-course
source lab.env
grep -E "offloaded|model buffer size|KV buffer size|llama_kv_cache: size|compute buffer size|n_slots" llama-server.log
curl -s http://127.0.0.1:8080/slots | python3 -c 'import json,sys; s=json.load(sys.stdin); print(len(s), "slots, n_ctx per slot:", sorted({x["n_ctx"] for x in s}))'

Output — what you should see

x.xx.xxx.xxx I load_tensors: offloaded 37/37 layers to GPU
x.xx.xxx.xxx I load_tensors:        CUDA0 model buffer size =  xxxx.xx MiB
x.xx.xxx.xxx I load_tensors:   CPU_Mapped model buffer size =   xxx.xx MiB
x.xx.xxx.xxx I llama_kv_cache:      CUDA0 KV buffer size = 11520.00 MiB
x.xx.xxx.xxx I llama_kv_cache: size = 11520.00 MiB (  4096 cells,  36 layers, 20/20 seqs), K (f16): 5760.00 MiB, V (f16): 5760.00 MiB
x.xx.xxx.xxx I sched_reserve:      CUDA0 compute buffer size =   xxx.xx MiB
x.xx.xxx.xxx I srv    load_model: initializing, n_slots = 20, n_ctx_slot = 4096, kv_unified = 'false'
20 slots, n_ctx per slot: [4096]

The device name is CUDA0 on Tracks S and N, Vulkan0 on X and MTL0 on M. On an eight-bit tier the KV lines read 6120.00 MiB (or 3060.00 MiB with 10 slots), as Task 1 predicted. A KV size that differs from your prediction means the configuration is not what you think: check n_ctx_slot and the cache type before measuring anything. If layers read 0/37, stop; everything after this would measure the CPU (Troubleshooting).

Record: LLAMA_VERSION, offloaded layers, both model buffers, the KV size line, the compute buffer, and n_slots/n_ctx_slot.

3. Prove the endpoint before loading it

RunnableAll tracks

terminal 2: one request, by hand

curl -s http://127.0.0.1:8080/v1/models | python3 -c 'import json,sys; d=json.load(sys.stdin)["data"][0]; print(d["id"], "n_ctx", d["meta"]["n_ctx"])'
curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "local-chat", "messages": [{"role": "user", "content": "Reply with one word: ready"}], "max_tokens": 8}'

Output — what you should see

local-chat n_ctx 4096
{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"","reasoning_content":"Okay, the user wants a"}}],"created":xxxxxxxxxx,"model":"local-chat",...,"usage":{"completion_tokens":8,"prompt_tokens":14,"total_tokens":22,"prompt_tokens_details":{"cached_tokens":0}},...,"timings":{"cache_n":0,"prompt_n":14,...}}

That is the shape the v0.4.0 server returned when this page’s scripts were tested against a CPU build with Qwen3-0.6B; the words and counts will differ. Read it closely, because it explains two choices in the load generator. Qwen3 thinks before it answers: all eight tokens went into reasoning_content and content is empty. A client that times only content would report no first token at all, so load-test.py counts reasoning text too. And with a 128-token budget almost every request stops on length while still thinking, which keeps output length identical across requests and engines; the generator records the finish reasons so you can confirm it.

A generator pointed at a broken endpoint produces a tidy table of failures; this check takes ten seconds.

Record: the n_ctx the server reports and the finish_reason.

4. The load generator, and the baseline sweep

RunnableAll tracks

load-test.py

#!/usr/bin/env python3
"""Concurrency load generator for an OpenAI-compatible chat completions endpoint.

Purpose: drive a served model at one or more concurrency levels and report the numbers
    Part 9 cares about - output token throughput, request throughput, goodput against a
    stated objective, time to first token and time per output token - with percentiles
    rather than means, plus the prompt and cached token counts the server reports.
Platform: all (spark, strix, mac, nvidia). Pure Python standard library, 3.9 or later:
    no pip install.
Minimum memory: 16 GB on the machine running the server. The generator itself is tiny
    and may run on a second machine on the same network.
Assumes: a server already listening and answering POST /v1/chat/completions with
    streaming, reachable over plain HTTP (this is a localhost or LAN lab tool; it does
    not speak TLS). The model name must be the one the server reports at /v1/models.
    Results are appended to the lab notebook as one JSON line per concurrency level.

Usage:
    python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
        --concurrency 1,5,10,20 --requests 40 --min-rounds 4 --max-tokens 128 \
        --prompt-set mixed --label llama-baseline --labbook labbook.md

    python3 load-test.py --base-url http://127.0.0.1:8000/v1 --model local-chat \
        --concurrency 1,10 --requests 40 --max-tokens 64 \
        --prompt-set shared-prefix --label vllm-prefix-shared --labbook labbook.md

    An API key, if the server needs one, is read from an environment variable named by
    --api-key-env. No key is ever written to this file or to the notebook.

Method, stated once:
    * Closed loop. Each level starts `concurrency` workers that each send a request, wait
      for it to finish and send the next, so that number of requests is in flight until
      the queue drains. Requests per level = max(--requests, --min-rounds x concurrency).
    * Before the first level, --warmup requests (default 1) with a short prompt that shares
      nothing with the prompt sets are sent and discarded.
    * Time to first token (TTFT) is measured to the first streamed delta carrying any text:
      answer text ("content") or reasoning text ("reasoning_content" or "reasoning"), so a
      model that thinks first is timed the same way on every engine.
    * Output tokens come from the server's usage block (requested with
      stream_options.include_usage); where a server sends none, one text delta is counted
      as one token and the notebook line says "deltas" instead of "usage".
    * Time per output token (TPOT) for one request = (end - TTFT) / (output tokens - 1),
      the same definition vllm bench serve uses. Percentiles are nearest-rank.
    * Output tokens per second = all output tokens at the level / wall-clock of the level.
    * Goodput = requests per second that completed with TTFT <= --slo-ttft and
      TPOT <= --slo-tpot.
    * Mean in flight = sum of request durations / wall-clock: close to `concurrency` when
      the loop held the load, lower when the client or the drain at the end did not.
    * Cached tokens: usage.prompt_tokens_details.cached_tokens (vLLM with
      --enable-prompt-tokens-details, llama-server, mlx_lm.server), or llama-server's
      timings.cache_n where that is the only count sent.
"""

from __future__ import annotations

import argparse
import asyncio
import json
import math
import os
import secrets
import statistics
import sys
import time
from collections import Counter
from urllib.parse import urlsplit

LAB_ID = "part-09/lab-serve-twenty-concurrent-users"
TEXT_FIELDS = ("content", "reasoning_content", "reasoning")

# --------------------------------------------------------------------------- prompts

TOOLS = [
    ("read_metrics", "Read the Prometheus metrics endpoint of one inference server and return the series whose names match a pattern.", {"server": "string: the server name from the inventory", "pattern": "string: a regular expression matched against metric names"}),
    ("tail_log", "Return the last lines of a server's log, optionally filtered by level.", {"server": "string", "lines": "integer: how many lines, at most 500", "level": "string: one of debug, info, warning, error"}),
    ("restart_server", "Stop and start one inference server with its recorded configuration. Refuses while requests are running unless force is true.", {"server": "string", "force": "boolean"}),
    ("set_option", "Change one recorded serving option for the next restart and return the previous value.", {"server": "string", "option": "string: for example max-num-seqs or ctx-size", "value": "string"}),
    ("memory_report", "Report total, used and free accelerator memory, and the part of it held by the key-value cache.", {"server": "string"}),
    ("list_models", "List the models a server reports at /v1/models, with their context length.", {"server": "string"}),
    ("run_load_test", "Start a load test against a server and return the path of the results once it finishes.", {"server": "string", "concurrency": "array of integers", "requests": "integer", "max_tokens": "integer"}),
    ("compare_runs", "Compare two recorded load-test runs level by level and return the differences in throughput and latency.", {"first_label": "string", "second_label": "string"}),
    ("disk_usage", "Report the disk space used by the model directory and by each model file in it.", {"path": "string"}),
    ("checksum_file", "Compute the SHA-256 of a model file and compare it with the checksum recorded beside it.", {"path": "string"}),
    ("gpu_temperature", "Return the accelerator temperature, power draw and any throttling reasons reported by the driver.", {"device": "integer"}),
    ("open_ticket", "Record an operational issue with a title, a severity and the evidence gathered so far.", {"title": "string", "severity": "string: one of low, medium, high", "evidence": "string"}),
    ("slot_status", "Return each parallel slot of a llama-server instance with its context size, whether it is processing and how many tokens it holds.", {"server": "string"}),
    ("queue_depth", "Return how many requests are running and how many are waiting on a server, sampled once per second for a number of seconds.", {"server": "string", "seconds": "integer: at most 60"}),
    ("prefix_cache_stats", "Return prefix cache queries, hits and the hit rate since the server started.", {"server": "string"}),
    ("download_model", "Download a model file from a named repository into the model directory and verify its checksum.", {"repository": "string", "file": "string"}),
    ("delete_model_file", "Delete one model file after confirming that no running server has it loaded.", {"path": "string", "confirm": "boolean"}),
    ("network_throughput", "Measure throughput between this machine and another on the local network for a number of seconds.", {"peer": "string: a host name from the inventory", "seconds": "integer"}),
    ("schedule_benchmark", "Schedule a benchmark to run at a quiet time and record it in the lab notebook when it finishes.", {"when": "string: an ISO 8601 time", "benchmark": "string"}),
    ("summarise_notebook", "Return the notebook lines recorded for one lab, grouped by label, as a table.", {"lab": "string", "label": "string"}),
]

RULES = [
    "Answer in at most four sentences unless the operator asks for a table.",
    "Quote numbers only from tool results, never from memory, and name the tool that produced them.",
    "Prefer the smallest change that tests one hypothesis, and say what result would refute it.",
    "Never restart a server that has requests running without asking the operator first.",
    "When memory is short, reduce concurrency or context before reducing model precision.",
    "Report latency as a median and a ninetieth percentile, never as a mean alone.",
]

# About 1,365 tokens under the Qwen3 tokeniser: a system-prompt-and-tool-schema preamble
# of the kind an agent sends on every step.
SHARED_PREAMBLE = (
    "You are the operations assistant for a small local inference service: one open-weight "
    "model served over an OpenAI-compatible interface on one machine. You can call the tools "
    "described below by answering with a JSON object naming the tool and its arguments.\n\n"
    "Rules:\n" + "\n".join(f"{i + 1}. {rule}" for i, rule in enumerate(RULES))
    + "\n\nTools:\n"
    + json.dumps([{"name": n, "description": d, "parameters": p} for n, d, p in TOOLS], indent=2)
)

SHARED_QUESTIONS = [
    "What happens to the key-value cache when I double the context length?",
    "Why is prompt processing so much faster than generation?",
    "Which measurement should I quote to a user who says the service feels slow?",
    "What does it mean when the number of waiting requests stays above zero?",
    "Why does a second concurrent request cost less than twice as much?",
    "How do I decide between more concurrent slots and a longer context?",
    "What is the first thing to check when memory runs out at startup?",
    "Why does the mean latency mislead me about a serving system?",
]

MIXED_PROMPTS = [
    "Explain in three sentences why decode speed is limited by memory bandwidth.",
    "Write a short shell function that reports how much disk a directory uses.",
    "Summarise the difference between tensor parallel and pipeline parallel.",
    "Give three reasons a language model server might refuse to start.",
    "What is a KV cache, and why does it grow with the conversation?",
    "Write a Python function that returns the median of a list, without imports.",
    "List four things that belong in a benchmark result besides the number itself.",
    "Explain quantisation to somebody who knows what a floating point number is.",
    "Describe what continuous batching does, in plain language.",
    "Name three failure modes of retrieval-augmented generation.",
    "Write a regular expression that matches an ISO date, and explain it.",
    "Why might a smaller model that fits beat a larger one that does not?",
    "Give a checklist for exposing a local API endpoint to a home network.",
    "What is the difference between a context window and a context length?",
    "Explain why two runs with the same seed can still produce different text.",
    "Write a short bash loop that retries a command three times with a pause.",
    "What does a high standard deviation in a benchmark run usually indicate?",
    "Describe the trade-off between batch size and per-user latency.",
    "Explain what a chat template does and what goes wrong without one.",
    "Give three signs that a machine is thermally limited during a benchmark.",
    "What is speculative decoding, and when does it not help?",
    "Explain the difference between total and active parameters.",
    "Write a JSON object describing a machine with a chip, memory and an operating system.",
    "Why is goodput a more useful headline number than raw throughput?",
]

WARMUP_PROMPT = "Reply with one word: ready"


def build_prompts(prompt_set: str, count: int) -> list[str]:
    """Returns `count` prompts, cycling the chosen set.

    mixed: 24 short, different questions, each behind a random tag, so that a prompt
        repeated later in the run (or in an earlier run) is not a cache hit.
    shared-prefix: the preamble, then one of eight questions; everything before the
        question is byte-identical across requests, so a prefix cache can reuse it.
    unique-prefix: the same text with a random tag as its first line, different for every
        request and every run, so nothing after the chat template's opening tokens can be
        reused. It is the control for shared-prefix: same length, no reusable prefix.
    """
    if prompt_set == "shared-prefix":
        pool = [f"{SHARED_PREAMBLE}\n\nQuestion: {q}" for q in SHARED_QUESTIONS]
        return [pool[i % len(pool)] for i in range(count)]
    if prompt_set == "unique-prefix":
        return [
            f"Request {secrets.token_hex(8)}.\n{SHARED_PREAMBLE}\n\n"
            f"Question: {SHARED_QUESTIONS[i % len(SHARED_QUESTIONS)]}"
            for i in range(count)
        ]
    if prompt_set == "mixed":
        return [
            f"Request {secrets.token_hex(8)}. {MIXED_PROMPTS[i % len(MIXED_PROMPTS)]}"
            for i in range(count)
        ]
    raise ValueError(f"unknown prompt set: {prompt_set}")


# ------------------------------------------------------------------- minimal HTTP/1.1


class RequestFailed(Exception):
    """Raised when a request did not complete with a usable stream."""


async def _read_headers(reader: asyncio.StreamReader) -> tuple[int, bool]:
    """Reads the status line and headers. Returns (status, chunked)."""
    status_line = await reader.readline()
    if not status_line:
        raise RequestFailed("server closed the connection without a response")
    parts = status_line.decode("latin-1", "replace").split()
    if len(parts) < 2 or not parts[1].isdigit():
        raise RequestFailed(f"unparseable status line: {status_line!r}")
    status = int(parts[1])
    chunked = False
    while True:
        line = await reader.readline()
        if line in (b"\r\n", b"\n", b""):
            break
        lowered = line.lower()
        if lowered.startswith(b"transfer-encoding:") and b"chunked" in lowered:
            chunked = True
    return status, chunked


async def _body_lines(reader: asyncio.StreamReader, chunked: bool):
    """Yields body lines, decoding chunked transfer encoding where it is used."""
    buf = b""
    if chunked:
        while True:
            size_line = await reader.readline()
            if not size_line:
                break
            head = size_line.strip().split(b";")[0]
            if not head:
                continue
            try:
                size = int(head, 16)
            except ValueError:
                break
            if size == 0:
                await reader.readline()
                break
            buf += await reader.readexactly(size)
            await reader.readexactly(2)
            while b"\n" in buf:
                line, buf = buf.split(b"\n", 1)
                yield line.rstrip(b"\r")
    else:
        while True:
            chunk = await reader.read(65536)
            if not chunk:
                break
            buf += chunk
            while b"\n" in buf:
                line, buf = buf.split(b"\n", 1)
                yield line.rstrip(b"\r")
    if buf.strip():
        yield buf.rstrip(b"\r")


def _cached_from(event: dict):
    """The cached prompt token count in a streamed event, if the server sent one."""
    usage = event.get("usage") or {}
    details = usage.get("prompt_tokens_details") or {}
    if isinstance(details, dict) and details.get("cached_tokens") is not None:
        return int(details["cached_tokens"])
    timings = event.get("timings") or {}
    if timings.get("cache_n") is not None:
        return int(timings["cache_n"])
    return None


async def stream_completion(cfg: dict, prompt: str, max_tokens: int) -> dict:
    """Sends one streaming chat completion and times it.

    Returns a dict with ok, ttft, total, tokens, token_source, prompt_tokens, cached,
    finish and (on failure) error.
    """
    body = json.dumps(
        {
            "model": cfg["model"],
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "temperature": cfg["temperature"],
            "stream": True,
            "stream_options": {"include_usage": True},
        }
    ).encode("utf-8")

    headers = [
        f"POST {cfg['path']} HTTP/1.1",
        f"Host: {cfg['host']}:{cfg['port']}",
        "Content-Type: application/json",
        "Accept: text/event-stream",
        f"Content-Length: {len(body)}",
        "Connection: close",
    ]
    if cfg["api_key"]:
        headers.append(f"Authorization: Bearer {cfg['api_key']}")

    started = time.perf_counter()
    ttft = None
    deltas = 0
    usage_tokens = None
    prompt_tokens = None
    cached = None
    finish = None
    writer = None
    try:
        reader, writer = await asyncio.wait_for(
            asyncio.open_connection(cfg["host"], cfg["port"]), timeout=cfg["timeout"]
        )
        writer.write(("\r\n".join(headers) + "\r\n\r\n").encode("latin-1") + body)
        await writer.drain()

        status, chunked = await asyncio.wait_for(_read_headers(reader), timeout=cfg["timeout"])
        if status != 200:
            detail = b""
            try:
                detail = await asyncio.wait_for(reader.read(300), timeout=5)
            except (OSError, asyncio.TimeoutError):
                pass
            raise RequestFailed(f"HTTP {status} {detail.decode('utf-8', 'replace').strip()}")

        async for raw in _body_lines(reader, chunked):
            if not raw.startswith(b"data:"):
                continue
            payload = raw[5:].strip()
            if payload in (b"[DONE]", b""):
                continue
            try:
                event = json.loads(payload)
            except json.JSONDecodeError:
                continue
            if event.get("error"):
                raise RequestFailed(f"server error in stream: {str(event['error'])[:200]}")
            for choice in event.get("choices") or []:
                delta = choice.get("delta") or {}
                if any(delta.get(field) for field in TEXT_FIELDS) or delta.get("tool_calls"):
                    deltas += 1
                    if ttft is None:
                        ttft = time.perf_counter() - started
                if choice.get("finish_reason"):
                    finish = choice["finish_reason"]
            usage = event.get("usage")
            if usage:
                if usage.get("completion_tokens") is not None:
                    usage_tokens = int(usage["completion_tokens"])
                if usage.get("prompt_tokens") is not None:
                    prompt_tokens = int(usage["prompt_tokens"])
            found = _cached_from(event)
            if found is not None:
                cached = found
    except (OSError, asyncio.TimeoutError, asyncio.IncompleteReadError, RequestFailed) as exc:
        return {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
    finally:
        if writer is not None:
            writer.close()
            try:
                await writer.wait_closed()
            except OSError:
                pass

    total = time.perf_counter() - started
    if deltas == 0 or ttft is None:
        return {"ok": False, "error": "stream produced no content deltas"}
    tokens = usage_tokens if usage_tokens else deltas
    return {
        "ok": True,
        "ttft": ttft,
        "total": total,
        "tokens": tokens,
        "token_source": "usage" if usage_tokens else "deltas",
        "prompt_tokens": prompt_tokens,
        "cached": cached,
        "finish": finish or "none",
    }


# ------------------------------------------------------------------------ measurement


def percentile(values: list[float], p: float) -> float:
    """Nearest-rank percentile: the smallest value with at least p% of values at or below it."""
    if not values:
        return 0.0
    ordered = sorted(values)
    rank = max(1, min(len(ordered), math.ceil(p / 100.0 * len(ordered))))
    return ordered[rank - 1]


def _summary(values: list[float], digits: int) -> dict:
    return {
        "p50": round(percentile(values, 50), digits),
        "p90": round(percentile(values, 90), digits),
        "p99": round(percentile(values, 99), digits),
        "mean": round(statistics.fmean(values), digits) if values else 0.0,
    }


def _token_source(sources: Counter):
    """"usage" or "deltas" when every request agrees, the counts when they do not, "none" if empty."""
    if not sources:
        return "none"
    if len(sources) == 1:
        return next(iter(sources))
    return dict(sources)


def _mean_or_none(values: list) -> float | None:
    present = [v for v in values if v is not None]
    return round(statistics.fmean(present), 1) if present else None


async def warm_up(cfg: dict, count: int) -> None:
    """Sends `count` short requests one at a time and discards their timings."""
    for _ in range(count):
        await stream_completion(cfg, WARMUP_PROMPT, 8)


async def run_level(cfg: dict, concurrency: int, prompts: list[str]) -> dict:
    """Runs every prompt through `concurrency` workers and aggregates the results."""
    queue: asyncio.Queue = asyncio.Queue()
    for prompt in prompts:
        queue.put_nowait(prompt)
    results: list[dict] = []

    async def worker() -> None:
        while True:
            try:
                prompt = queue.get_nowait()
            except asyncio.QueueEmpty:
                return
            results.append(await stream_completion(cfg, prompt, cfg["max_tokens"]))
            queue.task_done()

    started = time.perf_counter()
    await asyncio.gather(*[worker() for _ in range(concurrency)])
    wall = time.perf_counter() - started

    ok = [r for r in results if r["ok"]]
    failed = [r for r in results if not r["ok"]]
    ttfts = [r["ttft"] for r in ok]
    tpots = {id(r): (r["total"] - r["ttft"]) / (r["tokens"] - 1) for r in ok if r["tokens"] > 1}
    out_tokens = sum(r["tokens"] for r in ok)
    good = [
        r for r in ok
        if r["ttft"] <= cfg["slo_ttft"] and tpots.get(id(r), 0.0) <= cfg["slo_tpot"]
    ]
    sources = Counter(r["token_source"] for r in ok)

    return {
        "concurrency": concurrency,
        "requests": len(results),
        "completed": len(ok),
        "failed": len(failed),
        "wall_s": round(wall, 3),
        "output_tokens": out_tokens,
        "output_tokens_per_s": round(out_tokens / wall, 2) if wall > 0 else 0.0,
        "requests_per_s": round(len(ok) / wall, 3) if wall > 0 else 0.0,
        "goodput_requests_per_s": round(len(good) / wall, 3) if wall > 0 else 0.0,
        "within_slo": len(good),
        "mean_in_flight": round(sum(r["total"] for r in ok) / wall, 2) if wall > 0 else 0.0,
        "ttft_s": _summary(ttfts, 4),
        "tpot_s": _summary(list(tpots.values()), 5),
        "prompt_tokens_mean": _mean_or_none([r["prompt_tokens"] for r in ok]),
        "cached_tokens_mean": _mean_or_none([r["cached"] for r in ok]),
        "token_counts": _token_source(sources),
        "finish_reasons": dict(Counter(r["finish"] for r in ok)),
        "first_error": failed[0]["error"] if failed else None,
    }


def print_level(row: dict) -> None:
    """Prints one concurrency level as two aligned lines."""
    print(
        f"  c={row['concurrency']:>3}  "
        f"ok {row['completed']:>3}/{row['requests']:<3} "
        f"fail {row['failed']:<3} "
        f"wall {row['wall_s']:>7.2f}s  "
        f"out/s {row['output_tokens_per_s']:>8.2f}  "
        f"req/s {row['requests_per_s']:>6.3f}  "
        f"TTFT p50 {row['ttft_s']['p50']:.3f} p90 {row['ttft_s']['p90']:.3f}  "
        f"TPOT p50 {row['tpot_s']['p50']:.4f} p90 {row['tpot_s']['p90']:.4f}"
    )
    cached = row["cached_tokens_mean"]
    prompt = row["prompt_tokens_mean"]
    print(
        f"         goodput {row['goodput_requests_per_s']:.3f} req/s "
        f"({row['within_slo']}/{row['completed']} within SLO)  "
        f"in flight {row['mean_in_flight']:.1f}  "
        f"prompt {prompt if prompt is not None else '?'} tok, "
        f"cached {cached if cached is not None else '?'}  "
        f"tokens from {row['token_counts']}  finish {row['finish_reasons']}"
    )
    if row["first_error"]:
        print(f"         first error: {row['first_error']}")


def append_labbook(path: str, record: dict) -> None:
    """Appends one JSON line, in the course notebook format."""
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base-url", default="http://127.0.0.1:8000/v1",
                        help="OpenAI-compatible base URL, ending in /v1")
    parser.add_argument("--model", required=True, help="model name the server reports")
    parser.add_argument("--concurrency", default="1,5,10,20",
                        help="comma-separated concurrency levels to run in order")
    parser.add_argument("--requests", type=int, default=40,
                        help="requests sent at each concurrency level (at least)")
    parser.add_argument("--min-rounds", type=int, default=0,
                        help="send at least this many requests per worker at each level")
    parser.add_argument("--max-tokens", type=int, default=128)
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--timeout", type=float, default=300.0)
    parser.add_argument("--prompt-set", default="mixed",
                        choices=["mixed", "shared-prefix", "unique-prefix"])
    parser.add_argument("--warmup", type=int, default=1,
                        help="discarded short requests sent before the first level")
    parser.add_argument("--slo-ttft", type=float, default=2.0,
                        help="goodput objective for time to first token, in seconds")
    parser.add_argument("--slo-tpot", type=float, default=0.1,
                        help="goodput objective for time per output token, in seconds")
    parser.add_argument("--label", default="run", help="tag written into the notebook line")
    parser.add_argument("--engine", default="unknown",
                        help="engine name recorded in the notebook line, e.g. vllm")
    parser.add_argument("--engine-version", default="unknown",
                        help="engine version recorded in the notebook line")
    parser.add_argument("--host-desc", default="unknown",
                        help="machine description recorded in the notebook line")
    parser.add_argument("--quant", default="unknown",
                        help="weight format recorded in the notebook line, e.g. Q4_K_M or AWQ")
    parser.add_argument("--context-length", type=int, default=0,
                        help="context length per sequence the server was started with")
    parser.add_argument("--server-slots", type=int, default=0,
                        help="sequences the server may run at once (slots or max-num-seqs)")
    parser.add_argument("--api-key-env", default="LOADTEST_API_KEY",
                        help="environment variable holding the API key, if the server needs one")
    parser.add_argument("--labbook", default="labbook.md",
                        help="file to append one JSON line per concurrency level to")
    return parser.parse_args(argv)


def main(argv: list[str]) -> int:
    args = parse_args(argv)

    parts = urlsplit(args.base_url)
    if parts.scheme != "http":
        print("This tool speaks plain HTTP only; point it at a localhost or LAN endpoint.",
              file=sys.stderr)
        return 2
    if not parts.hostname:
        print(f"Could not read a host from --base-url {args.base_url!r}.", file=sys.stderr)
        return 2
    if args.requests < 1 or args.max_tokens < 2:
        print("--requests must be at least 1 and --max-tokens at least 2.", file=sys.stderr)
        return 2

    cfg = {
        "host": parts.hostname,
        "port": parts.port or 80,
        "path": (parts.path.rstrip("/") or "/v1") + "/chat/completions",
        "model": args.model,
        "max_tokens": args.max_tokens,
        "temperature": args.temperature,
        "timeout": args.timeout,
        "slo_ttft": args.slo_ttft,
        "slo_tpot": args.slo_tpot,
        "api_key": os.environ.get(args.api_key_env, ""),
    }

    try:
        levels = [int(x) for x in args.concurrency.split(",") if x.strip()]
    except ValueError:
        print(f"--concurrency must be a comma-separated list of integers, got {args.concurrency!r}",
              file=sys.stderr)
        return 2
    if not levels or min(levels) < 1:
        print("--concurrency listed no levels, or a level below 1.", file=sys.stderr)
        return 2

    print(f"==> {args.label}: {args.model} at {args.base_url}")
    print(f"    prompt set {args.prompt_set}, at least {args.requests} request(s) per level"
          f"{f' and {args.min_rounds} per worker' if args.min_rounds else ''}, "
          f"max_tokens {args.max_tokens}, temperature {args.temperature}")
    print(f"    SLO for goodput: TTFT <= {args.slo_ttft} s and TPOT <= {args.slo_tpot} s; "
          f"{args.warmup} warm-up request(s)")

    if args.warmup > 0:
        asyncio.run(warm_up(cfg, args.warmup))

    exit_code = 0
    for level in levels:
        count = max(args.requests, args.min_rounds * level)
        prompts = build_prompts(args.prompt_set, count)
        row = asyncio.run(run_level(cfg, level, prompts))
        print_level(row)
        if row["completed"] == 0:
            exit_code = 1
        record = {
            "lab": LAB_ID,
            "label": args.label,
            "engine": args.engine,
            "engine_version": args.engine_version,
            "host": args.host_desc,
            "quant": args.quant,
            "context_length": args.context_length or None,
            "server_slots": args.server_slots or None,
            "model": args.model,
            "base_url": args.base_url,
            "prompt_set": args.prompt_set,
            "max_tokens": args.max_tokens,
            "temperature": args.temperature,
            "slo_ttft_s": args.slo_ttft,
            "slo_tpot_s": args.slo_tpot,
            "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        }
        record.update(row)
        append_labbook(args.labbook, record)

    print(f"    appended {len(levels)} line(s) to {args.labbook}")
    if exit_code:
        print("    at least one level completed no requests; see the error above.",
              file=sys.stderr)
    return exit_code


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download load-test.py619 lines

The docstring states the method; here is what each number means, with one request worked through.

Measure	Definition in load-test.py	Worked example (illustrative inputs, not a measurement)
TTFT	time from sending the request to the first streamed delta carrying any text	first reasoning token after 0.40 s
TPOT	(end − TTFT) ÷ (output tokens − 1), the definition vllm bench serve uses	ended at 7.40 s with 128 tokens: 7.00 ÷ 127 = 0.0551 s
Output tokens	usage.completion_tokens from the final streamed chunk, requested with stream_options.include_usage; one per text delta only if a server sends no usage	128
Output tokens/s	all output tokens at the level ÷ the level’s wall-clock	40 requests × 128 tokens in 60 s = 85.3
Goodput	requests per second with TTFT ≤ --slo-ttft and TPOT ≤ --slo-tpot (defaults 2.0 s and 0.1 s)	36 of 40 within both, in 60 s: 0.600
p50, p90	nearest rank: the smallest value with at least that share of values at or below it	p90 of 40 values is the 36th smallest
Mean in flight	sum of request durations ÷ wall-clock	20 workers, 80 requests, drain at the end: somewhat under 20
Cached tokens	usage.prompt_tokens_details.cached_tokens, or llama-server’s timings.cache_n	about 4 on a tagged prompt: the chat template’s opening tokens

The objective is a choice, and the defaults are stated so you can change them: a first token within two seconds, and a pace of at least ten tokens per second, faster than anyone reads.

Concurrency is held by workers, not by an arrival rate. Concurrency 20 means twenty requests are in flight until the queue runs out: the saturated case, deliberately, because it finds the limit. It is not twenty users, who spend most of their time reading. vllm bench serve --request-rate models arrivals, and Task 8 uses its --max-concurrency as a cross-check. Each worker’s first requests start together, so the first round at a high level is also a burst of simultaneous prefills; --min-rounds 4 sends at least four requests per worker so that burst is a minority of the sample. The mixed prompts start with a random tag, so a prompt repeated across levels is not a cache hit.

RunnableAll tracks

terminal 2: the baseline sweep

cd ~/llm-course
source lab.env
python3 load-test.py \
    --base-url http://127.0.0.1:8080/v1 \
    --model local-chat \
    --concurrency 1,5,10,20 \
    --requests 40 \
    --min-rounds 4 \
    --max-tokens 128 \
    --prompt-set mixed \
    --engine llama.cpp \
    --engine-version "$LLAMA_VERSION" \
    --host-desc "$HOST_DESC" \
    --quant Q4_K_M \
    --context-length 4096 \
    --server-slots "$SLOTS" \
    --label llama-baseline \
    --labbook labbook.md

Output — what you should see

==> llama-baseline: local-chat at http://127.0.0.1:8080/v1
    prompt set mixed, at least 40 request(s) per level and 4 per worker, max_tokens 128, temperature 0.0
    SLO for goodput: TTFT <= 2.0 s and TPOT <= 0.1 s; 1 warm-up request(s)
  c=  1  ok  40/40  fail 0   wall   xx.xxs  out/s    xx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (40/40 within SLO)  in flight 1.0  prompt 3x.x tok, cached x.x  tokens from usage  finish {'length': 40}
  c=  5  ok  40/40  fail 0   wall   xx.xxs  out/s   xxx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (xx/40 within SLO)  in flight x.x  prompt 3x.x tok, cached x.x  tokens from usage  finish {'length': 40}
  c= 10  ok  40/40  fail 0   wall   xx.xxs  out/s   xxx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (xx/40 within SLO)  in flight x.x  prompt 3x.x tok, cached x.x  tokens from usage  finish {'length': 40}
  c= 20  ok  80/80  fail 0   wall   xx.xxs  out/s   xxx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (xx/80 within SLO)  in flight xx.x  prompt 3x.x tok, cached x.x  tokens from usage  finish {'length': 80}
    appended 4 line(s) to labbook.md

Check four things before trusting the numbers. fail 0 on every level. tokens from usage, so the token counts are the server’s. finish {'length': N}: every request generated the full 128 tokens; if some stopped early, their shorter answers inflate requests per second. And in flight close to the level: a figure well below it at c=20 (below about 15) means the client could not keep the load up, which is the “client is the bottleneck” row in Troubleshooting. The sweep takes a few minutes; while it runs, watch the server from a third terminal:

RunnableAll tracks

terminal 3: the server's view while the sweep runs (Ctrl-C to stop)

while sleep 2; do
    curl -s http://127.0.0.1:8080/metrics | grep -E '^llamacpp:(requests_processing|requests_deferred|n_busy_slots_per_decode)'
    echo "--"
done

Output — what you should see

llamacpp:requests_processing xx
llamacpp:requests_deferred 0
llamacpp:n_busy_slots_per_decode xx.xxxx
--

requests_processing should track the level, requests_deferred should stay at 0 (a nonzero value means more requests than slots), and n_busy_slots_per_decode is the batch size llama-server actually achieved per decode call.

Record: nothing by hand; the four JSON lines are the record. Note in prose anything odd you saw in terminal 3.

5. Read the curve

RunnableAll tracks

summarise-load.py

#!/usr/bin/env python3
"""Turn Part 9 load-test notebook lines into curves, comparisons and a pass/fail check.

Purpose: read the JSON lines load-test.py appended to the lab notebook and print, for one
    label, the throughput and latency curve with the derived numbers the lab asks for
    (scaling efficiency, the knee, a straight-line fit of time per output token against
    concurrency, and the operating point under the recorded objective); for two labels,
    a level-by-level comparison; and for the whole notebook, the lab's validation rules.
Platform: all (spark, strix, mac, nvidia). Python 3.9 or later, standard library only.
Minimum memory: none worth stating; it reads a text file.
Assumes: a notebook (default labbook.md) holding lines written by load-test.py, whose
    "lab" field is part-09/lab-serve-twenty-concurrent-users. Other lines, prose and
    other labs' records are skipped. When a label was run more than once, the most
    recent line per concurrency level is used.

Usage:
    python3 summarise-load.py --list
    python3 summarise-load.py --label llama-baseline
    python3 summarise-load.py --label llama-baseline --markdown --record
    python3 summarise-load.py --compare llama-baseline vllm-baseline
    python3 summarise-load.py --check
"""

from __future__ import annotations

import argparse
import json
import sys
import time

LAB_ID = "part-09/lab-serve-twenty-concurrent-users"
BASELINE_LEVELS = {1, 5, 10, 20}


def load_records(path: str) -> list[dict]:
    """Every load-test row for this lab, in file order."""
    rows = []
    try:
        with open(path, encoding="utf-8") as handle:
            for line in handle:
                line = line.strip()
                if not line.startswith("{"):
                    continue
                try:
                    obj = json.loads(line)
                except json.JSONDecodeError:
                    continue
                if obj.get("lab") == LAB_ID and "concurrency" in obj and "ttft_s" in obj:
                    rows.append(obj)
    except FileNotFoundError:
        sys.exit(f"summarise-load: {path} not found; run load-test.py first or pass --labbook")
    return rows


def by_label(rows: list[dict]) -> dict[str, dict[int, dict]]:
    """label -> concurrency -> the most recent row."""
    out: dict[str, dict[int, dict]] = {}
    for row in rows:
        out.setdefault(str(row.get("label")), {})[int(row["concurrency"])] = row
    return out


def fit_line(xs: list[float], ys: list[float]):
    """Least-squares y = a + b x. Returns (a, b) or None with fewer than two distinct x."""
    if len(set(xs)) < 2:
        return None
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    sxx = sum((x - mx) ** 2 for x in xs)
    b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sxx
    return my - b * mx, b


def curve(levels: dict[int, dict]) -> dict:
    """Derived numbers for one label's curve."""
    cs = sorted(c for c, r in levels.items() if r.get("completed"))
    derived = {"levels": cs, "knee": None, "operating_point": None, "fit": None}
    if not cs:
        return derived
    first = levels[cs[0]]
    per_request = first["output_tokens_per_s"] / cs[0]
    derived["per_request_tokens_per_s"] = per_request
    for prev, cur in zip(cs, cs[1:]):
        gain = (levels[cur]["output_tokens_per_s"] - levels[prev]["output_tokens_per_s"]) / (cur - prev)
        if gain < 0.5 * per_request:
            derived["knee"] = [prev, cur]
            break
    for c in cs:
        r = levels[c]
        if (r["failed"] == 0 and r["ttft_s"]["p90"] <= r.get("slo_ttft_s", 2.0)
                and r["tpot_s"]["p90"] <= r.get("slo_tpot_s", 0.1)):
            derived["operating_point"] = c
    slots = first.get("server_slots")
    fit_cs = [c for c in cs if not slots or c <= slots]
    line = fit_line([float(c) for c in fit_cs], [levels[c]["tpot_s"]["p50"] for c in fit_cs])
    if line:
        a, b = line
        derived["fit"] = {"a_s": a, "b_s": b, "levels_used": fit_cs}
    return derived


def bar(value: float, top: float, width: int = 24) -> str:
    filled = int(round(width * value / top)) if top > 0 else 0
    return "#" * filled + "." * (width - filled)


def print_curve(label: str, levels: dict[int, dict], markdown: bool) -> dict:
    d = curve(levels)
    cs = d["levels"]
    if not cs:
        print(f"{label}: no level completed any request")
        return d
    first = levels[cs[0]]
    print(f"==> {label}: {first.get('engine')} {first.get('engine_version')}, model {first.get('model')}, "
          f"prompt set {first.get('prompt_set')}, max_tokens {first.get('max_tokens')}")
    print(f"    host {first.get('host', 'unknown')}, quant {first.get('quant', 'unknown')}, "
          f"context {first.get('context_length')}, server slots {first.get('server_slots')}, "
          f"SLO TTFT <= {first.get('slo_ttft_s')} s, TPOT <= {first.get('slo_tpot_s')} s")
    head = ["c", "ok", "out tok/s", "scaling", "req/s", "goodput", "TTFT p50", "TTFT p90",
            "TPOT p50", "TPOT p90", "in flight", "cached"]
    table = []
    for c in cs:
        r = levels[c]
        scaling = r["output_tokens_per_s"] / (c * d["per_request_tokens_per_s"])
        table.append([str(c), f"{r['completed']}/{r['requests']}", f"{r['output_tokens_per_s']:.1f}",
                      f"{scaling:.2f}", f"{r['requests_per_s']:.3f}",
                      f"{r.get('goodput_requests_per_s', 0.0):.3f}",
                      f"{r['ttft_s']['p50']:.3f}", f"{r['ttft_s']['p90']:.3f}",
                      f"{r['tpot_s']['p50']:.4f}", f"{r['tpot_s']['p90']:.4f}",
                      f"{r.get('mean_in_flight', 0.0):.1f}", str(r.get("cached_tokens_mean"))])
    if markdown:
        print("| " + " | ".join(head) + " |")
        print("|" + "---|" * len(head))
        for row in table:
            print("| " + " | ".join(row) + " |")
    else:
        widths = [max(len(h), *(len(row[i]) for row in table)) for i, h in enumerate(head)]
        print("    " + "  ".join(h.rjust(w) for h, w in zip(head, widths)))
        for row in table:
            print("    " + "  ".join(v.rjust(w) for v, w in zip(row, widths)))

    top_out = max(levels[c]["output_tokens_per_s"] for c in cs)
    top_tpot = max(levels[c]["tpot_s"]["p90"] for c in cs)
    print("\n    output tokens/s                     TPOT p90 (s)")
    for c in cs:
        r = levels[c]
        print(f"    c={c:>3} {bar(r['output_tokens_per_s'], top_out)} {r['output_tokens_per_s']:>8.1f}"
              f"   {bar(r['tpot_s']['p90'], top_tpot, 16)} {r['tpot_s']['p90']:.4f}")

    print()
    if d["knee"]:
        a, b = d["knee"]
        print(f"    knee: between c={a} and c={b}, each added request gained less than half "
              f"of one request's throughput at c={cs[0]}")
    else:
        print("    knee: not reached at the levels run; every step still gained at least half "
              "of one request's throughput")
    if d["fit"]:
        a_s, b_s = d["fit"]["a_s"], d["fit"]["b_s"]
        print(f"    TPOT p50 ~ {a_s * 1000:.2f} ms + {b_s * 1000:.3f} ms x c  "
              f"(fitted on c = {', '.join(map(str, d['fit']['levels_used']))})")
        for c in d["fit"]["levels_used"]:
            predicted = c / (a_s + b_s * c) if a_s + b_s * c > 0 else float("nan")
            print(f"      c={c:>3}: c / fitted TPOT = {predicted:8.1f} tok/s if every request were "
                  f"always decoding; measured {levels[c]['output_tokens_per_s']:8.1f} tok/s")
        if b_s > 0:
            slo = first.get("slo_tpot_s", 0.1)
            print(f"    ceiling as c grows: 1 / b = {1 / b_s:.1f} tok/s; "
                  f"TPOT p50 reaches {slo} s at c = {(slo - a_s) / b_s:.1f}")
        else:
            print("    b <= 0: TPOT did not rise with concurrency at these levels, so no ceiling is implied")
    if not d["fit"]:
        print("    TPOT fit: needs at least two levels at or below the server's slot count")
    print(f"    operating point (no failures, both p90s within the SLO): "
          f"{'c=' + str(d['operating_point']) if d['operating_point'] else 'none of the levels run'}")
    return d


def record_summary(path: str, label: str, levels: dict[int, dict], d: dict) -> None:
    first = levels[d["levels"][0]]
    line = {
        "lab": LAB_ID, "record": "curve-summary", "label": label,
        "engine": first.get("engine"), "engine_version": first.get("engine_version"),
        "host": first.get("host"), "quant": first.get("quant"), "model": first.get("model"),
        "context_length": first.get("context_length"), "server_slots": first.get("server_slots"),
        "levels": d["levels"], "knee_between": d["knee"], "operating_point": d["operating_point"],
        "peak_output_tokens_per_s": max(levels[c]["output_tokens_per_s"] for c in d["levels"]),
        "tpot_fit_ms": ({"a": round(d["fit"]["a_s"] * 1000, 3), "b": round(d["fit"]["b_s"] * 1000, 4)}
                        if d["fit"] else None),
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    }
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(line, sort_keys=True) + "\n")
    print(f"    appended a curve-summary line for {label} to {path}")


def print_compare(a: str, b: str, labels: dict[str, dict[int, dict]]) -> int:
    for name in (a, b):
        if name not in labels:
            print(f"No rows for label {name}. Labels present: {', '.join(sorted(labels)) or 'none'}",
                  file=sys.stderr)
            return 1
    shared = sorted(set(labels[a]) & set(labels[b]))
    if not shared:
        print("The two labels share no concurrency level.", file=sys.stderr)
        return 1
    ra, rb = labels[a][shared[0]], labels[b][shared[0]]
    for field in ("model", "max_tokens", "prompt_set", "context_length", "server_slots", "quant", "engine"):
        if ra.get(field) != rb.get(field):
            print(f"    note: {field} differs: {ra.get(field)} vs {rb.get(field)}")
    print(f"==> {a} (A) against {b} (B)")
    head = ["c", "out/s A", "out/s B", "B/A", "goodput A", "goodput B", "TTFT p50 A", "TTFT p50 B",
            "TTFT p90 A", "TTFT p90 B", "TPOT p50 A", "TPOT p50 B", "cached A", "cached B"]
    rows = []
    for c in shared:
        x, y = labels[a][c], labels[b][c]
        ratio = y["output_tokens_per_s"] / x["output_tokens_per_s"] if x["output_tokens_per_s"] else float("nan")
        rows.append([str(c), f"{x['output_tokens_per_s']:.1f}", f"{y['output_tokens_per_s']:.1f}", f"{ratio:.2f}",
                     f"{x.get('goodput_requests_per_s', 0.0):.3f}", f"{y.get('goodput_requests_per_s', 0.0):.3f}",
                     f"{x['ttft_s']['p50']:.3f}", f"{y['ttft_s']['p50']:.3f}",
                     f"{x['ttft_s']['p90']:.3f}", f"{y['ttft_s']['p90']:.3f}",
                     f"{x['tpot_s']['p50']:.4f}", f"{y['tpot_s']['p50']:.4f}",
                     str(x.get("cached_tokens_mean")), str(y.get("cached_tokens_mean"))])
    widths = [max(len(h), *(len(r[i]) for r in rows)) for i, h in enumerate(head)]
    print("    " + "  ".join(h.rjust(w) for h, w in zip(head, widths)))
    for r in rows:
        print("    " + "  ".join(v.rjust(w) for v, w in zip(r, widths)))
    return 0


def check(labels: dict[str, dict[int, dict]]) -> int:
    results = []

    def verdict(ok: bool, text: str, level: str = "FAIL") -> None:
        results.append("PASS" if ok else level)
        print(f"    {'PASS' if ok else level:<4}  {text}")

    print("==> checking the notebook against the lab's validation rules")
    sweeps = [name for name, lv in labels.items()
              if BASELINE_LEVELS <= set(lv) and all(lv[c]["completed"] for c in BASELINE_LEVELS)
              and lv[1].get("prompt_set") == "mixed"]
    verdict(bool(sweeps), f"a mixed-prompt sweep with every level of {sorted(BASELINE_LEVELS)} completed: "
            f"{', '.join(sweeps) or 'none'}")
    for name in sweeps:
        lv = labels[name]
        failed = sum(lv[c]["failed"] for c in lv)
        verdict(failed == 0, f"{name}: failed requests across all levels = {failed}", "WARN")
        verdict(all(lv[c].get("token_counts") == "usage" for c in lv),
                f"{name}: output tokens counted from the server's usage block", "WARN")
        missing = [k for k in ("host", "quant", "context_length", "server_slots")
                   if lv[1].get(k) in (None, "unknown")]
        verdict(not missing, f"{name}: context recorded ({', '.join(missing) + ' missing' if missing else 'complete'})",
                "WARN")
    shared = [n for n, lv in labels.items() if any(r.get("prompt_set") == "shared-prefix" for r in lv.values())]
    unique = [n for n, lv in labels.items() if any(r.get("prompt_set") == "unique-prefix" for r in lv.values())]
    verdict(bool(shared) and bool(unique), f"prefix pair present: shared {shared or 'none'}, unique {unique or 'none'}")
    for name in shared:
        for c, r in sorted(labels[name].items()):
            if r.get("cached_tokens_mean") is None or not r.get("prompt_tokens_mean"):
                verdict(False, f"{name} c={c}: the server reported no cached-token count", "WARN")
                continue
            frac = r["cached_tokens_mean"] / r["prompt_tokens_mean"]
            verdict(frac > 0.5, f"{name} c={c}: mean cached tokens are {frac:.0%} of the prompt")
    for name in unique:
        for c, r in sorted(labels[name].items()):
            if r.get("cached_tokens_mean") is not None and r.get("prompt_tokens_mean"):
                frac = r["cached_tokens_mean"] / r["prompt_tokens_mean"]
                verdict(frac < 0.1, f"{name} c={c}: mean cached tokens are {frac:.0%} of the prompt (control)")
    fails = results.count("FAIL")
    print(f"    {results.count('PASS')} pass, {results.count('WARN')} warn, {fails} fail")
    return 1 if fails else 0


def main(argv: list[str]) -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--list", action="store_true", help="list the labels in the notebook")
    group.add_argument("--label", help="print the curve for one label")
    group.add_argument("--compare", nargs=2, metavar=("A", "B"), help="compare two labels level by level")
    group.add_argument("--check", action="store_true", help="run the lab's validation rules")
    parser.add_argument("--markdown", action="store_true", help="print the curve table as Markdown")
    parser.add_argument("--record", action="store_true",
                        help="with --label, append a curve-summary line to the notebook")
    args = parser.parse_args(argv)

    labels = by_label(load_records(args.labbook))
    if args.list:
        print(f"==> {len(labels)} label(s) in {args.labbook}")
        for name, lv in labels.items():
            r = lv[max(lv)]
            print(f"    {name:<28} {r.get('engine', '?'):<10} {r.get('prompt_set', '?'):<14} "
                  f"levels {','.join(map(str, sorted(lv)))}  last {r.get('recorded_at', '?')}")
        return 0
    if args.check:
        return check(labels)
    if args.compare:
        return print_compare(args.compare[0], args.compare[1], labels)
    if args.label not in labels:
        print(f"No rows for label {args.label}. Labels present: {', '.join(sorted(labels)) or 'none'}",
              file=sys.stderr)
        return 1
    d = print_curve(args.label, labels[args.label], args.markdown)
    if args.record and d["levels"]:
        record_summary(args.labbook, args.label, labels[args.label], d)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download summarise-load.py310 lines

RunnableAll tracks

terminal 2: the curve, derived

python3 summarise-load.py --label llama-baseline --record

Output — what you should see

==> llama-baseline: llama.cpp version: x.x.x (build xxxxx, commit xxxxxxx), model local-chat, prompt set mixed, max_tokens 128
    host Track ..., quant Q4_K_M, context 4096, server slots 20, SLO TTFT <= 2.0 s, TPOT <= 0.1 s
     c     ok  out tok/s  scaling  req/s  goodput  TTFT p50  TTFT p90  TPOT p50  TPOT p90  in flight  cached
     1  40/40       xx.x     1.00  x.xxx    x.xxx     x.xxx     x.xxx    x.xxxx    x.xxxx        1.0     x.x
     5  40/40      xxx.x     x.xx  x.xxx    x.xxx     x.xxx     x.xxx    x.xxxx    x.xxxx        x.x     x.x
    10  40/40      xxx.x     x.xx  x.xxx    x.xxx     x.xxx     x.xxx    x.xxxx    x.xxxx        x.x     x.x
    20  80/80      xxx.x     x.xx  x.xxx    x.xxx     x.xxx     x.xxx    x.xxxx    x.xxxx       xx.x     x.x

    output tokens/s                     TPOT p90 (s)
    c=  1 #####...................     xx.x   ####............ x.xxxx
    ...
    knee: between c=x and c=xx, each added request gained less than half of one request's throughput at c=1
    TPOT p50 ~ xx.xx ms + x.xxx ms x c  (fitted on c = 1, 5, 10, 20)
      c=  1: c / fitted TPOT =     xx.x tok/s if every request were always decoding; measured     xx.x tok/s
      ...
    ceiling as c grows: 1 / b = xxxx.x tok/s; TPOT p50 reaches 0.1 s at c = xx.x
    operating point (no failures, both p90s within the SLO): c=xx
    appended a curve-summary line for llama-baseline to labbook.md

The script reduces the curve to four derived quantities, and each has a mechanism behind it.

Scaling is output throughput at c divided by c times the throughput at c=1. It is 1.00 when every added request is free and falls as the batch stops being free.

The TPOT fit models a decode step as a fixed cost plus a cost per sequence in the batch: TPOT(c) ≈ a + b × c. a is roughly one sequence’s step, set by bandwidth as Task 1 computed; b is what each extra sequence adds, mostly compute and cache traffic. It follows that throughput is c ÷ (a + b × c), which rises towards a ceiling of 1 ÷ b and never reaches it, and that TPOT meets an objective s at c = (s − a) ÷ b.

The knee is the first step between levels where each added request gained less than half of what one request produced alone.

The operating point is the highest level with no failures and both ninetieth percentiles inside the objective. It can sit past the knee: the knee is about efficiency, the operating point about what users feel.

Worked through with illustrative inputs, not a measurement, a = 20 ms and b = 1.5 ms:

c	TPOT = a + b·c	c ÷ TPOT, tok/s	Scaling	Gain per added request, tok/s
1	21.5 ms	46.5	1.00
5	27.5 ms	181.8	0.78	(181.8 − 46.5) ÷ 4 = 33.8
10	35.0 ms	285.7	0.61	(285.7 − 181.8) ÷ 5 = 20.8, below half of 46.5: knee between 5 and 10
20	50.0 ms	400.0	0.43	(400.0 − 285.7) ÷ 10 = 11.4
any	approaches 1.5 ms · c	ceiling 1 ÷ b = 666.7
53	100 ms, the objective: c = (100 − 20) ÷ 1.5

However many requests that machine admits, its output never reaches the ceiling in the table, and its median TPOT meets a 0.1 s objective at about c = 53. Measured output throughput sits below c ÷ TPOT because it also pays for prefill, queueing and the drain at the end of each level; a gap that widens sharply at one level is where admission, not compute, became the limit.

Read your table against the shapes below. Each has a signature in two or three columns, which is why the generator records them all.

Shape in your table	Mechanism	What to try
Throughput rises, TPOT p50 nearly flat, scaling above 0.8	Decode steps are still bandwidth-bound; the extra sequences ride along	Nothing: you are below the knee; the next level is also cheap
Throughput flattens, TPOT rises in step with c	Compute per step is now the limit: b × c dominates a	Operate at or below the knee; more slots will not help
TTFT p90 climbs much faster than TPOT, requests_deferred above 0	Requests wait for a slot or for the block pool before prefill starts	More slots if memory allows (Task 7), or accept the queue
TPOT p90 far above p50 at high c	Other requests’ prefills are interleaved with your decode steps	On vLLM a smaller --max-num-batched-tokens; shorter prompts
Throughput at 20 below 10, in flight normal	Past saturation: scheduling and cache pressure cost more than batching gains; on vLLM, preemption	Record it; Task 9 shows the vLLM counter that confirms it
in flight well below c, a Python process at a full core	The client, not the server, is the limit	Run load-test.py on another machine and repeat

Fill in the recording sheet from the table the script printed; --markdown prints the same table in Markdown for the notebook.

Pending validationThroughput and latency against concurrency, llama-server — your recording sheet

Concurrency	Output tokens/s	Scaling	Goodput req/s	TTFT p50 s	TTFT p90 s	TPOT p50 s	TPOT p90 s
1	—	1.00	—	—	—	—	—
5	—	—	—	—	—	—	—
10	—	—	—	—	—	—	—
20	—	—	—	—	—	—	—

your machine: track, chip and memory, your operating system and version · llama.cpp llama-server LLAMA_VERSION from lab.env · Qwen3-8B, Q4_K_M, KV cache as in lab.env · 4,096 tokens of context · the date you ran it

Empty on purpose: fill it from summarise-load.py. Beside it, write the fitted a and b, the knee, the operating point, and the Task 1 aggregate ceiling at context 100 for comparison with your measured peak.

Record: the curve-summary line the script appended, plus one sentence: the knee, the operating point, and how the measured peak compares with Task 1’s bandwidth ceiling.

6. Prefix caching, measured

Prefix caching skips prefill for the part of a prompt the server has already computed. Each engine decides what “already computed” means differently:

Engine	What is kept	What a new request reuses
llama-server v0.4.0	Each slot keeps its last prompt’s KV (--cache-prompt, on by default)	The longest common prefix with the slot it is assigned; a slot is chosen by prompt similarity (--slot-prompt-similarity, default 0.10). A slot that has never seen the preamble reuses nothing
vLLM 0.28.0	Full 16-token blocks, hashed with everything before them, in the shared pool (enable_prefix_caching defaults to True)	Every leading full block whose hash matches, from any earlier request; never a partial block
mlx_lm.server 0.31.3	Up to --prompt-cache-size (default 10) finished caches, least recently used evicted	The nearest cached prompt’s matching prefix

Time to first token is where the saving appears:

Pseudocode — not a real command

TTFT ≈ queueing + (prompt tokens − cached tokens) ÷ prefill rate + one decode step

With illustrative inputs, not a measurement: a 1,390-token prompt at a prefill rate of 1,000 tokens per second costs about 1.39 s cold. If 1,376 tokens are cached, 14 are left, about 0.014 s plus one decode step. On vLLM the cached count is a multiple of 16. Your own prefill rate is the pp512 figure from Part 6’s lab. Decode does not change, because the cache still has to be read at every step.

The shared-prefix set puts an identical 1,365-token preamble (a system prompt and a tool schema, counted with the Qwen3 tokeniser) before one of eight short questions. The unique-prefix set is the control: the same text behind a random first line, different for every request, so the prompts are the same length and nothing past the chat template’s opening tokens can be reused. Run the control first:

RunnableAll tracks

terminal 2: the control, then the shared prefix

cd ~/llm-course
source lab.env
python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
    --concurrency 1,10 --requests 40 --max-tokens 64 --prompt-set unique-prefix \
    --engine llama.cpp --engine-version "$LLAMA_VERSION" --host-desc "$HOST_DESC" \
    --quant Q4_K_M --context-length 4096 --server-slots "$SLOTS" \
    --label llama-prefix-unique --labbook labbook.md
python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
    --concurrency 1,10 --requests 40 --max-tokens 64 --prompt-set shared-prefix \
    --engine llama.cpp --engine-version "$LLAMA_VERSION" --host-desc "$HOST_DESC" \
    --quant Q4_K_M --context-length 4096 --server-slots "$SLOTS" \
    --label llama-prefix-shared --labbook labbook.md
python3 summarise-load.py --compare llama-prefix-unique llama-prefix-shared

Output — what you should see

...
  c=  1  ok  40/40  fail 0   ...  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (xx/40 within SLO)  in flight 1.0  prompt 14xx.x tok, cached x.x  tokens from usage  finish {'length': 40}
...
  c=  1  ok  40/40  fail 0   ...  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
         goodput x.xxx req/s (xx/40 within SLO)  in flight 1.0  prompt 13xx.x tok, cached 13xx.x  tokens from usage  finish {'length': 40}
...
    note: prompt_set differs: unique-prefix vs shared-prefix
==> llama-prefix-unique (A) against llama-prefix-shared (B)
    c  out/s A  out/s B   B/A  goodput A  goodput B  TTFT p50 A  TTFT p50 B  TTFT p90 A  TTFT p90 B  TPOT p50 A  TPOT p50 B  cached A  cached B
    1     xx.x     xx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx       x.xxx       x.xxx      x.xxxx      x.xxxx       x.x    13xx.x
   10    xxx.x    xxx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx       x.xxx       x.xxx      x.xxxx      x.xxxx       x.x    1xxx.x

What the mechanism predicts, and what each deviation means:

Observation	Prediction	If it does not hold
Cached tokens, control	Single digits at both levels	Higher: the tag is not first, or the file is an old copy of load-test.py
Cached tokens, shared, c=1	Mean a little below the prompt length: every request but the first reuses the preamble	Near zero: prompt caching is off, or the server restarted between runs
Cached tokens, shared, c=10	Lower mean than at c=1 on llama-server: the first requests land on slots that never saw the preamble	Equal to c=1 is what vLLM should show, because its blocks are shared across the pool
TTFT p50, shared against control	Falls by roughly the prefill time of the cached tokens	Unchanged: compare the cached column first; if cached is high and TTFT did not fall, prefill was not your TTFT’s main cost (queueing was)
TTFT p90, shared, c=10	Still high on llama-server: the cold-slot requests are the tail
TPOT p50	About the same in both runs	Much lower in the shared run: the control’s long prefills were interrupting decodes at c=10, which is Task 5’s interleaving row

The engine keeps its own count, which should agree with the client’s:

RunnableAll tracks

terminal 2: the engine's own counters

curl -s http://127.0.0.1:8080/metrics | grep -E '^llamacpp:prompt_tokens_(total|cached_total)'

Output — what you should see

llamacpp:prompt_tokens_total xxxxxx
llamacpp:prompt_tokens_cached_total xxxxxx

The metric’s own HELP line in v0.4.0 describes prompt_tokens_total as prompt tokens processed “excluding cached tokens”, so the cached share is cached ÷ (cached + total) since the server started. The control is also the demonstration of the batching lesson’s warning: a variable first line destroys prefix reuse however much stable text follows it.

Record: the comparison table, and two sentences: the change in TTFT p50 at c=1 with the cached tokens that explain it, and why c=10 differs.

7. Slots, context and cache type: one change at a time

Three settings decide what llama-server allocates and how many sequences it batches. Predict each before running it; the KV sizes are Task 1’s formula.

Variant	Command change	KV size predicted (f16 tier)	What the c=20 level should do
Fewer slots	SLOTS=8	8 × 4,096 × 147,456 B = 4608.00 MiB	TPOT p50 falls (batch of 8), TTFT p90 climbs, requests_deferred reaches 12
Less context per slot	CTX_PER_SLOT=2048	20 × 2,048 × 147,456 B = 5760.00 MiB	Nearly unchanged: these prompts use about 170 tokens of each slot, so half the reservation was never used
The other cache type	CACHE_TYPE=$ALT_KV_TYPE	q8_0: 6120.00 MiB; q4_0 on the 16 GB tiers: 20 × 4,096 × 41,472 B = 3240.00 MiB	Memory halves; TPOT may rise or fall: fewer bytes to read per step against the work of dequantising them

On the eight-bit tiers the first two rows’ sizes are 2448.00 MiB and 3060.00 MiB; with Track M’s 10 slots, variants B and C are 1530.00 MiB and 1620.00 MiB. Run each variant as two blocks, stopping the running server in terminal 1 with Ctrl-C first.

RunnableAll tracks

variant A, terminal 1: eight slots

cd ~/llm-course && source lab.env
SLOTS=8 CACHE_TYPE=$KV_TYPE MODEL=$GGUF LLAMA_BIN=$LLAMA_BIN bash serve-llama-parallel.sh

RunnableAll tracks

variant A, terminal 2: allocation, then the c=20 level

cd ~/llm-course && source lab.env
grep -E "llama_kv_cache: size|n_slots" llama-server.log
python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
    --concurrency 20 --requests 80 --max-tokens 128 --prompt-set mixed \
    --engine llama.cpp --engine-version "$LLAMA_VERSION" --host-desc "$HOST_DESC" \
    --quant Q4_K_M --context-length 4096 --server-slots 8 \
    --label llama-slots8 --labbook labbook.md

RunnableAll tracks

variant B, terminal 1: 2,048 tokens per slot

cd ~/llm-course && source lab.env
SLOTS=$SLOTS CTX_PER_SLOT=2048 CACHE_TYPE=$KV_TYPE MODEL=$GGUF LLAMA_BIN=$LLAMA_BIN bash serve-llama-parallel.sh

RunnableAll tracks

variant B, terminal 2: allocation, then the c=20 level

cd ~/llm-course && source lab.env
grep -E "llama_kv_cache: size|n_slots" llama-server.log
python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
    --concurrency 20 --requests 80 --max-tokens 128 --prompt-set mixed \
    --engine llama.cpp --engine-version "$LLAMA_VERSION" --host-desc "$HOST_DESC" \
    --quant Q4_K_M --context-length 2048 --server-slots "$SLOTS" \
    --label llama-ctx2048 --labbook labbook.md

RunnableAll tracks

variant C, terminal 1: the other cache type

cd ~/llm-course && source lab.env
SLOTS=$SLOTS CACHE_TYPE=$ALT_KV_TYPE MODEL=$GGUF LLAMA_BIN=$LLAMA_BIN bash serve-llama-parallel.sh

RunnableAll tracks

variant C, terminal 2: allocation, then the c=20 level

cd ~/llm-course && source lab.env
grep -E "llama_kv_cache: size|n_slots" llama-server.log
python3 load-test.py --base-url http://127.0.0.1:8080/v1 --model local-chat \
    --concurrency 20 --requests 80 --max-tokens 128 --prompt-set mixed \
    --engine llama.cpp --engine-version "$LLAMA_VERSION" --host-desc "$HOST_DESC" \
    --quant "Q4_K_M, KV $ALT_KV_TYPE" --context-length 4096 --server-slots "$SLOTS" \
    --label llama-kv-alt --labbook labbook.md

Output — what you should see

x.xx.xxx.xxx I llama_kv_cache: size =  xxxx.00 MiB (  xxxx cells,  36 layers, xx/xx seqs), K (xxxx):  xxxx.00 MiB, V (xxxx):  xxxx.00 MiB
x.xx.xxx.xxx I srv    load_model: initializing, n_slots = xx, n_ctx_slot = xxxx, kv_unified = 'false'
==> llama-slots8: local-chat at http://127.0.0.1:8080/v1
...
  c= 20  ok  80/80  fail 0   wall   xx.xxs  out/s   xxx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx
...

Compare each against the baseline’s c=20 row with python3 summarise-load.py --compare llama-baseline llama-slots8 (and the same for llama-ctx2048 and llama-kv-alt). Leave the server running in the baseline configuration afterwards if you are on Track X without vLLM; otherwise stop it before Task 8.

✕Common mistake

Raising the slot count to fix a slow TTFT without checking memory. Slots are allocated whole at load. SLOTS=40 at 4,096 tokens and f16 asks for 23,040 MiB of cache; on a 24 GB card the load fails, and on a unified-memory Mac it can push the system into swap, where every number you then measure describes the disk. Do the Task 1 sum for the new value first.

Record: for each variant, the predicted and logged KV size, and the c=20 row beside the baseline’s, with one sentence saying whether the prediction held.

8. The same load against the other engine

Hold five things equal or the comparison says nothing: the model family and size, the context per sequence (4,096), the concurrency cap (20), a KV pool of the same number of tokens, and the same client, prompts and levels. Two things cannot be equal and go in the notebook instead: the weight format, because a Q4_K_M GGUF and an AWQ checkpoint are not the same bytes (Task 1 put the difference at about a gigabyte), and startup time.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

serve-vllm-container.sh

#!/usr/bin/env bash
# Purpose: start vLLM from NVIDIA's container image with the settings this lab measures,
#          publishing the port on the loopback address only, mounting the model library
#          read-only, and copying the log to a file on the host
# Platform: spark (NVIDIA's NGC vLLM image, the documented path on DGX Spark); usable on
#           nvidia with IMAGE overridden. Track X uses serve-vllm-native.sh; Track M has no
#           vLLM path
# Minimum memory: 16 GB of accelerator-visible memory for the four-bit 8B checkpoint
# Assumes: docker able to reach the GPU (--gpus all), the image already pulled, the
#          checkpoint downloaded under $MODELS_DIR on the host, nothing else answering on
#          $PORT; runs in the foreground, and Ctrl-C or "docker rm -f vllm-lab" stops it
#
# Usage: KV_BYTES=12079595520 bash serve-vllm-container.sh
#        MAX_SEQS=8 KV_BYTES=12079595520 bash serve-vllm-container.sh
#        KV_BYTES= MEM_FRACTION=0.30 bash serve-vllm-container.sh
#
# Environment:
#   IMAGE           container image                         (default: nvcr.io/nvidia/vllm:26.08-py3)
#   MODELS_DIR      host model library, mounted at /models  (default: $HOME/models)
#   MODEL           checkpoint path inside the container    (default: /models/Qwen/Qwen3-8B-AWQ)
#   SERVED_NAME     model name clients send                 (default: local-chat)
#   PORT            host port, bound to 127.0.0.1           (default: 8000)
#   CTX, MAX_SEQS, KV_BYTES, MEM_FRACTION, KV_DTYPE, PREFIX_CACHING
#                   as in serve-vllm-native.sh              (defaults: 4096, 20, empty, 0.90,
#                                                            auto, on)
#   CONTAINER_NAME  container name                          (default: vllm-lab)
#   LOGFILE         copy of the server log on the host      (default: ./vllm-server.log)
#
# On a unified-memory machine --gpu-memory-utilization is a share of memory the operating
# system also uses, so this lab sets KV_BYTES instead wherever it can. --ipc=host: vLLM's
# Docker documentation says PyTorch needs the host's shared memory to pass data between
# processes.

set -euo pipefail

IMAGE="${IMAGE:-nvcr.io/nvidia/vllm:26.08-py3}"
MODELS_DIR="${MODELS_DIR:-$HOME/models}"
MODEL="${MODEL:-/models/Qwen/Qwen3-8B-AWQ}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
PORT="${PORT:-8000}"
CTX="${CTX:-4096}"
MAX_SEQS="${MAX_SEQS:-20}"
KV_BYTES="${KV_BYTES:-}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
KV_DTYPE="${KV_DTYPE:-auto}"
PREFIX_CACHING="${PREFIX_CACHING:-on}"
CONTAINER_NAME="${CONTAINER_NAME:-vllm-lab}"
LOGFILE="${LOGFILE:-./vllm-server.log}"

die() { echo "serve-vllm-container: $*" >&2; exit 1; }

command -v docker >/dev/null 2>&1 || die "docker is not on PATH"
docker image inspect "$IMAGE" >/dev/null 2>&1 || die "image $IMAGE not present; run: docker pull $IMAGE"
case "$MODEL" in
    /models/*) [ -f "$MODELS_DIR/${MODEL#/models/}/config.json" ] \
        || die "$MODELS_DIR/${MODEL#/models/}/config.json not found; download the checkpoint first" ;;
esac
case "$PREFIX_CACHING" in
    on) PREFIX_FLAG="--enable-prefix-caching" ;;
    off) PREFIX_FLAG="--no-enable-prefix-caching" ;;
    *) die "PREFIX_CACHING must be on or off" ;;
esac
if [ -n "$KV_BYTES" ]; then
    MEMORY_FLAG="--kv-cache-memory-bytes"
    MEMORY_VALUE="$KV_BYTES"
else
    MEMORY_FLAG="--gpu-memory-utilization"
    MEMORY_VALUE="$MEM_FRACTION"
fi
if command -v curl >/dev/null 2>&1 \
    && curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$PORT/"; then
    die "something is already answering on 127.0.0.1:$PORT; stop it first (one server at a time)"
fi
if docker container inspect "$CONTAINER_NAME" >/dev/null 2>&1; then
    die "a container named $CONTAINER_NAME exists; remove it with: docker rm -f $CONTAINER_NAME"
fi

cat <<INFO
==> vllm serve, in a container
    image             $IMAGE
    model             $MODEL  (host: $MODELS_DIR, read-only)
    served as         $SERVED_NAME
    published on      http://127.0.0.1:$PORT/v1
    max model length  $CTX tokens per sequence
    max sequences     $MAX_SEQS
    KV cache memory   $MEMORY_FLAG $MEMORY_VALUE
    KV cache dtype    $KV_DTYPE
    prefix caching    $PREFIX_CACHING
    log copied to     $LOGFILE
    Stop it with Ctrl-C, or from another terminal: docker rm -f $CONTAINER_NAME
INFO

docker run --rm \
    --name "$CONTAINER_NAME" \
    --gpus all \
    --ipc=host \
    -v "$MODELS_DIR:/models:ro" \
    -p "127.0.0.1:$PORT:8000" \
    "$IMAGE" \
    vllm serve "$MODEL" \
    --host 0.0.0.0 \
    --port 8000 \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    "$MEMORY_FLAG" "$MEMORY_VALUE" \
    --kv-cache-dtype "$KV_DTYPE" \
    "$PREFIX_FLAG" \
    --enable-prompt-tokens-details 2>&1 | tee "$LOGFILE"

Download serve-vllm-container.sh109 lines

RunnableTrack S · DGX Spark

terminal 1: vLLM in the NGC container, equal pool

cd ~/llm-course && source lab.env
IMAGE=$VLLM_IMAGE MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-container.sh

RunnableTrack S · DGX Spark

terminal 2: its allocation, and its version

cd ~/llm-course && source lab.env
grep -E "Model loading took|memory for KV Cache|Available KV cache memory|GPU KV cache size|Application startup complete" vllm-server.log
export VLLM_VERSION="$(docker exec vllm-lab vllm --version | tail -n 1)"; echo "$VLLM_VERSION"

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

serve-vllm-native.sh

#!/usr/bin/env bash
# Purpose: start vLLM as an OpenAI-compatible server with the settings this lab measures -
#          context per sequence, sequence cap, KV cache size or memory fraction, KV cache
#          dtype and prefix caching - printing the configuration first and copying the log
#          to a file, so the run is reproducible from the terminal and searchable afterwards
# Platform: nvidia and strix (a native vLLM install in an activated virtual environment);
#           Track S uses serve-vllm-container.sh, Track M has no vLLM path
# Minimum memory: 16 GB of accelerator-visible memory for the four-bit 8B checkpoint
# Assumes: vllm on PATH (the install lesson's environment, activated), the checkpoint in
#          $MODEL (a local directory with config.json, or a Hugging Face repository id that
#          vLLM downloads), nothing else answering on $PORT; runs in the foreground
#
# Usage: MODEL=$HOME/models/Qwen/Qwen3-8B-AWQ KV_BYTES=12079595520 bash serve-vllm-native.sh
#        MAX_SEQS=8 MODEL=... bash serve-vllm-native.sh
#        KV_BYTES= MEM_FRACTION=0.80 MODEL=... bash serve-vllm-native.sh
#
# Environment:
#   MODEL           checkpoint directory or repository id  (default: $HOME/models/Qwen/Qwen3-8B-AWQ)
#   SERVED_NAME     model name clients send                (default: local-chat)
#   HOST            address to bind                        (default: 127.0.0.1)
#   PORT            port to listen on                      (default: 8000)
#   CTX             --max-model-len, tokens per sequence   (default: 4096)
#   MAX_SEQS        --max-num-seqs, sequences in flight    (default: 20)
#   KV_BYTES        --kv-cache-memory-bytes; empty to size the cache from MEM_FRACTION
#                                                          (default: empty)
#   MEM_FRACTION    --gpu-memory-utilization, used when KV_BYTES is empty (default: 0.90)
#   KV_DTYPE        --kv-cache-dtype: auto (the model's dtype) or fp8     (default: auto)
#   PREFIX_CACHING  on or off                               (default: on, vLLM 0.28.0's default)
#   TP_SIZE         --tensor-parallel-size                  (default: 1)
#   LOGFILE         copy of the server log                  (default: ./vllm-server.log)
#
# Read from vLLM 0.28.0's source: kv_cache_memory_bytes, when set, "ignores
# gpu_memory_utilization" (config/cache.py), and the worker then logs "reserved ... GiB memory
# for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling";
# without it the worker logs "Available KV cache memory: N GiB". Either way the engine logs
# "GPU KV cache size: N tokens, Maximum concurrency for N tokens per request: N.NNx".
# enable_prefix_caching defaults to True. --enable-prompt-tokens-details adds
# usage.prompt_tokens_details.cached_tokens to responses, which load-test.py records.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/Qwen/Qwen3-8B-AWQ}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8000}"
CTX="${CTX:-4096}"
MAX_SEQS="${MAX_SEQS:-20}"
KV_BYTES="${KV_BYTES:-}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
KV_DTYPE="${KV_DTYPE:-auto}"
PREFIX_CACHING="${PREFIX_CACHING:-on}"
TP_SIZE="${TP_SIZE:-1}"
LOGFILE="${LOGFILE:-./vllm-server.log}"

die() { echo "serve-vllm-native: $*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 \
    || die "vllm is not on PATH; activate the environment from the install lesson first"
case "$MODEL" in
    /*|./*|../*) [ -f "$MODEL/config.json" ] || die "$MODEL/config.json not found; download the checkpoint first" ;;
esac
case "$PREFIX_CACHING" in
    on) PREFIX_FLAG="--enable-prefix-caching" ;;
    off) PREFIX_FLAG="--no-enable-prefix-caching" ;;
    *) die "PREFIX_CACHING must be on or off" ;;
esac
if [ -n "$KV_BYTES" ]; then
    MEMORY_FLAG="--kv-cache-memory-bytes"
    MEMORY_VALUE="$KV_BYTES"
else
    MEMORY_FLAG="--gpu-memory-utilization"
    MEMORY_VALUE="$MEM_FRACTION"
fi
if command -v curl >/dev/null 2>&1 \
    && curl --silent --max-time 2 --output /dev/null "http://$HOST:$PORT/"; then
    die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi
if [ "$HOST" != "127.0.0.1" ] && [ "$HOST" != "localhost" ]; then
    echo "NOTE: binding to $HOST exposes an unauthenticated generation endpoint." >&2
    echo "      Set an API key and put it behind TLS before doing this on a real network." >&2
fi

echo "==> vllm serve, $(vllm --version 2>/dev/null | tail -n 1)"
cat <<INFO
    model             $MODEL
    served as         $SERVED_NAME
    listening on      http://$HOST:$PORT/v1
    max model length  $CTX tokens per sequence
    max sequences     $MAX_SEQS
    KV cache memory   $MEMORY_FLAG $MEMORY_VALUE
    KV cache dtype    $KV_DTYPE
    prefix caching    $PREFIX_CACHING
    tensor parallel   $TP_SIZE
    log copied to     $LOGFILE
    The first start profiles memory and captures CUDA graphs; wait for
    "Application startup complete." before sending requests.
INFO

# The log goes to the terminal and to $LOGFILE; exec keeps this shell's pid for the server.
exec > >(tee "$LOGFILE") 2>&1
exec vllm serve "$MODEL" \
    --host "$HOST" \
    --port "$PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    "$MEMORY_FLAG" "$MEMORY_VALUE" \
    --kv-cache-dtype "$KV_DTYPE" \
    "$PREFIX_FLAG" \
    --tensor-parallel-size "$TP_SIZE" \
    --enable-prompt-tokens-details

Download serve-vllm-native.sh111 lines

RunnableTrack X · Ryzen AI Max+

terminal 1: vLLM from the ROCm wheels, equal pool

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack X · Ryzen AI Max+

terminal 2: its allocation, and its version

cd ~/llm-course && source lab.env
grep -E "Model loading took|memory for KV Cache|Available KV cache memory|GPU KV cache size|Application startup complete" vllm-server.log
export VLLM_VERSION="$("$VLLM_VENV/bin/vllm" --version | tail -n 1)"; echo "$VLLM_VERSION"

The bf16 weights make vLLM read about three times the bytes per token that llama-server’s Q4_K_M does, so its c=1 row should be the slower one before any batching difference appears.

Track M — Apple siliconPartial

vLLM has no mainline Apple silicon path; the comparison on this track is llama-server against mlx_lm.server, which batches but allocates its cache on demand, so there is no equal pool to set.

RunnableTrack M · Apple silicon

terminal 1: mlx_lm.server through Part 8's script

cd ~/llm-course && source lab.env
PORT=8081 PROMPT_CACHE_SIZE=10 MODEL=$MLX_MODEL bash serve-mlx.sh

Output — what you should see

==> mlx_lm.server
    mlx-lm 0.31.3 / mlx x.xx.x
    model   /Users/you/models/mlx-community/Qwen3-8B-4bit
...
    send "model": "/Users/you/models/mlx-community/Qwen3-8B-4bit" in every request
... Starting httpd at 127.0.0.1 on port 8081...

The MLX server resolves a request’s model field as a model to load, so every command below sends the absolute path, $MLX_MODEL. PROMPT_CACHE_SIZE=10 restores the server’s default, which Part 8’s script lowers to 1 for its own measurement.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

serve-vllm-native.sh

#!/usr/bin/env bash
# Purpose: start vLLM as an OpenAI-compatible server with the settings this lab measures -
#          context per sequence, sequence cap, KV cache size or memory fraction, KV cache
#          dtype and prefix caching - printing the configuration first and copying the log
#          to a file, so the run is reproducible from the terminal and searchable afterwards
# Platform: nvidia and strix (a native vLLM install in an activated virtual environment);
#           Track S uses serve-vllm-container.sh, Track M has no vLLM path
# Minimum memory: 16 GB of accelerator-visible memory for the four-bit 8B checkpoint
# Assumes: vllm on PATH (the install lesson's environment, activated), the checkpoint in
#          $MODEL (a local directory with config.json, or a Hugging Face repository id that
#          vLLM downloads), nothing else answering on $PORT; runs in the foreground
#
# Usage: MODEL=$HOME/models/Qwen/Qwen3-8B-AWQ KV_BYTES=12079595520 bash serve-vllm-native.sh
#        MAX_SEQS=8 MODEL=... bash serve-vllm-native.sh
#        KV_BYTES= MEM_FRACTION=0.80 MODEL=... bash serve-vllm-native.sh
#
# Environment:
#   MODEL           checkpoint directory or repository id  (default: $HOME/models/Qwen/Qwen3-8B-AWQ)
#   SERVED_NAME     model name clients send                (default: local-chat)
#   HOST            address to bind                        (default: 127.0.0.1)
#   PORT            port to listen on                      (default: 8000)
#   CTX             --max-model-len, tokens per sequence   (default: 4096)
#   MAX_SEQS        --max-num-seqs, sequences in flight    (default: 20)
#   KV_BYTES        --kv-cache-memory-bytes; empty to size the cache from MEM_FRACTION
#                                                          (default: empty)
#   MEM_FRACTION    --gpu-memory-utilization, used when KV_BYTES is empty (default: 0.90)
#   KV_DTYPE        --kv-cache-dtype: auto (the model's dtype) or fp8     (default: auto)
#   PREFIX_CACHING  on or off                               (default: on, vLLM 0.28.0's default)
#   TP_SIZE         --tensor-parallel-size                  (default: 1)
#   LOGFILE         copy of the server log                  (default: ./vllm-server.log)
#
# Read from vLLM 0.28.0's source: kv_cache_memory_bytes, when set, "ignores
# gpu_memory_utilization" (config/cache.py), and the worker then logs "reserved ... GiB memory
# for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling";
# without it the worker logs "Available KV cache memory: N GiB". Either way the engine logs
# "GPU KV cache size: N tokens, Maximum concurrency for N tokens per request: N.NNx".
# enable_prefix_caching defaults to True. --enable-prompt-tokens-details adds
# usage.prompt_tokens_details.cached_tokens to responses, which load-test.py records.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/Qwen/Qwen3-8B-AWQ}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8000}"
CTX="${CTX:-4096}"
MAX_SEQS="${MAX_SEQS:-20}"
KV_BYTES="${KV_BYTES:-}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
KV_DTYPE="${KV_DTYPE:-auto}"
PREFIX_CACHING="${PREFIX_CACHING:-on}"
TP_SIZE="${TP_SIZE:-1}"
LOGFILE="${LOGFILE:-./vllm-server.log}"

die() { echo "serve-vllm-native: $*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 \
    || die "vllm is not on PATH; activate the environment from the install lesson first"
case "$MODEL" in
    /*|./*|../*) [ -f "$MODEL/config.json" ] || die "$MODEL/config.json not found; download the checkpoint first" ;;
esac
case "$PREFIX_CACHING" in
    on) PREFIX_FLAG="--enable-prefix-caching" ;;
    off) PREFIX_FLAG="--no-enable-prefix-caching" ;;
    *) die "PREFIX_CACHING must be on or off" ;;
esac
if [ -n "$KV_BYTES" ]; then
    MEMORY_FLAG="--kv-cache-memory-bytes"
    MEMORY_VALUE="$KV_BYTES"
else
    MEMORY_FLAG="--gpu-memory-utilization"
    MEMORY_VALUE="$MEM_FRACTION"
fi
if command -v curl >/dev/null 2>&1 \
    && curl --silent --max-time 2 --output /dev/null "http://$HOST:$PORT/"; then
    die "something is already answering on $HOST:$PORT; stop it first (one server at a time)"
fi
if [ "$HOST" != "127.0.0.1" ] && [ "$HOST" != "localhost" ]; then
    echo "NOTE: binding to $HOST exposes an unauthenticated generation endpoint." >&2
    echo "      Set an API key and put it behind TLS before doing this on a real network." >&2
fi

echo "==> vllm serve, $(vllm --version 2>/dev/null | tail -n 1)"
cat <<INFO
    model             $MODEL
    served as         $SERVED_NAME
    listening on      http://$HOST:$PORT/v1
    max model length  $CTX tokens per sequence
    max sequences     $MAX_SEQS
    KV cache memory   $MEMORY_FLAG $MEMORY_VALUE
    KV cache dtype    $KV_DTYPE
    prefix caching    $PREFIX_CACHING
    tensor parallel   $TP_SIZE
    log copied to     $LOGFILE
    The first start profiles memory and captures CUDA graphs; wait for
    "Application startup complete." before sending requests.
INFO

# The log goes to the terminal and to $LOGFILE; exec keeps this shell's pid for the server.
exec > >(tee "$LOGFILE") 2>&1
exec vllm serve "$MODEL" \
    --host "$HOST" \
    --port "$PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    "$MEMORY_FLAG" "$MEMORY_VALUE" \
    --kv-cache-dtype "$KV_DTYPE" \
    "$PREFIX_FLAG" \
    --tensor-parallel-size "$TP_SIZE" \
    --enable-prompt-tokens-details

Download serve-vllm-native.sh111 lines

RunnableTrack N · NVIDIA GPU

terminal 1: vLLM, equal pool

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack N · NVIDIA GPU

terminal 2: its allocation, and its version

cd ~/llm-course && source lab.env
grep -E "Model loading took|memory for KV Cache|Available KV cache memory|GPU KV cache size|Application startup complete" vllm-server.log
export VLLM_VERSION="$("$VLLM_VENV/bin/vllm" --version | tail -n 1)"; echo "$VLLM_VERSION"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Output — what you should see

... Model loading took x.xx GiB memory and xx.xxxxxx seconds
... Initial free memory xx.xx GiB, reserved 11.25 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. ...
... GPU KV cache size: 81,920 tokens, Maximum concurrency for 4,096 tokens per request: 20.00x
INFO:     Application startup complete.
0.28.0

Those lines are format strings in vLLM 0.28.0’s source; the prefix before them (process, time, file) varies. On the 16 GB tier the reservation reads 5.62 GiB and the tokens and concurrency are the same. A GPU KV cache size below 81,920 tokens or a concurrency below 20.00x means the pool is not equal: stop and check KV_BYTES before measuring. The first start takes minutes (profiling, graph capture); wait for Application startup complete.

Then the same sweep and prefix pair against port 8000 (8081 and --model "$MLX_MODEL" on Track M):

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

terminal 2: baseline sweep and prefix pair, vLLM

cd ~/llm-course && source lab.env
for set in mixed unique-prefix shared-prefix; do
    case $set in mixed) levels=1,5,10,20; tokens=128; label=vllm-baseline ;;
                 unique-prefix) levels=1,10; tokens=64; label=vllm-prefix-unique ;;
                 shared-prefix) levels=1,10; tokens=64; label=vllm-prefix-shared ;; esac
    python3 load-test.py --base-url http://127.0.0.1:8000/v1 --model local-chat \
        --concurrency $levels --requests 40 --min-rounds 4 --max-tokens $tokens --prompt-set $set \
        --engine vllm --engine-version "$VLLM_VERSION" --host-desc "$HOST_DESC" \
        --quant "$VLLM_QUANT" --context-length 4096 --server-slots 20 \
        --label $label --labbook labbook.md
done

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

terminal 2: baseline sweep and prefix pair, vLLM

cd ~/llm-course && source lab.env
for set in mixed unique-prefix shared-prefix; do
    case $set in mixed) levels=1,5,10,20; tokens=128; label=vllm-baseline ;;
                 unique-prefix) levels=1,10; tokens=64; label=vllm-prefix-unique ;;
                 shared-prefix) levels=1,10; tokens=64; label=vllm-prefix-shared ;; esac
    python3 load-test.py --base-url http://127.0.0.1:8000/v1 --model local-chat \
        --concurrency $levels --requests 40 --min-rounds 4 --max-tokens $tokens --prompt-set $set \
        --engine vllm --engine-version "$VLLM_VERSION" --host-desc "$HOST_DESC" \
        --quant "$VLLM_QUANT" --context-length 4096 --server-slots 20 \
        --label $label --labbook labbook.md
done

Track M — Apple silicon

RunnableTrack M · Apple silicon

terminal 2: baseline sweep and prefix pair, mlx_lm.server

cd ~/llm-course && source lab.env
for set in mixed unique-prefix shared-prefix; do
    case $set in mixed) levels=1,5,10,20; tokens=128; label=mlx-baseline ;;
                 unique-prefix) levels=1,10; tokens=64; label=mlx-prefix-unique ;;
                 shared-prefix) levels=1,10; tokens=64; label=mlx-prefix-shared ;; esac
    python3 load-test.py --base-url http://127.0.0.1:8081/v1 --model "$MLX_MODEL" \
        --concurrency $levels --requests 40 --min-rounds 4 --max-tokens $tokens --prompt-set $set \
        --engine mlx-lm --engine-version 0.31.3 --host-desc "$HOST_DESC" \
        --quant "MLX 4-bit" --context-length 4096 --server-slots 32 \
        --label $label --labbook labbook.md
done

Replace 0.31.3 with the version serve-mlx.sh printed if it differs. --server-slots 32 is the server’s default decode concurrency, not a slot allocation.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

terminal 2: baseline sweep and prefix pair, vLLM

cd ~/llm-course && source lab.env
for set in mixed unique-prefix shared-prefix; do
    case $set in mixed) levels=1,5,10,20; tokens=128; label=vllm-baseline ;;
                 unique-prefix) levels=1,10; tokens=64; label=vllm-prefix-unique ;;
                 shared-prefix) levels=1,10; tokens=64; label=vllm-prefix-shared ;; esac
    python3 load-test.py --base-url http://127.0.0.1:8000/v1 --model local-chat \
        --concurrency $levels --requests 40 --min-rounds 4 --max-tokens $tokens --prompt-set $set \
        --engine vllm --engine-version "$VLLM_VERSION" --host-desc "$HOST_DESC" \
        --quant "$VLLM_QUANT" --context-length 4096 --server-slots 20 \
        --label $label --labbook labbook.md
done

The loop runs the three sets in that order, so the prefix control again comes first. The output has the same shape as Tasks 4 and 6. Summarise and compare:

RunnableAll tracks

terminal 2: the second engine's curve, and the comparison

cd ~/llm-course
second=$(python3 summarise-load.py --list | awk '$1 ~ /^(vllm|mlx)-baseline$/ {print $1; exit}')
python3 summarise-load.py --label "$second" --record
python3 summarise-load.py --compare llama-baseline "$second"
python3 summarise-load.py --compare "${second%-baseline}-prefix-unique" "${second%-baseline}-prefix-shared"

Output — what you should see

==> vllm-baseline: vllm 0.28.0, model local-chat, prompt set mixed, max_tokens 128
...
    note: quant differs: Q4_K_M vs AWQ
    note: engine differs: llama.cpp vs vllm
==> llama-baseline (A) against vllm-baseline (B)
    c  out/s A  out/s B   B/A  goodput A  goodput B  TTFT p50 A  TTFT p50 B  ...
    1     xx.x     xx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx  ...
    5    xxx.x    xxx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx  ...
   10    xxx.x    xxx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx  ...
   20    xxx.x    xxx.x  x.xx      x.xxx      x.xxx       x.xxx       x.xxx  ...

Read B/A down the column rather than at one level. At c=1 it is mostly the weight formats and kernels, which Part 8 measured for one user. How it moves from c=1 to c=20 is the scheduling and memory management difference this part is about. In the prefix comparison, vLLM’s cached mean at c=10 should be close to its c=1 value because its blocks are shared across the whole pool; set that beside llama-server’s per-slot figure from Task 6.

Cross-check the client with vLLM’s own generator at the same concurrency cap:

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

terminal 2: vllm bench serve inside the container

cd ~/llm-course && source lab.env
docker exec vllm-lab vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 \
    --endpoint /v1/chat/completions --model "$VLLM_MODEL" --served-model-name local-chat \
    --dataset-name random --random-input-len 40 --random-output-len 128 --ignore-eos \
    --num-prompts 80 --max-concurrency 20 \
    --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90 --goodput ttft:2000 tpot:100

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

terminal 2: vllm bench serve

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 \
    --endpoint /v1/chat/completions --model "$VLLM_MODEL" --served-model-name local-chat \
    --dataset-name random --random-input-len 40 --random-output-len 128 --ignore-eos \
    --num-prompts 80 --max-concurrency 20 \
    --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90 --goodput ttft:2000 tpot:100

Track M — Apple siliconNot supported

vllm bench serve ships with vLLM, which this track does not install; the client check on Track M is the in-flight column and a second run from another machine.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

terminal 2: vllm bench serve

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 \
    --endpoint /v1/chat/completions --model "$VLLM_MODEL" --served-model-name local-chat \
    --dataset-name random --random-input-len 40 --random-output-len 128 --ignore-eos \
    --num-prompts 80 --max-concurrency 20 \
    --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90 --goodput ttft:2000 tpot:100

Output — what you should see

============ Serving Benchmark Result ============
Successful requests:                     80
Failed requests:                         0
Maximum request concurrency:             20
Benchmark duration (s):                  xx.xx
...
Request throughput (req/s):              x.xx
Request goodput (req/s):                 x.xx
Output token throughput (tok/s):         xxx.xx
...
---------------Time to First Token----------------
Mean TTFT (ms):                          xx.xx
Median TTFT (ms):                        xx.xx
P90 TTFT (ms):                           xx.xx
-----Time per Output Token (excl. 1st token)------
...

--model names the checkpoint the tool loads its tokeniser from, and --served-model-name the name it sends. --ignore-eos makes every request generate exactly 128 tokens, which load-test.py gets from Qwen3’s thinking instead. The two tools use the same TPOT definition, so the median TPOT and output throughput should agree with your vllm-baseline c=20 row within a few per cent; if load-test.py reports clearly less throughput, suspect the client. If the command reports a missing Python module, vLLM’s bench extra (vllm[bench]) adds the optional dependencies.

Record: the engine’s version, its allocation lines, the comparison table, the prefix comparison, and the bench cross-check’s output throughput and median TPOT beside your c=20 row.

9. vLLM: admission and memory fraction

Two more restarts, each with no other change. Admission: --max-num-seqs caps the batch the way SLOTS did, but the pool stays the same size. Memory fraction: without KV_BYTES, vLLM claims --gpu-memory-utilization of the device’s total memory, profiles what the weights and activations take, and gives the rest to the pool. Stop the server in terminal 1 before each start.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

terminal 1: cap of eight sequences, equal pool

cd ~/llm-course && source lab.env
MAX_SEQS=8 IMAGE=$VLLM_IMAGE MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-container.sh

RunnableTrack S · DGX Spark

terminal 1: memory fraction A

cd ~/llm-course && source lab.env
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_A IMAGE=$VLLM_IMAGE MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-container.sh

RunnableTrack S · DGX Spark

terminal 1: memory fraction B, after stopping A

cd ~/llm-course && source lab.env
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_B IMAGE=$VLLM_IMAGE MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-container.sh

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

terminal 1: cap of eight sequences, equal pool

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
MAX_SEQS=8 MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack X · Ryzen AI Max+

terminal 1: memory fraction A

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_A MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack X · Ryzen AI Max+

terminal 1: memory fraction B, after stopping A

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_B MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

lab.env’s fractions assume the 128 GB machine. On 64 GB, fraction × the GPU-visible total the device line reported must exceed the 15.26 GiB of bf16 weights with room for a pool; if the log reports No available memory for the cache blocks, raise both by 0.2 and record the values.

Track M — Apple siliconNot supported

vLLM does not run on this track, and mlx_lm.server allocates its cache on demand, so there is no sequence cap or memory fraction to set; Task 7's slot and cache-type variants are Track M's memory experiments.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

terminal 1: cap of eight sequences, equal pool

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
MAX_SEQS=8 MODEL=$VLLM_MODEL KV_BYTES=$VLLM_KV_BYTES KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack N · NVIDIA GPU

terminal 1: memory fraction A

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_A MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

RunnableTrack N · NVIDIA GPU

terminal 1: memory fraction B, after stopping A

cd ~/llm-course && source lab.env && source "$VLLM_VENV/bin/activate"
KV_BYTES= MEM_FRACTION=$MEM_FRACTION_B MODEL=$VLLM_MODEL KV_DTYPE=$VLLM_KV_DTYPE bash serve-vllm-native.sh

Start the eight-sequence server first, then run the c=20 level against it and sample the queue while it runs:

RunnableAll tracks

terminal 2 (not Track M): c=20 against eight sequences

cd ~/llm-course && source lab.env
python3 load-test.py --base-url http://127.0.0.1:8000/v1 --model local-chat \
    --concurrency 20 --requests 80 --max-tokens 128 --prompt-set mixed \
    --engine vllm --engine-version "$VLLM_VERSION" --host-desc "$HOST_DESC" \
    --quant "$VLLM_QUANT" --context-length 4096 --server-slots 8 \
    --label vllm-seqs8 --labbook labbook.md &
sleep 20
curl -s http://127.0.0.1:8000/metrics | grep -E '^vllm:(num_requests_running|num_requests_waiting|num_preemptions_total|kv_cache_usage_perc)'
wait

Output — what you should see

vllm:num_requests_running{model_name="local-chat",engine="0"} 8.0
vllm:num_requests_waiting{model_name="local-chat",engine="0"} 12.0
vllm:num_preemptions_total{model_name="local-chat",engine="0"} 0.0
vllm:kv_cache_usage_perc{model_name="local-chat",engine="0"} 0.0x
...
  c= 20  ok  80/80  fail 0   wall   xx.xxs  out/s   xxx.xx  req/s  x.xxx  TTFT p50 x.xxx p90 x.xxx  TPOT p50 x.xxxx p90 x.xxxx

Running at the cap and waiting at 12 is admission as the limit, with almost all of the pool idle. The periodic stats line in terminal 1 shows the same (Running: 8 reqs, Waiting: 12 reqs, GPU KV cache usage: x.x%, Prefix cache hit rate: x.x%) and adds Preemptions: N only when there were any. The warning text vLLM’s optimisation page quotes for a preemption does not appear in the 0.28.0 scheduler’s source; that counter and that field are how preemption shows up.

Then stop it and start fraction A, and later fraction B. After each fraction start, wait for Application startup complete. and read what the engine made of it:

RunnableAll tracks

terminal 2 (not Track M): the pool a fraction produced

cd ~/llm-course
grep -E "Model loading took|Available KV cache memory|GPU KV cache size" vllm-server.log

Output — what you should see

... Model loading took x.xx GiB memory and xx.xxxxxx seconds
... Available KV cache memory: x.xx GiB
... GPU KV cache size: xx,xxx tokens, Maximum concurrency for 4,096 tokens per request: xx.xxx

The arithmetic behind the last line, with an illustrative log value rather than a measurement (use 73,728 bytes per token on the fp8 tier):

Step	Formula	If the log says Available KV cache memory: 7.00 GiB
Blocks	available bytes ÷ (16 × 147,456)	7,516,192,768 ÷ 2,359,296 = 3,185 (whole blocks)
Tokens	blocks × 16	50,960
Maximum concurrency	blocks ÷ blocks per 4,096-token request (256)	12.44x

Moving from fraction A to B changes the pool by roughly the difference in fraction times the total, because the weights and activations do not change. On a unified-memory machine the fraction is a share of memory the operating system also uses, and vLLM refuses to start when free memory is below it; that is why lab.env uses smaller fractions on Tracks S and X, and why the equal-pool runs set bytes instead.

Record: the vllm-seqs8 row beside vllm-baseline at c=20, the metrics sample, and for each fraction the model memory, available KV memory, tokens and maximum concurrency, with your calculation from the available memory.

10. Record the results

Everything numeric is already in labbook.md as JSON lines. Two commands turn it into the record a later part can use:

RunnableAll tracks

terminal 2: the notebook tables and the check

cd ~/llm-course
python3 summarise-load.py --list
python3 summarise-load.py --label llama-baseline --markdown
python3 summarise-load.py --check

Output — what you should see

==> x label(s) in labbook.md
    llama-baseline               llama.cpp  mixed          levels 1,5,10,20  last 2026-xx-xxTxx:xx:xx+xxxx
    llama-prefix-unique          llama.cpp  unique-prefix  levels 1,10  last ...
    ...
| c | ok | out tok/s | scaling | req/s | goodput | TTFT p50 | TTFT p90 | TPOT p50 | TPOT p90 | in flight | cached |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 40/40 | xx.x | 1.00 | ... |
...
==> checking the notebook against the lab's validation rules
    PASS  a mixed-prompt sweep with every level of [1, 5, 10, 20] completed: llama-baseline, vllm-baseline
    PASS  llama-baseline: failed requests across all levels = 0
    PASS  llama-baseline: output tokens counted from the server's usage block
    PASS  llama-baseline: context recorded (complete)
    ...
    PASS  prefix pair present: shared ['llama-prefix-shared', 'vllm-prefix-shared'], unique ['llama-prefix-unique', 'vllm-prefix-unique']
    PASS  llama-prefix-shared c=1: mean cached tokens are xx% of the prompt
    ...
    xx pass, 0 warn, 0 fail

Paste the Markdown tables under a Part 9 heading in the notebook. Beneath them write, in your own words and with the numbers: the knee and operating point for each engine; the TTFT change prefix caching produced and the cached tokens behind it; the variant from Task 7 whose result surprised you; and the configuration you would run a service on, with the objective it meets. The capstone’s evidence collector reads the JSON lines, and the host, quant and context_length fields now carry the context that page asks for.

Validate the load generator’s denominator

Begin at one in-flight request. Confirm the response body, token accounting and latency fields before raising concurrency. Then use the same workload at each rung, preserving failures and completed request counts. The configured number of clients is not the number of successful simultaneous users.

Check the generator’s arrival model: a closed-loop client sends replacement work after a request finishes, which can reduce offered load when the server slows. Record this behaviour when interpreting the curve. Include the output-length distribution, since total completion time depends on how much text each model emits.

When the latency target is crossed, stop increasing load and inspect queueing, cache pressure and prefill interference. Choose an admission level below the unacceptable region and repeat a mixed short/long request workload. Finally stop the generator and verify the service returns to its idle state and answers a fresh smoke request. This demonstrates recovery from the experiment rather than leaving a saturated server behind. Keep the raw load file, server configuration and resulting admission recommendation for the gateway project.

Validation

Criterion	Command	Pass
A complete baseline curve with counts from the server	python3 summarise-load.py --check	PASS on the sweep, failed-requests, usage and context lines for llama-baseline
The prefix measurement is controlled	same	PASS prefix pair present and every shared-prefix line above 50 %, every control line below 10 %
llama-server allocated what Task 1 predicted	grep "llama_kv_cache: size" llama-server.log after restarting the baseline	the MiB figure from Task 1’s table for your tier
The comparison is at equal pool (S, N, X)	grep "GPU KV cache size" vllm-server.log from the equal-pool start	81,920 tokens and 20.00x
A second engine’s curve exists	python3 summarise-load.py --list	a vllm-baseline or mlx-baseline label with levels 1,5,10,20 (Track X without vLLM: state that in the notebook)
Each curve is summarised	grep -c '"record": "curve-summary"' labbook.md	2 or more (1 on Track X without vLLM)
Task 7’s variants ran	python3 summarise-load.py --list	llama-slots8, llama-ctx2048, llama-kv-alt

--check exits with status 1 when any line reads FAIL; WARN lines are worth a sentence in the notebook but do not fail the lab.

Expected outcome

A section of the notebook that describes your machine as a service: a throughput curve and a latency curve against concurrency for two engines, with medians and ninetieth percentiles; a fitted a and b for each with the ceiling and the knee they imply, set beside Task 1’s bandwidth ceiling; a controlled measurement of prefix caching in cached tokens and TTFT; three one-change llama-server experiments with predictions; and, on vLLM, the admission and memory-fraction readings.

The shapes the mechanism predicts, and which your numbers confirm or contradict:

• The first few concurrent requests are nearly free. Scaling above about 0.8 at c=5 with TPOT barely changed is the argument for a serving engine, and Task 1’s short-context ceiling says bandwidth allows it on every track.
• The curve bends well before twenty on a single accelerator, at the point where b × c becomes comparable to a. Twenty is chosen to take you past the knee, not because it is where to operate.
• The tail grows faster than the median as concurrency rises, which is why every table here has both.
• Prefix caching moves TTFT and leaves TPOT close to where it was; where TPOT also improved, the control’s long prefills were interfering with decodes.
• At equal pool the engines diverge as concurrency rises. Whether vLLM’s scheduler and block pool beat llama-server’s slots on your hardware, with your weight formats and prompts, is what the B/A column says; on Track X, with bf16 against Q4_K_M, the c=1 gap is mostly bytes per token.

Troubleshooting

Symptom	Cause	Fix
Every request fails: first error: stream produced no content deltas	The endpoint answered with no text: wrong --model, or a template failure	Compare --model with /v1/models (on Track M, the absolute path); read the server log for a template error
first error: ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 8080) (the errno differs on macOS)	Wrong port, or the server is still loading	llama-server 8080, vLLM 8000, MLX 8081; wait for listening on or Application startup complete.
first error: RequestFailed: HTTP 400 ... mentioning context or length	Prompt plus max_tokens exceeds the context per sequence, for example a prefix set (about 1,400 prompt tokens) against CTX_PER_SLOT=1024	Run the prefix sets only against 4,096 tokens per sequence
offloaded 0/37 layers or no GPU device in the load log	A CPU-only llama.cpp build, or LLAMA_BIN points at one	Rebuild with the GPU backend (Part 6); check --list-devices
llama-server exits during load after the KV cache line	The pool plus weights did not fit	Use the eight-bit lab.env for your tier, or lower SLOTS; record that it did not fit
vLLM: Free memory on device ... is less than desired GPU memory utilization	The fraction asks for more than is free at startup	Lower MEM_FRACTION_A/_B, close what holds memory, or set KV_BYTES
vLLM: No available memory for the cache blocks or To serve at least one request with the model's max seq len (4096) ...	Weights and activations used the whole fraction	Raise the fraction, or set KV_BYTES; on Track X check the GPU-visible cap (Part 5)
vLLM on the 16 GB tier rejects --kv-cache-dtype fp8	This GPU and attention backend combination lacks the kernel	Set VLLM_KV_DTYPE=auto and keep VLLM_KV_BYTES=6039797760: the log then reports 40,960 tokens and 10.00x; record that the pools differ
GPU KV cache size lower than Task 1 predicted with KV_BYTES set	A different block size or dtype than assumed	Check KV_DTYPE; record the logged tokens and compare at that pool
Throughput at c=20 below c=10	Past saturation; on vLLM, preemption	Record it; on vLLM check vllm:num_preemptions_total and the Preemptions: field
in flight well below c and a Python process at 100 % of a core	The client is the bottleneck	Run load-test.py on another machine through an SSH tunnel, ssh -N -L 8080:127.0.0.1:8080 you@server, so the server stays bound to 127.0.0.1, and repeat against http://127.0.0.1:8080/v1 on the client
Numbers vary by more than a few per cent between identical runs	Something else is using the machine, or it is thermally limited	Close everything, plug the laptop in, re-run; if the variance persists, record it
finish shows stop for some requests	The model finished thinking and answering within 128 tokens	Shorter outputs raise requests per second; compare output tokens per second instead, or raise --max-tokens
tokens from deltas instead of usage	The server ignored stream_options.include_usage	Counts are approximate; say so beside the numbers
cached shows None on vLLM	The server was started without --enable-prompt-tokens-details	Use this page’s serve scripts, which pass it

Cleanup

Stop the server in terminal 1 with Ctrl-C, then confirm nothing is left:

RunnableAll tracks

stop what the lab started

cd ~/llm-course
docker rm -f vllm-lab 2>/dev/null || true
for port in 8000 8080 8081; do
    curl --silent --max-time 2 --output /dev/null "http://127.0.0.1:$port/" && echo "port $port is still in use"
done; echo "port check finished"
ls -lh labbook.md llama-server.log vllm-server.log 2>/dev/null

Output — what you should see

port check finished
-rw-r--r-- 1 you you xxxK ... labbook.md
-rw-r--r-- 1 you you xxxK ... llama-server.log
-rw-r--r-- 1 you you xxxM ... vllm-server.log

Keep labbook.md, lab.env and both scripts: Part 20 and Part 22 run load-test.py against larger servers and compare against these numbers, Part 23 operates a service built on them, and the capstone collects them. The logs can go once their lines are in the notebook. Keep the model files if you have the disk; the gateway project on the next page loads the same GGUF.

⚠Warning

Removing a model directory deletes weights that took a long download to acquire, and the only way back is to download them again. If you need the space, remove one checkpoint rather than a tree (rm -r ~/models/Qwen/Qwen3-8B deletes only the 16.4 GB bf16 copy used on Track X), and note in the notebook which one, so that a later comparison does not silently become a different model.

What you learned

Objective	The observation that shows it	Recorded as
Predict what concurrency costs in memory	Whether each load log’s KV size matched Task 1’s pool, and whether vLLM reported 81,920 tokens at 20.00x	Task 2 and Task 8 log lines beside the prediction
Build and trust a load generator	fail 0, tokens from usage, finish length and in flight near c, and how closely vllm bench serve agreed	llama-baseline rows; bench figures
Read a throughput-against-latency curve	Scaling, the fitted a and b, the knee and the operating point, set against Task 1’s ceiling	curve-summary lines and the recording sheet
Measure prefix caching	Cached tokens and TTFT against a same-length control, and how the c=10 figures differ between per-slot and pooled caches	The two prefix comparisons
Relate batch size, context and memory	What fewer slots, less context per slot and the other cache type did to the KV size and the c=20 row; what --max-num-seqs did to the queue; how far each fraction moved the pool	llama-slots8, llama-ctx2048, llama-kv-alt, vllm-seqs8, the fraction readings
Compare two engines fairly	Equal pool, cap, context and client, and how B/A moves down the concurrency levels	The comparison table and its notes

Check your understanding

Question 1. Your output throughput rises sharply from concurrency 1 to 5, barely improves from 5 to 10, and falls from 10 to 20, while mean in flight stays close to each level. What is the most likely explanation for the fall?

• The load generator is broken
• Past the machine's knee the extra in-flight requests add scheduling and cache pressure without useful parallelism, and on vLLM preemption discards completed prefill work
• The model is too small
• The network is saturated

Show the answer and why

Answer: Past the machine's knee the extra in-flight requests add scheduling and cache pressure without useful parallelism, and on vLLM preemption discards completed prefill work

In-flight close to the level rules out the client. A falling curve past saturation is a real result. On vLLM 0.28.0 the confirmation is vllm:num_preemptions_total rising and a Preemptions field in the periodic stats line; the warning text quoted on the optimisation page does not appear in that version's scheduler source.

Question 2. Which of these llama-server starts gives twenty sequences 4,096 tokens each?

• llama-server --model qwen3-8b.gguf --parallel 20 --ctx-size 4096
• llama-server --model qwen3-8b.gguf --parallel 20 --ctx-size 81920
• llama-server --model qwen3-8b.gguf --parallel 1 --ctx-size 81920
• llama-server --model qwen3-8b.gguf --ctx-size 4096

Show the answer and why

Answer: llama-server --model qwen3-8b.gguf --parallel 20 --ctx-size 81920

--ctx-size is the total across slots. The first command leaves each of twenty slots about 200 tokens; the v0.4.0 build pads that to 256 and prints n_ctx_slot = 256 in the load log, and a prompt from the shared-prefix set would be rejected. The third gives one slot 81,920 tokens, so twenty requests queue behind it.

Question 3. Qwen3-8B caches 147,456 bytes per token at f16. You change a llama-server start from 20 slots of 4,096 tokens to 20 slots of 8,192 tokens. What happens to the KV allocation in the load log?

• Nothing, because the cache is paged
• It doubles, from 11520.00 MiB to 23040.00 MiB, allocated at load whether or not requests use it
• It grows only when a request uses more than 4,096 tokens
• It halves, because each slot is used less

Show the answer and why

Answer: It doubles, from 11520.00 MiB to 23040.00 MiB, allocated at load whether or not requests use it

20 × 8,192 × 147,456 bytes is 24,159,191,040 bytes, 23,040 MiB. llama-server with explicit --parallel allocates every slot whole at load, which is why Task 7 halved the context per slot and saw almost no change in throughput: the short prompts never used the reservation. vLLM's block pool is also reserved at startup, but the tokens in it can go to any request.

Question 4. After switching from the unique-prefix control to the shared-prefix set, llama-server's mean cached tokens at concurrency 10 are clearly lower than at concurrency 1, and TTFT p90 at concurrency 10 barely improved. What explains it?

• Prefix caching is off
• Each llama-server slot keeps only its own last prompt, so the requests that land on slots which never saw the preamble prefill it in full, and they are the tail
• The preamble is too long to cache
• Concurrency disables caching

Show the answer and why

Answer: Each llama-server slot keeps only its own last prompt, so the requests that land on slots which never saw the preamble prefill it in full, and they are the tail

The median improves because most requests hit a warm slot; the ninetieth percentile is the cold ones. vLLM hashes blocks into one shared pool, so after the first request any request can reuse the preamble, which is why its cached mean at concurrency 10 should stay close to its concurrency 1 value.

Question 5. Your fit gives TPOT p50 ≈ 25 ms + 2 ms × c. What does it predict?

• Output throughput approaches but never exceeds 1 ÷ 2 ms, about 500 tokens every second, however many requests are admitted
• Median TPOT reaches a 0.1 s objective at about c = 37
• At c = 20 the decode rate is 20 ÷ 65 ms, about 308 tokens every second across all requests
• Doubling the slots doubles the ceiling

Show the answer and why

Answer: Output throughput approaches but never exceeds 1 ÷ 2 ms, about 500 tokens every second, however many requests are admitted, Median TPOT reaches a 0.1 s objective at about c = 37, At c = 20 the decode rate is 20 ÷ 65 ms, about 308 tokens every second across all requests

Throughput is c ÷ (a + b·c), which tends to 1 ÷ b; (100 − 25) ÷ 2 is 37.5; 20 ÷ 0.065 s is 307.7. Slots raise the admission limit, not b, which is the per-sequence cost of a step, so they cannot move the ceiling. Measured throughput sits below these figures because it also pays for prefill and queueing.

Question 6. vLLM on a 24 GB card logs "GPU KV cache size: 50,960 tokens, Maximum concurrency for 4,096 tokens per request: 12.44x" and you run the concurrency-20 level. What should you expect?

• Twenty requests decode together, because --max-num-seqs is 20
• All twenty are admitted while their caches are short, but as they grow the pool cannot hold twenty full-length sequences, so the scheduler preempts or holds requests; with 128-token answers to short prompts it may never reach that point
• The server refuses to start
• Throughput doubles compared with 20.00x

Show the answer and why

Answer: All twenty are admitted while their caches are short, but as they grow the pool cannot hold twenty full-length sequences, so the scheduler preempts or holds requests; with 128-token answers to short prompts it may never reach that point

Maximum concurrency is computed for sequences at the full 4,096 tokens. The mixed prompts use under 200 tokens each, so twenty of them occupy a small share of 50,960 tokens and nothing is preempted. The shared-prefix set at 1,400 tokens plus output is where a pool this size starts to matter, which is why the equal-pool comparison sets the bytes explicitly.

Check my answersClear