Lab: Two-Machine Prefill and Decode with vLLM

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The versions, connectors and measurements each track was run with will be recorded here when the validation pass is done, and the pending table below is where they will land.

Objective

By the end of this lab you will have run one request through two machines, with the prompt read on one and the answer written on the other, and you will have four numbers proving what that cost or saved: time to first token and time per output token for the split pair, and the same two for the identical model on a single machine. You will also have a byte count from the network interface showing that a key-value cache actually crossed the link, of roughly the size the arithmetic in Lesson 2 predicted.

The deliverable is the comparison, not the split. On most home networks the split will lose, and a notebook line that says so, with the connector, the link and the payload size beside it, is exactly as valuable as one that says it won. What you must not end up with is a pair that appears to work and is quietly prefilling every prompt twice.

⌂Single-machine path

The primary path needs two machines with 24 GB each. If you have one, take the single-machine path: two vLLM processes on one device with the memory divided between them, running a smaller model. Every configuration step, every failure mode and every notebook entry is the same. What it cannot measure is a network, because the transfer crosses the loopback interface. LMCache’s own documentation says as much about this arrangement, calling single-node mode functional testing only because loopback is not RDMA. Record that you took this path; the honesty is the point.

Architecture

What you are building

• clientThe load generatorPart 9's load-test.py, run on either machine or a third one
• routerThe proxyvLLM's disagg_proxy_demo.py; calls prefill then decode; the only port a client talks to
• prefillPrefill hostvllm serve with kv_role kv_producer; the machine with more compute
• decodeDecode hostvllm serve with kv_role kv_consumer; the machine with more memory
• cacheShared mountthe key-value store for the shared connector, and the model library, both from Part 18

• The load generator ⟷ connected to The proxyhouse network; requests in, tokens out
• The proxy ⟷ connected to Prefill hostthe prompt, first
• The proxy ⟷ connected to Decode hostthe same request, second
• Prefill host ⟷ connected to Shared mountwrites one request's key-value blocks
• Shared mount ⟷ connected to Decode hostreads them back; this is the transfer you measure
• Shared mount ⟷ connected to Prefill hostmodel weights, at load time only

Both machines hold the whole model. The proxy sends the prompt twice and only the first call prefills it. On the point-to-point connectors the store disappears from this picture and the prefill host writes straight to the decode host, which is a shorter path and a longer configuration.

Choosing which machine takes which role is the first task and it follows from Lesson 1. Prefill is compute-bound and needs room for the prompts it is reading right now; decode is bandwidth-bound and holds every conversation in flight. So the machine with the most arithmetic throughput and the least memory prefills, and the machine with the most memory decodes. On the reference cluster that means an NVIDIA desktop or a DGX Spark for prefill and a Spark or the Ryzen box for decode.

Requirements

Every track needs the lab notebook from Part 1, Part 9’s load-test.py, the shared mount and the measured link figures from Part 18’s lab, and about ninety minutes, of which roughly sixty are attended. The model download is unattended; start it before you read the tasks.

Track S — NVIDIA DGX Spark

Primary path. Two machines with 24 GB or more each, at least one of which is a DGX Spark, with vLLM installed as in Part 9 on both. If both are Sparks with the ConnectX-7 cable from Part 20, you have the only link in this course where the transfer arithmetic favours a split, and you should run the lab twice: once with the shared connector and once with nixl over the cable. Around 20 GB of disk for Qwen3-8B at bf16, on the shared library both machines read.

A single Spark can also take the single-machine path, with two processes sharing the 128 GB.

Track X — AMD Ryzen AI Max+ 395Partial

vLLM's GPU installation page lists Ryzen AI MAX and AI 300 among its ROCm targets with pre-built wheels, but this course has not exercised that path, and these machines usually have 2.5 gigabit Ethernet, on which Lesson 2's arithmetic predicts the split will lose.

Run it if your vLLM installation works, and use CONNECTOR=shared: the point-to-point connectors want RDMA and this hardware normally does not have it. Expect the comparison to come out against the split and record it that way. The Ryzen box is an excellent decode host in a mixed pair with an NVIDIA machine doing prefill, and that is the arrangement worth trying.

If vLLM does not run for you, take Track M’s reduced path below on this machine and say so in the notebook.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS, so neither instance in a disaggregated pair can run with GPU acceleration on Apple silicon.

Reduced path. You cannot disaggregate, and the substitute is to do the thing disaggregation protects: run several conversations at once on llama-server with enough slots that a long prompt in one does not stall the others, then measure what happens as you take the slots away. That is a real experiment about prefill interference and it produces comparable notebook lines.

Use llama-cache-tiers.sh serve from the next lab’s files with LLAMA_PARALLEL set to 4, run the load generator against it at concurrency 4, then restart with LLAMA_PARALLEL set to 1 and run it again. The difference is the interference this whole part is about, measured on hardware that cannot remove it. Part 21 is where an Apple cluster is built.

Track N — NVIDIA desktop or laptop

Primary path. Two machines with 24 GB or more of device memory each, or one machine with two such cards, vLLM installed as in Part 9, and the link between them measured in Part 18. This is the track most likely to be on ordinary Ethernet, so it is also the track where Lesson 2’s prediction is most testable: work out the expected transfer time before you start and see whether your measurement matches.

On one machine with two cards, set CUDA_VISIBLE_DEVICES to a different card for each instance and run both scripts locally. That is a legitimate two-instance deployment whose transfer crosses the host rather than a network, and it sits between the two paths above.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-22-disaggregated-serving"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-22-disaggregated-serving and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Do the arithmetic before you install anything

Ten minutes here decides what the rest of the lab means.

Look up your model’s bytes per token on the model reference. For Qwen3-8B, which is Apache-2.0 and ungated, it is 147,456. Multiply by the prompt length you intend to test at, which in this lab is roughly 4,096 tokens for the shared-prefix prompt set. Divide the result by the throughput Part 18’s lab measured on the link between your two machines, in bytes per second.

Write three numbers in the notebook now, before any software is configured:

1. The predicted payload for one request.
2. The predicted transfer time on your measured link.
3. Your prediction for whether time to first token will improve or get worse.

∴Research note

Writing the prediction down first is not ceremony. A measurement you have already predicted teaches you something whichever way it comes out; a measurement you interpret afterwards mostly confirms whatever you already believed. Part 18’s lesson made the same argument about link speeds and this is the same discipline applied to a system.

2. Fill in the settings file

Every script in this part reads one file. Copy it, fill in your own machines, and keep it out of version control: every value in it is a fact about your house.

Fragment — not complete on its own

env-example.txt

# Purpose: every setting the Part 22 scripts read. Copy this file to `.env` beside the
#          scripts and fill in the empty lines for your own machines. Nothing here is a
#          secret except the Hugging Face token, which you generate rather than copy and
#          which is better exported in your shell than written here.
# Platform: all
# Minimum memory: 24 GB per machine for the two-machine path; 16 GB for the single-machine
#          path and for the offload lab
# Assumes: `cp env-example.txt .env`, then an editor. Every script loads it with
#          `set -a; . ./.env; set +a` when the file is present, so a value already set in
#          your shell always wins over a value in the file.
#
# Part 18's own .env already holds CLUSTER_IFACE, CLUSTER_PEERS, MODELS_DIR and LABBOOK.
# The names below are deliberately identical: copy your values across, or source Part 18's
# .env first and this one after it, and delete the duplicated lines.

# ------------------------------------------------------------------- the two machines
# The name of the machine that will do PREFILL, and the name of the machine that will do
# DECODE, as the other machines can resolve them. RFC 8375 reserves everything under
# `home.arpa` for names that mean something inside one house and nothing outside it, so
# names like `node-a.home.arpa` are exactly the intended use. Where a machine has a direct
# cable as well as a house connection, use its "-direct" name here: the name you type is
# what decides which cable the key-value cache crosses.
#
#   PREFILL_HOST=node-a-direct.home.arpa
#   DECODE_HOST=node-b-direct.home.arpa
#
# Leave BOTH empty for the single-machine path; the scripts then use the loopback address
# and say so in the notebook line.
PREFILL_HOST=
DECODE_HOST=

# The address THIS machine advertises for the connector handshake, when you are using a
# point-to-point connector. It must be an address the other machine can reach, which on a
# cluster with a direct cable is the address on the cable and not the one on the switch.
# Leave empty on the single-machine path.
SIDE_CHANNEL_ADDR=

# Distinct handshake ports. Two instances on ONE host must not share one.
PREFILL_SIDE_CHANNEL_PORT=5600
DECODE_SIDE_CHANNEL_PORT=5601

# The interface that carries cluster traffic on THIS machine, from Part 18's lab. The
# scripts read its byte counters around each load test, and pass it to the transfer
# library so that it does not pick the management interface on its own.
#   Linux   ip -br addr           e.g. enp1s0f1np1, eno1, enp5s0
#   macOS   networksetup -listallhardwareports   e.g. en0, en5
CLUSTER_IFACE=

# ---------------------------------------------------------------------- the connector
# Which connector the two instances use. The scripts accept:
#   shared    ExampleConnector over a directory both machines can read and write.
#             Needs no RDMA, no UCX and no handshake. Start here.
#   nixl      NixlConnector, point to point over UCX. Wants RDMA to be worth doing.
#   mooncake  MooncakeConnector. Needs the mooncake-transfer-engine package.
CONNECTOR=shared

# For CONNECTOR=shared only: a directory BOTH machines can read and write, on the shared
# mount from Part 18. It fills up with key-value blocks; put it somewhere you can delete.
KV_SHARED_PATH=

# ---------------------------------------------------------------------------- models
# The model both instances load. They must load the SAME one: two instances that disagree
# about the model, the quantisation or the block size produce blocks the other cannot use,
# and the symptom is a silent miss rather than an error.
# Qwen3-8B is Apache-2.0 and ungated; at bf16 it needs roughly 17 GB for weights alone, so
# use it on the 24 GB path and drop to Qwen3-1.7B for two processes on one small device.
MODEL=Qwen/Qwen3-8B

# The smaller model for the single-machine path, where one device is split between two
# engine processes. Qwen3-1.7B is Apache-2.0.
SMALL_MODEL=Qwen/Qwen3-1.7B

# The name clients send in the "model" field. Keep it identical on every path so the load
# generator's command line never changes between runs.
SERVED_NAME=local-chat

# Where models are cached on this machine. Part 18's shared library, or a local path.
HF_HOME=

# --------------------------------------------------------------------------- serving
# Ports. The proxy is the only one a client should ever talk to.
PREFILL_PORT=8100
DECODE_PORT=8200
PROXY_PORT=8000

# Where the engines bind. Keep this on the loopback address or on your cluster address;
# nothing in this part authenticates anything.
SERVE_HOST=127.0.0.1

# Context length to allocate, and how many sequences may be in flight. Both instances must
# use the same context length.
CTX=8192
MAX_SEQS=8

# Fraction of the device each instance may claim. On the single-machine path the two
# processes share one device, so this must be well below half for each of them.
MEM_FRACTION=0.85
SPLIT_MEM_FRACTION=0.40

# ------------------------------------------------------------- the offload lab (lab 2)
# Gibibytes of host memory the engine may use as a key-value tier. Start small: this is
# host memory taken away from everything else on the machine.
KV_OFFLOAD_GB=8

# Backend for that tier, as the vllm serve CLI reference documents it: "native" for vLLM's
# own host-memory offloading, or "lmcache" if you have installed LMCache.
KV_OFFLOAD_BACKEND=native

# For the llama.cpp reduced path on Tracks X and M: the GGUF file, the number of slots and
# where saved slot caches are written.
LLAMA_MODEL=
LLAMA_PORT=8080
LLAMA_PARALLEL=2
LLAMA_SLOT_SAVE_PATH=./slot-cache
LLAMA_CACHE_TYPE=q8_0

# ------------------------------------------------------------------ the load generator
# Part 9's load-test.py. Give the path to it; the wrapper refuses to run without it rather
# than reimplementing a load generator that already exists.
LOAD_TEST=../part-09-vllm-and-sglang/load-test.py

# Concurrency levels and requests per level for every run in this part. Keep them the same
# across runs or the comparison means nothing.
CONCURRENCY=1,4,8
REQUESTS=24
MAX_TOKENS=128

# The prompt set. "shared-prefix" is the workload a phase split is aimed at: a long shared
# preamble with a short different question. "mixed" is the control.
PROMPT_SET=shared-prefix

# Where every script appends its JSON line. The lab notebook from Part 1.
LABBOOK=labbook.md

# --------------------------------------------------------------------------- secrets
# Gated models need a Hugging Face token. Export it in your shell rather than writing it
# here if you can; if you do put it here, keep .env out of version control.
HF_TOKEN=

# The load generator reads an API key, if your server needs one, from the variable named
# here. It is never written to the notebook.
LOADTEST_API_KEY_ENV=LOADTEST_API_KEY

Download env-example.txt142 lines

RunnableAll tracks

make the settings file, on both machines

cp env-example.txt .env

Three lines decide the topology. PREFILL_HOST and DECODE_HOST are names, not addresses, and where a machine has a direct cable as well as a house connection you use its -direct name, because the name you type is what chooses the cable. CONNECTOR starts at shared. KV_SHARED_PATH is a directory on the mount both machines can write to.

Leave PREFILL_HOST and DECODE_HOST empty for the single-machine path, and export SINGLE_MACHINE=1 before running the serve scripts; they then use the smaller model and a memory fraction that lets two processes fit on one device.

3. Start the prefill instance

On the machine you chose for prefill.

RunnableAll tracks

serve-prefill.sh

#!/usr/bin/env bash
# Purpose: start the vLLM instance that reads prompts and writes their key-value blocks
#          out through the chosen connector, printing the whole configuration first so the
#          run is reproducible from the terminal log alone
# Platform: spark, nvidia (vLLM's GPU path; Track X only where your ROCm build works)
# Minimum memory: 24 GB for Qwen3-8B at bf16 on the two-machine path; use SMALL_MODEL and
#          SPLIT_MEM_FRACTION for two processes on one 16 GB device
# Assumes: vLLM installed as in Part 9 and on PATH; a .env copied from env-example.txt and
#          filled in; for CONNECTOR=shared, KV_SHARED_PATH exists and is writable from
#          BOTH machines; nothing else listening on PREFILL_PORT
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${MODEL:-Qwen/Qwen3-8B}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
PREFILL_PORT="${PREFILL_PORT:-8100}"
CTX="${CTX:-8192}"
MAX_SEQS="${MAX_SEQS:-8}"
MEM_FRACTION="${MEM_FRACTION:-0.85}"
CONNECTOR="${CONNECTOR:-shared}"
KV_SHARED_PATH="${KV_SHARED_PATH:-}"
SIDE_CHANNEL_ADDR="${SIDE_CHANNEL_ADDR:-}"
PREFILL_SIDE_CHANNEL_PORT="${PREFILL_SIDE_CHANNEL_PORT:-5600}"
CLUSTER_IFACE="${CLUSTER_IFACE:-}"
SPLIT="${SPLIT_MEM_FRACTION:-0.40}"
SINGLE_MACHINE="${SINGLE_MACHINE:-0}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Install it as Part 9 describes."

if [ "$SINGLE_MACHINE" = "1" ]; then
    MODEL="${SMALL_MODEL:-Qwen/Qwen3-1.7B}"
    MEM_FRACTION="$SPLIT"
    printf '    NOTE: single-machine path. Using %s at memory fraction %s so that two\n' "$MODEL" "$MEM_FRACTION"
    printf '          engine processes fit on one device. Record this in the notebook: the\n'
    printf '          transfer crosses the loopback interface, which measures no network.\n\n'
fi

# --- build the connector configuration ------------------------------------------------
case "$CONNECTOR" in
    shared)
        [ -n "$KV_SHARED_PATH" ] || fail "CONNECTOR=shared needs KV_SHARED_PATH set to a directory both machines can write."
        mkdir -p "$KV_SHARED_PATH"
        [ -w "$KV_SHARED_PATH" ] || fail "KV_SHARED_PATH ($KV_SHARED_PATH) is not writable by this account."
        KV_CONFIG="{\"kv_connector\":\"ExampleConnector\",\"kv_role\":\"kv_producer\",\"kv_connector_extra_config\":{\"shared_storage_path\":\"${KV_SHARED_PATH}\"}}"
        ;;
    nixl)
        [ -n "$SIDE_CHANNEL_ADDR" ] || fail "CONNECTOR=nixl needs SIDE_CHANNEL_ADDR: the address the OTHER machine can reach this one on."
        KV_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
        export VLLM_NIXL_SIDE_CHANNEL_HOST="$SIDE_CHANNEL_ADDR"
        export VLLM_NIXL_SIDE_CHANNEL_PORT="$PREFILL_SIDE_CHANNEL_PORT"
        [ -n "$CLUSTER_IFACE" ] && export UCX_NET_DEVICES="$CLUSTER_IFACE"
        ;;
    mooncake)
        [ -n "$SIDE_CHANNEL_ADDR" ] || fail "CONNECTOR=mooncake needs SIDE_CHANNEL_ADDR."
        KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
        ;;
    *)
        fail "CONNECTOR must be one of: shared, nixl, mooncake. Got '${CONNECTOR}'."
        ;;
esac

cat <<INFO
==> vLLM prefill instance (the producer)
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${PREFILL_PORT}
    connector         ${CONNECTOR}
    kv role           kv_producer
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}

    Two things to copy into the notebook from the startup log: the key-value cache size
    the engine settled on, and the maximum concurrency that implies at this context
    length. The decode instance must be started with the SAME model and the SAME context
    length, or its blocks will not match these and every request will silently re-prefill.

    --kv-transfer-config is documented on vLLM's disaggregated prefilling page. If your
    build rejects it, run "vllm serve --help | grep kv" and record what it does accept.

INFO

exec vllm serve "$MODEL" \
    --host "$SERVE_HOST" \
    --port "$PREFILL_PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    --gpu-memory-utilization "$MEM_FRACTION" \
    --kv-transfer-config "$KV_CONFIG"

Download serve-prefill.sh100 lines

RunnableAll tracks

the producer

bash serve-prefill.sh

Output — what you should see

==> vLLM prefill instance (the producer)
    model             Qwen/Qwen3-8B
    served as         local-chat
    listening on      http://127.0.0.1:8100
    connector         shared
    kv role           kv_producer
    max model length  8192
    ...
INFO ... Application startup complete.

Copy two lines out of the startup log into the notebook: the key-value cache size the engine settled on, and the maximum concurrency it reports at your context length. Part 9 taught you to read them and they matter more here, because the decode instance has to agree with this one about the model and the context length or the blocks it receives will not be blocks it can use.

4. Start the decode instance

On the machine you chose for decode, in its own terminal.

RunnableAll tracks

serve-decode.sh

#!/usr/bin/env bash
# Purpose: start the vLLM instance that receives key-value blocks from the prefill
#          instance and generates the answer, with the same model and context length so
#          that the blocks it receives are blocks it can use
# Platform: spark, nvidia (vLLM's GPU path; Track X only where your ROCm build works)
# Minimum memory: 24 GB for Qwen3-8B at bf16 on the two-machine path; use SMALL_MODEL and
#          SPLIT_MEM_FRACTION for two processes on one 16 GB device
# Assumes: vLLM installed as in Part 9 and on PATH; a .env copied from env-example.txt and
#          filled in with the SAME MODEL and CTX the prefill instance uses; for
#          CONNECTOR=shared, KV_SHARED_PATH readable from this machine; nothing else
#          listening on DECODE_PORT
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${MODEL:-Qwen/Qwen3-8B}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
DECODE_PORT="${DECODE_PORT:-8200}"
CTX="${CTX:-8192}"
MAX_SEQS="${MAX_SEQS:-8}"
MEM_FRACTION="${MEM_FRACTION:-0.85}"
CONNECTOR="${CONNECTOR:-shared}"
KV_SHARED_PATH="${KV_SHARED_PATH:-}"
SIDE_CHANNEL_ADDR="${SIDE_CHANNEL_ADDR:-}"
DECODE_SIDE_CHANNEL_PORT="${DECODE_SIDE_CHANNEL_PORT:-5601}"
CLUSTER_IFACE="${CLUSTER_IFACE:-}"
SPLIT="${SPLIT_MEM_FRACTION:-0.40}"
SINGLE_MACHINE="${SINGLE_MACHINE:-0}"
MONOLITHIC="${MONOLITHIC:-0}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Install it as Part 9 describes."

if [ "$SINGLE_MACHINE" = "1" ]; then
    MODEL="${SMALL_MODEL:-Qwen/Qwen3-1.7B}"
    MEM_FRACTION="$SPLIT"
fi

# The baseline run: one ordinary server, no connector, everything on one machine. This is
# the number the disaggregated pair is compared against, and it must use the same model,
# the same context length and the same served name so that only one thing differs.
if [ "$MONOLITHIC" = "1" ]; then
    cat <<INFO
==> vLLM monolithic baseline (no connector, no split)
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${DECODE_PORT}
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}

    Point the load generator at THIS port for the baseline, and at the proxy port for the
    disaggregated run. Nothing else about the two runs should differ.

INFO
    exec vllm serve "$MODEL" \
        --host "$SERVE_HOST" \
        --port "$DECODE_PORT" \
        --served-model-name "$SERVED_NAME" \
        --max-model-len "$CTX" \
        --max-num-seqs "$MAX_SEQS" \
        --gpu-memory-utilization "$MEM_FRACTION"
fi

case "$CONNECTOR" in
    shared)
        [ -n "$KV_SHARED_PATH" ] || fail "CONNECTOR=shared needs KV_SHARED_PATH set to the SAME directory the prefill instance writes to."
        [ -d "$KV_SHARED_PATH" ] || fail "KV_SHARED_PATH ($KV_SHARED_PATH) is not a directory on this machine. Is the shared mount up?"
        [ -r "$KV_SHARED_PATH" ] || fail "KV_SHARED_PATH ($KV_SHARED_PATH) is not readable by this account."
        KV_CONFIG="{\"kv_connector\":\"ExampleConnector\",\"kv_role\":\"kv_consumer\",\"kv_connector_extra_config\":{\"shared_storage_path\":\"${KV_SHARED_PATH}\"}}"
        ;;
    nixl)
        [ -n "$SIDE_CHANNEL_ADDR" ] || fail "CONNECTOR=nixl needs SIDE_CHANNEL_ADDR: the address the OTHER machine can reach this one on."
        KV_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
        export VLLM_NIXL_SIDE_CHANNEL_HOST="$SIDE_CHANNEL_ADDR"
        export VLLM_NIXL_SIDE_CHANNEL_PORT="$DECODE_SIDE_CHANNEL_PORT"
        [ -n "$CLUSTER_IFACE" ] && export UCX_NET_DEVICES="$CLUSTER_IFACE"
        ;;
    mooncake)
        [ -n "$SIDE_CHANNEL_ADDR" ] || fail "CONNECTOR=mooncake needs SIDE_CHANNEL_ADDR."
        KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
        ;;
    *)
        fail "CONNECTOR must be one of: shared, nixl, mooncake. Got '${CONNECTOR}'."
        ;;
esac

cat <<INFO
==> vLLM decode instance (the consumer)
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${DECODE_PORT}
    connector         ${CONNECTOR}
    kv role           kv_consumer
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}

    This instance is where the measurement lives. Its /metrics endpoint carries
    vllm:time_to_first_token_seconds, vllm:prefix_cache_hits and
    vllm:prefix_cache_queries, and a hit rate near zero here means the blocks are not
    arriving: the instance is quietly prefilling every prompt itself and returning
    correct answers, which is the failure this lab is designed to catch.

INFO

exec vllm serve "$MODEL" \
    --host "$SERVE_HOST" \
    --port "$DECODE_PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    --gpu-memory-utilization "$MEM_FRACTION" \
    --kv-transfer-config "$KV_CONFIG"

Download serve-decode.sh122 lines

RunnableAll tracks

the consumer

bash serve-decode.sh

The same script serves the baseline later, which is deliberate: one script, one set of flags, one variable changed, so that the two runs differ in exactly one thing.

✕Common mistake

The commonest configuration error here is two instances that disagree. A different model repository, a different --max-model-len, a different quantisation: any of them produces blocks the other cannot use, and none of them produces an error. Both scripts read the same .env, which is why they read the same .env. If you edit one machine’s copy, edit the other’s.

5. Get the proxy and start it

The proxy is not installed by pip install vllm. It lives in the vLLM source tree at examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py, and the script below either uses the copy you already have or fetches that one file from the tag you name.

RunnableAll tracks

run-proxy.sh

#!/usr/bin/env bash
# Purpose: run vLLM's own example disaggregated-prefill proxy in front of the prefill and
#          decode instances, after checking that both are answering, and tell you exactly
#          where to obtain the proxy if you do not have it
# Platform: all (the proxy is plain Python; it may run on any machine that can reach both
#          instances, including your workstation)
# Minimum memory: 1 GB; the proxy holds no model
# Assumes: python3 and curl on PATH; serve-prefill.sh and serve-decode.sh already running
#          and answering /v1/models; a .env copied from env-example.txt. The proxy file is
#          examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py in the vLLM
#          repository; it is NOT installed by `pip install vllm`.
#
# Usage: bash run-proxy.sh            start the proxy (the file must already be here)
#        bash run-proxy.sh --fetch    download the proxy from the tag in VLLM_TAG first
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

SERVED_NAME="${SERVED_NAME:-local-chat}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
PREFILL_HOST="${PREFILL_HOST:-127.0.0.1}"
DECODE_HOST="${DECODE_HOST:-127.0.0.1}"
PREFILL_PORT="${PREFILL_PORT:-8100}"
DECODE_PORT="${DECODE_PORT:-8200}"
PROXY_PORT="${PROXY_PORT:-8000}"
VLLM_TAG="${VLLM_TAG:-main}"
PROXY_FILE="${PROXY_FILE:-${HERE}/disagg_proxy_demo.py}"

REPO_PATH="examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py"
RAW_BASE="https://raw.githubusercontent.com/vllm-project/vllm"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v python3 >/dev/null 2>&1 || fail "python3 is not on PATH."
command -v curl >/dev/null 2>&1 || fail "curl is not on PATH."

if [ "${1:-}" = "--fetch" ]; then
    printf '==> Fetching the proxy from the vLLM repository\n'
    printf '    tag   %s\n' "$VLLM_TAG"
    printf '    path  %s\n' "$REPO_PATH"
    printf '    into  %s\n\n' "$PROXY_FILE"
    printf '    Use the SAME tag as the vLLM you installed. A proxy from a different\n'
    printf '    version than the engines is a supported way to waste an evening.\n\n'
    curl -fsSL "${RAW_BASE}/${VLLM_TAG}/${REPO_PATH}" -o "$PROXY_FILE"
    printf '    Downloaded. Read it before running it; it is a demonstration file with no\n'
    printf '    authentication of any kind.\n\n'
fi

if [ ! -f "$PROXY_FILE" ]; then
    cat >&2 <<MISSING
The proxy is not here and it is not installed by the vLLM wheel. It lives in the
vLLM source tree at:

    ${REPO_PATH}

Get it in whichever way suits you, at the SAME tag as your installed vLLM:

    git clone --depth 1 --branch <your vllm tag> https://github.com/vllm-project/vllm
    cp vllm/${REPO_PATH} ${PROXY_FILE}

or let this script fetch just that file:

    VLLM_TAG=<your vllm tag> bash run-proxy.sh --fetch

Then run this script again.
MISSING
    exit 1
fi

# --- both instances must be answering before the proxy is worth starting ---------------
check_instance() {
    local role="$1" host="$2" port="$3"
    if curl -fsS --max-time 5 "http://${host}:${port}/v1/models" >/dev/null 2>&1; then
        printf '    OK: the %s instance answers on %s:%s\n' "$role" "$host" "$port"
    else
        fail "The ${role} instance is not answering on ${host}:${port}. Start it first."
    fi
}

printf '==> Checking both instances\n'
check_instance prefill "$PREFILL_HOST" "$PREFILL_PORT"
check_instance decode "$DECODE_HOST" "$DECODE_PORT"
printf '\n'

cat <<INFO
==> Proxy
    model     ${SERVED_NAME}
    prefill   ${PREFILL_HOST}:${PREFILL_PORT}
    decode    ${DECODE_HOST}:${DECODE_PORT}
    listening http://${SERVE_HOST}:${PROXY_PORT}

    Send every measured request to the proxy port and nothing to the two instances
    directly. The proxy calls prefill first and decode second; if you see requests
    arriving only at the decode instance, you have measured a single-machine deployment
    wearing a disaggregated hat.

    No authentication, on any of the three. Keep them on your cluster network and put
    Part 9's gateway in front if anything outside this machine will reach them.

INFO

exec python3 "$PROXY_FILE" \
    --model "$SERVED_NAME" \
    --prefill "${PREFILL_HOST}:${PREFILL_PORT}" \
    --decode "${DECODE_HOST}:${DECODE_PORT}" \
    --port "$PROXY_PORT"

Download run-proxy.sh112 lines

RunnableAll tracks

fetch the proxy at the tag matching your installed vLLM, then run it

VLLM_TAG=v0.28.0 bash run-proxy.sh --fetch

Output — what you should see

==> Checking both instances
    OK: the prefill instance answers on node-a-direct.home.arpa:8100
    OK: the decode instance answers on node-b-direct.home.arpa:8200

==> Proxy
    model     local-chat
    prefill   node-a-direct.home.arpa:8100
    decode    node-b-direct.home.arpa:8200
    listening http://127.0.0.1:8000

Use the tag that matches the vLLM you installed. This course pins vLLM 0.28.0 · verified 2026-09-08; a proxy from one version driving engines from another is a documented way to lose an evening.

RunnableAll tracks

one request through the pair, before any measurement

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":32}'

=Expected result

Generated text comes back, and the prefill instance’s log shows a request arriving a moment before the decode instance’s does. If only the decode instance logs anything, the proxy is not calling both, and everything you measure afterwards will be a single-machine deployment with extra steps.

6. Load-test the pair

The generator is Part 9’s load-test.py and this lab does not replace it. The wrapper below reads your settings, reads the interface byte counters either side of the run, calls the generator, and appends a notebook line saying how many bytes crossed the link.

RunnableAll tracks

run-load.sh

#!/usr/bin/env bash
# Purpose: run Part 9's load generator against one endpoint with the settings from .env,
#          reading this machine's cluster-interface byte counters either side of the run,
#          and append one notebook line saying how many bytes crossed the link to produce
#          those tokens
# Platform: all (Linux reads /proc/net/dev; macOS reads netstat -ib)
# Minimum memory: 1 GB on the machine running the generator; it may be a third machine
# Assumes: python3 on PATH; LOAD_TEST in .env pointing at Part 9's load-test.py; a server
#          already answering at the URL given; CLUSTER_IFACE naming an interface on THIS
#          machine. Counters cover the whole interface, so run it on a quiet cluster and
#          read the byte figures as an order of magnitude, not an exact accounting.
#
# Usage: bash run-load.sh disagg   http://127.0.0.1:8000/v1
#        bash run-load.sh baseline http://127.0.0.1:8200/v1
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

LABEL="${1:-}"
BASE_URL="${2:-}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

[ -n "$LABEL" ] || fail "Usage: bash run-load.sh <label> <base-url ending in /v1>"
[ -n "$BASE_URL" ] || fail "Usage: bash run-load.sh <label> <base-url ending in /v1>"

LOAD_TEST="${LOAD_TEST:-../part-09-vllm-and-sglang/load-test.py}"
case "$LOAD_TEST" in
    /*) ;;
    *) LOAD_TEST="${HERE}/${LOAD_TEST}" ;;
esac
[ -f "$LOAD_TEST" ] || fail "LOAD_TEST does not point at a file: ${LOAD_TEST}. Set it in .env to Part 9's load-test.py."

command -v python3 >/dev/null 2>&1 || fail "python3 is not on PATH."

SERVED_NAME="${SERVED_NAME:-local-chat}"
CONCURRENCY="${CONCURRENCY:-1,4,8}"
REQUESTS="${REQUESTS:-24}"
MAX_TOKENS="${MAX_TOKENS:-128}"
PROMPT_SET="${PROMPT_SET:-shared-prefix}"
LABBOOK="${LABBOOK:-labbook.md}"
CLUSTER_IFACE="${CLUSTER_IFACE:-}"
CONNECTOR="${CONNECTOR:-shared}"
CTX="${CTX:-8192}"

# --- interface byte counters, before and after ----------------------------------------
# Prints "<received> <transmitted>" for one interface, or nothing if it cannot be read.
counters() {
    local iface="$1"
    [ -n "$iface" ] || return 0
    case "$(uname -s)" in
        Linux)
            awk -v want="${iface}:" '$1 == want { print $2, $10 }' /proc/net/dev
            ;;
        Darwin)
            netstat -ib | awk -v want="$iface" '$1 == want && $4 !~ /:/ { print $7, $10; exit }'
            ;;
        *)
            ;;
    esac
}

read -r RX_BEFORE TX_BEFORE <<<"$(counters "$CLUSTER_IFACE")"
START_EPOCH="$(date +%s)"

printf '==> %s against %s\n' "$LABEL" "$BASE_URL"
if [ -n "$CLUSTER_IFACE" ] && [ -n "${RX_BEFORE:-}" ]; then
    printf '    reading byte counters on %s\n' "$CLUSTER_IFACE"
else
    printf '    no interface counters (CLUSTER_IFACE unset or unreadable on this system)\n'
fi
printf '\n'

python3 "$LOAD_TEST" \
    --base-url "$BASE_URL" \
    --model "$SERVED_NAME" \
    --concurrency "$CONCURRENCY" \
    --requests "$REQUESTS" \
    --max-tokens "$MAX_TOKENS" \
    --prompt-set "$PROMPT_SET" \
    --label "$LABEL" \
    --engine vllm \
    --labbook "$LABBOOK"

read -r RX_AFTER TX_AFTER <<<"$(counters "$CLUSTER_IFACE")"
END_EPOCH="$(date +%s)"

RX_DELTA=""
TX_DELTA=""
if [ -n "${RX_BEFORE:-}" ] && [ -n "${RX_AFTER:-}" ]; then
    RX_DELTA=$(( RX_AFTER - RX_BEFORE ))
    TX_DELTA=$(( TX_AFTER - TX_BEFORE ))
fi

# --- one notebook line about the link, beside the generator's own lines ----------------
{
    printf '{"lab": "part-22/lab-two-machine-prefill-decode", "record": "link", '
    printf '"label": "%s", "base_url": "%s", "connector": "%s", ' "$LABEL" "$BASE_URL" "$CONNECTOR"
    printf '"context_length": %s, "prompt_set": "%s", "concurrency": "%s", "requests": %s, ' \
        "$CTX" "$PROMPT_SET" "$CONCURRENCY" "$REQUESTS"
    printf '"iface": "%s", "wall_s": %s, ' "$CLUSTER_IFACE" "$(( END_EPOCH - START_EPOCH ))"
    if [ -n "$RX_DELTA" ]; then
        printf '"rx_bytes": %s, "tx_bytes": %s, ' "$RX_DELTA" "$TX_DELTA"
    else
        printf '"rx_bytes": null, "tx_bytes": null, '
    fi
    printf '"recorded_at": "%s"}\n' "$(date +%Y-%m-%dT%H:%M:%S%z)"
} >>"$LABBOOK"

printf '\n'
if [ -n "$RX_DELTA" ]; then
    printf '==> Link during this run, on %s\n' "$CLUSTER_IFACE"
    printf '    received     %s bytes\n' "$RX_DELTA"
    printf '    transmitted  %s bytes\n' "$TX_DELTA"
    printf '    Divide the larger of these by the number of requests and compare against\n'
    printf '    the per-request key-value cache size the lesson had you compute. If it is\n'
    printf '    a thousand times smaller, no cache is crossing the link.\n'
else
    printf '==> No byte counters recorded. Set CLUSTER_IFACE in .env to measure the link.\n'
fi
printf '    Appended one link line to %s\n' "$LABBOOK"

Download run-load.sh127 lines

RunnableAll tracks

the disaggregated run, through the proxy

bash run-load.sh disagg http://127.0.0.1:8000/v1

Output — what you should see

==> disagg against http://127.0.0.1:8000/v1
    reading byte counters on enp1s0f1np1

==> local-chat at http://127.0.0.1:8000/v1
  c=  1  ok  24/24  fail 0    wall   ...
  c=  4  ok  24/24  fail 0    wall   ...
  c=  8  ok  24/24  fail 0    wall   ...

==> Link during this run, on enp1s0f1np1
    received     ... bytes
    transmitted  ... bytes

The prompt set matters. .env sets PROMPT_SET=shared-prefix, which is a long shared preamble with a short different question on the end, because that is the workload a phase split is aimed at. Run mixed afterwards if you want the control; do not compare one against the other.

7. Load-test the same model on one machine

Stop the proxy and the prefill instance. Restart the decode instance as an ordinary server with no connector, on the same machine, with the same model and the same context length.

RunnableAll tracks

the baseline: one machine, no split

MONOLITHIC=1 bash serve-decode.sh

RunnableAll tracks

the same load, against the baseline

bash run-load.sh baseline http://127.0.0.1:8200/v1

Nothing else about the two runs should differ: same model, same context length, same concurrency levels, same request count, same prompt set. The comparison script checks the ones it can see and tells you when they disagree.

8. Compare, and record

RunnableAll tracks

compare-disagg.py

#!/usr/bin/env python3
"""Compare a disaggregated run against a monolithic run, from the lab notebook.

Purpose: read the JSON lines Part 9's load generator wrote, pair the disaggregated run
    with the single-machine baseline at each concurrency level, and print what the split
    did to time to first token, time per output token and throughput. Where run-load.sh
    also recorded interface byte counters, report the bytes moved per request beside the
    key-value cache size the model's shape predicts, so that "the transfer happened" is a
    number rather than a hope. Appends one comparison line to the notebook.
Platform: all. Pure Python standard library: no pip install, and it may run anywhere the
    notebook file is, including a machine that took no part in the serving.
Minimum memory: 1 GB. It reads a text file.
Assumes: Python 3.9 or later; a labbook.md containing at least one line from each label,
    written by Part 9's load-test.py through run-load.sh. The two runs must have used the
    same model, the same context length, the same prompt set and the same concurrency
    levels, or the comparison is between two different experiments.

Usage:
    python3 compare-disagg.py --labbook labbook.md \\
        --disagg disagg --baseline baseline

    python3 compare-disagg.py --labbook labbook.md \\
        --disagg disagg-nixl --baseline baseline \\
        --kv-bytes-per-token 147456 --prompt-tokens 8192 \\
        --note "two Sparks over the QSFP cable, NixlConnector"

The --kv-bytes-per-token value comes from the course model reference: it is the model's
layers x key-value heads x head dimension x 2 x 2. Qwen3-8B is 147456; Qwen3-1.7B is
114688. Passing it turns the byte counters into a prediction you can check.
"""

from __future__ import annotations

import argparse
import json
import sys
import time
from pathlib import Path


def read_lines(path: Path) -> list[dict]:
    """Every JSON object in the notebook, in file order. Prose lines are skipped."""
    records = []
    try:
        text = path.read_text(encoding="utf-8")
    except OSError as exc:
        print(f"Could not read {path}: {exc}", file=sys.stderr)
        return records
    for raw in text.splitlines():
        raw = raw.strip()
        if not raw.startswith("{"):
            continue
        try:
            obj = json.loads(raw)
        except json.JSONDecodeError:
            continue
        if isinstance(obj, dict):
            records.append(obj)
    return records


def load_rows(records: list[dict], label: str) -> dict[int, dict]:
    """The most recent load-generator row per concurrency level, for one label."""
    rows: dict[int, dict] = {}
    for obj in records:
        if obj.get("label") != label:
            continue
        if obj.get("record") == "link":
            continue
        if "concurrency" not in obj or "ttft_s" not in obj:
            continue
        try:
            level = int(obj["concurrency"])
        except (TypeError, ValueError):
            continue
        rows[level] = obj
    return rows


def link_line(records: list[dict], label: str) -> dict | None:
    """The most recent interface-counter line for one label, if run-load.sh wrote one."""
    found = None
    for obj in records:
        if obj.get("record") == "link" and obj.get("label") == label:
            found = obj
    return found


def ratio(new: float, old: float) -> str:
    """A readable multiple, guarding against a zero denominator."""
    if not old:
        return "n/a"
    return f"{new / old:.2f}x"


def signed_pct(new: float, old: float) -> str:
    if not old:
        return "n/a"
    return f"{(new - old) / old * 100:+.1f}%"


def human_bytes(n: float) -> str:
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if abs(n) < 1000 or unit == "TB":
            return f"{n:.2f} {unit}" if unit != "B" else f"{n:.0f} B"
        n /= 1000
    return f"{n:.2f} TB"


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md",
                        help="notebook file to read and append to")
    parser.add_argument("--disagg", default="disagg",
                        help="label of the disaggregated run")
    parser.add_argument("--baseline", default="baseline",
                        help="label of the single-machine run it is compared against")
    parser.add_argument("--kv-bytes-per-token", type=int, default=0,
                        help="bytes of key-value cache per token, from the model reference")
    parser.add_argument("--prompt-tokens", type=int, default=0,
                        help="approximate prompt length used, for the predicted payload")
    parser.add_argument("--note", default="",
                        help="free text: topology, connector, link, anything")
    parser.add_argument("--lab", default="part-22/lab-two-machine-prefill-decode")
    parser.add_argument("--print-only", action="store_true",
                        help="print the comparison, write nothing")
    args = parser.parse_args()

    book = Path(args.labbook)
    records = read_lines(book)
    if not records:
        print(f"No JSON lines found in {book}. Run the load generator first.", file=sys.stderr)
        return 2

    disagg = load_rows(records, args.disagg)
    base = load_rows(records, args.baseline)

    missing = []
    if not disagg:
        missing.append(args.disagg)
    if not base:
        missing.append(args.baseline)
    if missing:
        print(f"No load-generator rows for label(s): {', '.join(missing)}.", file=sys.stderr)
        print("Labels present: " + ", ".join(sorted({
            str(r.get("label")) for r in records if r.get("label")
        })), file=sys.stderr)
        return 2

    shared = sorted(set(disagg) & set(base))
    if not shared:
        print("The two runs share no concurrency level, so nothing is comparable.",
              file=sys.stderr)
        print(f"  {args.disagg}: {sorted(disagg)}", file=sys.stderr)
        print(f"  {args.baseline}: {sorted(base)}", file=sys.stderr)
        return 2

    # --- a sanity check the reader will otherwise skip ---------------------------------
    warnings = []
    for level in shared:
        d, b = disagg[level], base[level]
        for field in ("model", "prompt_set", "max_tokens"):
            if d.get(field) != b.get(field):
                warnings.append(
                    f"  concurrency {level}: {field} differs "
                    f"({d.get(field)!r} against {b.get(field)!r})"
                )
    if warnings:
        print("These two runs are not the same experiment:")
        for line in warnings:
            print(line)
        print("  Fix the settings and run both again; the comparison below is not valid.\n")

    print(f"==> {args.disagg} against {args.baseline}, from {book}")
    print(f"    {'conc':>4}  {'TTFT p50':>18}  {'TPOT p50':>18}  {'out tok/s':>18}")
    comparison = []
    for level in shared:
        d, b = disagg[level], base[level]
        d_ttft = float(d["ttft_s"]["p50"])
        b_ttft = float(b["ttft_s"]["p50"])
        d_tpot = float(d["tpot_s"]["p50"])
        b_tpot = float(b["tpot_s"]["p50"])
        d_thru = float(d["output_tokens_per_s"])
        b_thru = float(b["output_tokens_per_s"])
        print(
            f"    {level:>4}  "
            f"{ratio(d_ttft, b_ttft):>8} {signed_pct(d_ttft, b_ttft):>9}  "
            f"{ratio(d_tpot, b_tpot):>8} {signed_pct(d_tpot, b_tpot):>9}  "
            f"{ratio(d_thru, b_thru):>8} {signed_pct(d_thru, b_thru):>9}"
        )
        comparison.append({
            "concurrency": level,
            "ttft_p50_disagg_s": round(d_ttft, 4),
            "ttft_p50_baseline_s": round(b_ttft, 4),
            "tpot_p50_disagg_s": round(d_tpot, 5),
            "tpot_p50_baseline_s": round(b_tpot, 5),
            "output_tokens_per_s_disagg": round(d_thru, 2),
            "output_tokens_per_s_baseline": round(b_thru, 2),
        })

    print("\n    A ratio below 1.00 on time to first token is the split helping. A ratio")
    print("    above 1.00 is the transfer costing more than the prefill it replaced, which")
    print("    is the expected result on ordinary Ethernet and is worth recording as such.")

    # --- the link, if the counters were read ------------------------------------------
    link = link_line(records, args.disagg)
    predicted = None
    observed = None
    if args.kv_bytes_per_token and args.prompt_tokens:
        predicted = args.kv_bytes_per_token * args.prompt_tokens
    if link and link.get("rx_bytes") is not None:
        total_requests = 0
        for level in shared:
            total_requests += int(disagg[level].get("requests", 0))
        moved = max(int(link["rx_bytes"]), int(link["tx_bytes"]))
        if total_requests:
            observed = moved / total_requests

    if observed is not None:
        print(f"\n==> Link on {link.get('iface')} during the {args.disagg} run")
        print(f"    bytes moved per request, observed   {human_bytes(observed)}")
        if predicted is not None:
            print(f"    one request's cache, from arithmetic {human_bytes(predicted)}")
            print("    Same order of magnitude means the cache is crossing the link.")
            print("    Three orders smaller means it is not, and the decode instance is")
            print("    quietly prefilling every prompt itself.")
    elif link:
        print(f"\n==> The {args.disagg} run recorded no byte counters. Set CLUSTER_IFACE.")
    else:
        print(f"\n==> No link line for label {args.disagg}. Run it through run-load.sh.")

    record = {
        "lab": args.lab,
        "record": "comparison",
        "disagg_label": args.disagg,
        "baseline_label": args.baseline,
        "model": disagg[shared[0]].get("model"),
        "prompt_set": disagg[shared[0]].get("prompt_set"),
        "connector": (link or {}).get("connector"),
        "iface": (link or {}).get("iface"),
        "kv_bytes_per_token": args.kv_bytes_per_token or None,
        "prompt_tokens": args.prompt_tokens or None,
        "predicted_bytes_per_request": predicted,
        "observed_bytes_per_request": round(observed, 1) if observed is not None else None,
        "levels": comparison,
        "same_experiment": not warnings,
        "note": args.note,
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    }

    if args.print_only:
        print("\n    --print-only: nothing written.")
        return 0

    with book.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    print(f"\n    Appended one comparison line to {book}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Download compare-disagg.py262 lines

RunnableAll tracks

pair the two runs and check the payload against the arithmetic

python3 compare-disagg.py \
  --labbook labbook.md \
  --disagg disagg \
  --baseline baseline \
  --kv-bytes-per-token 147456 \
  --prompt-tokens 4096 \
  --note "prefill on the compute machine, decode on the memory machine, shared connector"

Output — what you should see

==> disagg against baseline, from labbook.md
    conc            TTFT p50            TPOT p50           out tok/s
       1      ...        ...       ...       ...       ...       ...
       4      ...        ...       ...       ...       ...       ...
       8      ...        ...       ...       ...       ...       ...

==> Link on enp1s0f1np1 during the disagg run
    bytes moved per request, observed   ...
    one request's cache, from arithmetic 603.98 MB

Two readings, in this order. First, did the cache move? Observed bytes per request within an order of magnitude of the arithmetic means yes. Three orders of magnitude smaller means no, and nothing else in the output is worth interpreting until that is fixed. Second, what did it cost? A time-to-first-token ratio below one is the split helping; above one is the transfer costing more than the prefill it replaced.

9. Optional: the point-to-point connector

Only on a link with RDMA, which in this course means a DGX Spark pair over the ConnectX-7 cable or two Macs over Thunderbolt 5 from macOS 26.2. Set CONNECTOR=nixl and SIDE_CHANNEL_ADDR to each machine’s address on the cable, not on the switch, and run tasks 3 to 8 again with the label disagg-nixl.

✱Hardware note

SIDE_CHANNEL_ADDR set to a management address instead of a cluster address gives you a pair that works and is slow, and nothing in any log says so. It is the same failure Part 20 warns about for Ray and Part 19 warns about for RPC hosts, and the diagnosis is the same: the interface counters on the cable show nothing while the counters on the management interface show everything.

Follow one request through the complete handoff

Use three labelled terminals: prefill worker, decode worker and proxy/client control. Apply the same reviewed environment file in each appropriate shell, with role-specific addresses checked. Confirm both workers use the same model and compatible cache configuration before starting the proxy.

First issue one short request through the proxy. Collect logs that show prefill, transfer and decode for that request. Successful direct requests to both workers are not sufficient evidence of a working handoff. Stop if the connector reports incompatibility or falls back to a different behaviour than the experiment intends.

Run the colocated baseline and split deployment with the same prompt classes, answer limits and concurrency. Measure first-token latency, decode cadence, transfer cost and failure counts. Include a cancelled request and a worker restart in the isolated lab, then verify a fresh request succeeds. If long prompts improve while short prompts regress, report both and use that boundary in the routing decision. Keep the model identity, connector version, complete configuration, role logs and raw load results. A transfer path that works once is only the first checkpoint; the lab’s conclusion needs correctness and measured service behaviour under the intended workload.

Validation

You are done when all of the following are true. On a reduced path, skip the items naming machines you do not have and keep the rest; the notebook lines are required on every path because they are what the comparison is made of.

• .env exists on both machines with the same MODEL, CTX and SERVED_NAME.
• Both instances reported Application startup complete. and you recorded each one’s key-value cache size and maximum concurrency from its startup log.
• run-proxy.sh found both instances answering before it started, and a single curl through the proxy returned generated text with a request appearing in both instances’ logs.
• labbook.md contains load-generator lines with the label disagg and lines with the label baseline, at the same concurrency levels, from the same model and prompt set.
• labbook.md contains a link record for the disagg run with a non-null rx_bytes, unless your path has no cluster interface to read.
• compare-disagg.py printed a comparison and reported observed bytes per request within an order of magnitude of the arithmetic, or you have written down why it did not.
• Your prediction from task 1 is in the notebook next to the result, right or wrong.

Expected outcome

Pending validationWhat this lab measures — to be filled in by the validation pass and by you

Configuration	Link	TTFT p50 (s)	TPOT p50 (s)	Output tokens/s	Bytes per request
One machine, no split (baseline)	none	pending	pending	pending	none
Two machines, shared-directory connector	house Ethernet	pending	pending	pending	pending
Two machines, NixlConnector	direct RDMA cable	pending	pending	pending	pending
Two processes on one device	loopback	pending	pending	pending	pending
llama-server, 4 slots against 1 slot	none	pending	pending	pending	none

the reference cluster described in Part 18, DGX OS 7.x, Ubuntu 24.04 and macOS as each track requires · vLLM, and llama.cpp on the reduced path to be recorded by the validation pass · Qwen3-8B at bf16, or Qwen3-1.7B on the single-machine path, as listed · 8,192 tokens of context · 2026-09-09

No row here has been measured. The table exists so the validation pass can substitute measured values without rewriting the page, and so your own rows have a shape to go into. The four columns are what Part 9's load generator and this lab's wrapper report.

The result this lab expects, and which your own numbers may well contradict, is that on ordinary house Ethernet the split makes time to first token substantially worse, leaves time per output token roughly unchanged, and lowers total throughput a little. Lesson 2 predicted the first of those from one multiplication, and vLLM’s own page predicted the third in capital letters. On a direct RDMA cable the first should move much less, and the interference benefit should start to be visible at the higher concurrency levels rather than at concurrency one.

If your numbers say something else, that is the more interesting outcome and the notebook line with your connector, link and versions is what makes it worth reporting.

Troubleshooting

The decode instance answers, time to first token is unchanged, and nothing errors. The characteristic failure. The decode instance found nothing it recognised and prefilled the prompt itself, which is always a valid thing to do. Check the shared directory has files in it, check the decode instance’s vllm:prefix_cache_hits against vllm:prefix_cache_queries on its /metrics endpoint, and check the byte counters. One of the three will tell you where it stopped.

--kv-transfer-config is rejected. The option is documented on vLLM’s disaggregated prefilling page but on 2026-09-09 it was not on the vllm serve CLI reference page. Run vllm serve --help | grep kv on your installation and record what your build accepts. If it is absent entirely, your vLLM is older or newer than the documentation you are reading.

The proxy will not start: “the prefill instance is not answering”. It checks /v1/models on both before starting. A large model takes minutes to load; wait for Application startup complete. on both, then try again. If one instance is on another machine, check the name resolves from where the proxy runs, which is the Part 18 problem wearing a new hat.

Permission denied on the shared directory. The prefill instance writes and the decode instance reads, so the mount must be read-write for the account running the engines on both machines. Part 18 exported the model library read-only on purpose; the key-value store is a different directory with different needs and should not be inside the read-only export.

Out of memory when the second process starts, on the single-machine path. Two engines on one device each claim SPLIT_MEM_FRACTION of it, and the default sums to less than one for a reason. Lower it, or use the smaller model, or both. The startup log’s cache size tells you whether what is left can hold a single sequence at your context length.

Two instances on one host and the transfer fails to establish. With a point-to-point connector, each needs its own side-channel port. The settings file has PREFILL_SIDE_CHANNEL_PORT and DECODE_SIDE_CHANNEL_PORT as separate values for exactly this case, and LMCache’s documentation notes the same requirement.

Time per output token got worse, not just time to first token. That should not happen: the decode instance is doing the same work it would have done alone. Look at whether the decode machine is also running the proxy, the load generator or the other instance, and at whether the transfer is still in progress while decoding has started. Move the generator to a third machine and try again.

The interface counters show a huge number on the baseline run too. The counters cover the whole interface, including your model library mount and anything else on the machine. Run on a quiet cluster, and read the byte figures as an order of magnitude rather than an accounting.

Cleanup

Stop the proxy first, then the two instances. Nothing about the machines has changed except the files below.

RunnableAll tracks

remove the key-value store, which holds prompts in the model's representation

rm -rf "${KV_SHARED_PATH:?set KV_SHARED_PATH first}"

⚠Warning

That directory is the users’ prompts in the model’s internal representation, and this lab filled it up. Delete it when you are finished, and while it exists treat it as you would treat the transcripts: on storage you control, readable only by the accounts running the engines. The same argument Part 17 made about saved slot files applies with more force here, because a shared store is readable from every machine that mounts it.

Leave the network configuration and the shared mount alone. The next lab and the project both assume them, and Part 23 assumes them again.

What you learned

• The arithmetic decides, and it decides before you build. One multiplication from the model reference and one division by a measured link speed predicted the result of this lab. Doing it first turned an afternoon of configuration into a test of a prediction.
• A cache miss is silent. The decode instance prefills the prompt itself, returns a correct answer, and logs nothing unusual. Three independent checks catch it: the store’s contents, the prefix cache counters, and the interface byte counters against the predicted payload.
• One variable at a time, or nothing is comparable. The same script serves the split and the baseline because the temptation to change two things is otherwise irresistible, and the comparison script checks the settings it can see.
• Both machines hold the whole model. Nothing here let you run a model that did not fit. That is Parts 19, 20 and 21, and confusing the two is the commonest misunderstanding about this architecture.
• The interface counters are the ground truth. An engine can tell you what it thinks it did. The operating system tells you how many bytes actually crossed the cable, and the difference between those two statements is where the evening goes.
• A negative result, recorded properly, is a result. “On this link, with this model, at this prompt length, the split cost me time to first token” is a sentence with a measurement behind it, and it is worth more than a favourable number with no context.

Record in the notebook: the vLLM version on both machines and the proxy’s tag; the connector and, where relevant, the side-channel addresses and the interface named; the model, quantisation and context length; each instance’s key-value cache size and maximum concurrency from its startup log; the predicted payload and transfer time from task 1; the observed bytes per request; and the four load-generator numbers for each of the two runs at each concurrency level. Add one sentence saying which configuration you would actually use. The project at the end of this part is written from these entries.

Check your understanding

Question 1. Which machine should take the prefill role, and why?

• The one with the most memory, because the cache lives there
• The one with the most arithmetic throughput and the least memory, because prefill is compute-bound and only needs room for the prompts it is reading right now
• The one running the proxy, to save a hop
• Either; the roles are symmetrical

Show the answer and why

Answer: The one with the most arithmetic throughput and the least memory, because prefill is compute-bound and only needs room for the prompts it is reading right now

Decode holds every conversation in flight, so the memory-rich machine takes that role. Prefill processes one prompt at a time and needs arithmetic. Splitwise made this argument from measurements and concluded that token generation does not need the newest accelerator, which is the same statement from the other side.

Question 2. Your comparison shows time to first token unchanged and the interface counters show a few kilobytes per request. What has happened?

• The link is saturated
• No key-value cache is crossing the link: the decode instance is prefilling every prompt itself and returning correct answers, so nothing errors and nothing improves
• The proxy is compressing the cache
• The model is too small to benefit

Show the answer and why

Answer: No key-value cache is crossing the link: the decode instance is prefilling every prompt itself and returning correct answers, so nothing errors and nothing improves

A few kilobytes is the request text, not a cache. For Qwen3-8B at 4,096 tokens the arithmetic predicts about 0.6 GB. Check the store's contents and the decode instance's prefix cache hit counter next; the failure is upstream of anything about performance.

Question 3. Why does the lab insist that the baseline run uses the same script as the disaggregated run, with one variable changed?

• To save typing
• Because a comparison between two runs that differ in model, context length or prompt set is a comparison between two experiments, and the difference in the numbers then means nothing
• Because vLLM requires it
• Because the notebook format demands it

Show the answer and why

Answer: Because a comparison between two runs that differ in model, context length or prompt set is a comparison between two experiments, and the difference in the numbers then means nothing

This is why the comparison script checks the model, prompt set and token limit on both sides and warns when they differ. The single most common way to produce a meaningless benchmark is to change the thing you are measuring and something else at the same time.

Question 4. Which of these belong in the notebook entry for this lab? Select all that apply.

• The connector used, and for a point-to-point connector the interface it was bound to
• The predicted payload and transfer time from the arithmetic, written down before the run
• Each instance's key-value cache size and maximum concurrency from its startup log
• Only the run that produced the better numbers

Show the answer and why

Answer: The connector used, and for a point-to-point connector the interface it was bound to, The predicted payload and transfer time from the arithmetic, written down before the run, Each instance's key-value cache size and maximum concurrency from its startup log

Recording only the favourable run is how a notebook becomes marketing. The prediction written down first is what makes the result informative either way, and the startup-log figures are what let you tell a configuration problem from a network problem six months later.

Question 5. On the single-machine path, what can this lab measure and what can it not?

• Everything the two-machine path measures
• The configuration, the failure modes and the mechanism, but nothing about a network: the transfer crosses the loopback interface, which LMCache's own documentation describes as suitable for functional testing only
• Only the baseline
• Nothing useful

Show the answer and why

Answer: The configuration, the failure modes and the mechanism, but nothing about a network: the transfer crosses the loopback interface, which LMCache's own documentation describes as suitable for functional testing only

The procedure, the flags, the silent-miss failure and the notebook entry are all identical, and that is most of the value. What loopback cannot tell you is what a cable would have done, so the path is labelled on the page and should be labelled in your notebook line too.

Check my answersClear