Lab: KV Cache Offload and Sharing

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The versions and measurements each track was run with will be recorded here when the validation pass is done.

Objective

By the end of this lab you will have measured three different ways of not recomputing a key-value cache, and you will know which of them is worth having on your machine.

The first is a host-memory tier under the engine’s own cache, so that blocks pushed out of device memory are recalled rather than recomputed. The second is reuse across the turns of a conversation, which is the workload every chat client and every agent loop actually produces. The third is a store shared between two engine instances, so that a prefix one instance computed is available to the other, which is the thing a load-balanced pair of replicas otherwise cannot do.

Unlike the previous lab, all three of these are likely to pay on a home machine. The previous lab moved a gigabyte across a cable once per request; this one avoids moving anything.

⌂Single-machine path

This lab has no cluster path and needs none. Everything runs on one machine at 16 GB. The two-instance task uses two processes on one device, which is exactly how a shared store behaves between two machines minus the network, and the network is not the part being measured.

Requirements

Every track needs the lab notebook from Part 1, about sixty minutes of which fifty are attended, and the settings file from the previous lab. No model download beyond what Parts 6 and 9 already left you.

Track S — NVIDIA DGX Spark

Primary path. vLLM as installed in Part 9, 16 GB or more, and enough free host memory for KV_OFFLOAD_GB. On a 128 GB unified-memory machine the distinction between device memory and host memory is not what it is on a discrete card, and that is itself worth measuring: the offload tier may buy you much less here than it does on a machine with 24 GB of video memory and 64 GB beside it. Record which you have.

Track X — AMD Ryzen AI Max+ 395Partial

vLLM's GPU installation page lists Ryzen AI MAX and AI 300 among its ROCm targets with pre-built wheels, but this course has not exercised that path.

Run the vLLM tasks if your installation works. If it does not, the llama.cpp path below is a complete substitute for tasks 2, 3 and 5 and is not a lesser exercise: prompt caching, a quantised cache and slot save and restore are three tiers measured on the engine this track uses for everything else.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS, so the offloading backend and the shared-store connector are not available on Apple silicon.

Reduced path, and a good one. llama-server has three of the four ideas in this lab: prompt caching per slot, a quantised key-value cache through -ctk and -ctv, and a disk tier through slot save and restore. Follow tasks 1, 5 and 6, using llama-cache-tiers.sh throughout. What you cannot do here is share a store between two engine instances; say so in the notebook.

mlx-lm’s explicit prompt-cache file, which Part 17 introduced, is a fourth tier available on this track and is worth a fourth notebook line if you have the time.

Track N — NVIDIA desktop or laptop

Primary path. vLLM as installed in Part 9, and a card with 16 GB or more. This is the track where the host-memory tier should show the largest effect, because device memory is genuinely separate from host memory and genuinely scarcer. Set KV_OFFLOAD_GB to something your machine can spare and raise it only if the measurement rewards you.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-22-disaggregated-serving"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-22-disaggregated-serving and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Settings, and one distinction to get right

This lab reads the same settings file as the previous one. If you did that lab, you already have it.

Fragment — not complete on its own

env-example.txt

# Purpose: every setting the Part 22 scripts read. Copy this file to `.env` beside the
#          scripts and fill in the empty lines for your own machines. Nothing here is a
#          secret except the Hugging Face token, which you generate rather than copy and
#          which is better exported in your shell than written here.
# Platform: all
# Minimum memory: 24 GB per machine for the two-machine path; 16 GB for the single-machine
#          path and for the offload lab
# Assumes: `cp env-example.txt .env`, then an editor. Every script loads it with
#          `set -a; . ./.env; set +a` when the file is present, so a value already set in
#          your shell always wins over a value in the file.
#
# Part 18's own .env already holds CLUSTER_IFACE, CLUSTER_PEERS, MODELS_DIR and LABBOOK.
# The names below are deliberately identical: copy your values across, or source Part 18's
# .env first and this one after it, and delete the duplicated lines.

# ------------------------------------------------------------------- the two machines
# The name of the machine that will do PREFILL, and the name of the machine that will do
# DECODE, as the other machines can resolve them. RFC 8375 reserves everything under
# `home.arpa` for names that mean something inside one house and nothing outside it, so
# names like `node-a.home.arpa` are exactly the intended use. Where a machine has a direct
# cable as well as a house connection, use its "-direct" name here: the name you type is
# what decides which cable the key-value cache crosses.
#
#   PREFILL_HOST=node-a-direct.home.arpa
#   DECODE_HOST=node-b-direct.home.arpa
#
# Leave BOTH empty for the single-machine path; the scripts then use the loopback address
# and say so in the notebook line.
PREFILL_HOST=
DECODE_HOST=

# The address THIS machine advertises for the connector handshake, when you are using a
# point-to-point connector. It must be an address the other machine can reach, which on a
# cluster with a direct cable is the address on the cable and not the one on the switch.
# Leave empty on the single-machine path.
SIDE_CHANNEL_ADDR=

# Distinct handshake ports. Two instances on ONE host must not share one.
PREFILL_SIDE_CHANNEL_PORT=5600
DECODE_SIDE_CHANNEL_PORT=5601

# The interface that carries cluster traffic on THIS machine, from Part 18's lab. The
# scripts read its byte counters around each load test, and pass it to the transfer
# library so that it does not pick the management interface on its own.
#   Linux   ip -br addr           e.g. enp1s0f1np1, eno1, enp5s0
#   macOS   networksetup -listallhardwareports   e.g. en0, en5
CLUSTER_IFACE=

# ---------------------------------------------------------------------- the connector
# Which connector the two instances use. The scripts accept:
#   shared    ExampleConnector over a directory both machines can read and write.
#             Needs no RDMA, no UCX and no handshake. Start here.
#   nixl      NixlConnector, point to point over UCX. Wants RDMA to be worth doing.
#   mooncake  MooncakeConnector. Needs the mooncake-transfer-engine package.
CONNECTOR=shared

# For CONNECTOR=shared only: a directory BOTH machines can read and write, on the shared
# mount from Part 18. It fills up with key-value blocks; put it somewhere you can delete.
KV_SHARED_PATH=

# ---------------------------------------------------------------------------- models
# The model both instances load. They must load the SAME one: two instances that disagree
# about the model, the quantisation or the block size produce blocks the other cannot use,
# and the symptom is a silent miss rather than an error.
# Qwen3-8B is Apache-2.0 and ungated; at bf16 it needs roughly 17 GB for weights alone, so
# use it on the 24 GB path and drop to Qwen3-1.7B for two processes on one small device.
MODEL=Qwen/Qwen3-8B

# The smaller model for the single-machine path, where one device is split between two
# engine processes. Qwen3-1.7B is Apache-2.0.
SMALL_MODEL=Qwen/Qwen3-1.7B

# The name clients send in the "model" field. Keep it identical on every path so the load
# generator's command line never changes between runs.
SERVED_NAME=local-chat

# Where models are cached on this machine. Part 18's shared library, or a local path.
HF_HOME=

# --------------------------------------------------------------------------- serving
# Ports. The proxy is the only one a client should ever talk to.
PREFILL_PORT=8100
DECODE_PORT=8200
PROXY_PORT=8000

# Where the engines bind. Keep this on the loopback address or on your cluster address;
# nothing in this part authenticates anything.
SERVE_HOST=127.0.0.1

# Context length to allocate, and how many sequences may be in flight. Both instances must
# use the same context length.
CTX=8192
MAX_SEQS=8

# Fraction of the device each instance may claim. On the single-machine path the two
# processes share one device, so this must be well below half for each of them.
MEM_FRACTION=0.85
SPLIT_MEM_FRACTION=0.40

# ------------------------------------------------------------- the offload lab (lab 2)
# Gibibytes of host memory the engine may use as a key-value tier. Start small: this is
# host memory taken away from everything else on the machine.
KV_OFFLOAD_GB=8

# Backend for that tier, as the vllm serve CLI reference documents it: "native" for vLLM's
# own host-memory offloading, or "lmcache" if you have installed LMCache.
KV_OFFLOAD_BACKEND=native

# For the llama.cpp reduced path on Tracks X and M: the GGUF file, the number of slots and
# where saved slot caches are written.
LLAMA_MODEL=
LLAMA_PORT=8080
LLAMA_PARALLEL=2
LLAMA_SLOT_SAVE_PATH=./slot-cache
LLAMA_CACHE_TYPE=q8_0

# ------------------------------------------------------------------ the load generator
# Part 9's load-test.py. Give the path to it; the wrapper refuses to run without it rather
# than reimplementing a load generator that already exists.
LOAD_TEST=../part-09-vllm-and-sglang/load-test.py

# Concurrency levels and requests per level for every run in this part. Keep them the same
# across runs or the comparison means nothing.
CONCURRENCY=1,4,8
REQUESTS=24
MAX_TOKENS=128

# The prompt set. "shared-prefix" is the workload a phase split is aimed at: a long shared
# preamble with a short different question. "mixed" is the control.
PROMPT_SET=shared-prefix

# Where every script appends its JSON line. The lab notebook from Part 1.
LABBOOK=labbook.md

# --------------------------------------------------------------------------- secrets
# Gated models need a Hugging Face token. Export it in your shell rather than writing it
# here if you can; if you do put it here, keep .env out of version control.
HF_TOKEN=

# The load generator reads an API key, if your server needs one, from the variable named
# here. It is never written to the notebook.
LOADTEST_API_KEY_ENV=LOADTEST_API_KEY

RunnableAll tracks

the settings file, if you do not already have one

cp env-example.txt .env

Before anything runs, one distinction that costs people an afternoon.

!Important

--cpu-offload-gb and --kv-offloading-size are different features. vLLM’s engine argument documentation describes --cpu-offload-gb as “The space in GiB to offload to CPU, per GPU”, and explains it as “a virtual way to increase the GPU memory size”, with the example of a 24 GB GPU set to 10 behaving virtually like a 34 GB one. That is model weights, streamed back over the bus on every forward pass. This lab is about the key-value cache, which on the vllm serve CLI reference as read on 2026-09-09 is controlled by --kv-offloading-size, documented as “Size of the KV cache offloading buffer in GiB”, and --kv-offloading-backend, whose supported backends the same page gives as “‘native’ (vLLM native CPU offloading), ‘lmcache’”. Two options, two problems, similar names.

The same page’s disaggregated prefilling documentation describes a third route to the same place, the OffloadingConnector, “enable offloading of KV data to CPU memory, customizing the CPU block size (in tokens) and total CPU memory bytes to allocate”. The scripts here use the serve options because they are on the CLI reference page; if your build only has the connector, the configuration JSON is in Lesson 3 and the measurement is unchanged.

2. Measure without the tier, then with it

The baseline first, because a tier with nothing to compare against tells you nothing.

RunnableAll tracks

serve-offload.sh

#!/usr/bin/env bash
# Purpose: start one vLLM instance with a host-memory tier under its key-value cache, so
#          that blocks evicted from device memory are recalled instead of recomputed, and
#          print the one distinction that matters here: this offloads the CACHE, not the
#          weights
# Platform: spark, nvidia (vLLM's GPU path; Track X only where your ROCm build works)
# Minimum memory: 16 GB of device memory for Qwen3-8B at a four-bit or eight-bit
#          checkpoint, plus KV_OFFLOAD_GB of HOST memory that other programs will not get
# Assumes: vLLM installed as in Part 9 and on PATH; a .env copied from env-example.txt;
#          nothing else listening on DECODE_PORT. Run it once with OFFLOAD=0 for the
#          baseline and once with OFFLOAD=1, and compare the two notebook lines.
#
# Usage: OFFLOAD=0 bash serve-offload.sh     the baseline: device memory only
#        OFFLOAD=1 bash serve-offload.sh     with a host-memory tier of KV_OFFLOAD_GB
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${MODEL:-Qwen/Qwen3-8B}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
PORT="${DECODE_PORT:-8200}"
CTX="${CTX:-8192}"
MAX_SEQS="${MAX_SEQS:-8}"
MEM_FRACTION="${MEM_FRACTION:-0.85}"
KV_OFFLOAD_GB="${KV_OFFLOAD_GB:-8}"
KV_OFFLOAD_BACKEND="${KV_OFFLOAD_BACKEND:-native}"
OFFLOAD="${OFFLOAD:-1}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Install it as Part 9 describes."

case "$KV_OFFLOAD_BACKEND" in
    native|lmcache) ;;
    *) fail "KV_OFFLOAD_BACKEND must be 'native' or 'lmcache'. Got '${KV_OFFLOAD_BACKEND}'." ;;
esac

ARGS=(
    "$MODEL"
    --host "$SERVE_HOST"
    --port "$PORT"
    --served-model-name "$SERVED_NAME"
    --max-model-len "$CTX"
    --max-num-seqs "$MAX_SEQS"
    --gpu-memory-utilization "$MEM_FRACTION"
    --enable-prefix-caching
)

if [ "$OFFLOAD" = "1" ]; then
    ARGS+=(--kv-offloading-backend "$KV_OFFLOAD_BACKEND" --kv-offloading-size "$KV_OFFLOAD_GB")
    TIER="host memory, ${KV_OFFLOAD_GB} GiB, backend ${KV_OFFLOAD_BACKEND}"
    LABEL_HINT="offload-on"
else
    TIER="none: device memory only"
    LABEL_HINT="offload-off"
fi

cat <<INFO
==> vLLM with a key-value cache tier
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${PORT}
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    prefix caching    enabled
    cache tier        ${TIER}

    Suggested label for the measurement:  ${LABEL_HINT}

    The distinction to keep straight. --kv-offloading-backend and --kv-offloading-size
    move KEY-VALUE BLOCKS into host memory. --cpu-offload-gb is a different option that
    moves MODEL WEIGHTS there, and vLLM's engine argument documentation describes it as a
    virtual way to increase the GPU memory size. They solve different problems and this
    lab is about the first one. If your build does not accept these two options, run
    "vllm serve --help | grep -i offload" and record what it does accept.

    The tier is host memory taken away from everything else on this machine. Start with a
    small KV_OFFLOAD_GB and raise it only if the measurement says it helped.

INFO

exec vllm serve "${ARGS[@]}"

Download serve-offload.sh89 lines

RunnableAll tracks

baseline: device memory only, prefix caching on

OFFLOAD=0 bash serve-offload.sh

Note the key-value cache size in tokens from the startup log, and note the maximum concurrency it implies at your context length. Those two numbers are what the tier is trying to extend.

RunnableAll tracks

measure-reuse.py

#!/usr/bin/env python3
"""Measure key-value reuse across the turns of a conversation, and across two engines.

Purpose: send a growing multi-turn conversation to one or two OpenAI-compatible endpoints,
    time to the first token of each turn, and read each engine's prefix-cache counters
    either side of the run. With one endpoint this measures reuse across turns, which is
    what an agent loop or a chat client does. With two endpoints given, turns alternate
    between them, which measures whether a shared cache tier is letting one engine reuse
    what the other computed. Appends one notebook line per turn plus one summary line.
Platform: all (spark, strix, mac, nvidia). Pure Python standard library: no pip install.
Minimum memory: 1 GB for the tool itself; 16 GB on the machine serving the model.
Assumes: Python 3.9 or later; a server answering POST /v1/chat/completions with streaming
    over plain HTTP; the model name the server reports at /v1/models. Counters are read
    from /metrics where the engine exposes it, which vLLM does by default and llama-server
    does when started with --metrics. An API key, if the server needs one, is read from
    the environment variable named by --api-key-env and is never written to the notebook.

Usage:
    python3 measure-reuse.py --base-url http://127.0.0.1:8000/v1 --model local-chat \\
        --turns 6 --preamble-words 1500 --label offload-on --labbook labbook.md

    python3 measure-reuse.py \\
        --base-url http://127.0.0.1:8100/v1 \\
        --base-url http://127.0.0.1:8200/v1 \\
        --model local-chat --turns 6 --label shared-store --labbook labbook.md

Counting note: time to first token is measured from just before the request is written to
    the moment the first streamed chunk carrying content arrives. It therefore includes
    connection setup, which is small on a LAN and is the same for every turn, so it does
    not distort the comparison between turns.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.request
from http.client import HTTPResponse
from urllib.parse import urlsplit

PREAMBLE_SENTENCES = [
    "The operator runs one open-weight language model on machines they own.",
    "Prefill reads the prompt and is limited by arithmetic throughput.",
    "Decode writes the answer one token at a time and is limited by memory bandwidth.",
    "A key-value cache holds the key and value vectors for every token already read.",
    "Its size is layers times key-value heads times head dimension times two tensors.",
    "That figure is multiplied by the bytes per element and by the number of tokens.",
    "A prefix cache reuses those vectors when two prompts begin with the same tokens.",
    "The match must start at position zero and must be exact on token identifiers.",
    "Reuse of that kind is bit-identical, so it cannot change what the model produces.",
    "Moving cold blocks to host memory extends the cache without touching the network.",
    "Writing blocks to disk lets a cache survive a restart of the serving process.",
    "A shared store lets a second engine reuse blocks the first engine computed.",
    "The operator measures time to first token, time per output token and goodput.",
    "Throughput on its own answers a different question from goodput under a deadline.",
]

QUESTIONS = [
    "In one sentence, what limits decode speed?",
    "In one sentence, what does a prefix cache reuse?",
    "In one sentence, why must a prefix match start at position zero?",
    "In one sentence, what does moving blocks to host memory cost?",
    "In one sentence, when is a disk tier worth having?",
    "In one sentence, what does a shared store let a second engine do?",
    "In one sentence, how is a key-value cache size computed?",
    "In one sentence, what is the difference between throughput and goodput?",
    "In one sentence, why does an agent loop benefit from cache reuse?",
    "In one sentence, what does a cache miss look like from the outside?",
]


def build_preamble(words: int) -> str:
    """A stable block of text of roughly the requested length, identical on every run."""
    out: list[str] = []
    count = 0
    i = 0
    while count < words:
        sentence = PREAMBLE_SENTENCES[i % len(PREAMBLE_SENTENCES)]
        out.append(sentence)
        count += len(sentence.split())
        i += 1
    return " ".join(out)


def post_stream(base_url: str, body: dict, api_key: str, timeout: float):
    """Sends one streaming request. Returns (ttft_seconds, total_seconds, chunks, text)."""
    parts = urlsplit(base_url)
    url = f"{base_url.rstrip('/')}/chat/completions"
    if parts.scheme != "http":
        raise ValueError("This tool speaks plain HTTP only; use a localhost or LAN endpoint.")
    data = json.dumps(body).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=data, headers=headers, method="POST")

    started = time.perf_counter()
    ttft = None
    chunks = 0
    pieces: list[str] = []
    response: HTTPResponse
    with urllib.request.urlopen(request, timeout=timeout) as response:
        for raw in response:
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            try:
                obj = json.loads(payload)
            except json.JSONDecodeError:
                continue
            for choice in obj.get("choices", []):
                piece = (choice.get("delta") or {}).get("content")
                if not piece:
                    continue
                if ttft is None:
                    ttft = time.perf_counter() - started
                chunks += 1
                pieces.append(piece)
    total = time.perf_counter() - started
    return ttft, total, chunks, "".join(pieces)


def metrics(base_url: str, timeout: float = 10.0) -> dict:
    """Prefix-cache counters from the engine's /metrics endpoint, where it has one."""
    parts = urlsplit(base_url)
    root = f"{parts.scheme}://{parts.netloc}/metrics"
    wanted = {
        "vllm:prefix_cache_hits": "prefix_cache_hits",
        "vllm:prefix_cache_queries": "prefix_cache_queries",
        "vllm:kv_cache_usage_perc": "kv_cache_usage_perc",
        "llamacpp:prompt_tokens_total": "prompt_tokens_total",
        "llamacpp:tokens_predicted_total": "tokens_predicted_total",
    }
    found: dict[str, float] = {}
    try:
        with urllib.request.urlopen(root, timeout=timeout) as response:
            text = response.read().decode("utf-8", "replace")
    except (urllib.error.URLError, OSError, ValueError):
        return found
    for line in text.splitlines():
        if line.startswith("#") or not line.strip():
            continue
        name, _, rest = line.partition("{")
        if not rest:
            name, _, value = line.partition(" ")
        else:
            _, _, value = rest.partition("} ")
        key = wanted.get(name.strip())
        if not key:
            continue
        try:
            found[key] = found.get(key, 0.0) + float(value.strip())
        except ValueError:
            continue
    return found


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base-url", action="append", default=[],
                        help="OpenAI-compatible base URL ending in /v1; give twice to alternate")
    parser.add_argument("--model", required=True, help="model name the server reports")
    parser.add_argument("--turns", type=int, default=6, help="conversation turns to send")
    parser.add_argument("--preamble-words", type=int, default=1500,
                        help="approximate length of the shared preamble in words")
    parser.add_argument("--max-tokens", type=int, default=48)
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--timeout", type=float, default=300.0)
    parser.add_argument("--label", default="reuse", help="tag written into the notebook line")
    parser.add_argument("--api-key-env", default="LOADTEST_API_KEY",
                        help="environment variable holding the API key, if one is needed")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--lab", default="part-22/lab-kv-cache-offload-and-sharing")
    parser.add_argument("--note", default="", help="free text: tier, connector, anything")
    parser.add_argument("--print-only", action="store_true", help="print it, record nothing")
    args = parser.parse_args()

    urls = args.base_url or ["http://127.0.0.1:8000/v1"]
    if args.turns < 2:
        print("--turns must be at least 2; the point is the comparison between them.",
              file=sys.stderr)
        return 2

    api_key = os.environ.get(args.api_key_env, "")
    preamble = build_preamble(args.preamble_words)

    print(f"==> {args.label}: {args.model}")
    for url in urls:
        print(f"    endpoint {url}")
    print(f"    {args.turns} turns, preamble about {args.preamble_words} words, "
          f"max_tokens {args.max_tokens}")
    if len(urls) > 1:
        print("    turns alternate between the endpoints: a hit on turn two means the")
        print("    second engine reused what the first one computed.")
    print()

    before = {url: metrics(url) for url in urls}

    messages = [{"role": "system", "content": preamble}]
    rows = []
    for turn in range(args.turns):
        url = urls[turn % len(urls)]
        question = QUESTIONS[turn % len(QUESTIONS)]
        messages.append({"role": "user", "content": question})
        body = {
            "model": args.model,
            "messages": messages,
            "max_tokens": args.max_tokens,
            "temperature": args.temperature,
            "stream": True,
        }
        try:
            ttft, total, chunks, text = post_stream(url, body, api_key, args.timeout)
        except (urllib.error.URLError, OSError, ValueError) as exc:
            print(f"    turn {turn + 1}: failed against {url}: {exc}", file=sys.stderr)
            return 1
        messages.append({"role": "assistant", "content": text})
        rows.append({
            "turn": turn + 1,
            "endpoint": url,
            "ttft_s": round(ttft, 4) if ttft is not None else None,
            "total_s": round(total, 4),
            "chunks": chunks,
        })
        shown = f"{ttft:.3f}" if ttft is not None else "  none"
        print(f"    turn {turn + 1:>2}  {url:<34}  TTFT {shown} s  total {total:6.2f} s")

    after = {url: metrics(url) for url in urls}

    print()
    first = rows[0]["ttft_s"]
    later = [r["ttft_s"] for r in rows[1:] if r["ttft_s"] is not None]
    if first and later:
        best = min(later)
        print(f"    turn 1 time to first token   {first:.3f} s")
        print(f"    fastest later turn           {best:.3f} s  ({best / first:.2f}x of turn 1)")
        print("    A later turn much faster than the first is reuse working. A later turn")
        print("    as slow as the first means every turn is being prefilled from scratch,")
        print("    which is what an unhit cache looks like from outside.")

    deltas = {}
    for url in urls:
        b, a = before.get(url, {}), after.get(url, {})
        delta = {k: round(a[k] - b.get(k, 0.0), 3) for k in a if k in ("prefix_cache_hits",
                                                                      "prefix_cache_queries",
                                                                      "prompt_tokens_total",
                                                                      "tokens_predicted_total")}
        if "kv_cache_usage_perc" in a:
            delta["kv_cache_usage_perc_end"] = round(a["kv_cache_usage_perc"], 4)
        if delta:
            deltas[url] = delta
            hits = delta.get("prefix_cache_hits")
            queries = delta.get("prefix_cache_queries")
            if hits is not None and queries:
                print(f"    {url}: prefix cache {hits:.0f} hit tokens of "
                      f"{queries:.0f} queried, {hits / queries * 100:.1f}%")

    if not deltas:
        print("    No /metrics counters were readable. vLLM serves them by default;")
        print("    llama-server needs --metrics. The turn timings above still stand.")

    record = {
        "lab": args.lab,
        "record": "reuse",
        "label": args.label,
        "model": args.model,
        "endpoints": urls,
        "turns": args.turns,
        "preamble_words": args.preamble_words,
        "max_tokens": args.max_tokens,
        "rows": rows,
        "metric_deltas": deltas,
        "note": args.note,
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    }

    if args.print_only:
        print("\n    --print-only: nothing written.")
        return 0

    with open(args.labbook, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    print(f"\n    Appended one reuse line to {args.labbook}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Download measure-reuse.py295 lines

RunnableAll tracks

six turns of one conversation against the baseline

python3 measure-reuse.py \
  --base-url http://127.0.0.1:8200/v1 \
  --model local-chat \
  --turns 6 \
  --preamble-words 1500 \
  --label offload-off \
  --labbook labbook.md

Output — what you should see

==> offload-off: local-chat
    endpoint http://127.0.0.1:8200/v1
    6 turns, preamble about 1500 words, max_tokens 48

    turn  1  http://127.0.0.1:8200/v1            TTFT ..... s  total ...
    turn  2  http://127.0.0.1:8200/v1            TTFT ..... s  total ...
    ...
    turn 1 time to first token   ..... s
    fastest later turn           ..... s  (....x of turn 1)
    http://127.0.0.1:8200/v1: prefix cache ... hit tokens of ... queried, ...%

Then stop it and start the same server with the tier.

RunnableAll tracks

with a host-memory tier under the cache

OFFLOAD=1 bash serve-offload.sh

RunnableAll tracks

the same six turns, with the tier

python3 measure-reuse.py \
  --base-url http://127.0.0.1:8200/v1 \
  --model local-chat \
  --turns 6 \
  --preamble-words 1500 \
  --label offload-on \
  --labbook labbook.md

=Expected result

On a quiet server with one conversation, the two runs should look almost identical, because nothing has been evicted and there is nothing to recall. That is the correct result and it is the point of task 3: an offload tier does nothing until the device cache is under pressure. If you see a large difference here, something else changed between the runs.

3. Put the cache under pressure

A tier is worth measuring only when the device cache is full. Two ways to arrange that, and both are honest.

The direct way is to lower the memory fraction so the engine has a small cache to begin with. Set MEM_FRACTION to something well below your usual value in .env, restart with OFFLOAD=0, run the load generator from Part 9 at concurrency 8 with the shared-prefix prompt set, then repeat with OFFLOAD=1.

RunnableAll tracks

a full cache, without the tier

bash run-load.sh offload-off-load http://127.0.0.1:8200/v1

RunnableAll tracks

the same load, with the tier

bash run-load.sh offload-on-load http://127.0.0.1:8200/v1

The other way is to raise the context length until the cache no longer holds your concurrency. The startup log’s maximum concurrency figure tells you where that boundary is: below one and a single sequence does not fit, which is a different failure and not what you want here.

RunnableAll tracks

compare the two runs, as in the previous lab

python3 compare-disagg.py \
  --labbook labbook.md \
  --disagg offload-on-load \
  --baseline offload-off-load \
  --lab part-22/lab-kv-cache-offload-and-sharing \
  --note "host-memory tier under pressure, memory fraction lowered"

What the tier changes, on a 24 GB machine — arithmetic, not a measurement

Weights (bf16)

16.4 GB

Device key-value cache

4.5 GB

Engine buffers and activations

1.5 GB

Free

1.6 GB

Total

24 GB

Qwen3-8B at bf16 from the model reference leaves a few gigabytes of device memory for the cache, which at 147,456 bytes per token is a few tens of thousands of tokens: two or three conversations at 8,192 tokens each. The host tier does not appear on this bar at all, because it is not device memory. What it changes is how many conversations can be paused and resumed without paying for the prefill again.

4. Share a prefix between two engine instances

Two instances, one store. This is what a pair of replicas behind Part 9’s gateway cannot do on their own, and it is the reason the store tier exists.

RunnableAll tracks

serve-shared.sh

#!/usr/bin/env bash
# Purpose: start one of two vLLM instances that share a key-value store through a
#          directory both can read and write, so that a prefix one instance computed can
#          be reused by the other; run it twice, once as instance a and once as b
# Platform: spark, nvidia (vLLM's GPU path; Track X only where your ROCm build works)
# Minimum memory: 16 GB of device memory for two instances of a small model at
#          SPLIT_MEM_FRACTION each, or one instance per machine at MEM_FRACTION
# Assumes: vLLM installed as in Part 9 and on PATH; a .env copied from env-example.txt
#          with KV_SHARED_PATH set to a directory BOTH instances can read and write; on
#          two machines that is the shared mount from Part 18, on one machine any
#          directory will do. Nothing else listening on the port chosen.
#
# Usage: bash serve-shared.sh a      first instance,  port PREFILL_PORT
#        bash serve-shared.sh b      second instance, port DECODE_PORT
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

INSTANCE="${1:-}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Install it as Part 9 describes."

SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
CTX="${CTX:-8192}"
MAX_SEQS="${MAX_SEQS:-8}"
KV_SHARED_PATH="${KV_SHARED_PATH:-}"
SINGLE_MACHINE="${SINGLE_MACHINE:-0}"

case "$INSTANCE" in
    a) PORT="${PREFILL_PORT:-8100}" ;;
    b) PORT="${DECODE_PORT:-8200}" ;;
    *) fail "Usage: bash serve-shared.sh a|b" ;;
esac

if [ "$SINGLE_MACHINE" = "1" ]; then
    MODEL="${SMALL_MODEL:-Qwen/Qwen3-1.7B}"
    MEM_FRACTION="${SPLIT_MEM_FRACTION:-0.40}"
else
    MODEL="${MODEL:-Qwen/Qwen3-8B}"
    MEM_FRACTION="${MEM_FRACTION:-0.85}"
fi

[ -n "$KV_SHARED_PATH" ] || fail "KV_SHARED_PATH is not set. Both instances need the SAME directory."
mkdir -p "$KV_SHARED_PATH"
[ -w "$KV_SHARED_PATH" ] || fail "KV_SHARED_PATH (${KV_SHARED_PATH}) is not writable by this account."

# kv_both, not kv_producer or kv_consumer: this instance both writes blocks to the store
# and reads blocks from it, which is what sharing means as opposed to handing off.
KV_CONFIG="{\"kv_connector\":\"ExampleConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"shared_storage_path\":\"${KV_SHARED_PATH}\"}}"

cat <<INFO
==> vLLM instance ${INSTANCE}, sharing a key-value store
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${PORT}
    shared store      ${KV_SHARED_PATH}
    kv role           kv_both
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}

    Both instances must load the SAME model at the SAME context length. Two instances
    that disagree about either produce blocks the other cannot use, and the symptom is
    not an error: it is that nothing is ever reused and the store fills up.

    The measurement is measure-reuse.py with BOTH endpoints given, so that turns
    alternate. A fast second turn on the other endpoint is the store doing its job.

    The store is a directory of key-value blocks: it is the users' prompts in the
    model's own representation. Put it where you would put the transcripts, and delete
    it in cleanup.

INFO

exec vllm serve "$MODEL" \
    --host "$SERVE_HOST" \
    --port "$PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --max-num-seqs "$MAX_SEQS" \
    --gpu-memory-utilization "$MEM_FRACTION" \
    --enable-prefix-caching \
    --kv-transfer-config "$KV_CONFIG"

Download serve-shared.sh91 lines

RunnableAll tracks

the first instance, in its own terminal

SINGLE_MACHINE=1 bash serve-shared.sh a

RunnableAll tracks

the second instance, in another terminal

SINGLE_MACHINE=1 bash serve-shared.sh b

Both use kv_role set to kv_both, because each one writes blocks to the store and reads blocks from it. That is what distinguishes sharing from the handoff in the previous lab, where one instance only produced and the other only consumed.

RunnableAll tracks

turns alternating between the two instances

python3 measure-reuse.py \
  --base-url http://127.0.0.1:8100/v1 \
  --base-url http://127.0.0.1:8200/v1 \
  --model local-chat \
  --turns 6 \
  --preamble-words 1500 \
  --label shared-store \
  --labbook labbook.md \
  --note "two instances, one shared directory, kv_both on both"

The reading is simple. Turn 1 goes to instance A and is slow, because nothing is cached anywhere. Turn 2 goes to instance B. If turn 2 is nearly as slow as turn 1, the store is not being read. If it is much faster, instance B reused what instance A computed, and you have measured the thing this task exists for.

✦Tip

Run measure-reuse.py against a single instance with the store enabled as a control, using the same preamble and turn count. Two instances sharing a store should look like one instance from the client’s point of view, and comparing against a single instance is how you find out whether the store is adding latency of its own.

5. The disk tier, timed

llama.cpp gives the clearest view of a disk tier because you can time it directly: it saves a slot’s cache to a file on demand and reads it back on demand, so the tier’s latency is a stopwatch rather than an inference.

RunnableAll tracks

llama-cache-tiers.sh

#!/usr/bin/env bash
# Purpose: the reduced path for Tracks X and M, where vLLM's connectors are not available:
#          run llama-server with prompt caching, several slots and a quantised key-value
#          cache, then save a slot's cache to disk and bring it back, which is llama.cpp's
#          version of a disk tier
# Platform: strix, mac (and anywhere else llama.cpp runs; the flags are identical)
# Minimum memory: 16 GB; a four-bit 8B model with a 32,768-token cache fits comfortably
# Assumes: llama-server on PATH or in LLAMA_BIN, built for your accelerator as in Part 6;
#          a .env copied from env-example.txt with LLAMA_MODEL pointing at a GGUF file;
#          curl on PATH for the slot subcommands
#
# Usage: bash llama-cache-tiers.sh serve            start the server with the tiers on
#        bash llama-cache-tiers.sh slots            show what each slot is holding
#        bash llama-cache-tiers.sh save 0 name.bin  write slot 0's cache to disk
#        bash llama-cache-tiers.sh restore 0 name.bin  read it back into slot 0
#        bash llama-cache-tiers.sh erase 0          discard slot 0's cache
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

ACTION="${1:-serve}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
LLAMA_PORT="${LLAMA_PORT:-8080}"
LLAMA_MODEL="${LLAMA_MODEL:-}"
LLAMA_PARALLEL="${LLAMA_PARALLEL:-2}"
LLAMA_SLOT_SAVE_PATH="${LLAMA_SLOT_SAVE_PATH:-./slot-cache}"
LLAMA_CACHE_TYPE="${LLAMA_CACHE_TYPE:-q8_0}"
SERVED_NAME="${SERVED_NAME:-local-chat}"
CTX="${CTX:-8192}"
LLAMA_BIN="${LLAMA_BIN:-}"
BASE="http://${SERVE_HOST}:${LLAMA_PORT}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

SERVER="llama-server"
if [ -n "$LLAMA_BIN" ] && [ -x "${LLAMA_BIN}/llama-server" ]; then
    SERVER="${LLAMA_BIN}/llama-server"
fi

case "$ACTION" in
    serve)
        command -v "$SERVER" >/dev/null 2>&1 || [ -x "$SERVER" ] \
            || fail "llama-server is not on PATH and LLAMA_BIN does not contain it."
        [ -n "$LLAMA_MODEL" ] || fail "LLAMA_MODEL is not set in .env."
        [ -f "$LLAMA_MODEL" ] || fail "LLAMA_MODEL (${LLAMA_MODEL}) is not a file."
        mkdir -p "$LLAMA_SLOT_SAVE_PATH"

        cat <<INFO
==> llama-server with the cache tiers this track can reach
    model            ${LLAMA_MODEL}
    served as        ${SERVED_NAME}
    listening on     ${BASE}
    context          ${CTX}
    slots            ${LLAMA_PARALLEL}
    key-value type   ${LLAMA_CACHE_TYPE} for both keys and values
    slot save path   ${LLAMA_SLOT_SAVE_PATH}

    What each of these does, from the server README:
      --cache-prompt     prompt caching, which is enabled by default; passed explicitly
                         here so the run is self-documenting
      --parallel         the number of server slots, which is also the number of distinct
                         conversations that can stay warm at once
      -ctk / -ctv        the key and value cache data type; q8_0 halves the cache against
                         the f16 default, and unlike prompt caching this DOES change
                         outputs, so do not carry it into a comparison against an f16 run
      --slot-save-path   where "save slot kv cache" writes to; without it the save and
                         restore endpoints are disabled
      --metrics --slots  the two endpoints this lab reads

    The README also documents --cache-reuse, "min chunk size to attempt reusing from the
    cache via KV shifting", which is not used here and is not yet in this course's
    captured command reference. Check your build's --help before adding it.

INFO
        exec "$SERVER" \
            --model "$LLAMA_MODEL" \
            --alias "$SERVED_NAME" \
            --host "$SERVE_HOST" \
            --port "$LLAMA_PORT" \
            --ctx-size "$CTX" \
            --parallel "$LLAMA_PARALLEL" \
            --n-gpu-layers 999 \
            --cache-prompt \
            --cache-type-k "$LLAMA_CACHE_TYPE" \
            --cache-type-v "$LLAMA_CACHE_TYPE" \
            --slot-save-path "$LLAMA_SLOT_SAVE_PATH" \
            --metrics \
            --slots
        ;;

    slots)
        command -v curl >/dev/null 2>&1 || fail "curl is not on PATH."
        printf '==> What each slot is holding\n'
        curl -fsS "${BASE}/slots"
        printf '\n'
        ;;

    save|restore)
        command -v curl >/dev/null 2>&1 || fail "curl is not on PATH."
        SLOT="${2:-}"
        FILENAME="${3:-}"
        [ -n "$SLOT" ] || fail "Usage: bash llama-cache-tiers.sh ${ACTION} <slot> <filename>"
        [ -n "$FILENAME" ] || fail "Usage: bash llama-cache-tiers.sh ${ACTION} <slot> <filename>"
        printf '==> %s slot %s as %s\n' "$ACTION" "$SLOT" "$FILENAME"
        START="$(date +%s%N)"
        curl -fsS -X POST "${BASE}/slots/${SLOT}?action=${ACTION}" \
            -H "Content-Type: application/json" \
            -d "{\"filename\": \"${FILENAME}\"}"
        END="$(date +%s%N)"
        printf '\n    took %s ms\n' "$(( (END - START) / 1000000 ))"
        printf '    That figure is the disk tier''s latency for this conversation. Compare\n'
        printf '    it against the time the same prompt takes to prefill from cold.\n'
        ;;

    erase)
        command -v curl >/dev/null 2>&1 || fail "curl is not on PATH."
        SLOT="${2:-}"
        [ -n "$SLOT" ] || fail "Usage: bash llama-cache-tiers.sh erase <slot>"
        printf '==> Erasing slot %s. This discards its cache and cannot be undone.\n' "$SLOT"
        printf '    Check "bash llama-cache-tiers.sh slots" first if you are unsure.\n'
        curl -fsS -X POST "${BASE}/slots/${SLOT}?action=erase"
        printf '\n'
        ;;

    *)
        fail "Unknown action '${ACTION}'. Use: serve, slots, save, restore, erase."
        ;;
esac

Download llama-cache-tiers.sh134 lines

RunnableAll tracks

a server with prompt caching, several slots and a quantised cache

bash llama-cache-tiers.sh serve

RunnableAll tracks

send a long prompt, then look at what the slots hold

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Summarise this in one sentence."}],"max_tokens":16}' > /dev/null

bash llama-cache-tiers.sh slots

RunnableAll tracks

write slot 0 to disk, and time it

bash llama-cache-tiers.sh save 0 lab-22-session.bin

Output — what you should see

==> save slot 0 as lab-22-session.bin
    took ... ms
    That figure is the disk tier's latency for this conversation. Compare
    it against the time the same prompt takes to prefill from cold.

RunnableAll tracks

erase it, then bring it back, and time that too

bash llama-cache-tiers.sh erase 0
bash llama-cache-tiers.sh restore 0 lab-22-session.bin

Put the restore time next to the time to first token that the same prompt took from cold in task 2. The ratio between them is the whole argument for a disk tier, and it is a number you now have on your own hardware.

⌚Version note

--cache-prompt, --slot-save-path, -ctk and -ctv are all documented in the llama-server README read on 2026-09-09 against llama.cpp v0.4.0 · verified 2026-09-08. The same README documents --cache-reuse as “min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled (default: 0)”, which is not used by these scripts and is not yet in this course’s captured command reference. Check your build’s --help before adding it.

6. Decide, and write it down

Look at the notebook lines you now have and answer three questions in prose, in the notebook, in your own words.

Did the host tier pay? Compare offload-on-load against offload-off-load under pressure. If it did not, say at what memory fraction and concurrency you tested, because the answer depends on both.

Did the store pay? Compare the alternating run against a single instance with the same store.

What did the disk tier cost, and what would it save? The restore time against the cold prefill time.

When offload helps, and when it hurts

Your numbers are the authority for your machine. What follows is the set of conditions to test them against, so that a result you did not expect has somewhere to be filed.

It helps when the device cache is the binding constraint and the prompts repeat. A server whose startup log reports a maximum concurrency below the number of people using it is refusing or preempting work that a tier could have kept. A workload where the same corpus, the same system prompt or the same conversation comes back is a workload where the recalled blocks are blocks somebody wants. Both conditions have to hold: repeated prompts with a cache that never fills gains nothing, and a full cache with prompts that never repeat gains nothing either.

It helps when the tier is genuinely faster than recomputing. Recall costs a copy across a bus; prefill costs the whole prompt through the model. The longer the prompt, the more lopsided that comparison, which is why the tier is worth most to exactly the long-prompt workloads the previous lesson said a phase split was aimed at.

It hurts when the host memory it takes was doing something. KV_OFFLOAD_GB is not free memory discovered somewhere; it is taken from the page cache, from the other processes on the machine and, if you take too much, from the machine’s ability to avoid swapping. A tier that pushes a machine into swap is slower than having no tier at all, and the symptom is a general slowdown rather than a slow engine.

It hurts when device and host memory are the same memory. On the unified-memory tracks the copy that a discrete card pays to move blocks off the accelerator is a different and smaller operation, and the tier may return very little. That is a property of the machine and not a fault in the software, and it is one of the more useful things a Track S or Track X reader can put in the notebook, because most published guidance about offloading was written for discrete cards.

It hurts when it hides a configuration problem. A tier that makes a badly sized deployment tolerable is a tier that will be blamed later for the thing it was covering. If the maximum concurrency in the startup log is below one at your context length, no tier fixes that: a single sequence does not fit and the answer is a shorter context or a smaller model.

✕Common mistake

The measurement that is easiest to get wrong here is a comparison taken at different levels of pressure. Turning the tier on and also raising the memory fraction, or turning it on and also running fewer concurrent requests, produces a favourable number that means nothing. One variable, recorded, as in the previous lab.

Prove reuse rather than measuring ordinary warm-up

Use a cold request, an identical-prefix repeat and a changed-prefix control. Record model residency separately from cache state. If every request improves after the first, warm-up may explain the effect; cache-hit evidence and the negative control are needed to attribute it to reuse.

Before sharing between instances, verify the compatibility contract: model and adapter identity, tokeniser/template, cache dtype/layout and connector settings. Do not reuse a cache just because the filename or model alias matches. Test rejection or invalidation when an identity changes.

For offload, record bytes moved, storage tier, transfer time and the workload’s reuse frequency. Force the documented pressure condition gradually and retain evictions and misses. An offloaded cache can save capacity while increasing latency; report both. Keep sensitive cache data within the intended access and retention boundary. At cleanup, stop the lab services before removing their temporary cache files and retain configuration plus measurements. The completion criterion is a demonstrated hit/miss distinction and an explained cost/capacity tradeoff, not merely the existence of a cache directory or a successful second request.

Validation

• labbook.md contains reuse lines with labels offload-off and offload-on from the same model, turn count and preamble length.
• labbook.md contains load-generator lines with labels offload-off-load and offload-on-load at the same concurrency levels, taken with the cache under pressure, and you recorded the memory fraction used.
• labbook.md contains a reuse line labelled shared-store with two endpoints listed, or a note saying your track cannot run it and why.
• You recorded the save and restore times for one slot, and the time to first token the same prompt took from cold.
• You wrote three sentences answering task 6, with the settings each answer depends on.

Expected outcome

Pending validationWhat this lab measures — to be filled in by the validation pass and by you

Tier	How it was measured	Turn 1 TTFT (s)	Best later turn TTFT (s)	Prefix cache hit rate
None, quiet server	six turns, one instance	pending	pending	pending
Host memory, quiet server	six turns, one instance	pending	pending	pending
None, cache under pressure	load generator at concurrency 8	pending	pending	pending
Host memory, cache under pressure	load generator at concurrency 8	pending	pending	pending
Shared store, two instances	six turns alternating	pending	pending	pending
Disk, llama.cpp slot save and restore	save, erase, restore, timed	pending	pending	not applicable

the reference cluster machines described in Part 18, one at a time, DGX OS 7.x, Ubuntu 24.04 and macOS as each track requires · vLLM, and llama.cpp for the disk tier to be recorded by the validation pass · Qwen3-8B, or Qwen3-1.7B for two instances on one device, bf16 for vLLM; a four-bit GGUF for llama.cpp, with an eight-bit key-value cache · 8,192 tokens of context · 2026-09-09

No row here has been measured. Your own six rows go in the notebook; this table is where the validation pass will put its own.

What this lab expects: the host tier does nothing on a quiet server and something under pressure, with the size of the something depending on how separate your device memory is from your host memory. On a discrete card the gap between the two is real and the tier should show. On a unified-memory machine the two are the same physical memory, and a tier that copies between them may buy very little; that is a finding worth writing down rather than a disappointment.

The shared store should make turn 2 on the second instance much faster than turn 1 on the first. The disk tier’s restore should be dramatically faster than a cold prefill of the same prompt, which is what makes it worth having for a conversation you will come back to.

Troubleshooting

--kv-offloading-size is rejected. Run vllm serve --help | grep -i offload and record what your build accepts. The two options used here are on the CLI reference page as read on 2026-09-09; if yours has only --kv-transfer-config, use the OffloadingConnector form from Lesson 3 instead. Check you have not reached for --cpu-offload-gb, which offloads weights.

Nothing changes when the tier is enabled. Almost always because the device cache was never full. Lower MEM_FRACTION, raise the concurrency, or lengthen the preamble, and check the startup log’s maximum concurrency figure to confirm the cache is actually the constraint.

Turn 2 is as slow as turn 1 with a shared store. The two instances disagree about something. Check they loaded the same model repository at the same context length, that KV_SHARED_PATH is the same directory in both terminals, and that the directory has files in it after turn 1.

The machine starts swapping. KV_OFFLOAD_GB is host memory taken from everything else. Lower it. A tier that pushes the machine into swap is slower than recomputing the prefill, which is one of the two ways offload hurts.

Slot save returns an error. --slot-save-path must be set for the save and restore endpoints to exist at all; the README lists it as disabled by default. The script sets it from LLAMA_SLOT_SAVE_PATH and creates the directory, so check that variable and the account’s write permission on it.

The quantised cache changed the answers. Expected, and Part 17 said so. -ctk and -ctv round the stored keys and values, unlike prompt caching, which reuses them unchanged. Do not carry an eight-bit cache into a comparison against a sixteen-bit one.

Cleanup

Stop the servers. Then remove the two things this lab created.

RunnableAll tracks

remove the shared store and the saved slot files

rm -rf "${KV_SHARED_PATH:?set KV_SHARED_PATH first}" \
       "${LLAMA_SLOT_SAVE_PATH:-./slot-cache}"

⛨Security

Both of those directories hold conversations in the model’s internal representation, including anything private that was pasted into a prompt. A saved slot file is the conversation, as Part 17 put it, and a shared store is every conversation that passed through either instance. Delete them when you are done, and while they exist put them where you would put the transcripts and restrict them the same way. A store on a mount that every machine in the house can read is a different exposure from a file in your home directory.

What you learned

• A tier does nothing until the thing above it is full. The host-memory tier changed nothing on a quiet server and mattered under pressure. Any measurement of a cache tier that does not say what the pressure was is not a measurement of anything.
• Weights and cache are different offloads. --cpu-offload-gb moves model weights and --kv-offloading-size moves key-value blocks, and the two options solve different problems despite the similar names.
• Two replicas do not share a cache unless you give them one. A store is what turns two independent servers into two servers with one memory, and the alternating-turn measurement is how you prove it happened.
• The disk tier has a latency you can time. llama.cpp’s slot save and restore turn an abstract tier into a stopwatch, and the ratio of restore time to cold prefill time is the whole argument.
• On unified memory the tiers mean something different. Device memory and host memory being the same physical memory changes what an offload can buy, and noticing that on your own machine is worth more than repeating a rule of thumb from a discrete-card deployment.
• These techniques usually beat the split at home. The previous lab moved a gigabyte across a cable once per request. This one avoided moving anything, and on most home networks that is where the return is.

Record in the notebook: the engine and version; the model, quantisation and context length; the memory fraction and concurrency each measurement was taken at; the key-value cache size and maximum concurrency from each startup log; the six rows of the table above with your own numbers; the slot save and restore times against a cold prefill; and three sentences saying which tiers you will keep. The project uses all of it.

Check your understanding

Question 1. You enable the host-memory tier on an idle server, run six conversation turns, and nothing changes. What does that tell you?

• The tier is broken
• Nothing is wrong: the device cache was never full, so nothing was evicted and there was nothing to recall. A tier is only measurable under pressure.
• The model is too small
• The turns were too short

Show the answer and why

Answer: Nothing is wrong: the device cache was never full, so nothing was evicted and there was nothing to recall. A tier is only measurable under pressure.

This is why the lab lowers the memory fraction and raises the concurrency before drawing any conclusion. A cache-tier benchmark with no statement of what the pressure was cannot be interpreted, by you or by anyone reading your notebook.

Question 2. What is the difference between --cpu-offload-gb and --kv-offloading-size in vLLM?

• They are aliases for the same option
• --cpu-offload-gb offloads model weights to host memory, which the engine argument documentation describes as a virtual way to increase GPU memory size; --kv-offloading-size sets the size of the key-value cache offloading buffer
• --cpu-offload-gb is for AMD hardware
• --kv-offloading-size sets the device cache size

Show the answer and why

Answer: --cpu-offload-gb offloads model weights to host memory, which the engine argument documentation describes as a virtual way to increase GPU memory size; --kv-offloading-size sets the size of the key-value cache offloading buffer

Two options, similar names, different problems. Reaching for the weights option when you meant the cache one gives you a model that streams parameters over the bus on every forward pass, which is slow for a reason that has nothing to do with caching.

Question 3. In the two-instance task, both servers set kv_role to kv_both rather than producer and consumer. Why?

• Because the option is required on every instance
• Because sharing means each instance both writes blocks to the store and reads blocks from it, unlike the handoff in the previous lab where one instance only produced and the other only consumed
• Because it doubles throughput
• Because the store cannot tell the instances apart

Show the answer and why

Answer: Because sharing means each instance both writes blocks to the store and reads blocks from it, unlike the handoff in the previous lab where one instance only produced and the other only consumed

The roles describe what an instance does with the store. A producer writes and a consumer reads, which is a one-way handoff. Two replicas that should each benefit from the other's work both do both, which is what the combined role expresses.

Question 4. Which of these are honest conclusions from this lab? Select all that apply.

• On a unified-memory machine, a host-memory tier may buy very little because device and host memory are the same physical memory
• A tier that pushes the machine into swap is slower than recomputing the prefill
• The ratio of slot restore time to cold prefill time is the argument for a disk tier
• An eight-bit key-value cache is free, like prompt caching

Show the answer and why

Answer: On a unified-memory machine, a host-memory tier may buy very little because device and host memory are the same physical memory, A tier that pushes the machine into swap is slower than recomputing the prefill, The ratio of slot restore time to cold prefill time is the argument for a disk tier

Quantising the cache rounds the stored keys and values, so unlike prefix reuse it can change outputs. Part 17 drew the same line: reuse is bit-identical and quantisation is not, which is why the two belong in different categories even though both save memory.

Question 5. Why does this lab claim these techniques usually beat the two-machine split on a home network?

• Because the split is broken
• Because the split moves a request's whole key-value cache across a link once per request, while these techniques avoid moving it at all; on a link measured in Part 18 the transfer is often slower than the prefill it replaced
• Because vLLM optimises them better
• Because the tiers are newer

Show the answer and why

Answer: Because the split moves a request's whole key-value cache across a link once per request, while these techniques avoid moving it at all; on a link measured in Part 18 the transfer is often slower than the prefill it replaced

Lesson 2 did the arithmetic and the previous lab measured it. Not recomputing something is cheaper than moving it, and it needs no cable at all. That is why the second lab is the one most readers should keep the results of.

Check my answersClear