Prefix Caching and KV Reuse

Speculative decoding attacks the second half of a request. This lesson attacks the first half, and on the workloads most people actually run it is the larger win. By the end you will be able to say exactly what a prefix cache can and cannot reuse, turn it on and prove it is being hit on all four tracks, arrange your prompts so that hits happen instead of hoping they do, name the flags that quantise the cache and what they cost, and save a conversation’s cache to disk and bring it back.

What is actually reusable

A key-value cache holds, for every token the model has read, the key and value vectors each attention layer computed for it. Those vectors depend on the token and on every token before it, and on nothing else. So if two requests begin with the identical sequence of token ids, the cache entries for that shared beginning are identical too, and the second request does not need to compute them again. It only needs to prefill the part that differs.

Three consequences follow, and every practical detail in this lesson is one of them.

The match has to be a prefix, from position zero. A document that appears in the middle of two prompts shares nothing reusable if the text before it differs, because every key and value in the document depends on what preceded it. The match has to be on tokens, not on characters: a prompt that differs by one space may tokenise differently from that point on. And because the reuse is exact, the output cannot change. vLLM’s usage page says so plainly: caching “won’t change model outputs” while avoiding “redundant prompt computations”.

The same long document, asked two questions

Request 1, cold

prefill: document + question Adecode

Request 2, no cache

prefill: document + question Bdecode

Request 2, cache hit

reusedprefill Bdecode

Prefill is compute-bound and proportional to the prompt; decode is bandwidth-bound and proportional to the answer. On the second request the shared document is read from the cache instead of recomputed, so only the new question is prefilled. Nothing about decode changes, which is why prefix caching helps long prompts with short answers and does almost nothing for short prompts with long answers.

Four engines, four names for the same idea

llama.cpp calls it prompt caching and has it on already. The server README, as read on 2026-09-09, documents --cache-prompt and --no-cache-prompt as “whether to enable prompt caching (default: enabled)”, with a matching cache_prompt parameter on the completion endpoint. It works per slot: llama-server divides its context into slots, one per concurrent conversation, and a slot keeps the tokens it last processed so that the next request into that slot reuses the common prefix. The practical consequence is that the reuse is tied to a slot, so the number of parallel slots you configure is also the number of distinct conversations that can stay warm at once.

vLLM calls it automatic prefix caching, and its design page explains the mechanism: “we hash each kv-cache block by the tokens in the block and the tokens in the prefix before the block”, with the parent block’s hash folded in, plus extra hashes for things like adapter ids and multimodal inputs. Blocks are held in a free queue and evicted least-recently-used first, which the page describes as popping “the block from the head of the free queue. This is the LRU block to be evicted.” Because the hash covers the prefix, a hit is exact by construction rather than by coincidence. The usage page names the two workloads it is for: a “Long Document Query”, where “the user repeatedly queries the same long document (e.g. software manual or annual report) with different queries”, and a “Multi-Round Conversation”. It also sets expectations honestly, saying caching “in general does not reduce the performance of vLLM” and that it only reduces prefill time, with limited benefit when answers are long or prompts share no prefix.

SGLang calls it RadixAttention, and it is the most general of the four. The announcement post describes retaining “the KV cache for both prompts and generation results in a radix tree”, a structure that “enables efficient prefix search, insertion, and eviction”, with “a Least Recently Used (LRU) eviction policy, complemented by a cache-aware scheduling policy, to enhance the cache hit rate”. Caching the generated tokens as well as the prompt is what makes it fit branching workloads: the post names few-shot examples, questions in self-consistency, chat history in multi-turn chat, and search history in tree-of-thought as the reuse patterns it handles. The switch is negative: --disable-radix-cache turns it off.

mlx-lm does it differently, and the difference is instructive. There is no server-managed pool; there is a file. The README shows mlx_lm.cache_prompt writing a cache with --prompt-cache-file, and mlx_lm.generate reading it back with the same option. That is a weaker mechanism for a busy server and a stronger one for a workstation: the cache survives the process, and you can build it once for a document you will ask about for a week.

RunnableTrack M · Apple silicon

build a prompt cache once, then reuse it from the file

mlx_lm.cache_prompt \
  --model mlx-community/Qwen3-8B-4bit \
  --prompt - \
  --prompt-cache-file manual-cache.safetensors < manual.txt

mlx_lm.generate \
  --prompt-cache-file manual-cache.safetensors \
  --prompt "Summarise the section on power limits."

⌚Version note

The two mlx-lm options above are from the README and the argument parser in mlx_lm/generate.py, both read on 2026-09-09, and they are not yet in this course’s captured command reference for mlx-lm 0.31.3 · verified 2026-09-08. Check --help on your installed version before scripting them.

Proving a cache is being hit

A prefix cache that is enabled and never hit behaves exactly like one that is switched off, and the failure is silent. So measure it two ways and read them against each other.

The first way is the counters. vLLM’s production metrics page documents vllm:prefix_cache_hits (“Prefix cache hits, in terms of number of cached tokens”) and vllm:prefix_cache_queries (“Prefix cache queries, in terms of number of queried tokens”); the ratio between them over a run is the hit rate in tokens. SGLang publishes Prometheus metrics when started with --enable-metrics, and its running log line reports token usage against the cache pool. llama-server exposes /metrics when started with --metrics and a /slots endpoint when started with --slots.

The second way is the clock, and it works even where an engine exposes nothing. Send a long shared prefix followed by a short different question, with the answer capped at a single token so that almost all of the measured time is prefill, and compare the first request against the ones after it. A warm request should be dramatically shorter than the cold one; if it is not, either the cache is off, the prefix is not matching, or the prefix is too short to matter.

RunnableAll tracks

measure-prefix-cache.py

#!/usr/bin/env python3
"""Measure whether a server is really reusing the prefix it already computed.

Purpose: send one long shared prefix followed by several different short questions, with
    max_tokens set to 1 so that almost all of the measured time is prefill, and compare
    the first (cold) request against the later (warm) ones. Where the server exposes
    prefix-cache counters on a Prometheus endpoint, the script reports what the run added
    to them, so the timing and the counters can be read against each other. A cache that
    is on but never hit looks exactly like a cache that is off, and only these two
    measurements together tell them apart.
Platform: all (spark, strix, mac, nvidia). Pure Python standard library: no pip install.
Minimum memory: 8 GB on the machine running this script; the floor is set by the server.
Assumes: a server listening and answering POST /v1/chat/completions over plain HTTP (a
    localhost or LAN tool; it does not speak TLS). Run it twice, once with the engine's
    prefix cache enabled and once with it disabled, and compare the two notebook lines.
    Results are appended to the lab notebook as one JSON line per run.

Usage:
    python3 measure-prefix-cache.py --base-url http://127.0.0.1:8000/v1 \\
        --model local-chat --label vllm-apc-on --labbook labbook.md

    python3 measure-prefix-cache.py --base-url http://127.0.0.1:8080/v1 \\
        --model local-chat --prefix-words 1200 --questions 6 \\
        --label llama-server-cache-prompt --labbook labbook.md

    # a prefix of your own: a document, a system prompt, a tool schema
    python3 measure-prefix-cache.py --base-url http://127.0.0.1:8000/v1 \\
        --model local-chat --prefix-file manual.txt --label rag-manual

An API key, if the server needs one, is read from the environment variable named by
--api-key-env. No key is written to this file, to the output or to the notebook.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import statistics
import sys
import time
import urllib.error
import urllib.request
from urllib.parse import urlsplit

# --------------------------------------------------------------------------- prompts

# The synthetic prefix is deliberately dull and self-similar: what is being measured is
# how long the engine takes to read tokens it has already read, not what it makes of them.
PREFIX_SENTENCES = [
    "The service runs one open-weight language model on a single machine.",
    "Memory is divided between the model weights, the key-value cache and the engine's working buffers.",
    "Prefill reads the prompt and is compute-bound; decode writes the answer and is bandwidth-bound.",
    "The key-value cache is charged per token and comes out of a pool reserved at start-up.",
    "Every measurement is recorded with the hardware, the engine version and the date.",
    "A benchmark without its context is a number somebody will misquote.",
    "Concurrency is the size of the pool divided by the context length each conversation is allowed.",
    "The operator cares about time to first token, time per output token and goodput.",
]

QUESTIONS = [
    "Reply with the single word: one.",
    "Reply with the single word: two.",
    "Reply with the single word: three.",
    "Reply with the single word: four.",
    "Reply with the single word: five.",
    "Reply with the single word: six.",
    "Reply with the single word: seven.",
    "Reply with the single word: eight.",
]


def build_prefix(words: int, path: str | None) -> str:
    """A file's contents, or a synthetic block of roughly `words` words."""
    if path:
        with open(path, encoding="utf-8") as handle:
            text = handle.read().strip()
        if not text:
            raise SystemExit(f"{path} is empty")
        return text
    out: list[str] = []
    count = 0
    i = 0
    while count < words:
        sentence = PREFIX_SENTENCES[i % len(PREFIX_SENTENCES)]
        out.append(f"[{i:04d}] {sentence}")
        count += len(sentence.split()) + 1
        i += 1
    return "\n".join(out)


# ------------------------------------------------------------------------ HTTP


class ServerError(Exception):
    """The server did not answer in a way this script can use."""


def post_json(url: str, payload: dict, api_key: str, timeout: float) -> dict:
    """One non-streaming POST. Returns the decoded JSON body."""
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(url, data=body, method="POST")
    request.add_header("Content-Type", "application/json")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http only, checked in main
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:200]
        raise ServerError(f"HTTP {exc.code} from {url}: {detail}") from exc
    except (urllib.error.URLError, OSError, json.JSONDecodeError) as exc:
        raise ServerError(f"{type(exc).__name__} talking to {url}: {exc}") from exc


def get_text(url: str, api_key: str, timeout: float) -> str | None:
    """GET a text body, or None if the endpoint is absent or refuses."""
    request = urllib.request.Request(url, method="GET")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http only
            return response.read().decode("utf-8", "replace")
    except (urllib.error.URLError, OSError):
        return None


# ------------------------------------------------------------- Prometheus scraping

# Generic on purpose: engines rename metrics between releases, so this reports whatever
# the server publishes whose name mentions a cache, rather than asserting that a
# particular counter exists. Nothing matched is itself a result worth recording.
CACHE_METRIC = re.compile(r"(prefix_cache|cache_hit|cache_quer|radix|kv_cache)", re.IGNORECASE)
SAMPLE = re.compile(r"^(?P<name>[A-Za-z_:][A-Za-z0-9_:]*)(?P<labels>\{[^}]*\})?\s+(?P<value>[-+0-9.eE]+|NaN)$")


def scrape_metrics(metrics_url: str, api_key: str, timeout: float) -> dict[str, float]:
    """Reads a Prometheus exposition page and keeps the cache-related samples."""
    text = get_text(metrics_url, api_key, timeout)
    if text is None:
        return {}
    out: dict[str, float] = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        match = SAMPLE.match(line)
        if not match or not CACHE_METRIC.search(match.group("name")):
            continue
        try:
            value = float(match.group("value"))
        except ValueError:
            continue
        out[match.group("name") + (match.group("labels") or "")] = value
    return out


def metrics_delta(before: dict[str, float], after: dict[str, float]) -> dict[str, float]:
    """What this run added, so a warm server's history does not pollute the record."""
    keys = set(before) | set(after)
    return {k: round(after.get(k, 0.0) - before.get(k, 0.0), 4) for k in sorted(keys)}


# ------------------------------------------------------------------------ measurement


def one_request(cfg: dict, prefix: str, question: str) -> dict:
    """One prefix-plus-question request with max_tokens 1, timed end to end.

    max_tokens is 1 so that the measurement is dominated by prefill. It does not
    isolate prefill perfectly - the response still travels back over HTTP and one
    decode step still happens - but the constant is the same for every request in the
    run, so the cold-to-warm difference is the cache and not the overhead.
    """
    payload = {
        "model": cfg["model"],
        "messages": [
            {"role": "system", "content": prefix},
            {"role": "user", "content": question},
        ],
        "max_tokens": 1,
        "temperature": 0.0,
        "stream": False,
    }
    started = time.perf_counter()
    body = post_json(cfg["chat_url"], payload, cfg["api_key"], cfg["timeout"])
    elapsed = time.perf_counter() - started
    usage = body.get("usage") or {}
    return {
        "elapsed_s": elapsed,
        "prompt_tokens": usage.get("prompt_tokens"),
        "timings": body.get("timings") if isinstance(body.get("timings"), dict) else None,
    }


def run(cfg: dict, prefix: str, questions: list[str]) -> dict:
    """Cold request, then the warm ones, with the cache counters read either side."""
    before = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"])

    try:
        cold = one_request(cfg, prefix, questions[0])
    except ServerError as exc:
        return {"error": str(exc)}

    warm: list[dict] = []
    errors: list[str] = []
    for question in questions[1:]:
        try:
            warm.append(one_request(cfg, prefix, question))
        except ServerError as exc:
            errors.append(str(exc))

    after = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"])
    warm_times = [w["elapsed_s"] for w in warm]

    return {
        "prefix_chars": len(prefix),
        "prompt_tokens": cold["prompt_tokens"],
        "cold_s": round(cold["elapsed_s"], 4),
        "warm_s": {
            "count": len(warm_times),
            "p50": round(statistics.median(warm_times), 4) if warm_times else None,
            "mean": round(statistics.fmean(warm_times), 4) if warm_times else None,
            "min": round(min(warm_times), 4) if warm_times else None,
            "max": round(max(warm_times), 4) if warm_times else None,
        },
        "warm_over_cold": (
            round(statistics.median(warm_times) / cold["elapsed_s"], 4)
            if warm_times and cold["elapsed_s"] > 0 else None
        ),
        "server_timings_sample": cold["timings"],
        "cache_metrics_delta": metrics_delta(before, after),
        "failed": len(errors),
        "first_error": errors[0] if errors else None,
    }


def append_labbook(path: str, record: dict) -> None:
    """Appends one JSON line, in the course notebook format."""
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base-url", default="http://127.0.0.1:8000/v1",
                        help="OpenAI-compatible base URL, ending in /v1")
    parser.add_argument("--model", required=True, help="model name the server reports at /v1/models")
    parser.add_argument("--prefix-words", type=int, default=800,
                        help="approximate length of the synthetic shared prefix, in words")
    parser.add_argument("--prefix-file", default=None,
                        help="use this file as the shared prefix instead of the synthetic one")
    parser.add_argument("--questions", type=int, default=6,
                        help="requests to send: the first is cold, the rest should hit the cache")
    parser.add_argument("--timeout", type=float, default=300.0)
    parser.add_argument("--label", default="prefix-cache", help="tag written into the notebook line")
    parser.add_argument("--engine", default="unknown", help="engine name recorded in the notebook")
    parser.add_argument("--engine-version", default="unknown", help="engine version recorded in the notebook")
    parser.add_argument("--api-key-env", default="SPEC_API_KEY",
                        help="environment variable holding the API key, if the server needs one")
    parser.add_argument("--labbook", default="labbook.md")
    return parser.parse_args(argv)


def main(argv: list[str]) -> int:
    args = parse_args(argv)

    parts = urlsplit(args.base_url)
    if parts.scheme != "http":
        print("This tool speaks plain HTTP only; point it at a localhost or LAN endpoint.", file=sys.stderr)
        return 2
    if not parts.hostname:
        print(f"Could not read a host from --base-url {args.base_url!r}.", file=sys.stderr)
        return 2

    root = f"{parts.scheme}://{parts.netloc}"
    path = parts.path.rstrip("/") or "/v1"
    cfg = {
        "chat_url": f"{root}{path}/chat/completions",
        "metrics_url": f"{root}/metrics",
        "model": args.model,
        "timeout": args.timeout,
        "api_key": os.environ.get(args.api_key_env, ""),
    }

    if args.questions < 2:
        print("--questions must be at least 2: one cold request and one warm one.", file=sys.stderr)
        return 2
    questions = [QUESTIONS[i % len(QUESTIONS)] for i in range(args.questions)]
    prefix = build_prefix(args.prefix_words, args.prefix_file)

    print(f"==> {args.label}: {args.model} at {args.base_url}")
    print(f"    shared prefix of {len(prefix)} characters, {args.questions} request(s), max_tokens 1")

    result = run(cfg, prefix, questions)
    if "error" in result:
        print(f"    the first request failed: {result['error']}", file=sys.stderr)
        return 1

    print(f"    prompt      {result['prompt_tokens']} tokens (as counted by the server)")
    print(f"    cold        {result['cold_s']:.3f} s")
    if result["warm_s"]["p50"] is not None:
        print(f"    warm        p50 {result['warm_s']['p50']:.3f} s over {result['warm_s']['count']} request(s)")
        print(f"    ratio       warm/cold {result['warm_over_cold']:.3f}")
    if result["cache_metrics_delta"]:
        print("    cache counters this run added:")
        for name, value in result["cache_metrics_delta"].items():
            print(f"      {name} += {value}")
    else:
        print("    cache       the server exposed no cache counters; the timing is your only evidence")
    if result["first_error"]:
        print(f"    first error {result['first_error']}")

    record = dict(result)
    record.update({
        "lab": "part-17/measure-prefix-cache",
        "label": args.label,
        "engine": args.engine,
        "engine_version": args.engine_version,
        "model": args.model,
        "base_url": args.base_url,
        "prefix_source": args.prefix_file or f"synthetic-{args.prefix_words}-words",
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    })
    append_labbook(args.labbook, record)
    print(f"    appended 1 line to {args.labbook}")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download measure-prefix-cache.py332 lines

RunnableAll tracks

cold against warm, and the counters either side

python3 measure-prefix-cache.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model local-chat \
  --prefix-words 1200 \
  --questions 6 \
  --engine vllm \
  --label apc-on \
  --labbook labbook.md

Run it twice, once with the cache enabled and once with it disabled, and compare the two notebook lines. That pair is the evidence; a single warm-looking number is not.

RunnableTrack N · NVIDIA GPU

the same server with prefix caching off, for the comparison

vllm serve Qwen/Qwen3-8B \
  --served-model-name local-chat \
  --port 8000 \
  --max-model-len 8192 \
  --no-enable-prefix-caching

Arranging prompts so that hits happen

This is the part that pays for the lesson. A cache hit needs an exact token prefix, so anything that varies early destroys every hit behind it. The fix is always the same: put the stable material first and the varying material last.

• A timestamp, a request id or a session id at the top of a system prompt invalidates the whole prompt on every request. Move it to the end, or drop it.
• A tool or function list serialised from a dictionary with unstable ordering changes bytes without changing meaning. Sort it once and keep the order fixed.
• Per-user preamble before a shared corpus splits the cache by user. Put the corpus first and the user’s context after it.
• Retrieval that re-ranks the same passages into a different order on each turn shares nothing with the previous turn. Keep an order that is stable when the retrieved set is stable.
• A trailing whitespace difference or a re-wrapped paragraph re-tokenises from that point on. Build prompts from a template, not by string concatenation at three call sites.

✦Tip

Test the arrangement, do not reason about it. Send the same request twice through measure-prefix-cache.py with your real system prompt in --prefix-file, and look at whether the counters moved. Ten minutes of that finds the timestamp nobody remembered putting at the top.

Quantising the cache

The cache is charged per token, and on a long context it can rival the weights. For Qwen3-8B, whose recorded shape is thirty-six layers with eight key-value heads of head dimension 128, one token of cache at 16 bits costs both a key and a value in every layer.

Qwen3-8B at Q4_K_M with a 32,768-token context, on a 24 GB machine

Weights (Q4_K_M)

5 GB

KV cache at 16-bit, 32k tokens

4.8 GB

Engine buffers and activations

1.5 GB

Free

12.7 GB

Total

24 GB

Estimates from the shape recorded in the course model reference, not measurements: weights from the published file size, cache from layers × key-value heads × head dimension × 2 tensors × bytes per element × tokens. Halving the bytes per element halves the cache, and at this context length that is most of a gigabyte and a half back.

Every engine will store those tensors in fewer bits if you ask. llama.cpp’s server README documents --cache-type-k and --cache-type-v, both defaulting to f16, with q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, bf16 and f32 among the accepted types. vLLM and SGLang both take --kv-cache-dtype; SGLang’s server arguments page lists auto, fp8_e5m2, fp8_e4m3, bf16, nvfp4 and fp4_mx_block16, with auto meaning the model’s own type. mlx-lm’s generate takes --kv-bits (“Number of bits for KV cache quantization. Defaults to no quantization”) and --quantized-kv-start, which delays quantisation until a given step so that the earliest, most-attended tokens stay at full precision.

RunnableAll tracks

an 8-bit key-value cache on llama-server

llama-server \
  --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
  --alias local-chat \
  --port 8080 \
  --ctx-size 32768 \
  --n-gpu-layers 999 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --metrics \
  --slots

!Important

Quantising the cache is not free the way prefix caching is. Prefix caching reuses bytes that are bit-identical to the ones it replaced; cache quantisation rounds them, and rounding changes outputs. This lesson gives you the flags. The part on quantisation and evaluation measures what they cost on a held-out set, which is the only way to decide whether the memory you got back was worth it. Do not carry an 8-bit cache into a benchmark you intend to compare with a 16-bit one.

Saving and restoring a cache

An in-memory cache dies with the process, and on a workstation that is a real cost: reopening a long conversation after a restart means paying for the whole prefill again. llama-server can write a slot’s cache to disk. Started with --slot-save-path, it accepts POST /slots/{id_slot}?action=save, ?action=restore and ?action=erase, alongside GET /slots for the current state.

RunnableAll tracks

save slot 0 to disk, then bring it back

curl -s -X POST "http://127.0.0.1:8080/slots/0?action=save" \
  -H "Content-Type: application/json" \
  -d '{"filename": "review-session.bin"}'

curl -s -X POST "http://127.0.0.1:8080/slots/0?action=restore" \
  -H "Content-Type: application/json" \
  -d '{"filename": "review-session.bin"}'

⛨Security

A saved slot file is the conversation. It contains the model’s internal representation of every token in that prompt, including anything private that was pasted into it, and GET /slots returns what each slot is currently holding. Put the save path on the same disk you would put the transcripts on, restrict it as you would restrict them, and do not expose /slots beyond localhost. Part 9’s project put authentication in front of the gateway for this class of reason; the same rule applies here.

⚠Warning

?action=erase discards a slot’s cache and cannot be undone, and the files under the save path are ordinary files that a cleanup script will happily delete. Neither loses data you cannot regenerate, but both cost you the prefill again. Check which slot you are addressing before you erase it: GET /slots tells you what is in each one.

Who benefits most

Two later parts are built on workloads this lesson makes cheap.

Agents, in the part on tool calling and the agent loop, resend the entire transcript on every turn: the system prompt, the tool schemas, and every observation so far, with one new message on the end. That is the ideal shape for a prefix cache, because the growing part is appended and everything before it is unchanged. An agent loop without prefix caching pays for the whole history at every step, and the cost grows with the square of the number of steps. With it, each step prefills roughly what was added. This is why the arrangement rules above matter more for agents than anywhere else: one timestamp in the system prompt turns the good case back into the bad one.

Retrieval, in the part on putting models to work, sends the same corpus chunks to the model again and again as different users ask about the same documents. Ordering the retrieved context stably, and putting it before the user’s question rather than after it, is the difference between a corpus that is prefilled once and one that is prefilled per request.

Cache identity includes the computation that produced it

Identical text is insufficient if tokenisation, special tokens, model weights, adapter, position handling or cache representation changed. The cached tensors represent a particular computation on a particular token prefix. Record these identities when testing persistence or transfer between processes.

Use a cold request, an identical-prefix repeat and a changed-prefix control. Compare prompt latency and engine cache-hit evidence. Keep model residency and warm-up separate: loading the model once can accelerate all later requests without proving prefix reuse. Test eviction by adding competing prefixes, then repeating the original request.

For multi-user systems, define whether cache sharing is allowed across users and how sensitive prompts are isolated and expired. A performance cache can retain derived information even if the original request log is deleted. Follow the engine’s documented isolation controls and test the intended boundaries. Prefix caching preserves useful work when its identity and lifecycle are correct; it should not be treated as a semantic memory that recognises paraphrases or an unrestricted store shared by every caller.

Recap

A prefix cache reuses the key and value vectors for a shared token prefix, which is exact and cannot change the output, but only ever matches from position zero and only on identical tokens. llama.cpp enables prompt caching by default and works per slot; vLLM hashes each block together with its prefix and evicts least-recently-used; SGLang keeps prompts and generations in a radix tree with LRU eviction and cache-aware scheduling; mlx-lm writes an explicit cache file you carry between runs. Prove a hit with the engine’s counters and with a cold-versus-warm timing, because an unhit cache is invisible. Arrange prompts so the stable material comes first, which is the single change that turns a theoretical cache into a real one. Quantise the cache with --cache-type-k and --cache-type-v, --kv-cache-dtype or --kv-bits when you need the memory, remembering that unlike prefix caching this one does change outputs. And save a slot to disk when a long conversation is worth more than the file it occupies.

Check your understanding

Question 1. Your system prompt ends with the retrieved document and begins with the current time. Prefix caching is enabled but the hit counters barely move. Why?

• The document is too long to cache
• The cache matches only an exact token prefix from position zero, and a timestamp at the top changes the first block on every request, so nothing behind it can be reused
• Prefix caching does not work with retrieved documents
• The counters only update once per hour

Show the answer and why

Answer: The cache matches only an exact token prefix from position zero, and a timestamp at the top changes the first block on every request, so nothing behind it can be reused

Every key and value depends on all preceding tokens, so a change at position five invalidates position five hundred. Moving the timestamp to the end of the prompt, after the stable material, converts a permanent miss into a hit on everything above it. This is the single most common reason a cache that is switched on does nothing.

Question 2. Which of these are true of prefix caching as the engines document it? Select all that apply.

• It reduces prefill time and does not change the speed of decode
• It does not change the model's outputs
• It gives the largest benefit on long shared prompts with short answers
• It reduces the memory the key-value cache occupies

Show the answer and why

Answer: It reduces prefill time and does not change the speed of decode, It does not change the model's outputs, It gives the largest benefit on long shared prompts with short answers

Reuse is bit-identical, so outputs are unchanged, and vLLM's usage page says as much. It saves recomputation of the prompt only, which is why the benefit shrinks as answers grow. It does not shrink the cache: the blocks are retained rather than freed, which is precisely why every engine needs an eviction policy for them.

Question 3. You want more concurrent conversations on a 24 GB machine serving an 8B model at a 32k context, and you set an 8-bit key-value cache. What have you traded?

• Nothing: cache quantisation is lossless like prefix caching
• Roughly half the cache memory for a change in the model's outputs, whose size has to be measured on a held-out set before it is accepted
• Prefill speed for decode speed
• The ability to use prefix caching at the same time

Show the answer and why

Answer: Roughly half the cache memory for a change in the model's outputs, whose size has to be measured on a held-out set before it is accepted

Prefix caching reuses bytes unchanged; cache quantisation rounds them, and rounded keys and values produce different attention scores. The memory saving is real and often worth it, but it belongs in the same category as weight quantisation: a change to be measured, not assumed. The part on quantisation and evaluation shows how.

Question 4. True or false: an agent loop benefits from prefix caching more than a single-shot chat request does.

• True
• False

Show the answer and why

Answer: True

True. An agent resends the whole transcript each turn with one new message appended, which is exactly the append-only shape a prefix cache is built for. Without caching, the cost of a run grows with the square of the number of steps, because every step reprocesses every earlier step. With it, each step prefills roughly what was added.

Check my answersClear