The KV Cache as a Transferable Object: Mooncake, LMCache and NIXL

By the end of this lesson you will be able to compute, for any model in this course’s reference set, how many bytes of key-value cache one request produces; divide that by a link’s rated speed to get the time the transfer would take; name the three libraries this field is built on and say what each needs from the network; describe the host-memory and disk tiers that sit underneath them; and use the words connector, transfer engine and store to mean the three different things they mean.

The previous lesson ended with a question: can your link carry a request’s cache in less time than the prefill took? This lesson is the arithmetic that answers it.

What actually has to move

Part 17 defined the object. For every token the model has read, each attention layer computed a key vector and a value vector, and those stay in memory so that later tokens can attend to them. The whole set for one request is its key-value cache, and it is a function of the model’s shape and the number of tokens, not of the text.

The formula has five terms and this course records four of them for every model on the model reference:

Pseudocode — not a real command

bytes per token = layers
                × key-value heads
                × head dimension
                × 2            (one key tensor and one value tensor)
                × bytes per element

bytes per request = bytes per token × prompt tokens

Qwen3-8B, whose recorded shape is thirty-six layers with eight key-value heads of head dimension 128, therefore costs 36 × 8 × 128 × 2 × 2 bytes for every token at sixteen-bit precision. That is 147,456 bytes, or 144 kibibytes, per token. The model reference records exactly that figure, so you can read it off rather than multiplying, and the licence is Apache-2.0.

Three worked examples, each an estimate from that arithmetic rather than a measurement.

Qwen3-8B, one request with a 4,096-token prompt, on a 24 GB machine

Weights (bf16)

16.4 GB

One request's KV cache, 4,096 tokens

0.6 GB

Engine buffers and activations

1.5 GB

Free

5.5 GB

Total

24 GB

Arithmetic, not a measurement: 147,456 bytes per token from the model reference, times 4,096 tokens, is about 0.6 GB. The weights are the sixteen-bit figure from the same reference. This is one request. A decode machine holding eight of these conversations at once is carrying about 4.8 GB of cache, which is the number that decides how many people your decode node can serve.

Qwen3-8B, one request with a 32,768-token prompt, on a 24 GB machine

Weights (bf16)

16.4 GB

One request's KV cache, 32,768 tokens

4.8 GB

Engine buffers and activations

1.5 GB

Free

1.3 GB

Total

24 GB

The same arithmetic with eight times the prompt: about 4.8 GB for a single request. This is the shape of workload a phase split is aimed at, a long document with a short question, and it is also the payload that has to cross the cable. Both numbers are estimates from the recorded shape, not measurements.

Qwen3-32B, one request with an 8,192-token prompt, on a 128 GB machine

Weights (bf16)

65.6 GB

One request's KV cache, 8,192 tokens

2.1 GB

Engine buffers and activations

3 GB

Free

57.3 GB

Total

128 GB

Sixty-four layers rather than thirty-six, so 262,144 bytes per token: about 2.1 GB for an 8,192-token prompt, against about 1.2 GB for Qwen3-8B at the same length. Layer count and key-value head count drive this figure, not parameter count, which is why a mixture-of-experts model can have a small cache and enormous weights. Estimates from the recorded shape.

▤Memory budget

Compare Qwen3-30B-A3B with Qwen3-32B on the model reference and the point becomes obvious. The mixture-of-experts model has forty-eight layers but only four key-value heads, giving 98,304 bytes per token against 262,144 for the dense model of similar total size. Roughly a third of the cache traffic, for a model with twice as many parameters on disk. If you are choosing a model for a disaggregated setup, the key-value head count is the field to look at first.

Dividing by the link

Now put those payloads through Part 18’s link table. The byte rates below are the ones that lesson computed from the rated speeds, and the transfer times are the payload divided by the byte rate.

Pending validationTime to move one request's Qwen3-8B key-value cache, by prompt length and link class

Link class	Byte rate from arithmetic (GB/s)	4,096 tokens, ~0.60 GB (s)	8,192 tokens, ~1.21 GB (s)	32,768 tokens, ~4.83 GB (s)
2.5 gigabit Ethernet	0.31	1.95	3.9	15.58
10 gigabit Ethernet	1.25	0.48	0.97	3.87
25 gigabit Ethernet	3.13	0.19	0.39	1.54
Thunderbolt 4	5	0.12	0.24	0.97
100 gigabit Ethernet	12.5	0.05	0.1	0.39
Thunderbolt 5	15	0.04	0.08	0.32
200 gigabit Ethernet	25	0.02	0.05	0.19

The link classes and rated speeds in Part 18's table · none - arithmetic only; no link and no engine was run computed on 2026-09-09 from the rated speeds in Part 18 and the shape recorded for Qwen3-8B in the course model reference · Qwen3-8B (36 layers, 8 key-value heads, head dimension 128), sixteen-bit key-value cache; weight quantisation does not change these figures · 32,768 tokens of context · 2026-09-09

Every cell is arithmetic, not a measurement. It assumes the link runs at its rated speed for the whole transfer, ignores protocol overhead, ignores the cost of getting the bytes out of one device's memory and into another's, and ignores everything else using the link. Treat each figure as a floor: the real transfer cannot be faster than this and will be slower. The lab measures the real one.

Read the first row and the last row against each other, because that gap is the whole story of this part. On a direct 200 gigabit link a full 32k-token cache moves in under a fifth of a second by arithmetic. On the 2.5 gigabit Ethernet that comes onboard most mini PCs, the same payload takes over fifteen seconds at the rated speed, before any overhead at all.

!Important

The test to apply is a comparison, not a threshold. Is the transfer time smaller than the prefill time it replaced? If prefilling 8,192 tokens on your machine takes about a second, and moving the resulting cache takes about four, the split has made time to first token roughly five times worse and removed some interference in exchange. That is a bad trade and the arithmetic told you before you built anything. Part 5’s prefill measurement and this table are the two numbers you need.

There is a second cost the table cannot show, and it is the one NVIDIA’s Dynamo documentation is warning about when it says TCP over Ethernet is “200-500x slower for this transfer”. A rated speed is the ceiling on the wire. Getting a gigabyte out of GPU memory, through the host’s network stack, across the wire, back through another host’s stack and into a second GPU’s memory involves copies and context switches that the rated speed says nothing about. Part 18’s lesson on RDMA explained what remote direct memory access removes from that path; this is the workload where removing it matters most, because the payload is large and it arrives all at once.

Three libraries and what each wants

The field has converged on three pieces of software, and every engine in the next two lessons uses one or more of them. They are not in this course’s version reference, so they are named here with their repositories and their own documentation rather than with a pinned version.

NIXL, the NVIDIA Inference Xfer Library, at https://github.com/ai-dynamo/nixl. Its own description says it is “targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo, while providing an abstraction over various types of memory (e.g., CPU and GPU) and storage (e.g., file, block and object store) through a modular plug-in architecture”. The plug-ins include UCX for the network path, GPU Direct Storage and POSIX for files, and object stores. The repository states that “NIXL was tested with UCX version 1.23.x” and requires “a C++20 compatible compiler (GCC >= 11 or Clang >= 14)”. It is Apache-2.0, with a note that the Python wheels bundle NVIDIA proprietary modules under a separate licence. What the network has to provide: UCX, which will use RDMA where it exists and fall back to TCP where it does not. It runs on a plain network; its performance assumptions do not.

Mooncake, at https://github.com/kvcache-ai/Mooncake, Apache-2.0, and the subject of the paper “Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving” (Qin and others, submitted June 2024, revised September 2025). It is two things wearing one name. The Transfer Engine is described in the README as “a high-performance data transfer framework” giving “a unified interface for batched data movement across diverse storage, network, and accelerator environments”, and the README lists its transports as “TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, and Ascend-family transports”. The Mooncake Store is “a high-performance distributed key-value cache storage engine designed for LLM inference” that “stores and manages reusable KV caches and model weights across inference clusters”. The paper’s abstract describes the architecture as separating “the prefill and decoding clusters” and leveraging “the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache”. vLLM, SGLang and LMDeploy all integrate it. What the network has to provide: TCP is explicitly in the transport list, which makes it the one of the three that names a plain-Ethernet path in its own README.

LMCache, at https://docs.lmcache.ai/, which describes itself as “A KV cache management layer for LLM inference” that “turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality”. It is the layer above the transfer, not the transfer itself: its documentation describes moving “KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends”, with backends including a filesystem adapter, S3, Redis or Valkey, Mooncake and NIXL. What the network has to provide: whatever its chosen backend needs. Point it at a filesystem and it needs a filesystem; point it at NIXL and you are back to UCX.

⌚Version note

None of these three is in this course’s version table, so no <Validated> marker appears beside them and no version is pinned on this page. Their interfaces move quickly. Read the repository at the tag you install, check it against the engine version you are running, and write both into the notebook. The entries this course would need in order to pin them are listed in the report for this part.

The vocabulary

Three words get used interchangeably in blog posts and mean three different things in the documentation. Getting them straight makes the next two lessons much easier to read.

Where each word belongs

a request's key-value cache travelling from the prefill engine to the decode engine

1. EnginevLLM, SGLang, TensorRT-LLM. Owns the key-value blocks and decides which ones exist.
2. ConnectorThe engine-side plug-in that says when a block should leave or arrive: NixlConnector, MooncakeConnector, LMCacheConnectorV1, OffloadingConnector. Named in vLLM's --kv-transfer-config; SGLang calls the same idea a transfer backend.
3. Transfer engineThe library that actually moves bytes between two memories: NIXL over UCX, Mooncake's Transfer Engine. Its subject is registered memory regions, not attention.
4. StoreSomewhere a cache can sit instead of being sent point to point: the Mooncake Store, an LMCache filesystem or S3 or Redis backend, a shared directory. Adds persistence and sharing; adds latency.
5. TransportRDMA over converged Ethernet, InfiniBand, NVLink, or TCP. The thing Part 18 measured with iperf3.

vLLM's own page uses a third set of words for its internals, describing a Connector that retrieves key-value caches, a LookupBuffer that manages insertion and selection, and a Pipe that carries tensors in one direction. Those are the abstractions inside the engine; the layers above are how the pieces are named when you configure them.

The practical value of the distinction: you choose a connector in the engine’s configuration, and the connector chooses a transfer engine, and the transfer engine uses whatever transport your network provides. When a split is slow, the question is which of those three layers is the problem, and they have different diagnostics.

The tiers below the network

Sending a cache to another machine is one thing you can do with it. Two others are usually cheaper.

Host memory. A decode machine’s GPU memory is finite and its host memory is often much larger. Moving cold blocks from device memory to host memory over the machine’s own bus, and pulling them back on a hit, extends the effective cache without touching the network at all. vLLM’s connector list includes an OffloadingConnector for exactly this, documented as enabling “offloading of KV data to CPU memory, customizing the CPU block size (in tokens) and total CPU memory bytes to allocate”. LMCache calls the same thing a tier in its hierarchy. The next lab measures it.

Local disk. Slower again, and a different kind of useful: it survives a restart. LMCache’s filesystem backend is documented as “a pure file-system L2 adapter using async I/O (aiofiles)” that stores each cache object as a raw file, needs no NIXL, and works on any POSIX filesystem, with a required base_path and optional read_ahead_size and use_odirect settings. A cache on disk is worth having when the prompt is expensive and stable: a manual, a codebase, a system prompt shared by every user.

A shared filesystem. This is the tier that home clusters have and data centres mostly do not bother with, and it is worth taking seriously. Part 18 put the model library on NFS. A directory both engines can read and write is a place to put key-value blocks, and it turns a point-to-point transfer into two file operations. vLLM ships a connector that does precisely this: its ExampleConnector takes a shared_storage_path in its extra configuration, and the repository’s own example runs a prefill process and then a decode process against the same directory. It is slower than RDMA by a wide margin and it needs no special hardware whatsoever.

✦Tip

On a house network the shared-filesystem path is usually the one to try first, not last. It uses infrastructure Part 18 already built, its failure modes are file-permission errors rather than transport errors, and its performance is bounded by something you have already measured. Start there, get the mechanism working end to end, and only then find out whether a point-to-point connector beats it on your link.

Sharing a prefix between engines

There is a second use for all of this machinery that has nothing to do with splitting a request in half, and for most home setups it is the more valuable one.

Part 17’s prefix caching works inside one engine process. Two engine instances behind a load balancer each build their own cache, so a user whose second request lands on the other replica pays for the whole prefill again. A shared store fixes that: both instances write their blocks to the same place and read from it, so the corpus that instance A prefilled is available to instance B. LMCache states this as a design goal, describing key-value cache that can be “reused across multiple serving engines”, and it offers peer-to-peer sharing and multi-server coordination as separate features on top.

The same idea also extends prefix caching past its usual limit. LMCache documents a mechanism to “extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt”, recomputing selected tokens to recover quality. That is a real departure from the rule Part 17 established, that a match must begin at position zero, and it is worth knowing the rule can be bought out of rather than assuming it is a law. It also means the reuse is no longer bit-identical, so unlike ordinary prefix caching it can change outputs, which puts it in the category of things to measure on a held-out set before trusting.

TCP fallbacks, and what they cost

Every library here will run over TCP. None of them is designed for it, and the documentation is candid about what that means.

LMCache’s disaggregated prefill page, read on 2026-09-09, says of running both roles on one host that “On a single host, NIXL over localhost uses loopback/TCP, not RDMA, so latencies are not representative — single-node mode is for functional testing only”. That is the clearest statement in any of this documentation of what a TCP path is good for: proving the wiring, not measuring the result.

NVIDIA’s Dynamo RDMA page, read the same day, is the one with a number attached: “The alternative is TCP over Ethernet, which is 200-500x slower for this transfer: roughly 98s Time To First Token (TTFT) on TCP versus 200-500ms with RDMA.” Reported by NVIDIA, for Dynamo, on the hardware and model NVIDIA had in mind, and quoted here for the size of the gap rather than as a figure to expect on your own machines.

Mooncake is the exception worth naming, because TCP is in its supported transport list rather than being an unhappy fallback. That does not make TCP fast; it makes Mooncake a library that expected to meet one.

✱Hardware note

Only two link classes in this course reach RDMA. Track S has it between two DGX Sparks over the ConnectX-7 QSFP ports, which the DGX Spark user guide documents at up to 200 gigabits per second per port. Track M has it over Thunderbolt 5 from macOS 26.2, as Part 18 recorded. Track X and Track N machines on ordinary Ethernet do not, unless you have put a card in a spare slot. If you are on those tracks, the shared-filesystem connector and the offload tiers in the next lab are where the value is, and the two-machine split is an experiment whose result you should expect to be negative and record anyway.

Calculate transfer size and compatibility independently

For a conventional attention cache, count layers, cached tokens, KV heads, head dimension, key/value multiplicity and bytes per value. Multiply those terms for a raw storage estimate, then account for sharding, block layout, metadata and any compression used by the connector. Divide actual transferred bytes by measured link throughput for a first transfer-time estimate.

Even if the bytes fit, the receiver must interpret them correctly. Model revision, adapter, tokenisation, positions, cache dtype and layout are part of compatibility. Matching the checkpoint’s display name or the cache file’s size is insufficient.

Keep a test with the same prompt processed locally and through the transfer path. Compare the resulting application output and engine diagnostics under controlled settings. Then test an intentionally incompatible configuration and confirm it is rejected or handled according to the documented contract. Treat cache material as derived request data for access and retention decisions. A cache is reusable computation with a specific identity, not a portable semantic summary that any engine can consume.

Recap

One request’s key-value cache is layers times key-value heads times head dimension times two tensors times bytes per element, times the number of prompt tokens, and this course records the first four terms for every model. Qwen3-8B costs 147,456 bytes per token, so a 4,096-token prompt produces about 0.6 GB and a 32,768-token prompt about 4.8 GB. Key-value head count matters far more than parameter count, which is why the mixture-of-experts model has a third of the dense model’s cache traffic.

Divide the payload by Part 18’s byte rates and the answer to “should I split this” appears without building anything. A 200 gigabit direct link moves a full-length cache in a fraction of a second by arithmetic; 2.5 gigabit Ethernet takes over fifteen. The comparison that matters is against the prefill time the transfer replaces.

Three libraries carry this work. NIXL is a point-to-point transfer library over UCX with plug-ins for memory and storage; Mooncake is a transfer engine plus a distributed store, with TCP among its transports; LMCache is a management layer with tiers from GPU memory down to disk and remote backends, and with sharing across engine instances as an explicit goal. A connector is the engine-side plug-in, a transfer engine moves bytes, a store is somewhere bytes can rest, and a transport is the wire.

Below the network there are cheaper tiers: host memory, local disk, and the shared filesystem you already built in Part 18. On a house network those are usually where the return is, and the next two lessons show what each engine offers before the lab measures which one earns its place.

Check your understanding

Question 1. A model has 48 layers, 4 key-value heads and a head dimension of 128, at sixteen bits. How many bytes of key-value cache does one token cost, and what is the reasoning?

• 48 × 4 × 128 = 24,576 bytes, one value per head per layer
• 48 × 4 × 128 × 2 × 2 = 98,304 bytes: every layer stores both a key tensor and a value tensor, and each element is two bytes at sixteen-bit precision
• It depends on the number of parameters, which is not given
• 48 × 128 × 2 = 12,288 bytes; the head count does not enter

Show the answer and why

Answer: 48 × 4 × 128 × 2 × 2 = 98,304 bytes: every layer stores both a key tensor and a value tensor, and each element is two bytes at sixteen-bit precision

Two factors of two are the ones people drop: one for the pair of tensors, one for the bytes per element. That model is Qwen3-30B-A3B, and the course model reference records 98,304 bytes per token for it, so you can check the arithmetic against the table rather than trusting either on its own.

Question 2. Prefilling an 8,192-token prompt on your prefill machine takes about one second. Your two machines are joined by 2.5 gigabit Ethernet and the model is Qwen3-8B. What does the arithmetic say about splitting the phases?

• Split it: the transfer is much faster than the prefill
• Do not split it: about 1.2 GB has to cross a link whose rated byte rate is around 0.31 GB per second, so the transfer alone is roughly four seconds at the ceiling and slower in practice, against the one second of prefill it replaces
• Split it, because the transfer happens in parallel with prefill
• The arithmetic cannot say anything without measuring first

Show the answer and why

Answer: Do not split it: about 1.2 GB has to cross a link whose rated byte rate is around 0.31 GB per second, so the transfer alone is roughly four seconds at the ceiling and slower in practice, against the one second of prefill it replaces

The transfer happens after prefill finishes, because it is the result of the prefill. Adding four seconds to time to first token in order to remove some scheduling interference is a bad trade, and one multiplication found that out before any software was installed. The same machines might still gain a great deal from the offload and sharing tiers.

Question 3. Match the words. Which statements use the vocabulary correctly? Select all that apply.

• A connector is the engine-side plug-in that decides when a block should leave or arrive, and is what you name in the engine's configuration
• A transfer engine moves bytes between two memories and knows nothing about attention
• A store is a place a cache can sit rather than being sent point to point, which adds persistence and sharing at the cost of latency
• A transport is another name for a connector

Show the answer and why

Answer: A connector is the engine-side plug-in that decides when a block should leave or arrive, and is what you name in the engine's configuration, A transfer engine moves bytes between two memories and knows nothing about attention, A store is a place a cache can sit rather than being sent point to point, which adds persistence and sharing at the cost of latency

The transport is the wire and the protocol on it: RDMA over converged Ethernet, InfiniBand, NVLink or TCP, which is what Part 18 measured with iperf3. Keeping the four apart is what makes a slow split diagnosable, because each layer fails differently and each has its own logs.

Question 4. You have two machines on ordinary 2.5 gigabit Ethernet and a shared NFS mount from Part 18. Which approach does this lesson recommend trying first?

• Buy a 200 gigabit card for each machine
• The shared-filesystem path, because it reuses infrastructure you already measured, its failures are file-permission errors rather than transport errors, and its ceiling is a number you already know
• Run the transfer over Wi-Fi to avoid loading the switch
• Skip disaggregation entirely; nothing in this part applies

Show the answer and why

Answer: The shared-filesystem path, because it reuses infrastructure you already measured, its failures are file-permission errors rather than transport errors, and its ceiling is a number you already know

vLLM ships a connector that takes a shared storage path, so the mechanism can be proved end to end with no special hardware. Getting the wiring right on the cheap path first, and then finding out whether a point-to-point connector beats it, is a much shorter route than debugging UCX and the engine at the same time.

Question 5. True or false: LMCache documents a way to reuse cached blocks at positions other than the start of the prompt, which means that reuse is no longer bit-identical.

• True
• False

Show the answer and why

Answer: True

True. Its documentation describes extending reuse beyond prefix caching by reusing blocks at any position, with selective recomputation to recover quality. Part 17's rule that a match must start at position zero is a property of ordinary prefix caching, not a law of nature, but buying out of it means outputs can change, so it belongs in the category of things measured on a held-out set rather than assumed.

Check my answersClear