vLLM Disaggregated Prefill: Connectors and the Proxy

By the end of this lesson you will be able to start two vLLM instances with different key-value roles, choose a connector your network can actually carry, put the project’s own example proxy in front of the pair, set the environment variables that decide which interface the transfer uses, read the logs and metrics that show whether a cache actually moved, and state the feature’s support status with the date you checked it.

This is the implementation the first lab builds. Read it with Part 18’s measured link numbers and the previous lesson’s transfer arithmetic beside you, because they decide which of the connectors below is worth your evening.

The status line, first

Take this in before anything else, because it changes how you should treat everything after it.

⌚Version note

vLLM’s disaggregated prefilling page, read on 2026-09-09 against vLLM 0.28.0 · verified 2026-09-08, carries the heading “Disaggregated Prefilling (experimental)” and the warning “This feature is experimental and subject to change.” The same page states, in capitals, “Disaggregated prefill DOES NOT improve throughput.” Neither sentence is a reason to skip the lesson. Both are reasons to pin your version, to write the exact commit or release beside every number you record, and to expect the configuration syntax below to have moved by the time you read this.

The page gives two benefits and they are both about latency rather than volume. The first is tuning time to first token and inter-token latency separately, by giving each phase its own instance with its own parallel strategy. The second is controlling tail inter-token latency, on the grounds that a separated prefill cannot interrupt a decode in progress, which the page presents as more reliable than chunked prefill for that specific purpose.

Two instances and three roles

A disaggregated vLLM deployment is two ordinary vllm serve processes. Both load the whole model. What distinguishes them is one extra argument, --kv-transfer-config, whose value is a JSON object. Two of its fields decide the topology.

kv_connector names the plug-in that moves the blocks. kv_role says what this instance does with them: kv_producer for the prefill instance, kv_consumer for the decode instance, or kv_both for an instance that does both, which is what you use when a single instance is talking to a store rather than to a partner.

The LMCache documentation’s two-node example, read on 2026-09-09, shows the shape clearly: the prefill server is started with "kv_role":"kv_producer" and the decode server with "kv_role":"kv_consumer", both naming the same connector.

One request through a disaggregated pair

1. 1The client sends a request to the proxyAn ordinary OpenAI-compatible chat or completion request. The client knows nothing about the split.
2. 2The proxy asks the prefill instance for one tokenThe producer reads the whole prompt, writes the key-value blocks out through its connector, and returns almost immediately. Its answer is discarded.
3. 3The connector moves the blocksPoint to point over NIXL or Mooncake, or through a store, or into a shared directory. This is the transfer whose size the previous lesson computed.
4. 4The proxy sends the same request to the decode instanceThe consumer finds the blocks already present, skips the prefill it would otherwise have done, and starts generating.
5. 5The proxy streams the decode instance's answer backThe client sees one response from one endpoint, as it would from a single server.

The prompt is sent twice and prefilled once. That is why the proxy is not optional: something has to make both calls, in order, and hand the second one’s stream to the client.

The page also documents the client-side form of the same handshake, for programs that would rather drive the two instances themselves than run a proxy. The prefill call passes {"return_token_ids": True, "kv_transfer_params": {"do_remote_decode": True}} in the request body and the decode call passes {"kv_transfer_params": {"do_remote_prefill": True, "prompt_token_ids": ids}} with the token ids the first call returned. Worth knowing that this exists; the proxy is the easier path and the one the lab uses.

Choosing a connector for the network you have

The page lists nine connectors. Four of them matter for a home lab, and the choice is decided by the link, not by preference.

Connector	What it moves bytes with	What it needs	Where it fits at home
ExampleConnector	A shared directory, named by shared_storage_path in the extra configuration	A filesystem both instances can read and write	Any network at all, including a slow one. The Part 18 NFS mount is exactly this.
NixlConnector	NIXL over UCX	UCX; RDMA to be worth doing, TCP to merely work	Track S over the ConnectX-7 cable; Track M over Thunderbolt 5 from macOS 26.2
MooncakeConnector	Mooncake’s Transfer Engine	The mooncake-transfer-engine package; TCP is a supported transport	The plain-Ethernet point-to-point option, if you want one
OffloadingConnector	Host memory on the same machine	Nothing on the network	One machine, extending the cache into host memory. The second lab.

The one to reach for first on a house network is the shared-directory connector, and the reason is the same reason Part 18 built the NFS export before it built any cluster: it is the path whose failures you can already diagnose. vLLM’s own example for it, examples/disaggregated/example_connector/ in the repository, builds a KVTransferConfig with kv_connector="ExampleConnector", kv_role="kv_both" and kv_connector_extra_config={"shared_storage_path": "local_storage"}, then runs a prefill script and a decode script in sequence against that directory. Change the directory to a mount both machines see and the same mechanism spans two machines.

Fragment — not complete on its own

# The connector argument on the prefill instance. The path must be visible to BOTH
# instances; on the reference cluster that means the shared mount from Part 18.
--kv-transfer-config '{"kv_connector":"ExampleConnector","kv_role":"kv_producer","kv_connector_extra_config":{"shared_storage_path":"/mnt/models/kv-store"}}'

The point-to-point connectors are what the documentation is really built around, and on a link with RDMA they are the right answer. LMCache’s page shows a NixlConnector wrapped in a MultiConnector alongside an LMCache connector, so that the same deployment does the request handoff over NIXL and keeps a longer-lived tier behind it. That is a production shape; the lab uses the simpler form.

✕Common mistake

The failure to expect on a shared directory is not a transfer error. It is that the decode instance silently prefills the prompt itself, because it looked in the directory, found nothing it recognised, and carried on. Nothing crashes and the answers are correct. The only symptom is that time to first token did not improve and the directory is filling up with blocks nobody reads. Check the directory’s contents and the decode instance’s prefill counters before believing a null result.

The proxy

Something has to call the prefill instance and then the decode instance for every request. vLLM ships a demonstration of that, and the lab uses it rather than inventing one.

The file is examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py in the vLLM repository, and the directory’s README describes it as demonstrating “XpYd (X prefill instances, Y decode instances)”. Its module docstring calls it “a disaggregated prefilling proxy demo to demonstrate an example usage of XpYd disaggregated prefilling”. It takes four arguments: --model, --prefill with one or more host:port values, --decode with one or more, and --port for the proxy’s own listener, which defaults to 8000. The README’s usage example is:

Fragment — not complete on its own

# From the vLLM repository README for examples/disaggregated/disaggregated_serving,
# read 2026-09-09. Take the current file from the repository at the tag you installed.
python3 examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py  \
     --model $model_name  \
     --prefill localhost:8100 localhost:8101   \
     --decode localhost:8200 localhost:8201   \
     --port 8000

Two practical notes about obtaining it. It is not installed by pip install vllm; it lives in the source tree, so you need the repository checked out at the same tag as the wheel you installed, or at least the single file downloaded from that tag. And it is a demonstration, which the file’s own docstring says: it is the right thing to measure with and the wrong thing to leave running in front of anything that matters. Part 23 is where a front door gets built properly.

⛨Security

The proxy, the prefill instance and the decode instance all speak plain HTTP with no authentication. Bind every one of them to an address on your cluster network, never to a public interface, and put the authenticated gateway from Part 9 in front of the proxy if anything outside the machine is going to reach it. The same rule Part 18 stated for cluster traffic applies here with more force, because the payload crossing the link is the user’s prompt in the model’s own representation.

Configuration and environment

Beyond the connector JSON, four environment variables decide whether the transfer works at all, and three of them are about naming the right interface. The values below come from LMCache’s disaggregated prefill documentation, read on 2026-09-09.

VLLM_NIXL_SIDE_CHANNEL_HOST is the address this instance advertises for the handshake between producer and consumer. It must be an address the other machine can reach, which on a cluster with a direct cable means the address on the cable and not the one on the house switch. This is the same mistake Part 20 warns about for Ray, in a different costume.

VLLM_NIXL_SIDE_CHANNEL_PORT is the port for that handshake, and the documentation notes that two instances sharing a host need distinct ones. On the single-machine path, that is the line people miss.

UCX_NET_DEVICES restricts UCX to the interfaces you name. Part 20’s TensorRT-LLM lesson sets the same variable for the same reason: without it, a library will cheerfully choose the management interface and give you a slow cluster while reporting nothing at all.

NCCL_CUMEM_ENABLE appears in the documented examples alongside the others; set it as the documentation for your version shows rather than reasoning about it.

None of these applies to the shared-directory connector, which has no handshake and no transport. That is a third reason to start there.

Fragment — not complete on its own

# Shape only. The interface name and the addresses come from your own .env,
# never from a page: a name that is right in one house is wrong in every other.
export VLLM_NIXL_SIDE_CHANNEL_HOST="${PREFILL_LINK_ADDR}"
export VLLM_NIXL_SIDE_CHANNEL_PORT=5600
export UCX_NET_DEVICES="${CLUSTER_IFACE}"

✱Hardware note

--kv-transfer-config is documented on vLLM’s disaggregated prefilling page and used throughout the LMCache integration documentation, but on 2026-09-09 it did not appear on the vllm serve CLI reference page and it is not yet in this course’s captured command reference. Run vllm serve --help | grep kv on your own installation before scripting it, and record what your version accepts. The same check is worth doing for --kv-offloading-backend and --kv-offloading-size, which the CLI page does document and which the next lab uses.

Reading the logs and the metrics

A disaggregated pair has three processes and four places a request can go wrong. These are the things to look at, in order.

The startup log of each instance. Both should report the model loaded and the key-value cache size the engine settled on, exactly as in Part 9. If the two instances disagree about the model, the quantisation or the block size, the blocks one produces are not the blocks the other expects, and the symptom is a silent miss rather than an error.

The proxy’s own output. It shows which instance each request went to and in what order. A proxy that is sending everything to the decode instance and nothing to the prefill instance is a working single-machine deployment wearing a disaggregated hat, and it will look fine.

The metrics endpoint on each instance. Both serve /metrics in the Prometheus text format. vLLM’s production metrics page documents the four that matter here, and the interesting readings are comparisons between the two instances rather than absolute values:

Metric	Documented as	What it tells you here
vllm:time_to_first_token_seconds	“Histogram of time to first token in seconds.”	On the decode instance, this is the number the split is trying to improve
vllm:request_time_per_output_token_seconds	“Histogram of time_per_output_token_seconds per request.”	Should be essentially unchanged by the split; if it got worse, something is wrong
vllm:prefix_cache_hits	“Prefix cache hits, in terms of number of cached tokens.”	On the decode instance, a hit rate near zero when it should be high means the transfer is not landing
vllm:kv_cache_usage_perc	“KV-cache usage. 1 means 100 percent usage.”	On the decode instance, this is the occupancy that decides how many conversations fit

The interface counters. The engine cannot tell you how many bytes crossed the cable; the operating system can. Part 19’s approach of reading the interface byte counters either side of a request is exactly the right instrument here, and the first lab reuses it. A split that is working shows a large read on the decode machine’s cluster interface at the moment each request starts, of roughly the size the previous lesson’s arithmetic predicted. A split that is not working shows nothing, and the arithmetic tells you what “nothing” would have looked like.

=Expected result

The single clearest proof that the mechanism works: the decode instance’s time to first token is much lower than the same instance’s time to first token would be for the same prompt with the prefill instance switched off. Measure both. One number on its own proves nothing, and the pair is one run of the load generator each.

Per-track status

Track S — NVIDIA DGX Spark

Primary path. vLLM runs on DGX Spark, as Part 9 established, and a pair of Sparks joined by a ConnectX-7 QSFP cable is the only configuration in this course where the transfer arithmetic from the previous lesson comes out comfortably in favour of a split. Use NixlConnector over the cable, with UCX_NET_DEVICES naming the QSFP interface and VLLM_NIXL_SIDE_CHANNEL_HOST set to each node’s address on that cable. Start with the shared directory anyway: proving the mechanism on the cheap connector first turns a two-variable problem into two one-variable problems.

Track X — AMD Ryzen AI Max+ 395Partial

vLLM's GPU installation page lists Ryzen AI MAX and AI 300 (gfx1151/1150) among its ROCm targets with pre-built wheels, but this course has not exercised that path and no ROCm-specific disaggregation guidance exists in the vLLM disaggregated prefilling documentation.

vLLM is listed for this hardware and this course has not run it there, which is the same status Part 9 recorded. If your installation works, the shared-directory connector is the one to try: it does not touch UCX or RDMA, and 2.5 gigabit Ethernet is what these machines usually have, so the arithmetic already says the point-to-point path is not going to win. The page’s list of connectors does include a ROCm-only MoRIIOConnector; this course has not tested it and does not describe it further.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS, so neither instance in a disaggregated pair can run on Apple silicon with GPU acceleration.

Take the single-machine path with llama-server instead. It cannot disaggregate, but it can do the thing disaggregation is trying to protect: run several conversations at once without a long prompt stalling the others, with each conversation’s cache kept warm in its own slot. Start it with --parallel set to the number of concurrent conversations you want and prompt caching left at its default, and use the slot save and restore endpoints from Part 17 to keep an expensive prompt across restarts. Part 21 is where a two-Mac cluster is built; this particular feature is not part of it.

RunnableTrack M · Apple silicon

the reduced path: one server, several warm slots

llama-server \
  --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
  --alias local-chat \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 32768 \
  --parallel 4 \
  --n-gpu-layers 999 \
  --cache-prompt \
  --slot-save-path ./slot-cache \
  --metrics \
  --slots

Track N — NVIDIA desktop or laptop

Primary path. Two desktops or a desktop and a laptop, both running vLLM as installed in Part 9. Everything in this lesson applies unchanged. The thing to check before you start is the link: unless you have put a card in a spare slot, this is the track most likely to be on ordinary Ethernet, and the previous lesson’s table is the prediction your measurement should match. One machine with two GPUs is also a legitimate configuration here, with one instance pinned to each card through CUDA_VISIBLE_DEVICES and the transfer happening over the host rather than the network.

Trace the request through producer, transfer and consumer

Assign a request identifier and follow it through the entry proxy, prefill producer, connector and decode consumer. Record which process owns each phase and where errors are returned. A healthy producer and consumer tested separately do not prove that the handoff works.

Use the connector example corresponding to your installed version and transport. Configuration keys and topology assumptions are part of the implementation contract; do not mix a current example with a different pinned engine without reviewing the differences. Start with a small checkpoint and one request before attempting load.

Measure transfer latency, first-token latency and output cadence alongside failures. Test a producer restart, a consumer restart and a cancelled request in the isolated lab. Observe whether buffers are reclaimed and whether a later request succeeds. If a connector silently falls back or recomputes, document it rather than describing the result as successful cache transfer. The service-level acceptance test must establish both correct outputs and the intended division of work.

Recap

A disaggregated vLLM deployment is two ordinary servers with one extra argument. --kv-transfer-config carries a JSON object whose kv_connector names the plug-in and whose kv_role is kv_producer on the prefill instance and kv_consumer on the decode instance. The prompt goes to both; only the first prefills it.

The connector choice is a network decision. ExampleConnector with a shared_storage_path needs only a directory both machines can see, which the Part 18 NFS mount already provides; NixlConnector goes point to point over UCX and wants RDMA to be worth doing; MooncakeConnector names TCP among its transports; OffloadingConnector moves blocks into host memory on one machine and touches no network at all.

The proxy is disagg_proxy_demo.py in the vLLM repository’s disaggregated serving examples, taking --model, --prefill, --decode and --port. It is a demonstration, it is not installed by the wheel, and it has no authentication.

Four environment variables decide whether a point-to-point transfer uses the right cable, and VLLM_NIXL_SIDE_CHANNEL_HOST set to a management address instead of a cluster address is the classic way to build a split that works and is slow. The shared-directory connector has none of them.

Proof of life is the decode instance’s time to first token against the same instance without a producer, the prefix cache counters on the decode instance, and the interface byte counters showing a payload the size the arithmetic predicted. And the feature is marked experimental on its own page as of 2026-09-09, so pin the version and write it down.

Next: SGLang’s version of the same split, which uses different words and a bootstrap server, and NVIDIA Dynamo, which puts a scheduler above all of it.

Check your understanding

Question 1. What distinguishes the prefill instance from the decode instance in a disaggregated vLLM deployment?

• The prefill instance loads only the first half of the model
• One argument: --kv-transfer-config, whose kv_role is kv_producer on the prefill instance and kv_consumer on the decode instance. Both load the whole model.
• The prefill instance runs with a larger --max-model-len
• They are different programs

Show the answer and why

Answer: One argument: --kv-transfer-config, whose kv_role is kv_producer on the prefill instance and kv_consumer on the decode instance. Both load the whole model.

Both are ordinary vllm serve processes with the whole model. Nothing about the weights is split, which is why disaggregation is not a capacity technique. The role field is what tells each instance whether it writes key-value blocks out or expects to find them already there.

Question 2. Your two machines are joined by 2.5 gigabit Ethernet and share an NFS mount from Part 18. Which connector should you configure first, and why?

• NixlConnector, because it is the one the documentation is built around
• ExampleConnector with a shared_storage_path on the NFS mount, because it needs no UCX, no RDMA and no side-channel environment variables, so its failures are file-permission errors you can already diagnose
• MooncakeConnector, because TCP is in its transport list
• OffloadingConnector, because it is fastest

Show the answer and why

Answer: ExampleConnector with a shared_storage_path on the NFS mount, because it needs no UCX, no RDMA and no side-channel environment variables, so its failures are file-permission errors you can already diagnose

On a slow link the arithmetic already says the point-to-point path will not win, so the value of trying it first is low and the debugging cost is high. Proving the mechanism on the connector with the fewest moving parts turns one hard problem into two easy ones. OffloadingConnector is a different feature entirely: it moves blocks into host memory on one machine.

Question 3. You enable a shared-directory connector, run a load test, and time to first token is unchanged. The logs report nothing unusual and the answers are correct. What is the first thing to check?

• Whether the model is too small for the benefit to show
• Whether the decode instance is finding anything in the directory at all: a miss is silent, because the instance simply prefills the prompt itself and returns a correct answer
• Whether the proxy needs a larger timeout
• Whether the network cable is faulty

Show the answer and why

Answer: Whether the decode instance is finding anything in the directory at all: a miss is silent, because the instance simply prefills the prompt itself and returns a correct answer

This is the characteristic failure of the whole feature. Nothing crashes on a cache miss, because prefilling the prompt is always a valid thing to do. Look at the directory's contents, at the decode instance's prefix cache hit counter, and at the interface byte counters, and compare what you see against the payload size the arithmetic predicts.

Question 4. Which statements about the example proxy are accurate? Select all that apply.

• It lives in the vLLM repository under examples/disaggregated/disaggregated_serving and is not installed by the wheel
• It takes --model, --prefill with one or more host:port values, --decode with one or more, and --port
• It is described by its own docstring and README as a demonstration of the XpYd pattern
• It provides authentication and TLS for the pair

Show the answer and why

Answer: It lives in the vLLM repository under examples/disaggregated/disaggregated_serving and is not installed by the wheel, It takes --model, --prefill with one or more host:port values, --decode with one or more, and --port, It is described by its own docstring and README as a demonstration of the XpYd pattern

It has no authentication of any kind, and neither do the two instances behind it. Bind all three to a cluster address, and put the authenticated gateway from Part 9 in front if anything beyond the machine will reach it. Part 23 is where a front door is built to be exposed.

Question 5. On 2026-09-09, what did vLLM's own documentation say about disaggregated prefilling and throughput?

• That it roughly doubles throughput on two machines
• That it does not improve throughput, and that its benefits are separate tuning of time to first token and inter-token latency, and control of tail inter-token latency
• That throughput improves only with RDMA
• The page does not discuss throughput

Show the answer and why

Answer: That it does not improve throughput, and that its benefits are separate tuning of time to first token and inter-token latency, and control of tail inter-token latency

The sentence is on the page in capital letters, next to a warning that the feature is experimental and subject to change. Judging the feature by total tokens per second measures the thing it does not claim to do; the latency figures under a load with a deadline are what it is for.

Check my answersClear