SGLang PD Disaggregation and NVIDIA Dynamo

By the end of this lesson you will be able to start an SGLang prefill server and an SGLang decode server and put its router in front of them, name the transfer backends it accepts and the bootstrap mechanism that pairs the two roles, list the timeouts and buffers that field deployments end up setting, describe what NVIDIA Dynamo is and what its router does that a load balancer does not, and choose between the three implementations for a specific set of machines rather than in the abstract.

Part 9 taught SGLang on one machine and gave the reasons to choose it. Part 20’s lesson TensorRT-LLM and Dynamo on Spark Pairs introduced Dynamo as a survey and said that this part is where disaggregated serving is actually built. This lesson keeps both of those promises.

SGLang’s two roles

SGLang calls the feature PD disaggregation, for prefill and decode, and its documentation gives the same two motivations the previous lessons did. The first is prefill interruption: incoming prefill batches interrupt decode batches and cause substantial delays. The second is specific to data-parallel attention, where workers processing a mixed workload see increased decode latency.

The configuration is a small set of flags on the ordinary launch command, and unlike vLLM there is no JSON object: the role is a value, not a document.

RunnableTrack N · NVIDIA GPU

the prefill server, from SGLang's PD disaggregation page

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --port 30000 \
  --disaggregation-ib-device mlx5_roce0

RunnableTrack N · NVIDIA GPU

the decode server, on the same host in the documentation's example

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --port 30001 \
  --base-gpu-id 1 \
  --disaggregation-ib-device mlx5_roce0

Llama 3.1 8B Instruct is published under the Llama 3.1 Community licence, which the model reference records; check the model card before using it, as that licence is not one of the permissive ones.

Four flags carry the whole feature.

--disaggregation-mode takes prefill or decode. There is no third value for “both”, which is the interesting difference from vLLM: an SGLang instance is one or the other.

--disaggregation-transfer-backend takes mooncake, which the documentation gives as the default, nixl, or ascend. The page says plainly, “Currently, we support Mooncake and NIXL as the transfer engine.” Mooncake needs the mooncake-transfer-engine package; NIXL is installed with pip install nixl or built from source against a local UCX, and its backend can be steered with the SGLANG_DISAGGREGATION_NIXL_BACKEND environment variable, for which the page shows LIBFABRIC as an alternative to the UCX default.

--disaggregation-ib-device names the InfiniBand or RoCE device to transfer over. This is the same decision UCX_NET_DEVICES makes in the vLLM lesson, wearing SGLang’s name for it.

--disaggregation-bootstrap-port sets the port for the bootstrap exchange by which a decode instance and a prefill instance find each other. In the documentation’s multi-node examples the same value is set on both roles.

✱Hardware note

Notice what the documented examples have in common: mlx5_roce0, mlx5_1, mlx5_2. Every first-class example on the page names an RDMA device, and the transfer backends are a transfer engine that supports TCP and a UCX library that falls back to it. SGLang’s PD disaggregation is written for machines with RDMA between them, and on the reference cluster that means the DGX Spark pair and nothing else. On plain Ethernet it will run; the previous lesson’s arithmetic is what tells you whether running it is worth doing.

The router in front

SGLang ships its own front door for this arrangement, which vLLM does not: sglang_router, launched as a module and told where each role lives.

RunnableTrack N · NVIDIA GPU

the router, pairing one prefill server with one decode server

python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001 \
  --host 127.0.0.1 \
  --port 8000

That is a real difference from vLLM’s arrangement, where the proxy is an example file in the source tree. SGLang’s router is a component with its own module, health checking through --health-check-interval-secs, and a --mini-lb mode that the documentation’s larger examples use. It accepts several prefill and several decode endpoints, which is how a pool of each is built.

An SGLang PD deployment, minimum viable shape

• clientClients
• routersglang_router--pd-disaggregation, one or more --prefill and --decode endpoints, health checks
• prefillPrefill server--disaggregation-mode prefill; reads prompts, writes key-value state out
• decodeDecode server--disaggregation-mode decode; receives key-value state, generates
• storageModel librarythe same weights on both, from Part 18

• Clients ⟷ connected to sglang_routerOpenAI-compatible requests
• sglang_router ⟷ connected to Prefill serverthe prompt
• sglang_router ⟷ connected to Decode serverthe continuation
• Prefill server ⟷ connected to Decode serverkey-value state over the transfer backend, on the device named by --disaggregation-ib-device; paired through the bootstrap port
• Model library ⟷ connected to Prefill serverread at load time
• Model library ⟷ connected to Decode serverread at load time

The same topology as the vLLM pair, with SGLang's names. The bootstrap exchange is the piece vLLM handles through its side-channel environment variables; here it is a port on both servers.

What field deployments end up setting

The most useful part of SGLang’s page is not the happy-path commands. It is the collection of environment variables that exist because somebody’s deployment needed them, and reading that list tells you what goes wrong.

SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT and SGLANG_DISAGGREGATION_WAITING_TIMEOUT appear in the documented examples raised to 600 and, in one case, to 3600 seconds. A bootstrap timeout is what a pair does when one side comes up much later than the other, which on a home cluster is the normal case: a large model loads at whatever speed each machine’s disk allows, and the machine that finishes first waits. Raising these is not a workaround, it is configuration for a real property of your cluster.

SGLANG_DISAGG_STAGING_BUFFER and SGLANG_DISAGG_STAGING_POOL_SIZE_MB appear together, the second set to 4096 in the documented example. A staging buffer is memory the transfer passes through, and its existence is a reminder that the payload does not go straight from one GPU to the other by magic.

--disable-radix-cache appears in several of the page’s larger examples on both roles. That switch turns off the RadixAttention prefix cache Part 17 described. Seeing it in a disaggregation example is worth pausing on: the two features interact, and a configuration that is right for one workload is not automatically right for the other.

✕Common mistake

The failure that eats an evening is a bootstrap that never completes: the decode server sits waiting, the prefill server sits waiting, nothing is printed on either side, and the router reports both as healthy because both are answering health checks. Check the bootstrap port is identical on both and reachable between the machines before you check anything else, and raise the two timeouts before you start a large model on a slow disk.

NVIDIA Dynamo

Dynamo is not an engine. Part 20’s lesson established that and this lesson does not repeat the survey; what follows is the part that bears on disaggregation specifically.

Its architecture documentation names disaggregation as one of its design goals directly: “GPU efficiency: disaggregate prefill and decode so each can scale independently”, motivated by the observation that “Prefill/decode imbalance leaves GPUs underutilized when traffic mix shifts”. Prefill workers compute prompt key-value state, decode workers generate output tokens, and the decode side “receives KV state (typically via NIXL transfer path)”.

The piece worth studying, and the one with no counterpart in either vLLM or SGLang as this course uses them, is the router. It “selects workers based on load and KV overlap”, where KV overlap means the blocks a worker already holds that match the incoming request’s tokens, found by prefix-tree matching. Its documented cost function is worth reading even if you never run Dynamo, because it is an explicit statement of what a scheduler should weigh:

Pseudocode — not a real command

effective_device_credit = overlap_score_credit * overlap_score_credit_decay_factor
adjusted_prefill_blocks = max(0, (
    prefill_blocks
    - effective_device_credit * device_overlap_blocks
    - host_cache_hit_weight * host_overlap_blocks
    - disk_cache_hit_weight * disk_overlap_blocks
    - shared_cache_multiplier * shared_beyond_blocks
))
active_request_blocks = decode_active_request_weight * active_requests
cost = (
    prefill_load_scale * adjusted_prefill_blocks
    + potential_decode_blocks
    + active_request_blocks
)

Read what that says. The cost of sending a request to a worker is the prefill work it would have to do, discounted separately by how much of the prompt that worker already has in device memory, in host memory and on disk, with a different weight for each tier, plus the decode load it is already carrying. The lowest cost wins. Every idea in the previous two lessons is a term in that expression: the tiers from Lesson 2, the prefix overlap from Part 17, and the load balancing from Part 9’s gateway, combined into one number.

That is the intellectual content of this section, and it transfers whether or not you run the software. When you write your own routing rule in this part’s project, this is the shape it should have: not round-robin, but “who already has most of this prompt, and who is least busy”.

∴Research note

Dynamo also brings a block manager it calls KVBM, which its overview describes as managing “block reuse, eviction, and offload/recall across memory tiers” and as extending “effective cache capacity using multi-tier memory offload/recall”. That is Lesson 2’s tier list with a scheduler attached. The course records NVIDIA Dynamo 1.4.2 · verified 2026-09-08 as surveyed rather than taught, on the same grounds Part 20 gave: multi-node work requires RDMA, and GB10 is not named as a validated platform on Dynamo’s compatibility page.

The requirement is the wall, and it is the same one Part 20 quoted. Dynamo’s RDMA setup page says “Dynamo needs RDMA for disaggregated serving, where prefill workers generate KV cache and hand it to decode workers”, that “Dynamo reaches RDMA through NIXL, which transfers KV cache over either UCX or libfabric”, and that the alternative “is TCP over Ethernet, which is 200-500x slower for this transfer”. Aggregated deployments, it notes, transfer no key-value cache between workers and do not need RDMA at all.

Kubernetes, in one paragraph

All three implementations have a Kubernetes story and this course does not teach any of them. Dynamo’s documentation is substantially a Kubernetes documentation set, with an operator, custom resources and cloud-provider guides, and Part 20 said so. On the vLLM side, LMCache’s own Kubernetes deployment page names the vLLM Production Stack, describing it as “a specialized production-ready implementation for K8S-native cluster-wide deployment for vllm & lmcache”, and points at its quickstart and at a tutorial on offloading the key-value cache. If you already run a cluster, those are the doors to go through. If you do not, adding one in order to serve a model at home is a considerably larger project than the one you started, and the two labs in this part are deliberately written with processes and scripts so that the serving mechanism is visible rather than declarative. Part 23 operates what you build here with the tools this course does teach.

Choosing between the three

The honest form of this decision is a set of conditions, not a ranking.

If your situation is	Use	Because
Two machines, ordinary Ethernet, you want to learn the mechanism	vLLM with the shared-directory connector	No RDMA, no UCX, no bootstrap; the failures are file errors, and Part 18’s mount already exists
Two DGX Sparks with the QSFP cable, and vLLM already in use from Part 9	vLLM with NixlConnector	The transfer arithmetic works out, and it is one extra argument on the servers you already run
An RDMA link, and you already prefer SGLang from Part 9	SGLang PD with mooncake or nixl	The role flags are simpler than the JSON, and the router is a supported component rather than an example file
You want a cache tier shared across instances more than you want a phase split	vLLM or SGLang with LMCache	Both integrate it, and on a home network the tiers usually pay better than the split
You run Kubernetes already and need this in production	Dynamo, or the vLLM Production Stack	Both are built for it; neither is taught here
One machine	vLLM with OffloadingConnector, or two processes on one device	The second lab, and the single-machine path of the first

✦Tip

Do not choose an engine for this feature. Choose the engine for everything else it has to do, as Part 9 argued, and then use that engine’s disaggregation if the arithmetic says a split is worth building. The mechanism is nearly identical across the three, and the vocabulary in Lesson 2 is enough to read any of their documentation. Switching engines to get a connector name is a poor trade.

Track S — NVIDIA DGX Spark

Both SGLang and vLLM run here and the Spark pair has the RDMA link both of them want. SGLang’s documented device names are examples from other hardware, so find yours with the same tools Part 20’s pair preparation uses and put the result in your settings file rather than copying a name from a page.

Track X — AMD Ryzen AI Max+ 395Not supported

SGLang's PD disaggregation examples and transfer backends are documented against NVIDIA and Ascend hardware with RDMA devices; NVIDIA Dynamo's compatibility page lists NVIDIA GPU architectures only.

Neither implementation on this page has a documented path on this hardware. The transferable idea is the router’s cost function: a front door that prefers the backend already holding most of the prompt is worth building even with one engine and no disaggregation, and this part’s project asks you to design one.

Track M — Apple siliconNot supported

SGLang and NVIDIA Dynamo both target NVIDIA and other data-centre accelerators; neither has a macOS path.

Part 21 covers what an Apple cluster can do. For this feature specifically, the previous lesson’s llama-server path is the reduced version: several warm slots on one server, with prompt caching and slot save and restore doing the work a cache tier would do.

Track N — NVIDIA desktop or laptop

SGLang PD runs here, and the documentation’s own examples use a single host with --base-gpu-id separating the two roles onto two cards, which is exactly the two-GPU desktop case. On two machines over house Ethernet, expect the arithmetic from Lesson 2 to dominate and measure it anyway.

Compare architectures at the application boundary

Different frameworks expose different routers, worker roles and transfer mechanisms. Compare them using the same externally visible contract: model identity, permitted request sizes, concurrency, authentication, latency target and failure behaviour. Keep internal metrics for diagnosis, but do not substitute one framework’s differently defined latency metric for another’s without checking the definition.

Draw the complete topology for each candidate, including discovery or metadata services. Record versioned configuration and image identities. Verify a single request through the full route before running the workload mix, and retain a colocated control.

If one deployment uses more machines or keeps additional model copies resident, include those resources in the comparison. A throughput increase purchased with extra replicas is useful but different from a gain due to phase separation. Choose the design that meets the measured service requirement with an operating procedure you can maintain. A feature name shared by two frameworks does not establish equal connector support, cache compatibility or recovery semantics.

Recap

SGLang’s PD disaggregation is four flags: --disaggregation-mode set to prefill or decode with no combined value, --disaggregation-transfer-backend taking mooncake by default or nixl or ascend, --disaggregation-ib-device naming the RDMA device, and --disaggregation-bootstrap-port for the exchange that pairs the two roles. In front of them, sglang_router launched with --pd-disaggregation and one or more --prefill and --decode endpoints, which is a supported component rather than an example script. Every documented example names an RDMA device, and the bootstrap and waiting timeouts are raised in the field because a large model on a slow disk makes one side wait for the other.

NVIDIA Dynamo sits above an engine rather than being one. Its disaggregation goal is that prefill and decode scale independently; its decode workers receive key-value state over NIXL; and its router scores each worker by the prefill work a request would cost after discounting the blocks that worker already holds in device memory, host memory and on disk, plus the decode load it is carrying. That cost function is the most useful thing on this page even for readers who never install it, because it is the routing rule this part’s project asks you to design. Dynamo requires RDMA for disaggregated serving and its own documentation puts TCP at two to five hundred times slower for the transfer.

Kubernetes is where all three of these live in production, through Dynamo’s operator or the vLLM Production Stack, and this course does not go there.

Choose the engine for the rest of the job and use its disaggregation if the arithmetic favours a split. Next: the lab, which builds the split on two machines, measures it against one machine running the same model, and records whichever answer the measurement gives.

Check your understanding

Question 1. How does SGLang express the prefill and decode roles, compared with vLLM?

• Identically: both use a JSON object
• With flags rather than a document: --disaggregation-mode takes prefill or decode, and there is no combined value, whereas vLLM sets kv_role inside --kv-transfer-config and does have a kv_both
• SGLang does not support disaggregation
• SGLang uses two separate executables

Show the answer and why

Answer: With flags rather than a document: --disaggregation-mode takes prefill or decode, and there is no combined value, whereas vLLM sets kv_role inside --kv-transfer-config and does have a kv_both

The mechanisms are equivalent and the syntax differs. The absence of a combined mode in SGLang is a real difference: a vLLM instance with kv_both is how a single instance talks to a store rather than to a partner, and SGLang expresses that idea elsewhere.

Question 2. Every first-class example on SGLang's PD disaggregation page names a device like mlx5_roce0 in --disaggregation-ib-device. What should you conclude?

• That the flag is optional and the value is arbitrary
• That the feature is written for machines with an RDMA link between them, so on plain Ethernet the arithmetic from the previous lesson is what decides whether running it is worth doing
• That SGLang only runs on InfiniBand
• That the device name will be the same on your machines

Show the answer and why

Answer: That the feature is written for machines with an RDMA link between them, so on plain Ethernet the arithmetic from the previous lesson is what decides whether running it is worth doing

It will run without RDMA, because the transfer backends fall back. The documentation is telling you what it was designed and tested against, which is a different and more useful signal than a hard requirement. Copying a device name from a page is its own mistake: find yours on your own hardware.

Question 3. Dynamo's router scores each worker with a cost function. Which of these does that function account for? Select all that apply.

• The prefill work the request would need, discounted by blocks the worker already holds in device memory
• Separate discounts for blocks held in host memory and on disk, each with its own weight
• The decode load the worker is already carrying, through its active requests
• The network distance between the client and the worker

Show the answer and why

Answer: The prefill work the request would need, discounted by blocks the worker already holds in device memory, Separate discounts for blocks held in host memory and on disk, each with its own weight, The decode load the worker is already carrying, through its active requests

The documented expression subtracts a device-overlap term, a host-cache term and a disk-cache term from the prefill blocks, then adds potential decode blocks and a weighted active-request count. Nothing in it is about the client. It is the routing rule this part's project asks you to design in words: who already has most of this prompt, and who is least busy.

Question 4. Your SGLang prefill and decode servers both start, the router reports both healthy, and no request ever completes. Where do you look first?

• The model files, which are probably corrupt
• The bootstrap: check that --disaggregation-bootstrap-port is the same on both and reachable between the machines, and raise the bootstrap and waiting timeouts, because a pair waiting for each other prints nothing and answers health checks normally
• The router's policy setting
• The GPU memory fraction

Show the answer and why

Answer: The bootstrap: check that --disaggregation-bootstrap-port is the same on both and reachable between the machines, and raise the bootstrap and waiting timeouts, because a pair waiting for each other prints nothing and answers health checks normally

Health checks test that a server is answering HTTP, not that the pair has found each other. The documented examples raise the two timeouts to hundreds or thousands of seconds precisely because one side commonly finishes loading a large model long before the other.

Question 5. You already use vLLM from Part 9, have two machines on ordinary Ethernet, and want to learn how disaggregation works. What does this lesson recommend?

• Switch to SGLang, because its router is a supported component
• Stay with vLLM and use the shared-directory connector: it needs no RDMA, no UCX and no bootstrap, and the Part 18 mount already exists
• Install Dynamo and its Kubernetes operator
• Buy RDMA cards before starting

Show the answer and why

Answer: Stay with vLLM and use the shared-directory connector: it needs no RDMA, no UCX and no bootstrap, and the Part 18 mount already exists

Choose the engine for everything else it has to do and then use its disaggregation. The three implementations are near-identical in mechanism, and switching engines to obtain a particular connector name is a poor trade. On plain Ethernet the split is unlikely to win on latency anyway, which is a result worth measuring rather than a reason not to build it.

Check my answersClear