Skip to content
Local LLM Course

Disaggregated Serving: Prefill Machines and Decode Machines

Every engine in this course so far has done both halves of a request on the same device. It reads the prompt, then it writes the answer, and the two jobs take turns on hardware that suits one of them better than the other.

This part separates them. Prefill goes to the machine with more arithmetic throughput, decode goes to the machine with more memory, and the key-value cache the prompt produced travels from the first to the second. That is the architecture behind large-scale serving, and it is the reason the papers in this part exist. It is also the point in Level 4 where the honest answer is most likely to be “not on this network”, and the part is written to help you find that out with a measurement rather than an opinion.

The reference cluster has exactly the right ingredients. One machine has more compute than memory bandwidth; another has a great deal of memory at a modest price. Part 18 named those two jobs as separate roles for a reason, and this is the part that gives each of them its own box.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain why prefill is compute-bound and decode is bandwidth-bound, and what happens to the latency of each when they share one device;
  • define time to first token, time per output token and goodput, and say which of them a phase split is meant to improve and which it is not;
  • summarise what DistServe and Splitwise proposed, and what each one assumed about the network between the machines;
  • work out, from a model’s recorded shape, exactly how many bytes of key-value cache one request produces, and how long moving those bytes takes over each link class in Part 18’s table;
  • name the transfer libraries this field is built on, say what each needs from the network, and place the vocabulary correctly: a connector is not a transfer engine and neither is a store;
  • configure vLLM’s disaggregated prefill across two machines, choose a connector that a home network can actually carry, put the example proxy in front of the pair, and read the logs and metrics that tell you whether the cache moved;
  • describe SGLang’s prefill and decode roles and its transfer backends, and say what NVIDIA Dynamo adds above an engine and what it requires underneath one;
  • offload a key-value cache to host memory and to disk, measure what the tiers cost and what they return, and share a cached prefix between two engine instances;
  • design a tiered architecture for the machines you actually own, with a routing layer, a prefill pool, a decode pool and a cache tier, and justify every role with a number you measured.

What you need first

Section titled “What you need first”

Part 18, all of it. This part assumes the reference cluster’s names, the measured throughput and round-trip figures for each of your links, and the shared model library. The arithmetic in Lesson 2 is Part 18’s link table with a request-sized payload put through it, and without your own measured link numbers the second half of the part has nothing to compare against.

Part 17’s lesson on prefix caching and key-value reuse, because this part extends exactly that idea across a network and does not repeat it. If you have not proved a cache hit on one machine, proving one between two machines will be considerably harder.

Part 9, twice over: the vLLM and SGLang serving lessons are the engines this part configures, and load-test.py from that part’s lab is the measuring instrument used throughout. The gateway from Part 9’s project is one of the two candidates for the routing layer in this part’s project.

Parts 19, 20 and 21 are useful but not required. They split one model across machines; this part splits one request across machines, which is a different thing with different network consequences, and reading them in order makes the contrast sharper.

The first lab needs two machines with 24 GB each for its primary path. The single-machine path runs two engine processes on one device with the memory divided between them, and needs 16 GB. The second lab and the project run on one machine at 16 GB.

How to work through it

Section titled “How to work through it”

Read the four lessons in order. The first is the argument: two phases, two hardware profiles, and the interference that follows from running them together. The second is the arithmetic, and it is the lesson that decides whether the rest of the part is a good idea on your network. The third and fourth are the implementations, one in depth and two in survey.

Then do the first lab, which builds the split and measures it against a single machine running the same model. Expect the comparison to be unflattering on a house network. That is the intended result and the notebook line that records it is worth as much as a favourable one.

The second lab is the consolation prize, and it is a larger prize than it looks. Moving a cache between machines is expensive; keeping a cache instead of recomputing it is nearly free, and the tiers that make it possible are the same connectors and stores the first lab configured. Most home setups get more from the second lab than the first.

The project ties the level together: roles per machine, a routing layer in front, the cache tier underneath, and a design document whose every claim points at a measurement. A single-machine design is a valid submission as long as it says what a second machine would change and why.

One habit to carry out of this part: before you move a cache, work out how big it is. One multiplication done in advance decides whether a split is worth building, and it is the same multiplication whichever engine you end up using.

Part22 of 27Level4 — Cluster ArchitectPages7Estimated time6.0 hours

1 of 7 pages in this part need two or more machines for their primary path, marked below. Every one of them has a single-machine path, so the part can be completed with one computer.

0 / 7 lessons in this part completed