Why Prefill and Decode Want Different Hardware

By the end of this lesson you will be able to say why one request contains two workloads that want different machines, describe precisely how they get in each other’s way when they share one, define the three numbers that a phase split is judged on, summarise what DistServe and Splitwise proposed and what each assumed about the network, and state the one question that decides whether any of it applies to the machines in your house.

One request, two workloads

Part 3 introduced the split and every serving page since has used the vocabulary. Prefill reads the prompt: every token in it is available at once, so the engine can process them together as a large matrix multiplication, and the limit is how much arithmetic the device can do per second. Decode writes the answer: each token depends on the one before it, so the engine produces them one at a time, and for each one it must read the model’s weights out of memory. The limit there is memory bandwidth, not arithmetic.

That difference is not a detail of implementation. It is the shape of the computation, and it does not change with a better engine.

Two consequences follow immediately.

The two phases have different appetites for a device. A prefill of a few thousand tokens can saturate a GPU’s arithmetic units while barely touching its memory bandwidth. A decode step at batch size one does the opposite: it reads every weight the model needs for one token and does almost no arithmetic with each byte it read. This is why Part 5 had you measure both compute and bandwidth on your machine, and why the two figures did not predict each other.

They have different latency contracts. A user waiting for a reply notices the first token arriving as a pause; after that they notice the rate at which words appear. Those are two different measurements taken from two different phases, and a serving system can be good at one and poor at the other.

The three numbers

Every claim in this part is made in terms of three measurements, and Part 9’s load generator already reports all of them.

Time to first token is the interval between sending a request and receiving the first piece of the answer. It is dominated by prefill. vLLM exposes it as vllm:time_to_first_token_seconds, described in its production metrics page as a “Histogram of time to first token in seconds”.

Time per output token is the interval between subsequent tokens once the answer has started. It is dominated by decode. vLLM’s metric is vllm:request_time_per_output_token_seconds. Some papers and some engines call the same idea inter-token latency, and vLLM’s disaggregated prefilling page uses that name.

Goodput is the throughput that meets a latency requirement. Ordinary throughput counts every token the machine produced; goodput counts only the tokens produced by requests that stayed inside their time-to-first-token and time-per-output-token budgets. A machine can raise throughput and lower goodput at the same time by batching more aggressively, and that is exactly the trade the next section is about. DistServe’s title names goodput as the thing being optimised for precisely this reason.

!Important

Throughput and goodput answer different questions and the difference is the whole subject. If you have no latency requirement, batch everything and stop reading. If you do have one, the useful question is how many requests per second you can serve within it, and that number can fall while tokens per second rises.

Interference

Put both phases on one device and they compete, in a specific and observable way.

The engine schedules work in steps. In a step it can run a batch of decode operations, or it can run a prefill, or with chunked prefill it can run a slice of a prefill alongside some decodes. Whichever it chooses, the decoding requests that were mid-answer wait for that step to finish before they get their next token. A long prefill is a long step, and every user currently reading an answer sees a stall exactly as long as it.

SGLang’s disaggregation documentation names this first among its motivations, describing incoming prefill batches interrupting decode batches and causing substantial delays. vLLM’s disaggregated prefilling page makes the same argument from the other end, offering “controlling tail ITL” as one of the feature’s two benefits, on the grounds that separating the phases prevents prefill jobs from interrupting decode.

One device, two phases, and what a user reading an answer experiences

Shared, whole prefill

decodedecodeprefill for a new requestdecodedecode

Shared, chunked prefill

decodedec + chunkdec + chunkdec + chunkdecodedecode

Split across two machines

decodedecodedecodecache arrivesdecodedecodedecode

Schematic, not measured: the blocks show ordering rather than duration. In the top row a long prefill arrives while three users are mid-answer, and all three wait for it. In the middle row chunked prefill breaks the same work into slices, which shortens each stall but adds one to every step. In the bottom row the prefill happens on another machine entirely, and the decode stream is uninterrupted at the cost of moving the cache.

Chunked prefill, which Part 9 introduced and which vLLM enables by default, is the single-machine answer to this problem: cut the prefill into pieces small enough that no step gets very long. It works, and it is the right first move. It does not remove the competition, it spreads it, and every step now carries some prefill work whether or not there is a decode deadline approaching.

The second cost of sharing is subtler and it is the one the papers care about most. Because both phases run on the same device, they must also share every configuration decision: the same parallelism layout, the same memory split, the same batch policy. DistServe’s abstract puts it plainly, saying that colocating the phases “not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases”, and that under strict latency requirements a colocated system “has to prioritize one latency over the other, or over-provision compute resources to meet both”.

What the two papers proposed

Two 2023 and 2024 papers set the terms of this whole field, and both are worth reading in full.

DistServe (Zhong and others, submitted January 2024, revised June 2024) assigns prefill and decoding to different GPUs, which its abstract says eliminates the interference, and then co-optimises the resource allocation and parallelism strategy separately for each phase given the application’s time-to-first-token and time-per-output-token requirements. The paper reports serving “7.4x more requests or 12.6x tighter SLO” against what it called state-of-the-art systems in 2024, “while staying within latency constraints for > 90% of requests”. That is a claim about the systems it was compared against, on the hardware it used, and it is quoted here as the authors’ reported result rather than as a figure this course has reproduced.

One sentence in that abstract matters more than the headline number for anybody building this at home: DistServe “places the two phases according to the serving cluster’s bandwidth to minimize the communication caused by disaggregation”. The design assumes a bandwidth budget it can plan around, and it spends effort minimising traffic because the traffic is the cost.

Splitwise (Patel and others, submitted November 2023, revised May 2024) starts from a measurement rather than a scheduler. It characterises the two phases and finds that prompt computation is compute-intensive while token generation is memory-intensive, with, in the paper’s terms, distinct latency, throughput, memory and power characteristics. From that it draws the conclusion this part is built on: token generation “do not require the compute capability of the latest GPUs, and can be run with lower power and cost”.

That is a hardware argument, not a scheduling one. If decode does not need the newest accelerator, you can buy a cheaper or older one for it, and spend the budget where prefill lives. Splitwise reports 1.4 times higher throughput at 20 per cent lower cost against the designs it compared with, and 2.35 times more throughput at the same cost and power, again as the authors’ reported figures. It moves the state between machines over what it calls the fast back-plane interconnects available in today’s GPU clusters.

∴Research note

Read both abstracts with the deployment in mind. Every reported improvement in either paper was measured in a cluster with a data-centre interconnect between the machines. Neither paper claims anything about two boxes joined by house Ethernet, and neither should be quoted as if it did. The mechanism transfers; the numbers do not.

Heterogeneous hardware, which is what a house has

Splitwise’s conclusion is unusually convenient for a home lab, because a home lab is almost never a matched set. The reference cluster in Part 18 is four different machines, and their differences are exactly the ones the phase split cares about.

Matching the phase to the machine

• clientClientschat, an editor, an agent loop
• routerRouterthe gateway from Part 9, or the engine's own proxy
• prefillPrefill machinethe box with the most arithmetic throughput per second; its memory only has to hold the weights and one prompt at a time
• decodeDecode machinethe box with the most memory at a reasonable bandwidth; it holds every conversation in flight
• storageModel libraryboth machines load the same weights

• Clients ⟷ connected to Routerhouse network; requests in, tokens out
• Router ⟷ connected to Prefill machinethe prompt
• Prefill machine ⟷ connected to Decode machinethe whole prompt's key-value cache, once per request
• Router ⟷ connected to Decode machinethe request again, to be continued from the transferred cache
• Model library ⟷ connected to Prefill machineread at load time only
• Model library ⟷ connected to Decode machineread at load time only

Both machines hold the whole model. What moves between them is one request's key-value state, once, at the moment the prompt has been read. Part 18's role list called these prefill and decode; this is what it looks like wired up.

Three things about that picture are worth stating before the next lesson turns them into arithmetic.

Both machines hold the entire model. This is not a capacity technique. Nothing here lets you run a model that did not fit; Parts 19, 20 and 21 are where that happens. If a model fits on neither machine, disaggregation does not help.

The transfer is per request, not per token. Part 18’s second role layout made this contrast explicitly. A pipeline split sends a small activation across the link on every token; a phase split sends a large block once per request. Which of those a link can carry is a completely different question, and the second lesson answers it.

The decode machine is the one that needs memory. Every conversation in flight has its whole key-value cache sitting on the decode machine until it finishes. The prefill machine only needs room for the prompts it is currently reading. That asymmetry is what makes the unified-memory tracks good decode nodes and a small-memory machine with a fast GPU a good prefill node.

What a house network changes

Here is the sentence the rest of this part exists to make concrete: in a data centre the transfer is cheap, and in a house it usually is not.

NVIDIA’s Dynamo documentation states the case for its own stack in a form that is unusually direct about the size of the gap. Its RDMA setup page says “Dynamo needs RDMA for disaggregated serving, where prefill workers generate KV cache and hand it to decode workers”, and gives the alternative and its cost in the same paragraph: “The alternative is TCP over Ethernet, which is 200-500x slower for this transfer: roughly 98s Time To First Token (TTFT) on TCP versus 200-500ms with RDMA.” Those are the vendor’s own figures for the vendor’s own stack, read on 2026-09-09, and they are quoted here as reported by NVIDIA rather than measured by this course. It also notes that aggregated deployments, which run both phases in one worker, “transfer no KV cache between workers, and do not need RDMA”.

A comparison with two orders of magnitude in it is not a tuning problem. If a transfer that should take a fraction of a second takes a minute and a half, the split has made time to first token worse by more than the interference it removed, and the correct engineering response is to not do it.

That leaves three honest positions for a reader with two machines and a house switch, and this part teaches all three.

1. You have a fast direct link. Two DGX Sparks with a ConnectX-7 cable between them have RDMA over converged Ethernet, and the transfer arithmetic in the next lesson will show it is comfortable. This is the primary path in the first lab.
2. You have ordinary Ethernet. The transfer will dominate. Measure it anyway, record it, and read the result as the answer to a real question rather than as a failure. A notebook line that says “on 2.5 gigabit Ethernet the split cost me time to first token” is worth having, and it is the reason the second lab exists.
3. You have one machine. Two engine processes on one device reproduce every configuration step and every failure mode, and the transfer happens over the loopback interface, which teaches the mechanism and measures nothing about a network. That is exactly how the single-machine path is labelled, on the page and in the notebook.

?Reality check

The claim to be careful with is “disaggregation makes serving faster”. vLLM’s own page rejects the general form of it in three words on the page itself, saying “Disaggregated prefill DOES NOT improve throughput”. What it offers instead is the ability to tune time to first token and inter-token latency separately, and to control tail latency. Those are latency properties under a load with a deadline, not a speed-up, and a benchmark that shows one machine beating a split pair on total tokens per second has not contradicted anything.

When the split is worth considering at home

Collecting the argument into a short list, because the next three pages assume you have made this judgement already.

The workload has long prompts and short answers. Retrieval, document question-answering, classification over pasted text. Prefill dominates, and it is the phase you can move to better hardware. If your prompts are short and your answers long, almost all the work is decode and there is nothing worth splitting.

There are enough concurrent users for the interference to be real. With one user at a time there is nothing for a prefill to interrupt. The problem the papers solve appears when several people are mid-answer and a new long prompt arrives.

The two machines are genuinely different. If both boxes are the same, a phase split gains you only the scheduling isolation, and you could have had most of that from two replicas behind the Part 9 gateway with less to go wrong. The replica pool is the arrangement to reach for first, as Part 18 said, and this part does not change that advice.

The link can carry a request’s cache in less time than the prefill took. This is the arithmetic condition, it is the one people skip, and the next lesson is entirely about it.

Include the handoff in the latency budget

Splitting prefill and decode adds a transfer and coordination step between them. Write time to first token as queueing plus prefill plus handoff plus the first decode work, with the exact boundary defined by the implementation. A faster prefill device helps only if its saving exceeds the added cost for the workload you serve.

Prompt length changes both prefill work and the amount of state to transfer. Concurrency changes interference and queueing. Use short, medium and long prompt classes with comparable answer lengths and measure each separately. A mean across an undocumented mix hides which requests benefit.

Compare against a colocated baseline using the same total hardware where feasible, and also against the simpler deployment you would otherwise operate. Keep failure and cancellation behaviour in the test: a request whose cache transfer fails still consumes resources somewhere. Disaggregation is a service design for particular workload and topology conditions. It should follow evidence of interference or specialisation benefit, rather than being assumed to improve every two-machine system.

Recap

One request is two workloads. Prefill reads the prompt in parallel and is limited by arithmetic throughput; decode writes the answer serially and is limited by memory bandwidth. Sharing a device between them means a long prefill stalls every answer in flight, and it means both phases must accept the same parallelism and memory decisions. Chunked prefill spreads the stall; it does not remove the coupling.

DistServe put the phases on different GPUs to remove the interference and to let each phase be configured for its own latency target, and it places the phases according to the cluster’s bandwidth because the transfer is the cost. Splitwise measured the two phases, found that token generation does not need the newest accelerator, and split the phases across different hardware accordingly. Both reported large improvements in clusters with data-centre interconnects.

The three numbers to judge any of this by are time to first token, time per output token and goodput, and Part 9’s load generator reports all three. Throughput on its own answers a different question, and vLLM says outright that disaggregated prefill does not improve it.

What a house changes is the link. NVIDIA’s own documentation puts TCP over Ethernet at two to five hundred times slower than RDMA for this specific transfer. Next: exactly how many bytes have to move, worked from the model shapes this course already records, and how long that takes on each of the links Part 18 had you measure.

Check your understanding

Question 1. Your service has one user at a time, sending short prompts and asking for long answers. What would splitting prefill and decode onto two machines buy you?

• A large reduction in time to first token
• Very little: almost all the work is decode, there is no concurrent request for a prefill to interrupt, and you have added a network transfer to every request
• The ability to run a model twice as large
• Higher total throughput

Show the answer and why

Answer: Very little: almost all the work is decode, there is no concurrent request for a prefill to interrupt, and you have added a network transfer to every request

The interference a phase split removes only exists when a prefill arrives while other requests are decoding. With short prompts there is little prefill work to move, and with one user there is nothing to interrupt. Both machines still hold the whole model, so nothing about capacity changes either.

Question 2. Which of these are accurate summaries of what the two founding papers proposed? Select all that apply.

• DistServe assigns prefill and decoding to different GPUs and co-optimises the parallelism plan for each phase separately
• DistServe places the phases according to the cluster's bandwidth in order to minimise the communication that disaggregation causes
• Splitwise found that token generation does not require the compute capability of the latest GPUs and can run at lower power and cost
• Both papers measured their improvements over ordinary gigabit Ethernet between commodity machines

Show the answer and why

Answer: DistServe assigns prefill and decoding to different GPUs and co-optimises the parallelism plan for each phase separately, DistServe places the phases according to the cluster's bandwidth in order to minimise the communication that disaggregation causes, Splitwise found that token generation does not require the compute capability of the latest GPUs and can run at lower power and cost

The first three come straight from the abstracts. The fourth is the one to reject: Splitwise moves state over what it calls fast back-plane interconnects in GPU clusters, and DistServe explicitly optimises placement against the cluster bandwidth it has. Neither result is a claim about a house network, which is why this part measures rather than assumes.

Question 3. A colleague reports that after enabling disaggregated prefill their total tokens per second went down, and concludes the feature is broken. What is the better reading?

• They are right; a working split always raises throughput
• vLLM's own documentation states that disaggregated prefill does not improve throughput; the feature targets separate tuning of time to first token and inter-token latency, and control of tail latency, so throughput is the wrong measurement to judge it by
• They should increase the batch size until throughput recovers
• Throughput and goodput are the same measurement

Show the answer and why

Answer: vLLM's own documentation states that disaggregated prefill does not improve throughput; the feature targets separate tuning of time to first token and inter-token latency, and control of tail latency, so throughput is the wrong measurement to judge it by

The page says it in capitals. What a split offers is the ability to configure each phase for its own latency target and to stop prefill work from stalling answers in progress. The measurement that captures that is goodput under a latency requirement, or the two latency figures separately, not tokens per second in aggregate.

Question 4. Why is the decode machine the one that needs plenty of memory?

• Because decode needs a larger copy of the weights
• Because every conversation in flight keeps its whole key-value cache on the decode machine until the answer finishes, while the prefill machine only needs room for the prompts it is reading right now
• Because the decode machine also runs the router
• Because prefill runs at lower precision

Show the answer and why

Answer: Because every conversation in flight keeps its whole key-value cache on the decode machine until the answer finishes, while the prefill machine only needs room for the prompts it is reading right now

Both machines hold the same weights. The asymmetry is in the cache: a prompt's key-value state is produced once on the prefill machine and then lives on the decode machine for the whole answer. That is why unified-memory boxes make good decode nodes and a small-memory machine with a strong GPU makes a good prefill node.

Question 5. True or false: NVIDIA's documentation describes an aggregated deployment, with prefill and decode in one worker, as needing RDMA.

• True
• False

Show the answer and why

Answer: False

False, and the distinction is the practical one. The RDMA setup page says aggregated deployments transfer no key-value cache between workers and do not need RDMA. It is disaggregation specifically that creates the transfer, and it is that transfer the page describes as two to five hundred times slower over TCP than over RDMA.

Check my answersClear