Why One Box Runs Out: The Memory Wall and the Bandwidth Wall

By the end of this lesson you will be able to say which of three limits is actually binding on your setup, predict what a second machine does to each of them before you buy one, and name the costs that arrive with it. That last part is the reason the lesson exists. Adding a machine is the most expensive thing in this course, and the arithmetic that says whether it will help takes about two minutes.

There are three ways to run out on one box, and they are not interchangeable. Treating a bandwidth problem as a capacity problem is how people end up with a cluster that is slower than the machine they started with.

The capacity wall

The first wall is the one everybody meets. The weights plus the key-value cache are larger than the memory, and the machine either refuses to load the model or loads it by spilling to disk, at which point the token rate falls by orders of magnitude and you notice.

The arithmetic is Part 4’s, unchanged. Weights come from the file size at your chosen format; the cache is two bytes per element times layers times key-value heads times head dimension per token, times however many tokens you asked the engine to hold. Here is what that looks like for a model that sits just past the largest single machine in this course.

Qwen3-235B-A22B at Q4_K_M, 32k context, on one 128 GB machine

128 GB budget

Weights (Q4_K_M)

142 GB

KV cache, 32k tokens at FP16

6.3 GB

Requested

148.3 GB

Machine budget

128 GB

Over budget. 148.3 GB requested against a 128 GB machine - 20.3 GB over. Something here has to shrink: a smaller quantisation, a shorter context, or fewer of these reservations at once.

Weights from the file size recorded in src/data/models.json; the cache from the model's published bytes per token times 32,768. Neither figure was measured, and an engine adds overheads of its own. The point is the shape: this model is not close to fitting, so no amount of tuning on one 128 GB machine will load it.

Two 128 GB machines have 256 GB between them, and the same model fits with room for a longer context and for the operating system. That is the clean case for a cluster: nothing about the model changed, the machine simply became large enough to hold it. Qwen3-235B-A22B is Apache-2.0 licensed, as the model reference records, and so is gpt-oss-120b; GLM-4.6 is MIT. Licences matter less here than usual because you are not redistributing anything, but the habit of checking is the habit.

The same model across two 128 GB machines

Weights (Q4_K_M), split across both

142 GB

KV cache, 32k tokens at FP16

6.3 GB

Two operating systems and engine overhead, estimated

16 GB

Free

91.7 GB

Total

256 GB

The overhead segment is an estimate, not a measurement: two machines each need their own working set, and a distributed engine allocates buffers on both. Part 19's lab measures what is actually left.

▤Memory budget

Notice what the second diagram does not say. It does not say the model runs twice as fast. It says it runs at all. Capacity and speed are separate questions, and the next section is why.

The bandwidth wall

The second wall is subtler, and it is the one that produces disappointed cluster owners.

Decode is bandwidth-bound. Part 3 established the ceiling: tokens per second cannot exceed memory bandwidth divided by the bytes of active weights read per token. A machine with a lot of memory and modest bandwidth, which describes three of the four tracks, hits that ceiling long before it hits any compute limit.

Now put the model on two machines and ask what happens to the ceiling. The answer depends entirely on how you split it, and the two common splits give opposite answers.

Split the model by layers, so machine one holds the first half and machine two holds the second half. To produce one token, machine one reads its layers, sends a small activation vector across the link, and machine two reads its layers. The two reads happen one after the other, because machine two cannot start until machine one has finished. Total bytes read per token is unchanged, total time is unchanged, and the ceiling is exactly what it would have been on a single machine large enough to hold everything. You bought capacity, not speed.

Split the model within each layer, so both machines hold half of every weight matrix and both read at the same time. Now the two reads overlap, the bytes read per machine per token is halved, and the ceiling doubles, minus whatever the machines spend agreeing on the result at every layer boundary. That last term is the whole subject of the next lesson, and it is not small.

Pending validationWhat a second machine does to the decode ceiling, batch size 1

Arrangement	Active bytes read per machine per token (GB)	Machines reading at once	Ceiling from arithmetic (tokens/s)
One machine, if it were large enough	13.3	1	20.5
Two machines, split by layers	6.65	1	20.5
Two machines, split within layers	6.65	2	41

DGX Spark class (GB10, 128 GB), vendor bandwidth figure as recorded in src/data/hardware.json · none - arithmetic only, no engine was run computed from src/data/hardware.json and src/data/models.json on 2026-09-09 · Qwen3-235B-A22B (mixture of experts, 22B active of 235B total), Q4_K_M · 32,768 tokens of context · 2026-09-09

Estimates from arithmetic, not measurements. Active bytes approximate the per-token read as the file size scaled by the ratio of activated to total parameters, which ignores attention and embedding weights that are read for every token, so the true figure is higher and every ceiling lower. The third row also ignores the cost of the collective operation at each layer boundary, which is exactly what the network lesson makes you calculate. Part 19 and Part 20 measure the real thing.

The third row is the one to be suspicious of. It is the arithmetic before the network bill arrives. vLLM’s parallelism guidance is blunt about where that bill is payable: it says that “efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand”. A home network is usually not that, which is why the same page recommends keeping the within-layer split inside a machine and using the by-layer split between machines.

✕Common mistake

The commonest disappointment in home clustering: a model that already fitted on one machine is split across two “to make it faster”, using an engine that splits by layers. The reader gets the same token rate as before, plus a network hop, plus a second machine’s power bill. The measurement that would have predicted it takes two minutes and this lesson is it.

Prefill is a different question

One qualification, because it changes what “faster” means. The bandwidth wall is a decode wall. Prefill, reading the prompt, is compute-bound rather than bandwidth-bound, and it behaves better under a split than decode does: two machines each doing arithmetic on their own layers still do the arithmetic once each, but a serving engine can have one machine reading the next request’s prompt while the other is still writing the last request’s answer.

That overlap is real and it is why a layer split helps a busy server more than it helps you sitting at a prompt. It is also why the honest way to describe a cluster’s benefit is always with a workload attached. “Faster” for a single interactive conversation and “faster” for twenty concurrent requests are different claims about the same hardware, and Part 19’s lab measures both rather than reporting one and implying the other.

The concurrency wall

The third wall has nothing to do with the model fitting and everything to do with how many people are asking. One machine serving one conversation leaves most of its arithmetic idle, which is why Part 9 spent so long on batching: a served model gets dramatically more total throughput from sixteen concurrent requests than from one, at the cost of some latency for each.

But batching runs out too. The key-value cache grows with every concurrent sequence, and at some point the engine either refuses new requests or starts evicting. When that is your limit, the fix is the easiest one in this part: run a second complete copy of the model on a second machine and put a router in front of them. Nothing is split, nothing crosses the network per token, and the two machines do not need to talk to each other at all.

This is data parallelism, and at home it is by far the highest ratio of benefit to trouble. The gateway from Part 9 is already the router; adding a second backend to it is a configuration change. Part 23 operates a service built this way.

Which wall are you against?

1. 1Does the model load at all?If not, and no smaller quantisation is acceptable, this is a capacity problem. A second machine adds memory; splitting by layers is enough.
2. 2Does it load but decode too slowly for one user?This is a bandwidth problem. A second machine helps only with a within-layer split and a fast link, and a smaller or sparser model on one machine is usually the better answer.
3. 3Is it fast enough alone but not for everyone?This is a concurrency problem. Run a second copy behind the gateway. Nothing is split and no fast link is needed.
4. 4None of the above?Then you do not have a cluster problem, and Part 17 (caching and speculative decoding) or a better quantisation from Part 16 is where the time is.

The diagnosis before the purchase. Three walls, three different answers, and one of the answers is 'do not build a cluster'.

What a second machine costs

Capacity, bandwidth and concurrency are the benefits. These are the costs, and they are worth writing in the notebook next to the benefits before ordering anything.

Latency per token. Every layer boundary that crosses a cable adds the time to move the data plus the time for the two machines to agree that it moved. Even a small transfer carries a fixed cost. Over a home network, that fixed cost is measured in tens or hundreds of microseconds, and it is paid once per boundary per token.

Availability. A two-machine cluster is available when both machines are available. If each is up ninety-nine per cent of the time and their outages are independent, the pair is up about ninety-eight per cent of the time. Distributed engines are also worse at partial failure than single processes: llama.cpp’s RPC README describes the backend as being “in a proof-of-concept development stage” and says plainly that “the functionality is fragile and insecure”, which is an honest description of most of this software.

Complexity. A single-machine failure is a log file. A cluster failure is a log file on each machine, a network to check, a clock to compare and a name to resolve. This is why the lab in this part is entirely about the network layer: it converts a whole class of future confusion into a number you already measured.

Power and heat. Two machines draw roughly twice the power of one for the same conversation when the split is by layers, because both are reading weights and only one is doing useful work at a time. Nobody puts this in the benchmark tables and it belongs in yours.

The frontier of what the software supports. The Megatron scaling paper puts the general problem well: naive use of these methods “leads to fundamental scaling issues at thousands of GPUs, e.g., due to expensive cross-node communication or devices spending significant time waiting on other devices to make progress”. At home you are not at thousands of GPUs, but you are on the same curve, with a much slower network and much less mature tooling.

The option nobody costs: one bigger machine

Before the four honest cases, one comparison that is almost always missing from the discussion.

The alternative to two machines is not always one machine you already have. It is sometimes one larger machine. The course hardware reference records memory tiers up to 512 GB on a single Apple silicon Mac Studio, and 128 GB on a single DGX Spark or Ryzen AI Max+ 395 box. A single 256 GB machine holds the same model as two 128 GB machines, with none of the network, none of the availability multiplication, one operating system to keep patched, one power supply, and a decode ceiling set by its own memory bandwidth rather than by whichever half finishes last.

Two machines win on three counts: they can be bought one at a time, they can be different from each other, and one of them can fail while the other keeps serving something smaller. One machine wins on simplicity, on latency, and on every hour you would otherwise spend on the network. Neither answer is universal, which is exactly why the comparison belongs in the notebook next to the arithmetic rather than being settled by whichever option sounds more interesting.

✦Tip

Write both options down with their memory, their bandwidth and their price before you buy either. The comparison takes ten minutes and it is the single most valuable thing in this lesson for anyone who has not already bought the second machine.

The honest cases for clustering at home

Four cases justify the trouble.

A model class you cannot otherwise run. This is the strongest. If the 200B-class and 400B-class models are the reason you are here, a pair of 128 GB machines is the accessible way to hold one, and Parts 19 and 20 do exactly that.

More concurrent users than one box serves. A second replica behind the gateway. The simplest cluster, and the one most people actually need.

Role separation. One machine serving while another fine-tunes, or a machine that holds the shared model library and never runs an engine at all. Nothing is split; the machines simply have different jobs. This is the arrangement most home labs converge on and it is the reference cluster’s shape.

Learning the technique. A legitimate reason, provided you write it down as the reason. A cluster built to learn distributed inference has succeeded when you understand it, even if the tokens per second went down.

And one case that is not on the list: making an interactive chat feel faster. On a home network, with the engines available today, that is a job for a smaller model, a better quantisation, prefix caching or speculative decoding, all of which you already have from Parts 16 and 17.

⌂Single-machine path

Every page in Level 4 that needs more than one machine also gives you a way to do it on one, using separate processes, containers or virtual machines. You will not get the performance, but you will get the procedure, the failure modes and the vocabulary, and those are what transfer when you do have a second machine. The single-machine path is stated on each page rather than left implied.

Decide whether the second machine adds capacity or service rate

If a model does not fit on one machine, splitting it can make an otherwise impossible workload possible. That is a capacity result. If it already fits, a second machine must justify communication and coordination costs by reducing latency or serving more requests. Those are separate hypotheses and need separate baselines.

Draw the bytes resident on each machine and the bytes transferred during each request. Distinguish one-time model loading from communication repeated per layer or token. A large initial transfer can be tolerable for a long-lived service; a small transfer repeated thousands of times can dominate interactive latency.

For the decision record, compare a split model with independent replicas and with a smaller single-machine model that meets the quality target. Include aggregate power and operational complexity. A cluster that hosts a larger model at lower speed may still be the right choice if that model uniquely passes the task evaluation. Explain the requirement it satisfies instead of treating the existence of more hardware as evidence of acceleration.

Recap

Three walls, three different answers. Capacity: the model does not fit, a second machine adds memory, and a split by layers is enough to use it. Bandwidth: the model fits but decodes slowly, a second machine helps only with a within-layer split on a fast link, and the arithmetic usually points at a smaller model instead. Concurrency: one machine cannot serve everyone, a second full replica behind the gateway fixes it, and nothing needs to be split at all.

The costs are latency per layer boundary, availability that multiplies rather than adds, operational complexity, power drawn by machines that are waiting, and software that its own authors describe as fragile. The four honest reasons to pay them are a model that will not otherwise load, more users than one box serves, giving machines separate jobs, and learning the technique on purpose.

The next lesson makes the split precise: five kinds of parallelism, the collective operation each performs, and how often per token it runs.

Check your understanding

Question 1. A 70B-class model at Q4 fits on your 64 GB machine and decodes at a rate you find slow. You add a second identical machine and split the model by layers across both. What should you expect?

• Roughly twice the decode speed
• Roughly the same decode speed, plus a network hop
• Half the decode speed
• The model will no longer load

Show the answer and why

Answer: Roughly the same decode speed, plus a network hop

A split by layers reads the same total bytes per token, just on two machines one after the other. The reads do not overlap, so the bandwidth ceiling is unchanged and the link adds a small fixed cost per token. The split buys capacity you did not need. A within-layer split would overlap the reads, but it needs a link fast enough to carry a collective operation at every layer.

Question 2. Which limit does adding a second complete replica of the model, behind a router, actually relieve?

• Capacity: it lets you run a larger model
• Single-user decode speed
• Concurrency: more simultaneous requests at the same per-user speed
• Prefill speed for a single long prompt

Show the answer and why

Answer: Concurrency: more simultaneous requests at the same per-user speed

Two independent replicas serve twice as many conversations. Neither replica is larger, so the largest model you can run is unchanged, and neither is faster, so one user sees no difference. This is data parallelism, and it needs almost nothing from the network because no per-token data crosses it.

Question 3. True or false: two machines that are each available 99 per cent of the time give a cluster that is available 99 per cent of the time.

• True
• False

Show the answer and why

Answer: False

A cluster that needs both machines is available only when both are up. With independent outages that is about 98 per cent, and the failures are also harder to diagnose. Availability multiplies downwards as you add machines that are all required, which is an argument for replicas, where either machine can serve, over splits, where both must.

Question 4. You want to run a 235B-class mixture-of-experts model at Q4 and you own two 128 GB machines. Which statement is the most accurate description of what you are buying?

• A model that would not otherwise load, at roughly the decode ceiling a single 256 GB machine would have
• A model that would not otherwise load, at roughly twice the decode ceiling of a single machine
• The same models as before, served to twice as many people
• Faster prefill on long prompts, with decode unchanged

Show the answer and why

Answer: A model that would not otherwise load, at roughly the decode ceiling a single 256 GB machine would have

The pair supplies capacity. With a layer split the reads are sequential, so the ceiling matches a hypothetical single machine with enough memory rather than doubling. Doubling would need a within-layer split and a link fast enough to carry its collective at every layer, which is the subject of the next two lessons.

Question 5. Which of these are good reasons to build a home cluster? Select all that apply.

• A 400B-class model is the reason you are doing this at all
• Your interactive chat with an 8B model is slower than you would like
• Four people in the house use the service at once and requests queue
• You want to learn how distributed inference actually behaves

Show the answer and why

Answer: A 400B-class model is the reason you are doing this at all, Four people in the house use the service at once and requests queue, You want to learn how distributed inference actually behaves

Capacity, concurrency and deliberate learning are all sound. A sluggish 8B chat is a bandwidth or a configuration problem on one machine: prefix caching, speculative decoding, a better quantisation or a smaller model, all of which Parts 16 and 17 cover, and none of which a second machine improves.

Check my answersClear