Skip to content
Local LLM Course

Beyond One Machine: Parallelism and Networks

Everything up to here has run on one machine. Level 4 is where that stops being enough, and this part is the map of what happens next.

There are only two honest reasons to add a second machine to a language-model setup at home. The first is capacity: the weights and the key-value cache do not fit anywhere, and no amount of patience fixes that. The second is throughput: the model fits, but one machine cannot serve the requests you want to serve. Everything else, and there is a great deal of everything else on the internet, is either one of these two wearing a different hat or a project that would have been finished sooner on one box.

That framing matters because splitting a model is not free. A model split across two machines sends data over a cable at every step where it used to send data over a memory bus, and the cable is slower than the bus by a factor you can calculate before you buy it. Which of the five parallelism strategies you choose decides how much data crosses that cable and how often, and therefore whether the cluster is faster than one machine or a slower, more complicated version of it. Part 19’s challenge page is built entirely around a cluster that lost that argument.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • say which of capacity, bandwidth and concurrency is actually limiting your setup, and what a second machine does and does not do for each;
  • define tensor, pipeline, expert, data and sequence parallelism precisely, name the collective operation each one performs, and say how often per token it happens;
  • explain from the communication pattern alone why tensor parallelism is normally kept inside one machine and pipeline parallelism is what crosses between machines;
  • work out, symbolically, how many bytes cross a link per token for a given split, and compare that against a link’s rated speed to predict whether the link will be the bottleneck;
  • describe what 2.5, 10, 25, 100 and 200 gigabit Ethernet, Thunderbolt 4 and Thunderbolt 5 offer between two machines in a house, and what remote direct memory access removes from the path that a normal network stack keeps in it;
  • name the link class each of the four tracks can realistically reach, and choose one for the parallelism you intend to run rather than for the number on the box;
  • read the course’s reference cluster, understand the role each machine plays, and map it onto whatever machines you actually own, including the case where you own exactly one;
  • build the network layer of a cluster: addresses, names, keys, a measured throughput and latency figure per link, a verified maximum transmission unit, and shared model storage.

What you need first

Section titled “What you need first”

Part 5, because every conversation in this part is about memory capacity and memory bandwidth on the four platforms, and Part 5 is where you measured yours. Part 3’s inference lesson and Part 4’s memory-budget lesson, because the arithmetic here is the same arithmetic with a network term added. Part 9’s gateway, because a cluster nearly always ends up with a router in front of it and that is the thing you already built.

The lab needs two or more machines on the same network for its primary path, and 8 GB of memory on each. If you have one machine, the single-machine path runs the whole procedure between two containers on a bridge network: the measured numbers mean something different, but the method, the scripts and the notebook entries are the same, and the muscle memory transfers intact.

How to work through it

Section titled “How to work through it”

Read the four lessons in order. The first is a diagnosis, and it may well tell you that your problem is not a cluster problem, which is a good outcome for twenty-five minutes of reading. The second is the vocabulary that Parts 19 to 22 assume without re-explaining. The third is the part you should read before spending money on a network card or a cable, because it is written to stop you buying the wrong one. The fourth introduces the reference cluster and gives you a worksheet to fill in for your own machines.

Then do the lab. It builds no cluster and runs no model. It gives every machine a stable name, lets them talk to each other without passwords, measures every link in both directions, verifies the packet size the link will actually carry, and puts the models where all of them can read them. Parts 19, 20, 21 and 22 all begin by assuming that this is done, and every hour of debugging a distributed engine that turns out to be a name-resolution problem is an hour this lab was meant to save you.

One habit to carry out of this part: measure the link before you blame the engine. A distributed inference run that is disappointing has a network number behind it, and you will already have that number in your notebook.

Part18 of 27Level4 — Cluster ArchitectPages5Estimated time2.9 hours

1 of 5 pages in this part need two or more machines for their primary path, marked below. Every one of them has a single-machine path, so the part can be completed with one computer.

0 / 5 lessons in this part completed