Skip to content
Local LLM Course

Clustering with llama.cpp RPC

Part 18 explained why a second machine helps and what a network has to do to make it help. This part is the first place you actually build one, and it is deliberately the simplest tool that can do it.

llama.cpp’s RPC backend is two pieces. On every machine you want to lend to the cluster, a small program called ggml-rpc-server starts up, finds the accelerators that machine has, and offers them over a socket. On the machine you sit at, llama-server or llama-cli is given a list of those sockets and treats every remote device as though it were another card in the local box. The model’s layers are then spread across the whole set. Nothing else changes: the same GGUF file, the same flags you learned in Part 6, the same client.

That simplicity is why this part comes before the NVIDIA and Apple cluster parts rather than after them. Part 20 needs two DGX Sparks and a QSFP cable. Part 21 needs Apple silicon and Thunderbolt. This part needs whatever you have. A Spark, a mini PC and a laptop can serve one model together, with three different GPU backends, and the course knows of no other tool that will do that today.

What this part is honest about

Section titled “What this part is honest about”

The RPC backend’s own README opens with a warning in bold: it is a proof of concept, the functionality is fragile and insecure, and it must never be run on an open network. That is the project’s assessment of its own code, not the course being cautious on its behalf, and it shapes everything here. The lab binds every server to one interface on a private cluster link and nothing else, and the security section of the first lesson explains exactly what an unauthenticated device server on your network means.

The second honest thing is about speed. A layer split across machines does not make a model faster. It makes a model possible. If the model already fits on one of your machines, splitting it across two will almost certainly be slower, and the challenge at the end of this part is a worked example of somebody discovering that the hard way. Clustering is a capacity answer, and Part 18’s first lesson is where you decide whether capacity is your problem.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • start ggml-rpc-server on any machine you own, bound to the cluster link and nothing else, and say which device it is offering and why;
  • point a client at several of them, read the device list back, and know which position in that list each machine occupies before you write a split;
  • work out from a model’s file sizes and your machines’ memory whether it will fit across the cluster, using the same arithmetic Part 4 taught for one machine;
  • set --tensor-split deliberately, in proportion to memory or to speed, and explain which you chose and why;
  • run a 200B-class quantised model across two or three machines when no single one of them could hold it;
  • mix a CUDA host, a Vulkan host, a Metal host and a CPU host in one run, and predict from the split alone which of them will hold the run back;
  • move a link from TCP to RDMA where the hardware allows it, force it back to TCP for a comparison, and measure the difference rather than assume it;
  • measure prefill, decode and the bytes that actually crossed the cable during a generation, and compare the last of those against the arithmetic that predicted it;
  • diagnose a cluster that is slower than one of its own machines, in a fixed order, and prove the fix with a second measurement.

What you need first

Section titled “What you need first”

Part 6, because everything here is llama.cpp and this part assumes you can build it, recognise which backend you built, and read a load log. The build needs one extra flag, which the first lesson gives.

Part 4’s memory-budget lesson, because the fit arithmetic is identical, only now the total is the sum of several machines instead of one.

Part 18, in full and in advance. Its lab gives every machine a stable .home.arpa name, sets up passwordless access, measures every link in both directions, verifies the maximum transmission unit and puts the models where every machine can read them. This part begins by assuming all of that is done, and its challenge page exists largely because people skip it.

How to work through it

Section titled “How to work through it”

Read the two lessons first. The first is the mechanism: what the server exposes, how the client splits, what crosses the cable and how often, and why a layer split tolerates a slow link where a tensor split would not. The second is the measurement: transports, where the time goes, where the key-value cache lives, and how to attribute a result to one host in the cluster rather than to the cluster as a whole.

Then the two labs, in order. The first is the headline: a model that fits nowhere in your house, running. The second is the more useful one day to day, because it is about mixed hardware, unequal hosts and what happens when one of them disappears mid-generation.

Finish with the challenge. It hands you a cluster that is three times slower than one of its own machines and asks you to explain it in a fixed order, with evidence, before touching anything. That procedure is the transferable skill in this part. The cluster is not.

Part19 of 27Level4 — Cluster ArchitectPages5Estimated time3.9 hours

3 of 5 pages in this part need two or more machines for their primary path, marked below. Every one of them has a single-machine path, so the part can be completed with one computer.

0 / 5 lessons in this part completed