llama.cpp RPC: Layers Across Machines

By the end of this lesson you will be able to describe, in the right order, what happens when a model is split across machines with llama.cpp’s RPC backend: which program runs where, what each one knows, how the layers get divided, what crosses the cable and how often. You will be able to predict from arithmetic alone roughly how much traffic a split generates, which is what lets you decide whether the network you have is good enough before you spend an evening finding out that it is not.

One program per machine, and one flag on the client

The RPC backend is smaller than its reputation. There are exactly two moving parts.

On each machine you want to contribute, you run ggml-rpc-server. It starts, asks the ggml library what accelerators this build can see, and offers them over a TCP socket. Language models, tokens and GGUF files play no part in it. It is a device, reachable over a network, and that is the whole of its job. The README is explicit about the default: the server exposes every available accelerator device on the host, and if there are none, it exposes a single CPU device.

On the machine you sit at, you run the client you already know from Part 6, llama-cli or llama-server, with one extra option: --rpc, whose help text in the argument definitions reads “comma separated list of RPC servers to use in addition to standard devices”. That phrase is the mental model. The remote machines are not a cluster in the client’s eyes. They are extra devices in a list that already contains your own GPU.

Two hosts: the client's own device, plus one remote

• clientClientllama-server, holds the GGUF, layers 0 to 46
• workerRPC host Bggml-rpc-server, layers 47 to 93

• Client ⟷ connected to RPC host BTCP or RDMA, one hidden state per token

The client holds the model file and the tokeniser and decides everything. The RPC host holds a slice of the weights and computes when asked.

Three hosts, one of them smaller than the others

• clientClientllama-server, layers 0 to 40
• workerRPC host Bggml-rpc-server, layers 41 to 81
• workerRPC host Cggml-rpc-server, layers 82 to 93, smaller share

• Client ⟷ connected to RPC host Bboundary 1
• Client ⟷ connected to RPC host Cboundary 2

Adding a machine adds a boundary. The proportions are yours to choose; the default is each device's share of the cluster's free memory.

Note what the diagram does not show: a link between host B and host C. Every remote device is talked to by the client. The hosts do not know about each other, and adding a third machine adds another conversation for the client to hold, not another hop in a chain.

Building it, which is one flag

Both sides need the RPC backend compiled in. The README gives the recipe: add -DGGML_RPC=ON to the CMake options you already used in Part 6, alongside your platform’s backend flag.

RunnableTrack S · DGX Spark

the Part 6 CUDA build, with RPC added

cmake -S ~/llama.cpp -B ~/llama.cpp/build \
    -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(nproc)"

Substitute your track’s flag: -DGGML_VULKAN=1 on a Ryzen AI Max+, nothing at all on macOS where Metal is the default, -DGGML_CUDA=ON on an NVIDIA desktop. The README’s own example builds into a separate directory named build-rpc-cuda, which is worth copying if you want to keep a non-RPC build for comparison; otherwise adding the flag to your existing build directory is fine and gives you one set of binaries.

⌚Version note

The build guide, read on 2026-09-09, documents every backend flag in detail and does not mention GGML_RPC at all. The RPC option is documented only in the RPC tool’s own README. That is worth knowing before you go looking for it in the wrong file, and it is a fair signal of how the project regards this feature’s maturity. This course pins llama.cpp v0.4.0 · verified 2026-09-08.

The binary the build produces is ggml-rpc-server. Older material, and the course’s own command reference at the time of writing, calls it rpc-server; if a command you copied from elsewhere is not found, that rename is the first thing to check.

Starting a server, and reading what it says

The server takes six options and no configuration file. From its own usage text: -h, -t, --threads N, -d, --device <dev1,dev2,...>, -H, --host HOST, -p, --port PORT and -c, --cache. The default host is 127.0.0.1 and the default port is 50052.

Fragment — not complete on its own

ggml-rpc-server -H 10.x.x.x -p 50052 -c

The startup banner tells you three things worth reading every time.

Output — what you should see

Starting RPC server v3.0.0
  endpoint       : 127.0.0.1:50052
  local cache    : n/a
Devices:
  CUDA0: NVIDIA GeForce RTX 5090 (32109 MiB, 31588 MiB free)

The endpoint line is the address it is actually bound to, which is not always the address you meant. The cache line says whether -c took effect. The device list is what this host is offering, with its free memory, and that free-memory figure is what the client will use to decide this host’s share of the model unless you say otherwise.

To offer less than everything, name the devices: the README shows ggml-rpc-server --device CUDA0 -p 50052 as having the same effect as setting CUDA_VISIBLE_DEVICES=0 first. There is no option to cap the memory a host will offer. If you want a host to take a smaller slice of the model, that decision is made on the client, with --tensor-split, and the next section is about how.

How the layers get divided

With no instruction, llama.cpp distributes the model weights and the key-value cache across all available devices, local and remote alike, in proportion to each device’s available memory. Two 128 GB machines with nothing else running take roughly half each. A 128 GB machine and a 32 GB card take roughly four fifths and one fifth.

That default is right surprisingly often and wrong in two specific cases: when a device reports free memory it cannot really give a model, which is the normal situation on an integrated GPU with a cap on its share of system memory, and when the devices differ in speed rather than in size, which is the subject of this part’s second lab.

The override is --tensor-split, whose help text reads “how to split tensors across multiple devices, comma-separated list of proportions, e.g. 3,1”. The proportions are relative, so 3,1 and 0.75,0.25 mean the same thing.

Fragment — not complete on its own

llama-server -m model.gguf --rpc node-b.home.arpa:50052,node-c.home.arpa:50052 \
    -ngl 999 -c 8192 -ts 1,1,0.5

Three numbers for three devices: the local one first, then the RPC hosts in the order they appear in --rpc. Getting that order wrong is the most common mistake in this part, because nothing complains. A backwards split loads, runs and quietly gives your fastest machine the smallest slice.

✕Common mistake

The name --tensor-split describes what it does to the tensors, not the kind of parallelism you get. With the default split mode, layers are assigned to devices in contiguous blocks, so this is a pipeline-style layer split in the vocabulary of Part 18, not the tensor parallelism of Part 20. That distinction is the reason a slow link is survivable here and fatal there.

llama-bench accepts the same two ideas with slightly different spelling: its own usage text gives -rpc, --rpc <rpc_servers> and -ts, --tensor-split <ts0/ts1/..>, separating the proportions with slashes rather than commas. The lab’s run-split.sh converts between the two forms so that one variable in your environment file drives both.

Backends that can share a run

Because each server offers whatever devices its own build found, the cluster does not have to be uniform. The README’s own diagram shows exactly this: a host with two CUDA devices, a host with Metal, and a host offering both CUDA and CPU, all serving one client. A DGX Spark on CUDA, a Ryzen AI Max+ on Vulkan and a Mac on Metal can hold three slices of the same model at the same time.

What has to match is the model, not the hardware: one GGUF file, opened by the client, whose layers are then shipped out. Nothing requires the hosts to run the same operating system, the same processor architecture or even the same backend, and this is the capability that makes this part rather than Part 20 or Part 21 the right first cluster for most readers.

What does have to match is the build. Both ends need GGML_RPC compiled in, and a client whose build lacks it will reject --rpc outright rather than fall back to anything.

The local cache, which changes how loading feels

Every tensor a remote device holds has to get there somehow, and the first time it does, it goes over the cable. For a 125 GB model on a domestic link, that is the difference between a coffee and an afternoon.

The -c option turns on a local file cache, stored by default under $HOME/.cache/llama.cpp/rpc and relocatable with the LLAMA_CACHE environment variable. The README describes it as storing large tensors so they need not be transferred over the network, and says it can speed up model loading significantly, especially for large models. In practice it means the first load of a given model is slow and every later load of the same model is not, which matters a great deal when you are iterating on a split.

Turn it on everywhere. The only cost is disk on the host, and the host is holding a slice of a large model anyway.

The security warning, and what it means in a house

The README’s first block, before any usage, is an importance callout:

This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!

Take it literally. There is no authentication option in the server’s usage text, no token, no transport encryption and no access control of any kind. Anything that can reach the port can ask the machine to allocate memory and execute computation graphs.

⛨Security

Three rules for running this at home, and none of them is optional.

Bind to the cluster link, never to everything. The default host is 127.0.0.1, which reaches nothing; the temptation is to change it to 0.0.0.0, which reaches everything including your guest Wi-Fi. Bind to the address of the interface Part 18 had you measure, and nothing else. This part’s start-rpc-server.sh refuses a wildcard address for that reason.

Never expose the port to the internet. No port forwarding, no reverse proxy, no tunnel “just to try it from the café”. If you want remote access to your cluster, expose the llama-server API from Part 9 with authentication in front of it, and leave the RPC ports where they are.

Run the servers only while you need them. They are not a service. Start them for a session, stop them afterwards, and do not add them to your system’s startup.

What crosses the cable, as arithmetic

This is the part that decides whether your network is adequate, and you can do it on paper.

In a layer split, a machine computes a contiguous block of layers and hands the result to whoever holds the next block. What it hands over is one hidden state per token: a vector as wide as the model’s hidden size, in the activation precision. Qwen3-235B-A22B’s published configuration gives a hidden size of 4,096, so at 16 bits per element the boundary payload is 4,096 multiplied by 2, or 8 KiB, per token, per boundary.

During decode, one token is produced at a time, so that 8 KiB is all the model data that needs to cross per boundary per token. During prefill the tokens arrive in batches, so a 512-token batch carries 512 times as much, about 4 MiB per boundary, once.

Pending validationBoundary traffic per token from the model's own dimensions — arithmetic, not a measurement

Model	Hidden size	Layers	Bytes per boundary, decode	Bytes per boundary, 512-token prefill batch
Qwen3-235B-A22B	4,096	94	8 KiB	4 MiB
Qwen3-30B-A3B	2,048	48	4 KiB	2 MiB

none: this table is arithmetic from published model configurations · llama.cpp RPC, layer split v0.4.0 · as listed per row, independent of quantisation; activations are 16-bit · 512 tokens of context · 2026-09-09

Hidden sizes are from each model's config.json on Hugging Face. This is the model payload only: the RPC protocol also describes the computation graph on every step, so the measured traffic in this part's first lab will be larger. Treat these figures as the floor that tells you whether the link is plausibly adequate, and the lab's measurement as the truth.

Hold that against the link classes Part 18 catalogued. Even 2.5 gigabit Ethernet, the slowest wired link in that lesson, carries hundreds of megabytes each second. A few kilobytes per token is not a demand on it. That is the whole argument for why a layer split tolerates a slow link, and it is why this technique works between a mini PC and a laptop over an ordinary switch when the tensor parallelism of Part 20 would not.

There are two important qualifications, and they are where the surprises live.

Latency, not bandwidth, is what you feel. The boundary crossing is a synchronous handover: the next machine cannot start until the bytes arrive. Every token pays the round trip of every boundary. A link with plenty of capacity and a round trip in the tens of milliseconds, which describes Wi-Fi rather well, adds that delay to every single token. This is why the challenge page in this part starts by asking which interface the traffic used.

Loading is a different problem from running. Getting 125 GB of weights onto the hosts is a bulk transfer and it is entirely bandwidth-bound, which is what the local cache exists to solve. Do not judge a link by how long the first load took.

Verify placement rather than inferring it from two running processes

An RPC worker listening on another machine is only one prerequisite. The client must discover the intended device, allocate work there and use the planned split. Preserve the client’s device list and placement log alongside the worker log. A running remote process with idle hardware is not evidence of distributed inference.

Use a small checkpoint for the first end-to-end test, then the capacity target. Check the model path on the process that loads the file; do not assume every worker needs the same local file arrangement unless the selected path says so. Keep RPC traffic on the lab’s isolated network as required by the RPC instructions.

For measurement, separate loading, first request and warm generation. Cache effects can make a later remote load appear much faster without changing per-token communication. Stop one lab worker during a disposable request and record the actual failure and recovery procedure. The service should not be described as fault tolerant merely because it spans machines; splitting a single model can introduce additional dependencies for every request.

Recap

ggml-rpc-server offers one machine’s devices over a socket and knows nothing about models; the client adds them to its device list with --rpc and splits the model’s layers across everything it can see. The default division is proportional to each device’s free memory; --tensor-split overrides it with relative proportions, in the order local devices first and then the RPC hosts as listed. Both ends need -DGGML_RPC=ON, which the build guide does not document. Backends may be mixed freely, which is this tool’s unique advantage. The -c cache makes the second load of a model fast. The server has no authentication whatsoever and its own README says never to run it on an open network, so bind it to the cluster link and stop it when you are done. The per-token boundary payload is one hidden state, a few kilobytes, which any wired link carries easily; what actually hurts is round-trip latency, paid once per boundary per token.

Check your understanding

Question 1. You run the client with --rpc hostb:50052,hostc:50052 on a machine that has its own GPU. How many proportions does --tensor-split expect, and in what order?

• Two, for the two RPC hosts only
• Three: the local device first, then hostb, then hostc
• Three, but in reverse order, because the remote devices are registered first
• It depends on the number of layers in the model

Show the answer and why

Answer: Three: the local device first, then hostb, then hostc

The RPC hosts are added to the device list the client already has, so local devices come first and the remote ones follow in the order given to --rpc. Nothing warns you about a split in the wrong order: it loads and runs, with the shares on the wrong machines. Confirm the order with a device listing before writing a split.

Question 2. Why does a layer split across machines tolerate a much slower link than tensor parallelism does?

• Because it compresses the weights before sending them
• Because only one hidden state per token crosses each boundary, rather than a collective operation on every layer
• Because the RPC backend uses UDP instead of TCP
• Because each machine holds a complete copy of the model

Show the answer and why

Answer: Because only one hidden state per token crosses each boundary, rather than a collective operation on every layer

A layer split hands over one activation vector at each boundary, which is a few kilobytes per token; tensor parallelism performs a reduction across devices inside every layer. That difference of several orders of magnitude in traffic is why one crosses a house and the other stays inside a box.

Question 3. Which of these are true of ggml-rpc-server as its documentation describes it? Select all that apply.

• It has no authentication, and the README says never to run it on an open network
• It exposes every accelerator it finds by default, or a single CPU device if there are none
• It has an option to limit how much memory it will offer to a model
• Its local file cache avoids re-sending large tensors over the network on later loads

Show the answer and why

Answer: It has no authentication, and the README says never to run it on an open network, It exposes every accelerator it finds by default, or a single CPU device if there are none, Its local file cache avoids re-sending large tensors over the network on later loads

The usage text lists only help, threads, device, host, port and cache. There is no memory-limit option: to give a host a smaller share of the model you set the proportions on the client with --tensor-split. The other three statements come straight from the README.

Question 4. Your two machines are joined by a link with ample capacity but a round trip in the tens of milliseconds. What should you expect?

• Loading will be slow, but generation will be unaffected
• Generation will be badly affected, because every token pays the round trip at every boundary
• Nothing: llama.cpp batches the boundary crossings to hide latency
• The model will fail to load

Show the answer and why

Answer: Generation will be badly affected, because every token pays the round trip at every boundary

The handover at a layer boundary is synchronous, so the delay is added to every token rather than amortised. This is the shape of the Wi-Fi fault that this part's challenge page has you diagnose, and it is why the first thing to check on a slow cluster is which interface carried the traffic.

Check my answersClear