RDMA Transport, Tuning and Measuring the Split

The previous lesson got a model running across machines. This one is about the number that comes out of it. By the end you will be able to say which transport a connection is using and why, force it back to plain TCP so that a comparison means something, split a token’s elapsed time into transfer and computation, say where the key-value cache lives and what that costs, attribute a result to one machine in the cluster rather than to the cluster as a whole, and recognise the case where the right answer is a smaller model on one box.

Two transports, chosen without asking you

The RPC backend can carry its traffic over TCP or over remote direct memory access. The README is precise about how the choice is made: the transport is negotiated during the initial handshake, no changes to command-line usage are required, and the connection falls back to TCP unless both peers can use RDMA.

There is no flag to request RDMA. Either both ends were built with a provider library present and the connection was made over a capable link, or you are on TCP. Two providers are supported, each enabled by default when its library is found at build time: on Linux, RoCEv2-capable network cards through libibverbs; on macOS, RDMA over Thunderbolt on Apple silicon Macs with Thunderbolt 5, through librdma.

What RDMA removes from the path

one boundary handover

1. ggml RPC backendSerialises the computation graph and the activation tensor for the next block of layers.same either way
2. Transport negotiationAt handshake: RDMA if both peers can, TCP otherwise. No flag selects it.
3. Kernel network stackTCP path only: system calls, socket buffers, a copy into kernel memory, segmentation, interrupts.skipped by RDMA
4. Network adapterRoCEv2 on a ConnectX-class card, or the Thunderbolt 5 controller on Apple silicon.
5. The cableDirect attach, Thunderbolt, or a switch. Physically identical under both transports.

RDMA does not make the cable faster. It removes the copies and the context switches between the application and the adapter, which is why its benefit shows up in latency, and therefore in decode, more than in bulk transfer.

That distinction matters for expectation-setting. The previous lesson’s arithmetic showed the boundary payload is a few kilobytes per token, which no modern wired link struggles to carry. What a cluster is short of is not capacity but the fixed cost paid at every handover, and that fixed cost is exactly what RDMA attacks.

Getting onto RDMA, per track

Track S — NVIDIA DGX Spark

This is the best-supported RDMA path in the course. Each Spark has a ConnectX-7 with two QSFP ports, and NVIDIA publishes a playbook for wiring two of them together. Its prerequisites are two DGX Spark systems, one QSFP cable for a direct 200 gigabit Ethernet connection, SSH access to both, and the same username on both.

The playbook has you connect the cable to the same physical port on each machine, then run ibdev2netdev to see which RoCE device maps to which network interface and which of them came up. It notes that each physical port presents two logical interfaces, and that full bandwidth is available from a single cable. NVIDIA’s clustering documentation says the same thing from the hardware side: each QSFP port appears as two independent Linux Ethernet interfaces, each paired with its own RoCE device, so its mapping table for one Spark lists four Ethernet interfaces and four RoCE devices across the two ports. It also names the specific cables it has approved. Addresses are then assigned with netplan or by hand, and passwordless SSH is set up in both directions.

That documentation also sets the ceiling on how far this goes: NVIDIA Sync supports up to three Sparks connected directly by cable, and up to four through a switch. Beyond a pair, Part 20 is where the vendor path continues.

Part 18’s lab is where you did all of that. If you have not, do it before this part’s first lab: it is an hour, and the playbook says so itself.

Once the link is up, ibv_devices on each machine should list the RoCE devices. That listing, plus a connection made over the QSFP addresses, is what gets ggml-rpc-server onto RDMA.

Track X — AMD Ryzen AI Max+ 395Partial

A Ryzen AI Max+ machine's built-in networking is 2.5 gigabit Ethernet and USB4, neither of which offers RDMA. The path exists only if you fit a RoCE-capable card, which most of these machines cannot take.

Assume TCP on this track, and be unbothered by it. A layer split’s per-token payload is small enough that a 2.5 gigabit wired link is not the limiting factor; round-trip latency on a short switched path is a fraction of a millisecond, which is small against the time a Ryzen AI Max+ spends computing its slice.

If your machine has a spare slot and you have fitted a RoCE-capable card, the Linux path is the same as Track S: the provider is libibverbs, the check is ibv_devices, and the connection has to be made over that card’s address.

What you should not do is put this machine on Wi-Fi and expect the cluster to behave. That is the fault the challenge page in this part reproduces.

Track M — Apple silicon

Apple silicon Macs with Thunderbolt 5 can do RDMA over Thunderbolt, and llama.cpp uses it through librdma. Apple’s technote TN3205 gives the requirements as a Mac with Apple silicon and Thunderbolt 5, running macOS 26.2 or later. MLX’s distributed documentation, which Part 18 cites and Part 21 builds on, states the same version requirement and the same recovery-mode step, so the two Apple-side paths in this course agree about the prerequisite.

Enabling it is a one-off step that cannot be done from a running system. The technote’s procedure is to reboot into macOS Recovery, open Terminal from the Utilities menu, run rdma_ctl enable, and reboot back into macOS. Afterwards, ibv_devices lists one device per Thunderbolt port, named for the interface it belongs to.

RunnableTrack M · Apple silicon

confirm RDMA is enabled after the Recovery step

ibv_devices

Output — what you should see

    device                 node GUID
    ------              ----------------
    rdma_en2            xxxxxxxxxxxxxxxx
    rdma_en3            xxxxxxxxxxxxxxxx

Two points from the technote shape how you wire a cluster. The connection is point-to-point, so two Macs need one cable, three Macs fully connected need three, and four need six; beyond that the technote suggests a ring to work around the number of ports. And the hardware load-balances between RDMA and IP over Thunderbolt on the same link, so the ordinary network keeps working while RDMA is in use.

Track N — NVIDIA desktop or laptop

A desktop or laptop with ordinary Ethernet is a TCP host, and that is a perfectly good thing to be in this part. RDMA on this track means fitting a RoCE-capable adapter, which is a reasonable thing to do in a tower with a spare slot and not a reasonable thing to do in a laptop.

The Linux procedure is then identical to Track S: libibverbs present at build time, ibv_devices listing the adapter, and the RPC connection made over that adapter’s address. Inside WSL2, treat RDMA as unavailable for the purposes of this course; the course has not verified any path to it and will not describe one it has not checked.

!Important

RDMA is point-to-point, and the README says which local device gets used: the one whose GID matches the address the connection was made on. In plain terms, you must connect over the RDMA-capable link. With Thunderbolt, use the peer’s Thunderbolt address in --rpc; a connection made over any other interface stays on TCP no matter what hardware is installed.

This is the single most common reason a cluster with good hardware performs like a cluster without it, and it is invisible unless you look. Name resolution that returns the Wi-Fi address of a machine you meant to reach over Thunderbolt produces exactly this. Part 18’s naming scheme, with a separate name per link, is the defence.

Forcing TCP, so the comparison means something

“RDMA is faster” is a claim, and this course tests claims. The README gives you the switch: setting GGML_RPC_NO_RDMA on either peer forces plain TCP without rebuilding anything.

RunnableAll tracks

on each host: the same cluster, deliberately on TCP

. ./rpc.env
RPC_NO_RDMA=1 bash start-rpc-server.sh

That variable is this part’s own; start-rpc-server.sh translates it into the GGML_RPC_NO_RDMA the README documents, and prints which transport the run is allowed to use so the choice appears in your terminal rather than only in your memory.

Run your measurement twice, once with the variable set on the hosts and once without, changing nothing else, and you have a comparison that belongs to your hardware rather than to somebody else’s. Set it on the servers rather than the client so that you can be certain which side you changed, and record which way round you ran each measurement, because a table of results without that column is uninterpretable a week later.

Expect the difference to show up in decode rather than in loading, for the reason the stack diagram gives. Expect it to be larger on a link with a longer round trip. And be prepared for it to be small: if your hosts spend most of each token computing rather than waiting, removing some of the waiting changes little, and that is a result worth recording too.

Where a token’s time goes

A single decoded token, on a two-machine split, passes through this sequence.

One decoded token on a two-host layer split

Client

compute layers 0-46send hidden statewaitsample

RPC host

idlereceivecompute layers 47-93return logitsidle

token startsboundarytoken ends

Relative widths, not measured times. The important shape is that only one machine computes at a time: a layer split gives capacity, not concurrency. The two accent blocks are what the transport affects; everything else is the machines' own speed.

Two consequences follow from that picture and they are worth stating plainly.

A layer split does not use both machines at once. While the client computes its layers, the RPC host is idle, and while the host computes, the client waits. For a single request the cluster’s decode rate is bounded by the sum of the machines’ times, not by the faster of them. This is why a model that fits on one machine runs slower when split, and why the challenge page in this part exists.

Concurrency comes from requests, not machines. Serve several requests at once, as Part 9 taught, and the pipeline fills: the host works on one request’s later layers while the client works on another’s earlier ones. Throughput across many requests is the metric where a split cluster looks much better than this diagram suggests, and it is worth measuring both.

Reading the numbers out of the server

llama-server reports its own timings, so you do not need to hold a stopwatch. Every completion response carries a timings object with the prompt and prediction durations and their per-second rates: prompt_ms, predicted_ms, prompt_per_second and predicted_per_second among them. That is prefill and decode separated for you, on the run you just did, rather than a benchmark of a different run.

With --metrics, the server also exposes a Prometheus-compatible endpoint at /metrics, documented as accessible only when that flag is set. Among its series are llamacpp:prompt_tokens_seconds and llamacpp:predicted_tokens_seconds as gauges of average throughput, and llamacpp:n_decode_total as a count of decode calls. /slots reports per-slot state, which is how you see what a concurrent load is really doing.

The lab’s measure-split.py reads the timings object from one completion, samples this machine’s interface byte counters either side of it, and writes both into the notebook on one line. The counters are the part people leave out, and they are the part that turns “the cluster felt slow” into “the cluster moved this many bytes to produce this many tokens”.

Where the key-value cache lives

llama.cpp distributes the key-value cache along with the weights, across all available devices in proportion to available memory. That is the sentence in the README, and its practical meaning is good news: each machine holds the cache for the layers it computes, so attention reads its keys and values from local memory. No cache entries cross the cable during generation. The boundary payload stays one hidden state per token however long the context grows.

The consequences are about capacity rather than traffic.

• Context length is a cluster-wide budget. The cache is divided in the same proportions as the weights, so a host holding a fifth of the layers holds a fifth of the cache. Work out the total from Part 4’s arithmetic and then divide it the way you divided the model.
• Quantising the cache helps every host at once. The --cache-type-k and --cache-type-v options from Part 6 apply exactly as they did on one machine.
• A host that is tight on memory gets tighter as the context grows. A split that fitted at a short context can fail to allocate at a long one, and the failure appears at load time because the cache is allocated up front.

Attributing a result to one machine

When a cluster is disappointing, the useful question is which machine made it so. Three measurements, in order, answer it without guesswork.

Run each host alone. Start a benchmark on each machine on a model that fits it, using the methodology from Part 6’s lab. You now know each machine’s own decode rate, which is the input to a sensible split.

Count the layers. The load log names how many layers went to each device. Divide each host’s layer count by its own rate and you have an estimate of the time it contributes per token; the sum of those estimates should be close to the cluster’s observed time per token. A host whose contribution dominates the sum is your answer.

Watch the counters on each host, not just the client. The client’s interface sees all the traffic; a single host’s interface sees only its own. Comparing them tells you whether one link is carrying far more than its share, which usually means the split does not match the topology you think you have.

Pending validationPer-host contribution — the recording sheet for this part's labs

Host and track	Layers held	Own decode rate, tokens/s	Estimated share of time per token, ms	Transport
client, track ?	—	—	—	local
host B, track ?	—	—	—	—
host C, track ?	—	—	—	—

your cluster: one row per machine, with its track and memory · llama.cpp RPC, layer split the build number from llama-cli --version · the model you split, as run · 8,192 tokens of context · the date you ran it

Empty on purpose: the shape to fill in from your own cluster. The fourth column is arithmetic from the second and third, not a measurement, and the point of writing it down is to compare it against the cluster's measured time per token. A large gap between the sum of the estimates and the measurement is the network, and the byte counters will say so.

When a smaller model on one box wins

This is the most valuable judgement in the part, and it goes against the grain of building something that works.

A cluster is worth it when the model you need does not fit on any single machine you own. That is the case the first lab demonstrates: a 200B-class model at four bits does not fit in 128 GB, and no amount of tuning on one machine changes that.

A cluster is not worth it when a model that fits is split anyway. You pay a boundary crossing per token, you lose the ability to use both machines on one request at once, you add two more programs that can fail, and you gain nothing. On any machine in this course, a mixture-of-experts model in the 30B or 120B class is very likely to be the better answer than a 200B-class model split across the house, because it reads a fraction of itself per token and it runs on one box.

The honest comparison is easy to run and you should run it before you commit to a cluster. Benchmark the largest model that fits on your best single machine, then benchmark the split cluster, and put the two rates next to each other along with what each one cost you in complexity. Part 16’s evaluation methods are the way to check whether the larger model is actually better at your task, which is the other half of the decision and the half people skip.

?Reality check

The claim under examination is “more machines means more speed”. For a single request on a layer split, it is simply false: the machines take turns. What more machines buy is memory, and therefore access to a class of model that was previously out of reach, plus more throughput when several requests are in flight. Measure both before deciding which one you actually wanted.

Prove which transport carried the useful work

Record the configured transport and the observed transport separately. A system may fall back to TCP after an RDMA setup problem, and a successful model response alone does not identify the route. Collect engine logs, interface counters and a standalone transport test on the same interfaces.

Make an A/B comparison with the same checkpoint, split, context and request lengths. Restart only the lab’s relevant processes between configurations and document which caches were retained. A change in model placement or cold-start state would confound the transport comparison.

If RDMA produces little benefit, examine the fraction of time spent communicating. When computation or local memory traffic dominates, removing part of network overhead has a bounded effect. If the workload performs many tiny synchronisations, latency may matter more than bulk bandwidth. Use the measurement to decide whether to tune transport, alter placement or return to one machine. The existence of an RDMA-capable device is an opportunity to test, not a reason to attribute every result to RDMA.

Recap

The transport is negotiated at the handshake and falls back to TCP unless both peers can do RDMA; there is no flag to ask for it, only GGML_RPC_NO_RDMA to refuse it, which is how you run an honest comparison. RDMA is available over RoCE on Linux with a capable card, and over Thunderbolt 5 on Apple silicon running macOS 26.2 or later after a one-off rdma_ctl enable from Recovery. It is point-to-point, so the connection must be made over the capable link or it silently stays on TCP. A layer split runs one machine at a time per request, so its win is capacity and multi-request throughput rather than single-request speed. The key-value cache is distributed with the weights, so attention stays local and only one hidden state per token crosses a boundary. Attribute results by benchmarking each host alone, counting its layers, and reading its own interface counters. And when the model fits on one machine, run it on one machine.

Check your understanding

Question 1. Your two Macs have Thunderbolt 5, macOS 26.2, and rdma_ctl enable has been run on both. The cluster still behaves as though it is on TCP. What is the most likely cause?

• RDMA needs a command-line flag that was not passed
• The --rpc list names an address on a different interface, such as Wi-Fi or Ethernet, so the connection was not made over the Thunderbolt link
• macOS supports RDMA only between three or more machines
• The model is too small for RDMA to be negotiated

Show the answer and why

Answer: The --rpc list names an address on a different interface, such as Wi-Fi or Ethernet, so the connection was not made over the Thunderbolt link

The README says the transport is negotiated at the handshake and that each side uses the local device whose GID matches the address the connection was made on. Connect over the Thunderbolt address. A name that resolves to the wrong interface produces exactly this symptom, which is why Part 18 gives each link its own name.

Question 2. On a single request split across two machines by layers, what happens while the RPC host computes its block of layers?

• The client computes the next token in parallel
• The client waits, because the boundary handover is synchronous and the token is not finished
• Both machines compute the same layers for redundancy
• The client streams the key-value cache to the host

Show the answer and why

Answer: The client waits, because the boundary handover is synchronous and the token is not finished

A layer split is sequential for one request: each machine takes its turn. That is why the cluster is slower than one machine for a model that fits on one machine, and why concurrency has to come from serving several requests at once rather than from adding machines.

Question 3. Where does the key-value cache live in an RPC split, and what does that cost per token in network traffic?

• On the client, and each host fetches the entries it needs every token
• Distributed across the devices in proportion to memory, with each host holding the cache for its own layers, so no cache entries cross the link during generation
• Duplicated on every host, so it costs nothing to move but wastes memory
• In shared storage, read over the network on every attention operation

Show the answer and why

Answer: Distributed across the devices in proportion to memory, with each host holding the cache for its own layers, so no cache entries cross the link during generation

The README states that weights and the KV cache are both distributed across all available devices in proportion to available memory. Attention therefore reads local memory, and the per-token boundary payload stays one hidden state regardless of context length. The cost of a long context is memory on each host, and it is paid at load time.

Question 4. You want to prove whether RDMA is helping on your cluster. Which procedure gives a comparison you can defend?

• Run once with RDMA, and compare against a figure from someone else with similar hardware
• Run the same measurement twice, once with GGML_RPC_NO_RDMA set on the hosts and once without, changing nothing else, and record which run was which
• Compare the model load times, since loading is where the transport matters most
• Compare a run over Thunderbolt against a run over Wi-Fi

Show the answer and why

Answer: Run the same measurement twice, once with GGML_RPC_NO_RDMA set on the hosts and once without, changing nothing else, and record which run was which

One variable at a time, on your own hardware. Comparing against someone else changes hardware, build, model and methodology at once. Comparing Thunderbolt against Wi-Fi changes the link as well as the transport, so it cannot tell you what RDMA contributed.

Check my answersClear