Level 4 · Cluster ArchitectLabPart 19 · page 3 of 575 minSXMN 128 GB Two machines

75Minutes

4Tools

8Sources

All fourTracks

The primary path on this page needs two or more machines. Every cluster page carries a single-machine path — look for the callout below.

Tools used on this page4

Lab: Run a Model Bigger Than Any One Machine

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The build tag, the transport each link negotiated, and the versions each track was run with will be recorded here when the validation pass has run this lab on the reference cluster.

Objective

By the end of this lab you will have run a model that does not fit on any machine you own. Concretely: Qwen3-235B-A22B, quantised to IQ4_XS, whose three GGUF files come to roughly 125 GB and therefore fit in no single 128 GB machine once a key-value cache is added, generating text across two or three of your boxes.

You will also have four things that matter more than the demonstration.

The fit arithmetic done in advance, from the repository’s real file sizes, so you knew it would fit before you spent an evening downloading it.
A split you chose on purpose, with the device order confirmed rather than assumed.
A measurement of prefill, decode and the number of bytes that actually crossed the cluster link to produce them, held against the arithmetic from this part’s first lesson.
Where the hardware allows it, the same measurement on TCP and on RDMA, with one variable changed between them.

Architecture

The reference cluster’s primary path for this lab is two DGX Sparks joined by the QSFP link from Part 18, with the Ryzen AI Max+ box added as a third host when you want the extra memory headroom rather than the speed.

Reference cluster path: two Sparks, optionally a third host

clientDGX Spark ATrack S, 128 GB, CUDA. Runs llama-server and holds the model file.
workerDGX Spark BTrack S, 128 GB, CUDA. Runs ggml-rpc-server.
workerRyzen AI Max+ 395 boxTrack X, 128 GB, Vulkan. Optional third host, smaller share.
storagemodel storageThe shared model directory from Part 18, or a local copy on the client.

DGX Spark A connected to DGX Spark BQSFP direct link, RoCE where both ends negotiate it
DGX Spark A connected to Ryzen AI Max+ 395 boxwired Ethernet, TCP
DGX Spark A connected to model storagethe client is the only machine that reads the GGUF

The client is one of the Sparks. It holds the GGUF file, opens it, and ships each remote device its slice. The third host is optional and takes a deliberately smaller share.

Only the client opens the model file. The RPC hosts receive tensors over the link on the first load and keep them in their local cache afterwards, so shared storage is a convenience for the client rather than a requirement for the cluster.

The commands below address the hosts by the example names Part 18’s lab uses in its hosts file, node-b.home.arpa for the house network and node-b-direct.home.arpa for the direct cable. Substitute the names you gave your own machines. Which of the two you type is not a detail: it decides which cable carries the model traffic, and this part’s challenge page is built on somebody getting it wrong.

Does it fit? The arithmetic, before the download

The repository publishes the IQ4_XS quantisation as three files, which the file list gives as roughly 50, 50 and 26 GB, about 125 GB in total. That is the same weight budget the course model reference records as 118 GiB; the difference is only decimal gigabytes against binary ones, and it is worth noticing once so that a loader reporting a smaller number than your download did not alarm you.

Qwen3-235B-A22B has 94 layers, 4 key-value heads and a head dimension of 128, which by Part 4’s formula is 188 KiB of cache per token at 16 bits. At the 8,192-token context this lab uses, that is under 2 GB across the whole cluster.

Qwen3-235B-A22B at IQ4_XS on one 128 GB machine — the reason this lab needs two

Weights, IQ4_XS: 125.5 GB
KV cache, 8k context: 1.6 GB
Compute buffers and overhead: 2 GB
Requested: 129.1 GB
Machine budget: 128 GB

Over budget. 129.1 GB requested against a 128 GB machine - 1.1 GB over. Something here has to shrink: a smaller quantisation, a shorter context, or fewer of these reservations at once.

Over budget on a machine that exposes all 128 GB to the accelerator, and further over on one that does not. No quantisation of the cache and no shortening of the context rescues this: the weights alone are the problem.

The same model split evenly across two 128 GB machines — one machine's share

Weights, half the layers: 62.8 GB
KV cache for those layers: 0.8 GB
Compute buffers and overhead: 2 GB
Free: 62.4 GB
Total: 128 GB

Comfortable, with room for a much longer context if you want it. Two machines is not a tight fit for this model; three is generous.

Requirements

Every track needs: a llama.cpp build with -DGGML_RPC=ON on every machine in the cluster, the network from Part 18’s lab already built and measured, the hf tool from Part 4, and the lab notebook.

Time. About seventy-five minutes attended, once the model is on disk. The download is unattended and dominates the wall clock: roughly 125 GB on a domestic connection is an overnight job, and the first load across the cluster is a further bulk transfer of the same order. Start the download before you read the rest of this page.

Path	Machines	Model	Approximate download
Reference cluster	Two Sparks, optionally plus the Strix Halo box	Qwen3-235B-A22B IQ4_XS	125 GB, three files
Any two tracks	Two machines of at least 128 GB each, wired	Qwen3-235B-A22B IQ4_XS	125 GB, three files
Three smaller machines	Three machines of at least 64 GB each	Qwen3-235B-A22B IQ4_XS	125 GB, three files
Single machine	One machine of at least 16 GB	Qwen3-30B-A3B Q4_K_M	18.6 GB, one file

Both models are Apache-2.0 licensed and neither is gated, so no acceptance step or token is needed. The model reference records the licence for each.

Track S — NVIDIA DGX Spark

The primary path. Two Sparks give 256 GB between them, which holds this model with a great deal to spare, and the QSFP link from the NVIDIA playbook is the fastest link in the course. Build both machines with -DGGML_CUDA=ON -DGGML_RPC=ON.

Use the QSFP addresses in RPC_HOSTS, not the 10 gigabit Ethernet ones. Both work; only one of them can negotiate RDMA, and mixing them up is the fault this part’s challenge page is about. Part 18’s naming scheme gives each link its own name for exactly this reason.

With a third machine you can raise the context length considerably rather than run a bigger model, since 235B at four bits is the largest reference model the course carries.

Track X — AMD Ryzen AI Max+ 395Partial

A 128 GB Ryzen AI Max+ cannot expose all of its memory to the GPU, so it takes a deliberately smaller share of the split than its total memory suggests, and its 2.5 gigabit link is the slowest in the reference cluster.

Two of these machines have 256 GB between them on paper, but the GPU-visible share is capped below the total: about 96 GB on Windows, and on Linux by the graphics translation table limit, which has to be raised deliberately. Part 5’s lesson on this machine covers where that limit lives.

Two consequences for this lab. First, work out each machine’s usable figure and give that to plan-tensor-split.py, not the number on the box. Second, expect to set --tensor-split by hand rather than letting the default proportional split decide, because the free-memory figure a capped device reports is not the amount it can give a model.

Build with -DGGML_VULKAN=1 -DGGML_RPC=ON. As a third host beside two Sparks this machine works well with a small share; as the client of a two-Strix cluster, budget carefully.

Track M — Apple silicon

A 128 GB Mac Studio pairs with any other 128 GB machine, and two of them are an excellent cluster: Metal is enabled by default so the build needs only -DGGML_RPC=ON, and a Thunderbolt 5 link between two Apple silicon machines is the one consumer path to RDMA in this course.

A 256 GB or 512 GB Mac Studio can hold this model alone, which makes it the ideal machine on which to run this lab’s most instructive comparison: the same model split across two boxes, and on one. Do both, and record them side by side.

On a Mac with less than 128 GB, join a cluster as a second host with a smaller share, or take the single-machine path with the 30B-class model.

Track N — NVIDIA desktop or laptopPartial

A desktop card's VRAM is a hard ceiling and no consumer card reaches 128 GB, so this track contributes to a cluster rather than forming the primary path on its own.

A desktop with a 24 or 32 GB card is a useful host in a cluster whose other machines have unified memory: give it a proportion matching its VRAM and it holds a real slice of the model. Two such desktops alone cannot hold this model, and asking them to will spill weights into system memory across PCIe, which is slower than not clustering at all.

Build with -DGGML_CUDA=ON -DGGML_RPC=ON. If your desktop has 128 GB of system memory and no large card, the honest path is the single-machine route with the 30B-class model, and the Part 18 topology worksheet is where you record why.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-19-llama-cpp-rpc"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-19-llama-cpp-rpc and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Confirm the network before you trust it

Part 18’s lab left you with names, keys, a throughput and round-trip figure per link from its measure-links.sh, and a verified maximum transmission unit. Check the two things this lab depends on most, on the client:

RunnableAll tracks

getent hosts node-b-direct.home.arpa
ping -c 4 node-b-direct.home.arpa

On macOS, dscacheutil -q host -a name node-b-direct.home.arpa does the first job. What you are looking for is that the name resolves to the address of the cluster link and not to a Wi-Fi or general-purpose Ethernet address. If it resolves to the wrong one, fix it now; every number you produce later will otherwise be a measurement of the wrong cable.

2. Build every machine with RPC compiled in

On each machine in the cluster, including the client, add the RPC flag to the Part 6 build.

RunnableTrack S · DGX Spark

cmake -S ~/llama.cpp -B ~/llama.cpp/build \
    -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(nproc)"
ls ~/llama.cpp/build/bin/ggml-rpc-server

Substitute your track’s backend flag. The ls at the end is the check that matters: if that binary is missing, the flag did not take, and everything after this step will fail in a confusing way.

3. Fetch the model

Download to the client. The three IQ4_XS files live in their own directory in the repository.

RunnableAll tracks

hf download unsloth/Qwen3-235B-A22B-GGUF --include "IQ4_XS/*" \
    --local-dir ~/models/unsloth/Qwen3-235B-A22B-GGUF

RunnableAll tracks

hf download unsloth/Qwen3-30B-A3B-GGUF --include "*Q4_K_M*" \
    --local-dir ~/models/unsloth/Qwen3-30B-A3B-GGUF

Point every later command at the first shard, Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf. llama.cpp opens the rest itself; naming a later shard produces a confusing failure about a missing tensor.

4. Fill in the environment file

Every script in this part reads one environment file. Copy the template, edit it on each machine, and source it before running anything.

RunnableAll tracks

# Purpose: the environment every script in Part 19 reads. Copy this file to rpc.env,
#          fill in the values for your own machines, and source it before running anything.
#          Nothing in this file is committed to the course: the names below are placeholders
#          and the real ones live only on your machines.
# Platform: all
# Minimum memory: 8 GB
# Assumes: the names, addresses and shared storage set up in Part 18's cluster-network lab
#
# Usage: cp env-example.txt rpc.env && "${EDITOR:-nano}" rpc.env && . ./rpc.env
#
# Part 18's own .env already holds CLUSTER_IFACE, CLUSTER_PEERS, MODELS_DIR, SSH_USER and
# LABBOOK. The names here are deliberately identical, so copy your values across rather than
# inventing new ones; if you would rather keep one file, source Part 18's .env first and this
# one after it, and delete the duplicated lines below. SSH_USER is the account you use to
# reach each host to start its server; nothing in this part logs in on your behalf.

# --- Every host in the cluster ------------------------------------------------------
# The interface that carries cluster traffic on THIS machine, as the operating system
# names it. Part 18's lab is where you found and measured it. This is the single most
# important line in the file: bind to the wrong interface and the challenge page in this
# part is about you.
#   Linux   ip -br addr           e.g. enp1s0f1np1, eno1, enp5s0
#   macOS   networksetup -listallhardwareports   e.g. en0, en5, bridge0
CLUSTER_IFACE=CHANGE-ME

# Optional. Set this only when the interface has more than one address and you want a
# particular one. Leave it empty and the scripts read the address off CLUSTER_IFACE.
CLUSTER_ADDR=

# The port each ggml-rpc-server listens on. 50052 is the tool's own default.
RPC_PORT=50052

# Optional. Restrict which devices this host exposes over RPC, as ggml-rpc-server names
# them: CUDA0, Vulkan0, Metal0, CPU. Empty means "expose every accelerator you find".
RPC_DEVICE=

# Optional. CPU threads for the CPU device on this host. Empty uses the tool's default,
# which is half of the reported hardware concurrency.
RPC_THREADS=

# Where llama.cpp's binaries live on this machine.
LLAMA_BIN=$HOME/llama.cpp/build/bin

# --- The client host only -----------------------------------------------------------
# Every remote rpc-server, as host:port, comma separated, IN THE ORDER YOU WANT THE LAYERS
# ASSIGNED, because that order is what --tensor-split addresses. Use the names from Part 18,
# not addresses: a name you can read is a name you can debug, and it keeps addresses out of
# your notes. Prefer the "-direct" name of a machine that has one, because the name you type
# here decides which cable the model traffic uses.
#
# Leave this empty and the scripts build it from Part 18's CLUSTER_PEERS, appending RPC_PORT
# to each peer in the order that variable lists them. Set it explicitly whenever the order
# matters, when only some peers take part, or when a peer listens on another port.
#   RPC_HOSTS=node-b-direct.home.arpa:50052,node-c.home.arpa:50052
RPC_HOSTS=

# Only read when RPC_HOSTS is empty. This is Part 18's variable; if you sourced Part 18's
# .env first, leave it alone here.
CLUSTER_PEERS=

# The model. Point at the FIRST shard of a split GGUF; llama.cpp opens the rest itself.
MODELS_DIR=$HOME/models
MODEL=$MODELS_DIR/unsloth/Qwen3-235B-A22B-GGUF/IQ4_XS/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf

# Proportions for --tensor-split, comma separated, in device order: the local device
# first, then the RPC hosts in the order they appear in RPC_HOSTS. Leave it empty to let
# llama.cpp split in proportion to each device's free memory.
TENSOR_SPLIT=

# Context length to allocate, and how many layers to offload. 999 means "all of them".
CTX=8192
NGL=999

# Where the notebook lines go.
LABBOOK=labbook.md

Download env-example.txt75 lines

RunnableAll tracks

cp env-example.txt rpc.env
"${EDITOR:-nano}" rpc.env
. ./rpc.env

The variable names overlap with Part 18’s .env on purpose: CLUSTER_IFACE, CLUSTER_PEERS, MODELS_DIR, SSH_USER and LABBOOK mean the same things they meant there, so copy your values across rather than inventing new ones. If you leave RPC_HOSTS empty, the scripts build it from CLUSTER_PEERS by appending the port to each peer in the order that variable lists them; set it yourself as soon as the order matters, because the order is what --tensor-split addresses.

The one line to get right is CLUSTER_IFACE. It is the interface Part 18 had you measure, and it is what the servers bind to and what the measurement script reads counters from. Everything else in the file has a working default.

5. Do the arithmetic before you run anything

Give the planner each device’s usable memory, not its installed memory, and the model’s real size and layer count.

RunnableAll tracks

#!/usr/bin/env python3
"""Work out a --tensor-split before you run one, and say what each host would hold.

Purpose: turn a list of devices, each with the memory it can actually give the model and
    optionally the decode rate it managed on its own, into two candidate splits: one in
    proportion to memory, which is what llama.cpp does by default, and one in proportion
    to speed, which is what you want when the hosts are not equally fast. It prints the
    gigabytes and the layers each device would take under each plan, flags any device
    that would be asked to hold more than it has, and gives the split string in both the
    comma form llama-server takes and the slash form llama-bench takes.
Platform: all (pure Python; nothing platform-specific)
Minimum memory: 8 GB on whichever machine you run it on; it loads nothing
Assumes: Python 3.9 or later; the usable-memory figures come from your own measurements,
    not from the box, because the memory a device will give a model is smaller than the
    memory it has.

Usage: python3 plan-tensor-split.py --weights-gb 125.5 --layers 94 \
           --device node-a:110 --device node-b:110
       python3 plan-tensor-split.py --weights-gb 18.6 --layers 48 --kv-gb 1.6 \
           --device desktop:22:38 --device strix:90:12 --plan speed

A device given as name:memory_gb has no speed, so the speed plan falls back to equal
shares for it and says so. Numbers here are arithmetic, not measurements: they tell you
what will fit and roughly where the work will land, and the run tells you the rest.
"""

from __future__ import annotations

import argparse
import sys


def parse_device(spec: str) -> dict:
    parts = spec.split(":")
    if len(parts) not in (2, 3):
        raise SystemExit(f"--device {spec!r}: expected name:memory_gb or name:memory_gb:tokens_per_s")
    name = parts[0]
    try:
        memory = float(parts[1])
        rate = float(parts[2]) if len(parts) == 3 else None
    except ValueError as exc:
        raise SystemExit(f"--device {spec!r}: {exc}") from exc
    if memory <= 0:
        raise SystemExit(f"--device {spec!r}: usable memory must be greater than zero")
    return {"name": name, "memory_gb": memory, "rate": rate}


def normalise(values: list[float]) -> list[float]:
    total = sum(values)
    return [v / total for v in values] if total > 0 else [1.0 / len(values)] * len(values)


def whole_layers(shares: list[float], layers: int) -> list[int]:
    """Largest-remainder allocation, so the layers add up to exactly `layers`."""
    exact = [s * layers for s in shares]
    base = [int(x) for x in exact]
    remaining = layers - sum(base)
    order = sorted(range(len(exact)), key=lambda i: exact[i] - base[i], reverse=True)
    for i in order[:remaining]:
        base[i] += 1
    return base


def report(plan: str, devices: list[dict], shares: list[float], args) -> bool:
    payload = args.weights_gb + args.kv_gb
    layers = whole_layers(shares, args.layers)
    print(f"\n== split by {plan} ==")
    print(f"{'device':<20}{'share':>9}{'GB held':>10}{'usable GB':>12}{'layers':>9}")
    over = False
    for device, share, count in zip(devices, shares, layers):
        held = payload * share
        flag = ""
        if held > device["memory_gb"]:
            flag = "  <-- more than this device has"
            over = True
        print(
            f"{device['name']:<20}{share:>9.3f}{held:>10.1f}{device['memory_gb']:>12.1f}"
            f"{count:>9d}{flag}"
        )
    proportions = [f"{s:.3f}".rstrip("0").rstrip(".") for s in shares]
    print(f"\n  llama-server / llama-cli:  -ts {','.join(proportions)}")
    print(f"  llama-bench:               -ts {'/'.join(proportions)}")
    return over


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--device", action="append", default=[], metavar="NAME:GB[:TOK_S]",
                        help="one device, in the order the client will see it")
    parser.add_argument("--weights-gb", type=float, required=True, help="size of the weights on disk")
    parser.add_argument("--kv-gb", type=float, default=0.0, help="KV cache at your context length")
    parser.add_argument("--layers", type=int, required=True, help="the model's layer count")
    parser.add_argument("--plan", choices=["memory", "speed", "both"], default="both")
    args = parser.parse_args()

    if len(args.device) < 2:
        print("give at least two --device arguments; a split needs somewhere to split to", file=sys.stderr)
        return 1

    devices = [parse_device(d) for d in args.device]
    payload = args.weights_gb + args.kv_gb
    capacity = sum(d["memory_gb"] for d in devices)

    print(f"model payload : {payload:.1f} GB  ({args.weights_gb:.1f} GB weights"
          f" + {args.kv_gb:.1f} GB KV cache)")
    print(f"cluster memory: {capacity:.1f} GB across {len(devices)} device(s)")
    print(f"headroom      : {capacity - payload:.1f} GB")
    if payload > capacity:
        print("\nThis model does not fit the cluster at all. Take a smaller quantisation,"
              "\nshorten the context, or add a machine. No split rescues it.")
        return 2

    over_anywhere = False
    if args.plan in ("memory", "both"):
        over_anywhere |= report("memory", devices, normalise([d["memory_gb"] for d in devices]), args)

    if args.plan in ("speed", "both"):
        rates = [d["rate"] for d in devices]
        if any(r is None for r in rates):
            mean = sum(r for r in rates if r is not None) / max(1, sum(1 for r in rates if r is not None)) \
                if any(r is not None for r in rates) else 1.0
            rates = [r if r is not None else mean for r in rates]
            print("\n(one or more devices had no rate; they were given the average of the others)")
        over_anywhere |= report("speed", devices, normalise(rates), args)

    print("\nRead this before using either split:")
    print("  * The order above must match the order the client sees: local devices first,")
    print("    then the RPC hosts in the order they appear in RPC_HOSTS. Confirm it with")
    print("    probe-rpc-devices.sh rather than assuming.")
    print("  * Usable memory is not installed memory. Leave room for the KV cache, the")
    print("    compute buffers and whatever else the machine is doing.")
    print("  * Splitting by speed only helps while every share still fits. A device flagged")
    print("    above will fall back to host memory or fail to allocate.")
    return 3 if over_anywhere else 0


if __name__ == "__main__":
    raise SystemExit(main())

Download plan-tensor-split.py138 lines

RunnableAll tracks

python3 plan-tensor-split.py \
    --weights-gb 125.5 --kv-gb 1.6 --layers 94 \
    --device node-a:110 --device node-b:110

Output — what you should see

model payload : 127.1 GB  (125.5 GB weights + 1.6 GB KV cache)
cluster memory: 220.0 GB across 2 device(s)
headroom      : 92.9 GB

== split by memory ==
device                  share   GB held   usable GB   layers
node-a                  0.500      63.5       110.0       47
node-b                  0.500      63.5       110.0       47

  llama-server / llama-cli:  -ts 0.5,0.5
  llama-bench:               -ts 0.5/0.5

Two machines with roughly equal usable memory give a roughly equal split, which is what the default would have chosen anyway. Run it again with a third, smaller device and watch the proportions move; that is the case where writing the split down beats trusting the default.

6. Start a server on every host

On each RPC host, not the client:

RunnableAll tracks

#!/usr/bin/env bash
# Purpose: start one ggml-rpc-server on this machine, bound to the cluster link and nothing
#          else, with the local tensor cache enabled, and print exactly what it exposes
# Platform: all (CUDA, Vulkan, Metal or CPU builds; the backend is whatever this host built)
# Minimum memory: 8 GB
# Assumes: llama.cpp built with -DGGML_RPC=ON so that ggml-rpc-server exists in $LLAMA_BIN;
#          the interface named in CLUSTER_IFACE exists and has an IPv4 address; nothing else
#          is listening on RPC_PORT; the environment from env-example.txt has been sourced
#
# Usage: . ./rpc.env && bash start-rpc-server.sh
#        . ./rpc.env && bash start-rpc-server.sh --background
#
# Environment (all from env-example.txt):
#   CLUSTER_IFACE  interface carrying cluster traffic on this host   (required)
#   CLUSTER_ADDR   address to bind to, overriding the interface      (default: read from CLUSTER_IFACE)
#   RPC_PORT       port to listen on                                 (default: 50052)
#   RPC_DEVICE     devices to expose, e.g. CUDA0 or Vulkan0 or CPU   (default: every accelerator)
#   RPC_THREADS    CPU threads for the CPU device                    (default: the tool's own)
#   LLAMA_BIN      directory holding ggml-rpc-server                 (default: found on PATH)
#   RPC_NO_RDMA    set to 1 to force plain TCP for a comparison run  (default: unset)
#
# The RPC server has no authentication and no encryption, and its own README says so in
# capitals: never run it on an open network. This script therefore refuses to bind to a
# wildcard address, and prints the one address it did bind to so you can check it.

set -euo pipefail

BACKGROUND=0
[ "${1:-}" = "--background" ] && BACKGROUND=1

RPC_PORT="${RPC_PORT:-50052}"
CLUSTER_ADDR="${CLUSTER_ADDR:-}"
RPC_DEVICE="${RPC_DEVICE:-}"
RPC_THREADS="${RPC_THREADS:-}"

die() { echo "start-rpc-server: $*" >&2; exit 1; }

# --- 1. Find the binary ----------------------------------------------------------------
if [ -n "${LLAMA_BIN:-}" ]; then
  SERVER="$LLAMA_BIN/ggml-rpc-server"
else
  SERVER="$(command -v ggml-rpc-server || true)"
fi
[ -n "$SERVER" ] && [ -x "$SERVER" ] || die \
  "ggml-rpc-server not found. Set LLAMA_BIN, and check the build was configured with -DGGML_RPC=ON."

# --- 2. Work out which address to bind to ----------------------------------------------
if [ -z "$CLUSTER_ADDR" ]; then
  [ -n "${CLUSTER_IFACE:-}" ] || die "set CLUSTER_IFACE (or CLUSTER_ADDR); see env-example.txt"
  case "$(uname -s)" in
    Darwin)
      CLUSTER_ADDR="$(ipconfig getifaddr "$CLUSTER_IFACE" || true)"
      ;;
    *)
      CLUSTER_ADDR="$(ip -4 -o addr show dev "$CLUSTER_IFACE" 2>/dev/null \
        | awk '{ print $4 }' | cut -d/ -f1 | head -n 1)"
      ;;
  esac
fi
[ -n "$CLUSTER_ADDR" ] || die "$CLUSTER_IFACE has no IPv4 address; bring the link up first (Part 18)"

case "$CLUSTER_ADDR" in
  0.0.0.0|"*"|"")
    die "refusing to bind to every interface. Name the cluster link in CLUSTER_IFACE instead."
    ;;
esac

# --- 3. Say what is about to happen ------------------------------------------------------
echo "==> ggml-rpc-server"
echo "    binary     : $SERVER"
echo "    binding to : $CLUSTER_ADDR port $RPC_PORT (interface ${CLUSTER_IFACE:-set explicitly})"
echo "    cache      : on (-c); large tensors are kept on local disk instead of re-sent"
[ -n "$RPC_DEVICE" ] && echo "    devices    : $RPC_DEVICE" || echo "    devices    : every accelerator this build can see"
[ -n "$RPC_THREADS" ] && echo "    threads    : $RPC_THREADS"
if [ -n "${RPC_NO_RDMA:-}" ]; then
  echo "    transport  : TCP forced (GGML_RPC_NO_RDMA is set); unset it to allow RDMA"
else
  echo "    transport  : negotiated at handshake; RDMA if both peers can, TCP otherwise"
fi

# --- 4. Build the argument list ----------------------------------------------------------
ARGS=(-H "$CLUSTER_ADDR" -p "$RPC_PORT" -c)
[ -n "$RPC_DEVICE" ] && ARGS+=(-d "$RPC_DEVICE")
[ -n "$RPC_THREADS" ] && ARGS+=(-t "$RPC_THREADS")

if [ -n "${RPC_NO_RDMA:-}" ]; then
  export GGML_RPC_NO_RDMA=1
fi

# --- 5. Run it ---------------------------------------------------------------------------
if [ "$BACKGROUND" = "1" ]; then
  LOG="rpc-server-$(hostname -s)-${RPC_PORT}.log"
  nohup "$SERVER" "${ARGS[@]}" > "$LOG" 2>&1 &
  echo "    pid        : $! (log: $LOG)"
  echo "    stop it with: kill $!"
else
  echo "    press Ctrl-C to stop it"
  exec "$SERVER" "${ARGS[@]}"
fi

Download start-rpc-server.sh99 lines

RunnableAll tracks

. ./rpc.env
bash start-rpc-server.sh --background

Output — what you should see

==> ggml-rpc-server
    binary     : /home/you/llama.cpp/build/bin/ggml-rpc-server
    binding to : 10.x.x.x port 50052 (interface enp1s0f1np1)
    cache      : on (-c); large tensors are kept on local disk instead of re-sent
    devices    : every accelerator this build can see
    transport  : negotiated at handshake; RDMA if both peers can, TCP otherwise
    pid        : 12345 (log: rpc-server-node-b-50052.log)

Read the binding line every time. The script refuses a wildcard address, but it cannot know whether the interface you named is the one you meant.

For the single-machine path, run two of them on the same host on different ports:

RunnableAll tracks

CLUSTER_ADDR=127.0.0.1 RPC_PORT=50052 bash start-rpc-server.sh --background
CLUSTER_ADDR=127.0.0.1 RPC_PORT=50053 bash start-rpc-server.sh --background

7. Confirm the device order before you write a split

The proportions in --tensor-split are positional and nothing checks them. Find out what the client actually sees.

RunnableAll tracks

#!/usr/bin/env bash
# Purpose: ask every rpc-server in the cluster what device it is offering, one host at a
#          time, and write the device order the client will use into one short report
# Platform: all (run it on the client; the hosts may be any mixture of tracks)
# Minimum memory: 8 GB on the client; the hosts need only enough for their own share
# Assumes: llama.cpp built with -DGGML_RPC=ON on the client so llama-bench can register RPC
#          devices; one ggml-rpc-server running on every host in RPC_HOSTS; the environment
#          from env-example.txt has been sourced
#
# Usage: . ./rpc.env && bash probe-rpc-devices.sh [report.md]
#
# Environment:
#   RPC_HOSTS   host:port,host:port                      (required)
#   LLAMA_BIN   directory holding llama-bench            (default: found on PATH)
#
# Why one host at a time: --list-devices with every host registered prints one flat list,
# and nothing in that list says which host a device came from. Registering one host per
# run and recording the order is the only way to know what --tensor-split is addressing.

set -euo pipefail

REPORT="${1:-rpc-devices.md}"

die() { echo "probe-rpc-devices: $*" >&2; exit 1; }

if [ -n "${LLAMA_BIN:-}" ]; then
  BENCH="$LLAMA_BIN/llama-bench"
else
  BENCH="$(command -v llama-bench || true)"
fi
[ -n "$BENCH" ] && [ -x "$BENCH" ] || die "llama-bench not found; set LLAMA_BIN"

# --- Part 18 compatibility --------------------------------------------------------------
# Part 18's .env lists every machine in CLUSTER_PEERS, space separated, as names without
# ports. When RPC_HOSTS is not set, build it from those names in the order they appear,
# appending RPC_PORT to each. Set RPC_HOSTS yourself whenever the order matters, when only
# some peers take part, or when a peer listens on a different port: the order is what
# --tensor-split addresses.
RPC_PORT="${RPC_PORT:-50052}"
if [ -z "${RPC_HOSTS:-}" ] && [ -n "${CLUSTER_PEERS:-}" ]; then
  read -r -a PART18_PEERS <<< "$CLUSTER_PEERS"
  for peer in "${PART18_PEERS[@]}"; do
    RPC_HOSTS="${RPC_HOSTS:+$RPC_HOSTS,}${peer}:${RPC_PORT}"
  done
  echo "    RPC_HOSTS built from Part 18's CLUSTER_PEERS: $RPC_HOSTS"
fi

[ -n "${RPC_HOSTS:-}" ] || die "set RPC_HOSTS (or Part 18's CLUSTER_PEERS); see env-example.txt"

{
  echo "# RPC cluster device probe"
  echo
  echo "- client: $(hostname -s), $(uname -s) $(uname -m)"
  echo "- probed: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "- RPC_HOSTS: \`$RPC_HOSTS\`"
  echo
} > "$REPORT"

echo "==> local devices, which always come first in the split order"
{
  echo "## Local devices (position 1 onwards in --tensor-split)"
  echo
  echo '```'
} >> "$REPORT"
"$BENCH" --list-devices 2>&1 | tee -a "$REPORT"
echo '```' >> "$REPORT"
echo >> "$REPORT"

IFS=',' read -r -a HOSTS <<< "$RPC_HOSTS"
POSITION=1
for hp in "${HOSTS[@]}"; do
  host="${hp%%:*}"
  port="${hp##*:}"
  echo "==> $hp"
  {
    echo "## RPC host $POSITION: \`$hp\`"
    echo
  } >> "$REPORT"

  if (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null; then
    echo "    reachable"
    echo "- TCP connect: succeeded" >> "$REPORT"
  else
    echo "    NOT reachable: nothing is listening on $hp" >&2
    echo "- TCP connect: **failed** — no server listening, or the wrong interface" >> "$REPORT"
    echo >> "$REPORT"
    POSITION=$((POSITION + 1))
    continue
  fi

  # Round-trip time on the path the RPC traffic will take. A number in the tens of
  # milliseconds here means Wi-Fi or a router, and this part's challenge page is about that.
  if command -v ping >/dev/null; then
    rtt="$(ping -c 4 "$host" 2>/dev/null | tail -n 1 || true)"
    [ -n "$rtt" ] && echo "- round trip: \`$rtt\`" >> "$REPORT"
  fi

  {
    echo
    echo '```'
  } >> "$REPORT"
  "$BENCH" --rpc "$hp" --list-devices 2>&1 | tee -a "$REPORT"
  {
    echo '```'
    echo
  } >> "$REPORT"
  POSITION=$((POSITION + 1))
done

{
  echo "## Split order"
  echo
  echo "The proportions given to \`--tensor-split\` are read in this order: every local"
  echo "device listed above first, then the RPC hosts in the order they appear in"
  echo "RPC_HOSTS. Copy that order into your notebook beside the split you chose."
} >> "$REPORT"

echo "==> written to $REPORT"

Download probe-rpc-devices.sh118 lines

RunnableAll tracks

. ./rpc.env
bash probe-rpc-devices.sh rpc-devices.md

The report lists the local devices first, then one section per RPC host in the order they appear in RPC_HOSTS. That order is the order your proportions are read in. Copy it into the notebook next to the split you are about to use.

8. Run the split

RunnableAll tracks

#!/usr/bin/env bash
# Purpose: run one model across this machine and the RPC hosts, with an explicit or a
#          proportional split, either as a server to measure against or as a benchmark
# Platform: all (the client is whichever machine you sit at; the hosts can be any track)
# Minimum memory: 8 GB on the client; the model has to fit the sum of every device
# Assumes: llama.cpp built with -DGGML_RPC=ON on the client; one ggml-rpc-server already
#          running on every host in RPC_HOSTS; the model file readable at $MODEL; the
#          environment from env-example.txt has been sourced
#
# Usage: . ./rpc.env && bash run-split.sh                 # start llama-server
#        . ./rpc.env && MODE=bench bash run-split.sh      # run llama-bench instead
#        . ./rpc.env && MODE=probe bash run-split.sh      # list the devices and exit
#
# Environment (all from env-example.txt):
#   MODEL         first shard of the GGUF                      (required)
#   RPC_HOSTS     host:port,host:port in device order          (required)
#   TENSOR_SPLIT  comma-separated proportions, e.g. 1,1,0.5    (default: by free memory)
#   CTX           context length to allocate                   (default: 8192)
#   NGL           layers to offload                            (default: 999)
#   LLAMA_BIN     directory holding the binaries               (default: found on PATH)
#   HOST_PORT     port llama-server listens on locally         (default: 8080)
#   MODE          server, bench or probe                       (default: server)
#
# The device order that --tensor-split addresses is: this machine's local devices first,
# then the RPC hosts in the order they appear in RPC_HOSTS. Run MODE=probe once and read
# the list before you write a split; guessing the order is how a split ends up backwards.

set -euo pipefail

MODE="${MODE:-server}"
CTX="${CTX:-8192}"
NGL="${NGL:-999}"
HOST_PORT="${HOST_PORT:-8080}"
TENSOR_SPLIT="${TENSOR_SPLIT:-}"

die() { echo "run-split: $*" >&2; exit 1; }

bin() {
  if [ -n "${LLAMA_BIN:-}" ]; then
    echo "$LLAMA_BIN/$1"
  else
    command -v "$1" || true
  fi
}


# --- Part 18 compatibility --------------------------------------------------------------
# Part 18's .env lists every machine in CLUSTER_PEERS, space separated, as names without
# ports. When RPC_HOSTS is not set, build it from those names in the order they appear,
# appending RPC_PORT to each. Set RPC_HOSTS yourself whenever the order matters, when only
# some peers take part, or when a peer listens on a different port: the order is what
# --tensor-split addresses.
RPC_PORT="${RPC_PORT:-50052}"
if [ -z "${RPC_HOSTS:-}" ] && [ -n "${CLUSTER_PEERS:-}" ]; then
  read -r -a PART18_PEERS <<< "$CLUSTER_PEERS"
  for peer in "${PART18_PEERS[@]}"; do
    RPC_HOSTS="${RPC_HOSTS:+$RPC_HOSTS,}${peer}:${RPC_PORT}"
  done
  echo "    RPC_HOSTS built from Part 18's CLUSTER_PEERS: $RPC_HOSTS"
fi

[ -n "${RPC_HOSTS:-}" ] || die "set RPC_HOSTS (or Part 18's CLUSTER_PEERS); see env-example.txt"

# --- 1. Are the hosts actually there? ---------------------------------------------------
# A cluster that fails three minutes into loading a 125 GB model because one host is not
# listening is a cluster that wasted three minutes. Check first; it costs nothing.
IFS=',' read -r -a HOSTS <<< "$RPC_HOSTS"
for hp in "${HOSTS[@]}"; do
  host="${hp%%:*}"
  port="${hp##*:}"
  if ! (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null; then
    die "no ggml-rpc-server answering at $hp. Start it there, and check CLUSTER_IFACE on that host."
  fi
  echo "    reachable: $hp"
done

# --- 2. Probe: list every device this run would see, then stop ---------------------------
if [ "$MODE" = "probe" ]; then
  BENCH="$(bin llama-bench)"
  [ -x "$BENCH" ] || die "llama-bench not found; set LLAMA_BIN"
  echo "==> devices visible to this client, local first, then the RPC hosts in order"
  exec "$BENCH" --rpc "$RPC_HOSTS" --list-devices
fi

[ -n "${MODEL:-}" ] || die "set MODEL to the first shard of the GGUF; see env-example.txt"
[ -f "$MODEL" ] || die "$MODEL does not exist on this machine"

echo "==> splitting $(basename "$MODEL")"
echo "    rpc hosts    : $RPC_HOSTS"
echo "    context      : $CTX tokens, offloading $NGL layers"
if [ -n "$TENSOR_SPLIT" ]; then
  echo "    tensor split : $TENSOR_SPLIT (local devices first, then RPC hosts in order)"
else
  echo "    tensor split : none given, so llama.cpp splits in proportion to free memory"
fi

# --- 3. Benchmark mode --------------------------------------------------------------------
if [ "$MODE" = "bench" ]; then
  BENCH="$(bin llama-bench)"
  [ -x "$BENCH" ] || die "llama-bench not found; set LLAMA_BIN"
  ARGS=(-m "$MODEL" --rpc "$RPC_HOSTS" -ngl "$NGL" -p 512 -n 128 -r 3 -o json)
  if [ -n "$TENSOR_SPLIT" ]; then
    # llama-bench separates split proportions with "/" where llama-server uses ","
    ARGS+=(-ts "${TENSOR_SPLIT//,//}")
  fi
  exec "$BENCH" "${ARGS[@]}"
fi

# --- 4. Server mode -----------------------------------------------------------------------
SERVER="$(bin llama-server)"
[ -x "$SERVER" ] || die "llama-server not found; set LLAMA_BIN"

ARGS=(-m "$MODEL" --rpc "$RPC_HOSTS" -ngl "$NGL" -c "$CTX"
      --host 127.0.0.1 --port "$HOST_PORT" --metrics)
[ -n "$TENSOR_SPLIT" ] && ARGS+=(-ts "$TENSOR_SPLIT")

echo "    listening on : 127.0.0.1 port $HOST_PORT, with the metrics endpoint enabled"
echo "    loading now; a split model loads slowly the first time and quickly after that,"
echo "    because each host caches its own tensors on local disk."
exec "$SERVER" "${ARGS[@]}"

Download run-split.sh120 lines

RunnableAll tracks

. ./rpc.env
bash run-split.sh

The first load is slow, and most of that time is the weights crossing the link to each host. Watch the load log for the line naming how many layers went to each device; that number, not your intention, is what the run is doing. Once loaded, leave the server running for the measurements.

9. Measure it

RunnableAll tracks

#!/usr/bin/env python3
"""Measure one generation on a split cluster: prefill, decode, and bytes over the link.

Purpose: send one completion to a llama-server that is running across RPC hosts, read the
    prefill and decode timings the server reports, read this machine's interface byte
    counters either side of the request, and append one lab-notebook line describing the
    run. The point is not the tokens-per-second figure on its own: it is that figure next
    to the number of bytes the cluster link carried to produce it.
Platform: all (Linux reads /proc/net/dev, macOS reads netstat -ib; nothing else is needed)
Minimum memory: 8 GB on the client; the model has to fit the sum of the cluster's devices
Assumes: Python 3.9 or later; a llama-server started by run-split.sh and still loading or
    loaded; the interface name from CLUSTER_IFACE; the lab notebook from Part 1.

Usage: python3 measure-split.py --iface enp1s0f1np1 --labbook labbook.md
       python3 measure-split.py --iface en5 --server-url http://127.0.0.1:8080 \
           --prompt-words 400 --n-predict 128 --note "two Sparks over RoCE" \
           --transport rdma --tensor-split 1,1

Counters are for the whole interface, so anything else using that link during the run is
counted too. Run it on a quiet cluster, and treat the byte figures as the order of
magnitude they are, not as an exact accounting of the model's traffic.
"""

from __future__ import annotations

import argparse
import json
import platform
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path

PROMPT_SEED = (
    "A cluster is two machines and a cable, and the cable is the part that decides. "
    "Explain, carefully and at length, what a layer split sends between machines and why. "
)


def counters(iface: str) -> tuple[int | None, int | None]:
    """Bytes received and transmitted on one interface, or (None, None) if unreadable."""
    system = platform.system()
    if system == "Linux":
        try:
            for line in Path("/proc/net/dev").read_text(encoding="utf-8").splitlines():
                name, _, rest = line.partition(":")
                if name.strip() != iface:
                    continue
                fields = rest.split()
                return int(fields[0]), int(fields[8])
        except (OSError, ValueError, IndexError):
            return None, None
        return None, None
    if system == "Darwin":
        try:
            out = subprocess.run(
                ["netstat", "-ib"], capture_output=True, text=True, check=True, timeout=20
            ).stdout
        except (OSError, subprocess.SubprocessError):
            return None, None
        for line in out.splitlines():
            fields = line.split()
            # Name Mtu Network Address Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll
            if len(fields) >= 10 and fields[0] == iface:
                try:
                    return int(fields[6]), int(fields[9])
                except ValueError:
                    continue
        return None, None
    return None, None


def get_json(url: str, timeout: float = 10.0):
    try:
        with urllib.request.urlopen(url, timeout=timeout) as handle:  # noqa: S310 - local server
            return json.loads(handle.read().decode("utf-8"))
    except (urllib.error.URLError, OSError, ValueError):
        return None


def get_text(url: str, timeout: float = 10.0) -> str | None:
    try:
        with urllib.request.urlopen(url, timeout=timeout) as handle:  # noqa: S310 - local server
            return handle.read().decode("utf-8")
    except (urllib.error.URLError, OSError):
        return None


def post_completion(base: str, prompt: str, n_predict: int, timeout: float):
    body = json.dumps(
        {"prompt": prompt, "n_predict": n_predict, "temperature": 0.0, "cache_prompt": False}
    ).encode("utf-8")
    request = urllib.request.Request(
        f"{base}/completion", data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(request, timeout=timeout) as handle:  # noqa: S310 - local server
        return json.loads(handle.read().decode("utf-8"))


def pick(source: dict, *names, default=None):
    """First present, non-empty value among several possible key names."""
    for name in names:
        if isinstance(source, dict) and source.get(name) not in (None, ""):
            return source[name]
    return default


def metrics_of(text: str | None) -> dict:
    """Pull the llamacpp: gauges out of the Prometheus exposition text."""
    wanted = {
        "llamacpp:prompt_tokens_seconds": "metrics_prompt_tokens_per_s",
        "llamacpp:predicted_tokens_seconds": "metrics_predicted_tokens_per_s",
        "llamacpp:n_decode_total": "metrics_decode_calls_total",
    }
    found: dict = {}
    if not text:
        return found
    for line in text.splitlines():
        if line.startswith("#") or " " not in line:
            continue
        name, _, value = line.partition(" ")
        key = wanted.get(name.split("{")[0])
        if key:
            try:
                found[key] = float(value)
            except ValueError:
                pass
    return found


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--server-url", default="http://127.0.0.1:8080", help="llama-server base URL")
    parser.add_argument("--iface", default="", help="cluster interface to read counters from")
    parser.add_argument("--prompt-words", type=int, default=400, help="approximate prompt length")
    parser.add_argument("--n-predict", type=int, default=128, help="tokens to generate")
    parser.add_argument("--timeout", type=float, default=900.0, help="seconds to wait for the reply")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--lab", default="part-19/lab-a-model-bigger-than-any-one-machine")
    parser.add_argument("--note", default="", help="free text: topology, transport, anything")
    parser.add_argument("--transport", default="", help="tcp, rdma or unknown")
    parser.add_argument("--tensor-split", default="", help="the proportions this run used")
    parser.add_argument("--print-only", action="store_true", help="print it, record nothing")
    args = parser.parse_args()

    base = args.server_url.rstrip("/")
    props = get_json(f"{base}/props")
    if props is None:
        print(f"no llama-server answering at {base}; start run-split.sh first", file=sys.stderr)
        return 1

    words = PROMPT_SEED.split()
    prompt = " ".join((words * (args.prompt_words // len(words) + 1))[: args.prompt_words])

    rx0, tx0 = counters(args.iface) if args.iface else (None, None)
    wall0 = time.monotonic()
    try:
        reply = post_completion(base, prompt, args.n_predict, args.timeout)
    except (urllib.error.URLError, OSError, ValueError) as exc:
        print(f"the completion request failed: {exc}", file=sys.stderr)
        print("On a cluster this usually means a host went away mid-generation.", file=sys.stderr)
        return 1
    wall = time.monotonic() - wall0
    rx1, tx1 = counters(args.iface) if args.iface else (None, None)

    timings = reply.get("timings") or {}
    entry = {
        "record": "split-run",
        "lab": args.lab,
        "engine": "llama.cpp",
        "build": pick(props.get("build_info", {}) if isinstance(props.get("build_info"), dict) else {},
                      "build", "commit") or pick(props, "build_info"),
        "model": pick(props, "model_path", "model") or reply.get("model"),
        "client": f"{platform.system()}-{platform.machine()}",
        "transport": args.transport or "unknown",
        "tensor_split": args.tensor_split or "by free memory",
        "iface": args.iface or None,
        "prompt_tokens": pick(timings, "prompt_n", "tokens_evaluated"),
        "prompt_ms": pick(timings, "prompt_ms"),
        "prompt_tokens_per_s": pick(timings, "prompt_per_second"),
        "predicted_tokens": pick(timings, "predicted_n", "tokens_predicted"),
        "predicted_ms": pick(timings, "predicted_ms"),
        "predicted_tokens_per_s": pick(timings, "predicted_per_second"),
        "wall_s": round(wall, 3),
        "rx_bytes": (rx1 - rx0) if (rx0 is not None and rx1 is not None) else None,
        "tx_bytes": (tx1 - tx0) if (tx0 is not None and tx1 is not None) else None,
        "note": args.note,
        "measured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }
    entry.update(metrics_of(get_text(f"{base}/metrics")))

    def show(label: str, value, unit: str = "") -> None:
        text = f"{value:,.2f}" if isinstance(value, float) else ("n/a" if value is None else f"{value:,}")
        print(f"    {label:<28} {text:>16} {unit}")

    print("==> one completion across the cluster")
    show("prompt tokens", entry["prompt_tokens"])
    show("prefill", entry["prompt_tokens_per_s"], "tokens/s")
    show("generated tokens", entry["predicted_tokens"])
    show("decode", entry["predicted_tokens_per_s"], "tokens/s")
    show("wall clock", entry["wall_s"], "s")
    show("bytes received on link", entry["rx_bytes"])
    show("bytes sent on link", entry["tx_bytes"])
    if entry["tx_bytes"] and entry["predicted_tokens"]:
        per_token = (entry["tx_bytes"] + (entry["rx_bytes"] or 0)) / float(entry["predicted_tokens"])
        show("link bytes per token", round(per_token, 1))
        print("    Compare that against the hidden-state arithmetic in this part's first lesson.")
    if entry["rx_bytes"] is None:
        print("    No counters: pass --iface with the interface name from CLUSTER_IFACE.")

    if args.print_only:
        return 0

    notebook = Path(args.labbook)
    if not notebook.exists():
        print(f"    {notebook} does not exist; creating it", file=sys.stderr)
    with notebook.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(entry) + "\n")
    print(f"    recorded 1 line in {notebook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download measure-split.py226 lines

RunnableAll tracks

python3 measure-split.py \
    --iface "$CLUSTER_IFACE" \
    --transport rdma \
    --tensor-split "${TENSOR_SPLIT:-by free memory}" \
    --note "two Sparks over the QSFP link" \
    --labbook labbook.md

Output — what you should see

==> one completion across the cluster
    prompt tokens                             xxx
    prefill                                 xx.xx tokens/s
    generated tokens                          128
    decode                                  xx.xx tokens/s
    wall clock                              xx.xx s
    bytes received on link              xx,xxx,xxx
    bytes sent on link                 xx,xxx,xxx
    link bytes per token                  xx,xxx
    Compare that against the hidden-state arithmetic in this part's first lesson.
    recorded 1 line in labbook.md

The last line before the recording is the one to think about. The first lesson’s arithmetic said the model payload at a boundary is about 8 KiB per token for this model. Your measured figure will be larger, because the protocol also describes the computation graph on every step and because the counter sees everything else on that interface. How much larger is a property of your cluster worth writing down.

10. TCP against RDMA, one variable at a time

Only on a link where RDMA is available: a QSFP link between Sparks, or Thunderbolt 5 between Apple silicon Macs on macOS 26.2 or later. If neither describes your cluster, skip to the recording sheet and note that this comparison was not applicable, which is itself a result.

RunnableAll tracks

pkill -f ggml-rpc-server
. ./rpc.env
RPC_NO_RDMA=1 bash start-rpc-server.sh --background

Then restart run-split.sh on the client and repeat the measurement with --transport tcp. Change nothing else: same model, same split, same context, same prompt length, same number of generated tokens. Two runs that differ in one variable are a comparison; two runs that differ in three are an anecdote.

11. Record it in the course format

Pending validationA 200B-class model across the cluster — your recording sheet

Run	Machines	Transport	Prefill tokens/s	Decode tokens/s	Link bytes per generated token
split, default proportions	—	—	—	—	—
split, chosen proportions	—	—	—	—	—
split, forced TCP	—	tcp	—	—	—
largest model that fits one box	1	n/a	—	—	0

your cluster: one line per machine, with track, memory and link, the operating system and version of each machine · llama.cpp RPC, layer split the build number from llama-cli --version, which must match on every machine · Qwen3-235B-A22B, IQ4_XS · 8,192 tokens of context · the date you ran it

Empty on purpose. The last row is the one that makes the table an argument rather than a demonstration: run gpt-oss-120b, or whatever the largest model is that fits your best single machine, under the Part 6 methodology, and put its decode rate here. If that row is faster than the split rows, you have learned the most useful thing in this part.

Verify the capacity claim before loading the target

Write the memory budget per participating device, including weights, cache, overhead and headroom. Confirm that the target exceeds the usable capacity of each individual machine under the selected representation. Otherwise this run may demonstrate distribution but not the stated capacity claim.

Start RPC workers in their identified terminals and verify discovery from the client before loading the large model. Save device order and the actual split selected. A tensor-split vector must refer to that order; a vector calculated for another enumeration can assign work to the wrong device. Use the smaller checkpoint path first if discovery or placement remains uncertain.

After loading, issue one short request, then the intended context and output workload. Retain client and worker memory observations and transport evidence. If a worker fails, stop the client workload and collect the worker’s first error before restarting. Record the exact process IDs or terminal ownership so cleanup affects only this lab. Keep model identity, split plan, startup logs, network measurements and task output. A successful completion establishes that the planned model can run across these devices; speedup and fault tolerance require their own comparisons and failure tests.

Validation

You are done when all of the following are true:

ggml-rpc-server exists in the build directory on every machine, and each was started bound to the cluster link’s address rather than to a wildcard;
rpc-devices.md lists the local devices and every RPC host, in the order the client sees them, and you have written that order in the notebook;
the client’s load log shows the model’s layers distributed across the devices, with a count per device that matches the split you asked for to within a layer;
the server answered a completion and measure-split.py recorded a line in labbook.md with non-null prompt_tokens_per_s and predicted_tokens_per_s;
that line has non-null rx_bytes and tx_bytes, which means the interface name was right;
the recording sheet above is filled in, including the context block and the single-machine comparison row;
you can state, in one sentence each: which machine held which layers, how many bytes crossed the link per generated token, and how that compares with the 8 KiB the arithmetic predicted.

Expected outcome

A model running that could not run before, and a page of notes explaining what it cost.

Some shapes are common enough to predict, and noticing them is most of the value.

Loading dominates the first run and almost disappears on the second. The local cache is doing the work, and the difference between the two is a direct measurement of how much of your first evening was network.

Decode is slower than the same-sized model would be on one machine, if such a machine existed. The machines take turns, so the per-token cost is the sum of the two computations plus a boundary crossing. Capacity is what you bought.

The measured bytes per token exceed the arithmetic. The 8 KiB figure is the model payload at one boundary; the protocol also carries the graph description, and your interface counter sees any other traffic on that link. A measurement within a small multiple of the prediction means the split is behaving as designed.

RDMA’s effect, where you could test it, is visible in decode rather than in loading. Bulk transfer is bandwidth-bound and both transports have plenty of bandwidth; per-token latency is where the removed copies show up.

The single-box comparison row is uncomfortable. A 120B-class mixture-of-experts model on one Spark will very likely generate faster than a 235B-class model split across two. That is not a failure of the lab. It is the lab’s most important output, and Part 16’s evaluation methods are how you decide whether the larger model is actually better at your task.

Troubleshooting

--rpc is rejected as an unknown option. The client was built without -DGGML_RPC=ON. Rebuild it. This is the most common first failure and it is not subtle once you know it.

ggml-rpc-server: command not found. Either the build lacked the RPC flag, or you are looking for the older name rpc-server. Check for the binary in build/bin before anything else.

The client cannot connect, but the server is running. Almost always the bind address. Read the server’s endpoint line: if it says 127.0.0.1, it is reachable only from its own machine. start-rpc-server.sh prints the address it chose for exactly this reason.

Loading fails with an allocation error on one host. That host’s usable memory is smaller than the split assumed. On Track X this is the GPU-visible cap; on Track N it is the card’s VRAM. Re-run plan-tensor-split.py with the real figure and set --tensor-split by hand.

The model loads but the layer counts do not match your split. Check the device order in rpc-devices.md. Proportions are positional, local devices come first, and a split written in the wrong order loads silently.

Generation is far slower than the sum of the machines suggests. Stop and go to this part’s challenge page, which is a procedure for exactly this and which starts by asking which interface carried the traffic.

The second load is as slow as the first. The cache is off or unwritable. The servers must be started with -c, and $HOME/.cache/llama.cpp/rpc must be writable on each host; the README gives LLAMA_CACHE as the way to move it.

measure-split.py reports no byte counters. The interface name did not match. On Linux compare it against ip -br addr; on macOS the counters come from netstat -ib and the name is the short one such as en5.

A host disappears mid-generation. The client’s request fails rather than degrading. That is expected behaviour and the next lab measures it deliberately.

Cleanup

Stop the servers. They are not a service and they have no authentication.

RunnableAll tracks

pkill -f ggml-rpc-server

RunnableAll tracks

du -sh ~/.cache/llama.cpp/rpc

Keep the model if you have the disk; the next lab and the challenge both reuse this cluster, and Part 20 compares this result against a vLLM multi-node run on the same machines. Keep labbook.md, rpc-devices.md and rpc.env, and keep rpc.env out of anything you publish: it names your machines.

What you learned

A cluster is a memory answer. You ran a model that fits nowhere in your house, and the arithmetic told you it would work before the download started.
The split is positional and silent. Local devices first, then the RPC hosts in order. Confirming that order takes one command and prevents the most common failure in this part.
The default split follows reported free memory, which is not always usable memory. On a capped device you must set the proportions yourself.
The cache turns a nightly transfer into a one-off. The difference between your first and second load is a measurement of your link, not of the engine.
Bytes per token is the number that explains the cluster. Arithmetic predicted its order of magnitude; your counter measured the rest, and the gap between them is protocol overhead.
One machine may still be the right answer. The last row of your recording sheet is the one to look at hardest.

Record in the notebook: the build number on every machine and the confirmation that they match; the device order from rpc-devices.md; the split you used and why; the layers each device actually received; prefill, decode and link bytes per token for each run; the transport each link negotiated; and the decode rate of the largest model that fits your best single machine, beside it all.

Sources for this lesson

8 verified · checked 2026-09-09

01llama.cpp — RPC backend README§ Usage; Local cache; RDMA transport; Troubleshootinggithub.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md2026-09-09
02llama.cpp — tools/rpc/rpc-server.cpp§ print_usage and the argument parsergithub.com/ggml-org/llama.cpp/blob/master/tools/rpc/rpc-server.cpp2026-09-09
03llama.cpp — llama-server README§ Command-line options; GET /metrics; the timings objectgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md2026-09-09
04llama.cpp — llama-bench README§ Usage and optionsgithub.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md2026-09-09
05unsloth/Qwen3-235B-A22B-GGUF model repository§ Files and versions; the IQ4_XS directoryhuggingface.co/unsloth/Qwen3-235B-A22B-GGUF2026-09-09
06unsloth/Qwen3-30B-A3B-GGUF model repository§ Files and versionshuggingface.co/unsloth/Qwen3-30B-A3B-GGUF2026-09-09
07NVIDIA DGX Spark playbooks — Connect Two Sparks§ Prerequisites; physical hardware connectiongithub.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/connect-two-sparks/README.md2026-09-09
08NVIDIA DGX Spark — Clustering§ QSFP ports and RoCE devices; supported cluster sizesdocs.nvidia.com/dgx/dgx-spark/spark-clustering.html2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.

Lab: Run a Model Bigger Than Any One Machine

Objective

Architecture

Does it fit? The arithmetic, before the download

Requirements

Track S — NVIDIA DGX Spark

Track X — AMD Ryzen AI Max+ 395Partial

Track M — Apple silicon

Track N — NVIDIA desktop or laptopPartial

Tasks

Working directory and terminal roles

1. Confirm the network before you trust it

2. Build every machine with RPC compiled in

3. Fetch the model

4. Fill in the environment file

5. Do the arithmetic before you run anything

6. Start a server on every host

7. Confirm the device order before you write a split

8. Run the split

9. Measure it

10. TCP against RDMA, one variable at a time

11. Record it in the course format

Verify the capacity claim before loading the target

Validation

Expected outcome

Troubleshooting

Cleanup

What you learned

Check your understanding

Sources for this lesson