Lab: Build and Measure Your Cluster Network

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The cables, switches, tool versions and measured link figures for each track belong here once the validation pass has run this lab on the reference cluster.

Objective

By the end of this lab your machines will resolve each other by name, log in to each other without a password, and have a measured throughput and round-trip figure for every link between them, recorded in your lab notebook with the date. You will know which of your links can carry a jumbo frame end to end, whether an RDMA path exists between any pair, and where every machine reads its models from.

No model is loaded and nothing is split. That is deliberate. Parts 19 to 22 all begin by assuming this groundwork exists, and the commonest way to lose an afternoon in those parts is to debug a distributed engine that is actually suffering from a name that does not resolve, a link that is running at a tenth of its rated speed, or a maximum transmission unit that two machines disagree about. An hour spent here removes all three from the list of suspects for good.

⌂Single-machine path

The primary path needs two or more machines. If you have one, the section The single-machine path below runs every step between two containers on a bridge network Docker creates for you. The throughput numbers mean something different, because a bridge network is not a cable, but the procedure, the scripts, the failure modes and the notebook entries are identical. Do it that way, write down that you did, and everything later in this level still follows.

Architecture

What you are building: a network, not a cluster

• workerMachine Aholds the model library; runs the NFS server
• workerMachine Bmounts the library; runs an iperf3 server
• routerHouse switchnames, addresses, management traffic

• Machine A ⟷ connected to House switchwired Ethernet, default frame size
• Machine B ⟷ connected to House switchwired Ethernet, default frame size
• Machine A ⟷ connected to Machine Bdirect cable where you have one: the interface named in CLUSTER_IFACE

Two paths between the same pair of machines, measured separately. On Track S the direct path is a QSFP cable; on Track M a Thunderbolt cable; on Tracks X and N there may be no direct path at all, which is a perfectly good answer as long as you have measured the one you do have.

Requirements

Everyone needs the lab notebook from Part 1, iperf3 on every machine, and a wired connection between them. The course pins iperf3 3.21 · verified 2026-09-08. Nothing here downloads a model, so the memory floor of 8 GB per machine is really a floor on the machines being usable at all rather than on this lab’s own appetite.

Time. About sixty minutes attended for two machines, and roughly ten minutes more per extra machine, most of which is waiting for iperf3. There is no unattended time. If you intend to set up shared storage as well, add fifteen minutes and the time it takes to copy or re-download your model library once.

What to have ready. Physical access to each machine or a working login on it; the ability to add static entries to your router or to edit /etc/hosts on every machine; and a cable for the direct link if your machines have ports for one.

Track S — NVIDIA DGX Spark

The primary path, and the only track in this course with two machines of the same kind and a documented direct link. Measure both paths: the RJ-45 Ethernet through the house switch, and the direct QSFP cable between the two ConnectX-7 ports.

NVIDIA’s ConnectX-7 networking page shows that each Ethernet interface has a matching RoCE device, with names such as an enp1s0f0np0 paired with a rocep1s0f0. Write both down: the Ethernet name is what you address and set an MTU on, and the RoCE name is what the RDMA verification in task 7 opens. Task 7 is fully available on this track and this is the only track where it is.

Track X — AMD Ryzen AI Max+ 395Partial

These machines have 2.5 gigabit Ethernet and USB4-class ports rather than a RoCE adapter, so the RDMA verification in task 7 has nothing to open unless a RoCE-capable card has been fitted.

Everything except task 7. Two Ryzen AI Max+ 395 machines, or one of them and a machine from another track, connected through your switch, is a complete and useful cluster network for the layer splits Part 19 teaches, and the arithmetic in the networking lesson says 2.5 gigabit Ethernet is comfortable for that.

Jumbo frames are worth setting on this track precisely because the link is the narrowest in the course: fewer frames for the same bytes is a real saving when the model library is being read over it. Verify with task 6 rather than assuming the switch agrees.

Track M — Apple siliconPartial

The course reference cluster has one Mac, so the two-Mac Thunderbolt 5 path in task 7 is written from Apple's and MLX's documentation and has not been run on hardware here.

Tasks 1 to 6, 8 and 9 run on a Mac exactly as written, with two differences noted where they occur: the do-not-fragment flag in task 6 is -D on the BSD ping that macOS ships rather than -M do, and the NFS scripts are Linux-only so a Mac mounts the shared library through the Finder instead.

The Thunderbolt 5 pair, with RDMA on macOS 26.2 or later, is documented in task 7 and marked as documentation rather than validation. If you own two Macs, this is a genuine contribution: run it, record it, and you will have a measurement the course does not.

Track N — NVIDIA desktop or laptop

The second primary path and the one most readers will use, because two ordinary computers on a switch is what most houses have. Everything runs except task 7, unless the desktop has a RoCE-capable adapter fitted, in which case task 7 runs as written on Track S.

If one of your machines is a laptop, measure its wired link and its wireless link separately and keep both numbers. The comparison is the most persuasive line in the whole notebook and it settles the question of whether a wireless cluster member is worth trying, once, permanently.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-18-parallelism-and-networks"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-18-parallelism-and-networks and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Fill in the worksheet and the settings file

Start with the worksheet from the previous lesson, on paper: for every machine, its short name, its class and memory, its operating system and architecture, its interfaces, its roles, and where its models live. Then draw the links and mark each one with what it will carry: per-token traffic, per-request traffic, or load-time traffic only.

Now put the parts a script needs into a settings file. Copy the example and edit it on every machine; only MACHINE_NAME, MACHINE_ROLES and CLUSTER_PEERS differ between them.

RunnableAll tracks

env-example.txt

# Purpose: the settings measure-links.sh, setup-nfs-server.sh, mount-models.sh and
#          record-topology.py all read. Copy this file to `.env` beside them and fill
#          in the lines that are empty. Nothing here is a secret; nothing here should
#          ever be committed, because every value is a fact about your house.
# Platform: all
# Minimum memory: 8 GB
# Assumes: `cp env-example.txt .env` and then an editor. The scripts read it with
#          `set -a; . ./.env; set +a`, so a value with spaces needs quotes.

# ------------------------------------------------------------------- who is who
# A short name for THIS machine, used as the record key in the lab notebook. Use
# something you will still recognise in six months: the track letter and a number
# works well, for example s1, s2, x1, m1, n1.
MACHINE_NAME=

# The roles this machine holds, from the six in the lesson, separated by commas:
# prefill, decode, router, cache, storage, agent.
MACHINE_ROLES=

# The other machines in the cluster, separated by spaces, written as names rather
# than addresses. RFC 8375 reserves anything under `home.arpa` for names that mean
# something inside one house and nothing outside it, so a set of peers written as
#   CLUSTER_PEERS="node-b.home.arpa node-c.home.arpa"
# is exactly the intended use. Add them to /etc/hosts on every machine, or as static
# entries on your router, as the lab describes.
CLUSTER_PEERS=

# The user account the scripts use for ssh. Leave empty to use the account you are
# logged in as, which is the usual case when the same username exists everywhere.
SSH_USER=

# ---------------------------------------------------------------- the measurement
# iperf3's default port. Change it only if something else already listens there.
IPERF_PORT=5201

# Seconds per direction per link. iperf3's own default is 10; 20 gives a steadier
# figure on a link that ramps up slowly, at the cost of a longer run.
IPERF_SECONDS=20

# Parallel streams. One stream can leave a fast link idle because a single TCP flow
# does not always fill it; four is a reasonable default for anything above a gigabit.
IPERF_STREAMS=4

# How many echo requests per link when measuring round-trip time.
PING_COUNT=50

# Where the scripts append their JSON lines. The lab notebook from Part 1.
LABBOOK=labbook.md

# --------------------------------------------------------------------- the links
# The interface that will carry cluster traffic on THIS machine: the direct cable
# rather than the one going to the house switch, where you have both. Find it with
# `ip link show` on Linux or `ifconfig` on macOS.
CLUSTER_IFACE=

# The maximum transmission unit you intend that interface to carry. 1500 is the
# Ethernet default and is always safe; 9000 is the conventional jumbo frame and needs
# every device in the path to agree, which the lab makes you verify.
CLUSTER_MTU=1500

# ------------------------------------------------------------- the model library
# Absolute path to the shared model library on THIS machine: the directory the server
# exports, or the directory a client mounts it at. The same path on every machine
# saves a great deal of confusion later.
MODELS_DIR=

# For the client script only: the machine exporting the library, and the path it
# exports. Leave both empty on the machine that IS the server.
NFS_SERVER=
NFS_EXPORT=

# For the server script only: which clients may mount the export. NFS accepts a name
# pattern or an address range; the lab uses a name pattern so that nothing in this
# file is an address. For example: *.home.arpa
NFS_CLIENTS=

Download env-example.txt75 lines

RunnableAll tracks

on every machine

cp env-example.txt .env

Everything in that file is a fact about your house, which is why it is a file you fill in rather than a value written into a script. Nothing shipped by this course contains a name or an address of yours, and your .env should never be committed anywhere either.

2. Give every machine a stable address and a name

A machine whose address changes is a machine whose cluster breaks at three in the morning. Two steps fix it, and the second is the one people skip.

A stable address. The tidy way is a static lease on your router, sometimes called a DHCP reservation: the machine still asks for an address, and the router always gives it the same one. The alternative is configuring the address on the machine itself, which works and is easier to get wrong. Either is fine; doing neither is not.

A name. Addresses are miserable to type, impossible to remember and appear in logs where you will not recognise them. Give each machine a name under home.arpa, the domain RFC 8375 sets aside for exactly this: the RFC states that “‘home.arpa.’ is designated for non-unique use in residential home networks”, that “a name ending in ‘.home.arpa.’ is not globally unique”, and that queries for such names “MUST NOT be recursively forwarded to servers outside the logical boundaries of the homenet”. Part 7 used the same domain for the household chat service.

There are two ways to publish those names. A static DNS entry on the router is the better one, because every device in the house learns the name. An entry in /etc/hosts on each machine is the quicker one and needs no router that supports it; it is the path this lab assumes, because it also works for the direct cable, which your router knows nothing about.

Pseudocode — not a real command

# /etc/hosts on every machine, one line per peer.
# The address column is whatever the machine's address actually is; the course
# never prints an address of its own.

<address on the house network>   node-a.home.arpa   node-a
<address on the house network>   node-b.home.arpa   node-b

# And, on the two machines joined by a direct cable, the addresses at each end of
# that cable, under names of their own so you can choose the path deliberately:

<address at this end of the cable>   node-a-direct.home.arpa   node-a-direct
<address at the other end>           node-b-direct.home.arpa   node-b-direct

Giving the direct link its own names is the single most useful trick in this lab. Later, when you tell an engine which host to talk to, the name decides which cable the model traffic uses, and you will never again have to wonder which path a disappointing measurement went over.

Check the names resolve, from each machine, before going on.

RunnableAll tracks

every peer answers to its name

ping -c 3 node-b.home.arpa

=Expected result

Three replies with a round-trip time each, and a summary line. If the name does not resolve at all, the /etc/hosts entry is missing or misspelled on the machine you are typing on. If it resolves but nothing answers, the address is wrong or the machine is off.

3. SSH keys and passwordless access

Every distributed launcher in this level starts processes on other machines over SSH, and every one of them assumes it will not be asked for a password. MLX’s distributed documentation, for instance, takes a host file whose entries name each node by an ssh host.

Generate a key if you do not already have one. The ssh-keygen manual lists ed25519 among the possible types for -t and gives it as the default; -C sets the comment that appears beside the key in the file, which is worth using so that a key’s purpose is obvious in a year.

RunnableAll tracks

one key for the cluster

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_cluster -C "local LLM course cluster key"

Then install the public half on each peer. ssh-copy-id is documented as “a script that uses ssh(1) to log into a remote machine” and “enables logins with those keys on the remote server”; -i makes it “use only the key(s) contained in identity_file”.

Fragment — not complete on its own

ssh-copy-id -i ~/.ssh/id_ed25519_cluster.pub "${SSH_USER}@node-b.home.arpa"

⛨Security

A cluster key is a key that opens every machine you own. Give it a passphrase and use an agent, keep it out of any directory that is synchronised to a cloud service, and never commit it. If you already have a key you use for code hosting, make a separate one for this: the blast radius of a key is the set of things it opens, and there is no reason for those two sets to be the same.

Confirm that a login needs nothing from you.

Fragment — not complete on its own

ssh -i ~/.ssh/id_ed25519_cluster "${SSH_USER}@node-b.home.arpa" uname -a

4. Start an iperf3 server on every peer

iperf3 measures between exactly two endpoints: one runs as a server, the other as a client. The manual gives the defaults you should know: the port is 5201 and a test runs for ten seconds unless --time says otherwise.

Leave a server running on every machine other than the one you are measuring from. It listens, uses nothing while idle, and having them all up means one pass of the script measures every link.

RunnableAll tracks

on each peer, in a terminal you can leave open

iperf3 --server --port 5201

=Expected result

A line saying the server is listening on 5201, and then nothing until a client connects. Each completed test prints an interval report and a summary. Leave it running for the whole lab.

5. Measure every link, in both directions

Throughput is not symmetric, and the direction that matters is the one your cluster will use. The script measures both, adds a round-trip figure from ping, and appends one JSON line per link to your notebook.

RunnableAll tracks

measure-links.sh

#!/usr/bin/env bash
# Purpose: measure throughput in both directions and round-trip time to every named
#          cluster peer, then append one JSON line per link to the lab notebook so the
#          numbers are dated, attributed to a machine and never retyped by hand
# Platform: all (spark, strix, mac, nvidia); each peer must already be running
#           `iperf3 --server` on the port in .env, which task 4 of the lab starts
# Minimum memory: 8 GB
# Assumes: iperf3 and python3 on PATH, an .env beside this script copied from
#          env-example.txt with MACHINE_NAME and CLUSTER_PEERS filled in, and every
#          peer reachable by the name written in CLUSTER_PEERS. Raw iperf3 output is
#          kept under link-results/ so a surprising number can be re-read later.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${MACHINE_NAME:=}"
: "${CLUSTER_PEERS:=}"
: "${IPERF_PORT:=5201}"
: "${IPERF_SECONDS:=20}"
: "${IPERF_STREAMS:=4}"
: "${PING_COUNT:=50}"
: "${LABBOOK:=${HERE}/labbook.md}"
: "${CLUSTER_IFACE:=}"

RESULTS_DIR="${RESULTS_DIR:-${HERE}/link-results}"

if [ -z "$MACHINE_NAME" ]; then
    echo "MACHINE_NAME is empty in ${ENV_FILE}. Give this machine a short name first." >&2
    exit 1
fi

if [ -z "$CLUSTER_PEERS" ]; then
    echo "CLUSTER_PEERS is empty in ${ENV_FILE}. Nothing to measure." >&2
    echo "On the single-machine path, set it to the container name of the iperf3 server." >&2
    exit 1
fi

for tool in iperf3 ping python3; do
    if ! command -v "$tool" >/dev/null 2>&1; then
        echo "${tool} is not on PATH." >&2
        exit 1
    fi
done

mkdir -p "$RESULTS_DIR"

read -r -a PEERS <<< "$CLUSTER_PEERS"

# Turns one peer's raw output into a single JSON line for the notebook. Kept in Python
# because iperf3's JSON is nested and awk would be a poor place to learn that.
summarise() {
    # summarise <peer> <forward-json> <reverse-json> <ping-text>
    python3 - "$1" "$2" "$3" "$4" <<'PY'
import json
import os
import re
import sys
from datetime import date, datetime, timezone
from pathlib import Path

peer, forward_path, reverse_path, ping_path = sys.argv[1:5]


def read_iperf(path):
    """Sender and receiver bit rates from one iperf3 --json run, or None."""
    try:
        data = json.loads(Path(path).read_text(encoding="utf-8"))
    except (OSError, ValueError):
        return None
    if data.get("error"):
        return {"error": str(data["error"])}
    end = data.get("end", {})
    sent = end.get("sum_sent", {})
    received = end.get("sum_received", {})
    out = {}
    if "bits_per_second" in sent:
        out["sent_bits_per_second"] = round(float(sent["bits_per_second"]))
    if "bits_per_second" in received:
        out["received_bits_per_second"] = round(float(received["bits_per_second"]))
    if sent.get("retransmits") is not None:
        out["retransmits"] = sent["retransmits"]
    return out or None


def read_ping(path):
    """min/avg/max round trip in milliseconds, from either the Linux or the BSD summary."""
    try:
        text = Path(path).read_text(encoding="utf-8")
    except OSError:
        return None
    m = re.search(r"=\s*([\d.]+)/([\d.]+)/([\d.]+)", text)
    loss = re.search(r"([\d.]+)%\s*packet loss", text)
    out = {}
    if m:
        out["rtt_min_ms"] = float(m.group(1))
        out["rtt_avg_ms"] = float(m.group(2))
        out["rtt_max_ms"] = float(m.group(3))
    if loss:
        out["packet_loss_percent"] = float(loss.group(1))
    return out or None


record = {
    "lab": "part-18/build-and-measure-your-cluster-network",
    "record": "link",
    "date": date.today().isoformat(),
    "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
    "from_machine": os.environ.get("MACHINE_NAME", ""),
    "to_peer": peer,
    "interface": os.environ.get("CLUSTER_IFACE", "") or None,
    "iperf3_seconds": int(os.environ.get("IPERF_SECONDS", "0") or 0),
    "iperf3_streams": int(os.environ.get("IPERF_STREAMS", "0") or 0),
    "forward": read_iperf(forward_path),
    "reverse": read_iperf(reverse_path),
    "ping": read_ping(ping_path),
}
print(json.dumps(record, sort_keys=True))
PY
}

echo "==> measuring ${#PEERS[@]} link(s) from ${MACHINE_NAME}"
echo "    results kept in ${RESULTS_DIR}, notebook lines appended to ${LABBOOK}"
echo ""

FAILURES=0

for peer in "${PEERS[@]}"; do
    safe="${peer//[^A-Za-z0-9._-]/_}"
    forward="${RESULTS_DIR}/${safe}-forward.json"
    reverse="${RESULTS_DIR}/${safe}-reverse.json"
    pingout="${RESULTS_DIR}/${safe}-ping.txt"

    echo "--> ${peer}: sending"
    if ! iperf3 --client "$peer" --port "$IPERF_PORT" --time "$IPERF_SECONDS" \
        --parallel "$IPERF_STREAMS" --json > "$forward"; then
        echo "    iperf3 failed sending to ${peer}; is 'iperf3 --server' running there?" >&2
        FAILURES=$((FAILURES + 1))
    fi

    echo "--> ${peer}: receiving"
    if ! iperf3 --client "$peer" --port "$IPERF_PORT" --time "$IPERF_SECONDS" \
        --parallel "$IPERF_STREAMS" --reverse --json > "$reverse"; then
        echo "    iperf3 failed receiving from ${peer}." >&2
        FAILURES=$((FAILURES + 1))
    fi

    echo "--> ${peer}: round trip"
    if ! ping -c "$PING_COUNT" -q "$peer" > "$pingout" 2>&1; then
        echo "    ping failed for ${peer}; the name may not resolve." >&2
        FAILURES=$((FAILURES + 1))
    fi

    summarise "$peer" "$forward" "$reverse" "$pingout" | tee -a "$LABBOOK"
    echo ""
done

if [ "$FAILURES" -gt 0 ]; then
    echo "${FAILURES} step(s) failed. The notebook lines above show which fields are missing." >&2
    exit 1
fi

echo "All links measured. Read the lines back with:"
echo "  grep '\"record\": \"link\"' ${LABBOOK}"

Download measure-links.sh174 lines

RunnableAll tracks

from the machine you are sitting at

bash measure-links.sh

It uses --client to connect, --port for the port, --time for the duration, --parallel to open several streams at once, --reverse to turn the test around so the server sends, and --json so the result is parsed rather than eyeballed. Several streams matter more than people expect: one connection does not always fill a fast link, and a single-stream result on a 10 gigabit link can understate it badly.

Run it on each machine in turn, so that every link is measured from both ends. Then, if you gave your direct cable its own names in task 2, set CLUSTER_PEERS to those names and run it again. Two rows in your notebook for the same pair of machines, over two different cables, is the comparison the whole networking lesson was building towards.

Pending validationYour links, once this task is done

Link	Direction	Throughput (gigabits/s)	Round trip, average (ms)	Retransmits
A to B over the house switch	forward	from your run	from your run	from your run
A to B over the house switch	reverse	from your run	from your run	from your run
A to B over the direct cable	forward	from your run	from your run	from your run
A to B over the direct cable	reverse	from your run	from your run	from your run

your machines, named in the notebook record · iperf3 the version iperf3 --version prints on your machines · not applicable; this is a network measurement, not applicable · 1 tokens of context · the date of your run

The shape of the table your run fills in. The course does not print numbers from someone else's house here: a link measurement is only meaningful for the cable, the switch and the machines that produced it. Compare each figure against the rated speed from the networking lesson and write down the ratio, because that ratio is what tells you whether something is wrong.

✕Common mistake

A result far below the rated speed usually has a boring cause, and the order to check is: the wrong interface was used because a name resolved to the house network rather than the direct cable; the link negotiated a lower speed than the cable supports; a single stream did not fill the link; or something else on the network was busy. Only after those four is it worth suspecting the hardware.

6. Set and verify the maximum transmission unit

Jumbo frames are worth having on a link that carries model traffic, and only if every device in the path agrees. Look first at what you have.

RunnableAll tracks

Linux: what MTU does each interface carry?

ip link show

On macOS the same information comes from ifconfig with no arguments, which lists every interface with its MTU.

Raise it on the interface that will carry cluster traffic. On Linux, the ip link set synopsis takes the device and then mtu with a number.

Fragment — not complete on its own

sudo ip link set "${CLUSTER_IFACE}" mtu 9000

Now verify it, which is the step that separates a working jumbo link from a mysterious one. Send a ping carrying a payload just under the target size, with fragmentation forbidden. The Linux ping manual documents -M do as “set DF flag but subject to PMTU checks by kernel, packets too large will be rejected”, and -s as specifying “the number of data bytes to be sent”. The payload for a 9,000-byte unit is 8,972, because 20 bytes of IP header and 8 bytes of ICMP header travel with it.

RunnableTrack S · DGX Spark

Linux: does the whole path carry a jumbo frame?

ping -M do -s 8972 -c 4 node-b.home.arpa

macOS ships a BSD ping rather than the Linux one, and it spells the same idea differently: the FreeBSD manual, which Darwin’s version follows, documents -D as “disable fragmentation”. Confirm it with man ping on your own Mac before relying on it, because this is the one command in the lab where the course is reading a close relative of your manual rather than your manual itself.

RunnableTrack M · Apple silicon

macOS: the same test, BSD spelling

ping -D -s 8972 -c 4 node-b.home.arpa

=Expected result

Four replies means the entire path carries 9,000-byte frames and you can leave the setting in place. A message about the message being too long, or silence where the smaller ping succeeded, means something in the path is smaller than you think: put the interface back to 1500 and either find the device that disagrees or accept the default. Repeat the test with -s 1472 to confirm that a default-sized frame still passes, which proves the failure is about size and not about reachability.

⚠Warning

Changing an interface’s MTU can drop the link while it reconfigures. Do it from the console or over a different interface than the one you are changing, never over an SSH session that runs on the interface itself, or you will lose the session and the machine in the same second. The change made with ip link set does not survive a reboot, which in this case is a safety feature: reboot to undo it.

7. Verify the RDMA path, where you have one

RDMA falls back to TCP silently. llama.cpp’s RPC README says the transport “is negotiated during the initial handshake” and that “the connection falls back to TCP unless both peers can use RDMA”, which means a cluster with a broken RDMA path looks exactly like a cluster that is simply slower than you hoped. Check it here, once, rather than wondering later.

Track S, and Track N with a RoCE-capable adapter. The perftest package provides ib_write_bw, which its README describes as a “bandwidth test with RDMA write transactions”. The package’s tests all take the same shape: the test with its options on the server, and the same test with the same options plus the server’s address on the client, with the README warning that “the SAME OPTIONS must be passed to both server and client”.

Fragment — not complete on its own

# on the machine at one end of the direct cable
ib_write_bw

Fragment — not complete on its own

# on the machine at the other end, naming the first one
ib_write_bw node-a-direct.home.arpa

A table of message sizes and bandwidths means the RDMA path is live. An error about finding no device means the RoCE device is not present or not up, and NVIDIA’s ConnectX-7 networking page is where the interface and RoCE device names come from.

Track M, two Macs over Thunderbolt 5. This is documented rather than validated here, because the course’s reference cluster has one Mac. MLX’s distributed documentation states that “starting from macOS 26.2, RDMA over thunderbolt is available and enables low-latency communication between Macs with thunderbolt 5”, and that enabling it “cannot be done remotely even with sudo. In fact, it has to be done in macOS recovery”, where rdma_ctl enable is run. llama.cpp’s RPC README describes the same requirement for its macOS provider: “macOS 26.2 or later, with RDMA enabled once from macOS Recovery via rdma_ctl enable”. Setting up the Thunderbolt link itself is a per-cable procedure that MLX’s documentation describes: “disable the thunderbolt bridge interface”, find the interfaces for each cable on both machines, and “set up a unique subnetwork connecting the two nodes for the corresponding interfaces”. Part 21 does this in full.

Tracks X and N without a RoCE adapter. There is no RDMA path to verify, and that is a finding rather than a failure. Write “no RDMA path” in the notebook and move on; the layer splits Part 19 teaches do not need one.

8. Put the model library on shared storage

Two machines that downloaded a model separately will eventually disagree about which model they are running, usually while you are benchmarking. One library, read by everyone, removes the whole category.

On the machine that holds the library, run the server script. It prints its plan and changes nothing until you agree.

RunnableAll tracks

setup-nfs-server.sh

#!/usr/bin/env bash
# Purpose: export the shared model library read-only to the other machines in the
#          cluster, so that every worker loads the same bytes and no model is ever
#          downloaded twice. Prints exactly what it would do and changes nothing
#          until you pass --apply
# Platform: spark, strix, nvidia (Linux with the Ubuntu NFS server packages).
#           Track M is a client in this lab, not a server; see the lab page.
# Minimum memory: 8 GB
# Assumes: an .env beside this script with MODELS_DIR and NFS_CLIENTS filled in,
#          sudo available, and the machine reachable by name from the clients. It
#          adds one line to /etc/exports and never removes or rewrites another.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"
APPLY=0

for arg in "$@"; do
    case "$arg" in
        --apply) APPLY=1 ;;
        -h|--help)
            echo "Usage: $0 [--apply]"
            echo "  Without --apply it prints the plan and changes nothing."
            exit 0
            ;;
        *)
            echo "Unknown argument: ${arg}" >&2
            exit 1
            ;;
    esac
done

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${MODELS_DIR:=}"
: "${NFS_CLIENTS:=}"

if [ "$(uname -s)" != "Linux" ]; then
    echo "This script sets up the Linux NFS server packages and $(uname -s) is not Linux." >&2
    echo "On macOS, be the client instead: run mount-models.sh against a Linux server." >&2
    exit 1
fi

if [ -z "$MODELS_DIR" ]; then
    echo "MODELS_DIR is empty in ${ENV_FILE}." >&2
    exit 1
fi

if [ ! -d "$MODELS_DIR" ]; then
    echo "MODELS_DIR is ${MODELS_DIR}, which is not a directory on this machine." >&2
    exit 1
fi

if [ -z "$NFS_CLIENTS" ]; then
    echo "NFS_CLIENTS is empty in ${ENV_FILE}. Name the machines allowed to mount this." >&2
    echo "A name pattern such as the one in env-example.txt is the intended form." >&2
    exit 1
fi

# Read-only for the workers, because a worker never writes to the model library.
# sync is the Ubuntu documentation's default recommendation; no_subtree_check is what
# that documentation shows for an ordinary directory export.
EXPORT_LINE="${MODELS_DIR} ${NFS_CLIENTS}(ro,sync,no_subtree_check)"

echo "Plan for this machine:"
echo ""
echo "  1. install the server package:"
echo "       sudo apt install nfs-kernel-server"
echo ""
echo "  2. append this line to /etc/exports, if it is not there already:"
echo "       ${EXPORT_LINE}"
echo ""
echo "  3. apply the export table and make sure the service is running:"
echo "       sudo exportfs -a"
echo "       sudo systemctl start nfs-kernel-server.service"
echo ""
echo "  The export is read-only. root_squash, which maps a client's root to an"
echo "  unprivileged user, stays at its default: nothing here enables no_root_squash."
echo ""

if [ "$APPLY" -ne 1 ]; then
    echo "Nothing was changed. Re-run with --apply to carry out the plan above."
    exit 0
fi

echo "==> installing nfs-kernel-server"
sudo apt install nfs-kernel-server

if sudo grep -Fqx -- "$EXPORT_LINE" /etc/exports 2>/dev/null; then
    echo "==> /etc/exports already contains this export; leaving it alone"
else
    echo "==> appending the export line to /etc/exports"
    printf '%s\n' "$EXPORT_LINE" | sudo tee -a /etc/exports >/dev/null
fi

echo "==> applying the export table"
sudo exportfs -a

echo "==> starting the service"
sudo systemctl start nfs-kernel-server.service

echo ""
echo "Exported. Check what this machine now offers with:"
echo "  cat /etc/exports"
echo "Then run mount-models.sh on each client."

Download setup-nfs-server.sh113 lines

RunnableAll tracks

on the machine holding the models: look first

bash setup-nfs-server.sh

⚠Warning

The --apply run installs a package, appends a line to /etc/exports and starts a network service that offers a directory to other machines. Read the printed plan before running it, and confirm that NFS_CLIENTS in your .env names only machines you own. The export is read-only and leaves root_squash at its default; the Ubuntu documentation describes no_root_squash, which this lab does not use, as allowing “root-owned files to be modified by any client system’s root user”.

RunnableAll tracks

on the machine holding the models: apply it

bash setup-nfs-server.sh --apply

Then on each machine that reads the library:

RunnableAll tracks

mount-models.sh

#!/usr/bin/env bash
# Purpose: mount the cluster's shared model library on this machine, so that every
#          worker loads the same weights from one place. Prints exactly what it would
#          do and changes nothing until you pass --apply
# Platform: spark, strix, nvidia (Linux with the Ubuntu NFS client package).
#           Track M mounts the same export through the Finder; see the lab page.
# Minimum memory: 8 GB
# Assumes: an .env beside this script with NFS_SERVER, NFS_EXPORT and MODELS_DIR
#          filled in, sudo available, and setup-nfs-server.sh already run on the
#          machine holding the library. It creates one directory and mounts one
#          export; it never writes to /etc/fstab.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"
APPLY=0

for arg in "$@"; do
    case "$arg" in
        --apply) APPLY=1 ;;
        -h|--help)
            echo "Usage: $0 [--apply]"
            echo "  Without --apply it prints the plan and changes nothing."
            exit 0
            ;;
        *)
            echo "Unknown argument: ${arg}" >&2
            exit 1
            ;;
    esac
done

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${NFS_SERVER:=}"
: "${NFS_EXPORT:=}"
: "${MODELS_DIR:=}"

if [ "$(uname -s)" != "Linux" ]; then
    echo "This script uses the Linux NFS client packages and $(uname -s) is not Linux." >&2
    echo "On macOS, mount the same export from the Finder with Go > Connect to Server," >&2
    echo "then record the mount point in your notebook by hand." >&2
    exit 1
fi

for name in NFS_SERVER NFS_EXPORT MODELS_DIR; do
    if [ -z "${!name}" ]; then
        echo "${name} is empty in ${ENV_FILE}." >&2
        exit 1
    fi
done

if mountpoint -q "$MODELS_DIR" 2>/dev/null; then
    echo "${MODELS_DIR} is already a mount point. Nothing to do."
    exit 0
fi

if [ -d "$MODELS_DIR" ] && [ -n "$(ls -A "$MODELS_DIR" 2>/dev/null)" ]; then
    echo "${MODELS_DIR} already exists and is not empty." >&2
    echo "Mounting over it would hide what is there. Move it aside or choose another" >&2
    echo "path in .env before continuing." >&2
    exit 1
fi

echo "Plan for this machine:"
echo ""
echo "  1. install the client package:"
echo "       sudo apt install nfs-common"
echo ""
echo "  2. create the mount point:"
echo "       sudo mkdir -p ${MODELS_DIR}"
echo ""
echo "  3. mount the shared library:"
echo "       sudo mount ${NFS_SERVER}:${NFS_EXPORT} ${MODELS_DIR}"
echo ""
echo "  The export is read-only on the server side, so nothing on this machine can"
echo "  change the library by accident. The mount does not survive a reboot; add an"
echo "  /etc/fstab entry yourself once you are happy with it, as the lab page says."
echo ""

if [ "$APPLY" -ne 1 ]; then
    echo "Nothing was changed. Re-run with --apply to carry out the plan above."
    exit 0
fi

echo "==> installing nfs-common"
sudo apt install nfs-common

echo "==> creating ${MODELS_DIR}"
sudo mkdir -p "$MODELS_DIR"

echo "==> mounting ${NFS_SERVER}:${NFS_EXPORT}"
sudo mount "${NFS_SERVER}:${NFS_EXPORT}" "$MODELS_DIR"

echo ""
echo "Mounted. Confirm the machine can read the library with:"
echo "  ls ${MODELS_DIR}"
echo "Unmount again with:"
echo "  sudo umount ${MODELS_DIR}"

Download mount-models.sh107 lines

RunnableAll tracks

on each worker: look, then apply

bash mount-models.sh

The mount does not survive a reboot. When you are satisfied, add the equivalent line to /etc/fstab yourself, in the form the Ubuntu documentation shows, so that the library is there after a restart. A Mac mounts the same export through the Finder rather than with these scripts.

✦Tip

Use the same path on every machine. A model at one path on one machine and another path on another is a configuration difference that every distributed launcher will find, at the least convenient moment. MODELS_DIR in .env exists to make this easy to keep consistent.

9. Record the topology

The last step turns everything above into one durable line per machine.

RunnableAll tracks

record-topology.py

"""Record this machine's place in the cluster, and its measured links, in the lab notebook.

Purpose: capture the topology the rest of Level 4 assumes - which machine this is, what
         roles it holds, which interfaces it has and at what maximum transmission unit,
         whether an RDMA device is present, where the shared model library is, and the
         throughput and round-trip figures measure-links.sh produced - as one JSON line
         so that no number in a later part has to be remembered rather than looked up.
Platform: all (spark, strix, mac, nvidia). Probes that do not apply to a machine are
          absent from the record rather than causing a failure.
Minimum memory: 8 GB
Assumes: python3.9 or newer, and an .env beside this file copied from env-example.txt.
         Run measure-links.sh first if you want the link figures folded in; without it
         the topology is still recorded and the links are simply empty.

Usage: python3 record-topology.py --labbook labbook.md
       python3 record-topology.py --print          (show the record, write nothing)
       python3 record-topology.py --labbook labbook.md --note "jumbo frames enabled"
"""
import argparse
import json
import os
import platform
import re
import shutil
import subprocess
import sys
from datetime import date, datetime, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent


def read_env(path):
    """A small KEY=VALUE reader, so the script and the shell scripts share one .env."""
    values = {}
    try:
        text = Path(path).read_text(encoding="utf-8")
    except OSError:
        return values
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        value = value.strip()
        if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
            value = value[1:-1]
        values[key.strip()] = value
    return values


def run(cmd, timeout=20):
    """Run a command and return its stdout, or None if it is unavailable or fails."""
    if shutil.which(cmd[0]) is None:
        return None
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
    except (OSError, subprocess.SubprocessError):
        return None
    if out.returncode != 0:
        return None
    return out.stdout


def interfaces():
    """Every network interface with its maximum transmission unit and link state."""
    found = {}
    sysnet = Path("/sys/class/net")
    if sysnet.is_dir():
        for entry in sorted(sysnet.iterdir()):
            info = {}
            for field, name in (("mtu", "mtu"), ("operstate", "state"), ("speed", "speed_mbit")):
                try:
                    raw = (entry / field).read_text(encoding="utf-8").strip()
                except OSError:
                    continue
                if name in ("mtu", "speed_mbit"):
                    try:
                        info[name] = int(raw)
                    except ValueError:
                        continue
                else:
                    info[name] = raw
            if info:
                found[entry.name] = info
        return found

    # macOS and other BSD systems: ifconfig with no arguments lists every interface.
    text = run(["ifconfig"])
    if not text:
        return found
    for line in text.splitlines():
        match = re.match(r"^(?P<name>[A-Za-z0-9._-]+):\s.*\bmtu\s+(?P<mtu>\d+)", line)
        if match:
            found[match.group("name")] = {"mtu": int(match.group("mtu"))}
    return found


def rdma():
    """Whatever the machine will say about an RDMA path, per platform."""
    info = {}
    ib = Path("/sys/class/infiniband")
    if ib.is_dir():
        info["infiniband_devices"] = sorted(p.name for p in ib.iterdir())
    if shutil.which("ib_write_bw"):
        info["ib_write_bw_present"] = True
    if sys.platform == "darwin":
        info["rdma_ctl_present"] = shutil.which("rdma_ctl") is not None
        version = run(["sw_vers", "-productVersion"])
        if version:
            info["macos"] = version.strip()
    return info or None


def storage(models_dir):
    """Where the shared model library is on this machine, and whether it is a mount."""
    if not models_dir:
        return None
    path = Path(models_dir)
    info = {"path": str(path), "exists": path.is_dir()}
    if path.is_dir():
        try:
            info["is_mount_point"] = path.is_mount()
        except OSError:
            pass
        try:
            info["entries"] = len(list(path.iterdir()))
        except OSError:
            info["entries"] = None
    return info


def link_results(results_dir):
    """Fold in whatever measure-links.sh left behind, without re-running anything."""
    directory = Path(results_dir)
    if not directory.is_dir():
        return {}
    links = {}
    for forward in sorted(directory.glob("*-forward.json")):
        peer = forward.name[: -len("-forward.json")]
        entry = {}
        for label, filename in (("forward", forward),
                                ("reverse", directory / f"{peer}-reverse.json")):
            try:
                data = json.loads(Path(filename).read_text(encoding="utf-8"))
            except (OSError, ValueError):
                continue
            sent = data.get("end", {}).get("sum_sent", {})
            if "bits_per_second" in sent:
                entry[f"{label}_bits_per_second"] = round(float(sent["bits_per_second"]))
        ping_file = directory / f"{peer}-ping.txt"
        try:
            ping_text = ping_file.read_text(encoding="utf-8")
        except OSError:
            ping_text = ""
        match = re.search(r"=\s*([\d.]+)/([\d.]+)/([\d.]+)", ping_text)
        if match:
            entry["rtt_avg_ms"] = float(match.group(2))
        if entry:
            links[peer] = entry
    return links


def build_record(env, results_dir, note):
    peers = [p for p in env.get("CLUSTER_PEERS", "").split() if p]
    roles = [r.strip() for r in env.get("MACHINE_ROLES", "").split(",") if r.strip()]
    return {
        "lab": "part-18/build-and-measure-your-cluster-network",
        "record": "topology",
        "date": date.today().isoformat(),
        "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
        "machine": env.get("MACHINE_NAME", ""),
        "roles": roles,
        "peers": peers,
        "os": platform.platform(),
        "arch": platform.machine(),
        "python": platform.python_version(),
        "cluster_interface": env.get("CLUSTER_IFACE", "") or None,
        "intended_mtu": int(env["CLUSTER_MTU"]) if env.get("CLUSTER_MTU", "").isdigit() else None,
        "interfaces": interfaces(),
        "rdma": rdma(),
        "model_library": storage(env.get("MODELS_DIR", "")),
        "links": link_results(results_dir),
        "note": note,
    }


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default=None,
                        help="append one JSON line to this file (for example labbook.md)")
    parser.add_argument("--env", default=str(HERE / ".env"),
                        help="the settings file to read (default: .env beside this script)")
    parser.add_argument("--results-dir", default=str(HERE / "link-results"),
                        help="where measure-links.sh left its raw iperf3 and ping output")
    parser.add_argument("--note", default="", help="free text kept with the record")
    parser.add_argument("--print", dest="show", action="store_true",
                        help="print the record and write nothing")
    args = parser.parse_args()

    env = read_env(args.env)
    # Anything already exported in the shell wins over the file, so a one-off run can
    # override a value without editing .env.
    for key in ("MACHINE_NAME", "MACHINE_ROLES", "CLUSTER_PEERS", "CLUSTER_IFACE",
                "CLUSTER_MTU", "MODELS_DIR"):
        if os.environ.get(key):
            env[key] = os.environ[key]

    if not env.get("MACHINE_NAME"):
        print("MACHINE_NAME is not set. Fill it in in .env, or export it, so the record "
              "can be told apart from the other machines'.", file=sys.stderr)
        return 1

    record = build_record(env, args.results_dir, args.note)
    print(json.dumps(record, indent=2, sort_keys=True))

    if args.show or not args.labbook:
        if not args.labbook:
            print("\nNothing written: pass --labbook labbook.md to record this.")
        return 0

    path = Path(args.labbook)
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    print(f"\nrecorded in {path}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Download record-topology.py230 lines

RunnableAll tracks

on every machine

python3 record-topology.py --labbook labbook.md --note "first cluster network build"

It records the machine’s name and roles, its operating system and architecture, every interface with its MTU, whether an RDMA device is present, where the model library is and whether it is a mount, and the throughput and round-trip figures the previous tasks produced. Run it on every machine. Together the records describe your cluster completely enough that Part 19 can be started from the notebook alone.

The single-machine path

One computer, two containers, the same procedure. Docker’s bridge documentation gives the reason this uses a network of its own rather than the default one: “user-defined bridges provide automatic DNS resolution between containers”, so that on such a network “containers can resolve each other by name or alias”, whereas on the default bridge they “can only access each other by IP addresses”. Name resolution is the part of a cluster this path exists to rehearse.

RunnableAll tracks

iperf3.Dockerfile

# Purpose: a minimal image holding iperf3 and ping, so that the single-machine path of
#          this lab can run the same measurement procedure between two containers that
#          the multi-machine path runs between two machines
# Platform: all (spark, strix, nvidia with Docker Engine; Track M with Docker Desktop
#           or Podman). The base image is multi-architecture, so it builds on aarch64.
# Minimum memory: 8 GB
# Assumes: nothing but a working container runtime and a network connection to fetch
#          the packages once. Built by compose.yaml; there is no reason to build it
#          by hand.

FROM ubuntu:24.04

RUN apt-get update \
    && apt-get install --yes --no-install-recommends iperf3 iputils-ping \
    && rm -rf /var/lib/apt/lists/*

# No entrypoint: compose.yaml gives each container its own command, one serving and
# one idling until you exec into it.
CMD ["iperf3", "--version"]

Download iperf3.Dockerfile19 lines

RunnableAll tracks

compose.yaml

# Purpose: the single-machine path of this lab. Two containers on one user-defined
#          bridge network, one running an iperf3 server and one idle so you can exec
#          into it and run the client, which lets you rehearse the whole measurement
#          procedure - including resolving a peer by name - on one computer.
# Platform: spark, strix, nvidia (Docker Engine with the Compose plugin). Track M runs
#           the same file under Docker Desktop or Podman; see the lab page.
# Minimum memory: 8 GB
# Assumes: iperf3.Dockerfile beside this file. Nothing is published to the host: the
#          two containers can reach each other and nothing else can reach them.
#
#   docker compose up --build --detach
#   docker compose exec client iperf3 --client server --port 5201 --time 10 --json
#   docker compose down
#
# Docker's bridge documentation is the reason this uses a network of its own rather
# than the default bridge: on a user-defined bridge, containers resolve each other by
# name, which is the part of a real cluster this path is here to rehearse.

name: cluster-lab

services:
  server:
    build:
      context: .
      dockerfile: iperf3.Dockerfile
    restart: unless-stopped
    # The client reaches this container as "server", the service name, which Compose
    # publishes as a network alias on the bridge below.
    command: ["iperf3", "--server", "--port", "5201"]
    networks:
      - cluster-lab

  client:
    build:
      context: .
      dockerfile: iperf3.Dockerfile
    depends_on:
      - server
    # Idle. You exec into this container to run the client side, exactly as you would
    # log in to the second machine on the multi-machine path.
    command: ["sleep", "infinity"]
    networks:
      - cluster-lab

networks:
  cluster-lab:
    driver: bridge

Download compose.yaml47 lines

RunnableAll tracks

build and start both containers

docker compose up --build --detach

RunnableAll tracks

measure the link between them, by name

docker compose exec client iperf3 --client server --port 5201 --time 10 --json

The client reaches the server as server, its service name, which Compose publishes as an alias on the bridge. That is the same thing that happens when node-b.home.arpa resolves on a real network, and when it stops happening your cluster stops working, which is the lesson.

Work through tasks 1, 4, 5 and 9 this way, setting CLUSTER_PEERS to the service name. Task 2 has no equivalent, because Docker did the naming for you; task 3 has none, because there is no second machine to log in to; task 6 can be read but not usefully performed, because both ends of a bridge network are configured by the same kernel and cannot disagree; and task 7 has no RDMA path.

⌂Single-machine path

What this path measures is a loopback through a virtual bridge, not a cable, so its throughput figure says something about your kernel rather than about any network you might build. Record it as such. What it does teach, completely, is the procedure and the shape of the notebook, and those are the parts that transfer to two machines unchanged.

Use a per-link validation sheet

Make one row for every directed link: source host/interface, destination host/interface, address, MTU, measured latency, measured bulk throughput and observed transport. Test both directions; asymmetric routing or a configuration error can make them differ. Keep management and data networks in separate rows even when they connect the same hosts.

Run the reachability and SSH checks before bulk transfer, and bulk transfer before RDMA tests. If a stage fails, preserve the first error and both endpoint configurations. Do not change MTU and routing simultaneously. After an MTU change, repeat the full-path packet test, not only the local interface listing. Use the lesson’s dry-run setup scripts before applying storage or network changes.

For shared storage, verify that a client can read the intended model files with the correct user and that permissions do not expose unrelated directories. Separate model-loading throughput from the network’s standalone test. After cleanup, restore any temporary interface settings and confirm the management connection still works. The single-machine container path validates the procedural roles and measurement format; label physical cable bandwidth, RDMA and independent-host failure as not run. Keep the topology file and raw link results for every later cluster comparison.

Validation

Six checks. Each is one command and a clear answer.

Names resolve from every machine. ping -c 3 node-b.home.arpa from each machine to each other, including the direct-cable names where you made them.

Logins need nothing from you. ssh to each peer with the cluster key runs a command and returns, with no prompt.

Every link has two numbers. Search the notebook for the link records and confirm there is one per direction per link.

RunnableAll tracks

how many links did you measure?

grep -c '"record": "link"' labbook.md

The maximum transmission unit is confirmed, not assumed. A do-not-fragment ping at the size you configured succeeds, or you set the interface back and wrote down why.

The library is readable everywhere. ls in MODELS_DIR on each worker shows the same contents.

Every machine has a topology record. One per machine, with its own name.

RunnableAll tracks

how many machines are described?

grep -c '"record": "topology"' labbook.md

Expected outcome

Your notebook contains one topology record per machine and one link record per direction per link, all dated. You can state, from the notebook rather than from memory, which of your links is fastest, what the ratio is between its measured throughput and its rated speed, which links carry a jumbo frame, whether any pair of your machines has an RDMA path, and where every machine reads its models from.

You should also be able to answer one design question: which single link in your house would you put a model split across, and which link would you never. If the two paths between the same pair of machines gave noticeably different numbers, you now know why the naming trick in task 2 was worth the five minutes.

Troubleshooting

The name resolves on one machine and not another. /etc/hosts is per-machine. Every machine needs every peer’s line, including its own. A router DNS entry avoids this and is worth the setup if your router supports it.

iperf3 connects and reports a rate far below what the link should do. Check which interface the traffic actually used, by measuring against the direct-cable name rather than the house-network name. Then try more streams with --parallel. Then check whether the link negotiated the speed you expect: ip link show reports the state, and on many machines the interface’s speed is readable under /sys/class/net.

iperf3 refuses to connect at all. The server is not running on the peer, is listening on a different port, or a firewall is in the way. The client’s error names which.

The jumbo ping fails but the small one succeeds. Something in the path carries less than you configured. On a direct cable, one of the two ends was not changed. Through a switch, the switch is the usual answer, and many inexpensive switches do not carry jumbo frames at all. Put both ends back to the default rather than leaving one raised.

A distributed tool later hangs partway through loading a model. This is the MTU symptom, and it is why task 6 exists. Small exchanges succeed and the first large one disappears.

ssh-copy-id asks for a password every time and never stops. The account or the machine name is wrong, or the peer’s SSH configuration refuses key authentication. Try a plain ssh login first and read what it says.

The NFS mount succeeds but the directory is empty. You mounted over a directory that already had contents, or the server is exporting a different directory than you think. cat /etc/exports on the server settles it.

Everything works but the numbers move between runs. Something else in the house is using the network, or a laptop is on wireless. Measure again on a quiet network, and if a machine is wireless, measure it wired and record both.

Cleanup

Nothing here needs to be undone, and most of it you want to keep. Two things are worth reversing if you were only trying them out.

The MTU change made with ip link set disappears on reboot. To undo it now, set the interface back to 1500 the same way.

To stop sharing the model library, unmount it on each client with umount, then remove the line this lab added to /etc/exports on the server and run exportfs -a again. Stop the NFS service if you no longer want it running.

On the single-machine path, remove the containers and the network Compose created:

RunnableAll tracks

tear down the single-machine path

docker compose down

Leave the iperf3 servers running only while you are measuring; they are harmless but there is no reason for a listening port to outlive its purpose. Keep the names, the keys, the settings file and the notebook: Parts 19 to 22 all use them.

What you learned

Names before addresses. Every machine answers to a name under home.arpa, and the direct cable has names of its own so that a later configuration chooses a path deliberately rather than by accident. This is the single change that makes cluster configuration readable.

Passwordless login is a prerequisite, not a convenience. Distributed launchers assume it. You have a dedicated key with a purpose in its comment and a passphrase on it.

A link has a measured speed, and it is not the number on the box. You have throughput in both directions and a round-trip time for every link, and the ratio between measured and rated is written down. When a later part disappoints you, this is the first number to look at.

Frame size is a property of the whole path. You verified it rather than configuring it, and you know what the failure looks like: small things work, the first large thing vanishes.

RDMA is present or it is not, and it fails quietly. You checked, once, and recorded the answer, so no later measurement has to be re-run because the transport turned out to be TCP.

One library, read by everyone. No machine can be running a different copy of a model than its neighbours, because there is only one copy.

Record in the lab notebook: one topology line per machine from record-topology.py, one link line per direction per link from measure-links.sh, and, in your own words, one sentence naming the link you would split a model across and the link you would not. Part 19 starts by asking you for exactly that sentence.

Check your understanding

Question 1. Why does this lab give the direct cable between two machines its own set of names, separate from the machines' house-network names?

• Because a cable cannot be addressed without a name
• Because the name a later engine is given decides which path the model traffic takes, so two names make the choice explicit and measurable
• Because the router requires it
• Because RDMA only works with named interfaces

Show the answer and why

Answer: Because the name a later engine is given decides which path the model traffic takes, so two names make the choice explicit and measurable

A machine with two paths to another machine has two addresses on that machine, and a single name resolves to only one of them. Naming both means you can measure both, choose deliberately, and never again wonder which cable a disappointing benchmark went over. It costs two extra lines in /etc/hosts.

Question 2. A do-not-fragment ping with a payload of 8,972 bytes fails, while a plain ping to the same machine succeeds. What has this told you?

• The machine is unreachable
• The name does not resolve
• Something in the path carries a smaller frame than you configured, so the jumbo setting is not usable end to end
• The RDMA path is broken

Show the answer and why

Answer: Something in the path carries a smaller frame than you configured, so the jumbo setting is not usable end to end

The plain ping proves reachability and name resolution, so the failure is about size alone. Some device between the two endpoints, often a switch and sometimes the other machine, has not been raised to the same maximum transmission unit. Put both ends back to the default rather than leaving one raised, because a mismatch fails in a much more confusing way than a small MTU does.

Question 3. Which iperf3 options does the measurement script use, and what does each do? Select all that are correct.

• --reverse turns the test around so the server sends to the client
• --parallel opens several client streams at once
• --json produces machine-readable output instead of the human-readable default
• --time sets how long the test runs, replacing the ten-second default

Show the answer and why

Answer: --reverse turns the test around so the server sends to the client, --parallel opens several client streams at once, --json produces machine-readable output instead of the human-readable default, --time sets how long the test runs, replacing the ten-second default

All four are documented in the iperf3 manual and all four are used. Parallel streams matter more than people expect: a single TCP connection does not always fill a fast link, so a one-stream result on a fast link can understate it badly and send you looking for a fault that is not there.

Question 4. On the single-machine path, why do the containers sit on a network created with the bridge driver rather than on Docker's default bridge?

• The default bridge is slower
• Only a user-defined bridge lets containers resolve each other by name, which is the part of a real cluster this path exists to rehearse
• The default bridge cannot be used with Compose
• iperf3 requires it

Show the answer and why

Answer: Only a user-defined bridge lets containers resolve each other by name, which is the part of a real cluster this path exists to rehearse

Docker's documentation states that user-defined bridges provide automatic DNS resolution between containers and that on the default bridge containers can only reach each other by address. Since name resolution is where a large share of cluster debugging goes, a single-machine path that includes it teaches far more than one that does not.

Question 5. True or false: if a distributed tool reports that it is using TCP rather than RDMA, the cluster is broken and will not work.

• True
• False

Show the answer and why

Answer: False

It will work, and llama.cpp's RPC README says the connection falls back to TCP unless both peers can use RDMA. What it will not be is as fast as you expected. That is the danger: the fallback is silent, so the symptom is a disappointing measurement rather than an error, which is exactly why this lab verifies the RDMA path separately and writes the answer down.

Check my answersClear