Challenge: The Cluster That Is Slower Than One Machine

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The per-track faults, their symptoms and the versions they were reproduced with belong here once the validation pass has run this page on real machines.

Objective

Somebody has built the cluster from this part’s labs. They report: “I split a 30B model across two machines and it is three times slower than running it on one of them. Clustering is useless.” By the end of this page you will be able to answer that in about ten minutes, and you will have reproduced each candidate fault on purpose so that you recognise it when it is not yours.

The deliverable is not a fast cluster. It is a written diagnosis with four parts: the evidence you collected, the fault it pointed at, the one change you made, and the second measurement that proves the change worked. Evidence, then theory, then proof. Anything else is changing things until the symptom moves.

There is a twist worth naming now, because it is the most common answer of all: sometimes the diagnosis is that the cluster should not exist. A model that fits on one machine will run faster on that machine, every time, because a layer split makes the machines take turns. Being able to say that clearly, with a number beside it, is the most valuable outcome of this page.

⌂Single-machine path

One machine. Run two ggml-rpc-server processes on one host, as in this part’s first lab, and use Qwen3-8B at Q4_K_M against the 8 GB floor. Three of the six faults can be reproduced that way: a CPU device in the split, proportions in the wrong order, and the model that fitted on one box all the time. The three network faults cannot be, so read those sections, run the diagnostic script anyway to see what a healthy report looks like, and note in your write-up which faults you reproduced and which you only studied.

The procedure

A slow cluster: what to do, in order

1. 1Get two numbers, not one impressionThe cluster's decode rate, and the decode rate of the best single machine in it on a model that fits. "Slower" needs a baseline before it means anything.
2. 2Ask whether the cluster should existIf the model fits on one of your machines, stop. The answer is not a faster cluster; the answer is one machine, and the rest of the procedure is optional curiosity.
3. 3Find out which interface carried the trafficThe route to each RPC host names the interface. Wi-Fi, or the wrong wired port, explains more slow clusters than everything else put together.
4. 4Check the maximum transmission unit end to endA large packet sent with fragmentation forbidden either arrives or it does not. A link configured for jumbo frames that quietly is not is a slow link that reports nothing.
5. 5Read the device listA host whose GPU backend failed to load offers a CPU device and looks perfectly healthy. One command finds it.
6. 6Read the layer counts, not your intentionsThe load log says how many layers each device actually got. Compare that against the split you meant to write, in the order the client sees devices.
7. 7Change exactly one thing, then measure againSame model, same context, same prompt, same generated length. Two changes at once means you will not know which one worked.
8. 8Write the four parts downEvidence, fault, change, proof. Next time this takes two minutes, because you will have seen it.

The order is not arbitrary. Each step is cheap, and each eliminates a large fraction of the possibilities, so by the fifth you are usually looking at one candidate rather than six.

The short list of faults

Fault	What the evidence looks like
The model fitted on one machine	The model file plus its cache is smaller than one machine’s memory. The single-machine baseline is faster than the cluster. Nothing is broken.
Wi-Fi, or the wrong interface	The route to the RPC host names a wireless or general-purpose interface. Round-trip times are in the tens of milliseconds rather than fractions of one. Decode is far worse than prefill.
MTU mismatch	A 1500-byte path succeeds and a 9000-byte path fails, or vice versa, on a link configured for jumbo frames. Throughput is a fraction of the link’s rating and the failure is silent.
A CPU device in the split	The device list names CPU where you expected CUDA, Vulkan or Metal. That host’s build has no GPU backend, or its runtime failed to load.
Proportions in the wrong order	The load log’s layer counts are the reverse of what you asked for. The fastest machine has the smallest share. Nothing warned you.
RDMA silently not in use	ibv_devices lists hardware, but the --rpc addresses belong to a different interface, so the handshake fell back to TCP.

Notice what is not on the list: the model, the quantisation and the prompt. Those change speed by sensible amounts. The faults above change it by factors.

Requirements

The cluster from this part’s first two labs, still assembled; the rpc.env file; the notebook with each machine’s standalone decode rate in it. Forty-five minutes, all attended. No new downloads.

Track S — NVIDIA DGX Spark

Two faults are characteristic here and both come from the same cause: each Spark has a ConnectX-7 with two QSFP ports and a separate 10 gigabit Ethernet port, so there are several plausible addresses for the same machine and only one of them is the fast one.

Reproduce the wrong interface fault by naming the machine on the general-purpose Ethernet path instead of the QSFP one. Everything works. Only the numbers change, and only if you look. NVIDIA’s clustering documentation makes the trap concrete: each QSFP port appears as two independent Linux Ethernet interfaces, each paired with its own RoCE device, so the mapping table for a single Spark lists four Ethernet interfaces and four RoCE devices across its two ports. Add the 10 gigabit Ethernet port and Wi-Fi and one machine has six plausible names before you have typed anything, of which one carries the model traffic at the speed you paid for. That is why ibdev2netdev is the first command in the playbook and why section 3 of your report exists.

The NVIDIA playbook’s own troubleshooting table is worth reading beside your report: it lists network-unreachable errors, SSH failures and an invisible second node, all with the same underlying cause of an interface not configured. It also notes that each physical port presents two logical interfaces, which is a fine way to end up measuring the wrong one.

Track X — AMD Ryzen AI Max+ 395

The CPU device fault is the one to reproduce here, because it is the one this track hits by accident. A Vulkan build that cannot find its loader at run time, or a machine whose GPU-visible memory cap left no room, produces an rpc-server offering a CPU device and no error anybody reads.

The second candidate is the wrong proportions: a 128 GB Ryzen AI Max+ reports a great deal of free memory, so the default split hands it most of the layers even when the other machine computes considerably faster. That is not a fault in the tool. It is a decision the tool made on your behalf, and this page is where you take it back.

Track M — Apple silicon

Reproduce RDMA silently not in use. With two Thunderbolt 5 Macs on macOS 26.2 or later and rdma_ctl enable already run, put the Wi-Fi or Ethernet name of the peer in RPC_HOSTS instead of its Thunderbolt address. ibv_devices still lists the hardware, the cluster still runs, and the transport is TCP over a much worse link.

This is the clearest demonstration in the course of why Part 18 gives each link its own name: with one name per machine there is no way to say which cable you meant.

Track N — NVIDIA desktop or laptop

Reproduce the model that fitted on one machine. A desktop with a 24 or 32 GB card runs a 30B-class mixture-of-experts model at four bits comfortably on its own, and splitting it across the house is exactly the mistake this page is named after.

Measure it both ways and keep the pair of numbers. It is the most persuasive row in your notebook and the one to reach for when somebody proposes a cluster for a model that fits.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-19-llama-cpp-rpc"
cd "$LAB_DIR"
pwd
test -f "diagnose-rpc-cluster.sh"

Expected result: pwd ends in part-19-llama-cpp-rpc and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Turn the impression into two numbers

Never diagnose a feeling. Get the cluster’s rate, and get the baseline it should be compared against.

RunnableAll tracks

the cluster, as it stands

. ./rpc.env
bash run-split.sh

RunnableAll tracks

in another terminal: the cluster's rate

python3 measure-split.py \
    --iface "$CLUSTER_IFACE" \
    --lab part-19/challenge-the-cluster-slower-than-one-machine \
    --note "before: the cluster as reported" \
    --labbook labbook.md

RunnableAll tracks

the baseline: the same model on your best single machine

~/llama.cpp/build/bin/llama-bench \
    -m ~/models/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf \
    -p 512 -n 128 -r 3 -ngl 999 -fa on

If that second command runs at all, you have already found the most likely fault, and task 3 is where you deal with it. If it fails for want of memory, the cluster is justified and the remaining tasks are about making it behave.

2. Collect the evidence with one command

The script gathers everything the procedure asks for into one report: the build, every interface with its address and maximum transmission unit and whether it is wireless, the route and round trip to each RPC host, whether each server answers, whether RDMA hardware exists, and whether the model would have fitted here anyway. It changes nothing and needs no privileges.

RunnableAll tracks

diagnose-rpc-cluster.sh

#!/usr/bin/env bash
# Purpose: collect, into one report, every piece of evidence needed to explain a slow
#          llama.cpp RPC cluster: which interface carries the traffic, its MTU and whether
#          it is wireless, the route and round trip to each host, whether each rpc-server
#          answers, whether RDMA is available, the split in use, and whether the model
#          would have fitted on this machine alone
# Platform: all (Linux and macOS paths; each section says when it could not run)
# Minimum memory: 8 GB
# Assumes: run on the client; the environment from env-example.txt has been sourced; it
#          reads and measures only, changes nothing, and needs no privileges
#
# Usage: . ./rpc.env && bash diagnose-rpc-cluster.sh [report.md]
#
# Environment:
#   RPC_HOSTS      host:port,host:port                    (required)
#   CLUSTER_IFACE  the interface you believe is in use    (recommended)
#   MODEL          the model file being split             (optional but useful)
#   TENSOR_SPLIT   the proportions in use                 (optional)
#   LLAMA_BIN      directory holding the binaries         (default: found on PATH)
#
# Read the report top to bottom before forming a theory. Almost every slow cluster is
# explained by one of the first three sections, and guessing before reading them is how an
# afternoon disappears.

set -euo pipefail

REPORT="${1:-rpc-cluster-report.md}"
OS="$(uname -s)"

say() { printf '%s\n' "$*" >> "$REPORT"; }
section() { printf '\n## %s\n\n' "$*" >> "$REPORT"; }
code_start() { printf '```\n' >> "$REPORT"; }
code_end() { printf '```\n' >> "$REPORT"; }


# --- Part 18 compatibility --------------------------------------------------------------
# Part 18's .env lists every machine in CLUSTER_PEERS, space separated, as names without
# ports. When RPC_HOSTS is not set, build it from those names in the order they appear,
# appending RPC_PORT to each. Set RPC_HOSTS yourself whenever the order matters, when only
# some peers take part, or when a peer listens on a different port: the order is what
# --tensor-split addresses.
RPC_PORT="${RPC_PORT:-50052}"
if [ -z "${RPC_HOSTS:-}" ] && [ -n "${CLUSTER_PEERS:-}" ]; then
  read -r -a PART18_PEERS <<< "$CLUSTER_PEERS"
  for peer in "${PART18_PEERS[@]}"; do
    RPC_HOSTS="${RPC_HOSTS:+$RPC_HOSTS,}${peer}:${RPC_PORT}"
  done
  echo "    RPC_HOSTS built from Part 18's CLUSTER_PEERS: $RPC_HOSTS"
fi

[ -n "${RPC_HOSTS:-}" ] || { echo "diagnose-rpc-cluster: set RPC_HOSTS, or Part 18's CLUSTER_PEERS; see env-example.txt" >&2; exit 1; }

: > "$REPORT"
say "# RPC cluster diagnostic"
say ""
say "- client: $(hostname -s), $OS $(uname -m)"
say "- collected: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
say "- RPC_HOSTS: \`$RPC_HOSTS\`"
say "- TENSOR_SPLIT: \`${TENSOR_SPLIT:-not set, so llama.cpp splits by free memory}\`"

# --- 1. What is this client running? ----------------------------------------------------
section "1. Build"
code_start
if [ -n "${LLAMA_BIN:-}" ] && [ -x "$LLAMA_BIN/llama-cli" ]; then
  "$LLAMA_BIN/llama-cli" --version >> "$REPORT" 2>&1 || true
elif command -v llama-cli >/dev/null; then
  llama-cli --version >> "$REPORT" 2>&1 || true
else
  say "llama-cli not found; set LLAMA_BIN"
fi
code_end
say ""
say "A build without RPC compiled in cannot use \`--rpc\` at all, so if the client refused"
say "the option, rebuild it with \`-DGGML_RPC=ON\` before reading any further."

# --- 2. Interfaces, addresses and MTU ---------------------------------------------------
section "2. Interfaces on this machine"
code_start
case "$OS" in
  Darwin)
    ifconfig 2>/dev/null | grep -E '^[a-z0-9]+:|inet |media:' >> "$REPORT" || true
    ;;
  *)
    ip -br addr >> "$REPORT" 2>&1 || true
    echo "---" >> "$REPORT"
    ip -br link >> "$REPORT" 2>&1 || true
    ;;
esac
code_end
say ""
if [ -n "${CLUSTER_IFACE:-}" ]; then
  say "You said the cluster link is \`$CLUSTER_IFACE\`. Its MTU and kind:"
  code_start
  case "$OS" in
    Darwin)
      ifconfig "$CLUSTER_IFACE" >> "$REPORT" 2>&1 || true
      networksetup -listallhardwareports 2>/dev/null \
        | grep -B 1 "Device: $CLUSTER_IFACE\$" >> "$REPORT" || true
      ;;
    *)
      ip -d link show dev "$CLUSTER_IFACE" >> "$REPORT" 2>&1 || true
      if [ -e "/sys/class/net/$CLUSTER_IFACE/wireless" ] \
         || [ -d "/sys/class/net/$CLUSTER_IFACE/phy80211" ]; then
        echo "THIS INTERFACE IS WIRELESS" >> "$REPORT"
      fi
      ;;
  esac
  code_end
  say ""
  say "An MTU of 1500 on a link you configured for jumbo frames means the change did not"
  say "take, or a switch in the path did not take it. A wireless interface here is the"
  say "single most common reason a cluster is slower than one machine."
else
  say "CLUSTER_IFACE is not set, so this report cannot say which link you meant to use."
fi

# --- 3. Route and round trip to each host ------------------------------------------------
section "3. Route, reachability and round trip"
IFS=',' read -r -a HOSTS <<< "$RPC_HOSTS"
for hp in "${HOSTS[@]}"; do
  host="${hp%%:*}"
  port="${hp##*:}"
  say "### \`$hp\`"
  say ""
  code_start
  case "$OS" in
    Darwin) route get "$host" >> "$REPORT" 2>&1 || true ;;
    *)      ip route get "$(getent ahostsv4 "$host" 2>/dev/null | awk 'NR==1 {print $1}')" \
              >> "$REPORT" 2>&1 || echo "could not resolve $host" >> "$REPORT" ;;
  esac
  code_end
  say ""
  say "The interface in that route is the one that will actually carry the traffic. If it"
  say "is not \`${CLUSTER_IFACE:-your cluster link}\`, you have found the fault."
  say ""
  code_start
  if (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null; then
    echo "TCP connect to $hp: succeeded" >> "$REPORT"
  else
    echo "TCP connect to $hp: FAILED - no rpc-server listening, or it bound elsewhere" >> "$REPORT"
  fi
  ping -c 4 "$host" >> "$REPORT" 2>&1 || echo "ping failed or is blocked" >> "$REPORT"
  code_end
  say ""
  say "Path MTU, by sending a packet that must not be fragmented:"
  say ""
  code_start
  case "$OS" in
    Darwin) ping -c 2 -D -s 1472 "$host" >> "$REPORT" 2>&1 || echo "1500-byte path failed" >> "$REPORT" ;;
    *)      ping -c 2 -M "do" -s 1472 "$host" >> "$REPORT" 2>&1 || echo "1500-byte path failed" >> "$REPORT" ;;
  esac
  case "$OS" in
    Darwin) ping -c 2 -D -s 8972 "$host" >> "$REPORT" 2>&1 || echo "9000-byte path failed (no jumbo frames)" >> "$REPORT" ;;
    *)      ping -c 2 -M "do" -s 8972 "$host" >> "$REPORT" 2>&1 || echo "9000-byte path failed (no jumbo frames)" >> "$REPORT" ;;
  esac
  code_end
  say ""
done

# --- 4. Is RDMA available at all? ----------------------------------------------------------
section "4. RDMA"
code_start
if command -v ibv_devices >/dev/null; then
  ibv_devices >> "$REPORT" 2>&1 || true
else
  echo "ibv_devices not found: no verbs library on this machine, so this client is on TCP" >> "$REPORT"
fi
if [ -n "${GGML_RPC_NO_RDMA:-}" ]; then
  echo "GGML_RPC_NO_RDMA is set in this environment: RDMA is switched off deliberately" >> "$REPORT"
fi
code_end
say ""
say "RDMA is negotiated per connection and it is point-to-point: a connection made over a"
say "different interface from the RDMA-capable one stays on TCP however good the hardware is."

# --- 5. Would it have fitted on this machine? -----------------------------------------------
section "5. The question the challenge is really about"
if [ -n "${MODEL:-}" ] && [ -f "$MODEL" ]; then
  bytes="$(python3 -c 'import os,sys; print(os.path.getsize(sys.argv[1]))' "$MODEL" 2>/dev/null || echo 0)"
  say "- model file: \`$(basename "$MODEL")\`"
  say "- first shard: $((bytes / 1000000)) MB (later shards are not counted here)"
else
  say "- MODEL is not set or does not exist here, so the fit question cannot be answered."
fi
code_start
case "$OS" in
  Darwin) sysctl -n hw.memsize 2>/dev/null | awk '{ printf "physical memory: %d MB\n", $1/1000000 }' >> "$REPORT" || true ;;
  *)      grep -E '^(MemTotal|MemAvailable|SwapTotal)' /proc/meminfo >> "$REPORT" 2>&1 || true ;;
esac
code_end
say ""
say "If every shard of the model would have fitted in this machine's memory with room for"
say "the KV cache, the cluster was never going to win, and the fix is to stop splitting."

echo "==> written to $REPORT"
echo "    Read sections 2 and 3 first: interface, MTU, route."

Download diagnose-rpc-cluster.sh196 lines

RunnableAll tracks

collect the evidence

. ./rpc.env
bash diagnose-rpc-cluster.sh slow-cluster-report.md

Read it top to bottom before forming any theory. Sections two and three, the interfaces and the routes, settle most cases on their own.

✦Tip

Run this once on a cluster you believe is healthy and keep the report. A healthy report is what makes an unhealthy one legible: you will know what round trip your link normally shows, what maximum transmission unit succeeds, and what the device list normally says.

3. The fault everyone finds first: the model fitted

RunnableAll tracks

the fit question, answered with arithmetic

python3 plan-tensor-split.py \
    --weights-gb 18.6 --kv-gb 0.8 --layers 48 \
    --device this-machine:22 --device other-machine:22

Compare the payload line against one machine’s usable memory. If one machine could have held it, the cluster was never going to be faster: a layer split runs the machines in turn, so the time per token is the sum of their contributions plus a boundary crossing, and one machine’s time alone is smaller than that sum.

?Reality check

The claim being tested is “two machines are faster than one”. For a single request on a layer split it is false, and the arithmetic says so before any measurement: the participants take turns. What two machines buy is memory, and with it access to models that were out of reach. They also buy throughput when several requests are in flight at once, which is a different measurement from the one that made somebody unhappy here.

Write the two rates side by side in your notebook. That pair of numbers is worth more than any tuning you could do afterwards.

If the model does fit on one machine, the correct fix is to stop splitting, and your proof is the baseline you already measured. Record the diagnosis and continue through the remaining faults as exercises, because you will meet them on a cluster that is justified.

4. Reproduce the wrong interface

Part 18’s naming scheme gave every machine two names: node-b.home.arpa, which resolves on the house network, and node-b-direct.home.arpa, which resolves on the cable between the two machines. That is what makes this a one-line experiment rather than a re-cabling job: swap one name for the other and the cluster runs over a different link.

RunnableAll tracks

the same cluster, over the wrong cable

pkill -f llama-server
. ./rpc.env
RPC_HOSTS=node-b.home.arpa:50052 bash run-split.sh

On a machine whose house-network path is Wi-Fi, this is the full fault. On one where both paths are wired, it is the milder version of it, and the round-trip figures in the report will still separate them.

Measure it with the same command as task 1 and compare. Then look at what the evidence would have told you: the route section of your report names the interface, and the round-trip figures differ by orders of magnitude between a wired cluster link and Wi-Fi.

=Expected result

Prefill suffers less than decode. Prefill sends a batch’s worth of hidden states in one go, which is a bandwidth problem; decode pays the round trip once per boundary per token, which is a latency problem. A result where prefill looks acceptable and decode has collapsed is close to a signature for this fault.

5. Reproduce the maximum transmission unit mismatch

A link configured for jumbo frames where one end, or a switch in the middle, was not updated carries small packets fine and large ones badly, and nothing in any log mentions it.

RunnableAll tracks

does a 9000-byte path actually work?

ping -c 3 -M do -s 8972 node-b-direct.home.arpa
ping -c 3 -M do -s 1472 node-b-direct.home.arpa

On macOS the equivalent flag is -D rather than -M do. The diagnostic script runs both probes for every host and puts the results in the report.

A 1472-byte payload plus headers is a 1500-byte packet; 8972 plus headers is 9000. If the small one succeeds and the large one fails on a link you configured for jumbo frames, the setting did not take somewhere along the path. Part 18’s lab is where you fix it, and its CLUSTER_MTU setting is where you recorded what the link was supposed to carry. The point here is that you can find the mismatch in one command instead of suspecting the engine.

6. Reproduce the CPU device in the split

RunnableAll tracks

on one host: pretend the GPU backend failed to load

pkill -f ggml-rpc-server
. ./rpc.env
RPC_DEVICE=CPU bash start-rpc-server.sh --background

RunnableAll tracks

on the client: the device list tells you immediately

. ./rpc.env
bash probe-rpc-devices.sh cpu-fault.md

The report names CPU where a backend should be. That is the whole diagnosis, and it takes one command. Measure the cluster in this state so that you know what the damage looks like, then restart the host normally.

This fault arrives in the wild without anybody choosing it: a Vulkan loader that is missing, a CUDA runtime that does not match the driver, or a build that never had the backend flag. The host comes up, joins, and slows every token.

7. Reproduce the split in the wrong order

RunnableAll tracks

proportions reversed on purpose

pkill -f llama-server
. ./rpc.env
TENSOR_SPLIT=0.15,0.85 bash run-split.sh

Give your fastest device the smallest share and measure. Nothing warns you; it loads and runs. Then read the load log’s per-device layer counts and confirm they match the proportions you passed, in the order the device list gave. Comparing what you meant against what happened is the whole of this check.

8. Fix one thing, and prove it

Choose the fault your evidence actually supports. Change exactly that. Measure again with the same command, the same model, the same context length and the same generated length as task 1.

RunnableAll tracks

the proof

python3 measure-split.py \
    --iface "$CLUSTER_IFACE" \
    --lab part-19/challenge-the-cluster-slower-than-one-machine \
    --note "after: <the one change you made>" \
    --labbook labbook.md

Pending validationBefore and after, one change between them — your recording sheet

Run	Fault present	Interface used	Prefill tokens/s	Decode tokens/s	Link bytes per token
before	—	—	—	—	—
after the one change	none	—	—	—	—
best single machine, no cluster	n/a	n/a	—	—	0

your cluster: one line per machine, with track, memory, backend and link, the operating system and version of each machine · llama.cpp RPC, layer split the build number from llama-cli --version, identical on every machine · Qwen3-30B-A3B, Q4_K_M · 8,192 tokens of context · the date you ran it

Empty on purpose. The third row is not optional: a cluster result without the single-machine comparison beside it cannot answer the question that started this page. If the third row is the fastest, that is your diagnosis and the first two rows are the evidence for it.

9. Write the diagnosis

Four short paragraphs in the notebook, in this order, is the format to keep for the rest of your career with these machines.

1. Evidence. What the report said: the interface, the round trip, the maximum transmission unit, the device list, the layer counts, and whether the model fitted on one box.
2. Fault. Which of the six, and the specific line of evidence that ruled the other five out.
3. Change. Exactly one thing, stated so that somebody else could repeat it.
4. Proof. The second measurement, with the same settings as the first, and the difference.

Rule out an unnecessary cluster first

Start by proving the same checkpoint and context fit on one machine. If they do, measure that path under the same request and concurrency as the cluster. A larger model on the cluster is a different quality/capacity comparison and cannot establish a networking slowdown by itself.

Save the healthy topology and device order before introducing faults. For each fault, collect client placement, worker logs, selected interface and link counters. A low token rate could result from the wrong route, a slow CPU device, an inefficient split or ordinary communication overhead; use the evidence to distinguish them.

Restore one setting and rerun the same workload. Recheck MTU end to end after restoring it, and confirm the runtime actually uses the intended interface. If the correctly configured cluster still loses, calculate whether communication exceeds the work saved. That can be an expected architectural result rather than a bug. Finish with an incident note identifying the workload, cause, decisive evidence, repair and residual limitation. Keep the single-machine baseline as the control for future tuning; the objective is an explained result, not forcing the cluster to win.

Validation

You are done when all of the following are true:

• slow-cluster-report.md exists and you have read it end to end;
• you can name, from the report alone, which interface carried the traffic to each RPC host;
• the report shows a maximum transmission unit probe result for each host, and you can say whether the path carries the size you configured;
• you reproduced at least two faults deliberately and recorded the rate under each;
• labbook.md contains a before line and an after line for part-19/challenge-the-cluster-slower-than-one-machine, differing in exactly one setting;
• the recording sheet is filled in, including the single-machine row;
• the four-paragraph diagnosis is written, and its “change” paragraph names one change.

Expected outcome

A procedure you trust, and the recognition that comes from having caused each fault on purpose.

Most slow clusters are one of six things, and three of them are network. Which interface, which packet size, which transport. None of them requires reading engine code.

Latency, not bandwidth, is what a layer split feels. A few kilobytes per token is nothing for any wired link; a round trip paid once per boundary per token is everything. That is why Wi-Fi is so much worse than its rating suggests, and why the symptom is a collapsed decode rate beside an acceptable prefill rate.

A silently degraded host looks healthy. A CPU device where a GPU should be, or a TCP connection where RDMA was available, produces a working cluster and a bad one. Both are found by looking at a list rather than by measuring harder.

The default split is a decision somebody made for you. Memory-proportional is a reasonable default and it is frequently not what you want on unequal machines.

Sometimes the answer is to stop. The most valuable outcome of this page is the ability to say “this model fits on that machine, so run it there”, with two numbers beside it. Part 18’s first lesson makes the same argument before you buy any hardware; this page is what it looks like after you already have.

Troubleshooting

The diagnostic script reports nothing for the route section. The host name did not resolve. That is itself a finding: fix name resolution first, because a name that does not resolve on the client cannot be the name the cluster is using.

ping is blocked, so the round-trip figures are missing. Use the byte counters from measure-split.py and the prefill-against-decode shape instead. A firewall that drops ICMP also tends to break path maximum transmission unit discovery, which is worth noting as a candidate in its own right.

Both interfaces show similar round trips. Then the fault is probably not the interface. Continue to the device list and the layer counts rather than re-running the same check.

The fix made no difference. Either the fault was misidentified or there were two. Undo the change, go back to the evidence, and resist the urge to change a second thing while the first is still in place.

The cluster is now faster but still slower than one machine. That is the expected outcome for a model that fits on one machine, and it is not a fault. Record it and move on.

Cleanup

RunnableAll tracks

restore the cluster to a known state

pkill -f ggml-rpc-server
pkill -f llama-server

Restart the servers normally, with the correct interface and no RPC_DEVICE override, and confirm the device list is what it was before you started breaking things. Keep the reports; slow-cluster-report.md from a healthy cluster is a reference you will want.

Part 20 begins with the same two machines and a different engine, so leave the network and the names as they are.

What you learned

• Evidence before theory. Six checks, in a fixed order, each cheap, each eliminating a large fraction of the possibilities.
• The first question is whether the cluster should exist. Fit arithmetic answers it in seconds and no amount of tuning changes the answer.
• Which interface carried the traffic is the highest-yield question in distributed inference. Route, round trip, packet size, in that order.
• A degraded participant is invisible unless you list devices. A CPU where a GPU should be, or TCP where RDMA was available, look exactly like a healthy cluster from the outside.
• One change, then one measurement. A fix without a second measurement under identical settings is a hope.

Record in the notebook: the healthy report and the faulty one; the rate under each fault you reproduced; the four-paragraph diagnosis; and the pair of numbers for the cluster against your best single machine, which is the row you will quote most often.

Check your understanding

Question 1. A split cluster shows an acceptable prefill rate and a decode rate that has collapsed. What does that shape point at first?

• A model that is too heavily quantised
• A link with a long round trip, such as Wi-Fi, because decode pays the round trip once per boundary per token while prefill sends a whole batch at once
• A key-value cache that is too large
• A mismatch between the two machines' llama.cpp builds

Show the answer and why

Answer: A link with a long round trip, such as Wi-Fi, because decode pays the round trip once per boundary per token while prefill sends a whole batch at once

Prefill is a bandwidth problem and decode is a latency problem in a layer split. That asymmetry is close to a signature, and the route section of the diagnostic report confirms it in one line.

Question 2. Which single piece of evidence most often settles a slow-cluster case?

• The tokens-per-second figure itself
• The interface named in the route to each RPC host
• The model file's checksum
• The number of CPU threads on the client

Show the answer and why

Answer: The interface named in the route to each RPC host

Three of the six faults on the short list are about which cable carried the traffic and what it could carry. The route names the interface, and that one line eliminates or confirms half the list before anything is measured again.

Question 3. Your evidence shows the model would have fitted on one of the two machines. What is the correct fix?

• Tune the tensor split until the cluster beats the single machine
• Stop splitting and run it on the one machine, and record both rates as the proof
• Add a third machine so the shares are smaller
• Force RDMA, which will make the cluster faster than one machine

Show the answer and why

Answer: Stop splitting and run it on the one machine, and record both rates as the proof

A layer split runs the machines in turn, so its time per token is a sum over participants and cannot be smaller than the fastest participant alone. No split, transport or extra machine changes that. Clustering is a capacity answer.

Question 4. You changed the interface in RPC_HOSTS and also raised the context length, and the cluster is faster. What have you proved?

• That the interface was the fault
• Nothing about either change, because two variables moved at once
• That context length affects network speed
• That the cluster was correctly configured all along

Show the answer and why

Answer: Nothing about either change, because two variables moved at once

A comparison is only valid when everything not being compared is held fixed. Undo one change and measure again. This is the same discipline the Part 6 benchmark lab insists on, applied to a fault rather than to a model.

Check my answersClear