Lab: A Two-Mac Cluster over Thunderbolt 5

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The course’s reference lab has one Mac, an M5 MacBook Pro with 24 GB, so the two-Mac primary path is documentation-only until a second Mac joins it, and every results table on this page is marked pending. The single-Mac path at the end of the tasks is the one part of this page the reference lab can exercise, and it too is unvalidated until the validation pass runs it.

Objective

By the end of this lab two Macs will be joined by a Thunderbolt cable that carries model traffic and nothing else, an MLX distributed group will form across them on demand, one model will run split between them, and your notebook will hold four things: whether the group forms, how long a collective takes at several sizes, prefill and decode figures for the served model, and how many bytes crossed the cable while you measured.

You will do it twice over: once with MLX directly, where you write the host file and choose the split, and once with exo, where the cluster is discovered and the split is chosen for you. The two runs on the same hardware are the comparison the previous lesson set up.

⌂Single-machine path

The primary path needs two Macs. If you have one, the section The single-Mac path below runs every step with two ranks on the same machine, which MLX supports directly and its documentation recommends as the way to test a distributed script before involving a network. The group forms, the collectives run, the splits work and every script writes the same notebook lines. What you do not get is more memory, so the model has to be one that already fits. Do it that way, write down that you did, and the rest of Level 4 still follows.

Architecture

Two Macs, two networks, one job each

• workerMac Arank 0: runs mlx.launch, serves the endpoint, holds half the model
• workerMac Brank 1: started over ssh, holds the other half
• routerHouse switch or Wi-Finames, ssh, model downloads, exo discovery

• Mac A ⟷ connected to House switch or Wi-FiEthernet or Wi-Fi: how the machines are named and reached
• Mac B ⟷ connected to House switch or Wi-FiEthernet or Wi-Fi: how the machines are named and reached
• Mac A ⟷ connected to Mac BThunderbolt 5 cable: model traffic only, its own subnet, no bridge

The separation is the point. mlx.launch starts the remote rank over the house network; every tensor the split moves goes over the Thunderbolt cable. Keeping them apart is what makes the link measurement in task 8 mean anything, because it is the only traffic on that interface.

The memory picture is the reason for the whole exercise. Two 32 GB Macs are not a 64 GB Mac, but for a model whose weights are the problem they are close enough to be interesting.

One 32 GB Mac, against the model this lab serves

32 GB budget

Weights, Qwen3-32B at 8-bit

34.8 GB

Requested

34.8 GB

Machine budget

32 GB

Over budget. 34.8 GB requested against a 32 GB machine - 2.8 GB over. Something here has to shrink: a smaller quantisation, a shorter context, or fewer of these reservations at once.

The course's model reference records Qwen3-32B, an Apache-2.0 licensed dense model, at about 34.8 GB in 8-bit, which is why this model was chosen: it does not fit on one 32 GB Mac at all, before any key-value cache or any desktop. Splitting it in two puts roughly 17.4 GB of weights on each machine and leaves room to work in. If your Macs have less, use a smaller repository and say so in your notebook; if they have more, the interesting model is the next one up.

Requirements

Track M — Apple silicon

The primary path. Two Macs, each with 32 GB of unified memory or more, each with at least one free Thunderbolt port, joined by a cable rated for Thunderbolt 5. Both on macOS 26.2 or later if you want the RDMA backend, and on the same build, because exo’s README warns that “RDMA ports may be unable to discover each other on different versions of MacOS. Please ensure that OS versions match exactly (even beta version numbers) on all devices.”

Both machines need mlx-lm at the same Python path; the course pins mlx-lm 0.31.3 · verified 2026-09-08. Both need this lab’s directory at the same absolute path, because mlx.launch copies nothing. Both need password-less SSH from Mac A, which Part 18’s lab set up.

Two models, both Apache-2.0 licensed and neither gated: Qwen3-32B in an 8-bit MLX conversion for the split model, and Qwen3-8B in a 4-bit conversion as the single-Mac baseline. Both are in the course’s model reference; substitute your own if your Macs have more or less memory, and record the substitution.

Disk: with tensor parallelism both machines download the whole model repository, so budget the full size twice. With pipelining each downloads only its own shard. Plan for about 35 GB per machine for the suggested model, and more if your network is slow enough that you would rather copy the Hugging Face cache across than download twice.

Time. About seventy-five minutes attended once the machines are already named and reachable, of which perhaps twenty-five is the Thunderbolt configuration and the first group check. The model download is unattended and depends entirely on your connection. Add an hour and a reboot of each machine if you are enabling RDMA for the first time, because that part happens in Recovery and cannot be scripted.

Track S — NVIDIA DGX SparkNot supported

MLX's distributed backends for this lab are the ring and JACCL backends over Thunderbolt between Macs. A DGX Spark has ConnectX-7 rather than Thunderbolt, and its cluster path is NCCL and Ray.

Track S has its own two-machine lab in Part 20, over ConnectX-7 with RoCE, which is the same idea on better cabling. If you own one Mac as well as a Spark, the fallback section One Mac and one other machine, over Ethernet below joins them with llama.cpp RPC from Part 19, and the reality check that follows this lab is built around exactly that pairing.

Track X — AMD Ryzen AI Max+ 395Not supported

A Ryzen AI Max+ 395 machine has USB4-class ports and 2.5 gigabit Ethernet, not Thunderbolt 5 with RDMA, and exo runs on CPU only on Linux.

Track X reaches a cluster through Part 19’s llama.cpp RPC, which crosses platforms and needs nothing from the network beyond reachability. If you also own a Mac, use the fallback section below to join the two, and bring the result to the reality check.

Track N — NVIDIA desktop or laptopNot supported

This lab's primary path is Apple-silicon-only: it depends on Thunderbolt networking between Macs and on MLX, which has no path to a consumer NVIDIA GPU here.

Track N has the multi-GPU desktop material in Part 20 and the cross-platform RPC cluster in Part 19. As with the other two tracks, a Mac plus an NVIDIA desktop is a perfectly good pair for the fallback section and for the reality check’s comparison.

⚠Warning

Task 2 changes how each Mac boots and networks. Enabling remote direct memory access is done from Recovery and needs a reboot; configuring the link brings down the bridge0 interface macOS creates for Thunderbolt and replaces it with a per-cable subnet. Neither is destructive to your data, and both are reversible, but both will interrupt anything using a Thunderbolt-attached disk or display, and the bridge teardown will drop a session running over that interface. Do the configuration from the keyboard, or over Wi-Fi, and never over the cable you are about to reconfigure.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-21-apple-clusters"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-21-apple-clusters and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Prepare the ground on both Macs

Give both machines a name you can type, under home.arpa as RFC 8375 intends, and confirm SSH works without a password. Part 18’s lab did this; five minutes now saves an hour later.

RunnableTrack M · Apple silicon

from Mac A: can you reach Mac B without typing anything?

ssh -o BatchMode=yes mac-b.home.arpa true && echo "ssh is ready"

Then confirm the two facts mlx.launch will assume. The Python that has mlx-lm in it must be at the same absolute path on both machines: ask for the path with mlx.launch --print-python on each and compare the strings, not the versions. The lab directory must also be at the same absolute path on both, because the launcher runs the script where it is told and does not copy it.

Copy the settings file and fill it in on Mac A.

Fragment — not complete on its own

env-example.txt

# Purpose: the settings every script in this lab reads. Copy this file to `.env`
#          beside them and fill in the empty lines. Nothing here is a secret, and
#          nothing here should be committed: every value is a fact about your house.
# Platform: mac (Track M). The fallback sections of the lab reuse MACHINE_NAME,
#           PEER_SSH and LABBOOK and ignore the rest.
# Minimum memory: 32 GB per Mac for the primary path; 24 GB for the single-Mac path.
# Assumes: `cp env-example.txt .env` and then an editor. The shell scripts read it
#          with `set -a; . ./.env; set +a`, so a value containing spaces needs quotes.

# ----------------------------------------------------------------- who is who
# A short name for THIS Mac, used as the record key in the lab notebook. Keep it
# to something you will still recognise in six months, for example m1 or m2.
MACHINE_NAME=

# The name you can `ssh` to for the OTHER Mac, over your house network. RFC 8375
# reserves everything under `home.arpa` for names that mean something inside one
# house and nothing outside it, so a value such as
#   PEER_SSH=mac-b.home.arpa
# is exactly the intended use. Add it to /etc/hosts on both Macs, or as a static
# entry on your router, as task 1 describes. Leave empty on the single-Mac path.
PEER_SSH=

# The name the OTHER Mac can ssh to for THIS one. Needed only by the JACCL
# backend, whose configuration helper logs in to every node in turn.
SELF_SSH=

# The user account the scripts ssh as. Leave empty to use the account you are
# logged in as, which is the usual case when the same username exists on both.
SSH_USER=

# --------------------------------------------------------------- the cluster
# The host file mlx.launch reads, written by task 3. Keep it beside the scripts.
HOSTFILE=hosts.json

# The MLX distributed backend: ring or jaccl. Start with ring, which needs
# nothing beyond a working Thunderbolt link, and move to jaccl once `ibv_devices`
# lists devices on both machines. The single-Mac path uses ring.
MLX_BACKEND=ring

# How many ranks to launch. Two Macs means 2. On the single-Mac path this is also
# 2, and both ranks run on this machine.
RANKS=2

# The path to the python interpreter that has mlx-lm installed. It must be the
# SAME path on both Macs. `mlx.launch --print-python` prints the one it would use.
# Leave empty to let mlx.launch decide.
CLUSTER_PYTHON=

# The directory holding these scripts on BOTH Macs, at the same absolute path.
# mlx.launch copies nothing: it runs the script it is given, wherever it is told
# to look. Set this to that path, for example "$HOME/llm-course/part-21".
CLUSTER_DIR=

# ------------------------------------------------------------------ the model
# The MLX repository to serve. Choose one whose implementation supports the split
# you intend; the lesson lists which families support tensor parallelism and which
# support pipelining. The primary path wants a model larger than one Mac's memory.
MODEL_REPO=mlx-community/Qwen3-32B-8bit

# The split: leave empty for tensor parallelism, or set to 1 for pipelining.
# Pipelining downloads only each rank's own shard; tensor parallelism downloads
# the whole repository on every machine.
USE_PIPELINE=

# A smaller model that fits on one Mac, used for the single-machine path and for
# the one-machine baseline row of the results table.
BASELINE_REPO=mlx-community/Qwen3-8B-4bit

# ------------------------------------------------------------- the measurement
# Where mlx_lm.server listens on rank 0, and where measure-pair.py points.
SERVER_HOST=127.0.0.1
SERVER_PORT=8080

# exo's dashboard and API port. 52415 is its default; change it only if something
# else already listens there.
EXO_PORT=52415

# The directory you cloned exo into, used only to print the right start command.
# Leave empty if you installed the macOS application instead of running from source.
EXO_DIR=

# The model id to ask exo to place. exo accepts a short id or a full Hugging Face
# repository name; ask it what it knows with `curl http://localhost:52415/models`.
EXO_MODEL=mlx-community/Qwen3-32B-8bit

# Which placement to accept from exo's preview list, by index, counting from 0
# among the previews whose error field is null. Leave at 0 to take the first,
# or run `bash run-exo.sh previews` and pick deliberately.
EXO_PLACEMENT_INDEX=0

# The Thunderbolt interface to read byte counters from, as `networksetup
# -listallhardwareports` names it for the Thunderbolt port your cable is in.
# Leave empty and measure-pair.py records every interface it can see instead.
LINK_IFACE=

# How many requests per measurement, and how long each generation runs. Twenty
# short requests give a steadier decode figure than one long one.
REQUESTS=20
MAX_TOKENS=128

# The prompt length to use for the prefill measurement, in approximate tokens.
# Prefill is compute-bound and only shows itself on a prompt long enough to
# matter; 2048 is long enough on every Mac in this course.
PREFILL_TOKENS=2048

# ------------------------------------------------- the reality-check comparison
# The two sides compare-endpoints.sh drives, for the reality check that follows
# this lab. Each URL is the /v1 base of one machine's OpenAI-compatible endpoint;
# each name is what you want that side called in the notebook and the write-up.
# The two sides usually serve different models, because each serves the largest
# one its own memory can hold, and saying so in the name keeps that visible.
SIDE_A_NAME=
SIDE_A_URL=
SIDE_A_MODEL=
SIDE_B_NAME=
SIDE_B_URL=
SIDE_B_MODEL=

# Requests in flight for the batch measurement. Eight is enough to separate a
# machine that batches well from one that does not, without needing a load
# generator on a third machine.
BATCH_CONCURRENCY=8

# ------------------------------------------------------------------ the record
# The lab notebook every script appends its JSON line to.
LABBOOK=labbook.md

Download env-example.txt126 lines

RunnableTrack M · Apple silicon

settings, on Mac A

cp env-example.txt .env

=Expected result

ssh is ready, and two identical Python paths. If the paths differ, fix that now: install mlx-lm into the same location on both machines, or set CLUSTER_PYTHON in .env to a path that exists on both. A mismatch here produces a launch failure whose message is about a missing file rather than about Python, and it is the single commonest way to lose twenty minutes in this part.

2. Enable remote direct memory access, once per Mac

This step is optional. Skip it and the whole lab still works over the ring backend; do it and you get the JACCL backend to compare against. It cannot be automated, and both MLX and exo say so. MLX’s documentation: enabling it “cannot be done remotely even with sudo. In fact, it has to be done in macOS recovery.” The steps both projects give are identical:

1. Shut the Mac down.
2. Hold the power button for about ten seconds, until the boot options appear.
3. Choose Options to enter Recovery, and open Terminal from the Utilities menu.
4. Run rdma_ctl enable.
5. Reboot.

Repeat on the second Mac. Then check, on each:

RunnableTrack M · Apple silicon

did it take?

ibv_devices

=Expected result

A short table of device names and node identifiers, one row per Thunderbolt interface, with names of the form rdma_en2. An empty table or a missing command means RDMA is not enabled on this machine: repeat the Recovery step. There is no way to check this over SSH before the reboot, which is why it is worth doing both machines in one sitting.

✱Hardware note

exo’s README lists three physical caveats that apply whatever software you use. The cables “must support TB5”; every machine in an RDMA cluster “must be connected to all other devices in the cluster”, so a third Mac needs two more cables rather than one; and on a Mac Studio “you cannot use the Thunderbolt 5 port next to the Ethernet port”. Check the last one against the back of the machine before you cable it, not after.

3. Discover the topology and configure the link

macOS presents Thunderbolt cables as a single bridged network service. MLX takes that apart: the bridge comes down, and each cable becomes its own point-to-point subnet. The helper script checks everything first and then hands the work to mlx.distributed_config, which prints the commands rather than running them unless you have password-less sudo.

RunnableTrack M · Apple silicon

setup-thunderbolt-bridge.sh

#!/usr/bin/env bash
# Purpose: check that two Macs are ready to be clustered over Thunderbolt, then ask
#          mlx.distributed_config to work out the topology and write the host file.
#          It changes nothing by itself: every command that would touch a network
#          interface is printed for you to read and run, because taking the
#          Thunderbolt bridge down is a change to how the machine networks.
# Platform: mac (Track M) only. Needs macOS 26 or later on both machines, and macOS
#           26.2 or later plus Thunderbolt 5 for the jaccl backend.
# Minimum memory: 32 GB per Mac for the primary path; 24 GB for the single-Mac path.
# Assumes: mlx and mlx-lm installed in the python on PATH, an .env beside this script
#          copied from env-example.txt, password-less ssh to PEER_SSH already working
#          (Part 18's lab sets that up), and `dot` from Graphviz if you want the
#          topology drawing. Writes the host file named by HOSTFILE.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${MACHINE_NAME:=}"
: "${PEER_SSH:=}"
: "${SELF_SSH:=}"
: "${MLX_BACKEND:=ring}"
: "${HOSTFILE:=hosts.json}"

if [ "$(uname -s)" != "Darwin" ]; then
    echo "This script is Track M only: it reads macOS-specific tools." >&2
    echo "On another track, the lab's fallback section points at Part 19." >&2
    exit 1
fi

if [ -z "$MACHINE_NAME" ]; then
    echo "MACHINE_NAME is empty in ${ENV_FILE}." >&2
    exit 1
fi

echo "==> 1. This machine"
sw_vers
sysctl -n machdep.cpu.brand_string || true
echo ""

OS_VERSION="$(sw_vers -productVersion)"
OS_MAJOR="${OS_VERSION%%.*}"
OS_REST="${OS_VERSION#*.}"
OS_MINOR="${OS_REST%%.*}"
if [ "$OS_MINOR" = "$OS_VERSION" ]; then
    OS_MINOR=0
fi

echo "==> 2. Thunderbolt ports and the interfaces they map to"
echo "    system_profiler names the ports; networksetup names the interfaces."
system_profiler SPThunderboltDataType | sed -n '1,60p'
echo ""
networksetup -listallhardwareports | grep -A2 -i thunderbolt || \
    echo "    No hardware port with Thunderbolt in its name. Check the cable."
echo ""

echo "==> 3. The Thunderbolt bridge, which the cluster does not want"
echo "    macOS presents Thunderbolt cables as one bridged service. MLX takes them"
echo "    apart into per-cable subnets, and says the bridge must be down even for"
echo "    the RDMA backend, which does not use TCP/IP for model traffic at all."
ifconfig bridge0 2>/dev/null | sed -n '1,4p' || echo "    No bridge0 on this machine."
echo ""

if [ "$MLX_BACKEND" = "jaccl" ]; then
    echo "==> 4. RDMA devices"
    if [ "$OS_MAJOR" -lt 26 ] || { [ "$OS_MAJOR" -eq 26 ] && [ "$OS_MINOR" -lt 2 ]; }; then
        echo "    macOS ${OS_VERSION} is older than 26.2, which is the documented" >&2
        echo "    floor for RDMA over Thunderbolt. Use MLX_BACKEND=ring." >&2
        exit 1
    fi
    if ! command -v ibv_devices >/dev/null 2>&1; then
        echo "    ibv_devices is not on PATH; RDMA has not been enabled here." >&2
        echo "    Enable it once from macOS Recovery with 'rdma_ctl enable', reboot," >&2
        echo "    and run this script again. It cannot be done over ssh." >&2
        exit 1
    fi
    ibv_devices
    echo ""
else
    echo "==> 4. RDMA devices: skipped, MLX_BACKEND is ${MLX_BACKEND}"
    echo ""
fi

if [ -z "$PEER_SSH" ]; then
    echo "==> 5. PEER_SSH is empty: treating this as the single-Mac path."
    echo "    No host file is needed. Launch two ranks on this machine with:"
    echo "      mlx.launch -n 2 -- python check-group.py"
    exit 0
fi

echo "==> 5. The peer"
SSH_TARGET="$PEER_SSH"
if [ -n "${SSH_USER:-}" ]; then
    SSH_TARGET="${SSH_USER}@${PEER_SSH}"
fi
if ! ssh -o BatchMode=yes -o ConnectTimeout=10 "$SSH_TARGET" true; then
    echo "    Cannot ssh to ${PEER_SSH} without a password." >&2
    echo "    mlx.launch starts every rank over ssh, so fix this first." >&2
    exit 1
fi
echo "    ssh to ${PEER_SSH} works with no password."
ssh "$SSH_TARGET" sw_vers -productVersion | sed 's/^/    peer macOS: /'
echo ""

echo "==> 6. Both Macs, as mlx.distributed_config sees them"
HOSTS="${SELF_SSH:-localhost},${PEER_SSH}"
echo "    hosts: ${HOSTS}"
mlx.distributed_config --verbose --hosts "$HOSTS" --over thunderbolt --dot \
    > "${HERE}/topology.dot"
echo "    Topology written to ${HERE}/topology.dot."
if command -v dot >/dev/null 2>&1; then
    echo "    Render it with:  dot -Tpng topology.dot -o topology.png"
else
    echo "    Install Graphviz to render it: brew install graphviz"
fi
echo ""

echo "==> 7. The configuration commands, printed rather than run"
echo "    Read every line before you run any of them. They bring the bridge"
echo "    interface down and give each cable's interface its own tiny subnet."
echo "    Answer the helper's prompt only once you have run them on that machine."
mlx.distributed_config --verbose --hosts "$HOSTS" --over thunderbolt \
    --backend "$MLX_BACKEND" --output-hostfile "${HERE}/${HOSTFILE}"

echo ""
echo "Host file written to ${HERE}/${HOSTFILE}."
echo "Check it against hosts-example.json, then run:"
echo "  bash run-distributed.sh"

Download setup-thunderbolt-bridge.sh138 lines

RunnableTrack M · Apple silicon

check, discover, and write the host file

bash setup-thunderbolt-bridge.sh

The script’s first six steps are all reading: the macOS version, the chip, the Thunderbolt ports from system_profiler and the interface names they map to from networksetup, the state of bridge0, the RDMA devices if you enabled them, and whether the peer answers. Step seven is where the helper takes over and offers you the configuration.

Read the commands it prints before you run any of them. They do three things per cable: bring the bridge interface down, give the cable’s interface an address on its own small subnet, and add a route to the peer through that interface. That is the whole of what a Thunderbolt cluster network is, and having read it once you will never again wonder why the bridge has to go.

If Graphviz is installed, the script also leaves a topology.dot file behind. Render it. MLX’s documentation recommends exactly this when something is miscabled: it “makes it very easy to figure out which cable is not connected correctly”, which matters more with four machines than with two but costs nothing here.

4. Read the host file before you trust it

The helper writes hosts.json. Compare it against the shape below, which is what a two-Mac JACCL host file looks like with the values removed.

Fragment — not complete on its own

hosts-example.json

[
  {
    "ssh": "REPLACE-WITH-THE-NAME-YOU-SSH-TO-FOR-MAC-A",
    "ips": ["REPLACE-WITH-MAC-A-ADDRESS-ON-THE-THUNDERBOLT-CABLE"],
    "rdma": [null, "REPLACE-WITH-THE-RDMA-DEVICE-ON-MAC-A-FACING-MAC-B"]
  },
  {
    "ssh": "REPLACE-WITH-THE-NAME-YOU-SSH-TO-FOR-MAC-B",
    "ips": [],
    "rdma": ["REPLACE-WITH-THE-RDMA-DEVICE-ON-MAC-B-FACING-MAC-A", null]
  }
]

Download hosts-example.json12 lines

Three fields, three jobs. ssh is the name mlx.launch logs in to, over your house network. ips are the addresses the ranks bind to for model traffic, on the Thunderbolt cable; for the JACCL backend only rank 0 needs one, because it is used to exchange connection details before RDMA takes over. rdma names the device on this machine that faces each other machine, with null where a machine would be facing itself. A ring host file is the same thing without the rdma array.

✕Common mistake

The commonest error here is an ssh name that resolves to the Thunderbolt address instead of the house-network one. The launcher then tries to open an SSH session over a link that exists only because both machines are already up and configured, which works until it does not. Keep the two names apart: the home.arpa name is for SSH, the cable address is for tensors.

5. Prove the group forms, and time it

Before any model, run the smallest possible distributed program. It joins the group, performs an all-sum whose answer is known in advance, and then times repeated all-sums at four array sizes so that a latency-bound case and a bandwidth-bound case are both covered.

RunnableTrack M · Apple silicon

check-group.py

"""Prove that an MLX distributed group forms, and time one collective on it.

Purpose: the first thing to run on a new cluster, before any model is involved. It
    joins the distributed group, reports the rank and size every process sees, runs
    an all-sum whose result is known in advance so a wrong answer is obvious, and
    then times repeated all-sums at several array sizes so that the ring backend and
    the JACCL backend can be compared on the same cable. Rank 0 appends one JSON line
    to the lab notebook.
Platform: mac (Track M) for the two-Mac path; the same script runs anywhere MLX runs,
    including two ranks on one machine, which is the single-machine path.
Minimum memory: 32 GB per Mac for the primary path; 24 GB for the single-Mac path.
    This script itself allocates a few hundred megabytes at the largest size.
Assumes: mlx installed in the python that mlx.launch starts, and this file present at
    the SAME absolute path on every machine. Launched by mlx.launch, never directly:
    run with plain python it reports a group of size one, which is correct and dull.

Usage:
    mlx.launch -n 2 -- python check-group.py --label single-machine
    mlx.launch --backend ring --hostfile hosts.json -- \
        /path/to/python check-group.py --label two-macs-ring
    mlx.launch --backend jaccl --hostfile hosts.json -- \
        /path/to/python check-group.py --label two-macs-jaccl
    python3 check-group.py --print            (group of one; writes nothing)
"""

from __future__ import annotations

import argparse
import json
import platform
import time
from datetime import date, datetime, timezone
from pathlib import Path

import mlx.core as mx

HERE = Path(__file__).resolve().parent

# Array sizes to time, in float32 elements. The smallest is latency-dominated and
# the largest is bandwidth-dominated, which is exactly the difference between the
# ring backend and an RDMA backend that the lesson describes.
DEFAULT_SIZES = [1024, 262144, 4194304, 33554432]


def parse_args():
    p = argparse.ArgumentParser(description="MLX distributed group check")
    p.add_argument(
        "--backend",
        default="any",
        choices=["any", "ring", "jaccl", "mpi", "nccl"],
        help="Backend to request from mx.distributed.init(). Default: any.",
    )
    p.add_argument(
        "--sizes",
        default=",".join(str(s) for s in DEFAULT_SIZES),
        help="Comma-separated float32 element counts to time.",
    )
    p.add_argument("--repeat", type=int, default=20, help="All-sums per size.")
    p.add_argument("--warmup", type=int, default=5, help="Untimed all-sums per size.")
    p.add_argument(
        "--label",
        default="",
        help="A name for this run, for example two-macs-jaccl. Goes in the record.",
    )
    p.add_argument("--labbook", default=str(HERE / "labbook.md"))
    p.add_argument(
        "--print",
        dest="print_only",
        action="store_true",
        help="Show the record and write nothing.",
    )
    return p.parse_args()


def time_all_sum(elements: int, repeat: int, warmup: int) -> dict:
    """Average seconds per all-sum at this size, and the bytes each one moves."""
    x = mx.ones(elements, dtype=mx.float32)
    for _ in range(warmup):
        mx.eval(mx.distributed.all_sum(x))
    start = time.perf_counter()
    for _ in range(repeat):
        mx.eval(mx.distributed.all_sum(x))
    elapsed = time.perf_counter() - start
    return {
        "elements": elements,
        "bytes_per_array": elements * 4,
        "repeats": repeat,
        "seconds_per_all_sum": elapsed / repeat,
    }


def main() -> int:
    args = parse_args()

    world = mx.distributed.init(backend=args.backend)
    rank = world.rank()
    size = world.size()

    # A result every process can check without trusting the network: an array of
    # ones summed across `size` processes must be exactly `size` everywhere.
    probe = mx.distributed.all_sum(mx.ones(8, dtype=mx.float32))
    mx.eval(probe)
    expected = float(size)
    observed = [float(v) for v in probe.tolist()]
    correct = all(abs(v - expected) < 1e-6 for v in observed)

    print(f"rank {rank} of {size}: all_sum(ones) = {observed[0]}, expected {expected}")
    if not correct:
        print(f"rank {rank}: WRONG RESULT. The group formed but the maths did not.")
        return 1

    if size == 1:
        print(
            "Group size is 1, so every collective was a noop. That is the expected "
            "result for plain python; launch with mlx.launch to get a real group."
        )

    sizes = [int(s) for s in args.sizes.split(",") if s.strip()]
    timings = [time_all_sum(n, args.repeat, args.warmup) for n in sizes]

    record = {
        "lab": "part-21/two-mac-cluster-over-thunderbolt-5",
        "record": "group-check",
        "date": date.today().isoformat(),
        "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
        "label": args.label,
        "backend_requested": args.backend,
        "group_size": size,
        "all_sum_correct": correct,
        "machine": platform.node().split(".")[0],
        "platform": platform.platform(),
        "timings": timings,
    }

    if rank != 0:
        return 0

    text = json.dumps(record, sort_keys=True)
    if args.print_only:
        print(text)
        return 0
    with open(args.labbook, "a", encoding="utf-8") as fh:
        fh.write(text + "\n")
    print(text)
    print(f"Appended to {args.labbook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download check-group.py150 lines

RunnableTrack M · Apple silicon

two Macs, ring backend over Thunderbolt

mlx.launch --verbose --backend ring --hostfile hosts.json -- python check-group.py --label ring

Everything after the bare -- is the command each rank runs, so python there has to be the interpreter that has mlx-lm in it, at the same path on both machines. If a bare python is not that interpreter, write the full path you confirmed in task 1 instead; the wrapper script in the next task reads it from CLUSTER_PYTHON so you only have to get it right once.

If you enabled RDMA in task 2, run it again over JACCL. This is the cleanest comparison in the whole part: the same script, the same cable, the same machines, one word different.

RunnableTrack M · Apple silicon

the same pair, RDMA backend

mlx.launch --verbose --backend jaccl --hostfile hosts.json -- python check-group.py --label jaccl

=Expected result

Two lines, one per rank, each reporting all_sum(ones) = 2.0, expected 2.0, followed by a JSON line in your notebook holding the per-size timings. A wrong sum means the group formed but the transport is corrupting data, which is a hardware or cabling fault and not something to work around. Nothing at all, or a hang, means the ranks did not find each other: check the addresses in the host file against ifconfig on both machines.

The two records are the first thing worth looking at. The smallest array size is dominated by the cost of one round trip, and the largest by how fast the cable moves bytes. MLX’s documentation claims JACCL cuts latency by roughly a factor of ten against the ring backend; your smallest-size rows are where you can see whether that held on your machines.

6. Run one model across both machines

Now the model. This script loads it split across the group, generates once, and records prefill speed, decode speed and peak memory per rank.

RunnableTrack M · Apple silicon

sharded-generate.py

"""Generate once from a model split across the cluster, and record what it cost.

Purpose: the shortest path from a configured cluster to a number worth writing down.
    It loads one model across the distributed group, generates a completion, and
    records prefill speed, decode speed, token counts and peak memory per rank in the
    lab notebook. The --pipeline flag selects pipelining instead of tensor
    parallelism, which is the one decision this script makes differently from run to
    run and the one the lab asks you to compare.
Platform: mac (Track M). Runs on one machine too, as a group of one or as two ranks
    launched with `mlx.launch -n 2`, which is the single-machine path.
Minimum memory: 32 GB per Mac for a model larger than one machine; 24 GB for the
    single-Mac baseline with a smaller model.
Assumes: mlx-lm installed in the python mlx.launch starts, this file present at the
    SAME absolute path on every machine, and the model repository readable by every
    rank. With --pipeline each rank downloads only its own shard; without it every
    rank downloads the whole repository, so check the disk on both machines first.

Usage:
    mlx.launch --backend jaccl --hostfile hosts.json -- \
        /path/to/python sharded-generate.py --model mlx-community/Qwen3-32B-8bit \
        --label two-macs-jaccl-tensor

    mlx.launch --backend ring --hostfile hosts.json -- \
        /path/to/python sharded-generate.py --model mlx-community/Qwen3-32B-8bit \
        --pipeline --label two-macs-ring-pipeline

    python3 sharded-generate.py --model mlx-community/Qwen3-8B-4bit --label one-mac
"""

from __future__ import annotations

import argparse
import json
import platform
from datetime import date, datetime, timezone
from pathlib import Path

import mlx.core as mx
from mlx_lm import stream_generate
from mlx_lm.utils import sharded_load

HERE = Path(__file__).resolve().parent

# Long enough that prefill is measurable rather than noise, and boring enough that
# the answer is not the point. Repeat it to reach the prompt length you want.
DEFAULT_PROMPT = (
    "Explain, for an engineer who has just built a two-machine cluster, why the "
    "prompt-processing phase and the token-generation phase of a language model "
    "have different bottlenecks, what each phase asks of the link between the two "
    "machines, and which of the two a faster cable actually helps. Answer in no "
    "more than six sentences."
)


def parse_args():
    p = argparse.ArgumentParser(description="Distributed generation with mlx-lm")
    p.add_argument("--model", required=True, help="MLX repository or local path.")
    p.add_argument("--prompt", default=DEFAULT_PROMPT, help="The prompt text.")
    p.add_argument(
        "--prompt-file",
        default=None,
        help="Read the prompt from this file instead of --prompt.",
    )
    p.add_argument(
        "--prompt-repeat",
        type=int,
        default=1,
        help="Repeat the prompt this many times, to lengthen prefill.",
    )
    p.add_argument("--max-tokens", type=int, default=256)
    p.add_argument(
        "--pipeline",
        action="store_true",
        help="Use pipelining instead of tensor parallelism.",
    )
    p.add_argument("--label", default="", help="A name for this run, for the record.")
    p.add_argument("--labbook", default=str(HERE / "labbook.md"))
    p.add_argument(
        "--quiet",
        action="store_true",
        help="Do not stream the answer; report only the numbers.",
    )
    return p.parse_args()


def build_prompt(args) -> str:
    if args.prompt_file:
        text = Path(args.prompt_file).read_text(encoding="utf-8")
    else:
        text = args.prompt
    return ("\n\n".join([text] * max(1, args.prompt_repeat))).strip()


def main() -> int:
    args = parse_args()

    group = mx.distributed.init()
    rank = group.rank()
    size = group.size()

    def rprint(*a, **kw):
        if rank == 0:
            print(*a, **kw)

    pipeline_group = group if args.pipeline else None
    tensor_group = None if args.pipeline else group

    rprint(f"loading {args.model} across {size} rank(s)")
    rprint("split: " + ("pipeline" if args.pipeline else "tensor parallel"))
    model, tokenizer = sharded_load(args.model, pipeline_group, tensor_group)

    messages = [{"role": "user", "content": build_prompt(args)}]
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

    response = None
    for response in stream_generate(
        model, tokenizer, prompt, max_tokens=args.max_tokens
    ):
        if not args.quiet:
            rprint(response.text, end="", flush=True)
    if not args.quiet:
        rprint()

    if response is None:
        rprint("No tokens were generated. Check --max-tokens and the prompt.")
        return 1

    # Every rank knows its own peak memory; only rank 0 writes the notebook line,
    # so the peaks are gathered rather than reported separately per machine.
    peaks = mx.distributed.all_gather(mx.array([response.peak_memory], mx.float32))
    mx.eval(peaks)
    peak_by_rank = [round(float(v), 3) for v in peaks.tolist()]

    if rank != 0:
        return 0

    record = {
        "lab": "part-21/two-mac-cluster-over-thunderbolt-5",
        "record": "sharded-generate",
        "date": date.today().isoformat(),
        "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
        "label": args.label,
        "model": args.model,
        "group_size": size,
        "split": "pipeline" if args.pipeline else "tensor",
        "machine": platform.node().split(".")[0],
        "prompt_tokens": response.prompt_tokens,
        "prompt_tokens_per_second": round(response.prompt_tps, 3),
        "generation_tokens": response.generation_tokens,
        "generation_tokens_per_second": round(response.generation_tps, 3),
        "peak_memory_gb_by_rank": peak_by_rank,
    }
    text = json.dumps(record, sort_keys=True)
    with open(args.labbook, "a", encoding="utf-8") as fh:
        fh.write(text + "\n")
    print(text)
    print(f"Appended to {args.labbook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download sharded-generate.py162 lines

The wrapper builds the launcher command from .env, so switching backend or split is one variable rather than one retyped command.

RunnableTrack M · Apple silicon

run-distributed.sh

#!/usr/bin/env bash
# Purpose: launch one model across the cluster with mlx.launch, either as a one-shot
#          generation that records its own numbers or as an OpenAI-compatible server
#          that measure-pair.py can then drive. It builds the mlx.launch command from
#          .env so that the ring and jaccl backends, and the tensor and pipeline
#          splits, are one variable apart rather than one retyped command apart.
# Platform: mac (Track M). With PEER_SSH empty it launches both ranks on this Mac,
#           which is the single-machine path.
# Minimum memory: 32 GB per Mac for the primary path; 24 GB for the single-Mac path.
# Assumes: mlx-lm installed at the same python path on both Macs, this directory
#          present at the same absolute path on both, a host file written by
#          setup-thunderbolt-bridge.sh, and an .env copied from env-example.txt.
#          Serving mode runs in the foreground until you stop it with Ctrl-C.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${PEER_SSH:=}"
: "${MLX_BACKEND:=ring}"
: "${HOSTFILE:=hosts.json}"
: "${RANKS:=2}"
: "${CLUSTER_PYTHON:=}"
: "${CLUSTER_DIR:=$HERE}"
: "${MODEL_REPO:=}"
: "${BASELINE_REPO:=}"
: "${USE_PIPELINE:=}"
: "${SERVER_HOST:=127.0.0.1}"
: "${SERVER_PORT:=8080}"
: "${MAX_TOKENS:=256}"
: "${LABBOOK:=${HERE}/labbook.md}"

MODE="${1:-generate}"

usage() {
    cat <<'USAGE'
Usage: bash run-distributed.sh [generate|serve|baseline]

  generate   one distributed completion, numbers appended to the lab notebook
  serve      mlx_lm.server across the cluster, for measure-pair.py to drive
  baseline   one completion of BASELINE_REPO on this Mac alone, for the first row
USAGE
}

case "$MODE" in
    generate|serve|baseline) ;;
    -h|--help) usage; exit 0 ;;
    *) usage >&2; exit 1 ;;
esac

if ! command -v mlx.launch >/dev/null 2>&1; then
    echo "mlx.launch is not on PATH. Install mlx-lm in this python first." >&2
    exit 1
fi

if [ -z "$CLUSTER_PYTHON" ]; then
    CLUSTER_PYTHON="$(mlx.launch --print-python)"
    echo "==> using the python mlx.launch reports: ${CLUSTER_PYTHON}"
    echo "    Confirm the same path exists on the other Mac before going further."
fi

# ------------------------------------------------------- the launcher arguments
LAUNCH=(mlx.launch --verbose --backend "$MLX_BACKEND")

if [ -n "$PEER_SSH" ]; then
    if [ ! -f "${HERE}/${HOSTFILE}" ]; then
        echo "No ${HERE}/${HOSTFILE}. Run setup-thunderbolt-bridge.sh first." >&2
        exit 1
    fi
    LAUNCH+=(--hostfile "${HERE}/${HOSTFILE}")
else
    echo "==> PEER_SSH is empty: single-machine path, ${RANKS} ranks on this Mac."
    echo "    The mechanism is the same; the memory is not."
    LAUNCH+=(-n "$RANKS")
fi

LAUNCH+=(--cwd "$CLUSTER_DIR")

SPLIT_ARGS=()
if [ -n "$USE_PIPELINE" ]; then
    SPLIT_ARGS+=(--pipeline)
fi

LABEL="${MLX_BACKEND}-$([ -n "$USE_PIPELINE" ] && echo pipeline || echo tensor)"

# --------------------------------------------------------------------- baseline
if [ "$MODE" = "baseline" ]; then
    if [ -z "$BASELINE_REPO" ]; then
        echo "BASELINE_REPO is empty in ${ENV_FILE}." >&2
        exit 1
    fi
    echo "==> one Mac, ${BASELINE_REPO}, no distributed group"
    "$CLUSTER_PYTHON" "${CLUSTER_DIR}/sharded-generate.py" \
        --model "$BASELINE_REPO" \
        --max-tokens "$MAX_TOKENS" \
        --label "one-mac-baseline" \
        --labbook "$LABBOOK"
    exit 0
fi

if [ -z "$MODEL_REPO" ]; then
    echo "MODEL_REPO is empty in ${ENV_FILE}." >&2
    exit 1
fi

# --------------------------------------------------------------------- generate
if [ "$MODE" = "generate" ]; then
    echo "==> ${MODEL_REPO} across the group, backend ${MLX_BACKEND}, split ${LABEL}"
    "${LAUNCH[@]}" -- \
        "$CLUSTER_PYTHON" "${CLUSTER_DIR}/sharded-generate.py" \
        --model "$MODEL_REPO" \
        --max-tokens "$MAX_TOKENS" \
        --label "$LABEL" \
        --labbook "$LABBOOK" \
        "${SPLIT_ARGS[@]+"${SPLIT_ARGS[@]}"}"
    exit 0
fi

# ------------------------------------------------------------------------ serve
echo "==> serving ${MODEL_REPO} across the group on ${SERVER_HOST}:${SERVER_PORT}"
echo "    Rank 0 answers HTTP; the other ranks do their share and stay quiet."
echo "    Leave this running and drive it from another terminal with:"
echo "      python3 measure-pair.py --label ${LABEL}"
echo "    Stop it with Ctrl-C when the measurement is recorded."
"${LAUNCH[@]}" -- \
    "$CLUSTER_PYTHON" -m mlx_lm.server \
    --model "$MODEL_REPO" \
    --host "$SERVER_HOST" \
    --port "$SERVER_PORT" \
    "${SPLIT_ARGS[@]+"${SPLIT_ARGS[@]}"}"

Download run-distributed.sh140 lines

Start with the baseline: one Mac, a model that fits, no distributed group at all. Without this row the cluster numbers have nothing to be compared against.

RunnableTrack M · Apple silicon

the row you will compare everything to

bash run-distributed.sh baseline

Then the split model. The first run downloads weights, which is the unattended part of the lab.

RunnableTrack M · Apple silicon

one completion, split across both Macs

bash run-distributed.sh generate

Set USE_PIPELINE=1 in .env and run it again to get the other split. If the model refuses one of the two with a message about not supporting pipelining or not supporting sharding, that is the per-implementation limit the previous lesson described, and the answer is to record which splits this model supports and move on rather than to hunt for a flag.

▤Memory budget

Watch peak memory per rank in the notebook line, not just the total. Tensor parallelism divides the weights and the key-value cache; pipelining divides the weights and leaves the cache wherever the attention layers landed. On an uneven pair of Macs those are meaningfully different pictures, and the record captures both because every rank reports its own peak.

7. Serve it, so it can be measured properly

A single generation tells you the machine can do it. A served endpoint under repeated requests tells you what it is like to use.

RunnableTrack M · Apple silicon

on Mac A, and leave it running

bash run-distributed.sh serve

The launcher starts the server process on both ranks; rank 0 answers HTTP and the other rank does its share of the arithmetic quietly. Leave this terminal alone until the measurement is recorded.

8. Measure prefill, decode and the cable

From a second terminal on Mac A, drive the endpoint and read the interface counters around the run. The FreeBSD manual, which macOS follows here, documents netstat -i as showing “the state of all network interfaces” and -b as showing “the number of bytes in and out”; the script takes a reading before and after and records the difference.

RunnableTrack M · Apple silicon

measure-pair.py

"""Measure prefill, decode and link use on a served cluster, and record all three.

Purpose: drive an OpenAI-compatible endpoint that is being served by a cluster, and
    record the numbers that decide whether the cluster was worth building. With
    --concurrency 1 it answers the single-user question; with a larger value it
    answers the batch question, and the two are different questions.
    Prefill is measured as prompt tokens divided by time to first token on a long
    prompt. Decode is measured as generated tokens divided by the time between the
    first and last token. Link use is read from the interface byte counters before
    and after the run, so you can see how much traffic the split actually put on the
    cable rather than assuming. One JSON line per run goes to the lab notebook.
Platform: mac (Track M) for the link counters, which come from BSD `netstat -ibn`;
    the timing half runs anywhere python3 does, including against a server on another
    machine. On Linux the link section is recorded as unavailable rather than faked.
Minimum memory: none on the machine running this script. It streams text and does
    no inference of its own, so it can be run from a laptop against the cluster.
Assumes: a server already answering POST {base-url}/chat/completions with streaming,
    reachable over plain HTTP; this is a house-network tool and does not speak TLS.
    An API key, if the server wants one, is read from the environment variable named
    by --api-key-env and is never written to this file or to the notebook.

Counting note: one streamed delta carrying content counts as one output token, which
    is exact for mlx_lm.server and for exo as this course reads them. Prefill speed
    from time to first token includes queueing and template rendering, so it is a
    lower bound on the engine's own prefill rate; --exo-bench-url asks exo for its
    own figures instead, which are measured inside the server.

Usage:
    python3 measure-pair.py --label two-macs-jaccl-tensor

    python3 measure-pair.py --base-url http://127.0.0.1:8080/v1 \
        --model mlx-community/Qwen3-32B-8bit --requests 20 --max-tokens 128 \
        --iface en2 --label two-macs-ring-tensor --labbook labbook.md

    python3 measure-pair.py --base-url http://localhost:52415/v1 \
        --exo-bench-url http://localhost:52415 --label exo-jaccl

    python3 measure-pair.py --concurrency 8 --requests 40 --label batch-of-eight
"""

from __future__ import annotations

import argparse
import json
import os
import platform
import shutil
import statistics
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request
from datetime import date, datetime, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent

# Filler with no interesting answer in it, repeated to reach the prompt length the
# prefill measurement wants. The model is being timed, not questioned.
FILLER = (
    "A cluster is two machines and a cable. The weights are divided between them, "
    "the activations cross the cable, and every design question in this part is "
    "about how often that happens and how many bytes go each time. "
)

QUESTION = "In one sentence, what limits decode speed on a single machine?"

# Roughly four characters per token for English prose. This is only used to build a
# prompt of about the right length; the exact prompt token count comes back from the
# server in the usage field where the server reports one.
CHARS_PER_TOKEN = 4


def parse_args():
    p = argparse.ArgumentParser(description="Measure a served cluster")
    p.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    p.add_argument(
        "--model",
        default=None,
        help="Model name the server reports at /models. Default: ask the server.",
    )
    p.add_argument("--requests", type=int, default=20)
    p.add_argument(
        "--concurrency",
        type=int,
        default=1,
        help="Requests in flight at once. 1 is the single-user case; a larger "
        "number is the batch case, and the two answer different questions.",
    )
    p.add_argument("--max-tokens", type=int, default=128)
    p.add_argument(
        "--prefill-tokens",
        type=int,
        default=2048,
        help="Approximate prompt length for the prefill measurement.",
    )
    p.add_argument(
        "--iface",
        default=os.environ.get("LINK_IFACE", ""),
        help="Interface to read byte counters for. Empty records every interface.",
    )
    p.add_argument(
        "--exo-bench-url",
        default=None,
        help="exo base URL; use its /bench/chat/completions for server-side figures.",
    )
    p.add_argument("--api-key-env", default="LLM_API_KEY")
    p.add_argument("--label", default="", help="A name for this run, for the record.")
    p.add_argument("--labbook", default=str(HERE / "labbook.md"))
    p.add_argument("--timeout", type=float, default=600.0)
    p.add_argument(
        "--print",
        dest="print_only",
        action="store_true",
        help="Show the record and write nothing.",
    )
    return p.parse_args()


# ------------------------------------------------------------------ link counters

def read_link_counters(iface: str):
    """Bytes in and out per interface, from BSD netstat. None where unavailable."""
    if shutil.which("netstat") is None:
        return None
    cmd = ["netstat", "-ibn"]
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=20, check=False)
    except (OSError, subprocess.SubprocessError):
        return None
    if out.returncode != 0 or not out.stdout:
        return None

    lines = out.stdout.splitlines()
    if not lines:
        return None
    header = lines[0].split()
    try:
        i_name = header.index("Name")
        i_net = header.index("Network")
        i_in = header.index("Ibytes")
        i_out = header.index("Obytes")
    except ValueError:
        # A Linux netstat has none of these columns; say so rather than guess.
        return None

    counters = {}
    for line in lines[1:]:
        fields = line.split()
        if len(fields) <= max(i_in, i_out, i_net):
            continue
        name = fields[i_name]
        if iface and name != iface:
            continue
        # The link-layer row is the one that counts every packet on the interface;
        # the address rows repeat a subset and would double-count.
        if not fields[i_net].startswith("<Link"):
            continue
        try:
            counters[name] = {
                "bytes_in": int(fields[i_in]),
                "bytes_out": int(fields[i_out]),
            }
        except ValueError:
            continue
    return counters or None


def link_delta(before, after):
    if not before or not after:
        return None
    delta = {}
    for name, start in before.items():
        end = after.get(name)
        if not end:
            continue
        delta[name] = {
            "bytes_in": end["bytes_in"] - start["bytes_in"],
            "bytes_out": end["bytes_out"] - start["bytes_out"],
        }
    return delta or None


# ------------------------------------------------------------------- HTTP helpers

def post_json(url: str, body: dict, headers: dict, timeout: float):
    data = json.dumps(body).encode("utf-8")
    request = urllib.request.Request(url, data=data, headers=headers, method="POST")
    return urllib.request.urlopen(request, timeout=timeout)  # noqa: S310


def discover_model(base_url: str, headers: dict, timeout: float):
    url = base_url.rstrip("/") + "/models"
    try:
        request = urllib.request.Request(url, headers=headers)
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310
            payload = json.loads(response.read().decode("utf-8"))
    except (urllib.error.URLError, ValueError, OSError):
        return None
    entries = payload.get("data") or payload.get("models") or []
    if isinstance(entries, list) and entries:
        first = entries[0]
        if isinstance(first, dict):
            return first.get("id") or first.get("name")
        if isinstance(first, str):
            return first
    return None


def build_prompt(target_tokens: int) -> str:
    repeats = max(1, (target_tokens * CHARS_PER_TOKEN) // len(FILLER))
    return (FILLER * repeats) + "\n\n" + QUESTION


# ------------------------------------------------------------------ one streamed run

def stream_once(url, body, headers, timeout):
    """Time to first token, total time, and how many content deltas arrived."""
    started = time.perf_counter()
    first_token_at = None
    tokens = 0
    with post_json(url, body, headers, timeout) as response:
        for raw in response:
            line = raw.decode("utf-8", errors="replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            try:
                chunk = json.loads(payload)
            except ValueError:
                continue
            choices = chunk.get("choices") or []
            if not choices:
                continue
            delta = choices[0].get("delta") or {}
            content = delta.get("content")
            if not content:
                continue
            if first_token_at is None:
                first_token_at = time.perf_counter()
            tokens += 1
    finished = time.perf_counter()
    if first_token_at is None:
        return None
    return {
        "time_to_first_token_s": first_token_at - started,
        "decode_seconds": max(finished - first_token_at, 1e-9),
        "output_tokens": tokens,
        "total_seconds": finished - started,
    }


def bench_once(exo_base, body, headers, timeout):
    """exo's own prefill and decode figures, measured inside the server."""
    url = exo_base.rstrip("/") + "/bench/chat/completions"
    request_body = dict(body)
    request_body["stream"] = False
    with post_json(url, request_body, headers, timeout) as response:
        payload = json.loads(response.read().decode("utf-8"))
    keys = ("prompt_tps", "generation_tps", "prompt_tokens", "generation_tokens",
            "peak_memory_usage")
    return {k: payload[k] for k in keys if k in payload}


def summarise(values):
    if not values:
        return None
    ordered = sorted(values)
    return {
        "median": round(statistics.median(ordered), 4),
        "p90": round(ordered[min(len(ordered) - 1, int(0.9 * len(ordered)))], 4),
        "min": round(ordered[0], 4),
        "max": round(ordered[-1], 4),
        "n": len(ordered),
    }


def main() -> int:
    args = parse_args()

    headers = {"Content-Type": "application/json"}
    key = os.environ.get(args.api_key_env, "")
    if key:
        headers["Authorization"] = f"Bearer {key}"

    model = args.model or discover_model(args.base_url, headers, 30.0)
    if not model:
        print(
            "No model given and the server did not name one at /models. "
            "Pass --model.",
            file=sys.stderr,
        )
        return 1

    prompt = build_prompt(args.prefill_tokens)
    body = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": args.max_tokens,
        "temperature": 0.0,
        "stream": True,
    }
    url = args.base_url.rstrip("/") + "/chat/completions"

    print(f"==> warming up {model}")
    warm = dict(body)
    warm["max_tokens"] = 8
    try:
        stream_once(url, warm, headers, args.timeout)
    except (urllib.error.URLError, OSError) as exc:
        print(f"The server did not answer: {exc}", file=sys.stderr)
        print(f"Checked {url}", file=sys.stderr)
        return 1

    before = read_link_counters(args.iface)
    wall_start = time.perf_counter()

    runs = []
    failures = []
    lock = threading.Lock()
    pending = list(range(args.requests))
    concurrency = max(1, min(args.concurrency, args.requests))

    print(
        f"==> {args.requests} request(s), {args.max_tokens} tokens each, "
        f"{concurrency} in flight"
    )

    def worker():
        while True:
            with lock:
                if not pending:
                    return
                pending.pop()
            try:
                result = stream_once(url, body, headers, args.timeout)
            except (urllib.error.URLError, OSError) as exc:
                with lock:
                    failures.append(str(exc))
                continue
            if result:
                with lock:
                    runs.append(result)
                    print(f"    {len(runs)}/{args.requests}", end="\r", flush=True)

    threads = [threading.Thread(target=worker) for _ in range(concurrency)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    print("")
    for message in failures[:3]:
        print(f"    a request failed: {message}", file=sys.stderr)

    wall_seconds = time.perf_counter() - wall_start
    after = read_link_counters(args.iface)

    if not runs:
        print("No request completed. Nothing to record.", file=sys.stderr)
        return 1

    ttfts = [r["time_to_first_token_s"] for r in runs]
    decode_rates = [
        r["output_tokens"] / r["decode_seconds"] for r in runs if r["output_tokens"]
    ]
    # Prefill rate from time to first token, using the prompt length we asked for.
    prompt_tokens_estimate = len(prompt) // CHARS_PER_TOKEN
    prefill_rates = [prompt_tokens_estimate / t for t in ttfts if t > 0]

    exo_bench = None
    if args.exo_bench_url:
        try:
            exo_bench = bench_once(args.exo_bench_url, body, headers, args.timeout)
        except (urllib.error.URLError, ValueError, OSError) as exc:
            print(f"exo bench endpoint did not answer: {exc}", file=sys.stderr)

    record = {
        "lab": "part-21/two-mac-cluster-over-thunderbolt-5",
        "record": "served-measurement",
        "date": date.today().isoformat(),
        "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
        "label": args.label,
        "model": model,
        "base_url": args.base_url,
        "client_machine": platform.node().split(".")[0],
        "client_platform": platform.platform(),
        "requests_completed": len(runs),
        "requests_asked": args.requests,
        "requests_failed": len(failures),
        "concurrency": concurrency,
        "max_tokens": args.max_tokens,
        "prompt_tokens_estimate": prompt_tokens_estimate,
        "wall_seconds": round(wall_seconds, 3),
        "aggregate_output_tokens_per_second": round(
            sum(r["output_tokens"] for r in runs) / wall_seconds, 3
        ),
        "time_to_first_token_s": summarise(ttfts),
        "decode_tokens_per_second": summarise(decode_rates),
        "prefill_tokens_per_second_estimate": summarise(prefill_rates),
        "link_bytes": link_delta(before, after),
        "link_interface": args.iface or "all",
        "exo_bench": exo_bench,
    }

    if record["link_bytes"] is None:
        record["link_note"] = (
            "No BSD netstat interface counters were readable on this machine, so "
            "link use was not recorded. Run this script on one of the Macs."
        )

    text = json.dumps(record, sort_keys=True)
    if args.print_only:
        print(text)
        return 0
    with open(args.labbook, "a", encoding="utf-8") as fh:
        fh.write(text + "\n")
    print(text)
    print(f"Appended to {args.labbook}")
    print("Read the lines back with:")
    print(f"  grep 'served-measurement' {args.labbook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download measure-pair.py429 lines

RunnableTrack M · Apple silicon

twenty requests, with the link counters around them

python3 measure-pair.py --label two-macs-ring-tensor

Set LINK_IFACE in .env to the Thunderbolt interface from task 3 if you want only that interface in the record; leave it empty and every interface is recorded, which is more data and occasionally more revealing.

Repeat for each configuration you can reach: ring and JACCL, tensor and pipeline, and the single-Mac baseline. Change the --label each time. That is four to five runs of twenty requests, and it is the bulk of the attended time in this lab.

∴Research note

The prefill figure this script reports is prompt tokens divided by time to first token, which includes queueing, chat-template rendering and the network hop from your terminal. It is therefore a lower bound on the engine’s own prefill rate rather than a measurement of it. That is fine for comparing configurations on the same machine, which is what this lab does, and it is not fine for quoting against someone else’s number. exo’s benchmark endpoint reports the server’s own figures, which is why the next task uses it.

9. Run exo across the same pair

Now the other tool, on the same hardware, so the comparison is fair. Install exo on both Macs from its README, and start it on each in its own terminal with uv run exo. It will find the other machine by itself.

RunnableTrack M · Apple silicon

run-exo.sh

#!/usr/bin/env bash
# Purpose: drive an already-running exo cluster through its API: show which nodes it
#          found, list the placements it will accept for a model, create one, wait
#          until it is ready, and delete it afterwards. exo itself is started by hand
#          in its own terminal on each Mac, because it runs in the foreground and its
#          log is worth watching the first few times.
# Platform: mac (Track M). exo runs on Linux too, on CPU only as its README states,
#           which is why the other tracks use Part 19's llama.cpp RPC path instead.
# Minimum memory: 32 GB per Mac for the primary path; 24 GB for a single Mac with a
#                 model that fits on it.
# Assumes: exo running on this Mac and on the peer, curl and python3 on PATH, and an
#          .env copied from env-example.txt. Nothing here starts or stops exo itself.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${EXO_PORT:=52415}"
: "${EXO_DIR:=}"
: "${EXO_MODEL:=}"
: "${EXO_PLACEMENT_INDEX:=0}"

BASE="http://localhost:${EXO_PORT}"
MODE="${1:-status}"

usage() {
    cat <<'USAGE'
Usage: bash run-exo.sh [start-help|status|previews|place|delete <instance-id>]

  start-help  print the command that starts exo on each Mac, and stop
  status      the cluster exo currently sees, from GET /state
  previews    every placement exo will accept for EXO_MODEL, with its cost
  place       create the placement at EXO_PLACEMENT_INDEX and wait until ready
  delete      remove an instance and free the memory it holds
USAGE
}

require() {
    command -v "$1" >/dev/null 2>&1 || { echo "$1 is not on PATH." >&2; exit 1; }
}
require curl
require python3

exo_up() {
    curl -fsS --max-time 5 "${BASE}/node_id" >/dev/null 2>&1
}

if [ "$MODE" = "start-help" ] || [ "$MODE" = "-h" ] || [ "$MODE" = "--help" ]; then
    usage
    echo ""
    echo "Start exo on EACH Mac, in its own terminal, and leave it running:"
    if [ -n "$EXO_DIR" ]; then
        echo "  cd ${EXO_DIR} && uv run exo"
    else
        echo "  cd <your exo checkout> && uv run exo"
        echo "  (or launch the macOS application, which needs macOS 26.2 or later)"
    fi
    echo ""
    echo "Then come back here. The dashboard is at ${BASE}/"
    exit 0
fi

if ! exo_up; then
    echo "Nothing answers at ${BASE}. Start exo first:" >&2
    echo "  bash run-exo.sh start-help" >&2
    exit 1
fi

case "$MODE" in
    status)
        echo "==> node id"
        curl -fsS "${BASE}/node_id"
        echo ""
        echo "==> cluster state"
        curl -fsS "${BASE}/state" | python3 -m json.tool
        ;;

    previews)
        if [ -z "$EXO_MODEL" ]; then
            echo "EXO_MODEL is empty in ${ENV_FILE}." >&2
            exit 1
        fi
        echo "==> placements exo will accept for ${EXO_MODEL}"
        curl -fsS --get "${BASE}/instance/previews" \
            --data-urlencode "model_id=${EXO_MODEL}" \
            | python3 "${HERE}/exo-placement.py" --list
        echo ""
        echo "Set EXO_PLACEMENT_INDEX in ${ENV_FILE} to the number in brackets."
        ;;

    place)
        if [ -z "$EXO_MODEL" ]; then
            echo "EXO_MODEL is empty in ${ENV_FILE}." >&2
            exit 1
        fi
        echo "==> creating placement ${EXO_PLACEMENT_INDEX} for ${EXO_MODEL}"
        BODY="$(curl -fsS --get "${BASE}/instance/previews" \
            --data-urlencode "model_id=${EXO_MODEL}" \
            | python3 "${HERE}/exo-placement.py" --select "$EXO_PLACEMENT_INDEX")"
        curl -fsS -X POST "${BASE}/instance" \
            -H 'Content-Type: application/json' \
            -d "$BODY" | python3 -m json.tool
        echo ""
        echo "==> waiting until exo reports the instance ready"
        echo "    A first placement downloads weights, so this can take a while."
        curl -fsS -N --get "${BASE}/instance/await" \
            --data-urlencode "model_id=${EXO_MODEL}" \
            | sed -n '1,20p'
        echo ""
        echo "Ready. Measure it with:"
        echo "  python3 measure-pair.py --base-url ${BASE}/v1 \\"
        echo "      --model ${EXO_MODEL} --label exo --exo-bench-url ${BASE}"
        ;;

    delete)
        INSTANCE_ID="${2:-}"
        if [ -z "$INSTANCE_ID" ]; then
            echo "Usage: bash run-exo.sh delete <instance-id>" >&2
            echo "Find the id with: bash run-exo.sh status" >&2
            exit 1
        fi
        echo "==> deleting instance ${INSTANCE_ID}"
        curl -fsS -X DELETE "${BASE}/instance/${INSTANCE_ID}"
        echo ""
        echo "Memory freed on every node that held a shard."
        ;;

    *)
        usage >&2
        exit 1
        ;;
esac

Download run-exo.sh142 lines

RunnableTrack M · Apple silicon

exo-placement.py

"""Read exo's placement previews from standard input and make them legible.

Purpose: `GET /instance/previews` returns every way exo is willing to place a model
    across the cluster, and each entry carries four separate decisions: which nodes,
    pipeline or tensor sharding, which transport, and what it costs each node in
    memory. This turns that list into something you can read at a glance, and can
    also pick one entry out of it so a script does not have to parse JSON in shell.
Platform: all (spark, strix, mac, nvidia). Pure standard library; it only reads text.
Minimum memory: none of consequence; this is a filter, not a workload.
Assumes: the JSON body of GET /instance/previews on standard input. Placements whose
    `error` field is not null are shown but never selectable, because exo has already
    said why they will not work.

Usage:
    curl -fsS --get "$BASE/instance/previews" --data-urlencode "model_id=$MODEL" \
        | python3 exo-placement.py --list

    curl -fsS --get "$BASE/instance/previews" --data-urlencode "model_id=$MODEL" \
        | python3 exo-placement.py --select 0 > placement.json
"""

from __future__ import annotations

import argparse
import json
import sys


def parse_args():
    p = argparse.ArgumentParser(description="Read exo placement previews")
    mode = p.add_mutually_exclusive_group(required=True)
    mode.add_argument(
        "--list",
        dest="list_only",
        action="store_true",
        help="Print one line per placement, numbering the usable ones.",
    )
    mode.add_argument(
        "--select",
        type=int,
        metavar="N",
        help="Print the request body for usable placement N, counting from 0.",
    )
    return p.parse_args()


def usable_placements(payload) -> list:
    return [p for p in payload.get("previews", []) if p.get("error") is None]


def describe(index, preview) -> str:
    mark = f"[{index}]" if index is not None else "[--]"
    sharding = preview.get("sharding", "?")
    transport = preview.get("instance_meta", "?")
    lines = [f"{mark} sharding={sharding} transport={transport}"]
    for node, delta in (preview.get("memory_delta_by_node") or {}).items():
        lines.append(f"     {node}: {round(int(delta) / 1e9, 2)} GB")
    error = preview.get("error")
    if error is not None:
        lines.append(f"     not usable: {error}")
    return "\n".join(lines)


def main() -> int:
    args = parse_args()
    try:
        payload = json.load(sys.stdin)
    except ValueError as exc:
        print(f"Could not read JSON from standard input: {exc}", file=sys.stderr)
        return 1

    previews = payload.get("previews", [])
    if not previews:
        print("exo returned no placements at all for this model.", file=sys.stderr)
        print("Check the model id with: curl $BASE/models", file=sys.stderr)
        return 1

    if args.list_only:
        index = 0
        for preview in previews:
            if preview.get("error") is None:
                print(describe(index, preview))
                index += 1
            else:
                print(describe(None, preview))
        print("")
        print(f"{index} usable placement(s). Use --select with the number in [].")
        return 0

    usable = usable_placements(payload)
    if not usable:
        print("Every placement exo offered carries an error.", file=sys.stderr)
        return 1
    if args.select < 0 or args.select >= len(usable):
        print(
            f"Only {len(usable)} usable placement(s); {args.select} is out of range.",
            file=sys.stderr,
        )
        return 1
    print(json.dumps({"instance": usable[args.select]["instance"]}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download exo-placement.py105 lines

RunnableTrack M · Apple silicon

what does exo think it has?

bash run-exo.sh status

RunnableTrack M · Apple silicon

how will it split this model, and what will that cost?

bash run-exo.sh previews

The preview list is the interesting output of this whole task. Each entry names the split, the transport and the memory cost per node, and any entry exo will not accept says why. Choose one, set EXO_PLACEMENT_INDEX in .env, and create it.

RunnableTrack M · Apple silicon

place the model and wait for it

bash run-exo.sh place

Then measure it the same way, adding exo’s own benchmark endpoint so you get server-side prefill and decode figures alongside your client-side ones.

RunnableTrack M · Apple silicon

the same twenty requests, against exo

python3 measure-pair.py --base-url http://localhost:52415/v1 --exo-bench-url http://localhost:52415 --label exo

⛨Security

exo’s endpoint list has no authentication in it, and it can create and delete model instances. Keep it on a network you control. If anything outside the house needs to reach this cluster, put the authenticating reverse proxy from Part 9 in front of it rather than opening port 52415, and remember that an unauthenticated instance-creation endpoint is also a way for anyone who can reach it to fill both machines’ memory.

Validate the cable and collective independently of the model

Perform the operating-system and hardware prerequisites on both Macs. Save the original network configuration before running setup commands. Check that the host file names the intended addresses and Python environment on every node; a correct path on one Mac may be absent on the other.

Run the small group/collective check first. Confirm the expected rank count and result on both machines before allocating model shards. If it hangs, find the rank that failed to enter the collective and inspect its launcher or environment error. Then verify the active network path rather than inferring it from a connected Thunderbolt cable.

Measure the local baseline, distributed run and exo path with explicit model representations and cache states. Keep unsupported combinations as separate not-run rows. For failure recovery, stop one lab process during an expendable request, restore it and repeat the short probe. Preserve the host file, backend, model identity, placement and transport measurements. The single-Mac path can validate code and serving behaviour, while the physical link and independent-node failure remain untested. After cleanup, verify ordinary management access and any network settings you chose to restore.

Validation

Four checks, in order. Each one narrows where a problem can be.

The group forms and the maths is right. Every rank printed the same expected sum in task 5, and a group-check line exists in the notebook for each backend you tried.

The model loads split, not twice. In task 6, peak memory per rank is roughly half the model’s size rather than all of it. If each rank reports the full size, the group was size one and each machine loaded the whole model: check that mlx.launch really started two ranks.

RunnableTrack M · Apple silicon

what did the notebook actually record?

grep '"record": "sharded-generate"' labbook.md

The endpoint answers, and the numbers are stable. A served-measurement line exists per configuration, and the median and p90 time to first token in each are within a factor of two of each other. A p90 far above the median usually means something else was running on one of the Macs.

The cable carried the traffic. The link_bytes field for the Thunderbolt interface is non-zero and grows with the number of requests. If it is zero while the model clearly ran, the ranks are talking over the house network instead: the host file has the wrong addresses in it, and your cluster is working at Ethernet speed without telling you.

Expected outcome

A notebook containing, at minimum: one group-check line per backend, one sharded-generate line per split you could run, one served-measurement line per configuration including the single-Mac baseline, and one served-measurement line for exo. Enough, in other words, to fill this in:

Pending validationYour two Macs, once this lab is done

Configuration	Time to first token, median (s)	Decode (tokens/s)	Thunderbolt bytes out	Peak memory per Mac (GB)
One Mac, model that fits, no group	from your run	from your run	not applicable	from your run
Two Macs, ring, tensor parallel	from your run	from your run	from your run	from your run
Two Macs, ring, pipeline parallel	from your run	from your run	from your run	from your run
Two Macs, JACCL, tensor parallel	from your run	from your run	from your run	from your run
Two Macs, exo, its own choice of placement	from your run	from your run	from your run	from your run

your two Macs, with chip and memory named in the notebook record, macOS 26.2 or later, the same build on both · mlx-lm through mlx.launch, and exo the versions your machines report; the course pins mlx-lm 0.31.3 · the repository you served, from the course model reference, the MLX quantisation of that repository · 4,096 tokens of context · the date of your run

Empty by design: the course's reference lab has one Mac, so no row here has been measured. Fill it in from your own notebook lines. The comparisons that matter are ring against JACCL on the same split, which isolates the transport, and one Mac against two on a model that fits on one, which prices the split itself.

The honest possible outcomes include one that people do not expect: two Macs slower than one, on a model that fitted on one machine. That is the correct result when the split’s traffic costs more than the second machine’s arithmetic saves, it is exactly what Part 19’s challenge page is about, and it is worth recording rather than hiding. A cluster earns its place on capacity first.

Troubleshooting

mlx.launch reports that a file does not exist on the remote host. The script or the Python interpreter is at a different path on the second Mac. Compare the two paths character by character; --print-python on each machine gives you the interpreter, and pwd in the lab directory gives you the script.

The launcher hangs with no output. Almost always SSH. Run the same ssh command by hand from Mac A and see whether it asks anything: a host-key confirmation prompt is enough to stall a launcher that is not expecting one.

The ranks start but never form a group. The addresses in the host file are not reachable between the machines over the cable. Ping each address from the other machine. If the bridge came back up after a reboot, the per-cable addresses are gone and the configuration in task 3 has to be applied again; nothing that helper does survives a restart on its own.

JACCL fails while ring works. Check ibv_devices on both machines, check that both are on the same macOS build, and check that the rdma array in the host file names devices that actually exist. exo’s README warns specifically that RDMA ports “may be unable to discover each other on different versions of MacOS”.

The model refuses to load with a message about sharding. The model’s mlx-lm implementation does not support the split you asked for. Try the other split; if neither works, choose another model. This is a property of the implementation, not of your cluster. A related message says that pipeline loading is only supported for converted models: that one means the repository has no safetensors index for mlx-lm to read, so it cannot work out which files this rank needs. Use a repository from the MLX community conversions, or convert the model yourself as Part 8 describes.

Everything works but it is slower than one machine. Check link_bytes first. If the Thunderbolt interface moved almost nothing, the tensors went over Ethernet. If it moved a great deal, you are on the wrong split for the link: try pipelining, which crosses the cable once per block boundary instead of at every layer.

exo finds only one node. Both machines must be on the same discovery namespace and reachable over the house network. exo’s namespace defaults to its own version string, so two machines running different builds of exo will not see each other, which is the same trap as the macOS build mismatch one level up.

Cleanup

Stop the server with Ctrl-C in its terminal, and delete the exo instance so the memory comes back.

RunnableTrack M · Apple silicon

free the model exo is holding

bash run-exo.sh status

Take the instance id from that output and remove it.

Fragment — not complete on its own

bash run-exo.sh delete "${INSTANCE_ID}"

Stop exo on both Macs with Ctrl-C in their terminals. The Thunderbolt configuration is not persistent: the addresses and routes the helper applied are gone after a reboot, and bridge0 comes back on its own. If you want the machines back exactly as they were without rebooting, bring the bridge back up from the console on each machine.

⚠Warning

Leaving RDMA enabled is harmless and saves you the Recovery trip next time. If you do want to undo it, that is another Recovery-mode operation and not something to attempt over SSH. The model downloads are the large thing on disk: with tensor parallelism you have a full copy of the repository on each Mac, so decide deliberately whether to keep both.

The single-Mac path

MLX supports several ranks on one machine, and its documentation recommends it: “test locally first. You can use the pattern mlx.launch -n2 -- my_script.py to run a small scale test on a single node first.” Leave PEER_SSH empty in .env and every script in this lab uses that path.

RunnableTrack M · Apple silicon

two ranks, one Mac, ring backend

mlx.launch -n 2 -- python check-group.py --label single-machine

RunnableTrack M · Apple silicon

the same model split between two processes on one Mac

bash run-distributed.sh generate

Everything works: the group forms, the collectives run, the splits apply, the server serves and measure-pair.py records the same fields. Two things are different, and both belong in your notebook. The link never carries anything, because the ranks are talking through the machine rather than over a cable, so the collective timings are a floor that no cable will beat. And the memory does not add up: both ranks draw on the same unified memory, so the largest model you can run is still the largest model that fits on this Mac, minus the overhead of running two processes.

Set MODEL_REPO to something that fits comfortably, note in the notebook that these are single-machine rows, and read the two-Mac tables in this part as documentation rather than as something you reproduced.

One Mac and one other machine, over Ethernet

If your second machine is a Spark, a Ryzen AI Max+ 395 box or an NVIDIA desktop rather than a second Mac, MLX cannot help: the ring backend would run, but only the Mac would contribute Metal inference, and JACCL needs Thunderbolt at both ends.

The cross-platform path is Part 19’s, and it is a good one. llama.cpp’s RPC backend splits a model by layers across machines of different kinds over ordinary networking, and its README notes that its macOS RDMA provider covers “RDMA over Thunderbolt on Apple silicon Macs with Thunderbolt 5” with the same macOS 26.2 requirement, negotiated during the handshake so no command changes. Work through Lab: A Mixed-Platform Cluster in Part 19 with your Mac as one of the machines, record the same measurements, and bring them to the reality check that follows this page: a Mac paired with a GPU machine is exactly the comparison that page is built around.

What you learned

• A Thunderbolt cluster is not the Thunderbolt bridge. macOS gives you a bridged service; MLX takes it down and gives each cable its own subnet and route. You have now read the commands that do it, which means the topology is something you can debug rather than something that works or does not.
• RDMA is a one-time, in-person, per-machine step. Recovery mode, rdma_ctl enable, reboot, and ibv_devices to confirm. No software automates it, and both MLX and exo say so.
• Backend and split are independent choices. You ran the same script over two transports and two splits and have four notebook lines that price each combination on your own hardware.
• A served measurement needs a baseline. The single-Mac row is what makes every cluster row mean something, and without it a table of cluster numbers says nothing about whether the cluster helped.
• The link counters catch the silent failure. A cluster whose tensors quietly went over Ethernet looks exactly like a cluster that is disappointing for no reason, and the bytes on the Thunderbolt interface are the difference.

Record in your lab notebook, beyond the JSON lines the scripts wrote: the chip and memory of each Mac, the macOS build on each, the Thunderbolt generation of the cable, whether RDMA was enabled, the model repository and its quantisation, which splits that model supported, and one sentence saying whether the second machine earned its place for the workload you care about.

Check your understanding

Question 1. After a reboot, the cluster forms but runs at a fraction of the speed it did yesterday, and the Thunderbolt interface shows almost no bytes. What happened?

• RDMA was disabled by the reboot and has to be re-enabled from Recovery
• The per-cable addresses and routes did not survive the reboot, so the ranks fell back to the house network
• The model was re-downloaded in a different quantisation
• macOS updated and the two machines are now on different builds

Show the answer and why

Answer: The per-cable addresses and routes did not survive the reboot, so the ranks fell back to the house network

The configuration mlx.distributed_config applies is not persistent: bridge0 comes back and the per-cable addresses go away. If the host file names those addresses and they no longer exist, the ranks find each other some other way and you get Ethernet speed silently. Enabling RDMA, by contrast, does persist across reboots.

Question 2. You have two 32 GB Macs and a model whose MLX repository is 35 GB. Which statements are true?

• Neither Mac can load the model alone
• With pipelining, each Mac downloads only the weight files for its own layers
• With tensor parallelism, each Mac downloads the whole 35 GB repository
• Tensor parallelism also splits the key-value cache between the machines
• The cluster gives you 64 GB of usable memory for one model

Show the answer and why

Answer: Neither Mac can load the model alone, With pipelining, each Mac downloads only the weight files for its own layers, With tensor parallelism, each Mac downloads the whole 35 GB repository, Tensor parallelism also splits the key-value cache between the machines

The last is the one to be careful with. Two 32 GB machines do not behave as one 64 GB machine: each rank needs room for its own share of weights, its share of the cache, its activations and the operating system. The usable total is meaningfully less than the sum, which is why the lab suggests a model comfortably under it rather than exactly at it.

Question 3. Why does this lab measure the single-Mac baseline before anything else?

• To warm the model cache so the cluster runs are faster
• Because a cluster measurement with nothing to compare it against cannot answer whether the cluster helped
• Because mlx.launch requires a previous single-machine run
• To check that the Thunderbolt cable is connected

Show the answer and why

Answer: Because a cluster measurement with nothing to compare it against cannot answer whether the cluster helped

A table of cluster numbers on its own tells you what the cluster does, not whether it was worth building. The baseline is what turns it into a comparison. This is the same discipline as the load-test baseline in Part 9 and the one-machine row in Part 19's challenge.

Question 4. True or false: the prefill figure measure-pair.py reports can be compared directly against a prefill figure published by someone else.

• True
• False

Show the answer and why

Answer: False

False. It is prompt tokens divided by time to first token, which includes queueing, template rendering and a network hop, so it is a lower bound on the engine's prefill rate rather than a measurement of it. It is a fair way to compare configurations on your own machines, which is what this lab needs. exo's benchmark endpoint reports the server's own figures, which is the number to use when the comparison has to leave your house.

Check my answersClear