mlx.distributed: Ring, MPI and RDMA over Thunderbolt 5

Part 8 ended its MLX lesson with a single line about mlx.launch and a promise that Level 4 would explain it. This is that explanation. By the end you should be able to say which of MLX’s four distributed backends applies to a given set of machines, write the host file that describes them, predict what a tensor-parallel and a pipeline-parallel split each do to your download, your memory and your cable, and state exactly which of those things the documentation asserts and which you will have to measure yourself.

The smallest distributed MLX program

MLX’s distributed layer is small enough to fit in your head. A program joins a group, performs collective operations on arrays, and that is nearly all of it.

RunnableTrack M · Apple silicon

the whole idea, in five lines

import mlx.core as mx

world = mx.distributed.init()
x = mx.distributed.all_sum(mx.ones(10))
print(world.rank(), x)

The documentation’s own framing of that program is the design decision worth noticing: run it with plain python and only one process starts, “and no distributed communication takes place. Namely, all operations in mx.distributed are noops when the distributed group has a size of one.” A program written for four machines therefore runs unchanged on one, which is why the single-machine path through this part’s lab is a real path rather than a consolation prize.

Alongside all_sum the module documents all_gather, all_max, all_min, sum_scatter, and send, recv and recv_like for point-to-point exchange. A Group object answers rank() and size(). Everything a distributed language model does in MLX is built from those pieces.

Four backends, four transports

mx.distributed.init() takes a backend argument from {'any', 'ring', 'jaccl', 'mpi', 'nccl'}. The choice is about the wire, not about the mathematics: the collectives mean the same thing whichever backend carries them.

Backend	What it uses	What it needs	Where it applies
ring	TCP sockets in a ring	nothing beyond a network	every track; the default
jaccl	RDMA over Thunderbolt	macOS 26.2 or later, Thunderbolt 5, RDMA enabled from Recovery	Track M only
mpi	an MPI library, through mpirun	OpenMPI or similar on every node at the same path	any track with MPI installed
nccl	NVIDIA’s collective library	CUDA GPUs	Tracks S and N, not this part

MLX’s own one-line descriptions are worth quoting because they are unusually direct. The ring backend is “always available and usually faster than MPI”. MPI is “a full featured and mature distributed communications library”. JACCL is “low latency communication with RDMA over thunderbolt”, and the documentation adds that it is “necessary for things like tensor parallelism”. NCCL is “the backend of choice for CUDA environments”.

Where the backend sits on Track M

one program, four possible wires

1. Your program, or mlx-lmcalls all_sum, all_gather, send, recv on mx.arraysunchanged per backend
2. mx.distributedthe Group abstraction; a noop when the group has one member
3. Backend: ring, jaccl, mpi or ncclchosen by init(backend=...) or by mlx.launch --backend
4. TransportTCP sockets, ibverbs over Thunderbolt, an MPI byte-transfer layer, or NCCL
5. LinkThunderbolt 5 cable, Ethernet, or loopback on one machine
6. Metal and unified memorythe arrays the collectives move never leave unified memory on their own machinePart 5

Only the two middle layers change when you change backend. That is why the lab can run the same script over a Thunderbolt ring, over RDMA and on one machine, and compare the three.

The ring backend

The ring backend “does not depend on any third party library so it is always available”, uses TCP sockets, and connects the nodes in a ring: rank 1 talks to rank 0 and rank 2, rank 2 to rank 1 and rank 3, and so on. That topology has a consequence the documentation states plainly: send() and recv() “with arbitrary sender and receiver are not supported in the ring backend”. Collectives work; arbitrary point-to-point does not.

A ring can run over ordinary Ethernet, and MLX says it can beat MPI there. But the documentation is explicit that Ethernet is not the point: “although the ring backend can have benefits over MPI even for Ethernet, its main purpose is to use Thunderbolt rings for higher bandwidth communication”. Two Macs joined by one Thunderbolt cable are the smallest possible ring.

JACCL, and what macOS 26.2 changed

The newest backend is the one this part exists for. MLX’s documentation states that “starting from macOS 26.2, RDMA over thunderbolt is available and enables low-latency communication between Macs with thunderbolt 5”, and that the JACCL backend uses it “to achieve communication latency an order of magnitude lower than the ring backend”.

Three requirements come with it, and all three are documented rather than optional.

A recent macOS, and a trip to Recovery. The documentation is candid: “until the feature matures, enabling RDMA over thunderbolt is slightly more involved and cannot be done remotely even with sudo. In fact, it has to be done in macOS recovery.” The steps are to start in Recovery, open Terminal from the Utilities menu, run rdma_ctl enable, and reboot. Afterwards ibv_devices lists devices named after the Thunderbolt interfaces they belong to, which is how you confirm it worked.

A fully connected mesh. “The JACCL backend supports only fully connected topologies. Namely, there needs to be a thunderbolt cable connecting all pairs of Macs directly.” Two Macs need one cable. Three need three. Four need six, which is why published four-Mac clusters look like a cable sculpture, and why exo’s README notes that no Thunderbolt 5 switch exists to tidy them up.

Thunderbolt 5 specifically. Apple’s Mac Studio specification page lists four Thunderbolt 5 ports and gives their rated speed, which Part 18’s link table records alongside the other link classes. A Mac whose ports are Thunderbolt 4 still clusters over the ring backend; it does not get JACCL.

Ring versus mesh, for four machines

• workerMac 1ring: rank 0
• workerMac 2ring: rank 1
• workerMac 3ring: rank 2
• workerMac 4ring: rank 3

• Mac 1 ⟷ connected to Mac 2ring and mesh both need this cable
• Mac 2 ⟷ connected to Mac 3ring and mesh both need this cable
• Mac 3 ⟷ connected to Mac 4ring and mesh both need this cable
• Mac 4 ⟷ connected to Mac 1ring and mesh both need this cable
• Mac 1 ⟷ connected to Mac 3mesh only: the first diagonal
• Mac 2 ⟷ connected to Mac 4mesh only: the second diagonal

A ring of four needs four cables; the fully connected mesh JACCL requires needs six. The two diagonals are the difference between a ring cluster and an RDMA cluster, and they are also two more ports consumed on every machine.

⌚Version note

Every RDMA capability on Track M depends on macOS 26.2 or later. Three independent projects say so as read on 2026-09-09: MLX’s distributed documentation, llama.cpp’s RPC README (“requires macOS 26.2 or later, with RDMA enabled once from macOS Recovery via rdma_ctl enable”), and exo’s README (“RDMA is a new capability added to macOS 26.2”). exo’s README adds a caveat that has bitten people: “RDMA ports may be unable to discover each other on different versions of MacOS. Please ensure that OS versions match exactly (even beta version numbers) on all devices.”

MPI, and NCCL

MPI is the oldest path and still the most portable. With --backend mpi, mlx.launch “is a thin wrapper over mpirun”, with three consequences the documentation lists: IPs in the host file are ignored, every node must be able to SSH to every other node rather than just to rank 0, and mpirun must exist at the same path everywhere. Arguments reach mpirun through --mpi-arg, which is how you pin MPI to one interface. The documentation’s own advice about tuning it is to stop tuning it: “for faster all reduce consider using the ring backend either with Thunderbolt connections or over Ethernet”.

NCCL is the default backend in CUDA environments and belongs to Part 20’s world rather than this one. It appears here only so that the --backend values are not mysterious: MLX now runs on CUDA machines too, and mlx.launch --backend nccl is how a Mac drives a job on Linux GPU nodes.

Launching: mlx.launch and the host file

mlx.launch connects to each host over SSH, starts one process per rank, forwards their output, and terminates the rest if one dies. It also broadcasts standard input to every process, which means an interactive debugger works across a cluster.

The two minimal forms are the ones to remember:

RunnableTrack M · Apple silicon

two ranks on this machine, for learning the mechanics

mlx.launch -n 2 my_script.py

Fragment — not complete on its own

mlx.launch --hosts host-a.home.arpa,host-b.home.arpa my_script.py

Reading launch.py’s argument parser at MLX’s current source gives the full list, and it is worth having in front of you because the documentation page does not tabulate it: --print-python, --verbose, --hosts, --repeat-hosts (short form -n), --hostfile, --backend, --env, --mpi-arg, --connections-per-ip, --starting-port (short form -p, default 32323), --cwd, --nccl-port and --python. Anything after a bare -- is the command to run.

Host files

Command-line hosts are the quick path. A JSON host file is the complete one, and both the ring and the JACCL backend need one for anything non-trivial. The schema is “a list of objects that define each host via a hostname to ssh to and a list of IPs to utilize for the communication”:

Pseudocode — not a real command

[
  {"ssh": "<name you can ssh to>", "ips": ["<address on cable 1>", "<address on cable 2>"]},
  {"ssh": "<name you can ssh to>", "ips": ["<address on cable 1>", "<address on cable 2>"]}
]

The ssh field and the ips field do different jobs, and confusing them is the commonest setup error in this part. The ssh name is how the launcher reaches the machine to start a process, normally over your house Ethernet or Wi-Fi. The addresses are what the ranks bind to and connect to for model traffic, normally on the Thunderbolt cables. The documentation makes the same point from the other direction when it says that with --hosts, the ring backend “only accepts IPs and not hostnames”, so “if we need to ssh to a hostname that does not correspond to the IP we want to bind to we have to provide a hostfile”.

JACCL’s host file carries a third field. Because RDMA connections are opened per device rather than per address, the file needs “a list of rdma devices that connect each node to each other node”, as an rdma array whose entry for a node’s connection to itself is null. Rank 0 also needs one reachable address, because the ranks exchange RDMA connection metadata over a TCP side channel before switching to RDMA.

What has to be identical on every machine

The documentation gives a three-item checklist, and every item has cost somebody an evening: ssh hostname works with no password and no host-key confirmation; the Python binary is at the same path on every host, which mlx.launch --print-python will tell you; and the script is at the same path on every host. Shared model storage from Part 18’s lab solves the third item for weights; it does not solve it for the script, which you still have to copy.

Configuring the cables: mlx.distributed_config

Setting up Thunderbolt links by hand is, in the documentation’s own words, “a relatively tedious process”. MLX ships mlx.distributed_config to do it, and reading what it does is the best available description of what a Thunderbolt cluster actually is underneath.

The documented sequence is: SSH to all nodes to check they are reachable; run commands on each node to work out which node is connected to which; verify that the result is a valid fully connected mesh, or for the ring backend a valid ring; check that RDMA is enabled; extract each node’s Ethernet address from en0; disable the Thunderbolt bridge and set up peer-to-peer networks for each cable; and write the host file.

Two of those steps deserve a second look.

Disabling the Thunderbolt bridge is required, not optional. macOS presents Thunderbolt cables as a single bridged network service. MLX takes them apart again, because a bridge is the wrong shape for a set of point-to-point links each of which should be its own tiny subnet. The documentation says this even for JACCL, where TCP is not used for the model traffic: “even though TCP/IP is not used when communicating with Thunderbolt RDMA, disabling the thunderbolt bridge is still required as well as setting up isolated local networks for each thunderbolt connection”. exo says the same thing about its own setup script, which “will disable Thunderbolt Bridge and set dhcp on each RDMA port”.

The helper prefers to show you the commands. --auto-setup runs the configuration over SSH and “requires password-less sudo on each node. If it isn’t available then the configuration script will print commands to be run on each node.” Printing them is the better default the first time: the commands bring the bridge interface down, give each cable’s interface an address on its own tiny subnet, and add a route to the peer, and seeing that written out is what makes the topology concrete.

There is also a debugging mode that has saved more time than it looks like it should. With --dot, the helper “will export a GraphViz representation of the connections between the nodes which makes it very easy to figure out which cable is not connected correctly”.

✕Common mistake

The documentation page for mlx.distributed_config shows --output in its worked example, but the argument parser in MLX’s source at the version the course reads defines the option as --output-hostfile. The course uses the source. If a command in someone’s blog post fails with an unrecognised argument, this is why: check the parser, not the prose.

Tensor parallel or pipeline parallel, in mlx-lm

Part 18 defined both splits in the abstract. mlx-lm implements both, and the choice is one flag.

mlx-lm’s distributed inference example is the clearest statement of the API. It calls sharded_load(model, pipeline_group, tensor_group) and passes the process group as exactly one of the two, with --pipeline selecting pipelining “instead of tensor parallelism”. The same flag exists on mlx_lm.server and on mlx_lm chat, so a served endpoint and an interactive session choose the same way. The example’s own launch line pairs it with the RDMA backend:

Fragment — not complete on its own

mlx.launch --backend jaccl --env MLX_METAL_FAST_SYNCH=1 --hostfile hosts.json sharded_generate.py

What each split costs

The two splits differ in three ways that all matter on a home cluster.

Traffic per token. Tensor parallelism splits every matrix in every layer, so each machine computes a slice and the ranks combine results at every layer. Pipeline parallelism gives each machine a contiguous block of layers, so exactly one activation crosses the cable per block boundary per token. That is the Part 18 arithmetic, and it is why MLX’s documentation calls JACCL “necessary for things like tensor parallelism” while a ring over Ethernet is enough for pipelining.

Download and disk. This one surprises people. sharded_load reads the model’s model.safetensors.index.json, works out which weight files its own rank needs, and downloads only those when pipelining. In tensor-parallel mode it downloads the whole repository on every node. Two 32 GB Macs pipelining a 35 GB model each fetch about half of it; the same pair sharding it tensor-parallel each fetch all of it and then keep half in memory.

Memory per machine. Both splits divide the weights. Tensor parallelism also divides the attention heads, and with them the key-value cache, which is why the number of key-value heads must divide by the number of ranks. Qwen3-32B, which the course’s model reference records as Apache-2.0 licensed with 8 key-value heads, splits cleanly across two or four machines; a model with 2 key-value heads would not split across four.

Not every model supports both

This is the fact most likely to stop a first attempt, and it is not on any documentation page: it is in the model implementations. In mlx-lm mlx-lm 0.31.3 · verified 2026-09-08, tensor parallelism exists for a model whose implementation defines a shard method, and pipelining for one whose inner model mixes in the pipeline helper. The two sets are not the same and neither is universal.

Reading the source at that version, shard is defined for the Qwen3 and Qwen2 dense families, gpt-oss, Llama, DeepSeek v2, v3 and v3.2, the GLM 4 mixture-of-experts models, MiniMax, Kimi and several others. Pipelining is defined for far fewer: the GLM 4 mixture-of-experts models, DeepSeek v2 and v3, and Ministral 3. Qwen3’s own mixture-of-experts implementation, the one behind Qwen3-30B-A3B, defines neither, and sharded_load raises “the model does not support any sharding” rather than silently running on one machine.

!Important

Check the model before you buy the second Mac. The question “can these two machines run this model” has three parts: does the memory add up, does the model’s mlx-lm implementation support the split you intend, and does the cable carry the traffic that split generates. Only the first is obvious from the model card.

What is documented, and what you will measure

This lesson has quoted a lot and measured nothing, deliberately. Here is the honest division.

Documented, by Apple’s MLX project and by two independent projects that implement the same feature: that RDMA over Thunderbolt exists from macOS 26.2, that it needs Thunderbolt 5 and a one-time Recovery step, that JACCL needs a fully connected mesh, and that its latency is lower than the ring backend’s by roughly a factor of ten.

Claimed by a project about its own software, and therefore to be treated as a claim: exo’s README states that its RDMA support enables a “99% reduction in latency between devices” and that its tensor parallelism gives “up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices”. The next lesson looks at exo properly, and this part’s lab measures a pair of machines rather than repeating either figure.

Not established anywhere the course could check: what any of this does to tokens per second on your models, at your context lengths, on your cables. That is what the lab is for.

Pending validationThe comparison the lab fills in, per model and context length

Configuration	Prefill (prompt tokens/s)	Decode (tokens/s)	Peak memory per Mac (GB)
One Mac, model that fits	from your run	from your run	from your run
Two Macs, ring over Thunderbolt, tensor parallel	from your run	from your run	from your run
Two Macs, JACCL over Thunderbolt 5, tensor parallel	from your run	from your run	from your run
Two Macs, pipeline parallel, model too large for one	from your run	from your run	from your run

your two Macs, named with chip and memory in the notebook record, macOS 26.2 or later, the same build on both machines · mlx-lm through mlx.launch the version mlx_lm.generate --help reports on your machines · the model you chose, from the course model reference, the MLX quantisation of the repository you loaded · 8,192 tokens of context · the date of your run

An empty shape, not a prediction. The rows are chosen so that the two comparisons that matter fall out of the table: ring against JACCL on the same split isolates the transport, and one Mac against two on a model that fits isolates what the split costs when capacity was not the problem.

Validate a collective before loading model shards

A distributed array program needs consistent process membership and communication semantics. Have every rank report its rank and group size, then run a tiny collective with a result you can compute by hand. This tests the launcher and backend independently of model memory and tokenisation.

If a collective hangs, compare logs from every rank and verify that each reaches the same operation in the same order. A single process failing before the collective can leave the others waiting. Check host configuration, environment activation and interface selection before changing model parameters.

After the collective works, load a small supported sharded model and record placement. Keep the backend explicit in the measurement; a ring or MPI path and an RDMA path have different requirements. A single-Mac fallback can teach the program structure and tensor shapes, but it cannot validate a cable or cross-host collective. Preserve that distinction in your labbook so a later reader knows which layer of the distributed stack has actually been exercised.

Recap

MLX’s distributed layer is four collective operations and a group object, carried by one of four backends. The ring backend is always available and runs over TCP, including over Thunderbolt. JACCL runs over RDMA, needs macOS 26.2 or later, Thunderbolt 5, a one-time Recovery step and a fully connected mesh, and is what makes tensor parallelism across machines reasonable. MPI is the portable fallback that MLX itself suggests you skip. NCCL belongs to CUDA machines.

mlx.launch starts one process per rank over SSH and needs the same Python and the same script at the same path on every machine. Host files separate the name you SSH to from the addresses model traffic uses, and JACCL host files add the RDMA device names. mlx.distributed_config discovers Thunderbolt cables, disables the bridge, builds per-cable subnets and writes the host file, and will print its commands rather than run them.

In mlx-lm, one flag chooses between tensor parallelism and pipelining. Tensor parallelism moves data at every layer, splits the key-value cache, and downloads the whole model on every node. Pipelining moves one activation per boundary, downloads only each node’s own shard, and is supported by fewer model implementations. Check which your model supports before you plan around either.

Check your understanding

Question 1. Two Macs are joined by one Thunderbolt 5 cable, both on macOS 26.2. You want to add a third Mac and keep using the JACCL backend. What must be true?

• One more cable, joining the third Mac to either of the first two
• Two more cables, so that all three Macs are directly connected to each other
• Nothing extra; JACCL discovers the third machine over Ethernet
• A Thunderbolt 5 switch between the three machines

Show the answer and why

Answer: Two more cables, so that all three Macs are directly connected to each other

JACCL supports only fully connected topologies: every pair of Macs needs its own direct cable. Three machines need three cables, so adding the third machine means adding two. No Thunderbolt 5 switch exists to shortcut this, which is the practical ceiling on RDMA cluster size.

Question 2. Your two 32 GB Macs will run a model whose MLX repository is about 35 GB. You choose pipeline parallelism. What happens to the download?

• Both machines download the whole repository, about 35 GB each
• Each machine downloads roughly the half it will hold
• Rank 0 downloads everything and sends the other half over the cable
• Neither downloads anything until the first request arrives

Show the answer and why

Answer: Each machine downloads roughly the half it will hold

When a pipeline group is given, mlx-lm reads the safetensors index, works out which weight files its own rank needs, and downloads only those. Tensor parallelism is the other case: there, every node downloads the whole repository and then keeps its slice in memory.

Question 3. Which of these are stated requirements for the JACCL backend rather than choices?

• macOS 26.2 or later
• Thunderbolt 5 rather than Thunderbolt 4
• RDMA enabled once from macOS Recovery
• A host file rather than a bare --hosts list
• Password-less sudo on every node

Show the answer and why

Answer: macOS 26.2 or later, Thunderbolt 5 rather than Thunderbolt 4, RDMA enabled once from macOS Recovery, A host file rather than a bare --hosts list

The first three are stated by MLX, by llama.cpp and by exo alike. The host file is required for JACCL because it must carry the RDMA device names, which a comma-separated host list has nowhere to put. Password-less sudo is only needed if you want mlx.distributed_config to apply its configuration for you; without it the helper prints the commands and you run them.

Question 4. A distributed script fails immediately with an error saying the model does not support any sharding. What is the most likely cause?

• The Thunderbolt cable is not connected
• The two machines are on different macOS builds
• The model implementation in mlx-lm defines neither a shard method nor pipelining
• The host file lists the wrong addresses

Show the answer and why

Answer: The model implementation in mlx-lm defines neither a shard method nor pipelining

That message comes from sharded_load, which raises it after inspecting the loaded model class. Support for tensor parallelism and support for pipelining are per-implementation in mlx-lm, and some popular models have neither. A cabling or addressing fault fails earlier and differently, at group formation.

Question 5. True or false: because mx.distributed operations are noops in a group of size one, a script written for a two-Mac cluster runs unchanged on a single Mac.

• True
• False

Show the answer and why

Answer: True

True, and it is a deliberate design choice the documentation calls out. It is also the basis of this part’s single-machine path: mlx.launch -n 2 starts two ranks on one machine, so you can exercise the whole mechanism, including group formation and collectives, without a second Mac. What you cannot get that way is more memory.

Check my answersClear