exo: Automatic Partitioning Across Your Macs

The previous lesson was the mechanism: host files, backends, ranks, a flag that chooses a split. exo is the same mechanism with the ceremony removed. You start it on each Mac, it finds the others, it looks at what they have and how they are connected, it proposes ways to place a model across them, and it serves whichever one you pick behind an OpenAI-compatible endpoint.

By the end of this lesson you should be able to say what exo automates and what it does not, read one of its placement previews and understand what it is telling you, decide between exo and mlx.launch for a given job, and state the project’s platform support and status with a date rather than from memory.

What exo is

exo is an Apache-2.0 licensed clustering runtime for local models, maintained by exo labs. Its own one-line summary is that it “connects all your devices into an AI cluster”, and the README’s claim about why you would want that is worth reading as two separate statements: that it enables “running models larger than would fit on a single device”, which is capacity, and that RDMA support “makes models run faster as you add more devices”, which is throughput. Part 18 spent a lesson separating those two reasons, and exo’s README is a good example of how easily they get bundled.

Underneath, it is not a new inference engine. The README states that exo “uses MLX as an inference backend and MLX distributed for distributed communication”. Everything in the previous lesson is still what is happening; exo decides the arguments for you.

⌚Version note

This lesson describes exo from its README, its API reference and its argument parser as read on 2026-09-09, at the revision the course pins (exo main · verified 2026-09-08). exo is a fast-moving project with tagged releases, and its command-line interface has changed shape more than once. Run --help on the revision you install and compare before trusting any option list, including this one.

Discovery: the part that is genuinely automatic

The README’s first feature is the one that changes the experience most: “Devices running exo automatically discover each other - no manual configuration.” There is no host file, no list of addresses, no rank numbering. You run the same command on every machine.

Fragment — not complete on its own

uv run exo

Two details from the argument parser matter for a household with more than one experiment in it. Discovery is namespaced: --namespace sets a “discovery namespace, nodes with different namespaces will not connect”, and the README describes the same idea in the macOS application as an EXO_LIBP2P_NAMESPACE setting whose use cases are “running multiple separate exo clusters on the same network” and “preventing accidental cluster joining”. The ports are fixed and settable: --api-port (default 52415), --zenoh-port (52414) and --discovery-port (52413).

A node can also be told what job it is doing. --no-worker runs “exo without the worker component”, which the README recommends for “coordinator-only nodes that handle networking and orchestration but don’t execute inference tasks”, and --no-downloads disables the download coordinator so a node never fetches weights. On a home cluster the interesting case is a small machine with a good network position acting as the front door for two large machines behind it.

Topology-aware placement

The second feature is the one that replaces your judgement. The README describes it as: “exo figures out the best way to split your model across all available devices based on a realtime view of your device topology. It takes into account device resources and network latency/bandwidth between each link.”

That sentence is doing what Part 18 asked you to do by hand. Which machines have enough memory, which links are fast enough for a split that talks at every layer, and which are only fast enough for a split that talks at layer boundaries: exo answers those from a live view rather than from your notebook.

From a model name to a served endpoint

1. 1DiscoveryEvery node running exo finds the others in its namespace. No host file, no ranks.
2. 2TopologyResources per device and latency and bandwidth per link, kept current rather than measured once.
3. 3PreviewsGET /instance/previews returns every valid placement for a model: which nodes, which sharding, which transport.
4. 4You choosePick a placement. This is the one judgement exo does not make for you, and the interesting one.
5. 5Create the instancePOST /instance with the placement, then wait on GET /instance/await until it reports ready.
6. 6ServeRequests arrive at any node on port 52415 in OpenAI, Claude, Responses or Ollama dialect.
7. 7Delete the instanceDELETE /instance/{id} frees the memory. Instances are explicit objects, not a side effect of asking a question.

The shape of an exo session. Compared with mlx.launch, the host file and the backend flag have been replaced by the topology view and the preview list, and a model became a long-lived instance rather than an argument to a process.

Reading a placement preview

The preview endpoint is the most instructive thing in exo’s API, because it makes the partitioner show its work. GET /instance/previews?model_id=... “will preview all valid placements for your model”, and the README’s sample response has the fields that matter:

Pseudocode — not a real command

{
  "previews": [
    {
      "model_id": "<the MLX repository>",
      "sharding": "Pipeline",              # or Tensor
      "instance_meta": "MlxRing",          # the transport, ring or jaccl
      "instance": { ... },                 # the whole placement, to be posted back
      "memory_delta_by_node": { ... },     # what this placement costs each node
      "error": null                        # or why this placement will not work
    }
  ]
}

Four things are being decided at once and shown separately: which nodes take part, whether the split is pipeline or tensor, which transport carries it, and what it will cost each node in memory. A preview with a non-null error is exo telling you why a placement you might have expected is not available, which is more useful than silence.

The related GET /instance/placement computes a placement without creating it and accepts model_id, sharding, instance_meta and min_nodes as query parameters, so you can ask for a specific shape rather than choosing from what is offered.

!Important

Creating an instance is a separate, asynchronous step from asking a question. POST /instance returns a command id, not a ready model, and the API reference is explicit: “before sending inference requests, wait until the API sees the new instance”, which is what GET /instance/await is for. It streams and ends with a message whose type is either ready or timeout. A script that posts an instance and immediately posts a chat completion is a script that will fail intermittently on a slow download and pass on a warm cache.

The transports underneath

instance_meta in a preview is the same choice the previous lesson made with --backend. exo’s benchmarking tool spells the two values plainly in its filter: ring or jaccl. So a placement is a pipeline or tensor split, carried by the MLX ring backend over whatever network exists, or by JACCL over Thunderbolt RDMA.

Everything the previous lesson said about JACCL therefore applies here unchanged, and exo’s README repeats the requirements in its own words. RDMA “is a new capability added to macOS 26.2. It works on any Mac with Thunderbolt 5”, enabled by booting into Recovery and running rdma_ctl enable. The caveats section adds four practical ones: every device in an RDMA cluster “must be connected to all other devices in the cluster”; “the cables must support TB5”; on a Mac Studio “you cannot use the Thunderbolt 5 port next to the Ethernet port”; and the operating system versions must “match exactly (even beta version numbers) on all devices”.

The README also documents a setup script that “will disable Thunderbolt Bridge and set dhcp on each RDMA port”, which is the same bridge-teardown step mlx.distributed_config performs. Two projects independently arriving at “turn off the bridge macOS gives you” is a good sign that it is not optional.

✱Hardware note

The port restriction is easy to miss and produces a cluster that half works. On a Mac Studio the Thunderbolt 5 port next to the Ethernet jack is excluded, so a four-machine mesh needs its cables planned against the case, not just counted. Label both ends of every cable while you are plugging them in; mlx.distributed_config --dot will draw you the graph afterwards, but only if you can tell which cable it is complaining about.

The API surface

exo speaks four dialects on one port, which is the feature that makes it drop into an existing setup. The README lists compatibility with “OpenAI Chat Completions API, Claude Messages API, OpenAI Responses API, and Ollama API”, and the endpoint summary in the API reference confirms /v1/chat/completions, /v1/messages, /v1/responses and a family of /ollama/api/... paths, alongside /models, /v1/models, /models/search and /models/add.

For this course the important extra endpoint is POST /bench/chat/completions, which is “same as /v1/chat/completions, but also returns performance and generation statistics” including prompt_tps, generation_tps, prompt_tokens, generation_tokens and peak_memory_usage. That is prefill speed, decode speed and peak memory reported by the server itself, per request, which is exactly what this part’s lab needs to record and what a plain OpenAI-compatible endpoint makes you compute from timings.

⛨Security

The dashboard and API listen on port 52415 and there is no authentication in the endpoint list. Part 9’s gateway lesson applies here in full: keep exo bound to a network you control, and if anything outside the house needs to reach it, put the authenticating, rate-limiting reverse proxy you built in Part 9 in front rather than exposing the port. An unauthenticated endpoint that can create and delete model instances is also an endpoint that can fill a machine’s memory on request.

Installing it, and which platform you are on

There are two supported routes on macOS. The macOS application is a download or brew install --cask exo, and the README states that it “requires macOS Tahoe 26.2 or later”. From source, the sequence is a clone, a dashboard build with npm, uv sync --extra mlx and uv run exo, with Xcode, Homebrew, uv, node and a Rust nightly toolchain as prerequisites. The README notes that the application “will ask for permission to modify system settings and install a new Network profile”, which is the Thunderbolt reconfiguration described above.

Linux is supported and limited. The README is unambiguous: “Currently, exo runs on CPU on Linux. GPU support for Linux platforms is under development.” A Linux node can therefore join a cluster, serve the API and do useful coordination work, but it will not contribute GPU inference.

✱Hardware note

Tracks S, X and N can run exo, and on Linux it will use the CPU. That is worth knowing and rarely worth doing: a DGX Spark or a Ryzen AI Max+ 395 box has a fast accelerator that exo cannot reach on Linux today, so the sensible cross-platform cluster on those tracks is llama.cpp RPC from Part 19, not exo. This lesson is readable on every track; its subject is Track M.

Measuring it

exo ships its own benchmark driver, which is unusual and useful. bench/exo_bench.py “measures model prefill and token generation speed across different placement configurations”, driving the /bench/chat/completions endpoint. Its parameters are the ones you would want: --model, --pp for prompt sizes, --tg for generation lengths, --max-nodes, --instance-meta to filter by ring, jaccl or both, --sharding to filter by pipeline, tensor or both, --repeat, --warmup and --json-out.

That option list is a good summary of the whole part. The two axes that decide whether a Mac cluster is worth having are the transport and the split, and exo’s own benchmark tool is built to sweep exactly those two.

Limits, and when to use mlx.launch directly

exo automates the placement, and automation is the right default when the placement is not the thing you are studying. Four situations argue the other way.

You are learning the mechanism. A host file you wrote yourself is a topology you understand. The previous lesson exists because a cluster that fails is much easier to debug when you know what ssh, ips and rdma meant in the file.

You need a model or an option exo does not surface. mlx_lm.server has flags for prompt caching, draft models, concurrency and chat templates that a placement API does not expose. If your requirement is a specific mlx-lm feature, drive mlx-lm.

You want the comparison to be clean. Measuring ring against JACCL, or pipeline against tensor, on the same model and the same pair of machines is easiest when nothing is choosing for you. exo’s own benchmark filters exist for this reason, and they are the exo-shaped version of the same discipline.

The project’s status matters to you. exo is young, moves quickly, and its RDMA support arrived alongside the operating system feature it depends on. Jeff Geerling’s write-up of a four-machine Mac Studio cluster, published 2025-12-18, is candid about that: he was “working with prerelease software while testing”, and reports that “it was obvious RDMA over Thunderbolt is new. When it works, it works great. When it doesn’t…” Treat that as the current state of a moving target rather than a verdict, and check the project’s issues before planning around it.

The same two Macs, two ways to drive them

• workerMac Aexo node, or mlx.launch rank 0
• workerMac Bexo node, or mlx.launch rank 1
• clientYour clientOpenAI-compatible request

• Mac A ⟷ connected to Mac BThunderbolt 5: ring over TCP, or JACCL over RDMA
• Your client ⟷ connected to Mac Aexo: port 52415 on any node. mlx-lm: the port you gave mlx_lm.server on rank 0
• Mac A ⟷ connected to Mac Bhouse Ethernet: exo discovery, or the ssh mlx.launch uses to start ranks

The hardware is identical either way. What changes is who decides the split, where the endpoint lives, and how much of the configuration you can see. Both paths are run against each other in this part's lab.

Inspect the placement that automation selected

Automatic discovery and partitioning remove manual steps, but they do not remove the need to inspect the result. Record discovered hosts, available memory, selected model, partition assignment and transport before measuring. A node that appears in discovery may not receive a useful share of the computation.

Compare the automatically selected placement with a single-machine baseline for a model that fits locally. Then, if capacity is the goal, test the larger model separately. Keep prompt lengths and answer limits explicit; otherwise a change in output behaviour can look like a placement improvement.

Observe what happens when a node becomes unavailable during an expendable request and after the service is idle. Discovery recovery and successful recovery of an in-flight distributed computation are different properties. Save the operator steps required to restore service. Automation is valuable when it makes a supported topology easier to operate, but the acceptance criteria remain model correctness, resource use, latency and recoverability on your actual machines.

Recap

exo is an Apache-2.0 clustering runtime that discovers the Macs on your network, builds a live view of their resources and links, proposes valid placements for a model, and serves whichever you choose behind OpenAI, Claude, Responses and Ollama-compatible endpoints on port 52415. It is not its own engine: it runs on MLX and MLX distributed, so the ring and JACCL backends from the previous lesson are what actually move the tensors.

Its placement previews are worth reading even if you never use exo, because they name the four decisions a partitioner makes: nodes, sharding, transport and memory cost per node. Instances are explicit and asynchronous: create, await, use, delete.

Its RDMA requirements are macOS 26.2 or later, Thunderbolt 5, a Recovery-mode enable step, a full mesh of cables, matching operating system builds, and on a Mac Studio one port that cannot be used. On Linux it runs on CPU only, which makes llama.cpp RPC the better cross-platform choice on the other three tracks. Reach for mlx.launch directly when you are learning, when you need an mlx-lm feature exo does not expose, or when a clean comparison is the point.

Check your understanding

Question 1. A placement preview comes back with "sharding": "Pipeline" and "instance_meta": "MlxRing". What has exo decided?

• Each node holds a block of layers, and the nodes talk over TCP using the MLX ring backend
• Every layer is split across nodes, and the nodes talk over Thunderbolt RDMA
• The model runs entirely on one node and the others idle
• The model will be downloaded in full to every node before starting

Show the answer and why

Answer: Each node holds a block of layers, and the nodes talk over TCP using the MLX ring backend

The two fields are independent: sharding says how the model is divided, instance_meta says which MLX backend carries the traffic. Pipeline plus ring is the combination that tolerates a slower link, because one activation crosses per block boundary per token rather than data crossing at every layer.

Question 2. Which of these does exo take care of that mlx.launch makes you do yourself?

• Finding the other machines on the network
• Choosing between a pipeline and a tensor split
• Enabling RDMA from macOS Recovery
• Assigning ranks to machines
• Serving an OpenAI-compatible endpoint

Show the answer and why

Answer: Finding the other machines on the network, Choosing between a pipeline and a tensor split, Assigning ranks to machines, Serving an OpenAI-compatible endpoint

Discovery, the placement decision, ranking and the API are exo's job. Enabling RDMA is not: it is a one-time, in-person, Recovery-mode step on each Mac that no software can do for you, which both exo and MLX state plainly.

Question 3. You have a DGX Spark and a Ryzen AI Max+ 395 box, both on Linux, and one Mac. Is exo the right way to cluster them?

• Yes: exo discovers all three and uses each machine's GPU
• Yes, but only if all three run macOS 26.2
• No: exo runs on CPU only on Linux today, so the two Linux machines would contribute no GPU inference
• No: exo refuses to start on non-Apple hardware

Show the answer and why

Answer: No: exo runs on CPU only on Linux today, so the two Linux machines would contribute no GPU inference

exo runs on Linux and will join the cluster, but its README states that it currently runs on CPU there with GPU support under development. Two accelerators sitting idle is a poor cluster. Part 19's llama.cpp RPC path reaches all three machines' accelerators and is the right tool for that mix.

Question 4. True or false: a script can POST /instance and then immediately POST /v1/chat/completions for that model.

• True
• False

Show the answer and why

Answer: False

False. Creating an instance is asynchronous and returns a command id. The API reference says to wait until the instance is visible, which GET /instance/await does by streaming until it emits a ready or timeout message. A script that skips the wait works on a warm cache and fails when the weights have to be downloaded.

Check my answersClear