The Course Reference Cluster and the Single-Machine Path

By the end of this lesson you will know exactly which machines the rest of Level 4 was written against and how much of each page you can expect to reproduce on yours; you will be able to name the six roles a machine can hold in a local AI service and say which of them your machines are suited to; you will know which engines can mix platforms and which refuse; and you will have a worksheet to fill in, which the lab then turns into a working network.

The reference cluster

The course is written on a specific set of machines, described here by class and memory rather than by name, because a hostname is a fact about one house and is of no use to you.

The machines every Level 4 page was written against

• workerDGX Spark AGB10, 128 GB unified, DGX OS, aarch64
• workerDGX Spark BGB10, 128 GB unified, DGX OS, aarch64
• workerRyzen AI Max+ 395 mini PC128 GB unified, Radeon 8060S, Linux
• clientApple silicon MacBook ProM5, 24 GB unified, macOS
• workerNVIDIA laptopRTX 4090 laptop GPU, 16 GB VRAM, 128 GB system memory, Ubuntu
• routerHouse switch and routermanagement, downloads, client traffic

• DGX Spark A ⟷ connected to DGX Spark Bdirect ConnectX-7 cable, RoCE, no switch in the path
• DGX Spark A ⟷ connected to House switch and routerwired Ethernet
• DGX Spark B ⟷ connected to House switch and routerwired Ethernet
• Ryzen AI Max+ 395 mini PC ⟷ connected to House switch and routerwired Ethernet
• NVIDIA laptop ⟷ connected to House switch and routerwired Ethernet
• Apple silicon MacBook Pro ⟷ connected to House switch and routerWi-Fi, or a USB-C Ethernet adapter

One machine of each track, plus a second DGX Spark for the pair that Parts 19, 20 and 22 need. Every link speed and every measured number for this cluster is filled in by the validation pass; nothing on this page claims a measurement that has not been made.

Four things about this cluster are worth stating plainly, because they set the honesty of every Level 4 page.

It has one Mac. Track M’s two-machine pages, including the Thunderbolt 5 bridge in this part’s lab and the Mac pair in Part 21, are therefore written from Apple’s and MLX’s documentation and are marked as such on the page. They are not validated on hardware here, and the course says so rather than implying otherwise.

It has one machine per track except Track S. Mixed-platform clusters are built from a Spark plus the Ryzen box, or a Spark plus the laptop, and Part 19’s mixed-platform lab is written for exactly that shape.

The laptop is the small-memory member on purpose. With 16 GB of video memory it cannot hold what the 128 GB machines hold, and that is useful: it is the machine that shows what happens when one member of a cluster is the weak link, which Part 19’s challenge page turns into a diagnosis exercise.

The direct Spark link is separate from the house network. Model traffic crosses the QSFP cable; management, model downloads and client requests cross the switch. The previous lesson gave the reason. The lab measures both so the difference is a number rather than an assertion.

Pending validationReference cluster links, to be measured by the validation pass

Link	Class	Rated speed (gigabits/s)	Measured throughput (GB/s)	Measured round trip (ms)
Spark A to Spark B, direct	ConnectX-7 QSFP, RoCE	200	to be measured	to be measured
Spark to house switch	Ethernet	10	to be measured	to be measured
Ryzen AI Max+ 395 to switch	Ethernet	2.5	to be measured	to be measured
NVIDIA laptop to switch	Ethernet	to be recorded	to be measured	to be measured
MacBook Pro to switch	Wi-Fi, or a USB-C Ethernet adapter	to be recorded	to be measured	to be measured

The reference cluster described on this page · iperf3 and ping, once the validation pass runs them rated speeds from the vendor documentation cited on this page, read 2026-09-09 · not applicable; this table describes links, not applicable · 1 tokens of context · 2026-09-09

Rated speeds are vendor figures. The throughput and round-trip columns are deliberately empty until the validation pass runs this part's lab on the hardware; a rated speed is not a measurement and the course does not print one as if it were. Your own copy of this table, filled in with your own numbers, is the deliverable of the lab.

Six roles

A cluster is easier to reason about when you stop thinking about machines and start thinking about jobs. Six roles cover everything Level 4 and Level 5 build.

Prefill. Reads the prompt. Compute-bound, benefits from arithmetic throughput, and its work arrives in bursts. A machine with a fast GPU and modest memory is a good prefill node.

Decode. Writes the answer, one token at a time. Bandwidth-bound and steady. A machine with large, reasonably fast memory is a good decode node, which is why the unified-memory tracks suit it. Part 22 separates these two deliberately, which is the point of disaggregated serving.

Router. Accepts client requests, chooses a backend, holds the keys, records what was spent. This is the gateway from Part 9 and it needs almost no hardware.

Cache. Holds key-value state so that a repeated prefix is not recomputed, which Part 17 covered on one machine and Part 22 extends across machines. It needs memory and a fast link to whoever reads from it.

Storage. Holds the model library so that every machine reads the same bytes. It needs disk and a decent network and nothing else at all.

Agent host. Runs the loop that calls tools, which Level 5 builds. It is the machine that must be sandboxed, and it is deliberately not the machine holding your model library.

Role layout one: the capacity cluster (Parts 19 and 20)

• clientYour laptop or phone
• routerGatewayfrom Part 9; can share a machine
• workerWorker Afirst half of the layers
• workerWorker Bsecond half of the layers
• storageModel libraryshared, read-only to the workers

• Your laptop or phone ⟷ connected to Gatewayhouse network
• Gateway ⟷ connected to Worker Ahouse network; one request in, tokens out
• Worker A ⟷ connected to Worker Bfast direct link; one activation per token
• Model library ⟷ connected to Worker Ahouse network; read at load time only
• Model library ⟷ connected to Worker Bhouse network; read at load time only

One model, split across two workers. Only the worker-to-worker link needs to be fast, and only the storage link needs to be wide, and they are needed at different times: storage at load, the direct link at every token.

Role layout two: the replica pool (the concurrency answer)

• clientSeveral clients
• routerGatewayload-balances between identical backends
• workerReplica Acomplete copy of the model
• workerReplica Bcomplete copy of the model
• storageModel library

• Several clients ⟷ connected to Gatewayhouse network
• Gateway ⟷ connected to Replica Ahouse network
• Gateway ⟷ connected to Replica Bhouse network
• Model library ⟷ connected to Replica Aread at load time only
• Model library ⟷ connected to Replica Bread at load time only

Nothing is split, the replicas never speak to each other, and no link carries per-token traffic. If one replica fails the other still answers. This is the arrangement to reach for first, and the one Part 23 operates.

Role layout three: separated prefill and decode (Part 22)

• clientClients
• routerRouter
• prefillPrefill nodecompute-heavy; reads the prompt
• decodeDecode nodebandwidth-heavy; writes the answer
• cacheKV cacheholds state between the two
• storageModel library

• Clients ⟷ connected to Routerhouse network
• Router ⟷ connected to Prefill nodeprompt in
• Prefill node ⟷ connected to KV cachekey-value state, once per request; the big transfer
• KV cache ⟷ connected to Decode nodekey-value state, once per request
• Decode node ⟷ connected to Routertokens out
• Model library ⟷ connected to Prefill noderead at load time
• Model library ⟷ connected to Decode noderead at load time

The transfer here is once per request rather than once per token, but it is much larger: the whole prompt's key-value state. Part 22 works out how large, and what link that needs. Both machines still hold the whole model.

Mixing platforms, and where it stops working

The single most useful thing to know before planning a cluster from machines you already own: which engines will accept a mixed set.

Engine	Can mix platforms?	What it accepts	Where it is taught
llama.cpp RPC	Yes	CUDA, Metal, Vulkan and CPU hosts in one run; the server picks up whichever accelerators it was built for	Part 19
vLLM multi-node	No	Homogeneous NVIDIA nodes coordinated by Ray; macOS has no mainline GPU path	Part 20
MLX distributed	Macs only	Apple silicon over Ethernet, a Thunderbolt ring, or RDMA over Thunderbolt 5	Part 21
exo	Macs primarily	Its README states that on macOS it uses the GPU and on Linux it currently runs on CPU	Part 21

llama.cpp’s RPC backend is the only one of the four that will put a Mac, a Ryzen box and an NVIDIA laptop into one run, which is exactly why Part 19 comes first in this level. Its README describes the server as exposing “ggml devices remotely” and lists build options for CUDA and other backends per host, so each machine is built for its own accelerator and the client does not care.

The two limits to hold in mind are that the README is candid about the backend’s maturity, calling it “fragile and insecure” and warning “never run the RPC server on an open network or in a sensitive environment”, and that a mixed cluster runs at the pace its arrangement allows: a slow member holding a proportional share of the layers slows every token. Part 19’s mixed-platform lab measures this and its challenge page makes it the whole exercise.

⛨Security

Every clustering tool in this level assumes a trusted network. llama.cpp’s RPC README says so outright. Put cluster traffic on a wired network you control, never expose an RPC or worker port beyond it, and keep the authenticated gateway from Part 9 as the only thing clients talk to. Part 23 is where a service gets exposed properly, with authentication and TLS.

The single-machine path

Every page in this level that needs more than one machine also has a path through it on one. There are three ways to do that, and they teach different amounts.

Three ways to run a cluster on one computer

1. 1Separate processesTwo engine processes on two ports, talking over the loopback address. Teaches the configuration, the flags and the failure modes. Teaches nothing about the network, because there is not one.
2. 2Containers on a user-defined bridge networkTwo containers with their own addresses and names, on a network Docker creates. Adds name resolution and a real socket path between them, so the procedure matches the multi-machine one closely. This is the path this part's lab uses.
3. 3Virtual machinesClosest to the real thing: separate kernels, separate network stacks, separate MTU settings that can disagree. The most faithful and the most work, and the only single-machine path on which an MTU mismatch can actually be reproduced.

Choose by what you are trying to learn. For the mechanics of an engine, processes are enough. For the mechanics of a cluster, containers or virtual machines are worth the extra step.

Docker’s own documentation gives the reason the lab uses containers rather than processes: “user-defined bridges provide automatic DNS resolution between containers”, and on such a network “containers can resolve each other by name or alias”, whereas on the default bridge they “can only access each other by IP addresses”. Name resolution between members is half of what makes a real cluster work and half of what breaks it, so a single-machine path that includes it is worth much more than one that does not.

⌂Single-machine path

What the single-machine path cannot teach: the actual throughput and latency of a link, because a loopback or a bridge network is not a cable. What it teaches perfectly: the procedure, the configuration, the shape of the failures and the contents of your notebook. Do the whole lab this way if you have one machine, record that you did, and every later cluster page will still make sense.

The boring prerequisites that break clusters

Four things have nothing to do with parallelism and cause a disproportionate share of the confusion in this level. Fix them once, at the start, and they never come back.

The same username on every machine. Every distributed launcher in Parts 19 to 21 starts processes on other machines over SSH, and every one of them is easier when the account name does not change between them. Where it does change, you will be writing the user into a host file on every line, and forgetting one.

Matching engine builds. Two machines running different builds of the same engine is a bug that presents as a crash on model load, or worse as a subtly different result. Record the build on each machine and compare them before blaming anything else. This is not paranoia: llama.cpp publishes rolling build tags precisely because a build number is the only meaningful version for it, and the course’s version reference exists for the same reason.

Architecture, not just operating system. A DGX Spark is aarch64 and most desktops are x86_64. Part 5’s caveat follows you here: a container image or a wheel that runs on one is not necessarily the same software on the other, and on a mixed cluster the failure can be a slow machine rather than an error. Record the architecture in the topology, which is what the lab’s script does.

Clocks. Machines whose clocks disagree produce logs that cannot be lined up, which turns a ten-minute diagnosis into an afternoon. Every desktop operating system synchronises its clock by default; the point is to check rather than assume, once, and to notice if a machine has been offline long enough to drift.

None of these is interesting. All of them are cheaper to fix now than during a benchmark.

Shared model storage

Two machines that each downloaded a model separately are two machines that will eventually disagree about which model they are running. Shared storage is the fix, and it is worth setting up before the first cluster rather than after the first confusing benchmark.

Ubuntu’s server documentation covers the standard approach. The server package is nfs-kernel-server and the client package is nfs-common; exports are declared in /etc/exports, one line per export, giving the directory, then the clients allowed to mount it, then the options in brackets, exactly as the documentation’s example shows. The export is applied with exportfs -a, and mounted on the client with a plain mount naming the server and the exported path. The documentation explains the options it shows: sync has the server commit a change to stable storage before it replies, no_subtree_check turns off subdirectory verification, and no_root_squash “allows root-owned files to be modified by any client system’s root user”, which is the option you should not enable for a model library.

Three practical points for a model library specifically.

Export it read-only to the workers. They never write to it. A read-only export removes a whole class of accident, and one machine, the one you download on, keeps a read-write mount.

Expect it to be the load-time bottleneck and not care. Reading tens of gigabytes over 2.5 gigabit Ethernet takes minutes. That is a once-per-start cost, and the alternative is tens of gigabytes of duplicated disk on every machine.

Consider copying instead, if your network is slow and your disks are large. A synchronised copy on each machine, made from one source, gets you the version discipline without the load-time penalty. The property that matters is that there is one authoritative library, not that it is mounted rather than copied.

What to draw before you build

The lab in this part fills in a worksheet. Sketch it now, on paper, for the machines you actually own. Six columns:

1. Machine — a short name you will actually use, such as the track letter and a number.
2. Class and memory — track, accelerator, total memory, and video memory separately if the machine has a discrete card.
3. Operating system and architecture — including whether it is aarch64, because Part 5’s caveat follows you into every container tag in this level.
4. Links — every network interface that could carry cluster traffic, with its class.
5. Role or roles — from the six above. Most machines hold more than one.
6. Model library — whether this machine holds it, mounts it, or has its own copy.

Then draw the links between the machines and mark, on each one, whether it will carry per-token traffic, per-request traffic or load-time traffic only. That single annotation is what tells you which link has to be fast, and it is the thing most home clusters get wrong.

Define machine roles independently of hostnames

Assign roles such as gateway, model worker, retrieval service and evidence collector. A physical machine can hold several roles, and a role can move to another machine without changing the application contract. Record the role-to-host mapping, service addresses, ports and dependencies in a topology table.

The single-machine path should exercise the same logical contracts where possible: authenticated gateway access, explicit model aliases, task evaluation and service shutdown. It cannot measure physical interconnect behaviour or the failure of a separate host. Label those checks not run rather than treating loopback traffic as equivalent evidence.

Before cluster work, make local paths and shared paths explicit. A model available under one user’s home directory may not exist at the same path on another node. Verify readable files and consistent revisions on every participant. For a failure drill, identify which role is stopped and what the client should observe. This helps you distinguish the architecture you designed from the accidental arrangement produced by whichever terminal happened to start first.

Recap

The course’s reference cluster is two DGX Sparks joined by a direct ConnectX-7 cable, one Ryzen AI Max+ 395 mini PC with 128 GB, one Apple silicon MacBook Pro with 24 GB, and one NVIDIA laptop with 16 GB of video memory and 128 GB of system memory, all on a house switch. It has one Mac, so every two-Mac page in this level is written from documentation and labelled that way.

Six roles: prefill, decode, router, cache, storage and agent host. Most home machines hold several, and the useful design question is not “how many machines” but “which link carries per-token traffic”.

llama.cpp’s RPC backend is the only engine here that mixes platforms; vLLM expects homogeneous NVIDIA nodes; MLX and exo are Apple silicon. All of them assume a trusted network and say so.

The single-machine path runs through containers on a user-defined bridge network, which gives you name resolution and a real socket path, and the lab is written so that the same scripts and the same notebook entries work either way.

Next, the lab: names, addresses, keys, measured links, a verified maximum transmission unit, and shared storage. No model is loaded and nothing is split. It is the foundation every later page in this level assumes.

Check your understanding

Question 1. You own an Apple silicon Mac, an NVIDIA desktop and a Ryzen AI Max+ 395 mini PC, and you want all three to serve one model together. Which engine can do that?

• vLLM with Ray across the three
• MLX distributed
• llama.cpp with the RPC backend
• None; a mixed cluster is impossible

Show the answer and why

Answer: llama.cpp with the RPC backend

llama.cpp's RPC backend exposes ggml devices remotely and each host is built for its own accelerator, so a Metal host, a CUDA host and a Vulkan host can share one run. vLLM multi-node expects homogeneous NVIDIA nodes and has no mainline GPU path on macOS; MLX distributed is Apple silicon only. This is why Part 19 opens the practical half of the level.

Question 2. In the capacity-cluster layout, which link must be fast?

• The link between the gateway and the clients
• The link between the shared model library and the workers
• The link between the two workers
• All of them equally

Show the answer and why

Answer: The link between the two workers

The worker-to-worker link carries traffic on every token. The storage link carries a large volume once, at load time, so it needs width rather than low latency and a slow one merely delays the start. The client link carries a request and a stream of tokens, which is trivial. Marking each link with what it carries is the whole point of the worksheet.

Question 3. Why does this part's single-machine path use containers on a user-defined bridge network rather than two processes on the loopback address?

• Containers are faster
• A user-defined bridge gives the members names that resolve, which is the part of a real cluster that most often breaks
• Processes cannot open two ports on one machine
• Docker is required by the engines

Show the answer and why

Answer: A user-defined bridge gives the members names that resolve, which is the part of a real cluster that most often breaks

Docker's documentation states that user-defined bridges provide automatic DNS resolution between containers, so members address each other by name exactly as they would across machines. Two processes on the loopback address teach the flags but skip name resolution entirely, and name resolution is where a large share of cluster debugging time goes.

Question 4. Which statements about the course reference cluster are true? Select all that apply.

• It contains two DGX Sparks joined by a direct cable that does not pass through the house switch
• It contains two Macs, so the Thunderbolt pair pages are validated on hardware
• Its NVIDIA laptop has less video memory than the unified-memory machines have total memory, which makes it the useful weak member
• Every machine reads models from its own separate download

Show the answer and why

Answer: It contains two DGX Sparks joined by a direct cable that does not pass through the house switch, Its NVIDIA laptop has less video memory than the unified-memory machines have total memory, which makes it the useful weak member

The Spark pair is cabled directly so that model traffic never crosses the switch. The laptop is deliberately the small-memory member, which is what makes the slow-member diagnosis exercises real. There is only one Mac, so two-Mac pages are written from documentation and labelled as such, and the whole cluster reads from one shared model library rather than separate downloads.

Question 5. You export the model library over NFS to two worker machines. What is the sensible export mode for the workers?

• Read-write with no_root_squash, so the workers can update models
• Read-only, because workers never write to the library
• Read-write but only for the root user
• It does not matter; NFS is read-only by default

Show the answer and why

Answer: Read-only, because workers never write to the library

Workers only load weights. A read-only export removes a class of accident entirely, and the one machine you download on keeps a read-write mount. The Ubuntu documentation describes no_root_squash as allowing root-owned files to be modified by a client's root user, which is precisely what a model library does not need.

Check my answersClear