Networking for Home Clusters: 2.5 GbE to 200 GbE, Thunderbolt 5 and RDMA

By the end of this lesson you will be able to convert a link’s advertised speed into the number you actually care about, work out from the previous lesson’s formulas whether a given link can carry a given split, say what remote direct memory access removes from the path and when it is available to you, set a sensible maximum transmission unit and know why, and choose a link class for your track without buying the wrong thing.

This is the lesson to read before spending money. It names link classes rather than products, because product names date faster than the course does and the class is what determines whether your cluster works.

Gigabits, gigabytes and the factor of eight

Network links are sold in gigabits per second. Memory bandwidth, model sizes and everything else in this course are in gigabytes. The conversion is a division by eight, and forgetting it is the single commonest error in home-cluster planning: a link that sounds like it is in the same league as a memory bus turns out to be an order of magnitude or two below it.

Here is the ladder, with the arithmetic done.

Vendor specification, not measuredLink classes available to a home cluster, rated speeds

Link class	Rated speed (gigabits/s)	Byte rate from arithmetic (GB/s)	Where it appears in this course
2.5 gigabit Ethernet	2.5	0.31	Onboard on the Ryzen AI Max+ 395 mini PCs and the Mac mini
10 gigabit Ethernet	10	1.25	The DGX Spark RJ-45 port; Mac Studio; a Mac mini option
25 gigabit Ethernet	25	3.13	An add-in card in a desktop with a spare slot
Thunderbolt 4	40	5	Mac mini with an M6 chip; USB4 ports on other machines
100 gigabit Ethernet	100	12.5	An add-in card in a workstation or server
Thunderbolt 5	120	15	Mac Studio and the M5 Pro Mac mini, four and three ports
200 gigabit Ethernet	200	25	Each QSFP port on a DGX Spark

Link classes as named by their standards and by the vendor pages cited on this page · none - arithmetic only, no link was measured rated speeds read from vendor documentation on 2026-09-09; byte rate is the rated speed divided by eight · not applicable; this table describes links, not models, not applicable · 1 tokens of context · 2026-09-09

Vendor and standard figures, not measurements. Apple's specification pages list Thunderbolt 5 at up to 120 gigabits per second and Thunderbolt 4 at up to 40; NVIDIA's ConnectX-7 networking page states that each DGX Spark QSFP port provides up to 200 gigabits per second. Rated speed is a ceiling on the wire and never the throughput an application sees: framing, protocol headers, the host's own copying and the other end's ability to keep up all take a share. The lab in this part measures what your links actually deliver, which is the number that belongs in your notebook.

!Important

Every figure in that table is a vendor or standard rating. None of them is a measurement, and the gap between the two is routinely twenty per cent or more before you have tuned anything. Never plan a cluster against a rated speed you have not confirmed with iperf3 on your own cable.

Turning the parallelism arithmetic into a link

The previous lesson left two formulas. Feed them the link classes above and the answer stops being a matter of opinion.

For a 64-layer model with a hidden size of 4,096 at two bytes per element, the activation vector is eight kilobytes per token. A tensor-parallel split across two machines moves roughly two of those per layer, about one megabyte per token. A pipeline split across two machines moves one of them, eight kilobytes per token.

Pending validationTime to move one token's worth of traffic, by split and link class

Link class	Tensor parallel, ~1 MB per token (microseconds)	Pipeline parallel, ~8 KB per token (microseconds)
2.5 gigabit Ethernet	3,355	26
10 gigabit Ethernet	839	6.6
25 gigabit Ethernet	336	2.6
Thunderbolt 4	210	1.6
100 gigabit Ethernet	84	0.66
Thunderbolt 5	70	0.55
200 gigabit Ethernet	42	0.33

The link classes in the table above, at their rated speeds · none - arithmetic only, no engine and no link was run computed on 2026-09-09 from the rated speeds and the activation formulas in the previous lesson · a hypothetical 64-layer model with hidden size 4,096 at two bytes per element, bf16 activations · 1 tokens of context · 2026-09-09

Estimates from arithmetic, not measurements, and transfer time only: they assume the link runs at its rated speed and ignore every fixed cost of getting a message onto and off the wire. That fixed cost is the subject of the next table and it usually dominates. Read these figures as a floor on the delay a split adds per token.

Two things jump out. Pipeline parallelism is nearly free on every link in the table, which is why it is the home-cluster default. And tensor parallelism is not obviously impossible even on cheap Ethernet, if transfer time were all there was.

It is not. Every exchange has a fixed cost that does not depend on how many bytes you send: the application hands the data to the kernel, the kernel copies it, the network card sends it, the other card receives it, the other kernel copies it, the other application is woken up, and then the same thing happens on the way back. Tensor parallelism pays that cost twice per layer.

Pending validationWhat per-exchange latency alone costs, before any bytes move

Round trip per exchange (microseconds)	Tensor parallel: added delay per token (ms)	Ceiling from latency alone (tokens/s)	Pipeline parallel: added delay per token (ms)
20	2.56	391	0.02
50	6.4	156	0.05
200	25.6	39	0.2
1,000	128	8	1

Not a machine: a range of plausible round-trip times spanning a tuned RDMA link at the low end and a congested or wireless network at the high end · none - arithmetic only, no engine was run computed on 2026-09-09 as 128 exchanges per token for the tensor split and one for the pipeline split · the same hypothetical 64-layer model, not applicable · 1 tokens of context · 2026-09-09

Estimates from arithmetic, not measurements. The tensor-parallel column assumes the all-reduces are serialised with the arithmetic, which a good implementation partly avoids by overlapping them; treat the ceiling as pessimistic but the shape as real. The point is the ratio: latency multiplied by 128 is a different kind of problem from latency multiplied by one, and it is why the two splits belong on different rungs of the interconnect ladder.

That is the whole argument in one table. Tensor parallelism does not fail on a home network because the network is narrow. It fails because the network has a per-message cost and tensor parallelism sends hundreds of messages per token.

What RDMA removes

Remote direct memory access attacks exactly that fixed cost. NVIDIA’s networking documentation defines it as “the remote memory management capability that allows server-to-server data movement directly between application memory without any CPU involvement”.

Read that phrase carefully, because it is the whole benefit. In a conventional path, the data is copied from the application’s memory into the kernel’s buffers, formed into packets, handed to the card, and on the far side reversed, with an interrupt and a process wake-up at the end. With RDMA the network card reads the sending application’s memory directly and writes into the receiving application’s memory directly. The operating system arranged the permissions in advance and then stepped out of the path.

What is in the path, with and without RDMA

1. Application bufferWhere the activation tensor actually lives. Both paths start here.
2. Kernel socket bufferA copy. Present on the conventional path; absent with RDMA.
3. Protocol stackSegmentation, checksums, congestion control in software. Present on the conventional path; offloaded to the adapter with RDMA.
4. Network adapterBoth paths meet here. With RDMA the adapter has already been told which memory it may read and write.
5. The cableIdentical on both paths. RDMA does not make the wire faster; it removes the work at each end of it.

RDMA does not raise the rated speed of a link. It removes copies, context switches and per-packet software work, which is why it changes latency much more than it changes throughput.

RoCE, RDMA over Converged Ethernet, is how that arrives on Ethernet hardware. NVIDIA’s documentation describes two versions: RoCE v1 uses “ethertype 0x8915 and can be used with or without the VLAN tag”, which keeps it on one link-layer segment, while RoCE v2 adds an “IP header which allows traversal of IP L3 Routers and a UDP header”, using a well-known destination port of 4791, which lets it be routed.

There is a condition, and it is the one that catches people. The same documentation states that “in order to function reliably, RoCE requires a form of flow control” and that “the normal and optimal way to use RoCE is to use Priority Flow Control (PFC)”, which “must be enabled on all endpoints and switches in the flow path”. A direct cable between two machines has no switch in the path and is therefore the easy case. A cheap unmanaged switch in the middle is the hard one.

✱Hardware note

This is a strong argument for cabling two machines directly to each other for the model traffic, and leaving the house network for everything else. A direct link has no switch to configure, no other traffic to compete with, and no shared queue to overflow. The reference cluster does exactly this between its two Sparks, and the lab measures both paths so you can see the difference.

ConnectX-7 on the DGX Spark

Track S has the fastest home link in this course, and it is documented. NVIDIA’s ConnectX-7 networking page states that “each DGX Spark has two QSFP ports (sometimes called ‘ConnectX-7 ports’) on the back of the device” and that “each port provides up to 200 Gigabits per second (Gb/s)”. It also states that “each Ethernet interface has a corresponding RoCE interface (typically called a ‘RoCE device’) for InfiniBand communication”, which is the RDMA path.

The page names approved cable part numbers rather than leaving the choice open, and it shows the interface naming the operating system produces: an Ethernet device such as enp1s0f0np0 paired with a RoCE device such as rocep1s0f0. You will need both names in Part 20, because the Ethernet name is what you give an address to and the RoCE name is what an RDMA benchmark opens.

Part 5 recorded the documented topology limits: up to three systems cabled directly, up to four through a switch. Two is the arrangement this course validates.

A direct ConnectX-7 link, with the house network kept separate

• workerDGX Spark A128 GB unified memory
• workerDGX Spark B128 GB unified memory
• routerHouse switchmanagement, downloads, the gateway

• DGX Spark A ⟷ connected to DGX Spark Bdirect QSFP cable, RoCE, no switch in the path
• DGX Spark A ⟷ connected to House switch10 gigabit Ethernet
• DGX Spark B ⟷ connected to House switch10 gigabit Ethernet

Two networks, two jobs. Model traffic never touches the switch; management, model downloads and client requests never touch the direct link. This separation is also what makes the lab's measurements interpretable.

Thunderbolt 5 and RDMA on the Mac

Track M got the same capability by a different route. Apple’s specification pages list Thunderbolt 5 at up to 120 gigabits per second, with four ports on the Mac Studio and three on the M5 Pro Mac mini; the M6 Mac mini has Thunderbolt 4 at up to 40 instead, which is a purchasing point rather than a footnote.

Two Macs can be cabled directly to each other over Thunderbolt and treated as a network. MLX’s distributed documentation describes setting up a ring this way: you “disable the thunderbolt bridge interface”, then for each cable “find the interfaces corresponding to that cable” on both machines and “set up a unique subnetwork connecting the two nodes for the corresponding interfaces”. That is a manual, per-cable procedure, and it is the thing Part 21’s lab walks through.

The newer capability is RDMA. MLX’s documentation states that “starting from macOS 26.2, RDMA over thunderbolt is available and enables low-latency communication between Macs with thunderbolt 5”, and is candid about the ceremony: “until the feature matures, enabling RDMA over thunderbolt is slightly more involved and cannot be done remotely even with sudo. In fact, it has to be done in macOS recovery”, where rdma_ctl enable is run. MLX exposes it as a backend named JACCL, launched with mlx.launch --backend jaccl.

Two other projects have picked it up. llama.cpp’s RPC README lists two RDMA providers, “each enabled by default when its library is found at build time”: on Linux, “RoCEv2-capable NICs (e.g. Mellanox ConnectX), via libibverbs”, and on macOS, “RDMA over Thunderbolt on Apple silicon Macs with Thunderbolt 5, via librdma”, again requiring “macOS 26.2 or later, with RDMA enabled once from macOS Recovery via rdma_ctl enable”. The transport “is negotiated during the initial handshake”, so no command changes, and “the connection falls back to TCP unless both peers can use RDMA” — which is worth knowing, because a cluster that silently fell back to TCP looks exactly like a cluster that is disappointing for no reason. Setting GGML_RPC_NO_RDMA forces TCP, which makes the comparison measurable. exo’s README states that the project “ships with day-0 support for RDMA over Thunderbolt 5” and that its macOS application “requires macOS Tahoe 26.2 or later”; the performance figures on that page are the project’s own claims and Part 21 measures them rather than repeating them.

⌚Version note

Every RDMA capability on Track M depends on macOS 26.2 or later, stated by both MLX’s documentation and llama.cpp’s RPC README as read on 2026-09-09, and has to be enabled once from macOS Recovery. There is no remote way to do it. If the pair of Macs you plan to cluster is not on that release, the Thunderbolt link still works as ordinary networking; you simply do not get the latency benefit.

Jumbo frames, MTU and switches

The maximum transmission unit is the largest payload a link will carry in one frame. Ethernet’s default is 1,500 bytes. A jumbo frame raises it, conventionally to 9,000 bytes, which means six times fewer frames, six times fewer per-frame decisions and noticeably less processor time for the same number of bytes.

Three rules make this safe rather than mysterious.

Every device in the path must agree. Both machines and every switch between them. A machine sending 9,000-byte frames into a switch configured for 1,500 does not gracefully degrade; it produces a connection that completes a handshake and then stalls on the first large transfer, which is one of the most confusing failures in this part.

Set it on the interface that carries model traffic, not on everything. A direct cable between two machines is the ideal place for jumbo frames precisely because there is no switch to agree with. The house network can stay at the default.

Verify it rather than assuming it. The check is a ping carrying a payload just under the target size with fragmentation forbidden: if it arrives, the whole path carries that size; if it does not, something in the middle is smaller than you think. The payload for a 9,000-byte unit is 8,972 bytes, because 20 bytes of IP header and 8 bytes of ICMP header travel with it. The lab does this on both a jumbo and a default link so you can see both answers.

✕Common mistake

Raising the MTU on one machine and not the other is the classic way to build a cluster that passes every quick test and fails every real one. Small messages fit in one frame and work; the first large transfer disappears. If a distributed engine hangs partway through loading a model, check this before anything else.

Verifying a link before you trust it

Three tools, in the order you should reach for them.

iperf3 measures throughput between two machines. It runs as a server on one machine, with --server, and as a client on the other, with --client, connecting on port 5201 by default and running for ten seconds by default. --time changes the duration, --parallel opens several streams at once, --reverse turns the test around so the server sends, and --json produces output a script can parse. The course pins iperf3 3.21 · verified 2026-09-08.

ping measures round-trip time, which the tables above showed matters more than throughput for tensor parallelism, and with the right options it also probes the maximum transmission unit.

ib_write_bw, from the perftest package, exercises the RDMA path specifically. The README describes it as a “bandwidth test with RDMA write transactions” and its invocation follows the package’s general shape: the test with its options on the server, the same test with the same options plus the server’s address on the client, with the warning that “the SAME OPTIONS must be passed to both server and client”. If this runs and iperf3 also runs, you have confirmed both paths; if the RDMA test fails while iperf3 succeeds, your cluster will quietly use TCP.

A buying guide, by track

Link classes, not products. Prices move; classes do not.

Track S, DGX Spark. You already own the best link in this course. A second Spark and one approved QSFP cable gives a direct RoCE path with no switch to configure. Keep the 10 gigabit RJ-45 port for the house network and the direct cable for model traffic. There is nothing to buy beyond the cable, and buying a 200 gigabit switch to sit between two machines that could be cabled directly makes the setup slower, more expensive and harder to debug.

Track X, Ryzen AI Max+ 395. The hardware reference records 2.5 gigabit Ethernet and USB4 or Thunderbolt-class ports on these machines. That means a layer split over 2.5 gigabit Ethernet is your realistic cluster, and the arithmetic above says it is entirely adequate for one. If the box has a spare slot or you are willing to use a USB4 adapter, a 10 gigabit link is the next rung and is worth it mainly for moving model files around rather than for inference. Do not plan tensor parallelism on this track.

Track M, Apple silicon. Thunderbolt 5 between two Macs is the fast path, and on macOS 26.2 or later with RDMA enabled it is the only home link besides the Spark’s that removes the kernel from the data path. Check the Thunderbolt generation before buying a second Mac: the specification pages distinguish Thunderbolt 4 from Thunderbolt 5 by machine and by chip. The 10 gigabit Ethernet port on a Mac Studio is a good management and file-transfer network.

Track N, NVIDIA desktop or laptop. The hardware reference records 1 to 25 gigabit Ethernet for this class. A desktop with a spare slot can take a 10 or 25 gigabit adapter cheaply, and RoCE-capable adapters exist at that class, which is the only way a Track N reader gets RDMA. A laptop generally cannot, and should plan on a layer split over whatever wired Ethernet it has. If you have two GPUs in one desktop, note that the hardware reference records no NVLink on the GeForce 40 and 50 series, so the fast path between them is PCIe and Part 20 measures what that supports.

All tracks. Wired, always. Wi-Fi shares a medium with everything else in the house, reorders packets and varies from second to second. The lab measures a wireless link deliberately, once, so that you have the comparison in your notebook and never wonder again.

Test latency, bandwidth and route separately

A successful ping establishes basic reachability, not application bandwidth. A bulk throughput test measures sustained transfer, not the latency of frequent small collectives. RDMA additionally requires a working device, driver, addressing and transport configuration; Ethernet link speed alone does not prove that path is active.

Start by drawing the intended source interface, destination interface and route on both machines. Test each direction and record packet loss, message-size sensitivity and background traffic. If you change MTU, every relevant hop must support it; a locally accepted interface setting is not end-to-end evidence.

Keep network tests separate from model tests so you can attribute a later slowdown. A fast link can still be bypassed if the engine selects the management network, and a measured model transfer may be limited by storage rather than the cable. Before buying hardware, estimate how much repeated communication the chosen parallelism requires. Match the purchase to that measured need and the supported software path, rather than assuming a larger number printed on the port will improve every cluster workload.

Recap

Divide the advertised gigabits by eight to get bytes, then compare against the parallelism formulas. Pipeline parallelism moves about one activation vector per token and is comfortable on every link class down to 2.5 gigabit Ethernet. Tensor parallelism moves hundreds of messages per token, and it is the per-message cost rather than the width of the link that rules it out on a home network.

Remote direct memory access is the answer to per-message cost: the adapter reads and writes application memory directly and the operating system leaves the path. RoCE brings it to Ethernet, in a routable version 2 over UDP port 4791, and it requires priority flow control on every device in the path, which is a strong argument for a direct cable with no switch in it. Track S has it over ConnectX-7 QSFP ports; Track M has it over Thunderbolt 5 from macOS 26.2, enabled once from Recovery; Track N can have it with the right adapter; Track X plans without it.

Jumbo frames help, and only when every device in the path agrees. Verify with a do-not-fragment ping rather than trusting the configuration, measure throughput with iperf3 and latency with ping before blaming an engine, and confirm the RDMA path separately because it falls back to TCP silently.

The next lesson introduces the machines this course was written on and gives you a worksheet for your own.

Check your understanding

Question 1. A link is advertised at 10 gigabits per second. Roughly what byte rate does that correspond to, before any overhead?

• About 10 gigabytes per second
• About 1.25 gigabytes per second
• About 80 gigabytes per second
• About 0.8 gigabytes per second

Show the answer and why

Answer: About 1.25 gigabytes per second

Divide by eight. The habit matters because memory bandwidth is quoted in gigabytes and link speed in gigabits, so an unconverted comparison flatters the network by a factor of eight before overheads are even considered.

Question 2. Why does RDMA help tensor parallelism more than it helps pipeline parallelism?

• RDMA raises the rated speed of the cable
• Tensor parallelism sends far more messages per token, and RDMA reduces the fixed cost of each message
• Pipeline parallelism cannot use RDMA at all
• RDMA compresses the activation tensors

Show the answer and why

Answer: Tensor parallelism sends far more messages per token, and RDMA reduces the fixed cost of each message

RDMA removes copies, protocol work and a process wake-up from each exchange. Tensor parallelism pays that cost roughly twice per layer per token, so the saving is multiplied by the layer count. Pipeline parallelism pays it once per token, so the same saving is barely visible. The wire speed is identical either way.

Question 3. You raise the MTU to 9,000 on both machines but the switch between them is left at the default. What is the likely symptom?

• Nothing; the switch will fragment the frames
• Both machines refuse to bring the link up
• Small exchanges succeed and the first large transfer stalls
• Throughput drops by about half but everything still works

Show the answer and why

Answer: Small exchanges succeed and the first large transfer stalls

Frames that fit inside 1,500 bytes pass, so handshakes, pings and name lookups all succeed and the link looks healthy. The first frame larger than the switch will carry is dropped, and a distributed engine hangs partway through loading. Verify the path with a do-not-fragment ping rather than trusting the two endpoints.

Question 4. Which of these are true of RoCE as NVIDIA's documentation describes it? Select all that apply.

• Version 2 adds an IP and UDP header so the traffic can be routed
• It requires a form of flow control to work reliably, normally priority flow control
• It makes the cable itself carry more bits per second
• Version 1 uses its own ethertype and stays on one link-layer segment

Show the answer and why

Answer: Version 2 adds an IP and UDP header so the traffic can be routed, It requires a form of flow control to work reliably, normally priority flow control, Version 1 uses its own ethertype and stays on one link-layer segment

RoCE v1 uses ethertype 0x8915; v2 encapsulates in IP and UDP on port 4791 and is routable; and the documentation states that reliable operation needs flow control, normally priority flow control enabled on every endpoint and switch in the path. What RoCE does not do is change the wire rate: it removes work at the ends, not bits on the cable.

Question 5. You have two Ryzen AI Max+ 395 machines with 2.5 gigabit Ethernet and want to run a model that fits across them but not on either one. What should you plan?

• A tensor-parallel split, because 2.5 gigabit is enough bandwidth for one megabyte per token
• A layer split, because it moves one small transfer per token and 2.5 gigabit is comfortable for that
• Nothing; a cluster is impossible below 10 gigabit Ethernet
• A wireless link, to avoid a switch in the path

Show the answer and why

Answer: A layer split, because it moves one small transfer per token and 2.5 gigabit is comfortable for that

The arithmetic in this lesson puts a layer split at a few tens of microseconds per token on 2.5 gigabit Ethernet, which is negligible beside the time spent reading weights. Tensor parallelism would be ruled out by per-message latency rather than by bandwidth. And wireless is never the path a model is split across: it is shared, variable and reorders packets.

Check my answersClear