Connecting Two DGX Sparks over ConnectX-7

By the end of this lesson you will be able to identify which of a DGX Spark’s four QSFP-derived interfaces the cable actually brought up, give those interfaces addresses that survive a reboot, establish the passwordless SSH the playbooks assume, name the RoCE device that corresponds to each Ethernet interface, and set the environment variables that decide which wire your tensors travel on. You will also be able to say what NVIDIA claims a connected pair can run, and read the qualification NVIDIA attaches to that claim.

This is a procedure lesson. It is short on argument and long on names, because the names are what go wrong.

The ports, the cable, and what “two ports” means

Part 5 recorded the hardware. NVIDIA’s ConnectX-7 networking page states that “each DGX Spark has two QSFP ports (sometimes called ‘ConnectX-7 ports’) on the back of the device” and that “each port provides up to 200 Gigabits per second (Gb/s)”. It tells you which is which from behind the machine: “when seen from the back of the device, the left port is the QSFP port closest to the ethernet port.”

The cable is not a free choice. The same page names approved part numbers from Amphenol and Luxshare rather than describing a category, and the switch playbook points at NVIDIA’s own marketplace listing for a 0.4 m QSFP cable. Buy from that list. A QSFP cable that negotiates at a lower rate, or does not negotiate at all, is a fault you will spend an evening attributing to software.

Vendor specification, not measuredThe DGX Spark ConnectX-7 link, as NVIDIA documents it

Figure	What NVIDIA states	Where
QSFP ports per machine	Two	ConnectX-7 Networking
Rate per port	Up to 200 Gb/s	ConnectX-7 Networking
Linux interfaces per port	Two independent Ethernet interfaces	ConnectX-7 Networking
RDMA path	One RoCE device per Ethernet interface	ConnectX-7 Networking
Direct cabling limit	Up to three systems	ConnectX-7 Networking
Through a switch	Up to four systems	ConnectX-7 Networking
Largest model claimed	Up to 700 billion parameters across four systems	DGX Spark product page

NVIDIA DGX Spark (GB10), two systems, DGX OS 7.x · no engine; NVIDIA documentation retrieved 2026-09-09 · no model loaded, not applicable · 0 tokens of context · 2026-09-09

Every figure here is published by NVIDIA on the pages in this page's sources. None of it was measured by this course. Part 18's lab measures the link you actually have, in both directions, and that number is the one to trust when an engine disappoints you.

The sentence that surprises people is the third row. Each physical QSFP port appears to Linux as two independent Ethernet interfaces, so one machine with two ports shows four. The playbook prints an example of exactly that, and pairs each Ethernet name with its RDMA counterpart:

Output — what you should see

rocep1s0f0   port 1 ==> enp1s0f0np0   (Up)
rocep1s0f1   port 1 ==> enp1s0f1np1   (Up)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)

Read that as four pairs. The name beginning en is an ordinary Ethernet interface: it is what you give an address to, and what NCCL_SOCKET_IFNAME selects. The name beginning roce is the RDMA device for the same hardware: it is what an RDMA benchmark opens and what NCCL_IB_HCA filters on. They are two views of one port, and confusing them is the most common way to end up with a cluster that works and is slow.

✕Common mistake

The playbook is explicit about a physical mistake that produces a software-looking symptom: “make sure to use the same physical port on each device to prevent issues with NCCL tests.” Left port to left port. If none of the four interfaces reports (Up), the page’s advice is to “check the QSFP cable connection, reboot the systems and try again” before touching anything else.

Addressing the link

The command that tells you what the cable did is ibdev2netdev, run on both machines. Whichever interfaces report (Up) are the ones the cable brought up, and the playbook’s own example uses enp1s0f1np1 and enP2p1s0f1np1. Yours may differ, because it depends on which port you used.

The playbook then offers two ways to give them addresses, and says the two are mutually exclusive. Option 1 writes a netplan file, sets its permissions to 600, and applies it, which survives a reboot. Option 2 assigns addresses with ip addr add and brings the interfaces up by hand, with the explicit warning that “using this option, the IPs assigned to the interfaces will change if you reboot the system.” Use netplan for anything you intend to run more than once.

Two details in the addressing are worth understanding rather than copying.

One subnet per cable, not one for the pair. The playbook puts the first interface on one private class-C subnet and the second on another, so that each cable is its own point-to-point network with no routing decision to make. Both machines take the same host position in each subnet, differing only in the final octet.

One cable is enough for full rate. The playbook states it twice: “full bandwidth can be achieved with just one QSFP cable”, and adds that “when two QSFP cables are connected, all four interfaces must be assigned IP addresses to obtain full bandwidth.” A second cable half-configured is worse than no second cable, because the engine may pick the unaddressed pair.

⛨Security

These addresses live on a cable that goes nowhere else, which is the whole reason the arrangement is safe enough to run unencrypted collective traffic over. Keep it that way: do not bridge the QSFP subnets onto the house network, and do not let the serving port bind to them. vLLM’s own documentation is blunt about the risk, saying that traffic on this network “is unencrypted, and the endpoints exchange data in a format that can be exploited to execute arbitrary code if an adversary gains network access.”

Same user, no password

The playbooks assume two things about accounts that are easy to skip and expensive to skip.

The first is that both machines have the same username, and the playbook’s step 1 is to check with whoami and create a matching user if they differ. Open MPI launches remote ranks over SSH as the local user; a mismatch fails at the point where a distributed run starts, not at the point where you configured it.

The second is passwordless SSH in both directions. NVIDIA ships a discovery script, discover-sparks, that finds the nodes on the link and exchanges keys, printing a summary that ends “SSH setup complete! Both local and remote nodes can now SSH to each other without passwords.” If it fails, the documented fallback is manual: read each node’s address with ip addr show on the chosen interface, then run ssh-copy-id from each node to the other.

✕Common mistake

NVIDIA’s vLLM multi-node playbook records the failure that this script produces on a machine nobody has ever used SSH on: the script “writes its SSH key to ~/.ssh/ and fails if the directory does not exist.” Its fix is one line, run on both nodes before the script:

RunnableTrack S · DGX Spark

make the key directory the discovery script expects

mkdir -p ~/.ssh && chmod 700 ~/.ssh

Which wire, exactly: the variables that decide

A DGX Spark pair has at least two networks between it: the direct QSFP link and whatever the house switch or Wi-Fi provides. Every distributed framework has to be told which one to use, and each of them has its own variable for it. Getting this wrong does not produce an error. It produces a cluster that runs at the speed of the slow network.

Four consumers of an interface name, and what each one wants

1. 1Ray and vLLM bootstrapVLLM_HOST_IP, set per node to that node's own address on the chosen interface. vLLM's documentation says to set it to "an address on a private network segment".
2. 2NCCL, for the collectivesNCCL_SOCKET_IFNAME selects the Ethernet interface; NCCL_IB_HCA filters the RDMA devices. NCCL_SOCKET_IFNAME "will bypass the automatic interface selection algorithm".
3. 3Gloo and the PyTorch storeGLOO_SOCKET_IFNAME and TP_SOCKET_IFNAME, which the NVIDIA playbook sets to the same interface as NCCL.
4. 4Open MPI and UCXOMPI_MCA_btl_tcp_if_include and UCX_NET_DEVICES. TensorRT-LLM's multi-node path launches with mpirun, so these are the ones that matter there.

The NVIDIA vLLM multi-node playbook sets all of these to one interface name held in a shell variable, which is the pattern worth copying: name the interface once, export it into everything.

The NCCL documentation is worth reading for two sentences rather than for its full length. On interface selection it says that NCCL_SOCKET_IFNAME takes “a list of prefixes to filter interfaces to be used by NCCL”, that a leading = means an exact name and a leading ^ means exclude, and that setting it “will bypass the automatic interface selection algorithm”. On the RDMA side, NCCL_IB_HCA “specifies which Host Channel Adapter (RDMA) interfaces to use for communication”, again as a prefix list with the same = and ^ conventions. NCCL_IB_GID_INDEX “defines the Global ID index used in RoCE mode”, and its default is -1, meaning NCCL selects one; leave it alone until something tells you not to.

Two more are for diagnosis rather than configuration. NCCL_DEBUG accepts VERSION, WARN, INFO and TRACE, and NCCL_IB_DISABLE set to 1 “prevents the IB/RoCE transport from being used by NCCL. Instead, NCCL will fall back to using IP sockets.” That second one is a measurement tool: run the same job with and without it and the difference is what RDMA is worth on your pair.

!Important

There is a genuine inconsistency between two NVIDIA playbooks here, and it is not a mistake in either. The NCCL playbook exports NCCL_SOCKET_IFNAME as the management interface, because mpirun launches ranks over the address you SSH to and the collective then negotiates its own RDMA path. The vLLM multi-node playbook exports it as the QSFP interface, because Ray’s bootstrap and NCCL’s socket transport both run there. Follow whichever playbook you are actually in, and check with NCCL_DEBUG=INFO which transport was chosen rather than assuming.

The official pair, and what NVIDIA says it runs

The connect-two-Sparks playbook is a short document with a stated budget: “duration: 1 hour including validation”, “risk level: medium — involves network reconfiguration”, and a rollback that is simply removing the netplan file or deleting the addresses. That is the honest shape of this work. It is an hour of careful naming, and it is reversible.

The pair this part builds

• workerDGX Spark A128 GB unified memory; Ray head; the served endpoint
• workerDGX Spark B128 GB unified memory; Ray worker
• routerHouse networkSSH, model downloads, clients

• DGX Spark A ⟷ connected to DGX Spark Bdirect QSFP cable: RoCE, no switch, its own subnet
• DGX Spark A ⟷ connected to House networkmanagement interface
• DGX Spark B ⟷ connected to House networkmanagement interface

Two networks with two jobs. Model traffic never crosses the house network; clients never touch the QSFP link. Part 18's lab built the right-hand side of this picture; this part builds the left.

What does the pair buy? NVIDIA’s product page frames it from the model’s side: ConnectX networking “enables the connection of up to four NVIDIA DGX Spark systems to work with AI models of up to 700 billion parameters.” For two machines, the arithmetic is the one Part 4 taught: 256 GB of unified memory holds a four-bit checkpoint of a few hundred billion parameters, and whatever survives two operating systems, two containers and two engines is the key-value cache. On a large enough model that remainder is very small, which is the whole story of the next callout.

The playbooks are concrete about which models. The TensorRT-LLM playbook’s support matrix marks nvidia/Qwen3-235B-A22B-FP4 as “two Sparks only”, which is the vendor naming a 235B-class mixture-of-experts model as a pair-sized workload. That handle redirected to a repository named nvidia/Qwen3-235B-A22B-NVFP4 when the Hub was checked on 2026-09-09, which is the ordinary way a vendor repository gets renamed; follow the redirect and check the card. Qwen3-235B-A22B is Apache-2.0 licensed according to the model reference. The vLLM multi-node playbook’s main worked example is a 70B dense model, with a 405B model in an optional final step.

?Reality check

The claim worth holding up is “two Sparks run a 405B model”. NVIDIA’s own vLLM multi-node playbook does document it, using an INT4 AWQ conversion of Llama 3.1 405B Instruct, and it attaches this warning on the same page: “the 405B model has insufficient memory headroom for production use — testing only.” The serve command it gives asks for a maximum model length of 64 tokens and a maximum of one sequence, which is what “no headroom” looks like written as flags. So the claim is true and nearly useless: the pair can load the weights and generate, and it cannot hold a conversation. Llama 3.1 405B is published under the Llama 3.1 Community licence, whose terms you should read on the model card before using it for anything.

That is not a reason to skip the exercise. Loading a 400B-class model on hardware you own and watching it answer is worth doing once, and this part’s lab has you do it, then measure it, then compare it against a 120B-class mixture-of-experts model on one machine that will very probably be more useful. Both numbers go in the notebook. The comparison is the lesson.

Three Sparks, four Sparks, and a switch

The documented topology limits from Part 5 hold here. NVIDIA’s ConnectX-7 page states support for “up to three DGX Spark systems connected directly through cables, and up to four systems when using a switch”, and there is a playbook for each: connect two, connect three in a ring, and connect multiple through a switch.

The switch path is a different quantity of work. Its playbook budgets “2 hours including validation” against the pair’s one, and its prerequisites include knowing how to configure the switch itself: how to enable and disable QSFP ports, how to create a bridge, and “how to configure the link speed manually on the port and disable auto-negotiation if needed”. It asks for “a QSFP switch with at least 4 QSFP56-DD ports (at least 200Gbps each)”.

Part 18 explained why the switch is the harder case in a way that has nothing to do with convenience. RoCE needs flow control to work reliably, and NVIDIA’s networking documentation says the normal way to provide it is priority flow control enabled on every endpoint and every switch in the path. A direct cable has no switch in the path. That is why this course validates the pair and describes the switch.

Verifying, before an engine is involved

The order matters. Every check below is cheap, and each one rules out a class of fault that is expensive to diagnose once a model is loading.

1. ibdev2netdev on both nodes. The expected result is at least one interface (Up) on each, with the same name on both if you used the same physical port. This is a cable and firmware check.
2. ip addr show on the chosen interface, both nodes. The address you configured is present and the interface is UP. This is a netplan check.
3. A ping across the link. Reachability, and a first sense of latency.
4. SSH in both directions, by address, without a password. The playbook’s own verification is ssh <address> hostname from each node. This is what Open MPI and Ray will do.
5. A collective. NVIDIA’s NCCL playbook builds NCCL and nccl-tests from source on both nodes and runs all_gather_perf under mpirun, with a larger buffer size to “use more of your 200Gbps bandwidth”. This is the first thing that proves the GPUs can talk, and it takes about half an hour of building.

The troubleshooting table NVIDIA publishes for the pair has exactly three rows, and they map onto the checks above: “network unreachable” means the interfaces are not configured, so verify the netplan file and apply it; SSH authentication failures mean the keys were not distributed, so re-run the discovery script; and a node not visible in the cluster means a connectivity problem, so check the cable and the addresses.

⌂Single-machine path

None of this needs two machines to be understood, and some of it can be practised on one. A single Spark still has two QSFP ports and four interfaces, so ibdev2netdev shows you the naming and the Ethernet-to-RoCE pairing on the machine you own. The variables in this lesson are the same variables Part 22 will set for disaggregated serving. If you have one Spark, read this lesson, run the first check, and take the single-Spark path in this part’s lab.

Establish a two-node acceptance ladder

Check the pair in increasing order of cost: device and interface visibility, addressing and route, SSH using the intended account, bulk transfer, collective or RDMA test, then model serving. Save outputs from both nodes at every stage. A later failure is easier to localise when the lower layers already have evidence.

Keep the management connection distinct from the high-speed data link. Record which interface each runtime uses and confirm traffic with counters during a test. Hostname resolution can select a different address than the one you intended, so capture resolved endpoints as well as names.

Before loading a large checkpoint, compare software and image identities and verify the model files needed by each rank. Use a small distributed test first. If a worker fails, retain its first error; the coordinator’s timeout is often only the downstream symptom. A completed network setup is an input to the serving lab. It does not yet establish that a particular model, quantisation or parallelism configuration can run on the pair.

Recap

Two DGX Sparks are joined by one approved QSFP cable between the same physical port on each machine. Each port presents two Ethernet interfaces to Linux and each Ethernet interface has a matching RoCE device, so ibdev2netdev is the command that tells you what you actually have. Addresses go on the interfaces that came up, one private subnet per cable, written into netplan if you want them after a reboot; NVIDIA states that one cable reaches full rate and that a second cable must have all four interfaces addressed to help.

Both machines need the same username and passwordless SSH in both directions, which the discover-sparks script arranges and which fails on a machine that has never had a ~/.ssh directory. From there the work is naming: VLLM_HOST_IP for the bootstrap, NCCL_SOCKET_IFNAME and NCCL_IB_HCA for the collectives, GLOO_SOCKET_IFNAME and TP_SOCKET_IFNAME for PyTorch’s store, and OMPI_MCA_btl_tcp_if_include with UCX_NET_DEVICES for the MPI path, all pointed at one interface held in one shell variable.

NVIDIA documents up to three Sparks cabled directly and up to four through a switch, and claims up to 700 billion parameters across four systems. For two, the vendor’s own 405B example carries a warning that there is insufficient memory headroom for production use, which is the honest state of the art on this hardware and the reason the lab measures a 120B-class model on one machine alongside it.

Check your understanding

Question 1. ibdev2netdev on a Spark prints four lines, each pairing a name beginning "roce" with a name beginning "en". What is the relationship?

• They are two different cables
• They are two views of the same port: the "en" name is the Ethernet interface you give an address to, the "roce" name is the RDMA device an RDMA benchmark or NCCL_IB_HCA refers to
• The "roce" names are inactive backups
• One is for the CPU and one is for the GPU

Show the answer and why

Answer: They are two views of the same port: the "en" name is the Ethernet interface you give an address to, the "roce" name is the RDMA device an RDMA benchmark or NCCL_IB_HCA refers to

NVIDIA's ConnectX-7 page states that each Ethernet interface has a corresponding RoCE interface. You address the Ethernet name and select the RoCE name for RDMA. Mixing them up is why a cluster can be configured, working, and slow.

Question 2. You configured the link with ip addr add, ran a successful distributed job, then rebooted both machines. The job now fails. Why?

• The cable degraded
• The playbook warns that addresses assigned with ip addr add change on reboot; the netplan option is the one that persists
• NCCL caches the old addresses
• The firmware reset the ports

Show the answer and why

Answer: The playbook warns that addresses assigned with ip addr add change on reboot; the netplan option is the one that persists

The playbook offers netplan and command-line assignment as mutually exclusive options and notes explicitly that the command-line addresses do not survive a reboot. Anything you intend to run twice belongs in the netplan file, with permissions 600 as the playbook sets them.

Question 3. Which statements about NVIDIA's documented DGX Spark topologies are correct? Select all that apply.

• Up to three systems can be connected directly with cables
• Up to four systems can be connected through a switch
• Full bandwidth needs both QSFP cables connected
• NVIDIA states that up to four systems can work with AI models of up to 700 billion parameters

Show the answer and why

Answer: Up to three systems can be connected directly with cables, Up to four systems can be connected through a switch, NVIDIA states that up to four systems can work with AI models of up to 700 billion parameters

The third is the opposite of what the playbook says: full bandwidth is achievable with one cable, and a second cable only helps if all four interfaces are addressed. The other three are direct statements from the ConnectX-7 networking page and the product page.

Question 4. A distributed run across the pair works but is far slower than the link should allow. Which single check is most likely to explain it?

• Whether the GPUs are the same model
• Which interface NCCL selected, checked with NCCL_DEBUG=INFO, since an unset or wrongly set NCCL_SOCKET_IFNAME lets the automatic algorithm pick the management network
• Whether the model is quantised
• The version of Docker

Show the answer and why

Answer: Which interface NCCL selected, checked with NCCL_DEBUG=INFO, since an unset or wrongly set NCCL_SOCKET_IFNAME lets the automatic algorithm pick the management network

Nothing errors when collectives run over the house network instead of the cable; the job simply crawls. NCCL_SOCKET_IFNAME bypasses the automatic selection, and NCCL_DEBUG set to INFO tells you which transport was actually chosen. Measure before you tune.

Question 5. True or false: NVIDIA documents two DGX Sparks running a 405B-parameter model, so a pair is a practical way to serve one.

• True
• False

Show the answer and why

Answer:

The first half is true and the conclusion does not follow. NVIDIA's vLLM multi-node playbook documents the 405B step and warns on the same page that the model has insufficient memory headroom for production use, testing only, serving it with a maximum model length of 64 tokens and one sequence at a time. It loads and generates; it cannot hold a conversation.

Check my answersClear