Skip to content
Local LLM Course

NVIDIA Clusters: DGX Spark Pairs, vLLM Multi-Node and Multi-GPU Desktops

Part 18 gave you the vocabulary and the arithmetic, and Part 19 built a cluster with llama.cpp RPC that will run on almost anything. This part is the other NVIDIA answer: the one where a vendor has written the procedure down, tested it on the exact hardware, and published container images that already contain the matched CUDA, driver and Python builds.

That changes the character of the work. On Track S you are not assembling a distributed system from first principles; you are following NVIDIA’s connect-two-Sparks playbook, then NVIDIA’s vLLM multi-node playbook, and reading the logs carefully enough to know when the documented path has stopped describing your machine. On Track N you are doing something quite different, because two consumer cards in one box have no NVLink between them and every collective operation crosses PCIe. Both are in this part, because the same three questions decide both: how much data crosses the link per token, how fast the link is, and whether the split you chose puts that traffic on the critical path of every layer or only at a stage boundary.

The honest headline for Level 4 is that a pair of Sparks is the most accessible way to run a 400B-class model at home with a procedure somebody else has already debugged, and that NVIDIA’s own documentation is candid about how little headroom is left when you do. This part quotes that candour rather than smoothing it over.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • cable two DGX Sparks together, give the QSFP interfaces addresses that survive a reboot, establish passwordless SSH between them, and verify the link before any engine touches it;
  • name the RoCE device that corresponds to each Ethernet interface, and set the handful of environment variables that decide which interface NCCL, UCX and Open MPI actually use;
  • start a Ray cluster across two hosts, run one vllm serve command that uses both, and read the startup log well enough to know how many tokens of key-value cache the pair actually has;
  • choose tensor parallel size, pipeline parallel size and data parallel size for a given topology, and say why the choice is different on two one-GPU machines than on one eight-GPU machine;
  • explain what expert parallelism does for a mixture-of-experts model and when it is worth the extra dependencies;
  • predict what two consumer cards in one desktop will and will not do, from PCIe generation and lane count, and set up llama.cpp’s layer split or vLLM’s tensor parallel accordingly;
  • do the power and cooling arithmetic for a two-card desktop from the manufacturer’s own specification figures, before buying the second card;
  • describe TensorRT-LLM’s multi-node path on a Spark pair, say what NVIDIA Dynamo is for and what it requires, and state plainly where each belongs at home;
  • serve a very large model across two Sparks, load-test it, and record throughput, time to first token and time per output token against a single-machine baseline.

What you need first

Section titled “What you need first”

Part 18, all of it. This part uses tensor, pipeline, expert and data parallelism as settled vocabulary, and it assumes you have already built the network layer its lab describes: names, keys, a measured figure for every link, and model storage both machines can read. The two machines this part is written against are the two DGX Sparks of the course reference cluster that Part 18 introduces, joined by a direct ConnectX-7 cable with the house network kept separate. Part 9, because every serving option here is a vllm serve option you have already met on one machine, and because the load generator this part measures with is the one you used there. Part 5’s DGX Spark lesson for the machine itself, and Part 8’s TensorRT-LLM lesson for the container path and the quantisation support matrix.

Part 19 is not a prerequisite, but it is a useful contrast: llama.cpp RPC and vLLM multi-node solve overlapping problems with completely different tolerances for a slow link.

The primary path needs two DGX Sparks with 128 GB each and one approved QSFP cable. Every page states what to do with one machine instead, and the lab’s fallbacks are written out in full rather than mentioned: one Spark with a 120B-class mixture-of-experts model, or a desktop with two 24 GB cards. Tracks X and M are redirected by name to Parts 19 and 21, which are where AMD and Apple clusters actually live.

How to work through it

Section titled “How to work through it”

Read the first lesson with the cable in your hand if you have one, because it is a procedure rather than an argument, and the fault it is written to prevent is the one where a distributed engine fails for forty minutes over an interface name. The second lesson is the centre of the part and the one worth re-reading: Ray, the parallel sizes and the environment variables that decide whether tensors cross the fast cable or the house network. The third lesson is for Track N and can be read out of order; it is the one to read before buying a second graphics card. The fourth surveys NVIDIA’s own serving stack on the pair and says what this course validated and what it did not.

Then do the lab, which is long, and which needs both machines free for an hour and a half. It ends by comparing the pair against one machine on the same load, and that comparison is the point: a cluster is worth having when the number in your notebook says so.

Part20 of 27Level4 — Cluster ArchitectPages5Estimated time3.4 hours

1 of 5 pages in this part need two or more machines for their primary path, marked below. Every one of them has a single-machine path, so the part can be completed with one computer.

0 / 5 lessons in this part completed