Level 4 · Cluster ArchitectLessonPart 20 · page 2 of 532 min

32Minutes

3Tools

8Sources

Tools used on this page3

ray
vllm
docker

vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across

By the end of this lesson you will be able to start a Ray cluster across two machines and confirm it sees both of them, choose tensor, pipeline and data parallel sizes for a topology rather than copying them from a page written for a different one, say what expert parallelism changes for a mixture-of-experts model, set the environment variables that decide which cable your collectives use, read the two startup lines that predict whether the arrangement will serve your workload, and recognise the three ways it falls over.

This course pins vLLM 0.28.0 · verified 2026-09-08. Everything in this lesson comes from vLLM’s own distributed-serving documentation and NVIDIA’s DGX Spark playbook, both read on 2026-09-09, and neither has yet been executed by this course on hardware.

One command, two machines

The thing to hold on to about vLLM multi-node is that it does not change how you serve. You start a Ray cluster, and then you run one vllm serve command, on one node, exactly as if the machine had more GPUs than it does. vLLM’s documentation says so plainly: “once a Ray cluster is running, use vLLM as you would in a single-node setting. All resources across the Ray cluster are visible to vLLM, so a single vllm command on a single node is sufficient.”

Ray is the piece that makes that true. vLLM describes it as “a distributed computing framework for scaling Python programs”, and says that “vLLM uses Ray to manage the distributed execution of tasks across multiple nodes and control where execution happens”. It is an optional dependency, installed with pip install "ray[cgraph]", and it is the default runtime for multi-node work: the docs state that “the default distributed runtimes are Ray for multi-node inference and native Python multiprocessing for single-node inference.”

What happens between plugging in the cable and answering a request

Head node starts Rayray start --head, or NVIDIA's containerised run_cluster.sh with --head. It prints the address workers join on.
Worker joinsray start --address=<head address:port> on the second machine, or run_cluster.sh with --worker. Ray now advertises both machines' GPUs as one pool.
ray status confirms the poolRun from either node. vLLM's guidance is to "run ray status and ray list nodes to verify that Ray finds the expected number of nodes and GPUs".
One vllm serve commandRun on one node. The parallel sizes tell vLLM how to lay the model out over the pool; the API server listens on that node only.
Clients talk to one endpointThe second machine has no HTTP server on it and never needs one. It is a compute resource, not a service.

Starting the cluster, two ways

Ray’s own on-premises documentation gives the bare procedure. On the head node:

RunnableAll tracks

ray start --head --port=6379

Ray’s documentation notes that omitting --port makes Ray try 6379 first and fall back to a random port if that one is in use, which is a good reason to state it rather than leave it to chance. The command prints the cluster address; on the other machine you pass that address back:

Fragment — not complete on its own

# Substitute the address:port the head node printed.
ray start --address=<head-node-address:port>

Ray “auto-detects the resources (e.g., CPU) available on each node”, and --num-cpus and --num-gpus override that detection when you need to.

The second way is the one NVIDIA’s playbook uses, and it is the better one on a Spark. vLLM ships a helper script, examples/ray_serving/run_cluster.sh, which “starts containers across nodes and initializes Ray”. You run it with the container image, the head node’s address, a role flag, the Hugging Face cache directory to mount, and any environment variables to pass through:

Fragment — not complete on its own

# The shape of the helper script's invocation, from vLLM's documentation.
bash run_cluster.sh <image> <HEAD_NODE_IP> --head <huggingface-home-on-this-node> -e VLLM_HOST_IP=<this node>
bash run_cluster.sh <image> <HEAD_NODE_IP> --worker <huggingface-home-on-this-node> -e VLLM_HOST_IP=<this node>

Two properties of that script decide how you use it. VLLM_HOST_IP “is unique for each worker”: it is always the address of the machine you are typing on, never the head’s, except on the head itself where they coincide. And the script holds the container open, so “keep the shells running these commands open; closing any shell terminates the cluster.” NVIDIA’s playbook adds the operational consequence, which is to run both inside tmux or screen, because run_cluster.sh has an exit trap that stops the container when the shell dies.

Choosing the parallel sizes

vLLM’s guidance is a short ladder, and it is written for machines with several GPUs each:

one GPU, if the model fits, needs no distributed inference at all;
a model too large for one GPU but small enough for one node uses tensor parallelism, “for example, set tensor_parallel_size=4 when using a node with 4 GPUs”;
a model too large for one node combines the two: “set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes.”

That last line is the one everybody quotes, and on a DGX Spark pair it produces a strange-looking answer. Each Spark has one GPU. Tensor parallel size equal to GPUs per node is one, and pipeline parallel size equal to nodes is two. Follow the rule literally and you get a pipeline split, which is exactly what Part 18 said tolerates a slow link.

NVIDIA’s own playbook does the opposite. Its serve step across two Sparks sets tensor parallel size to two, spanning the pair, and the four-node section says to “set --tensor-parallel-size equal to your node count”. That is not a contradiction of the vLLM guidance so much as a consequence of having a link fast enough to make it work: the ConnectX-7 cable is closer in character to an intra-node interconnect than to household Ethernet, so the arrangement vLLM warns about between nodes is the arrangement NVIDIA validates between these nodes.

vLLM also documents an edge case that matters more at home than in a data centre. “If the model fits within a single node but the GPU count doesn’t evenly divide the model size, enable pipeline parallelism, which splits the model along layers and supports uneven splits”, setting tensor parallel size to one and pipeline parallel size to the number of GPUs. And it repeats the warning from Part 9 for machines without a fast interconnect: “if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead.”

Here is the serve command across a pair, in the shape this course uses. It is one command, run on one node, after ray status has shown two nodes:

RunnableTrack S · DGX Spark

vllm serve /models/the-model-directory \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name local-cluster \
    --tensor-parallel-size 2 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 4

Mixture-of-experts: data parallel with expert parallel

Everything above splits one copy of the model. For a mixture-of-experts model there is a better arrangement, and vLLM documents it as its own deployment mode.

The idea is to treat the attention layers and the expert layers differently. vLLM’s data-parallel page states it directly: “for MoE models, particularly those like DeepSeek that employ MLA (Multi-head Latent Attention), it can be advantageous to use data parallel for the attention layers and expert or tensor parallel (EP or TP) for the expert layers.” Data parallelism replicates weights across ranks so each rank handles its own requests; the expert layers are the part that is too big to replicate, so they are shared.

By default, sharing those layers means tensor parallelism: “by default, expert layers form a tensor parallel group of size DP × TP.” Setting --enable-expert-parallel changes that to expert parallelism, where whole experts live on single devices. vLLM’s expert-parallel page gives the sizing rule as EP_SIZE = TP_SIZE × DP_SIZE, and describes what changes: “when EP is enabled, different layers in MoE models behave differently”, with expert layers “sharded across all EP ranks” and attention layers replicated when tensor parallel size is one.

The catch is dependencies. The single-node case works with the default allgather_reducescatter backend, described as “general purpose, works with any EP+DP configuration”. The multi-node case is where the page lists prerequisites: DeepEP, DeepGEMM, gdrcopy, and a NCCL newer than the one PyTorch ships, since “the deepep_v2 backend requires NCCL >= 2.30.4”. That is a build project, not a flag.

The multi-node data-parallel launch is not one command. vLLM’s page says “running a single data parallel deployment across multiple nodes requires a different vllm serve to be run on each node, specifying which DP ranks should run on that node”, with a family of --data-parallel-* options naming the global size, this node’s share, its starting rank and the coordinator’s address, and --headless marking the nodes that run engines without an API server. A Ray variant, --data-parallel-backend=ray, reduces it back to “a single launch command (on any node) to start all local and remote DP ranks”.

One sizing trap is worth carrying: --max-num-seqs applies per rank, while the admission limits apply to the whole server. vLLM’s example is four ranks at 256 sequences each with a server-wide queue cap of 256, which “rejects new requests once 256 are in-flight in total, even though ranks could jointly run 1024”.

Pointing the traffic at the right cable

Part 18 measured your links and the previous lesson named the variables. Here is where they are consumed.

VLLM_HOST_IP is the address vLLM advertises for this process, and vLLM’s security note says to “set VLLM_HOST_IP to an address on a private network segment”, because “traffic sent over this network is unencrypted”. On a Spark pair that means the QSFP subnet, not the house network. NCCL_SOCKET_IFNAME selects the interface NCCL’s socket transport uses, and GLOO_SOCKET_IFNAME with TP_SOCKET_IFNAME do the same for PyTorch’s rendezvous machinery; NVIDIA’s playbook sets all three to one interface name held in one shell variable.

For the RDMA path, vLLM’s advice is to pass NCCL_IB_HCA through to the containers: “to set up the cluster to use InfiniBand, append additional arguments like --privileged -e NCCL_IB_HCA=mlx5 to the run_cluster.sh helper script.” And it gives the check that tells you whether any of it worked:

Fragment — not complete on its own

# Start the server with NCCL logging on, then read what transport it chose.
NCCL_DEBUG=TRACE vllm serve /models/the-model-directory --tensor-parallel-size 2

vLLM’s own reading of that log is the useful part. “If you find [send] via NET/IB/GDRDMA in the logs, then NCCL is using InfiniBand with GPUDirect RDMA, which is efficient. If you find [send] via NET/Socket in the logs, NCCL used a raw TCP socket, which is not efficient for cross-node tensor parallelism.” That one line separates a working pair from a disappointing one, and it costs nothing to look at.

Two container settings belong with this. vLLM’s GPUDirect RDMA section gives --ipc=host, --shm-size=16G and a /dev/shm mount as the Docker configuration. Part 8 met --ipc=host in the TensorRT-LLM container for the same underlying reason: shared-memory limits inside a default container break these frameworks in ways whose error messages do not mention shared memory.

Reading the startup log

vLLM prints the two most useful numbers in the whole exercise before it serves anything, and its documentation quotes them:

Output — what you should see

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens
INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

The first line is “the total number of tokens that can be stored in the GPU KV cache at once”, now summed across the whole cluster. The second is the first divided by your context length: “an estimate of how many requests can be served concurrently if each request requires the specified number of tokens”. vLLM’s advice about it is the right advice: “if these numbers are lower than your throughput requirements, add more GPUs or nodes to your cluster.”

Copy both into the notebook every time you change a serving option. On a pair running a very large model at four-bit precision, the cache line is where you discover that almost nothing is left after the weights, and it is much cheaper to discover that in the log than under load.

When it breaks

Three failure modes account for most of what goes wrong, and each has a distinct signature.

The worker never joins. ray status shows one node. Ray’s message is Unable to connect to GCS at ..., and its causes are the head not running, a version mismatch, a wrong address or a firewall. NVIDIA’s playbook table has the hardware version of the same row: “node not visible in Ray cluster … verify QSFP cable connection and IP configuration”. Check the address you gave the worker against what the head printed, on the QSFP interface rather than the management one.

The cluster dies when a terminal closes. This is not a fault. run_cluster.sh stops its container on exit, and vLLM’s documentation says to keep the shells open. Use tmux or screen, as NVIDIA’s playbook instructs, and treat the head shell as part of the running service.

Everything works and nothing fits. The engine starts, the log reports a tiny cache, and requests queue or are refused. NVIDIA’s troubleshooting row for “CUDA out of memory” is to “reduce --max-model-len and --max-num-seqs, or lower --gpu-memory-utilization”. On a Spark there is a unified-memory wrinkle underneath it: the same page notes that “some applications have not yet been updated for UMA, so you may hit memory issues even within capacity”, with a documented remedy of flushing the buffer cache.

Restarting is the ordinary repair, and the order matters: stop the serve command, stop the worker container, stop the head container, then bring them up head-first. A worker that joins a cluster whose head has been restarted underneath it is a stale-membership problem that looks like a network problem.

Track S — NVIDIA DGX Spark

Primary. Two Sparks over the QSFP cable from the previous lesson, NVIDIA’s NGC vLLM image on both, run_cluster.sh inside tmux, and one vllm serve on the head. This is the path the playbook validates and the one this part’s lab follows.

Track X — AMD Ryzen AI Max+ 395Not supported

This course does not have a documented multi-node vLLM path for Ryzen AI Max+ machines, and vLLM's ROCm wheels for this chip have not been exercised by the validation pass. Part 19's llama.cpp RPC cluster is the supported route on Track X.

Read this lesson for the vocabulary, then build your cluster with llama.cpp RPC in Part 19, which runs on Vulkan and does not need any of this machinery. If you have a working ROCm vLLM build and want to try Ray across two machines, the procedure is the same; treat it as your own experiment and record what you find.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS, so there is no Ray-based vLLM cluster on Apple silicon.

Part 21 is your lesson. MLX distributed and exo solve the same problem over Thunderbolt 5, with a launcher that plays the role Ray plays here.

Track N — NVIDIA desktop or laptop

Secondary, and in two forms. Two desktops on the same network can run exactly this procedure with the generic vLLM image; what changes is the link, which is household Ethernet rather than a QSFP cable, so measure it in Part 18’s lab and expect pipeline parallel to beat tensor parallel across the pair. Two cards in one desktop are a single-node case and need no Ray at all; the next lesson is about that machine.

The options that decide a cluster

Option	What it decides
`--tensor-parallel-size`	Devices each layer is split across; puts a collective on every layer’s critical path
`--pipeline-parallel-size`	Devices the layers are divided between; one activation transfer per stage boundary
`--data-parallel-size`	Independent replicas, each with its own key-value cache
`--max-model-len`	Context length, and therefore the divisor in the maximum-concurrency line
`--gpu-memory-utilization`	Share of each device the engine may claim
`--max-num-seqs`	Sequences in flight, per data-parallel rank
`--enforce-eager`	Skips graph capture; faster startup, slower steady state
`--distributed-executor-backend` †	`ray` or `mp`; Ray is the documented default for multi-node
`--enable-expert-parallel` †	Expert layers use expert parallelism rather than tensor parallelism
`--data-parallel-size-local` †	How many of the global data-parallel ranks run on this node
`--headless` †	This node runs engines only, with no API server

† Confirmed in vLLM’s documentation on 2026-09-09; not yet in this course’s captured command reference, so it appears here rather than in a runnable block.

Understand the two schedulers in the deployment

Ray places processes and resources across machines; the inference engine schedules model execution and requests. A healthy Ray dashboard or node listing does not prove that the model’s parallel groups were formed correctly. Conversely, a model may load while one rank is assigned a route or resource you did not intend.

Record node addresses, accelerator counts, placement and engine rank logs. Check the expected tensor and pipeline parallel dimensions against the available devices. Keep checkpoint paths and versions consistent across ranks, and verify the chosen communication interfaces.

For a fault investigation, identify whether the failure occurs during cluster membership, model loading, collective initialisation or a request. A missing worker resource is different from an incompatible kernel and different again from an exhausted cache under load. Use a small model and short request before the target deployment. Treat the distributed control plane as an internal service with access limited to the lab network; an application API credential does not secure every orchestration port automatically.

Recap

A vLLM cluster is a Ray cluster with one vllm serve command on top of it. You start Ray on a head node, join a worker to the address the head printed, confirm with ray status and ray list nodes that both machines and both GPUs are in the pool, and then serve as though the machine were bigger. NVIDIA’s run_cluster.sh path wraps the same thing in containers, requires the shells to stay open, and needs tmux.

The parallel sizes are a topology decision, not a formula. vLLM’s ladder says tensor parallel inside a node and pipeline parallel across nodes, and NVIDIA’s Spark playbook sets tensor parallel across the pair instead, because a direct ConnectX-7 cable behaves more like an intra-node link than like a network. For mixture-of-experts models, data parallel attention with expert or tensor parallel experts is a separate mode, enabled by --enable-expert-parallel, straightforward on one node and a dependency exercise across two.

Which cable the traffic uses is decided by VLLM_HOST_IP, NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME and TP_SOCKET_IFNAME, and whether RDMA was actually used is decided by reading the NCCL trace for NET/IB/GDRDMA rather than NET/Socket. The startup log’s cache-size and maximum-concurrency lines tell you before any request arrives whether the arrangement can serve your workload. And the three failure modes are a worker that never joins, a shell that was closed, and a model that fits with no room left for its cache.

Sources for this lesson

8 verified · checked 2026-09-09

01vLLM — Parallelism and Scaling§ Distributed inference strategies; Multi-node deployment; Ray cluster setup with containers; Optimizing network communicationdocs.vllm.ai/en/latest/serving/parallelism_scaling.html2026-09-09
02vLLM — Data Parallel Deployment§ Internal load balancing; multi-nodedocs.vllm.ai/en/latest/serving/data_parallel_deployment.html2026-09-09
03vLLM — Expert Parallel Deployment§ Configuration; layer behavior; backend selectiondocs.vllm.ai/en/latest/serving/expert_parallel_deployment.html2026-09-09
04vLLM — vllm serve CLI reference§ Optionsdocs.vllm.ai/en/latest/cli/serve.html2026-09-09
05Ray documentation — Launching an On-Premise Cluster§ Start the Head Node; Start Worker Nodes; Troubleshootingdocs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/on-premises.html2026-09-09
06NCCL documentation — Environment Variables§ NCCL_SOCKET_IFNAME; NCCL_IB_HCA; NCCL_DEBUG; NCCL_IB_DISABLEdocs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html2026-09-09
07DGX Spark playbook — Serve LLMs with vLLM, Multi-node servingbuild.nvidia.com/spark/vllm/multi-node2026-09-09
08DGX Spark playbook — Serve LLMs with vLLM, Troubleshootingbuild.nvidia.com/spark/vllm/troubleshooting2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.

vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across

One command, two machines

Starting the cluster, two ways

Choosing the parallel sizes

Mixture-of-experts: data parallel with expert parallel

Pointing the traffic at the right cable

Reading the startup log

When it breaks

Track S — NVIDIA DGX Spark

Track X — AMD Ryzen AI Max+ 395Not supported

Track M — Apple siliconNot supported

Track N — NVIDIA desktop or laptop

The options that decide a cluster

Understand the two schedulers in the deployment

Recap

Check your understanding

Sources for this lesson