Lab: Serve a 400B-Class Model on Two DGX Sparks

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The versions, container tags and measurements each track was run with will be recorded here when the validation pass is done, and the pending tables below are where they will land.

Objective

By the end of this lab you will have made two DGX Sparks behave as one machine, served a 400B-class model across them with a single command, driven it with the load generator you built in Part 9, and written down four numbers: output token throughput, request throughput, time to first token and time per output token. You will then have served a 120B-class mixture-of-experts model on one of the two machines and measured the same four numbers, and you will be able to say which of the two you would actually use, with evidence.

That last part is the real objective. Running an enormous model on hardware you own is a satisfying afternoon. Deciding whether it was worth it is the skill.

⌂Single-machine path

The primary path needs two Sparks and a QSFP cable. If you have one machine, take the single-Spark path in the fallbacks below: you will serve gpt-oss-120b, measure it properly, and have the baseline that the two-machine comparison is measured against. Everything except the cluster construction is identical, and the notebook entry has the same shape.

Architecture

What you are building

• workerDGX Spark ARay head; the vllm serve process; the only HTTP endpoint
• workerDGX Spark BRay worker; no HTTP server of its own
• clientYour workstationRuns the load generator, or run it on node A
• routerHouse networkSSH, model downloads, client requests

• DGX Spark A ⟷ connected to DGX Spark Bdirect QSFP cable: collectives, RoCE, its own subnet
• DGX Spark A ⟷ connected to House networkmanagement interface; the served port
• DGX Spark B ⟷ connected to House networkmanagement interface only
• Your workstation ⟷ connected to House networkrequests to node A

One model, two machines, one endpoint. Node B never answers a request; it holds half the model. Every measurement in this lab is taken from a client talking to node A, which is what a user would experience.

The memory picture is why this is hard, and it is worth looking at before you start.

Llama 3.1 405B at four bits across the pair — estimate from the course model reference

Weights, four-bit, split across both machines

205 GB

Free

51 GB

Total

256 GB

The pair has 256 GB. The four-bit weight figure is the estimate in the course model reference; an AWQ INT4 build can differ from it. Everything not shaded has to cover two operating systems, two containers, the engine's activations and captured graphs on both nodes, and the whole key-value cache. NVIDIA's own playbook step for this model asks for a maximum model length of 64 tokens and one sequence at a time, and warns on the same page that the model has insufficient memory headroom for production use. The startup log's cache line is the arbiter, and recording it is one of this lab's tasks.

gpt-oss-120b at MXFP4 on one Spark — estimate from the course model reference

Weights, native MXFP4

63 GB

Key-value cache, 32,768 tokens

2.4 GB

Free

62.6 GB

Total

128 GB

The same reference, one machine, a mixture-of-experts model that activates about 5B of its 117B parameters per token. Weights and a full-length cache leave most of the machine free. Hold the two pictures side by side while the lab runs: they are the argument the final comparison settles.

Requirements

Every track needs the lab notebook from Part 1, Part 9’s load-test.py, and about ninety minutes, of which the model download is unattended and can be much longer than the rest put together. A four-bit 405B checkpoint is a very large download; start it before you read the tasks.

Track S — NVIDIA DGX Spark

Primary path. Two DGX Sparks, 128 GB each, one approved QSFP cable between the same physical port on each machine, and the connect-two-Sparks playbook already completed: addresses on the QSFP interfaces, matching usernames, passwordless SSH in both directions. Docker usable without sudo on both nodes. The NGC vLLM image for this hardware pulled on both nodes at the same tag. Roughly 250 GB of free disk for the 405B checkpoint plus 70 GB for the comparison model, on each node or on storage both can read.

Attended time is about forty-five minutes. Downloads are unattended and dominate the wall clock.

Track X — AMD Ryzen AI Max+ 395Not supported

There is no documented multi-node vLLM path for Ryzen AI Max+ machines, and the DGX Spark playbooks this lab follows are specific to GB10 hardware.

Go to Part 19, Lab: Run a Model Bigger Than Any One Machine, which builds the equivalent cluster with llama.cpp RPC and runs on Vulkan. It measures the same four numbers, so its notebook line is directly comparable with anything a Track S reader records here.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS, so neither the Ray cluster nor the container images in this lab exist for Apple silicon.

Go to Part 21, Lab: A Two-Mac Cluster over Thunderbolt 5, which is the Apple equivalent of this lab: two machines, one model, one endpoint, the same four measurements.

Track N — NVIDIA desktop or laptop

Fallback path, two cards in one desktop. Two GPUs of 24 GB each in one machine, both visible to nvidia-smi, vLLM installed as in Part 9, and about 45 GB of disk for a 70B-class AWQ checkpoint. No Ray, no cluster, no cable. You will run the same load test against a tensor split and a pipeline split and compare them against each other and against the link width your slots negotiated.

Two desktops on the house network can also follow the primary path with a generic vLLM image instead of the NGC one. Expect the link to be the limit, and read Part 18’s measurement of it before choosing tensor parallel.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-20-nvidia-clusters"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-20-nvidia-clusters and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Fill in the settings file

Every script in this lab reads one file. Copy it, fill in the addresses your QSFP interfaces actually carry, and keep it out of version control.

Fragment — not complete on its own

env-example.txt

# Purpose: every setting the Part 20 scripts read. Copy this file to `.env` beside the
#          scripts and fill in the empty lines for your own machines. Nothing here is a
#          secret except the Hugging Face token, which you generate rather than copy, and
#          which the scripts read from the environment and never write anywhere.
# Platform: all (the Spark values matter on Track S; the two-card values on Track N)
# Minimum memory: 128 GB per node for the two-Spark path; 24 GB per GPU for the two-card path
# Assumes: `cp env-example.txt .env`, then an editor. Every script loads it with
#          `set -a; . ./.env; set +a` when the file is present, so a value set in your
#          shell always wins over a value in the file.

# ------------------------------------------------------------------- the two machines
# Addresses on the direct QSFP link, NOT on the house network. Take them from
# `ip addr show <interface>` after the connect-two-Sparks playbook has assigned them.
# The playbook's example puts the first port's pair on a 192.168.100.0/24 subnet and the
# second port's pair on 192.168.101.0/24, one host address per node in each.
# This node's own address on the link. Different on each machine.
NODE_SELF_ADDR=

# The head node's address on the link. The SAME value on both machines: on the head it
# equals NODE_SELF_ADDR, on the worker it is the head's address.
NODE_HEAD_ADDR=

# The other machine's address on the link, used by prepare-spark-pair.sh for its
# reachability and SSH checks. Different on each machine.
NODE_PEER_ADDR=

# The login name that exists on BOTH machines. The playbooks require it to match.
CLUSTER_USER=

# ------------------------------------------------------------------ the link itself
# The QSFP interface that came up, from `ibdev2netdev`. The playbook's validated example
# on a DGX Spark is enp1s0f1np1; confirm yours rather than copying this one.
CX7_IFNAME=enp1s0f1np1

# The RoCE device paired with that interface in the same ibdev2netdev output, without the
# trailing port number. Used as the NCCL_IB_HCA filter. Leave empty to let NCCL choose.
CX7_ROCE_DEV=

# The management interface you SSH in on, used only for the NCCL test suite's bootstrap,
# which NVIDIA's NCCL playbook launches over the management network.
MGMT_IFNAME=

# ------------------------------------------------------------------------ containers
# The NGC vLLM image for your hardware. Take the current tag from the DGX Spark vLLM
# playbook's Instructions tab; it changes, and both nodes must use the same one.
VLLM_IMAGE=

# The TensorRT-LLM release image, if you take the playbook path in the optional task.
# Three NVIDIA pages named three different tags on 2026-09-09: check NGC.
TRTLLM_IMAGE=

# ---------------------------------------------------------------------------- models
# Where models live on each node. The same path on both machines, as vLLM's multi-node
# documentation requires, or a filesystem both can read.
HF_HOME=

# The very large model served across the pair. The vLLM multi-node playbook's optional
# 405B step uses hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4, published under the
# Llama 3.1 Community licence: read the model card before using it.
PAIR_MODEL=hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4

# The single-machine comparison. gpt-oss-120b is Apache-2.0 with native MXFP4 weights.
SINGLE_MODEL=openai/gpt-oss-120b

# The two-card desktop fallback. hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 is
# the 70B-class option and is tight on 2x24 GB; Qwen/Qwen3-32B-AWQ is the comfortable one.
TWO_CARD_MODEL=hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4

# The name clients send in the "model" field. Keep it stable across every path so the
# load generator's command line does not change between runs.
SERVED_NAME=local-cluster

# ------------------------------------------------------------------------- serving
SERVE_HOST=127.0.0.1
SERVE_PORT=8000

# Context length. The vLLM playbook's 405B step uses 64 because there is no headroom;
# raise it for the single-machine and two-card paths and record what you used.
CTX=4096

# Fraction of each device the engine may claim, and sequences in flight.
MEM_FRACTION=0.90
MAX_SEQS=4

# --------------------------------------------------------------------------- secrets
# Gated models need a Hugging Face token. Export it in your shell rather than writing it
# here if you can; if you do put it here, keep .env out of version control.
HF_TOKEN=

# The load generator reads an API key, if the server needs one, from the variable named
# here. It is never written to the notebook.
LOADTEST_API_KEY_ENV=LOADTEST_API_KEY

Download env-example.txt92 lines

RunnableTrack S · DGX Spark

make the settings file

cp env-example.txt .env

NODE_SELF_ADDR differs on each machine and NODE_HEAD_ADDR is the same on both. That asymmetry is the single most common configuration error in this lab, and the scripts refuse to run when it looks wrong.

2. Check both nodes and the link

Run this on each node. It changes nothing: it reads the interfaces, checks the address, pings the peer, tries SSH, and writes the interface environment that every later script sources.

RunnableTrack S · DGX Spark

prepare-spark-pair.sh

#!/usr/bin/env bash
# Purpose: check one node of a DGX Spark pair before any engine is started - the QSFP
#          interface is up, it has the address you configured, the peer answers, SSH works
#          without a password - and write the interface environment the vLLM and
#          TensorRT-LLM paths both need into cluster-env.sh for the other scripts to source
# Platform: spark (run once on each of the two nodes); the checks are read-only
# Minimum memory: 128 GB per node for the models this lab serves; this script needs none
# Assumes: the NVIDIA connect-two-Sparks playbook has already assigned addresses to the
#          QSFP interfaces and distributed SSH keys; ibdev2netdev, ip and ssh on PATH; a
#          .env file beside this script, copied from env-example.txt and filled in.
#          Changes nothing on the machine: it prints the netplan stanza it would write
#          rather than writing it.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

CX7_IFNAME="${CX7_IFNAME:-}"
CX7_ROCE_DEV="${CX7_ROCE_DEV:-}"
NODE_SELF_ADDR="${NODE_SELF_ADDR:-}"
NODE_PEER_ADDR="${NODE_PEER_ADDR:-}"
CLUSTER_USER="${CLUSTER_USER:-$(id -un)}"
ENV_OUT="${ENV_OUT:-${HERE}/cluster-env.sh}"

fail() { printf '%s\n' "$*" >&2; exit 1; }
note() { printf '    %s\n' "$*"; }

[ -n "$CX7_IFNAME" ] || fail "CX7_IFNAME is not set. Run ibdev2netdev, pick the interface that reports (Up), and put it in .env."
[ -n "$NODE_SELF_ADDR" ] || fail "NODE_SELF_ADDR is not set. It is this machine's address on the QSFP link, from 'ip addr show ${CX7_IFNAME}'."
[ -n "$NODE_PEER_ADDR" ] || fail "NODE_PEER_ADDR is not set. It is the other machine's address on the same link."

for tool in ip ssh; do
    command -v "$tool" >/dev/null 2>&1 || fail "${tool} is not on PATH."
done

printf '==> 1. QSFP ports and their RoCE devices\n'
if command -v ibdev2netdev >/dev/null 2>&1; then
    ibdev2netdev | sed 's/^/    /'
    if ! ibdev2netdev | grep -q "${CX7_IFNAME}.*(Up)"; then
        fail "${CX7_IFNAME} is not reported as (Up). Check the cable, confirm both nodes use the same physical port, and reboot before looking at software."
    fi
    note "OK: ${CX7_IFNAME} is up."
else
    note "ibdev2netdev is not on PATH; skipping the RoCE pairing check."
    note "On DGX OS it ships with the ConnectX-7 tooling. Without it you cannot see which"
    note "RoCE device belongs to which Ethernet interface, which NCCL_IB_HCA needs."
fi

printf '\n==> 2. The address on this node\n'
if ip -4 addr show "$CX7_IFNAME" | grep -qw "$NODE_SELF_ADDR"; then
    note "OK: ${CX7_IFNAME} carries the address in NODE_SELF_ADDR."
else
    printf '    %s\n' "${CX7_IFNAME} does not carry NODE_SELF_ADDR." >&2
    printf '    %s\n' "Either .env is wrong, or the address was assigned with 'ip addr add' and lost" >&2
    printf '    %s\n' "on the last reboot. The playbook's netplan option is the one that persists." >&2
    printf '\n    A netplan stanza for this node would look like this. Review it, write it to\n'
    printf '    /etc/netplan/40-cx7.yaml yourself, chmod 600 it, and run "sudo netplan apply":\n\n'
    cat <<STANZA
    network:
      version: 2
      ethernets:
        ${CX7_IFNAME}:
          dhcp4: no
          addresses:
            - ${NODE_SELF_ADDR}/24
STANZA
    exit 1
fi

printf '\n==> 3. The peer answers on the link\n'
if command -v ping >/dev/null 2>&1 && ping -c 2 -W 2 "$NODE_PEER_ADDR" >/dev/null 2>&1; then
    note "OK: the peer replies on the QSFP subnet."
else
    fail "The peer does not reply on the QSFP subnet. Check the cable and both netplan files before anything else."
fi

printf '\n==> 4. Passwordless SSH, this node to the peer\n'
if ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 \
       "${CLUSTER_USER}"@"${NODE_PEER_ADDR}" hostname >/dev/null 2>&1; then
    note "OK: SSH to the peer as ${CLUSTER_USER} needs no password."
else
    printf '    %s\n' "SSH to the peer failed or asked for a password." >&2
    printf '    %s\n' "Run 'mkdir -p ~/.ssh && chmod 700 ~/.ssh' on both nodes, then NVIDIA's" >&2
    printf '    %s\n' "discover-sparks script, or ssh-copy-id from each node to the other." >&2
    printf '    %s\n' "Both machines must also share the same username; this node is using ${CLUSTER_USER}." >&2
    exit 1
fi

printf '\n==> 5. Writing the interface environment to %s\n' "$ENV_OUT"
{
    printf '# Generated by prepare-spark-pair.sh. Source it before launching an engine.\n'
    printf '# Every variable here names the same interface, which is what stops a collective\n'
    printf '# from quietly running over the house network instead of the QSFP cable.\n'
    printf 'export VLLM_HOST_IP="%s"\n' "$NODE_SELF_ADDR"
    printf 'export MASTER_ADDR="%s"\n' "${NODE_HEAD_ADDR:-$NODE_SELF_ADDR}"
    printf 'export NCCL_SOCKET_IFNAME="%s"\n' "$CX7_IFNAME"
    printf 'export GLOO_SOCKET_IFNAME="%s"\n' "$CX7_IFNAME"
    printf 'export TP_SOCKET_IFNAME="%s"\n' "$CX7_IFNAME"
    printf 'export UCX_NET_DEVICES="%s"\n' "$CX7_IFNAME"
    printf 'export OMPI_MCA_btl_tcp_if_include="%s"\n' "$CX7_IFNAME"
    if [ -n "$CX7_ROCE_DEV" ]; then
        printf 'export NCCL_IB_HCA="=%s"\n' "$CX7_ROCE_DEV"
    else
        printf '# NCCL_IB_HCA is unset: NCCL will pick a RoCE device itself. Set CX7_ROCE_DEV\n'
        printf '# in .env to the roce* name ibdev2netdev pairs with %s to pin it.\n' "$CX7_IFNAME"
    fi
} > "$ENV_OUT"
sed 's/^/    /' "$ENV_OUT"

cat <<'DONE'

==> This node is ready.
    Run this script on the other node too, then start the Ray head on the node whose
    NODE_SELF_ADDR equals NODE_HEAD_ADDR.

    To see which transport NCCL actually chose once an engine is running, set
    NCCL_DEBUG=TRACE and look for "NET/IB/GDRDMA" (RDMA, efficient) rather than
    "NET/Socket" (plain TCP, not efficient for cross-node tensor parallelism).
DONE

Download prepare-spark-pair.sh124 lines

RunnableTrack S · DGX Spark

verify this node, then repeat on the other

bash prepare-spark-pair.sh

Output — what you should see

==> 1. QSFP ports and their RoCE devices
    rocep1s0f0   port 1 ==> enp1s0f0np0   (Down)
    rocep1s0f1   port 1 ==> enp1s0f1np1   (Up)
    ...
    OK: enp1s0f1np1 is up.

==> 2. The address on this node
    OK: enp1s0f1np1 carries the address in NODE_SELF_ADDR.

==> 3. The peer answers on the link
    OK: the peer replies on the QSFP subnet.

==> 4. Passwordless SSH, this node to the peer
    OK: SSH to the peer as <your user> needs no password.

==> 5. Writing the interface environment to ./cluster-env.sh

=Expected result

All five checks pass on both nodes before you go further. Every minute spent here saves ten later: a distributed engine that fails because an interface name is wrong produces a stack trace about NCCL, not about netplan.

3. Fetch the Ray cluster helper

vLLM ships the script that starts a container and initialises Ray in it, as examples/ray_serving/run_cluster.sh in the vLLM repository. NVIDIA’s playbook pins it to a specific commit and patches it to install a newer Ray inside the container before starting. Download it to run-cluster.sh beside the lab scripts, on both nodes, taking the URL and the patch from the playbook at the time you run this rather than from here: it is the piece of this procedure most likely to have moved, and the two nodes must end up with the same copy.

RunnableTrack S · DGX Spark

check the helper is where the scripts expect it

ls -l run-cluster.sh && head -n 5 run-cluster.sh

4. Start the cluster

Both of these hold their shell open. Run each inside tmux or screen, because the helper stops its container when the shell exits, and vLLM’s documentation is explicit that closing any shell terminates the cluster.

RunnableTrack S · DGX Spark

start-ray-head.sh

#!/usr/bin/env bash
# Purpose: start the Ray head node of a two-machine vLLM cluster inside the NGC vLLM
#          container, with every network variable pointed at the QSFP interface so the
#          collectives use the direct cable rather than the house network
# Platform: spark (the head node of the pair); works on nvidia with a generic vLLM image
# Minimum memory: 128 GB on this node for the models this lab serves
# Assumes: prepare-spark-pair.sh has passed on both nodes and written cluster-env.sh;
#          vLLM's examples/ray_serving/run_cluster.sh has been downloaded beside this
#          script; docker works without sudo; a .env filled in from env-example.txt.
#          This script does not return: it holds the container open, so run it inside
#          tmux or screen. Closing the shell tears the cluster down.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi
if [ -f "${HERE}/cluster-env.sh" ]; then
    # shellcheck disable=SC1091  # written by prepare-spark-pair.sh
    . "${HERE}/cluster-env.sh"
fi

RUN_CLUSTER="${RUN_CLUSTER:-${HERE}/run-cluster.sh}"
VLLM_IMAGE="${VLLM_IMAGE:-}"
NODE_SELF_ADDR="${NODE_SELF_ADDR:-}"
NODE_HEAD_ADDR="${NODE_HEAD_ADDR:-$NODE_SELF_ADDR}"
CX7_IFNAME="${CX7_IFNAME:-}"
HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v docker >/dev/null 2>&1 || fail "docker is not on PATH."
docker ps >/dev/null 2>&1 || fail "docker ps failed. Add yourself to the docker group ('sudo usermod -aG docker \$USER' then 'newgrp docker') on both nodes."
[ -f "$RUN_CLUSTER" ] || fail "Ray cluster helper not found at ${RUN_CLUSTER}. Download vLLM's examples/ray_serving/run_cluster.sh to that path, or set RUN_CLUSTER."
[ -n "$VLLM_IMAGE" ] || fail "VLLM_IMAGE is not set. Take the tag for your hardware from the DGX Spark vLLM playbook and put it in .env; both nodes must use the same tag."
[ -n "$NODE_SELF_ADDR" ] || fail "NODE_SELF_ADDR is not set."
[ -n "$CX7_IFNAME" ] || fail "CX7_IFNAME is not set."

if [ "$NODE_SELF_ADDR" != "$NODE_HEAD_ADDR" ]; then
    fail "This node's NODE_SELF_ADDR is not NODE_HEAD_ADDR, so it is a worker. Run start-ray-worker.sh here instead."
fi

PASSTHROUGH=(
    -e "VLLM_HOST_IP=${NODE_SELF_ADDR}"
    -e "MASTER_ADDR=${NODE_HEAD_ADDR}"
    -e "NCCL_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "GLOO_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "TP_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "UCX_NET_DEVICES=${CX7_IFNAME}"
    -e "OMPI_MCA_btl_tcp_if_include=${CX7_IFNAME}"
    -e "RAY_memory_monitor_refresh_ms=0"
)
if [ -n "${NCCL_IB_HCA:-}" ]; then
    PASSTHROUGH+=(-e "NCCL_IB_HCA=${NCCL_IB_HCA}")
fi
if [ -n "${NCCL_DEBUG:-}" ]; then
    PASSTHROUGH+=(-e "NCCL_DEBUG=${NCCL_DEBUG}")
fi

cat <<INFO
==> Ray head node
    image             ${VLLM_IMAGE}
    this node         ${NODE_SELF_ADDR} on ${CX7_IFNAME}
    model cache       ${HF_HOME}
    passthrough       ${#PASSTHROUGH[@]} arguments, all naming ${CX7_IFNAME}

    This shell now holds the cluster open. Leave it running, start the worker on the
    other node, and do everything else from a second terminal. Confirm the pool with:

        docker exec \$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+\$') ray status

INFO

exec bash "$RUN_CLUSTER" "$VLLM_IMAGE" "$NODE_HEAD_ADDR" --head "$HF_HOME" "${PASSTHROUGH[@]}"

Download start-ray-head.sh77 lines

RunnableTrack S · DGX Spark

start-ray-worker.sh

#!/usr/bin/env bash
# Purpose: join the second DGX Spark to the Ray cluster started by start-ray-head.sh,
#          inside the same container image and with the same interface variables, so both
#          ranks agree about which cable carries the collectives
# Platform: spark (the worker node of the pair); works on nvidia with a generic vLLM image
# Minimum memory: 128 GB on this node for the models this lab serves
# Assumes: start-ray-head.sh is already running on the head node and still has its shell
#          open; prepare-spark-pair.sh has passed here; the SAME container image tag as
#          the head; vLLM's run_cluster.sh downloaded beside this script; docker without
#          sudo. This script does not return, so run it inside tmux or screen.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi
if [ -f "${HERE}/cluster-env.sh" ]; then
    # shellcheck disable=SC1091  # written by prepare-spark-pair.sh
    . "${HERE}/cluster-env.sh"
fi

RUN_CLUSTER="${RUN_CLUSTER:-${HERE}/run-cluster.sh}"
VLLM_IMAGE="${VLLM_IMAGE:-}"
NODE_SELF_ADDR="${NODE_SELF_ADDR:-}"
NODE_HEAD_ADDR="${NODE_HEAD_ADDR:-}"
CX7_IFNAME="${CX7_IFNAME:-}"
HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v docker >/dev/null 2>&1 || fail "docker is not on PATH."
docker ps >/dev/null 2>&1 || fail "docker ps failed. Add yourself to the docker group on this node too."
[ -f "$RUN_CLUSTER" ] || fail "Ray cluster helper not found at ${RUN_CLUSTER}. Download the same run_cluster.sh the head node uses."
[ -n "$VLLM_IMAGE" ] || fail "VLLM_IMAGE is not set. It must be the same tag the head node pulled."
[ -n "$NODE_SELF_ADDR" ] || fail "NODE_SELF_ADDR is not set. On a worker it is THIS machine's address, not the head's."
[ -n "$NODE_HEAD_ADDR" ] || fail "NODE_HEAD_ADDR is not set. It is the head node's address on the QSFP link."
[ -n "$CX7_IFNAME" ] || fail "CX7_IFNAME is not set."

if [ "$NODE_SELF_ADDR" = "$NODE_HEAD_ADDR" ]; then
    fail "NODE_SELF_ADDR equals NODE_HEAD_ADDR, so this is the head node. Run start-ray-head.sh here instead."
fi

PASSTHROUGH=(
    -e "VLLM_HOST_IP=${NODE_SELF_ADDR}"
    -e "MASTER_ADDR=${NODE_HEAD_ADDR}"
    -e "NCCL_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "GLOO_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "TP_SOCKET_IFNAME=${CX7_IFNAME}"
    -e "UCX_NET_DEVICES=${CX7_IFNAME}"
    -e "OMPI_MCA_btl_tcp_if_include=${CX7_IFNAME}"
    -e "RAY_memory_monitor_refresh_ms=0"
)
if [ -n "${NCCL_IB_HCA:-}" ]; then
    PASSTHROUGH+=(-e "NCCL_IB_HCA=${NCCL_IB_HCA}")
fi
if [ -n "${NCCL_DEBUG:-}" ]; then
    PASSTHROUGH+=(-e "NCCL_DEBUG=${NCCL_DEBUG}")
fi

cat <<INFO
==> Ray worker node
    image             ${VLLM_IMAGE}
    this node         ${NODE_SELF_ADDR} on ${CX7_IFNAME}
    joining head at   ${NODE_HEAD_ADDR}
    model cache       ${HF_HOME}

    VLLM_HOST_IP is this node's own address, not the head's: vLLM's documentation says it
    is unique per worker, and getting it wrong is the commonest reason a worker appears to
    join and then produces nothing.

    Leave this shell running. If Ray reports "Unable to connect to GCS", the head is not
    running, the address is wrong, the two nodes have different Ray versions, or a
    firewall is in the way.

INFO

exec bash "$RUN_CLUSTER" "$VLLM_IMAGE" "$NODE_HEAD_ADDR" --worker "$HF_HOME" "${PASSTHROUGH[@]}"

Download start-ray-worker.sh80 lines

RunnableTrack S · DGX Spark

on node A, in its own tmux window

bash start-ray-head.sh

RunnableTrack S · DGX Spark

on node B, in its own tmux window

bash start-ray-worker.sh

Then, from a third terminal on node A, confirm the pool:

RunnableTrack S · DGX Spark

does Ray see both machines and both GPUs?

docker exec "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" ray status

You are looking for two nodes and two GPUs. One node means the worker did not join, and the troubleshooting section below has the four documented reasons.

5. Download the model into the cluster

vLLM’s documentation asks you to “download the model on every node to the same path, or store the model on a distributed file system accessible by all nodes.” The container mounts the cache directory from HF_HOME, so downloading inside the container on each node puts it in the right place.

RunnableTrack S · DGX Spark

download inside the container, on each node

docker exec -it "$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')" \
    bash -c 'hf download hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4'

This is the unattended hour. hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 is the repository NVIDIA’s playbook names for its optional 405B step. Its card describes the conversion as being made “from FP16 down to INT4 using the GEMM kernels performing zero-point quantization with a group size of 128”, and names the licence as llama3.1. It is a conversion of Meta’s Llama 3.1 405B Instruct; read the Llama 3.1 Community licence terms on the model card before using it for anything beyond this lab.

⛨Security

A gated model needs a Hugging Face token. Export it in the shell you run the download from, never in a committed file, and never in the notebook. If a download hangs or refuses, NVIDIA’s troubleshooting table names the causes as a missing or invalid token, an unaccepted gated-model agreement, or a network problem, in that order.

6. Serve the model across the pair

One command, on node A, using both machines.

RunnableTrack S · DGX Spark

serve-across-pair.sh

#!/usr/bin/env bash
# Purpose: serve one model across both DGX Sparks with a single vllm serve command inside
#          the Ray head container, printing the configuration first so the run is
#          reproducible from the terminal log, and reminding you which two startup lines
#          to copy into the notebook
# Platform: spark (run on the head node only, after both Ray shells are up)
# Minimum memory: 128 GB per node; the pair's 256 GB is the ceiling for weights plus cache
# Assumes: start-ray-head.sh and start-ray-worker.sh are both running and "ray status"
#          reports two nodes; the model is already downloaded into HF_HOME on both nodes,
#          or the cache is shared; a .env filled in from env-example.txt. Nothing else is
#          listening on SERVE_PORT.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${PAIR_MODEL:-}"
SERVED_NAME="${SERVED_NAME:-local-cluster}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
SERVE_PORT="${SERVE_PORT:-8000}"
CTX="${CTX:-4096}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
MAX_SEQS="${MAX_SEQS:-4}"
TP_SIZE="${TP_SIZE:-2}"
PP_SIZE="${PP_SIZE:-1}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v docker >/dev/null 2>&1 || fail "docker is not on PATH."
[ -n "$MODEL" ] || fail "PAIR_MODEL is not set in .env."

CONTAINER="${VLLM_CONTAINER:-$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$' | head -n 1)}"
[ -n "$CONTAINER" ] || fail "No Ray container found. Is start-ray-head.sh still running in its own shell?"

printf '==> Ray sees:\n'
docker exec "$CONTAINER" ray status | sed 's/^/    /'
printf '\n'

if [ "$TP_SIZE" -ne 1 ] && [ "$PP_SIZE" -ne 1 ]; then
    printf '    NOTE: both parallel sizes are above one. On a pair of one-GPU machines that\n'
    printf '          asks for more devices than exist. Set one of TP_SIZE or PP_SIZE to 1.\n\n'
fi

cat <<INFO
==> vLLM across the pair
    container         ${CONTAINER}
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${SERVE_PORT}
    tensor parallel   ${TP_SIZE}
    pipeline parallel ${PP_SIZE}
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}

    Two lines in the startup log are the ones to record. One reports the GPU KV cache size
    in tokens, summed across the cluster. The other reports the maximum concurrency that
    implies at your context length. If the second is below the concurrency you intend to
    serve, stop now and change something: it will not improve when requests arrive.

    Startup on a very large model is slow, and most of it is loading weights. Watch for
    "Application startup complete."

INFO

EXEC_ENV=()
if [ -n "${NCCL_DEBUG:-}" ]; then
    EXEC_ENV+=(-e "NCCL_DEBUG=${NCCL_DEBUG}")
fi

exec docker exec -it "${EXEC_ENV[@]+"${EXEC_ENV[@]}"}" "$CONTAINER" \
    vllm serve "$MODEL" \
        --host "$SERVE_HOST" \
        --port "$SERVE_PORT" \
        --served-model-name "$SERVED_NAME" \
        --tensor-parallel-size "$TP_SIZE" \
        --pipeline-parallel-size "$PP_SIZE" \
        --max-model-len "$CTX" \
        --gpu-memory-utilization "$MEM_FRACTION" \
        --max-num-seqs "$MAX_SEQS"

Download serve-across-pair.sh85 lines

RunnableTrack S · DGX Spark

serve across both machines

bash serve-across-pair.sh

The script prints ray status first, then the configuration, then hands over to vLLM. Startup on a model this size is slow and almost all of it is reading weights. Two lines in the log are the ones this lab asks you to record:

Output — what you should see

INFO ... [kv_cache_utils.py:775] GPU KV cache size: <N> tokens
INFO ... [kv_cache_utils.py:779] Maximum concurrency for <CTX> tokens per request: <M>x

The first is the whole cluster’s cache in tokens. The second is the first divided by your context length. Write both into the notebook now, with the context length you asked for, before you run anything against the server.

✕Common mistake

If the maximum concurrency line reports a number below one, the cluster cannot hold even a single sequence at the context you asked for, and the server will fail or preempt on the first request. Lower CTX in .env and restart. NVIDIA’s own 405B example sets a maximum model length of 64 tokens for exactly this reason, and a maximum of one sequence with it.

Confirm it answers before measuring it:

RunnableTrack S · DGX Spark

one request, from node A

curl -s http://127.0.0.1:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "local-cluster", "prompt": "The three limits on a home cluster are", "max_tokens": 32, "temperature": 0}'

7. Check that the collectives used the cable

This is the step that separates a working pair from a disappointing one, and it costs one restart of the whole cluster. The variable has to reach the workers on both nodes, not only the process you type the command in, so set it in the shell before each of the three scripts: the head, the worker, then the serve command. All three pass it through when it is present.

RunnableTrack S · DGX Spark

on node A, then the same on node B for the worker

NCCL_DEBUG=TRACE bash start-ray-head.sh

RunnableTrack S · DGX Spark

then serve, with tracing on the driver too

NCCL_DEBUG=TRACE bash serve-across-pair.sh

vLLM’s documentation gives the reading. [send] via NET/IB/GDRDMA in the log means NCCL is using InfiniBand with GPUDirect RDMA, which it calls efficient. [send] via NET/Socket means a raw TCP socket, which it calls not efficient for cross-node tensor parallelism. Record which one you saw. If it is the socket, the interface variables from task 2 did not reach the container, and there is no point measuring anything until they do.

8. Measure the pair

Part 9’s load generator is the instrument. The wrapper below fills in the base URL, the served name and the notebook path from .env so that every run in this lab is comparable.

RunnableAll tracks

load-test-pair.sh

#!/usr/bin/env bash
# Purpose: run Part 9's load generator against whichever endpoint this lab has just
#          started, with the base URL, model name and notebook path filled in from .env, so
#          that the pair, the single machine and the two-card desktop are all measured by
#          the same tool with the same settings and produce comparable notebook lines
# Platform: all (it drives an HTTP endpoint; it can run on a third machine on the network)
# Minimum memory: none on the machine running this; the server needs the lab's floor
# Assumes: Part 9's load-test.py is on this machine and LOAD_TEST points at it; a server is
#          already answering on SERVE_HOST:SERVE_PORT with the name in SERVED_NAME; python3
#          on PATH; a .env filled in from env-example.txt. Pass a label as the first
#          argument: it is the tag written into the notebook line.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

LABEL="${1:-}"
LOAD_TEST="${LOAD_TEST:-${HERE}/../part-09-vllm-and-sglang/load-test.py}"
SERVED_NAME="${SERVED_NAME:-local-cluster}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
SERVE_PORT="${SERVE_PORT:-8000}"
CONCURRENCY="${CONCURRENCY:-1,2,4}"
REQUESTS="${REQUESTS:-16}"
MAX_TOKENS="${MAX_TOKENS:-128}"
PROMPT_SET="${PROMPT_SET:-mixed}"
ENGINE="${ENGINE:-vllm}"
ENGINE_VERSION="${ENGINE_VERSION:-unknown}"
LABBOOK="${LABBOOK:-labbook.md}"
LOADTEST_API_KEY_ENV="${LOADTEST_API_KEY_ENV:-LOADTEST_API_KEY}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

[ -n "$LABEL" ] || fail "Usage: $(basename "$0") <label>   e.g. 'pair-tp2' or 'single-spark-120b'"
command -v python3 >/dev/null 2>&1 || fail "python3 is not on PATH."
[ -f "$LOAD_TEST" ] || fail "Part 9's load generator was not found at ${LOAD_TEST}. Copy load-test.py from the Part 9 lab files, or set LOAD_TEST to its path."

BASE_URL="http://${SERVE_HOST}:${SERVE_PORT}/v1"

if ! curl -sf --max-time 10 "${BASE_URL}/models" >/dev/null 2>&1; then
    fail "Nothing is answering at ${BASE_URL}/models. Start a server first, and wait for 'Application startup complete.'"
fi

cat <<INFO
==> Load test
    label             ${LABEL}
    endpoint          ${BASE_URL}
    model name        ${SERVED_NAME}
    concurrency       ${CONCURRENCY}
    requests / level  ${REQUESTS}
    max tokens        ${MAX_TOKENS}
    prompt set        ${PROMPT_SET}
    notebook          ${LABBOOK}

    On a very large model with almost no cache headroom, start at concurrency 1 and raise
    it only if the server survives. A refused or preempted request is a result too: record
    the concurrency at which it started happening.

INFO

exec python3 "$LOAD_TEST" \
    --base-url "$BASE_URL" \
    --model "$SERVED_NAME" \
    --concurrency "$CONCURRENCY" \
    --requests "$REQUESTS" \
    --max-tokens "$MAX_TOKENS" \
    --prompt-set "$PROMPT_SET" \
    --label "$LABEL" \
    --engine "$ENGINE" \
    --engine-version "$ENGINE_VERSION" \
    --api-key-env "$LOADTEST_API_KEY_ENV" \
    --labbook "$LABBOOK"

Download load-test-pair.sh76 lines

RunnableAll tracks

measure the pair

bash load-test-pair.sh pair-405b-tp2

Start at low concurrency. On a model with almost no cache headroom, the interesting result is often the concurrency at which requests begin to be preempted, and that is a number worth recording rather than an error to avoid.

9. Measure one machine, for comparison

Stop the cluster on node A and node B, then serve the comparison model on one machine. It is the same served name and the same port, so the load-test command line is identical apart from its label.

RunnableTrack S · DGX Spark

serve-single-spark.sh

#!/usr/bin/env bash
# Purpose: serve the 120B-class mixture-of-experts comparison model on ONE DGX Spark, with
#          the same served name and port as the pair, so the same load-test command line
#          measures both and the two notebook lines are directly comparable
# Platform: spark (one machine; this is also the fallback path for readers with one Spark)
# Minimum memory: 128 GB of unified memory
# Assumes: vllm on PATH in an activated environment, or set USE_CONTAINER=1 and VLLM_IMAGE
#          to run it inside the NGC image instead; the model already downloaded into
#          HF_HOME; nothing else listening on SERVE_PORT. No Ray cluster is involved.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${SINGLE_MODEL:-openai/gpt-oss-120b}"
SERVED_NAME="${SERVED_NAME:-local-cluster}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
SERVE_PORT="${SERVE_PORT:-8000}"
CTX="${CTX:-4096}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
MAX_SEQS="${MAX_SEQS:-4}"
USE_CONTAINER="${USE_CONTAINER:-0}"
HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

cat <<INFO
==> vLLM on one Spark, the single-machine comparison
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${SERVE_PORT}
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}
    container         ${USE_CONTAINER}

    Record the same two startup lines you recorded for the pair: the KV cache size in
    tokens and the maximum concurrency at this context length. On one machine the cache is
    smaller and the model is smaller, and which way those two facts push the comparison is
    the question this lab exists to answer.

INFO

if [ "$USE_CONTAINER" = "1" ]; then
    command -v docker >/dev/null 2>&1 || fail "docker is not on PATH."
    [ -n "${VLLM_IMAGE:-}" ] || fail "USE_CONTAINER=1 needs VLLM_IMAGE set in .env."
    exec docker run --rm -it \
        --gpus all \
        --ipc=host \
        --network host \
        -v "${HF_HOME}:/root/.cache/huggingface" \
        "$VLLM_IMAGE" \
        vllm serve "$MODEL" \
            --host "$SERVE_HOST" \
            --port "$SERVE_PORT" \
            --served-model-name "$SERVED_NAME" \
            --max-model-len "$CTX" \
            --gpu-memory-utilization "$MEM_FRACTION" \
            --max-num-seqs "$MAX_SEQS"
fi

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Activate the environment from Part 9's install lesson, or set USE_CONTAINER=1."

exec vllm serve "$MODEL" \
    --host "$SERVE_HOST" \
    --port "$SERVE_PORT" \
    --served-model-name "$SERVED_NAME" \
    --max-model-len "$CTX" \
    --gpu-memory-utilization "$MEM_FRACTION" \
    --max-num-seqs "$MAX_SEQS"

Download serve-single-spark.sh75 lines

RunnableTrack S · DGX Spark

serve the mixture-of-experts model on one machine

bash serve-single-spark.sh

RunnableAll tracks

measure it the same way

bash load-test-pair.sh single-spark-120b

gpt-oss-120b is Apache-2.0 licensed according to the model reference, ships with native MXFP4 weights, and activates about 5B of its 117B parameters per token. Record the same two startup lines and the same four load-test numbers.

10. Optional: the TensorRT-LLM playbook path

If you want the vendor-validated alternative, NVIDIA’s TensorRT-LLM two-Spark playbook serves nvidia/Qwen3-235B-A22B-FP4, the checkpoint its support matrix marks “two Sparks only”. The shape is an MPI job rather than a Ray cluster: a container on each node with host networking and the InfiniBand device mapped in, an Open MPI hostfile on the primary node, and mpirun with trtllm-llmapi-launch wrapping the serve command on port 8355.

RunnableTrack S · DGX Spark

the serve command at the end of the playbook's procedure

trtllm-serve nvidia/Qwen3-235B-A22B-FP4 \
    --tp_size 2 \
    --backend pytorch \
    --max_num_tokens 32768 \
    --max_batch_size 4 \
    --extra_llm_api_options /tmp/extra-llm-api-config.yml \
    --port 8355

Take the container tag and the full container invocation from the playbook rather than from this page: three NVIDIA pages named three different tags on 2026-09-09, and the model handle redirected to a repository named nvidia/Qwen3-235B-A22B-NVFP4 when the Hub was checked on the same day. Then measure it, remembering that trtllm-serve reports the model under its repository name rather than the served name vLLM used:

RunnableTrack S · DGX Spark

measure the TensorRT-LLM path on its own port

SERVE_PORT=8355 SERVED_NAME=nvidia/Qwen3-235B-A22B-FP4 bash load-test-pair.sh pair-trtllm-235b

Put that third line in the notebook next to the other two.

The fallbacks

Track S — NVIDIA DGX Spark

One Spark. Skip tasks 2 to 8 entirely and run tasks 1, 9 and the validation. You will serve gpt-oss-120b at MXFP4 on one machine and measure it properly, which is the baseline every two-machine result in this course is compared against. The memory diagram above is your prediction; the startup log is the check.

Set PAIR_MODEL aside and leave SINGLE_MODEL as it is. Record in the notebook that this was the single-machine path, so the line is not mistaken for a cluster result later.

Track X — AMD Ryzen AI Max+ 395Not supported

No documented multi-node vLLM path exists for this hardware, and the container images this lab pulls are built for GB10 and x86 NVIDIA GPUs.

Go to Part 19. Its first lab, Run a Model Bigger Than Any One Machine, is the Track X equivalent: llama.cpp RPC across two machines over whatever network you have, with the same four measurements and the same notebook shape.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover macOS; there is no Ray-based cluster to build on Apple silicon.

Go to Part 21. Its lab, A Two-Mac Cluster over Thunderbolt 5, builds the Apple version of this cluster with MLX distributed and exo, and its reality-check page examines the claim that several Macs replace a GPU server.

Track N — NVIDIA desktop or laptop

Two 24 GB cards in one desktop. No cluster, no cable, no Ray. Serve a 70B-class AWQ model across both cards, twice: once as a tensor split and once as a pipeline split.

RunnableTrack N · NVIDIA GPU

serve-two-cards.sh

#!/usr/bin/env bash
# Purpose: serve one model across two GPUs in a single desktop, in either the tensor-split
#          or the pipeline-split arrangement, printing the PCIe generation and link width
#          each card negotiated so the measurement can be read against the link it ran on
# Platform: nvidia (one machine, two or more cards, no NVLink on recent GeForce parts)
# Minimum memory: 24 GB per GPU for the 70B-class AWQ model; less for a 32B-class one
# Assumes: vllm on PATH in an activated environment; both cards visible to nvidia-smi; the
#          model already downloaded; nothing else listening on SERVE_PORT. Set SPLIT=tp
#          for a tensor split or SPLIT=pp for a pipeline split, and run it both ways.
set -euo pipefail

HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
if [ -f "${HERE}/.env" ]; then
    set -a
    # shellcheck disable=SC1091  # written by the reader from env-example.txt
    . "${HERE}/.env"
    set +a
fi

MODEL="${TWO_CARD_MODEL:-Qwen/Qwen3-32B-AWQ}"
SERVED_NAME="${SERVED_NAME:-local-cluster}"
SERVE_HOST="${SERVE_HOST:-127.0.0.1}"
SERVE_PORT="${SERVE_PORT:-8000}"
CTX="${CTX:-4096}"
MEM_FRACTION="${MEM_FRACTION:-0.90}"
MAX_SEQS="${MAX_SEQS:-4}"
SPLIT="${SPLIT:-tp}"
DEVICES="${DEVICES:-2}"

fail() { printf '%s\n' "$*" >&2; exit 1; }

command -v vllm >/dev/null 2>&1 || fail "vllm is not on PATH. Activate the environment from Part 9's install lesson."
command -v nvidia-smi >/dev/null 2>&1 || fail "nvidia-smi is not on PATH; the driver is the first thing to check."

printf '==> The link these cards actually negotiated\n'
nvidia-smi --query-gpu=index,name,memory.total,pcie.link.gen.current,pcie.link.width.current \
    --format=csv | sed 's/^/    /'
printf '    Read this under load as well as at idle: the current values drop when a GPU is\n'
printf '    idle, so an idle reading understates the link.\n\n'

case "$SPLIT" in
    tp) TP_SIZE="$DEVICES"; PP_SIZE=1 ;;
    pp) TP_SIZE=1; PP_SIZE="$DEVICES" ;;
    *)  fail "SPLIT must be 'tp' (tensor parallel) or 'pp' (pipeline parallel); got '${SPLIT}'." ;;
esac

cat <<INFO
==> vLLM across ${DEVICES} GPUs in one machine
    model             ${MODEL}
    served as         ${SERVED_NAME}
    listening on      http://${SERVE_HOST}:${SERVE_PORT}
    split             ${SPLIT}  (tensor parallel ${TP_SIZE}, pipeline parallel ${PP_SIZE})
    max model length  ${CTX}
    memory fraction   ${MEM_FRACTION}
    max sequences     ${MAX_SEQS}

    Run this twice, once with SPLIT=tp and once with SPLIT=pp, and load-test each. Without
    NVLink the tensor split runs a collective on every layer over PCIe, and vLLM's own
    documentation suggests pipeline parallel on machines without that interconnect. Which
    wins on your slot is a measurement, and both numbers belong in the notebook next to
    the link width printed above.

INFO

exec vllm serve "$MODEL" \
    --host "$SERVE_HOST" \
    --port "$SERVE_PORT" \
    --served-model-name "$SERVED_NAME" \
    --tensor-parallel-size "$TP_SIZE" \
    --pipeline-parallel-size "$PP_SIZE" \
    --max-model-len "$CTX" \
    --gpu-memory-utilization "$MEM_FRACTION" \
    --max-num-seqs "$MAX_SEQS"

Download serve-two-cards.sh73 lines

RunnableTrack N · NVIDIA GPU

tensor split across both cards

SPLIT=tp bash serve-two-cards.sh

RunnableTrack N · NVIDIA GPU

pipeline split across both cards

SPLIT=pp bash serve-two-cards.sh

Measure each with bash load-test-pair.sh two-card-tp and bash load-test-pair.sh two-card-pp. The script prints the PCIe generation and link width each card negotiated before it starts, and that pair of values is what makes the two results interpretable.

hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 is the 70B-class option, quantised, in its card’s words, “from FP16 down to INT4 using the GEMM kernels performing zero-point quantization with a group size of 128”, and carrying the Llama 3.1 licence. The card is candid about the budget: “around 35 GiB of VRAM are needed only for loading the model checkpoint, without including the KV cache or the CUDA graphs, meaning that there should be a bit over that VRAM available.” Two 24 GB cards clear that with little to spare, so drop CTX until the maximum-concurrency line is above one.

Qwen/Qwen3-32B-AWQ, which its card gives as AWQ 4-bit under the Apache-2.0 licence with a native 32,768-token context, is the comfortable choice if you would rather measure the split than fight the budget. Qwen3-32B is in the model reference with its four-bit size.

Use gates before each expensive stage

Do not start the large download until both nodes pass the network and software checks. Record image identities, accelerator visibility, model licence/access requirements and free storage. Confirm the selected quantised representation fits the combined usable memory budget, including runtime state on both nodes.

Start the orchestration layer and verify both workers’ resources before launching the engine. Keep one terminal or log stream per node. If model initialisation fails, collect the first rank-local error; a coordinator timeout may only reflect a worker that already exited. Verify collectives use the intended cable with logs and interface activity.

Send a short completion before the load test. Then measure the defined workload and compare with the single-machine alternative specified by the lesson. If that alternative uses a smaller model, label the quality and identity difference; it is an operational comparison, not an equal-model speed test. For a failed capacity gate, use the documented fallback and record the large-model path as not run. Archive the settings file, source identities, placement logs, link evidence and client results. Cleanup should stop this lab’s engine and orchestration processes while preserving downloads and the working network configuration you intend to reuse.

Validation

You are done when all of the following are true. On a fallback path, skip the items that name machines you do not have and keep the rest: the notebook lines and the startup-log figures are required on every path, because they are what the comparison is made of.

• prepare-spark-pair.sh passed all five checks on both nodes, and cluster-env.sh exists on each with the same interface name in every variable.
• ray status reported two nodes and two GPUs from inside the container.
• The 405B server logged Application startup complete. and answered a curl request with generated text.
• You recorded the GPU KV cache size in tokens and the maximum concurrency from the startup log, for the pair and for the single machine, with the context length each was asked for.
• The NCCL trace showed which transport was used, and you wrote down which.
• labbook.md contains at least two load-test lines with different labels, one from the pair and one from a single machine, produced by the same generator with the same settings.
• On the Track N fallback, two lines with two-card-tp and two-card-pp labels, and the PCIe generation and link width recorded beside them.

Expected outcome

Two comparable notebook entries, and an opinion about them that you can defend.

Pending validationWhat this lab measures — to be filled in by the validation pass

Configuration	Output tokens/s	Requests/s	Time to first token (s)	Time per output token (s)
Two Sparks, 405B four-bit, tensor parallel 2	pending	pending	pending	pending
One Spark, gpt-oss-120b MXFP4	pending	pending	pending	pending
Two Sparks, Qwen3-235B-A22B NVFP4, TensorRT-LLM	pending	pending	pending	pending
Two 24 GB cards, 70B-class AWQ, tensor parallel 2	pending	pending	pending	pending
Two 24 GB cards, 70B-class AWQ, pipeline parallel 2	pending	pending	pending	pending

two NVIDIA DGX Sparks (GB10, 128 GB each) and a two-card NVIDIA desktop, DGX OS 7.x; Ubuntu 24.04 on the desktop · vLLM and TensorRT-LLM; to be recorded by the validation pass · as listed in the first column, as listed in the first column · 4,096 tokens of context · 2026-09-09

No row here has been measured. The table exists so that the validation pass can substitute measured values without rewriting the page, and so that your own five rows have somewhere to go. Fill in your copy in the notebook; the four columns are what Part 9's load generator reports.

The result this lab expects, and which you should be ready to have contradicted by your own numbers, is that the 400B-class model on the pair produces tokens slowly and holds almost no context, while the 120B-class mixture-of-experts model on one machine is faster and far more usable. Part 18 gave the reason in advance: a dense 405B model reads every one of its parameters for every token, while a mixture-of-experts model of a quarter the size reads a twentieth of them. Capacity is what the second machine bought. Speed is not.

If your numbers say otherwise, that is the more interesting outcome, and the notebook line with your versions and settings is what makes it worth reporting.

Troubleshooting

ray status shows one node. Ray’s own message is Unable to connect to GCS at ..., and it lists four causes: the head is not running, a different Ray version is at that address, the address is wrong, or a firewall is in the way. Check that the worker’s NODE_HEAD_ADDR is the head’s QSFP address and not its management address, and that both nodes pulled the same container tag.

The cluster vanishes when you close a terminal. Expected. The helper script stops its container on exit and vLLM’s documentation says to keep the shells open. Use tmux.

docker ps fails with a permission error. NVIDIA’s table gives the fix directly: run sudo usermod -aG docker $USER && newgrp docker, on both nodes.

The container will not start, or is mysteriously slow. On aarch64 this is usually an image built for the wrong architecture. NVIDIA lists “container startup fails / missing ARM64 image” as a DGX Spark row with the fix of using the default NGC image for the hardware. docker image inspect reports what you pulled.

CUDA out of memory at startup. NVIDIA’s fix is to “reduce --max-model-len and --max-num-seqs, or lower --gpu-memory-utilization”, which in this lab means CTX, MAX_SEQS and MEM_FRACTION in .env. On a Spark there is a second cause: the same page notes that some applications have not been updated for unified memory and “you may hit memory issues even within capacity”, with a documented remedy of flushing the buffer cache.

nvidia-smi reports no memory usage. Documented behaviour on unified memory, as Part 5 recorded. NVIDIA’s own note says the --query-gpu memory fields report N/A on these machines and to use plain nvidia-smi instead.

A UCX warning about a device not being available. NVIDIA’s TensorRT-LLM playbook says this one can be ignored when inference succeeds: it means only one of the two ConnectX-7 ports is in use.

The pair is slower than one machine and the NCCL trace says NET/Socket. The interface variables did not reach the engine. Re-run prepare-spark-pair.sh, confirm cluster-env.sh has the QSFP interface in every variable, and check the head and worker scripts passed them through.

On the two-card desktop, the tensor split is much slower than the pipeline split. Look at the PCIe generation and link width the script printed. A second slot wired for four lanes makes a per-layer collective the bottleneck, and vLLM’s documentation itself suggests pipeline parallel on machines without NVLink.

The generation slows down after twenty minutes. Read nvidia-smi’s clock throttle reasons. It distinguishes a software power cap from a hardware thermal slowdown, and the two have different fixes.

Cleanup

Stop the serving process first, then the worker container, then the head container, in that order. Closing the two tmux windows does it, because the helper script’s exit trap stops each container.

The Ray cluster leaves nothing behind on the machines except the container images and the model cache. Remove the 405B checkpoint if you need the disk back; it is the largest thing this course downloads. NVIDIA’s TensorRT-LLM playbook’s cleanup step is the same idea for that path: stop the container on each node, then delete the model directory.

⚠Warning

The network configuration is deliberately not part of cleanup. Parts 21 and 22 assume a working link, and undoing it costs another hour. If you do want to revert it, the connect-two-Sparks playbook’s rollback is to delete the netplan file and run sudo netplan apply on both nodes, or to remove the addresses you added by hand. Do not run either over a connection that depends on the interface you are removing.

What you learned

• A cluster is a network problem before it is a serving problem. Five read-only checks on each node, all passing, are what stops a NCCL stack trace from being a mystery. The address, the interface name and the username are the three things that go wrong.
• One command serves two machines. Ray makes the second machine’s GPU visible, and vllm serve treats the pair as one larger machine. Nothing about the serving interface changes.
• The startup log predicts the outcome. The cache size in tokens and the maximum concurrency at your context length are available before any request arrives, and they tell you whether the configuration can do the job.
• RDMA is a thing you verify, not a thing you assume. NET/IB/GDRDMA and NET/Socket are two different clusters wearing the same configuration, and one line of log separates them.
• Capacity and speed are different purchases. The pair held a model one machine could not. It did not make tokens arrive faster than a smaller mixture-of-experts model on one machine, and the comparison you just measured is the evidence.
• The vendor’s own caveats are the most valuable part of a playbook. “Insufficient memory headroom for production use — testing only” is a more useful sentence than any benchmark on the same page.

Record in the notebook: the container tag and engine version on each node; the QSFP interface name and which transport NCCL chose; for each configuration, the model, the quantisation, the parallel sizes, the context length, the memory fraction, the KV cache size in tokens and the maximum concurrency from the startup log; and for each load test, the label, the concurrency levels, and the four numbers the generator reports. Add one sentence saying which configuration you would actually use and why. Part 28’s capstone is written from these entries.

Check your understanding

Question 1. On the worker node, what should VLLM_HOST_IP be set to?

• The head node's address, so the worker can find it
• The worker's own address on the QSFP link; vLLM's documentation states it is unique for each worker
• The management address of either node
• It does not need to be set

Show the answer and why

Answer: The worker's own address on the QSFP link; vLLM's documentation states it is unique for each worker

The variable advertises this process's own address. Setting it to the head's address on both nodes is the commonest way to produce a worker that appears to join and then produces nothing. NODE_HEAD_ADDR is the value that is the same on both machines; NODE_SELF_ADDR is the one that differs.

Question 2. The startup log reports a maximum concurrency of 0.4x for the context length you asked for. What does that mean and what do you do?

• The cluster is running at 40% efficiency; tune the batch size
• The cluster cannot hold even one sequence of that length, so the server will fail or preempt on the first request; lower the context length and restart
• Forty per cent of requests will be refused
• It is a warning that can be ignored

Show the answer and why

Answer: The cluster cannot hold even one sequence of that length, so the server will fail or preempt on the first request; lower the context length and restart

That line is the cache size divided by your context length, so a value below one means a single sequence does not fit. NVIDIA's own 405B example sets a maximum model length of 64 tokens and one sequence for exactly this reason, which is what "insufficient memory headroom" looks like written as flags.

Question 3. Why does the lab serve gpt-oss-120b on one machine after serving a 405B model on two?

• To check the second machine still works
• Because the comparison is the point: the pair bought capacity, and the question is whether a smaller mixture-of-experts model on one machine is faster and more usable for the same task
• Because the 405B model is unlicensed
• To warm the cache

Show the answer and why

Answer: Because the comparison is the point: the pair bought capacity, and the question is whether a smaller mixture-of-experts model on one machine is faster and more usable for the same task

A dense 405B model reads all of its parameters per token; a mixture-of-experts model of a quarter the size reads a small fraction of its own. Part 18 predicts the outcome and this lab measures it. A cluster is worth having when the notebook says so.

Question 4. Which of these belong in the notebook entry for a run? Select all that apply.

• The container tag used on each node
• The transport NCCL chose, from the trace
• The KV cache size in tokens and the maximum concurrency from the startup log
• Your Hugging Face token, so the run can be repeated

Show the answer and why

Answer: The container tag used on each node, The transport NCCL chose, from the trace, The KV cache size in tokens and the maximum concurrency from the startup log

A credential never goes in the notebook, and the load generator is written so that a key is read from an environment variable and never recorded. Everything else is what makes a number reproducible six months later, which is the only kind of number worth writing down.

Question 5. On the two-card desktop fallback, why does the script print the PCIe generation and link width before serving?

• To confirm the driver is installed
• Because a tensor split runs a collective on every layer over that link, so the negotiated generation and width are what make the tensor-versus-pipeline comparison interpretable
• Because vLLM requires it
• To check the cards are the same model

Show the answer and why

Answer: Because a tensor split runs a collective on every layer over that link, so the negotiated generation and width are what make the tensor-versus-pipeline comparison interpretable

The same two commands on two different motherboards can give opposite answers, and the difference is usually the second slot's electrical width rather than anything about the software. Recording the link alongside the measurement is what turns a surprising result into an explainable one.

Check my answersClear