TensorRT-LLM and Dynamo on Spark Pairs

By the end of this lesson you will be able to describe how TensorRT-LLM runs across two DGX Sparks and how that differs from the Ray path, configure its parallelism through a YAML file rather than a pile of flags, name the container tags NVIDIA’s own pages disagree about and say what to do about it, explain what NVIDIA Dynamo is and what it needs, and say where each of the two belongs in a house rather than a data centre.

Part 8 taught trtllm-serve on one machine. This is the same software with MPI underneath it.

The two-Spark TensorRT-LLM path

The structural difference from Part 9’s Ray path is worth stating first, because it changes what you type and where you type it.

Two ways to make two machines behave as one

1. 1vLLM with RayA long-lived cluster: start a head, join a worker, then run one vllm serve on the head. The cluster outlives the server, and you can restart the server without rebuilding the cluster.
2. 2TensorRT-LLM with MPINo standing cluster. A container runs on each node; mpirun on the primary node launches one rank per node over SSH, and trtllm-llmapi-launch wraps trtllm-serve so it becomes the MPI job.
3. 3What both need underneathThe same thing: matching usernames, passwordless SSH in both directions, addresses on the QSFP interfaces, and the interface names exported into the container.

The MPI path has no cluster to inspect, which is why its failures show up as an mpirun error rather than as a missing node in ray status.

NVIDIA’s playbook for the pair is a twelve-step procedure and the shape of it is this. Both nodes get into the docker group. The primary node gets a plain-text hostfile listing both nodes’ QSFP addresses, one per line, which is what Open MPI reads to decide where ranks go. A container starts on each node with host networking, the InfiniBand device mapped in, memory-lock limits raised, the interface names exported, and an entrypoint script fetched from NVIDIA’s playbook repository. The hostfile is then copied into the primary node’s container, a small YAML configuration is written inside it, and the model is downloaded and served from that one container with mpirun.

The container invocation carries the settings that make RDMA and MPI work, and they are worth recognising rather than copying blindly:

Fragment — not complete on its own

# The load-bearing arguments from NVIDIA's two-Spark TensorRT-LLM playbook.
# Take the full command, and the current image tag, from the playbook itself.
--network host                      # MPI and NCCL need the host's interfaces
--device /dev/infiniband:/dev/infiniband
--ulimit memlock=-1                 # RDMA pins memory; an unlimited lock is required
--ulimit stack=67108864
-e UCX_NET_DEVICES="<your QSFP interfaces>"
-e NCCL_SOCKET_IFNAME="<your QSFP interfaces>"
-e OMPI_MCA_btl_tcp_if_include="<your QSFP interfaces>"
-e OMPI_MCA_orte_default_hostfile="/etc/openmpi-hostfile"
-e OMPI_MCA_rmaps_ppr_n_pernode="1" # exactly one rank per machine

OMPI_MCA_rmaps_ppr_n_pernode set to one is the line that encodes “one GPU per Spark”. The serve command then asks for a tensor parallel size of two, spanning the pair, exactly as the vLLM playbook does and for the same reason: the cable is fast enough.

RunnableTrack S · DGX Spark

the serve command NVIDIA's two-Spark playbook runs inside the container

trtllm-serve nvidia/Qwen3-235B-A22B-FP4 \
    --tp_size 2 \
    --backend pytorch \
    --max_num_tokens 32768 \
    --max_batch_size 4 \
    --extra_llm_api_options /tmp/extra-llm-api-config.yml \
    --port 8355

That command is launched under mpirun with trtllm-llmapi-launch in front of it, and the port is one of the two the playbook’s prerequisites tell you to open: “network: open TCP ports 8355 (LLM) and 8356 (VLM) on host for OpenAI-compatible serving.”

✦Tip

The playbook records a warning it expects you to see and to ignore: UCX WARN network device 'enp1s0f0np0' is not available, which it explains is “related to only one of your two CX-7 ports being used, and the other being left unused”. This is the sort of note that makes a vendor playbook worth more than a general installation guide. Nothing else tells you which warnings on this exact machine are noise.

The model in that command is the interesting part. NVIDIA’s model support matrix for TensorRT-LLM on Spark lists a couple of dozen checkpoints, mostly NVFP4 and FP8 conversions the vendor publishes itself, and exactly one row is annotated “two Sparks only”: nvidia/Qwen3-235B-A22B-FP4. That is NVIDIA naming a 235B-class mixture-of-experts model as the workload the pair exists for. Qwen3-235B-A22B is Apache-2.0 licensed according to the model reference; NVIDIA’s conversion is a separate repository whose own card you should check before relying on it, and on 2026-09-09 that handle redirected to a repository named nvidia/Qwen3-235B-A22B-NVFP4. The matrix also lists both gpt-oss models at MXFP4, which is the single-Spark comparison this part’s lab uses.

Configuring parallelism in a file, not in flags

TensorRT-LLM’s parallelism documentation lists six strategies rather than Part 18’s five: tensor, pipeline, data, expert, context and “Wide Expert Parallel”, which it describes as “an advanced form of expert parallelism that addresses the inherent workload imbalance in large-scale MoE models through intelligent load balancing and expert replication”. The one-line summaries match Part 18’s definitions exactly, with tensor parallelism “best for: small batch sizes, memory-constrained scenarios” and pipeline parallelism “best for: large models that don’t fit in single GPU memory”.

Where it differs from vLLM is that the interesting choices live in a YAML file passed with --extra_llm_api_options, not on the command line. For attention, the page gives tensor_parallel_size alongside enable_attention_dp, which switches attention from a tensor split to a data split. For mixture-of-experts layers it gives three patterns and one constraint:

Pattern	What the file says	What it does
Tensor parallel experts	moe_tensor_parallel_size equal to tensor_parallel_size	“Every expert’s weight matrix is sliced across all GPUs. Each GPU sees all tokens.”
Expert parallel	moe_expert_parallel_size equal to tensor_parallel_size	“Full weights of each expert reside on a single GPU. Each GPU only sees tokens routed to its local experts.”
Hybrid	both set, multiplying to tensor_parallel_size	“Each GPU stores a subset of experts (EP) and shards those weights further (TP)”

The constraint is arithmetic and the documentation states it: “the product of moe_tensor_parallel_size and moe_expert_parallel_size must equal tensor_parallel_size.” On a two-Spark pair that leaves exactly two possibilities, which makes it a cheap experiment: run the same load test under each and keep the file that wins.

The configuration file the playbook writes is short, and its contents are worth understanding because two of the three keys are ones you already know from Part 9 under different names: kv_cache_config with a free_gpu_memory_fraction of 0.9 is vLLM’s --gpu-memory-utilization, cuda_graph_config with padding enabled is the graph capture that --enforce-eager disables, and print_iter_log controls per-iteration logging.

Container tags, dated

⌚Version note

This course pins TensorRT-LLM 1.2.1 · verified 2026-09-08, the latest stable release; 1.3 is a release candidate. On 2026-09-09 three NVIDIA pages named three different container tags for the same software. The TensorRT-LLM playbook’s prerequisites check used release:1.3.0rc13; its two-Spark page used release:1.3.0rc5; the installation documentation’s container page used devel:1.3.0rc25 and release:1.3.0rc25. None of these is wrong, they are simply dated. Look up the current tag on NGC before you pull, use one tag consistently on both nodes, and write the tag you used into the notebook beside your numbers. A pair of machines running two different container tags is a failure mode with no clear error message.

The architecture question from Part 5 applies here too. NVIDIA publishes the TensorRT-LLM images multi-architecture, which is why the Spark can pull them at all, and the vLLM playbook’s own troubleshooting table lists “container startup fails / missing ARM64 image” as a DGX Spark row. Check what you pulled with docker image inspect before you spend an evening on a slow benchmark.

The playbook budgets “45-60 minutes for setup and API server deployment” and rates its own risk as medium because “container pulls and model downloads may fail due to network issues”. For a 235B-class NVFP4 checkpoint on a domestic connection, the download is the long pole and it is unattended.

Dynamo, and where it belongs

NVIDIA Dynamo is the third piece of this stack and the one this course surveys rather than teaches. Its documentation describes it as “a distributed inference runtime for generative AI systems that must operate at high throughput, low latency, and high reliability under changing traffic conditions”, and says it “is backend-agnostic (SGLang, TRT-LLM, vLLM, and others)”. It is not an engine. It is the thing that sits above one.

The architecture is three planes. The request plane has a frontend that “accepts and normalizes requests”, a router that “selects workers based on load and KV overlap”, and separate prefill and decode workers. The control plane has a planner that “computes scaling targets from live metrics” and a Kubernetes operator. The storage and events plane has KV events, a block manager that “manages block reuse, eviction, and offload/recall across memory tiers”, and NIXL, which “performs high-speed KV/data transfer across workers and memory domains”.

That last component is where the requirement bites. Dynamo’s RDMA page is unambiguous: “NVIDIA Dynamo uses RDMA to transfer KV cache between workers in disaggregated serving”, and “Dynamo needs RDMA for disaggregated serving, where prefill workers generate KV cache and hand it to decode workers.” It gives the alternative and its cost in the vendor’s own words: “the alternative is TCP over Ethernet, which is 200-500x slower for this transfer.” It also notes that aggregated deployments, where prefill and decode live in one worker, “transfer no KV cache between workers, and do not need RDMA.”

∴Research note

A DGX Spark pair does have RDMA, over RoCE on the ConnectX-7 link, so the requirement is not in itself a barrier. The gap is elsewhere. Dynamo’s compatibility page lists platform support as “GPU architectures: Blackwell, Hopper, Ada Lovelace, Ampere” and “OS: Ubuntu 24.04 (x86_64, ARM64)”, which a Spark satisfies on both counts as a description. It does not name GB10, the DGX Spark, or DGX OS anywhere on that page, and an architecture family being supported is not the same claim as a specific machine being validated. This course’s version table records Dynamo NVIDIA Dynamo 1.4.2 · verified 2026-09-08 as surveyed rather than taught, on the grounds that multi-node work requires RDMA and GB10 is untested. If you run it on a Spark pair, you are ahead of the documentation, and the course would like to hear what happened.

The other reason Dynamo is a survey here is that its centre of gravity is Kubernetes. The documentation has a Kubernetes user guide, an operator, custom resources, cloud-provider setup guides for three clouds, and a local command-line guide alongside them. The problems it solves are real problems: prefill and decode imbalance, KV recomputation, memory pressure across tiers, and dynamic demand. They are also problems that appear when traffic is bursty and machines are many, which is not the shape of a house.

Part 22 is where disaggregated serving is actually built, with vLLM’s own disaggregated prefill and SGLang’s prefill-decode split, on two machines and without an orchestrator. Read this section as the map: Dynamo is what the pattern looks like when it is productised, and knowing that helps when you read a benchmark that was measured on it.

When the playbook path beats vLLM

Both engines serve the same models across the same pair. Three things separate them in practice.

A tested combination. On aarch64, a playbook is NVIDIA saying “this image, this model, this command, on this machine”. Part 8 made this argument for one Spark and it is stronger for two, because the number of things that can be subtly wrong has roughly doubled.

Vendor checkpoints. TensorRT-LLM’s advantage is largely quantisation, and NVIDIA publishes NVFP4 conversions of models it supports so you do not have to make one. The nvidia/ repository prefix in the support matrix is the practical form of that advantage.

Configuration as a file. --extra_llm_api_options pointing at a YAML file is reproducible in a way that a forty-token shell line is not, and it is the difference between a serving configuration you can commit and one you retype.

Against that, vLLM’s Ray cluster is easier to inspect and easier to restart, its documentation covers the multi-node case in far more depth, and it is the engine the rest of this course uses. The honest recommendation is the one Part 8 gave: readers who want one engine for the whole course should stay with vLLM, and readers on Track S who want the vendor-validated path for a very large model should use the playbook and record which one they used.

Track S — NVIDIA DGX Spark

Both paths are yours. Start from the playbook you can reach: the TensorRT-LLM two-Spark page for the NVFP4 checkpoint NVIDIA marks “two Sparks only”, or the vLLM multi-node page from the previous lesson. Record the container tag, the model repository and the parallel configuration with every number.

Track X — AMD Ryzen AI Max+ 395Not supported

TensorRT-LLM and Dynamo both target NVIDIA GPUs; Dynamo's compatibility page lists only NVIDIA architectures.

The transferable idea is the configuration file. Part 19’s llama.cpp RPC cluster has no equivalent, and writing your own settings file that a script reads is a habit worth importing.

Track M — Apple siliconNot supported

TensorRT-LLM and Dynamo target NVIDIA GPUs; there is no macOS path for either.

Part 21 covers the Apple cluster. The comparison worth drawing is that Apple’s launcher, mlx.launch, plays the role mpirun plays here, and it has the same property of leaving no standing cluster behind.

Track N — NVIDIA desktop or laptopPartial

The software runs on x86 NVIDIA hardware, but the DGX Spark playbooks are written for GB10 and their container tags and model matrix are Spark-specific.

Everything above works on two x86 machines with the general TensorRT-LLM documentation instead of the playbook, and the parallelism configuration file is identical. What you lose is the tested combination, so expect to spend the time the playbook would have saved.

What this course validated, and what it did not

Stated plainly, because Level 4 is where the gap between reading and running is widest.

Every page in this part was written from the documentation cited on it, read on 2026-09-09. Nothing in this part has yet been executed on hardware by the course. The validation pass will run the primary path on the reference cluster’s two Sparks and record versions, container tags and measurements per track, and the pending tables in the lab are where those numbers will land.

Specifically not validated: the TensorRT-LLM two-Spark path on any container tag; NVIDIA Dynamo on GB10 in any configuration; expert parallelism across two machines with the DeepEP backends; and any claim about which of the two serving paths is faster on the pair. Where this part states a number, it is either NVIDIA’s own specification figure, marked as such with its source, or arithmetic labelled as arithmetic.

Keep engine execution separate from service orchestration

A specialised engine determines how the model executes. A serving framework can add routing, discovery, scheduling and separation of inference phases. Adding orchestration does not automatically fix an engine that cannot load the checkpoint or improve a single request whose bottleneck is local memory bandwidth.

Establish the smallest supported engine deployment first, with the exact image, model and configuration from the chosen playbook. Then add the pair and finally the orchestration features required by your workload. Save a baseline at each stage so the source of any latency or reliability change is visible.

List the extra services and ports introduced by orchestration, their persistent state and shutdown order. Test a worker restart and observe what happens to in-flight and subsequent requests. A production-style architecture is useful when it solves a measured scheduling or capacity problem; it also introduces components you must monitor and recover. Describe the supported stack you exercised and the workload that benefits rather than treating a complex launch as evidence of a better service.

Recap

TensorRT-LLM across two Sparks is an MPI job rather than a standing cluster: a container per node with host networking and the InfiniBand device mapped in, a hostfile listing both QSFP addresses, and mpirun with trtllm-llmapi-launch wrapping trtllm-serve at tensor parallel size two on port 8355. Its parallelism lives in a YAML file passed with --extra_llm_api_options, where moe_tensor_parallel_size times moe_expert_parallel_size must equal tensor_parallel_size, and NVIDIA’s model matrix marks one NVFP4 checkpoint of a 235B-class mixture-of-experts model as requiring two Sparks. Three NVIDIA pages named three different container tags on one day, so pin one tag on both nodes and record it.

Dynamo is a distributed inference runtime above the engines, with a router, a planner, a KV block manager and NIXL for cache transfer, and it requires RDMA for disaggregated serving because the TCP alternative is, in NVIDIA’s own comparison, hundreds of times slower for that transfer. Its compatibility page names Blackwell and ARM64 Ubuntu without naming GB10, and this course treats it as surveyed rather than taught. Part 22 builds disaggregated serving on two machines without it.

Check your understanding

Question 1. How does the TensorRT-LLM two-Spark path differ structurally from the vLLM Ray path?

• It uses a different network protocol
• There is no standing cluster: a container runs on each node and mpirun launches one rank per node, so failures appear as an mpirun error rather than as a missing node in ray status
• It only supports one GPU
• It does not need SSH

Show the answer and why

Answer: There is no standing cluster: a container runs on each node and mpirun launches one rank per node, so failures appear as an mpirun error rather than as a missing node in ray status

Ray gives you a long-lived cluster you can inspect and restart the server against. MPI gives you a job. Both need the same foundation underneath, which is matching usernames, passwordless SSH both ways and addresses on the QSFP interfaces.

Question 2. Your TensorRT-LLM configuration file sets tensor_parallel_size to 2. Which MoE settings are valid?

• moe_tensor_parallel_size 2 and moe_expert_parallel_size 2
• Either moe_tensor_parallel_size 2 with expert parallel 1, or expert parallel 2 with tensor parallel 1, because the documentation requires their product to equal tensor_parallel_size
• Any values at all
• MoE settings are not configurable

Show the answer and why

Answer: Either moe_tensor_parallel_size 2 with expert parallel 1, or expert parallel 2 with tensor parallel 1, because the documentation requires their product to equal tensor_parallel_size

The constraint is stated in the documentation as an equality, so on a pair there are exactly two legal configurations. That makes the comparison a cheap experiment rather than a design decision: run both under the same load and keep the file that wins.

Question 3. Three NVIDIA pages named three different TensorRT-LLM container tags on the same day. What is the right response?

• Use the newest one always
• Look up the current tag on NGC, use the same tag on both nodes, and record it beside your measurements
• Use the oldest, as it is most stable
• Build the container from source

Show the answer and why

Answer: Look up the current tag on NGC, use the same tag on both nodes, and record it beside your measurements

The pages are dated rather than wrong. What actually breaks a pair is two nodes running different tags, which produces failures with no clear message. One tag, both nodes, written down.

Question 4. Which statements about NVIDIA Dynamo are supported by its own documentation? Select all that apply.

• It is a distributed inference runtime and is backend-agnostic across SGLang, TensorRT-LLM and vLLM
• It uses RDMA to transfer KV cache between workers in disaggregated serving
• Aggregated deployments transfer no KV cache between workers and do not need RDMA
• Its compatibility page lists GB10 and the DGX Spark as validated platforms

Show the answer and why

Answer: It is a distributed inference runtime and is backend-agnostic across SGLang, TensorRT-LLM and vLLM, It uses RDMA to transfer KV cache between workers in disaggregated serving, Aggregated deployments transfer no KV cache between workers and do not need RDMA

The compatibility page lists GPU architectures including Blackwell and ARM64 Ubuntu 24.04, and does not name GB10 or the DGX Spark. An architecture family being listed is a weaker claim than a machine being validated, which is why this course records Dynamo as surveyed rather than taught.

Question 5. Which of the numbers in this part have been measured by the course on hardware?

• All of them
• None of them; every figure here is either an NVIDIA specification value with its source or arithmetic labelled as arithmetic, and the validation pass will fill the pending tables
• Only the memory figures
• Only the two-Spark figures

Show the answer and why

Answer: None of them; every figure here is either an NVIDIA specification value with its source or arithmetic labelled as arithmetic, and the validation pass will fill the pending tables

Every page in this part carries a line saying it was written from the documentation cited on it. That is not a disclaimer; it is the difference between a course that measures and a course that repeats, and the pending benchmark tables are where the measured values will replace the estimates.

Check my answersClear