TensorRT-LLM, NIM and the DGX Spark Playbooks

By the end of this lesson you will be able to describe how TensorRT-LLM turns a Hugging Face checkpoint into something it can serve, start trtllm-serve inside NVIDIA’s own container, read the quantisation support matrix for the GPU you own, say what a NIM microservice is and is not, and explain why the DGX Spark playbooks are the documented path on Track S rather than a marketing page.

Track S readers should work through this with a terminal open. Track N readers get the same software on x86, with one important difference covered at the end. Tracks X and M should read it for the argument about vendor stacks, which recurs in the next lesson with different names.

What TensorRT-LLM is

llama.cpp reads a GGUF file and runs it. TensorRT-LLM is closer to a compiler with a server attached: it takes a model definition and produces something specialised to one model, one precision, one GPU architecture and one range of shapes, then serves that. The specialisation is where the speed is supposed to come from, and it is also where the cost is, because a specialised artefact is by definition not portable.

Modern releases offer two ways through. The TensorRT engine workflow is the original one, in which a checkpoint is built into an engine ahead of time. The PyTorch backend loads a checkpoint more directly and is what the quick start uses, which is why the quick start’s first command is short enough to fit on one line.

From a checkpoint to a served endpoint

1. 1A Hugging Face checkpointSafetensors weights at bf16, or a pre-quantised checkpoint published by the model owner or by NVIDIA.
2. 2Optional: quantise with ModelOptNVIDIA TensorRT Model Optimizer produces an FP8 or NVFP4 checkpoint offline. The documentation gives scripts/huggingface_example.sh --model <card> --quant fp8 as the example.
3. 3Choose a backendThe PyTorch backend loads the checkpoint directly. The TensorRT engine workflow builds a specialised engine first, which takes longer and targets one GPU architecture.
4. 4trtllm-serveStarts an HTTP server around the model, with the parallelism, batching and KV-cache options passed as flags or in a YAML file.
5. 5OpenAI-compatible endpoints/v1/models, /v1/completions and /v1/chat/completions, plus /health, /metrics and /version.

⌚Version note

This course pins TensorRT-LLM 1.2.1 · verified 2026-09-08. The documentation this lesson was read from on 2026-09-09 states at the foot of every page that it was “Last updated on August 27, 2026” and generated from commit 785c948, and the container images it names carry 1.3 release-candidate tags. That gap between the pinned stable release and the tags in the documentation is normal for this project and is the single most common reason a command from a blog post does not work: check which version you are actually running before you debug anything else.

Serving a model

The quick start is genuinely short. Both of these are complete commands:

RunnableTrack S · DGX Spark

serve a small model, straight from the Hub

trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

RunnableTrack S · DGX Spark

serve a checkpoint that is already quantised

trtllm-serve "nvidia/Qwen3-8B-FP8"

The second one is worth pausing on. nvidia/Qwen3-8B-FP8 is an FP8 conversion of Qwen3-8B published by NVIDIA, and it is the shape of thing you will keep meeting on this track: the vendor publishes pre-quantised checkpoints in the formats its own hardware accelerates, so that the reader does not have to run the quantisation step. Qwen3-8B itself is Apache-2.0 licensed, as the course’s model reference records; check the conversion’s own card for its licence before you rely on it, because a conversion can carry different terms from the model it converts.

Once it is up, it answers the API you already know:

RunnableTrack S · DGX Spark

the same request shape as every other engine in this course

curl -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -d '{
        "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
        "messages":[{"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": "Where is New York? Tell me in a single sentence."}],
        "max_tokens": 32,
        "temperature": 0
    }'

The options that matter for a first serve are few. --host and --port do what you expect. --backend selects pytorch. --max_batch_size caps how many requests the engine will schedule at once, and --kv_cache_free_gpu_memory_fraction decides how much of the free memory the KV cache is allowed to take, which is the same idea as vLLM’s memory-utilisation flag in Part 9. --tp_size and --pp_size set tensor and pipeline parallelism for a machine with more than one GPU; Part 18 explains the difference and Part 20 uses them.

RunnableTrack S · DGX Spark

a serve command with the options spelled out

trtllm-serve "Qwen/Qwen3-8B" \
    --backend pytorch \
    --host 127.0.0.1 \
    --port 8000 \
    --max_batch_size 8 \
    --kv_cache_free_gpu_memory_fraction 0.6

Several more options are worth knowing by name because they answer questions the rest of the course asks. --max_seq_len caps the total sequence length, which is how you bound the KV cache directly. --kv_cache_dtype accepts fp8 and nvfp4 as well as auto, so the cache can be quantised independently of the weights. --tool_parser and --reasoning_parser select the per-model-family parsers that turn a model’s raw output into tool calls and reasoning blocks, which is exactly the machinery Part 9 covers for vLLM and Part 24 depends on. And --extra_llm_api_options takes a YAML file, which is how a configuration stops being a shell line you cannot reproduce.

Quantisation, and what your GPU can actually do

TensorRT-LLM’s advantage over a portable engine is largely a quantisation story, and the documentation publishes a support matrix by GPU architecture. The formats it lists are FP4, MXFP4, FP8 in per-tensor, block-scaling and rowwise variants, FP8 and NVFP4 KV caches, and the weight-only and weight-plus-activation AWQ and GPTQ combinations W4A16 and W4A8.

The matrix is the part to read carefully, because “TensorRT-LLM supports NVFP4” is only true of some hardware:

Architecture	What the matrix lists as supported
Blackwell (sm100/sm103)	NVFP4, MXFP4, all FP8 variants, FP8 and NVFP4 KV cache, W4A8 and W4A16 AWQ and GPTQ
Blackwell (sm120)	NVFP4, MXFP4, FP8 per tensor, FP8 KV cache
Hopper	FP8 variants, FP8 KV cache, W4A8 and W4A16 AWQ and GPTQ
Ada Lovelace	FP8 per tensor, FP8 KV cache, W4A8 and W4A16 AWQ and GPTQ
Ampere	FP8 KV cache, W4A16 AWQ and GPTQ

Read the row for your own card before choosing a checkpoint. An Ampere card asked to load an NVFP4 checkpoint does not run it slowly; it does not run it. This is the mirror image of the GGUF situation in Part 6, where any quantisation runs anywhere and only the speed changes.

Where a pre-quantised checkpoint does not exist, the documentation’s answer is to make one: “you can quantize it offline using ModelOpt”, the NVIDIA TensorRT Model Optimizer, whose example invocation is scripts/huggingface_example.sh --model <huggingface_model_card> --quant fp8, with --quant fp8 --kv_cache_quant nvfp4 shown for quantising the cache as well. That is a separate tool with its own installation, and it is the step this course does not ask you to run: Part 16 covers quantisation as a subject properly, with evaluation attached, rather than as a build step.

✱Hardware note

The DGX Spark’s GB10 is a Blackwell part, and gpt-oss models ship with native MXFP4 weights, which appears in the Blackwell rows above. That combination is why the Spark can serve a model like gpt-oss-20b in its published format without a conversion step, and it is a genuine advantage of this track. What the documentation does not do is publish a per-model list of what has been validated on GB10 specifically, so treat “supported on Blackwell” as a starting point to verify, not a promise.

Containers, because the install is the hard part

TensorRT-LLM is distributed as NGC containers, and on Track S that is not a convenience but the documented path: the machine is aarch64, and NVIDIA’s arm64 container images are where the matching CUDA, driver and Python builds are already assembled.

The installation documentation names two images, a devel image for building and a release image for running, both under nvcr.io/nvidia/tensorrt-llm/, tagged with the release-candidate version. It also gives one instruction that is easy to skip and expensive to skip: “please make sure to set --ipc=host as a docker run argument to avoid Bus error (core dumped)”.

RunnableTrack S · DGX Spark

check the container can see the GPU before anything else

docker run --rm --gpus all nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 nvidia-smi

That exact command, with that exact tag, is the validation step in NVIDIA’s own DGX Spark playbook for TensorRT-LLM, read on 2026-09-09. The installation documentation on the same date named 1.3.0rc25 for both images. Two NVIDIA pages, two tags, one day: check the current tag on NGC rather than copying either of them, and record the one you used in the notebook.

⚠Warning

Container flags change what the container may do to the machine. --gpus all gives it every GPU; --ipc=host shares the host’s inter-process communication namespace, which is what removes the shared-memory limit that causes the bus error. Both are in NVIDIA’s own instructions and both are wider than a default container. Run vendor containers on a machine you administer, not on a shared one, and stop them when you are done.

NIM: the same idea, packaged and licensed

A NIM is a step further along the same path. NVIDIA describes the family as “a set of easy-to-use microservices for accelerating the deployment of foundation models on any cloud or data center”, delivered as containers that are part of NVIDIA AI Enterprise, with production-grade runtimes and ongoing security updates. In practice a NIM is a container that already contains a model, an optimised runtime chosen for the GPU it finds, and an OpenAI-compatible API, so that deployment is docker run rather than a build.

Two things follow that matter to a reader of this course.

It is a commercial product with a licence attached. The documentation places NIM inside NVIDIA AI Enterprise. Access is through NGC with an API key, and the DGX Spark playbook for NIM checks the key before anything else, with a command that also tells you the expected shape of one:

RunnableTrack S · DGX Spark

the playbook's own check that a key is present

echo $NGC_API_KEY | grep -E '^[a-zA-Z0-9]{86}=='

It is the opposite trade from the rest of this part. Everywhere else you are buying speed with complexity. With a NIM you are buying simplicity and support with a licence and a narrower model catalogue. Whether that is a good trade depends on whether you are running a service for other people or learning how one works. This course is about the second, so it names NIM, shows where it lives, and moves on.

∴Research note

The NIM playbook for DGX Spark, read on 2026-09-09, states its prerequisites and its key check but defers the container invocation to a linked instructions page, so this lesson does not print a docker run line for a NIM. If you follow this path, take the command from the playbook’s instructions tab at the time you run it, and record the image tag in the notebook. A command copied from a course page months later is a command that has not been checked.

The Spark playbooks, and why they are the documented path

NVIDIA publishes a set of playbooks for the DGX Spark, described on the index page as “Detailed instructions to set up and run popular AI workflows on DGX Spark”. Read on 2026-09-09 the index listed forty-one of them, and the ones relevant to this course include trt-llm, vllm, sglang, llama-cpp, nim-llm, nvfp4-quantization, speculative-decoding, the fine-tuning set (unsloth, llama-factory, nemo-fine-tune, pytorch-fine-tune) and the clustering set (connect-two-sparks, connect-three-sparks, multi-sparks-through-switch), which Part 20 uses.

They matter for a specific reason. This machine is aarch64, and a large fraction of the Python and container ecosystem assumes x86. A playbook is NVIDIA stating “this combination works on this hardware”, which is a much stronger claim than a wheel existing on PyPI. When a Spark reader has a choice between a playbook and a general installation guide, the playbook is the better starting point and the general guide is the fallback.

The two inference playbooks worth comparing are trt-llm and vllm. The TensorRT-LLM one budgets “45-60 minutes for setup and API server deployment”, rates its own risk as medium because container pulls and model downloads fail on flaky networks, and opens TCP ports 8355 and 8356. The vLLM one estimates “30 MIN (longer on first run due to model download)” and points at the vLLM recipes site for copyable vllm serve commands rather than printing them. Both exist, both are supported, and this part’s lab uses the TensorRT-LLM one on Track S precisely because a documented playbook exists; readers who would rather stay with one serving engine for the whole course can substitute vLLM, which Part 9 teaches in depth.

Track N: the same stack on x86

On an NVIDIA desktop or laptop the software is the same and three things differ.

The wheels are ordinary. x86-64 is the architecture everything is built for first, so pip installation is a realistic alternative to containers in a way it is not on the Spark.

Memory is a hard ceiling. The Spark’s 128 GB is unified; a desktop card’s memory is not, and a model that does not fit does not run. That makes the quantisation matrix above more consequential on this track, not less: choosing FP8 or W4A16 is often the difference between a model fitting and not.

The architecture row is probably not Blackwell. Ada Lovelace and Ampere cards are still the common case, and their rows in the matrix are shorter. Check yours before planning around NVFP4.

Track S — NVIDIA DGX Spark

Your primary path. Start with the trt-llm playbook, confirm the container sees the GPU, then serve Qwen3-8B and keep the endpoint for this part’s lab. Record the container tag you actually pulled: it will not be the tag printed above for long.

Track X — AMD Ryzen AI Max+ 395Not supported

TensorRT-LLM targets NVIDIA GPUs. The AMD-native equivalents are in the next lesson.

Read the quantisation-matrix section anyway. The same lesson applies with different names: what your hardware accelerates decides which checkpoints are worth downloading.

Track M — Apple siliconNot supported

TensorRT-LLM targets NVIDIA GPUs; there is no macOS path.

The previous lesson is your native path. The comparison worth drawing is that both vendors solve the same problem the same way, by publishing a framework that assumes their own memory model.

Track N — NVIDIA desktop or laptop

Secondary path. Everything above applies, with pip installation as a realistic alternative to the container and with your card’s row of the quantisation matrix as the constraint that decides what you can run. If your card is Ampere or Ada, plan around FP8 and W4A16 rather than NVFP4.

Read a playbook as a stack specification

A playbook combines a model, hardware target, software versions, container and launch configuration. Copying only the final command drops the assumptions that make it meaningful. Read the prerequisites and identify which files are produced during preparation and which are downloaded as prebuilt artefacts.

Record an image digest as well as the human-readable tag, the checkpoint revision and the engine configuration. Check CPU architecture before pulling a large image. After launch, preserve logs showing the selected kernels or backend path and the configured limits. A service responding through a familiar API says little about how it was built underneath.

For an adoption decision, compare setup cost and steady-state service behaviour. A compiled or specialised path may be attractive for a stable model and workload while being inconvenient for frequent model changes. Use the same requests and acceptance criteria as your general engine baseline. If the supported configuration differs, explain that constraint rather than changing the baseline silently to make the specialised deployment look comparable.

Recap

TensorRT-LLM specialises a model for a GPU rather than running a portable file, and trtllm-serve puts an OpenAI-compatible API in front of the result on /v1/models, /v1/completions and /v1/chat/completions. Its quantisation support is per architecture, so the matrix decides which checkpoints your card can use; NVIDIA publishes pre-quantised checkpoints for the formats its hardware accelerates, and ModelOpt makes one where none exists. Distribution is through NGC containers, with --ipc=host a documented requirement, and on the aarch64 Spark the container is the path rather than a shortcut. A NIM takes the packaging one step further and adds a commercial licence. The DGX Spark playbooks are NVIDIA asserting that a specific combination works on this specific machine, which is why Track S starts there. Whether any of it beats llama-server on your workload is this part’s lab.

Check your understanding

Question 1. Why does the course teach TensorRT-LLM through NGC containers on the DGX Spark rather than through pip?

• A container runs faster than a native install
• The Spark is aarch64, and the NVIDIA container images are where the matching CUDA, driver and Python builds are already assembled for that architecture
• pip does not work on DGX OS
• The licence requires it

Show the answer and why

Answer: The Spark is aarch64, and the NVIDIA container images are where the matching CUDA, driver and Python builds are already assembled for that architecture

The architecture is the reason. On x86 the wheels are ordinary and pip is a realistic option; on arm64 the container is where the pieces are known to fit together, which is the same argument that made Part 6 build llama.cpp from source on this track.

Question 2. You have an Ampere card and download an NVFP4 checkpoint. What does the documented support matrix predict?

• It will run, more slowly than FP8
• It will not run: the Ampere row lists FP8 KV cache and W4A16 AWQ and GPTQ, not NVFP4
• It will run at the same speed as on Blackwell
• It will run only with --backend pytorch

Show the answer and why

Answer: It will not run: the Ampere row lists FP8 KV cache and W4A16 AWQ and GPTQ, not NVFP4

Quantisation support here is a hardware capability, not a software preference. This is the sharpest difference from llama.cpp, where any GGUF quantisation loads on any backend and only the speed changes.

Question 3. Which statements about NIM are supported by NVIDIA’s own documentation? Select all that apply.

• It is a set of microservices for deploying foundation models, delivered as containers
• It is part of NVIDIA AI Enterprise
• Access uses an NGC API key
• It is the same thing as TensorRT-LLM under a different name

Show the answer and why

Answer: It is a set of microservices for deploying foundation models, delivered as containers, It is part of NVIDIA AI Enterprise, Access uses an NGC API key

A NIM packages a model, a runtime and an API together; TensorRT-LLM is one of the runtimes such a package can contain. The distinction matters when you are deciding what you are actually adopting: a library, or a licensed product built on it.

Question 4. The TensorRT-LLM documentation says it was last updated on 27 August 2026 and names container tags carrying a 1.3 release candidate, while this course pins the 1.2.1 release. What should you do with a command from either source?

• Prefer the newest tag, always
• Check which version you are actually running before debugging anything, and record the tag you used with your results
• Ignore the version; the options are stable
• Use the course pin regardless of what is installed

Show the answer and why

Answer: Check which version you are actually running before debugging anything, and record the tag you used with your results

Most reported failures on this stack are version mismatches between a copied command and an installed build. The habit that prevents them is cheap: print the version, write it in the notebook next to the numbers, and treat every command found elsewhere as dated.

Check my answersClear