Installing vLLM: x86 CUDA, DGX Spark, ROCm and What Does Not Work

By the end of this lesson you will have vLLM answering an HTTP request on your machine, or a clear, sourced statement of why your track cannot run it and which engine you use instead. Both of those are successful outcomes. What is not acceptable is an hour lost to a compiler error that was never going to resolve.

The stack you are installing into

vLLM is a Python package that compiles against a specific accelerator runtime, which talks to a specific kernel driver, which drives specific silicon. Every install problem in this lesson is one of those layers disagreeing with the one below it.

What has to line up for vLLM to start

each layer needs the one below

1. Your clientcurl, the OpenAI SDK, LiteLLM, an editor, an agentspeaks HTTP only
2. vLLM OpenAI-compatible serverThe process you start with vllm serveyou configure this
3. vLLM Python packageScheduler, paged KV cache, compiled attention and GEMM kernelsmust match the runtime
4. PyTorch buildBuilt for one accelerator runtime: CUDA 12.x, CUDA 13.x or ROCmthe usual mismatch
5. Accelerator runtimeCUDA toolkit and libraries, or the ROCm stackversion-pinned
6. Kernel driverThe NVIDIA or AMD driver the operating system loaded at bootreboot to change
7. SiliconGB10, Ryzen AI Max+ 395, a GeForce or RTX card, or Apple siliconfixed

A container image is one pre-assembled slice of the four middle layers. That is the whole argument for using one: somebody else already made those four agree.

This is why the container path exists and why it is the recommended one on two of the four tracks. A container ships the vLLM package, the PyTorch build and the accelerator runtime already matched; your machine only has to supply a driver new enough for them.

⌚Version note

Everything below was read from the vLLM documentation on 2026-09-09, against vLLM 0.28.0 · verified 2026-09-08. vLLM moves quickly and its platform support changes between minor releases, so re-read the installation page rather than trusting a page of any age, including this one, if a command fails.

Install, by track

Track S — NVIDIA DGX Spark

Track S is aarch64 with a GB10 chip and unified memory, which is the awkward combination: most Python wheels in this ecosystem are built for x86-64 first. Use NVIDIA’s own container.

The NGC catalogue publishes a vLLM container as nvcr.io/nvidia/vllm:xx.xx-py3, where the version part follows NVIDIA’s monthly container calendar; the catalogue page listed 26.08-py3 as the newest tag when it was read on 2026-09-09, with monthly tags going back to 25.09-py3. The page states multi-architecture support, and mentions DGX Spark specifically in a troubleshooting note about memory caching rather than in the supported-platform list, so treat the container as the documented starting point and confirm the tag you pull actually runs on your machine before building anything on it.

NVIDIA also publishes a DGX Spark playbook, Serve LLMs with vLLM, on https://build.nvidia.com/spark. It lists DGX Spark with 128 GB of unified memory as a supported hardware platform and directs you to a container setup and a docker run workflow on its Instructions tab. That playbook, not a source build, is the path this course follows on Track S. Part 8 covers the same playbook collection for TensorRT-LLM.

RunnableTrack S · DGX Spark

pull the NVIDIA vLLM container

docker pull nvcr.io/nvidia/vllm:26.08-py3

Building vLLM from source on aarch64 is possible and is a long afternoon. Do it only if the container genuinely cannot do what you need, and record what you did, because you will be repeating it after the next upgrade.

Track X — AMD Ryzen AI Max+ 395

Track X is the track whose status changed. vLLM’s GPU installation page lists, among the GPUs supported by the ROCm build, “MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950), Radeon RX 7900 series (gfx1100/1101), Radeon RX 9000 series (gfx1200/1201), Ryzen AI MAX / AI 300 Series (gfx1151/1150)” (checked 2026-09-09). Your machine is gfx1151.

The same page gives the requirements as Linux, ROCm 6.3 or above, and Python 3.12 for the pre-built wheels, with wheel variants published for ROCm 7.0 and ROCm 7.2.1. The install is:

RunnableTrack X · Ryzen AI Max+

vLLM from the ROCm wheel index

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade

There is also an official ROCm image, vllm/vllm-openai-rocm:latest, documented on vLLM’s Docker page, which needs additional device and security flags that the NVIDIA image does not.

Two cautions. First, being named on a supported-GPU list is not the same as being a well-trodden path: an integrated GPU sharing system memory behaves differently from a discrete MI300, and the kernels that are fastest on one are not necessarily present for the other. Second, this listing is recent: earlier readings of the same page did not name the chip, and the course’s hardware and version data recorded it as unlisted until the reading of 2026-09-09. The documentation is the authority, so treat vLLM as worth trying here and llama-server from Part 6 as the path the course has actually run. Measure both in this part’s lab and record which one your machine preferred.

Track M — Apple siliconNot supported

vLLM's mainline GPU path does not cover Apple silicon. The documented macOS build is a CPU build, and the Metal path is a separate plugin outside the main repository.

Track M does not run vLLM in this course, and the reason is documented rather than inferred.

vLLM’s installation index lists “Apple Silicon” twice: once under GPU, qualified as available “via vLLM-Metal”, and once under CPU. The page also notes that third-party hardware plugins “live outside the main vllm repository”. The CPU installation page gives the macOS requirements as macOS Sonoma or later, Xcode 15.4 or later with Command Line Tools, and Apple Clang 15 or later, and then states three things that settle it: “Currently the CPU implementation for macOS supports FP32 and FP16 datatypes”, “Currently, there are no pre-built Apple silicon CPU wheels”, and “On macOS the VLLM_TARGET_DEVICE is automatically set to cpu, which is currently the only supported device”.

A source-built CPU engine that cannot use the GPU is not a serving engine for this course’s purposes, and the Metal plugin is a separate project that this course does not teach.

What you use instead. llama-server with parallel slots, from Part 6, which is a first-class path on this track and is measured against vLLM in this part’s lab. Or the mlx_lm.server from Part 8, which is Apple’s native engine. Every remaining page in this part works on Track M with one of those two behind it, and the gateway project at the end of this part is written to run natively on your machine.

Track N — NVIDIA desktop or laptop

Track N is the reference path and the one the vLLM documentation is written for. The GPU installation page gives the requirements as Linux, Python 3.10 to 3.13, CUDA 12.9 for the default binaries, and a GPU of “compute capability 7.5 or higher (e.g., T4, RTX20xx, A100, L4, H100, B200, etc.)”.

RunnableTrack N · NVIDIA GPU

vLLM into a fresh environment with uv

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

--torch-backend=auto lets uv pick the PyTorch build that matches the driver it can see, which is the single most common source of a broken install when done by hand.

The container alternative is vllm/vllm-openai:latest, and vLLM’s Docker page gives the run command in full:

RunnableTrack N · NVIDIA GPU

the official container, from the vLLM Docker page

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

Windows users: this is WSL2 territory, following the CUDA-on-WSL rules from Part 5. Install inside the Linux distribution, not on the Windows side.

✱Hardware note

--ipc=host in that container command is not decoration. The vLLM documentation explains that you need either it or --shm-size “to allow the container to access the host’s shared memory”, because “vLLM uses PyTorch, which uses shared memory to share data between processes under the hood, particularly for tensor parallel inference”. Leave it out and the server starts, then fails in a way that looks like a model problem.

Container or native?

Both work. The decision is about who owns the version matching.

Take the container when the accelerator runtime is awkward to install or upgrade, when you want the machine’s Python left alone, when you are on aarch64 or ROCm, or when you want to move the same setup to another machine later. The cost is a multi-gigabyte pull, a layer of indirection when something goes wrong, and mounting the model cache in so downloads are not repeated.

Install natively when you are on x86 with CUDA already working, when you want to read tracebacks without a container boundary, when you are going to patch or profile vLLM, or when you want the smallest possible startup time.

The course’s default is: container on Track S and where ROCm is involved, native with uv on Track N. Either way, put it in its own environment. vLLM pins a PyTorch build, and installing it into an environment that already has PyTorch is how a working training setup becomes a broken one.

Verify with a first serve

An install is not finished until a model has answered. Use a small model for this so the download is quick: Qwen3-1.7B is Apache-2.0 licensed and ungated, and the model reference records the licence.

RunnableAll tracks

serve a small model on localhost

vllm serve Qwen/Qwen3-1.7B \
    --host 127.0.0.1 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.80 \
    --served-model-name smoke-test

The vLLM quickstart states that the server runs at http://localhost:8000 by default, with --host and --port configurable. This course sets both explicitly on every serve command, because the default binds somewhere you should have chosen on purpose.

Startup is slow the first time and that is normal: the weights download, the engine profiles the accelerator to work out how many KV blocks it can afford, and it captures execution graphs. Wait for the line that says the server is up before you conclude anything.

Then ask it what it is serving:

RunnableAll tracks

the two checks that prove it works

curl -s http://127.0.0.1:8000/v1/models

curl -s http://127.0.0.1:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "smoke-test",
      "messages": [{"role": "user", "content": "Reply with the single word: ready"}],
      "max_tokens": 16
    }'

Output — what you should see

{"object":"list","data":[{"id":"smoke-test","object":"model", ... }]}

The first call proves the HTTP server is up and tells you the name clients must send. The second proves the engine loaded, the chat template rendered and a token came out the other end. If the first works and the second does not, the problem is the model or the template, not the install.

=Expected result

A successful start prints, among a great deal else, the number of KV cache blocks the engine allocated and the maximum concurrency that implies for your context length. That line is the memory arithmetic from the previous lesson, computed for you against your actual free memory. Read it: it is the single most useful line vLLM prints, and the next lesson is largely about changing it.

When it does not start

A CUDA or ROCm version mismatch, usually visible as an import error mentioning a missing symbol or a library. The PyTorch build and the runtime disagree. On Track N, reinstall with --torch-backend=auto into a clean environment; on Track X, check the wheel variant you pulled against your installed ROCm version; on Track S, use the container.

Out of memory during startup, before any request. vLLM reserves its KV cache at load. Lower --gpu-memory-utilization, lower --max-model-len, or both. The next lesson explains what each one actually does.

The model downloads and then fails to load. Either the architecture is not supported by this vLLM version, or the checkpoint is in a quantisation format this build has no kernel for. Check the architecture against vLLM’s supported-models list before assuming the install is at fault.

It works but uses the CPU. On a native install this means the PyTorch build has no accelerator support. Do not tune around it; reinstall.

Something on your machine is already on port 8000. Change --port. It is worth choosing a port per engine and writing it down; the gateway project at the end of this part assumes you have.

Verify the full import-to-request path

Installation succeeds in stages. A package manager can resolve dependencies while the accelerator extension cannot load. The extension can load while a model kernel is unsupported. A small checkpoint can generate while a requested quantisation or attention path fails. Preserve the first failing stage rather than treating all failures as an installation problem.

Record Python, framework, driver, engine and container identities together. Use a dedicated environment so another lab’s training dependencies cannot silently replace the serving stack. Confirm the CPU architecture and accelerator support for the chosen distribution before a large download or build.

Run a small documented model first, send a short completion, then test the application feature and intended context. Keep the startup log with device allocation and configuration. If the platform lacks the documented mainline path, use the stated alternative track and label it as such. A compatible client API does not turn that alternative into a vLLM hardware validation; it establishes a way to complete the application exercise with another engine.

Recap

vLLM sits on a stack of four things that must agree: the package, the PyTorch build, the accelerator runtime and the driver. A container pre-assembles the middle three, which is why it is the recommended path on Track S and a reasonable one on Track X. Track N installs natively with uv and --torch-backend=auto into its own environment. Track X is now named in vLLM’s own list of supported ROCm GPUs as gfx1151, which is newer information than this course’s version table carries, so try it and keep llama-server as the certain path. Track M does not run vLLM: the documented macOS build is a CPU-only build with no pre-built wheels, and the Metal path is a plugin outside the main repository, so Track M uses llama-server from Part 6 or the MLX server from Part 8. An install is verified when /v1/models lists your served name and a chat completion returns a token, and the KV-block line in the startup log is the memory arithmetic of the previous lesson done against your real machine.

Check your understanding

Question 1. Why does this course recommend a container for vLLM on Track S rather than a source build?

• Containers are faster at inference
• The container ships a matched vLLM, PyTorch and accelerator runtime for aarch64, which is the part that is awkward to assemble by hand on that architecture
• vLLM cannot be installed natively on Linux
• It uses less memory

Show the answer and why

Answer: The container ships a matched vLLM, PyTorch and accelerator runtime for aarch64, which is the part that is awkward to assemble by hand on that architecture

A container is a pre-matched slice of the middle layers of the stack. On x86 with CUDA those layers are easy to match with uv; on aarch64 they are not, and NVIDIA publishes both an NGC image and a DGX Spark playbook built around one.

Question 2. What does vLLM's documentation say about running it on Apple silicon?

• It is fully supported through Metal in the main package
• The documented macOS build is a CPU build with no pre-built wheels, supporting FP32 and FP16 only, and the Metal path is a separate plugin outside the main repository
• It works but only for models under 2B parameters
• It is supported on macOS Sonoma and later with no limitations

Show the answer and why

Answer: The documented macOS build is a CPU build with no pre-built wheels, supporting FP32 and FP16 only, and the Metal path is a separate plugin outside the main repository

The installation index lists Apple silicon under CPU and, under GPU, only "via vLLM-Metal", noting that hardware plugins live outside the main repository. The CPU page states that VLLM_TARGET_DEVICE is forced to cpu on macOS. Track M therefore uses llama-server or the MLX server.

Question 3. You start the official vLLM container without --ipc=host or --shm-size. What is the documented consequence?

• The container refuses to start
• The container cannot reach the host's shared memory, which PyTorch uses to pass data between processes, particularly for tensor parallel inference
• The model downloads twice
• The API listens on the wrong port

Show the answer and why

Answer: The container cannot reach the host's shared memory, which PyTorch uses to pass data between processes, particularly for tensor parallel inference

The Docker page names shared memory as the reason for the flag. The failure it produces looks like an engine or model fault rather than a container configuration fault, which is why it is worth knowing in advance.

Question 4. Your first serve fails with an out-of-memory error before any request is sent. What is the most likely cause?

• The prompt is too long
• vLLM reserves its KV cache at load time, so the reservation itself did not fit; lower the memory fraction, the maximum model length, or both
• The GPU driver is missing
• Too many clients connected at once

Show the answer and why

Answer: vLLM reserves its KV cache at load time, so the reservation itself did not fit; lower the memory fraction, the maximum model length, or both

A serving engine allocates its block pool up front so it has blocks to hand out. Nothing has to be requested for the reservation to fail. The two levers are how much of the device it may claim and how long a sequence it must be able to hold.

Check my answersClear