Serving with vLLM: Quantised Weights, Context, Memory and Multi-GPU

By the end of this lesson you will be able to start vLLM on a checkpoint you chose deliberately, set the context length and memory fraction from arithmetic rather than by trial and error, split a model across two GPUs, turn on the metrics endpoint, and read the startup log well enough to know what the engine decided on your behalf.

vllm serve --help prints several hundred options. Around a dozen decide anything.

The shape of the command

RunnableAll tracks

the serve command this course uses as a baseline

vllm serve Qwen/Qwen3-8B \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name local-chat \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 32

Qwen3-8B is Apache-2.0 licensed and ungated; the model reference records the licence for every model this course names. --served-model-name sets what clients put in their model field, and setting it to a stable name rather than a repository path is what makes the gateway project at the end of this part possible.

Everything else in this lesson is a variation on those six lines.

Quantised checkpoints

vLLM does not read GGUF as its normal diet. It reads safetensors checkpoints, and where those checkpoints are quantised, the format is one of a handful of schemes with kernels compiled into the engine. Its quantisation page carries a compatibility matrix, and the honest summary is that what runs depends on your GPU generation as much as on the format.

Format	What it is	Where it runs, per vLLM’s matrix (checked 2026-09-09)
AWQ	Activation-aware weight quantisation to four bits. vLLM describes it as reducing “the model’s precision from BF16/FP16 to INT4”, for “lower latency and memory usage”.	Turing onwards; not on AMD GPUs
GPTQ	Four- or eight-bit weight quantisation, served through the Marlin and Machete kernels	Volta onwards; not on AMD GPUs
FP8 (W8A8)	Eight-bit floating point weights and activations	Ada and Hopper; also AMD GPUs
NVFP4	NVIDIA’s four-bit floating point, from Model Optimizer checkpoints	Selected at load time from the available kernel backends; falls back to weight-only four bits via Marlin where there is no native support
GGUF	The llama.cpp format	Volta onwards, and AMD GPUs

Two details are worth carrying. First, FP8: vLLM states that “FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell)”, and that FP8 models will also “run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin”. The two formats are E4M3, “1 sign bit, 4 exponent bits, and 3 bits of mantissa”, and E5M2, with one more exponent bit and one less of mantissa. E4M3 is the one you will meet in checkpoints.

Second, online quantisation. vLLM can convert a BF16 or FP16 model to FP8 at load time “without any calibration data required”, which sounds free and is not: the page immediately adds that “latency improvements are limited in this mode”. Use it to fit a model that otherwise would not; do not use it expecting the speed of a properly quantised checkpoint.

RunnableTrack N · NVIDIA GPU

quantise to FP8 at load time, no calibration data

vllm serve Qwen/Qwen3-8B \
    --host 127.0.0.1 --port 8000 \
    --served-model-name local-chat \
    --quantization fp8 \
    --max-model-len 8192

A checkpoint that already carries its quantisation config is detected without --quantization, and that is the better path when a publisher has shipped one: an AWQ or FP8 build made with calibration data is what the kernels were designed for. Use the command above when no such build exists and the BF16 weights will not fit. Check the model card for what the publisher actually shipped before assuming either way.

✕Common mistake

A four-bit checkpoint for llama.cpp is not a four-bit checkpoint for vLLM. Q4_K_M is a GGUF quantisation with its own block structure and its own kernels. vLLM’s matrix does list GGUF, but the formats it is built around are AWQ, GPTQ, FP8 and the compressed-tensors family. Downloading a GGUF because it is small and expecting vLLM to be fast with it is a common and expensive mistake. Look for an AWQ or FP8 build of the same model instead.

Context length and the KV cache

--max-model-len is the longest sequence the engine will accept, prompt plus output. The engine arguments page gives its default as derived from the model config, which for a modern model means tens of thousands of tokens, and vLLM sizes its block pool so that it can serve at least one sequence of that length.

This is the same trap as -c in llama.cpp, with one difference: because the cache is paged, a long --max-model-len does not waste memory on short requests. It does still set a floor on how much cache must exist, and it interacts with concurrency, because the total block pool has to hold every running sequence at once.

The arithmetic from the first lesson applies unchanged. Qwen3-8B costs about 144 KiB per token of KV cache at FP16, so a block pool of 10 GB holds roughly 72,000 tokens: nine sequences at 8,192, or seventeen at 4,096.

--kv-cache-dtype changes the exchange rate. Its default is auto, meaning the model’s own dtype; setting it to an eight-bit type roughly halves what each token costs, at a quality cost you should measure rather than assume. Part 16 gives you the tools to measure it properly.

The memory fraction

--gpu-memory-utilization is the fraction of the device vLLM may claim, and the engine arguments page gives its default as 0.92. Everything the engine needs comes out of that fraction: weights, activations, compiled graphs and the KV block pool. What is left over is for everybody else on the device.

Qwen3-8B at eight bits on a 24 GB card, --gpu-memory-utilization 0.90 — estimates from the course model reference

Weights, eight-bit

8.7 GB

KV block pool

10.9 GB

Activations, graphs, engine overhead

2 GB

Free

2.4 GB

Total

24 GB

vLLM claims 21.6 GB of the 24 GB. Weights and overhead come out first; whatever remains becomes the KV block pool, and it is the pool that decides your concurrency. The 2.4 GB outside the fraction is what the display server and anything else on the card have to live in. Raising the fraction grows the pool and shrinks that margin; on a machine whose GPU also draws your desktop, it is the margin that stops the desktop from dying under load.

On a unified-memory machine, Tracks S, X and M, the fraction is a share of memory the operating system and every application are also using. Be more conservative there than you would be on a dedicated card, and remember that on Track X the GPU-visible share is capped below the machine total in the first place, as Part 5 explained.

For a model that does not quite fit, vLLM also has --cpu-offload-gb, documented as “Space in GiB to offload to CPU, per GPU”. The physics from Part 6 has not changed: what crosses the bus per token crosses it at bus speed. It buys “runs” rather than “runs well”.

▤Memory budget

When memory runs short at request time, vLLM does not fail; it preempts. The optimisation page shows the warning: “Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space”, and notes that “in vLLM V1, the default preemption mode is RECOMPUTE rather than SWAP”. Recompute means the evicted request’s prefill is thrown away and done again later. Seeing that line repeatedly in your log is the engine telling you that your concurrency and context do not fit; the documented remedies are to raise the memory fraction, lower max_num_seqs or max_num_batched_tokens, or add GPUs.

Batching knobs

--max-num-seqs caps how many sequences may be in flight at once. --max-num-batched-tokens caps how many tokens a single scheduler step may process, which is what chunked prefill uses to keep a long prompt from stalling the batch.

vLLM’s optimisation guide is unusually direct about the trade. Smaller values of the token budget, “e.g., 2048”, “achieve better ITL because there are fewer prefills slowing down decodes”; higher values “achieve better time to first token (TTFT) as you can process more prefill tokens in a batch”; and “for optimal throughput, we recommend setting max_num_batched_tokens > 8192 especially for smaller models on large GPUs”. Chunked prefill itself is “enabled by default whenever possible” in the V1 engine.

So: if your users complain that answers stutter when somebody else asks a long question, lower the token budget. If they complain that answers take too long to start, raise it. You cannot have both, and the lab in this part is where you find out which side of the trade your workload sits on.

Prefix caching

Automatic prefix caching is the feature from the first lesson: blocks are hashed by their tokens and their preceding prefix, kept in a pool, and reused by any later request whose prompt begins the same way. It is controlled by --enable-prefix-caching and its negation --no-enable-prefix-caching, with --prefix-caching-hash-algo selecting the hash. Check vllm serve --help on your installed version for the default rather than assuming it, because that default has changed across engine generations.

The gain is entirely workload-dependent, and it is large exactly where this course’s later parts live: a fixed system prompt, a repeated tool schema, a document being asked several questions. You measure it in this part’s lab by sending the same prefix twice and watching time to first token.

Tensor parallel across two GPUs

Track N is the track where this matters. vLLM’s parallelism guidance is short: if the model fits on one node with several GPUs, “use tensor parallelism. For example, set tensor_parallel_size=4”; across nodes, “set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes”.

RunnableTrack N · NVIDIA GPU

one model across two cards in one machine

vllm serve Qwen/Qwen3-32B \
    --host 127.0.0.1 --port 8000 \
    --served-model-name local-chat \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

Tensor parallel splits every layer across the devices, which means the devices talk to each other on every layer, which means the link between them is in the critical path. The same page adds a caveat worth remembering: “if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead”. Two consumer cards in PCIe slots are exactly that case, so measure both before deciding. Part 18 develops this properly and Part 20 measures it on real multi-GPU desktops.

--disable-custom-all-reduce exists for when the custom collective misbehaves on a particular topology. It is a debugging switch, not a tuning one.

The metrics endpoint

vLLM exposes Prometheus metrics on /metrics on the same server as the API. The metrics page gives the example as a plain fetch:

RunnableAll tracks

what the engine thinks it is doing, right now

curl -s http://127.0.0.1:8000/metrics

The names worth knowing are vllm:num_requests_running and vllm:num_requests_waiting, which tell you whether you are saturated or queueing; vllm:time_to_first_token_seconds and vllm:request_time_per_output_token_seconds, which are the two latency measurements from the first lesson; vllm:e2e_request_latency_seconds; vllm:kv_cache_usage_perc, which is your block pool filling up; vllm:prefix_cache_hits and vllm:prefix_cache_queries, whose ratio is your prefix cache actually working; and vllm:spec_decode_num_accepted_tokens_per_pos, which the speculative decoding lesson uses.

A waiting count that is persistently above zero while the running count sits at its cap means requests are queueing, which means --max-num-seqs or the block pool is your limit. A KV usage percentage pinned near its maximum means preemption is coming.

Reading the startup log

The most useful diagnostics vLLM produces happen before it serves anything, and none of them are in the documentation: they are printed by the engine at startup. The exact wording changes between releases, so read your own log rather than matching it against a page. What you are looking for is four statements, in roughly this order.

The model length it settled on. Not necessarily the one you asked for; the model’s own config can override an over-large request, and a silent adjustment here changes every memory number after it.

The size of the KV block pool, reported in tokens. This is the number the whole of the first lesson was about, computed against your machine’s actual free memory rather than against a specification.

The concurrency that pool implies at your context length. The engine does the division for you. If it reports a number smaller than the concurrency you intend to serve, stop and change something now: it will not improve when the requests arrive.

Graph capture. This is why startup takes as long as it does. --enforce-eager skips it, which the optimisation page describes as giving “the fastest possible startup, at the cost of steady-state decode performance”.

Copy those four values into your notebook every time you change a serving option. They are the cheapest possible record of what the engine decided on your behalf, and the lab on the next page asks for them.

The options that matter, and where the reference stands

⌚Version note

This course validates every command-line option against a captured --help run of the pinned tool version, and the entry for vLLM 0.28.0 · verified 2026-09-08 currently records only a subset of what the documentation describes. The runnable blocks on this page therefore use only recorded options; the ones named in the table below with a footnote are confirmed in the vLLM documentation as read on 2026-09-09 but are not yet in the course’s command reference. That reference is extended once the options are captured from the binaries on the validation machines, after which they will appear in runnable blocks here and on the command reference page.

Option	What it decides
--max-model-len	Longest sequence accepted; sets the floor on the block pool
--gpu-memory-utilization	Fraction of the device vLLM may claim; default 0.92
--kv-cache-dtype	Bytes per cached token; default auto
--max-num-seqs	Cap on sequences in flight
--tensor-parallel-size	Devices each layer is split across
--pipeline-parallel-size	Devices the layers are divided between
--quantization	Overrides the checkpoint’s own quantisation config
--served-model-name	The name clients send; keep it stable
--api-key	Requires a key in the request header
--enforce-eager	Skips graph capture: fast startup, slower steady state
--swap-space	CPU swap space in GB for the KV cache
--max-num-batched-tokens †	Token budget per scheduler step; the chunked-prefill trade
--enable-prefix-caching †	Prefix cache on or off, with --no-enable-prefix-caching
--cpu-offload-gb †	GiB per GPU offloaded to host memory

† Confirmed in the vLLM documentation on 2026-09-09; not yet in this course’s captured command reference, so it appears here in prose rather than in a runnable block.

Turn context and concurrency into an admission policy

The service must decide which requests to accept before memory exhaustion becomes the policy. Define the largest permitted input, response allowance, simultaneous work and queue wait. These interact: many short requests and a few long requests can have very different cache demands even when the number of clients is identical.

Test one short request, one maximum-sized permitted request and a mixed workload. Record successful responses and the server’s behaviour for an oversized request. An explicit rejection with a useful error is easier for a client to handle than an unexplained timeout after admission.

Keep the served model name stable for clients, but associate it with the exact checkpoint, template and launch configuration in your labbook. Test cancellation as well as completion if your client streams: closing a browser or timing out should not leave useful capacity occupied indefinitely. Do not infer reclamation from the client disappearing; observe server metrics or a subsequent controlled request. Capacity planning includes how the server handles work that does not finish normally.

Recap

The serve command is a model, a host and port, a served name, a context length and a memory fraction; everything else is a variation. vLLM’s native quantisation formats are AWQ, GPTQ, FP8 and the NVFP4 and compressed-tensors families rather than GGUF, and what runs depends on your GPU generation, with FP8 needing compute capability 8.9 for real FP8 arithmetic and falling back to weight-only below that. --max-model-len sets the floor on the block pool and --gpu-memory-utilization sets its ceiling; the pool divided by your context length is your concurrency, and vLLM prints exactly that number at startup. When the pool is too small the engine preempts and recomputes rather than failing, which shows up in the log as a warning you should act on. --max-num-batched-tokens is the chunked-prefill trade between time to first token and inter-token latency. Tensor parallel splits layers across cards and puts the interconnect in the critical path, so on machines without NVLink the documentation itself suggests trying pipeline parallel instead. And /metrics turns all of this into numbers you can watch while it happens.

Check your understanding

Question 1. You download a Q4_K_M GGUF of an 8B model and serve it with vLLM, expecting it to be fast. What is wrong with that plan?

• vLLM cannot read GGUF at all
• GGUF is a llama.cpp format with its own block structure; vLLM's matrix lists it, but the formats it is built around are AWQ, GPTQ, FP8 and compressed-tensors, so an AWQ or FP8 build of the same model is the right choice
• GGUF files are always larger
• vLLM requires BF16 weights

Show the answer and why

Answer: GGUF is a llama.cpp format with its own block structure; vLLM's matrix lists it, but the formats it is built around are AWQ, GPTQ, FP8 and compressed-tensors, so an AWQ or FP8 build of the same model is the right choice

Support and optimisation are different things. The compatibility matrix lists GGUF, but the kernels vLLM is built around are for the safetensors quantisation families. This is one of the most common ways people conclude that vLLM is slow.

Question 2. At startup vLLM reports a maximum concurrency of about nine for your requested context length of 8,192 tokens, and you intend to serve twenty concurrent users. What does that tell you?

• Nothing useful; the number is an estimate
• The block pool holds about nine sequences of that length, so twenty concurrent requests at 8,192 tokens will queue and be preempted; lower the context, raise the memory fraction, quantise the cache, or add a device
• You need to raise --max-num-seqs to twenty
• The model is too large for the GPU

Show the answer and why

Answer: The block pool holds about nine sequences of that length, so twenty concurrent requests at 8,192 tokens will queue and be preempted; lower the context, raise the memory fraction, quantise the cache, or add a device

That line is the block pool divided by your requested context. Raising the sequence cap without growing the pool changes nothing except how many requests are admitted before preemption starts. The four remedies all change either the pool or what each sequence takes from it.

Question 3. Users say answers begin quickly but stutter whenever somebody else pastes a long document. Which option does the documentation point at, and in which direction?

• Raise --gpu-memory-utilization
• Lower the per-step token budget, --max-num-batched-tokens, which the guide says gives better inter-token latency because fewer prefills slow down decodes
• Raise --max-model-len
• Set --enforce-eager

Show the answer and why

Answer: Lower the per-step token budget, --max-num-batched-tokens, which the guide says gives better inter-token latency because fewer prefills slow down decodes

Stutter during someone else's long prompt is prefill competing with decode inside a scheduler step. A smaller token budget chunks that prefill more finely. The cost is time to first token for the person who pasted the document, which is the trade the guide states explicitly.

Question 4. Which statements about --gpu-memory-utilization are correct? Select all that apply.

• It is the fraction of the device vLLM may claim, with a documented default of 0.92
• Weights, activations, compiled graphs and the KV block pool all come out of that fraction
• Whatever remains after weights and overhead becomes the KV block pool, which sets concurrency
• It limits how much memory a single request may use

Show the answer and why

Answer: It is the fraction of the device vLLM may claim, with a documented default of 0.92, Weights, activations, compiled graphs and the KV block pool all come out of that fraction, Whatever remains after weights and overhead becomes the KV block pool, which sets concurrency

The fraction is a claim on the whole device, not a per-request limit. Because weights and overhead are fixed, raising or lowering the fraction almost entirely changes the size of the block pool, which is why it is the first lever to reach for when concurrency is short.

Question 5. Two consumer GPUs in PCIe slots, no NVLink, and you want to serve a model too large for one. What does vLLM's own guidance suggest trying?

• Tensor parallel is always correct for a single machine
• Pipeline parallel is worth trying, because the documentation notes that without an NVLink interconnect it can give higher throughput and lower communication overhead than tensor parallel
• Run two servers and load-balance between them
• Offload half the model to the CPU

Show the answer and why

Answer: Pipeline parallel is worth trying, because the documentation notes that without an NVLink interconnect it can give higher throughput and lower communication overhead than tensor parallel

Tensor parallel puts a collective operation on the critical path of every layer, so the link speed matters enormously. The documentation names L40S as an example of the case and points at pipeline parallel. Measure both on your own pair rather than taking either on trust.

Check my answersClear