Multi-GPU Desktops: PCIe, Tensor Parallel Without NVLink and Expert Parallel

By the end of this lesson you will be able to look at a desktop with two graphics cards and say what the second one is actually buying, read the PCIe generation and link width the machine negotiated rather than the ones the box promised, choose between llama.cpp’s layer split and vLLM’s tensor split for a given model and pair of cards, say what expert parallelism does on one machine, work out what happens when the two cards are different sizes, and do the power and cooling arithmetic from the manufacturer’s own figures before spending money.

This is the Track N lesson of Part 20, and it is the one to read before a purchase rather than after one.

The link you actually have

Part 5 established the ceiling: the course hardware reference records that the GeForce 40 and 50 series have no NVLink, and NVIDIA’s own comparison page carries an NVLink and SLI row reading “No” for the RTX 4090 and 4080. Two cards in a desktop therefore talk to each other over PCIe, through the CPU’s root complex, and that link is the whole subject of this lesson.

Two properties of it are set by the machine rather than by the cards.

The generation. NVIDIA’s comparison page lists the RTX 5090 and 5080 as “Gen 5” and the RTX 4090 and 4080 as “Gen 4”. Each generation roughly doubles the per-lane rate of the one before it, so a Gen 5 card in a Gen 4 slot runs at Gen 4 rates, and a Gen 4 card in a Gen 3 board runs at Gen 3 rates. The card negotiates down silently.

The width. A slot is wired for some number of lanes, and the number on the motherboard box is frequently the physical connector rather than the electrical wiring. On many consumer boards the first slot is wired for sixteen lanes and the second for four or eight, and populating the second slot sometimes drops the first to eight as well because the processor has a fixed budget of lanes to divide.

You do not have to guess at either. nvidia-smi -q reports what NVIDIA’s documentation calls “GPU Link information: the PCIe link generation and bus width”, with a current and a maximum value, and the documentation adds a caution worth reading: the current figures “may be reduced when the GPU is not in use”, so take the reading under load rather than at idle. nvidia-smi topo -m prints the connection matrix between GPUs with a legend that distinguishes a connection traversing PCIe within a host bridge from one traversing the interconnect between processor sockets, which is how you find out that your two cards are on opposite sides of the machine.

RunnableTrack N · NVIDIA GPU

what the machine negotiated, not what the box promised

nvidia-smi --query-gpu=index,name,memory.total,pcie.link.gen.current,pcie.link.width.current --format=csv

✕Common mistake

A second card in a four-lane slot is the classic disappointing upgrade. The capacity is real: the model fits, and it would not have before. The throughput is not, if you chose a split that talks across the link on every layer. The card is not faulty and the driver is not misconfigured; the slot is doing exactly what the motherboard’s manual said it would, and nothing on screen mentions it. Check the negotiated width first, every time.

Two ways to split, and why they differ

Part 18 defined the strategies; here is what each engine actually does with two cards.

One box, two cards, three arrangements

1. llama.cpp layer split — the default-sm layer distributes layers and the KV cache across the cards. One small activation vector crosses PCIe at the boundary, per token. Tolerates a narrow slot.
2. llama.cpp row split-sm row splits weights across GPUs by rows, in parallel, with intermediate results and the KV cache held on the main GPU. More traffic, potentially more speed.
3. vLLM tensor parallel--tensor-parallel-size 2 splits every layer across both cards, with an all-reduce twice per layer. Fastest single conversation when the link can carry it; worst case on a narrow slot.
4. vLLM pipeline parallel--pipeline-parallel-size 2 is the layer split by another name, and vLLM recommends it explicitly on machines without NVLink.
5. Two independent serversOne model per card, both behind the Part 9 gateway. No cross-card traffic at all; twice the concurrency, no larger a model.

The first four are what the two engines offer for one model across two cards. Which is fastest is a property of the slot as much as of the software, which is why this page ends in a measurement rather than a recommendation.

llama.cpp exposes the choice as one flag. Its server README documents -sm, --split-mode {none,layer,row,tensor} as “how to split the model across multiple GPUs”, with none meaning “use one GPU only”, layer as the default which “split[s] layers and KV across GPUs (pipelined)”, row which “split[s] weight across GPUs by rows (parallelized)”, and tensor which “split[s] weights and KV across GPUs (parallelized, EXPERIMENTAL)”. The README marks the last one experimental, and this course treats it as such.

Two companion flags matter. -ts, --tensor-split takes a “comma-separated list of proportions, e.g. 3,1”, which is how you tell llama.cpp that your cards are not the same size. And -mg, --main-gpu selects “the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row)”, which is why the row split concentrates memory on one card rather than spreading it.

RunnableTrack N · NVIDIA GPU

a layer split weighted for a 24 GB card and a 12 GB card

llama-server \
    --model ~/models/the-model.gguf \
    --host 127.0.0.1 --port 8080 \
    --n-gpu-layers 999 \
    --split-mode layer \
    --tensor-split 2,1 \
    --ctx-size 8192

vLLM exposes the choice as two sizes. --tensor-parallel-size 2 splits every layer; --pipeline-parallel-size 2 deals the layers out. Part 9 quoted the guidance and it applies here without change: “if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead.” vLLM also names the case where pipeline parallel is the only option that works at all: “if the model fits within a single node but the GPU count doesn’t evenly divide the model size, enable pipeline parallelism, which splits the model along layers and supports uneven splits.”

RunnableTrack N · NVIDIA GPU

pipeline parallel across two cards, the arrangement vLLM suggests without NVLink

vllm serve Qwen/Qwen3-32B-AWQ \
    --host 127.0.0.1 --port 8000 \
    --served-model-name local-chat \
    --pipeline-parallel-size 2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

!Important

The reason tensor parallel over PCIe is worth trying anyway is that it is the only split that makes a single conversation faster. A layer split gives you capacity and, with several requests in flight, throughput; it does not make one user’s tokens arrive sooner, because only one card is working on that user’s token at a time. A tensor split has both cards reading their half of every weight simultaneously, which lifts the bandwidth ceiling Part 1 taught. Whether the collective on every layer costs more than the doubled read gains is exactly the sort of question a measurement answers in ten minutes and an argument never does.

Expert parallelism on one machine

For a mixture-of-experts model there is a third arrangement, and on one machine it is a flag rather than a project. vLLM’s expert-parallel page describes the single-node case as “enable EP by setting the --enable-expert-parallel flag”, with the size computed for you: “EP_SIZE = TP_SIZE × DP_SIZE”. The default communication backend, allgather_reducescatter, is described as “standard all2all using allgather/reducescatter primitives” and “general purpose, works with any EP+DP configuration”. The elaborate DeepEP backends, with their extra dependencies and their NCCL version requirement, are for the multi-node prefill and decode cases; on one desktop you do not need them.

What changes when you turn it on is which layers are shared and which are replicated. vLLM describes expert layers as “sharded across all EP ranks” while attention weights are “replicated across all DP ranks” when tensor parallel size is one. Without the flag, “MoE layers would use tensor parallelism (forming a TP group of size TP × DP), similar to dense models.”

The practical reading for a two-card desktop: a mixture-of-experts model whose experts dominate its size is the one case where two cards can hold something meaningfully larger than one, while the per-token traffic stays governed by how many experts a token visits rather than by the whole model. That is the same property that makes these models the interesting ones for home clusters at all, which Part 18 set out and Part 19 exploited.

Two cards that are not the same

Mixed sizes are the normal home case: the card you had, plus the card you bought. Both engines cope, differently.

llama.cpp’s --tensor-split takes proportions, so a 24 GB card and a 12 GB card are 2,1, and the engine puts twice as many layers on the larger one. Nothing else needs to change.

vLLM’s tensor parallelism assumes the devices are interchangeable, because every device holds an equal slice of every layer. Two cards of different sizes therefore behave as two copies of the smaller one, and the difference is wasted. Pipeline parallelism handles the imbalance better in principle, since layers can be dealt out unevenly, and it is what vLLM points at for uneven splits.

A 24 GB card and a 12 GB card, tensor-split evenly — estimate from the course model reference

Used on the 24 GB card

12 GB

Used on the 12 GB card

12 GB

Free

12 GB

Total

36 GB

An even split across unequal cards is limited by the smaller one, so 12 GB of the larger card cannot be used at all. This is an arithmetic consequence of the split, not a measurement: the same pair under llama.cpp's --tensor-split 2,1 uses all 36 GB. Check the actual figures with nvidia-smi under load before concluding either way.

✱Hardware note

Before mixing generations, check that both cards support the quantisation you intend to run. Part 8 printed TensorRT-LLM’s per-architecture matrix and Part 9 printed vLLM’s; the rule in both is that an older card is not slower at a format its architecture does not implement, it simply cannot use it. A pair consisting of one Ada card and one Ampere card is constrained by the Ampere row.

Power, cooling and the arithmetic to do first

This is the part that specification sheets answer directly and that people skip. NVIDIA publishes, for every card, a total graphics power and a required system power.

Vendor specification, not measuredPower figures NVIDIA publishes for cards a two-card desktop might use

Card	Total graphics power (W)	Required system power (W)	PCI Express
GeForce RTX 5090	575	1,000	Gen 5
GeForce RTX 5080	360	850	Gen 5
GeForce RTX 4090	450	850	Gen 4
GeForce RTX 4080	320	750	Gen 4
RTX PRO 6000 Blackwell	600	not stated on the product page	Gen 5

the cards named in the first column, not applicable · no engine; NVIDIA comparison and product pages retrieved 2026-09-09 · no model loaded, not applicable · 0 tokens of context · 2026-09-09

Specification figures published by NVIDIA, not measured by this course. Required system power is NVIDIA's recommendation for a system with one such card. The arithmetic for a second card is in the prose below and is an estimate, not a vendor recommendation.

The arithmetic is a single addition and it is worth doing before the order is placed. NVIDIA’s required system power for one card assumes one card; a second identical card adds its own total graphics power to the load, so for two RTX 4090s the arithmetic gives roughly thirteen hundred watts of supply, and for two RTX 5090s roughly sixteen hundred. Those are estimates from the vendor’s figures rather than NVIDIA recommendations, and a supply chosen with no headroom above them will shut the machine down under a sustained load rather than during the benchmark you used to test it.

Three physical constraints go with the electrical one, and none of them appears in a specification table.

Slot spacing. Two triple-slot cards need a board whose second usable slot is far enough from the first, and a case deep enough for both. Adjacent cards starve each other of air.

Sustained thermals. Inference is not a benchmark. A long generation run keeps both cards at load for hours, and the card nearest the other one runs hotter. nvidia-smi reports why clocks dropped: its documentation lists SW Power Cap, meaning “SW Power Scaling algorithm is reducing the clocks below requested clocks because the GPU is consuming too much power”, and HW Thermal Slowdown, “reducing the core clocks by a factor of 2 or more due to temperature being too high”. Read those before concluding that a model got slower.

Noise and heat in the room. A kilowatt of continuous draw is a room heater, and it is the reason a good number of home two-card builds end up in a cupboard with a duct.

✦Tip

Power limiting is a real tool here, not a compromise. nvidia-smi -pl “specifies maximum power limit in watts”, requires root, and takes a value “between Min and Max Power Limit as reported by nvidia-smi”. Capping both cards below their rated draw often costs very little decode speed, because decode is bandwidth-bound rather than compute-bound, and it can be the difference between a build that runs quietly for hours and one that trips a supply. Measure it: cap, rerun the same load test, and compare the two notebook lines.

What the second card is actually for

Pulling the threads together, in the order a purchase decision should consider them.

Capacity is the reliable gain. Two cards hold a model neither could hold alone, in every arrangement, on every link. If the reason for the second card is that a 32B-class model at four bits does not fit in 24 GB with the context you want, it will fit in two of them, and the split you choose barely matters.

Throughput is the likely gain. More cards, more requests in flight, whether you split one model across them or run two servers behind the gateway.

Single-conversation speed is the uncertain gain. It needs a tensor split, which needs the link to carry a collective on every layer, which is exactly what a consumer desktop without NVLink is worst at. It may still win on a Gen 5 slot at full width. It will very likely lose on a Gen 3 slot at four lanes. That is what the lab in this part measures on the two-card fallback path.

Nothing here is a substitute for memory bandwidth. Part 1’s arithmetic has not changed: decode speed is bounded by how fast the weights can be read. Two cards read their own memory in parallel under a tensor split, and read it in sequence under a layer split. Under a layer split, adding a card adds capacity without adding speed, and that is the correct expectation to have.

Compare model splitting with two independent replicas

If the model fits on one card, two replicas can increase service capacity without cross-card communication for each token. A split model may reduce per-device memory pressure but introduce synchronisation. Evaluate both designs under the same arrival pattern and quality requirements before assuming a split is faster.

Record physical topology: link type, PCIe path, device memory and whether another workload shares the bus. Two cards in one case do not necessarily have a high-bandwidth peer path. Mixed cards can make the slower device determine a synchronised stage’s progress.

Measure single-request latency, aggregate throughput, tail latency and total power after warm-up. Also observe host memory and cooling under sustained load. If splitting makes a larger checkpoint possible, report the capacity benefit separately from the replica comparison. A useful deployment can reserve one card for embeddings or a draft model, so the alternatives are broader than “one model across all cards”. Allocate devices according to the bottleneck measured in the complete application.

Recap

Two consumer cards in one desktop communicate over PCIe, with no NVLink on the recent GeForce generations, and the two properties that decide what that link can do are the generation NVIDIA lists for each card and the lane width the motherboard actually wired. Both are readable with nvidia-smi rather than guessable, and the current values should be read under load.

llama.cpp splits by layer by default, with --split-mode offering none, layer, row and an experimental tensor, and --tensor-split handling unequal cards as proportions. vLLM splits by tensor or by pipeline, recommends pipeline parallel on machines without NVLink, and needs pipeline parallel when the split is uneven. For a mixture-of-experts model, --enable-expert-parallel on one node shards experts instead of tensor-splitting them, with the default communication backend and none of the multi-node dependency work.

The gains, in order of certainty, are capacity, then throughput, then single-conversation speed, and the last one is the only one that depends on the link being fast. Before buying, add the second card’s total graphics power to NVIDIA’s required system power for the first, check the slot spacing and the airflow, and know that nvidia-smi -pl can cap both cards at a draw the machine can sustain for a cost you can measure.

Check your understanding

Question 1. You add a second RTX 4090 to a desktop, split a model with vLLM at tensor parallel size two, and get less throughput than one card gave on a smaller model. What should you check first?

• Whether the driver supports two cards
• The negotiated PCIe generation and link width with nvidia-smi, since a second slot wired for four lanes makes a per-layer collective the bottleneck
• Whether the model is quantised
• The power supply

Show the answer and why

Answer: The negotiated PCIe generation and link width with nvidia-smi, since a second slot wired for four lanes makes a per-layer collective the bottleneck

Tensor parallelism runs an all-reduce on every layer, so the link is on the critical path of every token. A narrow or older slot does not error, it just makes that collective slow. nvidia-smi reports the current link generation and width; read it under load, because the documentation notes the current values can be reduced when the GPU is idle.

Question 2. Which llama.cpp split mode is the default, and what does it move across the link?

• row; whole weight matrices
• layer; it splits layers and the KV cache across GPUs in a pipelined arrangement, so one small activation crosses at the boundary per token
• tensor; every layer's partial sums
• none; nothing crosses

Show the answer and why

Answer: layer; it splits layers and the KV cache across GPUs in a pipelined arrangement, so one small activation crosses at the boundary per token

The server README lists layer as the default and describes it as splitting layers and KV across GPUs, pipelined. That is why llama.cpp is comfortable on hardware where a tensor split would be painful, and it is the same property that makes its RPC backend work over ordinary Ethernet in Part 19.

Question 3. You have a 24 GB card and a 12 GB card. Which statements are correct? Select all that apply.

• llama.cpp can weight the split with --tensor-split 2,1 so both cards are used in proportion
• vLLM tensor parallelism splits every layer equally, so the pair behaves like two 12 GB cards
• vLLM points at pipeline parallelism for splits that do not divide evenly
• Mixing sizes is impossible

Show the answer and why

Answer: llama.cpp can weight the split with --tensor-split 2,1 so both cards are used in proportion, vLLM tensor parallelism splits every layer equally, so the pair behaves like two 12 GB cards, vLLM points at pipeline parallelism for splits that do not divide evenly

Mixed sizes work; what differs is how much of the larger card you get to use. A proportional layer split uses all of it, an equal tensor split uses as much of it as the smaller card has, and pipeline parallelism sits between the two because layers can be dealt out unevenly.

Question 4. NVIDIA lists the RTX 4090 at 450 W total graphics power and 850 W required system power. What supply should you plan for two of them?

• 850 W, because that is what NVIDIA states
• Roughly 1300 W by arithmetic - the stated requirement for one card plus the second card's total graphics power - with headroom above that, since NVIDIA states the requirement for a single-card system
• 900 W, twice the card power
• It does not matter; the cards throttle

Show the answer and why

Answer: Roughly 1300 W by arithmetic - the stated requirement for one card plus the second card's total graphics power - with headroom above that, since NVIDIA states the requirement for a single-card system

The published requirement already includes the rest of the machine, so the increment for a second identical card is its own draw. The result is an estimate from vendor figures rather than a vendor recommendation, and a supply with no margin above it fails under sustained load rather than during a short benchmark.

Question 5. A long generation run gets slower after twenty minutes on a two-card desktop. Where does the explanation come from?

• The model degrades over time
• nvidia-smi's clock throttle reasons, which distinguish SW Power Cap from HW Thermal Slowdown, so you learn whether the limit was the power budget or the temperature
• The context window filled up
• PCIe renegotiated

Show the answer and why

Answer: nvidia-smi's clock throttle reasons, which distinguish SW Power Cap from HW Thermal Slowdown, so you learn whether the limit was the power budget or the temperature

Both are documented throttle reasons with distinct fixes: a power cap points at the supply or at nvidia-smi -pl, and a thermal slowdown points at slot spacing and airflow. Reading the reason turns a vague slowdown into a specific, fixable cause, and it belongs in the notebook next to the number that dropped.

Check my answersClear