NVIDIA Desktops and Laptops: VRAM Tiers, CUDA and WSL2

By the end of this lesson you will be able to look at any NVIDIA card and say which tier of this course’s labs it can follow; install the driver and CUDA toolkit on Linux the way NVIDIA documents it; make the Windows decision once, with the one rule about WSL2 that everything else depends on; and predict what happens when a model is slightly too large for the card, which is the most common situation on this track.

This is the Track N lesson, and it is the track most readers are on.

The ladder

Track N is the only one of the four where the accelerator’s memory is separate from the machine’s, fixed at manufacture, and offered in a ladder of sizes. Knowing where your card sits on that ladder tells you almost everything about which pages of this course you can follow on their primary path.

Vendor specification, not measuredVRAM by card, from NVIDIA's own comparison pages

Card	Memory as published	Course tier
RTX PRO 6000 Blackwell	96 GB GDDR7 with ECC	96 GB
GeForce RTX 5090	32 GB GDDR7	32 GB
GeForce RTX 4090	24 GB GDDR6X	24 GB
GeForce RTX 3090	24 GB GDDR6X	24 GB
GeForce RTX 5080 / 5070 Ti	16 GB GDDR7	12-16 GB
GeForce RTX 4080 / 4080 SUPER / 4070 Ti SUPER	16 GB GDDR6X	12-16 GB
GeForce RTX 5070	12 GB GDDR7	12-16 GB
GeForce RTX 4070 / 4070 SUPER / 4070 Ti	12 GB	12-16 GB
GeForce RTX 5060 Ti / 4060 Ti	16 GB or 8 GB	8 GB or 12-16 GB
GeForce RTX 5060 / 4060 / 5050	8 GB	8 GB

the cards named in the first column, not applicable · no engine; NVIDIA comparison and product pages retrieved 2026-09-09 · no model loaded, not applicable · 0 tokens of context · 2026-09-09

Memory sizes as published by NVIDIA; the RTX 3090 row comes from the course hardware reference, as the current comparison page covers the 40 and 50 series. Laptop parts sold under these names generally have less memory and lower bandwidth than their desktop namesakes: check the specification for the machine, not for the name. Bandwidth figures for these cards are in the first lesson of this part.

Two things about that ladder shape the rest of Level 2.

Eight gigabytes is a real tier, not a consolation prize. It runs 8B-class models at four-bit quantisation with a useful context, which is enough for every Level 1 and Level 2 lab on its reduced path, and this course states a memory floor on every hands-on page precisely so that an 8 GB reader always knows where they stand.

Twenty-four gigabytes is where the shape of the course changes. At that tier a 32B-class model at four bits becomes possible, QLoRA fine-tuning of an 8B model fits, and the distillation teacher in Level 3 has somewhere to live. The hardware reference lists what each tier unlocks, and every lab states its floor.

Driver and toolkit on Linux

NVIDIA splits this into two documents, and the split confuses people, so here it is plainly: the driver is one installation and the CUDA toolkit is another. Many readers need only the first, because PyTorch wheels and prebuilt engine binaries bring their own CUDA runtime libraries with them. You need the toolkit when you are compiling something, which on this track means building llama.cpp in Part 6 or anything with custom kernels in Level 3.

NVIDIA’s CUDA Installation Guide for Linux documents CUDA 13.3 at the time this lesson was written and lists Ubuntu 22.04 LTS, 24.04 LTS and 26.04 LTS among its supported distributions. Its pre-installation checks are worth running before anything else, because each of them fails in a way that is otherwise mysterious later:

RunnableTrack N · NVIDIA GPU

the guide's pre-installation checks

lspci | grep -i nvidia
gcc --version
hostnamectl

The network-repository installation is three steps: install NVIDIA’s keyring package for your distribution and architecture, refresh the package lists, and install the toolkit metapackage. The guide’s post-installation step adds the toolkit’s bin directory to your PATH, with the CUDA version in the directory name.

Fragment — not complete on its own

# The keyring package URL contains your distribution and architecture; take the
# exact URL from the installation guide rather than from any tutorial.
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit

The guide notes that driver installation instructions now live in a separate driver installation guide rather than in the toolkit guide. On most distributions the packaged driver is the least troublesome route, and the only test that matters is whether nvidia-smi reports your card.

RunnableTrack N · NVIDIA GPU

the only driver test that matters

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

✕Common mistake

Installing the CUDA toolkit in the hope of fixing “PyTorch cannot see my GPU” is the most common wasted afternoon on this track. If nvidia-smi works and PyTorch does not, the toolkit is not the problem: the wheel was built against a CUDA newer than the driver supports, and the fix is a different wheel, not more toolkit. If nvidia-smi does not work, nothing above it will, and the driver is the only thing to look at.

Windows, answered once

Three arrangements are possible on a Windows machine, and this course picks one.

Linux, natively. Everything in the course works. If the machine can be dedicated to this, it is the least friction overall.

Windows, natively. llama.cpp, Ollama, LM Studio and the desktop tooling of Part 7 all run. Training frameworks and several serving engines do not, or do so awkwardly. This is a good place to be for Level 2 and a poor one for Level 3.

Windows with WSL2. A Linux userspace with GPU access, which is what this course’s Windows path assumes. Almost every Linux instruction then works unchanged.

The CUDA on WSL user guide has one rule that everything else depends on, and it states it twice. Install “NVIDIA GeForce Game Ready or NVIDIA RTX Quadro Windows 11 display driver on your system”, and then: “This is the only driver you need to install. Do not install any Linux display driver in WSL.” Elsewhere the same guide says that “users must not install any NVIDIA GPU Linux driver within WSL 2”. The Windows driver is projected into the Linux environment for you; installing a Linux driver on top of it breaks the projection.

The trap the guide warns about is that the obvious CUDA installation does exactly the forbidden thing, because the usual metapackages pull a driver in with them. In its words: “DO NOT choose the cuda, cuda-12-x, or cuda-drivers meta-packages under WSL 2”, and instead “Install the cuda-toolkit-12-x metapackage only” — that is, the versioned toolkit metapackage matching the release you want, and nothing that carries a driver.

!Important

The whole of the Windows path reduces to two sentences. Install the NVIDIA driver on Windows and nothing GPU-related inside WSL except a toolkit metapackage that does not include a driver. Then run nvidia-smi inside the WSL2 terminal: if it lists your card, you are done, and every Linux instruction in this course applies to you.

The guide is also honest about the limits, and two of them matter for this course. Pinned system memory is limited under WSL2, and the guide notes that some deep-learning training workloads can exceed that limit and may not work. And full managed-memory support is not available, so anything relying on CUDA unified memory behaves differently from native Linux. Neither stops Level 2. Both are worth knowing before you conclude that a training script is broken.

Track N, and where WSL2 inserts itself

what you install

1. Engines, frameworks and the course scriptsInstalled inside WSL2, exactly as on native Linux.yours
2. CUDA toolkit, if you are compilingInside WSL2, from the versioned toolkit metapackage only. Never a metapackage that carries a driver.yours
3. WSL2 Linux userspaceUbuntu by default. Sees the GPU through the projection, not through a driver of its own.
4. The projection layerWhat makes the Windows driver visible to Linux processes. Nothing to install and nothing to configure.do not touch
5. NVIDIA Windows display driverInstalled on Windows. The only driver on the machine.
6. The graphics card and its VRAMFixed capacity, high bandwidth, on the far side of PCIe from system memory.

On native Linux the middle layer is absent and everything else is the same. The rule about not installing a Linux driver exists because the driver layer is already occupied by the Windows one, projected through.

When the model does not fit

This is the situation Track N readers meet most, and it deserves a picture rather than a warning.

Take Qwen3-32B, published by Alibaba under the Apache-2.0 licence according to its model card, at Q4_K_M, with a full 32,768-token context. The model reference gives the weight size and the key-value cache cost per token; multiply the second by the context length and add.

Qwen3-32B at Q4_K_M with 32k of context, on a 24 GB card

24 GB budget

Weights, Q4_K_M

19.8 GB

Key-value cache, 32,768 tokens at FP16

8.6 GB

Requested

28.4 GB

Machine budget

24 GB

Over budget. 28.4 GB requested against a 24 GB machine - 4.4 GB over. Something here has to shrink: a smaller quantisation, a shorter context, or fewer of these reservations at once.

Figures from the course model reference. The weights alone leave room; the full context does not. This is the characteristic Track N failure: not a model that is too big, but a model plus the context you asked for.

Every engine offers the same three ways out, and they are not equivalent.

Ask for less context. The cheapest fix and usually the right one. Halving the context roughly halves the cache, and most work does not need the model’s maximum.

Quantise the cache. Storing keys and values at eight bits rather than sixteen roughly halves the same figure, at a quality cost that Part 6 shows you how to measure rather than assume.

Offload layers to system memory. The engine keeps some layers on the card and the rest in system RAM. It works, and it is slow, for the reason the first lesson of this part gave: every offloaded weight crosses PCIe on every token, and PCIe carries a fraction of what the card’s own memory does. The decode rate does not degrade gracefully as you offload; it falls off a cliff at the point where the offloaded fraction starts to dominate the time. Part 6’s challenge page has you find that cliff deliberately, because recognising it in a log is a skill.

▤Memory budget

The order to try them in is context, then cache quantisation, then a smaller quantisation of the weights, and only then offload. Offloading is the option that changes the machine’s performance class rather than the model’s quality, and most readers reach for it first because it is the one the engine offers in a flag.

More than one card

Two cards give you more VRAM and, with some engines, more speed. What they do not give you on this track is a fast link between them: the course hardware reference records that the 40 and 50 series have no NVLink, so cards communicate over PCIe.

That matters because the two ways of splitting a model across devices have very different appetites for that link. Pipeline parallel puts different layers on different cards and passes one small activation between them per token, which PCIe handles comfortably. Tensor parallel splits every layer across cards and exchanges data within each layer, which wants a link far faster than PCIe. Part 19 develops both properly. The practical summary for a two-card desktop is that adding a second card reliably buys capacity, and buys throughput only in the arrangements that tolerate a slow link.

✱Hardware note

Before buying a second card, check the physical realities that specifications do not mention: the number of PCIe slots that are actually wired for full width, the power supply’s headroom for two cards at their rated draw, and the case airflow when two cards are adjacent. The RTX PRO 6000 Blackwell is listed by NVIDIA at 600 W on its own. Two enthusiast cards in a desktop case is a power and thermal decision as much as a compute one.

Laptops

A laptop with an NVIDIA GPU is a Track N machine with three qualifications, and they compound.

The memory is smaller than the desktop card of the same name, often by a whole tier. The bandwidth is lower, as the course hardware reference notes for laptop parts. And the power budget moves: a laptop GPU’s sustained clock depends on the power limit the manufacturer chose and on whether the machine is plugged in, so the same script can produce noticeably different numbers on battery and on mains, and again after ten minutes when the chassis is warm.

None of that stops the course. It does mean that a laptop reader should record the power state and the run duration alongside every measurement, which is a field the lab notebook in the next lab has for exactly this reason, and should expect the second and third runs of a benchmark to be slower than the first.

Distinguish device memory, host memory and the power envelope

A discrete GPU has a local memory budget. Host RAM can store offloaded layers or staging buffers, but those bytes cross an interconnect or are computed on the CPU. Adding host RAM can make a larger workload possible while leaving it slower than a smaller fully resident model.

On laptops, repeat the same request under the power conditions in which you will use the machine. Record whether it is plugged in, the power profile and competing graphics work. A brief cold-device result and a sustained thermally limited result answer different questions. Do not compare them as if only the checkpoint changed.

For WSL, distinguish the host GPU driver from userspace libraries installed inside Linux. Follow the platform setup lesson and verify execution inside the environment where the server actually runs. An accelerator visible on the Windows host does not prove the Linux application uses it. Keep one small smoke workload and its startup log; it is the control you rerun after changing a driver, container or Python environment.

Recap

Track N is discrete memory: a fixed amount of fast VRAM on the card, with everything else on the far side of PCIe. The card’s memory size decides which tier of the course you can follow, from 8 GB up to 96 GB on the workstation Blackwell part, and laptop parts sit a tier below their desktop namesakes.

On Linux, the driver and the CUDA toolkit are separate installs and most readers need only the driver, with nvidia-smi as the single test that matters. On Windows the course uses WSL2, where the one firm rule is that the NVIDIA driver is installed on Windows and never inside WSL, and the toolkit metapackage chosen must be the one that does not carry a driver. The characteristic failure on this track is a model plus its context exceeding the card, and the ways out are less context, a quantised cache, smaller weights and, last, offloading to system memory, which changes the machine’s performance class. A second card adds capacity over PCIe, with no NVLink on the recent consumer generations, so it helps throughput only in the splitting arrangements that tolerate a slow link.

Check your understanding

Question 1. On Windows with WSL2, where do you install the NVIDIA driver?

• Inside WSL2, with apt
• On Windows only; the CUDA on WSL guide states that no NVIDIA GPU Linux driver may be installed within WSL 2
• In both places, so that each environment has its own
• Neither; the toolkit provides it

Show the answer and why

Answer: On Windows only; the CUDA on WSL guide states that no NVIDIA GPU Linux driver may be installed within WSL 2

The Windows driver is projected into the WSL2 environment. Installing a Linux driver inside WSL breaks that projection, which is why the guide also warns against the metapackages that would pull one in as a dependency.

Question 2. A 24 GB card loads Qwen3-32B at Q4_K_M happily with a short context, then fails when the context is raised to the model maximum. What happened?

• The weights grew
• The key-value cache grows with context length, and weights plus cache exceeded the card even though the weights alone did not
• The card overheated
• Q4_K_M does not support long contexts

Show the answer and why

Answer: The key-value cache grows with context length, and weights plus cache exceeded the card even though the weights alone did not

Weights are fixed; the cache is bytes-per-token times context length. This is the characteristic Track N budget failure, and the first thing to try is a smaller context rather than offloading.

Question 3. Rank these responses to "the model does not fit" from least to most damaging to performance.

• Offload layers, then quantise the cache, then reduce context
• Reduce context, then quantise the key-value cache, then use smaller weights, then offload layers to system memory
• Buy a second card, then offload, then reduce context
• They are all equivalent

Show the answer and why

Answer: Reduce context, then quantise the key-value cache, then use smaller weights, then offload layers to system memory

The first three trade a little quality or capability for memory and keep the model on the card. Offloading keeps the quality and moves weights across PCIe, which changes the machine's performance class rather than the model's.

Question 4. Which statements about two consumer cards in one desktop are correct? Select all that apply.

• They communicate over PCIe, with no NVLink on the recent consumer generations
• Pipeline parallelism tolerates the slow link better than tensor parallelism does
• Two cards always double decode speed
• Power supply headroom and case airflow are part of the decision

Show the answer and why

Answer: They communicate over PCIe, with no NVLink on the recent consumer generations, Pipeline parallelism tolerates the slow link better than tensor parallelism does, Power supply headroom and case airflow are part of the decision

Capacity adds reliably; speed depends on how the model is split and how much the arrangement talks across the link. Tensor parallelism wants a fast interconnect, which a PCIe-only desktop does not have.

Question 5. A laptop produces a good benchmark result, then a slower one on the second run and slower still on the third. What should be recorded?

• Only the best result, since the others are noise
• The power state, the run duration and the order of runs, because sustained clocks on a laptop depend on power limits and chassis temperature
• Nothing; laptop results are not comparable
• Only the average

Show the answer and why

Answer: The power state, the run duration and the order of runs, because sustained clocks on a laptop depend on power limits and chassis temperature

That pattern is thermal and power behaviour, not measurement error. It is real information about what the machine will do during a long job, and the lab notebook has fields for it.

Check my answersClear