Why Memory, Not FLOPS, Decides What You Can Run

By the end of this lesson you will be able to look at any machine, including one you are thinking of buying, and answer the only two hardware questions this course cares about: how much memory can the accelerator actually reach, and how fast can it read that memory. You will be able to say what unified memory is and what it costs, what crosses the PCIe bus on a machine with a graphics card and why that matters more than it sounds, and which of the four platform tracks your machine belongs to.

The two questions

Everything a machine does with a language model divides into two jobs, and the previous part named them: prefill, reading the prompt, and decode, writing the answer one token at a time. They stress different parts of the machine, and each of them is governed by one of two numbers.

Capacity decides what fits. A model’s weights have to be somewhere the accelerator can read at full speed while it runs. Add the key-value cache, which grows with the length of the conversation, and the activations of whichever layer is executing, and you have the memory budget Part 4 taught you to compute. If the total exceeds what the accelerator can reach, one of three things happens: the load fails, the engine quietly moves part of the model somewhere slower, or you reach for a smaller quantisation. All three are decisions, and the first thing to know about a machine is which of them you will be making.

Bandwidth decides how fast it decodes. Generating one token reads every weight the model uses for that token, once. The arithmetic done with each weight is a single multiply-and-add, which any modern accelerator finishes long before the next bytes arrive. So the token rate has a ceiling: the machine’s memory bandwidth divided by the bytes that have to be read per token. Nothing an engine does moves that ceiling. Engines can approach it, waste it, or dodge it by reading fewer bytes, and the whole of Level 2 is about the difference.

Peak floating-point throughput, the trillions of operations per second that appear at the top of every specification sheet, governs the third job: prefill, and training. It matters. It is just not what limits the thing most readers do most of the time, which is watch a model type at them.

!Important

Two machines with the same memory capacity and very different bandwidth will run exactly the same set of models at very different speeds. Two machines with the same peak operations per second and different bandwidth will read a long prompt at similar speeds and generate at different ones. When someone tells you a machine is fast at AI, the useful reply is “at prefill or at decode?”.

Where the memory is, and what stands between it and the chip

Memory is not one thing. On every machine there is a hierarchy, and the level that matters is the one the accelerator reads at full speed while computing. The four tracks split into two designs here, and the split is the single most important architectural fact in this part.

Discrete memory: a desktop or laptop with an NVIDIA graphics card (Track N)

further from the compute

1. GPU computeTensor cores and shader cores. Reads and writes VRAM.
2. VRAM on the cardGDDR6, GDDR6X or GDDR7. The full-speed level. Fixed at purchase and not upgradable.
3. PCIe linkThe only path between the card and the rest of the machine. Every byte the GPU needs from system memory crosses it, twice if it is written back.the bottleneck
4. System memoryDDR5 on the motherboard. Large and cheap, and reachable by the CPU at full speed, but from the GPU it is on the far side of PCIe.
5. NVMe storageWhere the model file lives between runs.

The GPU reads its own VRAM at full speed. Anything that does not fit there has to travel over PCIe on every use, and the PCIe link delivers a small fraction of what the card's own memory does, so a model that spills is not slightly slower, it is in a different regime.

Unified memory: DGX Spark, Ryzen AI Max+ 395 and Apple silicon (Tracks S, X and M)

further from the compute

1. CPU cores and GPU coresOn the same package, on the same memory controller.
2. Unified memory poolLPDDR5x soldered to the package or the board. Both processors address the same bytes. This is the full-speed level for the GPU and for the CPU at once.
3. A software-imposed ceilingThe GPU may not be allowed all of the pool: a wired-memory limit on macOS, a GTT limit on Linux, a firmware allocation on Windows. Raising it is a per-track procedure taught in the lessons that follow.per track
4. NVMe storageWhere the model file lives between runs.

One pool, addressed by both processors, with no copy and no bus between them. MLX's documentation puts it plainly: arrays live in unified memory, and 'any device can perform any operation' on them 'without needing to move them from one memory location to another'.

The trade is symmetric and there is no free side of it. Discrete memory is fast and small: it is built for graphics, sits on a wide bus a few millimetres from the compute, and costs enough per gigabyte that consumer cards stop where they stop. Unified memory is large and slower: it is ordinary system memory, so a desk-sized machine can carry more of it than any graphics card, at a bandwidth set by what an LPDDR5x memory controller can do rather than by what GDDR7 can do.

That is the whole map. A model that fits comfortably in a unified pool and would not fit on a graphics card will generate more slowly on the unified machine than a smaller model does on the card, and this is not a defect on either side. It is the two designs doing what they are for.

The four tracks

This course writes every hands-on page four times, once for each of these.

Track	Machines	Memory	Bandwidth	Compute
S NVIDIA DGX Spark	NVIDIA DGX Spark	128 GB unified	273 GB/s	Blackwell GPU with 20-core Arm CPU (GB10), CUDA 13 on aarch64
X AMD Ryzen AI Max+ 395	AMD Ryzen AI Max+ 395	64 / 128 GB unified	256 GB/s	Radeon 8060S integrated GPU (40 RDNA 3.5 compute units, gfx1151), 16 Zen 5 cores, XDNA 2 NPU; Vulkan and ROCm
M Apple silicon Mac	Apple silicon Mac	24 / 32 / 64 / 128 / 256 / 512 GB unified	546 GB/s*	Apple GPU with Metal; MLX
N NVIDIA desktop or laptop	NVIDIA desktop or laptop	8 / 12 / 16 / 24 / 32 / 48 / 96 GB VRAM	1792 GB/s*	CUDA 13; WSL2 on Windows

* Varies by chip or card: Apple silicon Mac: By chip, from Apple's specification pages read 2026-09-09: M4 120 GB/s, M4 Pro 273, M4 Max 546, M6 153 or 170, M5 Pro 307, M5 Max 460 or 614 depending on GPU configuration, M3 Ultra 819, M5 Ultra 1200. NVIDIA desktop or laptop: By card: RTX 3090 936 GB/s, RTX 4090 1008, RTX 5090 1792, RTX PRO 6000 Blackwell 1792; laptop parts are lower.

Memory tiers

Tier	What it can follow
8 GB	8B-class models at Q4; every Level 1 and 2 lab on its reduced path.
12–16 GB	14B-class at Q4 to Q6; LoRA fine-tuning of 1–4B models; the Track N validation tier.
24 GB	32B-class at Q4; QLoRA of 8B; the 14B-class distillation teacher; the Track M validation tier at 24 GB unified.
32 GB	30B-class MoE at Q8; gpt-oss-20b in MXFP4 with room; RTX 5090.
48–64 GB	70B-class at Q4; 30B-class teachers at Q8; M4 Pro and mid Mac Studio configurations.
96 GB	gpt-oss-120b in MXFP4; RTX PRO 6000 Blackwell.
128 GB	The DGX Spark, EVO-X2 and M4 Max/M5 Max tier: 120B-class MoE, 235B-class at IQ4 on Spark and Mac, 30B-class full fine-tunes with QLoRA.
256 GB	Two 128 GB machines clustered, or a 256 GB Mac Studio: 235B-class at Q8, 400B-class at FP4 on a Spark pair.
512 GB	M3 Ultra and M5 Ultra Mac Studio: 671B-class MoE at Q4.

Track S — NVIDIA DGX Spark. A GB10 Grace Blackwell machine with 128 GB of unified memory, running NVIDIA’s own Linux distribution with the full CUDA stack, on Arm rather than x86. It is the track with the fewest software surprises and the most memory per unit of frustration, and the one whose processor architecture changes which downloads you need.

Track X — AMD Ryzen AI Max+ 395. The same unified-memory idea on x86, in mini PCs such as the GMKtec EVO-X2, at 64 or 128 GB. It offers the best price per gigabyte of the four and asks for the most patience in return, because it has two GPU software paths of differing maturity and you have to know which one you are on.

Track M — Apple silicon. Mac mini and Mac Studio, where unified memory scales further than anywhere else on a desk and the bandwidth scales with the chip: Apple’s current Mac mini specifications list an M6 configuration and an M5 Pro configuration with very different figures, and the Mac Studio specifications list M5 Max and M5 Ultra options with higher ones again. No CUDA, a mature Metal and MLX stack, and one system setting that has to be changed before a large model will load.

Track N — an NVIDIA desktop or laptop. The class most readers already own, and the one where capacity is tightest and bandwidth is highest. A GeForce card’s memory is what it is: NVIDIA’s own comparison page puts the top consumer card of the current generation at 32 GB, and its workstation Blackwell part at 96 GB, with no upgrade path on either. Much of Level 2 is about living inside that ceiling.

The numbers the vendors publish

Here are the four tracks’ headline bandwidth figures as their vendors publish them. They are specifications, not measurements. Every one of them is a claim you will test in this part’s second lab, and the gap between a specification and what a real machine achieves is itself informative.

Vendor specification, not measured Memory bandwidth reported by the vendors, by track

1. Track X — Ryzen AI Max+ 395 (LPDDR5x)256 GB/s
2. Track S — DGX Spark (GB10, LPDDR5x)273 GB/s
3. Track M — Apple silicon, M4 Max class546 GB/s
4. Track N — RTX 5090 / RTX PRO 6000 Blackwell (GDDR7)1,792 GB/s

Vendor specification, not a measurement: the four platform tracks as specified by their vendors; vendor pages listed in this page's sources; 2026-09-09.

Show the numbers behind this chart

Row	Value (GB/s)	Note
Track X — Ryzen AI Max+ 395 (LPDDR5x)	256
Track S — DGX Spark (GB10, LPDDR5x)	273
Track M — Apple silicon, M4 Max class	546
Track N — RTX 5090 / RTX PRO 6000 Blackwell (GDDR7)	1,792

Published figures from the vendor pages listed in this page's sources, and from the course hardware reference. Nothing here was measured. The chart exists to show the shape of the spread: the fastest of the four reads memory several times faster than the two unified-memory x86 and Arm machines, which are in turn the two that carry the most memory for the money.

The spread within a track is often larger than the gap between tracks, which is why “Apple silicon” or “an NVIDIA card” is never a specification.

Pending validationThe spread inside two of the tracks, as published

Part	Track	Reported bandwidth GB/s	Memory as sold
M4	M	120	16-32 GB
M4 Pro	M	273	24-64 GB
M4 Max	M	546	36-128 GB
M5 Max	M	614	36-128 GB
M3 Ultra	M	819	96-512 GB
M5 Ultra	M	1,200	96-512 GB
GeForce RTX 3090	N	936	24 GB
GeForce RTX 4090	N	1,008	24 GB
GeForce RTX 5090	N	1,792	32 GB
RTX PRO 6000 Blackwell	N	1,792	96 GB

the parts named in the first column, not applicable · no engine; course hardware reference and vendor pages, retrieved 2026-09-09 · no model loaded, not applicable · 0 tokens of context · 2026-09-09

Specification figures carried by the course hardware reference and checked against the vendor pages cited on this page; laptop parts are lower than their desktop namesakes. Nothing in this table was measured. The second lab in this part replaces the row for your own machine with a measurement.

✱Hardware note

Read that table as a purchasing rule rather than a league table. Within Track M the bandwidth rises roughly in step with the memory ceiling, so a bigger Mac is faster as well as larger. Within Track N it does not: the workstation card and the top consumer card in the list report the same bandwidth and differ by a factor of three in capacity. Capacity and bandwidth are separate axes, and which one binds you depends entirely on the model you want to run.

Doing the arithmetic before you buy anything

The prediction is one division, and it is worth doing by hand once.

Take a model with about eight billion parameters, quantised to roughly four bits, so a little over half a byte per parameter. The weights are then somewhere around five gigabytes. Decode reads all of them for every token. On a machine whose memory delivers a few hundred gigabytes per second, that division gives a decode ceiling in the region of tens of tokens per second; on a machine delivering well over a thousand, the same division gives a ceiling several times higher. The exercise below has you do it properly with real figures.

Two corrections make the estimate honest. The first is that no machine achieves its specification: memory controllers have overheads, sustained transfers are slower than peak, and thermal limits bite on small enclosures. The second is that a mixture-of-experts model does not read all of its weights per token, only the active ones, which is why a model with thirty billion parameters and roughly three billion active can decode like a much smaller model while occupying memory like a large one. Part 4 covered that architecture; this part is where it becomes a hardware argument, because it is exactly the property that makes a large unified-memory machine worth owning.

▷Exercise

Before the lab, do it on paper for your own machine. Take the bandwidth figure for your track from the table above, take a model from the model reference with its size at Q4_K_M, and divide. Write the answer in your lab notebook with the date and the word “predicted”. The second lab in this part measures the bandwidth for real and repeats the division, and Part 6 runs the model and compares. Three numbers, each one checking the one before.

Prefill is the other half, and it flips the ranking

Reading a prompt is a different job. Every weight is fetched once and used for all of the prompt’s tokens at once, as one large matrix multiplication, so the arithmetic dominates and the bytes stop being the constraint. On this job the ranking of machines follows compute rather than bandwidth, and the gaps are large: the Blackwell parts in Tracks S and N have hardware support for four-bit floating-point formats that the other tracks do not, and NVIDIA quotes the Spark at up to one petaflop of FP4 for exactly this kind of work.

For a chat prompt of a few hundred tokens the difference is invisible. For a coding agent that re-reads a fifty-thousand-token context on every step, or for a fine-tuning run, it is most of the wall-clock time. This is why the course measures both numbers in the second lab rather than one, and why Part 6’s benchmarks report prompt processing and generation separately. A machine that is good at one is not necessarily good at the other.

▤Memory budget

The key-value cache is the third consumer of memory and the one that surprises people, because it grows with context length rather than with model size and it is read on every token like the weights are. A long conversation therefore costs memory you have to budget for and bandwidth you have to spend. Part 4 gave the bytes-per-token formula; Part 6 shows what happens when you ask for a context your machine cannot afford, and the answer is not a polite error message on every engine.

Use a roofline estimate with explicit assumptions

Arithmetic intensity is operations divided by bytes moved. Multiplying intensity by attainable memory bandwidth gives a bandwidth-limited operation rate; the device’s attainable compute rate supplies the other ceiling. The smaller ceiling is the starting estimate. It describes the particular kernel and traffic pattern, not the marketing category of the whole machine.

A matrix-vector operation at low batch size reuses weights less than a matrix-matrix operation across many tokens. That is why prompt processing and concurrent serving can shift the limiting resource. Quantisation changes byte traffic, but unpacking, scale loads and available kernels also change runtime.

As an exercise, draw two predictions for the same model: interactive single-request decode and a batch of independent requests. List which arrays are reused and which grow with the batch. Then compare against measurements while keeping prompt and answer lengths fixed. If performance is far below both ceilings, investigate placement, synchronisation, launch overhead and thermal limits. A roofline is useful because it tells you which experiment to run next, not because its upper bound is a promised token rate.

Recap

Two numbers describe a machine for this work: how much memory the accelerator can reach, and how fast it reads it. Capacity decides which models are possible; bandwidth divided by the bytes read per token sets a ceiling on decode speed that no engine can lift. Peak floating-point throughput governs prefill and training instead, which is real work but not the work most readers are watching.

The four tracks divide into discrete memory, where a graphics card has its own fast, small pool and everything else is across PCIe, and unified memory, where the CPU and GPU share one large, slower pool with no copy between them. Large and slow beats small and fast when the model does not fit at all, and loses when it does. Vendor bandwidth figures vary more inside a track than between tracks, so the part number matters, not the brand, and every published figure is a claim until you measure it, which is what the second lab in this part is for.

Check your understanding

Question 1. Two machines both have 128 GB of memory the accelerator can use. One reports roughly a third of the other's memory bandwidth. What differs when you run the same 70B model at Q4 on both?

• Only the faster machine can load the model
• Both load it; the faster machine decodes roughly three times as many tokens per second, and both read a long prompt at rates set by their compute rather than their bandwidth
• The slower machine produces lower-quality output
• Nothing differs; memory capacity is what matters

Show the answer and why

Answer: Both load it; the faster machine decodes roughly three times as many tokens per second, and both read a long prompt at rates set by their compute rather than their bandwidth

Capacity decides what fits and both machines fit it. Bandwidth divided by bytes-per-token sets the decode ceiling, so the ratio of decode speeds tracks the ratio of bandwidths. Prefill is compute-bound, so it is governed by a different number entirely.

Question 2. On a desktop with a 16 GB graphics card and 64 GB of system memory, an engine offers to keep half the layers of a large model in system RAM. Why is the result usually disappointing?

• System RAM cannot store model weights
• Every offloaded weight has to cross the PCIe link on every token, and that link carries a small fraction of what the card reads from its own memory
• The CPU cannot do matrix multiplication
• The model has to be re-quantised each time

Show the answer and why

Answer: Every offloaded weight has to cross the PCIe link on every token, and that link carries a small fraction of what the card reads from its own memory

Decode reads every active weight once per token. Weights on the far side of PCIe are read at PCIe speed, so the offloaded fraction dominates the time. It is not that the arrangement fails; it is that it moves the model into a different performance regime.

Question 3. Which statements about unified memory are correct? Select all that apply.

• The CPU and GPU address the same bytes with no copy between them
• The pool is usually larger than a consumer graphics card's VRAM
• Its bandwidth is typically higher than a discrete card's VRAM bandwidth
• The GPU may be limited by software to less than the whole pool

Show the answer and why

Answer: The CPU and GPU address the same bytes with no copy between them, The pool is usually larger than a consumer graphics card's VRAM, The GPU may be limited by software to less than the whole pool

Unified memory trades bandwidth for capacity: it is ordinary system memory, so it is larger and slower than GDDR on a card. All three unified tracks also impose a software ceiling on how much of the pool the GPU may claim, and raising it is a per-track procedure.

Question 4. A mixture-of-experts model has 30 billion total parameters and about 3 billion active per token. What does that change about the two questions?

• It reduces both the memory it occupies and the bytes read per token
• It reduces the memory it occupies but not the bytes read per token
• It reduces the bytes read per token, and therefore raises the decode ceiling, while the memory it occupies is still set by the total parameter count
• It has no effect on either

Show the answer and why

Answer: It reduces the bytes read per token, and therefore raises the decode ceiling, while the memory it occupies is still set by the total parameter count

All the experts must be resident because routing can select any of them, so capacity follows the total. Only the routed experts are read for a given token, so the decode ceiling follows the active count. That asymmetry is why mixture-of-experts models suit large unified-memory machines.

Question 5. True or false: a machine advertised with a very high AI operations-per-second figure will always generate text faster than one with a lower figure.

• True
• False

Show the answer and why

Answer:

That figure describes compute, which governs prefill and training. Generation is bandwidth-bound, so a machine with more compute and less bandwidth can read a prompt faster and still type its answer more slowly. Ask which of the two jobs the number describes.

Check my answersClear