Tensors, GPUs and Precision: Why Matrix Multiplication Is Everything

By the end of this lesson you will be able to read a tensor’s shape and say what it costs in bytes; count the operations in a matrix multiplication and predict when a chip is waiting on memory rather than computing; put a number on every level of memory between a model’s weights on disk and the arithmetic unit that uses them; and read a format name such as BF16, FP8 or Q4 and say what it keeps, what it loses, and why a scale factor travels with every weight of eight bits or fewer. Every figure in the tables is arithmetic from stated inputs, not a measurement: the config.json of Qwen3-8B (Apache-2.0), the file sizes in the model reference, and the vendor bandwidth figures in the hardware reference. The two timed snippets are one run on one CPU and are labelled so; your times will differ. The snippets need numpy and PyTorch, which this part’s lab installs; read them now and run them once that environment exists.

Tensors: numbers in shapes

A tensor is an n-dimensional array of numbers of one type, described by a shape, the tuple of its sizes along each dimension, and a dtype, the format of each element. A vector has one dimension, a matrix two, a batch of matrices three. The shape is metadata over a single flat buffer laid out row after row, so bytes = elements × bytes per element is the first piece of arithmetic in this course: a matrix of shape (4096, 12288) in 32-bit floating point is 50,331,648 elements in 201,326,592 bytes.

Every quantity in a model is a tensor, and the shapes come straight from the configuration file. The input to a layer is (batch, inputs), one row per token. A layer’s weights are (outputs, inputs) as PyTorch stores an nn.Linear, and its output is (batch, outputs). Read the config.json of Qwen3-8B and every matrix in the model follows:

Tensor (one layer, unless noted)	Shape as stored, (outputs, inputs)	From config.json	Elements	Bytes at BF16
embed_tokens (once)	(151936, 4096)	vocab_size, hidden_size	622,329,856	1,244,659,712
q_proj	(4096, 4096)	num_attention_heads × head_dim, hidden_size	16,777,216	33,554,432
k_proj and v_proj (each)	(1024, 4096)	num_key_value_heads × head_dim, hidden_size	4,194,304	8,388,608
o_proj	(4096, 4096)	hidden_size, num_attention_heads × head_dim	16,777,216	33,554,432
gate_proj and up_proj (each)	(12288, 4096)	intermediate_size, hidden_size	50,331,648	100,663,296
down_proj	(4096, 12288)	hidden_size, intermediate_size	50,331,648	100,663,296
lm_head (once; tie_word_embeddings is false)	(151936, 4096)	vocab_size, hidden_size	622,329,856	1,244,659,712
One layer’s seven matrices			192,937,984	385,875,968
36 layers plus the two vocabulary matrices			8,190,427,136	16,380,854,272

The last row is the 8.2 billion parameters in the model’s name and the 16.4 GB of its BF16 checkpoint in the model reference, less the small normalisation vectors this table omits. Part 2’s lab has you list the real tensors and reconcile that count against the file. Here is the same arithmetic done by the library, and the error you get when shapes disagree:

RunnableAll tracks

shapes.py

"""Tensors have a shape, a dtype and a size in bytes; matrix multiplication checks the shapes."""
import numpy as np

x = np.zeros((32, 4096), dtype=np.float32)      # a batch of 32 token vectors
w = np.zeros((4096, 12288), dtype=np.float32)   # one weight matrix as (inputs, outputs), which x @ w needs;
                                                 # PyTorch stores the transpose and multiplies by w.T
for name, t in (("x", x), ("w", w)):
    print(f"{name}: shape={t.shape}  ndim={t.ndim}  dtype={t.dtype}  "
          f"elements={t.size:,}  bytes={t.nbytes:,}")
y = x @ w
print(f"x @ w: shape={y.shape}   the inner sizes agree: {x.shape[1]} == {w.shape[0]}")
try:
    w @ x                                        # inner sizes 12288 and 32 do not agree
except ValueError as e:
    print("w @ x fails:", e)

Output — what you should see

shapes.py, numpy 2.5.2, CPU

x: shape=(32, 4096)  ndim=2  dtype=float32  elements=131,072  bytes=524,288
w: shape=(4096, 12288)  ndim=2  dtype=float32  elements=50,331,648  bytes=201,326,592
x @ w: shape=(32, 12288)   the inner sizes agree: 4096 == 4096
w @ x fails: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 32 is different from 12288)

Operations are defined on shapes. x @ w needs the inner sizes to agree, (32, 4096) times (4096, 12288) gives (32, 12288), and the message names the operand, the dimension and the two sizes that differ. When the mismatch is a matrix stored the other way round, the fix is a transpose, w.T, which changes the shape by changing how the buffer is read. A reshape to the same target shape also runs, but it scrambles which value sits where and produces wrong numbers while raising nothing. Reading shapes is the first debugging skill in this field.

One operation, repeated

Here is the sentence about accelerators. Almost all of the work in training or running a model is matrix multiplication, and a GPU is a machine for doing matrix multiplication in parallel.

C = A @ B         A is (m, k), B is (k, n), C is (m, n)
C[i, j] = sum over p of A[i, p] × B[p, j]       for every i below m and j below n
multiply-adds = m × n × k                        FLOPs = 2 × m × n × k

Each of the m × n outputs is a dot product of k terms, and no output depends on any other, so all of them can be computed at once. That independence is the whole trick. A CPU with a dozen cores does a dozen at a time; a GPU with thousands of simpler cores does thousands; and the units NVIDIA calls tensor cores multiply a small block of the matrix as one instruction, which is why its mixed-precision guide asks that “on FP16 inputs, all three dimensions (M, N, K) must be multiples of 8”. A processor does not run a model faster because it is faster in general. It runs it faster because the model’s work is one parallel operation and the processor is built for that operation.

The unit of this work is the FLOP, one floating-point operation, and every weight in a matrix costs one multiply and one add per row of input. For one token, m is 1 and the cost is two operations per parameter:

Job	Formula	Qwen3-8B, 8.2 billion parameters
One token through the model (decode)	2 × parameters	16.4 GFLOP
A 2,048-token prompt (prefill)	2 × parameters × tokens	33.6 TFLOP
One training token, forward and backward	6 × parameters	49.2 GFLOP
Pretraining on one billion tokens	6 × parameters × tokens	4.9 × 10^19 FLOP

The six comes from the backward pass costing twice the forward; nanochat’s source puts it in one line, “Each matmul weight parameter contributes 2 FLOPs (multiply *, accumulate +) in forward, and 2X that in backward => 2+4=6”, and Part 12’s scaling-laws lesson turns that line into a training budget. The difference between doing this work one operation at a time and doing it as one operation is visible on any machine:

RunnableAll tracks

matmul-cost.py

"""Count the multiply-adds in one matrix multiplication, then time Python loops against numpy."""
import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"        # one BLAS thread, so both sides use one core
import time
import timeit
import numpy as np

m, k, n = 128, 128, 128
rng = np.random.default_rng(0)
a = rng.standard_normal((m, k), dtype=np.float32)
b = rng.standard_normal((k, n), dtype=np.float32)
flops = 2 * m * n * k                            # one multiply and one add for every (i, j, p)
print(f"({m} x {k}) @ ({k} x {n}): {m * n * k:,} multiply-adds = {flops:,} FLOPs")

t0 = time.perf_counter()
c = np.zeros((m, n), dtype=np.float32)
for i in range(m):
    for j in range(n):
        s = 0.0
        for p in range(k):
            s += a[i, p] * b[p, j]
        c[i, j] = s
t_loops = time.perf_counter() - t0
t_numpy = min(timeit.repeat(lambda: a @ b, number=100, repeat=5)) / 100   # best of five runs, per call
print(f"python loops: {t_loops:9.4f} s per call   {flops / t_loops / 1e9:8.3f} GFLOP/s")
print(f"numpy a @ b : {t_numpy:9.6f} s per call   {flops / t_numpy / 1e9:8.3f} GFLOP/s")
print(f"same answer: max |difference| = {np.abs(c - a @ b).max():.1e}")

Output — what you should see

matmul-cost.py, numpy 2.5.2, one CPU core of the author's test box; your times will differ, the ratio will not by much

(128 x 128) @ (128 x 128): 2,097,152 multiply-adds = 4,194,304 FLOPs
python loops:    0.3056 s per call      0.014 GFLOP/s
numpy a @ b :  0.000018 s per call    237.917 GFLOP/s
same answer: max |difference| = 1.1e-05

Same 4,194,304 operations, same answer, seventeen thousand times faster on the same core: the loops ask the interpreter for each multiply, the library hands blocks of the matrix to vectorised code, and a GPU does the same thing with thousands of cores. Peak FLOPS, the trillions per second a vendor advertises, is this library number scaled up. It is also, for most of what this course does, not the number that matters.

Bytes moved versus operations done

Every multiply needs its operands, and the operands live in memory. A chip can only pull so many bytes per second, its memory bandwidth, so every job has two clocks running, and the slower one sets the time:

time_compute = FLOPs / peak_FLOP_per_second
time_memory  = bytes_moved / bandwidth_bytes_per_second
time         ≈ the larger of the two

arithmetic intensity = FLOPs / bytes_moved                    (FLOP per byte)
ridge point          = peak_FLOP_per_second / bandwidth       (FLOP per byte)
intensity below the ridge: bandwidth-bound.  Above it: compute-bound.

For a language model the bytes moved are dominated by the weights, and the FLOPs by the same weights, so the intensity of a job is easy to write down:

Job, Qwen3-8B	FLOPs	Bytes of weights read	Intensity, FLOP per byte
Decode, batch 1, BF16 (16.4 GB)	16.4 G	16.4 G	1.0
Decode, batch 1, Q4_K_M (5.0 GB file)	16.4 G	5.0 G	3.3
Decode, batch 8, BF16	131 G	16.4 G	8
Prefill, 2,048 tokens, BF16	33.6 T	16.4 G	2,048

Decode at batch 1 does one multiply-add per weight and then discards the weight: one FLOP per byte at BF16, three at four-bit. No accelerator has a ridge point that low; every one of them sits far above one FLOP per byte between its peak arithmetic and its bandwidth, so decode sits far below the ridge and the chip spends its time waiting for bytes. That is what bandwidth-bound means. Prefill fetches the same weights once and uses each of them for every token of the prompt, so its intensity is the prompt length, and it climbs above the ridge: compute-bound. Training, with big batches, lives there too. The same shift is visible on a CPU with one weight matrix and four batch sizes:

RunnableAll tracks

intensity.py

"""The same weight matrix, read once per call: batch 1 waits on bytes, batch 256 does arithmetic."""
import timeit
import torch

torch.manual_seed(0)
inputs, outputs = 4096, 12288                    # the shape of Qwen3-8B's gate_proj weight
w = torch.randn(inputs, outputs, dtype=torch.float32)
weight_bytes = w.numel() * w.element_size()      # 4096 * 12288 * 4 = 201 MB

print(f"weights: {weight_bytes / 1e6:.0f} MB, read once per call; {torch.get_num_threads()} CPU threads")
print(f"{'batch':>6} {'ms per call':>12} {'GFLOP/s':>9} {'weight GB/s':>12} {'FLOP per byte':>14}")
for batch in (1, 8, 64, 256):
    x = torch.randn(batch, inputs, dtype=torch.float32)
    dt = min(timeit.repeat(lambda: x @ w, number=1, repeat=20))   # best of 20 calls
    flops = 2 * batch * inputs * outputs
    print(f"{batch:>6} {dt * 1e3:>12.2f} {flops / dt / 1e9:>9.1f} "
          f"{weight_bytes / dt / 1e9:>12.1f} {flops / weight_bytes:>14.1f}")

Output — what you should see

intensity.py, torch 2.14.0, CPU, one run on the author's test box; your figures will differ, the shape will not

weights: 201 MB, read once per call; 12 CPU threads
 batch  ms per call   GFLOP/s  weight GB/s  FLOP per byte
     1         5.85      17.2         34.4            0.5
     8        18.19      44.3         11.1            4.0
    64        25.78     249.9          7.8            32.0
   256        58.58     439.9          3.4           128.0

The snippet’s weights are FP32, four bytes each, so its batch-1 intensity is 0.5 rather than the table’s 1.0 at BF16; the ratio between the rows is what matters. At batch 1 the call streamed the weights at this machine’s memory speed and got a handful of GFLOP/s for it: bandwidth-bound. At batch 256 it fetched exactly the same bytes, did two hundred and fifty-six times the arithmetic on them, and ran at twenty-five times the rate: compute-bound. On a GPU the gap is larger, not smaller. Part 5’s measurement lab has you measure both peaks for your own machine, and their quotient is your ridge point.

Rearranging the memory clock for decode gives the formula the course leans on hardest, worked here for one model on the four tracks from the vendor bandwidth figures in the hardware reference:

tokens_per_second (decode, batch 1) ≤ memory_bandwidth_bytes_per_second / bytes_of_weights_read_per_token

Track, machine	Vendor bandwidth (GB/s)	Qwen3-8B at BF16, 16.4 GB (tokens/s)	Qwen3-8B at Q4_K_M, 5.0 GB (tokens/s)
S, DGX Spark	273	16.6	54.6
X, Ryzen AI Max+ 395	256	15.6	51.2
M, Mac Studio with M4 Max	546	33.3	109.2
N, desktop with RTX 5090	1,792	109.3	358.4

These are ceilings, not predictions: nothing else has been allowed any time. Part 3’s inference lesson states the formula formally and extends it to mixture-of-experts models, Part 5 measures the bandwidth, and Part 6 measures the achieved speed, which lands below these numbers by a gap the rest of the course explains. What the table already settles is which number to read on a specification sheet:

What you are doing	Which clock dominates	The figure that predicts it
Generating text for one request	memory	bandwidth
Reading a long prompt; time to first token	compute	matrix-multiply throughput at your format
Serving many concurrent requests (Part 9)	moves from memory towards compute as the batch grows	both
Training and fine-tuning (Parts 11 to 15)	compute, once the batch is large	throughput at BF16

!Important

For running models at home, bandwidth is the number to read first. Two machines with the same memory capacity and different bandwidth fit the same models and generate at different speeds. Two with the same peak FLOPS and different bandwidth read a prompt at the same speed and generate at different speeds.

Where the memory is

Memory is not one thing. It is a hierarchy, fast and small next to the arithmetic and slow and large far from it, and the CUDA programming guide names the levels on the GPU side: registers “located on the SM”, shared memory, of which it says “the bandwidth is higher and the latency is lower than accessing global memory”, an L1 that “is physically located on each SM and is the same physical space used by shared memory”, an L2 that “is located on the device and is shared among all the SMs”, and global memory, the device’s own DRAM, which works “similar to RAM in a CPU system”. Below the device come the PCIe link, the host’s memory, storage and the network.

Level	Where it is	Size	Bandwidth (vendor figure, or a link ceiling)	Source
Registers	on each streaming multiprocessor	64 K 32-bit registers per SM, 256 KB (compute capability 12.x)	not published as a bandwidth	CUDA programming guide
Shared memory and L1	on each SM	100 KB per SM at compute capability 12.x (the RTX 50 series and the DGX Spark’s GB10, per NVIDIA’s CUDA GPUs list); 228 KB at 9.0 and 10.0; 164 KB at 8.0	“the bandwidth is higher and the latency is lower than accessing global memory”	CUDA programming guide
L2 cache	on the device, shared by all SMs	not in the guide’s table	—	CUDA programming guide
Device memory, HBM3	on the package, data-centre parts	—	3,350 GB/s (H100 SXM)	NVIDIA H100 page
Device memory, GDDR7	on the card, Track N	32 GB (RTX 5090)	1,792 GB/s	hardware reference
Unified memory, LPDDR5x	on the board, Tracks S, X and M	128 GB	273 (DGX Spark), 256 (Ryzen AI Max+ 395), 546 (M4 Max) GB/s	hardware reference
PCIe x16	between the card and the host	—	16 GB/s for an x16 Gen3 link (CUDA best-practices guide); 128 GB/s for the H100’s Gen5 x16 link (H100 page), which is the two directions summed: NVIDIA’s Hopper architecture post gives Gen5 x16 as “64 GB/sec in each direction” and Gen4 x16 as “32GB/sec in each direction”	NVIDIA
System memory, DDR5	on the motherboard; from the GPU, behind PCIe	up to hundreds of GB	89.6 GB/s for two channels of DDR5-5600: 2 × 5,600 million transfers × 8 bytes, a ceiling from the module’s name	arithmetic
NVMe storage	on four PCIe lanes	terabytes	a quarter of the per-direction x16 figures in the row above, four lanes against sixteen: 32 ÷ 4 = 8 GB/s at Gen4, 64 ÷ 4 = 16 GB/s at Gen5, as link ceilings; drives quote sequential reads below them	arithmetic from the NVIDIA per-direction figures
Network	between machines	—	1.25 GB/s for 10 GbE; 25 GB/s for the Spark’s 200 Gb/s ConnectX-7 port (bits ÷ 8)	hardware reference

The memory hierarchy, fastest at the top

larger, slower

1. Registers, shared memory, L1On each streaming multiprocessor. Hundreds of kilobytes. Where the multiply-adds happen.
2. L2 cacheOn the chip, shared by every SM. Megabytes.
3. Accelerator memoryHBM on data-centre parts, GDDR7 on a Track N card, LPDDR5x shared with the CPU on the DGX Spark, the Ryzen AI Max+ 395 and Apple silicon. Tens to hundreds of gigabytes. The level a model must fit in.
4. PCIe linkOn a discrete-memory machine, the only path between the card and everything below. A few per cent of what the memory on the card delivers.discrete only
5. System memoryDDR5 on the motherboard. Large and cheap; at full speed for the CPU, on the far side of PCIe for a discrete GPU.
6. Storage and networkNVMe, then Ethernet or RDMA to other machines. Where models live between runs, and how clusters share them (Parts 18 to 21).

Each level is larger and slower than the one above by a factor of four to ten. A model must fit in the level its compute reads at full speed: the card's own memory on a discrete GPU, the shared pool on a unified-memory machine. Everything below that level is the slow path.

What the hierarchy costs is easiest to see by reading one model’s weights once from each level. Qwen3-8B at BF16 is 16.4 GB; decode reads all of it once per token:

Level feeding the decode	Bandwidth used (GB/s)	Time to read 16.4 GB once (ms)	Ceiling (tokens/s)
HBM3, H100 SXM	3,350	4.9	204
GDDR7, RTX 5090	1,792	9.2	109
LPDDR5x, M4 Max	546	30.0	33
LPDDR5x, DGX Spark	273	60.1	17
PCIe Gen5 x16, one direction	64	256	3.9
PCIe Gen3 x16, one direction	16	1,025	1.0
NVMe on four Gen4 lanes	8	2,050	0.5
10 GbE	1.25	13,120	0.08

Two things to read off that table. Each step down is a factor of four to ten, not a few per cent, so the level a model sits in decides its speed class before any tuning. And the lower rows are per-token times only if the weights are re-read from that level on every token, which is exactly what happens when a model does not fit the level above: an engine that maps a file larger than memory reads it from disk for every token, and a model split between a card and system memory crosses PCIe for every token. The split case is worth doing with numbers, because the collapse is larger than intuition suggests:

Where the 16.4 GB sits	Bytes	Bandwidth (GB/s)	Time per token (ms)
VRAM of an RTX 5090	12.0 GB	1,792	6.7
System memory, reached over PCIe Gen5	4.4 GB	64	68.8
Total	16.4 GB		75.4, a ceiling of 13 tokens/s against 109 with everything in VRAM

A quarter of the weights in the wrong place takes nine tenths of the time. That is what “the speed collapses” means when Part 5’s NVIDIA lesson measures it.

✕Common mistake

“It fits in RAM, so it will run.” The symptom is single-digit tokens per second on a card whose arithmetic promised a hundred. The cause is that the engine put the layers it could not fit into system memory and is pulling them across PCIe on every token; its load log says how many layers went to the GPU, and Part 6’s two-tokens-per-second challenge shows you where to read it. The fix is a smaller quantisation or a smaller model, so that all of it sits in the full-speed level, not a faster PCIe slot.

The four tracks differ in which level is the full-speed one and how large it is. A desktop or laptop with an NVIDIA card has discrete memory: the GPU’s own VRAM, fast, fixed at purchase, joined to the rest of the machine by PCIe, of which the CUDA guide says that memory reached across that link has “higher latency and lower bandwidth compared to accessing device memory”. The DGX Spark, the Ryzen AI Max+ machines and Apple silicon have unified memory: one pool that both processors address. MLX’s documentation puts it in two sentences: “Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device types without performing data copies.”

Track	Design	The accelerator’s full-speed memory	Vendor bandwidth (GB/s)	The trade
N, NVIDIA desktop or laptop	discrete	8 to 32 GB of GDDR on the card; 96 GB on the RTX PRO 6000	936 to 1,792 by card	fastest decode, smallest ceiling on what fits
S, DGX Spark	unified	128 GB of LPDDR5x shared with the CPU	273	large models fit; each decodes more slowly
X, Ryzen AI Max+ 395	unified	64 or 128 GB shared, less a GPU-visible cap	256	the same, at a lower price
M, Apple silicon	unified	24 to 512 GB shared	120 (M4) to 1,200 (M5 Ultra) by chip	the widest range on both axes; the chip decides the bandwidth

Unified memory is what lets a model of a hundred gigabytes fit on a desk; its cost is that the pool is system memory with system-memory bandwidth, several times below a card’s, so the same model that fits comfortably decodes more slowly. Capacity against bandwidth is the central fact about the four tracks. Part 5’s memory lesson makes it concrete per track, including the software ceilings that stop a unified machine’s GPU from taking the whole pool.

Precision: what each format gives up

A number format is a way of splitting a fixed number of bits between a sign, an exponent, which sets the range, and a mantissa, which sets the precision. The layouts below are the ones PyTorch documents for its dtypes; the numerical columns are what torch.finfo reports for each:

Format	Bits	Sign, exponent, mantissa	Bytes per parameter	Step above 1.0 (eps)	Largest value	Smallest normal	What it gives up
FP32	32	1, 8, 23	4	1.19 × 10^-7	3.4 × 10^38	1.18 × 10^-38	nothing you will notice; twice the bytes of anything else
BF16	16	1, 8, 7	2	7.81 × 10^-3	3.4 × 10^38	1.18 × 10^-38	precision: about three significant digits
FP16	16	1, 5, 10	2	9.77 × 10^-4	65,504	6.10 × 10^-5	range: above 65,504 becomes inf, tiny values become 0
FP8 E4M3	8	1, 4, 3	1	0.125	448	1.56 × 10^-2	both; unusable without a scale
FP8 E5M2	8	1, 5, 2	1	0.25	57,344	6.10 × 10^-5	precision: about one significant digit (the step above 1.0 is 0.25); the gradient format
E2M1, the element of MXFP4 and NVFP4	4	1, 2, 1	0.5 plus scales	—	6	—	a grid of 0, 0.5, 1, 1.5, 2, 3, 4, 6 and their negatives
INT8	8	integer	1 plus scales	—	127 × scale	—	a grid of 256 steps per block
INT4	4	integer	0.5 plus scales	—	7 × scale	—	a grid of 16 steps per block

Bytes per parameter is the column you will use most: it is the divisor in the previous section’s decode ceiling. The rest of the table says what you pay for a smaller number in it, and the snippet shows each loss happening:

RunnableAll tracks

formats.py

"""What each floating-point format can represent, from torch.finfo, and what rounding does to values."""
import torch

def name(dt):
    return str(dt).replace("torch.", "")

print(f"{'dtype':<14}{'bits':>5}{'eps':>11}{'max':>12}{'smallest normal':>17}")
for dt in (torch.float32, torch.bfloat16, torch.float16, torch.float8_e4m3fn, torch.float8_e5m2):
    f = torch.finfo(dt)
    print(f"{name(dt):<14}{f.bits:>5}{f.eps:>11.2e}{f.max:>12.5g}{f.smallest_normal:>17.2e}")

def cast(value, dt):
    return torch.tensor(value, dtype=torch.float32).to(dt).float().item()

print()
for label, v in (("1.001", 1.001), ("1e-8, a small gradient", 1e-8), ("70000, a large activation", 70000.0)):
    cells = "  ".join(f"{name(dt)}={cast(v, dt):<10.7g}" for dt in (torch.float32, torch.bfloat16, torch.float16))
    print(f"{label:<27} {cells}")

print()
w = torch.tensor(0.5, dtype=torch.bfloat16)
print(f"bf16: 0.5 + 0.0001 = {(w + 0.0001).item()}    fp32: 0.5 + 0.0001 = {(torch.tensor(0.5) + 0.0001).item()}")

print()
torch.manual_seed(0)
weights = torch.randn(1_000_000) * 0.02          # weights of the size initializer_range=0.02 implies
def report(label, back):
    rel = ((back - weights).abs() / weights.abs()).mean().item()
    print(f"{label:<28} mean relative rounding error {rel:.4f}   values that became 0: {(back == 0).sum().item()}")
for dt in (torch.bfloat16, torch.float16, torch.float8_e4m3fn, torch.float8_e5m2):
    report(name(dt), weights.to(dt).float())
scale = 64.0                                     # one scale for the tensor: lift 0.02-sized values into E4M3's range
report("float8_e4m3fn, scaled x64", (weights * scale).to(torch.float8_e4m3fn).float() / scale)

Output — what you should see

formats.py, torch 2.14.0, CPU

dtype          bits        eps         max  smallest normal
float32          32   1.19e-07  3.4028e+38         1.18e-38
bfloat16         16   7.81e-03  3.3895e+38         1.18e-38
float16          16   9.77e-04       65504         6.10e-05
float8_e4m3fn     8   1.25e-01         448         1.56e-02
float8_e5m2       8   2.50e-01       57344         6.10e-05

1.001                       float32=1.001       bfloat16=1           float16=1.000977
1e-8, a small gradient      float32=1e-08       bfloat16=1.001172e-08  float16=0
70000, a large activation   float32=70000       bfloat16=70144       float16=inf

bf16: 0.5 + 0.0001 = 0.5    fp32: 0.5 + 0.0001 = 0.5001000165939331

bfloat16                     mean relative rounding error 0.0014   values that became 0: 0
float16                      mean relative rounding error 0.0002   values that became 0: 1
float8_e4m3fn                mean relative rounding error 0.1017   values that became 0: 38990
float8_e5m2                  mean relative rounding error 0.0454   values that became 0: 317
float8_e4m3fn, scaled x64    mean relative rounding error 0.0238   values that became 0: 624

The PyTorch documentation lists torch.finfo for the 32- and 16-bit types; it also answers for the two float8 dtypes in 2.14.0 on the CPU, which is how the last two rows of the first block were produced. Line by line:

BF16 against FP16. BF16 cannot hold 1.001, because its step above 1.0 is 0.0078; it keeps about three significant digits, where FP16 keeps 1.000977. FP16 in turn turns a gradient of 1e-8 into 0 and an activation of 70,000 into inf; BF16 keeps both, with error. NVIDIA’s mixed-precision guide gives the reason in two numbers: “Half precision dynamic range, including denormals, is 40 powers of 2. For comparison, single precision dynamic range including denormals is 264 powers of 2.” BF16 has FP32’s exponent and therefore FP32’s range. Loss scaling is the FP16 workaround, “You can scale the loss values computed in the forward pass, before starting backpropagation. By the chain rule, backpropagation ensures that all the gradient values of the same amount are scaled”, and PyTorch’s automatic mixed precision package implements it; hardware that computes in BF16 made it optional, which is why BF16 became the training format.

The lost update. 0.5 + 0.0001 is 0.5 in BF16. An update smaller than half the format’s step at the weight’s magnitude changes nothing, and a learning rate of 1e-4 times a gradient of 1 is such an update. This is why training keeps the weights in FP32 as well, “Maintain a primary copy of weights in FP32” in NVIDIA’s guide, and does the multiplications in BF16. Part 11’s memory arithmetic counts the bytes that copy costs.

Why eight bits always come with a scale. Cast weights of the size a model’s initializer_range of 0.02 implies straight into E4M3 and the mean rounding error is ten per cent, with four per cent of the weights collapsing to zero, because E4M3’s smallest normal value is 0.0156 and most weights sit below it, where the format falls back to subnormals down to about 0.002 with fewer bits of precision: values below about 0.001 round to zero and the rest are rounded coarsely. Multiply by 64 first, cast, divide back, and the error falls to 2.4 per cent with almost none lost: the scale moves the values into the part of the range where the format has its precision. Every format of eight bits or fewer therefore ships with a scale per tensor, per channel or per block, and the bits that scale occupies are part of the file.

INT8 and INT4 make the scale the whole design. A block of consecutive weights shares one scale and each weight becomes a small integer code:

scale = max|x| / qmax         qmax = 127 for 8-bit, 7 for 4-bit
q     = round(x / scale)      an integer in [-qmax - 1, qmax]
x'    = q × scale             what the engine multiplies with

RunnableAll tracks

block-quant.py

"""Quantise one block of 32 weights to 8-bit and 4-bit integers with one shared scale; count the bits."""
import numpy as np

rng = np.random.default_rng(0)
block = (rng.standard_normal(32) * 0.02).astype(np.float32)   # 32 consecutive weights of one tensor
rms = float(np.sqrt(np.mean(block ** 2)))

def quantise(x, bits):
    qmax = 2 ** (bits - 1) - 1                   # 127 for 8-bit, 7 for 4-bit: a symmetric integer grid
    scale = np.float16(np.abs(x).max() / qmax)   # one 16-bit scale for the whole block
    q = np.clip(np.round(x / np.float32(scale)), -qmax - 1, qmax).astype(np.int8)
    return q, scale, q * np.float32(scale)       # codes, scale, and the values they map back to

print("original    :", np.round(block[:6], 4).tolist(), f"  rms of block = {rms:.4f}")
for bits in (8, 4):
    q, scale, back = quantise(block, bits)
    err = np.abs(back - block)
    print(f"{bits}-bit codes : {q[:6].tolist()}   scale = {float(scale):.2e}")
    print(f"   restored : {np.round(back[:6], 4).tolist()}")
    print(f"   bits per weight = (32 x {bits} + 16) / 32 = {(32 * bits + 16) / 32:.2f}   "
          f"mean |error| = {err.mean():.1e} = {err.mean() / rms:.1%} of the block's rms   "
          f"codes equal to 0: {(q == 0).sum()} of 32")

Output — what you should see

block-quant.py, numpy 2.5.2, CPU

original    : [0.0024999999441206455, -0.0026000000070780516, 0.012799999676644802, 0.002099999925121665, -0.010700000450015068, 0.007199999876320362]   rms of block = 0.0163
8-bit codes : [7, -7, 35, 6, -29, 20]   scale = 3.66e-04
   restored : [0.0026000000070780516, -0.0026000000070780516, 0.012799999676644802, 0.002199999988079071, -0.010599999688565731, 0.007300000172108412]
   bits per weight = (32 x 8 + 16) / 32 = 8.50   mean |error| = 7.3e-05 = 0.5% of the block's rms   codes equal to 0: 0 of 32
4-bit codes : [0, 0, 2, 0, -2, 1]   scale = 6.64e-03
   restored : [0.0, 0.0, 0.013299999758601189, 0.0, -0.013299999758601189, 0.006599999964237213]
   bits per weight = (32 x 4 + 16) / 32 = 4.50   mean |error| = 1.4e-03 = 8.6% of the block's rms   codes equal to 0: 6 of 32

The bits-per-weight lines are exact, not approximate: ggml’s block_q8_0 is one 16-bit scale plus 32 one-byte codes, 34 bytes per 32 weights, 8.5 bits per weight, and block_q4_0 is one 16-bit scale plus 16 bytes of nibbles, 18 bytes per 32 weights, 4.5 bits per weight. Those are the two oldest GGUF types, and the snippet is their algorithm. The K-quants that Part 6’s GGUF lesson teaches spend the bits differently: two FP16 super-block scales per 256 weights, one for the block scales and one for the block minimums, plus a 6-bit scale and a 6-bit minimum per block of 32, which also comes to the 4.5 bits per weight the Hub’s GGUF table lists for Q4_K (block_q4_K is 2 + 2 + 12 + 128 = 144 bytes per 256 weights), and the _M mixes keep some tensors at a wider type, which is why Qwen3-8B’s Q4_K_M file is 5.0 GB, 0.61 bytes or 4.9 bits per parameter, rather than 0.5. Part 4’s memory-budget lesson carries the exact bytes per parameter for every format. The error line is the price: at 4 bits, six of the thirty-two weights became exactly zero and the mean error is nine per cent of the block’s typical size, against half a per cent at 8 bits. What that does to the model’s output is a measurement, and Part 16’s measurement lesson makes it.

The four-bit floating-point formats are the same idea with a floating-point element. MXFP4 stores E2M1 elements in blocks of 32 with one power-of-two scale per block; NVFP4 keeps the E2M1 element, shrinks the block to 16, gives each block a fractional E4M3 scale and adds “a second-level FP32 scalar applied per tensor”. Blackwell tensor cores multiply with them natively, and gpt-oss-20b (Apache-2.0) is the course’s example of a model shipped that way: its card says the models “were post-trained with MXFP4 quantization of the MoE weights”. Part 16’s Blackwell-formats lesson takes them apart bit by bit; here they are two more rows of bytes per parameter.

Format	Use it for	Needs	Where the course uses it
FP32	the primary copy of the weights and the optimiser’s state in training; reference numerics	nothing special	Part 11
BF16	training compute; unquantised inference; the baseline every quantisation is measured against	BF16 support in the framework build, which the PyTorch and MLX builds this part’s lab installs provide	Parts 2, 11 to 15
FP16	inference on older GPUs without BF16; training only with loss scaling	a check that activations stay below 65,504	avoided where BF16 exists
FP8 (E4M3 weights, E5M2 gradients)	serving throughput on Hopper and Blackwell; a half-size checkpoint	tensor cores with FP8 support	Parts 9 and 16
INT8, Q8_0	local inference at about half the bytes of BF16; the first quantised format to compare against BF16	any engine	Parts 6 and 16
INT4, Q4_K_M and relatives	memory-limited local inference; the default for the 8 GB tier	any engine, and a measurement of what was lost	Parts 6 and 16
MXFP4, NVFP4	models shipped in the format; native four-bit arithmetic on Blackwell	Blackwell tensor cores for native arithmetic; elsewhere the engine converts the blocks as it reads them	Part 16

✕Common mistake

A model trained in BF16 and run in FP16 can produce inf and then nan in its logits or its loss, and nothing else is reported, because activations that BF16 held comfortably exceed FP16’s largest value of 65,504, as the 70000 line above shows. The symptom is a generation that turns into garbage after a few tokens, or a training log whose loss reads nan. The fix is to load in BF16 or FP32; the loss scaling that rescues small gradients does nothing for large activations.

What precision costs, and where it does not

Lower precision is cheaper than intuition suggests, and the snippets say why. A rounding error on one weight is a small number of either sign, independent of the errors on its neighbours. A layer’s output is a dot product over thousands of such weights, so the errors partly cancel rather than add up, and the output moves by far less than the per-weight error. The network was trained by a noisy procedure on noisy data and learned to be robust to perturbations of about this kind. That is why a four-bit model with a nine-per-cent per-weight error can be indistinguishable from the original on most tasks.

Usually is not always. The cancellation argument is weakest where a few weights carry most of the signal, which is why the embeddings, the output head and some attention matrices are the ones a _M mixture keeps wider, and why the importance-weighted methods of Part 16’s post-training quantisation lesson scale the sensitive channels up before rounding. Small models suffer more than large ones, because they have fewer parameters to absorb the noise, and the damage shows on hard tasks before easy ones. The right response is the one this course takes everywhere: measure it, on your task, against the unquantised model, with the perplexity and divergence tools of Part 16’s measurement lesson, and keep the format the measurement supports.

Training is less tolerant than inference for a reason the 0.5 + 0.0001 line makes exact: a gradient step is a small change to a large number, and it has to survive the addition. That is why the training-memory arithmetic in Part 11 counts sixteen bytes per parameter where the inference arithmetic of Part 4 counts two.

Follow shapes through a language-model operation

Take a batch of two sequences, each with five token positions, and an embedding width of four. Token IDs have shape [2, 5]; looking up their vectors produces [2, 5, 4]. A projection with input width four and output width three produces [2, 5, 3]. The projection applies at every position; it does not combine the five positions. Attention is a separate operation that mixes information across positions.

Write these shapes beside the code before calculating memory. A tensor’s element count is the product of its dimensions, and its storage estimate is that count times bytes per element. A view may share storage with another tensor, whereas a copy creates another allocation. Summing the apparent sizes of all views can therefore double-count memory.

For diagnosis, distinguish three questions: is the shape correct, is the tensor on the intended device, and is the arithmetic using the intended dtype? A successful import answers none of them. A GPU allocation also does not prove every operation remains on the GPU; profile the actual forward and backward path when the measured runtime contradicts the estimate.

Recap

Everything in a model is a tensor with a shape, and the shape times the bytes per element is the memory. Almost all of the work is matrix multiplication, two operations per weight per token, and a GPU is a machine for doing that one operation in parallel. Whether a job is limited by arithmetic or by bytes is its intensity against the chip’s ridge point: decode at batch 1 is one FLOP per byte and waits on memory, so its ceiling is bandwidth divided by the bytes of weights per token; prefill and training reuse each weight thousands of times and are limited by compute. Memory is a hierarchy in which each level is several times slower than the one above, and a model must fit in the level its accelerator reads at full speed. Formats set the bytes per parameter, two at BF16, one at FP8 and Q8, about 0.6 at Q4, and each step down is a rounding error that inference mostly cancels and training mostly cannot.

Check your understanding

Question 1. Qwen3-8B at BF16 is 16.4 GB and its Q4_K_M file is 5.0 GB. On a machine whose vendor bandwidth figure is 273 gigabytes per second, what happens to the decode ceiling, in tokens per second, when you switch from BF16 to Q4_K_M?

• It rises by a factor of four, from a ceiling of about 17 to about 67, because 4 bits is a quarter of 16
• It rises by a factor of about 3.3, from a ceiling of about 17 to about 55, because the file is 0.61 bytes per parameter once the block scales are counted
• It does not change, because the arithmetic per token is the same
• It falls, because dequantising the weights costs extra operations

Show the answer and why

Answer: It rises by a factor of about 3.3, from a ceiling of about 17 to about 55, because the file is 0.61 bytes per parameter once the block scales are counted

The ceiling is bandwidth divided by bytes read per token: 273 / 16.4 is 16.6 and 273 / 5.0 is 54.6. The gain is the ratio of the file sizes, 3.3, not the ratio of nominal bit widths, because Q4_K_M spends 4.5 bits per weight on codes and sub-scales and keeps some tensors wider. Dequantising costs arithmetic, but decode at batch 1 has arithmetic to spare; it is waiting on bytes.

Question 2. Which of these training-loop fragments silently stops learning?

• master = model.float(); loss.backward(); optimizer.step() with the optimizer holding FP32 parameters
• w = w.to(torch.bfloat16); w -= 1e-4 * grad with grad of size about 1 and w of size about 0.5
• with torch.autocast(device_type='cuda', dtype=torch.bfloat16): loss = model(x)
• scaler = torch.amp.GradScaler(); scaler.scale(loss).backward() with FP16 activations

Show the answer and why

Answer: w = w.to(torch.bfloat16); w -= 1e-4 * grad with grad of size about 1 and w of size about 0.5

An update of 1e-4 to a BF16 weight of 0.5 rounds back to 0.5, because BF16's step near 0.5 is 0.0039 and anything below half of it vanishes; the loss stays flat and nothing is reported. The other three are the mixed-precision recipe: BF16 arithmetic under autocast, an FP32 primary copy in the optimiser, and loss scaling when the compute format is FP16.

Question 3. x has shape (32, 4096) and a checkpoint stored a weight as (12288, 4096). x @ w raises a shape error. Which fix is the bug?

• y = x @ w.T
• y = x @ w.reshape(4096, 12288)
• y = torch.nn.functional.linear(x, w), which computes x @ w.T
• y = (w @ x.T).T

Show the answer and why

Answer: y = x @ w.reshape(4096, 12288)

reshape keeps the buffer's element order and only relabels it, so every value lands in a different row and column than it belonged to; the multiplication runs, the shapes look right, and the numbers are wrong. A transpose changes how the same buffer is read and preserves which input feeds which output. The other three are equivalent to the correct product.

Question 4. A prompt of 2,048 tokens takes T seconds to prefill on your machine. Roughly what does a 4,096-token prompt take, and why?

• About T: the weights are read once either way
• About 2T: prefill is compute-bound, and the arithmetic grows with the number of tokens, with a little extra from attention
• About T/2: the batch is larger, so the intensity is higher and the chip is more efficient
• About 4T: every token attends to every other token

Show the answer and why

Answer: About 2T: prefill is compute-bound, and the arithmetic grows with the number of tokens, with a little extra from attention

Prefill sits above the ridge point, so its time is FLOPs divided by peak, and the matrix-multiplication FLOPs are 2 × parameters × tokens: double the tokens, double the work. Attention adds a term that grows with the square of the length, which is small at these lengths and dominant at very long ones. Higher intensity stops the chip waiting on bytes; it does not make the arithmetic free.

Question 5. A 20 GB quantised model is loaded on a card with 16 GB of VRAM whose bandwidth is 1,792 gigabytes per second, with the remaining 4 GB in system memory behind a PCIe Gen5 x16 link at about 64 gigabytes per second per direction. What is the decode ceiling, in tokens per second, compared with a card that held all 20 GB?

• About 90 either way; the 4 GB is only a fifth of the model
• About 14, because the 4 GB behind PCIe takes 62 ms per token while the 16 GB in VRAM takes 9 ms; a card holding all of it would allow about 90
• About 45: half the speed, because half the links are slow
• Zero: a model larger than VRAM cannot run

Show the answer and why

Answer: About 14, because the 4 GB behind PCIe takes 62 ms per token while the 16 GB in VRAM takes 9 ms; a card holding all of it would allow about 90

Time per token is the sum over levels of bytes divided by bandwidth: 16 / 1792 is 8.9 ms and 4 / 64 is 62.5 ms, so 71.4 ms per token and a ceiling of 14. The slow fifth takes seven eighths of the time. It runs, which is what makes this mistake easy to make; it just runs in a different speed class.

Question 6. Why does every 8-bit and 4-bit weight format carry a scale factor per block or per tensor, when BF16 does not?

• To record which quantisation tool produced the file
• Because the small formats have too little exponent range to place values of typical weight size, around 0.02, where their precision is: E4M3's smallest normal value is 0.0156, and a 4-bit integer has no exponent at all, so the scale moves each block into the format's usable range
• Because integer arithmetic units require it
• To make the file compatible with GGUF

Show the answer and why

Answer: Because the small formats have too little exponent range to place values of typical weight size, around 0.02, where their precision is: E4M3's smallest normal value is 0.0156, and a 4-bit integer has no exponent at all, so the scale moves each block into the format's usable range

BF16 keeps FP32's eight exponent bits, so it can place a value of any magnitude to three digits without help. E4M3 has four exponent bits and a floor of 0.0156, and cast directly it zeroes four per cent of typical weights and rounds the rest at ten per cent; multiplying the block by a scale first brings the error down to two per cent. An integer grid has no exponent, so the scale is the only thing that sets its range. The scale's bits are why Q4_0 is 4.5 bits per weight, not 4.

Check my answersClear