Memory Arithmetic for Training: Weights, Gradients, Optimiser States and Activations

By the end of this lesson you will be able to work out, on paper and before downloading anything, how much memory a full fine-tune, a LoRA run or a QLoRA run of a given model needs, and therefore which of the three your machine can actually do. This is the page Parts 12 to 17 cite most, and the page that decides whether a run in Part 13 finishes or dies twenty minutes in.

Part 4 answered the inference question: weights plus key-value cache plus headroom. Training is the same style of arithmetic with more terms and no cache at all. Five things occupy memory during a training step, and the first four are all proportional to the parameter count.

The five terms

Weights. The parameters themselves, in whatever precision the forward pass computes in. Two bytes each at BF16, the working precision of modern training.

Gradients. One number per trainable parameter, produced by the backward pass and held until the optimiser has used them. Two bytes each at BF16.

Optimiser states. Adam keeps running estimates of the first and second moments of the gradient; the paper describes the method as “based on adaptive estimates of lower-order moments”, and PyTorch’s AdamW page shows the algorithm initialising a first moment and a second moment per parameter. Two states, kept in FP32, is eight bytes per parameter. This is the term people forget, and it is the largest one.

Master weights. Mixed-precision training keeps a full-precision copy of the weights that the optimiser updates, because a small update added to a BF16 weight can round away to nothing. NVIDIA’s mixed-precision guidance states the requirement directly: “Maintain a primary copy of weights in FP32.” Four bytes per parameter.

Activations. Everything the forward pass computed that the backward pass will need. Unlike the first four, this term does not scale with the parameter count; it scales with batch size, sequence length and the model’s shape, and it is the term you control with settings rather than with the choice of model.

Bytes per parameter, by role

Role	Precision	Bytes per parameter	Where the figure comes from
Weights, forward pass	BF16	2	Sixteen bits
Gradients	BF16	2	One per trainable parameter
Adam first and second moments	FP32	8	Two states, four bytes each
Adam moments, 8-bit optimiser	INT8	2	bitsandbytes documents a 75% reduction
Master weights	FP32	4	NVIDIA: maintain a primary copy in FP32
Full fine-tune, total	mixed	16	2 + 2 + 8 + 4

Sixteen bytes per parameter is the number to memorise. A full fine-tune of a model needs about eight times what running that model at BF16 needs, and about twenty-six times what running it at Q4_K_M needs, before a single activation is stored.

▤Memory budget

The 8-bit optimiser row is the cheapest single saving available. The bitsandbytes documentation states that “large models can be finetuned with 75% less GPU memory without losing any accuracy compared to training with standard 32-bit optimizers”, through bnb.optim.Adam8bit and its relatives. Applied to the eight bytes of Adam state, that is two bytes per parameter, taking a full fine-tune from sixteen bytes per parameter to ten. The same page notes that tensors with fewer than 4,096 elements stay at 32 bits, so the practical figure is a little above two.

A full fine-tune, worked

Qwen3-1.7B has 1.7 billion parameters. At sixteen bytes each the fixed part of the budget is 27.2 GB: 3.4 GB of BF16 weights, 3.4 GB of gradients, 6.8 GB of FP32 master weights and 13.6 GB of Adam states. Its config gives 28 layers and a hidden size of 2,048, so the checkpointed activations at batch 1 and 1,024 tokens are small, and the logits are the larger part of what is left.

Full fine-tune of Qwen3-1.7B, batch 1 at 1,024 tokens, on a 32 GB machine

Weights, BF16

3.4 GB

Gradients, BF16

3.4 GB

FP32 master weights and Adam states

20.4 GB

Activations and logits

0.4 GB

Free

4.4 GB

Total

32 GB

Estimate from arithmetic, not a measurement. Weights, gradients, master weights and Adam states are 16 bytes per parameter against the published 1.7 billion; activations are the checkpointed figure from this lesson's formula plus the logits tensor. No reserve for the operating system is shown, and on a unified-memory machine you would need one.

Two things jump out. The optimiser and master weights are three quarters of the budget for a model whose weights are 3.4 GB. And a 32 GB machine can only just do this, with nothing left for the operating system, a larger batch or a longer sequence.

Now scale it by five. Qwen3-8B has 8.2 billion parameters, so the fixed part is 131.2 GB.

Full fine-tune of Qwen3-8B, batch 1 at 1,024 tokens, on a 128 GB machine

128 GB budget

Weights, BF16

16.4 GB

Gradients, BF16

16.4 GB

FP32 master weights and Adam states

98.4 GB

Activations and logits

0.6 GB

Requested

131.8 GB

Machine budget

128 GB

Over budget. 131.8 GB requested against a 128 GB machine - 3.8 GB over. Something here has to shrink: a smaller quantisation, a shorter context, or fewer of these reservations at once.

Estimate from arithmetic. The same 16 bytes per parameter against 8.2 billion parameters overruns a 128 GB machine before the operating system is counted, which is why a model of this size is fine-tuned with adapters on hardware of this size rather than in full.

That is the honest picture of full fine-tuning at home. A model that runs comfortably in five gigabytes at Q4_K_M does not fit in a hundred and twenty-eight for a full fine-tune. Everything else in this lesson is a way around that arithmetic.

Activations, and the term nobody predicts

Activations are the one term that depends on your settings rather than on the model alone.

With activation checkpointing, the stored activations are roughly

Pseudocode — not a real command

bytes = layers x batch x sequence x hidden_size x bytes_per_element

because one tensor is kept at each layer boundary and everything inside the block is recomputed. PyTorch’s checkpoint documentation describes exactly that trade: tensors produced inside the checkpointed function “are not kept alive until the backward pass. Instead … the unsaved tensors are recomputed by re-invoking function in the backward pass”. The technique comes from Training Deep Nets with Sublinear Memory Cost, whose abstract claims “O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch”, and reports taking a thousand-layer residual network from 48 GB to 7 GB for about thirty per cent more runtime.

Without checkpointing, every intermediate inside each block is kept: the attention projections, the multi-layer perceptron’s intermediate of size intermediate_size, the normalisations. The multiple depends on the architecture and on which attention implementation is in use, so this course does not publish a number for it. Measure it: run the same script with checkpointing on and off and read the peak memory from the verification script’s last line.

The logits are an activation too, and often the biggest one. The final projection produces a tensor of shape batch by sequence by vocabulary. Qwen3’s vocabulary is 151,936 entries, so at batch 4 and 1,024 tokens the BF16 logits alone are about 1.2 GB, and a cross-entropy that materialises a full-precision copy multiplies that. TRL’s memory guide names this directly: at large vocabulary sizes “the [batch × seq_len × vocab] logits tensor produced by the LM head is one of the dominant activations held in memory across forward and backward”. Its default loss_type="chunked_nll" avoids materialising it all at once, but the same page states that the chunked path is “not compatible with use_liger_kernel=True, PEFT, or VLM”. A LoRA run is a PEFT run, so in this part’s lab the full logits tensor is held, and doubling the batch size on a 0.6 billion parameter model costs more than intuition suggests.

✦Tip

Two settings move the activation term and nothing else. Gradient accumulation trains on an effective batch of batch_size × gradient_accumulation_steps while only ever holding batch_size in memory, because the gradients are summed before the optimiser steps. Sequence length is linear in this formula and worse than linear in attention, so trimming max_length to what your examples actually need is usually the cheapest fix for an out-of-memory error.

LoRA: freeze the base, train two small matrices

PEFT’s conceptual guide describes LoRA as representing weight updates with two smaller matrices through low-rank decomposition, with “the original weight matrix” frozen and receiving no further adjustments. The consequence for this lesson is precise, and it follows from the autograd rule in this part’s first lesson: no gradients and no optimiser states exist for frozen parameters.

So the budget becomes the base weights at their storage precision, plus the full sixteen bytes per parameter for the adapter alone, plus activations.

How big is the adapter? An adapter of rank r on a linear layer with in inputs and out outputs adds r × (in + out) parameters. Applying rank 16 to the seven projections of every block of Qwen3-8B, whose config gives 36 layers, a hidden size of 4,096 and an intermediate size of 12,288, gives about 43.7 million trainable parameters: half a per cent of the model. PEFT’s own print_trainable_parameters() prints exactly this ratio, and the lab’s script calls it.

LoRA fine-tune of Qwen3-8B, rank 16, batch 1 at 1,024 tokens, on a 24 GB machine

Frozen base weights, BF16

16.4 GB

Adapter weights, gradients and Adam states

0.7 GB

Activations and logits

0.6 GB

Reserved for the operating system

2 GB

Free

4.3 GB

Total

24 GB

Estimate from arithmetic. The frozen base is 2 bytes per parameter against 8.2 billion; the adapter is about 43.7 million parameters at 16 bytes each; activations are the checkpointed figure plus the logits tensor; the reserve is an allowance rather than a measurement.

The same model that overran a 128 GB machine for a full fine-tune fits on a 24 GB card. The adapter’s own three terms are under a gigabyte, and the base weights are simply resident, exactly as they would be for inference.

QLoRA: quantise the frozen part

If the base is frozen and never updated, it does not need to be stored at full precision either. The QLoRA paper describes an approach that “backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA)”, using 4-bit NormalFloat, double quantisation to quantise the quantisation constants, and paged optimisers for memory spikes, and reports finetuning a 65 billion parameter model “on a single 48GB GPU while preserving full 16-bit finetuning task performance”.

For arithmetic, treat a 4-bit base as roughly 0.55 bytes per parameter: four bits plus the quantisation constants, in the same spirit as Part 4’s per-block scale overhead. That takes the largest term down by a factor of nearly four.

QLoRA fine-tune of Qwen3-30B-A3B, rank 16, batch 1 at 1,024 tokens, on a 32 GB machine

Frozen base weights, 4-bit

16.8 GB

Adapter weights, gradients and Adam states

0.5 GB

Activations and logits

0.5 GB

Reserved for the operating system

4 GB

Free

10.2 GB

Total

32 GB

Estimate from arithmetic. The frozen base is about 0.55 bytes per parameter against 30.5 billion total parameters; the adapter is about 32.2 million parameters at 16 bytes each; activations use the model's 48 layers and hidden size of 2,048 plus the logits tensor; the reserve is an allowance for a unified-memory machine.

A thirty-billion-parameter mixture-of-experts model, fine-tuned on a 32 GB machine. Note what the mixture-of-experts architecture does and does not change: all 30.5 billion parameters must be resident, because any of them may be routed to, so the weight term uses the total rather than the active count. What it changes is speed, not this budget.

What fits per tier

The table below is the arithmetic of this lesson applied to five models at three methods. The three memory columns are the fixed part: weights, gradients, optimiser states and master weights. The two tier columns add the activations and logits at batch 1 and 1,024 tokens plus a 2 GB reserve, and name the smallest tier from the hardware reference that the total fits in.

Pending validationTraining memory by model and method, from arithmetic

Model	Full fine-tune, GB	LoRA, GB	QLoRA, GB	Smallest tier for LoRA	Smallest tier for QLoRA
Qwen3-0.6B	9.6	1.36	0.49	8 GB	8 GB
Qwen3-1.7B	27.2	3.68	1.21	8 GB	8 GB
Qwen3-4B	64	8.53	2.73	12-16 GB	8 GB
Qwen3-8B	131.2	17.1	5.21	24 GB	8 GB
Qwen3-30B-A3B	488	61.52	17.29	96 GB	24 GB

memory tiers from the course hardware reference, not one machine · PyTorch with Transformers, TRL and PEFT transformers 5.16.1, trl 1.12.0, peft 0.20.0 · as listed, at their published parameter counts, BF16 base for full fine-tune and LoRA; 4-bit base for QLoRA · 1,024 tokens of context · 2026-09-09

Arithmetic, not measurement: 16 bytes per parameter for a full fine-tune, 2 bytes per parameter plus a rank-16 adapter at 16 bytes per adapter parameter for LoRA, and about 0.55 bytes per parameter for a 4-bit base under QLoRA. Tier columns add checkpointed activations and the logits tensor at batch 1 and 1,024 tokens plus a 2 GB reserve. Two rows sit close to a boundary: Qwen3-8B under QLoRA lands just inside 8 GB with nothing to spare, and Qwen3-30B-A3B under LoRA lands just outside 48-64 GB. The validation pass replaces these with measured peak memory per track.

Read the columns rather than the rows. Full fine-tuning leaves the domestic range somewhere between one and two billion parameters. LoRA moves the boundary up by roughly a factor of eight, because the sixteen bytes per parameter apply to under one per cent of the model. QLoRA moves it up again by nearly four, because the term that is left is the frozen base.

Per track

Three specifics change the sums, and all three are in the hardware reference.

On Track N the budget is VRAM and only VRAM. Spilling to system memory across PCIe works for inference and is close to useless for training, because every step touches every weight several times.

On Track S, Track X and Track M the pool is shared with the operating system, so the reserve is real and larger than a card’s. Track X has the additional constraint from Part 5 that the memory visible to the GPU is capped below the machine’s total.

On Track M the arithmetic is the same but the framework is not: mlx-lm’s LoRA path holds the same five terms in unified memory, and the number of adapted layers is a command-line option rather than a list of module names.

✕Common mistake

An out-of-memory error at step 300 of a run that started fine is almost never a leak. It is the longest example in the dataset arriving in a batch. Activation memory scales with the actual sequence length of the batch being processed, so a run sized against the average will fail against the maximum. Set max_length deliberately, and check the length distribution of your data before you start rather than after the failure.

Audit the assumptions behind bytes per parameter

The familiar mixed-precision training estimate is an accounting example, not a universal constant. Weight dtype, gradient dtype, optimiser state, master copies, sharding and implementation details decide the actual fixed term. Some configurations do not keep the exact collection of copies assumed by a textbook estimate. Inspect the chosen optimiser and measure peak allocation before treating the estimate as a fit guarantee.

For an adapter run, split frozen base storage from trainable adapter storage. Optimiser states follow trainable parameters, but activations still arise from propagating through the base. Freezing parameters therefore does not make the forward/backward activation cost disappear.

Build a two-axis experiment: keep sequence length fixed while increasing microbatch size, then hold microbatch fixed while increasing sequence length. Record the phase of peak allocation, including the first optimiser step, which may initialise state lazily. Gradient accumulation increases effective batch without retaining all microbatch activations at once, but it does not shorten a single oversized example. Use the shape of the memory increase to decide which setting to reduce.

Recap

Training memory is four terms proportional to the trainable parameter count plus one that is not. Weights at two bytes, gradients at two, Adam’s two moments at eight, and FP32 master weights at four make sixteen bytes per trainable parameter for a full fine-tune; an 8-bit optimiser takes the Adam term from eight bytes to two. Activations depend on batch, sequence and the model’s shape, are cut by checkpointing at the cost of about one extra forward pass, and include a logits tensor whose size is batch by sequence by vocabulary. LoRA freezes the base so that only the adapter, well under one per cent of the model, carries gradients and optimiser states; QLoRA additionally quantises the frozen base to about 0.55 bytes per parameter. The practical consequence is a ladder: full fine-tuning at home stops at one to two billion parameters, LoRA reaches eight to fourteen billion on a mid-sized card, and QLoRA reaches a thirty-billion-parameter mixture of experts on a 32 GB machine.

Check your understanding

Question 1. A 3 billion parameter model is fully fine-tuned in mixed precision with AdamW. What is the fixed part of the budget, before activations?

• 6 GB, the weights at BF16
• 48 GB: 16 bytes per parameter for weights, gradients, master weights and Adam states
• 12 GB, weights plus gradients
• 24 GB, weights plus Adam states

Show the answer and why

Answer: 48 GB: 16 bytes per parameter for weights, gradients, master weights and Adam states

Two bytes of BF16 weights, two of gradients, four of FP32 master weights and eight of Adam moments is sixteen bytes per parameter, so 3 x 16 = 48 GB. The optimiser and master weights are three quarters of it.

Question 2. Why does LoRA reduce memory so much more than it reduces compute?

• Because the adapters are computed on the CPU
• Because gradients and optimiser states exist only for trainable parameters, and the frozen base still has to be read in the forward and backward passes
• Because the base model is quantised
• Because the batch size is smaller

Show the answer and why

Answer: Because gradients and optimiser states exist only for trainable parameters, and the frozen base still has to be read in the forward and backward passes

The base weights remain resident and are still multiplied by, so the arithmetic per step is barely changed. What disappears is the twelve bytes per parameter of gradients, master weights and optimiser states for the frozen part.

Question 3. Which of these reduce activation memory? Select all that apply.

• Gradient checkpointing
• Gradient accumulation with a smaller per-device batch
• A shorter max_length
• An 8-bit optimiser

Show the answer and why

Answer: Gradient checkpointing, Gradient accumulation with a smaller per-device batch, A shorter max_length

The first three all act on the batch, sequence and layer terms. An 8-bit optimiser reduces the optimiser-state term, which is proportional to the parameter count and has nothing to do with activations.

Question 4. A 0.6 billion parameter model with a 151,936-entry vocabulary runs out of memory when the batch size goes from 2 to 8. Which term grew most?

• The weights
• The Adam states
• The logits tensor, which is batch by sequence by vocabulary and does not depend on the parameter count
• The master weights

Show the answer and why

Answer: The logits tensor, which is batch by sequence by vocabulary and does not depend on the parameter count

The first four terms are fixed by the parameter count and do not change with batch size at all. The logits are batch by sequence by vocabulary, so quadrupling the batch quadruples them, which for a small model with a large vocabulary is the dominant activation.

Question 5. For a mixture-of-experts model such as Qwen3-30B-A3B, which parameter count belongs in the weight term of a QLoRA budget?

• The active parameters, because only those are computed with per token
• The total parameters, because any expert may be routed to and all of them must be resident
• Half the total, by convention
• Neither; the formula does not apply to mixtures of experts

Show the answer and why

Answer: The total parameters, because any expert may be routed to and all of them must be resident

Active parameters set the arithmetic per token and therefore the speed. Memory is set by what must be resident, which is the total. The same distinction governs the inference arithmetic in Part 4.

Check my answersClear