LoRA and QLoRA Explained
By the end of this lesson you will be able to write a LoraConfig deliberately rather than by
copying one, say what each field costs in memory and in capacity, explain what QLoRA quantises and
what it does not, and name the three things an adapter cannot do however you configure it.
Part 11 gave the arithmetic: sixteen bytes per trainable parameter for a full fine-tune, which takes an eight-billion-parameter model past a 128 GB machine before a single activation is stored. Adapters are the technique that makes the number small again, and they do it by changing what “trainable” means rather than by compressing anything.
The low-rank idea
Section titled “The low-rank idea”Start with one weight matrix inside the model, of shape in by out. Full fine-tuning replaces it
with the matrix plus an update of exactly the same shape, learned from your data. The update has as
many numbers in it as the original.
LoRA’s observation is that this update does not need that many numbers. PEFT’s conceptual guide
states the method in one sentence: “LoRA’s approach is to represent the weight updates with two
smaller matrices (called update matrices) through low-rank decomposition.” Instead of one in by
out update you train an in by r matrix and an r by out matrix, with r small: sixteen,
thirty-two, sixty-four. Their product has the shape of the update, so it can be added to the
original weight; but it has far fewer numbers in it, and only those numbers carry gradients.
The other half of the method is what happens to the original: “The original weight matrix remains frozen and doesn’t receive any further adjustments. To produce the final results, both the original and the adapted weights are combined.” Frozen means no gradient, and Part 11’s autograd rule then removes three of the five memory terms at a stroke: no gradient, no optimiser state and no full-precision master copy exists for a parameter that is not being trained.
The original paper puts the saving in its own terms. Relative to GPT-3 at 175 billion parameters fine-tuned with Adam, the abstract claims LoRA “can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times”, while adding “no additional inference latency”, because the two small matrices can be folded back into the weight they adapted once training is over.
Where the adapter sits
- Input activationsThe hidden state arriving at this projection, of width `in`.
- Frozen base matrix WShape `in` by `out`. Loaded at BF16 for LoRA, or at 4 bits for QLoRA. No gradient, no optimiser state, no master copy.not trained
- Adapter path: A then BA is `in` by r and B is r by `out`. A is initialised so the product starts at zero, so the model begins identical to the base.trained
- Scale by alpha over rA fixed scalar set at initialisation. rsLoRA divides by the square root of r instead.
- Sum, then continueThe base output plus the scaled adapter output. Identical in shape to the layer you replaced.
The initialisation matters more than it looks. PEFT initialises “with Kaiming-uniform for weight A and zeros for weight B resulting in an identity transform”. Because B starts at zero the adapter contributes nothing on the first step, so a freshly attached adapter reproduces the base model exactly. That is why a fine-tune that produces nonsense from step one is a data or template problem rather than an adapter problem.
Rank, alpha, dropout and target modules
Section titled “Rank, alpha, dropout and target modules”Four fields, and each one answers a different question.
Rank, r, is capacity. PEFT’s guide: “A higher rank means the model has more parameters to
train, but it also means the model has more learning capacity.” Rank 8 to 16 is enough for a format
or style change on a small model, which is what the lab in this part does. Rank 32 to 64 is the
range for a domain behaviour change on a few thousand examples, which is the project. Rank buys
capacity linearly and memory linearly, and past the point where your data can fill it, it buys
overfitting.
The size of the adapter follows from r directly. An adapter of rank r on a linear layer with
in inputs and out outputs adds r × (in + out) parameters, so the total is that sum over every
layer you targeted. Part 11’s memory lesson works this out for Qwen3-8B at rank 16 across the seven
projections of every block and gets about 43.7 million trainable parameters, half a per cent of the
model. PEFT’s print_trainable_parameters() prints exactly this ratio, and every script in this
part calls it, because a number that is not the one you expected means your target_modules did not
match what you thought.
Alpha is the scale. The adapter’s contribution is multiplied by a fixed scalar before it is
added, and PEFT documents it as lora_alpha/r in the original implementation. The practical
consequence of the division is that raising r at fixed alpha quietly lowers the scale, which is
why the common convention of alpha = 2r exists: it holds the scale constant while you change
capacity. There is an alternative. Rank-stabilised LoRA “uses lora_alpha/math.sqrt(r) which
stabilises the adapters and increases the performance potential from using a higher r”, enabled
with use_rslora=True, and it is the setting to reach for if a high-rank run is unstable.
Dropout is regularisation. lora_dropout drops activations on the adapter path during training
only. On a few hundred examples, 0.05 to 0.1 is a sensible default; the effect is small compared
with choosing the right number of epochs, and it is not a substitute for a held-out split.
Target modules decide where the adapters go. PEFT’s default “add trainable weights to the query
and value layers of each attention block”, which is the paper’s configuration and is cheap. QLoRA’s
configuration is broader: the same page notes that QLoRA “adds trainable weights to all the linear
layers of a transformer model” and “can provide performance equal to a fully finetuned model”, and
that the way to ask for it is target_modules="all-linear", which PEFT documents as choosing “all
linear/Conv1D modules … (if the model is a PreTrainedModel, the output layer excluded)”.
For the Qwen3 family the seven names are q_proj, k_proj, v_proj, o_proj, gate_proj,
up_proj and down_proj: the four attention projections and the three of the feed-forward block.
Adapting all seven costs about three times what adapting query and value alone costs, and on a
small model that is still under one per cent of the parameters, so this part’s scripts do it by
default.
What it costs, in memory
Section titled “What it costs, in memory”Here is the lab’s own configuration, drawn with Part 11’s arithmetic. A four-billion-parameter base at BF16, a rank-16 adapter on all seven projections, batch 1 at 1,024 tokens, on the 12 GB floor this part is written for.
LoRA fine-tune of Qwen3-4B, rank 16, batch 1 at 1,024 tokens, on a 12 GB machine
- Frozen base weights, BF16
- 8 GB
- Adapter weights, gradients and Adam states
- 0.5 GB
- Activations and logits
- 0.5 GB
- Reserved for the operating system
- 2 GB
- Free
- 1.0 GB
- Total
- 12 GB
Two readings. The adapter’s own three terms are half a gigabyte against a base of eight, which is the whole argument for the method. And the run fits with about a gigabyte to spare, which is why the lab’s default sequence length is short and why doubling the batch size is the first thing to fail on this tier.
Speed is a separate question from memory and the answer is less flattering. A LoRA step still runs the full forward pass through every frozen weight and still backpropagates through them to reach the adapters; what it saves is the optimiser update and the gradient storage. Expect a LoRA step to be faster than a full fine-tuning step on the same model and not by the factor the memory saving suggests, and expect the honest number to come from your own machine rather than from this page.
QLoRA: quantise the part that is frozen
Section titled “QLoRA: quantise the part that is frozen”If the base is never updated, it does not need to be stored at a precision suitable for updating. That is the whole idea, and it takes the largest term in the diagram above down by nearly a factor of four.
The paper describes an approach that “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance”, by backpropagating “gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA)”. Three innovations are named in the abstract: “4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights”, “double quantization to reduce the average memory footprint by quantizing the quantization constants”, and “paged optimizers to manage memory spikes”. The authors also report that their Guanaco model family reached “99.3% of the performance level of ChatGPT” on the Vicuna benchmark with 24 hours of fine-tuning on one GPU; that is a figure reported by the paper’s authors on a benchmark of that period, not a measurement this course makes.
In practice QLoRA is four keys on a configuration object. PEFT documents them together: set
load_in_4bit=True to quantise on load, bnb_4bit_quant_type="nf4" for the data type,
bnb_4bit_use_double_quant=True for “a nested quantization scheme to quantize the already quantized
weights”, and bnb_4bit_compute_dtype=torch.bfloat16 so the arithmetic happens in bfloat16. The
model is then passed through prepare_model_for_kbit_training() before the adapter is attached.
QLoRA fine-tune of Qwen3-8B, rank 16, batch 1 at 1,024 tokens, on a 16 GB machine
- Frozen base weights, 4-bit NF4
- 4.5 GB
- Adapter weights, gradients and Adam states
- 0.7 GB
- Activations and logits
- 0.6 GB
- Reserved for the operating system
- 2 GB
- Free
- 8.2 GB
- Total
- 16 GB
The eight-billion-parameter model that needed a 24 GB card for LoRA now has room to spare on the project’s 16 GB floor, and the space that opens up is usually better spent on a longer sequence length or a larger batch than on a larger model.
DoRA and the other variants
Section titled “DoRA and the other variants”DoRA is the variant most worth knowing, because it is one boolean away and it targets LoRA’s known weakness. The paper introduces “a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA”, and from it proposes decomposing “the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates”. The claimed effect is to enhance “both the learning capacity and training stability of LoRA while avoiding any additional inference overhead”.
PEFT exposes it as use_dora=True and documents the trade honestly: it “can improve the performance
of LoRA especially at low ranks”, “only supports linear and Conv2D layers”, and “introduces a bigger
overhead than pure LoRA, so it is recommended to merge weights for inference”. Read that as: try it
when you are rank-constrained by memory, expect a slower training step, and merge before serving.
Two others are worth a sentence each. use_rslora=True changes the scaling to divide by the square
root of the rank, which is the fix for an unstable high-rank run. And rank_pattern and
alpha_pattern take dictionaries keyed by layer name or regular expression, so different layers can
carry different ranks; it is a tuning knob for people who have already measured that some layers
matter more than others, and it is not where a first fine-tune should start.
What adapters cannot do
Section titled “What adapters cannot do”Three limits, and each one has produced a confused evening for somebody.
An adapter does not shrink the base model. The frozen weights are resident throughout training and throughout serving. LoRA changes what is trainable, not what is loaded, which is why the biggest segment in both diagrams above is still the base. A machine that cannot hold a model at inference cannot LoRA it either.
An adapter cannot change what is not adapted. Tokeniser, vocabulary and embedding matrices are
untouched unless you name them, and adding a new special token therefore requires modules_to_save
and a resized embedding layer rather than a bigger rank. This is the usual cause of a fine-tune that
will not produce a token you invented for it.
Merging into a quantised base is not the same as training against one. An adapter trained
against a 4-bit base and then merged into a full-precision base is being added to weights it never
saw; an adapter merged into a quantised base runs into the rounding of the destination. PEFT’s
quantisation guide is explicit about the general hazard for some backends, warning for torchao that
“merging only works correctly with LoRA and with quant_type = 'int8_weight_only'” and that other
combinations may error or produce incorrect results. The export lesson in this part gives the safe
procedure: merge into the base at BF16, then quantise the merged result. This part’s challenge
includes the worked case of somebody who did not.
Calculate adapter size for a single matrix
Section titled “Calculate adapter size for a single matrix”For a projection with input width 1,024 and output width 2,048, a rank-eight LoRA update uses matrices with 8 × 1,024 and 2,048 × 8 elements. Their total is 24,576 trainable values, compared with 2,097,152 in the original projection. This example explains the storage saving without assuming that all layers in a model have identical shapes.
Sum the actual selected projections across layers before comparing with the printed trainable-parameter count. Check additional trainable modules, such as a saved output head, which can materially change the total. Rank is a capacity choice; the scaling rule, learning rate, target modules and data jointly determine the result.
In QLoRA the base representation is quantised and frozen while the adapter is trained through the computation. That does not mean the optimiser updates four-bit base weights or that every intermediate tensor is four-bit. Distinguish storage dtype, compute dtype and optimiser-state dtype in the notebook. This accounting is what lets you predict which part of a memory reduction comes from freezing parameters and which comes from quantising their storage.
LoRA freezes a weight matrix and trains a low-rank detour around it, so only the detour carries
gradients, optimiser states and a master copy; the paper reports a ten-thousand-fold reduction in
trainable parameters and a threefold reduction in GPU memory for GPT-3 with Adam, and no added
inference latency once the detour is merged. Rank is capacity and costs r × (in + out) parameters
per adapted layer, alpha is a fixed scale of alpha/r that the convention alpha = 2r holds
constant, dropout is mild regularisation, and target modules decide where adapters go, with
"all-linear" as the QLoRA-style setting. QLoRA quantises the frozen base to 4-bit NormalFloat with
double quantisation and paged optimisers, taking the largest memory term down by nearly four; it is
documented for CUDA and, from ROCm 6.4.4, for the gfx1151 chip in Track X, while bitsandbytes
lists Apple silicon only as a CPU build. DoRA splits the update into magnitude and direction and
helps most at low rank, at the cost of a slower step. And no adapter shrinks the resident base,
touches modules it was not attached to, or merges cleanly into a base of a different precision.
Check your understanding
Sources for this lesson
8 verified · checked 2026-09-13
- 01LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., arXiv:2106.09685)§ Abstractarxiv.org/abs/2106.096852026-09-09
- 02QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., arXiv:2305.14314)§ Abstractarxiv.org/abs/2305.143142026-09-09
- 03DoRA: Weight-Decomposed Low-Rank Adaptation (Liu et al., arXiv:2402.09353)§ Abstractarxiv.org/abs/2402.093532026-09-09
- 04PEFT — LoRA conceptual guide§ Low-rank decomposition; rank; merginghuggingface.co/docs/peft/main/en/conceptual_guides/lora2026-09-09
- 05PEFT — LoRA developer guide§ Initialization; Rank and alpha; Target modules; rsLoRA; DoRA; Merging adaptershuggingface.co/docs/peft/developer_guides/lora2026-09-09
- 06PEFT — Quantization§ Quantize a model; LoraConfig; QLoRA-style traininghuggingface.co/docs/peft/main/en/developer_guides/quantization2026-09-09
- 07bitsandbytes — Installation§ NVIDIA CUDA; AMD ROCm; CPUhuggingface.co/docs/bitsandbytes/main/en/installation2026-09-09
- 08PEFT — target-module validation implementation§ inject_adapter target-module validationraw.githubusercontent.com/huggingface/peft/main/src/peft/tuners/tuners_utils.py2026-09-13
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.