Distillation, Pruning and Quantisation: How Small Models Get Good

By the end of this lesson you will be able to say, for any small model you download, which of three operations made it: a teacher training a new student, parameters deleted from a larger model, or every parameter rounded to fewer bits. You will have computed what each does to one 8B model, watched soft targets pull a student’s logits in a direction a label cannot, removed four layers from the model on your disk and measured the damage, rounded its real weights to four bits with and without an importance matrix, and trained a tiny network with the rounding inside the loop. You will also know which of Parts 13 to 17 teaches each technique with your hands on it.

Three operations, and what each does to one model

Operation	What it does	Parameter count afterwards	Training run?	Data needed	Where the cost goes
Distillation	Trains a new, smaller student to reproduce a larger teacher’s outputs	The student’s, chosen before training	Yes	Prompts, and the teacher’s answers or probabilities	Teacher generation, then student training: hours
Pruning	Deletes parameters from an existing model: layers, heads, MLP width, or individual weights	Fewer, for structured pruning	Usually, to repair the damage	Calibration text; a retraining corpus for the repair	The repair run
Quantisation	Keeps every parameter and stores each in fewer bits	Unchanged	No for post-training quantisation; yes for quantisation-aware training	None, or a calibration text	Minutes of conversion, or a download

Apply all three to Qwen3-8B, and compare with its smaller sibling Qwen3-4B (both Apache-2.0), with the architecture read from the 8B’s config.json (hidden_size 4096, intermediate_size 12288, 32 attention heads and 8 key-value heads of head_dim 128, 36 layers, vocab_size 151936, embeddings not tied) and file sizes from the course’s model data. This is arithmetic from those inputs, not a measurement:

Route	How the count is derived	Parameters	Weights at BF16	Same weights at Q4_K_M’s 4.9 bits
Original	36 × 192,946,432 per layer + 2 × 622,329,856 embedding and output head + 4,096 final norm	8,190,735,360	16.4 GB	5.0 GB, the published file
Quantise to Q4_K_M	nothing removed	8,190,735,360	not applicable	5.0 GB
Depth-prune 9 of 36 layers	minus 9 × 192,946,432	6,454,217,472	12.9 GB	about 4.0 GB
Width-prune the MLP from 12288 to 9216	minus 36 × 3 × 4096 × 3072	6,831,780,864	13.7 GB	about 4.2 GB
Distil into Qwen3-4B	a different, smaller model	about 4.0 billion	8.0 GB	2.5 GB, the published file

Three things fall out of the table. Each MLP holds 150,994,944 of a layer’s 192,946,432 parameters, which is why width pruning aims there. Quantisation shrinks the file more than removing a quarter of the layers does, and removes nothing. And the routes stack: the small models you run were usually distilled first and quantised afterwards.

Distillation: a big teacher and a small student

The idea dates from a short, readable 2015 paper. Hinton, Vinyals and Dean began from a deployment problem: predicting with an ensemble is “cumbersome and may be too computationally expensive to allow deployment to a large number of users”, and earlier work had shown it is possible to “compress the knowledge in an ensemble into a single model which is much easier to deploy”. They developed that compression further, and gave it its name.

The trick is what the student is trained on. A label says which answer is right. The teacher’s output distribution also says which wrong answers were nearly right, and in the paper’s words “the relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize”. Those soft targets are the teacher’s softmax, raised to a temperature T so the small probabilities become visible. You met temperature as a sampling setting in Part 2; here it shapes a training target:

q_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)          z = logits, T = temperature (1 is the ordinary softmax)

loss   = α · T² · CE( p_teacher(T), q_student(T) )  +  (1 − α) · CE( label, q_student(1) )

∂CE_soft / ∂z_i = ( q_i(T) − p_i(T) ) / T         the gradient on each student logit (equation 2)

CE is cross-entropy, p_teacher the teacher’s distribution, α the weight on the soft term. The paper found the best results with “a considerably lower weight on the second objective”, and adds the detail re-implementations drop: “since the magnitudes of the gradients produced by the soft targets scale as 1/T² it is important to multiply them by T²”. Put numbers through it:

RunnableAll tracks

soft-targets.py

"""Soft targets: what a teacher's distribution carries that a label does not, and what temperature does."""
import numpy as np

tokens = ["Brasília", "Rio", "São Paulo", "Lisbon", "banana"]
teacher = np.array([9.0, 5.5, 5.0, 2.5, -4.0])   # illustrative logits for "The capital of Brazil is"
student = np.array([5.0, 5.5, 1.0, 4.0, 2.0])    # a student that has the order wrong

def softmax(z, T=1.0):
    e = np.exp((z - z.max()) / T)
    return e / e.sum()

print(f"{'T':<3}" + "".join(f"{t:>11}" for t in tokens) + "   entropy")
for T in (1, 2, 4):
    p = softmax(teacher, T)
    h = -(p * np.log2(p)).sum()
    print(f"{T:<3}" + "".join(f"{x:11.4f}" for x in p) + f"   {h:.2f} bits")

T = 4
hard = np.eye(len(tokens))[0]                       # the label: token 0, nothing else
g_hard = softmax(student) - hard                    # d(cross-entropy)/d(logits) at T = 1
g_soft = (softmax(student, T) - softmax(teacher, T)) / T   # Hinton et al., equation 2
print("\ngradient on each student logit (positive = gradient descent lowers it)")
print(f"{'hard label, T=1':<22}" + "".join(f"{x:+11.3f}" for x in g_hard))
print(f"{'soft target, T=4':<22}" + "".join(f"{x:+11.3f}" for x in g_soft))

print("\nsize of the soft-target gradient as T rises")
for T in (1, 2, 4, 8, 16):
    g = (softmax(student, T) - softmax(teacher, T)) / T
    n = np.abs(g).sum()
    print(f"T={T:<3} sum|gradient| = {n:.4f}   x T^2 = {n * T * T:.3f}")

Output — what you should see

soft-targets.py, numpy 2.5.2

T     Brasília        Rio  São Paulo     Lisbon     banana   entropy
1       0.9524     0.0288     0.0174     0.0014     0.0000   0.33 bits
2       0.7411     0.1288     0.1003     0.0287     0.0011   1.19 bits
4       0.4949     0.2063     0.1821     0.0975     0.0192   1.86 bits

gradient on each student logit (positive = gradient descent lowers it)
hard label, T=1            -0.676     +0.534     +0.006     +0.119     +0.016
soft target, T=4           -0.057     +0.024     -0.021     +0.028     +0.027

size of the soft-target gradient as T rises
T=1   sum|gradient| = 1.2794   x T^2 = 1.279
T=2   sum|gradient| = 0.4919   x T^2 = 1.968
T=4   sum|gradient| = 0.1562   x T^2 = 2.500
T=8   sum|gradient| = 0.0441   x T^2 = 2.824
T=16  sum|gradient| = 0.0117   x T^2 = 2.986

Read it in three blocks. At T = 1 the teacher’s distribution is nearly a label, 0.33 bits of entropy; at T = 4 it carries 1.86 bits and a ranking of the alternatives. The hard label pushes every wrong token down, São Paulo included; the soft target pushes São Paulo up, because the teacher rated it above where the student has it, and pushes banana down. That sign flip is information no label contains. And the gradient shrinks towards 1/T² as T rises, so the × T² column levels off: change the temperature without the T² factor and you have silently reweighted the two losses, one of the faults in Part 15’s challenge.

A language model makes this decision at every position over its whole vocabulary, so there are three ways to distil one, and Part 15 takes each apart:

Family	The student trains on	Tokeniser condition	Resident while training	Cost driver
Sequence-level	Text the teacher wrote: supervised fine-tuning on a machine-written dataset	None	The student only; the teacher ran earlier	Generating the dataset once
Logit	The teacher’s full distribution at every position, with a loss like the one above	Same vocabulary, same ids, same logit width, for the standard loss; cross-tokeniser methods relax it, below	Teacher and student together	Two models’ forward passes per step
On-policy	The student’s own samples, scored against the teacher	As for logit, when the teacher’s probabilities are used	Both, plus generation inside the loop	Generation on every step

!Important

The usual claim is that distillation transfers format, style and problem-solving habits better than broad factual knowledge, which needs parameters to sit in, and the student has fewer. Test it on your own pair with the evaluation Part 15’s labs run: teacher, base student and distilled student on the same held-out tasks. If the problem is missing information, give the model the information at inference time with retrieval.

The small-student, big-teacher pattern

The pattern is the division of labour behind the Qwen3 small sizes this course uses and behind the DeepSeek-R1 paper’s distilled students: spend the expensive training (pretraining at scale, reinforcement learning) once, on a large model, and move the resulting behaviour into small ones by distillation. The DeepSeek-R1 abstract states the premise, that “the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models”, and the previous lesson read the same route off the Qwen3 report for the sizes you download. What published instances disclose lets you size the student side. The DeepSeek-R1 and Minitron papers appear as publications: their models are not in the course’s model reference, and their licences are not assessed here.

Instance	Teacher to student	What was trained	Arithmetic from the stated inputs
The DeepSeek-R1 paper’s distilled students	DeepSeek-R1 to base models from 1.5B to 70B	Supervised fine-tuning only: “2–3 epochs using the 800k data”, “batch size is 64”, “maximum context length is 32,768 tokens”	800,000 × 2 / 64 = 25,000 optimiser steps; at 3 epochs, 37,500
Qwen3 small sizes (Apache-2.0)	Qwen3-32B or Qwen3-235B-A22B to 0.6B, 1.7B, 4B, 8B, 14B and 30B-A3B	Off-policy distillation on the teachers’ outputs, then on-policy: the student’s own samples, its logits aligned with a teacher’s “to minimize the KL divergence”	Reported as “only 1/10 of the GPU hours compared to the four-stage training method”
The Minitron paper	A pretrained 15B model to 8B and 4B	Pruning, then distillation-based retraining “with a fraction (<3%) of the original training data”	Reported as “up to 40x fewer training tokens per model compared to training from scratch”

Distillation as it is actually run at home

1. 1Choose the pairA teacher your machine can serve and a student it can train, a few times smaller. A shared tokeniser makes the standard logit loss possible.
2. 2GenerateServe the teacher locally and have it answer a large set of prompts, keeping its outputs. This is where the hours go.
3. 3FilterVerify answers where a checker exists, deduplicate, and remove anything that overlaps the evaluation set.
4. 4Train the studentSupervised fine-tuning on the teacher outputs, or a distillation loss against the teacher probabilities where the tokenisers match.
5. 5Evaluate all threeTeacher, base student and distilled student on the same held-out tasks. The part of the gap that closed is the result.

Part 15 runs this end to end with the pair your memory allows: Qwen3-30B-A3B to Qwen3-4B at 128 GB, Qwen3-14B to Qwen3-4B at 24 GB, and Qwen3-8B to Qwen3-1.7B or Qwen3-4B at 12 to 16 GB in the sequence-level lab; Qwen3-8B to Qwen3-1.7B in the logit lab, with Qwen3-4B to Qwen3-0.6B as its reduced path. The teacher is the reader's own served model, so no data leaves the machine.

At home the pattern has one constraint that dominates the choice of pair: the teacher must be servable and the student trainable on the same machine, together for logit distillation and one after the other for sequence-level. Arithmetic from the course’s model data and Part 11’s sixteen bytes per parameter for a full fine-tune (weights and optimiser state only; activations, KV cache and the operating system come on top, and an adapter from Part 13 shrinks the student column sharply):

Teacher → student (all Apache-2.0)	Teacher at Q4_K_M, serving	Teacher at BF16	Student full fine-tune, 16 bytes per parameter	Sequence-level peak, one phase at a time	Logit, teacher BF16 + student full fine-tune
Qwen3-30B-A3B → Qwen3-4B	18.6 GB	61 GB	64.0 GB	64.0 GB	125.0 GB
Qwen3-14B → Qwen3-4B	9.0 GB	29.6 GB	64.0 GB	64.0 GB	93.6 GB
Qwen3-8B → Qwen3-1.7B	5.0 GB	16.4 GB	27.2 GB	27.2 GB	43.6 GB
Qwen3-4B → Qwen3-0.6B	2.5 GB	8.0 GB	9.6 GB	9.6 GB	17.6 GB

The logit column, with the standard loss, also needs the two models to share a vocabulary, and that takes thirty seconds to check. This downloads only tokeniser and config files:

RunnableAll tracks

same-tokeniser.py

"""Can this teacher and this student be paired for logit distillation? Compare tokenisers and logit widths."""
from transformers import AutoConfig, AutoTokenizer

STUDENT = "Qwen/Qwen3-0.6B"
TEACHERS = ["Qwen/Qwen3-4B", "Qwen/Qwen3-30B-A3B", "openai/gpt-oss-20b"]
TEXT = "Distil the teacher into the student."

def describe(name):                      # downloads only the tokeniser and config files, not the weights
    tok = AutoTokenizer.from_pretrained(name)
    width = AutoConfig.from_pretrained(name).vocab_size
    print(f"{name:<20} {len(tok.get_vocab()):>7} tokens, {width:>7} logits   ids {tok(TEXT)['input_ids']}")
    return tok.get_vocab(), width

s_vocab, s_width = describe(STUDENT)
for name in TEACHERS:
    t_vocab, t_width = describe(name)
    same = sum(1 for token, i in s_vocab.items() if t_vocab.get(token) == i)
    usable = t_vocab == s_vocab and t_width == s_width
    print(f"{'':<20} same token at the same id: {same:,} of {len(s_vocab):,}  ->  "
          f"{'same-id logit or sequence-level' if usable else 'no same-id logit loss'}")

Output — what you should see

same-tokeniser.py, transformers 5.16.1; Hub download messages omitted

Qwen/Qwen3-0.6B       151669 tokens,  151936 logits   ids [23356, 321, 279, 11079, 1119, 279, 5458, 13]
Qwen/Qwen3-4B         151669 tokens,  151936 logits   ids [23356, 321, 279, 11079, 1119, 279, 5458, 13]
                     same token at the same id: 151,669 of 151,669  ->  same-id logit or sequence-level
Qwen/Qwen3-30B-A3B    151669 tokens,  151936 logits   ids [23356, 321, 279, 11079, 1119, 279, 5458, 13]
                     same token at the same id: 151,669 of 151,669  ->  same-id logit or sequence-level
openai/gpt-oss-20b    200019 tokens,  201088 logits   ids [25617, 311, 290, 14044, 1511, 290, 6760, 13]
                     same token at the same id: 265 of 151,669  ->  no same-id logit loss

The logit width is larger than the token count because the output layer has spare rows; what a logit loss compares is those rows, position by position. gpt-oss-20b (Apache-2.0) spells the same sentence with different ids and has 49,152 more rows, so the same-id logit loss above cannot be used between them. It can teach a Qwen3 student through text, or through a cross-tokeniser method such as ULD and its extension GOLD, which TRL 1.12.0 ships as experimental (documentation checked 2026-09-12). GOLD “aligns the textual spans produced by both tokenizers and merges the associated logits”; where one tokeniser’s single token is several of the other’s, the merged probabilities are, in the same documentation’s words, a “reasonable approximation” for the tokens that were not written. The course does not use that method. The decision rule, in order: pick the teacher your memory can serve, the student your memory can train, then the family the tokeniser check allows. Part 15’s sequence-level lab does the full two-phase budget with activations and KV cache before anything runs.

Synthetic data

Synthetic data is training data a model produced: instructions, answers, preference pairs, test cases. Sequence-level distillation is its most common form. The Self-Instruct paper is the readable reference for doing it at scale: its pipeline generates instructions and answers from a model, then filters “invalid or similar ones before using them to finetune the original model”. Its diversity rule is concrete, “a new instruction is added to the task pool only when its ROUGE-L similarity with any existing instruction is less than 0.7”, and after filtering it kept “over 52K instructions and more than 82K instances”.

Every such pipeline has the same four stages, and each removes something you would otherwise train on. Here they are on a stand-in teacher that is right 88 times in 100:

RunnableAll tracks

synthetic-filter.py

"""Synthetic data in miniature: generate, verify, deduplicate, decontaminate, and count what each stage removes."""
import random

random.seed(0)
def question():
    return (random.randint(10, 99), random.randint(10, 99))    # a narrow seed template: two-digit products

def teacher(a, b):             # stands in for a served teacher: right 88 times in 100, wrong without warning
    return a * b if random.random() < 0.88 else a * b + random.choice([-10, -1, 1, 10])

evaluation = {question() for _ in range(300)}                   # questions you will score the student on
generated = [(a, b, teacher(a, b)) for a, b in (question() for _ in range(5000)) for _ in range(2)]

stages = [("generated, 2 samples per prompt", generated)]
verified = [(a, b, y) for a, b, y in generated if y == a * b]   # the verifier: recompute the answer
stages.append(("verifier: answer is correct", verified))
seen, unique = set(), []
for a, b, y in verified:
    if (a, b) not in seen:
        seen.add((a, b))
        unique.append((a, b, y))
stages.append(("deduplicated on the question", unique))
clean = [(a, b, y) for a, b, y in unique if (a, b) not in evaluation]
stages.append(("decontaminated against evaluation", clean))

for label, rows in stages:
    wrong = sum(1 for a, b, y in rows if y != a * b)
    print(f"{label:<36} {len(rows):>6} examples   {wrong:>5} with a wrong answer")
both = sum(g[2] == h[2] == g[0] * g[1] for g, h in zip(generated[::2], generated[1::2]))
print(f"\n{both:,} prompts with both samples correct; {len({(a, b) for a, b, _ in generated}):,} distinct questions")
print(f"kept {len(clean) / len(generated):.0%} of what the teacher generated")

Output — what you should see

synthetic-filter.py, Python 3.12

generated, 2 samples per prompt       10000 examples    1204 with a wrong answer
verifier: answer is correct            8796 examples       0 with a wrong answer
deduplicated on the question           3686 examples       0 with a wrong answer
decontaminated against evaluation      3559 examples       0 with a wrong answer

3,891 prompts with both samples correct; 3,742 distinct questions
kept 36% of what the teacher generated

Each line is a failure mode made countable. Without the verifier, 1,204 confidently wrong answers enter training with nothing to mark them; the student learns them as readily as the right ones. Deduplication removed 8,796 − 3,686 = 5,110, and the line after the stage table splits that in two. This is arithmetic on the script’s output:

Removed by deduplication	Count	Why
The second correct answer to the same prompt	3,891	Sampling twice and keeping one verified answer is rejection sampling; the duplicate is expected
The template asking a question it had already asked	5,110 − 3,891 = 1,219	5,000 draws from 90 × 90 = 8,100 possible pairs give only 3,742 distinct questions
Total	5,110	8,796 verified, 3,686 kept

Generated data is cheap in quantity and expensive in diversity, which is why Part 15 measures diversity rather than assuming it. Decontamination removed the 127 examples that would have let the student see the evaluation questions, a score you could no longer interpret. And the yield sets the budget: to keep 10,000 examples at this rate, generate about 28,000.

Two limits have no filter. Training on generated output repeatedly narrows a model’s distribution; the Shumailov paper reports “irreversible defects in the resulting models, where tails of the original content distribution disappear”. And some licences and hosted-provider terms restrict using outputs to train other models, which the licence lesson shows you how to find before you generate a single example.

Pruning

Pruning removes parameters from a trained model. Structured pruning removes whole components, so the matrices get smaller: depth pruning deletes layers, width pruning shrinks the hidden size, the number of heads or the MLP. Unstructured pruning zeroes individual weights and leaves every matrix its original shape. Semi-structured N:M sparsity sits between them: in the Wanda paper’s definition it “requires that at most N out of every M contiguous weights to be non-zero”, so 2:4 zeroes half the weights in a fixed pattern a kernel can be written for.

That difference decides whether anything gets smaller. A zero stored in a dense BF16 tensor still takes 16 bits, and an engine multiplying dense matrices spends the same time on it. Storing only the non-zeros needs an index per value; in a generic compressed layout with a 16-bit column index, half the weights at 16 bits plus their indices come to 0.5 × (16 + 16) = 16 bits per original weight, no saving at 50 per cent sparsity, and against a 4.9-bit quantised weight the index alone costs more than the value. No engine this course teaches reads such a format.

Kind	Removes	Shapes change?	Smaller and faster in llama.cpp, MLX or vLLM?	How the candidates are chosen
Depth	Whole layers	Layer count only	Yes, if the tool renumbers the remaining layers and updates the layer count (num_hidden_layers and any per-layer lists in config.json, block_count in GGUF); llama-quantize --prune-layers does both for GGUF	Block Influence, below
Width	Heads, hidden size, MLP width	Every affected matrix	Yes, if the tool writes a valid config	Activation-based importance, as in the Minitron paper, below
Semi-structured N:M (for example 2:4)	2 of every 4 weights	No	Only with kernels that exploit the pattern; check the engine’s documentation at the pinned version	Wanda’s score compared within each group of M
Unstructured	Individual weights	No	No, on dense kernels	Weight magnitude, or Wanda’s magnitude-times-input score

Wanda’s score is worth a second look. It prunes, in its abstract’s words, “weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis”, S_ij = abs(W_ij) × norm(X_j), where X_j is input column j across the calibration tokens. Its squared form, which the paper also writes out, is W_ij² times the sum of squared inputs on that column: the same per-column statistic llama.cpp’s importance matrix collects for quantisation, further down this page. Deciding what matters from how the model is used, rather than from the weights alone, is one idea that serves both techniques. For 2:4 the paper compares that score “among every M consecutive weights, for all weights connected to an output” and zeroes the lowest two of each four; the pattern “can leverage NVIDIA’s sparse tensor cores”, in its words, and whether an engine you run has such a kernel is a question for that engine’s documentation.

For width, the Minitron paper computes “the importance of each head, neuron and embedding channel by examining the activations produced by the MHA, MLP and LayerNorm layers, respectively”, from “a small (1024 samples) calibration dataset and only forward propagation passes”, aggregated over the batch and sequence (section 2.2). It then ranks each axis and performs “trimming (reshaping) of the corresponding weight matrices directly” (section 2.3): dropping MLP neuron i removes row i of the MLP’s input projections and column i of its output projection, and dropping an embedding channel narrows every matrix that reads or writes the hidden state.

For depth, ShortGPT defines Block Influence from how much a layer changes the hidden state passing through it, BI_i = 1 − mean over tokens of cos(x_in, x_out), and removes the layers with the lowest scores. Do it to Qwen3-0.6B (Apache-2.0), which Part 2’s parameters lesson downloaded. If you skipped that download (the Part 2 lab’s primary path uses Qwen3-1.7B), this fetches it, 1.5 GB in the Hub listing on 2026-09-12; that lesson shows what the command prints:

RunnableAll tracks

fetch Qwen3-0.6B if you do not have it

hf download Qwen/Qwen3-0.6B --local-dir ~/llm-course/models/qwen3-0.6b

RunnableAll tracks

block-influence.py

"""Depth pruning on a model you have: score each layer by Block Influence, remove four, measure the damage."""
from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = Path("~/llm-course/models/qwen3-0.6b").expanduser()   # or qwen3-1.7b: about 7 GB of float32 weights
DROP = 4
CALIBRATION = ("The river rose through the night, and by morning the lower streets of the town were under water. "
               "Shopkeepers carried what they could upstairs while the council opened the school as a shelter.")
HELD_OUT = ("A compiler translates a program written in one language into another, usually a lower-level one. "
            "It checks the program for errors, builds an intermediate representation, optimises it, and emits code.")

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32).eval().requires_grad_(False)
layers, bi = model.model.layers, {}
per_layer, total = sum(p.numel() for p in layers[0].parameters()), sum(p.numel() for p in model.parameters())
def score(i):                        # BI_i = 1 - mean cosine similarity between the layer's input and output
    def hook(module, args, kwargs, output):
        h_in = args[0] if args else kwargs["hidden_states"]
        h_out = output[0] if isinstance(output, tuple) else output
        bi[i] = 1 - torch.nn.functional.cosine_similarity(h_in[0], h_out[0], dim=-1).mean().item()
    return hook
handles = [layer.register_forward_hook(score(i), with_kwargs=True) for i, layer in enumerate(layers)]
model(**tok(CALIBRATION, return_tensors="pt"))
for h in handles:
    h.remove()
print("Block Influence by layer:")
for start in range(0, len(layers), 7):
    print("  " + "  ".join(f"{i:>2}: {bi[i]:.3f}" for i in range(start, min(start + 7, len(layers)))))

ids = tok(HELD_OUT, return_tensors="pt")["input_ids"]
def held_out_loss(kept):
    model.model.layers = torch.nn.ModuleList(layers[i] for i in kept)
    model.config.num_hidden_layers = len(kept)
    return model(input_ids=ids, labels=ids, use_cache=False).loss.item()
order = sorted(bi, key=bi.get)
print(f"\n{per_layer:,} parameters per layer; removing {DROP} removes {DROP * per_layer / total:.1%} of {total:,}")
for label, drop in (("nothing removed", []), ("lowest BI removed", order[:DROP]), ("highest BI removed", order[-DROP:])):
    loss = held_out_loss([i for i in range(len(layers)) if i not in drop])
    print(f"{label:<19} {str(sorted(drop)):<17} held-out loss {loss:.2f} nats")

Output — what you should see

block-influence.py on Qwen3-0.6B, CPU, torch 2.14.0; weight-loading progress bar omitted

Block Influence by layer:
   0: 0.939   1: 0.111   2: 0.122   3: 0.129   4: 0.130   5: 0.114   6: 0.102
   7: 0.115   8: 0.095   9: 0.090  10: 0.091  11: 0.067  12: 0.036  13: 0.031
  14: 0.076  15: 0.047  16: 0.061  17: 0.064  18: 0.056  19: 0.069  20: 0.062
  21: 0.054  22: 0.044  23: 0.026  24: 0.028  25: 0.017  26: 0.030  27: 0.102

15,730,944 parameters per layer; removing 4 removes 10.6% of 596,049,920
nothing removed     []                held-out loss 2.64 nats
lowest BI removed   [23, 24, 25, 26]  held-out loss 5.15 nats
highest BI removed  [0, 2, 3, 4]      held-out loss 15.09 nats

Layer 0 transforms its input almost completely and the late-middle layers barely change theirs, the redundancy ShortGPT’s title announces. The metric works: removing the four least influential layers costs 2.5 nats of loss on text the scoring never saw, removing the four most influential costs 12.5. It is still damage. A loss of 5.15 nats is a perplexity of about 172 against 14, for a tenth of the parameters, which is why pruning is followed by a repair run with the original model as teacher. The Minitron paper is the public account of that recipe, and Part 15’s pruning lesson reads it with the dated state of the tools. Your numbers on Qwen3-1.7B will differ; the ordering is what to check.

✕Common mistake

Pruning without the repair step, and calling the result a smaller model. llama-quantize at v0.4.0 documents a --prune-layers advanced option that removes listed layers while it converts; the file it writes is the “lowest BI removed” row at best, not a prune-and-distil model. The symptom can hide from multiple-choice benchmarks: ShortGPT’s authors report that removing a quarter of the layers of 7B models left multiple-choice scores less affected while generative tasks such as summarisation fell to nearly zero. The fix is to measure held-out loss and a generative task before and after, and to compare against the published model of the size you were aiming for.

Quantisation

Quantisation is the technique this course leans on daily. The arithmetic came from Part 1, which also rounded a block of random weights to four bits by hand:

weights_GB ≈ parameters × bits_per_parameter / 8 / 10⁹

Model (course model data)	Parameters	BF16	Q8_0	Q4_K_M or as shipped	Bits per parameter, from file size
Qwen3-8B	8.19 billion	16.4 GB	8.7 GB	5.0 GB	16.0, 8.5, 4.9
Qwen3-30B-A3B	30.5 billion	61 GB	32.5 GB	18.6 GB	16.0, 8.5, 4.9
gpt-oss-20b	21 billion	not shipped	not shipped	13 GB, MXFP4 as published	5.0

Q4_K_M lands above four bits because every block stores scales, and because the _M mix keeps some tensors at wider types. Rounding to about 4.9 bits makes the model roughly a third of its BF16 size, and, since decode speed is bandwidth divided by bytes read per token, the inference lesson predicts the speed-up from the same ratio.

Qwen3-30B-A3B on a 24 GB machine, four-bit weights and an 8k context

Weights, Q4_K_M

18.6 GB

KV cache, 8,192 tokens at FP16

0.8 GB

Free

4.6 GB

Total

24 GB

The same model's BF16 weights come to 61 GB, so on this machine the four-bit version is not an optimisation but the difference between running it and not. Weights from src/data/models.json. The KV figure is arithmetic from the config.json values recorded in the same file (48 layers × 2 × 4 key-value heads × 128 × 2 bytes = 98,304 bytes per token) times 8,192 tokens: an estimate, not a measurement.

Post-training quantisation

Post-training quantisation rounds a finished model with no training. The naive version is round-to-nearest: pick each block’s scale from its largest value and round every weight to the grid. Post-training methods are often close at eight bits and degrade below that, as LLM-QAT’s abstract reports (quoted under quantisation-aware training), and the reason is one line of algebra. A linear layer computes y = W · x, so rounding W to Ŵ changes the output by

Δy = (Ŵ − W) · x        the error on weight j of a row reaches the output multiplied by input x_j

A large rounding error on a weight whose input is always near zero does nothing; a small one on a weight whose input is large on every token does a great deal. The methods built to improve on round-to-nearest (AWQ, GPTQ, and llama.cpp’s importance matrix) answer “which weights matter” from activations; plain round-to-nearest and K-quants without an imatrix use the weights alone. AWQ’s abstract states it: “not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error”, and “to identify salient weight channels, we should refer to the activation distribution, not weights”. Rather than keep those weights at FP16, it scales them before rounding, Q(w · s) · (x / s), which cuts their relative error by about 1/s, and it chooses s = s_X^α per input channel, where s_X is the channel’s average activation magnitude and α comes from a grid search over [0, 1]. It “does not rely on any backpropagation or reconstruction”, which is why it is fast to apply.

llama.cpp, the engine of Part 6, takes the other route: keep the scaling, change the rounding. llama-imatrix runs calibration text through the model and, in its v0.4.0 source, accumulates x_j² for every input column of every tensor; its README calls the sum “the importance scores”. llama-quantize --imatrix then weights each weight’s rounding error by that score when it fits each block’s scale. For Q4_0 the v0.4.0 code weights weight j by imatrix_j × sqrt(σ² + w_j²), with σ² the row’s mean squared weight, and tries 19 stretches of the grid, keeping the one with the least weighted error. Here is that procedure in NumPy, on real weights and real activations from the same Qwen3-0.6B:

RunnableAll tracks

importance-matrix.py

"""Four-bit rounding of real weights, with and without an importance matrix, scored on what the layer outputs."""
from pathlib import Path
import numpy as np
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = Path("~/llm-course/models/qwen3-0.6b").expanduser()   # or qwen3-1.7b: about 7 GB of float32 weights
TENSORS = ["layers.2.mlp.down_proj", "layers.13.self_attn.q_proj", "layers.13.mlp.gate_proj"]
CALIBRATION = ("The river rose through the night, and by morning the lower streets of the town were under water. "
               "Shopkeepers carried what they could upstairs while the council opened the school as a shelter. "
               "def mean(xs):\n    return sum(xs) / len(xs)\nQuestion: what is 17 times 23? Answer: 17 times 23 is 391.")
HELD_OUT = ("A compiler translates a program written in one language into another, usually a lower-level one. "
            "It checks the program for errors, builds an intermediate representation, optimises it, and emits code.")
def q4_0(W, imatrix=None):            # ggml's Q4_0 layout: blocks of 32 along each row, one scale per block, codes -8..7
    B = W.reshape(W.shape[0], -1, 32)
    mx = np.take_along_axis(B, np.abs(B).argmax(-1)[..., None], -1) + 1e-30 * ~B.any(-1, keepdims=True)  # signed max
    if imatrix is None:                                   # no imatrix: plain round-to-nearest
        return (np.clip(np.floor(B / (mx / -8) + 0.5), -8, 7) * (mx / -8)).reshape(W.shape)
    w, best_err, best = imatrix.reshape(1, -1, 32) * np.sqrt((W ** 2).mean(1)[:, None, None] + B ** 2), np.inf, 0
    for step in range(-9, 10):                            # try 19 grid stretches, keep the least weighted error
        L = np.clip(np.rint(B * -(8 + 0.1 * step) / mx), -8, 7)
        d = (w * B * L).sum(-1, keepdims=True) / np.maximum((w * L * L).sum(-1, keepdims=True), 1e-30)
        err = (w * (B - d * L) ** 2).sum(-1, keepdims=True)
        best, best_err = np.where(err < best_err, d * L, best), np.minimum(err, best_err)
    return best.reshape(W.shape)
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32).eval().requires_grad_(False)
inputs = {n: [] for n in TENSORS}                         # the hooks keep each tensor's input, one array per text
for n in TENSORS:
    model.model.get_submodule(n).register_forward_pre_hook(lambda m, args, n=n: inputs[n].append(args[0][0].double().numpy()))
for text in (CALIBRATION, HELD_OUT):
    model(**tok(text, return_tensors="pt"))
rel = lambda a, b: np.linalg.norm(a - b) / np.linalg.norm(b)
print(f"{'tensor':<28}{'max/median':>11}   {'weight error':>19}   {'output error, held-out':>23}")
print(f"{'':<28}{'imatrix':>11}   {'plain':>9} {'imatrix':>9}   {'plain':>11} {'imatrix':>11}")
for name in TENSORS:
    (X_cal, X_new), W = inputs[name], model.model.get_submodule(name).weight.double().numpy()
    imatrix = (X_cal ** 2).mean(0)                        # what llama-imatrix accumulates: mean squared input per column
    plain, weighted = q4_0(W), q4_0(W, imatrix)
    print(f"{name:<28}{imatrix.max() / np.median(imatrix):>11,.0f}   {rel(plain, W):>9.2%} {rel(weighted, W):>9.2%}"
          f"   {rel(plain @ X_new.T, W @ X_new.T):>11.2%} {rel(weighted @ X_new.T, W @ X_new.T):>11.2%}")

Output — what you should see

importance-matrix.py on Qwen3-0.6B, CPU, torch 2.14.0; weight-loading progress bar omitted

tensor                       max/median          weight error    output error, held-out
                                imatrix       plain   imatrix         plain     imatrix
layers.2.mlp.down_proj       20,852,436       9.04%     9.06%         4.31%       1.90%
layers.13.self_attn.q_proj          558       8.68%     8.47%         6.87%       5.65%
layers.13.mlp.gate_proj             298       8.73%     8.44%         6.98%       5.56%

The second column is the argument. In layer 2’s down_proj one input column’s mean squared activation is about twenty million times the median column’s, so a handful of weights carry most of the output. Weighted rounding leaves the weight error where it was, about 9 per cent, and cuts the output error on text the calibration never contained from 4.31 to 1.90 per cent; on the two tensors with milder outliers the gain is smaller. It spends its accuracy where the activations are, which is also why a calibration text is an assumption about your workload. The script is a transcription of the idea for one type, simplified; the tools themselves are two commands, which Part 6 runs:

Fragment — not complete on its own

llama-imatrix -m Qwen3-8B-BF16.gguf -f calibration.txt -o imatrix.gguf
llama-quantize --imatrix imatrix.gguf Qwen3-8B-BF16.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M

Quantisation-aware training

Post-training quantisation is a guess made after the fact. Quantisation-aware training puts the rounding inside the training loop, so the model learns weights that survive it. The forward pass computes with rounded weights; rounding has no useful gradient, so the backward pass treats it as the identity, the straight-through estimator, and updates a full-precision master copy:

w_used = w + stop_gradient( Q(w) − w )      forward value Q(w); gradient with respect to w is 1

RunnableAll tracks

qat-in-miniature.py

"""Quantisation-aware training in miniature: round inside the forward pass, pass the gradient straight through."""
import copy
import torch
import torch.nn.functional as F

BITS = 4
torch.manual_seed(0)
direction, X, X_test = torch.randn(8), torch.randn(8192, 8), torch.randn(4096, 8)
Y, Y_test = (torch.sin(x @ direction).unsqueeze(1) for x in (X, X_test))   # a non-linear target to learn

def fake_quant(w):
    qmax = 2 ** (BITS - 1) - 1                            # 7 at 4 bits: codes -8..7, one scale per row
    scale = w.abs().amax(dim=1, keepdim=True).clamp(min=1e-8) / qmax
    q = torch.clamp(torch.round(w / scale), -qmax - 1, qmax) * scale
    return w + (q - w).detach()       # forward pass sees q; backward pass sees the identity (straight-through)
def run(net, x, quant):               # three linear layers with ReLU between, weights rounded when quant is True
    for n, layer in enumerate(net):
        x = F.linear(x, fake_quant(layer.weight) if quant else layer.weight, layer.bias)
        x = torch.relu(x) if n < len(net) - 1 else x
    return x
def train(net, steps, lr, quant):
    opt = torch.optim.Adam(net.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        F.mse_loss(run(net, X, quant), Y).backward()
        opt.step()
def report(label, net, quant):
    with torch.no_grad():
        print(f"{label:<40} test loss {F.mse_loss(run(net, X_test, quant), Y_test).item():.4f}")

print(f"{'always predicting the mean':<40} test loss {Y_test.var().item():.4f}")
trained = torch.nn.ModuleList([torch.nn.Linear(8, 64), torch.nn.Linear(64, 64), torch.nn.Linear(64, 1)])
train(trained, 3000, 3e-3, quant=False)
report("trained, full-precision weights", trained, False)
report(f"the same weights rounded to {BITS} bits", trained, True)
control, qat = copy.deepcopy(trained), copy.deepcopy(trained)
train(control, 500, 1e-3, quant=False)                    # 500 more steps without rounding, then round
report("500 more plain steps, then rounded", control, True)
train(qat, 500, 1e-3, quant=True)                         # the same 500 steps with rounding in the loop
report("500 steps with rounding in the loop", qat, True)

Output — what you should see

qat-in-miniature.py, CPU, torch 2.14.0; the last digits vary by machine

always predicting the mean               test loss 0.5067
trained, full-precision weights          test loss 0.0058
the same weights rounded to 4 bits       test loss 0.0600
500 more plain steps, then rounded       test loss 0.0422
500 steps with rounding in the loop      test loss 0.0115

Rounding the trained weights to four bits made the test loss ten times worse. Five hundred more ordinary steps, then rounding, barely helped; the same five hundred steps with the rounding in the loop recovered most of the gap, so the gain comes from training against the rounding, not from training longer. At billions of parameters this costs a real training run, which is why it is done by publishers. LLM-QAT gives the reason to pay for it, post-training methods “perform well down to 8-bits” and “break down at lower bit precision”, and its training signal is “a data-free distillation method that leverages generations produced by the pre-trained model”: the student is the same model in fewer bits.

The rounding a QAT run trains against is a specific format: a bit width, a scale per row or per block, a symmetric grid. Deploy the checkpoint in any other format and the gain can vanish. Append these lines to qat-in-miniature.py and run it again to see it; eight bits is a format neither network trained against:

Fragment — not complete on its own

append to qat-in-miniature.py

BITS = 8                              # deploy both in a format neither was trained against: more bits, not fewer
report("the plain network, rounded to 8 bits", control, True)
report("the QAT network, rounded to 8 bits", qat, True)
report("the QAT network, not rounded at all", qat, False)

Output — what you should see

the last three lines of output with those lines appended, CPU, torch 2.14.0; the last digits vary by machine

the plain network, rounded to 8 bits     test loss 0.0064
the QAT network, rounded to 8 bits       test loss 0.0406
the QAT network, not rounded at all      test loss 0.0403

More bits made the QAT network worse, not better, and several times worse than the plain one. Its master weights moved to where the four-bit per-row grid rounds them well, and unrounded they are poor weights. The symptom in practice is the control comparison failing: a QAT checkpoint that scores no better than a plain quantisation of the base model. The fix is to deploy exactly the format named on the QAT checkpoint’s card.

The step beyond is shipping the low-bit format as the model itself. gpt-oss-20b’s card states that the models “were post-trained with MXFP4 quantization of the MoE weights”, with the claim that this lets it “run within 16GB of memory”. The 13 GB file is 4.95 bits per parameter overall: the expert weights at 4.25 bits (a four-bit element plus an eight-bit scale shared by 32) and the rest wider. Part 16 takes MXFP4, NVFP4 and FP8 apart and says which track computes in which natively.

?Reality check

The claim you will meet is that four-bit quantisation “loses nothing”. The measurable version: on some tasks the difference from the unquantised model is too small to detect with the sample sizes people use, and on others it is not. Part 16 measures it for your model with KL divergence against the unquantised reference, perplexity, and a task evaluation.

Choosing between them

Most disappointment comes from applying one technique to another’s problem. Start from what you can measure:

What you observe	Reach for	Cost	Check before you start
Behaviour is right at BF16, but weights plus KV cache exceed your memory	Post-training quantisation: a published Q4_K_M or imatrix quant	A download, or minutes of conversion	KL divergence against BF16 and a task score at the width you pick
A model that fits scores below your target; a larger one you can serve scores above it	Distillation, sequence-level first	Teacher generation plus a training run: hours	A held-out set, and the teacher’s score on it
A model that fits is close, and you have examples of the right output	Supervised fine-tuning	A training run	A held-out set built before training
You need a size nobody publishes, and can pay for retraining	Pruning, then distillation	The largest: a repair run over many tokens	That no published model of that size already exists
Four-bit post-training quantisation measurably fails at the width you need	A published QAT checkpoint; QAT yourself only if you retrain anyway	A training run with rounding in the loop	That the failure shows in KL divergence or task score, not in one anecdote
The model lacks information: your documents, recent facts	None of these: retrieval	An index and a prompt	That the answer is in the documents

The map to Parts 13 to 17

Parts 11 and 12 build the training environment and pretrain a small model; these five parts then teach every technique above with your hands on it.

Part	What it takes from this lesson	Where to go	What you measure
13, Supervised Fine-Tuning	Sequence-level distillation is SFT on machine-written data; quantising the result	Building an SFT dataset, merging, exporting and quantising	Base against fine-tune on a held-out set
14, Preference Optimisation and RL	Reinforcement learning as the alternative route to reasoning in a small model	GRPO explained, the RL reality check	Pass@1 before and after, and a comparison with distillation on the same budget
15, Knowledge Distillation	Soft targets, the three families, teacher-student pairs, synthetic data, prune-and-distil	Sequence-level lab, logit lab, pruning	The fraction of the teacher-student gap closed, and its cost in tokens and hours
16, Quantisation and Evaluation	GPTQ, AWQ, K-quants and importance matrices; QAT and the native four-bit formats	PTQ in depth, quantise five ways	KL divergence, perplexity, task score, speed and memory per quant
17, Faster Inference	A draft model is a small student trained on the target’s own outputs	Training a draft model, the draft lab	Acceptance rate and decode speed, with the outputs unchanged

An improvement has a reference point and a deployment cost

Take a small model that fits your machine but fails a ticket-routing task. Try a clearer prompt on the unchanged checkpoint before collecting a training dataset. If retrieval supplies missing policy, evaluate retrieval separately. If the task still needs adaptation, use demonstrations whose labels can be checked, and preserve a held-out set before generating synthetic variants.

Now compare three deployed artefacts: the original small model, the adapted small model and the teacher. Report quality alongside latency, resident memory and preparation cost. A student can be useful without matching the teacher on every task; it must meet the requirements of the workload it replaces. Training time belongs in the decision when the task or source data changes frequently.

Avoid combining every technique in the first experiment. Distillation followed by pruning and quantisation produces a difficult attribution problem if quality falls. Save intermediate checkpoints and evaluate after each transformation. This creates a chain of evidence: which stage improved the target behaviour, which stage reduced resource use, and where a regression first appeared.

Recap

Distillation trains a new student on a teacher’s outputs; at temperature T the soft targets carry a ranking of the wrong answers that a label cannot, and the T² factor keeps the two losses in proportion. The small-student, big-teacher pattern spends expensive training once and moves the behaviour down, constrained at home by what one machine can serve and train and by whether the tokenisers match. Synthetic data needs a verifier, deduplication and decontamination, and the yield sets its budget. Pruning deletes parameters; only structured pruning makes a model smaller on the engines you use, and the damage needs a repair run. Post-training quantisation keeps every parameter, and its better methods protect the weights the activations lean on; quantisation-aware training trains against the rounding, and native four-bit formats ship the result as the model.

Check your understanding

Question 1. With the teacher and student logits of soft-targets.py, you raise the distillation temperature from 2 to 4 and change nothing else, and your trainer does not apply the T-squared factor. What happens to the soft-target term's pull on the student, relative to the hard-label term?

• It doubles, because a softer distribution carries more information
• It falls to roughly a third for these logits, and towards a quarter at higher temperatures, because the soft-target gradient approaches 1/T squared
• It is unchanged, because temperature only affects sampling
• It falls to a half, because the gradient is divided by T once

Show the answer and why

Answer: It falls to roughly a third for these logits, and towards a quarter at higher temperatures, because the soft-target gradient approaches 1/T squared

Equation 2 divides by T once, and the difference between two softened distributions shrinks by roughly another factor of T, so the magnitude approaches 1/T squared, as the x T^2 column of soft-targets.py levels off to show. Doubling T from 2 to 4 cuts the soft term's weight by about three for these logits (0.4919 to 0.1562 in soft-targets.py), approaching four as T grows, while the hard-label term, computed at T = 1, stays put. Multiplying by T squared is what keeps the balance you chose.

Question 2. Which of these commands produces a file whose quality is likely to be worse than its type name suggests?

• llama-quantize model-BF16.gguf model-Q4_K_M.gguf Q4_K_M
• llama-quantize --imatrix imatrix.gguf model-BF16.gguf model-Q4_K_M.gguf Q4_K_M
• llama-quantize --allow-requantize model-Q4_K_M.gguf model-Q3_K_M.gguf Q3_K_M
• llama-quantize --imatrix imatrix.gguf --output-tensor-type q8_0 model-BF16.gguf model-Q4_K_M.gguf Q4_K_M

Show the answer and why

Answer: llama-quantize --allow-requantize model-Q4_K_M.gguf model-Q3_K_M.gguf Q3_K_M

Its input is already quantised. The v0.4.0 README warns that --allow-requantize "can severely reduce quality compared to quantizing from 16bit or 32bit": the rounding error of the first quantisation is baked in before the second is applied. The last command is legitimate: --output-tensor-type q8_0 stores output.weight at Q8_0, where the v0.4.0 code would otherwise pick Q6_K (or Q8_0 when the tensor width does not fit its blocks) for a Q4_K_M file, so the result is at most slightly larger, not worse.

Question 3. An unstructured method sets half of Qwen3-8B's weights to exactly zero, and the result is saved as dense BF16. How large is the weights file?

• About 8.2 GB, half of 16.4 GB
• About 16.4 GB, the same as before
• About 5.0 GB, like Q4_K_M
• About 12.3 GB, because zeros compress

Show the answer and why

Answer: About 16.4 GB, the same as before

A dense tensor stores every position at 16 bits whatever its value, so 8,190,735,360 parameters at 2 bytes are still 16.4 GB, and a dense kernel multiplies by zero as fast as by anything else. Storing only the non-zeros needs an index per value, which at 50 per cent sparsity costs as much as it saves. Structured pruning, or quantisation, is what shrinks the file.

Question 4. You want to distil gpt-oss-20b into Qwen3-1.7B on a machine with enough memory for both. Which statement about the methods is correct?

• The standard same-id logit loss works, because both models are Apache-2.0
• Not the standard same-id logit loss, because the vocabularies differ (201,088 logits against 151,936, and the same sentence becomes different ids). Sequence-level distillation works, and so does a cross-tokeniser method that aligns spans.
• Neither logit nor sequence-level distillation, because a mixture-of-experts teacher cannot teach a dense student
• The standard same-id logit loss works if training is on-policy, because the student's own samples make the two sets of ids agree

Show the answer and why

Answer: Not the standard same-id logit loss, because the vocabularies differ (201,088 logits against 151,936, and the same sentence becomes different ids). Sequence-level distillation works, and so does a cross-tokeniser method that aligns spans.

A same-id logit loss compares the teacher's probability for id n with the student's probability for id n, which means nothing when id n is a different token, and sampling from the student does not change which token an id names. Text has no such condition, so the teacher can still write a dataset the student is fine-tuned on. Cross-tokeniser methods such as ULD and TRL 1.12.0's experimental GOLD trainer decode both token streams, group spans with the same visible text and merge the teacher's probabilities inside each group, which approximates the alignment rather than reproducing it. The licence is a separate question, and architecture is not a constraint on any of these.

Question 5. In importance-matrix.py, weighted rounding barely changes the weight error of layer 2's down_proj but cuts its output error by more than half. Why?

• The imatrix makes the weights smaller, so they round more accurately
• The output error is each weight's error multiplied by its input, and the weighting moves rounding error away from the few columns whose inputs are huge and onto columns whose inputs are small
• The held-out text happens to be easier than the calibration text
• The imatrix version uses more bits per weight

Show the answer and why

Answer: The output error is each weight's error multiplied by its input, and the weighting moves rounding error away from the few columns whose inputs are huge and onto columns whose inputs are small

Both versions store the same Q4_0 blocks, 4.5 bits per weight. What changes is which weights absorb the error: one input column's mean squared activation is about twenty million times the median, so an error there is amplified on every token. Total weight error is roughly conserved while output error falls, which is also why a calibration text unlike your workload can spend the protection in the wrong place.

Question 6. Why does qat-in-miniature.py train a control network for 500 plain steps before rounding it?

• To warm up the optimiser for the quantisation-aware run
• To show that the quantisation-aware run's improvement comes from training against the rounding, not from 500 extra steps of training
• Because quantisation-aware training cannot start from trained weights
• To measure the full-precision loss a second time

Show the answer and why

Answer: To show that the quantisation-aware run's improvement comes from training against the rounding, not from 500 extra steps of training

Without the control, the recovery from 0.0600 to 0.0115 could be credited to more training. The control got the same steps and learning rate and still rounded to 0.0422. Holding everything but one factor fixed is the habit Part 16 applies to real quantisation claims, where the comparison is a published QAT checkpoint against a plain quantisation of the same model.

Check my answersClear