Pruning and Compression: Prune-and-Distil

By the end of this lesson you will be able to tell depth pruning from width pruning and structured from unstructured, say why a method that reports impressive sparsity may make your model no faster at all, describe the prune-then-distil recipe in the terms its authors use, judge when it is worth doing rather than starting from a published small model, and name the tools with their state on a stated date. This is the one lesson in the part about what publishers do rather than what you will do this afternoon, and it says clearly which is which.

Part 3 drew the distinction between the three shrinking operations: distillation trains a new model, quantisation rounds the numbers, and pruning removes parameters. This lesson is the third one, and its punchline is that pruning on its own is rarely the answer, and pruning followed by distillation often is.

Two axes, and only one of them makes a model smaller in a useful way

Depth pruning removes whole layers, or whole transformer blocks. The model has fewer layers afterwards, every one of them full size. It is the crudest thing you can do and it is surprisingly effective, because a deep network’s later layers are more alike than they look.

Depth pruning: whole blocks removed from the middle

forward pass

1. EmbeddingUnchanged. Its size is set by the vocabulary, not by depth.
2. Blocks 1 to 12Kept. Early layers do the work that later layers build on and are the least redundant.
3. Blocks 13 and 14Removed. A redundancy metric decides which; the middle and later layers are where the candidates are.
4. Blocks 15 to 32Kept, and now sitting directly on the output of block 12.
5. Output headUnchanged. Vocabulary by hidden size, and usually tied to the embedding.

Depth pruning removes blocks and leaves the rest untouched, so the result loads in any engine that could load the original: the architecture is the same shape with a smaller layer count. The two removed blocks here are illustrative; which blocks to remove is what a method such as ShortGPT's Block Influence metric decides.

Width pruning shrinks the dimensions instead: fewer attention heads, a smaller hidden size, a narrower MLP intermediate dimension. Every layer survives and each is thinner. This is more surgical and more disruptive, because it changes the shape of every weight matrix in the model.

Both of these are structured: the removed parts disappear from the computation, so the model is genuinely faster and genuinely smaller on disk and in memory.

Unstructured pruning zeroes individual weights wherever they are, leaving the matrix the same shape with holes in it. This is where the sparsity numbers in the literature come from, and it is also where the disappointment comes from at home.

!Important

A model with half its weights set to zero is the same size in a GGUF file and the same speed in llama.cpp as a model with none of them set to zero, unless the kernel and the hardware can skip the zeroes. A dense kernel multiplies by zero exactly as fast as by anything else. Unstructured sparsity is a research result about how much a model can lose; on the four tracks in this course it is not, by itself, a way to make anything smaller or faster.

One-shot pruning: what the methods actually claim

“One-shot” means no retraining: prune the finished model and use it as it is.

SparseGPT showed the idea works at scale. Its abstract reports that large generative pretrained transformer models can reach “at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy”, pruning models such as OPT-175B and BLOOM-176B “in under 4.5 hours” and reaching “60% unstructured sparsity with negligible increase in perplexity”. The paper also notes it works with semi-structured patterns and “is compatible with weight quantization approaches”, which is what makes it interesting on hardware that has a fast path for a specific sparsity pattern.

Wanda made the same kind of result far cheaper to obtain. Its abstract describes pruning “weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis”, and stresses that it “requires no retraining or weight update, and the pruned LLM can be used as is”. The idea is the same one AWQ uses for quantisation, which Part 3 introduced: decide what matters from how the model is used, not from the weights alone.

ShortGPT attacks depth instead. It introduces a “Block Influence (BI)” metric to score how much each layer changes its input, removes the layers that change it least, and its abstract claims that the method “significantly outperforms previous” published pruning methods as of its 2024 publication, while remaining compatible with quantisation. Its title states the finding: layers are more redundant than you expect.

The practical reading of those three, for somebody with one machine:

• Depth pruning produces something you can actually serve, because the result is the same architecture with fewer layers.
• Width pruning produces something you can serve if the tooling writes a valid config, which is the part that breaks.
• Unstructured sparsity produces a research artefact unless you have a runtime that exploits the pattern.

Prune-then-distil, as its authors describe it

Removing a fifth of a network damages it. The recipe that makes pruning practical is to repair the damage with distillation, using the original model as the teacher.

NVIDIA’s Minitron work is the clearest public account. The first paper asks “if pruning an existing LLM and then re-training it with a fraction (<3%) of the original training data can be a suitable alternative to repeated, full retraining”, and answers it with “a set of practical and effective compression best practices for LLMs that combine depth, width, attention and MLP pruning with knowledge distillation-based retraining”.

The economics are the reason to care. The same abstract reports that deriving 8B and 4B models from an already pretrained 15B model “requires up to 40x fewer training tokens per model compared to training from scratch”, giving “compute cost savings of 1.8x for training the full model family”. Those are figures reported by the authors for their own family and hardware, not a prediction for yours, and the compression factor they work at is 2 to 4 times.

The follow-up paper is the practitioner’s version, applying the recipe to models the authors did not train: it “presents a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation”, exploring “(1) depth pruning and (2) joint hidden/attention/MLP (width) pruning”. Its most useful sentence for anyone working outside a model’s original lab is this one:

We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset.

That is a real, transferable finding. You will never have the teacher’s pretraining corpus. Bringing the teacher onto the distribution of the data you do have, before using it as a teacher, is a step you can take on any of the four tracks.

The published result is a model you can inspect. The Llama-3.1-Minitron-4B-Width-Base card, read on 2026-09-09, describes a model produced by pruning Llama-3.1-8B, reducing its embedding size and MLP intermediate dimension, then continuing training with distillation over 94 billion tokens from the Nemotron-4 15B corpus, released under the NVIDIA Open Model License Agreement. Note the name: it begins with “Llama”, which is what section 1.b.i of the Llama 3.1 Community License requires of a model created by training on the Llama Materials or their outputs.

When does this beat training small from scratch?

The comparison the Minitron papers make is with training the small model from scratch, and their answer is that pruning plus distillation wins on compute by a wide margin. But that comparison assumes you would otherwise have trained the small model yourself, which is the assumption to examine.

Prune-and-distil is the right answer when:

• You need a size that nobody publishes. A 3B model in a family that ships 1.7B and 4B, or a model with your context length and nothing else changed.
• You have the teacher and a large, relevant corpus, and 94 billion tokens of training is a sentence you can say without laughing.
• The teacher is your own fine-tune, so no published small model has its behaviour.

Training small from scratch is the right answer when: almost never, at home. Part 12 has you do it once, deliberately, at a size where it finishes in an afternoon, because doing it once teaches you what the pipeline is. It is not a route to a useful model on one machine.

Neither is the right answer when: a published model of the size you want already exists in the family you want. This is the common case and it deserves stating plainly. Qwen3 ships 0.6B, 1.7B, 4B, 8B, 14B, 30B and larger. Somebody has already spent the compute to make each of those as good as it can be at that size. Pruning the 8B to make a 4B, at home, will not beat the 4B they published, and the fair comparison is against that model rather than against your own from-scratch attempt.

?Reality check

The claim to be careful with is that prune-and-distil “recovers” the original model’s quality. What the papers report is that a pruned and distilled model is competitive with similarly sized models trained conventionally, using far less compute than training one from scratch. That is a statement about efficiency against a from-scratch baseline. It is not a claim that the 4B recovers the 8B, and the papers do not make it.

The tools, and their state on 2026-09-09

This is where the honest answer diverges from the interesting one.

NVlabs/Minitron publishes the recipe and the checkpoints. Its README describes the method as “We prune model embedding size, attention heads, and MLP intermediate dimension, following which, we perform continued training with distillation to arrive at the final models”, and lists releases including Llama-3.1-Minitron-4B in width and depth variants, Mistral-NeMo-Minitron-8B and the Nemotron-H and Nemotron-Nano models, under the NVIDIA Open Model License Agreement. What it gives you is the published models and the description; the pruning and retraining pipeline itself belongs to NVIDIA’s training stack rather than to a script you run on one machine.

LLM-Pruner is the closest thing to a runnable structured-pruning tool for a single machine. Its README describes it as compressing large language models to any size and lists support for Llama-3 and Llama-3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan and TinyLlama. Its most recent notable update, read on 2026-09-09, is dated 27 July 2024, adding grouped-query attention support, with the README noting that pruning results for newer models were still being tested. That is a year and a half old at the time of writing, and the model families this course uses are not on its list.

SparseGPT and Wanda publish reference implementations aimed at reproducing their papers. They produce sparse weights, and the gap between sparse weights and a faster model is the runtime, which for llama.cpp, MLX and the GGUF format is not there.

⌚Version note

Statuses read on 2026-09-09 from the repositories and model cards cited above. None of these tools is in the course’s version table, none is validated on the four tracks, and no lab in this course runs one. That is the finding, not an omission: structured pruning of a current model family, on one machine, with tooling somebody maintains, is not a path the course can currently recommend. If that changes, the place it will change first is a maintained pruning tool that names Qwen3 or a comparable family in its supported list.

What to do instead, on your machine

Three things, in order of usefulness.

Use the published small model. It is the fair baseline and it is usually the winner. Part 4’s landscape lesson and the model reference list what exists.

Distil into it. Everything in this part applies: take the published small model as the student, your large one as the teacher, and move the behaviour rather than the parameters. That is the same goal as prune-and-distil with the pruning step replaced by “somebody already did this properly”.

Quantise. If the problem is that the model does not fit, quantisation is a download or one command, no training, and Part 16 measures exactly what it costs you. Pruning is not a cheaper version of quantisation; it is a much more expensive version with a training run attached.

Distinguish fewer values from less work

Setting weights to zero produces sparsity, but a dense kernel may still load and multiply the full matrix. Removing channels, heads or layers changes shapes and can reduce work directly, provided the resulting architecture is valid and supported by the serving engine. Storage reduction and runtime acceleration are therefore different claims.

Before pruning, define the structural unit, importance criterion and recovery-training budget. Save the original checkpoint and configuration. After pruning, check shapes, residual connections, attention configuration and tokeniser compatibility before evaluating quality. Recovery distillation can compensate for some damage, but its benefit must be measured against the additional training cost.

Use a staged report: original, pruned before recovery, recovered and deployed representation. Compare task quality, file size, resident memory and actual latency at every relevant stage. If zeros compress on disk but latency stays unchanged, report that result accurately. It identifies a mismatch between the sparsity pattern and available kernels, not necessarily a failure of the pruning algorithm’s numerical objective.

Recap

Pruning removes parameters along two axes: depth takes whole blocks, width shrinks dimensions inside every block. Both are structured and both make the model genuinely smaller and faster. Unstructured pruning zeroes individual weights and, without a runtime that exploits the pattern, makes nothing smaller or faster on the hardware in this course. One-shot methods are strong and cheap: SparseGPT reaches high sparsity without retraining, Wanda does it with no weight update at all by combining weight magnitude with input activations, and ShortGPT scores whole layers by how little they change their input. Pruning damages a model, and the repair is distillation from the original: the Minitron papers report deriving a family this way with far fewer training tokens than from-scratch training, and note that when the original data is unavailable it helps to fine-tune the teacher on the distillation data first. At home, prune-and-distil is rarely the right call, because a published model of the size you want usually exists and starting from it, then distilling into it, is cheaper and better. The tooling picture on 2026-09-09 supports that conclusion: the maintained pruning tool this course could find last saw a significant update in July 2024 and does not list the model families the course uses.

Check your understanding

Question 1. You apply an unstructured pruning method that zeroes half the weights of a model you serve with llama.cpp. What changes?

• The model file halves in size and decoding roughly doubles in speed
• Nothing about size or speed, unless the kernel and hardware can skip the zeroes; a dense multiply costs the same whatever the value
• The model becomes unusable
• Memory bandwidth use halves

Show the answer and why

Answer: Nothing about size or speed, unless the kernel and hardware can skip the zeroes; a dense multiply costs the same whatever the value

The matrix is the same shape with zeroes in it. That is why the papers report sparsity and perplexity rather than speed, and why semi-structured patterns with hardware support are the ones that turn into performance.

Question 2. What does ShortGPT's Block Influence metric decide?

• Which individual weights to zero
• Which whole layers to remove, by scoring how little each one changes its input
• The quantisation type for each layer
• The learning rate schedule for retraining

Show the answer and why

Answer: Which whole layers to remove, by scoring how little each one changes its input

It is a depth-pruning method, and its title is its finding: layers are more redundant than expected. Depth pruning is also the friendliest kind for serving, because the result is the same architecture with a smaller layer count.

Question 3. The Minitron follow-up paper reports that with no access to the original training data it helps to slightly fine-tune the teacher on the distillation dataset first. Why does that matter to you?

• It does not; it is specific to their hardware
• Because you will never have a teacher's pretraining corpus either, so bringing the teacher onto the distribution of the data you do have is a step you can take on any track
• Because it means the teacher must be retrained from scratch
• Because it makes the student larger

Show the answer and why

Answer: Because you will never have a teacher's pretraining corpus either, so bringing the teacher onto the distribution of the data you do have is a step you can take on any track

It is the most transferable sentence in either paper for somebody working outside a model's original lab. The teacher is being asked to supervise on a distribution it was not tuned for, and a short fine-tune on that distribution first is cheap.

Question 4. You want a 4B model in a family that publishes 1.7B, 4B and 8B. Which is the right first move?

• Prune the 8B to 4B and distil
• Use the published 4B, and distil your behaviour into it if it is not good enough on your task
• Train a 4B from scratch
• Quantise the 8B to 4 bits and call it a 4B

Show the answer and why

Answer: Use the published 4B, and distil your behaviour into it if it is not good enough on your task

Somebody has already spent the compute to make that 4B as good as it can be at that size, and it is the fair baseline. Pruning the 8B at home will be compared against that model, not against a from-scratch attempt, and that is a comparison it is unlikely to win.

Question 5. Which statements about the state of pruning tools on 2026-09-09 are accurate? Select all that apply.

• NVIDIA publishes the Minitron recipe and checkpoints, but the pruning and retraining pipeline is part of its own training stack
• The maintained single-machine structured-pruning tool the course found last saw a significant update in July 2024
• No lab in this course runs a pruning tool
• Pruned checkpoints published by others can still be downloaded and served normally

Show the answer and why

Answer: NVIDIA publishes the Minitron recipe and checkpoints, but the pruning and retraining pipeline is part of its own training stack, The maintained single-machine structured-pruning tool the course found last saw a significant update in July 2024, No lab in this course runs a pruning tool, Pruned checkpoints published by others can still be downloaded and served normally

The absence of a recommended tool is a dated finding rather than an omission. Using somebody else's pruned checkpoint is unaffected by any of it: a depth-pruned model is an ordinary model with fewer layers, and its licence is on its card.

Check my answersClear