Lab: Build Your Model Shortlist
Validated on: written from the documentation cited above; not yet validated on hardware on any track. The tool versions each track was run with will be recorded here when the validation pass is done.
Objective
Section titled “Objective”Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.
By the end of this lab you will have a shortlist of models that fit your machine, chosen by
arithmetic you have checked by hand and recorded with the memory figure, reserve and context length
that produced it. You will have downloaded at least one with the Hugging Face CLI, verified it
against the SHA-256 the Hub publishes, proved that re-running the download is safe, audited the
result, and filed it in a model library at ~/models/ that every engine from Part 6 onward reads.
The download is the small part. The lab exists for three habits: decide what fits before you spend the bandwidth; never trust a multi-gigabyte file you have not checksummed; and never let the only record of what a file is be its name.
What this lab assumes from earlier parts
Section titled “What this lab assumes from earlier parts”Part 1’s lab created ~/llm-course/ with a .venv and started labbook.md in it. Part 2’s lab
installed huggingface_hub into that environment and used hf download once, to put the
Qwen3-1.7B safetensors checkpoint under ~/llm-course/models/qwen3-1.7b/. This lab keeps those
where they are: ~/llm-course/models/ is where the Python-side checkpoints for transformers live,
and ~/models/, which you create here, is the engine library of GGUF files. Part 5 then points the
Hugging Face cache at ~/models/hf/ so that everything ends up on one disk.
The arithmetic comes from Choosing a Model for a Memory Budget: bytes per parameter by format, the key-value cache formula, and the headroom rules. This lab does not re-teach it; it makes you do it once by hand and then automates it.
The library and who reads it
Section titled “The library and who reads it”One directory tree, four kinds of reader. The tree is the Hub namespace, then the repository name, then the file, exactly as the Hub spells them, with a checksum beside every file:
Output — what you should see
~/models/ README.md unsloth/ Qwen3-8B-GGUF/ Qwen3-8B-Q4_K_M.gguf Qwen3-8B-Q4_K_M.gguf.sha256 .cache/huggingface/ hf's own record of the download; leave it alone ggml-org/ gpt-oss-20b-GGUF/ gpt-oss-20b-MXFP4.gguf gpt-oss-20b-MXFP4.gguf.sha256 hf/ reserved for the Hugging Face cache (Part 5 sets HF_HOME here)| Reader | How it finds the file | Where the course sets it up |
|---|---|---|
| llama.cpp | a path on the command line, --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf |
Part 6 |
| Ollama | a Modelfile whose FROM line names the file; Ollama imports it into its own store once |
Part 7 |
| LM Studio | a symbolic link from its publisher/model/file tree into this one |
Part 7 |
| transformers, vLLM, SGLang, mlx-lm | the Hugging Face cache, moved onto the same disk with HF_HOME |
Part 5, Part 7 |
| A second machine | this directory exported read-only over NFS | Part 18 |
The layout is the one decision in this lab that is expensive to change later, because every later
part types these paths. Managing a Model
Library in Part 7 builds the tooling on
top of it. The namespaces are the ones the course model reference records: unsloth/Qwen3-*-GGUF
for Qwen3, ggml-org for gpt-oss. Qwen publishes its own GGUF conversions as well; the course
standardises on unsloth because that one namespace ships every tier the course uses, so paths and
sizes stay the same from Part 6 onwards, as the memory
lesson
explains.
Requirements
Section titled “Requirements”Every track needs the Part 1 environment, the hf CLI at Hugging Face CLI 1.30.0 · verified 2026-09-08 or
later, curl, a SHA-256 tool, and disk space for whatever you choose to download. A free Hugging
Face account is needed only for the optional gated step. Budget about 60 minutes of attention, 75
with the optional Task 11; the downloads run unattended and their length depends on your
connection, as the table below shows.
Track S — NVIDIA DGX Spark
A DGX Spark has 128 GB of unified memory, all of it visible to the GPU. Storage is the constraint to check first: the base configuration is 1 TB, and the 96–128 GB tier’s reference model alone is a 63 GB file.
The hf CLI is the same on aarch64 as anywhere else, and downloads are architecture
independent. If you work inside the NGC PyTorch container from Part 1, install the CLI on the
host and mount ~/models into the container, so that the library outlives any one container.
Track X — AMD Ryzen AI Max+ 395
A Ryzen AI Max+ 395 machine has 64 or 128 GB of unified memory, but the memory the GPU can address is capped below the machine total: the hardware reference records about 96 GB visible to the GPU on a 128 GB machine under Windows, and on Linux the driver’s GTT limit decides. Part 5’s Strix Halo lesson explains both, and its lab prints what the driver is actually offering; until you have run it, Task 3 says which figure to use. On Windows, run the lab’s scripts in WSL2.
Track M — Apple silicon
Unified memory, but macOS caps how much of it the GPU may hold resident through the
wired-memory limit, covered in Part 5’s Apple silicon
lesson; on a
24 GB Mac the models this lab picks are well under it, on a 128 GB Mac Studio the largest ones
are not until it is raised. macOS ships shasum rather than sha256sum; the lab’s scripts
handle both.
Track N — NVIDIA desktop or laptop
On a discrete card the number that matters is VRAM, not system memory. A machine with 64 GB of RAM and a 16 GB card runs 16 GB models, because anything that spills to system memory crosses PCIe and Part 5 measures how badly that goes.
On Windows, run the whole lab inside WSL2, so that the library lives on the Linux filesystem the later parts’ engines read.
Download sizes and times, so you can choose before you commit
Section titled “Download sizes and times, so you can choose before you commit”Every byte count below is the published size of a real file in the repository named, read from the Hub’s file listing on 13 September 2026. The minutes are arithmetic, bytes × 8 ÷ link speed, not measurements; your link will be slower than its label.
| Memory tier | A sensible first download | Repository and file | Bytes | GB | At 50 Mbit/s | At 500 Mbit/s |
|---|---|---|---|---|---|---|
| Slow link, any tier | Qwen3-1.7B, Q8_0 | unsloth/Qwen3-1.7B-GGUF, Qwen3-1.7B-Q8_0.gguf |
1,834,426,944 | 1.83 | 5 min | 0.5 min |
| 8 GB | Qwen3-4B, Q4_K_M | unsloth/Qwen3-4B-GGUF, Qwen3-4B-Q4_K_M.gguf |
2,497,281,312 | 2.50 | 7 min | 0.7 min |
| 8 GB (short context) or 12–16 GB | Qwen3-8B, Q4_K_M | unsloth/Qwen3-8B-GGUF, Qwen3-8B-Q4_K_M.gguf |
5,027,784,512 | 5.03 | 13 min | 1.3 min |
| 24 GB | Qwen3-14B, Q4_K_M | unsloth/Qwen3-14B-GGUF, Qwen3-14B-Q4_K_M.gguf |
9,001,753,984 | 9.00 | 24 min | 2.4 min |
| 32 GB | gpt-oss-20b, MXFP4 | ggml-org/gpt-oss-20b-GGUF, gpt-oss-20b-MXFP4.gguf |
12,109,566,624 | 12.11 | 32 min | 3.2 min |
| 48–64 GB | Qwen3-30B-A3B, Q4_K_M | unsloth/Qwen3-30B-A3B-GGUF, Qwen3-30B-A3B-Q4_K_M.gguf |
18,556,686,912 | 18.56 | 49 min | 4.9 min |
| 96–128 GB | gpt-oss-120b, MXFP4 | ggml-org/gpt-oss-120b-GGUF, gpt-oss-120b-MXFP4.gguf |
63,387,346,208 | 63.39 | 169 min | 16.9 min |
| Optional, any tier | Llama 3.1 8B Instruct, Q4_K_M (community conversion) | bartowski/Meta-Llama-3.1-8B-Instruct-GGUF, Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf |
4,920,739,232 | 4.92 | 13 min | 1.3 min |
Every model in the table is Apache-2.0 except the last, which is under the Llama 3.1 Community License; the whole reference set and the reasons for it are in the families lesson. Disk space: the file, plus the same again if you intend to add a second quantisation of the same model in Part 6, plus 10 GB of headroom so that the operating system never runs the disk to zero.
Preflight
Section titled “Preflight”Run every line before starting the tasks.
RunnableAll tracks
cd ~/llm-coursesource .venv/bin/activatepython3 --versionhf versioncurl --version | head -n 1command -v sha256sum || command -v shasumdf -h ~Output — what you should see
Python 3.12.x✓ hf version version: 1.30.0curl 8.x.x (...)/usr/bin/sha256sumFilesystem Size Used Avail Use% Mounted on/dev/nvme0n1p2 9xxG 1xxG 8xxG xx% /Look for four things. hf version prints version: followed by 1.30.0 or later; it may also print
a one-line hint about a newer release or an hf-cli skill, which you can ignore. The SHA-256 line
prints a path, /usr/bin/sha256sum on Linux and /usr/bin/shasum on macOS. Avail on the
filesystem holding your home directory exceeds the download you chose plus 10 GB. And
python3 --version is the environment’s Python, 3.9 or later.
Every hf output on this page is the terminal form. The CLI guide’s “Output formatting” section
documents a --format option on each command whose default, auto, picks human in a terminal
and agent when the CLI detects an AI coding agent from the variables such agents set (hf env
reports the verdict as Run by AI agent ?). Inside such an agent, version 1.30.0 prints
version=1.30.0 instead of the two lines above, and other commands print terser lines;
--format human forces the terminal form.
If hf version prints something older than 1.30.0, or hf: command not found, Task 1 fixes it.
If the disk is short, choose a smaller row now.
1. Make sure the hf CLI is the one you think it is
Section titled “1. Make sure the hf CLI is the one you think it is”The executable is hf, and it reaches your shell by one of four routes the CLI guide documents.
The Part 2 lab installed it into the course environment with huggingface_hub, so on most machines
the fix for an old version is an upgrade inside that environment:
RunnableAll tracks
cd ~/llm-coursesource .venv/bin/activateuv pip install -U huggingface_hubhf versionThe other three routes are the standalone installer, which the documentation recommends for a
machine-wide install (curl -LsSf https://hf.co/cli/install.sh | bash, then open a new terminal;
it also installs an hf-cli skill for AI agents unless you append -s -- --exclude-skill);
uvx hf, which runs the latest version in an isolated environment with no installation; and
brew install hf on a Mac. hf update later upgrades whichever route you used. Whatever the
route, confirm the executable answers:
RunnableAll tracks
hf --helpOutput — what you should see
Usage: hf [OPTIONS] [COMMAND] [ARGS]...
Hugging Face Hub CLI
Options: --install-completion Install completion for the current shell. ...Main commands: auth Manage authentication (login, logout, etc.). ... cache Manage local cache directory. ... download Download files from the Hub. ...The usage line differs between releases (the documentation shows COMMAND without brackets;
version 1.30.0 prints [COMMAND]). Check for Hugging Face Hub CLI and the auth, cache and
download commands, the three this lab uses. Record in the notebook’s Environment section: hf version and how it was installed (environment,
installer, uvx or brew).
2. Sign in only if you need to
Section titled “2. Sign in only if you need to”Every Qwen3 and gpt-oss repository in this lab is public and downloads without an account. Two
things change that. Gated repositories, which includes everything under meta-llama, need you to
have accepted the terms on the model page in a browser and to be authenticated locally. And an
unauthenticated download prints this warning, observed with version 1.30.0, because anonymous
requests are rate limited:
Output — what you should see
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.If you have an account, signing in is one command. The documentation describes the default flow
as a browser login: the command prints a URL and a short code, you open the URL, enter the code
and approve, and the CLI saves the token itself under ~/.cache/huggingface/token.
RunnableAll tracks
hf auth loginOutput — what you should see
? How would you like to log in? [Use arrows, Enter to confirm]> Log in with your browser Paste an access token
Open this URL in your browser: https://huggingface.co/oauth/device
And enter the code: ABCD-EFGH
Waiting for authorization...Token is valid....Login successful.The second option pastes a User Access Token from your account’s settings page; the documentation
recommends passing it from an environment variable, hf auth login --token $HF_TOKEN, rather than
typing it where it lands in your shell history. Read access is all this course needs. Check it took:
RunnableAll tracks
hf auth whoamiOutput — what you should see
your-usernameorgs: ...When not signed in, the same command prints Error: Not logged in and exits non-zero, which is the
state every public download in this lab works from. Record in the notebook whether this machine is
signed in and, if so, the name of the token you created for it.
3. Find the memory number for your machine
Section titled “3. Find the memory number for your machine”Every figure this lab uses is in decimal gigabytes of 109 bytes, the unit the Hub reports file sizes in, so convert what the command prints rather than rounding to the number on the box; the units section of the memory lesson explains the seven per cent difference.
Track S — NVIDIA DGX Spark
RunnableTrack S · DGX Spark
grep MemTotal /proc/meminfoOutput — what you should see
MemTotal: xxxxxxxxx kBThe kernel reports kibibytes; multiply by 1,024 and divide by 109. A 128 GB Spark reports a little under 128 because the firmware and kernel keep some; use 128 and the script’s default 10 GB reserve, which leaves 118 GB for weights and cache.
Track X — AMD Ryzen AI Max+ 395
RunnableTrack X · Ryzen AI Max+
grep MemTotal /proc/meminfoOutput — what you should see
MemTotal: xxxxxxxxx kBThat is the machine total on Linux. Under Windows, take the total from Windows itself: inside
WSL2, /proc/meminfo describes the WSL2 virtual machine. On a 128 GB machine under Windows the
total is not the number to give the fit script: use the documented GPU-visible figure,
about 96, per the hardware reference.
For a 64 GB machine, and for Linux at either size, the course has no documented figure: give
the script the machine total, label the run provisional, and re-run with the driver’s actual
limit once Part 5’s lab has read it; that re-run is the one to keep. Write both numbers in the
notebook, the total and the GPU-visible figure or the word provisional.
Track M — Apple silicon
RunnableTrack M · Apple silicon
sysctl -n hw.memsizeOutput — what you should see
25769803776Bytes. A “24 GB” Mac reports 25,769,803,776, which is 24 × 230; divided by 109 that is 25.8 decimal gigabytes, the figure to give the fit script. The script’s default reserve for that size is 3 GB, which covers macOS and a browser; leave 10 GB or more on a 128 GB Mac Studio if you intend to keep working while a model is loaded.
Track N — NVIDIA desktop or laptop
RunnableTrack N · NVIDIA GPU
nvidia-smi --query-gpu=name,memory.total --format=csvOutput — what you should see
name, memory.total [MiB]NVIDIA GeForce RTX xxxx, xxxxx MiBMebibytes; multiply by 1,048,576 and divide by 109. A card sold as 24 GB has
24 × 1,024 = 24,576 MiB, which is 25.8 decimal gigabytes, and reports a little less than that
because the driver keeps some. Give the fit script the figure the command prints, converted,
with an explicit --reserve-gb 1.5, because on a discrete card the operating system does not
share the pool and the reserve only needs to cover the engine’s buffers and the display.
Record in the notebook now, because every later part asks for it: memory_gb (decimal), whether
it is unified memory, GPU-visible unified memory or VRAM, and the reserve you will use.
4. Do the arithmetic once by hand
Section titled “4. Do the arithmetic once by hand”The fit script is only trustworthy if you have done its sums yourself once. Three numbers decide
a fit: the weights, from a published file size; the context, from four fields of the model’s
config.json; and the budget, your memory figure minus the reserve. The formula from the lesson:
Pseudocode — not a real command
kv_bytes_per_token = 2 x num_hidden_layers x num_key_value_heads x head_dim x bytes_per_elementkv_bytes = kv_bytes_per_token x context_lengthfits = weights_bytes + kv_bytes <= (memory_gb - reserve_gb) x 10^9bytes_per_element is 2 for the FP16 cache every engine defaults to and 1 for an 8-bit cache.
You already have a config.json on disk from Part 2. Read the four values out of it and price two
context lengths:
RunnableAll tracks
"""Read the four key-value cache numbers out of a config.json and price two context lengths."""import json, sys
path = sys.argv[1] if len(sys.argv) > 1 else "models/qwen3-1.7b/config.json"cfg = json.load(open(path, encoding="utf-8"))layers = cfg["num_hidden_layers"]kv_heads = cfg["num_key_value_heads"]head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]per_token_fp16 = 2 * layers * kv_heads * head_dim * 2
print(f"layers={layers} kv_heads={kv_heads} head_dim={head_dim} " f"(query heads={cfg['num_attention_heads']}, ignored on purpose)")print(f"bytes per token, FP16 cache: {per_token_fp16:,}")for ctx in (8192, 32768): print(f" {ctx:>6} tokens: {per_token_fp16 * ctx / 1e9:6.2f} GB at FP16, " f"{per_token_fp16 * ctx / 2 / 1e9:6.2f} GB at 8-bit")Save it as ~/llm-course/kv-from-config.py, then run it against the checkpoint Part 2 left on
disk:
RunnableAll tracks
cd ~/llm-coursepython3 kv-from-config.py models/qwen3-1.7b/config.jsonOutput — what you should see
layers=28 kv_heads=8 head_dim=128 (query heads=16, ignored on purpose)bytes per token, FP16 cache: 114,688 8192 tokens: 0.94 GB at FP16, 0.47 GB at 8-bit 32768 tokens: 3.76 GB at FP16, 1.88 GB at 8-bitThat output is exact arithmetic on values the Qwen3-1.7B config.json on the Hub declares:
num_hidden_layers: 28, num_key_value_heads: 8, head_dim: 128 and num_attention_heads: 16.
The line that matters most is the one about query heads: using 16 instead of 8 would double every
figure, the commonest mistake in this arithmetic.
Now the weights side, for the model you are most likely to download. The unsloth listing gives
Qwen3-8B-Q4_K_M.gguf as 5,027,784,512 bytes and the card gives 8.2 billion parameters:
RunnableAll tracks
python3 -c "print(round(5_027_784_512 * 8 / 8.2e9, 2), 'bits per weight')"Output — what you should see
4.91 bits per weightWhich is the lesson’s “about 4.9” for Q4_K_M, from the real file. Put the two halves together for Qwen3-8B at Q4_K_M on a 16 GB machine with the default 1.5 GB reserve, so a 14.50 GB budget, at three context lengths. Every figure is derived from the inputs above; none is measured.
| Context | Weights, Q4_K_M | KV cache at FP16 (147,456 bytes per token) | Total | Against 14.50 GB |
|---|---|---|---|---|
| 8,192 | 5.03 GB | 1.21 GB | 6.24 GB | fits |
| 32,768 | 5.03 GB | 4.83 GB | 9.86 GB | fits |
| 65,536 | 5.03 GB | 9.66 GB | 14.69 GB | does not fit |
The weights column does not move. The cache column is linear in context, so doubling the context doubles it, and at 64k it is larger than the whole model. Write in the notebook, for one model you care about: layers, key-value heads, head dimension, bytes per token, and the total at the context you intend to use.
5. Run the fit calculator
Section titled “5. Run the fit calculator”Download the script into ~/llm-course/ and read it before running it. Its model table is the
course reference set: every layer count, key-value head count and head dimension was read from the
model’s config.json, and every byte count marked published is the size of a real file in the
GGUF repository named beside it, read from the Hub on the dates in the docstring.
RunnableAll tracks
"""Work out which course reference models fit in a memory budget at a target context length.
Purpose: apply the memory arithmetic from Part 4 to one machine: weights from bytes per parameter (or from the published GGUF file size where one is known), key-value cache from the model's own config.json shape, plus a reserve for the operating system and the engine. Prints a fit table and a suggested shortlist, and can append the result to the lab notebook as one JSON line.Platform: all (pure Python 3.9+, standard library only; no accelerator needed)Minimum memory: 8 GBAssumes: nothing is downloaded and nothing is loaded. Every layer, key-value head and head dimension below was read from the model's config.json on the Hugging Face Hub on 2026-09-08, and every byte count marked "published" is the size of a real file in the GGUF repository named beside it, read from that repository's Hub file listing on 2026-09-13. The GGUF repositories are the ones the course model reference records: unsloth for Qwen3, ggml-org for gpt-oss, bartowski for Llama.
Usage: python3 fit-models.py --memory-gb 16 --context 8192 [--kv-bits 16] [--reserve-gb 1.5] [--labbook labbook.md] [--json] python3 fit-models.py --memory-gb 16 --context 8192 --explain qwen3-8b"""
from __future__ import annotations
import argparseimport jsonimport platformimport subprocessimport sysfrom datetime import datefrom pathlib import Path
# Bytes per parameter, including the per-block scales that the nominal bit width leaves# out. Derived in the "Choosing a Model for a Memory Budget" lesson from the block# structures documented at https://huggingface.co/docs/hub/gguf, and cross-checked# against the published file sizes below.BYTES_PER_PARAM = { "bf16": 2.0, "q8_0": 1.0625, "q6_k": 0.8203, "q5_k_m": 0.7135, "q4_k_m": 0.6125, "iq4_xs": 0.5313, "mxfp4": 0.5417,}
# Order used when reporting, heaviest first.QUANT_ORDER = ["bf16", "q8_0", "q6_k", "q5_k_m", "q4_k_m", "iq4_xs", "mxfp4"]
GB = 1_000_000_000
# The course reference set. "layers", "kv_heads" and "head_dim" come from each model's# config.json; "files" maps a quantisation to the file name and published size in bytes# in the GGUF repository named by "gguf_repo". A quantisation with no published file is# still reported, with its size estimated from bytes per parameter.MODELS = [ { "id": "qwen3-1.7b", "name": "Qwen3-1.7B", "repo": "Qwen/Qwen3-1.7B", "gguf_repo": "unsloth/Qwen3-1.7B-GGUF", "params_total_b": 1.7, "params_active_b": 1.7, "layers": 28, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-1.7B-Q8_0.gguf", 1_834_426_944), "q6_k": ("Qwen3-1.7B-Q6_K.gguf", 1_417_755_200), "q5_k_m": ("Qwen3-1.7B-Q5_K_M.gguf", 1_257_880_128), "q4_k_m": ("Qwen3-1.7B-Q4_K_M.gguf", 1_107_409_472), }, }, { "id": "qwen3-4b", "name": "Qwen3-4B", "repo": "Qwen/Qwen3-4B", "gguf_repo": "unsloth/Qwen3-4B-GGUF", "params_total_b": 4.0, "params_active_b": 4.0, "layers": 36, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-4B-Q8_0.gguf", 4_280_405_792), "q6_k": ("Qwen3-4B-Q6_K.gguf", 3_306_261_792), "q5_k_m": ("Qwen3-4B-Q5_K_M.gguf", 2_889_514_272), "q4_k_m": ("Qwen3-4B-Q4_K_M.gguf", 2_497_281_312), }, }, { "id": "qwen3-8b", "name": "Qwen3-8B", "repo": "Qwen/Qwen3-8B", "gguf_repo": "unsloth/Qwen3-8B-GGUF", "params_total_b": 8.2, "params_active_b": 8.2, "layers": 36, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-8B-Q8_0.gguf", 8_709_519_168), "q6_k": ("Qwen3-8B-Q6_K.gguf", 6_725_900_096), "q5_k_m": ("Qwen3-8B-Q5_K_M.gguf", 5_851_113_280), "q4_k_m": ("Qwen3-8B-Q4_K_M.gguf", 5_027_784_512), }, }, { # The course's cross-engine comparison model. Meta publishes no GGUF, so the # files are bartowski's community conversion (card: llama.cpp release b3472, # importance-matrix quantisation). Layer, head and head-dimension values are # from the course model reference; the original repository is gated. "id": "llama-3.1-8b", "name": "Llama-3.1-8B", "repo": "meta-llama/Llama-3.1-8B-Instruct", "gguf_repo": "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF", "params_total_b": 8.0, "params_active_b": 8.0, "layers": 32, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Meta-Llama-3.1-8B-Instruct-Q8_0.gguf", 8_540_775_840), "q6_k": ("Meta-Llama-3.1-8B-Instruct-Q6_K.gguf", 6_596_011_424), "q5_k_m": ("Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf", 5_732_992_416), "q4_k_m": ("Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf", 4_920_739_232), }, "note": "community conversion (bartowski); Llama 3.1 Community License; optional", }, { "id": "qwen3-14b", "name": "Qwen3-14B", "repo": "Qwen/Qwen3-14B", "gguf_repo": "unsloth/Qwen3-14B-GGUF", "params_total_b": 14.8, "params_active_b": 14.8, "layers": 40, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-14B-Q8_0.gguf", 15_698_534_784), "q6_k": ("Qwen3-14B-Q6_K.gguf", 12_121_938_304), "q5_k_m": ("Qwen3-14B-Q5_K_M.gguf", 10_514_570_624), "q4_k_m": ("Qwen3-14B-Q4_K_M.gguf", 9_001_753_984), }, }, { "id": "qwen3-30b-a3b", "name": "Qwen3-30B-A3B", "repo": "Qwen/Qwen3-30B-A3B", "gguf_repo": "unsloth/Qwen3-30B-A3B-GGUF", "params_total_b": 30.5, "params_active_b": 3.3, "layers": 48, "kv_heads": 4, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-30B-A3B-Q8_0.gguf", 32_483_932_736), "q6_k": ("Qwen3-30B-A3B-Q6_K.gguf", 25_092_532_800), "q5_k_m": ("Qwen3-30B-A3B-Q5_K_M.gguf", 21_725_581_888), "q4_k_m": ("Qwen3-30B-A3B-Q4_K_M.gguf", 18_556_686_912), }, }, { "id": "qwen3-32b", "name": "Qwen3-32B", "repo": "Qwen/Qwen3-32B", "gguf_repo": "unsloth/Qwen3-32B-GGUF", "params_total_b": 32.8, "params_active_b": 32.8, "layers": 64, "kv_heads": 8, "head_dim": 128, "quants": ["q8_0", "q6_k", "q5_k_m", "q4_k_m"], "files": { "q8_0": ("Qwen3-32B-Q8_0.gguf", 34_817_719_968), "q6_k": ("Qwen3-32B-Q6_K.gguf", 26_883_307_168), "q5_k_m": ("Qwen3-32B-Q5_K_M.gguf", 23_214_832_288), "q4_k_m": ("Qwen3-32B-Q4_K_M.gguf", 19_762_150_048), }, }, { "id": "gpt-oss-20b", "name": "gpt-oss-20b", "repo": "openai/gpt-oss-20b", "gguf_repo": "ggml-org/gpt-oss-20b-GGUF", "params_total_b": 21.0, "params_active_b": 3.6, "layers": 24, "kv_heads": 8, "head_dim": 64, "kv_is_upper_bound": True, "quants": ["mxfp4"], "files": {"mxfp4": ("gpt-oss-20b-MXFP4.gguf", 12_109_566_624)}, }, { "id": "gpt-oss-120b", "name": "gpt-oss-120b", "repo": "openai/gpt-oss-120b", "gguf_repo": "ggml-org/gpt-oss-120b-GGUF", "params_total_b": 117.0, "params_active_b": 5.1, "layers": 36, "kv_heads": 8, "head_dim": 64, "kv_is_upper_bound": True, "quants": ["mxfp4"], "files": {"mxfp4": ("gpt-oss-120b-MXFP4.gguf", 63_387_346_208)}, },]
def kv_bytes_per_token(model: dict, kv_bits: int) -> int: """2 x layers x kv_heads x head_dim x bytes per element - the formula from the lesson.""" bytes_per_element = kv_bits / 8 return int(2 * model["layers"] * model["kv_heads"] * model["head_dim"] * bytes_per_element)
def weight_bytes(model: dict, quant: str) -> tuple[int, str]: """Published file size where the course has read one, else parameters x bytes per parameter.""" published = model.get("files", {}).get(quant) if published is not None: return published[1], "published" per_param = BYTES_PER_PARAM[quant] return int(model["params_total_b"] * 1e9 * per_param), "estimated"
def default_reserve_gb(memory_gb: float) -> float: """The headroom rule from the lesson: OS, engine buffers and other applications.""" if memory_gb <= 16: return 1.5 if memory_gb <= 32: return 3.0 if memory_gb <= 64: return 8.0 return 10.0
def detect_memory_gb() -> float | None: """Best-effort total memory, so the script can suggest a value. Always overridable.""" system = platform.system() if system == "Linux": try: for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines(): if line.startswith("MemTotal:"): return int(line.split()[1]) * 1024 / GB except OSError: return None if system == "Darwin": try: out = subprocess.run( ["sysctl", "-n", "hw.memsize"], capture_output=True, text=True, check=True ) return int(out.stdout.strip()) / GB except (OSError, subprocess.CalledProcessError, ValueError): return None return None
def evaluate(memory_gb: float, context: int, kv_bits: int, reserve_gb: float) -> list[dict]: """One row per model and quantisation, with weights, cache and whether it fits.""" budget = memory_gb - reserve_gb rows = [] for model in MODELS: per_token = kv_bytes_per_token(model, kv_bits) kv_gb = per_token * context / GB for quant in model["quants"]: raw, source = weight_bytes(model, quant) weights_gb = raw / GB total_gb = weights_gb + kv_gb file_name = model.get("files", {}).get(quant, (None, None))[0] rows.append( { "model": model["id"], "name": model["name"], "quant": quant, "weights_gb": round(weights_gb, 2), "weights_source": source, "kv_bytes_per_token": per_token, "kv_gb": round(kv_gb, 2), "kv_is_upper_bound": bool(model.get("kv_is_upper_bound")), "total_gb": round(total_gb, 2), "fits": total_gb <= budget, "params_total_b": model["params_total_b"], "params_active_b": model["params_active_b"], "gguf_repo": model["gguf_repo"], "file": file_name, "downloadable": file_name is not None, "note": model.get("note"), } ) return rows
def shortlist(rows: list[dict]) -> list[dict]: """The heaviest quantisation that fits and can actually be downloaded, per model.""" best: dict[str, dict] = {} for row in rows: if not row["fits"] or not row["downloadable"]: continue current = best.get(row["model"]) if current is None or row["weights_gb"] > current["weights_gb"]: best[row["model"]] = row return sorted(best.values(), key=lambda r: r["params_total_b"])
def print_table(rows: list[dict], budget_gb: float) -> None: header = f"{'model':16s} {'quant':7s} {'weights':>9s} {'KV':>8s} {'total':>8s} fit" print(header) print("-" * len(header)) for row in rows: mark = "*" if row["weights_source"] == "estimated" else " " bound = "<" if row["kv_is_upper_bound"] else " " verdict = "yes" if row["fits"] else "no" print( f"{row['name']:16s} {row['quant']:7s} " f"{row['weights_gb']:8.2f}{mark} {row['kv_gb']:7.2f}{bound} " f"{row['total_gb']:8.2f} {verdict}" ) print() print(f"budget after reserve: {budget_gb:.2f} GB") if any(row["weights_source"] == "estimated" for row in rows): print("* weights estimated from bytes per parameter; no published file size was read") if any(row["kv_is_upper_bound"] for row in rows): print("< key-value cache is an upper bound: some layers use a sliding window")
def explain(model: dict, memory_gb: float, reserve_gb: float, context: int, kv_bits: int) -> None: """Show the arithmetic behind one model's rows, so the table can be checked by hand.""" per_token = kv_bytes_per_token(model, kv_bits) budget = memory_gb - reserve_gb print(f"{model['name']} ({model['repo']}; GGUF files from {model['gguf_repo']})") print(f" config.json: num_hidden_layers {model['layers']}, num_key_value_heads " f"{model['kv_heads']}, head_dim {model['head_dim']}") print(f" key-value cache per token = 2 x {model['layers']} x {model['kv_heads']} x " f"{model['head_dim']} x {kv_bits / 8:g} bytes = {per_token:,} bytes") for ctx in sorted({4096, 8192, 32768, context}): print(f" x {ctx:>6} tokens = {per_token * ctx / GB:6.2f} GB") print(" weights:") for quant in model["quants"]: raw, source = weight_bytes(model, quant) bits = raw * 8 / (model["params_total_b"] * 1e9) print(f" {quant:7s} {raw:>15,} bytes = {raw / GB:6.2f} GB {source:9s} " f"({bits:.2f} bits per weight over {model['params_total_b']} B parameters)") print(f" budget: {memory_gb} GB - {reserve_gb} GB reserve = {budget:.2f} GB") kv_gb = per_token * context / GB for quant in model["quants"]: raw, _ = weight_bytes(model, quant) total = raw / GB + kv_gb verdict = "fits" if total <= budget else "does not fit" print(f" {quant:7s} at {context} tokens: {raw / GB:.2f} + {kv_gb:.2f} = " f"{total:.2f} GB {verdict}") if model.get("kv_is_upper_bound"): print(" the cache figure is an upper bound: some layers use a sliding window") if model.get("note"): print(f" note: {model['note']}")
def main() -> None: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument( "--memory-gb", type=float, default=None, help="memory available to the model in GB: total unified memory, or VRAM on a discrete GPU", ) parser.add_argument("--context", type=int, default=8192, help="target context length in tokens") parser.add_argument( "--kv-bits", type=int, default=16, choices=[16, 8, 4], help="key-value cache element width" ) parser.add_argument( "--reserve-gb", type=float, default=None, help="memory held back for the OS, the engine and other applications", ) parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file") parser.add_argument("--json", action="store_true", help="print the full result as JSON") parser.add_argument( "--explain", metavar="MODEL_ID", default=None, help="print the arithmetic for one model id (for example qwen3-8b) and exit", ) args = parser.parse_args()
memory_gb = args.memory_gb if memory_gb is None: detected = detect_memory_gb() if detected is None: parser.error("could not detect memory on this platform; pass --memory-gb explicitly") memory_gb = round(detected, 1) print(f"detected total memory: {memory_gb} GB (override with --memory-gb)", file=sys.stderr)
reserve_gb = args.reserve_gb if args.reserve_gb is not None else default_reserve_gb(memory_gb) budget_gb = memory_gb - reserve_gb if budget_gb <= 0: parser.error(f"reserve of {reserve_gb} GB leaves nothing of {memory_gb} GB")
if args.explain: wanted = [m for m in MODELS if m["id"] == args.explain] if not wanted: parser.error(f"unknown model id {args.explain!r}; known: " + ", ".join(m["id"] for m in MODELS)) explain(wanted[0], memory_gb, reserve_gb, args.context, args.kv_bits) return
rows = evaluate(memory_gb, args.context, args.kv_bits, reserve_gb) picks = shortlist(rows) download_gb = round(sum(p["weights_gb"] for p in picks), 2)
result = { "lab": "part-04/fit-models", "date": date.today().isoformat(), "memory_gb": memory_gb, "reserve_gb": reserve_gb, "budget_gb": round(budget_gb, 2), "context": args.context, "kv_bits": args.kv_bits, "shortlist": [ { "model": p["model"], "quant": p["quant"], "weights_gb": p["weights_gb"], "kv_gb": p["kv_gb"], "total_gb": p["total_gb"], "gguf_repo": p["gguf_repo"], "file": p["file"], "note": p.get("note"), } for p in picks ], "download_gb": download_gb, }
if args.json: print(json.dumps(result, indent=2)) else: print(f"memory {memory_gb} GB, reserve {reserve_gb} GB, context {args.context} tokens, " f"KV at {args.kv_bits}-bit") print() print_table(rows, budget_gb) print() if picks: print("shortlist (heaviest quantisation that fits, per model):") for p in picks: print(f" {p['name']:16s} {p['quant']:7s} {p['weights_gb']:8.2f} GB " f"{p['gguf_repo']} {p['file']}") if p.get("note"): print(f" {'':16s} {'':7s} {'':11s} {p['note']}") print(f" total download if you took every row: {download_gb:.2f} GB") print(" this is a menu, not a shopping list: the lab downloads one or two of these") else: print("nothing in the reference set fits. Try a shorter context or --kv-bits 8.")
if args.labbook: with Path(args.labbook).open("a", encoding="utf-8") as handle: handle.write(json.dumps(result) + "\n") print(f"recorded in {args.labbook}")
if __name__ == "__main__": main()Run it with your memory figure and the context length you actually intend to use. The 16 GB, 8k example below is real output from the script; yours differs only in the budget line and the verdicts.
RunnableAll tracks
cd ~/llm-coursepython3 fit-models.py --memory-gb 16 --context 8192Output — what you should see
memory 16.0 GB, reserve 1.5 GB, context 8192 tokens, KV at 16-bit
model quant weights KV total fit---------------------------------------------------------Qwen3-1.7B q8_0 1.83 0.94 2.77 yesQwen3-1.7B q6_k 1.42 0.94 2.36 yesQwen3-1.7B q5_k_m 1.26 0.94 2.20 yesQwen3-1.7B q4_k_m 1.11 0.94 2.05 yesQwen3-4B q8_0 4.28 1.21 5.49 yes...Qwen3-8B q8_0 8.71 1.21 9.92 yesQwen3-8B q6_k 6.73 1.21 7.93 yesQwen3-8B q5_k_m 5.85 1.21 7.06 yesQwen3-8B q4_k_m 5.03 1.21 6.24 yesLlama-3.1-8B q8_0 8.54 1.07 9.61 yes...Qwen3-14B q8_0 15.70 1.34 17.04 noQwen3-14B q6_k 12.12 1.34 13.46 yes...Qwen3-30B-A3B q4_k_m 18.56 0.81 19.36 no...gpt-oss-20b mxfp4 12.11 0.40< 12.51 yesgpt-oss-120b mxfp4 63.39 0.60< 63.99 no
budget after reserve: 14.50 GB< key-value cache is an upper bound: some layers use a sliding window
shortlist (heaviest quantisation that fits, per model): Qwen3-1.7B q8_0 1.83 GB unsloth/Qwen3-1.7B-GGUF Qwen3-1.7B-Q8_0.gguf Qwen3-4B q8_0 4.28 GB unsloth/Qwen3-4B-GGUF Qwen3-4B-Q8_0.gguf Llama-3.1-8B q8_0 8.54 GB bartowski/Meta-Llama-3.1-8B-Instruct-GGUF Meta-Llama-3.1-8B-Instruct-Q8_0.gguf community conversion (bartowski); Llama 3.1 Community License; optional Qwen3-8B q8_0 8.71 GB unsloth/Qwen3-8B-GGUF Qwen3-8B-Q8_0.gguf Qwen3-14B q6_k 12.12 GB unsloth/Qwen3-14B-GGUF Qwen3-14B-Q6_K.gguf gpt-oss-20b mxfp4 12.11 GB ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf total download if you took every row: 47.59 GB this is a menu, not a shopping list: the lab downloads one or two of theseRead the table against Task 4: the Qwen3-8B Q4_K_M row is your 5.03 + 1.21 = 6.24. The < marks
the gpt-oss rows, whose cache figure is an upper bound because half their layers use a sliding
window. Now run it at a longer context:
RunnableAll tracks
cd ~/llm-coursepython3 fit-models.py --memory-gb 16 --context 32768Output — what you should see
memory 16.0 GB, reserve 1.5 GB, context 32768 tokens, KV at 16-bit...Qwen3-8B q4_k_m 5.03 4.83 9.86 yesQwen3-14B q8_0 15.70 5.37 21.07 noQwen3-14B q6_k 12.12 5.37 17.49 noQwen3-14B q5_k_m 10.51 5.37 15.88 noQwen3-14B q4_k_m 9.00 5.37 14.37 yes...shortlist (heaviest quantisation that fits, per model): ... Qwen3-14B q4_k_m 9.00 GB unsloth/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf ...The weights column is identical and the cache column is four times larger, which pushed Qwen3-14B from Q6_K down to Q4_K_M: context is a purchase, not a setting. If a model you want falls out at your preferred context, try an 8-bit cache, which halves the cache column; Part 6 measures what an 8-bit cache costs in quality.
RunnableAll tracks
cd ~/llm-coursepython3 fit-models.py --memory-gb 16 --context 32768 --kv-bits 8Output — what you should see
...shortlist (heaviest quantisation that fits, per model): ... Qwen3-14B q5_k_m 10.51 GB unsloth/Qwen3-14B-GGUF Qwen3-14B-Q5_K_M.gguf ...The --explain option prints the working for one model, which is the check that the script agrees
with your Task 4 arithmetic:
RunnableAll tracks
cd ~/llm-coursepython3 fit-models.py --memory-gb 16 --context 8192 --explain qwen3-8bOutput — what you should see
Qwen3-8B (Qwen/Qwen3-8B; GGUF files from unsloth/Qwen3-8B-GGUF) config.json: num_hidden_layers 36, num_key_value_heads 8, head_dim 128 key-value cache per token = 2 x 36 x 8 x 128 x 2 bytes = 147,456 bytes x 4096 tokens = 0.60 GB x 8192 tokens = 1.21 GB x 32768 tokens = 4.83 GB weights: q8_0 8,709,519,168 bytes = 8.71 GB published (8.50 bits per weight over 8.2 B parameters) q6_k 6,725,900,096 bytes = 6.73 GB published (6.56 bits per weight over 8.2 B parameters) q5_k_m 5,851,113,280 bytes = 5.85 GB published (5.71 bits per weight over 8.2 B parameters) q4_k_m 5,027,784,512 bytes = 5.03 GB published (4.91 bits per weight over 8.2 B parameters) budget: 16.0 GB - 1.5 GB reserve = 14.50 GB q8_0 at 8192 tokens: 8.71 + 1.21 = 9.92 GB fits q6_k at 8192 tokens: 6.73 + 1.21 = 7.93 GB fits q5_k_m at 8192 tokens: 5.85 + 1.21 = 7.06 GB fits q4_k_m at 8192 tokens: 5.03 + 1.21 = 6.24 GB fitsFinally, run it at the context you have decided on, with --labbook, so the result is recorded,
using your own figure from Task 3 (and --reserve-gb 1.5 on Track N).
RunnableAll tracks
cd ~/llm-coursepython3 fit-models.py --memory-gb 16 --context 8192 --labbook labbook.mdOutput — what you should see
...recorded in labbook.mdWhat to expect per tier, as the heaviest quantisation of each model that fits, taken from real runs of the script with its default reserves. These are the rows to compare your output against; a Track X or Track N reader with a smaller effective figure lands one row up.
| Memory figure | At 8k context | At 32k context |
|---|---|---|
| 8 | Qwen3-4B Q8_0; Qwen3-8B Q4_K_M | Qwen3-1.7B Q8_0 only (Qwen3-4B Q6_K with an 8-bit cache) |
| 16 | Qwen3-14B Q6_K; gpt-oss-20b | Qwen3-14B Q4_K_M; gpt-oss-20b |
| 24 | Qwen3-14B Q8_0; Qwen3-30B-A3B Q4_K_M | Qwen3-14B Q6_K; gpt-oss-20b |
| 32 | Qwen3-32B Q5_K_M; Qwen3-30B-A3B Q6_K | Qwen3-32B Q4_K_M; Qwen3-30B-A3B Q6_K |
| 64 | Qwen3-32B Q8_0; Qwen3-30B-A3B Q8_0 | the same |
| 96 and 128 | everything, including gpt-oss-120b MXFP4 | the same |
The script wrote the record for this task as one JSON line: memory_gb, reserve_gb, budget_gb,
context, kv_bits and the shortlist rows.
6. Choose your shortlist
Section titled “6. Choose your shortlist”The script prints a menu, not a shopping list. Choose three things from it, each by a rule with a measurable criterion:
| Role | Rule | Criterion | Example on the 16 GB, 8k run |
|---|---|---|---|
| The model to work with now | the largest model that fits at the context you actually use, at the heaviest quantisation that fits | fit is yes at your --context; total leaves at least 1 GB of the budget unused |
Qwen3-14B Q6_K (13.46 of 14.50 GB) or Qwen3-8B Q8_0 if you want more room |
| The small model | Qwen3-1.7B or Qwen3-4B, whichever your tier affords at Q8_0 | loads in seconds; total under a quarter of the budget | Qwen3-1.7B Q8_0 (2.77 GB) |
| The other architecture | if your first choice is dense, a mixture-of-experts model that fits; if it is a mixture of experts, a dense one | fit is yes; different architecture in the model reference |
gpt-oss-20b MXFP4 (12.51 GB; 3.6 B active of 21 B) |
The small model is the one you test a script, a chat template or an engine flag with, and it is the student in Part 15’s distillation labs. The other-architecture model is what makes Part 6’s benchmark lab informative: the architecture lesson predicted that a mixture-of-experts model decodes faster than a dense model of the same file size, and Part 6 is where you measure it. On an 8 GB machine no mixture-of-experts model fits; record that as the finding.
Write the shortlist in the notebook as three lines, each with the model, the quantisation, the repository, the file name, the total the script printed, and the reason.
7. Create the library
Section titled “7. Create the library”Download the README below into ~/llm-course/ beside the scripts, then create the tree and put the
README at its top.
Fragment — not complete on its own
# Model library
This directory is the single place model weights live on this machine. Every engine in theLocal LLM course is pointed at it, so a model is downloaded once and used by llama.cpp,Ollama, LM Studio, vLLM, SGLang and mlx-lm alike.
Fill in the machine line below and keep this file with the library.
- **Machine:**- **Platform track:** S / X / M / N- **Memory available to models (GB), and whether that is unified memory or VRAM:**- **Library created:**
## Layout
```~/models/ README.md this file <publisher>/ the Hugging Face namespace, spelled as the Hub spells it <model>/ the repository name, spelled as the Hub spells it <file> the weights <file>.sha256 the checksum verified at download time .cache/huggingface/ hf's own record of what it downloaded; leave it alone hf/ reserved: Part 5 points the Hugging Face cache (HF_HOME) here```
For example:
```~/models/ unsloth/ Qwen3-8B-GGUF/ Qwen3-8B-Q4_K_M.gguf Qwen3-8B-Q4_K_M.gguf.sha256 ggml-org/ gpt-oss-20b-GGUF/ gpt-oss-20b-MXFP4.gguf gpt-oss-20b-MXFP4.gguf.sha256```
Two rules keep the library usable a year from now. Keep the namespace, because`Qwen3-8B-GGUF` under `unsloth/` and under `Qwen/` are different conversions, with differentfiles, checksums and cards. And keep the file name unchanged, because it names thequantisation, which is the thing you will need to know when a benchmark number in the labnotebook does not match a later run.
The `.sha256` file is one line in the format `sha256sum --check` reads, the hash, two spaces,the file name, so any machine can re-check the file without the course's scripts.
## Adding a model
Use the lab's script, which looks up the checksum the Hub publishes, downloads with the `hf`CLI, verifies the file and writes the `.sha256` beside it:
```shbash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf ~/llm-course/labbook.md```
Re-running the same command is how you recover from an interrupted download: a file `hf`has recorded as complete is not downloaded again, one it has not is, and the checksum ischecked on every run. A file that fails the check is moved to `<file>.corrupt` togetherwith `hf`'s record of it, so that the next run downloads it afresh.
## Verifying the library
To re-check everything, for example after a disk problem or an unclean shutdown, run thelab's audit script, which reads every `.sha256` under the library, prints one line per fileand exits non-zero if any file has changed:
```shbash verify-library.sh ~/llm-course/labbook.md```
The same check without the script, on Linux and in WSL2:
```shfind ~/models -name '*.sha256' -execdir sha256sum --check {} \;```
On macOS, `shasum -a 256 --check` does the same job. The `hf` CLI can also check onerepository's directory against the Hub directly:
```shhf cache verify unsloth/Qwen3-8B-GGUF --local-dir ~/models/unsloth/Qwen3-8B-GGUF```
## Licences
Downloading weights does not change the licence they came under. Record the licence foreach model here as you add it, so that the answer is on the same machine as the files. Thecourse reference set, as recorded in the course model reference:
| Model | Licence | Gated | Notes || --- | --- | --- | --- || Qwen3 (all sizes) | Apache-2.0 | no | GGUF from unsloth, a third-party conversion; Qwen also publishes its own || gpt-oss-20b, gpt-oss-120b | Apache-2.0 | no | GGUF from ggml-org, the llama.cpp organisation || Llama 3.1 8B Instruct | Llama 3.1 Community License | yes (meta-llama) | GGUF is a community conversion (bartowski); accept the licence on the meta-llama page first || | | | |
Apache-2.0 and MIT models need no further thought. Anything else, a company-named licence,a Llama community licence, Gemma terms, has conditions, and the course's licence lesson inPart 3 covers what they are.
## Pointing tools at this directory
Later parts set these up properly. The short version:
- **llama.cpp** takes a path to the `.gguf` file, so nothing needs configuring.- **Ollama** keeps its own store by default; Part 7 covers importing a GGUF from here instead of downloading it a second time.- **LM Studio** expects `publisher/model/file` under its own models directory; Part 7 links it to this tree.- **Transformers, vLLM and SGLang** read the Hugging Face cache rather than this tree; Part 5 sets `HF_HOME` to `~/models/hf` so that cache lives on the same disk.
## What not to put here
Fine-tuned adapters, quantisations you made yourself and merged models belong in a separatetree, because this one is meant to hold files whose checksums match a published original.Part 13 onwards puts your own artefacts under `~/models-local/`.RunnableAll tracks
mkdir -p ~/modelscp ~/llm-course/library-readme.md ~/models/README.mdls -la ~/modelsOutput — what you should see
total xdrwxr-xr-x 2 you you 4096 ... .drwxr-x--- 20 you you 4096 ... ..-rw-r--r-- 1 you you xxxx ... README.mdOpen ~/models/README.md and fill in the four machine lines: the machine, the track, the memory
figure from Task 3 with its kind, and today’s date. The download script in the next task copies
the README for you if it is missing, but it cannot fill in the machine line.
What each rule of the layout costs when it is broken: without the namespace in the path
(~/models/qwen3-8b/ instead of ~/models/unsloth/Qwen3-8B-GGUF/), six months on there is no
record of which conversion you have. Without the Hub’s file name, the quantisation is gone from the
path and a benchmark disagreement cannot be traced. Without the .sha256 beside the file, the only
check left is the CLI’s own record, which Task 8 shows is not a check of the bytes. And do not put
weights in hf/ by hand; it is where Part 5 moves the Hugging Face cache.
8. Download and verify your first model
Section titled “8. Download and verify your first model”First, ask the CLI what a download would do, without doing it. --dry-run is documented to list
the files that would be downloaded and their sizes, and to show which are already present:
RunnableAll tracks
hf download unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf \ --local-dir ~/models/unsloth/Qwen3-8B-GGUF --dry-runOutput — what you should see
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.[dry-run] Will download 1 files (out of 1) totalling 5.0G.FILE SIZE-------------------- ----Qwen3-8B-Q4_K_M.gguf 5.0GThe line to check is [dry-run] Will download 1 files (out of 1) totalling 5.0G.: the size agrees
with the table, in the CLI’s own rounding. The table under it differs between releases (the
documentation heads its columns File and Bytes to download), so do not compare it character for
character. Now the real thing. The script below looks up the checksum the Hub publishes for the
file, refuses to start if the disk cannot hold it, downloads with hf download, checks the size and
the SHA-256 against what the Hub said, writes a .sha256 beside the file, and appends one JSON line
to the notebook. Read it before running it; it is the script you will use for every model in the
course.
RunnableAll tracks
#!/usr/bin/env bash# Purpose: download one model file from the Hugging Face Hub with the hf CLI, verify its# SHA-256 against the checksum the Hub publishes for the file, and file it in a# shared model library at ~/models/<publisher>/<model>/ that every engine in this# course can point at# Platform: all (Linux, macOS and WSL2; the hf CLI is identical on each)# Minimum memory: 8 GB# Assumes: hf, curl and python3 are on PATH; a sha256 tool (sha256sum or shasum) exists;# enough free disk for the file; HF_TOKEN is set or `hf auth login` has been run# if the repository is gated## Usage: bash fetch-model.sh <repo-id> <filename> [labbook.md]# e.g. bash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf labbook.md## Environment:# MODELS_DIR where the library lives (default: $HOME/models)# HF_REVISION branch, tag or commit to download (default: main)## Re-running the script is safe and is the way to recover from an interruption: hf skips a# file it has already recorded as complete and downloads one it has not, and this script# re-verifies the checksum on every run. A file that fails the size or checksum test is# moved aside together with hf's record of it, so the next run downloads it again.
set -euo pipefail
REPO="${1:-}"FILE="${2:-}"LABBOOK="${3:-}"MODELS_DIR="${MODELS_DIR:-$HOME/models}"HF_REVISION="${HF_REVISION:-main}"API="https://huggingface.co/api/models"
die() { echo "fetch-model: $*" >&2; exit 1; }
[ -n "$REPO" ] && [ -n "$FILE" ] || die "usage: bash fetch-model.sh <repo-id> <filename> [labbook.md]"case "$REPO" in */*) : ;; *) die "repo id must be <publisher>/<model>, got '$REPO'" ;;esac
for tool in hf curl python3; do command -v "$tool" >/dev/null || die "$tool is not installed or not on PATH"done
if command -v sha256sum >/dev/null; then sha256_of() { sha256sum "$1" | cut -d' ' -f1; }elif command -v shasum >/dev/null; then sha256_of() { shasum -a 256 "$1" | cut -d' ' -f1; }else die "no sha256 tool found (install coreutils for sha256sum, or use macOS shasum)"fi
PUBLISHER="${REPO%%/*}"MODEL="${REPO##*/}"DEST="$MODELS_DIR/$PUBLISHER/$MODEL"
# --- 1. Ask the Hub what the file should be -----------------------------------------# The repository tree endpoint reports, for every Git LFS file, an "lfs" object whose# "oid" is the file's SHA-256 and whose "size" is its length in bytes.echo "==> Looking up $FILE in $REPO@$HF_REVISION"AUTH_ARGS=()if [ -n "${HF_TOKEN:-}" ]; then AUTH_ARGS=(-H "Authorization: Bearer $HF_TOKEN")fi
TREE_JSON="$(curl -fsSL ${AUTH_ARGS[@]+"${AUTH_ARGS[@]}"} "$API/$REPO/tree/$HF_REVISION" 2>/dev/null || true)"EXPECTED=""if [ -n "$TREE_JSON" ]; then EXPECTED="$(printf '%s' "$TREE_JSON" | python3 -c 'import json, syswant = sys.argv[1]try: entries = json.load(sys.stdin)except json.JSONDecodeError: sys.exit(0)for entry in entries: if entry.get("path") == want: lfs = entry.get("lfs") or {} if lfs.get("oid"): print(lfs["oid"], lfs.get("size", entry.get("size", 0))) break' "$FILE")"fi
# Fallback: the Git LFS pointer served at /raw/ carries the same SHA-256 and size.if [ -z "$EXPECTED" ]; then echo " tree listing gave nothing; falling back to the LFS pointer" POINTER="$(curl -fsSL ${AUTH_ARGS[@]+"${AUTH_ARGS[@]}"} "https://huggingface.co/$REPO/raw/$HF_REVISION/$FILE" || true)" OID="$(printf '%s\n' "$POINTER" | sed -n 's/^oid sha256:\([0-9a-f]\{64\}\)$/\1/p')" SIZE="$(printf '%s\n' "$POINTER" | sed -n 's/^size \([0-9]\{1,\}\)$/\1/p')" [ -n "$OID" ] && EXPECTED="$OID $SIZE"fi
[ -n "$EXPECTED" ] || die "could not find a published SHA-256 for $FILE in $REPO (file names are case-sensitive: copy the name from the repository's file list; a private repository needs HF_TOKEN)"EXPECTED_SHA="${EXPECTED%% *}"EXPECTED_SIZE="${EXPECTED##* }"echo " expected sha256 $EXPECTED_SHA"echo " expected size $EXPECTED_SIZE bytes"
# --- 2. Prepare the library ----------------------------------------------------------mkdir -p "$DEST"TARGET="$DEST/$FILE"# hf keeps its record of each download here; removing it makes hf download the file again.METADATA="$DEST/.cache/huggingface/download/$FILE.metadata"
# Refuse to start a download the disk cannot hold. df -Pk is POSIX, so the column is the# same on Linux and macOS: available space in KiB.HAVE_BYTES=0[ -f "$TARGET" ] && HAVE_BYTES="$(python3 -c 'import os,sys; print(os.path.getsize(sys.argv[1]))' "$TARGET")"NEED_KB=$(( (EXPECTED_SIZE - HAVE_BYTES) / 1024 ))AVAIL_KB="$(df -Pk "$DEST" | awk 'NR == 2 { print $4 }')"if [ "$NEED_KB" -gt 0 ] && [ "$AVAIL_KB" -lt "$NEED_KB" ]; then die "not enough free space on $DEST: need $(( NEED_KB / 1024 )) MiB more, $(( AVAIL_KB / 1024 )) MiB free. Free some space or set MODELS_DIR to another disk."fiSCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"if [ ! -f "$MODELS_DIR/README.md" ]; then if [ -f "$SCRIPT_DIR/library-readme.md" ]; then cp "$SCRIPT_DIR/library-readme.md" "$MODELS_DIR/README.md" else printf '# Model library\n\nLayout: <publisher>/<model>/<file>. Each file has a .sha256 beside it.\n' \ > "$MODELS_DIR/README.md" fi echo "==> Wrote $MODELS_DIR/README.md"fi
# --- 3. Download ---------------------------------------------------------------------echo "==> Downloading into $DEST (safe to interrupt and re-run)"hf download "$REPO" "$FILE" --local-dir "$DEST" --revision "$HF_REVISION"
[ -f "$TARGET" ] || die "hf reported success but $TARGET does not exist"
# --- 4. Verify -----------------------------------------------------------------------# hf decides whether a file is complete from its own metadata record, not from the bytes# on disk, so a truncated or altered file with a record beside it would be kept forever.# Anything that fails here is moved aside and its record removed, so a re-run downloads it.set_aside() { mv "$TARGET" "$TARGET.corrupt" rm -f "$METADATA"}
ACTUAL_SIZE="$(python3 -c 'import os,sys; print(os.path.getsize(sys.argv[1]))' "$TARGET")"if [ "$ACTUAL_SIZE" != "$EXPECTED_SIZE" ]; then set_aside die "size mismatch: got $ACTUAL_SIZE bytes, expected $EXPECTED_SIZE. The file has been moved to $TARGET.corrupt; delete it and re-run to download it again."fi
[ -f "$TARGET.sha256" ] && echo " a previous run verified this file; verifying it again"echo "==> Verifying SHA-256 (this reads the whole file; expect a minute or two on a large one)"ACTUAL_SHA="$(sha256_of "$TARGET")"if [ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]; then set_aside die "checksum mismatch. Got $ACTUAL_SHA, expected $EXPECTED_SHA. The file has been moved to $TARGET.corrupt; delete it and re-run to download it again."fiprintf '%s %s\n' "$ACTUAL_SHA" "$FILE" > "$TARGET.sha256"echo " verified, checksum written to $TARGET.sha256"
# --- 5. Record -----------------------------------------------------------------------if [ -n "$LABBOOK" ]; then # In a terminal `hf version` prints "✓ hf version" and " version: x.y.z"; the agent # format is one "version=x.y.z" line. Ask for the agent format so the parse is stable. HF_VERSION="$(hf version --format agent 2>/dev/null | sed -n 's/^version=//p' | head -n 1)" python3 -c 'import json, sysfrom datetime import dateprint(json.dumps({ "lab": "part-04/fetch-model", "date": date.today().isoformat(), "hf_version": sys.argv[7] or "unknown", "repo": sys.argv[1], "revision": sys.argv[2], "file": sys.argv[3], "path": sys.argv[4], "bytes": int(sys.argv[5]), "gb": round(int(sys.argv[5]) / 1e9, 2), "sha256": sys.argv[6], "verified": True,}))' "$REPO" "$HF_REVISION" "$FILE" "$TARGET" "$ACTUAL_SIZE" "$ACTUAL_SHA" "$HF_VERSION" >> "$LABBOOK" echo " recorded in $LABBOOK"fi
echo "==> Done: $TARGET"RunnableAll tracks
cd ~/llm-coursebash fetch-model.sh unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf labbook.mdSubstitute the repository and file name from your own shortlist; the fit script printed both for
every row. For the Qwen3-8B example the first four lines are exact, because the checksum and size
are what the Hub publishes for that file, and the hash step takes about as long as reading the file
once. The ✓ Downloaded and path: lines are what version 1.30.0 prints in a terminal; the
documentation’s own hf download example shows a bare path, and in every form the last line names
the path.
Output — what you should see
==> Looking up Qwen3-8B-Q4_K_M.gguf in unsloth/Qwen3-8B-GGUF@main expected sha256 120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4 expected size 5027784512 bytes==> Downloading into /home/you/models/unsloth/Qwen3-8B-GGUF (safe to interrupt and re-run)Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads....progress...✓ Downloaded path: /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf==> Verifying SHA-256 (this reads the whole file; expect a minute or two on a large one) verified, checksum written to /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf.sha256 recorded in labbook.md==> Done: /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufWhere the expected checksum comes from. Large files on the Hub are stored through Git LFS, and
the Hub publishes each one’s SHA-256 in two places you can read without downloading anything. The
repository tree endpoint, GET /api/models/{namespace}/{repo}/tree/{rev}/{path} in the Hub’s
OpenAPI specification, returns one entry per file, and for an LFS file an lfs object whose oid
is the SHA-256 and whose size is the length in bytes; the script reads that first and falls back
to the Git LFS pointer served at /raw/, three lines carrying the same two values. Both are
readable with curl, so you can check the script’s inputs by hand:
RunnableAll tracks
curl -fsSL https://huggingface.co/unsloth/Qwen3-8B-GGUF/raw/main/Qwen3-8B-Q4_K_M.ggufOutput — what you should see
version https://git-lfs.github.com/spec/v1oid sha256:120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4size 5027784512Why the script hashes the file itself. hf download into a --local-dir keeps a record of
each file under .cache/huggingface/download/<file>.metadata: three lines, the commit hash, the
file’s etag (the same SHA-256 for an LFS file) and a timestamp. The documentation says this
“prevents re-downloading files if they’re already up-to-date”. The source of version 1.30.0 shows
how that decision is made, and it is worth knowing: the record is trusted as long as the file
“hasn’t been modified since the metadata was saved”, judged by the file’s modification time with a
one-second allowance; only if the file is newer than the record does the CLI “re-compute hash”. A
file whose hash matches is kept and its record rewritten, and a file that is missing, fails that
re-hash, or whose recorded etag no longer matches the Hub’s is deleted and downloaded afresh. What
is never re-checked is a file whose modification time is not newer than its record: one that was
truncated or altered without its modification time moving forward is kept as complete. The
independent hash in the script, and the .sha256 it leaves behind, are what turn “the CLI says it
is there” into “the bytes are the bytes the Hub published”.
The CLI has its own verification command, hf cache verify, which reads a local directory and
compares every file it recognises against the Hub. Run it on the directory you just filled:
RunnableAll tracks
hf cache verify unsloth/Qwen3-8B-GGUF --local-dir ~/models/unsloth/Qwen3-8B-GGUFOutput — what you should see
Warning: x remote file(s) are missing locally. Use --fail-on-missing-files for details.Warning: x local file(s) do not exist on the remote repo. Use --fail-on-extra-files for details.✓ Verified 1 file(s) for model 'unsloth/Qwen3-8B-GGUF'. All checksums match. repo_id: unsloth/Qwen3-8B-GGUF repo_type: model checked: 1 path: /home/you/models/unsloth/Qwen3-8B-GGUFThe two warnings are expected: the repository has other files you did not download, and your
directory has a .sha256 file the repository does not. The line that matters is All checksums match., with exit status 0; in the agent format the same result is one repo_id=… checked=1 …
line, and the documentation’s example words it differently, with a ✅. The wording moves between
releases and formats; the exit status and the absence of a failure line do not. On a mismatch, in
either format, the command prints the file with the expected and actual hashes and exits with
status 1:
Output — what you should see
❌ Checksum verification failed for the following file(s): - Qwen3-8B-Q4_K_M.gguf: expected 120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4 (sha256), got xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxWarning: x remote file(s) are missing locally. Use --fail-on-missing-files for details.Warning: x local file(s) do not exist on the remote repo. Use --fail-on-extra-files for details.Error: Verification failed for 'unsloth/Qwen3-8B-GGUF' (model) in /home/you/models/unsloth/Qwen3-8B-GGUF. Revision: mainThat block was observed with version 1.30.0 by altering one byte of a verified file; the ❌ line
and the Error: line are the two to grep for in a script.
Finally, confirm one checksum with a tool that is not the course’s, so you know the script is not
marking its own homework. The value to expect is the oid from the pointer above.
Track S — NVIDIA DGX Spark
RunnableTrack S · DGX Spark
sha256sum ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufTrack X — AMD Ryzen AI Max+ 395
RunnableTrack X · Ryzen AI Max+
sha256sum ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufTrack M — Apple silicon
RunnableTrack M · Apple silicon
shasum -a 256 ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufTrack N — NVIDIA desktop or laptop
RunnableTrack N · NVIDIA GPU
sha256sum ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufOutput — what you should see
120307ba529eb2439d6c430d94104dabd578497bc7bfe7e322b5d9933b449bd4 /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.ggufRecord, from the JSON line the script appended: repo, file, bytes, sha256, hf_version
and date. Add in prose that hf cache verify and your own sha256sum agreed with it.
9. Prove that re-running is safe
Section titled “9. Prove that re-running is safe”The claim to test: running the identical fetch-model.sh command a second time never harms a
complete file and always leaves you with a verified one. Test it on the largest file in your
shortlist that you intend to keep, so that the interrupted download is not wasted.
Start it, and while the progress is showing, press Ctrl-C:
RunnableAll tracks
cd ~/llm-coursebash fetch-model.sh unsloth/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf labbook.mdThe interrupted command ends with Aborted! or, depending on where in the transfer the interrupt
lands, with a Python traceback whose last line is RuntimeError: Task error: File reconstruction error: Internal Writer Error: Background writer channel closed; both were observed with version
1.30.0 and hf_xet 1.6.0, and the wording may differ by version. The script stops with the download.
Then look at what the interruption left behind, and run exactly the same command again:
RunnableAll tracks
ls -la ~/models/unsloth/Qwen3-14B-GGUF/ ~/models/unsloth/Qwen3-14B-GGUF/.cache/huggingface/download/bash fetch-model.sh unsloth/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf labbook.mdOutput — what you should see
/home/you/models/unsloth/Qwen3-14B-GGUF/:total xdrwxr-xr-x 3 you you 4096 ... .drwxr-xr-x 4 you you 4096 ... ..drwxr-xr-x 3 you you 4096 ... .cache
/home/you/models/unsloth/Qwen3-14B-GGUF/.cache/huggingface/download/:total x-rw-r--r-- 1 you you 0 ... Qwen3-14B-Q4_K_M.gguf.lock==> Looking up Qwen3-14B-Q4_K_M.gguf in unsloth/Qwen3-14B-GGUF@main ...==> Downloading into /home/you/models/unsloth/Qwen3-14B-GGUF (safe to interrupt and re-run)...progress...✓ Downloaded path: /home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf==> Verifying SHA-256 (this reads the whole file; expect a minute or two on a large one) verified, checksum written to ...==> Done: /home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.ggufWhat you should observe, and why. No .gguf and no partial file: the repository directory holds
only .cache/huggingface/, and under it a .gitignore, a CACHEDIR.TAG and the empty
Qwen3-14B-Q4_K_M.gguf.lock. While the transfer runs, the target file does not exist yet; the bytes
go to a process-unique file under .cache/huggingface/download/ whose name is a short hash of the
metadata file name, the SHA-256, eight hexadecimal characters and .incomplete, and the final name
appears only when the transfer is complete. The version 1.30.0 source of
_download_to_tmp_and_move in file_download.py deletes that file in a finally: block on any
failure, including Ctrl-C, with the comment that a partial file “could not be reused anyway since
the temporary name is unique to this download”. So at this version a re-run starts the transfer
again from byte zero: the safety of re-running comes from the unique temporary file, the rename
into place only after the transfer completes, and the metadata record, not from resuming bytes. A
.incomplete file survives only a hard kill or a power loss; that is the case the documentation
describes as “partial downloads left behind when a download is interrupted”, and the one hf cache prune cleans up in the cache. The second run therefore downloads the whole file again and
verifies it.
Then run the command a third time, with the file complete:
RunnableAll tracks
cd ~/llm-coursebash fetch-model.sh unsloth/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf labbook.mdOutput — what you should see
==> Looking up Qwen3-14B-Q4_K_M.gguf in unsloth/Qwen3-14B-GGUF@main ...==> Downloading into /home/you/models/unsloth/Qwen3-14B-GGUF (safe to interrupt and re-run)✓ Downloaded path: /home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf a previous run verified this file; verifying it again==> Verifying SHA-256 (this reads the whole file; expect a minute or two on a large one) verified, checksum written to ...==> Done: /home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.ggufThe download step returns in about a second, because the metadata record says the file is
complete; the verification step still reads the whole file, which is the point. Record: the file you
interrupted, what ls showed afterwards, and that the second and third runs both ended in
verified.
10. Audit the whole library
Section titled “10. Audit the whole library”Two files in, the library needs an audit tool, because the day you need one is the day after a
disk problem. The script below walks ~/models/, checks every file against the .sha256 beside
it with sha256sum or shasum, prints one line per file and a summary, exits non-zero on any
failure, and records the audit in the notebook.
RunnableAll tracks
#!/usr/bin/env bash# Purpose: re-check every model file in the library against the .sha256 that fetch-model.sh# wrote beside it at download time, print one line per file and a summary, exit# non-zero if anything failed, and optionally record the audit in the lab notebook# Platform: all (Linux, macOS and WSL2; uses sha256sum where present and shasum otherwise)# Minimum memory: 8 GB# Assumes: the library at $MODELS_DIR (default ~/models) was filled by fetch-model.sh, so# every weights file has a <file>.sha256 beside it; python3 is on PATH for the# notebook line## Usage: bash verify-library.sh [labbook.md]## Environment:# MODELS_DIR where the library lives (default: $HOME/models)## Reading every file takes about as long as copying it, so a 60 GB library takes minutes,# not seconds. Nothing is downloaded and nothing is changed.
set -euo pipefail
MODELS_DIR="${MODELS_DIR:-$HOME/models}"LABBOOK="${1:-}"
die() { echo "verify-library: $*" >&2; exit 1; }
[ -d "$MODELS_DIR" ] || die "$MODELS_DIR does not exist; run fetch-model.sh first"
# sha256sum --check reads "<hash> <file>" lines, which is exactly what fetch-model.sh# writes; shasum on macOS reads the same format.if command -v sha256sum >/dev/null; then check_sums() { sha256sum --check "$1"; }elif command -v shasum >/dev/null; then check_sums() { shasum -a 256 --check "$1"; }else die "no sha256 tool found (install coreutils for sha256sum, or use macOS shasum)"fi
ok=0failed=0missing=0bytes=0
echo "==> Verifying every *.sha256 under $MODELS_DIR"while IFS= read -r -d '' sums; do dir="$(dirname "$sums")" file="$(basename "${sums%.sha256}")" if [ ! -f "$dir/$file" ]; then echo "MISSING $dir/$file (checksum file present, weights file gone)" missing=$((missing + 1)) continue fi if (cd "$dir" && check_sums "$file.sha256" >/dev/null 2>&1); then echo "OK $dir/$file" ok=$((ok + 1)) size="$(wc -c < "$dir/$file" | tr -d ' ')" bytes=$((bytes + size)) else echo "FAILED $dir/$file (contents do not match the recorded SHA-256)" failed=$((failed + 1)) fidone < <(find "$MODELS_DIR" -type f -name '*.sha256' -print0)
total=$((ok + failed + missing))[ "$total" -gt 0 ] || die "no *.sha256 files under $MODELS_DIR; nothing to verify"
echo "==> $ok ok, $failed failed, $missing missing, of $total recorded file(s); $bytes bytes verified"
if [ -n "$LABBOOK" ] && command -v python3 >/dev/null; then python3 -c 'import json, sysfrom datetime import dateprint(json.dumps({ "lab": "part-04/verify-library", "date": date.today().isoformat(), "models_dir": sys.argv[1], "ok": int(sys.argv[2]), "failed": int(sys.argv[3]), "missing": int(sys.argv[4]), "bytes_verified": int(sys.argv[5]),}))' "$MODELS_DIR" "$ok" "$failed" "$missing" "$bytes" >> "$LABBOOK" echo " recorded in $LABBOOK"fi
[ "$failed" -eq 0 ] || exit 1RunnableAll tracks
cd ~/llm-coursebash verify-library.sh labbook.mdOutput — what you should see
==> Verifying every *.sha256 under /home/you/modelsOK /home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.ggufOK /home/you/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf==> 2 ok, 0 failed, 0 missing, of 2 recorded file(s); 14029538496 bytes verified recorded in labbook.mdA FAILED line means the bytes no longer match the checksum written at download time: delete the
file and re-run fetch-model.sh for it. A MISSING line means the .sha256 outlived its file,
which is what a half-finished manual clean-up looks like. The byte total is the sum of the two
published sizes, 5,027,784,512 + 9,001,753,984, one more check that the files are whole. Record:
ok, failed, missing and bytes_verified from the JSON line.
11. Optional: the gated comparison model
Section titled “11. Optional: the gated comparison model”Llama 3.1 8B Instruct is the course’s cross-engine comparison model, and this step is where the
course first meets a gate. Two facts to hold apart, both checked against the Hub’s API on 12
September 2026. The publisher’s repository, meta-llama/Llama-3.1-8B-Instruct, is gated with
manual approval, and it is under the Llama 3.1 Community License rather than Apache-2.0, with the
attribution, naming and large-user-base conditions that the licence
lesson
described. The GGUF conversion the course uses, bartowski/Meta-Llama-3.1-8B-Instruct-GGUF, is
not gated and downloads without an account, but its card carries the same licence text, and
converting the weights did not change the terms you use them under.
So the honest order is: accept the licence where the licence lives, prove that the gate opened,
then take the conversion. The documentation says requesting access “can only be done from your
browser”: open the meta-llama/Llama-3.1-8B-Instruct model page, read the licence, submit the
request, and wait; with manual approval that can take time. Then prove access with the smallest
file in the repository, which is also the file Task 4’s arithmetic wants:
RunnableAll tracks
hf auth whoamimkdir -p ~/llm-course/models/llama-3.1-8b-instructhf download meta-llama/Llama-3.1-8B-Instruct config.json \ --local-dir ~/llm-course/models/llama-3.1-8b-instructpython3 ~/llm-course/kv-from-config.py ~/llm-course/models/llama-3.1-8b-instruct/config.jsonOutput — what you should see
your-username✓ Downloaded path: /home/you/llm-course/models/llama-3.1-8b-instruct/config.jsonlayers=32 kv_heads=8 head_dim=128 (query heads=xx, ignored on purpose)bytes per token, FP16 cache: 131,072 8192 tokens: 1.07 GB at FP16, 0.54 GB at 8-bit 32768 tokens: 4.29 GB at FP16, 2.15 GB at 8-bitThe layer, key-value head and head-dimension values shown are what the course model reference
records for this model; the gated file itself was not readable by the author without accepting
the licence, so treat the output as the expected shape rather than a captured run. Before access is
granted, the same download fails with the exact message Error: Access denied. This repository requires approval., observed with version 1.31.0, and that is the message to expect if you skip
the browser step.
Now the conversion, with the lab’s script as for any other file:
RunnableAll tracks
cd ~/llm-coursebash fetch-model.sh bartowski/Meta-Llama-3.1-8B-Instruct-GGUF Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf labbook.mdOutput — what you should see
==> Looking up Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf in bartowski/Meta-Llama-3.1-8B-Instruct-GGUF@main expected sha256 7b064f5842bf... expected size 4920739232 bytes...==> Done: /home/you/models/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q4_K_M.ggufThe card is the level of disclosure to require before trusting a community conversion: it is
titled “Llamacpp imatrix Quantizations of Meta-Llama-3.1-8B-Instruct”, it names the llama.cpp
release used, b3472, and it links the original model. Record in the notebook that this file came
from a third party rather than from Meta, the converter, the llama.cpp release and the importance
matrix note from the card, and the licence. When a later benchmark disagrees with somebody
else’s, this is the first difference to check; Part 6’s GGUF
lesson explains what the
importance matrix changed.
12. Record the result
Section titled “12. Record the result”Your notebook should now carry four kinds of JSON line from this lab: one from the fit script, one
per download, one from the audit, and, if you did Task 11, a download line whose repo is a
third-party namespace.
RunnableAll tracks
cd ~/llm-coursegrep -c '"lab": "part-04' labbook.mdtail -n 3 labbook.mdOutput — what you should see
4{"lab": "part-04/fetch-model", "date": "2026-xx-xx", "hf_version": "1.3x.x", "repo": "unsloth/Qwen3-14B-GGUF", "revision": "main", "file": "Qwen3-14B-Q4_K_M.gguf", "path": "/home/you/models/unsloth/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf", "bytes": 9001753984, "gb": 9.0, "sha256": "5eaa0870bd81...", "verified": true}{"lab": "part-04/fetch-model", ...}{"lab": "part-04/verify-library", "date": "2026-xx-xx", "models_dir": "/home/you/models", "ok": 2, "failed": 0, "missing": 0, "bytes_verified": 14029538496}Above the JSON lines, write the prose that the lines cannot: the memory figure and its kind; the context length you budgeted for and why that one; the three shortlist rows and the reason for each; which row you downloaded first; and any row that fell out between 8k and 32k. Part 6 benchmarks every file in this library, and Part 16 measures what their quantisation cost.
Audit the shortlist as a decision document
Section titled “Audit the shortlist as a decision document”For each selected model, keep a completed row containing its immutable identity, licence, required serving path, weight estimate, cache estimate and intended task. Leave task quality unmeasured until you run the evaluation; a publisher score is a separate reported field.
Before the first download, compare the calculator’s memory total with your hand calculation. Check units and whether the available-memory value includes operating-system headroom. After downloading, verify the expected files and integrity record. Re-run the downloader as the lesson instructs and confirm it recognises completed artefacts rather than duplicating the library.
The final checkpoint is a library audit: every shortlisted file can be located, its recorded hash matches and its model card or licence record is retained. A checksum mismatch is a stop condition; do not proceed to loading an unexplained file. A candidate that does not fit can remain in the decision table as rejected, with the reason. That makes the shortlist useful when a future hardware upgrade changes capacity without requiring you to repeat the entire investigation.
Validation
Section titled “Validation”You are done when every row passes. Each check has a command and the output that means pass.
| Check | Command | Pass looks like |
|---|---|---|
| The arithmetic by hand agrees with the script | python3 fit-models.py --memory-gb <yours> --context 8192 --explain qwen3-1.7b |
2 x 28 x 8 x 128 x 2 bytes = 114,688 bytes, the figure Task 4 printed |
| The fit table has your budget | python3 fit-models.py --memory-gb <yours> --context 8192 |
budget after reserve: equals your memory minus your reserve |
| Context moves rows | the same at --context 32768 |
at least one row changed from yes to no, and you can name it |
| The library exists with its README | head -n 12 ~/models/README.md |
the four machine lines are filled in |
| A verified file is in place | ls ~/models/*/*/ |
at least one .gguf with a .sha256 beside it, under <namespace>/<repository>/ |
| The CLI agrees | hf cache verify <namespace>/<repo> --local-dir ~/models/<namespace>/<repo> |
exit status 0, All checksums match. in a terminal or checked=1 when run by an AI agent, and no line beginning ❌ or Error: |
| A re-run is harmless | bash fetch-model.sh <repo> <file> labbook.md on a complete file |
the download step returns immediately; verified is printed again |
| The audit passes | bash verify-library.sh labbook.md |
0 failed, 0 missing and exit status 0 |
| The notebook has the record | grep -c '"lab": "part-04' labbook.md |
3 or more |
Expected outcome
Section titled “Expected outcome”A model library at ~/models/ with a filled-in README, at least one verified model file under its
namespace and repository name with its checksum beside it, an audit that passes, and a lab notebook
holding Task 12’s JSON lines and prose. Every engine part from Part 6 onward starts from this
directory, and Part 6’s benchmark lab walks it.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Fix |
|---|---|---|
hf: command not found |
the CLI is in the course environment and the environment is not active, or the standalone installer’s PATH change has not reached this shell | cd ~/llm-course && source .venv/bin/activate; after the installer, open a new terminal |
version: 1.2x.x or older in preflight (version=1.2x.x inside an AI agent) |
an older huggingface_hub in the environment |
uv pip install -U huggingface_hub inside the environment; hf update for the other install routes |
Error: Not logged in from hf auth whoami |
no stored token | expected for public downloads; run hf auth login only for the gated step |
Error: Access denied. This repository requires approval. |
the repository is gated and your account has not been granted access, or you are not signed in | accept the terms on the model page in a browser, wait for approval, confirm with hf auth whoami, retry |
could not find a published SHA-256 for <file> |
the file name is wrong; names are case-sensitive | copy the name from the fit script’s output or the repository’s file list; curl -fsSL https://huggingface.co/<repo>/raw/main/<file> should print an oid sha256: line |
not enough free space on <dir> |
the script compared the file’s published size with df and refused |
free space, or set MODELS_DIR to a directory on a larger disk and keep using it |
size mismatch or checksum mismatch, file moved to .corrupt |
a file truncated or altered on disk after the CLI recorded it, or something between you and the Hub rewriting bytes | delete the .corrupt file and re-run the same command; the script also removed the CLI’s record, so the file is downloaded afresh. If it fails twice, suspect a proxy or filtering appliance |
httpx.TimeoutException ... Read timed out. (read timeout=10) |
a slow connection tripping the documented ten-second default | export HF_HUB_DOWNLOAD_TIMEOUT=30 and re-run, as the CLI guide advises |
Warning: You are sending unauthenticated requests to the HF Hub |
no token; anonymous requests are rate limited | harmless for one or two files; sign in if you download many |
| The download is much slower than your connection | the transfer layer is hf_xet, installed with huggingface_hub; a proxy in the middle is the usual culprit |
hf env shows the hf_xet version; the documentation offers HF_XET_HIGH_PERFORMANCE=1 to use all cores and bandwidth, and HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 for a spinning disk. hf_transfer is documented as deprecated; do not install it |
No space left on device partway through |
a download started before the preflight, or the disk filled from elsewhere | make room and re-run; if a .incomplete file was left by a hard kill, delete it under .cache/huggingface/download/; after a clean failure there is nothing to delete |
| A file manager shows 4.68 GiB for a “5.03 GB” file | the Hub reports 109 bytes and the file manager 230 | nothing is wrong; the script compares byte counts |
| Track X: the script says it fits but the model does not load, or loads and generates very slowly | the machine total was given instead of the GPU-visible figure | re-run with the smaller number; Part 5 reads and, where appropriate, raises the limit |
| Track M: a large model that fits by arithmetic fails to load | the wired-memory limit | Part 5’s Apple silicon lesson |
verify-library.sh prints MISSING |
a .sha256 whose file was deleted by hand |
delete the .sha256 too, or re-run fetch-model.sh for that file |
Cleanup
Section titled “Cleanup”Nothing here should be undone: the library is used by every remaining part of the course, and the
notebook lines are what Part 6 compares against. Keep ~/models/, its README, every .sha256, and
kv-from-config.py, fit-models.py, fetch-model.sh and verify-library.sh in ~/llm-course/.
If you downloaded a model you decided against, or a run left a .corrupt file behind, remove it and
its companions; nothing else in the tree depends on them.
If you experimented with hf download without --local-dir, the files went into the Hugging Face
cache under HF_HOME instead of the library. List and remove them with the CLI’s cache commands
rather than deleting cache directories by hand, so its index stays consistent; hf cache prune
also removes .incomplete files left by interrupted cache downloads.
RunnableAll tracks
hf cache lshf cache rm model/unsloth/Qwen3-8B-GGUFWhat you learned
Section titled “What you learned”- What fits is arithmetic, not opinion. Proved when
kv-from-config.pyprinted 114,688 bytes per token for Qwen3-1.7B,--explainprinted the same, and 5,027,784,512 bytes over 8.2 billion parameters came to 4.91 bits per weight. Recorded: layers, key-value heads, head dimension, bytes per token and the total at your context. - Context is a purchase. Proved when Qwen3-14B dropped from Q6_K at 8k to Q4_K_M at 32k with the weights column unchanged, and an 8-bit cache lifted it to Q5_K_M. Recorded: the fit script’s JSON line and the row that changed.
- The CLI’s record is not a check of the bytes. Proved when the script’s hash,
hf cache verifyandsha256sumall matched theoidthe Hub publishes, a check the CLI’s metadata alone does not make. Recorded:repo,file,bytes,sha256,hf_version,date. - Re-running is the recovery mechanism. Proved when Ctrl-C left no
.ggufand no partial file, the second run downloaded from the start and verified, and the third returned at once and verified again. Recorded: the file, thelslisting,verifiedon both runs. - A library is a decision you make once. Proved when
verify-library.shreported0 failed, 0 missingwith a byte total equal to the published sizes. Recorded:ok,failed,missing,bytes_verified. - Provenance is part of the model. Proved when the gate opened on the publisher’s repository and the public conversion named its converter, llama.cpp release and licence. Recorded: those three and the importance-matrix note.
Check your understanding
Sources for this lesson
22 verified · checked 2026-09-13
- 01Hugging Face Hub documentation — Command Line Interface (hf)§ Getting started; Output formatting; hf auth login; hf auth whoami; hf download; hf cache verify; hf cache prune; hf envhuggingface.co/docs/huggingface_hub/guides/cli2026-09-12
- 02Hugging Face Hub documentation — Download files from the Hub§ Download files to a local folder; Dry-run mode; Faster downloadshuggingface.co/docs/huggingface_hub/guides/download2026-09-12
- 03Hugging Face Hub documentation — Environment variables§ HF_HOME; HF_HUB_CACHE; HF_TOKEN; HF_HUB_DOWNLOAD_TIMEOUT; HF_HUB_DISABLE_XET; HF_XET_HIGH_PERFORMANCE; HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY; HF_HUB_ENABLE_HF_TRANSFER (deprecated)huggingface.co/docs/huggingface_hub/package_reference/environment_variables2026-09-12
- 04huggingface_hub v1.30.0 source — _local_folder.py, read_download_metadatagithub.com/huggingface/huggingface_hub/blob/v1.30.0/src/huggingface_hub/_local_folder.py2026-09-12
- 05huggingface_hub v1.30.0 source — file_download.py, local-directory up-to-date check and _download_to_tmp_and_movegithub.com/huggingface/huggingface_hub/blob/v1.30.0/src/huggingface_hub/file_download.py2026-09-12
- 06Hugging Face Hub documentation — Gated models§ Access gated models as a user; Download fileshuggingface.co/docs/hub/models-gated2026-09-12
- 07Hugging Face Hub OpenAPI specification (Markdown rendering)§ GET /api/models/{namespace}/{repo}/tree/{rev}/{path}huggingface.co/.well-known/openapi.md2026-09-12
- 08Hugging Face Hub documentation — GGUFhuggingface.co/docs/hub/gguf2026-09-08
- 09Qwen3-1.7B config.jsonhuggingface.co/Qwen/Qwen3-1.7B/raw/main/config.json2026-09-12
- 10unsloth/Qwen3-8B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-8B-GGUF/tree/main2026-09-13
- 11unsloth/Qwen3-4B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-4B-GGUF/tree/main2026-09-13
- 12unsloth/Qwen3-14B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-14B-GGUF/tree/main2026-09-13
- 13unsloth/Qwen3-30B-A3B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-30B-A3B-GGUF/tree/main2026-09-13
- 14unsloth/Qwen3-32B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-32B-GGUF/tree/main2026-09-13
- 15unsloth/Qwen3-1.7B-GGUF file listinghuggingface.co/api/models/unsloth/Qwen3-1.7B-GGUF/tree/main2026-09-13
- 16Qwen/Qwen3-8B-GGUF file listing (the publisher's own conversion)huggingface.co/api/models/Qwen/Qwen3-8B-GGUF/tree/main2026-09-13
- 17ggml-org/gpt-oss-20b-GGUF file listinghuggingface.co/api/models/ggml-org/gpt-oss-20b-GGUF/tree/main2026-09-12
- 18ggml-org/gpt-oss-120b-GGUF file listinghuggingface.co/api/models/ggml-org/gpt-oss-120b-GGUF/tree/main2026-09-12
- 19bartowski/Meta-Llama-3.1-8B-Instruct-GGUF file listinghuggingface.co/api/models/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main2026-09-12
- 20unsloth/Qwen3-8B-GGUF model cardhuggingface.co/unsloth/Qwen3-8B-GGUF2026-09-13
- 21bartowski/Meta-Llama-3.1-8B-Instruct-GGUF model cardhuggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF2026-09-12
- 22meta-llama/Llama-3.1-8B-Instruct model API record (gated field)huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct2026-09-12
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.