Model reference

The labs standardise on a small set of models so that results are comparable between readers and between platform tracks. The selection rules, in order: a permissive licence, so that no naming, user-count or non-commercial clause needs explaining in a lab, with three named exceptions stated in each entry’s notes; availability as safetensors, GGUF and MLX; documented tool calling; and a size that fills a memory tier without wasting it.

⌚Version note

Frontier open-weight releases move faster than any course can. The model-families lesson in Part 4 teaches the map and how to update it; the labs use the models below, which are stable, permissively licensed and available in every format the four tracks need. The set is reviewed twice a year, and a model is replaced only when its successor meets every rule above.

The memory-per-token column is the KV cache at FP16 for one token of context; multiply by your context length, halve it for an 8-bit cache, and add it to the weights to know whether a model fits. The memory-budget lesson in Part 4 works through the arithmetic.

25 models. Sizes are computed from bytes per parameter until the validation pass replaces them with measured file sizes; the date on each entry is when its card and licence were last read.

Model	Params (total / active, B)	Type	BF16 GB	Q8 GB	Q4 GB	KV per token (FP16)	Licence	Checked
Qwen3-0.6B	0.6 / 0.6	dense	1.2	0.6	0.4	112 KiB	Apache-2.0	2026-09-09
Qwen3-0.6B-Base	0.6 / 0.6	dense	1.2	—	—	112 KiB	Apache-2.0	2026-09-12
Qwen3-Embedding-0.6B	0.6 / 0.6	dense	1.2	0.7	—	112 KiB	Apache-2.0	2026-09-08
Qwen3-Reranker-0.6B	0.6 / 0.6	dense	1.2	—	—	112 KiB	Apache-2.0	2026-09-08
Parakeet TDT 0.6B v3	0.6 / 0.6	dense	1.2	—	—	—	CC-BY-4.0	2026-09-08
Qwen3-1.7B	1.7 / 1.7	dense	3.4	1.8	1.1	112 KiB	Apache-2.0	2026-09-08
Qwen3-1.7B-Base	1.7 / 1.7	dense	3.4	—	—	112 KiB	Apache-2.0	2026-09-08
Qwen3-4B	4 / 4	dense	8	4.3	2.5	144 KiB	Apache-2.0	2026-09-08
Gemma 4 E4B (it)	4.5 / 4.5	dense	9	—	2.8	280 KiB	Apache-2.0	2026-09-08
Llama 3.1 8B Instruct	8 / 8	dense	16.1	8.5	4.9	128 KiB	Llama-3.1-Community	2026-09-08
Qwen3-8B	8.2 / 8.2	dense	16.4	8.7	5	144 KiB	Apache-2.0	2026-09-08
Qwen3-VL-8B-Instruct	8.8 / 8.8	dense	17.6	—	5.4	144 KiB	Apache-2.0	2026-09-08
Qwen3-14B	14.8 / 14.8	dense	29.6	15.7	9	160 KiB	Apache-2.0	2026-09-08
gpt-oss-20b	21 / 3.6	moe	13	—	13	48 KiB	Apache-2.0	2026-09-08
Devstral Small 2	24 / 24	dense	48	25.5	14.5	160 KiB	Apache-2.0	2026-09-08
Nemotron 3 Nano 30B-A3B	30 / 3.5	hybrid	60	—	18	208 KiB	NVIDIA-Open-Model	2026-09-08
Qwen3-30B-A3B	30.5 / 3.3	moe	61	32.5	18.6	96 KiB	Apache-2.0	2026-09-08
Qwen3-Coder-30B-A3B-Instruct	30.5 / 3.3	moe	61	32.5	18.6	96 KiB	Apache-2.0	2026-09-08
Qwen3-32B	32.8 / 32.8	dense	65.6	34.8	19.8	256 KiB	Apache-2.0	2026-09-08
Qwen3-Next-80B-A3B	80 / 3	hybrid	160	—	48	48 KiB	Apache-2.0	2026-09-08
gpt-oss-120b	117 / 5.1	moe	63	—	63	72 KiB	Apache-2.0	2026-09-08
Qwen3-235B-A22B	235 / 22	moe	470	250	142	188 KiB	Apache-2.0	2026-09-08
GLM-4.6	357 / 32	moe	714	—	215	368 KiB	MIT	2026-09-08
Llama 3.1 405B Instruct	405 / 405	dense	810	—	205	504 KiB	Llama-3.1-Community	2026-09-09
Qwen3-Coder-480B-A35B-Instruct	480 / 35	moe	960	—	290	248 KiB	Apache-2.0	2026-09-08

qwen3-0.6b Alibaba · Qwen3 · student

The 0.5B-class model used for the first training run in Part 11; too small to serve, the right size to learn the toolchain on.

Sources:safetensors · gguf · mlx

Used in:Attention and the Transformer, Embeddings: Meaning as Geometry, Lab: Look Inside a Model, From Autocomplete to Assistant: Next-Token Prediction, Parameters, Layers and Model Size, Tokens, Tokenisers and Vocabulary, Inference: Prefill, Decode and Why Memory Bandwidth Rules, Distillation, Pruning and Quantisation: How Small Models Get Good, Open Weights, Open Source and Licences, Post-Training: SFT, Preference Tuning and Reinforcement Learning, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Lab: Prepare Your Machine, Challenge: The Model That Runs at Two Tokens per Second, Lab: Your First Training Run, Challenge: The Student That Learned the Teacher's Mistakes, Lab: Logit Distillation with TRL's Distillation Trainers, Three Kinds of Distillation: Logit, Sequence and On-Policy, Why Distillation Works, Lab: Train and Deploy a Draft for Your Model, Speculative Decoding Revisited: Acceptance Rates and When It Pays, Training a Draft Model: Medusa and EAGLE at Home

qwen3-0.6b-base Alibaba · Qwen3 · reference

The pretrained checkpoint behind Qwen3-0.6B; the base-versus-instruct comparison on the reduced path of the Part 2 lab. Hub API on 2026-09-12: apache-2.0, not gated, 596,049,920 parameters in safetensors.

Sources:safetensors

Used in:Lab: Look Inside a Model, Post-Training: SFT, Preference Tuning and Reinforcement Learning, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name

qwen3-embedding-0.6b Alibaba · Qwen3-Embedding · reference, embedding

Embedding model for the retrieval labs; paired with Qwen3-Reranker-0.6B.

Sources:safetensors · gguf

Used in:Capstone 1: Hardware and Model Plan, Capstone 2: The Inference Service, Embeddings: Meaning as Geometry, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Front-Ends: Open WebUI and Friends, Project: Your Local Model Gateway, Project: A Private Document Question-Answering Service, Retrieval-Augmented Generation: Embeddings, Chunking and Vector Stores, Backup, Upgrades and Reproducibility of a Model Estate, Lab: Dashboards for Your Cluster, Routing and Model Management: llama-swap, LiteLLM, NGINX and Health Checks, Agentic Retrieval and Research Agents, Project: A Multi-Agent System on Your Cluster

qwen3-reranker-0.6b Alibaba · Qwen3-Reranker · reference, reranker

Sources:safetensors

Used in:Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Project: A Private Document Question-Answering Service, Retrieval-Augmented Generation: Embeddings, Chunking and Vector Stores, Agentic Retrieval and Research Agents, Project: A Multi-Agent System on Your Cluster

parakeet-tdt-0.6b-v3 NVIDIA · Parakeet · reference, speech

Speech-to-text, 25+ languages; licence tag cc-by-4.0 confirmed through the Hugging Face model API on 2026-09-08.

Sources:safetensors

Used in:Open Weights, Open Source and Licences, Model Families and Who Makes Them, Vision, Speech and Documents: Multimodal Locally

qwen3-1.7b Alibaba · Qwen3 · reference, student

The ~1-2B tier; the distillation student.

Sources:safetensors · gguf · mlx

Used in:Capstone 5: The Agentic Workstation, Capstone 3: Cluster or Tiered Deployment, Capstone 1: Hardware and Model Plan, Capstone 4: Improve a Small Model, Attention and the Transformer, Embeddings: Meaning as Geometry, Lab: Look Inside a Model, From Autocomplete to Assistant: Next-Token Prediction, Parameters, Layers and Model Size, Tokens, Tokenisers and Vocabulary, Distillation, Pruning and Quantisation: How Small Models Get Good, Open Weights, Open Source and Licences, Post-Training: SFT, Preference Tuning and Reinforcement Learning, Lab: Build Your Model Shortlist, Model Families and Who Makes Them, Managing a Model Library: Storage, Naming and Versions, Installing vLLM: x86 CUDA, DGX Spark, ROCm and What Does Not Work, Speculative Decoding: Draft Models, EAGLE and n-gram, Lab: Your First Training Run, Memory Arithmetic for Training: Weights, Gradients, Optimiser States and Activations, What Pretraining Teaches You That Fine-Tuning Cannot, Challenge: The Fine-Tune That Got Worse, Fine-Tuning with TRL and PEFT, Lab: Fine-Tune a 1B to 4B Model to Follow Your Format, LoRA and QLoRA Explained, Merging, Exporting and Quantising a Fine-Tuned Model, What Fine-Tuning Changes and What It Cannot, DPO and Its Family: IPO, KTO, ORPO and SimPO, From Imitation to Preferences: Why Ranking Beats Copying, Reinforcement Learning with Verifiable Rewards: GRPO Explained, Lab: DPO a Model to Prefer Your Style, Lab: GRPO on a Maths or Code Task on One Machine, Reality Check: 'RL Makes Small Models Reason', Reward Functions: Maths, Code Tests, Format and Length, The RL Toolchain: TRL, Unsloth, verl, OpenRLHF and vLLM Rollouts, Challenge: The Student That Learned the Teacher's Mistakes, Lab: Logit Distillation with TRL's Distillation Trainers, Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student, Three Kinds of Distillation: Logit, Sequence and On-Policy, Project: The Distillation Pipeline, Pruning and Compression: Prune-and-Distil, Reasoning Distillation: Teacher Traces as Training Data, Why Distillation Works, Lab: Quantise Your Fine-Tune Five Ways and Measure Each, Lab: Run a Standard Benchmark Suite on Your Model, Lab: Train and Deploy a Draft for Your Model, Speculative Decoding Revisited: Acceptance Rates and When It Pays, Lab: KV Cache Offload and Sharing, Lab: Two-Machine Prefill and Decode with vLLM, Project: A Tiered Inference Architecture, vLLM Disaggregated Prefill: Connectors and the Proxy, Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster, Reality Check: 'Agents Are Just Loops', Distilling a Big Agent Model into a Small Local One, Fine-Tuning for Tool Use and Your Codebase, Lab: Fine-Tune a Small Model on Your Own Agent Trajectories and Measure the Gain, Reinforcement Learning on Agent Tasks: Tests as Rewards

qwen3-1.7b-base Alibaba · Qwen3 · reference

The pretrained checkpoint behind Qwen3-1.7B; the base-versus-instruct comparison in Part 2's lab.

Sources:safetensors

Used in:Lab: Look Inside a Model, Post-Training: SFT, Preference Tuning and Reinforcement Learning, Pretraining: Learning from Trillions of Tokens, Data for Pretraining: FineWeb-Edu, Cosmopedia and Training a Tokeniser, Project: A Domain Micro-Model, What Pretraining Teaches You That Fine-Tuning Cannot, Challenge: The Fine-Tune That Got Worse

qwen3-4b Alibaba · Qwen3 · reference, student

The ~4B tier; the fine-tuning workhorse at 12-16 GB.

Sources:safetensors · gguf · mlx

Used in:Capstone 4: Improve a Small Model, Attention and the Transformer, Distillation, Pruning and Quantisation: How Small Models Get Good, Reality Check: 'A Small Local Model Is as Good as the Frontier', Choosing a Model for a Memory Budget, Lab: Build Your Model Shortlist, Model Families and Who Makes Them, Lab: Run and Benchmark the Course Reference Models, Lab: A Private Chat Service for Your Home Network, LM Studio: GUI, MLX and Headless Serving, Managing a Model Library: Storage, Naming and Versions, Ollama: Models as a Service, Reality Check: 'The Default Context Is Enough', MLX and mlx-lm: Apple's Native Path, Lab: Benchmark Local Models on Your Own Tasks, Local Coding Assistants: Autocomplete and Chat in Your Editor, Project: A Private Document Question-Answering Service, Structured Output and JSON Mode, Memory Arithmetic for Training: Weights, Gradients, Optimiser States and Activations, Building an SFT Dataset, Challenge: The Fine-Tune That Got Worse, Fine-Tuning with TRL and PEFT, Lab: Fine-Tune a 1B to 4B Model to Follow Your Format, LoRA and QLoRA Explained, Merging, Exporting and Quantising a Fine-Tuned Model, Project: A Specialist Assistant, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, What Fine-Tuning Changes and What It Cannot, DPO and Its Family: IPO, KTO, ORPO and SimPO, From Imitation to Preferences: Why Ranking Beats Copying, Reinforcement Learning with Verifiable Rewards: GRPO Explained, Lab: DPO a Model to Prefer Your Style, Lab: GRPO on a Maths or Code Task on One Machine, Reality Check: 'RL Makes Small Models Reason', Reward Functions: Maths, Code Tests, Format and Length, The RL Toolchain: TRL, Unsloth, verl, OpenRLHF and vLLM Rollouts, Challenge: The Student That Learned the Teacher's Mistakes, Lab: Logit Distillation with TRL's Distillation Trainers, Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student, Three Kinds of Distillation: Logit, Sequence and On-Policy, Project: The Distillation Pipeline, Pruning and Compression: Prune-and-Distil, Reasoning Distillation: Teacher Traces as Training Data, Generating Synthetic Data with a Local Teacher, Why Distillation Works, Challenge: The Benchmark That Lied, Evaluation Harnesses: lm-evaluation-harness, lighteval, EvalPlus and Your Own, Lab: Quantise Your Fine-Tune Five Ways and Measure Each, Lab: Run a Standard Benchmark Suite on Your Model, LLM-as-Judge, Contamination and Honest Reporting, Measuring Quantisation Damage: Perplexity, KL Divergence and Task Evaluations, Post-Training Quantisation in Depth: GPTQ, AWQ, K-quants, imatrix and HQQ, Lab: Train and Deploy a Draft for Your Model, Training a Draft Model: Medusa and EAGLE at Home, Context Engineering: Memory, Compaction and the KV Budget, Function Calling End to End on Local Engines, Lab: A Minimal Agent from Scratch, Lab: Write and Connect an MCP Server, Reasoning Models in Agent Loops, What an Agent Is: The Loop, Tools and State, Project: A Local Agentic Coding Workstation, Which Local Models Can Actually Drive an Agent, Agent Frameworks Compared, Agentic Retrieval and Research Agents, Evaluating Agents: Trajectories, Success Rates and Cost, Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster, Reality Check: 'Agents Are Just Loops', Safety: Prompt Injection, Tool Permissions and Human-in-the-Loop, Distilling a Big Agent Model into a Small Local One, Fine-Tuning for Tool Use and Your Codebase, Lab: Fine-Tune a Small Model on Your Own Agent Trajectories and Measure the Gain, Reinforcement Learning on Agent Tasks: Tests as Rewards

gemma-4-e4b Google · Gemma 4 · alternative, vision

Licence apache-2.0 confirmed on the model card on 2026-09-08 (card: 4.5B effective parameters, 8B with embeddings, 128K context, text, image and audio input). Multimodal small-model example.

Sources:safetensors

Used in:Open Weights, Open Source and Licences, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name

llama-3.1-8b Meta · Llama 3.1 · comparison

The cross-engine comparison model: the most widely converted checkpoint. Gated on the Hub; Llama licence conditions stated on the page.

Sources:safetensors · gguf · mlx

Used in:Tokens, Tokenisers and Vocabulary, Open Weights, Open Source and Licences, Pretraining: Learning from Trillions of Tokens, Lab: Build Your Model Shortlist, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, GGUF and Quantisation Types, Challenge: The Student That Learned the Teacher's Mistakes, Pruning and Compression: Prune-and-Distil, Generating Synthetic Data with a Local Teacher, Challenge: The Benchmark That Lied, Evaluation Harnesses: lm-evaluation-harness, lighteval, EvalPlus and Your Own, Lab: Run a Standard Benchmark Suite on Your Model, Post-Training Quantisation in Depth: GPTQ, AWQ, K-quants, imatrix and HQQ, SGLang PD Disaggregation and NVIDIA Dynamo

qwen3-8b Alibaba · Qwen3 · reference

The ~8B tier; first real model on every track.

Sources:safetensors · gguf · mlx

Used in:Capstone 5: The Agentic Workstation, Capstone 3: Cluster or Tiered Deployment, Capstone 1: Hardware and Model Plan, Capstone 4: Improve a Small Model, Capstone 2: The Inference Service, Neural Networks, Activations and Backpropagation, Tensors, GPUs and Precision: Why Matrix Multiplication Is Everything, Attention and the Transformer, Embeddings: Meaning as Geometry, From Autocomplete to Assistant: Next-Token Prediction, Parameters, Layers and Model Size, Tokens, Tokenisers and Vocabulary, Inference: Prefill, Decode and Why Memory Bandwidth Rules, Distillation, Pruning and Quantisation: How Small Models Get Good, Open Weights, Open Source and Licences, Pretraining: Learning from Trillions of Tokens, Choosing a Model for a Memory Budget, Lab: Build Your Model Shortlist, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Lab: Measure Your Memory Bandwidth and Compute, Challenge: The Model That Runs at Two Tokens per Second, GGUF and Quantisation Types, Installing and Building llama.cpp on Your Platform, Lab: Run and Benchmark the Course Reference Models, llama-cli and llama-server, Sampling: Temperature, Top-p, Min-p, Repetition and Determinism, LM Studio: GUI, MLX and Headless Serving, Managing a Model Library: Storage, Naming and Versions, Ollama: Models as a Service, Reality Check: 'The Default Context Is Enough', Lab: Same Model, Every Engine, MLX and mlx-lm: Apple's Native Path, AMD-Native: ROCm Builds, Lemonade Server and the NPU Question, Specialist Engines: ExLlamaV3, ktransformers and mistral.rs, TensorRT-LLM, NIM and the DGX Spark Playbooks, Why a Second Kind of Engine: Batching, Paged Attention and Throughput, Installing vLLM: x86 CUDA, DGX Spark, ROCm and What Does Not Work, Lab: Serve a Model to Twenty Concurrent Users, Project: Your Local Model Gateway, Tool Calling and Structured Output on the Server Side, Serving with vLLM: Quantised Weights, Context, Memory and Multi-GPU, SGLang: When to Choose It, Speculative Decoding: Draft Models, EAGLE and n-gram, Lab: Benchmark Local Models on Your Own Tasks, Project: A Private Document Question-Answering Service, Prompting That Works Locally: System Prompts, Chat Templates and Thinking Modes, Structured Output and JSON Mode, Memory Arithmetic for Training: Weights, Gradients, Optimiser States and Activations, Challenge: The Fine-Tune That Got Worse, Fine-Tuning with TRL and PEFT, Lab: Fine-Tune a 1B to 4B Model to Follow Your Format, LoRA and QLoRA Explained, Merging, Exporting and Quantising a Fine-Tuned Model, Project: A Specialist Assistant, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, What Fine-Tuning Changes and What It Cannot, From Imitation to Preferences: Why Ranking Beats Copying, Lab: DPO a Model to Prefer Your Style, Reality Check: 'RL Makes Small Models Reason', Challenge: The Student That Learned the Teacher's Mistakes, Lab: Logit Distillation with TRL's Distillation Trainers, Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student, Three Kinds of Distillation: Logit, Sequence and On-Policy, Project: The Distillation Pipeline, Pruning and Compression: Prune-and-Distil, Reasoning Distillation: Teacher Traces as Training Data, Generating Synthetic Data with a Local Teacher, Why Distillation Works, Challenge: The Benchmark That Lied, Evaluation Harnesses: lm-evaluation-harness, lighteval, EvalPlus and Your Own, Lab: Quantise Your Fine-Tune Five Ways and Measure Each, Lab: Run a Standard Benchmark Suite on Your Model, LLM-as-Judge, Contamination and Honest Reporting, Measuring Quantisation Damage: Perplexity, KL Divergence and Task Evaluations, Post-Training Quantisation in Depth: GPTQ, AWQ, K-quants, imatrix and HQQ, Quantisation-Aware Training and the Blackwell Formats: FP8, NVFP4 and MXFP4, Lab: Train and Deploy a Draft for Your Model, Prefix Caching and KV Reuse, Speculative Decoding Revisited: Acceptance Rates and When It Pays, Training a Draft Model: Medusa and EAGLE at Home, Challenge: The Cluster That Is Slower Than One Machine, Lab: A Mixed-Platform Cluster, Lab: A Two-Mac Cluster over Thunderbolt 5, Lab: KV Cache Offload and Sharing, Lab: Two-Machine Prefill and Decode with vLLM, Project: A Tiered Inference Architecture, SGLang PD Disaggregation and NVIDIA Dynamo, The KV Cache as a Transferable Object: Mooncake, LMCache and NIXL, vLLM Disaggregated Prefill: Connectors and the Proxy, Why Prefill and Decode Want Different Hardware, Backup, Upgrades and Reproducibility of a Model Estate, Capacity Planning and Cost per Million Tokens at Home, Challenge: The 3 a.m. Out-of-Memory, Lab: Dashboards for Your Cluster, Observability: Metrics, Logs and Traces for LLM Serving, Routing and Model Management: llama-swap, LiteLLM, NGINX and Health Checks, Security for Exposed Endpoints and the Model Supply Chain, Context Engineering: Memory, Compaction and the KV Budget, Function Calling End to End on Local Engines, Lab: A Minimal Agent from Scratch, Lab: Write and Connect an MCP Server, Reasoning Models in Agent Loops, What an Agent Is: The Loop, Tools and State, Aider with Local Models, Claude Code with a Local Endpoint, OpenAI Codex CLI and OpenCode with Local Models, Editor Agents: Cline, Kilo Code, Continue and the Proprietary Editors, Lab: One Task, Six Agents, Project: A Local Agentic Coding Workstation, Which Local Models Can Actually Drive an Agent, Agent Frameworks Compared, Agentic Retrieval and Research Agents, Evaluating Agents: Trajectories, Success Rates and Cost, Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster, Reality Check: 'Agents Are Just Loops', Safety: Prompt Injection, Tool Permissions and Human-in-the-Loop, Distilling a Big Agent Model into a Small Local One

qwen3-vl-8b Alibaba · Qwen3-VL · reference, vision

Vision-language model for the multimodal lesson and agent screenshots.

Sources:safetensors · gguf · mlx

Used in:Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Vision, Speech and Documents: Multimodal Locally

qwen3-14b Alibaba · Qwen3 · reference, teacher-24gb

The ~14B tier; the 24 GB tier's distillation teacher.

Sources:safetensors · gguf · mlx

Used in:Distillation, Pruning and Quantisation: How Small Models Get Good, Reality Check: 'A Small Local Model Is as Good as the Frontier', Choosing a Model for a Memory Budget, Lab: Build Your Model Shortlist, Model Families and Who Makes Them, Lab: Run and Benchmark the Course Reference Models, Managing a Model Library: Storage, Naming and Versions, Serving with vLLM: Quantised Weights, Context, Memory and Multi-GPU, Building an SFT Dataset, Project: A Specialist Assistant, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, What Fine-Tuning Changes and What It Cannot, Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student, Project: The Distillation Pipeline, Reasoning Distillation: Teacher Traces as Training Data, Generating Synthetic Data with a Local Teacher, Why Distillation Works, Context Engineering: Memory, Compaction and the KV Budget

gpt-oss-20b OpenAI · gpt-oss · reference

Native MXFP4 weights; the native 4-bit floating-point example.

Sources:safetensors · gguf · mlx

Used in:Tensors, GPUs and Precision: Why Matrix Multiplication Is Everything, Parameters, Layers and Model Size, Distillation, Pruning and Quantisation: How Small Models Get Good, Open Weights, Open Source and Licences, Choosing a Model for a Memory Budget, Lab: Build Your Model Shortlist, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, GGUF and Quantisation Types, Lab: Run and Benchmark the Course Reference Models, llama-cli and llama-server, Sampling: Temperature, Top-p, Min-p, Repetition and Determinism, Managing a Model Library: Storage, Naming and Versions, TensorRT-LLM, NIM and the DGX Spark Playbooks, Tool Calling and Structured Output on the Server Side, Prompting That Works Locally: System Prompts, Chat Templates and Thinking Modes, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, Generating Synthetic Data with a Local Teacher, Quantisation-Aware Training and the Blackwell Formats: FP8, NVFP4 and MXFP4, Multi-GPU Desktops: PCIe, Tensor Parallel Without NVLink and Expert Parallel, Context Engineering: Memory, Compaction and the KV Budget, Function Calling End to End on Local Engines, Reasoning Models in Agent Loops, What an Agent Is: The Loop, Tools and State, Goose and OpenHands: Autonomous Agents and Sandboxing, Lab: One Task, Six Agents, Project: A Local Agentic Coding Workstation, Which Local Models Can Actually Drive an Agent

devstral-small-2 Mistral AI · Devstral · alternative, coder

Dense coder alternative. The Hugging Face repo is gated (anonymous fetch returned 401 on 2026-09-08), so the licence, sizes and context must be confirmed with a logged-in account during the hardware validation pass; Mistral's own API retired the hosted Devstral Small 2 endpoint in March 2026, which does not affect the downloadable weights.

Sources:safetensors

Used in:Capstone 5: The Agentic Workstation, Model Families and Who Makes Them, Which Local Models Can Actually Drive an Agent

nemotron-3-nano-30b-a3b NVIDIA · Nemotron 3 · alternative

Hybrid: 23 Mamba-2 and MoE layers plus 6 attention layers, 128 experts plus 1 shared, 6 active per token (card, 2026-09-08); 3.5B active parameters; 1M-token maximum context, 256k default. The hybrid-attention example. Licence is NVIDIA's own open-model licence, not Apache or MIT.

Sources:safetensors

Used in:Open Weights, Open Source and Licences, Dense, Mixture-of-Experts and Hybrid Architectures, Model Families and Who Makes Them

qwen3-30b-a3b Alibaba · Qwen3 · reference

The ~30B mixture-of-experts tier; the dense-versus-MoE teaching pair with gpt-oss-20b.

Sources:safetensors · gguf · mlx

Used in:Capstone 3: Cluster or Tiered Deployment, Capstone 1: Hardware and Model Plan, Capstone 4: Improve a Small Model, Parameters, Layers and Model Size, Inference: Prefill, Decode and Why Memory Bandwidth Rules, Distillation, Pruning and Quantisation: How Small Models Get Good, Reality Check: 'A Small Local Model Is as Good as the Frontier', Choosing a Model for a Memory Budget, Dense, Mixture-of-Experts and Hybrid Architectures, Lab: Build Your Model Shortlist, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Apple Silicon: Unified Memory, Metal and MLX, Lab: Measure Your Memory Bandwidth and Compute, Lab: Run and Benchmark the Course Reference Models, Sampling: Temperature, Top-p, Min-p, Repetition and Determinism, Managing a Model Library: Storage, Naming and Versions, Ollama: Models as a Service, Why a Second Kind of Engine: Batching, Paged Attention and Throughput, Serving with vLLM: Quantised Weights, Context, Memory and Multi-GPU, Memory Arithmetic for Training: Weights, Gradients, Optimiser States and Activations, LoRA and QLoRA Explained, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, Lab: Sequence-Level Distillation of a 30B-Class Teacher into a 4B Student, Three Kinds of Distillation: Logit, Sequence and On-Policy, Project: The Distillation Pipeline, Reasoning Distillation: Teacher Traces as Training Data, Generating Synthetic Data with a Local Teacher, Why Distillation Works, Tensor, Pipeline, Expert, Data and Sequence Parallelism, Why One Box Runs Out: The Memory Wall and the Bandwidth Wall, The Course Reference Cluster and the Single-Machine Path, Challenge: The Cluster That Is Slower Than One Machine, Lab: A Mixed-Platform Cluster, Lab: Run a Model Bigger Than Any One Machine, llama.cpp RPC: Layers Across Machines, RDMA Transport, Tuning and Measuring the Split, Multi-GPU Desktops: PCIe, Tensor Parallel Without NVLink and Expert Parallel, mlx.distributed: Ring, MPI and RDMA over Thunderbolt 5, Project: A Tiered Inference Architecture, The KV Cache as a Transferable Object: Mooncake, LMCache and NIXL, Why Prefill and Decode Want Different Hardware, Context Engineering: Memory, Compaction and the KV Budget, Reasoning Models in Agent Loops, Agent Frameworks Compared, Agentic Retrieval and Research Agents, Evaluating Agents: Trajectories, Success Rates and Cost, Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster, Reality Check: 'Agents Are Just Loops', Safety: Prompt Injection, Tool Permissions and Human-in-the-Loop, Distilling a Big Agent Model into a Small Local One

qwen3-coder-30b-a3b Alibaba · Qwen3-Coder · reference, coder

The coder for 24-32 GB tiers; native tool calling and 256k context.

Sources:safetensors · gguf · mlx

Used in:Capstone 5: The Agentic Workstation, Capstone 2: The Inference Service, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Base, Instruct, Thinking, Coder, Vision, Embedding: Reading a Model Name, Project: Your Local Model Gateway, Tool Calling and Structured Output on the Server Side, Speculative Decoding: Draft Models, EAGLE and n-gram, Local Coding Assistants: Autocomplete and Chat in Your Editor, Prompting That Works Locally: System Prompts, Chat Templates and Thinking Modes, Backup, Upgrades and Reproducibility of a Model Estate, Capacity Planning and Cost per Million Tokens at Home, Challenge: The 3 a.m. Out-of-Memory, Routing and Model Management: llama-swap, LiteLLM, NGINX and Health Checks, Security for Exposed Endpoints and the Model Supply Chain, Context Engineering: Memory, Compaction and the KV Budget, Function Calling End to End on Local Engines, Lab: A Minimal Agent from Scratch, Reasoning Models in Agent Loops, What an Agent Is: The Loop, Tools and State, Aider with Local Models, Challenge: The Agent That Escaped the Sandbox, Claude Code with a Local Endpoint, OpenAI Codex CLI and OpenCode with Local Models, Editor Agents: Cline, Kilo Code, Continue and the Proprietary Editors, Goose and OpenHands: Autonomous Agents and Sandboxing, Lab: One Task, Six Agents, Lab: Sandbox Your Agent: Containers, Permissions and Secrets, Project: A Local Agentic Coding Workstation, Which Local Models Can Actually Drive an Agent, Distilling a Big Agent Model into a Small Local One, Fine-Tuning for Tool Use and Your Codebase

qwen3-32b Alibaba · Qwen3 · reference, teacher-128gb

The ~32B dense tier; the same-tokeniser distillation teacher for the Qwen3 students.

Sources:safetensors · gguf · mlx

Used in:Distillation, Pruning and Quantisation: How Small Models Get Good, Choosing a Model for a Memory Budget, Dense, Mixture-of-Experts and Hybrid Architectures, Lab: Build Your Model Shortlist, Model Families and Who Makes Them, Lab: Measure Your Memory Bandwidth and Compute, NVIDIA Desktops and Laptops: VRAM Tiers, CUDA and WSL2, GGUF and Quantisation Types, Lab: Run and Benchmark the Course Reference Models, llama-cli and llama-server, Specialist Engines: ExLlamaV3, ktransformers and mistral.rs, Serving with vLLM: Quantised Weights, Context, Memory and Multi-GPU, Speculative Decoding: Draft Models, EAGLE and n-gram, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, Lab: Serve a 400B-Class Model on Two DGX Sparks, Multi-GPU Desktops: PCIe, Tensor Parallel Without NVLink and Expert Parallel, TensorRT-LLM and Dynamo on Spark Pairs, vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across, Lab: A Two-Mac Cluster over Thunderbolt 5, mlx.distributed: Ring, MPI and RDMA over Thunderbolt 5, The KV Cache as a Transferable Object: Mooncake, LMCache and NIXL

qwen3-next-80b-a3b Alibaba · Qwen3-Next · alternative

Hybrid linear attention; the KV-cache lesson's example. Only a quarter of the layers keep a growing cache, so bytesPerToken is an upper bound.

Sources:safetensors

Used in:Dense, Mixture-of-Experts and Hybrid Architectures, Model Families and Who Makes Them

gpt-oss-120b OpenAI · gpt-oss · reference, agent

The ~120B mixture-of-experts tier; the single-Spark fallback in the cluster labs; the best-documented agentic benchmark point among open models.

Sources:safetensors · gguf · mlx

Used in:Inference: Prefill, Decode and Why Memory Bandwidth Rules, Choosing a Model for a Memory Budget, Dense, Mixture-of-Experts and Hybrid Architectures, Lab: Build Your Model Shortlist, Reading a Model Card and a Benchmark, Model Families and Who Makes Them, Lab: Run and Benchmark the Course Reference Models, Unsloth, Axolotl, LLaMA-Factory and mlx-lm: Higher-Level Tools, Quantisation-Aware Training and the Blackwell Formats: FP8, NVFP4 and MXFP4, Tensor, Pipeline, Expert, Data and Sequence Parallelism, Why One Box Runs Out: The Memory Wall and the Bandwidth Wall, The Course Reference Cluster and the Single-Machine Path, Lab: Run a Model Bigger Than Any One Machine, RDMA Transport, Tuning and Measuring the Split, Connecting Two DGX Sparks over ConnectX-7, Lab: Serve a 400B-Class Model on Two DGX Sparks, TensorRT-LLM and Dynamo on Spark Pairs, vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across, mlx.distributed: Ring, MPI and RDMA over Thunderbolt 5, The KV Cache as a Transferable Object: Mooncake, LMCache and NIXL, Which Local Models Can Actually Drive an Agent, Agent Frameworks Compared, Evaluating Agents: Trajectories, Success Rates and Cost, Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster, Reality Check: 'Agents Are Just Loops'

qwen3-235b-a22b Alibaba · Qwen3 · reference, cluster

The ~200B+ tier; the llama.cpp RPC cluster model.

Sources:safetensors · gguf · mlx

Used in:Capstone 3: Cluster or Tiered Deployment, Neural Networks, Activations and Backpropagation, Distillation, Pruning and Quantisation: How Small Models Get Good, Choosing a Model for a Memory Budget, Dense, Mixture-of-Experts and Hybrid Architectures, Model Families and Who Makes Them, Lab: Run and Benchmark the Course Reference Models, Specialist Engines: ExLlamaV3, ktransformers and mistral.rs, Tensor, Pipeline, Expert, Data and Sequence Parallelism, Why One Box Runs Out: The Memory Wall and the Bandwidth Wall, The Course Reference Cluster and the Single-Machine Path, Lab: Run a Model Bigger Than Any One Machine, llama.cpp RPC: Layers Across Machines, RDMA Transport, Tuning and Measuring the Split, Connecting Two DGX Sparks over ConnectX-7, Lab: Serve a 400B-Class Model on Two DGX Sparks, TensorRT-LLM and Dynamo on Spark Pairs, vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across, Reality Check: 'Four Mac Studios Replace a GPU Server', Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out, Project: A Multi-Agent System on Your Cluster

glm-4.6 Z.ai · GLM · alternative

The MIT-licensed alternative at the 200B+ tier.

Sources:safetensors · gguf

Used in:Open Weights, Open Source and Licences, Model Families and Who Makes Them, Specialist Engines: ExLlamaV3, ktransformers and mistral.rs, Why One Box Runs Out: The Memory Wall and the Bandwidth Wall

llama-3.1-405b Meta · Llama 3.1 · cluster

The vendor-documented two-Spark playbook model: NVIDIA's vLLM two-Spark playbook (read 2026-09-09) serves an AWQ INT4 conversion of the 405B Instruct model, with the vendor's own caveat that it has insufficient memory headroom for production use and is for testing only; the FP4 two-Spark model in the TensorRT-LLM matrix is Qwen3-235B-A22B-NVFP4. Llama licence conditions apply.

Sources:safetensors

Used in:Open Weights, Open Source and Licences, Pretraining: Learning from Trillions of Tokens, Model Families and Who Makes Them, Connecting Two DGX Sparks over ConnectX-7, Lab: Serve a 400B-Class Model on Two DGX Sparks, vLLM Multi-Node with Ray: Tensor Parallel Inside, Pipeline Parallel Across

qwen3-coder-480b-a35b Alibaba · Qwen3-Coder · coder, cluster

The cluster-scale coder.

Sources:safetensors · gguf

Used in:Model Families and Who Makes Them, Which Local Models Can Actually Drive an Agent