Reality Check: 'A Small Local Model Is as Good as the Frontier'

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track model tags, Ollama versions and dates belong here once the validation pass has run this page on real machines.

?Reality check

The claim under test, in the form it usually appears: “a small local model is as good as the frontier”. It is not a statement you can agree or disagree with, because it does not say as good at what, measured how, or how much difference would count. This page turns it into something that could be shown false, and then runs that test on your machine.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this page you will have run twenty tasks through a 4B-class model and a larger local model on your own machine, scored every answer without knowing which model produced it, tested whether the gap you found is larger than chance alone would produce on a sample this small, and recorded the whole run in your lab notebook against a margin you wrote down before the first answer existed. Optionally you will have added a hosted frontier model you already have access to as a third column, and rerun the pair in thinking mode to see what that changes.

The point is less the answer than the method below, because every measurement in the rest of the course has its shape.

⌂Single-machine path

This is a single-machine page on every track. If you own a Spark pair or several Macs, use one machine; clustering starts in Part 18.

The method

From a slogan to a result

1. 1State the claimWrite the vague version down exactly as people say it, so you can see what it is missing.
2. 2Make it refutableName the models, the tasks, the scoring and the margin that would count. If no result could contradict it, it is not a claim.
3. 3Fix the rubric and the margin firstDecide what a 0, a 1 and a 2 look like, and how many points count as a real gap, before any answer exists. Afterwards is too late: you will score to your expectation.
4. 4Collect answersSame prompts, same settings, same thinking mode, one model at a time.
5. 5Blind the answersStrip the labels and shuffle. This is the step that makes the score about the answer rather than about the badge.
6. 6Score, then un-blindMark everything, then reveal which model wrote what. Not before.
7. 7Ask what chance would doTwenty tasks is a small sample. Count how often a gap this big arises between two equally good models before you call it a difference.
8. 8Report what happenedIncluding the part you did not want, and including how small the sample was.

Turning the slogan into a measurable question

A measurable question names four things: the population of tasks it is about, the metric each task is scored with, the comparator the model is being held against, and the margin that would count as a difference. The slogan supplies none of them, and each missing one is a place where a later reader, or you, can quietly choose whatever makes the answer come out right. Here is the slogan taken apart word by word, with the choice this page makes for each gap:

Word in the slogan	What it leaves open	This page’s choice
“a small local model”	Which model, at which quantisation, in which mode, with which sampling settings	qwen3:4b-q4_K_M: Qwen3-4B at Q4_K_M, thinking off, temperature 0, seed 1
“is as good as”	On which tasks, scored by whom, against what standard	Twenty tasks in five categories, scored 0 to 2 by you, against a rubric written before any answer exists
“the frontier”	Which comparator, run how	A 30B-class local model on the same machine with the same settings (a 14B-class one below 24 GB), plus a hosted model if you already have one
(unsaid)	How much difference would count	Four points out of forty, written into the task file before the run
(unsaid)	What the small model gives up to be “as good”	Decode speed, memory and where the data goes, reported alongside the score rather than folded into it

Written out with the gaps filled, the claim becomes a sentence a result could contradict:

On twenty tasks spread over factual recall, arithmetic and reasoning, code, summarisation and instruction following, scored 0 to 2 against a rubric written in advance, qwen3:4b-q4_K_M scores within four points out of forty of qwen3:30b-a3b-q4_K_M run on the same machine with the same settings, and within four points of a hosted frontier model.

Now the claim is falsifiable: a gap of fifteen points makes it wrong on this test set, and a margin fixed in advance cannot be picked later to suit the answer.

The same move works on every slogan this course later examines: “the default context is enough” becomes a conversation length at which the model loses its own beginning, measured in Part 7; “RL makes small models reason” becomes a before-and-after score on verifiable problems, measured in Part 14; and “a benchmark number” becomes something you reproduce or fail to in Part 16.

What a margin means on twenty tasks

Four points is a choice, not a law, and choosing it means knowing what chance alone does to a total out of forty. Each task contributes a difference between the two models of −2, −1, 0, +1 or +2 points. If the two models were equally good, every task on which they differed was as likely to have gone the other way, so you can count the sign patterns that produce a gap at least as large as yours; that count divided by the number of patterns is the probability that an equal pair would have produced your gap. The scoring script computes it for your run; here is the same arithmetic on some patterns you may recognise afterwards.

RunnableAll tracks

sign-permutation.py - how often a gap arises between two equally good models

"""How often a score gap arises between two equally good models, by exact enumeration.

Each task contributes a difference d = score_A - score_B in {-2, -1, 0, 1, 2}. If the two
models were equally good, every non-zero difference was as likely to have gone the other way,
so count the sign patterns (2^k for k tasks that differed) whose total is at least as far
from zero as the one observed. Pure arithmetic; nothing here was measured.
"""
from collections import defaultdict

def sign_permutation_p(differences):
    nonzero = [d for d in differences if d != 0]
    observed = abs(sum(nonzero))
    dist = {0: 1}
    for d in nonzero:
        nxt = defaultdict(int)
        for total, count in dist.items():
            nxt[total + d] += count
            nxt[total - d] += count
        dist = nxt
    extreme = sum(c for t, c in dist.items() if abs(t) >= observed)
    return extreme / 2 ** len(nonzero), len(nonzero)

cases = {
    "gap 2: two tasks, +1 each":               [1, 1] + [0] * 18,
    "gap 4: four tasks, +1 each":              [1, 1, 1, 1] + [0] * 16,
    "gap 4: eight tasks, six up two down":     [1, 1, 1, 1, 1, 1, -1, -1] + [0] * 12,
    "gap 6: three tasks, +2 each":             [2, 2, 2] + [0] * 17,
    "gap 8: ten tasks, nine up one down":      [1] * 9 + [-1] + [0] * 10,
    "gap 12: twelve tasks, +1 each":           [1] * 12 + [0] * 8,
    "gap 12: ten tasks, +2, +2, +1 x8":        [2, 2] + [1] * 8 + [0] * 10,
}
print(f"{'observed pattern':<40}{'gap':>5}{'tasks differing':>17}{'p (two-sided)':>15}")
for label, diffs in cases.items():
    p, k = sign_permutation_p(diffs)
    print(f"{label:<40}{sum(diffs):>+5}{k:>17}{p:>15.4f}")

Output — what you should see

observed pattern                          gap  tasks differing  p (two-sided)
gap 2: two tasks, +1 each                  +2                2         0.5000
gap 4: four tasks, +1 each                 +4                4         0.1250
gap 4: eight tasks, six up two down        +4                8         0.2891
gap 6: three tasks, +2 each                +6                3         0.2500
gap 8: ten tasks, nine up one down         +8               10         0.0215
gap 12: twelve tasks, +1 each             +12               12         0.0005
gap 12: ten tasks, +2, +2, +1 x8          +12               10         0.0020

An equal pair produces a gap of four between one run in eight and one run in three, depending on how many tasks differed, which is why four is a defensible “indistinguishable” line rather than a strict one. A gap of eight over ten differing tasks arises about once in fifty; a gap of twelve essentially never. So twenty tasks detect a large difference and not a small one, and both halves of that sentence belong in your report.

Choosing tasks that could tell the models apart

A task has discriminating power when the two models could plausibly score differently on it. A task both models get right tells you nothing, and neither does one both get wrong; a set built entirely of those produces a tie however different the models are. The twenty tasks here are spread over five categories because the part’s lessons predict where a size difference should show up, and a total that mixed the categories would hide exactly the shape you want to see:

Category	What it mainly draws on	Where that came from	Prediction from this part
Factual recall	Facts stored in the weights	Pretraining: knowledge needs parameters to live in	Favours the larger model
Arithmetic and reasoning	Multi-step procedure without a slip	Post-training: RL on verifiable rewards; thinking mode	Favours the larger model; thinking mode narrows it
Code	Recall of APIs plus procedure	Both of the above	Favours the larger model, less sharply
Summarisation	Following a constraint over given text	Post-training: behaviour, not knowledge	Close between sizes
Instruction following	Format and constraint obedience	Distillation: behaviour transfers cheaply	Close between sizes

The prediction column is the falsifiable part of the design: if instruction following separates the models and factual recall does not, something in this part’s account is wrong for these two models, and that is worth knowing.

Within each category, a task is only usable if it meets three tests. It must be scorable without running a model: a person with the rubric and the answer can mark it in under a minute. It must have a rubric that names the failure modes, not just the right answer: ar-02 says that 39 scores 0 because it comes from ignoring the order of operations, and co-04 says a LEFT JOIN with the date condition in the WHERE clause scores 1 because it silently drops the customers with no orders (ar-04 and if-02 are worked through below). And it must avoid a ceiling: fr-01 (what HTTP stands for) is there as an anchor that any working model should pass, and one such anchor is useful, but a set made of anchors measures nothing.

▷Exercise

Replace two or three of these tasks with work of your own: a question from your job, a document you actually need summarised, a bug you actually had. Write the rubric before you ask any model, apply the three tests above, and keep the category label so the breakdown still reads. A task set built from your own work tells you about your own use of these models.

Blinding, and the scoring rules

Blinding removes one specific thing: the label. Knowing which model wrote an answer changes how you score it, and the effect does not announce itself, so the run script gives every answer a six-character random identifier, shuffles the answers within each task so that the order tells you nothing, and keeps the mapping in a separate file you do not open until every score is in.

Blinding does not remove a model’s style fingerprint: after a few tasks you may suspect that the terse answers come from one model and the ones with headings from the other. Every blind comparison between two systems has this limitation, and the defence is the rubric: score the thing it names and nothing else, so that a guess about the author has nothing to attach to. The four rules in the task file exist for that reason:

Rule	What it stops
Score the answer in front of you, not the model you think produced it	The fingerprint leaking back in
Judge against the rubric, not against your own preferred answer	Marking down a correct answer for taking a different route
No marks for length, confidence or formatting the task did not ask for	The pull towards the longer, more assured answer
If a rubric is ambiguous for an answer, write down how you resolved it and apply the same resolution to every model	A ruling that drifts between the two models

Here is what the rubric does to three answers for ar-04 (“a jacket costs 48 pounds after a 20% discount; what was the price before?”). These are illustrative answers written for the example, not recorded model output.

Answer as written	Score	Why
“The original price was 57.60 pounds (48 plus 20%).”	0	The rubric names 57.60 as the answer that comes from the wrong operation
“48 / 0.8 = 60.00 pounds.”	2	Right answer, right method, in the form asked for
“60 pounds. Twenty percent off means 48 is 80% of the original, so the original is 48 divided by 0.8.”	2	The extra sentence of working is not something the task forbade, so it neither earns nor costs anything

And for if-02 (reply with valid JSON and nothing else): a correct object wrapped in a code fence scores 1, because exactly one instruction was broken; a correct object with a sentence of commentary before it also scores 1; a fenced object with commentary scores 0. The rubric decides these before you see who did what.

✕Common mistake

Scoring one model’s twenty answers, then the other’s, is the commonest way to let a standard drift: by the second pass you have seen what “good” looks like and you mark to it. The symptom is a re-score that disagrees with your first pass; step 9’s order, one task at a time across every response, with the first two tasks re-scored at the end, prevents and detects it.

Requirements

Every track needs Ollama (Ollama 0.33.3 · verified 2026-09-08), disk space for two models, a Python 3 interpreter for two dependency-free scripts, and the course directory and notebook from Part 1’s environment lab. Nothing here is gated, so no account or token is needed.

The model pair depends on memory, not on track. Both models are pulled by their explicit -q4_K_M tags (the next section says why), and the sizes are the ones the Ollama library listed for those tags on 12 September 2026:

Memory	Small model	Larger model	Download
24 GB or more (32 GB or more on a Mac)	qwen3:4b-q4_K_M, 2.6 GB	qwen3:30b-a3b-q4_K_M, 19 GB	21.6 GB
12 GB to 24 GB (24 GB on a Mac)	qwen3:4b-q4_K_M, 2.6 GB	qwen3:14b-q4_K_M, 9.3 GB	11.9 GB

All three are Apache-2.0 licensed: the 30B card states “30.5B in total and 3.3B activated” and gives the licence as Apache-2.0, the 4B card gives Apache 2.0, and the course’s model reference records the same for the 14B.

Why the explicit tags

An Ollama tag is a pointer, and a short tag can be repointed. On the library’s tag list read on 12 September 2026, the short tag qwen3:4b carried the same digest as qwen3:4b-thinking-2507-q4_K_M and qwen3:30b-a3b the same digest as qwen3:30b-a3b-thinking-2507-q4_K_M: both short tags had been moved to the July 2025 thinking-only checkpoints, while qwen3:14b still matched qwen3:14b-q4_K_M, the original model. A comparison that pulled the short tags would put a thinking-only 4B against a thinking-only 30B in one memory tier and against an original hybrid 14B in the other, which is not the same experiment twice. The explicit -q4_K_M tags name the original Qwen3 checkpoints the model cards above describe, all three at the same quantisation, and all three hybrid models whose card documents both a thinking and a non-thinking mode. Whatever tag you pull, record the digest ollama list prints: the digest is the fact and the tag is a label.

Memory arithmetic

Ollama loads the weights plus a key-value cache for the context length it allocates. The script asks for a 4,096-token context, and the cache per token is the model’s published figure from the course’s model reference; the arithmetic below is from those stated inputs, not a measurement.

Model	Weights, Q4_K_M	KV cache per token, FP16	KV cache at 4,096 tokens	Weights plus cache
qwen3:4b-q4_K_M	2.6 GB	2 × 36 layers × 8 KV heads × 128 × 2 bytes = 147,456 B	0.60 GB	3.2 GB
qwen3:14b-q4_K_M	9.3 GB	2 × 40 × 8 × 128 × 2 = 163,840 B	0.67 GB	10.0 GB
qwen3:30b-a3b-q4_K_M	19 GB	2 × 48 × 4 × 128 × 2 = 98,304 B	0.40 GB	19.4 GB

The memory floor: qwen3:14b-q4_K_M at a 4,096-token context on a 12 GB card

Weights, Q4_K_M

9.3 GB

KV cache, 4,096 tokens, FP16

0.7 GB

Free

2.0 GB

Total

12 GB

Arithmetic from the Ollama tag size and the model reference's bytes per token, not a measurement. The engine's own overhead is not shown; ollama ps reports the real loaded size once the model is up, and the difference between that and the file size is what the cache and the overhead actually cost. Below 12 GB the 14B model spills to system memory and the comparison is no longer a fair one.

Ollama’s own defaults would not give you this budget: its Modelfile reference documents num_ctx as defaulting to 2048, its FAQ says the server uses 4096, and the source of 0.33.3 describes the OLLAMA_CONTEXT_LENGTH setting as default: 4k/32k/256k based on VRAM, so the server picks one of three sizes by the graphics memory it finds. The script passes num_ctx explicitly so that the two models are compared under the same allocation on every track, which is also why Part 7’s reality check exists.

Time and speed

Attended time is about twenty-five minutes: the preflight, the one question by hand, and the scoring, which is the part that needs you. Everything else runs unattended.

Stage	Arithmetic	Unattended time
Download, 24 GB pair	21.6 GB × 8 bits ÷ 100 Mbit/s ≈ 1,730 s; ÷ 1 Gbit/s ≈ 175 s	About 30 minutes on a 100 Mbit/s connection, a few minutes on gigabit
Download, 12 GB pair	11.9 GB × 8 ÷ 100 Mbit/s ≈ 950 s	About 16 minutes at 100 Mbit/s
Generation, thinking off	20 tasks × roughly 50 to 200 answer tokens per model	A few minutes per model, including the first load
Generation, thinking on (optional)	20 tasks × up to 4,096 tokens per model	Tens of minutes per model; leave it running

Before you run, predict the decode speed you will see. The inference lesson’s arithmetic is bandwidth divided by active bytes per token; the active bytes are the tag’s file size for a dense model and the file size scaled by 3.3 ÷ 30.5 for the mixture of experts. Step 10 sets your measured median against this ceiling.

Pending validationCeiling on decode speed for the comparison pairs, batch size 1, tokens per second

Track and machine	Bandwidth (GB/s)	qwen3:4b-q4_K_M (2.6 GB)	qwen3:14b-q4_K_M (9.3 GB)	qwen3:30b-a3b-q4_K_M (~2.1 GB active)
S: DGX Spark	273	105	29	133
X: Ryzen AI Max+ 395	256	98	28	125
M: Mac, M4 (24 GB)	120	46	13	58
M: Mac, M4 Pro	273	105	29	133
M: Mac, M4 Max	546	210	59	266
N: RTX 3090 (24 GB)	936	360	101	455
N: RTX 4090 (24 GB)	1,008	388	108	490
N: RTX 5090 (32 GB)	1,792	689	193	872

The four course tracks; vendor bandwidth figures as recorded in src/data/hardware.json · none - arithmetic only, no engine was run computed from src/data/hardware.json and the Ollama tag sizes read on 2026-09-12 · Qwen3-4B, Qwen3-14B (dense) and Qwen3-30B-A3B (mixture of experts), Q4_K_M · 4,096 tokens of context · 2026-09-12

Estimates, not measurements: bandwidth divided by bytes of active weights per token. The mixture-of-experts figure scales the 19 GB file by 3.3 of 30.5 activated parameters and so ignores the attention and embedding weights read every token; its real active figure is higher and its ceiling lower. Nothing here accounts for the KV cache, kernel efficiency, a shared desktop, or a model that only just fits. Expect the measured median to land below these, often by half; a factor of five or more is a symptom, and the troubleshooting section names the usual cause.

Track S — NVIDIA DGX Spark

128 GB of unified memory, so run qwen3:4b-q4_K_M against qwen3:30b-a3b-q4_K_M. Ollama’s Linux install script selects the arm64 build on DGX OS; the Linux documentation also lists a manual ollama-linux-arm64.tar.zst archive. Both models fit in memory at the same time, so ollama ps can show both loaded after step 4, each at 100% GPU.

Track X — AMD Ryzen AI Max+ 395Partial

The Linux path is complete. On Windows 11 the native Ollama application is the accelerated path and the two Python scripts run there from PowerShell, but the shell steps on this page (the preflight, the curl request, the notebook one-liner, the validation checks and the cleanup) are written for a POSIX shell and have not been translated to or verified in PowerShell.

64 GB or 128 GB of unified memory, so run qwen3:4b-q4_K_M against qwen3:30b-a3b-q4_K_M. On Linux, Ollama’s GPU documentation lists gfx1151 (Ryzen AI Max+ 395) among the GPUs it supports through ROCm, and says a ROCm v7 driver is required; it also ships Vulkan support, enabled by default, as the alternative path, and OLLAMA_VULKAN=0 disables it. ollama ps should show 100% GPU on either path after step 4.

On Windows 11, which the course’s hardware notes list as a Strix operating system, install the native Windows application. The ollama commands on this page are the same there, and the two scripts run from PowerShell as python run-blind-comparison.py and python score-blind-comparison.py with any Python 3.9 or later, since they import only the standard library. The other shell steps (mkdir -p, df -h, curl -d with a single-quoted body, the LARGE variable, python3 -c, tail, grep -c, rm) have no PowerShell equivalent here; the Linux path is the one the course validates on this track.

A sparsely activated model such as qwen3:30b-a3b-q4_K_M suits this machine: large in memory, reading only a slice of itself per token, as the ceiling table above shows.

Track M — Apple silicon

Ollama’s download page states macOS 14 Sonoma or later, and its macOS documentation says that on first start the application offers to link the ollama command into /usr/local/bin; accept the prompt. At 32 GB and above run qwen3:4b-q4_K_M against qwen3:30b-a3b-q4_K_M. At 24 GB the arithmetic above puts the 30B model at 19.4 GB before macOS and your browser are counted, so run qwen3:14b-q4_K_M as the larger model; if you try the 30B first and the machine starts swapping, fall back to the 14B and record the substitution. The GPU path is Metal, and ollama ps should show 100% GPU.

Track N — NVIDIA desktop or laptop

Memory here means VRAM. At 24 GB or more run qwen3:4b-q4_K_M against qwen3:30b-a3b-q4_K_M, which needs 19.4 GB plus overhead and fits a 24 GB card. At 12 to 16 GB run qwen3:4b-q4_K_M against qwen3:14b-q4_K_M: the 30B model would spill into system memory over PCIe and the run would take far longer without teaching you anything extra.

The course’s Windows path is WSL2: install Ollama inside it with the Linux command and run everything as on Linux. The native Windows application is the other documented option, but the shell steps on this page are written for a POSIX shell, so follow the page inside WSL2 and do not run a native Ollama at the same time, so that ollama list and the scripts see the same server.

Tasks

0. Preflight

Confirm every prerequisite before anything is downloaded. Each command’s expected output is shown; a step that does not match is fixed from the troubleshooting section before you go on.

RunnableAll tracks

course directory, notebook and Python

mkdir -p ~/llm-course/part-03
cd ~/llm-course/part-03
ls -l ../labbook.md
python3 --version
df -h .

Output — what you should see

-rw-r--r--  1 you  staff  xxxx  Sep 12 10:00 ../labbook.md
Python 3.x.y
Filesystem      Size  Used Avail Use% Mounted on
/dev/xxx        xxxG  xxxG   xxG  xx% /

labbook.md must exist from Part 1; python3 must be 3.9 or later (any build, since the scripts import only the standard library; on native Windows the command is python); and Avail must exceed the download column of the pair you chose by a comfortable margin. If you would rather use the course environment from Part 1, source ~/llm-course/.venv/bin/activate first and use python in every command below; either works.

Then check the memory you actually have, because the pair depends on it:

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

total memory

free -g

Output — what you should see

               total        used        free      shared  buff/cache   available
Mem:             1xx           x         1xx           0           x         1xx

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

total memory (Linux)

free -g

Output — what you should see

               total        used        free      shared  buff/cache   available
Mem:              6x           x          5x           0           x          5x

On Windows 11 read the figure from Settings, System, About.

Track M — Apple silicon

RunnableTrack M · Apple silicon

total memory in gigabytes

sysctl -n hw.memsize | awk '{ printf "%.0f GB\n", $1 / 1073741824 }'

Output — what you should see

xx GB

The divisor is 1,024 cubed, so the figure matches the one in About This Mac and the tiers in the Requirements table.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

VRAM on the card

nvidia-smi --query-gpu=name,memory.total --format=csv

Output — what you should see

name, memory.total [MiB]
NVIDIA GeForce RTX xxxx, xxxxx MiB

Divide the MiB figure by 1,024 to get the tier the Requirements table uses; 12,288 MiB is 12 GB, 24,576 MiB is 24 GB.

Record in the notebook: the track, the memory figure, the pair you will run, and the free disk.

1. Install Ollama and confirm the server answers

Ollama is used here because it is the shortest path from nothing to a model generating text, and because the engines this course actually teaches come later: Part 6 is llama.cpp and Part 7 covers Ollama and LM Studio properly, including what this one-line installer just did to your machine.

RunnableAll tracks

install Ollama (Linux, macOS, WSL2)

curl -fsSL https://ollama.com/install.sh | sh

On Windows, the README gives a PowerShell equivalent:

RunnableTrack N · Windows

install Ollama (Windows)

irm https://ollama.com/install.ps1 | iex

⛨Security

Both of those pipe a script from the internet straight into a shell. That is the vendor’s documented install path, and it is also a pattern worth being deliberate about: fetch the script to a file, read it, then run it, if that is your habit. The macOS and Windows download page also offers ordinary signed installers if you would rather not pipe anything.

On Linux the installer registers a systemd service; on macOS and Windows the application runs the server. Confirm the command is on your path, that a server is answering, and which version it is:

RunnableAll tracks

version, server and model list

ollama --version
curl -s http://localhost:11434/api/version
echo
ollama list

Output — what you should see

ollama version is 0.33.3
{"version":"0.33.3"}
NAME    ID    SIZE    MODIFIED

Three things to check. The version line must be a single ollama version is x.y.z: on 0.33.3, with no server reachable, it prints two lines instead, Warning: could not connect to a running Ollama instance and Warning: client version is 0.33.3; the troubleshooting entry for that warning says how to start the server. The /api/version line must return JSON, which confirms the port the script will use; Ollama’s FAQ states that it binds 127.0.0.1 port 11434 by default. And an empty model list is the correct result at this point.

Record: the Ollama version.

2. Pull the two models and record what you got

Use the pair your memory allows, from the Requirements table.

RunnableAll tracks

pull the model pair, 24 GB and above

LARGE=qwen3:30b-a3b-q4_K_M
ollama pull qwen3:4b-q4_K_M
ollama pull "$LARGE"

RunnableAll tracks

pull the model pair, 12 to 24 GB

LARGE=qwen3:14b-q4_K_M
ollama pull qwen3:4b-q4_K_M
ollama pull "$LARGE"

The first line names the larger model in a shell variable, and every later block on this page writes it as "${LARGE:?set in step 2}": the shell substitutes the tag you chose here, and if the variable is unset, as it is in a new terminal, the command stops with LARGE: set in step 2 instead of running with an empty name. Set it again whenever you open a fresh terminal. The output blocks that follow show the 30B tag; on the 12 GB pair, read qwen3:14b-q4_K_M in its place.

The larger download takes a while. When both are present, list them and read what you have:

RunnableAll tracks

confirm the models, read the digest, the licence and the capabilities

ollama list
ollama show qwen3:4b-q4_K_M
ollama show qwen3:4b-q4_K_M --license

Output — what you should see

NAME                    ID              SIZE      MODIFIED
qwen3:30b-a3b-q4_K_M    0b28110b7a33    19 GB     x minutes ago
qwen3:4b-q4_K_M         2bfd38a7daaf    2.6 GB    x minutes ago
  Model
    architecture        qwen3
    parameters          x.xxB
    context length      40960
    embedding length    xxxx
    quantization        Q4_K_M

  Capabilities
    completion
    tools
    thinking

  Parameters
    ...

  License
    Apache License
    Version 2.0, January 2004
    ...

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/
...

The ID column is the first twelve characters of the digest. The digests shown are the ones the library listed for these tags on 12 September 2026; if yours differ, the tag has been repointed since, and your notebook should say so. ollama show confirms three things the run depends on: the quantisation is Q4_K_M for both models, the context length the model supports comfortably exceeds the 4,096 the script will allocate, and thinking is among the capabilities, which is what lets the script turn it off for both. ollama show --license prints the licence text the model was packaged with, the habit the licence lesson asked for.

Record: both tags, both ID digests, both SIZE figures.

3. Ask one question by hand, and see where the model is running

Before automating anything, see the models work and confirm they are on the accelerator. --think=false switches thinking off for the run, which is the setting the comparison uses, and --verbose prints the timing counters afterwards; both flags are in ollama run --help on 0.33.3.

RunnableAll tracks

one question, each model, thinking off, with timings

ollama run qwen3:4b-q4_K_M --think=false --verbose "In one sentence, what is a KV cache?"
ollama run "${LARGE:?set in step 2}" --think=false --verbose "In one sentence, what is a KV cache?"
ollama ps

Output — what you should see

A KV cache is ...

total duration:       x.xxs
load duration:        x.xxs
prompt eval count:    xx token(s)
prompt eval cached:   x token(s)
prompt eval duration: x.xxs
prompt eval rate:     xx.xx tokens/s
eval count:           xx token(s)
eval duration:        x.xxs
eval rate:            xx.xx tokens/s

NAME                    ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:30b-a3b-q4_K_M    0b28110b7a33    xx GB     100% GPU     xxxxx      4 minutes from now
qwen3:4b-q4_K_M         2bfd38a7daaf    x.x GB    100% GPU     xxxxx      4 minutes from now

The timing labels shown are the ones 0.33.3 prints and may differ in wording by version; the lines to find are eval count, eval duration and eval rate, which are the same counters the script records. The first response of a session is slow because the model is being loaded (the load duration line), and later ones are not. The eval rate for the smaller model is very probably higher, which is the bandwidth arithmetic from the inference lesson; if the 30B model is faster, that is the mixture-of-experts effect the same lesson predicted. ollama run allocates the server’s default context (see the memory arithmetic above), so CONTEXT and SIZE here are not the script’s allocation, and the PROCESSOR column is judged in step 4, once both models have been loaded at the context the script uses.

Record: the eval rate for each model.

4. Make the API call the script makes, by hand, to each model

The script sends one HTTP request per task and model. Send one to each model yourself so that the request and the response fields are not a black box. This is the documented non-streaming POST to /api/chat, with the sampling settings the Modelfile reference lists in options. The two ollama stop lines first unload what step 3 loaded, so that each request, which carries num_ctx 4096, loads its model at the context the script uses. The second request is the same body with the model name taken from the variable: -d @- makes curl read the body from standard input, which the <<EOF block supplies with $LARGE substituted.

RunnableAll tracks

one chat request per model by hand, and the decode speed from the counters

ollama stop qwen3:4b-q4_K_M
ollama stop "${LARGE:?set in step 2}"
curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:4b-q4_K_M",
  "messages": [{"role": "user", "content": "In one sentence, what is a KV cache?"}],
  "stream": false,
  "think": false,
  "options": {"temperature": 0, "seed": 1, "num_predict": 256, "num_ctx": 4096}
}' > one-response.json
curl -s http://localhost:11434/api/chat -d @- > one-response-large.json <<EOF
{
  "model": "$LARGE",
  "messages": [{"role": "user", "content": "In one sentence, what is a KV cache?"}],
  "stream": false,
  "think": false,
  "options": {"temperature": 0, "seed": 1, "num_predict": 256, "num_ctx": 4096}
}
EOF
python3 -c "import json, sys
for f in sys.argv[1:]:
    d = json.load(open(f))
    print(d['model'], '->', d['message']['content'])
    print({k: d[k] for k in ('done_reason', 'prompt_eval_count', 'eval_count', 'eval_duration')})
    print('decode tokens/s:', round(d['eval_count'] / d['eval_duration'] * 1e9, 1))
" one-response.json one-response-large.json
ollama ps

Output — what you should see

qwen3:4b-q4_K_M -> A KV cache is ...
{'done_reason': 'stop', 'prompt_eval_count': xx, 'eval_count': xx, 'eval_duration': xxxxxxxxxx}
decode tokens/s: xx.x
qwen3:30b-a3b-q4_K_M -> A KV cache is ...
{'done_reason': 'stop', 'prompt_eval_count': xx, 'eval_count': xx, 'eval_duration': xxxxxxxxxx}
decode tokens/s: xx.x
NAME                    ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:30b-a3b-q4_K_M    0b28110b7a33    xx GB     100% GPU     4096       4 minutes from now
qwen3:4b-q4_K_M         2bfd38a7daaf    x.x GB    100% GPU     4096       4 minutes from now

The counters are documented: eval_count is the number of tokens in the response, eval_duration is the time in nanoseconds spent generating it, and the API documentation gives the decode speed as eval_count / eval_duration * 10^9, which is the line the script and the scoring script both use. done_reason is not enumerated in the API documentation; the source of 0.33.3 gives stop when the model finished and length when num_predict cut it off, and the script flags every length so a truncated answer is never mistaken for a short one. With think false there is no thinking field in the message; with it true, the thinking documentation says the trace arrives in message.thinking and the answer in message.content, which is how the script keeps them apart.

The settings are the experiment’s controls. Temperature zero with a fixed seed makes a rerun reproduce the run, so that the comparison is not partly a lottery. The Modelfile reference documents the defaults the script is overriding: temperature 0.8, seed 0, num_predict −1 for unlimited generation, and num_ctx 2048. The script’s cap of 1,024 generated tokens is far above any answer these tasks call for in non-thinking mode. The Qwen3 card recommends temperature 0.7, top_p 0.8, top_k 20 and min_p 0 in non-thinking mode; --temperature 0.7 --top-p 0.8 --top-k 20 --min-p 0 on the run script restores the card’s setting for both models if you prefer it to greedy decoding.

ollama ps now decides whether the run is a fair one. CONTEXT reads 4096 because each request carried num_ctx 4096; the FAQ documents 100% GPU as the model entirely on the accelerator, 100% CPU as entirely in system memory, and a split such as 48%/52% CPU/GPU as a model that did not fit. SIZE is your first measurement of the memory arithmetic: weights plus the 4,096-token cache plus the engine’s overhead, so its difference from the file size in step 2 is what the cache and overhead cost. On a 12 GB card the pair does not fit together (3.2 GB plus 10.0 GB); at 16 GB it may; so expect either one line for qwen3:14b-q4_K_M or two lines with the earlier model split, and judge the line for the model just used. The run script asks one model at a time, so that is fine.

Record: the PROCESSOR column for each model, the SIZE column next to the file size from step 2, and the decode speed the one-liner printed for each. Delete one-response.json and one-response-large.json or keep them as a reference.

=Expected result

Both models answer in English, on topic, in roughly a sentence, done_reason is stop for both, and the ollama ps line for each model, loaded at 4096, says 100% GPU. If either says CPU or a split, stop here and use the troubleshooting entry: a comparison in which one model ran on the accelerator and the other did not measures the machine, not the models.

5. Take the task set, read it, and let the script check it

Everything the comparison runs is in one file. Download it into ~/llm-course/part-03 and read it before you run anything: the rubrics are the part that decides what the result means, and you should disagree with any you think are wrong now, while changing them is still honest.

Fragment — not complete on its own

tasks.json

{
  "$comment": "Task set for the Part 3 reality check: 'a small local model is as good as the frontier'. Twenty tasks in five categories, four each, chosen so that a difference between models could show up if there is one. Every task has a rubric that a person can apply without running anything, scored 0, 1 or 2. Edit or replace these tasks freely: the scripts read this file and do not care what is in it, and a task set built from your own work is more informative than this one.",
  "version": 1,
  "scoring": {
    "max_score": 2,
    "preregistered_margin": 4,
    "margin_note": "The margin, in points out of 40, within which two models count as indistinguishable on this task set. It is written here, before any answer exists, so that it cannot be chosen to fit the result. Change it only before you run, and say why in your notebook.",
    "levels": {
      "0": "Wrong, unusable, or ignores what was asked.",
      "1": "Partly right: the right idea with a defect, or the right answer with the wrong shape.",
      "2": "Fully right, in the form the task asked for."
    },
    "rules": [
      "Score the answer in front of you, not the model you think produced it. That is what the blind step is for.",
      "Judge against the rubric, not against your own preferred answer.",
      "Do not award marks for length, confidence or formatting the task did not ask for.",
      "If a rubric is ambiguous for a particular answer, write down how you resolved it and apply the same resolution to every model."
    ]
  },
  "categories": [
    "factual-recall",
    "arithmetic-reasoning",
    "code",
    "summarisation",
    "instruction-following"
  ],
  "tasks": [
    {
      "id": "fr-01",
      "category": "factual-recall",
      "prompt": "What do the letters in the acronym HTTP stand for? Answer in one line, with no explanation.",
      "rubric": "2 = gives HyperText Transfer Protocol (any capitalisation or spacing of 'hypertext'). 1 = three of the four words correct, or the right expansion buried in extra commentary. 0 = anything else.",
      "max_score": 2
    },
    {
      "id": "fr-02",
      "category": "factual-recall",
      "prompt": "Name the four nucleobases found in DNA. List them and nothing else.",
      "rubric": "2 = adenine, guanine, cytosine and thymine, all four, with no extras. Uracil is an extra and costs the point. 1 = three of the four correct with no more than one wrong addition. 0 = anything else.",
      "max_score": 2
    },
    {
      "id": "fr-03",
      "category": "factual-recall",
      "prompt": "Name the seven SI base units and the quantity each one measures.",
      "rubric": "2 = metre (length), kilogram (mass), second (time), ampere (electric current), kelvin (thermodynamic temperature), mole (amount of substance) and candela (luminous intensity), all seven with the right quantity against each. 1 = five or six units correct with their quantities. 0 = four or fewer, or units paired with the wrong quantities.",
      "max_score": 2
    },
    {
      "id": "fr-04",
      "category": "factual-recall",
      "prompt": "Which planets of the Solar System have no natural satellites? Name them and nothing else.",
      "rubric": "2 = Mercury and Venus, and only those two. 1 = one of the two named with no incorrect planets added. 0 = anything else, including adding a planet that does have moons.",
      "max_score": 2
    },
    {
      "id": "ar-01",
      "category": "arithmetic-reasoning",
      "prompt": "A train leaves at 09:47 and the journey takes 2 hours and 38 minutes. At what time does it arrive? Give the answer as a 24-hour clock time.",
      "rubric": "2 = 12:25. 1 = correct method visible but an arithmetic slip of no more than a few minutes. 0 = anything else.",
      "max_score": 2
    },
    {
      "id": "ar-02",
      "category": "arithmetic-reasoning",
      "prompt": "Compute 17 x 24 - 96 / 8, following the usual order of operations. Show your working, then give the final answer on its own line.",
      "rubric": "2 = 396, with working that shows 17 x 24 = 408 and 96 / 8 = 12. 1 = the right final answer with no working, or correct working with a copying error in the final line. 0 = anything else, including 39 (which comes from working left to right).",
      "max_score": 2
    },
    {
      "id": "ar-03",
      "category": "arithmetic-reasoning",
      "prompt": "A box contains 3 red balls and 5 blue balls. Two balls are drawn at random without replacement. What is the probability that both are blue? Give an exact fraction in lowest terms.",
      "rubric": "2 = 5/14, from (5/8) x (4/7). An equivalent unreduced fraction such as 20/56 with the reduction shown also scores 2. 1 = the correct method stated but the arithmetic wrong, or 25/64 (which is the with-replacement answer) accompanied by the correct without-replacement reasoning. 0 = anything else.",
      "max_score": 2
    },
    {
      "id": "ar-04",
      "category": "arithmetic-reasoning",
      "prompt": "A jacket costs 48 pounds after a 20% discount. What was the price before the discount? Give the answer to the nearest penny.",
      "rubric": "2 = 60 pounds, from 48 / 0.8. 1 = the correct method (dividing by 0.8, not adding 20%) with an arithmetic error. 0 = 57.60 (which comes from adding 20% to 48) or anything else.",
      "max_score": 2
    },
    {
      "id": "co-01",
      "category": "code",
      "prompt": "Write a Python function with the signature is_palindrome(s: str) -> bool. It returns True if s reads the same forwards and backwards, ignoring case and ignoring every character that is not a letter or a digit. Use only the Python standard library. Return the function and nothing else.",
      "rubric": "2 = the code runs as written and returns True for 'A man, a plan, a canal: Panama', False for 'hello', and True for the empty string. 1 = the logic is right but the code has one small defect, such as not handling the empty string or a missing import. 0 = it does not run, or it does not ignore case and punctuation.",
      "max_score": 2
    },
    {
      "id": "co-02",
      "category": "code",
      "prompt": "The Python function below is meant to return the sum of a list of numbers, but it returns the wrong answer. Say what the bug is in one sentence, then give the corrected function.\n\ndef total(xs):\n    result = 1\n    for x in xs:\n        result += x\n    return result",
      "rubric": "2 = identifies that the accumulator starts at 1 instead of 0, and returns a corrected function that starts at 0. 1 = fixes the code without naming the bug, or names the bug without giving working code. 0 = misidentifies the bug, or the corrected function is still wrong.",
      "max_score": 2
    },
    {
      "id": "co-03",
      "category": "code",
      "prompt": "Write a single shell pipeline that prints the five largest regular files under the current directory, with human-readable sizes, largest first. It must work on Linux and macOS. Return the pipeline and nothing else.",
      "rubric": "2 = one pipeline that considers regular files only (not directories), sorts by size descending and stops at five, using only options that exist in the tools it calls. 1 = the right shape but includes directories in the result, or passes an option the tool does not have. 0 = does not sort by size, does not limit to five, or would not run.",
      "max_score": 2
    },
    {
      "id": "co-04",
      "category": "code",
      "prompt": "Given the tables customers(id, name) and orders(id, customer_id, placed_on DATE), write a SQL query returning each customer's id, name and the number of orders they placed in 2025, including customers who placed none. Return the query and nothing else.",
      "rubric": "2 = a LEFT JOIN from customers to orders with the 2025 condition in the join (or in a subquery), a GROUP BY on the customer, and a COUNT of an orders column rather than COUNT(*), so that customers with no 2025 orders come back with zero. 1 = a LEFT JOIN with the date condition in the WHERE clause, or COUNT(*), either of which quietly drops or miscounts the customers with no orders. 0 = an inner join, or no grouping.",
      "max_score": 2
    },
    {
      "id": "su-01",
      "category": "summarisation",
      "prompt": "Summarise the passage below in one sentence of no more than 25 words, naming what replaced what and the main result. Output the sentence only.\n\nThe Kirkmoor village water scheme was completed in March after eleven months of work. Two boreholes replaced a surface intake that had failed four times in the previous decade, usually after heavy rain washed silt into the filters. The parish council raised the money in two parts: a regional grant covered rather more than half, and a loan repaid through water charges covered the rest. Since commissioning, turbidity readings have stayed within limits through two storms that would previously have forced a boil notice. The engineers noted that the pumps are oversized for present demand, which costs a little electricity now but leaves capacity for the forty houses approved on the eastern edge of the village.",
      "rubric": "2 = one sentence of 25 words or fewer that says two boreholes replaced a failing surface intake and that water quality has held through storms since. 1 = one of those two facts, or both facts but over the word limit or in more than one sentence. 0 = neither fact, or a summary that introduces something the passage does not say.",
      "max_score": 2
    },
    {
      "id": "su-02",
      "category": "summarisation",
      "prompt": "Summarise the passage below in no more than three bullet points: what was trialled, what improved, and what limited the improvement. Output the bullets only.\n\nA regional distribution centre trialled twelve autonomous shuttles on its slowest aisle for six weeks. The shuttles moved cases from the pick face to the packing benches, a job previously done by four staff with pallet trucks. Throughput on that aisle rose by about a fifth, but the gain was eaten by a bottleneck at the packing benches, which had never been the constraint before. Two shuttles were taken out of service after they stalled on a floor joint that was within specification but at the limit of it. Staff were redeployed rather than released, and the site manager's report recommends fixing the floor and rebalancing the benches before extending the trial.",
      "rubric": "2 = three bullets or fewer covering all of: the twelve-shuttle trial on one aisle, the throughput rise of about a fifth, and the packing-bench bottleneck that absorbed the gain. 1 = two of those three. 0 = one or none, or more than three bullets.",
      "max_score": 2
    },
    {
      "id": "su-03",
      "category": "summarisation",
      "prompt": "Summarise the passage below in exactly two sentences: the first saying what was done, the second saying what is still outstanding. Output the two sentences only.\n\nOver the winter the county library service moved its catalogue from a system it had run since 2003 to a hosted replacement. The migration itself took a single weekend, but the preparation took seven months, most of it spent reconciling records for items that existed in the old system twice under different spellings. Roughly 40,000 of 1.1 million records needed manual attention. Borrowing was unaffected because the old system stayed available in read-only mode for a fortnight. The service reports that searches now return results faster and that staff spend less time on catalogue maintenance, but that three reports the finance team relied on have no equivalent in the new system and are being rebuilt.",
      "rubric": "2 = exactly two sentences; the first names the catalogue migration to a hosted system, the second names the three finance reports with no equivalent that are being rebuilt. 1 = both facts present but not in exactly two sentences, or two sentences carrying only one of the facts. 0 = neither fact, or a summary that states something the passage does not.",
      "max_score": 2
    },
    {
      "id": "su-04",
      "category": "summarisation",
      "prompt": "Write a single sentence of no more than 30 words that states how the scheme grew and what condition is attached to the contract extension. Output the sentence only.\n\nThe city's bicycle hire scheme finished its second year with 480 bicycles at 62 docking stations. Journeys rose by a third on the first year, and the average journey shortened from 2.4 kilometres to 1.9, which the operator attributes to new stations in the centre rather than to a change in who is riding. Maintenance is the largest cost and rose faster than journeys, driven by vandalism at four stations that account for more than half of all damage. The council has extended the contract by two years on condition that those four stations are relocated or better lit, and that the operator publishes monthly availability figures.",
      "rubric": "2 = one sentence of 30 words or fewer that says journeys rose by about a third and that the two-year extension depends on dealing with the four vandalised stations, or on publishing monthly availability. 1 = one of the two facts, or both but over the limit or in more than one sentence. 0 = neither fact.",
      "max_score": 2
    },
    {
      "id": "if-01",
      "category": "instruction-following",
      "prompt": "Reply with exactly three bullet points on reasons to keep a backup of your files. Each bullet must be fewer than ten words. Write nothing before the bullets and nothing after them.",
      "rubric": "2 = exactly three bullets, every one under ten words, and no other text of any kind (no preamble, no closing line, no heading). 1 = exactly one of those three rules broken. 0 = two or more broken.",
      "max_score": 2
    },
    {
      "id": "if-02",
      "category": "instruction-following",
      "prompt": "Reply with valid JSON and nothing else: no code fence, no commentary, no leading or trailing text. The object must have exactly the keys \"name\", \"year\" and \"tags\", where name is a string, year is an integer, and tags is a list of exactly three strings. Choose any values you like.",
      "rubric": "2 = the whole reply parses as JSON, has exactly those three keys, the types are right, tags holds exactly three strings, and there is no code fence or commentary. 1 = exactly one of those conditions broken (most often a code fence around otherwise correct JSON). 0 = two or more broken, or the reply does not parse.",
      "max_score": 2
    },
    {
      "id": "if-03",
      "category": "instruction-following",
      "prompt": "Rewrite this sentence in the passive voice. Output only the rewritten sentence, with no explanation and no quotation marks: The committee approved the revised budget on Friday.",
      "rubric": "2 = a grammatical passive rewrite, such as 'The revised budget was approved by the committee on Friday', and nothing else in the reply. 1 = a correct passive rewrite accompanied by an explanation, or a reply with no explanation whose sentence is not actually passive. 0 = neither passive nor clean.",
      "max_score": 2
    },
    {
      "id": "if-04",
      "category": "instruction-following",
      "prompt": "List five cities in the United Kingdom in alphabetical order, numbered 1. to 5., one per line, with nothing else in the reply.",
      "rubric": "2 = exactly five lines, numbered 1. to 5., every entry a UK city, in alphabetical order, and no other text. 1 = exactly one of those rules broken (out of order, four or six entries, or a stray line of commentary). 0 = two or more broken, or an entry that is not a UK city.",
      "max_score": 2
    }
  ]
}

Download tasks.json169 lines

Twenty tasks, four in each of five categories, and a scoring block that carries the margin. Then download the run script into the same directory as tasks.json and let it check the file, the server and the models without sending a single prompt:

RunnableAll tracks

run-blind-comparison.py

"""Run every task in tasks.json against two or more models and write the answers out blind.

Purpose: produce a scoring sheet in which every answer carries a random identifier and
         nothing else, so the reality check's twenty tasks can be marked without knowing
         which model wrote which answer. Local models are called through Ollama's HTTP
         API on this machine; answers produced elsewhere can be supplied as a JSON file.
Platform: all (spark, strix, mac, nvidia). Pure standard library: no packages to install.
Minimum memory: 12 GB, enough for a 4B-class and a 14B-class model one at a time. The
         30B-class comparison wants 24 GB or more.
Assumes: Ollama is reachable at $OLLAMA_HOST or http://localhost:11434, and every model
         named with --model has already been pulled with `ollama pull`. Writes three files
         into the output directory and overwrites them if they exist.

What it sends: one non-streaming POST /api/chat per task and model, with the prompt as a
single user message, `think` set to false unless --think is given, and an `options` object
that fixes the sampling settings so a rerun reproduces the run. What it keeps: the answer
text, the length of any thinking trace, `done_reason`, and Ollama's token and timing counters.

Usage: python run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:30b-a3b-q4_K_M
       python run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:14b-q4_K_M --dry-run
       python run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:14b-q4_K_M \\
           --external frontier=frontier-answers.json --out-dir .
"""
import argparse
import json
import os
import random
import re
import secrets
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path

DEFAULT_HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
REQUIRED_TASK_FIELDS = ("id", "category", "prompt", "rubric")
COUNTERS = ("prompt_eval_count", "prompt_eval_duration", "eval_count", "eval_duration",
            "total_duration", "load_duration")

# Sampling settings. Non-thinking runs are greedy (temperature 0) so that a rerun gives the
# same answers. The Qwen3 model card says not to use greedy decoding in thinking mode and
# recommends temperature 0.6, top_p 0.95, top_k 20, min_p 0 there, so --think switches to
# those unless you set the values yourself.
GREEDY = {"temperature": 0.0}
THINKING_SAMPLING = {"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0}


def normalise_host(host: str) -> str:
    """Ollama's own OLLAMA_HOST is often set as host:port with no scheme."""
    host = host.strip().rstrip("/")
    if not host.startswith(("http://", "https://")):
        host = f"http://{host}"
    return host


def post_json(url: str, payload: dict, timeout: float) -> dict:
    data = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def get_json(url: str, timeout: float) -> dict:
    with urllib.request.urlopen(url, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def server_version(host: str, timeout: float) -> str:
    return str(get_json(f"{host}/api/version", timeout).get("version", "unknown"))


def installed_models(host: str, timeout: float) -> dict:
    """Model name -> {digest, size} for everything Ollama has locally, from GET /api/tags."""
    tags = get_json(f"{host}/api/tags", timeout)
    out = {}
    for m in tags.get("models", []):
        out[m.get("name", "")] = {"digest": str(m.get("digest", ""))[:12], "size": m.get("size")}
    return out


def resolve_model(name: str, present: dict):
    """The name as Ollama lists it: `qwen3:4b` is stored as itself, `qwen3` as `qwen3:latest`."""
    for candidate in (name, f"{name}:latest"):
        if candidate in present:
            return candidate
    return None


def strip_inline_thinking(text: str) -> tuple:
    """Remove a leading <think>...</think> block if an engine put the trace in the answer text."""
    match = re.match(r"^\s*<think>.*?</think>\s*", text, flags=re.DOTALL)
    if not match:
        return text, 0
    return text[match.end():], len(match.group(0))


def ask(host: str, model: str, prompt: str, options: dict, think, timeout: float) -> dict:
    """One non-streaming POST to /api/chat. Returns the answer text and the counters."""
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": options,
    }
    if think is not None:
        payload["think"] = think
    started = time.monotonic()
    try:
        body = post_json(f"{host}/api/chat", payload, timeout)
        think_sent = think
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", errors="replace")
        # A model whose template has no thinking support may reject the field; run it without.
        if exc.code == 400 and think is not None and "think" in detail.lower():
            payload.pop("think")
            body = post_json(f"{host}/api/chat", payload, timeout)
            think_sent = "omitted"
        else:
            raise urllib.error.URLError(f"HTTP {exc.code}: {detail[:200]}") from exc
    wall = time.monotonic() - started

    message = body.get("message", {})
    text, inline_chars = strip_inline_thinking(message.get("content", "") or "")
    thinking_chars = len(message.get("thinking", "") or "") + inline_chars
    result = {
        "text": text.strip(),
        "thinking_chars": thinking_chars,
        "done_reason": body.get("done_reason"),
        "think": think_sent,
        "wall_seconds": round(wall, 3),
    }
    for key in COUNTERS:
        result[key] = body.get(key)
    return result


def empty_result(text: str) -> dict:
    result = {"text": text, "thinking_chars": 0, "done_reason": None, "think": None,
              "wall_seconds": None}
    for key in COUNTERS:
        result[key] = None
    return result


def load_tasks(path: Path) -> dict:
    """Read the task file and refuse to run on one that is malformed."""
    try:
        doc = json.loads(path.read_text(encoding="utf-8"))
    except FileNotFoundError:
        raise SystemExit(f"{path} not found. Download tasks.json into this directory first.")
    except json.JSONDecodeError as exc:
        raise SystemExit(f"{path} is not valid JSON: {exc}")
    tasks = doc.get("tasks")
    if not isinstance(tasks, list) or not tasks:
        raise SystemExit(f"{path} has no tasks in it.")
    seen = set()
    for i, task in enumerate(tasks, start=1):
        missing = [f for f in REQUIRED_TASK_FIELDS if not task.get(f)]
        if missing:
            raise SystemExit(f"{path}: task {i} is missing {', '.join(missing)}.")
        if task["id"] in seen:
            raise SystemExit(f"{path}: task id {task['id']!r} appears twice.")
        seen.add(task["id"])
        if not all(f"{level} =" in task["rubric"] for level in ("0", "1", "2")):
            raise SystemExit(
                f"{path}: task {task['id']} has a rubric without explicit '2 =', '1 =' and '0 =' "
                "levels. Every rubric must say what each score looks like before any answer exists."
            )
    return doc


def describe_tasks(doc: dict) -> None:
    tasks = doc["tasks"]
    counts = {}
    for task in tasks:
        counts[task["category"]] = counts.get(task["category"], 0) + 1
    max_score = doc.get("scoring", {}).get("max_score", 2)
    margin = doc.get("scoring", {}).get("preregistered_margin")
    print(f"tasks: {len(tasks)} in {len(counts)} categories, {max_score} points each, "
          f"{len(tasks) * max_score} possible", file=sys.stderr)
    for category, n in counts.items():
        print(f"  {category:<22} {n:>2} tasks  {n * max_score:>2} points", file=sys.stderr)
    if margin is not None:
        print(f"pre-registered margin: {margin} points", file=sys.stderr)


def load_external(spec: str) -> tuple:
    """--external NAME=FILE, where FILE maps task id to answer text."""
    if "=" not in spec:
        raise SystemExit(f"--external wants NAME=FILE, got {spec!r}")
    name, _, filename = spec.partition("=")
    try:
        answers = json.loads(Path(filename).read_text(encoding="utf-8"))
    except FileNotFoundError:
        raise SystemExit(f"--external: {filename} not found.")
    except json.JSONDecodeError as exc:
        raise SystemExit(f"--external: {filename} is not valid JSON: {exc}")
    if not isinstance(answers, dict):
        raise SystemExit(f"{filename} must be a JSON object mapping task id to answer text.")
    return name.strip(), answers


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--tasks", default="tasks.json", help="task file to read")
    parser.add_argument("--model", action="append", default=[], metavar="NAME",
                        help="an Ollama model to run; repeat for each model compared")
    parser.add_argument("--external", action="append", default=[], metavar="NAME=FILE",
                        help="answers produced elsewhere, as a JSON object of task id to text")
    parser.add_argument("--host", default=DEFAULT_HOST, help="Ollama base URL")
    parser.add_argument("--out-dir", default=".", help="directory for the three output files")
    parser.add_argument("--think", action="store_true",
                        help="run every model in thinking mode (default: thinking off for all)")
    parser.add_argument("--temperature", type=float, default=None,
                        help="sampling temperature (default 0 without --think, 0.6 with it)")
    parser.add_argument("--top-p", type=float, default=None, help="nucleus sampling cut-off")
    parser.add_argument("--top-k", type=int, default=None, help="top-k sampling cut-off")
    parser.add_argument("--min-p", type=float, default=None, help="min-p sampling cut-off")
    parser.add_argument("--seed", type=int, default=1, help="sampling seed passed to Ollama")
    parser.add_argument("--num-predict", type=int, default=1024,
                        help="cap on generated tokens per answer, thinking included")
    parser.add_argument("--num-ctx", type=int, default=4096, help="context length to allocate")
    parser.add_argument("--timeout", type=float, default=900.0, help="seconds per request")
    parser.add_argument("--shuffle-seed", type=int, default=None,
                        help="seed for the blinding shuffle; omit for an unpredictable one")
    parser.add_argument("--dry-run", action="store_true",
                        help="check the task file, the server and the models, then stop")
    args = parser.parse_args()

    host = normalise_host(args.host)
    externals = [load_external(s) for s in args.external]
    if len(args.model) + len(externals) < 2:
        raise SystemExit("Give at least two answer sources: two --model values, or one of each.")

    doc = load_tasks(Path(args.tasks))
    tasks = doc["tasks"]
    describe_tasks(doc)

    sampling = dict(THINKING_SAMPLING if args.think else GREEDY)
    for key, value in (("temperature", args.temperature), ("top_p", args.top_p),
                       ("top_k", args.top_k), ("min_p", args.min_p)):
        if value is not None:
            sampling[key] = value
    options = {**sampling, "seed": args.seed, "num_predict": args.num_predict,
               "num_ctx": args.num_ctx}
    think = True if args.think else False

    version = None
    model_info = {}
    if args.model:
        try:
            version = server_version(host, timeout=30)
            present = installed_models(host, timeout=30)
        except (urllib.error.URLError, TimeoutError, OSError) as exc:
            raise SystemExit(
                f"Cannot reach Ollama at {host} ({exc}). Start it with `ollama serve`, "
                "or set --host if it runs elsewhere."
            ) from exc
        print(f"ollama server version {version} at {host}", file=sys.stderr)
        missing = []
        for name in args.model:
            resolved = resolve_model(name, present)
            if resolved is None:
                missing.append(name)
                continue
            info = present[resolved]
            model_info[name] = {"listed_as": resolved, **info}
            size_gb = (info["size"] or 0) / 1e9
            print(f"  {name:<28} digest {info['digest']}  {size_gb:5.1f} GB on disk",
                  file=sys.stderr)
        if missing:
            raise SystemExit(
                "These models are not present locally: " + ", ".join(missing)
                + "\nPull each one first, for example: ollama pull " + missing[0]
            )
    for name, mapping in externals:
        supplied = sum(1 for task in tasks if task["id"] in mapping)
        print(f"  {name:<28} external file, answers for {supplied}/{len(tasks)} tasks",
              file=sys.stderr)

    print(f"think: {think}   options: {json.dumps(options)}", file=sys.stderr)
    if args.dry_run:
        print("dry run: nothing was sent to a model.", file=sys.stderr)
        return

    answers = []
    for model in args.model:
        print(f"\n{model}", file=sys.stderr)
        for i, task in enumerate(tasks, start=1):
            print(f"  [{i:2d}/{len(tasks)}] {task['id']}", end="", file=sys.stderr, flush=True)
            try:
                result = ask(host, model, task["prompt"], options, think, args.timeout)
            except (urllib.error.URLError, TimeoutError, OSError) as exc:
                print(f"  FAILED: {exc}", file=sys.stderr)
                result = empty_result(f"[no answer: {exc}]")
            else:
                flag = ""
                if result["done_reason"] == "length":
                    flag = "  CUT OFF by num_predict"
                elif not result["text"]:
                    flag = "  EMPTY answer"
                print(f"  {result['wall_seconds']:6.1f} s  {result['eval_count'] or 0:5d} tok"
                      f"  {result['done_reason']}{flag}", file=sys.stderr)
            answers.append({"task": task["id"], "source": model, **result})

    for name, mapping in externals:
        for task in tasks:
            text = str(mapping.get(task["id"], "[no answer supplied]")).strip()
            answers.append({"task": task["id"], "source": name, **empty_result(text)})

    rng = random.Random(args.shuffle_seed) if args.shuffle_seed is not None else random.SystemRandom()
    for answer in answers:
        answer["response_id"] = secrets.token_hex(3)

    by_task = {task["id"]: [] for task in tasks}
    for answer in answers:
        by_task[answer["task"]].append(answer)
    for group in by_task.values():
        rng.shuffle(group)

    out_dir = Path(args.out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    blind_path = out_dir / "blind-responses.md"
    key_path = out_dir / "blind-key.json"
    scores_path = out_dir / "scores.csv"

    lines = [
        "# Blind responses",
        "",
        "Score every response below against its task's rubric, 0, 1 or 2, and write the score",
        "into scores.csv next to the matching response id. Do not open blind-key.json until you",
        "have finished scoring: it says which model wrote which answer.",
        "",
    ]
    for task in tasks:
        lines += [
            f"## {task['id']} - {task['category']}",
            "",
            "**Prompt**",
            "",
            "```text",
            task["prompt"].rstrip(),
            "```",
            "",
            f"**Rubric** {task['rubric']}",
            "",
        ]
        for answer in by_task[task["id"]]:
            lines += [
                f"### Response {answer['response_id']}",
                "",
                "```text",
                (answer["text"] or "[empty answer]").rstrip(),
                "```",
                "",
            ]
    blind_path.write_text("\n".join(lines) + "\n", encoding="utf-8")

    kept = ("task", "source", "thinking_chars", "done_reason", "think", "wall_seconds") + COUNTERS
    key_path.write_text(json.dumps({
        "generated": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "host": host,
        "ollama_version": version,
        "tasks_file": str(args.tasks),
        "preregistered_margin": doc.get("scoring", {}).get("preregistered_margin"),
        "think": think,
        "options": options,
        "sources": args.model + [name for name, _ in externals],
        "models": model_info,
        "responses": {a["response_id"]: {k: a[k] for k in kept} for a in answers},
    }, indent=2) + "\n", encoding="utf-8")

    ordered = [a["response_id"] for task in tasks for a in by_task[task["id"]]]
    scores_path.write_text(
        "response_id,score\n" + "".join(f"{rid},\n" for rid in ordered), encoding="utf-8")

    cut = sum(1 for a in answers if a["done_reason"] == "length")
    empty = sum(1 for a in answers if not a["text"] or a["text"].startswith("[no answer"))
    print(f"\nwrote {blind_path}  ({len(answers)} responses)", file=sys.stderr)
    print(f"wrote {key_path}  (do not read it until you have scored)", file=sys.stderr)
    print(f"wrote {scores_path}  (fill in the score column)", file=sys.stderr)
    if cut:
        print(f"\n{cut} answer(s) were cut off by --num-predict {args.num_predict}. Score them "
              "as they are, or rerun every model with a higher cap so the comparison stays fair.",
              file=sys.stderr)
    if empty:
        print(f"{empty} answer(s) are empty or failed; see the troubleshooting section.",
              file=sys.stderr)


if __name__ == "__main__":
    main()

Download run-blind-comparison.py393 lines

RunnableAll tracks

dry run: check the task file, the server and the models

python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model "${LARGE:?set in step 2}" --dry-run

Output — what you should see

tasks: 20 in 5 categories, 2 points each, 40 possible
  factual-recall          4 tasks   8 points
  arithmetic-reasoning    4 tasks   8 points
  code                    4 tasks   8 points
  summarisation           4 tasks   8 points
  instruction-following   4 tasks   8 points
pre-registered margin: 4 points
ollama server version 0.33.3 at http://localhost:11434
  qwen3:4b-q4_K_M              digest 2bfd38a7daaf    2.6 GB on disk
  qwen3:30b-a3b-q4_K_M         digest 0b28110b7a33   19.0 GB on disk
think: False   options: {"temperature": 0.0, "seed": 1, "num_predict": 1024, "num_ctx": 4096}
dry run: nothing was sent to a model.

The dry run stops with a message if a task is missing a field, a rubric does not spell out its three levels (an implicit level is one you would fill in from the answer), the server is unreachable, or a model has not been pulled; each message says what to do.

6. Pre-register the margin

The margin is already in tasks.json as preregistered_margin: 4, where the scoring script reads it. Pre-registration puts it in your notebook too, dated, before the first answer exists:

RunnableAll tracks

write the pre-registration into the notebook before running

python3 -c "import json, datetime; open('../labbook.md', 'a').write(json.dumps({'lab': 'part-03/reality-check-small-local-versus-frontier', 'preregistration': True, 'date': str(datetime.date.today()), 'models': ['qwen3:4b-q4_K_M', '${LARGE:?set in step 2}'], 'margin_points': 4, 'tasks': 20, 'think': False}) + '\n')"
tail -n 1 ../labbook.md

Output — what you should see

{"lab": "part-03/reality-check-small-local-versus-frontier", "preregistration": true, "date": "2026-xx-xx", "models": ["qwen3:4b-q4_K_M", "qwen3:30b-a3b-q4_K_M"], "margin_points": 4, "tasks": 20, "think": false}

If you changed the margin or swapped tasks in step 5, change the line to match, and add a "why" field. What you may not do is come back to this line afterwards.

7. Optionally, add a frontier model

If you have access to a hosted assistant, you can add it as a third column. Paste each task’s prompt into it, copy the answer back, and save the answers as a JSON object of task id to answer text:

Pseudocode — not a real command

{
  "fr-01": "the answer the hosted model gave to fr-01",
  "fr-02": "...",
  ... one entry per task id in tasks.json ...
}

Save it as frontier-answers.json in the same directory. Use the same prompt text with no additions and a fresh conversation for each task, or the comparison is not like for like. A task you skip appears in the blind file as [no answer supplied] and is scored 0, which is the honest consequence.

If you have no hosted model, skip this step; the local pair is the part that tells you most about your own machine.

✕Common mistake

Do not use a hosted model to score the answers on this page. A model grading models is a real technique with real pitfalls, and Part 16 teaches it properly with its biases named. Here you are the judge, because the point of the exercise is to practise judging.

8. Run the comparison

RunnableAll tracks

run the twenty tasks against both models

python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model "${LARGE:?set in step 2}"

Add your hosted answers if you made them:

RunnableAll tracks

run with a third column from a hosted model

python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model "${LARGE:?set in step 2}" \
    --external frontier=frontier-answers.json

It repeats the dry-run checks, then prints one line per task as it goes and writes three files:

Output — what you should see

tasks: 20 in 5 categories, 2 points each, 40 possible
  ...
think: False   options: {"temperature": 0.0, "seed": 1, "num_predict": 1024, "num_ctx": 4096}

qwen3:4b-q4_K_M
  [ 1/20] fr-01     x.x s     xx tok  stop
  [ 2/20] fr-02     x.x s     xx tok  stop
  ...
  [20/20] if-04     x.x s     xx tok  stop

qwen3:30b-a3b-q4_K_M
  [ 1/20] fr-01     x.x s     xx tok  stop
  ...

wrote blind-responses.md  (40 responses)
wrote blind-key.json  (do not read it until you have scored)
wrote scores.csv  (fill in the score column)

Each progress line carries the wall-clock seconds, the generated token count and the done_reason. Three things can appear after it and each has a consequence. CUT OFF by num_predict means the answer hit the cap; the script says so again at the end, and the fix is to rerun both models with --num-predict 2048, not to score a truncated answer against a complete one. EMPTY answer means the model returned nothing, which on a hybrid model with thinking off is rare and worth noting. FAILED: with an error means the request itself did not complete, the answer is recorded as [no answer: ...], and the troubleshooting section has the causes.

The first line of each model is slow because it includes the load. When the run finishes, blind-responses.md has every task once with its prompt, its rubric and the answers under random identifiers in a random order; blind-key.json has the mapping, the digests, the options and every counter; scores.csv has one row per response with an empty score column.

Record: nothing yet. Do not open blind-key.json.

9. Score blind

Open blind-responses.md in an editor, and scores.csv next to it. Score each response 0, 1 or 2 against its rubric and write the number into the score column:

Output — what you should see

response_id,score
99137d,2
0c4ac4,1
731ae1,2

Work one task at a time across all of its responses, apply the rubric as written, and when a rubric does not settle a case, write the ruling in your notebook and apply it to every response of that task. This takes about twenty minutes, and you will notice the pull towards the longer, more confident or better-formatted answer that the scoring rules exist to stop.

When every row has a score, go back to the first two tasks and score their responses again on a piece of paper without looking at what you wrote the first time. If any score differs, your standard moved during the session; resolve it, and write in the notebook that it happened. That is a one-person version of the agreement check an evaluation runs between two raters.

Record: any rulings you made on ambiguous rubrics, and whether the re-score of the first two tasks agreed.

10. Un-blind, total, and test the gap against chance

Download the scoring script into the same directory as tasks.json and run it from there:

RunnableAll tracks

score-blind-comparison.py

"""Un-blind the scored responses, total them per model and per category, and record the run.

Purpose: turn the filled-in scores.csv from run-blind-comparison.py into a result: a table of
         scores per model overall and by category, the gap between each pair of models set
         against the pre-registered margin, an exact sign-permutation test that says how often
         a gap that large arises between two equally good models, the median decode speed each
         local model reached, and one JSON line appended to the lab notebook so the run can be
         compared with later ones.
Platform: all (spark, strix, mac, nvidia). Pure standard library: no packages to install.
Minimum memory: 12 GB, the same floor as the run it scores; this script itself needs almost none.
Assumes: run-blind-comparison.py has been run in this directory, every row of scores.csv has a
         score of 0, 1 or 2, and labbook.md is the notebook started in Part 1.

Usage: python score-blind-comparison.py --labbook ../labbook.md --note "first run"
       python score-blind-comparison.py --per-task --no-labbook
"""
import argparse
import csv
import json
import statistics
import sys
from collections import defaultdict
from pathlib import Path


def load_scores(path: Path) -> dict:
    scores = {}
    blank = []
    try:
        fh = path.open(encoding="utf-8", newline="")
    except FileNotFoundError:
        raise SystemExit(f"{path} not found: run run-blind-comparison.py first.")
    with fh:
        for row in csv.DictReader(fh):
            rid = (row.get("response_id") or "").strip()
            raw = (row.get("score") or "").strip()
            if not rid:
                continue
            if raw == "":
                blank.append(rid)
                continue
            try:
                value = int(raw)
            except ValueError:
                raise SystemExit(f"{path}: response {rid} has a score of {raw!r}; use 0, 1 or 2.")
            if value not in (0, 1, 2):
                raise SystemExit(f"{path}: response {rid} scored {value}; use 0, 1 or 2.")
            scores[rid] = value
    if blank:
        raise SystemExit(
            f"{path}: {len(blank)} response(s) have no score yet, starting with {blank[0]}. "
            "Score every response before running this, or the comparison is not like for like."
        )
    return scores


def load_json(path: Path, what: str) -> dict:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except FileNotFoundError:
        raise SystemExit(f"{path} not found: {what}")
    except json.JSONDecodeError as exc:
        raise SystemExit(f"{path} is not valid JSON: {exc}")


def decode_tokens_per_second(record: dict):
    """Ollama reports eval_count tokens generated in eval_duration nanoseconds."""
    count, duration = record.get("eval_count"), record.get("eval_duration")
    if not count or not duration:
        return None
    return count / (duration / 1_000_000_000)


def sign_permutation_p(differences: list) -> tuple:
    """Exact two-sided sign-permutation test on paired per-task differences.

    If the two models were equally good, each task's difference would be as likely to have
    gone the other way. Enumerate every way of flipping the signs of the non-zero differences
    (2^k patterns, counted exactly by convolution) and return the fraction whose total is at
    least as far from zero as the observed total, plus k, the number of tasks that differed.
    """
    nonzero = [d for d in differences if d != 0]
    observed = abs(sum(nonzero))
    dist = {0: 1}
    for d in nonzero:
        nxt = defaultdict(int)
        for total, count in dist.items():
            nxt[total + d] += count
            nxt[total - d] += count
        dist = nxt
    patterns = 2 ** len(nonzero)
    extreme = sum(count for total, count in dist.items() if abs(total) >= observed)
    return extreme / patterns, len(nonzero)


def bar(fraction: float, width: int = 20) -> str:
    filled = round(fraction * width)
    return "#" * filled + "." * (width - filled)


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--scores", default="scores.csv")
    parser.add_argument("--key", default="blind-key.json")
    parser.add_argument("--tasks", default="tasks.json")
    parser.add_argument("--labbook", default="labbook.md")
    parser.add_argument("--margin", type=int, default=None,
                        help="pre-registered margin in points (default: scoring.preregistered_margin "
                             "in the task file, else 4)")
    parser.add_argument("--per-task", action="store_true",
                        help="also print every task's score per source, so you can see where a gap came from")
    parser.add_argument("--no-labbook", action="store_true", help="print the result but record nothing")
    parser.add_argument("--note", default="", help="a line of your own to store with the result")
    args = parser.parse_args()

    scores = load_scores(Path(args.scores))
    key = load_json(Path(args.key), "run run-blind-comparison.py first.")
    tasks = load_json(Path(args.tasks), "the task file the run used must be present.")
    task_order = [t["id"] for t in tasks["tasks"]]
    category_of = {t["id"]: t["category"] for t in tasks["tasks"]}
    max_score = tasks.get("scoring", {}).get("max_score", 2)
    margin = args.margin
    if margin is None:
        margin = key.get("preregistered_margin")
    if margin is None:
        margin = tasks.get("scoring", {}).get("preregistered_margin", 4)
    responses = key["responses"]

    unknown = sorted(set(scores) - set(responses))
    if unknown:
        raise SystemExit(f"{args.scores} scores responses that are not in {args.key}: {unknown[:5]}")
    unscored = sorted(set(responses) - set(scores))
    if unscored:
        raise SystemExit(f"{len(unscored)} response(s) in {args.key} were never scored, e.g. {unscored[0]}.")

    totals = defaultdict(int)
    per_category = defaultdict(lambda: defaultdict(int))
    counts = defaultdict(int)
    per_category_counts = defaultdict(lambda: defaultdict(int))
    per_task = defaultdict(dict)
    speeds = defaultdict(list)
    cut_off = defaultdict(int)
    thinking_chars = defaultdict(list)

    for rid, score in scores.items():
        record = responses[rid]
        source, task_id = record["source"], record["task"]
        category = category_of.get(task_id, "uncategorised")
        totals[source] += score
        counts[source] += 1
        per_category[source][category] += score
        per_category_counts[source][category] += 1
        per_task[task_id][source] = score
        speed = decode_tokens_per_second(record)
        if speed is not None:
            speeds[source].append(speed)
        if record.get("done_reason") == "length":
            cut_off[source] += 1
        if record.get("thinking_chars"):
            thinking_chars[source].append(record["thinking_chars"])

    sources = sorted(totals, key=lambda s: (-totals[s], s))
    categories = tasks.get("categories") or sorted({c for c in category_of.values()})
    width = max(len(s) for s in sources)

    print()
    print(f"Scored {len(scores)} responses from {len(sources)} sources, {max_score} points each.")
    if key.get("ollama_version"):
        print(f"Ollama {key['ollama_version']}, thinking {'on' if key.get('think') else 'off'}, "
              f"options {json.dumps(key.get('options'))}")
    print()
    for source in sources:
        possible = counts[source] * max_score
        fraction = totals[source] / possible if possible else 0.0
        print(f"  {source:<{width}}  {totals[source]:>3} / {possible:<3}  {bar(fraction)}  {fraction:6.1%}")
    print()
    print("  By category (score out of the category maximum):")
    header = "    " + " " * width + "".join(f"  {c[:14]:>14}" for c in categories)
    print(header)
    for source in sources:
        cells = ""
        for category in categories:
            got = per_category[source][category]
            possible = per_category_counts[source][category] * max_score
            cells += f"  {f'{got}/{possible}':>14}" if possible else f"  {'-':>14}"
        print(f"    {source:<{width}}{cells}")

    if args.per_task:
        print()
        print("  Per task (score per source; a task where every source scored the same tells you nothing):")
        print("    " + f"{'task':<8}" + "".join(f"  {s[:14]:>14}" for s in sources))
        for task_id in task_order:
            row = per_task.get(task_id, {})
            cells = "".join(f"  {row.get(s, '-'):>14}" for s in sources)
            mark = "" if len({row.get(s) for s in sources}) > 1 else "   (same)"
            print(f"    {task_id:<8}{cells}{mark}")

    print()
    print(f"  Pre-registered margin: {margin} points out of {len(task_order) * max_score}.")
    pairs = []
    for i, a in enumerate(sources):
        for b in sources[i + 1:]:
            diffs = [per_task[t].get(a, 0) - per_task[t].get(b, 0) for t in task_order]
            gap = sum(diffs)
            p_value, differing = sign_permutation_p(diffs)
            within = abs(gap) <= margin
            pairs.append({"a": a, "b": b, "gap": gap, "tasks_differing": differing,
                          "p_two_sided": round(p_value, 4), "within_margin": within})
            print(f"  {a} vs {b}: gap {gap:+d} point(s) over {differing} task(s) that differed; "
                  f"{'within' if within else 'outside'} the margin.")
            print(f"    If the two were equally good, a gap at least this large would arise in "
                  f"{p_value:.1%} of runs ({'not ' if p_value > 0.05 else ''}rare by the usual 5% rule).")

    if speeds:
        print()
        print("  Median decode speed reported by Ollama, tokens per second:")
        for source in sources:
            if speeds[source]:
                print(f"    {source:<{width}}  {statistics.median(speeds[source]):.1f}")
            else:
                print(f"    {source:<{width}}  not measured (answers came from elsewhere)")
    if any(cut_off.values()) or any(thinking_chars.values()):
        print()
        for source in sources:
            notes = []
            if cut_off[source]:
                notes.append(f"{cut_off[source]} answer(s) cut off by num_predict")
            if thinking_chars[source]:
                notes.append(f"median thinking trace {statistics.median(thinking_chars[source]):.0f} characters")
            if notes:
                print(f"    {source:<{width}}  " + "; ".join(notes))

    print()
    print(f"  {len(task_order)} tasks is a small sample. A margin of one or two points is not a finding.")
    print()

    record = {
        "lab": "part-03/reality-check-small-local-versus-frontier",
        "generated": key.get("generated"),
        "ollama_version": key.get("ollama_version"),
        "think": key.get("think"),
        "options": key.get("options"),
        "models": key.get("models"),
        "tasks_file": args.tasks,
        "tasks": len(task_order),
        "max_score": max_score,
        "preregistered_margin": margin,
        "totals": {s: {"score": totals[s], "possible": counts[s] * max_score} for s in sources},
        "by_category": {s: {c: {"score": per_category[s][c],
                               "possible": per_category_counts[s][c] * max_score}
                            for c in categories if per_category_counts[s][c]} for s in sources},
        "pairs": pairs,
        "median_decode_tokens_per_second": {
            s: round(statistics.median(v), 2) for s, v in speeds.items() if v},
        "cut_off_by_num_predict": {s: cut_off[s] for s in sources if cut_off[s]},
        "note": args.note,
    }

    if args.no_labbook:
        print(json.dumps(record))
        return
    labbook = Path(args.labbook)
    if not labbook.exists():
        print(f"{labbook} does not exist; creating it.", file=sys.stderr)
    with labbook.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    print(f"  recorded in {labbook}")


if __name__ == "__main__":
    main()

Download score-blind-comparison.py271 lines

RunnableAll tracks

un-blind, total, test and record

python3 score-blind-comparison.py --per-task --labbook ../labbook.md --note "first run, thinking off"

Output — what you should see

Scored 40 responses from 2 sources, 2 points each.
Ollama 0.33.3, thinking off, options {"temperature": 0.0, "seed": 1, "num_predict": 1024, "num_ctx": 4096}

  qwen3:30b-a3b-q4_K_M   xx / 40   ################....   xx.x%
  qwen3:4b-q4_K_M        xx / 40   ##############......   xx.x%

  By category (score out of the category maximum):
                          factual-recall  arithmetic-rea            code   summarisation  instruction-fo
    qwen3:30b-a3b-q4_K_M             x/8             x/8             x/8             x/8             x/8
    qwen3:4b-q4_K_M                  x/8             x/8             x/8             x/8             x/8

  Per task (score per source; a task where every source scored the same tells you nothing):
    task      qwen3:30b-a3b-  qwen3:4b-q4_K_
    fr-01                  2               2   (same)
    fr-02                  2               1
    ...

  Pre-registered margin: 4 points out of 40.
  qwen3:30b-a3b-q4_K_M vs qwen3:4b-q4_K_M: gap +x point(s) over x task(s) that differed; within the margin.
    If the two were equally good, a gap at least this large would arise in xx.x% of runs (not rare by the usual 5% rule).

  Median decode speed reported by Ollama, tokens per second:
    qwen3:30b-a3b-q4_K_M  xx.x
    qwen3:4b-q4_K_M       xx.x

  20 tasks is a small sample. A margin of one or two points is not a finding.

  recorded in ../labbook.md

The digits shown as x are yours. Read the output in this order. The totals answer the pre-registered question: within four points or not. The category rows are the shape, and they are usually more informative than the total. The per-task rows show where any gap came from, and a (same) on most rows is what a close result looks like at task level. The margin line applies the rule you wrote down in step 6. The chance line is the sign-permutation arithmetic from the method section applied to your differences, and “not rare” means the sample cannot tell the models apart even if the gap is outside the margin. The decode-speed lines are the medians of eval_count / eval_duration over the twenty answers, from the counters you read by hand in step 4.

Record: the script has already appended one JSON line to the notebook with the totals, the category breakdown, the per-pair gap, chance figure and whether it was within the margin, the options, the digests and the median decode speed per model, so this run can be compared with a later one after you have quantised or fine-tuned something. Add by hand the ratio of each measured median to the ceiling in the Requirements table for your machine.

11. Optionally, rerun in thinking mode

Both models are hybrids whose card documents a thinking mode, and the arithmetic category is where the part predicts it matters most. --think sends think: true for every model and, because the Qwen3 card says not to use greedy decoding in thinking mode, switches the sampling to the card’s recommended temperature 0.6, top_p 0.95, top_k 20 and min_p 0; the cap must rise because thinking tokens count against it, and the context with it.

RunnableAll tracks

the same twenty tasks with thinking on, into a separate directory

python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model "${LARGE:?set in step 2}" \
    --think --num-predict 4096 --num-ctx 8192 --out-dir think-run

Doubling num_ctx doubles the cache column of the memory table: 1.3 GB for the 14B model, which still fits the 12 GB floor with the weights at 9.3 GB, but only just. Leave the run for its tens of minutes per model, then score it as before, from inside think-run, with a different note:

RunnableAll tracks

score the thinking run against the same margin

cd think-run
python3 ../score-blind-comparison.py --tasks ../tasks.json --per-task --labbook ../../labbook.md --note "second run, thinking on"
cd ..

The output gains a line per model with the number of answers cut off by the cap and the median length of the thinking trace. Compare the two notebook lines: what thinking did to the arithmetic category, what it did to the median decode speed and to the wall-clock, and whether the gap between the models moved. A rerun that closes the arithmetic gap at the price of ten times the tokens is a finding about when a small model is as good as a larger one, which is a more useful sentence than the slogan.

Keep the blind comparison blind and the denominator fixed

Before generating answers, freeze the task file and decision margin. Save its checksum and the endpoint/model settings. Keep the identity mapping out of the scoring view until all answers have been graded. If you recognise a model’s style, note the possible bias rather than changing the rubric.

Use this acceptance sequence: verify one request to each endpoint; validate the task file; generate the full response set; check that every task has an outcome for every model; score blind; reveal the mapping; compute the paired result. An HTTP failure or empty response is an outcome to record, not a task to drop from one side. If a hosted comparison is omitted, label that arm not run.

Inspect disagreements by task category before interpreting the aggregate. A smaller model may meet the chosen task contract while losing on another category. Report that boundary explicitly. Keep the original results when changing thinking mode or sampling: those changes define another experiment. The claim you can support is about the recorded task distribution, settings and margin; it does not establish equivalence between model families on all possible work.

Validation

You are done when all of the following are true:

Check	Command	Pass looks like
Both models present	ollama list	Both tags listed with the digests you recorded
Every task has one response per model	grep -c '^### Response' blind-responses.md	40 (or 60 with a hosted column)
No response is empty or an error	grep -c -E 'empty answer|no answer' blind-responses.md	0
Every score is 0, 1 or 2	grep -c -E '^[0-9a-f]{6},[012]\r?$' scores.csv	40 (or 60)
The scoring script accepted the sheet	python3 score-blind-comparison.py --no-labbook	A result block and a JSON line, no SystemExit message
The notebook has the pre-registration and the result	grep -c 'reality-check-small-local-versus-frontier' ../labbook.md	2 or more, and the pre-registration line is dated before the result line
The result line parses	tail -n 1 ../labbook.md | python3 -m json.tool | head -n 3	The first three lines of a JSON object whose lab field is part-03/reality-check-small-local-versus-frontier
You can say the verdict	none	One sentence, without looking anything up, stating whether the pre-registered four-point margin was met and what the chance figure was

Expected outcome

A number, a breakdown, a probability and an honest sentence. What the number will be depends on your models, your tasks and your scoring, which is why you ran it rather than reading someone else’s result. Two patterns are common enough to prepare for: the categories usually separate in the direction the prediction table in the method section gives, and the decode-speed line usually favours the small model, though against the 30B mixture of experts the order may reverse for the reason step 3 gave.

Reporting a result you did not want

Whichever way it went, the report has the same shape, and it is the shape of every measurement report in this course:

Field	Where it comes from
The claim, in its measurable form	The method section, with your models named
Models, tags, digests, quantisation, Ollama version	Step 2 and the notebook line
Settings: thinking mode, temperature, seed, num_predict, num_ctx	The options in the notebook line
Machine, track, memory, PROCESSOR column	Steps 0 and 4
The margin, and when it was written down	The pre-registration line, dated
Totals, category scores, gap, chance figure	The result line
Any rulings made while scoring, and the re-score check	Step 9
What the score does not capture	The next section
The one-sentence verdict	You

The three outcomes and the sentence each one earns:

What happened	The sentence to write
Gap within the margin	“On these twenty tasks, with this rubric, the two models were indistinguishable; the sample is too small to detect a gap under about six points, so this is not evidence that they are equal at everything.”
Larger model ahead by more than the margin, chance figure small	“On this test set the larger model is ahead, by a gap an equal pair would produce in about x% of runs; the gap sits in the categories the part predicted, or it does not, and here is the breakdown.”
Small model ahead by more than the margin	“On this test set the small model scored higher; on twenty tasks that is a surprise to report, not a conclusion to draw, and the per-task table shows where it came from.”

!Important

Report the result you got. If the small model came within your margin, say so, and that it was on twenty tasks of your choosing; if it did not, say that too, even if you have just bought a machine on the assumption that it would.

Four things people do at this point turn a measurement back into an opinion. Moving the margin after the result: the margin was the only part of the test that could make it fail, and a margin chosen to fit the data fails nothing. Dropping tasks the losing model did badly on, “because they were unfair”: a task that was fair enough to run is fair enough to count, and the place for that judgement was step 5. Rerunning with another seed until the totals agree with you: with sampling on, every rerun is a fresh draw, and keeping the best of five draws biases the total upward by roughly the spread between draws; the fix is to report every run or to fix the seed, as this page does. Re-scoring after un-blinding: the label is now attached to every answer, and the effect the blinding removed is back.

Where small models genuinely win

A fair report also says what a total out of forty does not capture. Each row is a measurable criterion, and the decision rule is the same each time: state the task, the metric and the number, then choose.

Criterion	The measurable question	Where the number comes from	The decision it supports
Latency	Median decode speed, and time to first token, for this model on this machine	The decode-speed line in step 10; the prompt eval rate and eval rate lines in step 3	For anything interactive, an answer that starts immediately and flows faster than you read beats a better answer that takes ten times as long
Cost	Energy per answer and marginal cost per token	Machine draw × time: a 100 W draw for one hour is 0.1 kWh; there is no per-token charge and no bill that scales with an agent in a loop	Volume work, batch jobs and agents that run all day
Privacy	Does any byte leave the machine	ollama ps shows the model here; the API is on 127.0.0.1	For a document you may not send anywhere, the local model competes with not doing the task, not with the frontier
Availability	Works without network, account, rate limit or deprecation notice	The pull you made is the model you have in three years; the digest proves which	Anything that must keep working unattended
Fit within the categories you need	Per-category score, not the total	The category rows in step 10	If your work is instruction following and summarisation, a small model that is close on those rows is close on your work
Narrow tasks	Score on your task after fine-tuning or distillation	Part 13 and Part 15, which start from a task set like this one	The honest route to “as good as” for a specific definition of good

The right question. “As good as the frontier” is rarely the question anyone needs answered. “Good enough for this job, at this speed, at this cost, on this machine” is, and it is answerable, which is why the rest of this course is built around answering it.

Troubleshooting

Symptom	Cause	Fix
ollama: command not found right after installing	The shell has not picked up the new path	Open a new terminal. On Linux the install script links ollama into the first of /usr/local/bin, /usr/bin or /bin that is on your PATH (usually /usr/local/bin); command -v ollama prints which. The manual archive install extracts into /usr, so /usr/bin/ollama. On macOS the application offers to link it into /usr/local/bin on first start; on Windows the installer adds %LOCALAPPDATA%\Programs\Ollama to your user PATH
ollama --version prints Warning: could not connect to a running Ollama instance	The server is not running	Linux: sudo systemctl status ollama, and journalctl -e -u ollama for the log. macOS and Windows: start the Ollama application. Any platform: ollama serve in a terminal runs it in the foreground
The script says Cannot reach Ollama at http://localhost:11434 while ollama list works	OLLAMA_HOST is set to something other than the default, so the CLI and the script disagree	Run the script with --host set to the same value, for example --host 127.0.0.1:11434; the script accepts host:port without a scheme, as the variable is usually written
The script says These models are not present locally: ...	The tag was not pulled, or was pulled under a different name	ollama pull the exact tag; qwen3:30b-a3b-q4_K_M is not the same string as qwen3:30b-a3b, and the two currently carry different digests
ollama ps shows 100% CPU or a CPU/GPU split	The model did not fit the accelerator’s memory, or the GPU was not found	Use the smaller pair from the Requirements table and record the substitution; if a 4B model shows CPU on a machine with a GPU, read the server log named in the row above and see Part 7’s “Did it find the GPU?”, which is where the driver questions are answered
The large model is extremely slow or the machine becomes unresponsive	Spill into system memory over PCIe, or swapping on a unified-memory machine	Stop it with ollama stop <tag>, use the smaller pair, and note the substitution. ollama ps shows the split
The measured median is a factor of five or more below the ceiling table	Almost always a split load, a swapped machine, or something else large running	Check ollama ps and the memory; close the other thing; rerun
Progress lines end in CUT OFF by num_predict	The cap was reached, which with thinking off means an unusually long answer	Rerun both models with --num-predict 2048 so the comparison stays fair; with --think, use --num-predict 4096 or more
Progress lines say EMPTY answer	The model returned no content; with thinking on, it spent the whole budget on the trace	Raise --num-predict and rerun both models; if it persists with thinking off, record it as a real difference between the models
A progress line says FAILED: with a timeout	One request exceeded --timeout, 900 s by default, which on a fitting model means it is not fitting	Check ollama ps; use the smaller pair; the answer was recorded as [no answer: ...] and the run needs repeating
The scoring script says N response(s) have no score yet	scores.csv has an empty score cell	Fill every row; the script refuses a partial sheet because a partial sheet is not like for like
The scoring script says response ... scored 3; use 0, 1 or 2	A typo in the sheet	Fix the cell and rerun
A model produced its working instead of an answer, with thinking off	The model wrote reasoning into the answer text rather than a thinking field	The script strips a leading <think> block if one appears inline; anything else is scored as what the task asked for, and the behaviour is recorded in the notebook as a real difference
The two models gave identical answers to several tasks	Expected on the short factual tasks, where there is one right answer	If it happens on the long summarisation tasks too, check blind-key.json after scoring: if sources lists the same tag twice, you ran one model twice

Cleanup

Leave labbook.md, tasks.json and the two scripts; the model comparison is repeated in later parts. scores.csv and blind-key.json are the only record of your hand scoring: a rerun issues fresh random identifiers and an empty sheet, so it does not recreate them. Unload the models, then keep the scored run, and the thinking-mode run if you made one, in a dated directory:

RunnableAll tracks

unload the models and keep the scored run

ollama stop qwen3:4b-q4_K_M
ollama stop "${LARGE:?set in step 2}"
mkdir -p "runs/$(date +%F)-think-off"
mv blind-responses.md blind-key.json scores.csv "runs/$(date +%F)-think-off/"
[ -d think-run ] && mv think-run "runs/$(date +%F)-think-on"
ls runs

Output — what you should see

2026-xx-xx-think-off  2026-xx-xx-think-on

ollama stop unloads a model from memory immediately rather than after the five-minute keep-alive the FAQ documents; it does not delete anything. The small model is worth keeping if you plan to repeat this comparison after Part 13 or Part 16, which is the point of the notebook line; note that Part 7 pulls the short tag qwen3:4b, a different digest, so this download does not stand in for that one.

⚠Warning

The next block deletes files without asking. rm -f removes whatever is left of the generated files in the current directory, and rm -rf think-run removes the thinking-mode directory with everything in it, including its scored sheet if you did not move it above; a rerun does not bring the scores back. Check that ls runs shows your copies before you run it.

RunnableAll tracks

remove the generated files

rm -f blind-responses.md blind-key.json scores.csv one-response.json one-response-large.json
rm -rf think-run

ollama rm deletes downloaded weights permanently and the only way back is to download them again, which for the 30B tag is a 19 GB transfer. The block removes the large model and keeps the small one; check the name in ollama list before you press return.

RunnableAll tracks

remove the large model if you need the disk

ollama list
ollama rm "${LARGE:?set in step 2}"

What you learned

Objective	The observation that showed it	Where the numbers are
Turn the slogan into a claim a result could refute	Your measurable sentence, with a margin that a gap could exceed	The report’s first field
Fix the rubric and margin before any answer existed	The pre-registration line dated before the result line	Step 6 line: models, margin_points, date
Run both models under identical, reproducible settings	Each model at 100% GPU when loaded at CONTEXT 4096, every progress line ending stop	Step 2 digests; step 4 PROCESSOR and SIZE; options in the result line
Score without knowing the author	A complete sheet, and a re-score of the first two tasks that agreed or a drift you resolved	Step 9 rulings and re-score result
Tell a real gap from chance	The chance figure next to the gap: twenty tasks detect a large difference, not a small one	pairs: gap, tasks_differing, p_two_sided, within_margin
See where “better” lives	Category rows that separate, or not, in the direction the prediction table gave	by_category
Check speed against the bandwidth arithmetic	The measured median set against the ceiling table, with the gap left for Part 5 to explain	median_decode_tokens_per_second; your measured-to-ceiling ratio
Pin what you ran	The digest, not the tag: qwen3:4b and qwen3:4b-q4_K_M carried different digests on 12 September 2026	Step 2 ID column; models in the result line
Optionally, see what thinking changes	A second result line to set against the first	note “second run, thinking on”; the arithmetic category, cut-offs and median decode speed

Check your understanding

Question 1. Why are the answers shuffled and stripped of their model names before scoring?

• To make the file smaller
• Because knowing which model wrote an answer changes how it is scored, and the point is to score the answer
• To protect the models' licences
• Because the scoring script requires it

Show the answer and why

Answer: Because knowing which model wrote an answer changes how it is scored, and the point is to score the answer

Expectation is a strong effect and it does not announce itself. Blinding is the cheapest experimental control there is, and it costs nothing here because a script does it. What it cannot remove is a model's style fingerprint, which is why the rubric rules exist as well.

Question 2. Which of these runs is the bug?

• python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:30b-a3b-q4_K_M
• python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:30b-a3b-q4_K_M --think --num-predict 4096 --num-ctx 8192
• Running the 4B model with the default settings, then the 30B model with --think, and scoring the two blind files together
• python3 run-blind-comparison.py --model qwen3:4b-q4_K_M --model qwen3:14b-q4_K_M --dry-run

Show the answer and why

Answer: Running the 4B model with the default settings, then the 30B model with --think, and scoring the two blind files together

The script runs every model under one set of settings for a reason: a model in thinking mode with sampling on is a different experiment from a model with thinking off at temperature zero. Compare like with like; if you want to see what thinking does, rerun both models with --think, as step 11 does.

Question 3. The 4B model scores 34 and the 30B model scores 36, out of 40, and the two differed on four tasks. What is the supportable conclusion?

• The 4B model is as good as the 30B model at everything
• On these twenty tasks, with this rubric, the two were within the pre-registered margin; a two-point gap that an equal pair would produce about half the time is not evidence of a difference either way
• The 30B model is better and the difference is proven
• The test was invalid

Show the answer and why

Answer: On these twenty tasks, with this rubric, the two were within the pre-registered margin; a two-point gap that an equal pair would produce about half the time is not evidence of a difference either way

The conclusion is bounded by the test set and the sample size. Two points over four differing tasks is a pattern an equal pair produces often, as the sign-permutation table shows. "Indistinguishable on this test set" is a smaller claim than the slogan, and it is the one the evidence carries.

Question 4. You raise the context from 4,096 to 8,192 tokens for the thinking-mode rerun. What happens to the KV cache the 14B model allocates, and does the 12 GB floor still hold?

• It stays the same; the cache depends only on the answer length
• It doubles, from about 0.67 GB to about 1.3 GB, and with 9.3 GB of weights it still fits 12 GB, though with little to spare
• It doubles and no longer fits, so the 14B pair cannot run in thinking mode
• It quadruples, because attention is quadratic in the context

Show the answer and why

Answer: It doubles, from about 0.67 GB to about 1.3 GB, and with 9.3 GB of weights it still fits 12 GB, though with little to spare

The cache is bytes per token times the context length allocated, and Ollama allocates the length you ask for. Doubling num_ctx doubles it. Attention time grows with the context, but the cache memory is linear in it.

Question 5. Which of these are genuine advantages of a small local model that the score out of 40 does not capture? Select all that apply.

• Lower time per output token, from reading fewer active bytes
• No per-token cost as usage grows
• Data never leaving the machine
• A larger context window than any hosted model

Show the answer and why

Answer: Lower time per output token, from reading fewer active bytes, No per-token cost as usage grows, Data never leaving the machine

Speed, cost and privacy are real and are invisible in a quality score. Context window is a property of the specific model, not of being small or local, and small models often have shorter ones.

Question 6. Your per-category table shows the small model close on instruction following and summarisation, and well behind on factual recall. Which explanation fits what this part taught?

• The small model was quantised more heavily
• Format and instruction following are behaviours, which post-training and distillation transfer cheaply, while facts need parameters to be stored in
• The factual tasks were unfair
• The larger model had a longer context

Show the answer and why

Answer: Format and instruction following are behaviours, which post-training and distillation transfer cheaply, while facts need parameters to be stored in

It is the same boundary the post-training and distillation lessons drew: behaviour moves between models cheaply, knowledge does not. Both models here are at the same quantisation and the same context length, which is what makes the category shape attributable to size.

Check my answersClear