Reality Check: 'The Default Context Is Enough'

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track engine versions, models and measured truncation points belong here once the validation pass has run this page on real machines.

?Reality check

The claim under test, in the form it usually appears on a forum: “the default context is enough”. Sometimes it comes with a reason attached: longer contexts are slow, or nobody has conversations that long anyway. This page does not argue with it. It reads what the default on your machine actually is, then measures the conversation length at which your assistant stops being shown how the conversation started.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end you will have, for your own machine and in your lab notebook: the context each engine gives a conversation, read from the engine rather than from a page; the conversation length at which Ollama and llama-server stop showing the model its first message, and how each one fails; the memory a larger window costs, predicted from the model’s configuration and then read back from the engine’s log; the model’s own ceiling; and a context length for the household service from the lab, chosen from that service’s own request log and proved in place.

The claim becomes testable once it is stated as two conditions on one machine:

Condition	Measured by	The claim holds if
The default holds the conversations this household really has	the largest request in the service’s log (preflight) against the default (task 1)	largest request, plus room for the answer, is below the default
The default costs memory the machine can carry	the engine’s key-value cache line and loaded size (task 6)	the model stays 100% GPU with the other resident software running

Either condition failing refutes the claim for that machine, in one of two opposite directions: too small, and conversations lose their beginning; too large, and memory goes to positions nobody uses.

Why the default is not a number you can look up

For Ollama, three official pages give three different answers. All three still say the same at the pinned tag, read on 13 September 2026, and the source code at that tag adds a fourth:

Source at v0.33.3	What it says the default is
Modelfile reference, num_ctx	“(Default: 2048)”
FAQ, “How can I specify the context window size?”	“By default, Ollama uses a context window size of 4096 tokens.”
Context length page	4k below 24 GiB of VRAM, 32k from 24 to 48 GiB, 256k at 48 GiB and above
server/routes.go	4096 below 23 GiB, 32768 from 23 GiB, 262144 from 47 GiB (“slightly lower thresholds … to account for small differences”)

The documentation disagrees with itself; the source agrees with the context-length page, and it makes the answer a property of the memory Ollama detects at start-up, so no page can tell you your number. The machine can, in one log line, and task 1 reads it.

How Ollama 0.33.3 arrives at a number

Ollama takes the first of these that is set, then applies two corrections:

Order	Setting	Where it comes from
1	num_ctx in a request’s options	the native API; Open WebUI’s num_ctx (Ollama) advanced parameter sends it
2	PARAMETER num_ctx	a Modelfile baked into a model name
3	OLLAMA_CONTEXT_LENGTH	the server’s environment (the lab’s .env)
4	the VRAM tier above	the total GPU memory found at start-up, logged as vram-based default context

The first correction caps the number at the model’s trained window, logging requested context size too large for model. The second applies only to the automatic tier: if the load runs out of memory, Ollama retries once at 32768 or 4096. It then starts its bundled llama-server with -c set to the context times OLLAMA_NUM_PARALLEL and -np set to the slot count, so every slot gets the whole context and the memory multiplies, which is the FAQ’s “Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH”. The OpenAI-compatible endpoint cannot send options, as the compatibility page says, which is why a Modelfile is the documented way to give a /v1 client a different window.

How llama-server v0.4.0 arrives at a number

--ctx-size defaults to “0, 0 = loaded from model”, meaning the trained window in the GGUF metadata. --parallel defaults to auto, which the server resolves to four slots sharing one unified buffer, each allowed the whole window. --fit defaults to on and adjusts arguments you did not set until the load fits the device, lowering an unset context towards --fit-ctx (4096) before anything else; at --verbose it logs context size reduced from … to …. An explicit --ctx-size with an explicit --parallel is divided among the slots, the opposite of Ollama. And a slot larger than the trained window is capped with a warning unless RoPE scaling is enabled (task 7).

How LM Studio arrives at a number

The context belongs to the loaded instance: lms load takes --context-length, and the v1 REST API’s GET /api/v1/models reports each loaded instance’s config.context_length and parallel beside the model’s max_context_length. The documentation read for this page does not state what a load without the flag gets, so task 5 reads it.

What each engine does when the conversation is longer

Engine	Mechanism	What the person chatting sees
Ollama 0.33.3	before generating, drops whole messages from the front of the history until the rendered prompt fits, always keeping system messages and the last message (server/prompt.go); logged only at debug level	an answer that makes sense for a shorter conversation
llama-server v0.4.0	refuses a prompt at or above the slot’s context with HTTP 400, type exceed_context_size_error; context shift, off by default, only applies to text generated past the limit	an error instead of an answer
LM Studio 0.4.23	the SDK documents three overflow policies: stopAtLimit, truncateMiddle (“Keep the system prompt and the first user message, truncate middle”) and rollingWindow; which one its OpenAI-compatible server applies was not found in the documentation	measured in task 5

∴Research note

This page tests one thing: at what conversation length a planted word stops coming back, and whether the server counted every token it was sent. It does not test whether answers get subtly worse as the window fills, a harder question that needs the evaluation methods of Part 16. A binary “was the word recalled” is crude, and crude is what leaves no room to read the result the way you hoped.

What runs where

Everything happens on the machine that hosts the household service, in two terminals.

Terminal	What runs there	Address
1	every command, the experiment script, the notebook ~/home-chat/labbook.md
—	Ollama: on Tracks S, X and N the lesson’s native service, while the lab’s engine container is stopped; on Track M the Ollama application, which is also the household engine	127.0.0.1:11434
2	llama-server from Part 6, tasks 5 and 7 only	127.0.0.1:8080
—	LM Studio’s server, optional in task 5	127.0.0.1:1234

Requirements

The primary path uses Qwen3-4B (Apache-2.0, not gated) at Q4_K_M on every track, so the memory floor is 8 GB and every command is the same for every tier. Pinned versions: Ollama 0.33.3 · verified 2026-09-08, llama.cpp v0.4.0 · verified 2026-09-08 (the Part 6 build in ~/llama.cpp/build/bin), LM Studio 0.4.23 · verified 2026-09-08 for the optional LM Studio step, and Open WebUI 0.11.3 · verified 2026-09-08 for the household service. Python 3.8 or later runs the script; it needs no packages.

Download, sizes read 13 September 2026	Size	Needed when
qwen3:4b-q4_K_M, digest 2bfd38a7daaf	2.6 GB (Ollama tags page)	not already in the Ollama store you use here; the lab pulled it into the container volume, which the native service cannot see
unsloth/Qwen3-4B-GGUF, Qwen3-4B-Q4_K_M.gguf	2,497,281,312 bytes	not already in ~/models from Part 6 or the lab’s llama-server path

Plan on about an hour attended: the manifest’s 40 minutes was set before this page grew its llama-server and ceiling tasks. Unattended time is the downloads, 2.6 GB × 80 seconds, about three and a half minutes each at an example 100 Mbit/s, and the longer experiment steps on a machine whose default turns out to be large.

Track S — NVIDIA DGX Spark

Linux, with the lab’s stack in ~/home-chat and the native Ollama service from the Ollama lesson, stopped by the lab’s preflight. The preflight here stops the engine container and starts the native service, so the household service is offline until task 8 brings it back. Which Ollama tier this machine lands in is whatever total_vram its log reports; if it is 47 GiB or more, expect the cap to the model’s window to be the finding.

Track X — AMD Ryzen AI Max+ 395

As Track S. If Ollama logs library=cpu, the experiment still works and every step takes longer; the tier then follows from zero GPU memory. A Windows reader on this track boots Linux, as for the lab.

Track M — Apple silicon

The Ollama application is both the engine for this page and the household engine, and the lab gave it OLLAMA_CONTEXT_LENGTH with launchctl. The preflight lifts that setting so the default can be seen; the household service keeps answering, at the default, until task 8 sets the number you choose. Unified memory means the key-value cache competes with the desktop, so do task 6’s arithmetic before raising anything.

Track N — NVIDIA desktop or laptop

Memory means video memory, and a 12 or 16 GB card sits below Ollama’s first threshold. Linux is the primary path, as in the lab. On Windows, run the experiment against Ollama for Windows in PowerShell: ollama commands are identical, use python where the page says python3, type the number in place of a shell variable, and read the log with the Preflight block for this tab; the household change in task 8 happens in WSL2, exactly as on Linux.

Preflight

Keep terminal 1 for the whole page; the function and variables defined here live in it.

Record what the household has actually sent

Do this first, before this page adds its own requests to the same logs. Ollama’s bundled llama-server, and llama-server on its own, log one line per finished request carrying the tokens the slot held: the prompt plus the answer, thinking included. The largest of those is the context your household has needed so far.

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

the five largest requests the service has handled

cd ~/home-chat
docker compose logs engine | grep -o 'stop processing: n_tokens = [0-9]*' | awk '{print $5}' | sort -n | tail -n 5

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

the five largest requests the service has handled

cd ~/home-chat
docker compose logs engine | grep -o 'stop processing: n_tokens = [0-9]*' | awk '{print $5}' | sort -n | tail -n 5

Track M — Apple silicon

RunnableTrack M · Apple silicon

the five largest requests the service has handled

grep -o 'stop processing: n_tokens = [0-9]*' ~/.ollama/logs/server.log | awk '{print $5}' | sort -n | tail -n 5

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

the five largest requests the service has handled

cd ~/home-chat
docker compose logs engine | grep -o 'stop processing: n_tokens = [0-9]*' | awk '{print $5}' | sort -n | tail -n 5

Output — what you should see

xxx
xxxx
xxxx
xxxx
xxxxx

The last line is the largest. No output means nobody has used the service since the engine started: have one realistic long conversation in the web interface first (paste a two-page document and ask three follow-up questions), then run the block again. Record: household_max_tokens (the last line), and the date range the log covers.

Free the memory and start Ollama with nothing set

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

stop the household engine, start the native service

cd ~/home-chat
docker compose stop engine
sudo systemctl start ollama
systemctl is-active ollama
ollama --version
ollama_log() { sudo journalctl -u ollama --no-pager --since today; }

Output — what you should see

... Container home-chat-engine-1  Stopped
active
ollama version is 0.33.3

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

stop the household engine, start the native service

cd ~/home-chat
docker compose stop engine
sudo systemctl start ollama
systemctl is-active ollama
ollama --version
ollama_log() { sudo journalctl -u ollama --no-pager --since today; }

Output — what you should see

... Container home-chat-engine-1  Stopped
active
ollama version is 0.33.3

Track M — Apple silicon

RunnableTrack M · Apple silicon

lift the lab's context setting and restart the application

launchctl getenv OLLAMA_CONTEXT_LENGTH
launchctl unsetenv OLLAMA_CONTEXT_LENGTH
osascript -e 'quit app "Ollama"'
sleep 5
open -a Ollama
sleep 10
ollama --version
ollama_log() { cat ~/.ollama/logs/server.log; }

Output — what you should see

8192
ollama version is 0.33.3

8192 is the lab’s setting; an empty first line means none was set.

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

stop the household engine, start the native service

cd ~/home-chat
docker compose stop engine
sudo systemctl start ollama
systemctl is-active ollama
ollama --version
ollama_log() { sudo journalctl -u ollama --no-pager --since today; }

Output — what you should see

... Container home-chat-engine-1  Stopped
active
ollama version is 0.33.3

Ollama for Windows writes its log to %LOCALAPPDATA%\Ollama\server.log; in PowerShell, read it with this, and use Select-String -Pattern wherever a later block pipes ollama_log into grep -E with the same pattern:

RunnableTrack N · Windows

PowerShell: the log reader for this page

function ollama_log { Get-Content "$env:LOCALAPPDATA\Ollama\server.log" }
ollama_log | Select-String -Pattern 'OLLAMA_CONTEXT_LENGTH:[0-9]+|vram-based default context'

Compose’s progress wording varies by version; the container must end Stopped. A Warning: client version is line means the CLI and the server differ; update to the pinned version first. Then confirm the server started with no context setting of its own:

RunnableAll tracks

what the server was started with

ollama_log | grep -o 'OLLAMA_CONTEXT_LENGTH:[0-9]*' | tail -n 1
ollama_log | grep 'vram-based default context' | tail -n 1
ollama_log | grep 'inference compute' | tail -n 1

Output — what you should see

OLLAMA_CONTEXT_LENGTH:0
time=... level=INFO source=routes.go:2062 msg="vram-based default context" total_vram="x.x GiB" default_num_ctx=xxxx
time=... level=INFO source=types.go:50 msg="inference compute" id=... library=... name=... total="x.x GiB" available="x.x GiB"

Pass: OLLAMA_CONTEXT_LENGTH:0. On Linux each line is prefixed with the journal’s date, host and ollama[pid]:. Record: total_vram, default_num_ctx, and the library, total and available fields of the device line.

Tools, model files and the script

Save context-truncation-test.py from the download link under task 2 into ~/home-chat first.

RunnableAll tracks

tools, model files, the script, and a free port

cd ~/home-chat
python3 --version
ollama pull qwen3:4b-q4_K_M
ollama list | grep 'qwen3:4b-q4_K_M'
ls -l ~/llama.cpp/build/bin/llama-server ~/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf
python3 context-truncation-test.py --help | head -n 1
curl -sS http://127.0.0.1:8080/health

Output — what you should see

Python 3.x.x
pulling manifest
...
success
qwen3:4b-q4_K_M    2bfd38a7daaf    2.6 GB    x seconds ago
-rwxr-xr-x 1 you you    xxxxxxx ... /home/you/llama.cpp/build/bin/llama-server
-rw-r--r-- 1 you you 2497281312 ... /home/you/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf
usage: context-truncation-test.py [-h] [--base-url BASE_URL] --model MODEL
curl: (7) Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server

Pass: Python 3.8 or later, the digest 2bfd38a7daaf, both files listed, the usage line, and a refused connection on 8080 (curl’s wording varies by version; error 7 is the pass). A different digest means the tag has moved since 13 September 2026: record the one you got. A missing GGUF comes from hf download unsloth/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir ~/models/unsloth/Qwen3-4B-GGUF. On Track M the paths start /Users/you.

Tasks

1. Ask Ollama what it gives a conversation

Load the model with an empty request, which the API documentation describes as loading it into memory without generating anything, and keep it loaded for thirty minutes:

RunnableAll tracks

load at the default, then ask three places

curl -sS http://127.0.0.1:11434/api/generate -d '{"model": "qwen3:4b-q4_K_M", "keep_alive": "30m"}'
ollama ps
ollama_log | grep -E 'requested context size too large|starting llama-server|llama_kv_cache: size|n_ctx_slot' | tail -n 4

Output — what you should see

{"model":"qwen3:4b-q4_K_M","created_at":"...","response":"","done":true,...}
NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b-q4_K_M    2bfd38a7daaf    x.x GB    100% GPU     4096       29 minutes from now
time=... level=INFO source=llama_server.go:433 msg="starting llama-server" cmd="... --no-webui --offline -c 4096 -np 1 ... --context-shift --keep 4"
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
srv    load_model: initializing, n_slots = 1, n_ctx_slot = 4096, kv_unified = 'false'

The four readings must agree, and each says something different:

Line	Field	What it tells you
ollama ps	CONTEXT	the context each conversation slot got
ollama ps	PROCESSOR	100% GPU, 100% CPU, or a split such as 48%/52% CPU/GPU
starting llama-server	-c, -np	total cells allocated and the slot count; -c is CONTEXT × slots
llama_kv_cache: size	MiB, cells, layers	what that context costs; 576.00 MiB is 147,456 bytes × 4,096, task 6
n_ctx_slot	tokens	the per-slot limit llama-server enforces

The output shows the lowest tier. In the middle tier the numbers read 32768 and 4608.00 MiB. In the top tier the log adds requested context size too large for model num_ctx=262144 n_ctx_train=40960, and every number reads 40960 and 5760.00 MiB: the model capped the tool.

Put the number in a variable the later tasks use, and record it before you have any reason to want it to be a particular value:

RunnableAll tracks

the default, from the running server

DEFAULT_CTX=$(curl -sS http://127.0.0.1:11434/api/ps | python3 -c 'import json, sys; print([m["context_length"] for m in json.load(sys.stdin)["models"] if m["name"] == "qwen3:4b-q4_K_M"][0])')
echo "DEFAULT_CTX=$DEFAULT_CTX"

Output — what you should see

DEFAULT_CTX=4096

Record: CONTEXT, PROCESSOR, SIZE, the -c and -np values and the llama_kv_cache line. An IndexError from Python means the model is not loaded; run the first block again.

2. Read the experiment before you run it

RunnableAll tracks

context-truncation-test.py

#!/usr/bin/env python3
"""Find the conversation length at which a chat server stops showing the model its beginning.

Purpose: measure, rather than assume, what a server's context length does to a long
         conversation. A badge word is planted in the first message, the conversation is
         grown with filler exchanges of constant token size, and at every step the model is
         asked for the word back. Two small calibration requests first measure exactly how
         many prompt tokens the fixed part and each filler exchange cost, so every step knows
         how many tokens it sent; comparing that with the prompt tokens the server reports
         shows a server dropping part of the conversation even when nothing errors.
Platform: all (spark, strix, mac, nvidia; also Windows with Python 3). Standard library only.
Minimum memory: 8 GB, for a 4B-class model at the server's own context length. Raising the
         server's context is what costs memory; this script only sends requests.
Assumes: an OpenAI-compatible chat server is running and --model is a name it serves:
         Ollama http://127.0.0.1:11434/v1, llama-server http://127.0.0.1:8080/v1,
         LM Studio http://127.0.0.1:1234/v1. Python 3.8 or later. Writes one JSON line to
         --labbook (default labbook.md in the current directory) and nothing else.

Verdicts, one per step:
  fits        the word came back and the server counted every token it was sent
  cut-lost    the server counted fewer tokens than it was sent and the word did not come back
  cut-kept    the server counted fewer tokens than it was sent but the word still came back
  model       the word was lost although the server counted every token: the model missed it
  refused     the server answered with an HTTP error instead (it fails loudly)
  no-answer   empty answer that hit the token limit: thinking or --max-tokens, not context
  recall-only the server reported no token count, so only the answer can be judged

Usage: python3 context-truncation-test.py --base-url http://127.0.0.1:11434/v1 \\
           --model qwen3:4b-q4_K_M --around 4096 --labbook labbook.md
       python3 context-truncation-test.py --base-url http://127.0.0.1:8080/v1 \\
           --model qwen3-4b --steps 2048,3072,4096,6144 --no-labbook
"""
import argparse
import datetime
import json
import random
import re
import sys
import time
import urllib.error
import urllib.request

# Ordinary words no model is likely to produce by accident when asked for something else.
MARKER_WORDS = [
    "PELICAN", "BASALT", "MARZIPAN", "LANTERN", "OBSIDIAN",
    "TAMARIND", "QUARTZITE", "HALYARD", "JUNIPER", "CINNABAR",
]

# One filler exchange. The numbers are zero-padded to four digits so that every exchange
# tokenises to the same length under any tokeniser: that is what makes the calibration exact.
FILLER_USER = (
    "Note item {n:04d} for the inventory: {n:04d} crates of dried goods arrived at the north "
    "warehouse on day {n:04d}, checked in by the day shift, stacked in aisle {n:04d}, and "
    "recorded against the standing order. Reply with only: noted {n:04d}."
)
FILLER_ASSISTANT = "noted {n:04d}"
RECALL_QUESTION = (
    "What was the badge word I gave you in my very first message? "
    "Answer with that single word and nothing else."
)
AROUND_FRACTIONS = (0.5, 0.8, 0.95, 1.05, 1.25, 1.5, 2.0)
CALIBRATION_PAIRS = (1, 9)


def build_messages(marker, pairs, plant_in):
    """The planted word, `pairs` filler exchanges, then the question."""
    if plant_in == "system":
        messages = [{"role": "system",
                     "content": f"The badge word for this conversation is {marker}."}]
    else:
        messages = [
            {"role": "user",
             "content": f"Remember this for the whole conversation. The badge word is "
                        f"{marker}. Reply with only: noted."},
            {"role": "assistant", "content": "noted"},
        ]
    for n in range(1, pairs + 1):
        messages.append({"role": "user", "content": FILLER_USER.format(n=n)})
        messages.append({"role": "assistant", "content": FILLER_ASSISTANT.format(n=n)})
    messages.append({"role": "user", "content": RECALL_QUESTION})
    return messages


def http_json(url, payload=None, api_key="", timeout=30.0):
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    data = json.dumps(payload).encode("utf-8") if payload is not None else None
    req = urllib.request.Request(url, data=data, headers=headers)
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))


def strip_thinking(text):
    """A thinking block, closed or cut off, is not the answer."""
    text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
    return re.sub(r"<think>.*", "", text, flags=re.DOTALL).strip()


def ask(args, messages):
    payload = {"model": args.model, "messages": messages, "stream": False,
               "temperature": 0, "max_tokens": args.max_tokens}
    if args.reasoning_effort:
        payload["reasoning_effort"] = args.reasoning_effort
    started = time.time()
    body = http_json(f"{args.base_url.rstrip('/')}/chat/completions", payload,
                     args.api_key, args.timeout)
    choice = (body.get("choices") or [{}])[0]
    usage = body.get("usage") or {}
    return {
        "answer": strip_thinking(str((choice.get("message") or {}).get("content") or "")),
        "finish_reason": choice.get("finish_reason"),
        "prompt_tokens": usage.get("prompt_tokens"),
        "seconds": round(time.time() - started, 2),
    }


def server_facts(args):
    """What this particular server says about itself. Best effort; no endpoint is required."""
    root = re.sub(r"/v1/?$", "", args.base_url.rstrip("/"))
    facts = {}
    try:  # llama-server
        props = http_json(f"{root}/props", api_key=args.api_key, timeout=10)
        facts["llama_server_n_ctx_per_slot"] = props["default_generation_settings"]["n_ctx"]
        facts["llama_server_total_slots"] = props.get("total_slots")
    except (urllib.error.URLError, OSError, ValueError, KeyError, TypeError):
        pass
    wanted = {args.model, f"{args.model}:latest"}  # Ollama adds :latest to untagged names
    try:  # Ollama
        for m in http_json(f"{root}/api/ps", api_key=args.api_key, timeout=10).get("models", []):
            if wanted & {m.get("name"), m.get("model")}:
                facts["ollama_context_length"] = m.get("context_length")
                facts["ollama_size_bytes"] = m.get("size")
                facts["ollama_size_vram_bytes"] = m.get("size_vram")
    except (urllib.error.URLError, OSError, ValueError, AttributeError):
        pass
    try:  # LM Studio
        for m in http_json(f"{root}/api/v1/models", api_key=args.api_key, timeout=10).get("models", []):
            for inst in m.get("loaded_instances") or []:
                if args.model in (inst.get("id"), m.get("key")):
                    facts["lmstudio_context_length"] = (inst.get("config") or {}).get("context_length")
                    facts["lmstudio_parallel"] = (inst.get("config") or {}).get("parallel")
                    facts["lmstudio_max_context_length"] = m.get("max_context_length")
    except (urllib.error.URLError, OSError, ValueError, AttributeError):
        pass
    return facts


def verdict(recalled, sent, counted, finish_reason, answer):
    if not answer and finish_reason == "length":
        return "no-answer"
    if counted is None:
        return "recall-only"
    dropped = sent - counted > max(16, 0.02 * sent)
    if recalled:
        return "cut-kept" if dropped else "fits"
    return "cut-lost" if dropped else "model"


def main():
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--base-url", default="http://127.0.0.1:11434/v1",
                   help="OpenAI-compatible base URL, ending in /v1")
    p.add_argument("--model", required=True, help="model name exactly as the server lists it")
    group = p.add_mutually_exclusive_group()
    group.add_argument("--around", type=int, metavar="TOKENS",
                       help="context length to test around: steps at 0.5x to 2x of it")
    group.add_argument("--steps", help="comma-separated prompt sizes in tokens")
    p.add_argument("--plant-in", choices=("user", "system"), default="user",
                   help="put the badge word in the first user message or a system message")
    p.add_argument("--max-tokens", type=int, default=64)
    p.add_argument("--reasoning-effort", default="none",
                   help="sent as reasoning_effort; 'none' turns thinking off on Ollama and "
                        "llama-server. Pass an empty string to leave the field out")
    p.add_argument("--keep-going", action="store_true",
                   help="do not stop after two steps in a row that lose the word")
    p.add_argument("--api-key", default="", help="bearer token, if the server requires one")
    p.add_argument("--timeout", type=float, default=900.0, help="seconds per request")
    p.add_argument("--seed", type=int, default=7, help="chooses the badge word")
    p.add_argument("--labbook", default="labbook.md")
    p.add_argument("--no-labbook", action="store_true", help="print the record instead")
    p.add_argument("--note", default="", help="a line of your own stored with the record")
    args = p.parse_args()

    marker = random.Random(args.seed).choice(MARKER_WORDS)
    print(f"badge word: {marker}   model: {args.model}   planted in: {args.plant_in} message")

    # Calibration: two small conversations give the fixed cost and the cost per exchange.
    counts = []
    for pairs in CALIBRATION_PAIRS:
        try:
            out = ask(args, build_messages(marker, pairs, args.plant_in))
        except urllib.error.HTTPError as exc:
            detail = exc.read().decode("utf-8", "replace")[:300]
            sys.exit(f"calibration request failed with HTTP {exc.code}: {detail}\n"
                     "If the server rejects reasoning_effort, rerun with --reasoning-effort ''.")
        except (urllib.error.URLError, OSError) as exc:
            sys.exit(f"cannot reach {args.base_url}: {exc}")
        if not out["answer"] and out["finish_reason"] == "length":
            sys.exit(f"calibration with {pairs} exchange(s): an empty answer that hit --max-tokens "
                     f"({args.max_tokens}). The model spent its budget thinking: keep "
                     "--reasoning-effort none, or raise --max-tokens for a server that ignores it.")
        if marker.lower() not in out["answer"].lower():
            sys.exit(f"calibration with {pairs} exchange(s): the answer was {out['answer']!r} "
                     f"(finish_reason {out['finish_reason']}). The model did not return the word "
                     "from a short conversation, so no context length can be measured with it: "
                     "the model, not the window, is the limit. See the page's troubleshooting.")
        counts.append(out["prompt_tokens"])
    if None in counts:
        sys.exit("the server reported no usage.prompt_tokens, so the step sizes cannot be "
                 "calibrated. Use a server that reports usage (Ollama, llama-server, LM Studio do).")
    per_pair = (counts[1] - counts[0]) / (CALIBRATION_PAIRS[1] - CALIBRATION_PAIRS[0])
    fixed = counts[0] - per_pair * CALIBRATION_PAIRS[0]
    print(f"calibration: {fixed:.0f} fixed tokens + {per_pair:.1f} tokens per filler exchange")

    facts = server_facts(args)
    for key, value in facts.items():
        print(f"server reports {key} = {value}")

    if args.around:
        targets = [int(args.around * f) for f in AROUND_FRACTIONS]
    elif args.steps:
        targets = [int(s) for s in args.steps.split(",") if s.strip()]
    else:
        targets = [1024, 2048, 4096, 8192, 16384, 32768]

    print()
    print(f"{'target':>8} {'pairs':>6} {'sent':>7} {'counted':>8} {'seconds':>8}  "
          f"{'recalled':<8}  {'verdict':<11}  answer")
    results, losses = [], 0
    for target in targets:
        pairs = max(1, round((target - fixed) / per_pair))
        sent = int(round(fixed + per_pair * pairs))
        row = {"target_tokens": target, "filler_pairs": pairs, "sent_tokens": sent}
        try:
            out = ask(args, build_messages(marker, pairs, args.plant_in))
        except urllib.error.HTTPError as exc:
            detail = exc.read().decode("utf-8", "replace")[:300]
            try:  # llama-server and Ollama put the reason in error.message or error
                err = json.loads(detail).get("error")
                reason = err.get("message") if isinstance(err, dict) else str(err)
            except (ValueError, AttributeError):
                reason = detail
            print(f"{target:>8} {pairs:>6} {sent:>7} {'-':>8} {'-':>8}  {'-':<8}  "
                  f"{'refused':<11}  HTTP {exc.code}: {reason}")
            row.update({"verdict": "refused", "http_status": exc.code, "detail": detail})
            results.append(row)
            break
        except (urllib.error.URLError, OSError) as exc:
            sys.exit(f"cannot reach {args.base_url}: {exc}")
        recalled = marker.lower() in out["answer"].lower()
        v = verdict(recalled, sent, out["prompt_tokens"], out["finish_reason"], out["answer"])
        counted = "-" if out["prompt_tokens"] is None else out["prompt_tokens"]
        shown = out["answer"].replace("\n", " ")[:32] or f"(empty, {out['finish_reason']})"
        print(f"{target:>8} {pairs:>6} {sent:>7} {counted:>8} {out['seconds']:>8}  "
              f"{'yes' if recalled else 'NO':<8}  {v:<11}  {shown}")
        row.update({"counted_tokens": out["prompt_tokens"], "seconds": out["seconds"],
                    "recalled": recalled, "verdict": v, "finish_reason": out["finish_reason"],
                    "answer": out["answer"][:120]})
        results.append(row)
        losses = 0 if recalled else losses + 1
        if losses >= 2 and not args.keep_going:
            print("two steps in a row lost the word; stopping (use --keep-going to continue)")
            break

    fitted = [r["sent_tokens"] for r in results if r["verdict"] == "fits"]
    cuts = [r for r in results if r["verdict"] in ("cut-lost", "cut-kept", "refused")]
    misses = [r["sent_tokens"] for r in results if r["verdict"] == "model"]
    other = sorted({r["verdict"] for r in results} & {"no-answer", "recall-only"})
    print()
    print(f"largest conversation that fitted: {max(fitted) if fitted else 'none'} tokens sent")
    if cuts:
        print(f"first step the server cut or refused: {cuts[0]['sent_tokens']} tokens sent "
              f"({cuts[0]['verdict']})")
    else:
        print("no step was cut or refused: the context is larger than the largest step sent")
    if misses:
        print(f"steps the model missed with every token in view: {misses}")
    if other:
        print(f"steps that could not be judged: {', '.join(other)} (see the page's troubleshooting)")

    record = {
        "lab": "part-07/reality-check-the-default-context-is-enough",
        "recorded_at": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
        "base_url": args.base_url, "model": args.model, "marker": marker,
        "plant_in": args.plant_in, "reasoning_effort": args.reasoning_effort,
        "calibration": {"fixed_tokens": round(fixed, 1), "tokens_per_pair": round(per_pair, 1)},
        "server_facts": facts,
        "largest_fit_sent_tokens": max(fitted) if fitted else None,
        "first_cut": ({"sent_tokens": cuts[0]["sent_tokens"], "verdict": cuts[0]["verdict"]}
                      if cuts else None),
        "model_miss_sent_tokens": misses,
        "steps": results, "note": args.note,
    }
    if args.no_labbook:
        print(json.dumps(record))
        return
    with open(args.labbook, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download context-truncation-test.py304 lines

A badge word goes into the first message, filler exchanges grow the conversation, and the last message asks for the word. Two signals are read at every step: whether the word came back, and whether the server counted every token it was sent. The second is what separates a server that cut the conversation from a model that missed a word it was shown.

How the script knows what it sent. Each filler exchange has its numbers zero-padded to four digits, so every exchange costs the same number of tokens. Two calibration requests, with 1 and 9 exchanges, give two points on a straight line:

fixed      = tokens(1) − per_pair
per_pair   = (tokens(9) − tokens(1)) / 8
pairs      = round((target − fixed) / per_pair)
sent       = fixed + per_pair × pairs

Under llama-server v0.4.0 and the Qwen3 chat template the course’s CPU check counted these, and the server’s own error message later confirmed the prediction to the token:

Quantity	Tokens
tokens(1), tokens(9)	152, 824
per_pair = (824 − 152) / 8	84
fixed = 152 − 84	68
target 4,300: pairs = round((4,300 − 68) / 84)	50
sent = 68 + 84 × 50	4,268

Ollama renders the same messages with its own template, so its fixed may differ by a few tokens; the script calibrates against whichever server it talks to. The request sets reasoning_effort to none, which both Ollama’s compatibility page and llama-server’s README document as turning thinking off; without it Qwen3 spends the answer budget thinking and returns an empty answer that would read as a lost word.

Where the steps land. --around N sends targets of 0.5, 0.8, 0.95, 1.05, 1.25, 1.5 and 2 times N, so the boundary falls between the third and fourth steps. After two lost steps in a row the script stops, which bounds the slowest rows.

What each step’s verdict means, read before the numbers exist:

Word came back?	Server counted every token?	Verdict	Meaning
yes	yes	fits	the whole conversation was shown to the model
no	no	cut-lost	the server dropped part of it, and the word went with it
yes	no	cut-kept	the server dropped part of it, but not the part with the word
no	yes	model	every token was there and the model still missed the word
—	—	refused	the server returned an HTTP error: it fails loudly
empty	—	no-answer	the answer hit --max-tokens: thinking, not context

“Counted every token” allows a difference of 16 tokens or 2 per cent, whichever is larger.

3. Run it against Ollama’s default

RunnableAll tracks

the experiment at the default, word in the first user message

cd ~/home-chat
python3 context-truncation-test.py --base-url http://127.0.0.1:11434/v1 --model qwen3:4b-q4_K_M --around "$DEFAULT_CTX" --labbook labbook.md --note "Ollama default"

Output — what you should see

badge word: TAMARIND   model: qwen3:4b-q4_K_M   planted in: user message
calibration: xx fixed tokens + xx.x tokens per filler exchange
server reports ollama_context_length = 4096
server reports ollama_size_bytes = xxxxxxxxxx
server reports ollama_size_vram_bytes = xxxxxxxxxx

  target  pairs    sent  counted  seconds  recalled  verdict      answer
    2048     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    3276     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    3891     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    4300     xx    xxxx     xxxx     x.xx  NO        cut-lost     noted
    5120     xx    xxxx     xxxx     x.xx  NO        cut-lost     noted
two steps in a row lost the word; stopping (use --keep-going to continue)

largest conversation that fitted: xxxx tokens sent
first step the server cut or refused: xxxx tokens sent (cut-lost)
recorded in labbook.md

This is the shape Ollama’s source at v0.33.3 implies, reproduced on the course’s CPU check with a proxy that applies Ollama’s message-dropping rule in front of llama-server; the course has not run it against Ollama on hardware. Read the counted column on the cut-lost rows: it stays just under 4,096 while sent keeps rising, because whole messages were dropped until the rest fitted. An answer like noted is the model answering honestly about the conversation it was given, which begins somewhere in the filler.

!Important

Notice what did not happen. Nothing errored, the front-end showed nothing, and the server log at its default level says nothing: Ollama’s only trace is a truncating input messages which exceed context length line written at debug level. The failure of a too-small context on this engine is not a crash. It is an assistant that has been shown less than you said and has no way to tell you.

Now plant the same word in a system message instead, which Ollama’s truncation keeps:

RunnableAll tracks

the same run, word in a system message

python3 context-truncation-test.py --base-url http://127.0.0.1:11434/v1 --model qwen3:4b-q4_K_M --around "$DEFAULT_CTX" --plant-in system --labbook labbook.md --note "Ollama default, system message"

Output — what you should see

badge word: TAMARIND   model: qwen3:4b-q4_K_M   planted in: system message
...
    3891     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    4300     xx    xxxx     xxxx     x.xx  yes       cut-kept     TAMARIND
    5120     xx    xxxx     xxxx     x.xx  yes       cut-kept     TAMARIND
...

Expected: cut-kept from the fourth step onwards, with counted flat again. The conversation was still cut; what survives is decided by the role of the message, which matters in task 8. Record for both runs: largest_fit_sent_tokens, the first_cut sent tokens and verdict, and any model steps. On a top-tier machine the run goes to 81,920 tokens and the cut steps take longer than their size suggests, because Ollama re-tokenises the history once for each message it drops.

4. Change one thing and watch the point move

A finding you cannot reverse is a coincidence. Change only the context and the cut should move with it. Pick the new value from your default, so it stays affordable on the 8 GB tier:

RunnableAll tracks

the new context: 16384 if the default was 4096, otherwise 8192

if [ "$DEFAULT_CTX" -le 4096 ]; then NEW_CTX=16384; else NEW_CTX=8192; fi
echo "NEW_CTX=$NEW_CTX"
printf 'FROM qwen3:4b-q4_K_M\nPARAMETER num_ctx %s\n' "$NEW_CTX" > Modelfile.ctxtest
cat Modelfile.ctxtest
ollama create qwen3-4b-ctxtest -f Modelfile.ctxtest

Output — what you should see

NEW_CTX=16384
FROM qwen3:4b-q4_K_M
PARAMETER num_ctx 16384
gathering model components
using existing layer sha256:...
...
writing manifest
success

On Windows, write the file in PowerShell with the value typed in, then run the same ollama create:

RunnableTrack N · Windows

PowerShell: the Modelfile

Set-Content -Path Modelfile.ctxtest -Value "FROM qwen3:4b-q4_K_M`nPARAMETER num_ctx 16384"

Unload the default, load the variant, and run the experiment around the new value:

RunnableAll tracks

the variant, loaded and tested

ollama stop qwen3:4b-q4_K_M
curl -sS http://127.0.0.1:11434/api/generate -d '{"model": "qwen3-4b-ctxtest", "keep_alive": "30m"}'
ollama ps
python3 context-truncation-test.py --base-url http://127.0.0.1:11434/v1 --model qwen3-4b-ctxtest --around "$NEW_CTX" --labbook labbook.md --note "Modelfile num_ctx $NEW_CTX"

Output — what you should see

{"model":"qwen3-4b-ctxtest","created_at":"...","response":"","done":true,...}
NAME                       ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3-4b-ctxtest:latest    xxxxxxxxxxxx    x.x GB    100% GPU     16384      29 minutes from now
badge word: TAMARIND   model: qwen3-4b-ctxtest   planted in: user message
...
server reports ollama_context_length = 16384
...
first step the server cut or refused: xxxxx tokens sent (cut-lost)
recorded in labbook.md

Expected: the boundary sits between the 0.95 and 1.05 steps again, now of NEW_CTX. Record the same fields as task 3, plus the variant’s CONTEXT and SIZE. If CONTEXT still shows the old number, the variant was not the model you loaded; see Troubleshooting.

5. The same question to llama-server, and to LM Studio

Free the memory Ollama holds first, then start llama-server in terminal 2 with nothing set but the model, to see its default:

RunnableAll tracks

terminal 1: unload both Ollama models

ollama stop qwen3-4b-ctxtest
ollama ps

RunnableAll tracks

terminal 2: llama-server at its defaults

~/llama.cpp/build/bin/llama-server --model ~/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf --alias qwen3-4b --n-gpu-layers 999 --host 127.0.0.1 --port 8080

Output — what you should see

...
x.xx.xxx.xxx I srv    load_model: initializing, n_slots = 4, n_ctx_slot = xxxxx, kv_unified = 'true'
x.xx.xxx.xxx I srv  llama_server: listening on http://127.0.0.1:8080

n_slots = 4 is the automatic slot count, sharing one buffer. n_ctx_slot = 40960 means the whole trained window fitted; a smaller number is --fit lowering it for your memory, which the 8 GB tier should expect: the weights and a 40,960-token cache alone come to 8.54 GB (task 6). Check the same number over HTTP in terminal 1, then stop the server with Ctrl-C:

RunnableAll tracks

terminal 1: the server describes itself

curl -sS http://127.0.0.1:8080/props | python3 -c 'import json, sys; d = json.load(sys.stdin); print(d["default_generation_settings"]["n_ctx"], d["total_slots"])'

Output — what you should see

xxxxx 4

Record: n_slots, n_ctx_slot. Now give llama-server a window equal to the lowest Ollama tier, one slot, and run the experiment against it:

RunnableAll tracks

terminal 2: an explicit 4096-token slot

~/llama.cpp/build/bin/llama-server --model ~/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf --alias qwen3-4b --n-gpu-layers 999 --ctx-size 4096 --parallel 1 --host 127.0.0.1 --port 8080

RunnableAll tracks

terminal 1: the experiment against llama-server

python3 context-truncation-test.py --base-url http://127.0.0.1:8080/v1 --model qwen3-4b --around 4096 --labbook labbook.md --note "llama-server ctx-size 4096"

Output — what you should see

badge word: TAMARIND   model: qwen3-4b   planted in: user message
calibration: xx fixed tokens + xx.x tokens per filler exchange
server reports llama_server_n_ctx_per_slot = 4096
server reports llama_server_total_slots = 1

  target  pairs    sent  counted  seconds  recalled  verdict      answer
    2048     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    3276     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    3891     xx    xxxx     xxxx     x.xx  yes       fits         TAMARIND
    4300     xx    xxxx        -        -  -         refused      HTTP 400: request (xxxx tokens) exceeds the available context size (4096 tokens), try increasing it

largest conversation that fitted: xxxx tokens sent
first step the server cut or refused: xxxx tokens sent (refused)
recorded in labbook.md

Terminal 2 shows the same refusal as E srv send_error: task id = xx, error: request (xxxx tokens) exceeds the available context size (4096 tokens), try increasing it. The request (xxxx tokens) figure is the server’s own count of what the script sent; compare it with the script’s sent for that row. Same weights, same window, opposite failure: this engine fails loudly. Record the refused row’s sent and the server’s count, then stop llama-server with Ctrl-C.

Optional, if you use LM Studio. With its server running (lms server start --port 1234, from the LM Studio lesson), load the model without --context-length, read what the instance got, and run the experiment around it:

RunnableAll tracks

LM Studio: the default, then the experiment

lms load qwen3-4b --identifier ctx-probe
LMS_CTX=$(curl -sS http://127.0.0.1:1234/api/v1/models | python3 -c 'import json, sys; print([i["config"]["context_length"] for m in json.load(sys.stdin)["models"] for i in m["loaded_instances"] if i["id"] == "ctx-probe"][0])')
echo "LMS_CTX=$LMS_CTX"
python3 context-truncation-test.py --base-url http://127.0.0.1:1234/v1 --model ctx-probe --around "$LMS_CTX" --labbook labbook.md --note "LM Studio default"
lms unload --all

Output — what you should see

...
LMS_CTX=xxxx
...
first step the server cut or refused: xxxx tokens sent (xxxxxxxx)

The verdict on the first step past the boundary is LM Studio’s overflow policy, measured: refused for a server that stops, cut-lost for a rolling window, cut-kept for one that keeps the first user message. Record LMS_CTX, the parallel value if the script printed one, and that verdict.

6. Price the window you just used

The key-value cache holds one key vector and one value vector per layer, per key-value head, per token, so its size is a product. Part 3 derived it and Part 6’s lab checked it against llama.cpp’s own log; here it prices the numbers you just saw:

KV bytes = 2 × layers × kv_heads × head_dim × bytes_per_element × context × slots
Qwen3-4B and Qwen3-8B, config.json: num_hidden_layers 36, num_key_value_heads 8, head_dim 128
bytes_per_element: f16 2;  q8_0 34 per 32 elements;  q4_0 18 per 32 elements (ggml block sizes)

RunnableAll tracks

kv-price.py

# Price a context length for Qwen3-4B and Qwen3-8B, which share one KV shape.
LAYERS, KV_HEADS, HEAD_DIM = 36, 8, 128           # config.json: num_hidden_layers,
                                                  # num_key_value_heads, head_dim
BYTES_PER_ELEMENT = {"f16": 2.0, "q8_0": 34 / 32, "q4_0": 18 / 32}  # ggml block sizes
WEIGHTS = {"Qwen3-4B-Q4_K_M": 2_497_281_312,      # Hub file sizes, 2026-09-13
           "Qwen3-8B-Q4_K_M": 5_027_784_512}
GB = 1e9

def kv_bytes(context, cache="f16", slots=1):
    per_token = 2 * LAYERS * KV_HEADS * HEAD_DIM * BYTES_PER_ELEMENT[cache]
    return per_token * context * slots

print(f"bytes per token, f16: {kv_bytes(1):,.0f}")
print(f"{'context':>8} {'f16 GB':>8} {'q8_0 GB':>8} {'q4_0 GB':>8} "
      f"{'4B+f16 GB':>10} {'8B+f16 GB':>10}")
for ctx in (4096, 8192, 16384, 32768, 40960):
    f16, q8, q4 = (kv_bytes(ctx, c) / GB for c in ("f16", "q8_0", "q4_0"))
    w4, w8 = (WEIGHTS[k] / GB for k in WEIGHTS)
    print(f"{ctx:>8} {f16:>8.2f} {q8:>8.2f} {q4:>8.2f} {w4 + f16:>10.2f} {w8 + f16:>10.2f}")

Output — what you should see

bytes per token, f16: 147,456
 context   f16 GB  q8_0 GB  q4_0 GB  4B+f16 GB  8B+f16 GB
    4096     0.60     0.32     0.17       3.10       5.63
    8192     1.21     0.64     0.34       3.71       6.24
   16384     2.42     1.28     0.68       4.91       7.44
   32768     4.83     2.57     1.36       7.33       9.86
   40960     6.04     3.21     1.70       8.54      11.07

This is arithmetic from the configuration and the file sizes, not a measurement, and it leaves out the compute buffers. The last row is why the 8 GB tier sees llama-server’s --fit lower its default, and why a small Ollama tier exists at all:

Estimated: Qwen3-4B Q4_K_M at Ollama's lowest tier, on an 8 GiB card

Weights, Q4_K_M file

2.5 GB

KV cache, 4,096 tokens, f16

0.6 GB

Free

5.5 GB

Total

8.6 GB

From the file size and the formula above. 8 GiB is 8.59 GB. Compute buffers and anything else on the card are not included.

Estimated: the same model at its trained window of 40,960 tokens

Weights, Q4_K_M file

2.5 GB

KV cache, 40,960 tokens, f16

6.0 GB

Free

0.1 GB

Total

8.6 GB

The weights did not move; the cache is ten times larger because it is linear in the context. 0.05 GB is left before a single compute buffer.

Now check the prediction against what Ollama allocated. Load each model in turn and read the size the server reports and the cache it logged:

RunnableAll tracks

loaded size and KV line, at the default and at NEW_CTX

for m in qwen3:4b-q4_K_M qwen3-4b-ctxtest; do
  curl -sS http://127.0.0.1:11434/api/generate -d "{\"model\": \"$m\", \"keep_alive\": \"10m\"}" > /dev/null
  curl -sS http://127.0.0.1:11434/api/ps | python3 -c 'import json, sys; [print(m["name"], m["context_length"], m["size"], m["size_vram"]) for m in json.load(sys.stdin)["models"]]'
  ollama_log | grep 'llama_kv_cache: size' | tail -n 1
  ollama stop "$m"
done

Output — what you should see

qwen3:4b-q4_K_M 4096 xxxxxxxxxx xxxxxxxxxx
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
qwen3-4b-ctxtest:latest 16384 xxxxxxxxxx xxxxxxxxxx
llama_kv_cache: size = 2304.00 MiB ( 16384 cells,  36 layers,  1/1 seqs), K (f16): 1152.00 MiB, V (f16): 1152.00 MiB

Work the gap out in a table in your notebook, one row per model:

Quantity	How to get it
predicted KV	147,456 × context, bytes; 576.00 MiB is 603,979,776 bytes
logged KV	the MiB figure × 1,048,576
size difference	second size − first size, bytes
predicted difference	147,456 × (NEW_CTX − DEFAULT_CTX); 1,811,939,328 bytes for 4096 to 16384
overhead	size − 2,497,281,312 (weights file) − logged KV: compute buffers and everything else

The logged KV should equal the prediction to the MiB; a mismatch means a quantised cache or a different model than you think. The size difference is larger than the predicted difference by whatever the compute buffers grew. size_vram smaller than size is the part that went to system memory. Record: both size, size_vram, logged KV and overhead figures.

Slots multiply, or divide. Ollama gives every slot the whole context: OLLAMA_NUM_PARALLEL=2 at 8,192 tokens costs 147,456 × 8,192 × 2 = 2,415,919,104 bytes, the cache of one slot at 16,384. llama-server with explicit --ctx-size 8192 --parallel 2 allocates 8,192 cells and gives each slot 4,096, the course’s CPU check logged n_slots = 2, n_ctx_slot = 4096. LM Studio’s lms load --estimate-only prints an estimate for a given --context-length without loading, if you want its view. Quantising the cache halves (q8_0) or quarters (q4_0) the KV column; the FAQ calls the q8_0 loss “very small” and the q4_0 loss “small-medium … more noticeable at higher context sizes”, and Part 17 covers the flags per engine.

7. Find where the model, not the tool, is the limit

The model has a window of its own, and it is not a single number:

RunnableAll tracks

the window the file declares

ollama show qwen3:4b-q4_K_M

Output — what you should see

  Model
    architecture        qwen3
    parameters          4.0B
    context length      40960
    embedding length    2560
    quantization        Q4_K_M
...

Figure	Source	Meaning
32,768	Qwen3-4B card; the course’s model reference	“natively supports context lengths of up to 32,768 tokens”
40,960	max_position_embeddings in config.json; context_length in the GGUF	the card: “reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts”
131,072	the card, “Processing Long Texts”	validated “using the YaRN method”, which must be switched on

Ask Ollama for more than the file declares and watch the model cap the tool:

RunnableAll tracks

a variant that asks for 65,536 tokens

printf 'FROM qwen3:4b-q4_K_M\nPARAMETER num_ctx 65536\n' > Modelfile.ctx65k
ollama create qwen3-4b-ctx65k -f Modelfile.ctx65k
curl -sS http://127.0.0.1:11434/api/generate -d '{"model": "qwen3-4b-ctx65k", "keep_alive": "5m"}' > /dev/null
ollama ps
ollama_log | grep -E 'requested context size too large|llama_kv_cache: size' | tail -n 2
ollama stop qwen3-4b-ctx65k

Output — what you should see

NAME                      ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3-4b-ctx65k:latest    xxxxxxxxxxxx    x.x GB    100% GPU     40960      4 minutes from now
time=... level=WARN source=server.go:114 msg="requested context size too large for model" num_ctx=65536 n_ctx_train=40960
llama_kv_cache: size = 5760.00 MiB ( 40960 cells,  36 layers,  1/1 seqs), K (f16): 2880.00 MiB, V (f16): 2880.00 MiB

The warning’s wording and fields are from the v0.33.3 source; the source= line number is inferred from it. On an 8 GB card, expect PROCESSOR to show a CPU/GPU split: 8.54 GB of weights and cache does not fit, which is task 6’s last row arriving.

llama-server treats the same request differently, and the difference is memory. With 16 GB or more of GPU-visible memory, start it with 65,536 tokens and read the lines that matter (the KV size needs --verbose, which also prints a line for every token later; stop it with Ctrl-C once listening appears):

RunnableAll tracks

terminal 2, 16 GB and above: a slot bigger than the trained window

~/llama.cpp/build/bin/llama-server --model ~/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf --n-gpu-layers 999 --ctx-size 65536 --parallel 1 --host 127.0.0.1 --port 8080 --verbose 2>&1 | grep -E 'training context|llama_kv_cache: size|n_ctx_slot|listening'

Output — what you should see

W llama_context: n_ctx_seq (65536) > n_ctx_train (40960) -- possible training context overflow
I llama_kv_cache: size = 9216.00 MiB ( 65536 cells,  36 layers,  1/1 seqs), K (f16): 4608.00 MiB, V (f16): 4608.00 MiB
W srv    load_model: the slot context (65536) exceeds the training context of the model (40960) - capping
I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 40960, kv_unified = 'false'
I srv  llama_server: listening on http://127.0.0.1:8080

The course’s CPU check produced exactly this sequence with a Qwen3 GGUF at v0.4.0 (the MiB figures above are the formula’s for the 4B). The server allocated 65,536 cells and will use 40,960 of them: the memory bill without the capability. With --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768, the flags the Qwen3 card gives for llama-server, the same check logged n_ctx_train adjusted to 131072 and a 65,536-token slot. The Modelfile parameter table at v0.33.3 lists no RoPE or YaRN parameter, so Ollama offers no documented equivalent.

The second face of the model limit needs no flags: a model verdict, where the server counted every token and the word still did not come back. It is a property of the weights at that fill, and the card itself warns that static YaRN “may potentially degrade model performance” on shorter texts.

Situation	Rule
context wanted ≤ trained window	set it; the tool is the only limit
context wanted > trained window, no scaling	do not; Ollama caps it, llama-server allocates it and caps the slot
context wanted > trained window, scaling enabled as the card says	only after an evaluation at that length (Part 16) shows it still works
model verdicts inside the window	a longer context will not help; a larger model or a shorter conversation might

Record: the three window figures and their sources, the capped CONTEXT, and, if you ran it, the KV size and n_ctx_slot from the 65,536-token load.

8. Choose the household’s number and prove it took

You now have the demand (preflight), the price per token and the overhead (task 6), the ceiling (task 7) and each engine’s failure (tasks 3 and 5). Put them into one calculation:

Pseudocode — not a real command

need    = household_max_tokens × 2, rounded up to a multiple of 4096
            (the doubling is headroom for the longest conversation not yet seen; choose your own factor)
ceiling = 40960 for Qwen3-4B, without scaling
budget  = memory the engine may use, from the "inference compute" line's total
cost(C) = weights + overhead from task 6 + 147,456 × C × OLLAMA_NUM_PARALLEL
choose C = the smallest multiple of 4096 that is at least need, if C ≤ ceiling and cost(C) ≤ budget

RunnableAll tracks

house-context.py: fill in the first five lines from your notebook

household_max_tokens = 3000          # preflight: the last line of the largest-requests block
overhead_bytes = 400_000_000         # task 6: size - weights file - logged KV
budget_bytes = 8 * 1024**3           # the device line's total, GiB x 1024**3
slots = 1                            # OLLAMA_NUM_PARALLEL in ~/home-chat/.env
weights_bytes = 2_497_281_312        # Qwen3-4B-Q4_K_M
per_token, ceiling = 147_456, 40_960

need = -(-household_max_tokens * 2 // 4096) * 4096
cost = weights_bytes + overhead_bytes + per_token * need * slots
print(f"need {need} tokens, cost {cost / 1e9:.2f} GB of {budget_bytes / 1e9:.2f} GB, "
      f"within the model's window: {need <= ceiling}, fits: {cost <= budget_bytes}")

Output — what you should see

need 8192 tokens, cost 4.11 GB of 8.59 GB, within the model's window: True, fits: True

The first four lines above are placeholders for the shape of the calculation, not figures for your machine; the output is what those placeholders produce. Replace them and read your own line:

Your result	Do this
fits, and need is at or below what .env has	keep the value; the claim held for this household, and you can now say why
fits, and need is above it	set OLLAMA_CONTEXT_LENGTH to need
does not fit	first OLLAMA_KV_CACHE_TYPE=q8_0 (halves the KV term), then fewer slots, then a smaller model; rerun the calculation after each
need above 40,960	the model is the limit: a model with a longer declared window, or conversations that start fresh

Two findings from the experiment belong in the decision as well. On the Ollama engine an undersized context loses the start of a conversation silently, while on the llama-server engine it returns an error to the person chatting; size with more headroom on the silent one. And anything that must survive every conversation belongs in a system prompt, which Ollama’s truncation keeps (task 3). Open WebUI’s num_ctx (Ollama) advanced parameter, if anyone switches it to Custom, starts at 2048 and outranks everything in .env; leave it on Default.

Write the number and the reason into ~/home-chat/.env with an editor, on the OLLAMA_CONTEXT_LENGTH= line and the comment line above it, then set the shell variable to the same number:

RunnableAll tracks

your chosen number (16384 shown; use yours) and the file

HOUSE_CTX=16384
grep -B 1 '^OLLAMA_CONTEXT_LENGTH=' ~/home-chat/.env

Output — what you should see

# Largest request 2026-09-13: xxxx tokens; doubled and rounded; 4B + KV fits with 100% GPU (reality check)
OLLAMA_CONTEXT_LENGTH=16384

Then put the engine back and prove the number took:

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

native Ollama off, the stack back, the context checked

sudo systemctl stop ollama
cd ~/home-chat
docker compose up --detach --wait
docker compose exec engine ollama run qwen3:4b-q4_K_M "Reply with one word: ready"
docker compose exec engine ollama ps

Output — what you should see

...
ready
NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b-q4_K_M    2bfd38a7daaf    x.x GB    100% GPU     16384      29 minutes from now

On the llama-server path the check is docker compose logs engine | grep n_ctx_slot | tail -n 1, which must show your number.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

native Ollama off, the stack back, the context checked

sudo systemctl stop ollama
cd ~/home-chat
docker compose up --detach --wait
docker compose exec engine ollama run qwen3:4b-q4_K_M "Reply with one word: ready"
docker compose exec engine ollama ps

Output — what you should see

...
ready
NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b-q4_K_M    2bfd38a7daaf    x.x GB    100% GPU     16384      29 minutes from now

On the llama-server path the check is docker compose logs engine | grep n_ctx_slot | tail -n 1, which must show your number.

Track M — Apple silicon

RunnableTrack M · Apple silicon

the application gets the number, restarts, and shows it

launchctl setenv OLLAMA_CONTEXT_LENGTH "$HOUSE_CTX"
osascript -e 'quit app "Ollama"'
sleep 5
open -a Ollama
sleep 10
ollama run qwen3:4b-q4_K_M "Reply with one word: ready"
ollama ps

Output — what you should see

...
ready
NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b-q4_K_M    2bfd38a7daaf    x.x GB    100% GPU     16384      29 minutes from now

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

native Ollama off, the stack back, the context checked

sudo systemctl stop ollama
cd ~/home-chat
docker compose up --detach --wait
docker compose exec engine ollama run qwen3:4b-q4_K_M "Reply with one word: ready"
docker compose exec engine ollama ps

Output — what you should see

...
ready
NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b-q4_K_M    2bfd38a7daaf    x.x GB    100% GPU     16384      29 minutes from now

On the llama-server path the check is docker compose logs engine | grep n_ctx_slot | tail -n 1, which must show your number.

ollama run may print a thinking trace before ready. docker compose up recreates the engine because its environment changed; the front-end and the proxy stay as they were. Record: HOUSE_CTX, the reason line, CONTEXT and PROCESSOR from the check.

Distinguish truncation from failed recall

For each context setting, retain the sent token count, the server’s reported count, the response status and the location of the inserted evidence. If the server accepted fewer tokens than were sent, a missing answer can be a transport or truncation outcome. If all tokens were processed but the model missed the evidence, the failure is a different one.

Begin with a short positive control whose inserted fact is easy to retrieve. If that fails, repair the task or request format before interpreting a long-context sweep. Move the evidence position as a separate experiment, because a model can behave differently near the beginning, middle and end.

After changing the configured context, restart or reload as required by the selected engine and verify the effective setting. Do not infer it from a configuration file that the running process never reread. Save both the capacity boundary and the model’s task-performance boundary. The setting you choose for the household should satisfy the actual conversation workload with a response allowance and memory headroom; the largest accepted request is not automatically the most useful one.

Validation

Check	Command	Pass
the default was read from the machine	grep -c '"note": "Ollama default"' ~/home-chat/labbook.md	1 or more, and DEFAULT_CTX recorded with its log lines
the default run found a boundary	the task 3 record	a fits step and a cut-lost step, or every step fits on a machine whose default exceeds the largest step (then add larger --steps)
the change moved it	the task 4 record	first_cut near NEW_CTX, not near DEFAULT_CTX
llama-server failed loudly	the task 5 record	a refused step whose server count equals the script’s sent
the prediction met the log	task 6 table	logged KV equal to 147,456 × context for both loads
the ceiling was seen	task 7	CONTEXT 40960 for the 65,536 variant
every run was recorded	grep -c 'reality-check-the-default-context-is-enough' ~/home-chat/labbook.md	5 or more (6 with LM Studio)
the household number took	task 8 check	CONTEXT equal to HOUSE_CTX, PROCESSOR 100% GPU

Expected outcome

A number and an argument, both yours, and an engine that runs with them.

On a machine with less than 23 GiB of GPU memory, the likely result is that the default was 4,096 tokens, that the household’s own longest request was already close to it or past it, and that Ollama said nothing when it dropped the beginning. On a large unified-memory machine, the likely result is the opposite: an automatic 262,144 capped by the model to 40,960, and 6.04 GB of cache per slot reserved for conversations the log shows nobody has. Both refute “the default context is enough” as stated, in opposite directions. It was never a claim about context; it was a claim that someone else’s guess about your memory and your conversations was right.

!Important

Report what you got. If your household’s largest request sat comfortably inside the default and the memory held, the claim held for you: say so, with the machine, the engine, the model and the date, because those are the conditions under which it held.

Troubleshooting

Symptom	Cause	Fix
calibration exits with an empty answer that hit --max-tokens	the server ignored reasoning_effort and the model spent its budget thinking	--max-tokens 1024; the empty-answer check then passes on servers that think
calibration exits with did not return the word from a short conversation	the model misses the word with only a few hundred tokens in view	use Qwen3-4B or larger; this is the model limit, not a context finding
calibration fails with HTTP 400 and a message about reasoning_effort	a server that rejects the field	--reasoning-effort ''
HTTP 401 from LM Studio	its server requires an API token	--api-key with the token from its settings
the server reported no usage.prompt_tokens	a server that does not fill in usage	use one of the three engines on this page
every step fits, including the largest	the context is larger than twice --around	--steps with larger values, and expect long prefill
IndexError from the DEFAULT_CTX or LMS_CTX line	the model is not loaded, or listed under another name	run the load block again; ollama ps or lms ps shows the name
CONTEXT for the variant still shows the old number	the name you loaded is not the variant, or the old model was still loaded	ollama stop both, load qwen3-4b-ctxtest, check ollama ps
OLLAMA_CONTEXT_LENGTH: is not 0 after the preflight	a systemd override (Linux) or a remaining setting (Mac); the context-length page also documents a slider in the Ollama app	systemctl edit ollama to remove it, or record the value as this machine’s configured default and carry on
CONTEXT lower than the tier, and a reducing automatic context and retrying once log line	the automatic value did not fit and Ollama retried smaller	a finding: record it; memory is the limit before the model is
PROCESSOR shows a CPU/GPU split	weights plus cache exceed GPU memory	expected for the 65,536 variant on 8 GB; elsewhere, a smaller context or OLLAMA_KV_CACHE_TYPE=q8_0
llama-server logs failed to fit params to free device memory	memory is still held by Ollama or another process	ollama ps must be empty; stop other GPU users
the household’s answers look truncated after task 8	the front-end sends its own num_ctx, which outranks .env	set Open WebUI’s num_ctx (Ollama) parameter back to Default
no stop processing lines in the preflight	the engine was recreated since the household last used it	have a long conversation, then read again
the machine starts swapping	on unified memory the cache competes with everything else	stop, lower the context, and use task 6’s arithmetic first

Cleanup

Keep context-truncation-test.py and labbook.md; the script works against any OpenAI-compatible server, including the vLLM server of Part 9, where memory per conversation is allocated differently. Stop llama-server in terminal 2 with Ctrl-C if it is still running. Task 8 already stopped the native Ollama service on Tracks S, X and N and restarted the household engine.

⚠Warning

ollama rm deletes a model from the store it runs against, without asking. The two variants were built from Modelfiles in seconds and share the base model’s weights; check each name before you press return, because the command does not distinguish them from qwen3:4b-q4_K_M, which the household service needs. On Tracks S, X and N the variants live in the native service’s store, so start it for these commands and stop it afterwards; on Track M they are in the application’s store, which is also the household’s.

RunnableAll tracks

remove the two variants and their Modelfiles

ollama list
ollama rm qwen3-4b-ctxtest
ollama rm qwen3-4b-ctx65k
rm ~/home-chat/Modelfile.ctxtest ~/home-chat/Modelfile.ctx65k

Output — what you should see

NAME                       ID              SIZE      MODIFIED
qwen3-4b-ctx65k:latest     xxxxxxxxxxxx    2.6 GB    x minutes ago
qwen3-4b-ctxtest:latest    xxxxxxxxxxxx    2.6 GB    x minutes ago
qwen3:4b-q4_K_M            2bfd38a7daaf    2.6 GB    x minutes ago
deleted 'qwen3-4b-ctxtest'
deleted 'qwen3-4b-ctx65k'

What you learned

Objective	The observation that proved it	Recorded
the default is a property of the machine	vram-based default context and CONTEXT agreeing on a number no page gave you	total_vram, default_num_ctx, DEFAULT_CTX
Ollama’s failure is silent	cut-lost rows with counted flat under the context and a successful HTTP status	largest_fit_sent_tokens, first_cut
what survives is decided by role	the system-message run turning cut-lost into cut-kept	both records
the context caused it	the boundary moving to NEW_CTX	the task 4 record
llama-server’s failure is loud	refused, with the server’s token count equal to the script’s	the refused row
context is bought by the token and by the slot	the logged KV equal to 147,456 × context; Ollama multiplying by slots, llama-server dividing	task 6 table
the model is a ceiling	CONTEXT 40960 for a 65,536 request; llama-server allocating 65,536 and capping the slot	window figures and sources
a chosen default beats an inherited one	CONTEXT equal to HOUSE_CTX on the household engine, with the reason in .env	HOUSE_CTX and the reason line

Check your understanding

Question 1. Your household engine is Ollama 0.33.3 with OLLAMA_CONTEXT_LENGTH=8192 in .env. A family member opens Open WebUI's advanced parameters, switches num_ctx (Ollama) from Default to Custom and leaves the slider where it starts. What context do their conversations get?

• 8192, because the server environment variable wins
• 2048, because a num_ctx sent in the request outranks the Modelfile and the environment, and the control starts at 2048
• 40960, because Ollama uses the model's trained window when the front-end sends a value
• 4096, the VRAM tier default

Show the answer and why

Answer: 2048, because a num_ctx sent in the request outranks the Modelfile and the environment, and the control starts at 2048

Ollama applies the first setting it finds, in order: request options, Modelfile PARAMETER, OLLAMA_CONTEXT_LENGTH, then the VRAM tier. Open WebUI 0.11.3's control sets 2048 the moment it is switched to Custom. Because the request's num_ctx differs from the loaded model's, Ollama also reloads the model, so the change shows up in ollama ps as well as in shorter memories.

Question 2. Two runs of the experiment at the same --around value. Run A: word planted in the first user message, verdict cut-lost from the fourth step. Run B: word planted in a system message, verdict cut-kept from the fourth step. Which engine behaviour explains both?

• llama-server refusing the prompt
• A rolling window that drops tokens from the middle of each message
• Ollama dropping whole messages from the front of the history while keeping system messages and the last message
• The model ignoring system messages

Show the answer and why

Answer: Ollama dropping whole messages from the front of the history while keeping system messages and the last message

Both runs were cut at the same length, which the flat counted column shows; only what survived differed. server/prompt.go at v0.33.3 removes messages from the front until the rendered prompt fits and always re-adds system messages. A refusal would have produced refused, not a cut, and a model ignoring the system message would have lost the word in run B.

Question 3. Qwen3-4B has 36 layers, 8 key-value heads and a head dimension of 128. Ollama runs it with OLLAMA_NUM_PARALLEL=2 and OLLAMA_CONTEXT_LENGTH=16384 at an f16 cache. How much key-value cache does it allocate?

• 1.21 GB
• 2.42 GB
• 4.83 GB
• 9.66 GB

Show the answer and why

Answer: 4.83 GB

2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes per token; × 16,384 tokens × 2 slots = 4,831,838,208 bytes. Ollama starts llama-server with -c equal to the context times the slots, so every slot gets the whole window. The same two settings on llama-server (--ctx-size 16384 --parallel 2) allocate half that and give each slot 8,192 tokens.

Question 4. Which of these llama-server v0.4.0 commands allocates key-value cache for positions the model will never be allowed to use?

• llama-server --model Qwen3-4B-Q4_K_M.gguf --ctx-size 16384 --parallel 1
• llama-server --model Qwen3-4B-Q4_K_M.gguf --ctx-size 65536 --parallel 1
• llama-server --model Qwen3-4B-Q4_K_M.gguf --ctx-size 65536 --parallel 1 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768
• llama-server --model Qwen3-4B-Q4_K_M.gguf

Show the answer and why

Answer: llama-server --model Qwen3-4B-Q4_K_M.gguf --ctx-size 65536 --parallel 1

The GGUF declares 40,960 tokens. At 65,536 without scaling, llama-server allocates 65,536 cells, warns "possible training context overflow" and caps the slot at 40,960: the memory bill without the capability. With the YaRN flags from the Qwen3 card, it adjusts the trained context to 131,072 and uses the whole slot, whose quality then needs an evaluation. With no --ctx-size it uses the trained window, lowered by --fit if memory is short.

Question 5. In a run, one step shows recalled NO with counted equal to sent, and the next larger step shows recalled yes and fits. What does that step tell you?

• The server truncated silently at that step
• The context length is too small
• The model missed a word that was entirely in view: a model limit at that fill, not a context finding
• The calibration was wrong, so the whole run is invalid

Show the answer and why

Answer: The model missed a word that was entirely in view: a model limit at that fill, not a context finding

A server that dropped tokens would report fewer than were sent; this one counted all of them. The next step fitting shows the window was not the constraint. That is the model verdict, and it is why the script records both signals: watching only the answer would have put the truncation point at the wrong step.

Check my answersClear