Context Engineering: Memory, Compaction and the KV Budget

The thing that stops a local agent is almost never the model’s ability. It is that the transcript grew, the key-value cache grew with it, and the machine ran out of memory or the turns got slow enough that you stopped waiting.

By the end of this lesson you will be able to compute what a token of context costs for each of this course’s reference models, trace how a transcript grows over a realistic task, choose a context length for your memory tier before you start rather than after it fails, and apply the four techniques that keep the budget under control: compaction, notes outside the window, retrieval, and prefix caching.

What a token of context costs

Part 4 derived the formula: bytes per token equals layers times key-value heads times head dimension times two tensors times bytes per element. The course model reference records the result for each model, so the arithmetic is a multiplication.

Model	Layers	KV heads	Head dim	Bytes per token at 16-bit
Qwen3-4B and Qwen3-8B	36	8	128	147,456
Qwen3-14B	40	8	128	163,840
Qwen3-30B-A3B and Qwen3-Coder-30B-A3B	48	4	128	98,304
gpt-oss-20b	24	8	64	49,152

Multiply by the context length and you have the cache. In gigabytes of 10⁹ bytes, from the same figures:

Model	8k	32k	64k	128k	256k
Qwen3-4B, Qwen3-8B	1.21	4.83	9.66	19.33	38.65
Qwen3-14B	1.34	5.37	10.74	21.47	42.95
Qwen3-30B-A3B, Qwen3-Coder	0.81	3.22	6.44	12.88	25.77
gpt-oss-20b	0.40	1.61	3.22	6.44	12.88

Two things fall out of that table immediately.

The mixture-of-experts models are cheaper per token of context than the dense 8B. Qwen3-30B-A3B has four key-value heads where Qwen3-8B has eight, so it costs two thirds as much per token despite being nearly four times the size. For an agent, where context is the scarce resource, that changes which model you want on a given machine.

A long context can cost more than the weights. Qwen3-8B at Q4_K_M is a 5.0 GB file. Its cache at 128k tokens is nearly four times that.

Qwen3-8B at Q4_K_M with a 128k agent context, on a 32 GB machine

Weights, Q4_K_M

5 GB

KV cache, 131,072 tokens at 16-bit

19.3 GB

Engine buffers and activations

1.5 GB

Free

6.2 GB

Total

32 GB

Estimate, not a measurement: weights from the file size recorded in the course model reference, cache from 147,456 bytes per token at 16-bit times 131,072 tokens, and an allowance for engine buffers. The cache is the largest thing on the machine, and almost all of it is transcript the agent will never look at again.

Qwen3-Coder-30B-A3B at Q4_K_M with a 64k agent context, on a 32 GB machine

Weights, Q4_K_M

18.6 GB

KV cache, 65,536 tokens at 16-bit

6.4 GB

Engine buffers and activations

2 GB

Free

5.0 GB

Total

32 GB

Estimate on the same basis: 98,304 bytes per token at 16-bit times 65,536 tokens. Four key-value heads instead of eight is why a 30B model with twice the context fits alongside its own weights where the dense 8B at 128k does not leave room for much else.

▤Memory budget

Both figures halve if you quantise the cache, with --cache-type-k q8_0 --cache-type-v q8_0 on llama-server or --kv-cache-dtype on vLLM. Part 17 gives the flags and the warning that goes with them: prefix caching reuses bytes unchanged, cache quantisation rounds them, and rounding changes outputs. For an agent that is emitting structured calls, test that the calls still parse before you accept the memory back.

How a transcript actually grows

Abstract growth is easy to ignore. Here is a concrete eight-message trace of a five-turn task, with plausible token counts for a coding-ish job on a small repository.

After	Added	Transcript tokens	Prefilled this turn
System prompt and four tool schemas	900	900	—
The task	60	960	960
Turn 1: call, then a search result	40 + 700	1,700	1,700
Turn 2: call, then a directory listing	40 + 1,200	2,940	2,940
Turn 3: call, then a file read	40 + 2,400	5,380	5,380
Turn 4: call, then a test summary	40 + 300	5,720	5,720
Turn 5: the answer	250	5,970	—

The transcript ends at under six thousand tokens, which sounds harmless. But look at the last column: the model prefilled 16,700 tokens over the task, for 5,970 tokens of distinct content. Every turn re-reads everything before it. That is quadratic in the number of turns, and it is why an agent that feels fine at turn three feels broken at turn fifteen.

Three separate costs come out of the same growth, and they are worth naming separately because they have different fixes:

• Prefill time, which is the re-reading above, and which prefix caching removes almost entirely.
• Cache memory, which is the transcript length times bytes per token, and which only compaction, retrieval or a quantised cache reduce.
• Attention, which nothing reduces except having less in the window. Anthropic’s context-engineering post names this directly, describing “context rot”: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases”, because models have “an ‘attention budget’ that they draw on when parsing large volumes of context”.

✕Common mistake

Raising --ctx-size when the agent starts forgetting. It is the natural response and it makes two of the three costs worse: more cache memory allocated, and more room for the transcript to fill with material that dilutes the instruction at the top. A larger window is the right fix only when the agent is genuinely being truncated. When it is wandering, the fix is less context, not more.

Four ways to keep it affordable

Compaction

Anthropic’s post defines it as “the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary”. In an agent loop that means: when the transcript passes a threshold you choose, ask the model to write a short summary of what has been done and learnt so far, then start the next turn with the system prompt, the task, that summary, and the last one or two observations.

What to keep in the summary, in order of importance: the task as originally stated, decisions already made and why, facts discovered that are not re-derivable, files or identifiers touched, and what remains. What to drop: everything the agent read in full, every failed attempt whose lesson is already in the summary, and all reasoning text.

The threshold is a policy, not a constant. A reasonable starting point is to compact at about half the context length, because the summary turn itself needs room and because the second half of a window is where recall falls off fastest.

Notes outside the window

Compaction is lossy by design. The complement is what the post calls structured note-taking, where the agent “regularly writes notes persisted to memory outside of the context window” and pulls them back “at later times”. Concretely, that is a write_note tool and a read_note tool over one scratch directory. The agent writes what it found; the transcript keeps one line saying a note exists; the content comes back only when it is needed.

This is the same idea as the post’s just-in-time approach, where agents “maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools”. A file path is a dozen tokens. The file is two thousand.

Retrieval as memory

Part 10 built a document index with embeddings, a reranker and a query interface. In an agent that index is long-term memory: instead of putting the corpus in the context, the agent asks a question and gets back the three passages that answer it. The first lab’s third tool is exactly this, pointed at the Part 10 index if you built it and at a plain-text search if you did not.

The trade is worth stating plainly. Retrieval keeps the window small and puts a recall problem in its place: if the passage is not retrieved, the agent does not know it. That is a failure you can measure with the evaluation harness from Part 10, which is more than can be said for a fact lost somewhere in the middle of a hundred thousand tokens.

Prefix caching

Part 17 showed that a prefix cache reuses key and value vectors for a shared token prefix, exactly and without changing the output, but only from position zero and only on an identical prefix. An agent transcript is the ideal shape for it, because every turn is the previous turn plus an appendix.

Which makes the arrangement rules from that lesson load-bearing here. Four of them apply directly to a loop:

• Nothing variable at the top. A timestamp, a session id or a turn counter in the system prompt invalidates the entire cache on every single turn. This is the most common way an agent loses its cache, and the symptom is that every turn feels like the first one.
• A stable tool order. Serialise the tool list once and reuse the string. The MCP specification makes the same point from the server side, recommending that servers return tools deterministically because it “improves LLM prompt cache hit rates when tools are included in model context”.
• Append, never rewrite. Editing an earlier message, even to fix whitespace, discards the cache from that point on. Compaction deliberately breaks the prefix, which is one more reason to compact rarely rather than continuously.
• Stable retrieval ordering. If the same three passages come back in a different order, nothing matches.

With prefix caching working, the trace above costs about 5,970 tokens of prefill across the whole task instead of 16,700. Without it, you pay the whole transcript on every turn.

✦Tip

Prove the cache is being hit rather than assuming it. llama-server’s --cache-prompt is documented as enabled by default and its /slots and /metrics endpoints report what each slot holds; vLLM publishes prefix-cache counters in its metrics. Part 17’s measurement script sends the same request twice and looks at whether the counters moved, which is ten minutes that finds the timestamp nobody remembered putting at the top of the system prompt.

Choosing a context length for your track

Pick the number before you start, from your memory tier, the model you are serving and the length of task you intend to run. The table below is a starting point computed from the bytes-per-token figures above, leaving room for weights and an allowance for the engine and the rest of the machine.

Tier	A reasonable agent model	Context to ask for	Why
8 GB	Qwen3-4B at Q4_K_M	16k	2.4 GB of cache on top of 2.5 GB of weights; compaction will be frequent, so build it in from the start
12–16 GB	Qwen3-8B at Q4_K_M	32k	4.8 GB of cache on top of 5.0 GB of weights; enough for a ten-turn task without compacting
24 GB	Qwen3-8B at Q8_0, or gpt-oss-20b	64k	Cheap cache on gpt-oss makes a long agent window affordable at this tier
32 GB	Qwen3-Coder-30B-A3B at Q4_K_M	64k	Four key-value heads keep the cache to 6.4 GB beside 18.6 GB of weights
64 GB and up	Qwen3-Coder-30B-A3B at Q4_K_M or Q8_0	128k	Room for long file reads without compacting mid-task

Two caveats. The context length you ask the engine for is not the model’s trained context window: Qwen3-8B’s card gives a native 32,768 tokens extensible to 131,072 with YaRN, while Qwen3-Coder-30B-A3B’s card gives 262,144 natively. Asking for more than the native window without the documented scaling is asking for degraded behaviour, not more memory. And a longer window that you actually fill is slower on every turn, so the right size is the smallest one your task fits in.

✱Hardware note

On the unified-memory tracks, Apple silicon and the Ryzen AI Max+, the cache competes with everything else on the machine rather than with a dedicated card. Reserve more than the 1.5 to 2 GB used in the diagrams above: Part 4 suggests 8 GB on a 64 GB unified machine, and an agent that also has a browser and an editor open needs that headroom to be real.

Preserve decisions and evidence when compacting

Compaction is a lossy transformation of the conversation. Define what must survive: the user’s objective, constraints, authorised actions, current files or task state, unresolved questions and references to evidence. Long tool outputs can be replaced with summaries and stable paths, but a summary must not invent success for a step that failed.

Create a small test conversation with a changed requirement, a denied action and a partially completed task. Compact it, then ask the agent to continue. Check that it follows the latest requirement, preserves the denial and resumes from the actual state. Compare against continuation with the original context.

Store authoritative state outside prose where possible: task status, operation IDs and file hashes are easier to validate as structured records. Retrieved material and earlier model guesses should retain their provenance rather than becoming trusted instructions through summarisation. Context management is successful when the next action remains correct after the transformation, not merely when the token count becomes smaller.

Recap

A token of context costs layers times key-value heads times head dimension times two tensors times bytes per element, which is 147,456 bytes for Qwen3-8B at 16-bit and 98,304 for the 30B mixture-of-experts models, so the larger model is the cheaper one to give a long window to. An agent transcript grows by an observation per turn and is re-read in full on every turn, which costs prefill time quadratically, cache memory linearly, and attention in a way nothing but a smaller window fixes. Compaction replaces the history with a summary when it passes a threshold; notes and retrieval keep material outside the window and fetch it by name; prefix caching removes the re-reading entirely, provided nothing at the top of the prompt varies and the tool list is serialised in a stable order. Choose the context length from your memory tier before you start, keep it as small as the task allows, and remember that asking the engine for more than the model’s native window is not the same as the model being able to use it.

Check your understanding

Question 1. Qwen3-30B-A3B is nearly four times the parameter count of Qwen3-8B, yet its key-value cache costs two thirds as much per token. Why?

• It is quantised more aggressively
• The cache depends on layers, key-value heads and head dimension, not on parameter count, and it has four key-value heads where the 8B has eight
• Mixture-of-experts models do not cache attention
• It has a shorter context window

Show the answer and why

Answer: The cache depends on layers, key-value heads and head dimension, not on parameter count, and it has four key-value heads where the 8B has eight

Cache size and weight size are independent. That is why the model reference records layers, key-value heads and head dimension separately: for a workload where context is the constraint, a wide mixture-of-experts model with few key-value heads can be the cheaper choice despite being much larger on disk.

Question 2. A five-turn task ends with a 5,970-token transcript, but the model prefilled 16,700 tokens. Which single change removes most of that difference?

• Quantising the key-value cache
• Prefix caching, since every turn is the previous turn plus an appendix and the shared prefix is reused exactly
• A larger context length
• A smaller model

Show the answer and why

Answer: Prefix caching, since every turn is the previous turn plus an appendix and the shared prefix is reused exactly

The gap is re-reading, and a prefix cache reuses the key and value vectors for an identical prefix without changing the output. Cache quantisation reduces memory rather than prefill work, and a larger window makes the problem bigger. The condition is that the prefix really is identical, which is what the arrangement rules are for.

Question 3. Which of these will silently destroy an agent's prefix-cache hits on every turn? Select all that apply.

• A timestamp at the top of the system prompt
• Serialising the tool list from a dictionary whose key order varies between runs
• Appending a new tool result to the end of the transcript
• Re-ranking the same retrieved passages into a different order each turn

Show the answer and why

Answer: A timestamp at the top of the system prompt, Serialising the tool list from a dictionary whose key order varies between runs, Re-ranking the same retrieved passages into a different order each turn

A prefix cache matches from position zero, so anything variable early invalidates everything after it. Appending is the one operation that preserves the whole existing prefix, which is exactly why the loop shape suits caching so well.

Question 4. Your agent is losing track of instructions on long tasks but is not being truncated. What does the context-engineering guidance suggest?

• Raise the context length so more fits
• Reduce what is in the window: compact the history, move material to notes or retrieval, and fetch it back by reference when needed
• Repeat the system prompt every turn
• Switch to a reasoning model

Show the answer and why

Answer: Reduce what is in the window: compact the history, move material to notes or retrieval, and fetch it back by reference when needed

This is context rot rather than truncation: recall falls as the window fills, because the attention budget is spread thinner. Adding room adds more to compete with. Compaction, notes and just-in-time retrieval all shrink what the model has to read, which is the lever that actually applies.

Check my answersClear