Project: A Tiered Inference Architecture

Validated on: written from the documentation cited above and from the two labs in this part; not yet validated on hardware on any track. This project produces a document rather than a measurement, and the measurements it rests on are yours.

Objective

By the end of this project you will have a design document for a language-model serving architecture on the machines you actually own, in which every performance claim points at a measurement in your own lab notebook, and a script will have confirmed that none of the claims are missing their evidence.

The document has six moving parts: roles per machine, the routing layer in front, a prefill pool, a decode pool, a cache tier underneath, and the measurements that justify each of those choices. It ends with the question that separates a design from a wish list: what would you change with one more machine, and which measurement makes you say so.

⌂Single-machine path

A single-machine design is a complete submission. It is not a lesser answer, and on a house network it is frequently the right one. What makes it complete is section 8 of the template: what a second machine would take on, which link it would need, and which of your measurements says so. A design that concludes “one machine, with a host-memory tier and a shared store I do not currently need” is a good design if the numbers support it.

Requirements

Every track needs the lab notebook from Part 1 with entries from Part 18’s cluster-network lab and from both labs in this part, about ninety minutes, and no model download at all. This is a writing and arithmetic exercise built on measurements you already took.

Track S — NVIDIA DGX Spark

All of it applies. If you have a Spark pair with the direct cable, you are the one reader whose per-request link arithmetic may favour splitting the phases, so your document should say explicitly whether it does on your measurements and at what prompt length the answer changes.

Track X — AMD Ryzen AI Max+ 395

All of it applies. These machines are usually memory-rich and network-poor, which makes them excellent decode nodes and poor participants in a per-request transfer. Both of those belong in the document with the measurement behind them.

Track M — Apple silicon

All of it applies, with two substitutions. Where the template asks about the engine’s offloading backend, use llama.cpp’s slot save and restore and mlx-lm’s prompt-cache file, both from the previous lab and from Part 17. Where it asks about a store shared between engine instances, say that your track has no such mechanism in this course and what you would use instead.

Track N — NVIDIA desktop or laptop

All of it applies, and this is the track most likely to have genuinely separate device and host memory, which makes the tier arithmetic in section 6 the most interesting part of your document.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-22-disaggregated-serving"
cd "$LAB_DIR"
pwd
test -f "measurements-example.json"

Expected result: pwd ends in part-22-disaggregated-serving and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Gather the measurements you already have

Open your notebook and collect the lines this project needs. You should have, from earlier work:

• Link throughput and round-trip time for every link between your machines, from Part 18’s lab.
• A baseline load-test line: one machine, no split, no tier, from Part 9 or from this part’s first lab.
• A split line and its comparison, from this part’s first lab, unless you took a reduced path.
• Reuse lines with and without a host-memory tier, from this part’s second lab.
• A shared-store line, if your track could run it.
• A startup-log figure per configuration: the key-value cache size in tokens and the maximum concurrency at your context length.

If any of those is missing, either go and take it or plan to write “not measured” in the document’s final section. Both are acceptable; inventing a number is not.

2. Fill in the measurements file

Fragment — not complete on its own

measurements-example.json

{
  "$comment": "The schema summarise-architecture.py reads. Copy this file to measurements.json, replace every value with your own, and delete this comment. Every number here is a placeholder written as null or as an obviously fictional value: nothing in this file is a measurement, and the script refuses to summarise a file whose measurements are still null.",
  "schema": "part-22/tiered-architecture/1",
  "author_note": "One or two sentences on what this design is for: who uses it, for what, and what the latency requirement is. A design with no requirement cannot be judged.",
  "requirement": {
    "ttft_budget_s": 2.0,
    "tpot_budget_s": 0.06,
    "concurrent_users": 4,
    "typical_prompt_tokens": 4096,
    "typical_answer_tokens": 300
  },
  "model": {
    "id": "qwen3-8b",
    "quant": "bf16",
    "context_length": 8192,
    "kv_bytes_per_token": 147456
  },
  "machines": [
    {
      "name": "node-a",
      "track": "nvidia",
      "accelerator": "a discrete GPU; name it as your own system reports it",
      "device_memory_gb": 24,
      "host_memory_gb": 64,
      "roles": ["prefill", "router"],
      "notes": "Chosen for prefill because it has the most arithmetic throughput and the least memory."
    },
    {
      "name": "node-b",
      "track": "strix",
      "accelerator": "an integrated GPU on unified memory",
      "device_memory_gb": 96,
      "host_memory_gb": 128,
      "roles": ["decode", "cache", "storage"],
      "notes": "Chosen for decode because every conversation in flight keeps its cache here."
    }
  ],
  "links": [
    {
      "from": "node-a",
      "to": "node-b",
      "class": "2.5 gigabit Ethernet",
      "carries": "per-request",
      "measured_gbps": null,
      "measured_rtt_ms": null,
      "source": "Part 18 lab, labbook line label"
    },
    {
      "from": "node-a",
      "to": "clients",
      "class": "house network",
      "carries": "client",
      "measured_gbps": null,
      "measured_rtt_ms": null,
      "source": "Part 18 lab, labbook line label"
    }
  ],
  "tiers": [
    {
      "tier": "device",
      "where": "node-b device memory",
      "size_gb": 20,
      "measured_hit_rate": null,
      "source": "labbook line label"
    },
    {
      "tier": "host",
      "where": "node-b host memory, through the engine's offloading backend",
      "size_gb": 8,
      "measured_hit_rate": null,
      "source": "labbook line label"
    },
    {
      "tier": "shared-store",
      "where": "the Part 18 shared mount, as a directory of key-value blocks",
      "size_gb": 50,
      "measured_hit_rate": null,
      "source": "labbook line label"
    }
  ],
  "routing": {
    "layer": "litellm",
    "policy": "Describe the rule in one sentence. Round-robin is a rule; prefer-the-backend-that-already-holds-most-of-this-prompt is a better one.",
    "source": "Part 9 project, or vLLM's example proxy"
  },
  "measurements": [
    {
      "label": "baseline",
      "configuration": "one machine, no split, no offload",
      "concurrency": 4,
      "ttft_p50_s": null,
      "tpot_p50_s": null,
      "output_tokens_per_s": null,
      "source": "labbook.md, label baseline"
    },
    {
      "label": "disagg",
      "configuration": "prefill on node-a, decode on node-b, shared-directory connector",
      "concurrency": 4,
      "ttft_p50_s": null,
      "tpot_p50_s": null,
      "output_tokens_per_s": null,
      "source": "labbook.md, label disagg"
    },
    {
      "label": "offload-on",
      "configuration": "one machine, host-memory cache tier of 8 GiB",
      "concurrency": 4,
      "ttft_p50_s": null,
      "tpot_p50_s": null,
      "output_tokens_per_s": null,
      "source": "labbook.md, label offload-on"
    }
  ],
  "decisions": [
    {
      "decision": "State one decision, such as: decode runs on node-b.",
      "because": "State why, in one sentence that a reader could disagree with.",
      "evidence": ["baseline", "disagg"]
    },
    {
      "decision": "State the next decision, such as: the phases are not split.",
      "because": "State why.",
      "evidence": ["disagg", "offload-on"]
    }
  ],
  "one_more_machine": "What you would change if you had one more machine of the kind you would actually buy, and which measurement makes you say so. A design that cannot answer this has not been thought about."
}

Download measurements-example.json128 lines

RunnableAll tracks

start from the example and replace every value

cp measurements-example.json measurements.json

Six blocks to complete.

requirement is the latency budget the design has to meet, the number of concurrent users, and the typical prompt and answer lengths. Write it before anything else. A design with no requirement cannot be judged and cannot be defended, and picking the requirement afterwards to match what you measured is the oldest trick in benchmarking.

model names the model, the quantisation, the context length and the bytes of key-value cache per token from the model reference. That last field is what turns the rest of the file into arithmetic. Qwen3-8B is 147,456; Qwen3-30B-A3B is 98,304, and choosing between those two on cache traffic rather than on parameter count is itself a design decision worth recording.

machines lists each machine with its track, its device and host memory and the roles it holds, from Part 18’s six: prefill, decode, router, cache, storage, agent. Most home machines hold several and one machine may hold all of them.

links lists each link with its class, its measured throughput and round-trip time, and the single most important field: what it carries. Per-token, per-request, load-time or client. Part 18 called this the annotation most home clusters get wrong.

tiers lists the cache tiers with their sizes and the hit rate you measured for each.

measurements and decisions are the heart of it: a labelled measurement for every number you will quote, and a decision that cites the labels supporting it.

3. Design the routing layer

This is the section with the most room for thought, so do not rush it.

Two implementations are available to you from this course. LiteLLM, from Part 9’s gateway project, is the one with authentication, keys and quotas, and it is the front door Part 23 will operate. vLLM’s example proxy from this part’s first lab is the one that understands prefill and decode as separate backends, and SGLang’s router is the equivalent on that engine. They are not mutually exclusive: an authenticated gateway in front of a phase-aware proxy is a perfectly sensible arrangement, and it is what the reference layout below draws.

A reference layout to argue with, not to copy

• clientClientschat, an editor, an agent loop from Level 5
• routerAuthenticated gatewayLiteLLM from Part 9: keys, quotas, model aliases, the only thing clients reach
• routerPhase-aware proxythe engine's own proxy or router, where the phases are split; absent otherwise
• prefillPrefill poolone or more instances on the compute-rich machines
• decodeDecode poolone or more instances on the memory-rich machines
• cacheCache tierdevice, then host memory, then a shared store on the Part 18 mount
• storageModel libraryone authoritative copy, read-only to the workers

• Clients ⟷ connected to Authenticated gatewayhouse network, authenticated
• Authenticated gateway ⟷ connected to Phase-aware proxycluster network; the only hop that knows about phases
• Phase-aware proxy ⟷ connected to Prefill poolthe prompt
• Phase-aware proxy ⟷ connected to Decode poolthe continuation
• Prefill pool ⟷ connected to Cache tierwrites key-value blocks
• Cache tier ⟷ connected to Decode poolreads them; per-request traffic
• Model library ⟷ connected to Prefill poolload time only
• Model library ⟷ connected to Decode poolload time only

Collapse whichever boxes your measurements do not justify. A single-machine design collapses everything except the gateway, the model library and the tier, and that is a legitimate reading of this diagram rather than a failure to reach it.

Then write the routing rule, which matters more than the software. Round-robin is a rule. “Prefer the backend that already holds most of this prompt, and among those prefer the least busy” is a better one, and it is the shape of the cost function Dynamo’s router design documents: the prefill work a request would cost, discounted separately for the blocks the worker already has in device memory, in host memory and on disk, plus the decode load it is already carrying, lowest cost wins.

▷Exercise

Write your routing rule as a sentence with an “and” in it, naming what it prefers and how it breaks ties. Then write the one workload that would make your rule behave badly. Every routing rule has one, and a design that names its own is more trustworthy than one that does not.

4. Size the two pools, or decide there is only one

Whether or not you split the phases, the design has to say how many engine instances exist and what constrains each of them, and the two pools are constrained by different things.

The decode pool is constrained by cache. Every conversation in flight holds its whole key-value state until the answer finishes, so the number of concurrent conversations a decode instance can serve is the key-value cache size from its startup log divided by the context length you allow. That figure, not the parameter count and not the accelerator’s name, is what tells you how many people the pool serves. Write it down per instance and add them up. If the total is below the concurrent user count in your requirement, you have found the constraint before your users did.

The prefill pool is constrained by arithmetic and by arrival rate. A prefill instance only holds the prompts it is reading right now, so its memory is nearly all weights and its capacity is a question of how fast it gets through a queue. The useful number is the prefill time for your typical prompt, measured, against how often such a prompt arrives. One instance that takes a second per prompt cannot absorb four prompts a second whatever its memory says.

Where there is one pool, say which constraint you are living with. A consolidated design is constrained by both at once, and the interference from Lesson 1 is the mechanism by which they collide. Naming which of the two you expect to hit first, and at what load, is what makes the design testable rather than merely plausible.

▤Memory budget

Two numbers from the startup log carry this whole section: the key-value cache size in tokens and the maximum concurrency the engine reports at your context length. They are available before a single request arrives, they cost nothing to record, and they answer the capacity question more directly than any benchmark. Part 9 introduced them and every serving page since has asked for them.

5. Write the document

Fragment — not complete on its own

architecture-template.md

# Tiered inference architecture — <your service's name>

<!--
Purpose: the design document for Part 22's project. Copy this file to
         architecture.md, fill in every section, and delete these comments and every
         angle-bracket placeholder as you go.
Platform: all
Minimum memory: 16 GB on at least one machine; a single-machine design is a valid
         submission if section 8 says what a second machine would change.
Assumes: measurements.json filled in from measurements-example.json and passing
         `python3 summarise-architecture.py --file measurements.json --strict`, and the
         link figures from Part 18's lab in your notebook.

Rule for the whole document: every claim about performance names a measurement label
from measurements.json. A sentence that does not is either a fact from a vendor page,
with the page named, or arithmetic, said to be arithmetic.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Model:** <id from the course model reference>
**Engine and version:** <as reported by the engine> · **Measurements file:** `measurements.json`

---

## 1. What this serves, and the requirement

<Two or three sentences. Who uses it, for what, and how many at once. Then the numbers
the design has to meet: a time-to-first-token budget, a time-per-output-token budget, a
concurrent-user count, a typical prompt length and a typical answer length. Copy them
from the `requirement` block of measurements.json so the two files cannot disagree.>

<A design with no requirement cannot be judged, and neither can it be defended. If you
genuinely have no latency budget, say so and say what you are optimising instead.>

## 2. The machines and their roles

| Machine | Track | Accelerator | Device memory | Host memory | Roles | Why this role |
| --- | --- | --- | --- | --- | --- | --- |
| <name> | <S/X/M/N> | <as your system reports it> | <GB> | <GB> | <from: prefill, decode, router, cache, storage, agent> | <one sentence> |

<Part 18 named six roles. Most home machines hold several. The "why" column is the part
that matters: "it has the most arithmetic throughput and the least memory, so it prefills"
is a reason; "it is the newest" is not.>

<If one machine holds every role, say so plainly here. That is a design, not an absence
of one, and the rest of this document still applies.>

## 3. The links, and what each one carries

| From | To | Class | Measured throughput | Measured round trip | Carries |
| --- | --- | --- | --- | --- | --- |
| <machine> | <machine> | <2.5 GbE, Thunderbolt 5, ConnectX-7 QSFP, …> | <from Part 18's lab> | <from Part 18's lab> | <per-token / per-request / load-time / client> |

<The last column is the whole point. A link carrying per-token traffic has to be fast. A
link carrying per-request traffic has to move one request's key-value cache in less time
than the prefill it replaced. A link carrying load-time traffic only has to be wide, and a
slow one merely delays the start.>

**One request's key-value cache:** <bytes per token from the model reference> ×
<typical prompt tokens> = <result>. **Time to move it on the per-request link, from
arithmetic at the measured throughput:** <result>. **Prefill time for the same prompt,
measured:** <label from measurements.json>.

<Those three figures next to each other are the argument for or against splitting the
phases. State which way they point, in one sentence.>

## 4. The routing layer

**What sits in front:** <LiteLLM from Part 9's gateway project, vLLM's example proxy,
SGLang's router, or something you wrote.>

**The rule it applies:** <One paragraph. Round-robin is a rule. "Prefer the backend that
already holds most of this prompt, and among equals prefer the least busy" is a better
one, and it is the shape of the cost function in Dynamo's router design that Lesson 4
quoted. Say which you implemented and which you would implement with more time.>

**Authentication and exposure:** <Who can reach it, on which interface, with what
credential. Nothing in Part 22 authenticates anything by itself; Part 23 is where a
service gets exposed properly. Say what is true today rather than what you intend.>

## 5. The prefill pool and the decode pool

**Prefill:** <Which machines, how many instances each, what context length and memory
fraction, and what happens when they are all busy.>

**Decode:** <The same, plus the number that actually constrains it: how many concurrent
conversations fit, from the key-value cache size in the startup log divided by your
context length.>

**Or neither:** <If you decided not to split the phases, say so here and point at the
measurement that decided it. That is the expected outcome on ordinary Ethernet and it is
a result, not a failure.>

## 6. The cache tier

| Tier | Where | Size | Measured hit rate | What it saved |
| --- | --- | --- | --- | --- |
| Device | <machine's device memory> | <GB> | <label> | <one sentence> |
| Host | <machine's host memory, through the offloading backend> | <GB> | <label> | |
| Disk or shared store | <path, and which machines can read it> | <GB> | <label> | |

<Say explicitly whether any tier is shared between engine instances, and if so what the
measurement showed about cross-instance reuse. Say what happens to the tier on a restart.
And say where it lives on disk, because a store of key-value blocks is the users' prompts
in the model's own representation and belongs wherever you keep transcripts.>

## 7. The measurements that justify each decision

| Decision | Because | Evidence |
| --- | --- | --- |
| <one decision> | <one sentence a reader could disagree with> | <labels from measurements.json> |

<This table and the `decisions` block of measurements.json say the same thing; the script
checks that every label you cite exists. Aim for four to six decisions. A decision with no
evidence is a preference, and a preference is fine as long as it is labelled as one.>

## 8. What I would change with one more machine

<Name the machine you would actually buy or repurpose, not an ideal one. Say which role it
would take, which link it would need, and which measurement in this document makes you say
so. Then say what you would measure to find out whether you were right.>

<If your design is a single machine, this section is the most important one in the
document and it is what makes a single-machine submission complete.>

## 9. What is not true here

<The honest section. What did you not measure? Which figures are arithmetic rather than
measurements? Which track did you not run? What did you copy from documentation without
verifying? Every course page in Level 4 carries a version of this section, and a design
document without one is asking to be trusted further than it has earned.>

Download architecture-template.md130 lines

RunnableAll tracks

start from the template

cp architecture-template.md architecture.md

Nine sections, each with a job.

1. What this serves, and the requirement. Who, for what, how many at once, and the budget.
2. The machines and their roles. The “why this role” column is the one that carries the section.
3. The links, and what each carries. With the three-figure comparison: payload, transfer time from arithmetic, prefill time measured.
4. The routing layer. Software, rule, and what is authenticated today rather than what you plan.
5. The prefill pool and the decode pool. Or a paragraph saying you did not split, with the measurement that decided it.
6. The cache tier. Each tier, its size, its measured hit rate and what it saved.
7. The measurements that justify each decision. The same content as the decisions block, in prose.
8. What you would change with one more machine. A machine you would actually buy.
9. What is not true here. What you did not measure, what is arithmetic, what you copied from documentation without verifying.

!Important

Section 9 is not a disclaimer and it is not optional. Every Level 4 page in this course carries a version of it, because the difference between a design document and a sales document is whether it tells you what it does not know. Write it last and write it honestly; it is the section a reader will trust the rest of the document on the strength of.

6. Check it

RunnableAll tracks

summarise-architecture.py

#!/usr/bin/env python3
"""Check a tiered-architecture design against its own measurements, and summarise it.

Purpose: read the measurements.json a reader wrote for this part's project and answer
    three questions a design document should not be allowed to dodge. Does every decision
    cite a measurement that exists in the file? Is every measurement filled in rather than
    left as null? And does the arithmetic agree with the design: how long would one
    request's key-value cache take to cross each link that is marked as carrying
    per-request traffic, at the throughput that link was measured at? Prints the answers
    and appends one summary line to the lab notebook.
Platform: all. Pure Python standard library: no pip install and no server needed.
Minimum memory: 1 GB. It reads a JSON file.
Assumes: Python 3.9 or later; a measurements.json copied from measurements-example.json
    and filled in; the measured link throughput figures from Part 18's lab. The script
    checks structure and arithmetic. It cannot check whether your reasoning is good, which
    is what the written document is for.

Usage:
    python3 summarise-architecture.py --file measurements.json --labbook labbook.md

    python3 summarise-architecture.py --file measurements.json --strict
        exit non-zero if any check fails, which is what to run before you call it done
"""

from __future__ import annotations

import argparse
import json
import sys
import time
from pathlib import Path

SCHEMA = "part-22/tiered-architecture/1"

REQUIRED_TOP = [
    "schema", "requirement", "model", "machines", "links",
    "tiers", "routing", "measurements", "decisions", "one_more_machine",
]
REQUIRED_MACHINE = ["name", "track", "device_memory_gb", "roles"]
REQUIRED_LINK = ["from", "to", "class", "carries"]
REQUIRED_MEASUREMENT = ["label", "configuration", "concurrency",
                        "ttft_p50_s", "tpot_p50_s", "output_tokens_per_s"]
KNOWN_ROLES = {"prefill", "decode", "router", "cache", "storage", "agent"}
KNOWN_CARRIES = {"per-token", "per-request", "load-time", "client"}
KNOWN_TRACKS = {"spark", "strix", "mac", "nvidia"}


def human_bytes(n: float) -> str:
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if abs(n) < 1000 or unit == "TB":
            return f"{n:.0f} {unit}" if unit == "B" else f"{n:.2f} {unit}"
        n /= 1000
    return f"{n:.2f} TB"


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--file", default="measurements.json",
                        help="the design's measurements file")
    parser.add_argument("--labbook", default="labbook.md", help="notebook to append to")
    parser.add_argument("--lab", default="part-22/project-a-tiered-inference-architecture")
    parser.add_argument("--strict", action="store_true",
                        help="exit non-zero when any check fails")
    parser.add_argument("--print-only", action="store_true", help="write nothing")
    args = parser.parse_args()

    path = Path(args.file)
    try:
        design = json.loads(path.read_text(encoding="utf-8"))
    except OSError as exc:
        print(f"Could not read {path}: {exc}", file=sys.stderr)
        return 2
    except json.JSONDecodeError as exc:
        print(f"{path} is not valid JSON: {exc}", file=sys.stderr)
        return 2

    problems: list[str] = []
    notes: list[str] = []

    # --- structure ---------------------------------------------------------------------
    if design.get("schema") != SCHEMA:
        problems.append(f'schema should be "{SCHEMA}", found {design.get("schema")!r}')
    for key in REQUIRED_TOP:
        if key not in design:
            problems.append(f"missing top-level key: {key}")

    machines = design.get("machines") or []
    if not machines:
        problems.append("no machines listed; a design with no machines is not a design")
    names = set()
    roles_present = set()
    for i, machine in enumerate(machines):
        for key in REQUIRED_MACHINE:
            if key not in machine:
                problems.append(f"machine {i}: missing {key}")
        name = machine.get("name")
        if name in names:
            problems.append(f"two machines are called {name!r}")
        names.add(name)
        if machine.get("track") not in KNOWN_TRACKS:
            problems.append(f"machine {name!r}: track should be one of "
                            f"{sorted(KNOWN_TRACKS)}, found {machine.get('track')!r}")
        for role in machine.get("roles") or []:
            if role not in KNOWN_ROLES:
                problems.append(f"machine {name!r}: unknown role {role!r}; "
                                f"Part 18 names {sorted(KNOWN_ROLES)}")
            roles_present.add(role)

    for role in ("prefill", "decode", "router"):
        if role not in roles_present:
            problems.append(f"no machine holds the {role} role; every design needs one, "
                            f"even if one machine holds all three")

    links = design.get("links") or []
    for i, link in enumerate(links):
        for key in REQUIRED_LINK:
            if key not in link:
                problems.append(f"link {i}: missing {key}")
        if link.get("carries") not in KNOWN_CARRIES:
            problems.append(f"link {i}: carries should be one of {sorted(KNOWN_CARRIES)}, "
                            f"found {link.get('carries')!r}; that annotation is the whole "
                            f"point of the link list")
        for end in ("from", "to"):
            other = link.get(end)
            if other not in names and other != "clients":
                notes.append(f"link {i}: {end} is {other!r}, which is not a machine in "
                             f"this design (use \"clients\" for the outside world)")

    measurements = design.get("measurements") or []
    by_label: dict[str, dict] = {}
    for i, m in enumerate(measurements):
        for key in REQUIRED_MEASUREMENT:
            if key not in m:
                problems.append(f"measurement {i}: missing {key}")
        label = m.get("label")
        if label:
            by_label[label] = m
        for key in ("ttft_p50_s", "tpot_p50_s", "output_tokens_per_s"):
            if m.get(key) is None:
                problems.append(f"measurement {label!r}: {key} is still null; the design "
                                f"is not finished until every number is real")

    decisions = design.get("decisions") or []
    if not decisions:
        problems.append("no decisions listed")
    for i, d in enumerate(decisions):
        evidence = d.get("evidence") or []
        if not evidence:
            problems.append(f"decision {i}: no evidence cited; every decision in this "
                            f"project has to point at a measurement")
        for label in evidence:
            if label not in by_label:
                problems.append(f"decision {i}: cites measurement {label!r}, which is not "
                                f"in this file")
        if not str(d.get("because", "")).strip():
            problems.append(f"decision {i}: no reason given")

    if not str(design.get("one_more_machine", "")).strip():
        problems.append("one_more_machine is empty; a design that cannot say what a "
                        "second or third machine would change has not been thought about")

    # --- arithmetic --------------------------------------------------------------------
    model = design.get("model") or {}
    per_token = model.get("kv_bytes_per_token")
    prompt_tokens = (design.get("requirement") or {}).get("typical_prompt_tokens")
    payload = None
    if isinstance(per_token, int) and isinstance(prompt_tokens, int):
        payload = per_token * prompt_tokens

    print(f"==> {path}")
    print(f"    machines {len(machines)}, links {len(links)}, "
          f"measurements {len(measurements)}, decisions {len(decisions)}")
    if payload:
        print(f"    one typical request's key-value cache: {human_bytes(payload)} "
              f"({per_token} bytes per token x {prompt_tokens} tokens)")

    print("\n==> Links marked as carrying per-request traffic")
    per_request = [x for x in links if x.get("carries") == "per-request"]
    if not per_request:
        print("    None. This design does not move a key-value cache between machines,")
        print("    which is a perfectly good answer and should be stated as a decision.")
    for link in per_request:
        gbps = link.get("measured_gbps")
        label = f"{link.get('from')} to {link.get('to')} ({link.get('class')})"
        if not payload:
            print(f"    {label}: no payload arithmetic (set kv_bytes_per_token and "
                  f"typical_prompt_tokens)")
            continue
        if not isinstance(gbps, (int, float)) or gbps <= 0:
            problems.append(f"link {label}: carries per-request traffic but has no "
                            f"measured_gbps; Part 18's lab is where that number comes from")
            print(f"    {label}: not measured")
            continue
        seconds = payload / (gbps * 1e9 / 8)
        print(f"    {label}: about {seconds:.2f} s to move one request's cache "
              f"at the measured {gbps} gigabits per second")
        budget = (design.get("requirement") or {}).get("ttft_budget_s")
        if isinstance(budget, (int, float)) and seconds > budget:
            print(f"      That is longer than the whole time-to-first-token budget of "
                  f"{budget} s. Say so in the document.")

    # --- report ------------------------------------------------------------------------
    print()
    if notes:
        print("==> Notes")
        for note in notes:
            print(f"    - {note}")
        print()
    if problems:
        print(f"==> {len(problems)} thing(s) to fix")
        for problem in problems:
            print(f"    - {problem}")
    else:
        print("==> Every check passed. Every decision cites a measurement that exists,")
        print("    every measurement has a number in it, and every per-request link has a")
        print("    measured throughput. The reasoning is still yours to defend.")

    record = {
        "lab": args.lab,
        "record": "architecture",
        "file": str(path),
        "machines": [m.get("name") for m in machines],
        "roles": sorted(roles_present),
        "links_per_request": len(per_request),
        "measurement_labels": sorted(by_label),
        "decisions": len(decisions),
        "kv_bytes_per_request": payload,
        "problems": problems,
        "passed": not problems,
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    }

    if not args.print_only:
        with open(args.labbook, "a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True) + "\n")
        print(f"\n    Appended one architecture line to {args.labbook}")

    if problems and args.strict:
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Download summarise-architecture.py244 lines

RunnableAll tracks

check the design against its own measurements

python3 summarise-architecture.py \
  --file measurements.json \
  --labbook labbook.md \
  --strict

Output — what you should see

==> measurements.json
    machines 2, links 2, measurements 3, decisions 4
    one typical request's key-value cache: 603.98 MB (147456 bytes per token x 4096 tokens)

==> Links marked as carrying per-request traffic
    node-a to node-b (2.5 gigabit Ethernet): about ... s to move one request's cache
      at the measured ... gigabits per second

==> Every check passed. Every decision cites a measurement that exists,
    every measurement has a number in it, and every per-request link has a
    measured throughput. The reasoning is still yours to defend.

The script checks four things a design document can otherwise get away with: that every decision cites a measurement label that actually exists in the file, that no measurement is still null, that every link marked as carrying per-request traffic has a measured throughput behind it, and that the “one more machine” section is not empty. It also does the transfer arithmetic for you and says when the answer exceeds your own stated latency budget.

What it cannot check is whether your reasoning is any good. That is what section 7 of the document is for, and it is why the script says so in its own output.

7. Read it back as somebody else

Twenty minutes, and it is the task most likely to change the document.

Read your own design as if you were being asked to operate it, and mark every sentence that would make you ask a question. Three questions in particular:

“How do you know?” on every performance claim. If the answer is not a label in measurements.json, either take the measurement or downgrade the claim to a guess and say so.

“What happens when this breaks?” on every component. The decode machine reboots; the shared store’s mount goes away mid-request; the prefill instance runs out of memory at concurrency 12. You do not have to solve these here, because Part 23 is where operations happen, but a design that has never been asked is a design that will be asked at three in the morning instead.

“Why not one machine?” on every second machine. This is the question the whole part has been building towards, and if the honest answer is “no reason”, the design should say so and become simpler.

Connect each routing rule to measured evidence

Bring forward the raw baseline, disaggregation and cache-tier results before editing the architecture template. For every proposed route, name the workload condition that selects it and the measurement supporting that decision. Avoid filling missing measurements with optimistic estimates from another machine.

Check the data flow for a short request, a long request, an unavailable worker and a cancelled request. Identify who owns the request, where cache state resides and which process releases resources. Write the fallback rule and whether it changes model quality, privacy location or latency targets.

Use the summariser as an arithmetic check, then inspect the source values manually. A correctly calculated table can still contain mismatched prompt lengths or different models. Include a simpler single-pool alternative and explain why added tiers are justified, or choose that simpler alternative when the evidence does not support splitting. The final document should contain the topology, capacity assumptions, routing policy, failure procedure and links to result files. Another operator should be able to determine both how to start the system and when to abandon a tier that is no longer beneficial under a changed workload.

Validation

• measurements.json exists, is valid JSON, and summarise-architecture.py --strict exits without reporting anything to fix.
• architecture.md has all nine sections filled in with no angle-bracket placeholders left.
• Every performance claim in the prose names a measurement label that appears in measurements.json.
• Section 8 names a specific machine, a role for it, a link it would need, and the measurement that motivates it, and it is filled in even if your design is a single machine.
• Section 9 lists at least three things you did not measure or could not verify.
• labbook.md contains the architecture record the script appended.

Expected outcome

Pending validationThe shape of a finished design — to be filled in with your own

Section	What it commits to	Evidence label
Requirement	a time-to-first-token budget, a time-per-output-token budget and a user count	stated, not measured
Roles	which machine prefills, which decodes, which routes	pending
Links	which link carries per-request traffic, and how long a request's cache takes on it	pending
Routing	the rule, and the workload that would defeat it	pending
Pools	whether the phases are split at all, and at what prompt length the answer changes	pending
Cache tier	which tiers exist, their sizes and their measured hit rates	pending
One more machine	the machine, its role, its link and the measurement behind it	pending

whatever machines you own; the reference cluster for the validation pass · the engine your design uses to be recorded with your document · the model named in your measurements file, as your design states · 8,192 tokens of context · 2026-09-09

This table describes the document rather than measuring anything. The evidence column in your own copy holds the labels from measurements.json, which is what the checking script verifies.

Two documents count as finished, and they look very different.

The split design. Two or more machines, a per-request link fast enough that the arithmetic favours it, a prefill pool and a decode pool, a proxy that knows about both, and a cache tier underneath. Section 7 cites a first-lab comparison in which the split improved time to first token. This is the rarer outcome at home and it needs an RDMA link to be honest.

The consolidated design. One or two machines, no phase split, an authenticated gateway in front, a host-memory tier and possibly a shared store, and a section 5 that says plainly why the phases are not separated with the measurement that decided it. Section 8 names the machine that would change the answer. This is the commoner outcome and it is the better piece of engineering when the numbers say so.

?Reality check

The claim worth resisting while you write this is that a more elaborate architecture is a better one. The measurement that matters is whether the design meets its requirement, and a design that meets it on one machine has beaten a design that meets it on three. Level 4 spent five parts showing you how to add machines; this project is where you decide whether to.

Troubleshooting

The script says a decision cites a measurement that is not in the file. Either the label is misspelled or the measurement was never recorded. Both are worth catching: a decision resting on a measurement you did not take is exactly what this project exists to prevent.

The script says a per-request link has no measured throughput. Go back to Part 18’s lab notebook and copy the iperf3 figure in. If you never measured that link, mark the link as carrying something else, or take the measurement; a rated speed from a box is not a measurement and the field is named measured_gbps deliberately.

Every measurement is null because you took a reduced path. Fill in the ones you have, delete the measurement entries you cannot fill, and delete the decisions that depended on them. A shorter document where everything is true beats a complete one with holes in it.

You cannot decide between two routing rules. Write both into section 4, say what would distinguish them, and put the experiment in section 8 alongside the extra machine. An undecided question that names its own test is a legitimate part of a design.

The design keeps growing. A common failure and the template is partly to blame. Cut it back to the machines you own and the requirement you stated. Everything else belongs in section 8.

Cleanup

Nothing to clean up: this project starts no servers. Keep architecture.md and measurements.json beside your notebook, because Part 23 operates what you designed here and the capstone in Part 28 is written from these documents.

What you learned

• A design is a set of decisions with evidence, not a diagram. The script checks the second half of that sentence, and the checking is what makes the document worth re-reading in six months.
• The link annotation is the design. Marking each link as carrying per-token, per-request, load-time or client traffic tells you which link has to be fast, and Part 18 named it as the thing home clusters most often get wrong.
• The routing rule matters more than the routing software. Dynamo’s cost function is a scheduler’s whole opinion written as arithmetic, and writing your own rule as a sentence with an “and” in it is the transferable skill.
• Stating the requirement first is what makes the rest judgeable. A budget chosen after the numbers are in is not a budget: it is a description of what happened.
• Fewer machines is a result. Every part of Level 4 taught a way to add one. Deciding not to, with a measurement behind the decision, is what the level was for.
• Section 9 is the one that earns trust. A document that says what it did not measure is a document whose other claims are worth reading.

Record in the notebook: the architecture line the script appended, and one paragraph in your own words on what surprised you between the design you expected to write at the start of this part and the one you wrote at the end.

Check your understanding

Question 1. Your design uses one machine, no phase split, and a host-memory cache tier. Is it a complete submission?

• No; the project requires at least two machines
• Yes, provided section 8 says what a second machine would take on, which link it would need, and which measurement motivates it
• Only if you also design a two-machine version
• Only on Track M

Show the answer and why

Answer: Yes, provided section 8 says what a second machine would take on, which link it would need, and which measurement motivates it

On a house network the consolidated design is frequently the correct one, and Lesson 2's arithmetic explains why. What makes it a design rather than an absence of one is that it names the condition under which the answer would change.

Question 2. What is the single most important field in the links block of the measurements file?

• The link class
• What the link carries: per-token, per-request, load-time or client traffic, because that is what says whether the link has to be fast
• The round-trip time
• The source of the measurement

Show the answer and why

Answer: What the link carries: per-token, per-request, load-time or client traffic, because that is what says whether the link has to be fast

Part 18 named this annotation as the thing most home clusters get wrong. A per-token link must be fast; a per-request link must move one request's cache in less time than the prefill it replaced; a load-time link merely delays the start. The checking script requires a measured throughput on any link marked per-request.

Question 3. Which of these does summarise-architecture.py check? Select all that apply.

• That every decision cites a measurement label that exists in the file
• That no measurement is still null
• That every link carrying per-request traffic has a measured throughput
• That your reasoning is sound

Show the answer and why

Answer: That every decision cites a measurement label that exists in the file, That no measurement is still null, That every link carrying per-request traffic has a measured throughput

The script says so itself in its own output. It can verify that evidence exists and that arithmetic was done; it cannot verify that the conclusion drawn from the evidence is a good one. That is what section 7 of the document is for and why the project asks you to read it back as somebody else.

Question 4. Why does the template ask you to write the requirement before anything else?

• Because the file format demands it
• Because a design with no latency budget cannot be judged, and a budget chosen after the numbers are in is a description of what happened rather than a criterion
• Because it is the shortest section
• Because the router needs it

Show the answer and why

Answer: Because a design with no latency budget cannot be judged, and a budget chosen after the numbers are in is a description of what happened rather than a criterion

This is the same discipline the first lab applied by asking you to write your prediction down before running anything. A criterion chosen after the result is not a criterion, and the project is designed so that the temptation is at least visible.

Question 5. A colleague reviews your document and asks "how do you know?" about a sentence claiming the decode machine handles eight concurrent conversations. What should you do?

• Cite the machine's memory size
• Point at a measurement label in measurements.json, or downgrade the claim to a stated assumption and move it to section 9
• Remove the sentence
• Cite the vendor specification for the accelerator

Show the answer and why

Answer: Point at a measurement label in measurements.json, or downgrade the claim to a stated assumption and move it to section 9

Every performance claim in the prose has to name a label. The startup log's key-value cache size and maximum concurrency figure is the right evidence for that particular claim, and if you did not record it, saying so in section 9 is the honest alternative to implying you did.

Check my answersClear