# Tiered inference architecture — <your service's name>

<!--
Purpose: the design document for Part 22's project. Copy this file to
         architecture.md, fill in every section, and delete these comments and every
         angle-bracket placeholder as you go.
Platform: all
Minimum memory: 16 GB on at least one machine; a single-machine design is a valid
         submission if section 8 says what a second machine would change.
Assumes: measurements.json filled in from measurements-example.json and passing
         `python3 summarise-architecture.py --file measurements.json --strict`, and the
         link figures from Part 18's lab in your notebook.

Rule for the whole document: every claim about performance names a measurement label
from measurements.json. A sentence that does not is either a fact from a vendor page,
with the page named, or arithmetic, said to be arithmetic.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Model:** <id from the course model reference>
**Engine and version:** <as reported by the engine> · **Measurements file:** `measurements.json`

---

## 1. What this serves, and the requirement

<Two or three sentences. Who uses it, for what, and how many at once. Then the numbers
the design has to meet: a time-to-first-token budget, a time-per-output-token budget, a
concurrent-user count, a typical prompt length and a typical answer length. Copy them
from the `requirement` block of measurements.json so the two files cannot disagree.>

<A design with no requirement cannot be judged, and neither can it be defended. If you
genuinely have no latency budget, say so and say what you are optimising instead.>

## 2. The machines and their roles

| Machine | Track | Accelerator | Device memory | Host memory | Roles | Why this role |
| --- | --- | --- | --- | --- | --- | --- |
| <name> | <S/X/M/N> | <as your system reports it> | <GB> | <GB> | <from: prefill, decode, router, cache, storage, agent> | <one sentence> |

<Part 18 named six roles. Most home machines hold several. The "why" column is the part
that matters: "it has the most arithmetic throughput and the least memory, so it prefills"
is a reason; "it is the newest" is not.>

<If one machine holds every role, say so plainly here. That is a design, not an absence
of one, and the rest of this document still applies.>

## 3. The links, and what each one carries

| From | To | Class | Measured throughput | Measured round trip | Carries |
| --- | --- | --- | --- | --- | --- |
| <machine> | <machine> | <2.5 GbE, Thunderbolt 5, ConnectX-7 QSFP, …> | <from Part 18's lab> | <from Part 18's lab> | <per-token / per-request / load-time / client> |

<The last column is the whole point. A link carrying per-token traffic has to be fast. A
link carrying per-request traffic has to move one request's key-value cache in less time
than the prefill it replaced. A link carrying load-time traffic only has to be wide, and a
slow one merely delays the start.>

**One request's key-value cache:** <bytes per token from the model reference> ×
<typical prompt tokens> = <result>. **Time to move it on the per-request link, from
arithmetic at the measured throughput:** <result>. **Prefill time for the same prompt,
measured:** <label from measurements.json>.

<Those three figures next to each other are the argument for or against splitting the
phases. State which way they point, in one sentence.>

## 4. The routing layer

**What sits in front:** <LiteLLM from Part 9's gateway project, vLLM's example proxy,
SGLang's router, or something you wrote.>

**The rule it applies:** <One paragraph. Round-robin is a rule. "Prefer the backend that
already holds most of this prompt, and among equals prefer the least busy" is a better
one, and it is the shape of the cost function in Dynamo's router design that Lesson 4
quoted. Say which you implemented and which you would implement with more time.>

**Authentication and exposure:** <Who can reach it, on which interface, with what
credential. Nothing in Part 22 authenticates anything by itself; Part 23 is where a
service gets exposed properly. Say what is true today rather than what you intend.>

## 5. The prefill pool and the decode pool

**Prefill:** <Which machines, how many instances each, what context length and memory
fraction, and what happens when they are all busy.>

**Decode:** <The same, plus the number that actually constrains it: how many concurrent
conversations fit, from the key-value cache size in the startup log divided by your
context length.>

**Or neither:** <If you decided not to split the phases, say so here and point at the
measurement that decided it. That is the expected outcome on ordinary Ethernet and it is
a result, not a failure.>

## 6. The cache tier

| Tier | Where | Size | Measured hit rate | What it saved |
| --- | --- | --- | --- | --- |
| Device | <machine's device memory> | <GB> | <label> | <one sentence> |
| Host | <machine's host memory, through the offloading backend> | <GB> | <label> | |
| Disk or shared store | <path, and which machines can read it> | <GB> | <label> | |

<Say explicitly whether any tier is shared between engine instances, and if so what the
measurement showed about cross-instance reuse. Say what happens to the tier on a restart.
And say where it lives on disk, because a store of key-value blocks is the users' prompts
in the model's own representation and belongs wherever you keep transcripts.>

## 7. The measurements that justify each decision

| Decision | Because | Evidence |
| --- | --- | --- |
| <one decision> | <one sentence a reader could disagree with> | <labels from measurements.json> |

<This table and the `decisions` block of measurements.json say the same thing; the script
checks that every label you cite exists. Aim for four to six decisions. A decision with no
evidence is a preference, and a preference is fine as long as it is labelled as one.>

## 8. What I would change with one more machine

<Name the machine you would actually buy or repurpose, not an ideal one. Say which role it
would take, which link it would need, and which measurement in this document makes you say
so. Then say what you would measure to find out whether you were right.>

<If your design is a single machine, this section is the most important one in the
document and it is what makes a single-machine submission complete.>

## 9. What is not true here

<The honest section. What did you not measure? Which figures are arithmetic rather than
measurements? Which track did you not run? What did you copy from documentation without
verifying? Every course page in Level 4 carries a version of this section, and a design
document without one is asking to be trusted further than it has earned.>