# Inference service checklist — <the service you are building>

<!--
Purpose: the deliverable of Capstone 2. Not a list of intentions: every row names the
         check that proves the item is true, the command that produced it and what it
         printed. A row whose check has not been run is unfinished, and marking it so is
         the useful state.
Platform: all
Minimum memory: 8 GB for the smallest configuration; the memory floor is whichever model
         your service serves, from the Capstone 1 plan
Assumes: the gateway from Part 9's project, the front-end and reverse proxy from Part 7's
         lab, the monitoring stack from Part 23's lab, and the backup and upgrade practice
         from Part 23. This checklist assembles them into one service rather than teaching
         them again.

Status values: "done" with evidence, "partial" with what is missing, or "not done" with
the reason. Delete every comment block, including this one.
No address, hostname or key belongs in this file if you intend to show it to anybody.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Service name:** <name> ·
**Machines:** <from the Capstone 1 plan>

---

## 1. Engines, one row per model instance

| Machine | Engine and version | Model | Quantisation | Context configured | Started by | Check that it serves | What the check printed | Status |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| <name> | <e.g. llama.cpp bNNNN> | <id> | <e.g. Q4_K_M> | <n tokens> | <service unit, container, script> | <a request for the model list, and one completion> | <the model id it reported> | <done/partial/not done> |

**Key-value cache reported at startup, per instance:** <the figure the engine logged, in
tokens> · **Maximum concurrency it implies at your context:** <cache tokens ÷ context>

<Those two numbers are the capacity of your service and they cost nothing to read. Record
them here before any load test, because the load test will only confirm what they said.>

**Restart behaviour:** <what happens to an in-flight request when an engine restarts, and
whether the engine comes back by itself.>

## 2. The gateway

| Item | Value | Check | What it printed | Status |
| --- | --- | --- | --- | --- |
| Software and version | <LiteLLM, llama-swap, or both> | <version command> | | |
| Address it listens on | <loopback or the private interface; not written here> | <listening-socket listing> | | |
| Model names it exposes | <the permanent aliases, e.g. chat, coder, embed> | <the model-list endpoint> | | |
| Which engine each name reaches | <table or config file reference> | <one completion per name> | | |
| Keys, one per application | <how many, and what each is for> | <a request with the wrong key> | <the refusal> | |
| Per-key limits | <requests or tokens per minute, budgets, expiry> | <a request past the limit> | <the refusal> | |
| Fallbacks | <what happens when the first backend is down> | <stop a backend, send a request> | | |
| Usage logging | <where it goes and how long it is kept> | <a log line for a known request> | | |

<The gateway is the only thing a client should ever address, which is what makes the
permanent names worth more than the software: a model can be replaced underneath a name
without touching a single client.>

## 3. The front-end

| Item | Value | Check | Status |
| --- | --- | --- | --- |
| Software and version | <e.g. Open WebUI> | <version> | |
| Accounts | <how many, and whether registration is open> | <a second account's first login> | |
| Which gateway name it uses | <alias> | <a conversation that reaches the right engine> | |
| Context length per conversation | <n tokens, and what happens past it> | <a conversation long enough to find out> | |
| Where its data lives | <path or volume> | <its size> | |

## 4. Dashboards and alerts

**The questions the dashboard answers, written before the panels were built:**

1. <e.g. is the service up and answering?>
2. <e.g. how long is a request waiting?>
3. <e.g. how full is the key-value cache?>
4. <e.g. how much memory is each machine using?>
5. <e.g. how much power is each machine drawing?>
6. <e.g. what failed, and when?>

| Question | Panel | Metric source | Status |
| --- | --- | --- | --- |
| <1> | <panel name> | <the engine's metrics endpoint, the node exporter, the accelerator exporter> | |

| Alert | Condition | For how long | Where it goes | Proved to fire on | Status |
| --- | --- | --- | --- | --- | --- |
| <name> | <in words, not the query> | <duration> | <log, file, notification> | <YYYY-MM-DD, and what you did to trigger it> | |

<An alert that has never fired is an untested alert. Part 23's lab asks you to trigger one
deliberately; do it again here, on this service, and write the date.>

## 5. Authentication and transport security

| Item | Value | Check | What it printed | Status |
| --- | --- | --- | --- | --- |
| What is reachable from another machine | <the reverse proxy only> | <listening sockets on every machine> | | |
| TLS in front | <software, certificate kind: internal authority or public> | <a request from another device> | | |
| Certificate renewal | <automatic, and what happens if it fails> | <the expiry date the server reports> | | |
| Authentication on the gateway | <keys, one per application> | <an unauthenticated request> | <the refusal> | |
| Authentication on the front-end | <accounts; registration open or closed> | <an anonymous visit> | | |
| Engines reachable directly? | <they should not be> | <a request straight to an engine port from another machine> | <connection refused> | |
| Secrets | <where they live; never in a committed file> | <a search of the repository for the key> | | |
| Model provenance | <hash recorded beside each file; safetensors preferred> | <the verification command> | | |

<The row that matters most is the one about engines being reachable directly. A gateway
with keys in front of an engine that anybody on the house network can also address is not
an authenticated service; it is an authenticated service and an unauthenticated one.>

## 6. Backup and rollback

| Item | Value | Check | Status |
| --- | --- | --- | --- |
| What is backed up | <configuration, adapters, evaluation sets, the notebook, dashboards, front-end data> | <the archive's contents listing> | |
| What is not, deliberately | <model weights, replaced by repository, revision and hash> | <the manifest> | |
| Where it goes | <destination, and whether it survives the machine's failure> | | |
| Schedule | <how often, and what triggers it> | | |
| **Restore performed on** | <YYYY-MM-DD, into a scratch location> | <the comparison that showed it was intact> | |
| Version pinning | <container digests, not moving tags; package versions> | <the recorded digests> | |
| **Rollback performed on** | <YYYY-MM-DD> | <the upgrade, the return, and the request that worked afterwards> | |

<A backup that has not been restored is a hypothesis. Both dated rows above are the point
of this section, and both are commonly skipped.>

## 7. The service as a whole

| Question | Answer | Evidence |
| --- | --- | --- |
| What does a client have to know to use it? | <one address, one key, one model name> | |
| What happens when the machine reboots? | <which parts come back by themselves> | <a reboot, and what was running afterwards> |
| What happens when a model file is missing? | | |
| What is the slowest thing a user will notice? | | <a measurement label> |
| Who else uses it, and did it work for them? | <the household test> | <what they said> |

## 8. What is not done, and why

<Every "partial" and "not done" above, one line each, with the reason. This section is the
one a reader trusts the rest of the checklist on the strength of, and a checklist with
nothing in this section is a checklist that has not been read carefully.>