# Service runbook — Ridgeway Lane home lab

Course-authored sample document. Invented throughout, so that the question set has checkable
answers. Replace with your own runbook once the pipeline works.

Last reviewed: 2026-08-30.

## Services and who depends on them

| Service | Host | Port | Who notices when it is down |
| --- | --- | --- | --- |
| Chat front-end | tern | 443 | Everybody in the house |
| Model gateway | tern | 4000 | The chat front-end and the editor plugins |
| Inference engine | tern | 8080 | The gateway only |
| Document index | tern | 8090 | The question-answering service |

The inference engine is never published on the house network. Only the front-end is, and
only through the reverse proxy on port 443.

## Starting and stopping

Start everything: `runlab up` from `/srv/lab` on tern. Stop everything: `runlab down`.
Restarting a single service is `runlab restart <name>` and takes about forty seconds, most
of which is the model loading back into memory.

Never stop the gateway while the front-end is serving a request. The front-end does not
retry, and the user sees a blank reply rather than an error.

## Backups

The backup window is 02:00 to 04:00 on Sundays. During that window the front-end is stopped
for roughly ninety seconds while its database is copied, because the database is SQLite and
a copy taken mid-write has not been shown to restore.

What is backed up:

- the front-end database, which holds accounts, conversations and uploaded files
- the document index
- `/srv/lab/config`, which holds every configuration file and no secrets

What is deliberately not backed up:

- model weights, which are re-downloadable
- the reverse proxy's certificate authority, which is regenerated and re-trusted if lost
- logs

Backups are written to the spinning disk on tern and copied to an external drive on the
first Sunday of each month. A restore is tested into a scratch volume once a quarter. The
last successful restore test was 2026-07-05.

## Logging and retention

The gateway logs one line per request: timestamp, caller key name, model, token counts and
latency. It does not log prompt or completion text. Logs are rotated weekly and deleted
after 28 days.

Everybody with an account has been told, in writing, that request metadata is logged and
that message content is not.

## On-call

There is no on-call. If something breaks overnight it is fixed the next morning. The house
was told this before the service was made available, and it is the reason nothing critical
depends on it.

## Escalation

If tern is unrecoverable: bring skua up, run the front-end and gateway on it, and point the
gateway at petrel for inference. Expect the service to be slower and to be unavailable
whenever petrel is switched off, which is most nights.