Skip to content
Local LLM Course

Operating a Local AI Service

Part 9 gave you one address, three stable model names and a key. Everything since has been built on top of it: the retrieval service in Part 10, the fine-tuned adapters in Part 13, the evaluation harness in Part 16, the clusters in Parts 19 to 22. Somewhere in the middle of that, the thing on your desk stopped being an experiment and became infrastructure.

Nobody announces that transition. It shows up as a message from somebody in the house asking why the assistant is slow, or as a script that failed overnight, or as a moment six months later when you cannot remember which quantisation is behind local/coder and the file that would have told you was never written. This part is about the interval between the demo working and the service being trustworthy, which is where most home setups quietly stop.

The discipline is not specific to models. RunBook Academy’s other courses teach it for databases and web services: measure the thing, know what it costs, harden the edge, be able to go back to yesterday. What is specific to models is what to measure, because a model server fails in ways a web server does not. Its memory use depends on how long the conversations are. Its throughput depends on how many people are talking at once and on which model is loaded, which may not be the one you think. Its cost is dominated by electricity and by a machine you already bought, not by an invoice. And its worst failure mode is not an error page but an out-of-memory condition at three in the morning, after a week of working perfectly, caused by nothing changing at all.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • put one authenticated, rate-limited, TLS-terminated front door in front of every model on your machine, and explain what each of the three layers behind it is for;
  • name the metrics endpoint of every engine this course teaches, say which numbers each publishes, and know which questions none of them can answer;
  • read GPU power and memory on your own track, whichever of the four it is, and get those numbers into the same dashboard as the engine’s;
  • deploy Prometheus and Grafana from configuration files rather than from a set-up wizard, so that the dashboard is a file you can commit and rebuild;
  • write two alerts that fire when something is actually wrong, and explain why most alerts people write do not;
  • work out what a million tokens costs on your own hardware, from your own power measurement, your own tariff and your own purchase price, and compare that honestly with a hosted price you look up yourself;
  • state what changes about security when an endpoint leaves the loopback address, and apply the controls that follow: authentication, transport security, rate limits, input size ceilings, and a logging policy somebody could actually read;
  • check a model file for the two supply-chain problems that occur in practice, and record the provenance of everything you have downloaded;
  • upgrade an engine with a rollback path you tested before you needed it, and reproduce your whole serving setup on a new machine from a manifest rather than from memory;
  • diagnose a memory failure from the record it left behind, name which of the known faults it was, fix it, and add the guard rail that stops it recurring.

What you need first

Section titled “What you need first”

Part 9, in full and running. This part operates the gateway that project built: the same llama-swap.yaml, the same litellm-config.yaml, the same three aliases, and the check-gateway.sh that proves it works. If your gateway is not currently up, bring it up before starting, because every page here reads from it.

Part 5’s lab, for the habit of measuring the machine rather than believing a specification, and for its method: repeat the run, record the conditions, treat variance as information. The power measurement in this part’s third lesson is that method applied to watts.

Part 7’s lab, which put a reverse proxy and TLS in front of a chat front-end on your local network. This part extends that to an endpoint that programs talk to, which changes the threat model and adds rate limiting to the list.

Part 10’s security lesson, which introduced what an open endpoint is, what an API key does and does not achieve, why logs are prompts, and where prompt injection came from. This part does not repeat it. It picks it up at the point where the endpoint is no longer something you are experimenting with, and asks the harder questions: who is allowed to use it, how much, what happens when they misbehave, and what you can prove afterwards.

A machine with at least 8 GB of memory. The monitoring stack is small, the lab and the challenge both run on one machine, and the cluster parts are useful context rather than a prerequisite. Readers who did build a cluster in Parts 19 to 22 get more out of the lab, because the dashboard is the first thing that makes two machines feel like one system.

How to work through it

Section titled “How to work through it”

The first two lessons are the spine. Read the routing lesson with your gateway directory open, because it is largely about configuration you already have and options you did not use. Read the observability lesson slowly: it is the longest, it names a lot of metrics, and the point is not to memorise them but to know which question each one answers, so that you can find the right one when something is wrong and you are tired.

The third lesson, on capacity and cost, is arithmetic and a script. It is the page to read before somebody asks you whether running this at home is worth it, and it is deliberately free of prices: the formula takes your tariff, your machine, your measured throughput and a hosted price you look up on the day, because a price printed in a course is out of date before the ink dries.

The fourth and fifth lessons are the ones people skip and then wish they had not. Security for an exposed endpoint is short and consequential. Backup, upgrades and reproducibility is the page that decides whether a bad upgrade costs you ten minutes or a weekend.

Then the lab, which builds the dashboard and takes an hour, and the challenge, which uses it. The challenge is written as a diagnosis rather than a repair: you introduce a memory fault deliberately, read the graphs it leaves behind, name it, fix it, and add the guard rail. That is the shape of every operational problem you will meet afterwards, and it is worth practising once on a machine where the stakes are only your own afternoon.

Part23 of 27Level4 — Cluster ArchitectPages7Estimated time4.1 hours

0 / 7 lessons in this part completed