Routing and Model Management: llama-swap, LiteLLM, NGINX and Health Checks

By the end of this lesson you will be able to say, for any request arriving at your machine, which component decided it was allowed, which decided which model would answer, which decided whether that model was running, and which of those four layers to look at first when the answer is wrong. You will also be able to change the model behind a name without anything that uses the name noticing.

Part 9 built two of those layers and left the front door for later. This is later.

Three layers, three jobs, one address

The temptation, when a second person starts using your machine, is to add the missing piece to whatever you already have. The engine gets an API key. Then the engine gets a second port for the second model. Then something needs a certificate, so the certificate goes on the engine too. Six months of that produces a system where nothing can be changed because everything knows about everything.

The alternative is three small programs, each of which knows only about the one below it.

A request arriving from the network, on its way to a model

each layer answers one question and hands the request down

1. NGINXIs this connection encrypted, is this client asking too often, and is this body a sensible size? Terminates TLS, applies the rate limit, publishes one port.transport and abuse
2. LiteLLMWho is asking, may they have this model, have they used their allowance, and where should it go? Keys, budgets, aliases, retries, fallbacks, usage records.identity and policy
3. llama-swapIs the right engine running for the model that was asked for? Starts it if not, stops something else if it must, unloads it when idle.memory as a schedule
4. Enginesllama-server, vLLM, SGLang or an MLX server, each holding one model and a fixed number of slots
5. Accelerator and memoryThe one physical resource all of the above are competing for

A fault at any layer looks like a different symptom, which is the practical reason for the separation: a 503 from the rate limiter, a 401 from the router and a timeout from a cold model load are three distinguishable failures rather than one ambiguous one.

The value of the arrangement is not architectural elegance. It is that each layer fails visibly and separately, and that you can replace any one of them without touching the others. Swap llama-server for vLLM behind an alias and no client changes. Put a second machine behind the router and no engine changes. Move the certificate from a private authority to a public one and nothing below NGINX notices.

llama-swap: memory as a schedule

llama-swap v255 · verified 2026-09-08 is a single Go binary whose entire job is to make sure the engine that can answer a request is the one that is running. Part 9 configured it. What Part 9 did not do is treat its timing keys as operational settings, which is what they are.

Four keys decide how a machine behaves under a day of mixed use. The example configuration documents healthCheckTimeout as “number of seconds to wait for a model to be ready to serve requests” and checkEndpoint as the “URL path to check if the server is ready”. Those two together are how llama-swap knows a model has finished loading rather than merely having been started, which matters because a large model on a slow disk can take longer to become ready than any default patience allows.

The other two are about giving memory back. ttl is documented as “automatically unload the model after ttl seconds”, and unloadTimeout as the “graceful timeout in seconds when unloading a model (manual, API, or ttl expiry)”. A short ttl on a desktop you also use for other things is correct. The same short ttl on a machine serving a household is a recipe for everybody waiting through a model load several times a day, and it is the most common cause of the complaint that the assistant is “sometimes slow for no reason”.

Groups decide what may coexist. The documentation describes groups as letting you “keep some models loaded while others swap out”, with swap controlling “how members of this group swap among themselves”, exclusive controlling “how this group affects other groups”, and persistent meaning that “other groups cannot unload this group’s members”. Part 9’s configuration puts Qwen3-8B and Qwen3-Coder-30B-A3B in one swapping, exclusive group and Qwen3-Embedding-0.6B in a persistent one, which is the right shape for almost every single-machine estate: the big models take turns, the small always-needed one stays.

✦Tip

Read /running and /api/models/unload from the README as operational tools rather than curiosities. The first answers “what is actually loaded right now”, which is a question you will ask during every incident and which no amount of remembering answers reliably. The second gives you the memory back without stopping the service, which is what you want at the moment somebody else needs the machine.

Swapping a model without anybody waiting

Model swapping is not free and the documentation does not pretend otherwise: when a request arrives for a model that is not loaded, llama-swap starts the right engine, and the caller waits for that. The README’s description is plain: if the wrong upstream server is running, it “will be replaced with the correct one”. Replacement takes as long as loading takes.

There is a way to make the upgrade case free, and it is worth knowing because it is the case that happens on purpose. The trick is to treat the model id as versioned and the alias as the thing that moves.

Rolling a new model in behind an unchanged name

1. 1Add the new model under a new idA second entry in llama-swap.yaml, for example a dated id, pointing at the new file. Nothing refers to it yet, so nothing changes.
2. 2Put it in a group that can coexist, if memory allowsOn a machine with room, the new model can load while the old one is still serving. On a machine without room, accept a gap and do this when nobody is asking.
3. 3Warm it with one requestSend a request naming the new id directly. It loads, it answers, and now it is resident. This is the step that moves the waiting off your users and onto you.
4. 4Compare it against the old oneYour Part 16 evaluation set, on both ids, before anything else points at it. A model that is worse on your own tasks is a model you should not have rolled.
5. 5Move the aliasOne line in litellm-config.yaml: the alias local/chat now points at the new id. Clients keep sending the same name.
6. 6Keep the old id for a weekBoth entries stay. If something regresses, you move one line back rather than re-downloading a model at speed under pressure.
7. 7Remove it deliberatelyWhen the week has passed and nothing complained, delete the old entry and record the change in the changelog.

The alias is the interface and the model id is the implementation. Every part of this procedure exists so that the moment of change is one edit to one line, and so that undoing it is the same edit backwards.

LiteLLM: who is asking, and how much may they have

LiteLLM 1.100.0 · verified 2026-09-08 is the layer that knows about people. Part 9 used it for aliases, retries and fallbacks. Three of its facilities become load-bearing the moment more than one thing is calling.

Keys per caller. The virtual keys documentation states the requirements plainly: a Postgres database reachable through DATABASE_URL, and a master key that must start with sk-. /key/generate takes a models list, metadata, and a duration, so a key can be limited to the models one application needs and expire on its own. The master key is the proxy admin key that creates other keys; applications should carry the keys it issues rather than the key itself.

Budgets and rate limits. The same interface takes max_budget with a budget_duration written as a period such as 30d, plus tpm_limit and rpm_limit for tokens and requests per minute. These can be set on a key, on a team or on an internal user, and the documentation notes that budgets require a database. On a home service the money is not the point, because the marginal cost of your own machine is electricity. The point is the shape of the limit: a runaway script that would otherwise occupy your accelerator all night hits rpm_limit and stops, and the record of it hitting the limit is in the usage table.

Routing across deployments. The configuration documentation shows several entries in model_list sharing one model_name with different api_base values, which is how one alias is served by more than one machine. routing_strategy selects between simple-shuffle, least-busy, usage-based-routing and latency-based-routing. For a pair of machines from Part 20 or Part 21, least-busy is the honest default: it sends work to whichever is not currently occupied, which is exactly the property a two-machine home cluster has that a single machine does not.

✕Common mistake

Giving every application the master key because virtual keys needed a database. It is five minutes of work to add Postgres and it is the difference between revoking one key and rotating a secret that is now in four configuration files, one shell history and a screenshot. If you genuinely cannot run a database, at least give the master key to nothing that runs unattended, and write down every place you pasted it.

Health, which is three questions

The health documentation is unusually clear that these endpoints answer different things, and the difference matters because a monitor asking the wrong one gets a confidently wrong answer.

GET /health/liveliness means “the process is up. No dependencies are checked”. It needs no authentication. Use it for a process supervisor: if this fails, restart the container.

GET /health/readiness means “the worker is ready to accept traffic”, and reports whether the database is "connected", "disconnected" or "Not connected". Use it to decide whether to send a request at all.

GET /health is different in kind. The documentation says it “runs a real test request against every configured model, so it costs a few tokens per model”. On a swapping gateway that means loading every model in turn. This is a useful thing for a human to ask once and a destructive thing for a monitor to ask every fifteen seconds, which is why background_health_checks: true with health_check_interval: 300 exists: the checks run on a timer and the endpoint serves the cached answer.

Underneath, llama-server’s own /health is simpler and just as useful: the README documents it as returning 503 while “the model is still being loaded” and 200 with {"status": "ok"} when “the model is successfully loaded and the server is ready”. That is precisely the signal llama-swap’s checkEndpoint is reading.

Fallbacks, and when they are wrong

The reliability documentation gives three kinds. fallbacks sends an alias to another one when it fails outright. context_window_fallbacks sends it elsewhere specifically when the context window was exceeded, which on a local estate is a genuinely good idea: a request too long for the 8B model can go to the one with the longer context rather than failing. content_policy_fallbacks covers refusals. Around them, num_retries retries within one deployment, allowed_fails takes a deployment out of rotation after that many failures and cooldown_time says for how long.

Part 9 made the design point and it is worth repeating in operational terms: a fallback converts a visible failure into a quiet degradation. For a person waiting, that is a good trade. For an overnight batch writing numbers into a results table, it is a corrupted result set that looks fine. Decide per alias, and write the decision down.

The front door

Part 7 put Caddy in front of a chat front-end, and Caddy was the right choice there: its documentation describes generating “its own certificate authority (CA)” to “serve non-public sites over HTTPS”, installing the root into the system trust store on first use, and offering caddy trust to retry that “as a privileged user”. For a name that exists only inside your house, that is the shortest path to working TLS and Part 7’s procedure still stands.

This part uses NGINX for the API front door, for one reason: rate limiting. The limit_req module gives a documented, well-understood control that an API endpoint needs and a chat front-end mostly does not. Its two directives are limit_req_zone key zone=name:size rate=rate; in the http context, which the documentation notes will hold “about 16,000” states per megabyte for the $binary_remote_addr key, and limit_req zone=name [burst=number] [nodelay]; where requests within the burst are queued and those beyond it are refused, by default with status 503.

If you would rather keep one reverse proxy across the whole house, keep Caddy and put the rate limiting somewhere else, such as rpm_limit on every issued key. If you would rather have the limit at the edge, where it costs the machine nothing to enforce, this is the file:

RunnableAll tracks

gateway-front.conf

# Purpose: put one hardened front door in front of the Part 9 gateway - TLS for the whole
#          conversation, a per-client request rate limit, a body-size ceiling, and
#          streaming that is not ruined by response buffering
# Platform: all (this is an NGINX server block; on Linux it goes in
#           /etc/nginx/conf.d/, on macOS under the Homebrew prefix's nginx/servers/)
# Minimum memory: 8 GB, which is what the models behind it need; NGINX itself needs almost
#           nothing
# Assumes: the LiteLLM gateway from Part 9 listening on 127.0.0.1:4000, a certificate and
#          key you already have, and NGINX with the http_ssl_module. NGINX does not read
#          environment variables from its configuration, so the three values that differ
#          per machine - the server name and the two certificate paths - are written here
#          as examples and you edit them once. Nothing secret belongs in this file: the
#          key is referenced by path, never inlined, and the API keys live in LiteLLM.

# ---------------------------------------------------------------------------------------
# The rate-limiting zone. NGINX's documentation gives the syntax as
#   limit_req_zone key zone=name:size rate=rate;
# and notes that one megabyte holds about 16,000 of the 64-byte states that
# $binary_remote_addr produces. One request per second sounds absurdly low until you
# remember that a chat request occupies a model for seconds, and that the burst below is
# what absorbs a person typing quickly. This directive belongs in the http context: if
# your distribution's nginx.conf includes conf.d/*.conf inside http, it belongs here; if
# it includes it at the top level, move this one line into nginx.conf itself.
limit_req_zone $binary_remote_addr zone=llmapi:10m rate=2r/s;

# Redirect the plain-HTTP port rather than serving on it. A client that talks to port 80
# has already sent its key in clear text, so the redirect is a convenience for humans
# typing an address, not a security control.
server {
    listen 80;
    server_name gateway.home.arpa;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name gateway.home.arpa;

    # NGINX's HTTPS guide gives this pair as the minimum. The certificate can come from
    # your own certificate authority for a name that only exists inside the house, or from
    # a public one for a name that resolves publicly; the directives are the same either
    # way. Since nginx 1.27.3 the default protocols are TLSv1.2 and TLSv1.3, which is what
    # you want, so they are not repeated here.
    ssl_certificate     /etc/ssl/local/gateway.crt;
    ssl_certificate_key /etc/ssl/local/gateway.key;

    # The documented optimisation for a machine serving more than one client: one megabyte
    # of this cache holds about 4000 sessions, so handshakes are not repeated all day.
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;

    # A prompt is small. A prompt with a hundred-megabyte "document" attached is somebody
    # finding out what your machine does when it runs out of memory. Raise this
    # deliberately if you serve long documents, and know what you raised it to.
    client_max_body_size 2m;

    # Nothing about the gateway's internals belongs in a response header.
    server_tokens off;

    location / {
        # burst absorbs a short flurry from one client; requests beyond it are refused
        # rather than queued, because a queued LLM request is a request that will time out
        # somewhere less visible. The refusal status is 503 by default, which the
        # documentation states and which is what your dashboard will show.
        limit_req zone=llmapi burst=10 nodelay;

        proxy_pass http://127.0.0.1:4000;

        # Version 1.1 is the documented default and is what keepalive connections need.
        proxy_http_version 1.1;

        # Without these the gateway's log records NGINX as every caller, and the rate
        # limit you configured is the only place a client's address still exists.
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Token streaming is a long response sent slowly. Buffering it means the reader
        # sees nothing and then sees everything, which reads as a hang.
        proxy_buffering off;

        # The documented default read timeout is 60 seconds, which is shorter than a cold
        # model load. This is the single most common cause of "the gateway works from the
        # machine itself and times out from anywhere else".
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }

    # The two cheap health endpoints, exempt from the rate limit so that a monitor polling
    # every fifteen seconds does not consume a real client's allowance. Neither of them
    # touches a model and neither of them needs a key, which is exactly why they are the
    # ones a monitor should poll.
    location /health/liveliness {
        proxy_pass http://127.0.0.1:4000;
        access_log off;
    }

    location /health/readiness {
        proxy_pass http://127.0.0.1:4000;
        access_log off;
    }

    # Prometheus, Grafana, llama-swap's web page and LiteLLM's own /metrics are operator
    # surfaces. They are not published here, and the stack keeps them on the loopback
    # address. If you need them from another machine, reach the machine over a VPN rather
    # than adding a location block to this file.
}

Download gateway-front.conf108 lines

Four things in it are worth reading twice, because each corresponds to a failure people hit and then misdiagnose.

proxy_buffering off is there because token streaming is a long response delivered slowly, and the documented default buffers it. A reader who sees nothing for twenty seconds and then sees everything reports the service as hanging.

proxy_read_timeout 600s overrides a documented default of 60 seconds. A cold model load plus a long generation exceeds a minute easily, and the symptom is the memorable one: it works from the machine itself and times out from everywhere else.

client_max_body_size is a security control as much as an operational one. It is the cheapest possible answer to somebody pasting a very large document into a service that will try to tokenise all of it.

And the health endpoints get their own location blocks with the rate limit not applied, because a monitor polling every fifteen seconds should not consume a real client’s allowance.

⛨Security

Publish exactly one port. Prometheus, Grafana, llama-swap’s web interface and LiteLLM’s own metrics endpoint are operator surfaces: they show what everybody asked and what the machine is doing, and none of them has authentication worth relying on. Keep them on the loopback address and reach the machine over a VPN when you need them from elsewhere, exactly as Part 10 said about the engine itself. The same reasoning applies to llama-server’s --props and --slots, which the README describes as changing global properties and exposing slot monitoring: useful on your own machine, wrong on a shared one.

Define what a fallback is allowed to change

A fallback can preserve availability while changing model quality, context support, output schema reliability or privacy location. Define permitted substitutions per alias. A local-only application must not route to a hosted provider simply because the local worker is unavailable.

Test the route with the primary healthy, loading, unavailable and over capacity. Record whether the gateway queues, rejects, retries or selects another model. A retry needs a bound, and a request that can trigger external effects needs an idempotency policy above the model service.

Separate liveness, readiness and task correctness. A process can be alive before its model loads; a ready engine can still fail your required tool-call contract. Keep a lightweight health check and a deeper application probe. Record alias-to-artefact mappings so a fallback result can be attributed to the model that actually answered. A stable client name is useful only when the operator can still reconstruct its routing decisions.

Recap

Four layers, each answering one question: NGINX for transport and abuse, LiteLLM for identity and policy, llama-swap for what is loaded, engines for the model itself. llama-swap turns memory into a schedule through ttl, unloadTimeout and groups, and its checkEndpoint and healthCheckTimeout are what let it distinguish a started engine from a ready one. Rolling a new model in without downtime means versioning the model id, warming the new one, comparing it on your own evaluation set, and then moving the alias by one line. LiteLLM’s virtual keys, budgets and per-minute limits need a database and are worth the database; its routing strategies are how one alias covers two machines. Health means three different things and asking the expensive one on a timer is how a monitor brings down a swapping gateway. Fallbacks trade a visible failure for a quiet degradation, which is right for a person and wrong for a pipeline. And the front door terminates TLS, refuses oversized bodies, limits request rate per client, and does not buffer the stream.

Check your understanding

Question 1. Your monitoring system polls one endpoint every fifteen seconds to decide whether the gateway is healthy. Which one, and why?

• GET /health, because it is the most thorough
• GET /health/liveliness for the supervisor and GET /health/readiness for whether to send traffic; /health runs a real request against every configured model and would load each of them in turn
• GET /v1/models, because it is cheap
• Any of them; they return the same information

Show the answer and why

Answer: GET /health/liveliness for the supervisor and GET /health/readiness for whether to send traffic; /health runs a real request against every configured model and would load each of them in turn

The documentation is explicit that liveliness checks nothing but the process, readiness reports whether the worker and its database are ready, and /health costs a few tokens per model. On a gateway that loads models on demand, polling /health frequently is a way of keeping the accelerator permanently busy answering the monitor.

Question 2. You want to replace the model behind local/chat with a newer one, without anybody waiting and without breaking a rollback. What is the order of operations?

• Edit the model path in llama-swap.yaml and restart
• Add the new model under a new id, warm it with a direct request, compare it against your evaluation set, then move the alias in litellm-config.yaml and keep the old id for a week
• Point the alias at both models and let the router choose
• Delete the old model file so nothing can use it by accident

Show the answer and why

Answer: Add the new model under a new id, warm it with a direct request, compare it against your evaluation set, then move the alias in litellm-config.yaml and keep the old id for a week

The alias is the interface and the model id is the implementation. Editing the path in place means the change and the rollback are both a model load under pressure; adding a second id means the change is one line of YAML and the rollback is the same line backwards.

Question 3. Which of these belong at the LiteLLM layer rather than on the engine? Select all that apply.

• A key per application, revocable without restarting anything
• A requests-per-minute limit attached to one caller
• The number of parallel slots the model server allocates
• A record of how many tokens each application used, by model

Show the answer and why

Answer: A key per application, revocable without restarting anything, A requests-per-minute limit attached to one caller, A record of how many tokens each application used, by model

Identity, allowances and accounting belong where every request passes regardless of which engine served it. Slot count is a property of one engine process and a memory decision, so it belongs in the command line that starts it.

Question 4. A client reports that streaming responses arrive all at once, after a long silence, when they come through the reverse proxy but not when they talk to the gateway directly. What is the likely cause?

• The model is too slow
• Response buffering is on at the proxy, which is the documented default; token streaming needs it turned off
• The API key is wrong
• The rate limit is delaying the request

Show the answer and why

Answer: Response buffering is on at the proxy, which is the documented default; token streaming needs it turned off

NGINX documents proxy_buffering as on by default, which means it collects the response before passing it on. For a long response delivered a token at a time, that turns a live stream into a pause followed by a wall of text.

Question 5. You configure limit_req with a burst and nodelay. A client sends a rapid flurry of requests. What happens to the ones beyond the burst?

• They are queued until the rate allows them
• They are refused, by default with status 503, rather than delayed
• They are sent to the fallback model
• They are silently dropped with no response

Show the answer and why

Answer: They are refused, by default with status 503, rather than delayed

Without nodelay, excessive requests within the burst are delayed to fit the rate. With nodelay they are served immediately up to the burst and anything beyond it is terminated with an error, whose default status the module documents as 503. Refusing quickly is usually right for an API, because a queued model request tends to time out somewhere less visible.

Check my answersClear