Security for Exposed Endpoints and the Model Supply Chain

Part 10 asked what an open endpoint is. This lesson asks a harder question: what does it take to be responsible for one that other people use, on a machine you also live on, for longer than a weekend.

By the end you will be able to give every caller its own credential with an expiry, put one authenticated and encrypted front door in front of everything, set limits that survive both a stranger and your own runaway script, apply the injection controls that belong at the service layer rather than in a prompt, verify the provenance of every model file you have, keep every secret out of every file you commit, and write down a logging policy somebody could actually read.

Part 10 established the ground this stands on: an endpoint is free compute, a data sink and eventually a path into your tools; an engine’s API key is a shared secret and nothing more; a container’s published port is not filtered by your host firewall; logs contain prompts; and indirect prompt injection arrived in your system the moment you fed a model text you did not write. None of that is repeated here. This lesson is what you do about it once the thing is a service.

Credentials with a life cycle

The difference between a key and a credential is that a credential has a beginning, an owner, a scope and an end.

LiteLLM’s virtual keys give you all four, and the requirements are modest: a Postgres database reachable through DATABASE_URL, and a master key that must start with sk-. The /key/generate endpoint takes a models list that scopes the key, metadata that records who it is for, and a duration after which it stops working. /key/info reports what it has spent. That is a credential.

Four habits follow from having them.

One key per thing, never one key per person. Your editor gets a key. Your agent experiments get a different one. The overnight evaluation job gets a third. When one of them misbehaves you can see which, and when one of them leaks you revoke exactly one thing.

Scope it to the models it needs. A key limited to local/coder cannot be used to run the coder model’s memory-hungry neighbour, and it cannot quietly become the credential somebody uses for everything because it was the one lying around.

Give it an expiry, especially for experiments. A key with a duration cleans itself up. The keys that cause trouble are the ones issued for an afternoon three years ago.

Write down where each one went. Not the key. The fact that key alias editor is in one machine’s editor settings and nowhere else. When you rotate, that note is the difference between a five-minute job and an afternoon of discovering configuration files.

What to do when a key ends up somewhere it should not be

1. 1Revoke it first, explain laterDelete the key. Something will break; that is the point. A key you are still thinking about is a key that is still working.
2. 2Issue a replacement with the same scopeSame models, same metadata, new secret. Put it in the one place your note says the old one lived.
3. 3Read the usage records for that keyLiteLLM keeps spend and token counts per key. Look at the window between the leak and the revocation and see whether anything used it.
4. 4Ask why it leakedA key in a repository means it was in a file that should have read it from the environment. A key in a chat window means somebody was debugging. Each has a different fix.
5. 5Fix the cause, not the instanceAdd the file to .gitignore before the next key goes in it, or move the value into an environment file, or stop pasting configuration into issues.
6. 6Write the incident in the changelogThree lines: what leaked, what you did, what you changed. This is the entry your future self reads when it happens again.

The order matters. Every step after the first is easier with the key already dead, and the temptation to investigate before revoking is how a small exposure becomes a long one.

The engine’s own key remains what Part 10 said it was. llama-server documents --api-key-file as reading keys from a file, one per line, which is the right way to pass one because a key on a command line is in the shell history and in ps output. But it is still a shared secret with no identity, no revocation without a restart and no per-caller record. It is the lock on an interior door, and the gateway is the front one.

One front door, encrypted

The three reasons for TLS from Part 10 hold: prompts cross the network in clear text without it, browsers restrict features on insecure origins, and a certificate is what makes a name mean something. What changes at service scale is the life cycle.

Whichever proxy you use, the minimal configuration is small. NGINX’s HTTPS guide gives listen 443 ssl with ssl_certificate and ssl_certificate_key, and notes that the default protocols since version 1.27.3 are TLSv1.2 and TLSv1.3, so “configuring them explicitly is generally not needed”. Caddy, which Part 7 used, generates its own certificate authority for non-public names and installs the root for you. Both are correct choices; the routing lesson explains why this part’s API front door is NGINX.

The operational questions are the ones nobody writes down.

When does it expire, and what happens then? A certificate from a public authority is usually renewed automatically and you find out it was not when the service stops. A certificate from your own authority expires too. Put the expiry date in the changelog and check it during your quarterly review, or automate the renewal and then test that the automation works by looking at the certificate rather than by assuming.

Where is the private key, and is it in your backup? The backup script in this part excludes *.key and *.pem on purpose. That is the right default and it means you must know where those files are and have decided separately how they are protected.

Does the name mean anything? A certificate for a name that only exists in your own DNS is fine, and it only helps if the clients actually verify it. A client configured to skip verification has TLS without authentication, which is encryption against a passive listener and nothing against an active one.

Limits, and the abuse that is usually your own

Abuse on a home model service rarely looks like an attacker. It looks like a script with a retry loop and no back-off, a notebook cell that was run twice, or an agent from Part 26 that has found a way to call itself. The defences are the same either way, and there are four of them.

A request rate ceiling at the edge. limit_req with a burst absorbs a person typing quickly and refuses a loop, by default with status 503. It costs the machine nothing because the request never reaches a model.

A per-key rate and token ceiling at the router. rpm_limit and tpm_limit do the same job with identity attached, so you can be generous with the editor and strict with the experiment. This is the limit that tells you who ran away.

A body size ceiling. client_max_body_size is the cheapest possible answer to a very large document arriving at a service that will try to tokenise all of it. Set it to something you would actually send.

A timeout you chose. A generous proxy_read_timeout is needed for a cold model load, and generous is not infinite. A request that has been open for ten minutes is a request that has gone wrong.

!Important

Set the limits before you need them, and set them on the day you build the service, because the moment you need them is the moment the machine is too busy to log in to. A limit that refuses one of your own requests occasionally is working. A limit that has never refused anything is untested.

Injection, at the service layer

Part 10 introduced prompt injection, direct and indirect, and OWASP’s definition: “A Prompt Injection Vulnerability occurs when user prompts alter the LLM’s behavior or output in unintended ways.” Its prevention list includes constraining behaviour through the system prompt, defining and validating output formats, segregating and marking external content, and enforcing privilege control so the model reaches only what it must.

Those are prompt-level and application-level controls. Four more belong at the service layer, which is to say they are configuration rather than prompting, and they apply to every caller regardless of what any application does.

An input size ceiling. Injection payloads tend to be long, because they are trying to outweigh a system prompt. A body size limit and a maximum context length are blunt and they raise the cost of the attempt.

A privilege boundary per key. A key scoped to one model is a key that cannot reach the model with the tools attached. When Part 24 adds tool calling, this scoping becomes the mechanism by which a document-summarising credential cannot call a shell.

A tool-call allowlist, when there are tools. Part 24 builds this properly. The service- layer principle is worth stating now so that you build towards it: the set of tools a given credential may invoke is a property of the credential, enforced by the gateway, not a suggestion made in a system prompt that the model may or may not follow.

Enough logging to reconstruct. Not the text. Which key, which model, how many tokens, when, and whether it succeeded. When something odd happens, that is what lets you say “this key made two hundred requests in ten minutes overnight” rather than “something odd happened”.

⛨Security

For a service that only answers questions, the consequence of a successful injection is a wrong answer, which is bad. For a service that can act, it is an action taken by somebody else through your model, which is a different category. The dividing line is tools, and it arrives in Part 24. Everything in this section is what you build before you cross it, not after.

The model supply chain, as a standing practice

Part 10 gave the two checks. Making them a practice rather than an intention needs three things you keep.

Prefer safetensors. The documentation describes it as “a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy)”, and the Transformers documentation is blunt about the reason: PyTorch weights were traditionally serialised with “the pickle utility which is known to be insecure”, while “safetensor files are more secure and faster to load”.

Treat trust_remote_code as installing software. The Transformers documentation says to “take extra precaution when loading a custom model”, noting that although the Hub scans repositories, “you should still be careful to avoid inadvertently executing malicious code”, and gives the mitigation directly: load “from a specific revision to avoid loading model code that may have changed”, using a full commit hash. GGUF sidesteps the Python question but carries a chat template your server executes when it formats a conversation, so it is not inert either.

Record provenance, and verify it. Part 4’s downloader wrote a .sha256 beside each model file. The Hub’s own verification command exists for the same purpose: hf cache verify is documented as validating “local files against their checksums on the Hub”, for a cache snapshot or a local directory, and it exits non-zero when something does not match. The estate-manifest.py script in the next lesson collects those hashes into one file, which is what turns “I think this is the model I downloaded” into something checkable.

The check	What it answers	When to do it
Format is safetensors or GGUF	Can this file execute code when it is opened	Before the first load, every time
No trust_remote_code, or a pinned commit	Whose code runs in my Python process	Before the first load, and again after any update
Hash matches what the Hub published	Is this the file the publisher released	On download, and whenever the estate manifest is rebuilt
Publisher and licence recorded	May I use this, and who do I ask about it	On download, into the model library note from Part 7
Chat template read once	What Jinja does my server run per request	Before putting a new model behind an alias

Secrets in configuration

One rule, applied without exception: a secret lives in the environment, and configuration files refer to it by name.

LiteLLM’s configuration documentation gives the mechanism as os.environ/<YOUR-ENV-VAR>, which resolves from the environment at runtime, so no key is ever written into litellm-config.yaml. Compose files take values from .env the same way. Grafana’s provisioning supports environment lookups with $ENV_VAR_NAME or ${ENV_VAR_NAME}, which is why the admin password in this part’s stack comes from the environment and not from the YAML.

The consequences to internalise:

• .env is the only file in your gateway or monitoring directory that holds a secret, and it is the only one that must never be committed, pasted into an issue, or copied into a chat window while debugging.
• Add it to .gitignore before the first key goes in it, not after. Removing a secret from a repository’s history is harder than not putting it there.
• A password on a command line is visible in ps to every user on the machine and in the shell history afterwards. Use a file or an environment variable.
• Backups need a decision of their own. This part’s backup script excludes .env, keys and certificates deliberately, which means you must store them somewhere else, on purpose.

✦Tip

This course runs its own release check over every file before publishing, and it refuses private keys, tokens, e-mail addresses, addresses, machine paths and anything that looks like a credential assigned a literal value. Those rules are the course’s own and they are not a standard, but the habit behind them is worth copying: run something mechanical over your files before you push them somewhere, because a human reading their own configuration sees what they meant rather than what is there.

Logging and privacy, written down

The five decisions Part 10 listed become a document once other people are involved: what is captured, where it goes, who can read it, how long it is kept, and whether the users were told. Three practical notes for the service case.

Default to metadata. turn_off_message_logging prevents “messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tracked”. That is the correct default for a machine in a home, and the per-request header x-litellm-enable-message-redaction: true exists for the caller who wants more redaction than the default.

Retention means deletion. Rotation that archives is not retention, it is a slower accumulation. Choose a period, implement the deletion, and check once that it actually happened.

Tell people, in words they will read. A household service used by people who did not configure it needs one paragraph somewhere they will see it: what is recorded, for how long, who can read it, and how to ask for it to be removed. If that paragraph would embarrass you, the problem is the logging policy rather than the paragraph.

Model the artefact path and the request path separately

The artefact path brings code, model files, tokenisers and dependencies into the service. The request path brings prompts, documents and tool results to a running application. Integrity checks and pinned revisions help the former; authentication, input limits and permission checks help the latter. Neither replaces the other.

Create a small threat table with an asset, an entry point, a likely misuse and an enforced control. Examples include untrusted model code executing at load time, an unauthenticated client exhausting memory, and retrieved text asking an agent to disclose a credential. Test the control with an observable denied action rather than assuming the policy text is enough.

Keep a credential rotation and incident recovery procedure. After a suspected leak, removing a string from a prompt is insufficient if it remains valid in logs or backups. Revoke or rotate the credential, inspect exposure and restore service with a known configuration. Security is an operating property of the complete system, including downloads, outbound integrations and administrators, not an intrinsic consequence of hosting weights locally.

Recap

Credentials have an owner, a scope and an end; virtual keys give you all three and the revocation procedure starts by revoking. TLS is small to configure and its life cycle is what needs writing down: expiry, where the private key lives, and whether clients actually verify. Four limits defend a home service against a stranger and against your own retry loop: a rate ceiling at the edge, a per-key rate and token ceiling at the router, a body size ceiling, and a timeout you chose. Injection controls at the service layer are input size, per-key model scope, a tool allowlist when Part 24 brings tools, and enough metadata logging to reconstruct what happened. The supply chain is five checks made into a habit and recorded in a manifest. Secrets live in the environment and configuration refers to them by name, with .env excluded from version control before the first key goes in it. And the logging policy is a paragraph the people using your service can read.

Check your understanding

Question 1. A key you issued for an experiment turns up in a screenshot in a public issue. What is the first thing to do?

• Check the logs to see whether it was used
• Revoke the key, then issue a replacement, then read the usage records for the window between the exposure and the revocation
• Rotate the master key
• Change the gateway port

Show the answer and why

Answer: Revoke the key, then issue a replacement, then read the usage records for the window between the exposure and the revocation

Investigation is easier with the key already dead, and every minute spent deciding is a minute the credential still works. Reading the usage records is the second step, not the first, and this is precisely why keys are issued per application rather than shared.

Question 2. Which limits protect against a runaway script of your own, and which against a stranger?

• Only the edge rate limit matters in either case
• Both: the edge rate limit refuses the traffic cheaply without identity, and the per-key rate and token limits refuse it with identity, which is what tells you which of your own applications ran away
• Per-key limits are only useful for billing
• A body size limit is the only control that helps

Show the answer and why

Answer: Both: the edge rate limit refuses the traffic cheaply without identity, and the per-key rate and token limits refuse it with identity, which is what tells you which of your own applications ran away

They are complementary. The edge limit costs the machine nothing and works on anybody; the per-key limit is the one that turns an incident into a name. On a home service the runaway is usually your own script, which is exactly the case where identity is what you need.

Question 3. Which of these belong in a file that is committed to version control? Select all that apply.

• The model aliases and which model each points at
• The rate limits and their zone sizes
• The LiteLLM master key
• The name of the environment variable the master key is read from

Show the answer and why

Answer: The model aliases and which model each points at, The rate limits and their zone sizes, The name of the environment variable the master key is read from

Configuration is public interface and belongs under version control; secrets are read from the environment by name. LiteLLM documents os.environ references for exactly this, so the configuration can be committed while the value never is.

Question 4. You are about to load a community model that ships its own modelling code. What does the documentation suggest, and why?

• Nothing special; the Hub scans for malware
• Treat trust_remote_code as running code somebody else wrote, and load from a specific full commit hash so that the code reviewed today is the code that runs tomorrow
• Convert it to GGUF first, which removes all risk
• Only load it inside a virtual environment

Show the answer and why

Answer: Treat trust_remote_code as running code somebody else wrote, and load from a specific full commit hash so that the code reviewed today is the code that runs tomorrow

The Transformers documentation notes the Hub scans repositories and still tells you to take extra precaution, and gives revision pinning as the mitigation. Converting to GGUF does remove the Python question but not the chat template, which is a program your server runs on every request.

Question 5. What is the difference between rotating logs and having a retention policy?

• They are the same thing
• Rotation moves data into older files; retention means the old data is actually deleted after a period you chose, and you have checked once that it was
• Retention applies only to prompt text
• Rotation is required for compliance and retention is optional

Show the answer and why

Answer: Rotation moves data into older files; retention means the old data is actually deleted after a period you chose, and you have checked once that it was

Rotation with archiving is accumulation at a slower rate. The policy that matters names a period and deletes at the end of it, and the only way to know it works is to look once rather than to assume.

Check my answersClear