Observability: Metrics, Logs and Traces for LLM Serving

By the end of this lesson you will be able to name the metrics endpoint of every engine this course teaches, say which of your operational questions each one can answer and which it cannot, read power and memory from the accelerator on your own track, and put all of it into one Prometheus instance and one dashboard. You will also have a defensible answer to the question of which alerts are worth having, which is a shorter list than most people expect.

Start from the questions, not the metrics

A dashboard assembled from whatever the exporters happened to publish is a wall of graphs nobody reads. A dashboard assembled from questions is a page you glance at and learn something from. There are six questions a local model service has to be able to answer, and everything else is detail.

The six questions, and what each one is for

1. 1Is it up?Whether the gateway answers at all. The cheapest signal there is, and the one that catches the largest share of real incidents.
2. 2Is it keeping up?Requests running versus requests waiting. Running is use; waiting is the earliest honest sign that capacity has been exceeded.
3. 3How long does a caller wait?Time to first token, and the end-to-end latency the router sees. These diverge when a model is being loaded, which is exactly when you want to know.
4. 4How full is the memory?Accelerator memory in use, and key-value cache occupancy. This is the graph the three-in-the-morning failure is written on.
5. 5What is it costing?Power draw, which the next lesson turns into a cost per million tokens.
6. 6Who is using it, and for what?Tokens by model and by key. The question that decides whether a limit is needed and where.

Every metric named in this lesson exists to answer one of these. A number that answers none of them can go on a second page, or nowhere.

What each engine publishes

The engines disagree about names, and they disagree about which questions they can answer at all. That is not carelessness on anybody’s part: they have different internals, and a metric that means something in one is meaningless in another.

llama.cpp. The server README documents --metrics as enabling a “prometheus compatible metrics endpoint (default: disabled)”, and it is disabled by default, which is the single most common reason a scrape target is up and empty. What it publishes covers throughput and queueing well: llamacpp:prompt_tokens_total and llamacpp:tokens_predicted_total as counters, llamacpp:prompt_tokens_seconds and llamacpp:predicted_tokens_seconds as average throughput gauges, llamacpp:requests_processing and llamacpp:requests_deferred as the running and waiting counts, and llamacpp:n_tokens_max as the “high watermark of the context size observed”. Speculative decoding from Part 17 gets its own counters, including draft tokens generated and accepted, which is the cleanest way to measure whether it is earning its place. What llama.cpp does not publish is a key-value cache occupancy figure; the context high-water mark is the nearest thing, and the lesson to draw is that on this engine you infer memory pressure from the accelerator rather than from the engine.

vLLM. The metrics page for vLLM 0.28.0 · verified 2026-09-08 lists exactly the numbers a serving engine should expose: vllm:num_requests_running and vllm:num_requests_waiting as gauges, vllm:kv_cache_usage_perc as the cache utilisation where one means completely full, vllm:prompt_tokens and vllm:generation_tokens as counters, and histograms for vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, vllm:inter_token_latency_seconds and vllm:e2e_request_latency_seconds. The cache utilisation gauge is the one worth putting at the top of the dashboard, because it is the only place in this whole stack where an engine tells you directly how close it is to the wall.

SGLang. Metrics are published when the server is started with --enable-metrics, and the names follow the same convention: sglang:num_running_reqs, sglang:num_queue_reqs, sglang:token_usage, sglang:cache_hit_rate, sglang:gen_throughput, sglang:time_to_first_token_seconds, sglang:time_per_output_token_seconds and sglang:e2e_request_latency_seconds. The cache hit rate is distinctive and is worth watching if you followed Part 17: it is the prefix cache doing its job, and it moves when your prompt template changes.

llama-swap. The README lists /metrics as “system and GPU metrics for prometheus”. Its value is not the numbers so much as the layer they come from: this is the component that knows which model is loaded, and a memory graph is far more legible beside a record of what was resident when.

LiteLLM. Metrics are off until litellm_settings contains callbacks: ["prometheus"], after which they appear on the proxy port. The set is about people rather than tensors: litellm_total_tokens_metric, litellm_input_tokens_metric and litellm_output_tokens_metric, labelled by model, hashed key, key alias, team and user; litellm_request_total_latency_metric and litellm_llm_api_latency_metric as histograms; litellm_deployment_state where zero is healthy, one is a partial outage and two is a complete one; litellm_deployment_success_responses and its failure counterpart; and litellm_in_flight_requests. This is where the sixth question gets answered, and there is nowhere else it can be.

✕Common mistake

A scrape target that is up and publishes nothing. This is the characteristic failure of this whole area and it has three causes, in descending order of frequency: llama-server started without --metrics, SGLang started without --enable-metrics, and LiteLLM without the Prometheus callback in its configuration. In all three cases the port answers, so the target is green, and the panels are empty. Check by asking the endpoint itself what it has: curl -s http://127.0.0.1:8080/metrics | head tells you in one line.

Reading the accelerator, on each of the four tracks

Engine metrics tell you what the engine thinks. The accelerator is the thing that actually runs out, and every track reads it differently.

Track S — NVIDIA DGX Spark

NVIDIA’s DCGM exporter is the supported path and it is a container. The installation guide gives the command as docker run -d --rm --name dcgm-exporter --gpus all --cap-add SYS_ADMIN -p 9400:9400 against the nvcr.io/nvidia/k8s/dcgm-exporter image, listening on port 9400 with metrics on the default path. Which fields it collects comes from a collector CSV, described as having “exactly three columns: DCGM field, Prometheus metric type, help message”, with the default at /etc/dcgm-exporter/default-counters.csv and a custom one passed with -f.

The four fields worth having on the dashboard are DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED for framebuffer memory in use, DCGM_FI_DEV_POWER_USAGE and DCGM_FI_DEV_GPU_TEMP. Confirm what your tag actually exports before you write an alert against a field name; the counters file is the authority for your installation.

For a quick look without an exporter, nvidia-smi --query-gpu takes a comma-separated list of properties with --format=csv, and --loop repeats it on an interval. The documentation directs you to nvidia-smi --help-query-gpu for the property list rather than enumerating it, so read that on your own machine rather than trusting a list you found somewhere.

Track X — AMD Ryzen AI Max+ 395Partial

AMD publishes no equivalent of the DCGM exporter for this class of machine, so this track uses a small exporter shipped with the lab that reads the command-line tool and republishes two numbers.

The command-line tools are documented; a Prometheus exporter is not. AMD’s own documentation describes AMD SMI as “the successor to ROCm SMI”, and its CLI guide gives amd-smi metric -p for power, amd-smi metric -m for memory usage per block and amd-smi metric -v for total video memory, with --json for machine-readable output on most commands. The older rocm-smi with --showpower, --showmemuse and --json is what this course’s command reference records and what Part 8 used on this track.

Because there is no official exporter, the lab ships rocm-exporter.py: it runs the command-line tool, matches the keys by pattern rather than by an exact spelling that changes between releases, and republishes power and memory under names the dashboard and the alerts use. Run it with --once first: it prints every key the tool returned and which ones it matched, which is the fastest way to find out that your ROCm version calls something by a different name.

Track M — Apple siliconPartial

Apple publishes the powermetrics manual only as a manual page on the machine, and unified memory has no separate video memory to report, so this track measures package power and machine memory instead.

macOS ships powermetrics, which samples power by subsystem and normally needs root. Apple does not publish its manual on the web, so read man powermetrics on your own Mac and confirm the sampler names before relying on them; this course has not executed it on Apple hardware.

Unified memory changes the question rather than answering it differently. There is no separate pool of video memory to report, so “accelerator memory” on this track is the machine’s memory, which is also the honest answer to whether a model fits. The lab’s mac-exporter.py reports total memory from sysctl hw.memsize and in-use memory computed from vm_stat as active plus wired plus compressed pages, and publishes a package power figure from powermetrics when it is run with the privileges that needs. Run it with --once before pointing Prometheus at it, and with --no-power if you would rather not run a scraper as root.

Track N — NVIDIA desktop or laptop

Identical to Track S: the DCGM exporter container on port 9400, the same field names, and nvidia-smi --query-gpu with --format=csv for a look without an exporter.

Two differences matter in practice. On a discrete card, framebuffer memory is a hard wall rather than a share of a pool, so DCGM_FI_DEV_FB_USED against the card’s capacity is a genuinely predictive number and the memory alert built on it is the most useful one you will have. And inside WSL2 the exporter’s access to the GPU depends on the driver path Microsoft documents; if the container starts but reports nothing, that is where to look rather than at the exporter.

Whichever track you are on, add the node exporter as well. Its README gives the default port as 9100 and enables collectors for CPU, memory, filesystem and disk statistics by default. It is what tells you the difference between the engine deciding not to allocate and the machine having nothing left to give, and that distinction is the first fork in every memory investigation.

Prometheus and Grafana, from files

Prometheus’s own getting-started guide gives the whole shape: a global section with a scrape_interval, a scrape_configs list where each entry has a job_name and static_configs with targets, a default listen port of 9090, and a start command of ./prometheus --config.file=prometheus.yml. The configuration reference adds evaluation_interval for how often rules are evaluated and rule_files as “a list of globs” from which “rules and alerts are read”.

Grafana is the part people configure by clicking, and then cannot rebuild. Its provisioning documentation is the answer: data sources come from YAML in the provisioning directory with apiVersion, name, type, access and url, and dashboards come from a provider file naming a directory of JSON. Provisioning also supports environment variables, with the syntax given as $ENV_VAR_NAME or ${ENV_VAR_NAME}, which is how a password stays out of a file you commit. The lab in this part ships all four files, and the point of shipping them is that a rebuilt stack comes back identical rather than approximately.

⌚Version note

Prometheus, Grafana, the node exporter and the DCGM exporter are not in this course’s version table, because the course has not pinned or checked them on the lab machines. The shipped .env therefore uses latest, with a comment telling you to replace it with a release tag you read yourself, and the next lesson shows how to move from a tag to a digest. Treat that as an instruction rather than a formality: a moving tag is how a stack that worked in the morning stops working in the afternoon.

Token accounting, and what a log costs you

The sixth question is the one with a policy attached. LiteLLM’s token counters are labelled by model and by hashed key, so sum by (model) (rate(litellm_total_tokens_metric[5m])) answers “what is this machine being used for” and the same query grouped by key alias answers “by whom”. That is enough for every decision you will actually make: whether to raise a limit, whether a model is earning its memory, whether that overnight job is really running every night.

It is enough without recording what anybody typed, which is the important part. Part 10 established that a model server’s log at any useful verbosity contains prompts, and that prompts are the sensitive thing. LiteLLM’s logging documentation gives the control: setting turn_off_message_logging “will prevent the messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tracked”. There is also a per-request header, documented as x-litellm-enable-message-redaction: true, for the cases where one caller wants redaction and the rest do not.

Full request tracing, in the distributed-systems sense of a span per hop with a trace id carried through, is available through the callback integrations the logging page documents under success_callback, failure_callback and callbacks. On a home service it is usually more machinery than the problem deserves: with four layers you can correlate by timestamp, and the router’s own log already tells you which alias was called and what happened. Reach for tracing when you have a request path you genuinely cannot follow by reading two logs side by side, which for most people is when agents arrive in Part 26.

Alerts that are worth being woken by

Prometheus’s alerting documentation separates the two halves cleanly: “Alerting rules in Prometheus servers send alerts to an Alertmanager. The Alertmanager then manages those alerts, including silencing, inhibition, aggregation and sending out notifications”. You do not need the second half to benefit from the first. Rules evaluated by Prometheus are visible on its own pages, and for a household service that is often the whole of what you need.

The rule syntax is groups, each with a name and a list of rules, each rule having an alert name, an expr, an optional for duration before it fires, plus labels and annotations. The for clause is what separates a useful alert from a noisy one.

Three alerts earn their place, and the lab sets the first two.

The service is unreachable. up{job="gateway"} == 0, held for a couple of minutes. It fires for a crash, a reboot, a container stopped to free memory and forgotten, and a misconfigured port. It is the cheapest alert there is and it catches more real incidents than anything else.

Work is queueing. Requests waiting above zero, held for ten minutes. Sustained queueing is the earliest honest signal that the capacity you planned for has been exceeded, and it usually appears hours before the failure it precedes.

The accelerator is nearly full. Memory above a high fraction of capacity, held for fifteen minutes. This is the state a service sits in quietly for days before it dies, and catching it is the whole point of the challenge at the end of this part.

Notice what is not on the list. Nothing alerts on throughput, because a slow model is annoying and not urgent. Nothing alerts on GPU utilisation, because low utilisation is the normal state of a machine waiting for somebody to type. Nothing alerts on temperature, because the hardware already protects itself. An alert you do not act on trains you to ignore alerts, which is worse than not having it.

Define the metric before drawing the dashboard

For every plotted value, write its unit, measurement boundary and aggregation. Does first-token latency include queueing? Does token throughput count generated tokens from failed requests? Is memory a process allocation, device usage or system total? Similar labels can hide different definitions across engines.

Use counters for accumulated events, gauges for current state and histograms for latency distributions where supported. A mean hides tails; a percentile computed from too few observations can be unstable. Avoid labels containing full prompts, user identifiers or unbounded request IDs, which create privacy and cardinality problems.

Design one alert from an operator action: for example, sustained queue growth alongside a latency breach should prompt investigation of admitted load and worker health. Test the alert with a controlled failure in the lab, then verify recovery clears it. A dashboard is useful when it links a symptom to evidence and a next step. A page full of moving graphs without definitions cannot reliably distinguish overload, cold loading, network delay and a model generating unexpectedly long answers.

Recap

Six questions: is it up, is it keeping up, how long does a caller wait, how full is the memory, what is it costing, and who is using it. llama.cpp publishes throughput and queueing but no cache occupancy and needs --metrics; vLLM publishes the full set including vllm:kv_cache_usage_perc; SGLang needs --enable-metrics and adds a cache hit rate; llama-swap knows what is loaded; LiteLLM knows who asked, labelled by model and key. The accelerator is read through the DCGM exporter on the NVIDIA tracks, through a shipped exporter over the ROCm command-line tool on Track X, and through powermetrics and vm_stat on Track M, with the node exporter everywhere to separate the engine running out from the machine running out. Prometheus and Grafana are both configured from files, so the dashboard is code. Token accounting by model and by key answers the usage question without recording anybody’s prompts, which turn_off_message_logging is there to prevent. And three alerts are enough: unreachable, queueing, nearly full.

Check your understanding

Question 1. Your Prometheus target for llama-server is green, but every panel that reads llamacpp: metrics is empty. What is the first thing to check?

• The scrape interval is too long
• llama-server was started without --metrics, which the README documents as disabled by default; the port answers, so the target is up, but the metrics endpoint publishes nothing
• Prometheus cannot resolve the host name
• The dashboard is querying the wrong data source

Show the answer and why

Answer: llama-server was started without --metrics, which the README documents as disabled by default; the port answers, so the target is up, but the metrics endpoint publishes nothing

This is the characteristic failure of engine metrics, and it looks like a healthy target because the process really is running. The same shape occurs with SGLang without --enable-metrics and LiteLLM without the Prometheus callback in its configuration.

Question 2. You want one panel showing how close the engine is to running out of cache. Which engine gives you a direct answer, and what do you use on llama.cpp?

• All engines publish the same cache metric
• vLLM publishes vllm:kv_cache_usage_perc directly; llama.cpp publishes no cache occupancy figure, so you infer pressure from accelerator memory and use the context high-water mark as the nearest engine-side signal
• llama.cpp publishes it and vLLM does not
• Only the DCGM exporter can answer this

Show the answer and why

Answer: vLLM publishes vllm:kv_cache_usage_perc directly; llama.cpp publishes no cache occupancy figure, so you infer pressure from accelerator memory and use the context high-water mark as the nearest engine-side signal

The engines have different internals and expose different things. Knowing which question your engine can and cannot answer is what stops you building a panel that is permanently empty and then trusting it.

Question 3. Which of these belong in the set of alerts a home model service should have? Select all that apply.

• The gateway has not answered a scrape for a couple of minutes
• Requests have been waiting for a slot for ten minutes
• Accelerator memory has been near capacity for fifteen minutes
• GPU utilisation has been below half for an hour

Show the answer and why

Answer: The gateway has not answered a scrape for a couple of minutes, Requests have been waiting for a slot for ten minutes, Accelerator memory has been near capacity for fifteen minutes

The first three are states you would act on. Low utilisation is the normal condition of a machine waiting for somebody to type, so an alert on it fires constantly and trains you to ignore the others.

Question 4. What does turn_off_message_logging change, and what does it leave alone?

• It disables all logging
• The documentation states it prevents messages and responses from being logged to your logging provider, while request metadata such as spend is still tracked
• It encrypts the log file
• It stops the proxy recording which model was called

Show the answer and why

Answer: The documentation states it prevents messages and responses from being logged to your logging provider, while request metadata such as spend is still tracked

That split is exactly what a home service wants: enough accounting to answer who used what, and none of the text. Token counts labelled by model and by key answer every operational question you actually have, without creating a store of the prompt text itself.

Question 5. Why does this lesson add the node exporter alongside a GPU exporter?

• To measure network throughput between machines
• To distinguish the engine declining to allocate from the machine having nothing left, which is the first fork in every memory investigation
• Because Prometheus requires at least two targets
• To replace the GPU exporter on tracks that have no GPU

Show the answer and why

Answer: To distinguish the engine declining to allocate from the machine having nothing left, which is the first fork in every memory investigation

An engine that reports a failed allocation and a host that is swapping are different faults with different fixes. Without host memory on the same dashboard you cannot tell them apart, and the challenge at the end of this part turns on exactly that distinction.

Check my answersClear