Lab: Dashboards for Your Cluster

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The monitoring image tags, the exporter versions and the per-track notes for each machine this was run on belong here once the validation pass has done so.

Objective

By the end of this hour your machine will answer the six questions from the observability lesson on one screen, from data it collected itself, and it will tell you when two of the answers become alarming. Concretely, you will have Prometheus scraping the gateway, at least one engine and your track’s accelerator; Grafana serving a dashboard that came from a file rather than from clicking; two alert rules loaded and one of them proved to fire; and a row in your lab notebook recording what this machine looks like when it is well.

That last item is the deliverable that matters. Every diagnosis you make afterwards is a comparison against it, and a dashboard with no remembered baseline tells you what is happening without telling you whether it is unusual.

⌂Single-machine path

One machine is enough for every step. The page is written for a cluster because the configuration is the same either way: a second machine is one more block in the scrape configuration and one more label on the graphs. If you have two, do the whole lab on one first and add the second in the last task.

Architecture

Everything here reads. Nothing in this lab can change what your gateway does, which is why it is safe to build on a service you are already using.

What scrapes what

• cachePrometheusPulls every target on a timer, stores the series, evaluates the alert rules
• clientGrafanaQueries Prometheus, renders the provisioned dashboard
• routerLiteLLM gatewayTokens and latency by model and by key
• workerllama-swapWhich model is loaded, and system metrics
• workerEnginellama-server, vLLM or SGLang: throughput, queueing, cache
• workerAccelerator exporterDCGM on Tracks S and N, the shipped exporter on X and M
• workernode exporterHost memory, disk and processor
• workerA second machineOptional: the same three exporters, scraped over the LAN

• Prometheus ⟷ connected to LiteLLM gatewayscrape /metrics
• Prometheus ⟷ connected to llama-swapscrape /metrics
• Prometheus ⟷ connected to Enginescrape /metrics
• Prometheus ⟷ connected to Accelerator exporterscrape /metrics
• Prometheus ⟷ connected to node exporterscrape /metrics
• Prometheus ⟷ connected to A second machinescrape over the LAN
• Grafana ⟷ connected to Prometheusquery

Prometheus pulls; nothing pushes to it. That is why adding a machine is a configuration change on one host rather than an agent installation on every host, and it is why a target that disappears is itself a signal.

Requirements

The Part 9 gateway, running. At least one engine with a model you can send a request to. Docker with the Compose plugin on the three Linux tracks; Docker Desktop or Homebrew on Track M. About sixty minutes, all of it attended, with no downloads beyond four container images. The memory floor is 8 GB because that is what the service being watched needs; the monitoring stack itself is small.

Track S — NVIDIA DGX Spark

Container path throughout, including the accelerator exporter. Bring the stack up with the nvidia profile so that the DCGM exporter starts beside Prometheus and Grafana.

Look up the DCGM exporter tag on NGC before your first run and put the whole reference in .env: NVIDIA’s installation guide writes the image with a tag variable rather than publishing a moving one, so there is nothing sensible for this course to fill in.

Track X — AMD Ryzen AI Max+ 395Partial

No official Prometheus exporter exists for this class of AMD accelerator, so this track runs a small exporter shipped with the lab that reads the command-line tool.

Prometheus, Grafana and the node exporter come up from Compose as on every track. The accelerator exporter is rocm-exporter.py, run natively rather than in a container so that it can reach the device without any device-passing configuration.

Run it with --once before anything else. It prints every key the command-line tool returned and which ones it matched, and if your ROCm version spells them differently that is where you will see it, in ten seconds rather than after an hour of empty panels.

Track M — Apple siliconPartial

Apple ships no exporter and publishes the powermetrics manual only on the machine, so the power reading here is taken by a shipped script that needs root and has not been executed on Apple hardware by this course.

Two ways to run the stack. Docker Desktop runs compose-monitoring.yaml unchanged except for the node exporter, whose host-root mount and pid: host do not behave the same way inside Docker Desktop’s virtual machine; either accept that its numbers describe the virtual machine, or install Prometheus, Grafana and the node exporter natively from Homebrew as Part 7 did for its own stack and point them at the same configuration files.

The accelerator exporter is mac-exporter.py, run natively. It needs root for the power reading; --no-power runs it without root and publishes memory only, which is enough for every panel except one.

Track N — NVIDIA desktop or laptop

Identical to Track S: the nvidia profile starts the DCGM exporter.

Inside WSL2, the exporter’s access to the GPU depends on the driver path Microsoft documents for CUDA on WSL. If the container starts and publishes nothing, check that before suspecting the exporter, and record which it was, because it is the kind of thing you will meet again.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-23-operating-a-local-ai-service"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-23-operating-a-local-ai-service and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Take the files and read them

Eight files. Read each one before you run it: they are commented as the explanation, and this page does not repeat what the comments say.

RunnableAll tracks

env-example.txt

# Purpose: every setting the Part 23 front door and monitoring stack read. Copy this file
#          to `.env` beside compose-monitoring.yaml and fill in the empty lines. Nothing
#          here is a secret except the Grafana admin password, which you generate rather
#          than copy, and which is the only reason this file must never be committed.
# Platform: all
# Minimum memory: 8 GB
# Assumes: `cp env-example.txt .env` and then an editor. Docker Compose reads `.env`
#          automatically from the directory you run it in; the shell scripts in this part
#          read it with `set -a; . ./.env; set +a`.

# ------------------------------------------------------------- what the stack listens on
# Prometheus and Grafana stay on the loopback address. They are operator tools, they hold
# a picture of everything your machine does, and there is no reason for anything on the
# network to reach them directly. Put them behind the same front door as the gateway if
# you want them from another machine.
MONITORING_HOST=127.0.0.1
PROMETHEUS_PORT=9090
GRAFANA_PORT=3000

# The Grafana administrator password. Generate one with: openssl rand -hex 16
# Grafana's Docker page documents GF_SECURITY_ADMIN_PASSWORD as the environment variable
# for this setting, and GF_SECURITY_ADMIN_PASSWORD__FILE for reading it from a file.
# Leave this empty and Grafana starts with its own default, which is the single most
# common way a home dashboard ends up readable by the whole house.
GRAFANA_ADMIN_PASSWORD=

# ------------------------------------------------------------------ what gets scraped
# The gateway from Part 9. These are the ports its .env published; change them here if
# you changed them there. Prometheus reaches host services through host.docker.internal,
# which compose-monitoring.yaml maps to the host gateway.
GATEWAY_PORT=4000
SWAP_PORT=9292

# The engines. llama-server needs --metrics for its endpoint to exist at all; vLLM and
# SGLang publish theirs on the same port as the API.
LLAMA_SERVER_PORT=8080
VLLM_PORT=8000
SGLANG_PORT=30000

# The exporters. node exporter's default port is 9100 and the DCGM exporter's is 9400,
# both from their own documentation. The two exporters this part ships are given the next
# free numbers so that nothing collides on a machine running more than one.
NODE_EXPORTER_PORT=9100
DCGM_EXPORTER_PORT=9400
ROCM_EXPORTER_PORT=9401
MAC_EXPORTER_PORT=9402

# ---------------------------------------------------------------------------- images
# None of these four is in this course's version table, so the course does not pin them
# for you: it has not checked them on the lab machines. Read the current release tag on
# each project's own releases page, write it here in place of `latest` before your first
# run, and record what you chose in the changelog that Part 23's fifth lesson asks you to
# keep. `latest` is a moving target and moving targets are how a stack that worked in the
# morning stops working in the afternoon.
#
# Once the stack matters, replace the tag with a digest. Docker's documentation describes
# pulling by digest as specifying exactly which version of an image to pull, so that the
# image you run today is the image you ran yesterday. Read a pulled image's digest with:
#   docker image inspect --format '{{index .RepoDigests 0}}' <image>
PROMETHEUS_IMAGE=prom/prometheus:latest
GRAFANA_IMAGE=grafana/grafana:latest
NODE_EXPORTER_IMAGE=quay.io/prometheus/node-exporter:latest

# NVIDIA's installation guide writes this image as
#   nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}
# and does not publish a moving tag to fill in for you. Look the tag up on NGC and write
# the whole reference here. The service that uses it sits behind a Compose profile, so the
# stack starts without it on the tracks that have no NVIDIA GPU.
DCGM_EXPORTER_IMAGE=

# ------------------------------------------------------------------ the front door
# Only needed if you put NGINX in front of the gateway. The name is whatever you gave the
# machine in your own DNS or hosts file; home.arpa is the domain reserved for exactly this
# purpose, so it is what the shipped configuration uses as an example.
FRONT_SERVER_NAME=gateway.home.arpa
FRONT_TLS_CERT=/etc/ssl/local/gateway.crt
FRONT_TLS_KEY=/etc/ssl/local/gateway.key

# ------------------------------------------------------------------------- the estate
# Where estate-manifest.py and backup-estate.sh look. Absolute paths, no trailing slash.
MODELS_DIR=
GATEWAY_DIR=
BACKUP_DIR=

Download env-example.txt83 lines

RunnableAll tracks

compose-monitoring.yaml

# Purpose: the metrics stack - Prometheus scraping the gateway, the engines and the
#          machine, Grafana provisioned from files so the dashboard is code rather than
#          clicks, node exporter for the host, and the NVIDIA DCGM exporter behind a
#          profile for the tracks that have an NVIDIA GPU
# Platform: spark, strix, nvidia (Linux with Docker Engine and the Compose plugin), and
#           mac where Docker Desktop is installed. Track M can also run Prometheus and
#           Grafana natively from Homebrew; the page says which files to point them at.
# Minimum memory: 8 GB
# Assumes: a .env beside this file, copied from env-example.txt and filled in, with
#          GRAFANA_ADMIN_PASSWORD set to something you generated. The four configuration
#          files in this directory are mounted read-only; the two named volumes hold the
#          time series and Grafana's own state and are the only things worth backing up.
#          Prometheus reaches services running on the host through host.docker.internal,
#          which the extra_hosts line below maps to the host gateway.
#
#   docker compose -f compose-monitoring.yaml up -d                     (every track)
#   docker compose -f compose-monitoring.yaml --profile nvidia up -d    (Tracks S and N)

name: local-llm-monitoring

services:
  prometheus:
    image: "${PROMETHEUS_IMAGE}"
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      # Thirty days of history is enough to see a weekly pattern and to answer the
      # question the challenge page asks, and small enough not to matter on any disk.
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    volumes:
      - ./prometheus-config.yaml:/etc/prometheus/prometheus.yml:ro
      - ./alert-rules.yaml:/etc/prometheus/alert-rules.yaml:ro
      - prometheus-data:/prometheus
    extra_hosts:
      # Everything Prometheus scrapes on this machine - the gateway, llama-swap, the
      # engines, the per-track GPU exporter - is on the host, not in this network.
      - "host.docker.internal:host-gateway"
    ports:
      # Loopback only. Prometheus holds a complete picture of what this machine does and
      # has no authentication of its own.
      - "${MONITORING_HOST}:${PROMETHEUS_PORT}:9090"

  grafana:
    image: "${GRAFANA_IMAGE}"
    restart: unless-stopped
    depends_on:
      - prometheus
    environment:
      # Grafana's Docker documentation gives the convention as GF_<SectionName>_<KeyName>,
      # and GF_<SectionName>_<KeyName>__FILE for reading a value from a file. The password
      # comes from .env and is never written into this file.
      GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
      # Nobody signs up for a household dashboard.
      GF_USERS_ALLOW_SIGN_UP: "false"
      # Anonymous access is off. It is the default; it is stated here so that turning it
      # on is a visible edit rather than an absence nobody noticed.
      GF_AUTH_ANONYMOUS_ENABLED: "false"
    volumes:
      # Provisioning: Grafana reads datasources and dashboard providers from files under
      # its provisioning directory at start-up, so the dashboard is a file you can commit
      # rather than a thing you clicked together and cannot rebuild.
      - ./grafana-datasource.yaml:/etc/grafana/provisioning/datasources/prometheus.yaml:ro
      - ./grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/local-llm.yaml:ro
      - ./grafana-dashboard.json:/var/lib/grafana/dashboards/local-llm.json:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "${MONITORING_HOST}:${GRAFANA_PORT}:3000"

  node-exporter:
    image: "${NODE_EXPORTER_IMAGE}"
    restart: unless-stopped
    # The exporter's own README gives this shape: the host's root mounted read-only and
    # --path.rootfs pointing at it, so that filesystem and memory figures describe the
    # machine rather than the container.
    pid: host
    command:
      - "--path.rootfs=/host"
    volumes:
      - "/:/host:ro,rslave"
    ports:
      - "${MONITORING_HOST}:${NODE_EXPORTER_PORT}:9100"

  # Tracks S and N only. NVIDIA's installation guide gives the container command as
  # `--gpus all --cap-add SYS_ADMIN -p 9400:9400` against the nvcr.io image, which is what
  # this reproduces. It stays behind a profile so that `docker compose up -d` on a machine
  # with no NVIDIA GPU does not fail on an image it cannot use.
  dcgm-exporter:
    image: "${DCGM_EXPORTER_IMAGE}"
    profiles: ["nvidia"]
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "${MONITORING_HOST}:${DCGM_EXPORTER_PORT}:9400"

volumes:
  prometheus-data:
  grafana-data:

Download compose-monitoring.yaml107 lines

RunnableAll tracks

prometheus-config.yaml

# Purpose: tell Prometheus what to scrape, how often, and where the alert rules live.
#          Mounted into the container as /etc/prometheus/prometheus.yml, which is the path
#          compose-monitoring.yaml passes to --config.file.
# Platform: all
# Minimum memory: 8 GB
# Assumes: the Part 9 gateway and llama-swap running on this host, an engine with its
#          metrics endpoint enabled, and one GPU exporter for your track. Every job below
#          that your machine does not run will show as down in Prometheus: delete the ones
#          that do not apply rather than leaving them red, because a dashboard with a
#          permanently failing target is a dashboard nobody looks at.
#          Reload after an edit without restarting:
#            curl -s -X POST http://127.0.0.1:9090/-/reload

global:
  # Fifteen seconds is short enough to see a request burst and long enough that scraping
  # costs nothing. The default in Prometheus's own documentation is one minute.
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    # Set this to a name for the machine when you have more than one. It ends up on every
    # series and it is what lets a two-machine dashboard tell them apart.
    host: primary

rule_files:
  - /etc/prometheus/alert-rules.yaml

scrape_configs:
  # Prometheus watching itself. Keep it: when nothing else is being scraped, this is how
  # you tell "the service is down" from "Prometheus is down".
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  # ------------------------------------------------------------------ the gateway
  # LiteLLM exposes Prometheus metrics on the proxy port once litellm_settings has
  # callbacks: ["prometheus"]. Without that line this target scrapes successfully and
  # returns almost nothing, which is a confusing failure: check the config first.
  - job_name: gateway
    static_configs:
      - targets: ["host.docker.internal:4000"]
        labels:
          component: router

  # llama-swap's README describes /metrics as system and GPU metrics for Prometheus. It is
  # the layer that knows which model is loaded, which is exactly the thing you want beside
  # a memory graph.
  - job_name: swap
    static_configs:
      - targets: ["host.docker.internal:9292"]
        labels:
          component: model-manager

  # ------------------------------------------------------------------- the engines
  # llama-server publishes nothing at all unless it was started with --metrics. If this
  # target is up and empty, that flag is missing from the command line in llama-swap.yaml.
  - job_name: llama-server
    static_configs:
      - targets: ["host.docker.internal:8080"]
        labels:
          engine: llama.cpp

  # vLLM serves its metrics on the same port as the API. Delete this job on a machine that
  # does not run vLLM.
  - job_name: vllm
    static_configs:
      - targets: ["host.docker.internal:8000"]
        labels:
          engine: vllm

  # SGLang publishes metrics only when the server was started with --enable-metrics.
  - job_name: sglang
    static_configs:
      - targets: ["host.docker.internal:30000"]
        labels:
          engine: sglang

  # ------------------------------------------------------------------- the machine
  - job_name: node
    static_configs:
      - targets: ["host.docker.internal:9100"]

  # --------------------------------------------------------- one of these, per track
  # Tracks S and N: NVIDIA's DCGM exporter, listening on 9400 with the metrics on the
  # default path. Started by the nvidia profile in compose-monitoring.yaml.
  - job_name: dcgm
    static_configs:
      - targets: ["host.docker.internal:9400"]
        labels:
          accelerator: nvidia

  # Track X: the small exporter this part ships, because AMD publishes no equivalent of
  # the DCGM exporter for a Ryzen AI Max+ desktop. It reads rocm-smi and republishes the
  # two numbers this dashboard needs.
  - job_name: rocm
    static_configs:
      - targets: ["host.docker.internal:9401"]
        labels:
          accelerator: amd

  # Track M: the powermetrics exporter this part ships. It needs root, so it is run
  # deliberately and by hand rather than from Compose.
  - job_name: mac
    static_configs:
      - targets: ["host.docker.internal:9402"]
        labels:
          accelerator: apple

Download prometheus-config.yaml106 lines

RunnableAll tracks

alert-rules.yaml

# Purpose: the three alerts this part asks you to keep - the service is unreachable, work
#          is queueing, and the accelerator is close to full. Loaded by Prometheus through
#          the rule_files entry in prometheus-config.yaml.
# Platform: all
# Minimum memory: 8 GB
# Assumes: the scrape jobs named in prometheus-config.yaml. Prometheus evaluates these
#          rules and shows the result at /alerts and /rules; sending a notification
#          anywhere needs Alertmanager, which the documentation describes as the separate
#          component that does silencing, inhibition, aggregation and delivery. This file
#          is deliberately Alertmanager-free: three rules you can see on a page you look
#          at is a better starting point than a notification pipeline you never finish.
#
#          Confirm every metric name against your own /metrics output before trusting a
#          rule. An expression that names a metric nothing publishes never fires, and an
#          alert that never fires looks exactly like an alert with nothing wrong.
#          Check what a target actually exposes with:
#            curl -s http://127.0.0.1:9400/metrics | grep -E '^# HELP'

groups:
  - name: local-llm-service
    rules:
      # ---------------------------------------------------------------------- alert one
      # The cheapest and most valuable alert there is. `up` is the synthetic metric
      # Prometheus writes for every scrape, so this fires whether the gateway crashed, the
      # machine rebooted, or somebody stopped the container to free memory and forgot.
      # Two minutes of patience keeps a restart from paging you.
      - alert: GatewayUnreachable
        expr: up{job="gateway"} == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: The model gateway has not answered a scrape for two minutes
          description: >-
            Prometheus cannot reach the LiteLLM proxy. Everything that talks to this
            machine is getting connection errors. Check `docker compose ps` in the gateway
            directory and the router's own log before anything else.

      # ---------------------------------------------------------------------- alert two
      # Work arriving faster than the machine finishes it. Each engine names this
      # differently, so the expression asks all three and takes whichever exists: an
      # operand whose metric nothing publishes contributes an empty vector and is ignored.
      # Sustained queueing is the earliest honest warning that capacity has been exceeded,
      # and it usually precedes the memory failure by hours.
      - alert: RequestsQueueing
        expr: >-
          llamacpp:requests_deferred > 0
          or vllm:num_requests_waiting > 0
          or sglang:num_queue_reqs > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Requests have been waiting for a slot for ten minutes
          description: >-
            The engine has more work than slots. Either the concurrency you planned for is
            being exceeded, or a long request is holding a slot. Look at the running
            request count and the context length before you raise any limit.

      # -------------------------------------------------------------------- alert three
      # The guard rail the challenge page asks you to add: accelerator memory close to
      # full, held for long enough that a burst does not trigger it. Two expressions,
      # because the tracks report memory through different exporters, joined with `or` so
      # that one file works everywhere.
      #
      # On Tracks S and N the two DCGM framebuffer fields come from the exporter's default
      # counters file; confirm both appear in your own /metrics before relying on this.
      # On Tracks X and M the names are the ones rocm-exporter.py and mac-exporter.py in
      # this directory publish, so they are exactly what you will see.
      - alert: AcceleratorMemoryHigh
        expr: >-
          DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.92
          or local_llm_accelerator_memory_used_bytes
             / local_llm_accelerator_memory_total_bytes > 0.92
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: Accelerator memory has been above ninety-two per cent for fifteen minutes
          description: >-
            There is no headroom left for a longer prompt or one more concurrent request.
            This is the state a service sits in for hours before it fails, and it is the
            one worth catching. Check which model is loaded, what context length it was
            started with, and whether anything was meant to have been unloaded.

Download alert-rules.yaml84 lines

RunnableAll tracks

grafana-datasource.yaml

# Purpose: give Grafana its Prometheus data source from a file, so that a rebuilt stack
#          comes back with the same connection rather than a set-up wizard
# Platform: all
# Minimum memory: 8 GB
# Assumes: mounted at /etc/grafana/provisioning/datasources/prometheus.yaml, which is where
#          Grafana reads data source provisioning from at start-up. The URL is the Compose
#          service name on the stack's own network, so it needs no address and no host name.
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # A fixed uid so that grafana-dashboard.json can name this data source and keep
    # naming it after the stack is rebuilt. Without one, Grafana generates a new uid each
    # time and the dashboard's panels come back empty.
    uid: prometheus
    # proxy: Grafana's own process queries Prometheus. The alternative sends the browser
    # there directly, which would mean publishing Prometheus to whoever opens a dashboard.
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    # Grafana deletes and recreates a provisioned data source when this file changes, so
    # editing the file is how you change it; changes made in the interface are overwritten.
    editable: false
    jsonData:
      # Match the scrape interval in prometheus-config.yaml so that graphs do not invent
      # detail the data does not have.
      timeInterval: 15s

Download grafana-datasource.yaml28 lines

RunnableAll tracks

grafana-dashboards.yaml

# Purpose: tell Grafana to load dashboards from a directory of JSON files, so the dashboard
#          is a file under version control rather than a thing somebody clicked together
# Platform: all
# Minimum memory: 8 GB
# Assumes: mounted at /etc/grafana/provisioning/dashboards/local-llm.yaml, with the JSON
#          dashboards mounted under the path named below. Grafana rescans that directory on
#          the interval given here, so editing the JSON and saving is enough to see the
#          change; there is nothing to import and nothing to remember to export.
apiVersion: 1

providers:
  - name: local-llm
    type: file
    # Leave the dashboards editable in the interface so you can experiment, but understand
    # what that means: a change made in the browser is lost the next time this provider
    # reloads the file. Edit the JSON when you want to keep something.
    allowUiUpdates: true
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

Download grafana-dashboards.yaml21 lines

RunnableAll tracks

grafana-dashboard.json

{
  "uid": "local-llm-service",
  "title": "Local LLM service",
  "description": "Purpose: the four things worth watching on a local model service - throughput, latency, accelerator memory and power - on one screen, with one panel per question rather than one panel per metric. Platform: all. Minimum memory: 8 GB. Assumes: the scrape jobs in prometheus-config.yaml and the data source uid 'prometheus' from grafana-datasource.yaml. Each panel queries every engine it might find and shows whichever answers, so the same file works on all four tracks. A panel that stays empty means the metric is not published under that name on your machine: read your own /metrics output and correct the query here, in the file, not in the browser.",
  "tags": ["local-llm", "part-23"],
  "timezone": "browser",
  "editable": true,
  "schemaVersion": 39,
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-6h", "to": "now" },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Generation throughput",
      "description": "Output tokens per second, as each engine reports it. llama.cpp publishes an average rate directly; vLLM and SGLang publish counters, so this takes their rate over five minutes. Counter names differ by a _total suffix between client library versions, which is why both spellings are asked for.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "llamacpp:predicted_tokens_seconds",
          "legendFormat": "llama.cpp {{instance}}"
        },
        {
          "refId": "B",
          "expr": "rate(vllm:generation_tokens[5m]) or rate(vllm:generation_tokens_total[5m])",
          "legendFormat": "vLLM {{model_name}}"
        },
        {
          "refId": "C",
          "expr": "sglang:gen_throughput",
          "legendFormat": "SGLang {{instance}}"
        }
      ]
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "Requests: running and waiting",
      "description": "The two numbers that separate 'busy' from 'over capacity'. Running requests rising is use. Waiting requests rising is the warning, and it is what the RequestsQueueing alert watches.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "llamacpp:requests_processing",
          "legendFormat": "llama.cpp running"
        },
        {
          "refId": "B",
          "expr": "llamacpp:requests_deferred",
          "legendFormat": "llama.cpp waiting"
        },
        {
          "refId": "C",
          "expr": "vllm:num_requests_running",
          "legendFormat": "vLLM running"
        },
        {
          "refId": "D",
          "expr": "vllm:num_requests_waiting",
          "legendFormat": "vLLM waiting"
        },
        {
          "refId": "E",
          "expr": "sglang:num_running_reqs",
          "legendFormat": "SGLang running"
        },
        {
          "refId": "F",
          "expr": "sglang:num_queue_reqs",
          "legendFormat": "SGLang waiting"
        }
      ]
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Time to first token, 95th percentile",
      "description": "How long a caller waits before anything appears. Only vLLM and SGLang publish this as a histogram; on llama.cpp the closest available answer is the router's own end-to-end latency, which is the third query here and which includes the model load on a cold start.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
      "fieldConfig": { "defaults": { "unit": "s", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))",
          "legendFormat": "vLLM"
        },
        {
          "refId": "B",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(sglang:time_to_first_token_seconds_bucket[5m])))",
          "legendFormat": "SGLang"
        },
        {
          "refId": "C",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(litellm_request_total_latency_metric_bucket[5m])))",
          "legendFormat": "gateway, end to end"
        }
      ]
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "Key-value cache occupancy",
      "description": "The fraction of the cache that is holding conversation. This is the number that creeps up over a week as prompts get longer, and it is the one the three-in-the-morning failure is written on. Published by vLLM directly; on llama.cpp the high-water mark of context observed is the nearest equivalent and is shown beside it.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "vllm:kv_cache_usage_perc",
          "legendFormat": "vLLM KV cache"
        },
        {
          "refId": "B",
          "expr": "sglang:token_usage",
          "legendFormat": "SGLang token usage"
        },
        {
          "refId": "C",
          "expr": "llamacpp:n_tokens_max",
          "legendFormat": "llama.cpp context high-water mark"
        }
      ]
    },
    {
      "id": 5,
      "type": "timeseries",
      "title": "Accelerator memory in use",
      "description": "Tracks S and N read this from the DCGM exporter, in mebibytes, so it is converted here. Tracks X and M read it from the exporter this part ships, in bytes. One panel, two sources, whichever your machine has.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "DCGM_FI_DEV_FB_USED * 1024 * 1024",
          "legendFormat": "NVIDIA GPU {{gpu}}"
        },
        {
          "refId": "B",
          "expr": "local_llm_accelerator_memory_used_bytes",
          "legendFormat": "{{accelerator}} used"
        },
        {
          "refId": "C",
          "expr": "local_llm_accelerator_memory_total_bytes",
          "legendFormat": "{{accelerator}} total"
        }
      ]
    },
    {
      "id": 6,
      "type": "timeseries",
      "title": "Power draw",
      "description": "Watts, from the DCGM exporter on the NVIDIA tracks and from the shipped exporter on the other two. This is the panel the cost model reads: take the average over a sustained generation run and put it in cost-inputs.json as load_watts, and the average over an idle hour as idle_watts.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "DCGM_FI_DEV_POWER_USAGE",
          "legendFormat": "NVIDIA GPU {{gpu}}"
        },
        {
          "refId": "B",
          "expr": "local_llm_accelerator_power_watts",
          "legendFormat": "{{accelerator}}"
        }
      ]
    },
    {
      "id": 7,
      "type": "timeseries",
      "title": "Tokens by model, through the gateway",
      "description": "Who is using what. LiteLLM labels its token counters by model and by hashed key, so this is where per-application accounting lives. If the panel is empty, either callbacks: [\"prometheus\"] is missing from litellm-config.yaml or your client library spells the counter with a _total suffix; both spellings are asked for here.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "sum by (model) (rate(litellm_total_tokens_metric[5m])) or sum by (model) (rate(litellm_total_tokens_metric_total[5m]))",
          "legendFormat": "{{model}}"
        }
      ]
    }
  ]
}

Download grafana-dashboard.json196 lines

RunnableAll tracks

check-monitoring.sh

#!/usr/bin/env bash
# Purpose: prove the monitoring stack is doing its job - Prometheus healthy, the rules
#          loaded, at least one engine target up, the gateway target up, a GPU exporter
#          answering, and Grafana serving the provisioned dashboard - with one pass or
#          fail line for each
# Platform: all (spark, strix, mac, nvidia)
# Minimum memory: 8 GB, which is what the service being watched needs
# Assumes: curl on PATH, the stack from compose-monitoring.yaml running, and a .env beside
#          this script. Nothing is printed that would reveal the Grafana password: the
#          Grafana check uses its unauthenticated health endpoint only.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
fi

: "${MONITORING_HOST:=127.0.0.1}"
: "${PROMETHEUS_PORT:=9090}"
: "${GRAFANA_PORT:=3000}"

PROM="http://${MONITORING_HOST}:${PROMETHEUS_PORT}"
GRAFANA="http://${MONITORING_HOST}:${GRAFANA_PORT}"

command -v curl >/dev/null 2>&1 || { echo "curl is not on PATH." >&2; exit 1; }

PASSES=0
FAILURES=0
SKIPS=0

skip() {
    # skip <name> <reason> - a check that could not be made, which is not a failure
    SKIPS=$((SKIPS + 1))
    printf '  SKIP  %s  (%s)\n' "$1" "$2"
}

report() {
    # report <name> <ok:0|1> [detail]
    if [ "$2" -eq 0 ]; then
        PASSES=$((PASSES + 1))
        printf '  PASS  %s\n' "$1"
    else
        FAILURES=$((FAILURES + 1))
        printf '  FAIL  %s%s\n' "$1" "${3:+  (${3})}"
    fi
}

# Ask Prometheus a question and return the first sample value, or an empty string.
# The instant-query response puts each sample as "value":[<timestamp>,"<value>"], and the
# value is the quoted second element. No jq, because this has to run on a bare machine.
promql() {
    curl -sf --get "${PROM}/api/v1/query" --data-urlencode "query=$1" 2>/dev/null \
        | grep -oE '"value":\[[0-9.eE+-]+,"[^"]+"' \
        | head -n 1 \
        | sed -e 's/.*,"//' -e 's/"$//' || true
}

# Count the series a query returns, which is the only thing most checks here need.
count_series() {
    curl -sf --get "${PROM}/api/v1/query" --data-urlencode "query=$1" 2>/dev/null \
        | grep -o '"metric"' | wc -l | tr -d ' '
}

echo "==> checking ${PROM}"

if curl -sf "${PROM}/-/healthy" >/dev/null; then
    report "Prometheus is healthy" 0
else
    report "Prometheus is healthy" 1 "is the stack up?"
    echo "  Nothing else can pass while Prometheus is down. Stopping here." >&2
    exit 1
fi

RULES="$(curl -sf "${PROM}/api/v1/rules" || true)"
for rule in GatewayUnreachable RequestsQueueing AcceleratorMemoryHigh; do
    if printf '%s' "$RULES" | grep -q -- "$rule"; then
        report "alert rule ${rule} is loaded" 0
    else
        report "alert rule ${rule} is loaded" 1 "check rule_files and the container mount"
    fi
done

# The gateway from Part 9. If this is down, the dashboard has no request data at all.
if [ "$(promql 'up{job="gateway"}')" = "1" ]; then
    report "gateway target is up" 0
else
    report "gateway target is up" 1 "is the Part 9 gateway running on this machine?"
fi

# At least one engine, whichever it is. A machine only needs one for the dashboard to work.
ENGINES_UP="$(count_series 'up{job=~"llama-server|vllm|sglang"} == 1')"
if [ "${ENGINES_UP:-0}" -ge 1 ]; then
    report "at least one engine target is up" 0
else
    report "at least one engine target is up" 1 "llama-server needs --metrics to publish anything"
fi

# One GPU exporter, whichever your track uses.
GPU_SERIES="$(count_series 'DCGM_FI_DEV_POWER_USAGE or local_llm_accelerator_power_watts')"
if [ "${GPU_SERIES:-0}" -ge 1 ]; then
    report "a GPU or accelerator exporter is publishing power" 0
else
    report "a GPU or accelerator exporter is publishing power" 1 "run your track's exporter"
fi

MEM_SERIES="$(count_series 'DCGM_FI_DEV_FB_USED or local_llm_accelerator_memory_used_bytes')"
if [ "${MEM_SERIES:-0}" -ge 1 ]; then
    report "accelerator memory is being recorded" 0
else
    report "accelerator memory is being recorded" 1 "the memory panel will stay empty"
fi

# The host itself, which is what tells you the machine ran out of memory rather than the
# engine deciding not to allocate.
if [ "$(promql 'up{job="node"}')" = "1" ]; then
    report "node exporter target is up" 0
else
    report "node exporter target is up" 1 "host memory and disk will be missing"
fi

echo "==> checking ${GRAFANA}"

if curl -sf "${GRAFANA}/api/health" >/dev/null; then
    report "Grafana is healthy" 0
else
    report "Grafana is healthy" 1 "check the container log for a provisioning error"
fi

# This endpoint needs a login, so it is only checked when the admin password is available
# from .env. The password is passed to curl and never printed.
if [ -n "${GRAFANA_ADMIN_PASSWORD:-}" ]; then
    if curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" \
        "${GRAFANA}/api/dashboards/uid/local-llm-service" >/dev/null 2>&1; then
        report "the provisioned dashboard is present" 0
    else
        report "the provisioned dashboard is present" 1 \
            "check the Grafana container log for a provisioning error"
    fi
else
    skip "the provisioned dashboard is present" \
        "GRAFANA_ADMIN_PASSWORD is not set; open the dashboard in a browser instead"
fi

echo ""
echo "    ${PASSES} passed, ${FAILURES} failed, ${SKIPS} skipped"
if [ "$FAILURES" -gt 0 ]; then
    echo "    Open ${PROM}/targets to see which scrape jobs are failing and why." >&2
    exit 1
fi

Download check-monitoring.sh154 lines

Put them in one directory, beside but separate from your gateway directory. Track X and Track M also need their exporter, which is on this page further down.

RunnableAll tracks

your settings, and a password you generated

cp env-example.txt .env

openssl rand -hex 16

Edit .env. Put the hex string in GRAFANA_ADMIN_PASSWORD. Replace the three latest image tags with release tags you look up now, and write down which ones you chose. On Tracks S and N, put the full DCGM exporter reference in DCGM_EXPORTER_IMAGE.

⛨Security

.env is the only file here that holds a secret. Add it to .gitignore before you put the password in it, not after. Grafana with its own default password on a machine other people can reach is the single most common way a household dashboard becomes readable by the household, and the stack publishes Grafana on the loopback address for the same reason.

2. Turn the metrics endpoints on

This is the step that decides whether the rest of the hour works, and it is the one people skip because the services are already running.

The engine. llama-server publishes nothing without --metrics, which its README documents as disabled by default. Part 9’s llama-swap.yaml already includes it in the shared server macro; confirm it is still there. If you serve with vLLM, its metrics are on the API port with no flag. If you serve with SGLang, add --enable-metrics.

The router. LiteLLM publishes its metrics only when the Prometheus callback is on. Add the line to litellm-config.yaml under litellm_settings and restart the proxy:

Fragment — not complete on its own

litellm_settings:
  callbacks: ["prometheus"]

Check both before going further. A target that is up and empty is the failure this lab exists to teach you to recognise, and it is much cheaper to find now.

RunnableAll tracks

does anything actually publish?

curl -s http://127.0.0.1:8080/metrics | head -n 5

curl -s http://127.0.0.1:4000/metrics | head -n 5

curl -s http://127.0.0.1:9292/metrics | head -n 5

Output — what you should see

# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 0

Empty output from any of the three means that component is not publishing, and no amount of configuring Prometheus will change that.

3. Bring the stack up

RunnableTrack X · Ryzen AI Max+

every track, without an NVIDIA GPU

docker compose -f compose-monitoring.yaml up -d

docker compose -f compose-monitoring.yaml ps

RunnableTrack N · NVIDIA GPU

Tracks S and N, with the DCGM exporter

docker compose -f compose-monitoring.yaml --profile nvidia up -d

docker compose -f compose-monitoring.yaml --profile nvidia ps

Prometheus listens on the port its documentation gives as the default, and the compose file publishes it on the loopback address only. Open its targets page in a browser now, before Grafana, because the targets page is where the truth is.

4. Read the targets page, and delete what does not apply

Most of the jobs in the shipped configuration will be down, because no machine runs every engine in this course. That is expected and it is also a trap: a permanently red target teaches you to ignore red targets.

Open the targets page and, for each job that is down, decide which it is. A job you will never run gets deleted from prometheus-config.yaml. A job you meant to run and that is down is a fault to fix now. Then reload without restarting:

RunnableAll tracks

apply a configuration change in place

curl -s -X POST http://127.0.0.1:9090/-/reload

=Expected result

When you have finished, every remaining target should be up. That state is worth achieving because from now on a red target means something, and the moment you accept one permanently red job you have given up the cheapest alert in the whole system.

5. Add your track’s accelerator exporter

Track S — NVIDIA DGX Spark

Already running, from the nvidia profile. Confirm it is publishing what the dashboard expects.

RunnableTrack S · DGX Spark

what the DCGM exporter actually exports

curl -s http://127.0.0.1:9400/metrics | grep -E '^# HELP DCGM_FI_DEV_(FB|POWER|GPU)'

Note which field names appear. The exporter collects what its counters file lists, and the memory alert in alert-rules.yaml names two framebuffer fields; if one of them is absent here, that alert will never fire and you should say so in your notebook rather than assume it is watching.

Track X — AMD Ryzen AI Max+ 395

RunnableTrack X · Ryzen AI Max+

rocm-exporter.py

#!/usr/bin/env python3
"""Prometheus exporter for an AMD accelerator, built on rocm-smi.

Purpose: publish accelerator power and memory for Track X in the shape Prometheus scrapes,
    because AMD ships no equivalent of NVIDIA's DCGM exporter for a Ryzen AI Max+ desktop.
    It runs rocm-smi, reads two numbers out of its JSON, and serves them at /metrics.
Platform: strix (AMD Ryzen AI Max+ 395 and other ROCm machines). Not for the other tracks:
    Tracks S and N use the DCGM exporter, Track M uses mac-exporter.py in this directory.
Minimum memory: 8 GB, which is the service being watched; this process needs almost none.
Assumes: rocm-smi on PATH and a user who may run it. The JSON key names rocm-smi uses have
    changed between ROCm releases and AMD now describes amd-smi as the successor to
    rocm-smi, so this script does not hard-code a single spelling: it matches keys by
    pattern and tells you plainly which ones it found. Run it once with --once first and
    read the output before you point Prometheus at it.

Usage: python3 rocm-exporter.py --once
       python3 rocm-exporter.py --port 9401
       ROCM_SMI=/opt/rocm/bin/rocm-smi python3 rocm-exporter.py --port 9401
"""
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

ACCELERATOR = "amd"

# Key patterns, in the order they are preferred. rocm-smi's JSON keys are human sentences
# rather than identifiers, and they differ between releases, so each metric is described by
# a list of case-insensitive patterns and the first key that matches wins.
PATTERNS = {
    "power_watts": [r"average.*power.*\(w\)", r"\bpower\b.*\(w\)", r"socket.*power"],
    "memory_total_bytes": [r"vram total memory", r"total (vram|memory).*\(b\)"],
    "memory_used_bytes": [r"vram total used memory", r"used (vram|memory).*\(b\)"],
}

state = {"scrapes": 0, "errors": 0, "last_error": ""}


def rocm_smi_binary():
    return os.environ.get("ROCM_SMI", "rocm-smi")


def read_rocm_smi(timeout=15):
    """Run rocm-smi once and return its parsed JSON, or None with the reason recorded."""
    binary = rocm_smi_binary()
    if shutil.which(binary) is None:
        state["last_error"] = f"{binary} is not on PATH"
        return None
    cmd = [binary, "--showpower", "--showmemuse", "--json"]
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
    except (OSError, subprocess.TimeoutExpired) as exc:
        state["last_error"] = f"running {binary} failed: {exc}"
        return None
    if out.returncode != 0:
        state["last_error"] = f"{binary} exited {out.returncode}: {out.stderr.strip()[:200]}"
        return None
    try:
        return json.loads(out.stdout)
    except json.JSONDecodeError as exc:
        state["last_error"] = f"{binary} did not print JSON: {exc}"
        return None


def to_number(value):
    """rocm-smi prints numbers as strings, sometimes with a unit stuck on the end."""
    if isinstance(value, (int, float)):
        return float(value)
    match = re.search(r"-?\d+(?:\.\d+)?", str(value))
    return float(match.group(0)) if match else None


def extract(card_fields):
    """Pull the three numbers out of one card's fields, by pattern rather than by name."""
    found = {}
    for metric, patterns in PATTERNS.items():
        for pattern in patterns:
            for key, value in card_fields.items():
                if re.search(pattern, key, re.IGNORECASE):
                    number = to_number(value)
                    if number is not None:
                        found[metric] = (key, number)
                        break
            if metric in found:
                break
    return found


def collect():
    """Return a list of (card, metric, source key, value) rows, one set per card."""
    data = read_rocm_smi()
    if data is None:
        state["errors"] += 1
        return []
    rows = []
    for card, fields in data.items():
        if not isinstance(fields, dict):
            continue
        for metric, (key, value) in extract(fields).items():
            rows.append((card, metric, key, value))
    if not rows:
        state["errors"] += 1
        state["last_error"] = ("rocm-smi answered but none of its keys matched the patterns "
                               "this script looks for; run with --once to see the keys")
    return rows


def render(rows):
    """Prometheus text exposition format. No client library, so it is written out here."""
    state["scrapes"] += 1
    lines = [
        "# HELP local_llm_accelerator_power_watts Accelerator power draw reported by rocm-smi.",
        "# TYPE local_llm_accelerator_power_watts gauge",
        "# HELP local_llm_accelerator_memory_used_bytes Accelerator memory in use.",
        "# TYPE local_llm_accelerator_memory_used_bytes gauge",
        "# HELP local_llm_accelerator_memory_total_bytes Accelerator memory fitted.",
        "# TYPE local_llm_accelerator_memory_total_bytes gauge",
        "# HELP local_llm_exporter_scrape_errors_total Failed collections since start.",
        "# TYPE local_llm_exporter_scrape_errors_total counter",
        "# HELP local_llm_exporter_scrapes_total Collections served since start.",
        "# TYPE local_llm_exporter_scrapes_total counter",
    ]
    names = {
        "power_watts": "local_llm_accelerator_power_watts",
        "memory_used_bytes": "local_llm_accelerator_memory_used_bytes",
        "memory_total_bytes": "local_llm_accelerator_memory_total_bytes",
    }
    for card, metric, _key, value in rows:
        lines.append(f'{names[metric]}{{accelerator="{ACCELERATOR}",device="{card}"}} {value}')
    lines.append(f"local_llm_exporter_scrape_errors_total {state['errors']}")
    lines.append(f"local_llm_exporter_scrapes_total {state['scrapes']}")
    return "\n".join(lines) + "\n"


class Handler(BaseHTTPRequestHandler):
    """Two paths: /metrics for Prometheus, anything else for a human who guessed."""

    def do_GET(self):  # noqa: N802 - the name is fixed by BaseHTTPRequestHandler
        if self.path.split("?")[0] != "/metrics":
            self.send_response(404)
            self.send_header("Content-Type", "text/plain; charset=utf-8")
            self.end_headers()
            self.wfile.write(b"Nothing here. The metrics are at /metrics.\n")
            return
        body = render(collect()).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        """Quiet by default: a scrape every fifteen seconds is not news."""
        if os.environ.get("EXPORTER_ACCESS_LOG"):
            sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))


def print_once():
    """What the reader should run first: show the raw keys and what was matched."""
    data = read_rocm_smi()
    if data is None:
        print(f"rocm-smi could not be read: {state['last_error']}", file=sys.stderr)
        return 1
    print("Keys rocm-smi returned, per card:")
    for card, fields in data.items():
        if not isinstance(fields, dict):
            continue
        print(f"  {card}")
        for key, value in fields.items():
            print(f"    {key} = {value}")
    print()
    rows = collect()
    if not rows:
        print("No key matched the patterns this script looks for. Edit PATTERNS at the top",
              file=sys.stderr)
        print("of this file to match the key names your ROCm version prints.", file=sys.stderr)
        return 1
    print("Matched:")
    for card, metric, key, value in rows:
        print(f"  {card}  {metric:<20} from {key!r} = {value}")
    print()
    print("What Prometheus would receive:")
    print(render(rows))
    return 0


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--port", type=int, default=9401, help="port to serve /metrics on")
    parser.add_argument("--host", default="127.0.0.1",
                        help="address to bind; keep the default unless you know why not")
    parser.add_argument("--once", action="store_true",
                        help="print what rocm-smi says and what was matched, then exit")
    args = parser.parse_args()

    if args.once:
        sys.exit(print_once())

    server = ThreadingHTTPServer((args.host, args.port), Handler)
    print(f"rocm-exporter listening on http://{args.host}:{args.port}/metrics")
    print("Stop it with Ctrl-C. Run with --once first if a panel stays empty.")
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nstopped")
    finally:
        server.server_close()


if __name__ == "__main__":
    main()

Download rocm-exporter.py215 lines

RunnableTrack X · Ryzen AI Max+

see what the tool says before serving it

python3 rocm-exporter.py --once

Read the key list it prints. If the matched section is empty, edit the patterns at the top of the file to match the names your ROCm version uses, and run it again. Then serve:

RunnableTrack X · Ryzen AI Max+

serve the two numbers Prometheus needs

python3 rocm-exporter.py --port 9401

Leave it running in its own terminal for the rest of the lab.

Track M — Apple silicon

RunnableTrack M · Apple silicon

mac-exporter.py

#!/usr/bin/env python3
"""Prometheus exporter for Apple silicon, built on powermetrics and vm_stat.

Purpose: publish power draw and unified-memory use for Track M in the shape Prometheus
    scrapes. Apple ships no Prometheus exporter, and the tools that do report power on a
    Mac are command-line ones, so this runs them and republishes two numbers at /metrics.
Platform: mac (Apple silicon). Not for the other tracks: Tracks S and N use NVIDIA's DCGM
    exporter, Track X uses rocm-exporter.py in this directory.
Minimum memory: 8 GB, which is the service being watched; this process needs almost none.
Assumes: macOS, with `powermetrics`, `vm_stat` and `sysctl` in their usual places.
    powermetrics needs root: run this exporter with sudo, or run it without and accept
    that the power gauge will be absent while the memory gauges still work. Apple
    publishes the powermetrics manual only as a manual page on the machine itself, so
    before trusting the sampler names below, read `man powermetrics` on your own Mac and
    confirm them. Nothing here has been executed on Apple hardware by this course.

    Unified memory has no separate video memory to report, so "accelerator memory" here is
    the machine's memory: total from `sysctl hw.memsize`, and in-use computed from vm_stat
    as active plus wired plus compressed pages. That is a choice, it is stated on the
    dashboard, and it is the number that decides whether a model fits.

Usage: sudo python3 mac-exporter.py --once
       sudo python3 mac-exporter.py --port 9402
       python3 mac-exporter.py --port 9402 --no-power     (memory only, no root needed)
"""
import argparse
import os
import re
import shutil
import subprocess
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

ACCELERATOR = "apple"

# powermetrics prints power as lines such as "CPU Power: 1234 mW". The combined figure is
# preferred where it appears; otherwise the parts are added up. Confirm the sampler names
# against `man powermetrics` before relying on this.
POWER_LINE = re.compile(r"^\s*(combined|cpu|gpu|ane|package)\s+power:\s*([\d.]+)\s*mw",
                        re.IGNORECASE | re.MULTILINE)
VM_STAT_LINE = re.compile(r"^(.*?):\s+(\d+)\.?$", re.MULTILINE)
PAGE_SIZE_HINT = re.compile(r"page size of (\d+) bytes")

state = {"scrapes": 0, "errors": 0, "last_error": ""}


def run(cmd, timeout=20):
    if shutil.which(cmd[0]) is None:
        state["last_error"] = f"{cmd[0]} is not on PATH"
        return None
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
    except (OSError, subprocess.TimeoutExpired) as exc:
        state["last_error"] = f"running {cmd[0]} failed: {exc}"
        return None
    if out.returncode != 0:
        state["last_error"] = f"{cmd[0]} exited {out.returncode}: {out.stderr.strip()[:200]}"
        return None
    return out.stdout


def read_power(sample_ms):
    """One powermetrics sample, in watts, or None with the reason recorded."""
    text = run(["powermetrics", "--samplers", "cpu_power,gpu_power",
                "-n", "1", "-i", str(sample_ms)], timeout=sample_ms / 1000 + 20)
    if text is None:
        return None
    parts = {name.lower(): float(value) for name, value in POWER_LINE.findall(text)}
    if not parts:
        state["last_error"] = ("powermetrics ran but printed no power line this script "
                               "recognises; run with --once and read its output")
        return None
    for combined in ("combined", "package"):
        if combined in parts:
            return parts[combined] / 1000.0
    total = sum(parts.get(k, 0.0) for k in ("cpu", "gpu", "ane"))
    return total / 1000.0 if total else None


def read_memory():
    """Total and in-use memory in bytes, from sysctl and vm_stat."""
    total_text = run(["sysctl", "-n", "hw.memsize"])
    vm_text = run(["vm_stat"])
    if total_text is None or vm_text is None:
        return None, None
    try:
        total = int(total_text.strip())
    except ValueError:
        state["last_error"] = "sysctl hw.memsize did not return a number"
        return None, None

    hint = PAGE_SIZE_HINT.search(vm_text)
    page = int(hint.group(1)) if hint else 4096
    counts = {name.strip().lower(): int(value)
              for name, value in VM_STAT_LINE.findall(vm_text)}

    def pages(*keys):
        for key in keys:
            for name, value in counts.items():
                if name.startswith(key):
                    return value
        return 0

    used_pages = (pages("pages active")
                  + pages("pages wired down", "pages wired")
                  + pages("pages occupied by compressor", "pages stored in compressor"))
    return total, used_pages * page


def collect(sample_ms, want_power):
    rows = []
    power = read_power(sample_ms) if want_power else None
    if want_power and power is None:
        state["errors"] += 1
    elif power is not None:
        rows.append(("local_llm_accelerator_power_watts", power))

    total, used = read_memory()
    if total is None:
        state["errors"] += 1
    else:
        rows.append(("local_llm_accelerator_memory_total_bytes", float(total)))
        rows.append(("local_llm_accelerator_memory_used_bytes", float(used)))
    return rows


def render(rows):
    state["scrapes"] += 1
    lines = [
        "# HELP local_llm_accelerator_power_watts Package power draw reported by powermetrics.",
        "# TYPE local_llm_accelerator_power_watts gauge",
        "# HELP local_llm_accelerator_memory_used_bytes Unified memory active, wired and compressed.",
        "# TYPE local_llm_accelerator_memory_used_bytes gauge",
        "# HELP local_llm_accelerator_memory_total_bytes Unified memory fitted.",
        "# TYPE local_llm_accelerator_memory_total_bytes gauge",
        "# HELP local_llm_exporter_scrape_errors_total Failed collections since start.",
        "# TYPE local_llm_exporter_scrape_errors_total counter",
        "# HELP local_llm_exporter_scrapes_total Collections served since start.",
        "# TYPE local_llm_exporter_scrapes_total counter",
    ]
    for name, value in rows:
        lines.append(f'{name}{{accelerator="{ACCELERATOR}"}} {value}')
    lines.append(f"local_llm_exporter_scrape_errors_total {state['errors']}")
    lines.append(f"local_llm_exporter_scrapes_total {state['scrapes']}")
    return "\n".join(lines) + "\n"


def make_handler(sample_ms, want_power):
    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):  # noqa: N802 - the name is fixed by BaseHTTPRequestHandler
            if self.path.split("?")[0] != "/metrics":
                self.send_response(404)
                self.send_header("Content-Type", "text/plain; charset=utf-8")
                self.end_headers()
                self.wfile.write(b"Nothing here. The metrics are at /metrics.\n")
                return
            body = render(collect(sample_ms, want_power)).encode("utf-8")
            self.send_response(200)
            self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def log_message(self, fmt, *args):
            if os.environ.get("EXPORTER_ACCESS_LOG"):
                sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))

    return Handler


def print_once(sample_ms, want_power):
    if want_power and os.geteuid() != 0:
        print("powermetrics normally needs root. Re-run with sudo, or pass --no-power.",
              file=sys.stderr)
    rows = collect(sample_ms, want_power)
    if state["last_error"]:
        print(f"note: {state['last_error']}", file=sys.stderr)
    if not rows:
        return 1
    print("What Prometheus would receive:")
    print(render(rows))
    return 0


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--port", type=int, default=9402, help="port to serve /metrics on")
    parser.add_argument("--host", default="127.0.0.1",
                        help="address to bind; keep the default unless you know why not")
    parser.add_argument("--sample-ms", type=int, default=500,
                        help="how long each powermetrics sample takes, in milliseconds")
    parser.add_argument("--no-power", action="store_true",
                        help="skip powermetrics and publish memory only, without root")
    parser.add_argument("--once", action="store_true",
                        help="collect once, print what would be served, and exit")
    args = parser.parse_args()

    want_power = not args.no_power
    if args.once:
        sys.exit(print_once(args.sample_ms, want_power))

    server = ThreadingHTTPServer((args.host, args.port),
                                 make_handler(args.sample_ms, want_power))
    print(f"mac-exporter listening on http://{args.host}:{args.port}/metrics")
    if want_power:
        print("Each scrape runs powermetrics, which needs root and takes a moment. Keep the")
        print("Prometheus scrape interval at fifteen seconds or longer.")
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nstopped")
    finally:
        server.server_close()


if __name__ == "__main__":
    main()

Download mac-exporter.py217 lines

RunnableTrack M · Apple silicon

check it once, with the privileges powermetrics needs

sudo python3 mac-exporter.py --once

If the power line is missing, read man powermetrics on your own machine and confirm the sampler names; the script says which line it was looking for. Then serve:

RunnableTrack M · Apple silicon

serve power and unified memory

sudo python3 mac-exporter.py --port 9402

Prefer not to run a scraper as root? --no-power publishes memory only, and every panel except the power one still works. Write down which you chose, because it changes what the cost lesson can use.

Track N — NVIDIA desktop or laptop

Already running, from the nvidia profile. Confirm the field names as on Track S:

RunnableTrack N · NVIDIA GPU

what the DCGM exporter actually exports

curl -s http://127.0.0.1:9400/metrics | grep -E '^# HELP DCGM_FI_DEV_(FB|POWER|GPU)'

On a discrete card, framebuffer used against the card’s capacity is the most predictive single number you will have, so it is worth confirming that both halves of that ratio are present.

6. Open the dashboard and understand what it is showing

Open Grafana on its published port and sign in as admin with the password from .env. The dashboard is already there: the provider file told Grafana to load JSON from a directory, so there is nothing to import.

Work down the panels and, for each, say out loud which of the six questions it answers and where its number comes from. Two of them will probably be empty, because no machine runs every engine, and an empty panel whose reason you understand is fine. An empty panel whose reason you do not understand is the thing to fix now: read your own /metrics output and correct the query in grafana-dashboard.json, not in the browser, because the provider reloads the file and will overwrite a change made in the interface.

7. Put load through it and watch

A dashboard is meaningless at rest. Use the load generator from Part 9 to give it something to draw, or send a handful of requests by hand.

RunnableAll tracks

something to look at

for i in 1 2 3 4 5; do
    curl -s http://127.0.0.1:4000/v1/chat/completions \
        -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
        -H "Content-Type: application/json" \
        -d '{"model":"local/chat",
             "messages":[{"role":"user","content":"Write two sentences about tides."}],
             "max_tokens":128}' > /dev/null &
done
wait

Watch three things while it runs. Requests running rises and comes back down. Accelerator memory steps up when the model loads and stays up. And the tokens-by-model panel starts having something in it, labelled with the alias rather than the file name, which is the gateway doing the accounting the engines cannot.

Then ask for a different model and watch the swap happen on the memory graph. That step change is what a model load looks like from the outside, and recognising it is most of the challenge on the next page.

8. Make an alert fire

Two rules are loaded. Prove one of them works, because an alert nobody has ever seen fire is an alert nobody knows is broken.

The cheapest proof is the gateway one. Stop the gateway, wait past the for duration, and watch the alert move from inactive to pending to firing on Prometheus’s own alerts page.

Run the first and last of these from your gateway directory, where its compose.yaml lives, and the middle one from anywhere.

RunnableAll tracks

break it deliberately, then put it back

docker compose -f compose.yaml stop gateway

curl -s http://127.0.0.1:9090/api/v1/rules | head -c 400

docker compose -f compose.yaml start gateway

=Expected result

The rule passes through three states. Inactive while the expression matches nothing; pending once it matches but the for duration has not elapsed; firing after that. Watching the pending state is worth doing once, because it is what stops a two-minute restart from waking you up, and understanding it is what lets you choose sensible durations for your own rules.

Prometheus evaluates these rules and shows them. Sending a notification anywhere needs Alertmanager, which the documentation describes as the separate component handling silencing, inhibition, aggregation and delivery. Adding it is a reasonable next step and it is not part of this lab: a rule you can see on a page you look at is worth more than a notification pipeline that is half-finished.

9. Record the baseline

This is the deliverable. Fill in the sheet below from what the dashboard shows with the service idle and then under the load from task seven.

Pending validationBaseline for this machine — your own recording sheet

State	Accelerator memory used	Host memory available	Power (W)	Requests running	Tokens/s
Idle, nothing loaded	—	—	—	—	—
Idle, model resident	—	—	—	—	—
Under the load from task 7	—	—	—	—	—

your machine: track, chip, memory, and how the power figure was taken, your operating system and version · the engine and the slot count it was started with the engine version, from its own --version output · the model behind local/chat, its quantisation · 16,384 tokens of context · the date you measured it

Empty on purpose. Three rows, taken within the same hour on the same machine. The middle row is the one people forget and the one that matters most: a resident model with nobody using it is the state your machine spends most of its life in.

RunnableAll tracks

record it in the notebook

bash check-monitoring.sh

Validate a metric from request to dashboard

Start with one engine target and one known request. Check the raw metrics endpoint, the monitoring target status and the dashboard series in that order. If a graph is empty, determine whether the engine exported the metric, the collector scraped it and the dashboard queried the correct labels. Do not install every platform exporter on every host.

Give each measurement a definition in your notebook: unit, interval and whether queueing or failed requests are included. Generate the controlled load from the lesson and confirm the relevant counter or latency distribution changes. Compare accelerator metrics with the device actually serving the model; a host with several devices can expose unrelated utilisation.

Trigger one alert deliberately in the lab and record firing, notification and recovery. Then verify the dashboard after restarting the monitoring stack so an unsaved UI edit cannot conceal missing provisioning. Keep redacted scrape configuration, dashboard JSON, alert rule and the load evidence. Remove targets that do not exist rather than leaving permanent failures in the view. The lab is complete when a graph and alert can be traced to a known event and an operator can explain the next diagnostic action, not merely when the dashboard renders.

Validation

RunnableAll tracks

the whole stack, checked

bash check-monitoring.sh

Output — what you should see

==> checking http://127.0.0.1:9090
  PASS  Prometheus is healthy
  PASS  alert rule GatewayUnreachable is loaded
  PASS  alert rule RequestsQueueing is loaded
  PASS  alert rule AcceleratorMemoryHigh is loaded
  PASS  gateway target is up
  PASS  at least one engine target is up
  PASS  a GPU or accelerator exporter is publishing power
  PASS  accelerator memory is being recorded
  PASS  node exporter target is up
==> checking http://127.0.0.1:3000
  PASS  Grafana is healthy
  PASS  the provisioned dashboard is present

    11 passed, 0 failed, 0 skipped

The dashboard check needs a Grafana login, so the script uses the admin password from .env and never prints it. Without that variable the check is reported as skipped rather than failed, because a check that could not be made is not the same as a check that failed, and conflating the two is how a suite of checks stops being believed.

You are done when the script passes and when all of the following are also true.

• Every target on Prometheus’s targets page is up, because you deleted the jobs you will never run rather than tolerating them.
• Every panel on the dashboard either has data or has a reason you can state.
• You have watched one alert move from pending to firing and back to inactive.
• The baseline sheet has three complete rows.
• The lab notebook has an entry naming the image tags you chose and, on Tracks X and M, which exporter mode you ran.

Expected outcome

One page that answers the six questions, and a machine that will tell you when two of the answers become alarming.

The result you should notice in the following weeks is a change in how you investigate. The question “is it slow?” becomes “slower than the baseline, or the same as always”. The question “did something break overnight?” has an answer on a graph rather than a shrug. And the memory panel acquires a shape you recognise, which is the single most useful thing this lab gives you, because the challenge on the next page is entirely about recognising when that shape is wrong.

Troubleshooting

A target is up and every panel from it is empty. The endpoint answers and publishes nothing. In order of likelihood: llama-server without --metrics, SGLang without --enable-metrics, LiteLLM without the Prometheus callback. Ask the endpoint directly with curl before touching Prometheus.

Prometheus cannot reach anything on the host. The container reaches host services through host.docker.internal, which the compose file maps with an extra_hosts entry. If your Docker version does not map it, either run Prometheus with host networking or put the host’s address in the targets, and note in your changelog that you did.

Grafana starts and the dashboard is not there. Look at the container log: a provisioning error names the file and the line. The usual causes are a mount path that does not match the provider’s options.path and a YAML file with a tab in it.

The dashboard is there and every panel says “No data”. The data source uid in the dashboard JSON has to match the uid in the data source file. Both are set to the same fixed string in the shipped files; if you edited one, edit the other.

The DCGM exporter starts and publishes nothing. On Track N inside WSL2 this is usually the GPU not being visible to the container rather than a fault in the exporter. Check with nvidia-smi inside a container first, and treat that as the actual diagnosis.

rocm-exporter.py --once matches nothing. Your ROCm version spells its JSON keys differently. The script prints every key it received; edit the patterns at the top of the file to match, and record what you changed, because you will do it again after the next upgrade.

mac-exporter.py reports no power. Either it was not run with root, or the sampler output does not contain the line the script looks for. Read man powermetrics on your own machine, run the sampler by hand once, and see what it prints.

An alert never fires even when the condition is obviously true. The expression names a metric nothing on your machine publishes. This is why the check script confirms the rules are loaded and why task five had you look at the exporter’s own help output: a rule that names an absent metric is silent in exactly the same way as a rule with nothing wrong.

Cleanup

The stack is meant to stay. It costs little and it is the reason the next page is possible.

RunnableAll tracks

stop it, keeping the history

docker compose -f compose-monitoring.yaml down

⚠Warning

docker compose -f compose-monitoring.yaml down -v also removes the named volumes, which here means every metric Prometheus has collected and everything Grafana stores. The dashboard comes back from its file; the history does not. Use plain down unless you have decided to start again.

Stop the native exporter on Tracks X and M with Ctrl-C in its terminal. Nothing else on the machine was changed: the only edits outside this directory were --metrics on the engine and the Prometheus callback in the router, and both of those are worth keeping.

What you learned

• A dashboard is built from questions, not from metrics. Six questions, one panel each, and anything that answers none of them goes on a second page or nowhere.
• A target being up says nothing about it publishing. The characteristic failure of this whole area is a green target and an empty panel, and the cure is to ask the endpoint directly before configuring anything downstream.
• Provisioning turns a dashboard into code. A file you can commit, rebuild and diff beats an arrangement of panels that exists only in one browser’s memory.
• A permanently red target is worse than no target. Delete the jobs you will never run, so that red keeps meaning something.
• An alert has three states and the middle one is the useful one. The for duration is what separates an alert worth having from one that fires on every restart.
• Prometheus evaluates, Alertmanager notifies. You can have the first without the second, and for a household service that is often the right amount of machinery.
• The baseline is the deliverable. A graph tells you what is happening; only a remembered baseline tells you whether it is unusual.

Record in the notebook: the image tags you chose, the exporter mode on Tracks X and M, which jobs you deleted from the scrape configuration and why, which panels were empty and for what reason, the three rows of the baseline sheet, and the date. Then open the dashboard once a day for a week without a reason to, because the shape of a healthy week is the thing you are actually learning.

Check your understanding

Question 1. A scrape target shows as up, but every panel fed by it is empty. What is happening?

• Prometheus is scraping too slowly
• The port answers, so the target is healthy, but the component publishes no metrics: llama-server without --metrics, SGLang without --enable-metrics, or LiteLLM without the Prometheus callback
• Grafana is querying the wrong time range
• The alert rules are consuming the data

Show the answer and why

Answer: The port answers, so the target is healthy, but the component publishes no metrics: llama-server without --metrics, SGLang without --enable-metrics, or LiteLLM without the Prometheus callback

Target health is a property of the HTTP connection, not of the payload. This is why the lab has you curl each endpoint before configuring Prometheus at all: it separates two failures that look identical from the targets page.

Question 2. Why does this lab tell you to delete scrape jobs for engines you will never run, rather than leaving them red?

• Each failing target uses significant resources
• A permanently red target teaches you to ignore red targets, which gives up the cheapest signal in the whole system
• Prometheus refuses to evaluate rules while any target is down
• Grafana cannot render a dashboard with failing targets

Show the answer and why

Answer: A permanently red target teaches you to ignore red targets, which gives up the cheapest signal in the whole system

The value of the targets page is that red means something. One tolerated permanent failure is enough to break that, and the up metric is the basis of the most useful alert you have.

Question 3. You edit a panel in the Grafana interface and your change disappears later. Why?

• Grafana does not save panel edits
• The dashboard is provisioned from a file and the provider rescans it on an interval, so the file overwrites interface edits; the change belongs in the JSON
• The data source is read-only
• The dashboard JSON is mounted read-only, which prevents all edits

Show the answer and why

Answer: The dashboard is provisioned from a file and the provider rescans it on an interval, so the file overwrites interface edits; the change belongs in the JSON

That is the trade provisioning makes and it is the right one: the dashboard is code, so it can be rebuilt, committed and diffed. Experiment in the browser freely, then put anything worth keeping into the file.

Question 4. Which of these are legitimate reasons a panel on this dashboard is empty? Select all that apply.

• Your machine runs llama.cpp, and the panel queries a vLLM-only metric
• llama.cpp publishes no key-value cache occupancy figure
• Your track has no DCGM exporter because it has no NVIDIA GPU
• The panel is empty and you have no idea why

Show the answer and why

Answer: Your machine runs llama.cpp, and the panel queries a vLLM-only metric, llama.cpp publishes no key-value cache occupancy figure, Your track has no DCGM exporter because it has no NVIDIA GPU

The first three are the dashboard correctly showing you that a number does not exist on this machine. The fourth is the one to chase, and the way to chase it is to read your own /metrics output rather than to adjust the query until something appears.

Question 5. An alert rule is loaded, the condition is plainly true, and it never fires. What is the most likely explanation?

• The for duration is too short
• The expression names a metric nothing on this machine publishes, so it matches no series and stays inactive
• Alertmanager is not installed
• Prometheus only evaluates rules on restart

Show the answer and why

Answer: The expression names a metric nothing on this machine publishes, so it matches no series and stays inactive

A rule over an absent metric is silent in exactly the way a rule with nothing wrong is silent, which is why the memory alert in this part names the fields it depends on and why the lab has you confirm they appear in your own exporter output.

Check my answersClear