{
  "uid": "local-llm-service",
  "title": "Local LLM service",
  "description": "Purpose: the four things worth watching on a local model service - throughput, latency, accelerator memory and power - on one screen, with one panel per question rather than one panel per metric. Platform: all. Minimum memory: 8 GB. Assumes: the scrape jobs in prometheus-config.yaml and the data source uid 'prometheus' from grafana-datasource.yaml. Each panel queries every engine it might find and shows whichever answers, so the same file works on all four tracks. A panel that stays empty means the metric is not published under that name on your machine: read your own /metrics output and correct the query here, in the file, not in the browser.",
  "tags": ["local-llm", "part-23"],
  "timezone": "browser",
  "editable": true,
  "schemaVersion": 39,
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-6h", "to": "now" },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Generation throughput",
      "description": "Output tokens per second, as each engine reports it. llama.cpp publishes an average rate directly; vLLM and SGLang publish counters, so this takes their rate over five minutes. Counter names differ by a _total suffix between client library versions, which is why both spellings are asked for.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "llamacpp:predicted_tokens_seconds",
          "legendFormat": "llama.cpp {{instance}}"
        },
        {
          "refId": "B",
          "expr": "rate(vllm:generation_tokens[5m]) or rate(vllm:generation_tokens_total[5m])",
          "legendFormat": "vLLM {{model_name}}"
        },
        {
          "refId": "C",
          "expr": "sglang:gen_throughput",
          "legendFormat": "SGLang {{instance}}"
        }
      ]
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "Requests: running and waiting",
      "description": "The two numbers that separate 'busy' from 'over capacity'. Running requests rising is use. Waiting requests rising is the warning, and it is what the RequestsQueueing alert watches.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "llamacpp:requests_processing",
          "legendFormat": "llama.cpp running"
        },
        {
          "refId": "B",
          "expr": "llamacpp:requests_deferred",
          "legendFormat": "llama.cpp waiting"
        },
        {
          "refId": "C",
          "expr": "vllm:num_requests_running",
          "legendFormat": "vLLM running"
        },
        {
          "refId": "D",
          "expr": "vllm:num_requests_waiting",
          "legendFormat": "vLLM waiting"
        },
        {
          "refId": "E",
          "expr": "sglang:num_running_reqs",
          "legendFormat": "SGLang running"
        },
        {
          "refId": "F",
          "expr": "sglang:num_queue_reqs",
          "legendFormat": "SGLang waiting"
        }
      ]
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Time to first token, 95th percentile",
      "description": "How long a caller waits before anything appears. Only vLLM and SGLang publish this as a histogram; on llama.cpp the closest available answer is the router's own end-to-end latency, which is the third query here and which includes the model load on a cold start.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
      "fieldConfig": { "defaults": { "unit": "s", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))",
          "legendFormat": "vLLM"
        },
        {
          "refId": "B",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(sglang:time_to_first_token_seconds_bucket[5m])))",
          "legendFormat": "SGLang"
        },
        {
          "refId": "C",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(litellm_request_total_latency_metric_bucket[5m])))",
          "legendFormat": "gateway, end to end"
        }
      ]
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "Key-value cache occupancy",
      "description": "The fraction of the cache that is holding conversation. This is the number that creeps up over a week as prompts get longer, and it is the one the three-in-the-morning failure is written on. Published by vLLM directly; on llama.cpp the high-water mark of context observed is the nearest equivalent and is shown beside it.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "vllm:kv_cache_usage_perc",
          "legendFormat": "vLLM KV cache"
        },
        {
          "refId": "B",
          "expr": "sglang:token_usage",
          "legendFormat": "SGLang token usage"
        },
        {
          "refId": "C",
          "expr": "llamacpp:n_tokens_max",
          "legendFormat": "llama.cpp context high-water mark"
        }
      ]
    },
    {
      "id": 5,
      "type": "timeseries",
      "title": "Accelerator memory in use",
      "description": "Tracks S and N read this from the DCGM exporter, in mebibytes, so it is converted here. Tracks X and M read it from the exporter this part ships, in bytes. One panel, two sources, whichever your machine has.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "DCGM_FI_DEV_FB_USED * 1024 * 1024",
          "legendFormat": "NVIDIA GPU {{gpu}}"
        },
        {
          "refId": "B",
          "expr": "local_llm_accelerator_memory_used_bytes",
          "legendFormat": "{{accelerator}} used"
        },
        {
          "refId": "C",
          "expr": "local_llm_accelerator_memory_total_bytes",
          "legendFormat": "{{accelerator}} total"
        }
      ]
    },
    {
      "id": 6,
      "type": "timeseries",
      "title": "Power draw",
      "description": "Watts, from the DCGM exporter on the NVIDIA tracks and from the shipped exporter on the other two. This is the panel the cost model reads: take the average over a sustained generation run and put it in cost-inputs.json as load_watts, and the average over an idle hour as idle_watts.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "DCGM_FI_DEV_POWER_USAGE",
          "legendFormat": "NVIDIA GPU {{gpu}}"
        },
        {
          "refId": "B",
          "expr": "local_llm_accelerator_power_watts",
          "legendFormat": "{{accelerator}}"
        }
      ]
    },
    {
      "id": 7,
      "type": "timeseries",
      "title": "Tokens by model, through the gateway",
      "description": "Who is using what. LiteLLM labels its token counters by model and by hashed key, so this is where per-application accounting lives. If the panel is empty, either callbacks: [\"prometheus\"] is missing from litellm-config.yaml or your client library spells the counter with a _total suffix; both spellings are asked for here.",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
      "fieldConfig": { "defaults": { "unit": "none", "min": 0 }, "overrides": [] },
      "targets": [
        {
          "refId": "A",
          "expr": "sum by (model) (rate(litellm_total_tokens_metric[5m])) or sum by (model) (rate(litellm_total_tokens_metric_total[5m]))",
          "legendFormat": "{{model}}"
        }
      ]
    }
  ]
}