Challenge: The 3 a.m. Out-of-Memory

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track reproductions, the exact log wording each engine produced and the versions they were seen with belong here once the validation pass has run this page on real machines.

Objective

The service worked all week. Nobody deployed anything, nobody changed a configuration file, and at some point in the small hours it stopped answering. In the morning it starts again first time and looks perfectly healthy.

By the end of this page you will have a procedure that answers that in about ten minutes, and you will have run it on a fault you introduced deliberately, so that you recognise the shape when the fault is not yours. The deliverable is not a fixed machine. It is a written diagnosis with four parts: the evidence you collected, the fault it points at, the change you made, and the guard rail you added so that the same fault announces itself next time instead of waiting until three in the morning.

⌂Single-machine path

Single machine on every track. This needs the monitoring stack from the previous page, with at least a few hours of history in it, and nothing else.

The procedure

A service died overnight: what to do, in order

1. 1Do not restart it yetIf it is still down, the state it is in is evidence. Read the accelerator memory and the process list first; a restart destroys the only direct observation you will get.
2. 2Establish when, from the recordThe gateway target going down gives you a timestamp. Everything else is read relative to that moment rather than to now.
3. 3Read memory backwards from the failureDid it climb steadily, step up and stay, or spike? Those are three different faults and the shape distinguishes them before you read a single log line.
4. 4Ask whether the load changedRequests running and waiting over the same window. If they did not change, the fault is on the memory side and not the traffic side.
5. 5Then read the logNow you know what you are looking for and roughly when. The log confirms or refutes the theory the graphs suggested; it is a poor place to start.
6. 6Name the fault from the short listFour candidates. If the evidence fits more than one, collect more evidence rather than choosing the interesting one.
7. 7Change one thing, then reproduce the loadThe proof of a fix is the same load producing a different graph, not the absence of a failure for a day.
8. 8Add the guard rail and write it downAn alert, a limit, or a ttl. A fixed fault with no guard rail is a fault you will diagnose again from scratch.

The order is not arbitrary. Each step is cheap, each eliminates a large fraction of the possibilities, and the two that people naturally do first, restarting and reading the log, are deliberately placed fourth and fifth because doing them first destroys evidence and produces theories in the wrong order.

Four faults, worked

Almost every overnight memory failure on a local model service is one of these. Each has a distinctive shape on the memory graph, which is why the previous page insisted on a baseline.

Context creep: the same traffic, longer

What happened. Nothing changed except the conversations. People kept talking to the assistant, so their conversations got longer, so each request carried more context, so the key-value cache held more. Memory that comfortably fitted four short conversations does not fit the same four after an hour, because cache use scales with concurrency multiplied by how far each sequence has got.

What it looks like. A memory line that climbs across days rather than hours, with a sawtooth on top from individual requests. On vLLM, vllm:kv_cache_usage_perc rises towards one. On SGLang, sglang:token_usage does the same. On llama.cpp there is no cache occupancy metric, so the signal is llamacpp:n_tokens_max, the “high watermark of the context size observed”, rising over the same period. Requests running is unchanged, which is the distinguishing evidence: same load, more memory.

The fix. Size the context for the conversations you actually have rather than for the model’s maximum. --ctx-size on llama-server is a memory decision, not a capability decision, and quantising the cache with --cache-type-k and --cache-type-v buys back a substantial fraction of it, as Part 6 showed.

The guard rail. Three, in increasing order of effort. The AcceleratorMemoryHigh alert from the lab, which catches the slow climb days before it matters. A ttl on the model in llama-swap.yaml, documented as unloading “after ttl seconds”, which resets the cache whenever the service is genuinely idle. And context_window_fallbacks at the router, which LiteLLM documents as routing elsewhere specifically on a context-window error, so that a request too long for this model goes to one with room instead of failing.

Concurrency beyond the plan

What happened. More things called at once than the engine was started for. Often that is a new user; more often on a home service it is your own script with a retry loop, or an agent from Part 26 that found a way to call itself, or a cron job that started overlapping with itself because it now takes longer than its interval.

What it looks like. Requests waiting above zero for a sustained period, which is exactly what the RequestsQueueing alert watches. Memory rises with the number of active sequences rather than over days. If the engine allocates its cache per slot up front, memory may not move at all and the symptom is latency and queueing rather than a memory failure, which is the friendlier version of this fault.

The fix. Decide the slot count deliberately and set --parallel to it, having checked the memory at that concurrency and that context length rather than at rest.

The guard rail. rpm_limit and tpm_limit on the key that ran away, which turns the next occurrence into a refusal with a name attached. limit_req at the edge as the blunt backstop that costs the machine nothing. And the queueing alert, which fires hours before anything fails.

Fragmentation, or a plain leak

What happened. Memory was allocated and not fully returned, either because something genuinely leaked or because the allocator could not reuse the space it got back. Long-running processes with varying allocation sizes are exactly the conditions for this, and a model server serving prompts of every length for a week is that.

What it looks like. The distinguishing shape: memory that ratchets. Each burst raises the floor a little, and the floor never returns to where it started even when the service is completely idle. That is different from context creep, where the rise tracks the cache metric, and different from concurrency, where it tracks the request count.

The fix. There is rarely a configuration that fixes it, and the honest answer is often a periodic restart. That is not a defeat: a service restarted at a chosen time when nobody is using it is strictly better than one restarted at three in the morning by an allocation failure.

The guard rail. A ttl that forces an unload during genuinely idle periods gives the memory back without a restart at all. Beyond that, the memory alert with a long for duration, so it catches the ratchet rather than the bursts. And a note in the changelog, with the engine version, because a leak fixed upstream is worth knowing about at the next upgrade.

A model swap that never unloaded

What happened. A second model was loaded and the first was not released. Either the group configuration allows them to coexist and the machine cannot actually hold both, or an unload failed quietly, or a client is holding the old engine open so it never becomes idle.

What it looks like. A step change in memory that does not come back down, at a moment that corresponds to a different model being asked for. The decisive evidence is a disagreement: llama-swap’s /running says one model is loaded, and the accelerator says memory for two is resident. Those two disagreeing is the fault, and no other fault produces it.

The fix. Put the models that cannot coexist in one group with swap: true and exclusive: true, which the documentation describes as controlling “how members of this group swap among themselves” and “how this group affects other groups”. Part 9’s configuration does this and it is worth re-reading rather than assuming.

The guard rail. The memory alert catches the resident-but-forgotten model. /running compared against the memory panel is the ten-second manual check, and it is worth doing during any incident. POST /api/models/unload is the documented way to get the memory back immediately without stopping the service.

✕Common mistake

Restarting first, and then investigating. A restart clears the state that would have told you which of these four it was, and the service then works for another week, so the investigation never happens and the same night repeats. If the service is down and somebody is waiting, take one screenshot of the memory panel and one copy of the log before restarting. That is thirty seconds and it is the difference between a diagnosis and a recurring mystery.

Requirements

The monitoring stack from the previous page, with at least a few hours of history. The Part 9 gateway. Forty-five minutes, all of it attended. No downloads.

Track S — NVIDIA DGX Spark

Reproduce concurrency beyond the plan. It is the safest of the four to induce on a machine with a lot of memory, because the symptom is queueing rather than a failure, and it is the one this track will meet in practice when the household starts using it.

Start the engine with a deliberately small slot count, send more concurrent requests than that, and watch requests waiting rise on the dashboard.

Track X — AMD Ryzen AI Max+ 395

Reproduce context creep. Remember Part 5’s finding about the cap on GPU-visible memory: the accelerator’s share is smaller than the machine’s total, so the cache fills sooner than the machine’s memory figure suggests, and that gap is what makes this fault arrive earlier here than readers expect.

Send progressively longer prompts at a fixed concurrency and watch memory rise while the request count stays flat.

Track M — Apple siliconPartial

Deliberately filling unified memory affects the whole machine rather than one device, so this track reproduces the fault in its mildest form and stops as soon as the shape is visible.

Reproduce context creep, with care. Unified memory means the pressure lands on the whole machine, so use a model comfortably inside your tier, raise the context length rather than the model size, and stop as soon as the shape appears on the graph.

Do not run this on a machine doing anything else you care about, and be ready to stop the engine. Part 6’s challenge made the same warning about memory pressure on this track and it applies here for the same reason.

Track N — NVIDIA desktop or laptop

Reproduce a model swap that never unloaded, which is the most instructive of the four on a discrete card because framebuffer memory is a hard wall rather than a share of a pool.

Temporarily change the group configuration so that two models may coexist, ask for both, and watch memory step up twice without coming down. Change it back afterwards.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-23-operating-a-local-ai-service"
cd "$LAB_DIR"
pwd
test -f "diagnose-oom.py"

Expected result: pwd ends in part-23-operating-a-local-ai-service and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Look at a healthy week first

Open the dashboard with a range of several days. You are not looking for a problem; you are learning what normal looks like on this machine, so that abnormal is recognisable.

Three things to notice and write down. Where the memory floor sits when a model is resident and nobody is using it, which is the state the machine spends most of its life in. What the daily rhythm looks like, because most home services have one. And whether the floor is the same on the last day as on the first, which is the leak question answered in one glance.

2. Collect the evidence with one command

The script asks Prometheus for the window you name, reads the container log if you give it one, and writes a report arranged in the order of the procedure above. It changes nothing and starts nothing.

RunnableAll tracks

diagnose-oom.py

#!/usr/bin/env python3
"""Read the night back off the dashboards and say which memory fault it looks like.

Purpose: pull the hours around a failure out of Prometheus, pull the engine's own log out
    of the container runtime, and turn both into a written diagnosis: what the evidence
    shows, which of the known faults fits it, and which do not. It changes nothing and
    starts nothing; it only reads.
Platform: all (spark, strix, mac, nvidia). Pure Python, no dependencies.
Minimum memory: 8 GB, which is the service being diagnosed; this script needs almost none.
Assumes: the Prometheus from this part's lab, reachable and holding the window you care
    about, and optionally a container name whose log can be read with `docker logs`.
    A query whose metric your machine does not publish returns nothing, and the report
    says so rather than guessing. The verdict is a ranked shortlist, not a conclusion:
    it is where to look first, and the last section lists what would confirm each one.

Usage: python3 diagnose-oom.py --since 24h
       python3 diagnose-oom.py --since 12h --container local-gateway-swap-1
       python3 diagnose-oom.py --since 24h --output oom-report.md --labbook labbook.md
"""
import argparse
import json
import re
import shutil
import subprocess
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
from pathlib import Path

DURATION = re.compile(r"^(\d+)([smhd])$")
SECONDS = {"s": 1, "m": 60, "h": 3600, "d": 86400}

# Patterns that mean "the machine ran out of memory", in the words each stack uses. None of
# these is a guess: they are the strings the engines and the kernel print.
LOG_PATTERNS = [
    ("cuda-oom", re.compile(r"CUDA (?:error: )?out of memory|cudaErrorMemoryAllocation", re.I)),
    ("hip-oom", re.compile(r"hipErrorOutOfMemory|HIP out of memory", re.I)),
    ("metal-oom", re.compile(r"Insufficient Memory|failed to create (?:MTL|Metal) buffer", re.I)),
    ("host-oom", re.compile(r"std::bad_alloc|Cannot allocate memory|failed to allocate", re.I)),
    ("killed", re.compile(r"\bKilled\b|Out of memory: Kill(?:ed)? process|exit(?:ed)? .*137", re.I)),
    ("kv-cache", re.compile(r"KV cache|kv_cache|context size|n_ctx", re.I)),
    ("slot", re.compile(r"slot (?:is )?unavailable|no slot available|all slots are busy", re.I)),
]

# Every query is asked of every machine. A metric nothing publishes returns an empty result
# and is reported as absent, which is itself evidence about what you can and cannot see.
QUERIES = {
    "gateway_up": 'up{job="gateway"}',
    "engine_up": 'up{job=~"llama-server|vllm|sglang"}',
    "accelerator_memory_used": (
        'DCGM_FI_DEV_FB_USED * 1024 * 1024 or local_llm_accelerator_memory_used_bytes'
    ),
    "accelerator_memory_total": (
        '(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 1024 * 1024 '
        'or local_llm_accelerator_memory_total_bytes'
    ),
    "kv_cache_usage": 'vllm:kv_cache_usage_perc or sglang:token_usage',
    "context_high_water": 'llamacpp:n_tokens_max',
    "requests_running": (
        'llamacpp:requests_processing or vllm:num_requests_running or sglang:num_running_reqs'
    ),
    "requests_waiting": (
        'llamacpp:requests_deferred or vllm:num_requests_waiting or sglang:num_queue_reqs'
    ),
    "host_memory_available": 'node_memory_MemAvailable_bytes',
}


def parse_duration(text):
    match = DURATION.match(text)
    if not match:
        sys.exit(f"diagnose-oom: --since takes a number and one of s, m, h, d, for example 24h; "
                 f"got {text!r}")
    return int(match.group(1)) * SECONDS[match.group(2)]


def query_range(base, expression, start, end, step):
    """One Prometheus range query. Returns a list of (labels, samples) or None on failure."""
    params = urllib.parse.urlencode({
        "query": expression, "start": f"{start:.3f}", "end": f"{end:.3f}", "step": step,
    })
    url = f"{base.rstrip('/')}/api/v1/query_range?{params}"
    try:
        with urllib.request.urlopen(url, timeout=30) as response:  # noqa: S310 - a URL you gave
            payload = json.loads(response.read().decode("utf-8"))
    except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError) as exc:
        print(f"diagnose-oom: querying Prometheus failed: {exc}", file=sys.stderr)
        return None
    if payload.get("status") != "success":
        print(f"diagnose-oom: Prometheus refused the query: {payload.get('error')}", file=sys.stderr)
        return None
    return [(series.get("metric", {}),
             [(float(t), float(v)) for t, v in series.get("values", []) if v not in ("NaN",)])
            for series in payload["data"].get("result", [])]


def series_shape(samples):
    """Describe one series in the terms the diagnosis needs, without inventing precision."""
    values = [v for _, v in samples]
    if not values:
        return None
    first_third = values[: max(1, len(values) // 3)]
    last_third = values[-max(1, len(values) // 3):]
    start_level = sum(first_third) / len(first_third)
    end_level = sum(last_third) / len(last_third)
    peak = max(values)
    trough = min(values)
    rose = end_level > start_level * 1.15 and peak > trough * 1.15
    fell_at_end = values[-1] < peak * 0.5 < peak
    return {
        "samples": len(values),
        "start_level": start_level,
        "end_level": end_level,
        "peak": peak,
        "trough": trough,
        "rising": rose,
        "collapsed_at_end": fell_at_end,
        "ever_above_zero": peak > 0,
    }


def collect_metrics(base, start, end, step):
    out = {}
    for name, expression in QUERIES.items():
        result = query_range(base, expression, start, end, step)
        if result is None:
            out[name] = {"available": False, "reason": "Prometheus could not be queried"}
            continue
        if not result:
            out[name] = {"available": False, "reason": "no series; nothing publishes this here"}
            continue
        shapes = []
        for labels, samples in result:
            shape = series_shape(samples)
            if shape:
                shape["labels"] = labels
                shapes.append(shape)
        out[name] = {"available": bool(shapes), "series": shapes,
                     "reason": "" if shapes else "series present but empty over this window"}
    return out


def read_logs(container, since, runtime="docker"):
    if shutil.which(runtime) is None:
        return None, f"{runtime} is not on PATH"
    try:
        out = subprocess.run([runtime, "logs", "--since", since, container],
                             capture_output=True, text=True, timeout=60, check=False)
    except (OSError, subprocess.TimeoutExpired) as exc:
        return None, f"reading the log failed: {exc}"
    if out.returncode != 0:
        return None, f"{runtime} logs exited {out.returncode}: {out.stderr.strip()[:200]}"
    return (out.stdout or "") + (out.stderr or ""), ""


def scan_logs(text):
    hits = {}
    for name, pattern in LOG_PATTERNS:
        found = [line.strip() for line in text.splitlines() if pattern.search(line)]
        if found:
            hits[name] = found[-5:]
    return hits


def judge(metrics, log_hits):
    """Rank the four faults this part teaches against the evidence actually collected."""
    verdicts = []

    def shapes(name):
        block = metrics.get(name, {})
        return block.get("series", []) if block.get("available") else []

    def measured(name):
        """Absence of data is not evidence. Only reason from series that exist."""
        return bool(metrics.get(name, {}).get("available"))

    memory = shapes("accelerator_memory_used")
    kv = shapes("kv_cache_usage")
    context = shapes("context_high_water")
    waiting = shapes("requests_waiting")
    running = shapes("requests_running")

    memory_rising = any(s["rising"] for s in memory)
    kv_rising = any(s["rising"] for s in kv)
    context_rising = any(s["rising"] for s in context)
    was_queueing = any(s["ever_above_zero"] for s in waiting)
    running_peak = max((s["peak"] for s in running), default=0.0)
    memory_never_fell = memory and not any(s["collapsed_at_end"] for s in memory)

    # 1. Context creep: the cache fills because prompts got longer, not because there are
    #    more of them.
    score = 0
    why = []
    if kv_rising:
        score += 2
        why.append("key-value cache occupancy rose across the window")
    if context_rising:
        score += 2
        why.append("the high-water mark of context observed rose across the window")
    if measured("requests_waiting") and not was_queueing:
        score += 1
        why.append("nothing was queueing, so the load was not more requests")
    if "kv-cache" in log_hits:
        score += 1
        why.append("the log mentions the key-value cache or the context size")
    verdicts.append(("Context creep: the same traffic, with longer prompts", score, why))

    # 2. Concurrency beyond plan.
    score = 0
    why = []
    if was_queueing:
        score += 3
        why.append("requests were waiting for a slot")
    if running_peak > 1:
        score += 1
        why.append("more than one request was in flight at once")
    if "slot" in log_hits:
        score += 2
        why.append("the log says a slot was unavailable")
    verdicts.append(("Concurrency beyond what the engine was started for", score, why))

    # 3. Fragmentation or a leak: memory rises and never comes back down.
    score = 0
    why = []
    if memory_rising:
        score += 2
        why.append("accelerator memory in use rose across the window")
    if measured("accelerator_memory_used") and memory_never_fell:
        score += 2
        why.append("memory never returned to its starting level between bursts")
    if not kv_rising and not context_rising and memory_rising:
        score += 1
        why.append("memory rose while the cache and the context did not, so something else grew")
    verdicts.append(("Fragmentation or a leak: memory that is never given back", score, why))

    # 4. A model swap that never unloaded.
    score = 0
    why = []
    if memory_rising and memory_never_fell and measured("accelerator_memory_used"):
        score += 1
        why.append("memory stepped up and stayed up, which is what a load without an unload "
                   "looks like")
    engine = shapes("engine_up")
    if len(engine) > 1:
        score += 2
        why.append(f"{len(engine)} engine targets were up in this window, so more than one was running")
    verdicts.append(("A model swap that loaded without unloading", score, why))

    # 5. The plain one: something outside the engine took the memory.
    score = 0
    why = []
    if "host-oom" in log_hits or "killed" in log_hits:
        score += 3
        why.append("the log shows an allocation failure or a killed process")
    host = shapes("host_memory_available")
    if any(s["rising"] is False and s["end_level"] < s["start_level"] * 0.5 for s in host):
        score += 2
        why.append("host memory available fell by more than half across the window")
    verdicts.append(("The machine ran out, not the engine: something else took the memory", score, why))

    for name in ("cuda-oom", "hip-oom", "metal-oom"):
        if name in log_hits:
            verdicts.append((f"Confirmed by the log: an allocation on the accelerator failed ({name})",
                             5, [f"the log contains: {log_hits[name][-1][:160]}"]))

    verdicts.sort(key=lambda row: row[1], reverse=True)
    return verdicts


def human_bytes(value):
    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
        if abs(value) < 1024 or unit == "TiB":
            return f"{value:,.1f} {unit}"
        value /= 1024
    return f"{value:,.1f} TiB"


def write_report(metrics, log_hits, log_note, verdicts, window, out):
    lines = ["# Memory failure: what the record shows", ""]
    lines.append(f"Window examined: {window}")
    lines.append(f"Written: {datetime.now(timezone.utc).isoformat(timespec='seconds')}")
    lines.append("")
    lines.append("## What was measurable")
    lines.append("")
    for name, block in metrics.items():
        if not block.get("available"):
            lines.append(f"- **{name}**: nothing to read ({block.get('reason')})")
            continue
        for shape in block["series"]:
            label = ", ".join(f"{k}={v}" for k, v in shape["labels"].items()
                              if k not in ("__name__",)) or "no labels"
            fmt = human_bytes if "memory" in name else (lambda v: f"{v:,.3f}")
            lines.append(
                f"- **{name}** [{label}]: started near {fmt(shape['start_level'])}, "
                f"ended near {fmt(shape['end_level'])}, peaked at {fmt(shape['peak'])}"
                f"{'; rising across the window' if shape['rising'] else ''}"
                f"{'; collapsed at the end' if shape['collapsed_at_end'] else ''}")
    lines.append("")
    lines.append("## What the log said")
    lines.append("")
    if log_note:
        lines.append(f"- No log was read: {log_note}")
    elif not log_hits:
        lines.append("- The log was read and none of the known memory-failure patterns appeared. "
                     "That is worth knowing: it moves the question towards the host and away "
                     "from the engine.")
    else:
        for name, found in log_hits.items():
            lines.append(f"- **{name}**, most recent occurrence:")
            for line in found[-2:]:
                lines.append(f"  - `{line[:200]}`")
    lines.append("")
    lines.append("## Where to look first")
    lines.append("")
    lines.append("Ranked by how much of the evidence fits, not by how likely each is in general. "
                 "A candidate with no supporting evidence is listed so that you can see it was "
                 "considered and found wanting.")
    lines.append("")
    for name, score, why in verdicts:
        lines.append(f"### {name}")
        lines.append("")
        lines.append(f"Evidence for it: {score if score else 'none'}")
        lines.append("")
        if why:
            for reason in why:
                lines.append(f"- {reason}")
        else:
            lines.append("- Nothing in the collected evidence points here.")
        lines.append("")
    lines.append("## What would settle it")
    lines.append("")
    lines.extend([
        "- **Context creep**: compare the average prompt length this week with last week. "
        "If it doubled, the cache filling is arithmetic rather than a fault.",
        "- **Concurrency**: count the distinct callers in the gateway's usage records over the "
        "hour before the failure, and compare with the slot count the engine was started with.",
        "- **Fragmentation or a leak**: restart the engine with the same load and watch whether "
        "memory returns to the same starting level. If it does not, it is the process.",
        "- **A swap that never unloaded**: ask the model manager what it thinks is loaded, and "
        "compare with what the accelerator says is resident. Those two disagreeing is the fault.",
        "- **The machine, not the engine**: look at host memory and at what else was running. "
        "The engine is often the largest process rather than the guilty one.",
        "",
        "Then change one thing, put the same load back, and watch the same graph. A fix that "
        "cannot be seen on the graph that showed the fault has not been proved.",
    ])
    text = "\n".join(lines) + "\n"
    if out:
        Path(out).expanduser().write_text(text, encoding="utf-8")
        print(f"wrote {out}")
    else:
        print(text)
    return text


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--prometheus", default="http://127.0.0.1:9090",
                        help="base URL of your Prometheus")
    parser.add_argument("--since", default="24h", help="how far back to look, e.g. 24h or 3d")
    parser.add_argument("--step", default="5m", help="resolution of the range queries")
    parser.add_argument("--container", default=None, help="container whose log to read")
    parser.add_argument("--runtime", default="docker", help="docker or podman")
    parser.add_argument("--output", default=None, help="write the report here")
    parser.add_argument("--labbook", default=None, help="append one JSON line here")
    args = parser.parse_args()

    seconds = parse_duration(args.since)
    end = time.time()
    start = end - seconds
    window = (f"the last {args.since}, from "
              f"{datetime.fromtimestamp(start, timezone.utc).isoformat(timespec='seconds')} "
              f"to {datetime.fromtimestamp(end, timezone.utc).isoformat(timespec='seconds')}")

    metrics = collect_metrics(args.prometheus, start, end, args.step)
    log_hits, log_note = {}, "no --container was given"
    if args.container:
        text, note = read_logs(args.container, args.since, args.runtime)
        if text is None:
            log_note = note
        else:
            log_hits, log_note = scan_logs(text), ""

    verdicts = judge(metrics, log_hits)
    write_report(metrics, log_hits, log_note, verdicts, window, args.output)

    if args.labbook:
        top = verdicts[0] if verdicts else ("no candidate", 0, [])
        record = {
            "lab": "part-23/challenge-oom",
            "window": args.since,
            "top_candidate": top[0],
            "evidence_score": top[1],
            "log_patterns_seen": sorted(log_hits.keys()),
            "metrics_unavailable": sorted(k for k, v in metrics.items() if not v.get("available")),
            "recorded": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        }
        with Path(args.labbook).expanduser().open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record, sort_keys=True) + "\n")
        print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download diagnose-oom.py407 lines

RunnableAll tracks

the healthy report, taken now

python3 diagnose-oom.py --since 24h --output healthy-report.md

Read healthy-report.md end to end while everything is working. Knowing what a healthy report looks like is what makes an unhealthy one legible, and it also tells you which metrics your machine does not publish, which is information you want before an incident rather than during one.

It queries Prometheus over its HTTP API, which documents /api/v1/query_range taking a query, a start, an end and a step, and returning a JSON envelope with a status and a data object. Nothing exotic; you could do the same with curl and patience.

=Expected result

On most machines several of the metrics will be reported as absent, because no machine runs every engine in this course. The report says which and why. A candidate fault with no supporting evidence is still listed, with its evidence count at none, so that you can see it was considered rather than forgotten.

3. Break it on purpose, on your track

Track S — NVIDIA DGX Spark

Start the engine with fewer slots than you are about to use, then exceed them.

RunnableTrack S · DGX Spark

one slot, on purpose

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 8192 --parallel 1 --cont-batching --metrics

RunnableTrack S · DGX Spark

six callers, one slot

for i in 1 2 3 4 5 6; do
    curl -s http://127.0.0.1:8080/v1/chat/completions \
        -H "Content-Type: application/json" \
        -d '{"model":"local/chat",
             "messages":[{"role":"user","content":"Explain tides in six sentences."}],
             "max_tokens":256}' > /dev/null &
done
wait

Watch requests waiting on the dashboard while it runs. That is the queueing shape, and it is what your service looks like an hour before it stops being pleasant to use.

Track X — AMD Ryzen AI Max+ 395

Fill the cache with length rather than with callers. Start the engine with a large context and send prompts that grow.

RunnableTrack X · Ryzen AI Max+

a generous context, on purpose

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 32768 --parallel 4 --cont-batching --metrics

Build each request body with a short script rather than by hand, because the point is a prompt whose length you control.

RunnableAll tracks

make four request bodies, each longer than the last

import json
from pathlib import Path

for n in (200, 800, 3200, 12800):
    prompt = "The tide came in. " * n
    body = {"model": "local/chat", "max_tokens": 16,
            "messages": [{"role": "user", "content": prompt}]}
    Path(f"body-{n}.json").write_text(json.dumps(body))
    print(f"body-{n}.json  {len(prompt)} characters")

RunnableTrack X · Ryzen AI Max+

send them in order, watching the memory panel

for n in 200 800 3200 12800; do
    curl -s http://127.0.0.1:8080/v1/chat/completions \
        -H "Content-Type: application/json" \
        --data-binary "@body-${n}.json" > /dev/null
done

Watch the memory panel and the request count together. Memory rising while the request count is flat is the whole signature of this fault.

Track M — Apple silicon

The same shape as Track X, in its mildest form, and stop as soon as you see it.

RunnableTrack M · Apple silicon

a generous context, on a model well inside your tier

~/llama.cpp/build/bin/llama-server \
    --model ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 \
    --ctx-size 32768 --parallel 2 --cont-batching --metrics

Send two or three progressively longer prompts by hand, watching the memory panel between each. When the line has visibly stepped up and not come back down, you have the shape. Stop the server.

⚠Warning

On unified memory the pressure lands on the whole machine, not on a device. If the desktop becomes sluggish, stop the engine immediately; the system is paging and it recovers as soon as the process ends. Do this once, deliberately, on a machine with nothing else important running.

Track N — NVIDIA desktop or laptop

Make two models coexist that should not. Copy llama-swap.yaml, change the accelerator group in the copy so that exclusive is false, and point llama-swap at the copy.

RunnableTrack N · NVIDIA GPU

ask for both, in turn

curl -s http://127.0.0.1:9292/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat","messages":[{"role":"user","content":"hello"}],"max_tokens":8}' > /dev/null

curl -s http://127.0.0.1:9292/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"local/coder","messages":[{"role":"user","content":"hello"}],"max_tokens":8}' > /dev/null

curl -s http://127.0.0.1:9292/running

Compare what /running reports with what the memory panel shows. Two models resident when the machine can hold one is the fault, and on a discrete card it ends in an allocation failure rather than in slowness. Put the original configuration back afterwards.

4. Diagnose from the report, not from memory

RunnableAll tracks

the report for the window you just broke

python3 diagnose-oom.py --since 1h --output broken-report.md

Open broken-report.md and healthy-report.md side by side and work down the sections. For each, write one line saying what differs. The report is arranged in the order of the procedure on purpose, so the first section that differs usually names the fault.

Then, separately, write your own diagnosis in one sentence before reading the script’s ranked list. The script is a fast reader of graphs and nothing more; the point of writing your own first is to find out whether you can, because on the night it matters the script may be the thing that is down.

✕Common mistake

Trusting the ranked list because it is a list. It scores four candidates against whatever evidence was collectable, which on a machine that publishes half these metrics is not much. Its value is that it shows its working: every candidate lists the evidence for it, and a candidate at the top with one weak reason behind it is telling you that you need more evidence rather than that it is the answer.

5. Fix one thing, and prove it with the same load

Make the single change your diagnosis implies, and then run the same load again.

• Context creep: lower --ctx-size to the plan, or quantise the cache with --cache-type-k and --cache-type-v.
• Concurrency: raise --parallel to the planned slot count, having checked the memory at that count, or limit the caller with rpm_limit.
• Fragmentation or a leak: restart the engine and confirm the floor returns; then set a ttl so that idle periods do it for you.
• A swap that never unloaded: restore exclusive: true on the group, and confirm with /running and the memory panel together.

The proof is that the graph changed shape under the same load, not that nothing failed for an hour. A memory line that now returns to its floor, or a waiting count that stays at zero, is evidence. An absence of failure is not.

Pending validationBefore and after, one fault, one fix — your recording sheet

State	Fault	Memory floor	Memory peak	Requests waiting, peak	What the log said
Healthy baseline	—	—	—	—	—
Fault reproduced	—	—	—	—	—
After one change	—	—	—	—	—

your machine: track, chip, memory, and how memory was read, your operating system and version · the engine and the flags it was started with, for each row the engine version, from its own --version output · the model in use, its quantisation · 8,192 tokens of context · the date you ran it

Empty on purpose. Three rows, one change between the second and the third. Record the engine flags separately for each row, because the flags are usually what changed and six months later that is the only thing you will want to know.

6. Add the guard rail

A fault fixed without a guard rail is a fault you will diagnose again from scratch. Add one now, while the evidence is fresh, and make it the one that matches what you found.

• The AcceleratorMemoryHigh rule from the lab, if it is not already firing usefully. Check that its expression names metrics your machine actually publishes; a rule over an absent metric is silent in exactly the way a healthy machine is silent.
• A ttl in llama-swap.yaml, if the fault involved memory that was never given back.
• An rpm_limit on the key that ran away, if the fault was concurrency.
• A context_window_fallbacks entry, if the fault was a request too long for the model.

Then reload the rules, confirm they are loaded, and write the change in the changelog.

RunnableAll tracks

apply and confirm the rule change

curl -s -X POST http://127.0.0.1:9090/-/reload

curl -s http://127.0.0.1:9090/api/v1/rules | head -c 400

Identify the phase of allocation failure

Record whether the failure occurs while loading weights, allocating cache, prefilling a long prompt, adding concurrent requests or switching models. These phases imply different memory terms. Preserve the startup configuration, recent request sizes, device memory and process logs before restarting.

Introduce only the lesson’s controlled fault in the isolated service. Compare the same workload before and after. If reducing concurrency fixes it, check whether per-request state caused the peak; if model switching triggers it, inspect residency and exclusion groups. Do not label every rising allocation a leak without testing whether it is bounded cache growth or another resident model.

Apply one guardrail, such as an input limit, admission bound or corrected unloading policy, then repeat both the failing request and an ordinary request. The ordinary path must remain usable while the excessive workload is rejected or managed as designed. Record the error clients see and how the service recovers. Finish with a short runbook entry: symptom, first evidence, diagnosis, recovery and prevention. Keep the failed run’s evidence so a later incident can be compared against an actual memory failure rather than a remembered anecdote.

Validation

You are done when all of the following are true.

• healthy-report.md and broken-report.md both exist, and you can point at the section that differs between them.
• You wrote a one-sentence diagnosis before reading the script’s ranked list, and you can say whether they agreed.
• The recording sheet has three complete rows, including the engine flags for each.
• The third row’s graph shape differs from the second’s under the same load.
• One guard rail is in place, loaded, and confirmed to name metrics your machine publishes.
• The changelog has an entry with the fault, the evidence, the change and the guard rail.
• Anything you changed temporarily to reproduce the fault has been changed back.

Expected outcome

A procedure you can run from memory, and a calibrated sense of what each of the four faults looks like before it becomes a failure.

The four shapes, which your own graphs will make concrete. Context creep climbs slowly across days while the request count stays flat. Concurrency shows up as waiting requests before it shows up in memory at all. A leak ratchets, so the floor rises and never returns. And a swap that never unloaded is a step that does not come back down, confirmed by the model manager and the accelerator disagreeing about what is resident.

=Expected result

After the fix, the same load should produce a memory line that returns to the floor you recorded in the lab’s baseline sheet, or a waiting count that stays at zero, depending on which fault you reproduced. If the number recovered but the shape did not, you have changed something other than the fault, and the investigation is not finished.

Troubleshooting

The report says every metric is absent. Prometheus is unreachable or holds no data for that window. Check the address you passed and that the stack from the previous page is running; the script prints the error it got rather than swallowing it.

The report has data but the memory section is empty. Your accelerator exporter is not running, or it publishes under different names. This is the specific case the lab warned about, and it means the memory alert is also silent.

The log section says no known pattern appeared. That is a result, not a failure. It moves the question towards the host and away from the engine: look at host memory available and at what else was running, because the engine is often the largest process rather than the guilty one.

Two candidates score the same. Collect more evidence rather than choosing. The two that are most often confused are context creep and a leak, and the discriminator is whether the rise tracks the cache metric or is independent of it.

The fault will not reproduce. Usually the machine has more headroom than the fault needs. Increase the context length, or the concurrency, or use a larger model, one change at a time. If it still will not, that is worth writing down: your configuration is more robust than you thought, and you now know by how much.

The service failed for real during the exercise. Stop, take the evidence, and treat it as the exercise. That is the whole skill and it is more valuable on a real failure than on an induced one.

Cleanup

Put back everything you changed to reproduce the fault: the slot count, the context length, the group configuration. Confirm with the Part 9 check that the gateway is as it was.

RunnableAll tracks

confirm the gateway is back to normal

bash check-gateway.sh

Keep both reports and the changelog entry. The monitoring stack stays running; it is now holding the history that makes the next incident tractable.

What you learned

• Evidence before theory, and the graph before the log. The memory shape distinguishes the four faults before you read a single line of output, and the log is where a theory is confirmed rather than where one is formed.
• A restart destroys the evidence. Thirty seconds of screenshot and log copy is the difference between a diagnosis and a mystery that repeats every week.
• The four shapes are distinguishable. Slow climb with flat load, waiting requests, a ratcheting floor, and a step that never returns. Learn the four and you have learned the diagnosis.
• Two sources disagreeing is the strongest evidence there is. The model manager saying one model is loaded while the accelerator holds two is not ambiguous, and no other fault produces it.
• A fix proves itself on the same load. The absence of a failure is not evidence; a changed graph under an unchanged load is.
• A fault without a guard rail recurs. An alert, a limit or a ttl, chosen to match what you found, is what converts one night of work into a permanent improvement.
• Nothing changed is a cause, not an alibi. Context length grows on its own, allocators fragment on their own, and cron jobs start overlapping on their own. A service that has been running for a week is a service whose conditions have been changing for a week.

Record in the changelog: the fault you reproduced, the evidence that identified it, the single change that fixed it, the three rows of the recording sheet, the guard rail you added, and the date.

Check your understanding

Question 1. A service that has been running unchanged for a week fails overnight. Why is "nothing changed" not a reason to look outside the service?

• Something always changed; somebody is not telling you
• The conditions change on their own: conversations lengthen, allocators fragment, and scheduled jobs start overlapping as they get slower, so a week of no changes is a week of drifting conditions
• Overnight failures are always hardware
• It is a reason; the fault is usually elsewhere on the network

Show the answer and why

Answer: The conditions change on their own: conversations lengthen, allocators fragment, and scheduled jobs start overlapping as they get slower, so a week of no changes is a week of drifting conditions

Three of the four faults in this challenge need no change of any kind to arrive. That is what makes them hard to anticipate and why the memory graph over days, rather than the configuration, is where the investigation starts.

Question 2. Memory climbs steadily over several days while the number of requests running stays flat. Which fault does that shape point at?

• Concurrency beyond the plan
• Context creep: the same traffic with longer conversations, so cache use grows without the request count changing
• A model swap that never unloaded
• A failing disk

Show the answer and why

Answer: Context creep: the same traffic with longer conversations, so cache use grows without the request count changing

The flat request count is the discriminator. Concurrency raises the request count, a swap produces a step rather than a climb, and a leak rises independently of the cache metric. A climb that tracks cache occupancy or the context high-water mark, with unchanged load, is context creep.

Question 3. llama-swap reports one model running, and the accelerator reports enough memory in use for two. What have you found?

• A measurement error in the exporter
• A model that was loaded and never released: the model manager and the device disagreeing about what is resident is the signature of that fault and of no other
• A memory leak in the engine
• Normal behaviour, because the exporter includes the operating system

Show the answer and why

Answer: A model that was loaded and never released: the model manager and the device disagreeing about what is resident is the signature of that fault and of no other

Two independent sources disagreeing is the strongest evidence available in an investigation, and this particular disagreement has one cause. The immediate remedy is the documented unload endpoint; the lasting one is an exclusive group so the two models cannot coexist.

Question 4. Which of these count as proof that your fix worked? Select all that apply.

• The same load now produces a memory line that returns to its recorded floor
• The same load now produces a waiting count that stays at zero
• The service has not failed since you made the change
• The changed graph is visible on the same panel that showed the fault

Show the answer and why

Answer: The same load now produces a memory line that returns to its recorded floor, The same load now produces a waiting count that stays at zero, The changed graph is visible on the same panel that showed the fault

A fix proves itself by changing the evidence that identified the fault, under the load that produced it. An absence of failure over a short window is exactly what you had before the first failure, which is why it proves nothing.

Question 5. Your machine runs llama.cpp, and the diagnose script reports that key-value cache usage is unavailable. What should you conclude?

• The exporter is broken
• llama.cpp publishes no cache occupancy metric, so that signal genuinely does not exist here; the context high-water mark and the accelerator memory reading are what you have instead
• Prometheus is misconfigured
• The engine must be restarted with a different flag

Show the answer and why

Answer: llama.cpp publishes no cache occupancy metric, so that signal genuinely does not exist here; the context high-water mark and the accelerator memory reading are what you have instead

Knowing which questions your engine cannot answer is part of the diagnosis. The report says the metric is absent rather than reporting zero, which matters: zero would be a claim about the cache, and absence is a claim about the instrumentation.

Check my answersClear