Backup, Upgrades and Reproducibility of a Model Estate

By the end of this lesson you will be able to say which parts of your model estate you could lose without much regret and which would take months to reconstruct, take a backup of the second category that you have actually tested, pin every engine to something that cannot move underneath you, upgrade one with a rollback you tested before you needed it, and hand a new machine enough information to become this one.

The test for all of it is a single question: if this machine’s disk failed tonight, what would you be unable to get back?

What is actually irreplaceable

The instinct is to back up the largest thing, which is the model library. That is almost exactly backwards.

What	Replaceable?	What it really costs to lose
Model weights	Yes, if you recorded the repository, revision and hash	A download, and an afternoon if the file was later withdrawn
Engine binaries and images	Yes, if you pinned a version or a digest	A rebuild; much worse if you only recorded “latest”
Gateway and monitoring configuration	Only from memory	Days of rediscovering settings that took you months to converge on
Fine-tuned adapters from Part 13	No	The training run, the data preparation, and the hyperparameters you no longer have
Your own evaluation sets from Parts 10 and 16	No	The most valuable thing on the machine. Nobody else has your test cases
The lab notebook	No	Every measurement you made, and the reason behind every decision
Dashboards and alert rules	Only if they were clicked rather than provisioned	An evening, and the ones you forget to rebuild are the ones you needed
Secrets and certificates	Regenerable, at a cost	Reissuing every key and reconfiguring every client

The pattern is that the big files are cheap and the small ones are dear. Your evaluation set is a few hundred kilobytes of questions with expected answers, it is the only thing that can tell you whether next year’s model is better for your work, and it exists nowhere else in the world.

!Important

Weights are the one category where a hash substitutes for a copy. If you have recorded the repository, the revision and the SHA-256 of every file, you can prove that a re-downloaded file is the same file. That turns a large backup problem into a small manifest, which is why the manifest below is the load-bearing part of this lesson and the archive is the easy part.

RunnableAll tracks

backup-estate.sh

#!/usr/bin/env bash
# Purpose: take a restorable backup of everything about your model estate that is not the
#          weights - configurations, adapters, evaluation sets, the lab notebook, the
#          dashboard, and a manifest recording every model file's hash - verify the
#          archive can be read back, and record what was taken
# Platform: all (Linux, macOS and WSL2)
# Minimum memory: 8 GB, which is the service being backed up; this script needs almost none
# Assumes: tar on PATH, python3 for the manifest step, and enough free space in the
#          destination. Model weights are excluded on purpose: they are large, they are
#          re-downloadable, and the manifest records the hash of each one so you can prove
#          that what you download next year is what you had this year. Pass
#          --include-weights when you have somewhere to put them and a reason.
#
# Usage: bash backup-estate.sh --gateway ~/gateway --models ~/models --dest ~/backups
#        bash backup-estate.sh --gateway ~/gateway --models ~/models --dest ~/backups \
#             --extra ~/adapters --extra ~/evals --labbook labbook.md
set -euo pipefail

GATEWAY_DIR=""
MODELS_DIR=""
DEST="${HOME}/backups"
LABBOOK=""
INCLUDE_WEIGHTS=0
EXTRAS=()

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"
if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
    DEST="${BACKUP_DIR:-$DEST}"
fi

while [ $# -gt 0 ]; do
    case "$1" in
        --gateway)         GATEWAY_DIR="${2:?--gateway needs a directory}"; shift 2 ;;
        --models)          MODELS_DIR="${2:?--models needs a directory}"; shift 2 ;;
        --dest)            DEST="${2:?--dest needs a directory}"; shift 2 ;;
        --extra)           EXTRAS+=("${2:?--extra needs a directory}"); shift 2 ;;
        --labbook)         LABBOOK="${2:?--labbook needs a file}"; shift 2 ;;
        --include-weights) INCLUDE_WEIGHTS=1; shift ;;
        -h|--help)         sed -n '2,22p' "$0"; exit 0 ;;
        *) echo "unknown argument: $1" >&2; exit 2 ;;
    esac
done

command -v tar >/dev/null 2>&1 || { echo "tar is not on PATH." >&2; exit 1; }
[ -n "$GATEWAY_DIR" ] || { echo "Give --gateway <directory>: the Part 9 gateway directory." >&2; exit 1; }
[ -d "$GATEWAY_DIR" ] || { echo "No such directory: ${GATEWAY_DIR}" >&2; exit 1; }

mkdir -p "$DEST"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
ARCHIVE="${DEST}/estate-${STAMP}.tar.gz"

echo "==> Writing the manifest"
MANIFEST="${WORK}/estate-manifest.json"
if [ -f "${HERE}/estate-manifest.py" ] && command -v python3 >/dev/null 2>&1; then
    MANIFEST_ARGS=(--gateway "$GATEWAY_DIR" --output "$MANIFEST")
    [ -n "$MODELS_DIR" ] && MANIFEST_ARGS+=(--models "$MODELS_DIR")
    python3 "${HERE}/estate-manifest.py" "${MANIFEST_ARGS[@]}"
else
    echo "    estate-manifest.py is not beside this script; skipping the manifest." >&2
    printf '{"note":"no manifest was written"}\n' > "$MANIFEST"
fi

echo "==> Collecting"
STAGE="${WORK}/estate"
mkdir -p "$STAGE"
cp "$MANIFEST" "${STAGE}/estate-manifest.json"

# The gateway directory, minus the one file in it that must never leave the machine.
mkdir -p "${STAGE}/gateway"
tar -cf - -C "$(dirname "$GATEWAY_DIR")" \
    --exclude='.env' --exclude='*.key' --exclude='*.pem' \
    "$(basename "$GATEWAY_DIR")" | tar -xf - -C "${STAGE}/gateway"

for extra in ${EXTRAS+"${EXTRAS[@]}"}; do
    if [ -d "$extra" ]; then
        echo "    including ${extra}"
        mkdir -p "${STAGE}/extra"
        tar -cf - -C "$(dirname "$extra")" "$(basename "$extra")" | tar -xf - -C "${STAGE}/extra"
    else
        echo "    skipping ${extra}: not a directory" >&2
    fi
done

if [ -n "$LABBOOK" ] && [ -f "$LABBOOK" ]; then
    cp "$LABBOOK" "${STAGE}/labbook.md"
fi

if [ "$INCLUDE_WEIGHTS" -eq 1 ]; then
    [ -n "$MODELS_DIR" ] || { echo "--include-weights needs --models." >&2; exit 1; }
    echo "    including the model library, which will take a while and a lot of space"
    mkdir -p "${STAGE}/models"
    tar -cf - -C "$(dirname "$MODELS_DIR")" "$(basename "$MODELS_DIR")" \
        | tar -xf - -C "${STAGE}/models"
fi

echo "==> Archiving"
tar -czf "$ARCHIVE" -C "$WORK" estate

echo "==> Verifying the archive can be read back"
ENTRIES="$(tar -tzf "$ARCHIVE" | wc -l | tr -d ' ')"
BYTES="$(wc -c < "$ARCHIVE" | tr -d ' ')"
if [ "$ENTRIES" -lt 2 ]; then
    echo "The archive contains almost nothing; something went wrong." >&2
    exit 1
fi
if ! tar -tzf "$ARCHIVE" | grep -q 'estate/estate-manifest.json'; then
    echo "The manifest is not in the archive; something went wrong." >&2
    exit 1
fi

echo "    ${ARCHIVE}"
echo "    ${ENTRIES} entries, ${BYTES} bytes"
echo "    .env, private keys and certificates were excluded on purpose. Keep those"
echo "    somewhere else, deliberately, and know where."

if [ -n "$LABBOOK" ]; then
    printf '{"lab":"part-23/backup","archive":"%s","entries":%s,"bytes":%s,"weights_included":%s,"taken":"%s"}\n' \
        "$(basename "$ARCHIVE")" "$ENTRIES" "$BYTES" "$INCLUDE_WEIGHTS" "$STAMP" >> "$LABBOOK"
    echo "    recorded in ${LABBOOK}"
fi

echo "==> Done. Now restore it somewhere harmless and check that it is what you think."

Download backup-estate.sh129 lines

The script archives configuration, adapters, evaluation sets and the notebook, writes a manifest of every model file beside them, verifies that the archive can be read back, and excludes .env, private keys and certificates on purpose. That exclusion is a decision, not an oversight: those things need somewhere else, chosen deliberately, and an archive you copy to another machine is not it.

RunnableAll tracks

a backup of everything except the weights

bash backup-estate.sh \
    --gateway ~/gateway \
    --models ~/models \
    --extra ~/adapters \
    --extra ~/evals \
    --labbook labbook.md \
    --dest ~/backups

⚠Warning

A backup you have never restored is a hypothesis. Restore this one now, into a scratch directory, and look at what came out: does it contain the configuration you expected, does the manifest list the models you have, is the notebook the current one? Ten minutes today is the difference between a recovery and a discovery.

Pinning, so that nothing moves underneath you

Three kinds of thing can change without you doing anything, and each has its own pin.

Engine versions. Every tool this course teaches is in its version table, which exists so that a page cannot describe behaviour from a version nobody checked. Your estate deserves the same: record the exact version of each engine you run, from its own --version output rather than from what you believe you installed.

Container digests. A tag is a name that points at an image today. Docker’s documentation describes pulling by digest as specifying “exactly which version of an image to pull”, which “allows you to ‘pin’ an image to that version”, and it is candid about the trade: a pinned digest “does therefore not pull updated versions of an image, which may include security updates. If you want to pull an updated image, you need to change the digest accordingly”. That is the whole bargain. You get reproducibility and you take on the job of deciding when to move.

RunnableAll tracks

turn a tag into something that cannot move

docker image pull ghcr.io/mostlygeek/llama-swap:unified-cuda13

docker image inspect --format '{{index .RepoDigests 0}}' \
    ghcr.io/mostlygeek/llama-swap:unified-cuda13

Take the string that comes back, with its @sha256: part, and put that in .env instead of the tag. LiteLLM’s deployment page makes the same recommendation for its own image, and Part 9’s environment file already pins LiteLLM 1.100.0 · verified 2026-09-08 by tag for exactly this reason.

Model revisions. The Hub’s download guide documents revision for a branch, a tag or a commit hash, noting that a commit hash “must be the full-length hash instead of a 7-character commit hash”. Record the revision you downloaded, not just the repository name. Then hf cache verify, documented as validating “local files against their checksums on the Hub”, is how you confirm later that what is on your disk is what was published.

RunnableAll tracks

confirm the library is what the Hub says it is

hf cache verify Qwen/Qwen3-8B-GGUF

An upgrade that tests itself

The upgrade that goes wrong is never the one you were careful about. It is the routine one, done on a Tuesday, where the new image reads one configuration key differently and the service comes back up looking healthy and answering nothing.

An engine upgrade, with the rollback built into the procedure

1. 1Resolve the new image to a digestPull the tag, read the digest, and record that. From here on the upgrade refers to something that cannot change.
2. 2Start it beside the live oneA second container on a spare port, with the same configuration mounted read-only, and without the model directory so that it starts without loading weights.
3. 3Check the canaryDoes it become healthy, and does it publish the model list your configuration describes? Those two questions catch a configuration format change, which is the usual fault.
4. 4Optionally ask it for a real answerOnly if the machine has room for a second copy of a model. Most do not, which is why this step is opt-in rather than default.
5. 5Swap the live serviceWrite the new digest into the environment, keep the old one beside it, and recreate the service.
6. 6Re-run the gateway checkPart 9 check-gateway.sh, unchanged. This is the moment the upgrade is either confirmed or refuted.
7. 7Roll back automatically if it failsThe old digest is one environment variable away, and the script puts it back without asking, because a broken service is not a good place to have a conversation.
8. 8Record it either wayFrom, to, and what happened. A successful upgrade is as worth recording as a failed one, because next time you will want to know when this version arrived.

RunnableAll tracks

upgrade-engine.sh

#!/usr/bin/env bash
# Purpose: upgrade the container image behind the gateway without finding out in
#          production whether the new one works - pull it, resolve it to a digest, run it
#          beside the live one on a spare port, check it, swap it in, re-check the
#          gateway, and put the old digest back automatically if the re-check fails
# Platform: spark, strix, nvidia (Docker Engine with the Compose plugin), and mac where
#           Docker Desktop is installed. A native install has no image to swap; the same
#           procedure applies with a second binary and is described on the page.
# Minimum memory: 8 GB. The canary is started without loading a model, so it costs almost
#           nothing; --deep asks it to answer a real request and needs room for a second
#           copy of the model, which most machines do not have.
# Assumes: run from the gateway directory built in Part 9, holding compose.yaml and a .env
#          with LLAMA_SWAP_IMAGE in it. This script edits exactly two keys in .env,
#          LLAMA_SWAP_IMAGE and LLAMA_SWAP_IMAGE_PREVIOUS, and touches nothing else. It
#          never prints the contents of .env, which holds your keys.
#
# Usage: bash upgrade-engine.sh --image ghcr.io/example/engine:sometag
#        bash upgrade-engine.sh --image ghcr.io/example/engine:sometag --deep
#        bash upgrade-engine.sh --rollback
set -euo pipefail

COMPOSE_FILE="compose.yaml"
SERVICE="swap"
IMAGE_KEY="LLAMA_SWAP_IMAGE"
CANARY_PORT="9293"
CHANGELOG=""
NEW_IMAGE=""
DEEP=0
ROLLBACK=0
ASSUME_YES=0

usage() { sed -n '2,20p' "$0"; }

while [ $# -gt 0 ]; do
    case "$1" in
        --image)      NEW_IMAGE="${2:?--image needs an image reference}"; shift 2 ;;
        --compose)    COMPOSE_FILE="${2:?--compose needs a file}"; shift 2 ;;
        --service)    SERVICE="${2:?--service needs a compose service name}"; shift 2 ;;
        --key)        IMAGE_KEY="${2:?--key needs an .env variable name}"; shift 2 ;;
        --port)       CANARY_PORT="${2:?--port needs a port}"; shift 2 ;;
        --changelog)  CHANGELOG="${2:?--changelog needs a file}"; shift 2 ;;
        --deep)       DEEP=1; shift ;;
        --rollback)   ROLLBACK=1; shift ;;
        --yes)        ASSUME_YES=1; shift ;;
        -h|--help)    usage; exit 0 ;;
        *) echo "unknown argument: $1" >&2; exit 2 ;;
    esac
done

command -v docker >/dev/null 2>&1 || { echo "docker is not on PATH." >&2; exit 1; }
command -v curl >/dev/null 2>&1 || { echo "curl is not on PATH." >&2; exit 1; }
[ -f "$COMPOSE_FILE" ] || { echo "no such compose file: ${COMPOSE_FILE}" >&2; exit 1; }
[ -f ".env" ] || { echo "no .env beside ${COMPOSE_FILE}; run this from the gateway directory." >&2; exit 1; }

# Read one key out of .env without sourcing the whole file, so that a stray line in a file
# holding your keys cannot execute.
env_value() {
    grep -E "^$1=" .env | tail -n 1 | cut -d= -f2- || true
}

# Replace one key in .env, or append it. A temporary file and a move, so an interrupted
# write cannot leave you with half a .env.
set_env_value() {
    local key="$1" value="$2" tmp
    tmp="$(mktemp)"
    if grep -qE "^${key}=" .env; then
        sed "s|^${key}=.*|${key}=${value}|" .env > "$tmp"
    else
        cp .env "$tmp"
        printf '%s=%s\n' "$key" "$value" >> "$tmp"
    fi
    mv "$tmp" .env
}

confirm() {
    [ "$ASSUME_YES" -eq 1 ] && return 0
    printf '%s [y/N] ' "$1"
    read -r answer
    case "$answer" in y|Y|yes|YES) return 0 ;; *) return 1 ;; esac
}

record() {
    # record <action> <from> <to> <result>
    [ -n "$CHANGELOG" ] || return 0
    printf '{"lab":"part-23/upgrades","action":"%s","service":"%s","from":"%s","to":"%s","result":"%s","at":"%s"}\n' \
        "$1" "$SERVICE" "$2" "$3" "$4" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$CHANGELOG"
    echo "    recorded in ${CHANGELOG}"
}

CURRENT="$(env_value "$IMAGE_KEY")"
PREVIOUS="$(env_value "${IMAGE_KEY}_PREVIOUS")"

if [ "$ROLLBACK" -eq 1 ]; then
    [ -n "$PREVIOUS" ] || { echo "No ${IMAGE_KEY}_PREVIOUS in .env; nothing to roll back to." >&2; exit 1; }
    echo "==> Rolling back ${SERVICE}"
    echo "    from ${CURRENT}"
    echo "    to   ${PREVIOUS}"
    confirm "Proceed?" || { echo "Nothing changed."; exit 0; }
    set_env_value "$IMAGE_KEY" "$PREVIOUS"
    set_env_value "${IMAGE_KEY}_PREVIOUS" "$CURRENT"
    docker compose -f "$COMPOSE_FILE" up -d "$SERVICE"
    echo "==> Rolled back. Run check-gateway.sh to confirm."
    record rollback "$CURRENT" "$PREVIOUS" "applied"
    exit 0
fi

[ -n "$NEW_IMAGE" ] || { echo "Give --image <reference>, or --rollback." >&2; usage >&2; exit 2; }

echo "==> Pulling ${NEW_IMAGE}"
docker image pull "$NEW_IMAGE" >/dev/null

# A tag moves; a digest does not. Resolve now, so that what is recorded in .env and in the
# changelog is the thing that actually ran, not a name that pointed at it this morning.
DIGEST="$(docker image inspect --format '{{if .RepoDigests}}{{index .RepoDigests 0}}{{end}}' "$NEW_IMAGE" 2>/dev/null || true)"
if [ -z "$DIGEST" ]; then
    echo "    This image has no repository digest, which means it was built locally rather"
    echo "    than pulled. Continuing with the reference you gave, which is not pinned."
    DIGEST="$NEW_IMAGE"
fi
echo "    resolved to ${DIGEST}"

echo "==> Starting a canary on port ${CANARY_PORT}, beside the live service"
CANARY="upgrade-canary-$$"
CLEANED=0
cleanup_canary() {
    [ "$CLEANED" -eq 1 ] && return 0
    CLEANED=1
    docker rm -f "$CANARY" >/dev/null 2>&1 || true
}
trap cleanup_canary EXIT

# The canary runs the same configuration as the live service, read-only, on a spare port.
# It is not given the model directory unless --deep was asked for, so it starts, answers
# and stops without ever loading weights.
CANARY_ARGS=(run -d --name "$CANARY" -p "127.0.0.1:${CANARY_PORT}:8080"
             -v "$(pwd)/llama-swap.yaml:/app/config.yaml:ro")
if [ "$DEEP" -eq 1 ]; then
    MODELS_DIR="$(env_value MODELS_DIR)"
    [ -n "$MODELS_DIR" ] || { echo "--deep needs MODELS_DIR in .env." >&2; exit 1; }
    CANARY_ARGS+=(-v "${MODELS_DIR}:/models:ro" -e "MODELS_DIR=/models" -e "LLAMA_BIN=llama-server")
fi
docker "${CANARY_ARGS[@]}" "$NEW_IMAGE" --config /app/config.yaml --listen 0.0.0.0:8080 >/dev/null

echo "==> Checking the canary"
CANARY_URL="http://127.0.0.1:${CANARY_PORT}"
OK=1
for _ in $(seq 1 30); do
    if curl -sf "${CANARY_URL}/health" >/dev/null 2>&1; then OK=0; break; fi
    sleep 2
done
if [ "$OK" -ne 0 ]; then
    echo "    FAIL  the canary never became healthy. Its log follows; nothing was changed." >&2
    docker logs --tail 40 "$CANARY" >&2 || true
    record upgrade "$CURRENT" "$DIGEST" "canary-failed"
    exit 1
fi
echo "    PASS  health endpoint answers"

if curl -sf "${CANARY_URL}/v1/models" | grep -q '"id"'; then
    echo "    PASS  the model list is published"
else
    echo "    FAIL  the model list is empty; the new image reads your configuration differently." >&2
    docker logs --tail 40 "$CANARY" >&2 || true
    record upgrade "$CURRENT" "$DIGEST" "canary-failed"
    exit 1
fi

if [ "$DEEP" -eq 1 ]; then
    echo "    ... asking the canary for a real completion; this loads a model and takes a while"
    if curl -sf --max-time 600 "${CANARY_URL}/v1/chat/completions" \
        -H "Content-Type: application/json" \
        -d '{"model":"local/chat","messages":[{"role":"user","content":"Reply with one word: ready"}],"max_tokens":8}' \
        | grep -q '"content"'; then
        echo "    PASS  the canary generated a completion"
    else
        echo "    FAIL  the canary could not generate. Nothing was changed." >&2
        docker logs --tail 40 "$CANARY" >&2 || true
        record upgrade "$CURRENT" "$DIGEST" "canary-failed"
        exit 1
    fi
fi

cleanup_canary

echo "==> Swapping the live service"
echo "    from ${CURRENT}"
echo "    to   ${DIGEST}"
confirm "Proceed?" || { echo "Nothing changed."; exit 0; }

set_env_value "${IMAGE_KEY}_PREVIOUS" "$CURRENT"
set_env_value "$IMAGE_KEY" "$DIGEST"
docker compose -f "$COMPOSE_FILE" up -d "$SERVICE"

echo "==> Re-checking the gateway"
if [ -x ./check-gateway.sh ] || [ -f ./check-gateway.sh ]; then
    if bash ./check-gateway.sh; then
        echo "==> Upgrade complete. The previous digest is in .env as ${IMAGE_KEY}_PREVIOUS."
        record upgrade "$CURRENT" "$DIGEST" "applied"
        exit 0
    fi
    echo "    The gateway check failed after the swap. Putting the old digest back." >&2
    set_env_value "$IMAGE_KEY" "$CURRENT"
    set_env_value "${IMAGE_KEY}_PREVIOUS" "$DIGEST"
    docker compose -f "$COMPOSE_FILE" up -d "$SERVICE"
    record upgrade "$CURRENT" "$DIGEST" "reverted-automatically"
    exit 1
fi

echo "    check-gateway.sh from Part 9 is not in this directory, so the swap was not"
echo "    verified. Run your own check now, and 'bash upgrade-engine.sh --rollback' if it"
echo "    is unwell."
record upgrade "$CURRENT" "$DIGEST" "applied-unverified"

Download upgrade-engine.sh212 lines

RunnableTrack N · NVIDIA GPU

upgrade, with the canary and the automatic rollback

bash upgrade-engine.sh \
    --image ghcr.io/mostlygeek/llama-swap:unified-cuda13 \
    --changelog changelog.md

RunnableTrack N · NVIDIA GPU

put yesterday back

bash upgrade-engine.sh --rollback

The script edits exactly two keys in .env, the image and its predecessor, and touches nothing else. That constraint is deliberate: an upgrade tool that rewrites your configuration is an upgrade tool you will stop trusting the first time it reformats something.

⌂Single-machine path

On a native installation there is no image to swap, and the same procedure applies with one substitution: build or install the new engine into a second directory, point a spare llama-swap at it on another port, check it, and then change the binary path in your configuration. Keep the old directory for a week. The shape is identical; only the thing being pinned changes from a digest to a path.

Reproducing this machine somewhere else

The manifest is what turns a pile of files into a description.

RunnableAll tracks

estate-manifest.py

#!/usr/bin/env python3
"""Write down exactly what your model estate is, so it can be rebuilt.

Purpose: produce one JSON file listing every model file with its size and hash, every
    engine with its version, every container image with its digest, and every
    configuration file with its hash, so that a new machine can be brought to the same
    state and an old one can be proved to have changed.
Platform: all (spark, strix, mac, nvidia). Pure Python, no dependencies.
Minimum memory: 8 GB, which is the estate being described; this script needs almost none.
Assumes: a model library laid out as Part 4 built it, the gateway directory from Part 9,
    and whichever engines are installed on PATH. Hashing a large model library reads every
    byte of it, so hashes are taken from the .sha256 sidecar files Part 4's downloader
    wrote where those exist, and computed only for files with --hash. Nothing is modified.

Usage: python3 estate-manifest.py --models ~/models --gateway ~/gateway
       python3 estate-manifest.py --models ~/models --gateway ~/gateway --output estate.json
       python3 estate-manifest.py --models ~/models --hash --output estate.json
"""
import argparse
import hashlib
import json
import platform
import re
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

WEIGHT_SUFFIXES = {".gguf", ".safetensors", ".bin", ".pt", ".pth", ".npz"}
CONFIG_SUFFIXES = {".yaml", ".yml", ".json", ".conf", ".toml", ".md", ".txt"}

# Engines the course installs, and the argument that makes each print its version. A tool
# that is not on PATH is recorded as absent rather than omitted, because "not installed" is
# part of the description of a machine.
ENGINE_VERSION_COMMANDS = [
    ("llama-server", ["llama-server", "--version"]),
    ("llama-swap", ["llama-swap", "--version"]),
    ("vllm", ["vllm", "--version"]),
    ("litellm", ["litellm", "--version"]),
    ("mlx_lm.server", ["mlx_lm.server", "--help"]),
    ("ollama", ["ollama", "--version"]),
    ("docker", ["docker", "--version"]),
]


def run(cmd, timeout=20):
    if shutil.which(cmd[0]) is None:
        return None
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
    except (OSError, subprocess.TimeoutExpired):
        return None
    text = (out.stdout or "") + (out.stderr or "")
    return text.strip().splitlines()[0] if text.strip() else ""


def sha256_of(path, chunk=1024 * 1024):
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for block in iter(lambda: handle.read(chunk), b""):
            digest.update(block)
    return digest.hexdigest()


def sidecar_hash(path):
    """Part 4's downloader writes <file>.sha256 beside each model file it verified."""
    sidecar = path.with_name(path.name + ".sha256")
    if not sidecar.is_file():
        return None
    text = sidecar.read_text(encoding="utf-8", errors="replace").strip()
    match = re.search(r"\b[0-9a-f]{64}\b", text)
    return match.group(0) if match else None


def describe_models(models_dir, compute_hashes, verbose):
    rows = []
    if models_dir is None:
        return rows
    root = Path(models_dir).expanduser()
    if not root.is_dir():
        print(f"estate-manifest: no such model directory: {root}", file=sys.stderr)
        return rows
    for path in sorted(root.rglob("*")):
        if not path.is_file() or path.suffix.lower() not in WEIGHT_SUFFIXES:
            continue
        stat = path.stat()
        digest = sidecar_hash(path)
        source = "sidecar" if digest else None
        if digest is None and compute_hashes:
            if verbose:
                print(f"    hashing {path.name} ...", file=sys.stderr)
            digest = sha256_of(path)
            source = "computed"
        rows.append({
            "path": str(path.relative_to(root)),
            "bytes": stat.st_size,
            "modified": datetime.fromtimestamp(stat.st_mtime, timezone.utc).isoformat(),
            "sha256": digest,
            "sha256_source": source or "not recorded",
        })
    return rows


def describe_engines():
    rows = []
    for name, cmd in ENGINE_VERSION_COMMANDS:
        version = run(cmd)
        rows.append({
            "tool": name,
            "present": version is not None,
            "version_line": version if version else "not installed",
        })
    return rows


def image_references(gateway_dir):
    """Every image name mentioned in the gateway's .env or compose files."""
    names = set()
    if gateway_dir is None:
        return names
    root = Path(gateway_dir).expanduser()
    for candidate in list(root.glob("*.yaml")) + list(root.glob("*.yml")) + [root / ".env"]:
        if not candidate.is_file():
            continue
        for line in candidate.read_text(encoding="utf-8", errors="replace").splitlines():
            match = re.search(r"(?:image:\s*|_IMAGE=)[\"']?([A-Za-z0-9._/-]+(?::[A-Za-z0-9._-]+)?)",
                              line)
            if not match:
                continue
            reference = match.group(1).strip("\"'")
            # A bare word is a Compose variable that was not expanded, not an image.
            if "/" in reference or ":" in reference:
                names.add(reference)
    return names


def describe_images(gateway_dir):
    rows = []
    if shutil.which("docker") is None:
        return rows
    for name in sorted(image_references(gateway_dir)):
        digest = run(["docker", "image", "inspect", "--format",
                      "{{if .RepoDigests}}{{index .RepoDigests 0}}{{end}}", name])
        rows.append({
            "reference": name,
            "digest": digest if digest else "not pulled on this machine",
        })
    return rows


def describe_configs(gateway_dir):
    rows = []
    if gateway_dir is None:
        return rows
    root = Path(gateway_dir).expanduser()
    if not root.is_dir():
        print(f"estate-manifest: no such gateway directory: {root}", file=sys.stderr)
        return rows
    for path in sorted(root.rglob("*")):
        if not path.is_file() or path.suffix.lower() not in CONFIG_SUFFIXES:
            continue
        # A secrets file is named here so that its absence from the list is deliberate
        # rather than accidental, and its contents are never read.
        if path.name.startswith(".env"):
            rows.append({"path": str(path.relative_to(root)), "sha256": "not recorded",
                         "note": "excluded on purpose: this file holds keys"})
            continue
        rows.append({"path": str(path.relative_to(root)), "sha256": sha256_of(path)})
    return rows


def describe_machine():
    return {
        "system": platform.system(),
        "release": platform.release(),
        "machine": platform.machine(),
        "python": platform.python_version(),
        "processor": platform.processor(),
    }


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--models", default=None, help="model library directory")
    parser.add_argument("--gateway", default=None, help="the Part 9 gateway directory")
    parser.add_argument("--output", default=None, help="write here instead of standard output")
    parser.add_argument("--hash", dest="compute_hashes", action="store_true",
                        help="compute SHA-256 for model files with no .sha256 beside them")
    parser.add_argument("--quiet", action="store_true", help="do not report progress")
    args = parser.parse_args()

    if args.models is None and args.gateway is None:
        parser.error("give --models, --gateway, or both; there is nothing to describe otherwise")

    manifest = {
        "written": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "machine": describe_machine(),
        "engines": describe_engines(),
        "images": describe_images(args.gateway),
        "configs": describe_configs(args.gateway),
        "models": describe_models(args.models, args.compute_hashes, not args.quiet),
    }
    manifest["summary"] = {
        "model_files": len(manifest["models"]),
        "model_bytes": sum(m["bytes"] for m in manifest["models"]),
        "model_files_without_hash": sum(1 for m in manifest["models"] if not m["sha256"]),
        "config_files": len(manifest["configs"]),
        "engines_present": sum(1 for e in manifest["engines"] if e["present"]),
    }

    text = json.dumps(manifest, indent=2, sort_keys=True)
    if args.output:
        Path(args.output).expanduser().write_text(text + "\n", encoding="utf-8")
        if not args.quiet:
            print(f"wrote {args.output}: {manifest['summary']['model_files']} model file(s), "
                  f"{manifest['summary']['config_files']} configuration file(s)")
            missing = manifest["summary"]["model_files_without_hash"]
            if missing:
                print(f"{missing} model file(s) have no recorded hash. Run again with --hash "
                      "when you have time to read every byte.")
    else:
        print(text)


if __name__ == "__main__":
    main()

Download estate-manifest.py227 lines

RunnableAll tracks

write down what this machine is

python3 estate-manifest.py \
    --models ~/models \
    --gateway ~/gateway \
    --output estate-manifest.json

It records every model file with its size and its hash, taking the hash from the .sha256 sidecars Part 4’s downloader wrote and computing it only when you ask with --hash; every engine on the path with the version it reports; every container image mentioned in the gateway’s files with the digest that is actually pulled; every configuration file with its hash; and the machine itself. Nothing is modified and no secret is read: .env appears in the list with a note saying it was excluded on purpose, which is better than being silently absent.

Rebuilding a machine from it is then a procedure rather than an archaeology:

1. Install the engines at the recorded versions.
2. Pull the images at the recorded digests.
3. Download the models at the recorded repositories and revisions, and verify the hashes against the manifest.
4. Restore the configuration from the backup archive and check the hashes match.
5. Recreate the secrets, which are the one category that is deliberately not in either file.
6. Run check-gateway.sh from Part 9 and check-monitoring.sh from this part’s lab.

The sixth step is the point of the first five. A rebuild is finished when the same checks pass, not when the files are in place.

Restoring, and the quarterly ten minutes

The two things that make all of the above real are a restore you have actually performed and a short review you actually repeat.

Restore into a scratch directory, not over the original. Extract the archive somewhere harmless and look at what came out. Does the configuration directory contain the files you expected? Does the manifest list the models you believe you have? Is the notebook the current one or one from before you last worked on it? Those three questions catch the two common backup faults, which are a script that has been silently failing since you changed a path, and a backup that was working perfectly on the wrong directory.

Compare hashes rather than eyeballing files. The manifest already holds the hash of every configuration file, so a restored copy either matches it or does not. That comparison takes seconds and it is the only way to know that the archive is intact rather than merely present.

Check that the secrets you deliberately excluded are somewhere. This is the step that turns an exclusion into a decision. If you cannot say, in one sentence and without looking, where your .env and your private key would come from after a disk failure, you have not finished designing the backup; you have only finished writing the script.

Then there is the review, which takes about ten minutes and belongs in the calendar rather than in your intentions. Four questions, once a quarter:

1. Has anything you pinned moved on? Look at the releases page for each engine and each image. You are not obliged to upgrade. You are obliged to know that you are three versions behind, so that the decision is a decision.
2. Does the model library still match its hashes? hf cache verify against the repositories you care about answers this, and it occasionally finds a file that a disk corrupted quietly months ago.
3. When does the certificate expire? If the answer is “before the next review”, renew it now rather than discovering it on the day.
4. Does the backup still restore? Run the restore into a scratch directory again. A backup script that broke when you moved a directory will otherwise be discovered at the worst possible time, and the whole point of this lesson is to move that discovery earlier.

✦Tip

Write the review’s four answers into the changelog as one entry with the date, even when all four are “unchanged”. A quarterly line saying nothing happened is how you find out, a year later, that the review stopped happening in March.

The changelog

One file, appended to, never reorganised. It is the cheapest tool in this part and the one you will be most grateful for.

What goes in it: every model that went behind an alias and when, with the quantisation and where it came from; every engine or image upgrade with the digest before and after; every configuration change that was not obvious; every limit you set and why; every incident, including the ones that turned out to be nothing; every key issued and revoked, by alias rather than by value; and the date of every certificate expiry you are relying on.

What does not go in it: anything secret, and anything you would have to maintain. A changelog that has to be kept tidy stops being kept.

✦Tip

The two scripts in this lesson both take a --changelog or --labbook argument and append a single JSON line. That is not a formatting preference: a line per event, appended, is the only record-keeping discipline that survives being busy. You can read it with your eyes and parse it with anything, and it never has a merge conflict.

Perform a restore that cannot rely on the original service

A backup is useful only if it contains enough state to reconstruct the intended service. Test restoration into an isolated directory or host with different temporary ports so the original service cannot accidentally answer the probes. Restore configuration, required credentials through the approved secret mechanism, model identities and persistent application data.

Check a known user-visible object and a representative model request. Compare hashes for immutable artefacts and inspect application-level records for databases. A successful archive extraction proves only that files were unpacked. Record elapsed restoration time and any undocumented manual step.

For upgrades, keep old and new configurations and run the same acceptance suite before promotion. Include template-dependent features, not just a health endpoint. Define rollback before changing the active alias. Some state migrations are not reversible by swapping an image tag, so test the documented data rollback path too. Retain the evidence that the restored or rolled-back service works independently of the machine you hoped never to lose.

Recap

The big files are replaceable and the small ones are not: configuration, adapters, evaluation sets and the notebook are what you cannot get back, and weights are replaceable if you recorded the repository, revision and hash. Back up the second category, exclude secrets on purpose, and restore it once so that it is a fact rather than a hope. Pin engines by version, images by digest, and models by revision, accepting the documented trade that a pinned digest does not receive updates until you move it. Upgrade by starting the new image beside the live one, checking it, swapping, re-running the check, and rolling back automatically when the check fails. Keep a manifest that describes the machine well enough to rebuild it, and finish every rebuild by running the same checks rather than by looking at the files. And append one line to the changelog every time something changes, because the question you will ask in six months is always “when did this start”, and there is no other way to answer it.

Check your understanding

Question 1. Your disk fails. Which of these is the most expensive thing to have lost?

• The model weights, because they are the largest
• Your own evaluation set, because it is the only thing that can tell you whether a new model is better for your work and it exists nowhere else
• The container images
• The Prometheus time series

Show the answer and why

Answer: Your own evaluation set, because it is the only thing that can tell you whether a new model is better for your work and it exists nowhere else

Weights are a download away if you recorded the repository, revision and hash. An evaluation set built from your own tasks is small, irreplaceable, and the thing every future model decision depends on.

Question 2. What is the documented trade-off when you pin a container image by digest instead of by tag?

• Digests are slower to pull
• You get exactly the same image every time, and you no longer receive updated versions, which may include security updates, until you change the digest yourself
• Digests cannot be used with Compose
• There is no trade-off; digests are strictly better

Show the answer and why

Answer: You get exactly the same image every time, and you no longer receive updated versions, which may include security updates, until you change the digest yourself

Docker documents both halves plainly. Reproducibility and update responsibility are the same decision viewed from two sides, which is why the changelog entry for a digest change matters as much as the pin itself.

Question 3. Why does the upgrade script start the new image without the model directory by default?

• Because the new image cannot read models
• So the canary starts without loading weights, which means a second copy of a model is not needed; most machines do not have room for one, so the deep check that does load a model is opt-in
• To make the upgrade faster to type
• Because model files must never be mounted twice

Show the answer and why

Answer: So the canary starts without loading weights, which means a second copy of a model is not needed; most machines do not have room for one, so the deep check that does load a model is opt-in

The two questions the canary answers cheaply are whether the new image becomes healthy and whether it reads your configuration the same way. Both are answered without any weights, and they catch the fault that actually occurs: a configuration format that changed between versions.

Question 4. Which of these belong in the changelog? Select all that apply.

• The digest of the image before and after an upgrade
• The date a certificate you depend on expires
• The value of a key you issued
• An incident that turned out to be nothing

Show the answer and why

Answer: The digest of the image before and after an upgrade, The date a certificate you depend on expires, An incident that turned out to be nothing

Digests, expiries and incidents are all answers to the question "when did this start". A key value is a secret and belongs in the environment; the changelog records that a key with a given alias was issued and later revoked, which is what you actually need.

Check my answersClear