Level 2 · Local OperatorProjectPart 09 · page 8 of 875 minSXMN 8 GB

75Minutes

4Tools

10Sources

All fourTracks

Tools used on this page4

Project: Your Local Model Gateway

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The image tags, versions and per-track notes each track was run with will be recorded here when the validation pass is done.

Objective

By the end of this project your machine will have one address that every piece of software in the rest of this course talks to. Behind it, models load when they are asked for and unload when they are not; three stable names hide which engine and which file are actually answering; a key is required; a failed model falls back to another one instead of returning an error; health is checkable; and what was used is logged. You will also have written it all down in a file that Parts 10, 13, 24, 25 and the capstone are going to read.

This is the last page of Level 2, and it is placed here deliberately. Everything after it assumes a stable endpoint. The alternative, which is what most people do, is remembering that the coding model is on port 8081 unless you restarted it, and that the embedding server is the one you have to start by hand. That works until it does not, and it stops working exactly when you are trying to debug something else.

How the pieces fit

Two programs, each doing one job.

llama-swap watches for a request naming a model, works out whether the right engine is running, and if not, stops what is running and starts the right one. Its README describes it as “reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc.” It is a single Go binary with no dependencies, and its whole configuration is a YAML file listing model ids and the command line that starts each.

LiteLLM sits in front and is the thing you actually talk to. It gives you stable aliases, authentication, retries, fallbacks, health checks, usage records and a second API shape, without any of that being the engines’ problem.

The gateway, from a client's request down to the weights

ClientsYour editor, an agent, an evaluation harness, a notebook, curlknows only a name and a key
LiteLLM proxyOne address, one key per application. Aliases, retries, fallbacks, health checks, usage records, OpenAI and Anthropic shapesyou configure this
llama-swapModel ids to command lines. Starts and stops engines on demand; groups decide what may be loaded at the same timeyou configure this
Enginesllama-server processes, one per loaded model, each on its own port; a vLLM or MLX server can be added beside them
Model filesGGUF files in ~/models from Part 4, mounted read-onlynever written to
Accelerator and memoryThe one physical resource all of the above are competing for

Each layer only knows the one below it by an address. That is what lets you swap a model file, or put vLLM behind an alias instead of llama-server, without touching a single client.

The load-bearing idea is the alias. A client asks for local/coder. It does not know which model that is, which engine serves it, which port it is on, or whether it was even running a second ago. When you upgrade the coder model in six months you change one line of YAML, and everything that ever used the name keeps working.

Requirements

Every track needs the model library from Part 4, keys you generate yourself, and about seventy-five minutes. The three models this project wires up are Qwen3-8B for chat, Qwen3-Coder-30B-A3B for code and Qwen3-Embedding-0.6B for retrieval; all three are Apache-2.0 licensed and ungated, and the model reference records the licence for each.

Memory. The floor is 8 GB, because llama-swap loads one accelerator-backed model at a time. At 8 GB run the chat and embedding models and leave the coder model out of llama-swap.yaml until you have the memory for it; the file is commented so you can see exactly what to remove.

Time. About seventy-five minutes of attended work, plus whatever the downloads take if you do not already have the three model files.

Track S — NVIDIA DGX Spark

Container path. Use compose.yaml with the compose-nvidia.yaml override, and set LLAMA_SWAP_IMAGE to the unified-cuda13 tag, which the llama-swap README describes as multi-architecture and covering NVIDIA Ampere through Blackwell.

128 GB of unified memory means all three models can be resident at once if you want them to be. Do the swapping configuration first anyway, because understanding what swapping costs is the point of the exercise, and then move the models you use constantly into a persistent group.

Track X — AMD Ryzen AI Max+ 395

Container path with the compose-amd.yaml override, and LLAMA_SWAP_IMAGE set to the unified-vulkan tag, which the README lists for AMD and other Vulkan-capable GPUs. Check with id -nG that your user is in the video and render groups before the first run.

Remember Part 5’s GPU-visible memory cap. The machine’s total is not what the engines may use, and the model-fitting decisions in llama-swap.yaml come out of the smaller number.

Track M — Apple silicon

Native path. start-gateway-native.sh runs the same two programs without containers, using the Metal-built llama-server from Part 6.

One thing is unavailable on this path: virtual keys and usage records need a Postgres database, which the container path provides and this script does not start. Your gateway has one key, the master key, and the tasks below say which steps to skip. If you want the full set, install Postgres locally, set DATABASE_URL, and every step applies.

Track N — NVIDIA desktop or laptop

Container path with the compose-nvidia.yaml override. Pick the llama-swap tag for your card: the README lists unified-cuda13 for Ampere through Blackwell and unified-cuda for older Pascal through Ada cards.

On a 16 GB card the coder model at Q4_K_M does not fit alongside a large context. Start with chat and embeddings, get the whole thing working, and add the coder model afterwards with a context length you have checked.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-09-vllm-and-sglang"
cd "$LAB_DIR"
pwd
test -f "env-example.txt"

Expected result: pwd ends in part-09-vllm-and-sglang and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Take the files and read them

Six configuration files and two scripts. Read each one before you run it; they are commented as the explanation, and the page does not repeat what the comments say.

RunnableAll tracks

# Purpose: the settings compose.yaml, litellm-config.yaml, llama-swap.yaml and the native
#          start script all read. Copy this file to `.env` beside compose.yaml and fill in
#          the lines that are empty. Nothing here is a secret except the two keys, which
#          you generate rather than copy.
# Platform: all
# Minimum memory: 8 GB
# Assumes: `cp env-example.txt .env` and then an editor. Docker Compose reads `.env`
#          automatically from the directory you run it in; the native start script reads
#          it with `set -a; . ./.env; set +a`.

# ---------------------------------------------------------------- where things listen
# The gateway is the only thing anything else talks to. Keep it on the loopback address
# until Part 23 puts authentication and TLS in front of it properly.
GATEWAY_HOST=127.0.0.1
GATEWAY_PORT=4000

# llama-swap's own port. Nothing outside this stack should need it; it is published only
# so you can look at its web page while you are setting the gateway up.
SWAP_HOST=127.0.0.1
SWAP_PORT=9292

# ------------------------------------------------------------------------ your models
# Absolute path to the model library you built in Part 4. llama-swap reads GGUF files
# from here; the container path mounts it read-only.
MODELS_DIR=

# Absolute path to the llama-server binary, for the native path only. The container image
# brings its own. Leave it empty if you are using Compose.
LLAMA_BIN=

# ------------------------------------------------------------------------------- keys
# Generate with: openssl rand -hex 24
# The proxy admin key. It must begin with sk-, so write it as sk-<the hex you generated>.
# It is the key that creates other keys; it is not the key your applications should use.
LITELLM_MASTER_KEY=

# Generate with: openssl rand -hex 24
# Used to encrypt stored credentials. Changing it invalidates what is already stored.
LITELLM_SALT_KEY=

# The key the gateway presents to llama-swap, if you set one there. Leave it empty when
# llama-swap is only reachable from inside the stack, which is the default here.
LOCAL_API_KEY=

# ------------------------------------------------------------------- the spend database
# Virtual keys and usage records need a database. These values are read only by the
# container path; the native path runs without a database and the page says what you
# lose. Generate the password with: openssl rand -hex 16
GATEWAY_DB_USER=gateway
GATEWAY_DB_PASSWORD=
GATEWAY_DB_NAME=gateway

# ---------------------------------------------------------------------------- images
# llama-swap publishes one image with several tags. Pick the one for your accelerator:
#   unified-cuda13   multi-architecture, NVIDIA Ampere through Blackwell  (Tracks S and N)
#   unified-cuda     amd64 only, older NVIDIA Pascal through Ada          (Track N)
#   unified-vulkan   amd64, AMD and other Vulkan-capable GPUs             (Track X)
LLAMA_SWAP_IMAGE=ghcr.io/mostlygeek/llama-swap:unified-cuda13

# LiteLLM's deployment page says to pin a version tag rather than a moving one so that
# rollbacks are deterministic. This course pins LiteLLM 1.100.0; confirm the tag exists
# on the registry before your first run, and change it here rather than in compose.yaml.
LITELLM_IMAGE=ghcr.io/berriai/litellm:v1.100.0

POSTGRES_IMAGE=postgres:17-alpine

Download env-example.txt65 lines

RunnableAll tracks

# Purpose: teach llama-swap which models exist, how to start each one, and when to unload
#          it, so that a request naming a model causes the right engine to be running
# Platform: all (spark, strix, mac, nvidia). The container path mounts this file read-only
#           at /app/config.yaml; the native path passes it with --config.
# Minimum memory: 8 GB, because only one accelerator-backed model is loaded at a time
# Assumes: GGUF files under the directory MODELS_DIR points at, laid out as Part 4 built it,
#          and a llama-server binary that LLAMA_BIN points at on the native path. Every
#          ${env.NAME} below is read from the process environment, so the same file works
#          for the container and native paths with different values. The three file names
#          below are the ones this course's model library produces; list your own
#          directory and correct them before the first run rather than after it.

healthCheckTimeout: 300
logLevel: info

# Base for the ${PORT} macro. Each model gets the next free port from here, so no two
# engines can collide and nothing has to be assigned by hand.
startPort: 10001

macros:
  # Everything every model shares. Keeping it here means a change to the offload policy
  # is one edit rather than four.
  server: >-
    ${env.LLAMA_BIN}
    --host 127.0.0.1
    --port ${PORT}
    --n-gpu-layers 99
    --flash-attn on
    --metrics

  models-dir: ${env.MODELS_DIR}

models:
  # ------------------------------------------------------------------ the chat model
  # The name on the left is what a client sends. Keep these stable: every later part of
  # the course refers to these three names, and GATEWAY.md is where you record them.
  local/chat:
    name: General chat
    description: Qwen3-8B at Q4_K_M, four slots, sixteen thousand tokens of shared context
    cmd: |
      ${server}
      --model ${models-dir}/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
      --alias local/chat
      --ctx-size 16384
      --parallel 4
      --cont-batching
      --cache-type-k q8_0
      --cache-type-v q8_0
      --jinja
    checkEndpoint: /health
    # Unload after fifteen idle minutes. A desktop machine gets its memory back; a
    # machine that serves all day should raise this or move the model into a
    # persistent group.
    ttl: 900

  # ----------------------------------------------------------------- the coding model
  local/coder:
    name: Coding assistant
    description: Qwen3-Coder-30B-A3B at Q4_K_M, longer context, two slots
    cmd: |
      ${server}
      --model ${models-dir}/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
      --alias local/coder
      --ctx-size 32768
      --parallel 2
      --cont-batching
      --cache-type-k q8_0
      --cache-type-v q8_0
      --jinja
    checkEndpoint: /health
    ttl: 900

  # -------------------------------------------------------------- the embedding model
  # Small, cheap to keep resident, and asked for in short bursts by whatever is indexing.
  local/embed:
    name: Embeddings
    description: Qwen3-Embedding-0.6B, resident, used by the retrieval work in Part 10
    cmd: |
      ${server}
      --model ${models-dir}/Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf
      --alias local/embed
      --ctx-size 8192
      --embeddings
    checkEndpoint: /health
    ttl: 3600

groups:
  # The two big models share one accelerator, so only one may be loaded at a time and
  # loading either evicts the other. This is the whole point of llama-swap: a request for
  # local/coder unloads local/chat and starts the coder, without you doing anything.
  accelerator:
    swap: true
    exclusive: true
    members:
      - local/chat
      - local/coder

  # The embedding model is small enough to sit alongside whichever big model is loaded,
  # so it lives in its own group and is not swapped out by the others.
  resident:
    swap: false
    exclusive: false
    persistent: true
    members:
      - local/embed

Download llama-swap.yaml105 lines

RunnableAll tracks

# Purpose: the router half of the gateway. Turns three stable model aliases into calls to
#          whatever engine is behind them, adds retries and fallbacks, and gives the whole
#          machine one authenticated endpoint in both the OpenAI and Anthropic shapes.
# Platform: all (spark, strix, mac, nvidia). Passed to the proxy with --config.
# Minimum memory: 8 GB; the proxy itself is a small Python process and the engines behind
#          it are what use the machine.
# Assumes: llama-swap listening at LLAMA_SWAP_BASE_URL with the model ids from
#          llama-swap.yaml, and LITELLM_MASTER_KEY set in the environment. Every value
#          written as os.environ/NAME is read from the environment at runtime, so no key
#          is ever stored in this file.

model_list:
  # The name on the left is what your applications send for the rest of this course.
  # The model on the right is prefixed openai/ because llama-swap presents an
  # OpenAI-compatible interface; the part after the prefix is the id llama-swap knows.
  - model_name: local/chat
    litellm_params:
      model: openai/local/chat
      api_base: os.environ/LLAMA_SWAP_BASE_URL
      api_key: os.environ/LOCAL_API_KEY

  - model_name: local/coder
    litellm_params:
      model: openai/local/coder
      api_base: os.environ/LLAMA_SWAP_BASE_URL
      api_key: os.environ/LOCAL_API_KEY

  - model_name: local/embed
    litellm_params:
      model: openai/local/embed
      api_base: os.environ/LLAMA_SWAP_BASE_URL
      api_key: os.environ/LOCAL_API_KEY

  # A second route to the chat model, bypassing llama-swap and pointing straight at a
  # server you started by hand. It exists so that the fallback below has somewhere to go
  # when llama-swap itself is the thing that is unwell, and so that you can point one
  # alias at vLLM instead while you compare them. Comment it out if you are not running
  # a direct server; an unreachable deployment is a slow failure rather than a fast one.
  - model_name: local/chat-direct
    litellm_params:
      model: openai/local-chat
      api_base: os.environ/DIRECT_BASE_URL
      api_key: os.environ/LOCAL_API_KEY

litellm_settings:
  # Retry twice on the same alias before giving up on it.
  num_retries: 2
  # A local model that is still loading can take a long time to answer the first request.
  request_timeout: 600
  # Where each alias goes when its own retries are exhausted. The coder falls back to the
  # chat model, which is worse at code and much better than an error; the chat model falls
  # back to the direct route.
  fallbacks:
    - local/coder: ["local/chat"]
    - local/chat: ["local/chat-direct"]
  # Take a deployment out of rotation after three failures in a minute, for a minute.
  allowed_fails: 3
  cooldown_time: 60
  # Log the fact of every call. Set to true only if you are certain you want prompt and
  # response text kept, which on a shared machine you usually do not.
  turn_off_message_logging: true

general_settings:
  # Run health checks on a timer and serve the cached result, so that hitting /health does
  # not fire a real request at every model every time something polls it.
  background_health_checks: true
  health_check_interval: 300
  health_check_details: true

# The proxy reads LITELLM_MASTER_KEY, LITELLM_SALT_KEY and DATABASE_URL from the
# environment. The master key is the admin key that creates virtual keys; it is not the
# key your applications should carry. Virtual keys and the spend records behind them need
# DATABASE_URL to point at a Postgres database, which the container path provides and the
# native path does not.

Download litellm-config.yaml74 lines

RunnableAll tracks

# Purpose: the gateway stack - llama-swap loading GGUF models on demand, LiteLLM routing
#          stable aliases to them with keys, retries, fallbacks and health checks, and a
#          Postgres database holding virtual keys and usage records
# Platform: spark, strix, nvidia (Linux with Docker Engine and the Compose plugin).
#           Track M runs the same two processes natively; see start-gateway-native.sh.
# Minimum memory: 8 GB
# Assumes: a .env file beside this one, copied from env-example.txt and filled in, plus
#          llama-swap.yaml and litellm-config.yaml beside it. Add an accelerator override:
#            docker compose -f compose.yaml -f compose-nvidia.yaml up -d   (Tracks S, N)
#            docker compose -f compose.yaml -f compose-amd.yaml up -d      (Track X)
#          Without an override the engines run on the CPU, which works and is slow.

name: local-gateway

services:
  swap:
    image: "${LLAMA_SWAP_IMAGE}"
    restart: unless-stopped
    command:
      - "--config"
      - "/app/config.yaml"
      - "--listen"
      - "0.0.0.0:8080"
    environment:
      # The unified llama-swap images ship a llama-server binary on the PATH. Confirm it
      # before the first run with:
      #   docker run --rm --entrypoint sh "$LLAMA_SWAP_IMAGE" -c 'command -v llama-server'
      # and set an absolute path here if your image puts it somewhere else.
      LLAMA_BIN: llama-server
      MODELS_DIR: /models
    volumes:
      - ./llama-swap.yaml:/app/config.yaml:ro
      # Read-only on purpose: the gateway has no business writing to your model library.
      - "${MODELS_DIR}:/models:ro"
      - swap-cache:/root/.cache
    ports:
      # Published on one address only, and only so you can open llama-swap's own web page
      # while setting the gateway up. Nothing else needs to reach it: the router talks to
      # it over the private network below.
      - "${SWAP_HOST}:${SWAP_PORT}:8080"
    networks:
      - private

  gateway:
    image: "${LITELLM_IMAGE}"
    restart: unless-stopped
    depends_on:
      - swap
      - db
    command:
      - "--config"
      - "/app/config.yaml"
      - "--port"
      - "4000"
    environment:
      LLAMA_SWAP_BASE_URL: "http://swap:8080/v1"
      # The direct route the fallback in litellm-config.yaml uses. Point it at a server you
      # started by hand, or at the same llama-swap instance if you have no second engine.
      DIRECT_BASE_URL: "http://swap:8080/v1"
      LOCAL_API_KEY: "${LOCAL_API_KEY}"
      LITELLM_MASTER_KEY: "${LITELLM_MASTER_KEY}"
      LITELLM_SALT_KEY: "${LITELLM_SALT_KEY}"
      DATABASE_URL: "postgresql://${GATEWAY_DB_USER}:${GATEWAY_DB_PASSWORD}@db:5432/${GATEWAY_DB_NAME}"
      STORE_MODEL_IN_DB: "True"
    volumes:
      - ./litellm-config.yaml:/app/config.yaml:ro
    ports:
      # This is the one endpoint everything else in the course talks to.
      - "${GATEWAY_HOST}:${GATEWAY_PORT}:4000"
    networks:
      - private

  db:
    image: "${POSTGRES_IMAGE}"
    restart: unless-stopped
    environment:
      POSTGRES_USER: "${GATEWAY_DB_USER}"
      POSTGRES_PASSWORD: "${GATEWAY_DB_PASSWORD}"
      POSTGRES_DB: "${GATEWAY_DB_NAME}"
    volumes:
      - gateway-db:/var/lib/postgresql/data
    # No published port. The database is reachable only from the private network, which is
    # the whole of its security model here.
    networks:
      - private
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s

networks:
  private:
    driver: bridge

volumes:
  swap-cache:
  gateway-db:

Download compose.yaml99 lines

RunnableTrack N · NVIDIA GPU

# Purpose: override for compose.yaml that gives the llama-swap container the machine's
#          NVIDIA GPUs. Used on Track S (DGX Spark, aarch64) and Track N.
# Platform: spark, nvidia
# Minimum memory: 8 GB
# Assumes: the NVIDIA Container Toolkit is installed and `nvidia-ctk runtime configure`
#          has been run, so `docker run --gpus=all` already works on this host, and that
#          LLAMA_SWAP_IMAGE in .env is a CUDA tag rather than the CPU or Vulkan one.
#
# Usage: docker compose -f compose.yaml -f compose-nvidia.yaml up -d

services:
  swap:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    # Several inference paths pass data between processes through the host's shared memory.
    # Without this the container starts and then fails in ways that look like a model fault.
    ipc: host

Download compose-nvidia.yaml22 lines

RunnableTrack X · Ryzen AI Max+

# Purpose: override for compose.yaml that gives the llama-swap container the machine's AMD
#          GPU. Used on Track X.
# Platform: strix
# Minimum memory: 8 GB
# Assumes: an AMD driver stack on the host, that the container is allowed to open /dev/kfd
#          and /dev/dri, and that LLAMA_SWAP_IMAGE in .env is set to the Vulkan tag rather
#          than a CUDA one. On SELinux systems run `sudo setsebool container_use_devices=1`
#          first. Check your own group membership with `id -nG` before assuming this works.
#
# Usage: docker compose -f compose.yaml -f compose-amd.yaml up -d

services:
  swap:
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video
      - render
    ipc: host

Download compose-amd.yaml20 lines

RunnableTrack M · Apple silicon

#!/usr/bin/env bash
# Purpose: run the gateway without containers - llama-swap loading models on demand and
#          LiteLLM routing aliases in front of it - and stop both cleanly on Ctrl-C
# Platform: mac primarily, and any track that would rather not run containers. The
#           container path (compose.yaml) is the one Tracks S, X and N normally use.
# Minimum memory: 8 GB
# Assumes: llama-swap and litellm on PATH, llama-server built as in Part 6, and a .env
#          beside this script copied from env-example.txt with MODELS_DIR and LLAMA_BIN
#          filled in. No database is started here, so virtual keys and spend records are
#          unavailable on this path; the master key is the only key. Logs are written to
#          ./gateway-logs/.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"
LOG_DIR="${LOG_DIR:-${HERE}/gateway-logs}"

if [ ! -f "$ENV_FILE" ]; then
    echo "No .env found at ${ENV_FILE}." >&2
    echo "Copy env-example.txt to .env and fill in MODELS_DIR, LLAMA_BIN and the keys." >&2
    exit 1
fi

set -a
# shellcheck source=/dev/null
. "$ENV_FILE"
set +a

: "${GATEWAY_HOST:=127.0.0.1}"
: "${GATEWAY_PORT:=4000}"
: "${SWAP_HOST:=127.0.0.1}"
: "${SWAP_PORT:=9292}"

for required in MODELS_DIR LLAMA_BIN LITELLM_MASTER_KEY; do
    if [ -z "${!required:-}" ]; then
        echo "${required} is empty in ${ENV_FILE}. Fill it in and run this again." >&2
        exit 1
    fi
done

for binary in llama-swap litellm; do
    if ! command -v "$binary" >/dev/null 2>&1; then
        echo "${binary} is not on PATH." >&2
        echo "See the project page for how to install it on your track." >&2
        exit 1
    fi
done

if [ ! -x "$LLAMA_BIN" ]; then
    echo "LLAMA_BIN points at ${LLAMA_BIN}, which is not an executable file." >&2
    exit 1
fi

if [ ! -d "$MODELS_DIR" ]; then
    echo "MODELS_DIR points at ${MODELS_DIR}, which is not a directory." >&2
    exit 1
fi

# The router needs to know where llama-swap is. Both are exported so that
# litellm-config.yaml's os.environ/ lookups resolve.
export LLAMA_SWAP_BASE_URL="http://${SWAP_HOST}:${SWAP_PORT}/v1"
export DIRECT_BASE_URL="${DIRECT_BASE_URL:-http://${SWAP_HOST}:${SWAP_PORT}/v1}"
export LOCAL_API_KEY="${LOCAL_API_KEY:-}"

mkdir -p "$LOG_DIR"

SWAP_PID=""
GATEWAY_PID=""

stop_all() {
    echo ""
    echo "==> stopping"
    if [ -n "$GATEWAY_PID" ] && kill -0 "$GATEWAY_PID" 2>/dev/null; then
        kill "$GATEWAY_PID" 2>/dev/null || true
        wait "$GATEWAY_PID" 2>/dev/null || true
    fi
    if [ -n "$SWAP_PID" ] && kill -0 "$SWAP_PID" 2>/dev/null; then
        kill "$SWAP_PID" 2>/dev/null || true
        wait "$SWAP_PID" 2>/dev/null || true
    fi
    echo "    both processes stopped. Logs are in ${LOG_DIR}."
}
trap stop_all EXIT INT TERM

echo "==> llama-swap on http://${SWAP_HOST}:${SWAP_PORT}"
llama-swap \
    --config "${HERE}/llama-swap.yaml" \
    --listen "${SWAP_HOST}:${SWAP_PORT}" \
    >>"${LOG_DIR}/llama-swap.log" 2>&1 &
SWAP_PID=$!

# Give it a moment to bind before the router starts probing it.
for _ in $(seq 1 30); do
    if ! kill -0 "$SWAP_PID" 2>/dev/null; then
        echo "llama-swap exited immediately. Last lines of its log:" >&2
        tail -n 20 "${LOG_DIR}/llama-swap.log" >&2 || true
        exit 1
    fi
    if command -v curl >/dev/null 2>&1 &&
        curl -sf "http://${SWAP_HOST}:${SWAP_PORT}/health" >/dev/null 2>&1; then
        break
    fi
    sleep 1
done

echo "==> LiteLLM on http://${GATEWAY_HOST}:${GATEWAY_PORT}"
litellm \
    --config "${HERE}/litellm-config.yaml" \
    --host "$GATEWAY_HOST" \
    --port "$GATEWAY_PORT" \
    >>"${LOG_DIR}/litellm.log" 2>&1 &
GATEWAY_PID=$!

cat <<INFO

    gateway   http://${GATEWAY_HOST}:${GATEWAY_PORT}
    swap      http://${SWAP_HOST}:${SWAP_PORT}
    logs      ${LOG_DIR}/

    Check it with:  bash check-gateway.sh
    Stop it with:   Ctrl-C in this terminal

INFO

wait

Download start-gateway-native.sh125 lines

RunnableAll tracks

#!/usr/bin/env bash
# Purpose: prove the gateway is doing its job - liveness, readiness, the three model
#          aliases, an OpenAI-shaped call, an embedding, an Anthropic-shaped call, and a
#          rejected anonymous request - and print a pass or fail line for each
# Platform: all (spark, strix, mac, nvidia)
# Minimum memory: 8 GB, which is what the models behind the gateway need
# Assumes: curl on PATH, the gateway running, and a .env beside this script holding
#          LITELLM_MASTER_KEY. The key is read from the environment and never printed.
#          The first model call loads a model, so allow a minute for it on a cold start.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
fi

: "${GATEWAY_HOST:=127.0.0.1}"
: "${GATEWAY_PORT:=4000}"
BASE="http://${GATEWAY_HOST}:${GATEWAY_PORT}"
KEY="${LITELLM_MASTER_KEY:-}"

if [ -z "$KEY" ]; then
    echo "LITELLM_MASTER_KEY is not set. Put it in ${ENV_FILE} or export it." >&2
    exit 1
fi

if ! command -v curl >/dev/null 2>&1; then
    echo "curl is not on PATH." >&2
    exit 1
fi

PASSES=0
FAILURES=0

report() {
    # report <name> <ok:0|1> [detail]
    if [ "$2" -eq 0 ]; then
        PASSES=$((PASSES + 1))
        printf '  PASS  %s\n' "$1"
    else
        FAILURES=$((FAILURES + 1))
        printf '  FAIL  %s%s\n' "$1" "${3:+  (${3})}"
    fi
}

check_contains() {
    # check_contains <name> <body> <needle>
    if printf '%s' "$2" | grep -q -- "$3"; then
        report "$1" 0
    else
        report "$1" 1 "did not contain ${3}"
    fi
}

echo "==> checking ${BASE}"

# 1. Liveness: the process is up. No dependencies are checked.
if curl -sf "${BASE}/health/liveliness" >/dev/null; then
    report "liveness endpoint answers" 0
else
    report "liveness endpoint answers" 1 "is the gateway running?"
    echo "  Nothing else can pass while the gateway is down. Stopping here." >&2
    exit 1
fi

# 2. Readiness: the worker is ready to accept traffic.
if curl -sf "${BASE}/health/readiness" >/dev/null; then
    report "readiness endpoint answers" 0
else
    report "readiness endpoint answers" 1 "check the database connection on the container path"
fi

# 3. The three aliases every later part of the course expects.
MODELS_BODY="$(curl -sf -H "Authorization: Bearer ${KEY}" "${BASE}/v1/models" || true)"
for alias_name in "local/chat" "local/coder" "local/embed"; do
    check_contains "alias ${alias_name} is published" "$MODELS_BODY" "$alias_name"
done

# 4. An OpenAI-shaped chat completion. This is the call that loads a model, so it is the
#    slow one on a cold start.
CHAT_BODY="$(curl -sf --max-time 300 "${BASE}/v1/chat/completions" \
    -H "Authorization: Bearer ${KEY}" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat",
         "messages":[{"role":"user","content":"Reply with one word: ready"}],
         "max_tokens":8}' || true)"
check_contains "OpenAI-shaped chat completion returns content" "$CHAT_BODY" '"content"'

# 5. An embedding, which proves the second engine is reachable and the router picked it.
EMBED_BODY="$(curl -sf --max-time 300 "${BASE}/v1/embeddings" \
    -H "Authorization: Bearer ${KEY}" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/embed","input":"a sentence to embed"}' || true)"
check_contains "embeddings endpoint returns a vector" "$EMBED_BODY" '"embedding"'

# 6. The Anthropic-shaped surface, over the same alias and the same engine.
MSG_BODY="$(curl -sf --max-time 300 "${BASE}/v1/messages" \
    -H "x-api-key: ${KEY}" \
    -H "anthropic-version: 2023-06-01" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat",
         "messages":[{"role":"user","content":"Reply with one word: ready"}],
         "max_tokens":8}' || true)"
check_contains "Anthropic-shaped messages endpoint answers" "$MSG_BODY" '"content"'

# 7. The gateway must refuse an unauthenticated request. A gateway that answers without a
#    key is not a gateway.
ANON_STATUS="$(curl -s -o /dev/null -w '%{http_code}' "${BASE}/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat","messages":[{"role":"user","content":"hello"}],"max_tokens":4}' || true)"
if [ "$ANON_STATUS" = "401" ] || [ "$ANON_STATUS" = "403" ]; then
    report "anonymous request is refused" 0
else
    report "anonymous request is refused" 1 "got HTTP ${ANON_STATUS}"
fi

echo ""
echo "    ${PASSES} passed, ${FAILURES} failed"
if [ "$FAILURES" -gt 0 ]; then
    echo "    See gateway-logs/ on the native path, or 'docker compose logs' on the" >&2
    echo "    container path, for what the router and the engines actually said." >&2
    exit 1
fi

Download check-gateway.sh128 lines

Put them all in one directory. That directory is your gateway from now on; it is worth putting under version control, with .env excluded.

2. Check the model paths, before anything else

llama-swap.yaml names three files. They are the ones this course’s model library produces, and the names on your disk may differ by a quantisation or a repository revision.

RunnableAll tracks

ls -1 ~/models/unsloth/Qwen3-8B-GGUF/
ls -1 ~/models/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF/
ls -1 ~/models/Qwen/Qwen3-Embedding-0.6B-GGUF/

Correct the three --model paths in llama-swap.yaml to match. A wrong path produces an engine that exits immediately, and llama-swap will report it as a model that will not start, which is a longer route to the same answer.

3. The environment file and the keys

RunnableAll tracks

cp env-example.txt .env

openssl rand -hex 24
openssl rand -hex 16

Edit .env. Set MODELS_DIR to the absolute path of your model library. Set LITELLM_MASTER_KEY to sk- followed by the first hex string: LiteLLM’s documentation states that the master key “must begin with sk-”. Set LITELLM_SALT_KEY to a second hex string, and note that the deployment page describes it as encrypting stored credentials and says to “set once, never change it”. Set the database password to the shorter hex string. On the native path, also set LLAMA_BIN to your llama-server binary.

4. Start llama-swap on its own, and watch it swap

Get the bottom half working before adding the top half. On the container path, bring up only the model service; on the native path, run llama-swap by hand.

RunnableTrack N · NVIDIA GPU

docker compose -f compose.yaml -f compose-nvidia.yaml up -d swap

docker compose logs -f swap

RunnableTrack M · Apple silicon

MODELS_DIR="$HOME/models" LLAMA_BIN="$HOME/llama.cpp/build/bin/llama-server" \
    llama-swap --config llama-swap.yaml --listen 127.0.0.1:9292

Then ask it for a model, and watch the log while you do.

RunnableAll tracks

curl -s http://127.0.0.1:9292/v1/models

curl -s http://127.0.0.1:9292/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat",
         "messages":[{"role":"user","content":"Reply with one word: ready"}],
         "max_tokens":8}'

llama-swap also serves its own pages: the README lists /ui, /logs, /health, /metrics and /api/models/unload alongside the OpenAI, Anthropic and llama-server endpoints. Open the web page in a browser now; it is the fastest way to see what is loaded and what is not.

5. Add the router

RunnableTrack N · NVIDIA GPU

docker compose -f compose.yaml -f compose-nvidia.yaml up -d

docker compose ps

RunnableTrack M · Apple silicon

bash start-gateway-native.sh

LiteLLM’s deployment page gives the proxy’s default port as 4000, which is what compose.yaml publishes and what .env sets. From here on, port 4000 is the only address anything needs.

RunnableAll tracks

curl -s http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat",
         "messages":[{"role":"user","content":"Reply with one word: ready"}],
         "max_tokens":8}'

The openai/ prefix in litellm-config.yaml is the piece that makes this work. LiteLLM’s configuration page documents that prefix as the way to route to an OpenAI-compatible endpoint, so openai/local/chat with an api_base pointing at llama-swap means “speak the OpenAI protocol to that address, asking for the model called local/chat”.

6. Virtual keys

Skip this step on the native path, which has no database; the rest of the project works without it.

LiteLLM’s virtual keys page states the requirements plainly: a Postgres database reachable through DATABASE_URL, and a master key that begins with sk-. compose.yaml provides both.

RunnableTrack N · NVIDIA GPU

curl -s http://127.0.0.1:4000/key/generate \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" \
    -d '{"models": ["local/coder"], "metadata": {"application": "editor"}}'

The response contains a new key. Give that key to your editor, and now the editor can reach the coder model and nothing else. Generate a second one for your agent experiments with local/chat and local/embed, and a third for whatever you are evaluating this month.

7. Prove the fallback works

litellm-config.yaml sends local/coder to local/chat when the coder route fails, and local/chat to a direct route after that. LiteLLM’s reliability page documents this shape, together with num_retries, allowed_fails and cooldown_time.

Test it by breaking something on purpose. Stop the coder model’s ability to start, by renaming its file for a minute, and send a request for local/coder.

RunnableAll tracks

curl -s http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/coder",
         "messages":[{"role":"user","content":"Write a one-line shell command that lists files by size."}],
         "max_tokens":64}'

An answer should come back, from the chat model, after a delay. Put the file back afterwards.

8. Health checks and logging

LiteLLM exposes three health endpoints with different meanings, documented as: /health/liveliness, where “the process is up. No dependencies are checked”; /health/readiness, where “the worker is ready to accept traffic”; and /health, which runs real test requests against every configured model.

RunnableAll tracks

curl -s http://127.0.0.1:4000/health/liveliness

curl -s http://127.0.0.1:4000/health/readiness

curl -s http://127.0.0.1:4000/health -H "Authorization: Bearer $LITELLM_MASTER_KEY"

The last one is expensive, because it makes a real request to each model, and on a swapping gateway that means loading them. litellm-config.yaml therefore sets background_health_checks: true with health_check_interval: 300, which the documentation describes as running the checks on a timer and having /health “serve the last cached result”. Use liveliness for a process supervisor, readiness for whether to send traffic, and /health when a human wants to know what is wrong.

For logging, this project keeps two things and deliberately does not keep a third.

The request log is the router’s own output, which the container path collects and the native path writes to gateway-logs/. It tells you which alias was called, when, and whether it worked.

Usage records live in the database on the container path, which is what makes per-key spend and volume visible.

Prompt and response text is not recorded. litellm-config.yaml sets turn_off_message_logging: true, which the logging page describes as preventing “messages and responses from being logged to your logging provider, but request metadata - e.g. spend, will still be tracked”. That is the right default for a machine you also use for personal things. Turn it off deliberately, and only when you know why you want the text.

RunnableTrack N · NVIDIA GPU

docker compose logs --tail 50 gateway

LiteLLM’s logging page also documents a list of callback integrations, configured under litellm_settings with success_callback, failure_callback and callbacks, for sending records to an external observability system. This project does not use one: a local gateway’s log belongs on the local machine.

9. The Anthropic shape

The same models, the same aliases, a different protocol. LiteLLM’s documentation describes the endpoint as letting you “call all your LLM APIs in the Anthropic v1/messages format”, and states that it supports all LiteLLM providers rather than only Anthropic ones.

RunnableAll tracks

curl -s http://127.0.0.1:4000/v1/messages \
    -H "x-api-key: $LITELLM_MASTER_KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "Content-Type: application/json" \
    -d '{"model":"local/chat",
         "messages":[{"role":"user","content":"Reply with one word: ready"}],
         "max_tokens":16}'

Note the different authentication header and the required version header. This matters in Part 25, where some coding agents speak this shape and not the other, and it is why the gateway is a better place to solve the problem than each client is.

10. Write it down

RunnableAll tracks

# Gateway

Purpose: the record of what your local model gateway is, for the parts of this course that
come after Part 9 and for you in six months. Save this file as `GATEWAY.md` beside your
`compose.yaml`, fill in every angle-bracketed blank, and update it whenever you change the
stack. Platform: all. Minimum memory: 8 GB. Assumes: the gateway from Part 9's project is
running. Nothing in this file is a secret: keys live in `.env`, which is not committed.

---

## What this machine serves

| Field | Value |
| --- | --- |
| Machine name | `<the name you call it>` |
| Track | `<S, X, M or N>` |
| Accelerator and memory | `<chip and total memory>` |
| Operating system | `<name and version>` |
| Gateway address | `http://127.0.0.1:<port>` |
| Started by | `<docker compose up -d, or bash start-gateway-native.sh>` |
| Last changed | `<date>` |

## Endpoints

| Path | Shape | What it is for |
| --- | --- | --- |
| `/v1/chat/completions` | OpenAI | Every chat and agent client in Parts 10, 24, 25 and 26 |
| `/v1/completions` | OpenAI | Fill-in-the-middle and raw completion |
| `/v1/embeddings` | OpenAI | The retrieval work in Part 10 |
| `/v1/models` | OpenAI | What a client asks to discover the aliases below |
| `/v1/messages` | Anthropic | Clients that speak the Anthropic message format |
| `/health/liveliness` | Health | The process is up |
| `/health/readiness` | Health | The worker is ready to accept traffic |
| `/health` | Health | A real request against each configured model |

## Model aliases

These names are the contract. Later parts of the course use them, and changing one means
changing everything that refers to it, so add rather than rename.

| Alias | Model and quantisation | Context | Engine | Notes |
| --- | --- | --- | --- | --- |
| `local/chat` | `<model, quantisation>` | `<tokens>` | `<engine and version>` | General conversation |
| `local/coder` | `<model, quantisation>` | `<tokens>` | `<engine and version>` | Code; used from Part 25 |
| `local/embed` | `<model, quantisation>` | `<tokens>` | `<engine and version>` | Retrieval; used from Part 10 |
| `<any alias you added>` | | | | |

Licences of the models above, from the course model reference:
`<one line per model>`

## Keys

| Key | Who holds it | What it may reach | Created |
| --- | --- | --- | --- |
| Master key | You, in `.env` only | Everything, including key creation | `<date>` |
| `<a virtual key name>` | `<which application>` | `<which aliases>` | `<date>` |

The master key is not an application credential. Applications get a virtual key with a model
list and a budget, so that revoking one does not disturb the others.

## Routing and fallbacks

| Alias | Retries | Falls back to | Why |
| --- | --- | --- | --- |
| `local/coder` | `<n>` | `<alias>` | `<one sentence>` |
| `local/chat` | `<n>` | `<alias>` | `<one sentence>` |

Model swapping: `<which models share the accelerator and are swapped, and which stay resident>`

Idle unload: `<the ttl values you chose, and why>`

## Logging

| What | Where | Retention |
| --- | --- | --- |
| Router request log | `<path or docker compose logs gateway>` | `<how long you keep it>` |
| Engine logs | `<path or docker compose logs swap>` | `<how long you keep it>` |
| Usage and spend records | `<the database, or "not enabled on this path">` | `<how long you keep it>` |

Prompt and response text: `<recorded, or not recorded, and the setting that decides it>`

## Operating notes

Starting from cold: `<the command, and how long the first request takes while a model loads>`

Changing a model: `<edit llama-swap.yaml, then the command that reloads it>`

What breaks first under load: `<from your own measurements in the Part 9 lab>`

Known limitations on this machine: `<for example, which models do not fit, or which track
features are unavailable>`

## Verification

Last run of `check-gateway.sh`: `<date>`, `<n>` passed, `<n>` failed.

Anything that failed, and why it is acceptable: `<one line each, or "nothing">`

Download GATEWAY.md (template)97 lines

Save it as GATEWAY.md beside compose.yaml and fill in every blank. This is not paperwork. Part 10 asks which alias serves embeddings. Part 13 asks what the coder model is so it can fine-tune something comparable. Part 25 asks for the endpoint and the key shape. The capstone asks you to defend the whole design. All of those are ten seconds of work if this file is current and twenty minutes of archaeology if it is not.

Test routing as an application contract

Before adding the browser interface, test every alias directly through the gateway using its intended credential. Confirm that the returned model identity and backend logs match the alias mapping. Then send an invalid model name, no credential and an invalid credential, preserving the error responses.

Exercise model switching under the memory-exclusion policy. Load one large alias, call another, then return to the first. Observe unloading, cold-start delay and any queued requests. If both backends remain resident unexpectedly, stop and inspect group membership before increasing load.

For fallback, stop only the primary backend of the alias under test. Record whether the gateway uses the explicitly permitted fallback or returns the intended failure. Check that fallback does not silently change the privacy location or remove a required feature. Restore the primary and repeat the request. Save the final alias map, redacted configuration, key ownership and restart procedure. The gateway is complete when another client can use its stable names and an operator can identify which artefact answered, including during failure and recovery.

Validation

RunnableAll tracks

bash check-gateway.sh

Output — what you should see

==> checking http://127.0.0.1:4000
  PASS  liveness endpoint answers
  PASS  readiness endpoint answers
  PASS  alias local/chat is published
  PASS  alias local/coder is published
  PASS  alias local/embed is published
  PASS  OpenAI-shaped chat completion returns content
  PASS  embeddings endpoint returns a vector
  PASS  Anthropic-shaped messages endpoint answers
  PASS  anonymous request is refused

    9 passed, 0 failed

You are done when the script passes every check, and when all of the following are also true.

Asking for local/coder after local/chat visibly swaps the engine in the llama-swap log, and asking for local/embed does not.
A request with no key is refused, and a request with a virtual key limited to one model is refused for a different model. On the native path, only the first half of that applies.
A request for a model whose engine cannot start still returns an answer, from the fallback.
GATEWAY.md is filled in, including the licences of the three models and the date.
The gateway survives a restart of the machine, or you have written down in GATEWAY.md exactly what you type to bring it back.

Expected outcome

One address on your machine that serves every model you own, in two API shapes, behind a key, with models loading on demand and unloading when idle. Concretely:

http://127.0.0.1:4000 answers /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models and /v1/messages.
Three names, local/chat, local/coder and local/embed, that will not change for the rest of the course even when the models behind them do.
Applications that hold their own keys, each limited to the models it needs, on the container path.
A GATEWAY.md that answers, in under a minute, every question a later part of this course asks about your setup.

The result you should notice within a week is negative: you stop thinking about ports. That is the whole value, and it is why this is the last thing Level 2 asks you to build.

Troubleshooting

llama-swap reports a model that will not start. Almost always a wrong path in a cmd line, or a llama-server binary that is not where LLAMA_BIN says. Read the engine’s own output in llama-swap’s /logs page, which shows the failing command’s stderr.

command -v llama-server finds nothing inside the container. The unified images ship one, but if your tag does not, set LLAMA_BIN in compose.yaml to an absolute path inside the image. The comment in compose.yaml gives the one-line command that tells you where it is.

The router returns an error about a model that is not in the model list. LiteLLM matches on model_name, which is the left-hand name in litellm-config.yaml, not on llama-swap’s id. They are the same strings in this project on purpose; if you rename one, rename both.

Everything works through llama-swap on port 9292 and nothing works through the gateway on 4000. Check LLAMA_SWAP_BASE_URL. On the container path it is http://swap:8080/v1, using the service name on the private network; on the native path it is a loopback address and the port from .env. A trailing /v1 that is present in one place and missing in the other is the usual cause.

The proxy will not start, complaining about the database. The database container takes a few seconds longer than the router does. Wait and look again; if it persists, check that the password in .env matches what the database was first initialised with, because changing it later does not change the stored password.

A request hangs for a long time and then answers. A cold model is being loaded. That is what the generous request_timeout in litellm-config.yaml is for. If it happens on every request, your ttl values are unloading models faster than you are using them.

Two applications keep evicting each other’s models. They are in the same swapping group and the machine can only hold one. Either accept it, or give the smaller model its own persistent group, or buy memory. The lab on the previous page tells you which of those your machine actually needs.

Cleanup

The gateway is meant to stay. Stop it when you need the memory back:

RunnableTrack N · NVIDIA GPU

docker compose down

RunnableAll tracks

curl -s -X POST http://127.0.0.1:9292/api/models/unload

Keep GATEWAY.md, .env and the whole directory. Later parts start by asking you to bring the gateway up.

What you learned

An alias is an interface. local/chat is a promise to every client that a chat model will answer at this address, and it is what lets the model, the quantisation and even the engine change underneath without anything breaking.
Loading on demand turns memory into a schedule. One accelerator can serve three models when they are needed at different times, and groups are how you say which of them may coexist.
A key per application is cheap and a shared key is expensive. Revoking one virtual key disturbs nothing else; rotating a shared key means finding every place it was pasted.
Health has three different meanings and asking the wrong one gives you a misleading answer, which matters as soon as anything automated is watching.
A fallback is a decision, not a feature. Silently substituting a worse model is right for a human waiting and wrong for a pipeline recording results, and the choice belongs in writing.
Logging metadata without logging text is the default that lets you operate a service on a machine you also live on.

Record in GATEWAY.md: the machine, the address, the three aliases with their models, quantisations, context lengths, engines and licences, the keys you issued and what each may reach, the fallback chain and why you chose it, where the logs go and how long you keep them, what breaks first under load from the previous page’s lab, and the date. Then put the directory under version control, with .env excluded, and note the repository in your lab notebook.

Sources for this lesson

10 verified · checked 2026-09-09

01llama-swap — README§ Configuration; command-line flags; endpoints; container imagesgithub.com/mostlygeek/llama-swap2026-09-09
02llama-swap — example configuration§ Macros; models; groupsgithub.com/mostlygeek/llama-swap/blob/main/docs/config.example.yaml2026-09-09
03LiteLLM — Proxy overview§ Starting the proxy; config.yamldocs.litellm.ai/docs/simple_proxy2026-09-09
04LiteLLM — Proxy config.yaml§ model_list; os.environ references; OpenAI-compatible endpointsdocs.litellm.ai/docs/proxy/configs2026-09-09
05LiteLLM — Virtual keys§ Requirements; key generationdocs.litellm.ai/docs/proxy/virtual_keys2026-09-09
06LiteLLM — Reliability and fallbacks§ fallbacks; num_retries; cooldowndocs.litellm.ai/docs/proxy/reliability2026-09-09
07LiteLLM — Health checks§ Endpoints; background health checksdocs.litellm.ai/docs/proxy/health2026-09-09
08LiteLLM — Logging§ Callbacks; message redactiondocs.litellm.ai/docs/proxy/logging2026-09-09
09LiteLLM — Anthropic /v1/messages§ Usagedocs.litellm.ai/docs/anthropic_unified2026-09-09
10LiteLLM — Deployment§ Container image; required environment variables; default portdocs.litellm.ai/docs/proxy/deploy2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.

Project: Your Local Model Gateway

Objective

How the pieces fit

Requirements

Track S — NVIDIA DGX Spark

Track X — AMD Ryzen AI Max+ 395

Track M — Apple silicon

Track N — NVIDIA desktop or laptop

Tasks

Working directory and terminal roles

1. Take the files and read them

2. Check the model paths, before anything else

3. The environment file and the keys

4. Start llama-swap on its own, and watch it swap

5. Add the router

6. Virtual keys

7. Prove the fallback works

8. Health checks and logging

9. The Anthropic shape

10. Write it down

Test routing as an application contract

Validation

Expected outcome

Troubleshooting

Cleanup

What you learned

Check your understanding

Sources for this lesson