Claude Code with a Local Endpoint

This lesson is different from the other tool lessons in this part, because the tool’s own vendor publishes a sentence that decides how you should read everything after it. By the end you will know exactly which variables point Claude Code at a server you own, which two servers can answer it, what degrades when you do this, and — the useful part — how to decide whether it is worth doing at all.

The course records Claude Code current · verified 2026-09-08, which is a moving target rather than a pin; the tool updates itself and its documentation moves with it. Every claim below carries the date it was read.

The sentence that frames the lesson

Claude Code’s “Other LLM gateways” page states:

Any gateway that exposes a supported API format works. Anthropic doesn’t endorse, maintain, or audit third-party gateway products, and doesn’t support routing Claude Code to non-Claude models through any gateway.

!Important

Read that twice. The mechanism is documented and general — a gateway that speaks the format works — and the specific use this lesson describes is named and declined. Nothing here is a hack or a loophole; it is a documented configuration surface being used for a purpose its author does not support. That means no support channel, nothing promised about compatibility across updates, and a real chance that a version bump changes behaviour without warning. Plan accordingly, and do not put this configuration on a path anyone else depends on.

The honest reason to do it anyway is narrow and legitimate: you already use this tool, you want to see how far a local model gets inside a harness you know well, and you want that comparison in the lab’s results table beside five open-source tools. That is a good reason. “It will replace my subscription” is not one, and the last section of this lesson explains why.

The variables

Claude Code reads its endpoint and credentials from environment variables, documented on its environment-variables page.

ANTHROPIC_BASE_URL — “Override the API endpoint to route requests through a proxy or gateway.” The same entry adds two consequences worth knowing before you set it: “When set to a non-first-party host, MCP tool search is disabled by default. Set ENABLE_TOOL_SEARCH=true if your proxy forwards tool_reference blocks”, and that a remote-control feature is disabled when the base URL points anywhere other than Anthropic’s own host.

ANTHROPIC_AUTH_TOKEN — “Custom value for the Authorization header (the value you set here will be prefixed with Bearer )”. The authentication page describes when to reach for it: “Use this when routing through an LLM gateway or proxy that authenticates with bearer tokens rather than Anthropic API keys.” That is exactly what a LiteLLM virtual key is.

The model variables. ANTHROPIC_MODEL is the “Name of the model setting to use”. Three aliases resolve separately: ANTHROPIC_DEFAULT_HAIKU_MODEL is the “Model ID that the haiku alias resolves to, also used for background functionality”, with ANTHROPIC_DEFAULT_SONNET_MODEL and ANTHROPIC_DEFAULT_OPUS_MODEL for the other two. Setting only the first leaves background work pointed somewhere else, which is the commonest half-configured state.

CLAUDE_CODE_MAX_OUTPUT_TOKENS matters more here than it looks. Its documentation says Claude Code “defaults to 32000 for model IDs it doesn’t recognize, such as gateway-specific names”, and that raising it “reduces the effective context window available before auto-compaction triggers”. A gateway alias is by definition an unrecognised model id, so you get that default whether you wanted it or not, and on a local model at a modest context length that is a large fraction of the window reserved for output that will never be produced.

Traffic controls. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC set to a non-empty value disables “auto-updates, telemetry, error reporting, the /feedback command, Claude-drafted feedback, release notes, the PR and MR status badge checks, and availability checks”. DISABLE_TELEMETRY and DISABLE_ERROR_REPORTING are narrower versions of the same idea. Setting the model endpoint local does nothing about any of this; it is the Part 10 distinction between where the tokens go and what the tool itself sends, and it applies here with full force.

RunnableAll tracks

claude-code-local-settings.json

{
  "_readme": [
    "Purpose: a project-scoped Claude Code settings file that points the tool at a local",
    "Anthropic-compatible endpoint, names the three model aliases, keeps non-essential",
    "network traffic off, and denies reads of credential files. Platform: all (spark,",
    "strix, mac, nvidia). Minimum memory: 16 GB for the model behind the endpoint.",
    "Assumes: saved as .claude/settings.json in the project, or ~/.claude/settings.json",
    "for every project. Every key is from the Claude Code settings, environment-variable",
    "and permissions documentation read on 2026-09-09. Read the lesson first: Anthropic's",
    "own gateway documentation states that routing Claude Code to non-Claude models",
    "through a gateway is not supported, so this configuration is an experiment you run",
    "with your eyes open, not a supported deployment.",
    "Delete this _readme key if your editor's schema validation objects to it."
  ],
  "$schema": "https://json.schemastore.org/claude-code-settings.json",
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:4000",
    "ANTHROPIC_AUTH_TOKEN": "${GATEWAY_KEY}",
    "ANTHROPIC_MODEL": "local/coder",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "local/coder",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "local/chat",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "local/coder",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "8192",
    "MAX_MCP_OUTPUT_TOKENS": "8000",
    "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
    "BASH_DEFAULT_TIMEOUT_MS": "300000"
  },
  "permissions": {
    "defaultMode": "default",
    "allow": [
      "Bash(python3 -m pytest *)",
      "Bash(git diff *)",
      "Bash(git status)"
    ],
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(~/.ssh/**)",
      "Read(~/.aws/**)"
    ]
  }
}

Download claude-code-local-settings.json42 lines

The settings file is the tidier way to carry all of that. The documented scopes are ~/.claude/settings.json for you on this machine, .claude/settings.json shared in a project, .claude/settings.local.json for you in one project, and a managed file an organisation deploys, with precedence running from managed down to user. The env key “Set[s] environment variables for every session and its subprocesses”, which is why the local endpoint belongs there rather than in your shell profile: it is a property of this project, not of your login.

Two servers that can answer

Getting an Anthropic-shaped request answered by a local model

1. 1Claude Code builds a Messages requestSystem prompt, tools, conversation, in the Anthropic wire format. This part is not configurable.
2. 2ANTHROPIC_BASE_URL sends it to youPlus the bearer token from ANTHROPIC_AUTH_TOKEN, and any headers from ANTHROPIC_CUSTOM_HEADERS.
3. 3Option A: llama-server /v1/messagesThe direct path. One process, no translation. Tool use requires --jinja.
4. 4Option B: LiteLLM /v1/messagesThe gateway path. Translates to whatever the alias points at, and keeps your usage log and virtual keys.
5. 5The engine answersllama.cpp, vLLM or an MLX server behind the alias, in whatever shape it natively speaks.
6. 6Claude Code parses the replyAnd either finds a tool call it can execute, or does not, which is where a weak model ends the loop.

Option A is fewer moving parts and better for finding out whether this works at all. Option B is what you keep, because it gives you the same aliases, keys and usage records as every other tool in this part.

Option A: llama.cpp directly

The llama-server README documents the endpoint under a heading of its own, and the hedge in it is worth quoting in full because it sets the right expectation:

Given a list of messages, returns the assistant’s response. Streaming is supported via Server-Sent Events. While no strong claims of compatibility with the Anthropic API spec are made, in our experience it suffices to support many apps.

The README also states the condition that decides whether an agent works at all: “Tool use requires --jinja flag.” Without it you get an endpoint that answers questions and never edits a file, which is precisely the failure the first lesson warned about and which looks like a model problem.

RunnableAll tracks

start-anthropic-server.sh

#!/usr/bin/env bash
# Purpose: start llama-server with the Anthropic-compatible Messages endpoint and tool
#          calling enabled, so an agent that speaks the Anthropic wire format can be
#          pointed at it, and print the two environment variables that agent needs.
# Platform: all (spark, strix, mac, nvidia); the binary must be a build for your backend
# Minimum memory: 16 GB for a 30B-class mixture-of-experts coder at a long context
# Assumes: llama-server on PATH or at $LLAMA_BIN, a GGUF model at $MODEL, and that you
#          have read the lesson: the README makes no strong compatibility claim for this
#          endpoint, and tool use through it requires --jinja.

set -euo pipefail

MODEL="${MODEL:-$HOME/models/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf}"
LLAMA_BIN="${LLAMA_BIN:-llama-server}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8080}"
CTX="${CTX:-65536}"
ALIAS="${ALIAS:-local-agent-model}"
NGL="${NGL:-999}"

if ! command -v "$LLAMA_BIN" >/dev/null 2>&1 && [ ! -x "$LLAMA_BIN" ]; then
    echo "llama-server not found. Set LLAMA_BIN to the binary from your Part 6 build." >&2
    exit 1
fi

if [ ! -f "$MODEL" ]; then
    echo "Model file not found: $MODEL" >&2
    echo "Set MODEL to a GGUF from your Part 4 model library." >&2
    exit 1
fi

# --jinja is not optional here. The server README states that tool use through the
# Anthropic endpoint requires it, because the tool-call format comes from the model's own
# chat template. Without it the endpoint answers and the agent never sees a tool call.
#
# --cache-reuse is what stops every agent turn re-reading the whole conversation. It is
# the single largest speed setting for agent work; see this part's second lesson.
echo "Starting llama-server on http://${HOST}:${PORT}"
echo "  model:   ${MODEL}"
echo "  context: ${CTX}"
echo
echo "Point an Anthropic-shaped agent at it with:"
echo "  export ANTHROPIC_BASE_URL=http://${HOST}:${PORT}"
echo "  export ANTHROPIC_MODEL=${ALIAS}"
echo

exec "$LLAMA_BIN" \
    --model "$MODEL" \
    --alias "$ALIAS" \
    --ctx-size "$CTX" \
    --n-gpu-layers "$NGL" \
    --jinja \
    --cache-reuse 256 \
    --parallel 2 \
    --host "$HOST" \
    --port "$PORT"

Download start-anthropic-server.sh56 lines

--cache-reuse is in there for the reason the second lesson gave: it is the difference between an agent whose turns cost a constant amount and one that gets slower with every step.

RunnableAll tracks

point the tool at the direct server

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
export ANTHROPIC_MODEL=local-agent-model
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1

Option B: the gateway

LiteLLM publishes the same endpoint path. Its documentation shows the Anthropic SDK pointed at the proxy’s base URL with a model name that is one of your aliases, and states that the bridge supports “All LiteLLM supported providers openai, anthropic, bedrock, vertex_ai, gemini, azure, azure_ai, etc.”

It also has a page whose title is “Claude Code Quickstart”, which gives the two variables directly: export ANTHROPIC_BASE_URL="http://0.0.0.0:4000" for the unified endpoint, and export ANTHROPIC_AUTH_TOKEN="$LITELLM_KEY" for the key. On your machine the address is your gateway’s, which the Part 9 project bound to the loopback address.

That page also documents the tool-search consequence from the other side: “Claude Code turns its tool search off when ANTHROPIC_BASE_URL is not a first-party Anthropic host, so /context shows every MCP tool schema inlined instead of loaded on-demand.” Read that as a token-budget warning. If you wire in five MCP servers from Part 24, every one of their tool schemas is in the prompt on every turn, and on a local model with a 32k window that can be a substantial fraction of your context before the conversation starts.

▤Memory budget

Inlined tool schemas plus a 32,000-token output reservation plus the system prompt is a large fixed cost on every turn, and it is charged against a context you are paying for in memory. On a 16 GB machine this is the difference between an agent that works and one that compacts its conversation every three turns. Lower CLAUDE_CODE_MAX_OUTPUT_TOKENS to something your model will actually produce, and wire in the two MCP servers you need rather than the eight you have.

Permission modes

Whatever endpoint you use, the permission model is the tool’s own, and it is worth setting deliberately. The documented modes are default (reads only, and the CLI names it Manual, accepting manual as an alias), acceptEdits (reads, file edits and common filesystem commands), plan, auto, dontAsk (only pre-approved tools) and bypassPermissions (everything). The CLI flag is --permission-mode, and --dangerously-skip-permissions is documented as “Equivalent to --permission-mode bypassPermissions”.

The permissions block in settings gives finer control: allow to “Approve listed tool uses without a prompt”, deny to “Block listed tool uses, including reads of files that hold secrets”, ask to always prompt, and defaultMode to set the mode new sessions start in. The deny list is the interesting one for this part: it is a place to say, in the project’s own configuration, that the agent may not read your environment file, and the settings documentation’s own example does exactly that.

⛨Security

A deny rule in a settings file is a good habit and it is not a boundary. It is enforced by the program you are running, in your user account, with your file permissions, and it protects you from the agent’s mistakes rather than from anything determined. The boundary is the container in the next lab. Use both: the deny list stops the ordinary accident, the container stops the rest.

MCP configuration is unchanged by any of this. Servers are declared in .mcp.json under an mcpServers key with a type of http, stdio or ws, added with claude mcp add, and scoped local, project or user. The sse transport is documented as deprecated: “Use HTTP servers instead, where available.”

What works and what does not, dated 2026-09-09

Works, as far as the documentation goes. The base URL and bearer token are documented configuration. llama.cpp publishes a Messages endpoint and says tool use through it needs --jinja. LiteLLM publishes the same path and a page about pointing this specific tool at it. The permission model, the settings file and MCP are unaffected by where the model lives.

Degrades. MCP tool search is off by default against a non-first-party host, so tool schemas are inlined rather than fetched on demand. A remote-control feature is disabled. The output-token default assumes a model id it does not recognise and reserves a large slice of your window.

Not confirmed. How much of the Anthropic Messages surface llama.cpp implements: the README declines to claim strong compatibility, and this course has not validated it on hardware. Whether prompt caching behaves as the tool expects through either path. Whether extended-thinking controls translate to a local model at all. Every one of these belongs in your own notebook after you have tried it, and none of them should be repeated as fact from this page.

When this is worth the effort

Rarely, and the honest accounting matters more than the configuration.

It is worth it when you already work in this tool every day and want a single comparison point in the lab’s results table; when you need the tool’s permission model and MCP wiring specifically; or when you are testing whether a local model can hold up inside a harness you know well enough to notice small regressions.

It is not worth it as a way to get the tool’s behaviour with none of its cost. The harness is one part of what you are used to and the model is the other, and swapping the model for a 30B-class local one changes the thing you actually noticed. The second lesson’s benchmark tables give the shape of that gap, and no amount of configuration closes it.

And there is a cheaper alternative for most people. OpenCode and Codex CLI are open source, documented for local endpoints, and cost nothing to try. If your goal is an agent driven by your own model, start there; if your goal is this agent driven by your own model, this lesson is how, with the vendor’s own sentence attached.

Test the provider contract beyond a plain completion

An endpoint that accepts one request in an Anthropic-compatible shape may still differ in streaming events, tool-result handling, token accounting or model capabilities. Test a read-only tool round trip before evaluating a coding task, and retain the raw error when the client and server disagree.

Use the documented configuration for the installed client and serving engine, with the actual model name from that endpoint. A local route should be verified in server logs and network configuration; changing a model label alone does not establish where requests go. Keep other configured integrations visible when making a privacy claim.

Run the task in the same isolated repository used for other agents and score the final diff independently. Record client-specific limitations and manual interventions as part of the workflow result. Compatibility work can be valuable, but a route that happens to respond is not equivalent to the fully supported hosted service contract. Choose it for the features you have actually exercised and can reproduce.

Recap

ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN point Claude Code at a server you own, the three model-alias variables decide what haiku, sonnet and opus resolve to, and a settings file’s env block is the tidy place for all of them. Two servers can answer: llama.cpp’s /v1/messages, which requires --jinja for tool use and makes no strong compatibility claim, and LiteLLM’s bridge, which keeps your Part 9 aliases, keys and usage log. Setting a non-first-party base URL turns MCP tool search off, which inlines every tool schema into every turn, and the output-token default for unrecognised model ids reserves a large slice of a local model’s context. The permission modes and deny lists are worth setting and are not a security boundary. And Anthropic’s own documentation says routing this tool to non-Claude models through a gateway is not supported, which is the first fact to state whenever you describe this setup to somebody else.

Check your understanding

Question 1. What does Claude Code's own documentation say about routing it to non-Claude models through a gateway?

• It is supported for open-weight models only
• It is documented as a supported enterprise deployment
• Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway"
• The documentation does not mention gateways

Show the answer and why

Answer: Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway"

The "Other LLM gateways" page says the mechanism works with any gateway exposing a supported format, and in the same breath declines to support this specific use. Both halves matter: the configuration surface is real, and there is no support behind it.

Question 2. Why does setting ANTHROPIC_BASE_URL to a local address change how MCP tools are presented to the model?

• Local servers cannot host MCP servers
• MCP tool search is disabled by default against a non-first-party host, so tool schemas are inlined into the prompt instead of being loaded on demand
• MCP is disabled entirely
• The tool switches to the stdio transport

Show the answer and why

Answer: MCP tool search is disabled by default against a non-first-party host, so tool schemas are inlined into the prompt instead of being loaded on demand

Both the Claude Code environment-variable documentation and LiteLLM's quickstart describe this. The practical effect is a fixed token cost per turn proportional to how many MCP servers you wired in, which on a local model with a modest context window is a budget problem.

Question 3. You point the tool at llama-server's Messages endpoint. It answers questions but never edits a file. What is the first thing to check?

• The bearer token
• That the server was started with --jinja, which the README states is required for tool use
• The permission mode
• The context length

Show the answer and why

Answer: That the server was started with --jinja, which the README states is required for tool use

The README says it directly: tool use through that endpoint requires --jinja, because the tool-call format comes from the model's own chat template. Without it the endpoint works and the loop cannot advance, which looks exactly like a model that will not use tools.

Question 4. Why does CLAUDE_CODE_MAX_OUTPUT_TOKENS deserve attention when using a gateway alias?

• It controls the model temperature
• Unrecognised model ids get a default of 32000 output tokens, and that reservation reduces the context available before auto-compaction, which is expensive on a local model
• It must equal the context length
• It is ignored when a base URL is set

Show the answer and why

Answer: Unrecognised model ids get a default of 32000 output tokens, and that reservation reduces the context available before auto-compaction, which is expensive on a local model

A gateway alias is by definition a model id the tool does not recognise, so the default applies whether you meant it or not. Setting it to something your model will actually produce gives the conversation back a large slice of window.

Question 5. Which of these are true about a deny rule in the permissions block? Select all that apply.

• It can block reads of files that hold secrets
• It is enforced by the tool, in your user account, with your file permissions
• It replaces the need for a container around the agent
• It is a good default that stops ordinary accidents

Show the answer and why

Answer: It can block reads of files that hold secrets, It is enforced by the tool, in your user account, with your file permissions, It is a good default that stops ordinary accidents

A deny list is a guard rail inside a program you are trusting. It stops the agent from wandering into your environment file by accident. It is not a boundary, because everything enforcing it runs with your privileges, which is what the sandbox lab replaces.

Check my answersClear