Function Calling End to End on Local Engines

Part 9 explained the two pieces of server machinery that turn tokens into a tool_calls array, and how to switch them on for each engine. This lesson is the client side of the same story: the four messages that make one complete round trip, what each model family actually emits, what Ollama and LM Studio add on top of the OpenAI shape, how parallel calls and reasoning content interact, and how to find out how often any of it works for the model you have.

By the end you will be able to write the request by hand, read every field of the reply, recognise a format mismatch from the response alone, decide whether to run one call or several, and produce a number for the reliability of your own model, quantisation and engine that Part 25 will ask you for.

One complete round trip

A tool call is four messages, and two of them are yours.

What a single tool call looks like on the wire

1. 1Request 1: messages plus toolsYou send the conversation and the tool list. Every tool is {"type": "function", "function": {name, description, parameters}} where parameters is a JSON Schema object.
2. 2Reply 1: an assistant message with tool_callsThe message has an empty or short content and a tool_calls array. Each entry has an id, a type, and a function object holding a name and an arguments string. Arguments arrive as a JSON string, not an object.
3. 3Request 2: the same messages, plus the assistant message, plus one tool message per callYou append the assistant message exactly as you received it, then one {"role": "tool", "tool_call_id": ..., "content": ...} message carrying the result as text. Dropping the assistant message is the most common way to break the second turn.
4. 4Reply 2: the answer, or another tool callThe model either answers using the result or asks for another call. That branch is the whole agent loop; everything in the first lab is bookkeeping around it.

The two requests are identical in shape. An agent loop is this diagram with a while statement around it and a counter.

Here is the first request against the Part 9 gateway, with one tool.

RunnableAll tracks

a tools request against the gateway

curl -s http://127.0.0.1:4000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $LITELLM_KEY" \
    -d '{
      "model": "local/chat",
      "messages": [
        {"role": "user", "content": "How many gibibytes is 5.03 GB?"}
      ],
      "tools": [{
        "type": "function",
        "function": {
          "name": "convert_memory",
          "description": "Convert a memory figure between gigabytes and gibibytes.",
          "parameters": {
            "type": "object",
            "properties": {
              "value": {"type": "number", "description": "The figure to convert."},
              "from_unit": {"type": "string", "enum": ["GB", "GiB"]},
              "to_unit": {"type": "string", "enum": ["GB", "GiB"]}
            },
            "required": ["value", "from_unit", "to_unit"],
            "additionalProperties": false
          }
        }
      }],
      "tool_choice": "auto"
    }'

A working reply, trimmed to the parts that matter:

Output — what you should see

{"choices": [{"finish_reason": "tool_calls",
  "message": {"role": "assistant", "content": null,
    "tool_calls": [{"id": "call_a1b2", "type": "function",
      "function": {"name": "convert_memory",
                   "arguments": "{\"value\": 5.03, \"from_unit\": \"GB\", \"to_unit\": \"GiB\"}"}}]}}]}

Three details in that reply cause most client bugs. arguments is a string containing JSON, so it needs a second json.loads, and it can be malformed even when the surrounding response is not. content may be null rather than an empty string. And the id is what ties your tool message back to the call: send it back verbatim, because with parallel calls it is the only thing that says which result belongs to which request.

✕Common mistake

Sending the tool result without the assistant message that asked for it. The second request must contain the assistant message with its tool_calls array, followed by the tool messages. A client that appends only the tool result produces a conversation in which a result arrives with nothing having asked for it, and the model’s template has no way to render that. The symptoms range from a template error to a model that answers a question nobody asked.

The format is the model’s; the parser is the engine’s

Part 9 established that the chat template renders your tool list into whatever the model was trained on, and a per-family parser reads it back. It is worth knowing what those formats look like, because when the parser is wrong you will see the raw format as visible text and the shape tells you immediately which family you are dealing with.

Family	What the model emits	Where it is documented
Qwen3 (8B, 4B, 30B-A3B)	Hermes-style: a JSON object naming the function, wrapped in <tool_call> tags	vLLM lists hermes as the parser for the Qwen 2.5 and QwQ line; the Qwen3 card points at Qwen-Agent for agentic use
Qwen3-Coder	An XML-shaped call; vLLM ships a separate qwen3_xml parser for it	vLLM’s parser table
gpt-oss	The harmony format: output split across analysis, commentary and final channels, with function calls placed on the commentary channel	The harmony repository
Llama 3.1 to 3.3	A JSON object, with built-in tool names for wolfram_alpha, web_search / brave_search and code_interpreter	llama.cpp’s function-calling document
Llama 4, and some others	Python-looking calls, handled by a pythonic family of parsers	vLLM’s parser table

Two consequences worth carrying around.

gpt-oss is not optional about its format. The harmony repository states that “gpt-oss should not be used without using the harmony format as it will not work correctly”, and the model card repeats it: the models “were trained on our harmony response format and should only be used with the harmony format”. If you serve gpt-oss through something that does not implement harmony, the failure is not a missing feature, it is wrong behaviour.

Qwen3-Coder is a non-thinking model. Its card states plainly that it “supports only non-thinking mode and does not generate <think></think> blocks in its output”, with a native context length of 262,144 tokens. That combination, no thinking and a very long context, is exactly what you want in a tool loop, and the fourth lesson explains why.

⌚Version note

Format and parser names move between releases. Everything in the table above was read on 2026-09-09 against vLLM 0.28.0 · verified 2026-09-08, llama.cpp v0.4.0 · verified 2026-09-08 and the model cards linked in the sources. Before you debug a model, check the engine’s own parser list for the version you are running rather than trusting a table, including this one.

What the desktop engines add

Ollama and LM Studio both accept the OpenAI shape, and both add something on top that is worth knowing before you write a client against them.

Ollama takes a tools array in its own /api/chat request as well as through its OpenAI-compatible endpoint, and its tool-calling documentation adds two fields the OpenAI shape does not have. A think boolean turns thinking on or off per request, and the reply may carry message.thinking alongside message.content and message.tool_calls. Tool results go back as messages with role: "tool" and a tool_name field. The documentation’s own guidance on streaming is the useful part: “Gather every chunk of thinking, content, and tool_calls, then return those fields together with any tool results in the follow-up request.” Its single-call example carries the caveat that it is “only recommended for models which only return a single tool call”.

LM Studio documents two tiers of support, and the distinction explains a class of confusing behaviour. Models with a chat template trained for tool use get native tool use; the page names Qwen2.5-7B-Instruct, Llama-3.1 and Llama-3.2, and Ministral-8B-Instruct. Every other model gets default tool use, in which LM Studio supplies its own system prompt and asks the model to emit calls wrapped in [TOOL_REQUEST] and [END_TOOL_REQUEST] tags, then parses those back into the OpenAI shape. Either way the response carries finish_reason: "tool_calls" and the calls arrive in choices[0].message.tool_calls, and streamed calls arrive in pieces under delta.tool_calls that you have to reassemble.

!Important

Default tool use means a model with no tool training at all will still produce something that parses. That is genuinely useful and it is also a trap when you are measuring: a high parse rate under default tool use tells you LM Studio’s prompt is working, not that the model is good at choosing tools. Compare models under the same tier, and record which tier you used.

llama-server sits between the two. Its README gives --jinja as enabled by default, and the function-calling document ties tool support to it. The document is also where the parallel-call switch lives, and it is a request field rather than a flag: “Multiple/parallel tool calling is supported on some models but disabled by default, enable it by passing "parallel_tool_calls": true in the completion endpoint payload.”

Parallel and sequential calls

Two calls in one reply is a different thing from two calls in two turns, and the difference is about whether the second call depends on the first.

Parallel means the model returns a tool_calls array with more than one entry, because the calls are independent: read three files, look up two records. You run them all, append one tool message per call with the matching tool_call_id, and send one follow-up request. It saves a round trip and therefore a whole prefill of the growing transcript.

Sequential is the ordinary loop: one call, one result, one more decision. It is what you get when the second call’s arguments come from the first call’s output, and it is what you should assume by default.

Assume sequential because parallel calling varies more between models than anything else in this lesson. vLLM’s documentation says parallel calls “are not supported for Llama 3, but it is supported in Llama 4 models”, that Mistral 7B “struggles to generate parallel tool calls correctly”, and that Llama’s smaller models “frequently fail to emit tool calls in the correct format”. llama.cpp defaults it off. Your loop needs to handle an array of length n regardless, because a model that was not supposed to do it sometimes will.

✦Tip

Handle the array, not the first element. Iterate tool_calls, run each one, and append a tool message for every single tool_call_id you were sent. A loop that runs tool_calls[0] and silently drops the rest will produce a conversation with an unanswered call in it, and the model’s next turn will be about that gap rather than about the task.

Reasoning content in the same reply

A reasoning model in a tool loop emits thinking as well as calls, and the two travel in separate fields. Where they end up depends on the engine, and the field names have been moving.

vLLM’s reasoning-outputs page documents --reasoning-parser with names per family, including deepseek_r1 for the DeepSeek R1 series and QwQ-32B, qwen3 for the Qwen3 series, and others for Gemma, Granite, GLM and more. Two statements on that page matter for an agent. The first is a rename: “Reasoning used to be called reasoning_content. To migrate, directly replace reasoning_content with reasoning.” The second is the behavioural rule: “The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the content field, not from the reasoning.” A model that writes its call inside its thinking will not have that call extracted.

llama-server does the same job with --reasoning-format, documented as controlling “whether thought tags are allowed and/or extracted from the response, and in which format they’re returned”, with none leaving thoughts unparsed in message.content, deepseek putting them in message.reasoning_content, and deepseek-legacy keeping the <think> tags in the content while also populating message.reasoning_content. There is a companion --reasoning-budget, documented as a “token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget”. Ollama returns thinking in message.thinking and takes a think boolean in the request.

⌚Version note

--reasoning-format and --reasoning-budget were read from the llama-server README on 2026-09-09 and are not yet in this course’s captured command reference for llama.cpp v0.4.0 · verified 2026-09-08, so they appear here in prose rather than in a runnable block. The vLLM reasoning field name replaced reasoning_content in the documentation read on the same date; a client written against an older vLLM may still be looking for the old name, and the symptom is an empty thinking field rather than an error.

For the client, the rule is short: read the thinking field if you want to log it, never send it back in the next request unless the model’s own documentation tells you to, and never parse tool calls out of it.

Measuring it, for your model

None of the above tells you whether your model at your quantisation on your engine emits valid calls often enough to build on. That is a measurement, it takes ten minutes, and the rest of the course asks for the result.

The script sends a fixed set of twenty prompts through the same tool list every time. Fifteen should produce exactly one named call with checkable arguments; five should produce no call at all. It reports six rates.

Rate	Question it answers	What a low value means
Call rate	Did a call come back when one was wanted?	The model is answering from its own knowledge instead of using tools, or tool_choice is wrong
Parse rate	Of the calls, how many had arguments that parsed as JSON?	The model emits nearly-JSON: trailing commas, single quotes, prose inside the object
Right tool	Was it the tool the case expected?	Overlapping or vague tool descriptions
Schema valid	Right parameter names and types, no invented parameters?	The schema is not reaching the model, or the model is guessing parameter names
Arguments correct	Do the values match what the prompt asked for?	The model can call but cannot fill in; this is what fine-tuning in Part 27 targets
False calls	How often did a call appear when none was wanted?	tool_choice: "required" left on, or descriptions that over-claim

RunnableAll tracks

tool-prompts.json

{
  "$comment": "The fixed test set for tool-call-reliability.py (Part 24). Four tools with deliberately different argument shapes - two strings, an integer with bounds, an enum, an optional parameter - and twenty prompts, fifteen of which should produce exactly one named call and five of which should produce no call at all. The no-call cases are the half people leave out, and they are what catch a model or a tool_choice setting that calls something on every turn. Edit the cases to match your own tools once you have run it as shipped; keep the no-call cases.",
  "version": "1",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "search_notes",
        "description": "Search the local note collection for passages matching a query. Returns matching passages with their file names. Use this when the answer might be written down in the user's own notes rather than known in general.",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "The search terms, as words rather than a question."
            },
            "limit": {
              "type": "integer",
              "description": "How many passages to return, between 1 and 20. Defaults to 5.",
              "minimum": 1,
              "maximum": 20
            }
          },
          "required": ["query"],
          "additionalProperties": false
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "read_file",
        "description": "Read one text file from the workspace and return its contents. The path must be relative to the workspace root and must not contain '..'.",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {
              "type": "string",
              "description": "A path relative to the workspace root, for example 'notes/backup.md'."
            }
          },
          "required": ["path"],
          "additionalProperties": false
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "run_tests",
        "description": "Run the project's test suite and return the summary line. Optionally restrict the run to one target.",
        "parameters": {
          "type": "object",
          "properties": {
            "target": {
              "type": "string",
              "description": "A test file or directory to run. Omit to run everything."
            }
          },
          "required": [],
          "additionalProperties": false
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "convert_memory",
        "description": "Convert a memory figure between gigabytes and gibibytes. Use this whenever a unit conversion is needed rather than doing the arithmetic yourself.",
        "parameters": {
          "type": "object",
          "properties": {
            "value": {
              "type": "number",
              "description": "The figure to convert."
            },
            "from_unit": {
              "type": "string",
              "description": "The unit the figure is in.",
              "enum": ["GB", "GiB"]
            },
            "to_unit": {
              "type": "string",
              "description": "The unit to convert to.",
              "enum": ["GB", "GiB"]
            }
          },
          "required": ["value", "from_unit", "to_unit"],
          "additionalProperties": false
        }
      }
    }
  ],
  "cases": [
    {
      "id": "search-1",
      "prompt": "What did I write in my notes about the backup window?",
      "expect_tool": "search_notes",
      "expect_args": { "query": { "contains": "backup" } }
    },
    {
      "id": "search-2",
      "prompt": "Look through my notes for anything about the office printer and show me the three best matches.",
      "expect_tool": "search_notes",
      "expect_args": { "query": { "contains": "printer" }, "limit": { "equals": 3 } }
    },
    {
      "id": "search-3",
      "prompt": "Did I record a decision about which quantisation to use? Check my notes.",
      "expect_tool": "search_notes",
      "expect_args": { "query": { "contains": "quantis" } }
    },
    {
      "id": "search-4",
      "prompt": "Find the passages in my notes that mention the wifi upgrade.",
      "expect_tool": "search_notes",
      "expect_args": { "query": { "contains": "wifi" } }
    },
    {
      "id": "read-1",
      "prompt": "Show me the contents of notes/backup.md.",
      "expect_tool": "read_file",
      "expect_args": { "path": { "equals": "notes/backup.md" } }
    },
    {
      "id": "read-2",
      "prompt": "Open README.md and tell me what the project is for.",
      "expect_tool": "read_file",
      "expect_args": { "path": { "equals": "README.md" } }
    },
    {
      "id": "read-3",
      "prompt": "I need to see src/config/settings.toml.",
      "expect_tool": "read_file",
      "expect_args": { "path": { "equals": "src/config/settings.toml" } }
    },
    {
      "id": "read-4",
      "prompt": "Read the file docs/decisions.md for me.",
      "expect_tool": "read_file",
      "expect_args": { "path": { "equals": "docs/decisions.md" } }
    },
    {
      "id": "tests-1",
      "prompt": "Run the test suite and tell me whether it is green.",
      "expect_tool": "run_tests",
      "expect_args": {}
    },
    {
      "id": "tests-2",
      "prompt": "Please run only the tests in tests/test_parser.py.",
      "expect_tool": "run_tests",
      "expect_args": { "target": { "contains": "test_parser" } }
    },
    {
      "id": "tests-3",
      "prompt": "Are the tests passing right now?",
      "expect_tool": "run_tests",
      "expect_args": {}
    },
    {
      "id": "convert-1",
      "prompt": "A file listing says 5.03 GB. How many gibibytes is that?",
      "expect_tool": "convert_memory",
      "expect_args": {
        "value": { "equals": 5.03 },
        "from_unit": { "equals": "GB" },
        "to_unit": { "equals": "GiB" }
      }
    },
    {
      "id": "convert-2",
      "prompt": "Convert 24 GiB into gigabytes.",
      "expect_tool": "convert_memory",
      "expect_args": {
        "value": { "equals": 24 },
        "from_unit": { "equals": "GiB" },
        "to_unit": { "equals": "GB" }
      }
    },
    {
      "id": "convert-3",
      "prompt": "My machine reports 119.2 GiB of memory. What is that in GB?",
      "expect_tool": "convert_memory",
      "expect_args": {
        "value": { "equals": 119.2 },
        "from_unit": { "equals": "GiB" },
        "to_unit": { "equals": "GB" }
      }
    },
    {
      "id": "convert-4",
      "prompt": "Express 128 GB as gibibytes, using the conversion tool rather than doing it in your head.",
      "expect_tool": "convert_memory",
      "expect_args": {
        "value": { "equals": 128 },
        "from_unit": { "equals": "GB" },
        "to_unit": { "equals": "GiB" }
      }
    },
    {
      "id": "none-1",
      "prompt": "In one sentence, what is the difference between prefill and decode?",
      "expect_tool": null
    },
    {
      "id": "none-2",
      "prompt": "Thanks, that is all for now.",
      "expect_tool": null
    },
    {
      "id": "none-3",
      "prompt": "Explain why a mixture-of-experts model needs more memory than its active parameter count suggests.",
      "expect_tool": null
    },
    {
      "id": "none-4",
      "prompt": "Which of the tools you have would you use to find out what is in a file, and why?",
      "expect_tool": null
    },
    {
      "id": "none-5",
      "prompt": "Write me a two-line summary of what an agent loop is.",
      "expect_tool": null
    }
  ]
}

Download tool-prompts.json227 lines

RunnableAll tracks

tool-call-reliability.py

#!/usr/bin/env python3
"""Measure how reliably a model and server emit valid tool calls.

Purpose: the course's tool-calling reliability test. Sends a fixed set of prompts -
    fifteen that should produce exactly one named call and five that should produce
    none - to any OpenAI-compatible endpoint with the same tool list every time, and
    reports six rates: call rate, parse rate, right-tool rate, schema validity,
    argument correctness and false-call rate. Nothing here asks a model for an
    opinion; every check is a comparison against the schema in tool-prompts.json.
    Part 25 reuses this script to compare coding models, and Part 27 uses the
    per-case failures to decide what to fine-tune.
Platform: all (pure Python over HTTP; the server may be on any track or another machine)
Minimum memory: 8 GB on the machine running the model; this script needs almost none
Assumes: Python 3.9 or later and no third-party packages. An OpenAI-compatible
    /v1/chat/completions endpoint reachable at --base-url: llama-server from Part 6,
    Ollama or LM Studio from Part 7, vLLM from Part 9, or the Part 9 gateway. The
    server must already be configured for tool calling (Part 9 covers the per-engine
    flags). tool-prompts.json sits beside this file.

Usage: python3 tool-call-reliability.py --base-url http://127.0.0.1:4000/v1 \
           --model local/chat --repeat 5 --labbook labbook.md
       python3 tool-call-reliability.py --base-url http://127.0.0.1:8080/v1 \
           --model local-chat --tool-choice required --temperature 0
       python3 tool-call-reliability.py --base-url http://127.0.0.1:8080/v1 \
           --model local-chat --no-think --json > reliability.json
"""

from __future__ import annotations

import argparse
import json
import os
import re
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple

HERE = Path(__file__).resolve().parent
DEFAULT_PROMPTS = HERE / "tool-prompts.json"

# Fragments that mean "the model tried to call a tool and nothing parsed it back".
# Each one is a real format from a model family in this course's reference set, seen
# as visible text: Hermes-style XML from Qwen, harmony channels from gpt-oss, the
# pythonic form, and LM Studio's default format for models with no native template.
UNPARSED_MARKERS = (
    "<tool_call>",
    "</tool_call>",
    "<|channel|>commentary",
    "functions.",
    "[TOOL_REQUEST]",
    "<function=",
    "<tool_use>",
)
LOOSE_JSON_CALL = re.compile(r'\{\s*"(?:name|tool|function)"\s*:\s*"', re.IGNORECASE)


# --------------------------------------------------------------------------------------
# A JSON Schema check small enough to read, covering what tool schemas actually use
# --------------------------------------------------------------------------------------

JSON_TYPES = {
    "string": str,
    "integer": int,
    "number": (int, float),
    "boolean": bool,
    "array": list,
    "object": dict,
}


def schema_errors(arguments: Any, schema: Dict[str, Any]) -> List[str]:
    """Every way `arguments` fails `schema`, as short human-readable strings.

    Deliberately not a full JSON Schema implementation: it covers object type,
    required, additionalProperties, per-property type, enum and numeric bounds,
    which is everything a tool schema in this course uses. A validator you can
    read is worth more here than one you have to trust.
    """
    errors: List[str] = []
    if not isinstance(arguments, dict):
        return ["arguments are not a JSON object"]

    properties = schema.get("properties", {}) or {}
    for name in schema.get("required", []) or []:
        if name not in arguments:
            errors.append("missing required parameter %s" % name)

    if schema.get("additionalProperties") is False:
        for name in arguments:
            if name not in properties:
                errors.append("invented parameter %s" % name)

    for name, value in arguments.items():
        spec = properties.get(name)
        if not isinstance(spec, dict):
            continue
        wanted = spec.get("type")
        expected = JSON_TYPES.get(wanted) if isinstance(wanted, str) else None
        if expected is not None:
            # JSON has no integer type at the wire level, so a whole float is an integer.
            if wanted == "integer" and isinstance(value, float) and value.is_integer():
                value = int(value)
            if isinstance(value, bool) and wanted != "boolean":
                errors.append("%s is a boolean, expected %s" % (name, wanted))
            elif not isinstance(value, expected):
                errors.append("%s is %s, expected %s" % (name, type(value).__name__, wanted))
        if "enum" in spec and value not in spec["enum"]:
            errors.append("%s=%r is not one of %s" % (name, value, spec["enum"]))
        if isinstance(value, (int, float)) and not isinstance(value, bool):
            if "minimum" in spec and value < spec["minimum"]:
                errors.append("%s is below the minimum" % name)
            if "maximum" in spec and value > spec["maximum"]:
                errors.append("%s is above the maximum" % name)
    return errors


def argument_errors(arguments: Dict[str, Any], expected: Dict[str, Any]) -> List[str]:
    """Check the values a case says the call should carry.

    Three comparisons, which is all the test set needs: `equals` for an exact value
    (numbers compared numerically so 24 and 24.0 agree), `contains` for a
    case-insensitive substring, and `one_of` for a small set of acceptable answers.
    """
    errors: List[str] = []
    for name, rule in (expected or {}).items():
        if name not in arguments:
            errors.append("expected %s in the arguments" % name)
            continue
        got = arguments[name]
        if "equals" in rule:
            want = rule["equals"]
            same = (
                abs(float(got) - float(want)) < 1e-9
                if isinstance(want, (int, float)) and isinstance(got, (int, float))
                and not isinstance(got, bool)
                else got == want
            )
            if not same:
                errors.append("%s=%r, expected %r" % (name, got, want))
        if "contains" in rule and rule["contains"].lower() not in str(got).lower():
            errors.append("%s=%r does not contain %r" % (name, got, rule["contains"]))
        if "one_of" in rule and got not in rule["one_of"]:
            errors.append("%s=%r is not one of %s" % (name, got, rule["one_of"]))
    return errors


# --------------------------------------------------------------------------------------
# Talking to the server
# --------------------------------------------------------------------------------------

def post_chat(base_url: str, payload: Dict[str, Any], api_key: Optional[str],
              timeout: int) -> Dict[str, Any]:
    """One /v1/chat/completions request. Raises RuntimeError with a readable message."""
    url = base_url.rstrip("/") + "/chat/completions"
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(url, data=body, method="POST")
    request.add_header("Content-Type", "application/json")
    if api_key:
        request.add_header("Authorization", "Bearer " + api_key)
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:400]
        raise RuntimeError("HTTP %s from %s: %s" % (exc.code, url, detail)) from exc
    except urllib.error.URLError as exc:
        raise RuntimeError("could not reach %s: %s" % (url, exc.reason)) from exc


def first_call(message: Dict[str, Any]) -> Tuple[Optional[str], Optional[str], int]:
    """The name and raw argument string of the first tool call, and how many there were."""
    calls = message.get("tool_calls") or []
    if not calls:
        return None, None, 0
    function = calls[0].get("function", {}) or {}
    return function.get("name"), function.get("arguments"), len(calls)


def looks_like_an_unparsed_call(text: str) -> bool:
    """True when the content carries a tool-call format that nothing extracted.

    This is the single most useful diagnostic in the whole test: it separates
    "the model would not call the tool" from "the model called it and the server's
    parser did not recognise the format", which are fixed in completely different
    places.
    """
    if not text:
        return False
    if any(marker in text for marker in UNPARSED_MARKERS):
        return True
    return bool(LOOSE_JSON_CALL.search(text))


# --------------------------------------------------------------------------------------
# The measurement
# --------------------------------------------------------------------------------------

def run_case(case: Dict[str, Any], tools: List[Dict[str, Any]], args: argparse.Namespace,
             schemas: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
    """Send one case `--repeat` times and count what came back."""
    counts = {
        "attempts": 0, "called": 0, "parsed": 0, "right_tool": 0,
        "schema_valid": 0, "args_correct": 0, "unparsed_in_text": 0,
        "multiple_calls": 0, "errors": 0,
    }
    notes: List[str] = []

    payload_base: Dict[str, Any] = {
        "model": args.model,
        "messages": [
            {"role": "system", "content": args.system},
            {"role": "user", "content": case["prompt"]},
        ],
        "tools": tools,
        "temperature": args.temperature,
        "max_tokens": args.max_tokens,
    }
    if args.tool_choice != "auto":
        payload_base["tool_choice"] = args.tool_choice
    if args.no_think:
        # Documented by Qwen for its own models and accepted by vLLM as a per-request
        # template argument. Servers that do not know the key ignore it.
        payload_base["chat_template_kwargs"] = {"enable_thinking": False}

    for _ in range(args.repeat):
        counts["attempts"] += 1
        try:
            response = post_chat(args.base_url, dict(payload_base), args.api_key, args.timeout)
        except RuntimeError as exc:
            counts["errors"] += 1
            notes.append(str(exc)[:160])
            continue

        message = (response.get("choices") or [{}])[0].get("message", {}) or {}
        name, raw_arguments, how_many = first_call(message)
        content = message.get("content") or ""

        if name is None:
            if looks_like_an_unparsed_call(content):
                counts["unparsed_in_text"] += 1
                notes.append("call format in content: " + content.strip()[:120])
            continue

        counts["called"] += 1
        if how_many > 1:
            counts["multiple_calls"] += 1

        try:
            arguments = json.loads(raw_arguments) if isinstance(raw_arguments, str) else raw_arguments
            if not isinstance(arguments, dict):
                raise ValueError("arguments are not an object")
        except (TypeError, ValueError) as exc:
            notes.append("unparseable arguments (%s): %r" % (exc, str(raw_arguments)[:100]))
            continue
        counts["parsed"] += 1

        if case.get("expect_tool") is None or name != case["expect_tool"]:
            if case.get("expect_tool") is not None:
                notes.append("called %s, expected %s" % (name, case["expect_tool"]))
            continue
        counts["right_tool"] += 1

        problems = schema_errors(arguments, schemas.get(name, {}))
        if problems:
            notes.append("; ".join(problems[:3]))
            continue
        counts["schema_valid"] += 1

        wrong = argument_errors(arguments, case.get("expect_args", {}))
        if wrong:
            notes.append("; ".join(wrong[:3]))
            continue
        counts["args_correct"] += 1

    # Keep the report short: the first three distinct notes say what went wrong.
    seen: List[str] = []
    for note in notes:
        if note not in seen:
            seen.append(note)
    return {"id": case["id"], "expect_tool": case.get("expect_tool"),
            "counts": counts, "notes": seen[:3]}


def summarise(results: List[Dict[str, Any]]) -> Dict[str, Any]:
    """The six rates, computed over the cases each one is meaningful for."""
    expected = [r for r in results if r["expect_tool"] is not None]
    unexpected = [r for r in results if r["expect_tool"] is None]

    def total(rows: List[Dict[str, Any]], key: str) -> int:
        return sum(r["counts"][key] for r in rows)

    def rate(numerator: int, denominator: int) -> Optional[float]:
        return round(numerator / denominator, 4) if denominator else None

    wanted = total(expected, "attempts")
    called = total(expected, "called")
    return {
        "call_rate": rate(called, wanted),
        "parse_rate": rate(total(expected, "parsed"), called),
        "right_tool_rate": rate(total(expected, "right_tool"), wanted),
        "schema_valid_rate": rate(total(expected, "schema_valid"), wanted),
        "args_correct_rate": rate(total(expected, "args_correct"), wanted),
        "false_call_rate": rate(total(unexpected, "called"), total(unexpected, "attempts")),
        "unparsed_in_text": total(results, "unparsed_in_text"),
        "multiple_calls": total(results, "multiple_calls"),
        "request_errors": total(results, "errors"),
        "cases": len(results),
        "attempts": total(results, "attempts"),
    }


def print_report(results: List[Dict[str, Any]], rates: Dict[str, Any], args: argparse.Namespace) -> None:
    print("\n%-12s %-16s %8s %8s %8s %8s" % ("case", "expected tool", "called", "parsed", "valid", "correct"))
    for row in results:
        counts = row["counts"]
        print("%-12s %-16s %8d %8d %8d %8d" % (
            row["id"], row["expect_tool"] or "(none)",
            counts["called"], counts["parsed"], counts["schema_valid"], counts["args_correct"]))
        for note in row["notes"]:
            print("             %s" % note)

    def show(label: str, value: Optional[float]) -> str:
        return "%-22s %s" % (label, "n/a" if value is None else "%.1f%%" % (value * 100))

    print("\n%s, %s, temperature %s, %d attempts per case" % (
        args.model, args.base_url, args.temperature, args.repeat))
    print(show("call rate", rates["call_rate"]))
    print(show("parse rate", rates["parse_rate"]))
    print(show("right tool", rates["right_tool_rate"]))
    print(show("schema valid", rates["schema_valid_rate"]))
    print(show("arguments correct", rates["args_correct_rate"]))
    print(show("false calls", rates["false_call_rate"]))
    if rates["unparsed_in_text"]:
        print("\n%d response(s) contained a tool-call format as visible text. That is a parser or "
              "chat-template mismatch on the server, not a limitation of the model: check the "
              "engine's tool-call parser against the model family before concluding anything."
              % rates["unparsed_in_text"])
    if rates["request_errors"]:
        print("%d request(s) failed outright; the rates above are computed over the rest."
              % rates["request_errors"])


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base-url", default="http://127.0.0.1:4000/v1",
                        help="OpenAI-compatible base URL, ending in /v1")
    parser.add_argument("--model", required=True, help="the served model name or alias")
    parser.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY"),
                        help="bearer token, if the endpoint needs one; defaults to $OPENAI_API_KEY")
    parser.add_argument("--prompts", default=str(DEFAULT_PROMPTS), help="the test set to run")
    parser.add_argument("--repeat", type=int, default=5, help="attempts per case")
    parser.add_argument("--temperature", type=float, default=0.7)
    parser.add_argument("--max-tokens", type=int, default=512)
    parser.add_argument("--tool-choice", default="auto", choices=["auto", "required", "none"])
    parser.add_argument("--no-think", action="store_true",
                        help="ask the server to disable thinking mode for this run")
    parser.add_argument("--system", default=(
        "You are a careful assistant with tools. Call a tool when one of them can answer the "
        "request. Answer directly when none of them applies."))
    parser.add_argument("--timeout", type=int, default=180)
    parser.add_argument("--json", action="store_true", help="print the whole report as JSON")
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    parser.add_argument("--notes", default=None, help="free text recorded with the run")
    args = parser.parse_args()

    if args.repeat > 1 and args.temperature == 0:
        print("note: at temperature 0 every attempt is the same sample, so --repeat measures "
              "nothing but server determinism.", file=sys.stderr)

    suite = json.loads(Path(args.prompts).read_text(encoding="utf-8"))
    tools = suite["tools"]
    schemas = {t["function"]["name"]: t["function"].get("parameters", {}) for t in tools}

    started = time.time()
    results = []
    for case in suite["cases"]:
        results.append(run_case(case, tools, args, schemas))
        print(".", end="", flush=True)
    elapsed = time.time() - started
    rates = summarise(results)

    if args.json:
        print(json.dumps({"rates": rates, "results": results}, indent=2))
    else:
        print_report(results, rates, args)
        print("\n%d requests in %.0f s" % (rates["attempts"], elapsed))

    if args.labbook:
        record = {
            "lab": "part-24/tool-call-reliability",
            "model": args.model,
            "base_url": args.base_url,
            "prompts": os.path.basename(args.prompts),
            "prompts_version": suite.get("version"),
            "repeat": args.repeat,
            "temperature": args.temperature,
            "tool_choice": args.tool_choice,
            "thinking_disabled": bool(args.no_think),
            "rates": rates,
            "seconds": round(elapsed, 1),
            "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S"),
            "notes": args.notes,
        }
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print("recorded in %s" % args.labbook)


if __name__ == "__main__":
    main()

Download tool-call-reliability.py413 lines

Run it against the gateway, twice, once per model you are choosing between:

RunnableAll tracks

measure two models against the same test set

python3 tool-call-reliability.py \
    --base-url http://127.0.0.1:4000/v1 \
    --model local/chat \
    --repeat 5 \
    --labbook labbook.md \
    --notes "Q4_K_M, llama-server, --jinja"

python3 tool-call-reliability.py \
    --base-url http://127.0.0.1:4000/v1 \
    --model local/coder \
    --repeat 5 \
    --labbook labbook.md \
    --notes "Q4_K_M, llama-server, --jinja"

=Expected result

A working setup gives a high call rate and parse rate, a right-tool rate close to the call rate, and a false-call rate near zero. The interesting output is the per-case list underneath, which names the cases that failed and why: a wrong unit, an invented top_k parameter, a call to read_file when search_notes was wanted. Those are the cases to look at, not the totals.

One diagnostic in the report deserves its own paragraph. The script scans the text content of every reply for tool-call formats that arrived as visible text: <tool_call> tags, a harmony commentary channel marker, [TOOL_REQUEST], a functions. prefix, or a bare JSON object with a name key. When that count is not zero, the model tried to call the tool and the server did not recognise the format. That is a parser or template problem, fixed on the server with the flags in Part 9, and no amount of prompt-writing will improve it.

∴Research note

Record the conditions beside the rates, which is why the script writes model, base URL, temperature, tool_choice, whether thinking was disabled, and the prompt-set version into labbook.md. Two runs of the same model at different temperatures are two different measurements. Part 16 makes the same point about quantisation: format-following degrades before prose quality does, so the Q4 model that still writes well may have stopped calling tools reliably, and only this test will tell you.

Make retries safe at the execution boundary

Suppose a tool creates a ticket, but the response is lost before the agent receives it. Retrying the same proposed action can create a duplicate ticket. A tool-call identifier connects messages in a conversation; it is not automatically an application-level idempotency key enforced by the external system.

Separate validation, authorisation and execution. Validate the argument schema and permitted identifiers, then attach a stable operation identity where the tool supports it. Record whether the action completed before returning its result. For an uncertain outcome, query the operation state or ask for review instead of blindly repeating a consequential call.

Return structured errors that distinguish invalid arguments, denied permission, temporary unavailability and unknown completion state. The loop can then make a bounded decision about repair or retry. Test a multi-call response too: independent read operations may be parallelisable, while dependent writes must respect ordering. API compatibility covers message shape; reliable function calling additionally requires correct state management and execution semantics.

Recap

One tool call is four messages: your request with tools, an assistant message carrying a tool_calls array whose arguments is a JSON string, your second request containing that same assistant message plus one tool message per tool_call_id, and the reply. Every model family emits its own format, from Hermes-style tags for Qwen3 to harmony channels for gpt-oss, and the engine’s parser has to match it, so a format visible as text in the content is a server problem rather than a model one. Ollama adds think and message.thinking and a tool_name field on results; LM Studio distinguishes native tool use from a default mode that wraps calls in [TOOL_REQUEST] tags, which makes weak models parse without making them good. Parallel calls are a per-model capability that is off by default in llama.cpp and unreliable in several families, so write the loop for an array and assume one entry. Reasoning content arrives in a separate field whose name has changed recently, and calls are never parsed out of it. And none of that is worth assuming: run the reliability test, record the six rates with the conditions, and keep the number.

Check your understanding

Question 1. The reply contains a tool_calls array, and your client passes message.tool_calls[0].function.arguments straight into your function as keyword arguments. What breaks?

• Nothing; arguments is already an object
• arguments is a JSON string, so it has to be parsed first, and it can be malformed even when the rest of the response is valid
• The tool_call_id is missing
• The content field must be read instead

Show the answer and why

Answer: arguments is a JSON string, so it has to be parsed first, and it can be malformed even when the rest of the response is valid

The OpenAI shape carries arguments as a string containing JSON. That second parse is the place a model's malformed output surfaces, which is why the reliability test counts parse rate separately from call rate: a call that arrived but would not parse is a different fault from no call at all.

Question 2. Your report shows a call rate near zero and eight responses whose content contains <tool_call> tags as visible text. What do you change?

• The system prompt, to ask more firmly for tool use
• The server: the model is emitting Hermes-style calls and the engine is not running a parser that recognises them
• The model, to a larger one
• The temperature

Show the answer and why

Answer: The server: the model is emitting Hermes-style calls and the engine is not running a parser that recognises them

The format appearing as text is the signature of a parser or chat-template mismatch. The model did its part. Fix it with the engine flags from Part 9 - the right --tool-call-parser on vLLM, --jinja on llama-server - and rerun the test before drawing any conclusion about the model.

Question 3. Which statements about parallel tool calls are supported by the documentation cited here? Select all that apply.

• llama.cpp disables multiple tool calling by default and enables it with "parallel_tool_calls": true in the request payload
• vLLM states that parallel tool calls are not supported for Llama 3 but are supported in Llama 4
• Every model that supports tool calling supports parallel calls
• A client should iterate the tool_calls array and append one tool message per tool_call_id

Show the answer and why

Answer: llama.cpp disables multiple tool calling by default and enables it with "parallel_tool_calls": true in the request payload, vLLM states that parallel tool calls are not supported for Llama 3 but are supported in Llama 4, A client should iterate the tool_calls array and append one tool message per tool_call_id

Support varies by family and by engine default, which is exactly why the third option is wrong. Handling the array properly costs three lines and prevents a conversation with an unanswered call in it, which derails the following turn.

Question 4. A reasoning model is serving through vLLM with a reasoning parser enabled, and your loop sees thinking but never a tool call. What does the documentation say to check first?

• That the model supports tools at all
• That the call is being emitted in the content field rather than inside the reasoning, since tool calling only parses functions from the content field
• That the temperature is above zero
• That parallel calls are enabled

Show the answer and why

Answer: That the call is being emitted in the content field rather than inside the reasoning, since tool calling only parses functions from the content field

vLLM states that tool calling parses functions from the content field, not from the reasoning. A model that writes its call inside its thinking produces exactly this symptom. Disabling thinking for tool-heavy turns, or using a non-thinking model, is the usual fix, and the next lesson works out when that trade is worth making.

Check my answersClear