Tool Calling and Structured Output on the Server Side

By the end of this lesson you will be able to make a local server return function calls and schema-valid JSON the way a hosted API does, name the two independent things that have to be right for that to work, and design a small test that tells you how reliably your model and server actually do it.

This is the least glamorous lesson in the part and the one Level 5 depends on most. Part 24 builds an agent loop, Part 25 puts a coding agent on a local model, Part 26 builds multi-agent systems. All three are a client sending a tool list and expecting a parseable tool call back. When that fails, it fails as a silent behavioural fault, not as an error.

A tool call is not a model feature

The OpenAI-shaped conversation is: the client sends messages and a list of tools; the model decides to call one; the response comes back with a tool_calls array holding a function name and a JSON argument object; the client runs the function and sends the result back as another message.

Underneath, none of that exists. The model emits tokens. Two pieces of server-side machinery turn tokens into that clean structure, and either can be wrong independently.

What a server does between a tools list and a tool_calls array

1. 1The client sends messages and toolsOrdinary JSON over the chat completions endpoint. The client knows nothing about the model.
2. 2The chat template renders them into one token sequenceEach model family was trained with a specific way of presenting available functions. The template is shipped with the model and rendered by a Jinja engine.
3. 3The model generates tokensIt emits whatever pattern its training associated with calling a function: a JSON blob, a Python-looking call, a special token followed by arguments, or text in a named channel.
4. 4The tool-call parser reads them backA per-family parser recognises that pattern and extracts the name and arguments. Choose the wrong parser and the call arrives as ordinary text.
5. 5The server emits a tool_calls arrayNow it looks like a hosted API, and the client can run the function.
6. 6The result goes back as a tool messageThe template renders it again, in the form the model was trained to read results in, and the loop continues.

Template and parser are two halves of one convention, and they must agree with each other and with the model. Nothing reports a mismatch at any layer: the client simply sees a chatty answer where it expected a structured one.

✕Common mistake

The classic symptom is a tool call that arrives as prose. The response has no tool_calls array, and the content is a plausible-looking JSON blob, or a line of Python, or something like <tool_call>{"name": ...} rendered as visible text. That is not the model refusing to use tools. It is the parser not recognising what the model emitted. Check which parser the server is running before you conclude anything about the model, and check the model card for which format it was trained on.

Turning it on, per engine

Every engine needs to be told two things: use tool calling, and use this parser.

Engine	What you pass	Notes
vLLM	--enable-auto-tool-choice together with --tool-call-parser <name>, optionally --chat-template <file>	--tool-parser-plugin registers a custom parser
SGLang	--tool-call-parser <name>	The documented launch form is python3 -m sglang.launch_server --model-path <MODEL> --tool-call-parser <PARSER_NAME>
llama-server	--jinja	The function-calling document states that function calling “is supported for all models” when the server is started with it
Ollama	Nothing at the server; pass tools in the request	The API reference documents tools as “list of tools in JSON for the model to use if supported”

⌚Version note

The vLLM and SGLang tool and grammar options are confirmed in their documentation as read on 2026-09-09, and are not yet in this course’s captured command reference for vLLM 0.28.0 · verified 2026-09-08 or SGLang 0.5.19 · verified 2026-09-08, so they appear here in prose and in the table rather than in runnable blocks. The reference is extended once the options are captured from the binaries on the validation machines. The llama-server flags below are recorded and appear in runnable blocks as normal.

Choosing the parser

The parser name is a property of the model family, not of the engine, and the two engines spell them differently. A short comparison of the ones this course’s reference models need, from both documentation pages as read on 2026-09-09:

Model family	vLLM parser	SGLang parser
Qwen3, general	hermes	qwen
Qwen3-Coder	qwen3_xml	qwen3_coder
gpt-oss	openai	gpt-oss
Llama 3.1 to 3.3	llama3_json	llama3
Llama 4	llama4_pythonic	llama4
DeepSeek-V3	deepseek_v3	deepseekv3
Mistral	mistral	mistral
GLM	glm45, glm47	glm
Models emitting Python-style calls	pythonic	pythonic

vLLM’s list is longer than this and includes parsers for Granite, Jamba, InternLM, Kimi, Hunyuan, xLAM, Olmo and others; SGLang’s includes Apertus, Step-3 and the DeepSeek 3.1 and 3.2 variants. Read the engine’s own page for the model you are actually serving, because the sets change with each release.

✕Common mistake

A serving configuration is not portable between engines. --tool-call-parser hermes on vLLM and --tool-call-parser qwen on SGLang can be serving the same weights. Copying a command line from a blog post about the other engine is a reliable way to produce the prose-instead-of-tool-call symptom above.

What tool_choice does

vLLM documents four behaviours: "auto", where the model generates tool calls when it judges them appropriate; "required", where it “must generate at least one tool call”; "none", where no tool calls are generated; and named function calling, where the request specifies exactly which function to call.

"required" is the one that saves agent loops. When your control flow has no sensible branch for a chatty answer, forcing a call turns a class of runtime surprise into a schema you can rely on.

vLLM also states limitations plainly, and they are worth reading before you design around a model: Llama 3 does not support parallel tool calls while Llama 4 does; Mistral 7B’s parallel calling is described as degraded; some models produce malformed parameter formats; and the pythonic parser “requires models to output tool calls without mixed text generation”, so a model that likes to explain itself first will break it.

llama-server

Track M and any track without vLLM does this with llama.cpp, and it works well.

RunnableAll tracks

a server that speaks OpenAI-style function calls

~/llama.cpp/build/bin/llama-server \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 \
    --alias local-chat \
    -c 8192 -ngl 99 \
    --jinja

The server README gives --jinja as enabled by default, and the function-calling document ties tool calling to it explicitly. That document lists the families with native tool-call formats supported, naming Llama 3.1, 3.2 and 3.3 including built-in tools, Functionary v3.1 and v3.2, Hermes 2 and 3, Qwen 2.5, Mistral Nemo and Command R7B. For anything else, “generic tool call is supported when the template isn’t recognized by native format handlers”, with the honest caveat that “generic support may consume more tokens and be less efficient than a model’s native format”.

When a GGUF’s packaged template is wrong or missing, --chat-template-file replaces it, and the document offers replacements for several model families. Part 6’s /apply-template endpoint is how you check what the server is actually rendering before you blame the model.

Structured output

Tool calling is a special case of a more general capability: constraining generation so that the output conforms to a grammar. The general case is worth having on its own, because a great deal of practical work is extraction and classification where you want a schema, not a paragraph.

The mechanism is the same everywhere. At each step the model produces a probability distribution over the vocabulary; the constraint engine works out which tokens could legally come next given the schema and what has been emitted so far, and sets the rest to zero probability. The output is then valid by construction rather than by hope.

The engines that do this locally:

• xgrammar — SGLang’s default backend, supporting “JSON schema, regular expression, and EBNF constraints”. Also available in vLLM.
• outlines — supports “JSON schema and regular expression constraints”.
• llguidance / guidance — JSON schema, regex and EBNF.
• llama.cpp’s own GBNF grammars — the server README documents --grammar as a “BNF-like grammar to constrain generations”, --grammar-file to read one from a file, and -j, --json-schema to constrain to a JSON schema directly.

vLLM’s structured-outputs page names xgrammar, guidance, outlines, lm-format-enforcer and auto, selected with --structured-outputs-config.backend, and describes auto as choosing a backend from the details of each request. It also notes that the older guided_json, guided_regex, guided_choice and guided_grammar request fields are deprecated from version 0.12.0 in favour of a structured_outputs object, so "guided_json" -> {"structured_outputs": {"json": ...}}. SGLang takes json_schema, regex or ebnf, exactly one per request, with --grammar-backend choosing between XGrammar, Outlines and Llguidance.

The portable way to ask for JSON across all of them is the OpenAI response_format field:

RunnableAll tracks

a schema-constrained extraction, over the OpenAI-compatible API

curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "local-chat",
      "messages": [
        {"role": "system", "content": "Extract the fields. Reply with JSON only."},
        {"role": "user", "content": "The DGX Spark has 128 GB of unified memory and one GB10 chip."}
      ],
      "response_format": {
        "type": "json_schema",
        "json_schema": {
          "name": "machine",
          "schema": {
            "type": "object",
            "properties": {
              "machine": {"type": "string"},
              "memory_gb": {"type": "integer"},
              "chip": {"type": "string"}
            },
            "required": ["machine", "memory_gb", "chip"],
            "additionalProperties": false
          }
        }
      }
    }'

Ollama does the same thing through its own format parameter, documented as “the format to return a response in. Format can be json or a JSON schema”, on both /api/generate and /api/chat. LM Studio serves an OpenAI-compatible API from its developer surface, covered in Part 7; its documentation indexes the OpenAI compatibility page under Developer, and its Advanced section covers parallel requests and speculative decoding for the same server.

!Important

Constrained decoding makes output parseable, not correct. A schema forces well-formed JSON with the right field names and types. It has nothing to say about whether memory_gb is the right number. A model that would have got the answer wrong in prose will get it wrong inside a valid object, and now your program will accept it without complaint. Validate the values, not just the shape.

Measuring whether it works

Do not take any of this on trust for your model, your quantisation and your server settings. The failure modes are quiet and they are quantisation-sensitive: a heavily quantised model often keeps its prose quality while losing its ability to emit a format reliably.

The test is small and worth keeping. Write twenty prompts that should produce a tool call, and five that should not. Send each one ten times at the temperature you actually use. Then count four things.

Call rate. How often a tool call was produced when one was expected.

Parse rate. How often the response contained a well-formed tool_calls array rather than prose.

Argument validity. How often the arguments matched the function’s schema: right names, right types, no invented parameters.

False call rate. How often a call was produced for one of the five prompts where none was wanted. This is the one people forget, and it is the one that makes an agent loop wander.

Record model, quantisation, engine, engine version, parser, temperature and date beside the four numbers. Part 10’s evaluation lab turns this into a harness you keep; Part 16 uses the same test to show what a quantisation step costs; and Part 25 is the part that will thank you, because a coding agent on a model with an argument validity of four in five is not a coding agent, it is a source of plausible nonsense with a retry loop.

⛨Security

A tool call is a request from a token predictor to run code on your machine. Nothing about the mechanism in this lesson makes the request trustworthy, and a prompt injected into a document the model read can produce a perfectly well-formed call to a function you did not want called. Validate arguments against a schema, allow-list the functions a session may reach, and keep anything with real consequences behind a confirmation. Part 10 introduces prompt injection properly and Part 24 sandboxes the agent loop; until then, keep tool calling on localhost and on functions that cannot do damage.

Separate transport, syntax, semantics and authority

A tool response has several independent success conditions. The HTTP request must succeed; the client must receive a tool-call object; the arguments must parse and validate; the selected tool and values must be appropriate; and the application must authorise the action. Passing an earlier condition does not imply the next one.

Build a small probe set containing a valid call, an ambiguous request, a request needing no tool and an invalid argument. Inspect the raw response before connecting a framework that may repair or hide errors. If a model emits textual JSON instead of a tool-call object, inspect template and parser configuration. If it selects the wrong customer identifier in valid JSON, investigate the task and validation logic.

The application owns execution. Validate arguments against the declared schema, resolve identifiers against permitted resources and apply the caller’s permissions before invoking the function. Feed a structured success or error result back into the conversation with the matching call identifier. Constrained decoding can help syntax; it cannot grant authority or establish that the proposed action is correct.

Recap

A tool call is produced by two pieces of server machinery working together: a chat template that renders the tool list into the format the model was trained on, and a parser that recognises the model’s output and turns it back into a tool_calls array. Either can be wrong on its own, and the symptom in both cases is a structured call arriving as prose rather than an error. vLLM needs --enable-auto-tool-choice with a --tool-call-parser, SGLang needs a --tool-call-parser whose names differ from vLLM’s for the same models, llama-server needs --jinja and has a generic fallback for unrecognised templates, and Ollama takes tools in the request. tool_choice: "required" removes a whole class of surprise from an agent loop. Structured output is the same idea generalised, with xgrammar, outlines, llguidance and llama.cpp’s GBNF grammars all masking illegal tokens at each step, and it makes output parseable without making it correct. And none of it should be assumed: measure call rate, parse rate, argument validity and false calls for your model and quantisation, and record the conditions beside the numbers.

Check your understanding

Question 1. A client sends a tools list and gets back a chatty answer containing a JSON blob as visible text, with no tool_calls array. What is the most likely cause?

• The model is too small to use tools
• The server's tool-call parser does not match the format this model family emits, so the call was never recognised and extracted
• The client sent malformed JSON
• The temperature is too high

Show the answer and why

Answer: The server's tool-call parser does not match the format this model family emits, so the call was never recognised and extracted

The model emitted the pattern it was trained to emit; nothing downstream recognised it. Check the parser name against the model family in the engine's own documentation. It is also worth checking the chat template, because a template that never presented the tools would produce the same visible symptom.

Question 2. You have a working vLLM command line with --tool-call-parser hermes for a Qwen3 model and you move to SGLang. What do you need to change?

• Nothing; parser names are standardised
• The parser name: SGLang documents qwen for the Qwen series, and its names differ from vLLM's for the same models
• The model must be re-quantised
• The client has to send a different request format

Show the answer and why

Answer: The parser name: SGLang documents qwen for the Qwen series, and its names differ from vLLM's for the same models

The client API is portable between the two engines; the serving configuration is not. Both documentation pages list their own parser names, and copying a command line between engines is a common source of the prose-instead-of-tool-call failure.

Question 3. Grammar-constrained decoding is enabled and every response is valid against your JSON schema. What have you established?

• The answers are correct
• The answers are well formed: the right fields with the right types. Whether the values are right is a separate question the schema cannot answer
• The model has been fine-tuned for your task
• Tool calling will now work as well

Show the answer and why

Answer: The answers are well formed: the right fields with the right types. Whether the values are right is a separate question the schema cannot answer

Constrained decoding masks illegal tokens at each step, so the shape is valid by construction. A wrong number in a valid object is more dangerous than a wrong sentence, because your program will now accept it silently. Validate values as well as shape.

Question 4. Which measurements belong in a tool-calling reliability test? Select all that apply.

• How often a call was produced when one was expected
• How often the arguments matched the function's schema, with no invented parameters
• How often a call was produced on prompts where none was wanted
• The model's score on a public reasoning benchmark

Show the answer and why

Answer: How often a call was produced when one was expected, How often the arguments matched the function's schema, with no invented parameters, How often a call was produced on prompts where none was wanted

The three that matter are call rate, argument validity and false calls, all recorded with the model, quantisation, engine, parser, temperature and date. A public benchmark score tells you nothing about whether this quantisation of this model emits this parser's format reliably, and quantisation degrades format-following before it degrades prose.

Check my answersClear