Structured Output and JSON Mode

Asking politely for JSON works most of the time, and “most of the time” is a synonym for “breaks in production”. By the end of this lesson you will be able to make a local model return an object that matches a schema on every request, explain the difference between constraining the sampler and validating the reply, and name the failure modes that neither of those two fixes.

Three places to enforce a shape

There are exactly three points at which output structure can be imposed, and they are not alternatives. A serious pipeline uses all three.

The three layers, and what each one can and cannot do

1. 1The promptAsk for JSON, show an example. Costs nothing, enforces nothing: the model may still add prose, a code fence, or an apology.
2. 2The samplerA schema or grammar sent with the request masks every token that would break the structure. The reply is structurally valid by construction.
3. 3The validatorParse the reply into a typed object. Catches truncation, wrong types the grammar allowed, and values that are the right shape and the wrong answer.
4. 4The repair loopOn a validation failure, show the model its own output and the error, and ask again. Bounded, and it fails loudly when the bound is reached.

Each layer catches what the one before it lets through. Only the last one can reject an object whose shape is right and whose contents are wrong.

A schema in the request

The OpenAI-compatible chat endpoint carries structure in a response_format field, and local servers implement it. The llama-server README documents support for “both plain JSON output (e.g. {"type": "json_object"}) and schema-constrained JSON”, the latter through a json_schema object. Its native /completion endpoint takes the same thing under two parameter names: json_schema to “Set a JSON schema for grammar-based sampling”, and grammar to “Set grammar for grammar-based sampling”.

The distinction between the two response_format types matters. json_object says only “emit syntactically valid JSON”; the keys and types are still whatever the model felt like. json_schema says “emit JSON that matches this shape”. Use the second one. The first is worth knowing about only because it is what you get when a server does not support the second, and because a reply that parses but has none of your fields is the symptom.

vLLM exposes the same capability with more knobs. Its structured-outputs page documents constraining output to one of a set of choices, to a regular expression, to a JSON schema, to a context-free grammar, or to a JSON schema “within a set of specified tags”. The OpenAI-compatible response_format with "type": "json_schema" works there too, which is why the same client code runs against both engines and against the gateway from Part 9.

What the constraint actually does

Underneath, both engines do the same thing: at every decoding step they compute which tokens could continue a valid document and set the probability of all the others to zero. This is worth internalising because it explains both the power and the limits.

llama.cpp’s format for expressing those rules is GBNF, described in its guide as “a format for defining formal grammars to constrain model outputs in llama.cpp”. It is Backus-Naur Form with regex-like conveniences: a root rule as the entry point, terminals written as literal characters or ranges such as [1-9], negation with [^\n], and the repetition operators *, +, ?, {m} and {m,n}. You rarely write one by hand for JSON, because the project ships json_schema_to_grammar.py to convert a schema ahead of time, and the server accepts a schema directly. You do write one by hand when the output is not JSON: a chess move, a SQL identifier, a date, a single word from a fixed list.

RunnableAll tracks

constrain every reply on this server to one grammar

llama-server \
  --model ~/models/qwen3-4b/Qwen3-4B-Q4_K_M.gguf \
  --alias qwen3-4b \
  --jinja \
  --grammar-file ~/grammars/verdict.gbnf \
  --host 127.0.0.1 \
  --port 8080

vLLM’s page names its backends: xgrammar, guidance (also called llguidance), outlines and lm-format-enforcer, selected with --structured-outputs-config.backend and defaulting to auto, which “will try to choose an appropriate backend based on the details of the request”. You will meet these names in issue threads; they are implementations of the same idea, differing in which schema features they support and how much time they spend compiling the constraint before the first token.

✕Common mistake

The grammar constrains the first token as much as the last. Point it at a model in thinking mode and you have forbidden the <think> block the model was trained to open with, which is a good way to get a valid, empty and useless object. Turn thinking off for structured tasks, using the switches from the previous lesson, or route the structured call to a non-thinking model.

!Important

The GBNF guide lists real limits on schema conversion, and one of them is quiet: “unsupported features are silently skipped”. additionalProperties defaults to false, properties cannot be mixed with anyOf or oneOf, and uniqueItems, contains and the if/then/else conditionals are not supported. A constraint you wrote and that was skipped is worse than no constraint, because you stopped checking. This is the argument for keeping the validator even when the sampler is constrained.

One class, two jobs

Pydantic is the piece that keeps the schema and the check from drifting apart. It is described in its own documentation as “the most widely used data validation library for Python”, where “schema validation and serialization are controlled by type annotations”. Define the shape once as a class, and it gives you both the JSON Schema to send with the request and the validator to run on the reply.

Fragment — not complete on its own

class ModelRecord(BaseModel):
    name: str
    publisher: Optional[str] = None
    total_params_b: Optional[float] = None
    licence: Optional[str] = None
    quantisations: List[str] = Field(default_factory=list)

# the same class, used twice
schema = ModelRecord.model_json_schema()          # goes in the request
record = ModelRecord.model_validate_json(reply)   # checks what came back

When validation fails, Pydantic “will raise an error with a breakdown of what was wrong”, listing for each problem the error type, the field location, the message and the offending input. That breakdown is not just for your log: it is the most useful thing you can put in a repair prompt, because it tells the model precisely which field to fix.

The failure modes that survive constrained decoding

A constrained sampler makes the shape correct. It does nothing about the four failures below, which is why the validator and the repair loop exist.

Truncation. The grammar decides which tokens are allowed; the token budget decides when generation stops. Hit max_tokens in the middle of an object and you get a valid prefix of an invalid document. The symptom is a JSON decode error on long extractions, and the fix is a larger budget or a smaller schema, not a better prompt.

Right shape, wrong values. "total_params_b": 70.0 for an 8B model satisfies every constraint you wrote. Schemas cannot check facts. Narrow the type where you can, with enumerations, ranges and patterns, and accept that the rest is an evaluation problem, which is what this part’s lab is for.

Silent nulls and empty lists. A model that finds nothing will often return an empty array rather than say so, and an empty array is valid. If “found nothing” is a meaningful outcome, give it a field of its own and make it required.

Invented fields filled in confidently. With additionalProperties false the model cannot add fields, so instead it puts something plausible in the fields you did give it. This is the same behaviour that makes ungrounded question answering dangerous, and it is handled the same way: ask for the evidence alongside the value, and check that the evidence is really in the input.

Schemas that survive contact with a model

A schema that is valid JSON Schema is not automatically a schema a model handles well. Five rules, learned the same way everybody learns them.

Flat beats nested. Every level of nesting is another structure the model has to hold open while it writes, and another place a repair loop has to describe when something goes wrong. Two flat objects and a join in your own code are usually better than one deeply nested object.

Enumerate wherever you can. A field typed as a string can hold anything; a field typed as an enumeration of five values can hold five things. Constrained decoding turns that from a preference into an impossibility, which is the strongest form of validation available and costs one line.

Name fields the way a person would. The field name is in the prompt, in effect: a model fills total_params_billions more reliably than tpb. Short names save tokens that you were not short of.

Make “not found” expressible. If the model cannot honestly fill a field, it has three options: leave it null, invent something, or fail. Only the first is useful, and it is available only if you made the field nullable and said so in its description.

Put the evidence next to the value. For extraction, a field holding the span of input the value came from turns an unverifiable claim into a checkable one: your code can confirm that the quoted span really appears in the input, and reject the record when it does not. This is the same move the retrieval lesson makes with citations, applied one level down.

✦Tip

A description on a field constrains nothing. It is not a structural rule, so the grammar built from your schema ignores it, and whether the model ever sees it depends on your client. Write the descriptions anyway, because they are the text you will paste into the system prompt when a field keeps coming back wrong, and keeping them beside the field is how the two stay in agreement.

The script

This is the whole pattern in one file: a schema, a request, a validation, and a bounded repair loop that shows the model the validator’s own complaint. It runs against any OpenAI-compatible endpoint, so llama-server from Part 6 and the gateway from Part 9 are both fine.

RunnableAll tracks

structured-extract.py

#!/usr/bin/env python3
"""Get a validated JSON object out of a local model, with a schema and a repair loop.

Purpose: demonstrate the three layers that make structured output reliable on a local
    server: a JSON Schema sent with the request so the sampler is constrained, Pydantic
    validation of what comes back, and a bounded repair loop that shows the model its own
    error. Two tasks are included, extraction and classification, plus a --free-form mode
    that sends no schema so the difference is visible rather than asserted.
Platform: all (pure Python over HTTP; the server may be on any track or on another machine)
Minimum memory: 8 GB on the machine running the model; this script needs almost none
Assumes: Python 3.9 or later, pydantic 2.x installed in the active environment, and an
    OpenAI-compatible endpoint (llama-server from Part 6, or the gateway from Part 9)
    reachable at --base-url.

Usage: python3 structured-extract.py --base-url http://127.0.0.1:8080/v1 --model qwen3-8b --demo
       python3 structured-extract.py --base-url http://127.0.0.1:8080/v1 --model qwen3-8b \
           --task classify --input message.txt --labbook labbook.md
       python3 structured-extract.py --base-url http://127.0.0.1:8080/v1 --model qwen3-8b \
           --demo --free-form        # no schema: see how often it still parses
"""

from __future__ import annotations

import argparse
import json
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import List, Optional

try:
    from pydantic import BaseModel, Field, ValidationError
except ImportError:  # pragma: no cover - environment check, not logic
    sys.exit("pydantic is not installed. Run: uv pip install 'pydantic>=2'")


# --------------------------------------------------------------------------------------
# The two schemas. These are the contract: the same class produces the JSON Schema that
# constrains the sampler and the validator that checks the reply, so the two cannot drift.
# --------------------------------------------------------------------------------------

class ModelRecord(BaseModel):
    """One open-weight model as described in a paragraph of release notes."""

    name: str = Field(description="The model name exactly as written in the text")
    publisher: Optional[str] = Field(default=None, description="Who released it, or null")
    total_params_b: Optional[float] = Field(
        default=None, description="Total parameters in billions, or null if not stated"
    )
    licence: Optional[str] = Field(default=None, description="Licence name, or null if not stated")
    quantisations: List[str] = Field(
        default_factory=list, description="Quantisation formats named in the text"
    )


class Extraction(BaseModel):
    """The top-level object for the extraction task."""

    models: List[ModelRecord]


class Classification(BaseModel):
    """A single-label classification with a short justification."""

    label: str = Field(description="One of: bug, question, feature-request, documentation, other")
    confidence: float = Field(ge=0.0, le=1.0, description="0 to 1")
    rationale: str = Field(description="One sentence, at most 25 words")


LABELS = ["bug", "question", "feature-request", "documentation", "other"]

TASKS = {
    "extract": {
        "schema_model": Extraction,
        "system": (
            "You extract structured records from release notes. "
            "Report only what the text states. Use null for anything it does not state."
        ),
        "demo_input": (
            "This week's roundup. Alibaba published Qwen3-8B, a dense 8.2B model under "
            "Apache-2.0, with community GGUF builds at Q4_K_M and Q8_0. OpenAI released "
            "gpt-oss-20b, a mixture-of-experts model with 21B total parameters shipped with "
            "native MXFP4 weights, also Apache-2.0. A third release was mentioned on a forum "
            "but the licence was not given."
        ),
    },
    "classify": {
        "schema_model": Classification,
        "system": (
            "You classify an incoming message into exactly one of these labels: "
            + ", ".join(LABELS)
            + ". Choose 'other' when none of the rest fits."
        ),
        "demo_input": (
            "After I upgrade the server my long conversations start returning empty replies "
            "once they pass about eight thousand tokens. Nothing in the log looks wrong."
        ),
    },
}


def post_chat(base_url: str, api_key: Optional[str], payload: dict, timeout: int) -> dict:
    """One POST to /chat/completions. Raises RuntimeError with a readable message."""
    url = base_url.rstrip("/") + "/chat/completions"
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:500]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


def build_payload(args, messages: list, schema_model) -> dict:
    payload = {
        "model": args.model,
        "messages": messages,
        "temperature": args.temperature,
        "max_tokens": args.max_tokens,
    }
    if args.seed is not None:
        payload["seed"] = args.seed
    if not args.free_form:
        payload["response_format"] = {
            "type": "json_schema",
            "json_schema": {
                "name": schema_model.__name__.lower(),
                "schema": schema_model.model_json_schema(),
                "strict": True,
            },
        }
    return payload


def run(args) -> dict:
    task = TASKS[args.task]
    schema_model = task["schema_model"]
    text = task["demo_input"] if args.demo else Path(args.input).read_text(encoding="utf-8")

    messages = [
        {"role": "system", "content": task["system"]},
        {"role": "user", "content": text},
    ]

    started = time.time()
    last_error = None
    for attempt in range(1, args.retries + 2):
        payload = build_payload(args, messages, schema_model)
        reply = post_chat(args.base_url, args.api_key, payload, args.timeout)
        content = reply["choices"][0]["message"]["content"] or ""
        try:
            parsed = schema_model.model_validate_json(content)
        except ValidationError as exc:
            last_error = f"validation failed: {exc.error_count()} error(s)"
            complaint = json.dumps(exc.errors(include_url=False)[:5], default=str)
        except json.JSONDecodeError as exc:
            last_error = f"not JSON at all: {exc}"
            complaint = str(exc)
        else:
            elapsed = time.time() - started
            print(json.dumps(parsed.model_dump(), indent=2))
            print(f"\nvalid on attempt {attempt} of {args.retries + 1}  ({elapsed:.1f} s)")
            return {
                "ok": True,
                "attempts": attempt,
                "seconds": round(elapsed, 2),
                "result": parsed.model_dump(),
            }

        # The repair turn: the model is shown its own output and the validator's complaint.
        print(f"attempt {attempt}: {last_error}", file=sys.stderr)
        messages = messages + [
            {"role": "assistant", "content": content},
            {
                "role": "user",
                "content": (
                    "That reply did not validate against the schema. The validator reported: "
                    f"{complaint}. Return the corrected JSON object and nothing else."
                ),
            },
        ]

    elapsed = time.time() - started
    print(f"gave up after {args.retries + 1} attempt(s): {last_error}", file=sys.stderr)
    return {"ok": False, "attempts": args.retries + 1, "seconds": round(elapsed, 2), "error": last_error}


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1",
                        help="OpenAI-compatible base URL: llama-server, or the Part 9 gateway")
    parser.add_argument("--api-key", default=None, help="sent as a bearer token when set")
    parser.add_argument("--model", required=True, help="model name or alias the server answers to")
    parser.add_argument("--task", choices=sorted(TASKS), default="extract")
    parser.add_argument("--input", default=None, help="file to read; omit with --demo")
    parser.add_argument("--demo", action="store_true", help="use the built-in sample text")
    parser.add_argument("--free-form", action="store_true",
                        help="send no schema, so the reply is unconstrained (teaching mode)")
    parser.add_argument("--retries", type=int, default=2, help="repair attempts after the first")
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--max-tokens", type=int, default=1024)
    parser.add_argument("--seed", type=int, default=None)
    parser.add_argument("--timeout", type=int, default=180)
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    args = parser.parse_args()

    if not args.demo and not args.input:
        parser.error("give --input FILE or --demo")
    if args.input and not Path(args.input).is_file():
        parser.error(f"--input {args.input} does not exist")

    outcome = run(args)

    if args.labbook:
        record = {
            "lab": "part-10/structured-extract",
            "task": args.task,
            "model": args.model,
            "base_url": args.base_url,
            "constrained": not args.free_form,
            "temperature": args.temperature,
            "seed": args.seed,
            "ok": outcome["ok"],
            "attempts": outcome["attempts"],
            "seconds": outcome["seconds"],
            "date": time.strftime("%Y-%m-%d"),
        }
        with Path(args.labbook).open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        print(f"recorded in {args.labbook}")

    sys.exit(0 if outcome["ok"] else 1)


if __name__ == "__main__":
    main()

Download structured-extract.py244 lines

RunnableAll tracks

extraction, with the schema doing the work

python3 structured-extract.py \
  --base-url http://127.0.0.1:8080/v1 \
  --model qwen3-8b \
  --demo \
  --labbook labbook.md

Then run it again with --free-form, which sends no response_format at all, and again with --task classify. Three runs, three lessons: how often an unconstrained model returns something a parser will accept, how a constrained one behaves on the same input, and what a single-label decision looks like when the label set is part of the schema.

▷Exercise

Change one field of ModelRecord to a type the text cannot satisfy, for example make licence a required float, and run the demo again. Watch the repair loop: the model is shown the validator’s complaint and tries again, and after the retry budget the script exits non-zero rather than pretending. An extraction pipeline that fails loudly is worth more than one that returns a partially-filled object with no signal that anything went wrong.

Classification is a special case worth treating specially

Single-label classification does not need a JSON object at all. The cleanest formulation is a constraint that permits only the label strings, which vLLM exposes directly as a choice constraint and which is four lines of GBNF in llama.cpp:

Pseudocode — not a real command

root ::= "bug" | "question" | "feature-request" | "documentation" | "other"

That is one or two decoded tokens per classification instead of an object with a rationale, which on a batch of ten thousand messages is a different program entirely. Add the rationale field back only when a person is going to read it. The demo script includes the rationale because a person is: you, checking whether the labels mean anything.

When not to force a schema

Structured output is a hammer, and some jobs are not nails. Forcing a summary into {"summary": "..."} buys you nothing over returning text and costs you the tokens spent on punctuation. Forcing a long chain of reasoning into nested objects tends to produce worse reasoning, because the model is spending its decoding budget satisfying a shape. The rule that has held up: constrain the output when a program is the consumer, and leave it free when a person is.

A valid object can still represent the wrong answer

Suppose an invoice extractor returns a valid object with a numeric total and a currency string. The total might belong to a different invoice, omit tax or be copied from a subtotal. Schema validation establishes types and allowed structure; domain validation establishes relationships such as subtotal plus tax equalling total, within the application’s rounding rules.

Make unavailable information representable. A nullable field with an explicit missing-evidence status is preferable to forcing a number that the document does not contain. Reject or route conflicts for review according to the business rule. Do not repair a malformed response by silently inserting values that were never present in the source.

Evaluate parsing success, schema validity and semantic correctness separately. Include truncated outputs, refusals, empty evidence and contradictory passages in the test set. Even a constrained sampler can hit an output limit before completing the object. The application must inspect termination and validation results before consuming fields. A bounded retry can fix some formatting failures; repeated retries cannot manufacture absent source evidence.

Recap

Structure is enforced in three places and they compose: the prompt asks, the sampler makes structurally invalid output impossible, and the validator rejects what is structurally valid and semantically wrong. response_format with a JSON schema is the portable way to reach the sampler on both llama-server and vLLM, and GBNF is what llama.cpp compiles that schema into, with documented limits including features that are skipped without a warning. Pydantic gives you the schema and the validator from one class definition, and its error breakdown is the best content for a repair turn. Constrained decoding does not prevent truncation, wrong values, empty results or confident invention, so the validator stays and the retry loop is bounded and loud. For single-label decisions, constrain to the labels rather than to an object.

Check your understanding

Question 1. You send response_format {"type": "json_object"} and get back valid JSON with none of your fields in it. What went wrong?

• The model is too small
• Nothing went wrong: json_object only requires syntactically valid JSON, while json_schema is what constrains the keys and types
• The server does not support JSON output
• The temperature was too high

Show the answer and why

Answer: Nothing went wrong: json_object only requires syntactically valid JSON, while json_schema is what constrains the keys and types

The two response formats do different jobs. json_object is a syntax constraint; json_schema is a shape constraint. Asking for the first and expecting the second is a common and confusing mistake, because everything looks like it worked.

Question 2. A grammar-constrained extraction returns a JSON decode error on long documents only. What is the most likely cause?

• The grammar is invalid
• Generation hit the token budget mid-object, so the output is a valid prefix of an invalid document
• The model does not support grammars
• The schema has too many fields

Show the answer and why

Answer: Generation hit the token budget mid-object, so the output is a valid prefix of an invalid document

The grammar controls which tokens may be chosen, not how many are generated. When max_tokens runs out mid-document the sampler stops in a legal but incomplete state, which is why this shows up as a length-dependent failure.

Question 3. Why does the course keep a Pydantic validator even though the sampler is already constrained by the same schema?

• Because Pydantic is faster
• Because schema-to-grammar conversion documents features that are silently skipped, and because a structurally valid object can still hold wrong values, be truncated, or be empty
• Because the server may ignore response_format
• Because Pydantic can fix errors automatically

Show the answer and why

Answer: Because schema-to-grammar conversion documents features that are silently skipped, and because a structurally valid object can still hold wrong values, be truncated, or be empty

The GBNF guide states that unsupported schema features are silently skipped, so the constraint you believe you applied may be partial. The validator is the layer that turns that from an invisible problem into a caught one.

Question 4. You point a JSON schema constraint at a Qwen3 model in thinking mode and get an empty object. Why?

• Thinking mode disables structured output on the server
• The constraint applies from the first token, so the <think> block the model would open with is not an allowed continuation
• The schema is invalid in thinking mode
• The model ran out of context

Show the answer and why

Answer: The constraint applies from the first token, so the <think> block the model would open with is not an allowed continuation

Constrained decoding starts at token one. Anything the model was trained to emit before its answer, a reasoning block in particular, is forbidden by a grammar that only permits an object. Turn thinking off for the structured call, or send it to a non-thinking model.

Check my answersClear