Level 3 · Model BuilderLessonPart 17 · page 1 of 425 min

25Minutes

2Tools

8Sources

Tools used on this page2

Speculative Decoding Revisited: Acceptance Rates and When It Pays

Part 9 showed you where the speculative-decoding switch is. This lesson answers the question that comes next and that nobody answers for you: should it be on, for your model, on your machine, for the work you actually do. By the end you will be able to derive the speed-up rather than quote it, find the acceptance rate below which speculation costs you time, say why the setting that helps one workload hurts another, and measure all of it with a script that also proves the answers did not change.

Why a checked guess is not an approximation

The claim from Part 9 was that speculative decoding changes the speed and not the output. It is worth seeing why, because the reason is the whole reason to trust the technique.

Take greedy decoding first, where the model always takes its highest-probability token. The draft proposes some tokens; the target model runs once over all of them and reports what it would have chosen at each position. Wherever the proposal matches the target’s own choice, keeping it is by definition the same as having generated it. At the first mismatch the proposal is thrown away, and the target’s choice at that position is taken instead. Nothing about this can produce a token the target would not have produced. It is bookkeeping.

Sampling is the interesting case, because now the target has a distribution rather than a choice, and naively accepting whatever the draft sampled would bias the output towards the draft. Both founding papers solve this the same way. Leviathan, Kalman and Matias describe “a novel sampling method” that makes “exact decoding from the large models faster… without changing the distribution”. Chen and colleagues name the mechanism: “a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics”.

The scheme itself is short. Let q be the draft’s distribution at a position and p the target’s. The draft samples a token x from q. Accept it with probability min(1, p(x) / q(x)): if the target liked the token at least as much as the draft did, keep it unconditionally; if the target liked it less, keep it in proportion. When it is rejected, do not fall back to sampling from p, because the rejected mass has to be accounted for. Sample instead from the normalised positive part of p − q, the residual distribution of everything the draft under-weighted. The two branches together are exactly p.

The acceptance test at one position

The draft samples x from its own distribution qThis is the cheap step, and it is the only place the draft has any influence at all.
The target pass reports p, its distribution at the same positionOne pass over the expensive weights produces p at every proposed position at once. That parallelism is the source of the speed-up.
Draw u uniformly between 0 and 1The randomness that makes the acceptance rule exact rather than approximate.
If u is at most p(x)/q(x), accept x and move onA token the target rates at least as highly as the draft did is always accepted; one it rates lower is accepted in proportion to the ratio.
Otherwise reject, and sample from the normalised positive part of p minus qNot from p. The residual distribution puts back exactly the mass the rejection removed, which is what makes the two branches together identical to sampling from p.
Discard every proposal after the first rejectionThey were conditioned on a token that will not exist, so they carry no information about the sequence that does.

The greedy case is the same picture with the distributions collapsed onto single tokens, which is why greedy speculation can be checked byte for byte and sampled speculation cannot. Chen and colleagues state the preservation holds 'within hardware numerics': floating-point reductions over a different batch shape can still shift a logit in the last place.

The speed-up, derived

Two quantities decide everything. The acceptance rate, written α, is the probability that a proposal survives verification. The draft cost, written c, is what one drafted token costs as a fraction of one target step. The chain length k is how many tokens are drafted per iteration, and it is yours to choose.

Treat acceptance as independent at each position, which is the standard simplification and is wrong in a way we return to below. The chain survives the first position with probability α, the first two with α², and so on, so the expected number of accepted proposals is α + α² + … + α^k. The target pass also produces one token at the first rejected position, and that token is free: it is the token ordinary decoding would have produced. So the expected yield of one iteration is

E(tokens) = 1 + α + α² + … + α^k = (1 − α^(k+1)) / (1 − α)

and its cost, in units of one target step, is 1 + kc. The speed-up over ordinary decoding is the ratio:

S = (1 − α^(k+1)) / ((1 − α)(1 + kc))

Every property people find surprising about speculative decoding falls out of this one expression. Setting S = 1 and solving for α gives the break-even: the acceptance rate below which speculation is a loss. The table below is arithmetic, not measurement, and it is worth reading as a map of where you have to land.

Chain length k	Draft cost c = 0.10	c = 0.20	c = 0.35
2	break even near α = 0.17	α = 0.31	α = 0.48
4	α = 0.29	α = 0.46	α = 0.62
8	α = 0.45	α = 0.62	α = 0.76

The numbers in that table are solutions of the equation above, rounded, and no machine was involved in producing them. Two things follow immediately. A cheap draft has a low bar: at a tenth of a target step and a chain of two, a draft that agrees about one time in six already pays for itself. And an expensive draft has a punishing one: at roughly a third of a target step and a chain of eight, a draft has to be right about three times in four before it breaks even, which is a standard almost nothing meets on open-ended text.

Where the chain length stops helping

The numerator of S saturates. As k grows, (1 − α^(k+1)) / (1 − α) approaches 1 / (1 − α) and stops moving, because the probability of getting that far down the chain is α^k and decays geometrically. The denominator, meanwhile, grows without limit: every drafted token costs c whether it survives or not.

So there is an interior optimum, and it depends on α. At an acceptance rate around a half the best chain is one or two tokens long; at an acceptance rate around nine in ten, chains of six to ten still earn their keep. This is why engines expose the chain length as a knob and why a value copied from someone else’s blog post is a guess. llama.cpp’s server README, as read on 2026-09-09, gives --spec-draft-n-max a default of three, which is a sensible starting point for a chain rather than an answer. SGLang’s advice is to tune its three related options together, and its documentation warns that speculative decoding “may increase GPU memory usage because draft tree, CUDA graphs, and verification buffers consume additional VRAM”.

Acceptance is not one number

The derivation assumed α is a constant. It is not, in three separate ways, and each one matters when you go to measure.

It falls along the chain. The first proposal is conditioned only on real text; the fifth is conditioned on four guesses. Later positions are accepted less often, which is why vLLM publishes vllm:spec_decode_num_accepted_tokens_per_pos rather than a single average. Watching acceptance per position tells you where to cut the chain: if position four is almost never accepted, you are paying c for it every iteration and getting nothing.

It depends on the workload. Code editing where the model quotes back a function, structured extraction, and summarisation that reuses the source’s phrasing all have high acceptance, because the next token is nearly determined. Open-ended writing at a high temperature has low acceptance, because the target’s distribution is genuinely broad and the draft has to guess which broad thing it will pick. The same server, the same draft and the same chain length will land in different rows of the table above depending on what you ask it.

It depends on the context, not only the position. This is the finding behind EAGLE-2. Its authors write that most methods, EAGLE included, “use a static draft tree, implicitly assuming that the acceptance rate of draft tokens depends only on their position”, and that they “found that the acceptance rate of draft tokens is also context-dependent”. Their answer is a context-aware dynamic draft tree that spends its budget where the draft is confident; they report speed-up ratios of 3.05 to 4.26 times and an improvement of twenty to forty per cent over EAGLE. The practical lesson for a reader with one machine is smaller: an average acceptance rate over mixed traffic hides most of what you need to know, so measure per workload.

What the draft actually costs

The parameter ratio is a first estimate of c and usually an optimistic one. A draft with a tenth of the target’s parameters does not cost a tenth of a target step, because a decode step is not pure weight reading. There is a fixed cost per step in kernel launches, sampling and Python-level scheduling that does not shrink with the model, and on a small draft that fixed cost is a larger fraction of the total. On a machine where the target is large enough to be firmly bandwidth-bound and the draft is small enough to be latency-bound, the measured c can be several times the parameter ratio.

There is a memory cost too, and Part 9’s lesson priced it: a draft’s weights come out of the same pool as the key-value cache, so a draft bought with gigabytes is concurrency sold. What this lesson adds is that the draft also needs its own key-value cache for the chain it is generating, and the verification pass holds activations for k + 1 positions rather than one. Neither is large beside the weights. Both are enough to stop a server starting that started yesterday.

Concurrency eats the gain

The single most counter-intuitive property of speculative decoding is that it helps least exactly where a busy server needs help most, and the reason is that it and continuous batching are spending the same resource.

Decode is bandwidth-bound: the arithmetic units idle while weights stream in. Batching fills that idle capacity with other people’s requests. Speculation fills it with the same conversation’s future tokens. At a concurrency of one there is nothing else to spend the capacity on and speculation is nearly free; at a concurrency of twenty the capacity is already sold, and the verification pass over k + 1 positions per sequence is real work competing with real requests.

vLLM’s documentation puts this in a method-selection table with separate columns for low and high request rates: EAGLE is rated “High gain” at low QPS and “Medium to high gain” at high QPS, a draft model “High gain” then “Medium gain”, and the n-gram and suffix methods “Low to medium gain” then “Medium gain”. The page then adds the sentence to keep: “Real gains depend on your model family, traffic pattern, hardware, and sampling settings.” The gain does not vanish at scale, and EAGLE-3’s authors report a throughput improvement of about 1.38 times at a batch size of 64 in SGLang, but the shape of the curve is downwards, and the arithmetic above explains why.

Measuring it on your track

The measurement is a comparison, not a reading. Start two servers on two ports with the same weights at the same quantisation and the same context length, one with a draft and one without, and send both the identical prompt set at temperature zero and concurrency one. The script below does that, and it does the three things the arithmetic says you need.

RunnableAll tracks

#!/usr/bin/env python3
"""Measure what speculative decoding does to one server, and prove it changed nothing else.

Purpose: run one fixed prompt set through an OpenAI-compatible endpoint with speculation
    off and again with speculation on, and report the three things that decide whether
    speculation earns its place: output tokens per second, whatever acceptance statistics
    the server exposes on its Prometheus endpoint, and a byte-for-byte comparison of the
    greedy completions. Speculative decoding is a latency technique, not a quality trade,
    so a run whose text changed is a run to investigate rather than to publish.
Platform: all (spark, strix, mac, nvidia). Pure Python standard library: no pip install.
Minimum memory: 8 GB on the machine running this script; the memory floor is set by the
    server, not by the generator, and the generator may run on another machine on the LAN.
Assumes: one or two servers already listening and answering POST /v1/chat/completions over
    plain HTTP (this is a localhost or LAN tool and does not speak TLS). Both endpoints
    must be serving the same weights at the same quantisation, differing only in whether
    speculation is enabled, or the comparison means nothing. Results are appended to the
    lab notebook as one JSON line per endpoint plus one for the comparison.

Usage:
    # measure a single endpoint, e.g. before you have a draft at all
    python3 measure-speculative.py --baseline-url http://127.0.0.1:8080/v1 \\
        --model local-chat --label llama-server-no-draft --labbook labbook.md

    # compare two endpoints: one without a draft, one with
    python3 measure-speculative.py \\
        --baseline-url http://127.0.0.1:8080/v1 \\
        --speculative-url http://127.0.0.1:8081/v1 \\
        --model local-chat --max-tokens 192 --repeats 2 \\
        --label qwen3-8b-draft-0.6b --labbook labbook.md

    # your own prompts, one per line, instead of the built-in mixed set
    python3 measure-speculative.py --baseline-url http://127.0.0.1:8080/v1 \\
        --model local-chat --prompts my-prompts.txt --prompt-set file

An API key, if a server needs one, is read from the environment variable named by
--api-key-env. No key is written to this file, to the output or to the notebook.

Counting note: token counts come from the "usage" object the server returns, so they are
    the server's own count rather than an estimate. A server that omits "usage" is
    reported with a null token count and its tokens-per-second figure is left out rather
    than guessed.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import statistics
import sys
import time
import urllib.error
import urllib.request
from urllib.parse import urlsplit

# --------------------------------------------------------------------------- prompts

# Three shapes of work, because speculation behaves completely differently on each.
# "copy" repeats material that is already in the prompt, which is where lookup-based
# drafting wins; "closed" has one predictable answer; "open" is high-entropy writing,
# where a draft agrees with the target least often.
BUILTIN_PROMPTS = [
    # copy-heavy
    "Here is a function:\n\ndef total(rows):\n    out = 0\n    for r in rows:\n        out += r['amount']\n    return out\n\nRewrite it to skip rows whose 'amount' key is missing, and show the whole function.",
    "Reformat this list as a JSON array of objects with keys name and port: "
    "llama-server 8080, vllm 8000, sglang 30000, litellm 4000.",
    "Copy this sentence exactly, then explain it in one sentence: "
    "decode is bandwidth-bound because every active weight is read once per token.",
    "Take this shell line and add error handling, showing the complete result: "
    "curl -s http://127.0.0.1:8080/v1/models | jq .data",
    # closed
    "In two sentences, what does a KV cache hold and why does it grow with the conversation?",
    "Name the two halves of a generation step and say which one is compute-bound.",
    "What is an acceptance rate in speculative decoding? Answer in one sentence.",
    "List four fields that must accompany a tokens-per-second measurement.",
    # open
    "Write a short paragraph, in your own words, about why a smaller model that fits in memory can beat a larger one that does not.",
    "Describe, as if to a colleague over coffee, what surprised you most about running language models on your own hardware.",
    "Invent a plausible name and one-line description for a tool that records benchmark context automatically.",
    "Write three sentences of encouragement for somebody whose first fine-tune made their model worse.",
]


def load_prompts(prompt_set: str, path: str | None) -> list[str]:
    """The built-in mixed set, or one prompt per non-blank line of a file."""
    if prompt_set == "builtin":
        return list(BUILTIN_PROMPTS)
    if not path:
        raise SystemExit("--prompt-set file needs --prompts pointing at a file")
    with open(path, encoding="utf-8") as handle:
        prompts = [line.strip() for line in handle if line.strip()]
    if not prompts:
        raise SystemExit(f"{path} contained no prompts")
    return prompts


# ------------------------------------------------------------------------ HTTP


class ServerError(Exception):
    """The server did not answer in a way this script can use."""


def post_json(url: str, payload: dict, api_key: str, timeout: float) -> dict:
    """One non-streaming POST. Returns the decoded JSON body."""
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(url, data=body, method="POST")
    request.add_header("Content-Type", "application/json")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http(s) only, checked in main
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:200]
        raise ServerError(f"HTTP {exc.code} from {url}: {detail}") from exc
    except (urllib.error.URLError, OSError, json.JSONDecodeError) as exc:
        raise ServerError(f"{type(exc).__name__} talking to {url}: {exc}") from exc


def get_text(url: str, api_key: str, timeout: float) -> str | None:
    """GET a text body, or None if the endpoint is absent or refuses."""
    request = urllib.request.Request(url, method="GET")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http(s) only
            return response.read().decode("utf-8", "replace")
    except (urllib.error.URLError, OSError):
        return None


# ------------------------------------------------------------- Prometheus scraping

# Deliberately generic. Engines rename their metrics between releases, so this script
# reports whatever the server publishes whose name mentions speculation or the draft,
# rather than asserting that a particular metric name exists. An empty result means
# "this server exposed nothing under those names", which the page tells you to record.
SPEC_METRIC = re.compile(r"(spec_decode|speculat|draft|accept)", re.IGNORECASE)
SAMPLE = re.compile(r"^(?P<name>[A-Za-z_:][A-Za-z0-9_:]*)(?P<labels>\{[^}]*\})?\s+(?P<value>[-+0-9.eE]+|NaN)$")


def scrape_metrics(metrics_url: str, api_key: str, timeout: float, keep: re.Pattern[str]) -> dict[str, float]:
    """Reads a Prometheus exposition page and keeps the samples whose name matches."""
    text = get_text(metrics_url, api_key, timeout)
    if text is None:
        return {}
    out: dict[str, float] = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        match = SAMPLE.match(line)
        if not match or not keep.search(match.group("name")):
            continue
        try:
            value = float(match.group("value"))
        except ValueError:
            continue
        key = match.group("name") + (match.group("labels") or "")
        out[key] = value
    return out


def metrics_delta(before: dict[str, float], after: dict[str, float]) -> dict[str, float]:
    """What the run itself added, so a warm server's history does not pollute the record."""
    keys = set(before) | set(after)
    return {k: round(after.get(k, 0.0) - before.get(k, 0.0), 4) for k in sorted(keys)}


# ------------------------------------------------------------------------ measurement


def run_endpoint(cfg: dict, prompts: list[str], repeats: int) -> dict:
    """Sends every prompt `repeats` times, sequentially, and times each request.

    Sequential and single-stream on purpose: speculation spends idle arithmetic
    capacity, so concurrency one is where it has the most to gain, and it is the
    condition under which the acceptance arithmetic on the lesson page applies.
    """
    completions: list[str] = []
    latencies: list[float] = []
    token_counts: list[int | None] = []
    timings: list[dict] = []
    errors: list[str] = []

    metrics_before = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"], SPEC_METRIC)
    started = time.perf_counter()

    for _ in range(repeats):
        for prompt in prompts:
            payload = {
                "model": cfg["model"],
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": cfg["max_tokens"],
                "temperature": 0.0,
                "top_p": 1.0,
                "seed": cfg["seed"],
                "stream": False,
            }
            t0 = time.perf_counter()
            try:
                body = post_json(cfg["chat_url"], payload, cfg["api_key"], cfg["timeout"])
            except ServerError as exc:
                errors.append(str(exc))
                continue
            latencies.append(time.perf_counter() - t0)
            choices = body.get("choices") or [{}]
            message = choices[0].get("message") or {}
            completions.append(message.get("content") or "")
            usage = body.get("usage") or {}
            completion_tokens = usage.get("completion_tokens")
            token_counts.append(int(completion_tokens) if isinstance(completion_tokens, int) else None)
            if isinstance(body.get("timings"), dict):
                timings.append(body["timings"])

    wall = time.perf_counter() - started
    metrics_after = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"], SPEC_METRIC)

    counted = [t for t in token_counts if t is not None]
    total_tokens = sum(counted) if counted else None
    complete = len(counted) == len(latencies) and bool(counted)

    return {
        "label": cfg["label"],
        "base_url": cfg["base_url"],
        "model": cfg["model"],
        "requests": len(prompts) * repeats,
        "completed": len(latencies),
        "failed": len(errors),
        "first_error": errors[0] if errors else None,
        "wall_s": round(wall, 3),
        "latency_s": {
            "p50": round(statistics.median(latencies), 4) if latencies else None,
            "mean": round(statistics.fmean(latencies), 4) if latencies else None,
            "max": round(max(latencies), 4) if latencies else None,
        },
        "output_tokens": total_tokens,
        "output_tokens_per_s": (
            round(total_tokens / wall, 2) if complete and total_tokens and wall > 0 else None
        ),
        "tokens_counted_by_server": complete,
        "server_timings_sample": timings[0] if timings else None,
        "spec_metrics_delta": metrics_delta(metrics_before, metrics_after),
        "completions": completions,
    }


def compare_text(baseline: list[str], speculative: list[str]) -> dict:
    """Byte-for-byte comparison of the two sets of greedy completions.

    Verification means the accepted tokens are the ones the target model would have
    produced, so at temperature zero the two runs should agree exactly. They can still
    differ for reasons that have nothing to do with the draft - a different batch
    composition changes the order of floating-point reductions, and the engines say so -
    which is why this reports where the first divergence is rather than only that there
    was one.
    """
    pairs = list(zip(baseline, speculative))
    identical = [i for i, (a, b) in enumerate(pairs) if a == b]
    divergent = []
    for i, (a, b) in enumerate(pairs):
        if a == b:
            continue
        cut = 0
        for cut, (ca, cb) in enumerate(zip(a, b)):
            if ca != cb:
                break
        else:
            cut = min(len(a), len(b))
        divergent.append(
            {
                "prompt_index": i,
                "first_difference_at_char": cut,
                "baseline_tail": a[cut : cut + 60],
                "speculative_tail": b[cut : cut + 60],
            }
        )
    return {
        "compared": len(pairs),
        "identical": len(identical),
        "divergent": len(divergent),
        "divergences": divergent[:5],
    }


# ---------------------------------------------------------------------------- output


def print_endpoint(row: dict) -> None:
    """One endpoint's result, as an aligned block."""
    print(f"  {row['label']}  ({row['base_url']})")
    print(f"    requests    {row['completed']}/{row['requests']} completed, {row['failed']} failed")
    print(f"    wall clock  {row['wall_s']:.2f} s")
    if row["output_tokens_per_s"] is not None:
        print(f"    output      {row['output_tokens']} tokens, {row['output_tokens_per_s']:.2f} tokens/s")
    else:
        print("    output      the server did not return a usage object; tokens/s not computed")
    if row["latency_s"]["p50"] is not None:
        print(f"    latency     p50 {row['latency_s']['p50']:.3f} s, max {row['latency_s']['max']:.3f} s")
    if row["spec_metrics_delta"]:
        print("    acceptance statistics from the server's /metrics during this run:")
        for name, value in row["spec_metrics_delta"].items():
            print(f"      {name} += {value}")
    else:
        print("    acceptance  the server exposed no speculation metrics; record that as the finding")
    if row["first_error"]:
        print(f"    first error {row['first_error']}")


def append_labbook(path: str, record: dict) -> None:
    """Appends one JSON line, in the course notebook format."""
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def endpoint_config(base_url: str, args: argparse.Namespace, label: str) -> dict:
    """Derives the chat and metrics URLs from an OpenAI-compatible base URL."""
    parts = urlsplit(base_url)
    if parts.scheme != "http":
        raise SystemExit("This tool speaks plain HTTP only; point it at a localhost or LAN endpoint.")
    if not parts.hostname:
        raise SystemExit(f"Could not read a host from {base_url!r}.")
    root = f"{parts.scheme}://{parts.netloc}"
    path = parts.path.rstrip("/") or "/v1"
    return {
        "base_url": base_url,
        "chat_url": f"{root}{path}/chat/completions",
        "metrics_url": f"{root}/metrics",
        "model": args.model,
        "max_tokens": args.max_tokens,
        "seed": args.seed,
        "timeout": args.timeout,
        "api_key": os.environ.get(args.api_key_env, ""),
        "label": label,
    }


def parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--baseline-url", required=True,
                        help="OpenAI-compatible base URL of the server WITHOUT speculation, ending in /v1")
    parser.add_argument("--speculative-url", default=None,
                        help="base URL of the server WITH speculation; omit to measure one endpoint only")
    parser.add_argument("--model", required=True, help="model name the server reports at /v1/models")
    parser.add_argument("--max-tokens", type=int, default=192)
    parser.add_argument("--seed", type=int, default=0,
                        help="sent as the request seed; ignored by servers that do not accept one")
    parser.add_argument("--repeats", type=int, default=1, help="times to send the whole prompt set")
    parser.add_argument("--timeout", type=float, default=300.0)
    parser.add_argument("--prompt-set", default="builtin", choices=["builtin", "file"])
    parser.add_argument("--prompts", default=None, help="file of prompts, one per line, for --prompt-set file")
    parser.add_argument("--label", default="speculative-run", help="tag written into the notebook lines")
    parser.add_argument("--engine", default="unknown", help="engine name recorded in the notebook, e.g. llama.cpp")
    parser.add_argument("--engine-version", default="unknown", help="engine version recorded in the notebook")
    parser.add_argument("--api-key-env", default="SPEC_API_KEY",
                        help="environment variable holding the API key, if the servers need one")
    parser.add_argument("--labbook", default="labbook.md")
    return parser.parse_args(argv)


def main(argv: list[str]) -> int:
    args = parse_args(argv)
    prompts = load_prompts(args.prompt_set, args.prompts)

    print(f"==> {args.label}: {args.model}, {len(prompts)} prompt(s) x {args.repeats} repeat(s), "
          f"max_tokens {args.max_tokens}, temperature 0")

    baseline = run_endpoint(endpoint_config(args.baseline_url, args, f"{args.label}/baseline"), prompts, args.repeats)
    print_endpoint(baseline)

    speculative = None
    comparison = None
    if args.speculative_url:
        speculative = run_endpoint(
            endpoint_config(args.speculative_url, args, f"{args.label}/speculative"), prompts, args.repeats
        )
        print_endpoint(speculative)
        comparison = compare_text(baseline["completions"], speculative["completions"])
        print("  comparison")
        print(f"    text        {comparison['identical']}/{comparison['compared']} completions identical, "
              f"{comparison['divergent']} divergent")
        for d in comparison["divergences"]:
            print(f"      prompt {d['prompt_index']}: first difference at character {d['first_difference_at_char']}")
        if baseline["output_tokens_per_s"] and speculative["output_tokens_per_s"]:
            ratio = speculative["output_tokens_per_s"] / baseline["output_tokens_per_s"]
            print(f"    speed       {ratio:.2f}x the baseline token rate on this prompt set")

    stamp = time.strftime("%Y-%m-%dT%H:%M:%S%z")
    for row in (baseline, speculative):
        if row is None:
            continue
        record = dict(row)
        # The completions are the evidence for the comparison, not part of the record:
        # a notebook line has to stay readable.
        record.pop("completions", None)
        record.update({
            "lab": "part-17/measure-speculative",
            "engine": args.engine,
            "engine_version": args.engine_version,
            "max_tokens": args.max_tokens,
            "repeats": args.repeats,
            "prompt_count": len(prompts),
            "recorded_at": stamp,
        })
        append_labbook(args.labbook, record)

    if comparison is not None:
        append_labbook(args.labbook, {
            "lab": "part-17/measure-speculative",
            "label": f"{args.label}/comparison",
            "engine": args.engine,
            "engine_version": args.engine_version,
            "model": args.model,
            "baseline_tokens_per_s": baseline["output_tokens_per_s"],
            "speculative_tokens_per_s": speculative["output_tokens_per_s"] if speculative else None,
            "text_comparison": comparison,
            "recorded_at": stamp,
        })

    print(f"    appended {2 if comparison else 1} line(s) to {args.labbook}")

    if baseline["completed"] == 0 or (speculative is not None and speculative["completed"] == 0):
        print("    at least one endpoint completed no requests; see the error above.", file=sys.stderr)
        return 1
    if comparison is not None and comparison["divergent"]:
        print("    completions differed between the two servers; the page explains what to check.",
              file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download measure-speculative.py434 lines

It reports the token rate from each server’s own usage object rather than estimating it. It scrapes each server’s Prometheus endpoint before and after the run and reports what the run added to any counter whose name mentions speculation, drafting or acceptance, which is how you get an acceptance rate without trusting a log line. And it compares the completions byte for byte and tells you where the first difference is, so a run that changed the answers announces itself.

RunnableAll tracks

python3 measure-speculative.py \
  --baseline-url http://127.0.0.1:8080/v1 \
  --model local-chat \
  --engine llama.cpp \
  --label baseline-no-draft \
  --labbook labbook.md

The prompt set is deliberately three kinds of work: copy-heavy prompts that quote the input back, closed questions with one predictable answer, and open-ended writing. Run the comparison once per kind, with --prompt-set file and your own prompts, and you will have the only table that answers the question this lesson opened with.

Pending validationSpeculative decoding on your own track: the table this lesson asks you to fill in

Workload	Baseline tokens/s	With draft tokens/s	Ratio	Acceptance	Completions identical
Copy-heavy (quote a function back)	pending	pending	pending	pending	expected: yes
Closed questions	pending	pending	pending	pending	expected: yes
Open-ended writing	pending	pending	pending	pending	expected: yes
Concurrency 10, mixed	pending	pending	pending	pending	expected: yes

your track, as installed · llama.cpp or vLLM the build you measured · Qwen3-8B or your fine-tune, the same on both servers · 4,096 tokens of context · the day you ran it

Filled in by the reader from measure-speculative.py. The last two columns are the ones that make the run reproducible: a ratio without an acceptance rate cannot be compared with anybody else's, and a run whose completions differ is a bug report rather than a result.

Use a measured break-even calculation

Suppose a target-only step takes time T. A speculative iteration spends D drafting and V verifying, then keeps an average of K tokens including any correction or bonus token defined by the algorithm. The measured speed ratio is approximately K × T / (D + V) for that portion of generation. It exceeds one only when the saved target work outweighs both costs.

Use measured iteration totals where possible instead of substituting a parameter-count ratio for draft time. Verification cost changes with draft length, kernels and batch shape; acceptance changes with prompt domain and position in the chain. A single average acceptance rate can obscure several different operating regimes.

Repeat the comparison under the concurrency your application will admit. Include memory used by the draft and any reduction in available cache capacity. If speculative mode wins for one interactive request but loses under load, both observations can be correct. Choose the configuration that serves the workload, and record the crossover instead of promoting a universal speedup claim.

Recap

Speculative decoding is exact because the target model decides every token: greedily by agreement, and under sampling by an acceptance rule that keeps a proposal with probability min(1, p/q) and, on rejection, samples the normalised residual of p − q. The speed-up is (1 − α^(k+1)) / ((1 − α)(1 + kc)), and it explains everything: the break-even acceptance rate rises steeply with the draft’s cost, the chain length has an interior optimum because the yield saturates while the cost does not, and a cheaper draft beats a cleverer one more often than not. Acceptance is not a single number; it falls along the chain, changes with the workload, and, as EAGLE-2’s authors showed, depends on the context rather than only the position. And the gain shrinks as concurrency rises, because batching and speculation are both spending the accelerator’s idle arithmetic. All of which is measurable in about twenty minutes with two servers, one prompt set and a byte-for-byte comparison.

Sources for this lesson

8 verified · checked 2026-09-09

01Fast Inference from Transformers via Speculative Decoding§ Abstract; the speculative sampling methodarxiv.org/abs/2211.171922026-09-09
02Accelerating Large Language Model Decoding with Speculative Sampling§ Abstract; modified rejection samplingarxiv.org/abs/2302.013182026-09-09
03vLLM — Speculative Decoding§ Configuration keys; method selection; limitationsdocs.vllm.ai/en/latest/features/speculative_decoding2026-09-09
04vLLM — Production metrics§ Speculative decoding metricsdocs.vllm.ai/en/latest/usage/metrics.html2026-09-09
05SGLang — Speculative Decoding§ EAGLE and EAGLE3 launch commands; memory notesdocs.sglang.io/advanced_features/speculative_decoding.html2026-09-09
06EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees§ Abstract; context-dependent acceptancearxiv.org/abs/2406.168582026-09-09
07EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test§ Abstract; throughput at batch size 64arxiv.org/abs/2503.018402026-09-09
08llama.cpp — llama-server README§ Speculative decoding options; metrics endpointgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.