Lab: Train and Deploy a Draft for Your Model

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The versions, commits and wall-clock each track was run with will be recorded here when the validation pass is done.

Objective

By the end of this lab you will have two servers running the same model side by side, one with speculative decoding and one without, and a table in your lab notebook that says how much faster the speculative one was on three kinds of work, what its acceptance rate was, and whether its completions were identical to the baseline’s. That last column is the point. Every other speed technique in this course trades something; this one is supposed to trade nothing, and you will have checked.

Tracks S and N additionally train a draft head for their own model, which means collecting the model’s own generations, running a training script from one of the published repositories, and finding out what a draft costs in hours before it costs anything in latency. Tracks X and M take the no-training path, which is a small draft model of the same family, and reach the same measurement table by a shorter road. Both are real results; only one of them takes an afternoon.

⌂Single-machine path

Single-machine on every track. Two servers on two ports on the same box is the whole architecture, and the lab is written so that both fit inside the memory floor. If you own two machines you may put the baseline on one and the speculative server on the other, but then say so in the notebook, because they are no longer sharing a thermal envelope.

Requirements

Everyone needs the lab notebook from Part 1, a served model from Part 9 or Part 6, and the hf command-line tool from Part 4. The target model is your own fine-tune from Part 13 if you have one, and Qwen3-8B otherwise; both are Apache-2.0 and neither is gated, and the model reference records the licence for each. The draft model on the no-training path is Qwen3-0.6B, also Apache-2.0, chosen because it shares the Qwen3 tokeniser with the target, which is not optional.

Time. About seventy-five minutes of attended work. The unattended time is where the tracks diverge: the no-training path adds nothing, while collecting a few hundred generations is a background job of tens of minutes and training a head is measured in hours. Start the download and the data collection before you read the rest of the page.

Memory. The floor is 24 GB because two servers hold a copy of the target’s weights at the same time, plus a draft and two key-value caches. If you have less, serve the two configurations one after the other rather than side by side and say so in the notebook, because a sequential comparison is weaker evidence than a simultaneous one.

Track S — NVIDIA DGX Spark

The primary path, and the only track where every step of this lab has a documented route. 128 GB of unified memory holds the target at bfloat16, the draft, both servers and the training run without arithmetic. Use vLLM for the serving half so that --speculative-config and the Prometheus acceptance counters are available, and the CUDA stack from Part 11 for the training half.

Start the data collection first, in another terminal, and read the rest of the page while it runs. Budget disk for the hidden-state files if you take the EAGLE route: they are the largest thing this lab writes.

Track X — AMD Ryzen AI Max+ 395Partial

Neither Medusa's nor SpecForge's documentation names a ROCm training path as read on 2026-09-09, so the training tasks are not claimed on this track. Every other task runs.

Do tasks 1, 2, 3, 6, 7, 8 and 9 with the no-training path: Qwen3-0.6B at Q4_K_M drafting for Qwen3-8B at Q4_K_M, both GGUF, both served by llama-server, which has a Vulkan and a HIP build from Part 6. That gives you the full measurement table and the byte-for-byte check without a training run.

You can still do task 3, the data collection, because it only talks to a server, and it is worth doing: the file is portable, and it is what you would hand to a machine that can train. Attempting the training tasks on ROCm is a legitimate experiment, but record it as an experiment rather than as a lab result, and note the ROCm version you tried.

Track M — Apple siliconPartial

No training repository in this part documents an Apple silicon path as read on 2026-09-09. Serving with a draft model is fully supported on Metal.

Take the no-training path with llama-server’s Metal build: Qwen3-0.6B drafting for Qwen3-8B, both GGUF. A 24 GB Mac holds both models at Q4_K_M and two contexts; a 16 GB Mac should drop the target to Qwen3-4B and record the substitution.

mlx-lm offers a second route worth measuring alongside it. Its generate command takes --draft-model and --num-draft-tokens, confirmed in its argument parser on 2026-09-09, so you can compare llama.cpp’s draft speculation against MLX’s on the same machine. That comparison is not published anywhere reliable and it is a genuinely useful thing to put in the notebook.

Track N — NVIDIA desktop or laptop

The other primary path, and the one the memory floor was written for. At 12 to 16 GB of VRAM, train a Medusa-style head against an 8B backbone loaded at four bits, as the previous lesson budgets, and serve the measurement half with llama-server and a small draft GGUF so that the target and draft fit alongside each other.

At 24 GB or more you have the choice of the bfloat16 backbone for training and vLLM for serving, which is the Track S procedure at a smaller scale. Whichever you pick, write down which, because a head trained against four-bit features and served against bfloat16 ones is a different experiment.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-17-faster-inference"
cd "$LAB_DIR"
pwd
test -f "serve-with-draft.sh"

Expected result: pwd ends in part-17-faster-inference and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Record what you are measuring with, and start the baseline

A speculative result is not reproducible without the engine build and the acceptance rate. Capture the first one now.

RunnableAll tracks

the build that produced these numbers

llama-server --version

Put the version and the date in the notebook’s Environment section. Then start the baseline server: the same weights, the same context length, no draft.

RunnableAll tracks

serve-with-draft.sh

#!/usr/bin/env bash
# Purpose: start one server with speculative decoding enabled, on llama.cpp or on vLLM,
#          using the same target weights as the baseline server so that the pair can be
#          measured against each other by measure-speculative.py
# Platform: llamacpp: spark, strix, mac, nvidia. vllm: spark and nvidia (vLLM's GPU
#           installation page lists ROCm wheels for the Ryzen AI Max+ that the course has
#           not exercised, and no mainline macOS GPU path).
# Minimum memory: the target weights plus the draft plus a KV cache. For an 8B target at
#           Q4_K_M with a 0.6B draft at Q4_K_M that is roughly 6 GB of weights before the
#           cache, which is why this lab's floor is 24 GB rather than 8.
# Assumes: llama-server on PATH (built in Part 6) for ENGINE=llamacpp, or vllm on PATH
#          (Part 9) for ENGINE=vllm; the target and draft model paths exist. Nothing here
#          writes to a model directory or deletes anything.
#
# Usage: bash serve-with-draft.sh ENGINE TARGET DRAFT [PORT]
#   ENGINE   llamacpp | vllm
#   TARGET   GGUF file (llamacpp) or Hugging Face id / directory (vllm)
#   DRAFT    draft GGUF file (llamacpp) or draft model id / directory (vllm);
#            pass the literal word none to start the baseline server with no draft
#   PORT     default 8081 with a draft, 8080 without
#
# Environment: HOST (default 127.0.0.1), CTX (default 4096), NGL (default 999),
#              NGLD (draft layers on the accelerator, default 999), DRAFT_MAX (default 4),
#              DRAFT_MIN (default 0), DRAFT_P_MIN (default 0.75), ALIAS (default local-chat),
#              GPU_UTIL for vLLM (default 0.85).
set -euo pipefail

ENGINE="${1:-}"
TARGET="${2:-}"
DRAFT="${3:-}"

HOST="${HOST:-127.0.0.1}"
CTX="${CTX:-4096}"
NGL="${NGL:-999}"
NGLD="${NGLD:-999}"
DRAFT_MAX="${DRAFT_MAX:-4}"
DRAFT_MIN="${DRAFT_MIN:-0}"
DRAFT_P_MIN="${DRAFT_P_MIN:-0.75}"
ALIAS="${ALIAS:-local-chat}"
GPU_UTIL="${GPU_UTIL:-0.85}"

die() { echo "serve-with-draft: $*" >&2; exit 1; }

[[ -n "$ENGINE" && -n "$TARGET" && -n "$DRAFT" ]] ||
  die "usage: bash serve-with-draft.sh ENGINE TARGET DRAFT [PORT]"
[[ "$ENGINE" == "llamacpp" || "$ENGINE" == "vllm" ]] ||
  die "ENGINE must be llamacpp or vllm (got '$ENGINE')"

if [[ "$DRAFT" == "none" ]]; then
  PORT="${4:-8080}"
  ROLE="baseline, no draft"
else
  PORT="${4:-8081}"
  ROLE="speculative, drafting up to $DRAFT_MAX token(s) per step"
fi

echo "==> $ENGINE on $HOST:$PORT - $ROLE"
echo "    target $TARGET"
[[ "$DRAFT" == "none" ]] || echo "    draft  $DRAFT"
echo "    Serve the baseline and the speculative server at the same time, on two ports,"
echo "    so measure-speculative.py can compare them without a reload in between."
echo

if [[ "$ENGINE" == "llamacpp" ]]; then
  command -v llama-server >/dev/null || die "llama-server is not on PATH; build it as in Part 6"
  [[ -f "$TARGET" ]] || die "$TARGET is not a file; llama.cpp wants a GGUF path"

  if [[ "$DRAFT" == "none" ]]; then
    llama-server \
      --model "$TARGET" \
      --alias "$ALIAS" \
      --host "$HOST" \
      --port "$PORT" \
      --ctx-size "$CTX" \
      --n-gpu-layers "$NGL" \
      --metrics \
      --slots
  else
    [[ -f "$DRAFT" ]] || die "$DRAFT is not a file; llama.cpp wants a GGUF path for the draft too"
    # The draft must share the target's tokeniser. A draft from another family will either
    # be refused at start-up or accepted and never agree with the target; both are wasted time.
    llama-server \
      --model "$TARGET" \
      --alias "$ALIAS" \
      --host "$HOST" \
      --port "$PORT" \
      --ctx-size "$CTX" \
      --n-gpu-layers "$NGL" \
      --metrics \
      --slots \
      --spec-draft-model "$DRAFT" \
      --spec-draft-n-max "$DRAFT_MAX" \
      --spec-draft-n-min "$DRAFT_MIN" \
      --spec-draft-p-min "$DRAFT_P_MIN" \
      -ngld "$NGLD"
  fi
else
  command -v vllm >/dev/null || die "vllm is not on PATH; install it as in Part 9"

  if [[ "$DRAFT" == "none" ]]; then
    vllm serve "$TARGET" \
      --served-model-name "$ALIAS" \
      --host "$HOST" \
      --port "$PORT" \
      --max-model-len "$CTX" \
      --gpu-memory-utilization "$GPU_UTIL" \
      --seed 0
  else
    # One JSON object carries the whole speculative configuration. "draft_model" is the
    # method for a separate small model; the lab page gives the ngram and eagle3 shapes.
    SPEC_CONFIG="{\"method\": \"draft_model\", \"model\": \"$DRAFT\", \"num_speculative_tokens\": $DRAFT_MAX}"
    echo "    speculative-config: $SPEC_CONFIG"
    vllm serve "$TARGET" \
      --served-model-name "$ALIAS" \
      --host "$HOST" \
      --port "$PORT" \
      --max-model-len "$CTX" \
      --gpu-memory-utilization "$GPU_UTIL" \
      --seed 0 \
      --speculative-config "$SPEC_CONFIG"
  fi
fi

Download serve-with-draft.sh122 lines

RunnableAll tracks

baseline server on port 8080, no draft

bash serve-with-draft.sh llamacpp \
  ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
  none 8080

The literal word none in the draft position is what selects the baseline. The script starts llama-server with --metrics and --slots so that the measurement scripts have something to scrape.

2. Measure the baseline

RunnableAll tracks

measure-speculative.py

#!/usr/bin/env python3
"""Measure what speculative decoding does to one server, and prove it changed nothing else.

Purpose: run one fixed prompt set through an OpenAI-compatible endpoint with speculation
    off and again with speculation on, and report the three things that decide whether
    speculation earns its place: output tokens per second, whatever acceptance statistics
    the server exposes on its Prometheus endpoint, and a byte-for-byte comparison of the
    greedy completions. Speculative decoding is a latency technique, not a quality trade,
    so a run whose text changed is a run to investigate rather than to publish.
Platform: all (spark, strix, mac, nvidia). Pure Python standard library: no pip install.
Minimum memory: 8 GB on the machine running this script; the memory floor is set by the
    server, not by the generator, and the generator may run on another machine on the LAN.
Assumes: one or two servers already listening and answering POST /v1/chat/completions over
    plain HTTP (this is a localhost or LAN tool and does not speak TLS). Both endpoints
    must be serving the same weights at the same quantisation, differing only in whether
    speculation is enabled, or the comparison means nothing. Results are appended to the
    lab notebook as one JSON line per endpoint plus one for the comparison.

Usage:
    # measure a single endpoint, e.g. before you have a draft at all
    python3 measure-speculative.py --baseline-url http://127.0.0.1:8080/v1 \\
        --model local-chat --label llama-server-no-draft --labbook labbook.md

    # compare two endpoints: one without a draft, one with
    python3 measure-speculative.py \\
        --baseline-url http://127.0.0.1:8080/v1 \\
        --speculative-url http://127.0.0.1:8081/v1 \\
        --model local-chat --max-tokens 192 --repeats 2 \\
        --label qwen3-8b-draft-0.6b --labbook labbook.md

    # your own prompts, one per line, instead of the built-in mixed set
    python3 measure-speculative.py --baseline-url http://127.0.0.1:8080/v1 \\
        --model local-chat --prompts my-prompts.txt --prompt-set file

An API key, if a server needs one, is read from the environment variable named by
--api-key-env. No key is written to this file, to the output or to the notebook.

Counting note: token counts come from the "usage" object the server returns, so they are
    the server's own count rather than an estimate. A server that omits "usage" is
    reported with a null token count and its tokens-per-second figure is left out rather
    than guessed.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import statistics
import sys
import time
import urllib.error
import urllib.request
from urllib.parse import urlsplit

# --------------------------------------------------------------------------- prompts

# Three shapes of work, because speculation behaves completely differently on each.
# "copy" repeats material that is already in the prompt, which is where lookup-based
# drafting wins; "closed" has one predictable answer; "open" is high-entropy writing,
# where a draft agrees with the target least often.
BUILTIN_PROMPTS = [
    # copy-heavy
    "Here is a function:\n\ndef total(rows):\n    out = 0\n    for r in rows:\n        out += r['amount']\n    return out\n\nRewrite it to skip rows whose 'amount' key is missing, and show the whole function.",
    "Reformat this list as a JSON array of objects with keys name and port: "
    "llama-server 8080, vllm 8000, sglang 30000, litellm 4000.",
    "Copy this sentence exactly, then explain it in one sentence: "
    "decode is bandwidth-bound because every active weight is read once per token.",
    "Take this shell line and add error handling, showing the complete result: "
    "curl -s http://127.0.0.1:8080/v1/models | jq .data",
    # closed
    "In two sentences, what does a KV cache hold and why does it grow with the conversation?",
    "Name the two halves of a generation step and say which one is compute-bound.",
    "What is an acceptance rate in speculative decoding? Answer in one sentence.",
    "List four fields that must accompany a tokens-per-second measurement.",
    # open
    "Write a short paragraph, in your own words, about why a smaller model that fits in memory can beat a larger one that does not.",
    "Describe, as if to a colleague over coffee, what surprised you most about running language models on your own hardware.",
    "Invent a plausible name and one-line description for a tool that records benchmark context automatically.",
    "Write three sentences of encouragement for somebody whose first fine-tune made their model worse.",
]


def load_prompts(prompt_set: str, path: str | None) -> list[str]:
    """The built-in mixed set, or one prompt per non-blank line of a file."""
    if prompt_set == "builtin":
        return list(BUILTIN_PROMPTS)
    if not path:
        raise SystemExit("--prompt-set file needs --prompts pointing at a file")
    with open(path, encoding="utf-8") as handle:
        prompts = [line.strip() for line in handle if line.strip()]
    if not prompts:
        raise SystemExit(f"{path} contained no prompts")
    return prompts


# ------------------------------------------------------------------------ HTTP


class ServerError(Exception):
    """The server did not answer in a way this script can use."""


def post_json(url: str, payload: dict, api_key: str, timeout: float) -> dict:
    """One non-streaming POST. Returns the decoded JSON body."""
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(url, data=body, method="POST")
    request.add_header("Content-Type", "application/json")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http(s) only, checked in main
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:200]
        raise ServerError(f"HTTP {exc.code} from {url}: {detail}") from exc
    except (urllib.error.URLError, OSError, json.JSONDecodeError) as exc:
        raise ServerError(f"{type(exc).__name__} talking to {url}: {exc}") from exc


def get_text(url: str, api_key: str, timeout: float) -> str | None:
    """GET a text body, or None if the endpoint is absent or refuses."""
    request = urllib.request.Request(url, method="GET")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http(s) only
            return response.read().decode("utf-8", "replace")
    except (urllib.error.URLError, OSError):
        return None


# ------------------------------------------------------------- Prometheus scraping

# Deliberately generic. Engines rename their metrics between releases, so this script
# reports whatever the server publishes whose name mentions speculation or the draft,
# rather than asserting that a particular metric name exists. An empty result means
# "this server exposed nothing under those names", which the page tells you to record.
SPEC_METRIC = re.compile(r"(spec_decode|speculat|draft|accept)", re.IGNORECASE)
SAMPLE = re.compile(r"^(?P<name>[A-Za-z_:][A-Za-z0-9_:]*)(?P<labels>\{[^}]*\})?\s+(?P<value>[-+0-9.eE]+|NaN)$")


def scrape_metrics(metrics_url: str, api_key: str, timeout: float, keep: re.Pattern[str]) -> dict[str, float]:
    """Reads a Prometheus exposition page and keeps the samples whose name matches."""
    text = get_text(metrics_url, api_key, timeout)
    if text is None:
        return {}
    out: dict[str, float] = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        match = SAMPLE.match(line)
        if not match or not keep.search(match.group("name")):
            continue
        try:
            value = float(match.group("value"))
        except ValueError:
            continue
        key = match.group("name") + (match.group("labels") or "")
        out[key] = value
    return out


def metrics_delta(before: dict[str, float], after: dict[str, float]) -> dict[str, float]:
    """What the run itself added, so a warm server's history does not pollute the record."""
    keys = set(before) | set(after)
    return {k: round(after.get(k, 0.0) - before.get(k, 0.0), 4) for k in sorted(keys)}


# ------------------------------------------------------------------------ measurement


def run_endpoint(cfg: dict, prompts: list[str], repeats: int) -> dict:
    """Sends every prompt `repeats` times, sequentially, and times each request.

    Sequential and single-stream on purpose: speculation spends idle arithmetic
    capacity, so concurrency one is where it has the most to gain, and it is the
    condition under which the acceptance arithmetic on the lesson page applies.
    """
    completions: list[str] = []
    latencies: list[float] = []
    token_counts: list[int | None] = []
    timings: list[dict] = []
    errors: list[str] = []

    metrics_before = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"], SPEC_METRIC)
    started = time.perf_counter()

    for _ in range(repeats):
        for prompt in prompts:
            payload = {
                "model": cfg["model"],
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": cfg["max_tokens"],
                "temperature": 0.0,
                "top_p": 1.0,
                "seed": cfg["seed"],
                "stream": False,
            }
            t0 = time.perf_counter()
            try:
                body = post_json(cfg["chat_url"], payload, cfg["api_key"], cfg["timeout"])
            except ServerError as exc:
                errors.append(str(exc))
                continue
            latencies.append(time.perf_counter() - t0)
            choices = body.get("choices") or [{}]
            message = choices[0].get("message") or {}
            completions.append(message.get("content") or "")
            usage = body.get("usage") or {}
            completion_tokens = usage.get("completion_tokens")
            token_counts.append(int(completion_tokens) if isinstance(completion_tokens, int) else None)
            if isinstance(body.get("timings"), dict):
                timings.append(body["timings"])

    wall = time.perf_counter() - started
    metrics_after = scrape_metrics(cfg["metrics_url"], cfg["api_key"], cfg["timeout"], SPEC_METRIC)

    counted = [t for t in token_counts if t is not None]
    total_tokens = sum(counted) if counted else None
    complete = len(counted) == len(latencies) and bool(counted)

    return {
        "label": cfg["label"],
        "base_url": cfg["base_url"],
        "model": cfg["model"],
        "requests": len(prompts) * repeats,
        "completed": len(latencies),
        "failed": len(errors),
        "first_error": errors[0] if errors else None,
        "wall_s": round(wall, 3),
        "latency_s": {
            "p50": round(statistics.median(latencies), 4) if latencies else None,
            "mean": round(statistics.fmean(latencies), 4) if latencies else None,
            "max": round(max(latencies), 4) if latencies else None,
        },
        "output_tokens": total_tokens,
        "output_tokens_per_s": (
            round(total_tokens / wall, 2) if complete and total_tokens and wall > 0 else None
        ),
        "tokens_counted_by_server": complete,
        "server_timings_sample": timings[0] if timings else None,
        "spec_metrics_delta": metrics_delta(metrics_before, metrics_after),
        "completions": completions,
    }


def compare_text(baseline: list[str], speculative: list[str]) -> dict:
    """Byte-for-byte comparison of the two sets of greedy completions.

    Verification means the accepted tokens are the ones the target model would have
    produced, so at temperature zero the two runs should agree exactly. They can still
    differ for reasons that have nothing to do with the draft - a different batch
    composition changes the order of floating-point reductions, and the engines say so -
    which is why this reports where the first divergence is rather than only that there
    was one.
    """
    pairs = list(zip(baseline, speculative))
    identical = [i for i, (a, b) in enumerate(pairs) if a == b]
    divergent = []
    for i, (a, b) in enumerate(pairs):
        if a == b:
            continue
        cut = 0
        for cut, (ca, cb) in enumerate(zip(a, b)):
            if ca != cb:
                break
        else:
            cut = min(len(a), len(b))
        divergent.append(
            {
                "prompt_index": i,
                "first_difference_at_char": cut,
                "baseline_tail": a[cut : cut + 60],
                "speculative_tail": b[cut : cut + 60],
            }
        )
    return {
        "compared": len(pairs),
        "identical": len(identical),
        "divergent": len(divergent),
        "divergences": divergent[:5],
    }


# ---------------------------------------------------------------------------- output


def print_endpoint(row: dict) -> None:
    """One endpoint's result, as an aligned block."""
    print(f"  {row['label']}  ({row['base_url']})")
    print(f"    requests    {row['completed']}/{row['requests']} completed, {row['failed']} failed")
    print(f"    wall clock  {row['wall_s']:.2f} s")
    if row["output_tokens_per_s"] is not None:
        print(f"    output      {row['output_tokens']} tokens, {row['output_tokens_per_s']:.2f} tokens/s")
    else:
        print("    output      the server did not return a usage object; tokens/s not computed")
    if row["latency_s"]["p50"] is not None:
        print(f"    latency     p50 {row['latency_s']['p50']:.3f} s, max {row['latency_s']['max']:.3f} s")
    if row["spec_metrics_delta"]:
        print("    acceptance statistics from the server's /metrics during this run:")
        for name, value in row["spec_metrics_delta"].items():
            print(f"      {name} += {value}")
    else:
        print("    acceptance  the server exposed no speculation metrics; record that as the finding")
    if row["first_error"]:
        print(f"    first error {row['first_error']}")


def append_labbook(path: str, record: dict) -> None:
    """Appends one JSON line, in the course notebook format."""
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def endpoint_config(base_url: str, args: argparse.Namespace, label: str) -> dict:
    """Derives the chat and metrics URLs from an OpenAI-compatible base URL."""
    parts = urlsplit(base_url)
    if parts.scheme != "http":
        raise SystemExit("This tool speaks plain HTTP only; point it at a localhost or LAN endpoint.")
    if not parts.hostname:
        raise SystemExit(f"Could not read a host from {base_url!r}.")
    root = f"{parts.scheme}://{parts.netloc}"
    path = parts.path.rstrip("/") or "/v1"
    return {
        "base_url": base_url,
        "chat_url": f"{root}{path}/chat/completions",
        "metrics_url": f"{root}/metrics",
        "model": args.model,
        "max_tokens": args.max_tokens,
        "seed": args.seed,
        "timeout": args.timeout,
        "api_key": os.environ.get(args.api_key_env, ""),
        "label": label,
    }


def parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--baseline-url", required=True,
                        help="OpenAI-compatible base URL of the server WITHOUT speculation, ending in /v1")
    parser.add_argument("--speculative-url", default=None,
                        help="base URL of the server WITH speculation; omit to measure one endpoint only")
    parser.add_argument("--model", required=True, help="model name the server reports at /v1/models")
    parser.add_argument("--max-tokens", type=int, default=192)
    parser.add_argument("--seed", type=int, default=0,
                        help="sent as the request seed; ignored by servers that do not accept one")
    parser.add_argument("--repeats", type=int, default=1, help="times to send the whole prompt set")
    parser.add_argument("--timeout", type=float, default=300.0)
    parser.add_argument("--prompt-set", default="builtin", choices=["builtin", "file"])
    parser.add_argument("--prompts", default=None, help="file of prompts, one per line, for --prompt-set file")
    parser.add_argument("--label", default="speculative-run", help="tag written into the notebook lines")
    parser.add_argument("--engine", default="unknown", help="engine name recorded in the notebook, e.g. llama.cpp")
    parser.add_argument("--engine-version", default="unknown", help="engine version recorded in the notebook")
    parser.add_argument("--api-key-env", default="SPEC_API_KEY",
                        help="environment variable holding the API key, if the servers need one")
    parser.add_argument("--labbook", default="labbook.md")
    return parser.parse_args(argv)


def main(argv: list[str]) -> int:
    args = parse_args(argv)
    prompts = load_prompts(args.prompt_set, args.prompts)

    print(f"==> {args.label}: {args.model}, {len(prompts)} prompt(s) x {args.repeats} repeat(s), "
          f"max_tokens {args.max_tokens}, temperature 0")

    baseline = run_endpoint(endpoint_config(args.baseline_url, args, f"{args.label}/baseline"), prompts, args.repeats)
    print_endpoint(baseline)

    speculative = None
    comparison = None
    if args.speculative_url:
        speculative = run_endpoint(
            endpoint_config(args.speculative_url, args, f"{args.label}/speculative"), prompts, args.repeats
        )
        print_endpoint(speculative)
        comparison = compare_text(baseline["completions"], speculative["completions"])
        print("  comparison")
        print(f"    text        {comparison['identical']}/{comparison['compared']} completions identical, "
              f"{comparison['divergent']} divergent")
        for d in comparison["divergences"]:
            print(f"      prompt {d['prompt_index']}: first difference at character {d['first_difference_at_char']}")
        if baseline["output_tokens_per_s"] and speculative["output_tokens_per_s"]:
            ratio = speculative["output_tokens_per_s"] / baseline["output_tokens_per_s"]
            print(f"    speed       {ratio:.2f}x the baseline token rate on this prompt set")

    stamp = time.strftime("%Y-%m-%dT%H:%M:%S%z")
    for row in (baseline, speculative):
        if row is None:
            continue
        record = dict(row)
        # The completions are the evidence for the comparison, not part of the record:
        # a notebook line has to stay readable.
        record.pop("completions", None)
        record.update({
            "lab": "part-17/measure-speculative",
            "engine": args.engine,
            "engine_version": args.engine_version,
            "max_tokens": args.max_tokens,
            "repeats": args.repeats,
            "prompt_count": len(prompts),
            "recorded_at": stamp,
        })
        append_labbook(args.labbook, record)

    if comparison is not None:
        append_labbook(args.labbook, {
            "lab": "part-17/measure-speculative",
            "label": f"{args.label}/comparison",
            "engine": args.engine,
            "engine_version": args.engine_version,
            "model": args.model,
            "baseline_tokens_per_s": baseline["output_tokens_per_s"],
            "speculative_tokens_per_s": speculative["output_tokens_per_s"] if speculative else None,
            "text_comparison": comparison,
            "recorded_at": stamp,
        })

    print(f"    appended {2 if comparison else 1} line(s) to {args.labbook}")

    if baseline["completed"] == 0 or (speculative is not None and speculative["completed"] == 0):
        print("    at least one endpoint completed no requests; see the error above.", file=sys.stderr)
        return 1
    if comparison is not None and comparison["divergent"]:
        print("    completions differed between the two servers; the page explains what to check.",
              file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download measure-speculative.py434 lines

RunnableAll tracks

the number every later result is compared against

python3 measure-speculative.py \
  --baseline-url http://127.0.0.1:8080/v1 \
  --model local-chat \
  --max-tokens 192 \
  --engine llama.cpp \
  --label baseline \
  --labbook labbook.md

Output — what you should see

==> baseline: local-chat, 12 prompt(s) x 1 repeat(s), max_tokens 192, temperature 0
  baseline/baseline  (http://127.0.0.1:8080/v1)
    requests    12/12 completed, 0 failed
    wall clock  ...
    output      ... tokens, ... tokens/s
    acceptance  the server exposed no speculation metrics; record that as the finding
    appended 1 line to labbook.md

The acceptance line saying nothing was exposed is correct here: there is no draft yet, so there is nothing to accept. Seeing it now tells you the scraper is reaching /metrics at all, which is worth knowing before it matters.

3. Collect your model’s own answers

This is the self-distillation step from the previous lesson, and it is the same on every track because it only talks to a server.

RunnableAll tracks

make-draft-data.py

#!/usr/bin/env python3
"""Build the training data a draft head needs: the target model's own answers, and its features.

Purpose: a draft is trained to agree with one particular target model, so its training set
    is not human text but the target's own generations. This script has two stages. The
    "generate" stage sends your prompts through the served model and writes the answers as
    a ShareGPT-shaped JSONL file, which is the format the Medusa and EAGLE training
    repositories read. The "hidden-states" stage runs the same conversations back through
    the target with transformers and saves the second-to-top-layer hidden state per token,
    which is what an EAGLE-style draft regresses on. Run stage one on any track; run stage
    two only if the recipe you chose needs features.
Platform: generate: all (spark, strix, mac, nvidia; standard library only, talks to a
    server). hidden-states: spark, strix, nvidia and mac, and needs torch and transformers
    installed; it is the only stage that loads the weights locally.
Minimum memory: 8 GB for the generate stage. The hidden-states stage needs the target model
    resident: roughly 2 bytes per parameter at bfloat16, so about 17 GB for an 8B target and
    about 9 GB for a 4B one, plus the activations for one sequence.
Assumes: for generate, a served model reachable over plain HTTP at an OpenAI-compatible
    endpoint (Part 9's gateway, or llama-server, or vLLM). For hidden-states, torch and
    transformers in the active environment and the target's Hugging Face directory or id.
    Both stages append one record to the lab notebook through draftlog.py, which must sit
    next to this file.

Usage:
    # stage 1: the target's own answers to your prompts
    python3 make-draft-data.py generate --base-url http://127.0.0.1:8000/v1 \\
        --model my-fine-tune --prompts prompts.txt --out data/draft-data.jsonl \\
        --max-tokens 384 --temperature 0.0 --labbook labbook.md

    # stage 2: the features an EAGLE-style draft regresses on
    python3 make-draft-data.py hidden-states --target ./runs/my-fine-tune-merged \\
        --data data/draft-data.jsonl --out-dir data/hidden --limit 512 --labbook labbook.md

An API key, if the server needs one, is read from the environment variable named by
--api-key-env. No key is written to this file, to the output or to the notebook.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from urllib.parse import urlsplit

import draftlog

# A starter prompt set for readers who have not collected their own yet. Twelve prompts is
# far too few to train a usable draft; it is enough to prove the pipeline runs end to end,
# and the page says where the real prompts should come from.
STARTER_PROMPTS = [
    "Summarise what a key-value cache is and why it grows with the conversation.",
    "Rewrite this shell command with error handling: curl -s localhost:8080/v1/models",
    "Explain the difference between prefill and decode to a colleague.",
    "Write a Python function that returns the median of a list without imports.",
    "Give four fields that must accompany a tokens-per-second measurement.",
    "Describe continuous batching in plain language.",
    "What is an acceptance rate in speculative decoding?",
    "List three reasons a language model server refuses to start.",
    "Turn this into JSON with keys name and port: llama-server 8080, vllm 8000.",
    "Explain quantisation to somebody who knows what a floating point number is.",
    "Why can a smaller model that fits beat a larger one that does not?",
    "Write three sentences about what to record after a benchmark run.",
]


# ------------------------------------------------------------------------ HTTP


class ServerError(Exception):
    """The server did not answer in a way this script can use."""


def post_json(url: str, payload: dict, api_key: str, timeout: float) -> dict:
    """One non-streaming POST. Returns the decoded JSON body."""
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(url, data=body, method="POST")
    request.add_header("Content-Type", "application/json")
    if api_key:
        request.add_header("Authorization", f"Bearer {api_key}")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310 - http only, checked below
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:200]
        raise ServerError(f"HTTP {exc.code} from {url}: {detail}") from exc
    except (urllib.error.URLError, OSError, json.JSONDecodeError) as exc:
        raise ServerError(f"{type(exc).__name__} talking to {url}: {exc}") from exc


# --------------------------------------------------------------------- stage 1


def read_prompts(path: str | None) -> list[str]:
    """One prompt per non-blank line, or the starter set."""
    if not path:
        return list(STARTER_PROMPTS)
    with open(path, encoding="utf-8") as handle:
        prompts = [line.strip() for line in handle if line.strip()]
    if not prompts:
        raise SystemExit(f"{path} contained no prompts")
    return prompts


def stage_generate(args: argparse.Namespace) -> int:
    """Collect the target model's own answers and write them as ShareGPT-shaped JSONL."""
    parts = urlsplit(args.base_url)
    if parts.scheme != "http":
        print("This tool speaks plain HTTP only; point it at a localhost or LAN endpoint.", file=sys.stderr)
        return 2
    if not parts.hostname:
        print(f"Could not read a host from --base-url {args.base_url!r}.", file=sys.stderr)
        return 2
    chat_url = f"{parts.scheme}://{parts.netloc}{parts.path.rstrip('/') or '/v1'}/chat/completions"
    api_key = os.environ.get(args.api_key_env, "")

    prompts = read_prompts(args.prompts)
    out_path = Path(args.out)
    out_path.parent.mkdir(parents=True, exist_ok=True)

    print(f"==> generating {len(prompts)} answer(s) from {args.model} at {args.base_url}")
    if args.temperature > 0:
        print("    temperature is above zero: the draft will learn a sampled slice of the target's")
        print("    behaviour, which is usually what you want for coverage. Record the value.")

    kept = 0
    failed = 0
    started = time.perf_counter()
    with out_path.open("w", encoding="utf-8") as out:
        for i, prompt in enumerate(prompts):
            payload = {
                "model": args.model,
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": args.max_tokens,
                "temperature": args.temperature,
                "stream": False,
            }
            try:
                body = post_json(chat_url, payload, api_key, args.timeout)
            except ServerError as exc:
                failed += 1
                print(f"    [{i + 1}/{len(prompts)}] failed: {exc}", file=sys.stderr)
                continue
            choices = body.get("choices") or [{}]
            answer = ((choices[0].get("message") or {}).get("content") or "").strip()
            if not answer:
                failed += 1
                print(f"    [{i + 1}/{len(prompts)}] the server returned an empty answer", file=sys.stderr)
                continue
            out.write(json.dumps({
                "id": f"draft-{i:06d}",
                "conversations": [
                    {"from": "human", "value": prompt},
                    {"from": "gpt", "value": answer},
                ],
            }, ensure_ascii=False) + "\n")
            kept += 1
            if (i + 1) % 20 == 0:
                print(f"    [{i + 1}/{len(prompts)}] {kept} kept, {failed} failed")

    wall = time.perf_counter() - started
    print(f"==> wrote {kept} example(s) to {out_path} in {wall:.1f} s ({failed} failed)")

    draftlog.record(
        labbook=args.labbook,
        lab="part-17/make-draft-data/generate",
        model=args.model,
        dataset={
            "path": str(out_path),
            "sha256": draftlog.file_sha256(out_path),
            "examples": kept,
            "prompt_source": args.prompts or "built-in starter set",
        },
        hyperparameters={
            "stage": "generate",
            "max_tokens": args.max_tokens,
            "temperature": args.temperature,
            "base_url": args.base_url,
        },
        seed=args.seed,
        notes=f"{failed} prompt(s) failed or returned nothing",
    )
    print(f"    recorded the run in {args.labbook}")
    return 1 if kept == 0 else 0


# --------------------------------------------------------------------- stage 2


def stage_hidden_states(args: argparse.Namespace) -> int:
    """Save the target's second-to-top-layer hidden states for each conversation.

    An EAGLE-style draft does not predict tokens from tokens; it predicts the target's
    own features one step ahead, and it is trained against the features the target
    actually produced. That is why this stage exists and why it needs the weights
    locally: no serving API exposes hidden states.
    """
    try:
        import torch  # noqa: PLC0415 - optional heavy dependency, only this stage needs it
        from transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: PLC0415
    except ImportError as exc:
        print(f"This stage needs torch and transformers in the active environment: {exc}", file=sys.stderr)
        return 2

    data_path = Path(args.data)
    if not data_path.is_file():
        print(f"{data_path} does not exist; run the generate stage first.", file=sys.stderr)
        return 2
    out_dir = Path(args.out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)

    device = "cuda" if torch.cuda.is_available() else (
        "mps" if getattr(torch.backends, "mps", None) is not None and torch.backends.mps.is_available() else "cpu"
    )
    dtype = torch.bfloat16 if device == "cuda" else torch.float32
    print(f"==> loading {args.target} on {device} in {dtype}")

    tokenizer = AutoTokenizer.from_pretrained(args.target)
    model = AutoModelForCausalLM.from_pretrained(args.target, dtype=dtype)
    model.to(device)
    model.eval()

    index_path = out_dir / "index.jsonl"
    written = 0
    skipped = 0
    started = time.perf_counter()

    with data_path.open(encoding="utf-8") as handle, index_path.open("w", encoding="utf-8") as index:
        for line_no, line in enumerate(handle):
            if args.limit and written >= args.limit:
                break
            line = line.strip()
            if not line:
                continue
            example = json.loads(line)
            turns = example.get("conversations") or []
            messages = [
                {"role": "user" if t.get("from") == "human" else "assistant", "content": t.get("value", "")}
                for t in turns
            ]
            if len(messages) < 2:
                skipped += 1
                continue
            text = tokenizer.apply_chat_template(messages, tokenize=False)
            encoded = tokenizer(text, return_tensors="pt", truncation=True, max_length=args.max_length)
            input_ids = encoded["input_ids"].to(device)
            with torch.no_grad():
                out = model(input_ids, output_hidden_states=True)
            # hidden_states[0] is the embedding output and hidden_states[-1] the final layer,
            # so [-2] is the second-to-top layer the EAGLE paper describes.
            features = out.hidden_states[-2][0].to(torch.float16).cpu()
            target_path = out_dir / f"{example.get('id', f'row-{line_no:06d}')}.pt"
            torch.save({"input_ids": input_ids[0].cpu(), "hidden_states": features}, target_path)
            index.write(json.dumps({
                "id": example.get("id", f"row-{line_no:06d}"),
                "file": target_path.name,
                "tokens": int(input_ids.shape[1]),
                "hidden_size": int(features.shape[-1]),
            }) + "\n")
            written += 1
            if written % 25 == 0:
                print(f"    {written} example(s) written")

    wall = time.perf_counter() - started
    print(f"==> wrote {written} feature file(s) to {out_dir} in {wall:.1f} s ({skipped} skipped)")
    print("    These files are large. Check the size before you start a full run:")
    print(f"    du -sh {out_dir}")

    draftlog.record(
        labbook=args.labbook,
        lab="part-17/make-draft-data/hidden-states",
        model=args.target,
        dataset={
            "path": str(data_path),
            "sha256": draftlog.file_sha256(data_path),
            "examples": written,
            "features_dir": str(out_dir),
        },
        hyperparameters={
            "stage": "hidden-states",
            "layer": "second-to-top (hidden_states[-2])",
            "max_length": args.max_length,
            "device": device,
            "dtype": str(dtype),
        },
        seed=args.seed,
        notes=f"{skipped} row(s) skipped for having fewer than two turns",
    )
    print(f"    recorded the run in {args.labbook}")
    return 1 if written == 0 else 0


# ---------------------------------------------------------------------------- CLI


def parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md")
    parser.add_argument("--seed", type=int, default=0)
    sub = parser.add_subparsers(dest="stage", required=True)

    gen = sub.add_parser("generate", help="collect the target model's own answers through a served endpoint")
    gen.add_argument("--base-url", default="http://127.0.0.1:8000/v1")
    gen.add_argument("--model", required=True, help="model name the server reports at /v1/models")
    gen.add_argument("--prompts", default=None, help="file of prompts, one per line")
    gen.add_argument("--out", default="data/draft-data.jsonl")
    gen.add_argument("--max-tokens", type=int, default=384)
    gen.add_argument("--temperature", type=float, default=0.0)
    gen.add_argument("--timeout", type=float, default=300.0)
    gen.add_argument("--api-key-env", default="SPEC_API_KEY")

    hid = sub.add_parser("hidden-states", help="save the target's second-to-top-layer features per token")
    hid.add_argument("--target", required=True, help="Hugging Face directory or id of the target model")
    hid.add_argument("--data", default="data/draft-data.jsonl")
    hid.add_argument("--out-dir", default="data/hidden")
    hid.add_argument("--max-length", type=int, default=1024)
    hid.add_argument("--limit", type=int, default=0, help="stop after this many examples; 0 means all")

    return parser.parse_args(argv)


def main(argv: list[str]) -> int:
    args = parse_args(argv)
    if args.stage == "generate":
        return stage_generate(args)
    return stage_hidden_states(args)


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download make-draft-data.py334 lines

RunnableAll tracks

draftlog.py

"""Append one machine-readable record per draft-model run to the lab notebook.

Purpose: Part 17's self-contained copy of the run-log format defined in Part 11, so that
         this part's scripts record a data-preparation run, a draft-training run or a
         serving measurement identically without depending on Part 11's files being on
         the path. Every field is filled in or written as null, because a reader a month
         later has to be able to tell "not recorded" from "not applicable".
Platform: all (standard library only; torch, transformers and mlx are inspected for their
          version strings only if they happen to be installed)
Minimum memory: 8 GB
Assumes: Python 3.10 or newer. The lab notebook is created if it does not exist. git is
         optional and is used only to record the commit the configuration was at.

Usage: imported by this part's Python scripts:
           import draftlog; draftlog.record(labbook="labbook.md", lab="part-17/make-draft-data", ...)
       or called from a shell script with the run's own fields as JSON on stdin:
           python3 draftlog.py --record --labbook labbook.md < fields.json
       or run with no arguments to print the field list and exit.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import platform
import secrets
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

# The same field set Parts 11 and 13 define, so a Part 17 line can be read by the same
# tooling. A record missing any of them is refused: a partial record is harder to
# interpret than no record at all.
FIELDS = (
    "run_id", "lab", "date", "config_commit", "model", "dataset",
    "hyperparameters", "seed", "hardware", "versions", "losses", "scores", "notes",
)


def new_run_id() -> str:
    """A short identifier that sorts by time and does not collide between runs."""
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


def file_sha256(path: str | os.PathLike[str], chunk: int = 1 << 20) -> str | None:
    """Content hash of a data file, so a run can be tied to the exact bytes it read."""
    p = Path(path)
    if not p.is_file():
        return None
    digest = hashlib.sha256()
    with p.open("rb") as handle:
        while True:
            block = handle.read(chunk)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def git_commit(path: str | os.PathLike[str] = ".") -> str | None:
    """The commit the configuration is at, or None outside a repository."""
    if shutil.which("git") is None:
        return None
    target = Path(path)
    cwd = target if target.is_dir() else target.parent
    try:
        rev = subprocess.run(
            ["git", "rev-parse", "--short", "HEAD"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
        dirty = subprocess.run(
            ["git", "status", "--porcelain"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
    except (subprocess.SubprocessError, OSError):
        return None
    return f"{rev}-dirty" if dirty else rev


def describe_hardware() -> dict[str, Any]:
    """What the run was executed on, as far as it can be established without extra packages."""
    info: dict[str, Any] = {
        "os": f"{platform.system()} {platform.release()}",
        "machine": platform.machine(),
        "python": platform.python_version(),
        "accelerator": "cpu",
        "device_name": None,
    }
    try:
        import torch  # noqa: PLC0415 - optional, and only for reporting
    except ImportError:
        return info
    if torch.cuda.is_available():
        info["accelerator"] = "cuda"
        info["device_name"] = torch.cuda.get_device_name(0)
    else:
        mps = getattr(torch.backends, "mps", None)
        if mps is not None and mps.is_available():
            info["accelerator"] = "mps"
            info["device_name"] = platform.processor() or "Apple silicon"
    return info


def package_versions(
    names: tuple[str, ...] = ("torch", "transformers", "trl", "peft", "datasets", "accelerate", "mlx", "mlx-lm"),
) -> dict[str, str | None]:
    """Version strings for the packages that decide what a run actually did."""
    from importlib.metadata import PackageNotFoundError, version  # noqa: PLC0415

    out: dict[str, str | None] = {}
    for name in names:
        try:
            out[name] = version(name)
        except PackageNotFoundError:
            out[name] = None
    return out


def build_record(
    lab: str,
    model: str,
    dataset: dict[str, Any],
    hyperparameters: dict[str, Any],
    seed: int,
    losses: dict[str, Any] | None = None,
    scores: dict[str, Any] | None = None,
    config_path: str | os.PathLike[str] | None = None,
    notes: str | None = None,
) -> dict[str, Any]:
    """Assemble the record. Kept separate from writing so it can be inspected first."""
    return {
        "run_id": new_run_id(),
        "lab": lab,
        "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "config_commit": git_commit(config_path if config_path is not None else "."),
        "model": model,
        "dataset": dataset,
        "hyperparameters": hyperparameters,
        "seed": seed,
        "hardware": describe_hardware(),
        "versions": package_versions(),
        "losses": losses or {},
        "scores": scores or {},
        "notes": notes,
    }


def append(record: dict[str, Any], labbook: str | os.PathLike[str] = "labbook.md") -> Path:
    """Append one JSON line. Missing keys are an error: a partial record is worse than none."""
    missing = [f for f in FIELDS if f not in record]
    if missing:
        raise ValueError(f"run record is missing required fields: {', '.join(missing)}")
    path = Path(labbook)
    if not path.exists():
        path.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    return path


def record(labbook: str | os.PathLike[str] = "labbook.md", **kwargs: Any) -> dict[str, Any]:
    """Build and append in one call; returns the record so a caller can print it."""
    rec = build_record(**kwargs)
    append(rec, labbook)
    return rec


def main() -> None:
    parser = argparse.ArgumentParser(description="Run-log helper for the Part 17 draft-model lab.")
    parser.add_argument("--record", action="store_true", help="read the run's own fields as JSON on stdin")
    parser.add_argument("--labbook", default="labbook.md")
    args = parser.parse_args()
    if not args.record:
        print(__doc__)
        print("Fields in every record:", ", ".join(FIELDS))
        return
    fields = json.load(sys.stdin)
    allowed = {"lab", "model", "dataset", "hyperparameters", "seed", "losses", "scores", "config_path", "notes"}
    unknown = set(fields) - allowed
    if unknown:
        raise SystemExit(f"unknown field(s) on stdin: {', '.join(sorted(unknown))}")
    rec = record(labbook=args.labbook, **fields)
    print(f"recorded run {rec['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download draftlog.py192 lines

Write a prompt file first. The prompts decide where the draft will agree, so they should look like your traffic: pull them from the evaluation set you built in Part 10, from the fine-tuning set from Part 13, or from your gateway’s logs.

RunnableAll tracks

generate the training set through the server you already have

python3 make-draft-data.py generate \
  --base-url http://127.0.0.1:8080/v1 \
  --model local-chat \
  --prompts my-prompts.txt \
  --out data/draft-data.jsonl \
  --max-tokens 384 \
  --temperature 0.0 \
  --labbook labbook.md

Running with no --prompts uses a twelve-prompt starter set. That is enough to prove the pipeline and far too few to train a usable draft; a few thousand conversations is the order of magnitude the published recipes work at, and you should expect to leave this running.

✦Tip

Generate at a temperature above zero if you want coverage rather than only the single most likely continuation. A draft trained on greedy output learns the centre of the distribution and nothing about its shoulders. Record the value either way: it is a hyperparameter of the draft, not a detail of the data collection.

4. Extract hidden states, if your recipe needs them

Skip this on the no-training path and on the Medusa route; only an EAGLE-style draft regresses on the target’s features.

RunnableTrack S · DGX Spark

the target's second-to-top-layer features, per token

python3 make-draft-data.py hidden-states \
  --target ~/models/my-fine-tune-merged \
  --data data/draft-data.jsonl \
  --out-dir data/hidden \
  --max-length 1024 \
  --limit 64 \
  --labbook labbook.md

The --limit 64 is deliberate: run a small batch first and look at what it wrote.

RunnableAll tracks

check the size before you commit to the full pass

du -sh data/hidden

▤Memory budget

This is the step that fills a disk. Each example stores a vector per token at the model’s hidden width, and the directory grows linearly in both the number of conversations and their length. If sixty-four examples took more space than you expected, cap --max-length rather than hoping.

5. Train the draft

Tracks S and N. Tracks X and M skip to task 6 with the reason recorded on the Requirements tab.

RunnableAll tracks

train-draft.sh

#!/usr/bin/env bash
# Purpose: train a draft head for one target model with whichever of the three published
#          recipes fits the machine - Medusa heads on a frozen backbone, EAGLE-3 through
#          the EAGLE repository, or EAGLE-3 through SpecForge - by fetching the upstream
#          repository at a pinned commit and running its own documented training command
#          against the data make-draft-data.py wrote
# Platform: spark, strix, nvidia (CUDA, or ROCm which also presents as cuda to PyTorch).
#           Track M is not supported here: neither upstream trainer documents an Apple
#           silicon path, and the lab page gives Track M a reduced path that needs no
#           training at all.
# Minimum memory: 24 GB for a Medusa head on an 8B target at bfloat16; 12-16 GB is enough
#           for a Medusa head on a 4B target. EAGLE-3 training wants considerably more and
#           the lab page states what each track can realistically attempt.
# Assumes: git, python3 and a Python environment with torch installed; the recipe's own
#          requirements are installed by this script into that environment; the data file
#          from make-draft-data.py exists; draftlog.py sits next to this script. Nothing
#          here deletes anything: an existing checkout is reused, not replaced.
#
# Usage: bash train-draft.sh RECIPE TARGET_MODEL DATA_FILE [OUT_DIR]
#   RECIPE        medusa | eagle3 | specforge
#   TARGET_MODEL  Hugging Face id or local directory of the model the draft will serve
#   DATA_FILE     the JSONL make-draft-data.py generate wrote
#   OUT_DIR       where the trained draft is written (default drafts/RECIPE)
#
# Environment: WORK_DIR (default $HOME/draft-training) for the upstream checkouts,
#              GPUS (default 1), EPOCHS, LR, MEDUSA_NUM_HEADS, MEDUSA_NUM_LAYERS,
#              MAX_LENGTH, SPECFORGE_CONFIG, LABBOOK, DRY_RUN=1 to print the training
#              command and stop without running it.
set -euo pipefail

RECIPE="${1:-}"
TARGET="${2:-}"
DATA="${3:-}"
OUT_DIR="${4:-drafts/${RECIPE:-unset}}"

WORK_DIR="${WORK_DIR:-$HOME/draft-training}"
GPUS="${GPUS:-1}"
EPOCHS="${EPOCHS:-2}"
LR="${LR:-1e-3}"
MEDUSA_NUM_HEADS="${MEDUSA_NUM_HEADS:-3}"
MEDUSA_NUM_LAYERS="${MEDUSA_NUM_LAYERS:-1}"
MAX_LENGTH="${MAX_LENGTH:-1024}"
LABBOOK="${LABBOOK:-labbook.md}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Pinned so that a rerun trains the same thing. Replace these with the commit you actually
# used and record it in the notebook; both repositories move.
MEDUSA_REPO="https://github.com/FasterDecoding/Medusa"
EAGLE_REPO="https://github.com/SafeAILab/EAGLE"
SPECFORGE_REPO="https://github.com/sgl-project/SpecForge"

die() { echo "train-draft: $*" >&2; exit 1; }

[[ -n "$RECIPE" && -n "$TARGET" && -n "$DATA" ]] ||
  die "usage: bash train-draft.sh RECIPE TARGET_MODEL DATA_FILE [OUT_DIR]"
[[ "$RECIPE" == "medusa" || "$RECIPE" == "eagle3" || "$RECIPE" == "specforge" ]] ||
  die "RECIPE must be medusa, eagle3 or specforge (got '$RECIPE')"
[[ -f "$DATA" ]] || die "$DATA does not exist; run make-draft-data.py generate first"
[[ -f "$HERE/draftlog.py" ]] || die "draftlog.py is not next to this script"
command -v git >/dev/null || die "git is not on PATH"
command -v python3 >/dev/null || die "python3 is not on PATH"
python3 -c "import torch" 2>/dev/null || die "torch is not importable in this environment"

EXAMPLES="$(wc -l < "$DATA" | tr -d ' ')"
[[ "$EXAMPLES" -gt 0 ]] || die "$DATA has no rows"

mkdir -p "$WORK_DIR" "$OUT_DIR"

echo "==> recipe        $RECIPE"
echo "    target        $TARGET"
echo "    data          $DATA ($EXAMPLES example(s))"
echo "    output        $OUT_DIR"
echo "    checkouts in  $WORK_DIR"
echo

# Clones once and leaves an existing checkout alone, so a second run does not throw away
# local edits or re-download hundreds of megabytes.
fetch_repo() {
  local url="$1" dir="$2"
  if [[ -d "$dir/.git" ]]; then
    echo "    reusing existing checkout at $dir"
  else
    echo "    cloning $url into $dir"
    git clone --depth 1 "$url" "$dir"
  fi
  git -C "$dir" rev-parse --short HEAD
}

run_or_print() {
  echo
  echo "    the command about to run:"
  printf '      %s\n' "$*"
  echo
  if [[ "${DRY_RUN:-0}" == "1" ]]; then
    echo "    DRY_RUN=1, so stopping here."
    return 0
  fi
  "$@"
}

START="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
SECONDS=0
COMMIT=""

case "$RECIPE" in
  medusa)
    DIR="$WORK_DIR/Medusa"
    COMMIT="$(fetch_repo "$MEDUSA_REPO" "$DIR")"
    echo "    Medusa at commit $COMMIT (Apache-2.0)"
    echo "    Installing the repository into the active environment"
    python3 -m pip install --quiet -e "$DIR"
    command -v torchrun >/dev/null || die "torchrun is not on PATH; it ships with torch"
    # The option names are Medusa's own, from the training command in its README.
    run_or_print torchrun --nproc_per_node="$GPUS" "$DIR/medusa/train/train_legacy.py" \
      --model_name_or_path "$TARGET" \
      --data_path "$DATA" \
      --bf16 True \
      --output_dir "$OUT_DIR" \
      --num_train_epochs "$EPOCHS" \
      --per_device_train_batch_size 1 \
      --gradient_accumulation_steps 4 \
      --save_strategy "no" \
      --learning_rate "$LR" \
      --weight_decay 0.0 \
      --warmup_ratio 0.1 \
      --lr_scheduler_type "cosine" \
      --logging_steps 1 \
      --model_max_length "$MAX_LENGTH" \
      --lazy_preprocess True \
      --medusa_num_heads "$MEDUSA_NUM_HEADS" \
      --medusa_num_layers "$MEDUSA_NUM_LAYERS"
    ;;

  eagle3)
    DIR="$WORK_DIR/EAGLE"
    COMMIT="$(fetch_repo "$EAGLE_REPO" "$DIR")"
    echo "    EAGLE at commit $COMMIT (Apache-2.0)"
    echo "    Its README recommends SpecForge for out-of-the-box EAGLE-3 training;"
    echo "    this branch runs the repository's own trainer for readers who want it."
    command -v deepspeed >/dev/null || die "deepspeed is not on PATH; install it or use RECIPE=specforge"
    [[ -f "$DIR/eagle/traineagle3/main.py" ]] ||
      die "eagle/traineagle3/main.py is missing from the checkout; the layout has changed, read the README"
    [[ -f "$DIR/eagle/traineagle3/ds_config.json" ]] ||
      die "eagle/traineagle3/ds_config.json is missing; the layout has changed, read the README"
    echo "    Set the dataset and target paths inside the repository's own config before this runs;"
    echo "    the trainer reads them from there and not from this script's arguments."
    ( cd "$DIR/eagle/traineagle3" &&
      run_or_print deepspeed main.py --deepspeed_config ds_config.json )
    ;;

  specforge)
    DIR="$WORK_DIR/SpecForge"
    COMMIT="$(fetch_repo "$SPECFORGE_REPO" "$DIR")"
    echo "    SpecForge at commit $COMMIT (MIT)"
    python3 -m pip install --quiet -e "$DIR"
    command -v specforge >/dev/null || die "the specforge command is not on PATH after installation"
    CONFIG="${SPECFORGE_CONFIG:-$DIR/examples/configs/online/disaggregated/external/qwen3-8b-eagle3-disaggregated.yaml}"
    [[ -f "$CONFIG" ]] ||
      die "config $CONFIG not found; list $DIR/examples/configs and set SPECFORGE_CONFIG"
    # SpecForge takes every setting from one YAML file and accepts dotted overrides on the
    # command line, so the target and the data are passed as overrides rather than flags.
    run_or_print specforge train --config "$CONFIG" \
      "data.train_data_path=$DATA" \
      "output_dir=$OUT_DIR"
    ;;
esac

ELAPSED="$SECONDS"

if [[ "${DRY_RUN:-0}" == "1" ]]; then
  echo "==> DRY_RUN=1: nothing was trained and nothing was recorded."
  exit 0
fi

echo
echo "==> training finished in $((ELAPSED / 60)) minute(s); recording the run"

python3 "$HERE/draftlog.py" --record --labbook "$LABBOOK" <<JSON
{
  "lab": "part-17/train-draft/$RECIPE",
  "model": "$TARGET",
  "dataset": {"path": "$DATA", "examples": $EXAMPLES},
  "hyperparameters": {
    "recipe": "$RECIPE",
    "upstream_commit": "$COMMIT",
    "gpus": $GPUS,
    "epochs": $EPOCHS,
    "learning_rate": "$LR",
    "medusa_num_heads": $MEDUSA_NUM_HEADS,
    "medusa_num_layers": $MEDUSA_NUM_LAYERS,
    "model_max_length": $MAX_LENGTH,
    "output_dir": "$OUT_DIR"
  },
  "seed": 0,
  "losses": {},
  "scores": {},
  "notes": "started $START, ran for $ELAPSED seconds; acceptance rate and tokens per second come from measure-speculative.py, not from here"
}
JSON

echo "    draft written under $OUT_DIR"
echo "    next: serve it with serve-with-draft.sh and measure it with measure-speculative.py"

Download train-draft.sh202 lines

The script fetches the upstream repository, prints the exact training command it is about to run, and then runs it. Look at the printed command before you let it go: it is the repository’s own documented command with your paths substituted, and reading it is how you learn what the recipe actually does.

RunnableAll tracks

print the command and stop, so you can read it first

DRY_RUN=1 bash train-draft.sh medusa ~/models/my-fine-tune-merged data/draft-data.jsonl drafts/medusa

RunnableTrack N · NVIDIA GPU

a Medusa head on a frozen backbone

GPUS=1 EPOCHS=2 MEDUSA_NUM_HEADS=3 \
  bash train-draft.sh medusa ~/models/my-fine-tune-merged data/draft-data.jsonl drafts/medusa

RunnableTrack S · DGX Spark

EAGLE-3 through SpecForge, whose own guide is the reference

bash train-draft.sh specforge ~/models/my-fine-tune-merged data/draft-data.jsonl drafts/eagle3

SpecForge takes every setting from one YAML file and accepts dotted overrides, so the script passes the data path and the output directory as overrides rather than as flags. Point SPECFORGE_CONFIG at a different example configuration if the default one does not match your target’s family.

⚠Warning

Both training routes install a repository into your active Python environment with an editable install, which can move the versions of torch, transformers or accelerate that Parts 11 to 14 depend on. Do this in a dedicated virtual environment, not in the one holding your fine-tuning work. If you have only one environment, record its package versions before you start: the run log written by draftlog.py captures them for you.

6. Serve the same model with a draft

Now the second server, on a second port, with everything else identical.

RunnableAll tracks

fetch a small draft of the same family, then serve the pair

hf download unsloth/Qwen3-0.6B-GGUF --include "*Q4_K_M*" \
  --local-dir ~/models/unsloth/Qwen3-0.6B-GGUF

ls ~/models/unsloth/Qwen3-0.6B-GGUF

The pattern is safer than an exact file name, which repositories rename. Take the path the ls printed and pass it as the draft.

RunnableAll tracks

the speculative server on port 8081

bash serve-with-draft.sh llamacpp \
  ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
  ~/models/unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf \
  8081

RunnableTrack S · DGX Spark

the same thing on vLLM, where the acceptance counters live

bash serve-with-draft.sh vllm Qwen/Qwen3-8B Qwen/Qwen3-0.6B 8001

The script builds vLLM’s --speculative-config object for you with the draft_model method. The other documented shapes from the same page, as read on 2026-09-09, are {"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5} and {"method": "suffix", "num_speculative_tokens": 8, "suffix_decoding_max_tree_depth": 24}; eagle3 names an EAGLE-3 head. Try the n-gram one on copy-heavy prompts: it needs no second checkpoint at all and it is frequently the largest easy win.

⌚Version note

llama.cpp’s server README, read on 2026-09-09, also documents --spec-type, “comma-separated list of types of speculative decoding to use”, with n-gram modes named ngram-simple, ngram-map-k and ngram-mod alongside draft-model variants. That option is not yet in this course’s captured command reference for llama.cpp v0.4.0 · verified 2026-09-08, so it is named here rather than used in a runnable block. Check --help on your own build; if it is present, an n-gram mode is the no-download equivalent of the draft-model path above and worth measuring in task 7 as a third configuration.

✱Hardware note

Track M can add a third measurement here with MLX, using the two options confirmed in mlx-lm’s argument parser on 2026-09-09:

RunnableTrack M · Apple silicon

the MLX draft path, for comparison against llama.cpp

mlx_lm.generate \
  --model mlx-community/Qwen3-8B-4bit \
  --draft-model mlx-community/Qwen3-0.6B-bf16 \
  --num-draft-tokens 4 \
  --prompt "Rewrite this function to skip missing keys, showing the whole function."

This one is not an OpenAI-compatible server, so it is timed by hand rather than by the script. Record it as a separate note in the notebook rather than in the comparison table.

7. Measure the pair

Both servers are warm. Send them the identical prompt set at temperature zero.

RunnableAll tracks

the comparison this whole lab is for

python3 measure-speculative.py \
  --baseline-url http://127.0.0.1:8080/v1 \
  --speculative-url http://127.0.0.1:8081/v1 \
  --model local-chat \
  --max-tokens 192 \
  --repeats 2 \
  --engine llama.cpp \
  --label qwen3-8b-draft-0.6b \
  --labbook labbook.md

Output — what you should see

  qwen3-8b-draft-0.6b/speculative  (http://127.0.0.1:8081/v1)
    requests    24/24 completed, 0 failed
    output      ... tokens, ... tokens/s
    acceptance statistics from the server's /metrics during this run:
      ... += ...
  comparison
    text        12/12 completions identical, 0 divergent
    speed       ...x the baseline token rate on this prompt set
    appended 2 line(s) to labbook.md

8. Repeat it on the three kinds of work

One number is a curiosity. Three are an answer. Put your copy-heavy prompts, your closed questions and your open-ended prompts into three files and run task 7 three times, changing only --prompts and --label.

RunnableAll tracks

the same comparison on copy-heavy work

python3 measure-speculative.py \
  --baseline-url http://127.0.0.1:8080/v1 \
  --speculative-url http://127.0.0.1:8081/v1 \
  --model local-chat \
  --prompt-set file \
  --prompts prompts-copy-heavy.txt \
  --label draft-copy-heavy \
  --labbook labbook.md

The expected shape of the result is the whole lesson made concrete: a large gain on copy-heavy work, a moderate one on closed questions, and little or none on open-ended writing.

9. Record the run

Every script here has already appended its own JSON line. What is left is the interpretation, which no script can write for you: three or four sentences in the notebook saying which configuration you would actually deploy, on which endpoint, and why.

Prove the deployment path before spending the training budget

Identify the draft method supported by your engine and the artefact its loader expects. A separate draft checkpoint and a hidden-state predictor are not interchangeable. Verify the external recipe’s requirements, pin its commit and perform its smallest export/load check before collecting a large training dataset. The wrapper’s printed recipe is a review point, not proof that an arbitrary recipe works on your hardware.

Save a target-only baseline on each task category. Generate training data from the exact target configuration you intend to serve, preserving prompt provenance and held-out separation. During draft training, record the required hidden-state or token signal and the resulting artefact path.

Run speculative and target-only serving with the same target, representation, context and workload. Inspect acceptance and measured latency together, then repeat under the intended concurrency. Include draft memory in the capacity result. If acceptance improves without a speed gain, report that outcome and inspect draft/verification costs. Preserve the baseline, training recipe, target identity, draft artefact and task-level measurements. A draft is ready for deployment when the complete serving path provides a measured benefit under the target’s quality contract.

Validation

Four checks, in this order. Any one of them failing invalidates the numbers.

The two servers really are the same model. Ask both, and compare.

RunnableAll tracks

same weights, same name, two ports

curl -s http://127.0.0.1:8080/v1/models | head -20
curl -s http://127.0.0.1:8081/v1/models | head -20

The completions are identical. The script’s comparison line should read 12/12 completions identical, 0 divergent at temperature zero. Anything else goes to Troubleshooting before you look at a speed number.

The acceptance statistics moved. The speculative endpoint should have added something to at least one counter mentioning speculation, drafting or acceptance. If the block is empty on a server you know is speculating, the counters are not exposed under those names on your build, and the honest record is “not exposed”, not “zero”.

The speed difference is larger than the noise. Run task 7 twice with no changes and look at how much the ratio moves between runs. A speed-up smaller than that spread is not a result.

Expected outcome

You are done when the lab notebook holds, at minimum:

• one baseline line and one speculative line per kind of work, each carrying the engine, the version, the model, the quantisation and the context length;
• a comparison line per kind of work with the ratio, the acceptance statistics and the text comparison;
• for Tracks S and N, one training line from draftlog.py naming the recipe, the upstream commit and the number of examples;
• for every track, a sentence saying which endpoint you would enable speculation on and which you would not.

Pending validationYour result: speculative decoding on your own model and track

Configuration	Workload	Baseline tokens/s	Speculative tokens/s	Ratio	Acceptance	Identical
Draft model, 0.6B for 8B	Copy-heavy	pending	pending	pending	pending	expected: yes
Draft model, 0.6B for 8B	Closed questions	pending	pending	pending	pending	expected: yes
Draft model, 0.6B for 8B	Open-ended	pending	pending	pending	pending	expected: yes
Trained head (S and N)	Copy-heavy	pending	pending	pending	pending	expected: yes
n-gram, if your build has it	Copy-heavy	pending	pending	pending	pending	expected: yes

your track, as installed · llama.cpp or vLLM the build recorded in task 1 · Qwen3-8B or your Part 13 fine-tune, the same on both servers · 4,096 tokens of context · the day you ran it

A ratio without an acceptance rate beside it cannot be compared with anyone else's, and a row whose completions were not identical is a bug report rather than a result.

Troubleshooting

The speculative server refuses to start, out of memory. Expected, and it is the arithmetic from the first lesson: the draft’s weights and its own key-value cache come out of the same pool as the target’s. Lower CTX, or lower the draft’s quantisation, or drop the target a size. Note which you did.

The draft loads but acceptance is near zero. Almost always a mismatch. Confirm the draft shares the target’s tokeniser, which for the Qwen3 family it does and across families it does not. Confirm both servers apply the same chat template. And confirm you are not measuring open-ended writing and calling it representative.

The completions differ between the two servers. Check three things in order. Is the temperature genuinely zero on both, including any server-side default? Are both applying the same chat template, which a differently-built GGUF may not? Are the differences a single character deep in a long answer, which is the “within hardware numerics” case the first lesson described, or does the text diverge structurally, which is a real bug worth reporting upstream with both completions attached?

The speculative server is slower. Also expected, on the wrong workload or with the wrong chain length. Put your measured acceptance rate into the break-even table from the first lesson before changing anything: if you are below the line, a longer chain makes it worse.

Training exits immediately with an import error. The upstream repositories pin versions the course does not. Read the error, make a fresh virtual environment, and install the repository’s own requirements there. Do not fix it by upgrading the environment your Part 13 work lives in.

Training runs but the head never improves. Check the data first: wc -l on the JSONL, and read three rows. A file of empty completions trains a head that predicts nothing, and the generate stage prints a count of failures for exactly this reason.

hf download fails or is slow. The model reference lists alternative repositories for each model. Any Q4_K_M GGUF of Qwen3-0.6B will do, provided it is the Qwen3 tokeniser.

Cleanup

RunnableAll tracks

stop the servers

pkill -f llama-server || true
pkill -f "vllm serve" || true

The hidden-state directory is the only thing here that is both large and disposable.

⚠Warning

rm -rf on the wrong path is the one irreversible step in this lab. Look at what you are about to delete before you delete it, and keep data/draft-data.jsonl, which is cheap to store and expensive to regenerate.

RunnableAll tracks

check, then remove the feature files

du -sh data/hidden
ls data/hidden | head
rm -rf data/hidden

Keep the trained head, the notebook and the prompt files. The upstream checkouts under $HOME/draft-training can go too, and the script will clone them again if you return.

What you learned

• A draft is exact, and you proved it. The comparison line in your notebook is a byte-for-byte check on greedy completions, and it is the only claim in this part that can be falsified in thirty seconds.
• Acceptance rate is a measurement, not a property. You have it for three kinds of work on one model, and the three numbers differ. That is the answer to “should I turn this on”, and it is workload-shaped rather than model-shaped.
• The break-even is real. If any of your configurations came out below one, you have seen the first lesson’s arithmetic happen rather than read it.
• A draft head is trainable at home, within limits. Tracks S and N have a run log naming a recipe, an upstream commit and an example count. Tracks X and M have a documented reason why not, dated, which is a result rather than a gap.
• Serving decides training. You chose the recipe by asking what your engine could load, not by asking which paper reported the largest number.

Record in the lab notebook: the engine build from task 1; one line per configuration and workload with token rate, acceptance and the text comparison; the training recipe, upstream commit and example count if you trained one; the memory headroom you had left with both servers up; and your one-paragraph answer to which endpoint you would deploy this on. Part 23’s operations work reuses that paragraph, and the capstone asks you to defend it.

Check your understanding

Question 1. Why does this lab insist on two servers running at once rather than restarting one server between the two measurements?

• Because restarting takes too long
• Because a restart changes the page cache, the allocator state and often the context length, so the difference between the two runs would no longer isolate the draft
• Because llama-server cannot be restarted safely
• Because the acceptance metrics reset on restart

Show the answer and why

Answer: Because a restart changes the page cache, the allocator state and often the context length, so the difference between the two runs would no longer isolate the draft

The measurement is a difference, and a difference is only meaningful if one thing changed. Two warm servers on two ports, sent the same prompts, isolate the draft. The lab accepts a sequential comparison on a machine below the memory floor, but asks you to record that you did it, because the evidence is weaker.

Question 2. Your Track X machine cannot run the training tasks. What should go in the lab notebook?

• Nothing; skip the lab
• That neither training repository documented a ROCm path on 2026-09-09, together with the full measurement table from the no-training draft-model path
• An estimate of what the training would have produced
• The Track S numbers, marked as approximate

Show the answer and why

Answer: That neither training repository documented a ROCm path on 2026-09-09, together with the full measurement table from the no-training draft-model path

A dated statement about what a project does not document is a result. An estimate of an unrun experiment is not, and a number copied from different hardware is worse than no number. The no-training path reaches the same measurement table, which is the part of this lab that generalises.

Question 3. Which of these would you check first if the speculative server's completions differ from the baseline's? Select all that apply.

• That the temperature is genuinely zero on both servers, including any server-side default
• That both servers apply the same chat template
• That the draft and target share a tokeniser
• That the draft model is large enough to be accurate

Show the answer and why

Answer: That the temperature is genuinely zero on both servers, including any server-side default, That both servers apply the same chat template, That the draft and target share a tokeniser

All three of the first options change what is being compared rather than how fast it runs. The draft's size does not: a poor draft lowers the acceptance rate and therefore the speed, but verification means it cannot change the text. If the text changed, look for a configuration difference, not a quality one.

Question 4. True or false: the acceptance statistics block printed as empty, so the acceptance rate on that run was zero.

• True
• False

Show the answer and why

Answer: False

False. An empty block means the server published no counter whose name mentioned speculation, drafting or acceptance, which on some builds is simply the case. The honest notebook entry is "not exposed on this build", and the evidence for speculation working is then the token-rate difference and the identical completions. Recording a zero you did not measure is how a benchmark becomes fiction.

Check my answersClear