Level 5 · Agentic EngineerProjectCapstone · page 7 of 790 minSXMN 8 GB

90Minutes

1Tools

2Sources

All fourTracks

Tools used on this page1

hf

Capstone 6: The Report

Validated on: written from the documentation cited above and from the whole course; not yet validated on hardware on any track. This deliverable produces a document, and the material it is made of is your own notebook.

Objective

By the end of this project you will have a report that lets somebody who was not there see what you built, judge every number in it, reproduce the parts they care about, and find out plainly what you did not measure.

It is the deliverable the rubric weighs most heavily, and not because writing is the point. It is because a platform whose owner cannot say why it is configured as it is, what it costs, or whether last month’s fine-tune helped is the failure this course was written to prevent. The report is where that question gets answered or where it becomes clear that it cannot be.

Architecture

Six steps, and only two of them are writing.

From notebook to defended document

1. Collectthe script reads labbook.md, groups the records by the lab that wrote them, and prints the tables in the course format
2. Fill the gapsthe script names the context keys the notebook never recorded; each one is a number a reader could not judge
3. Draw what runsthe architecture as it is today, not as the plan imagined it; every box exists and nothing running is missing
4. Write the proseevery performance sentence names a measurement label; the rest are marked as arithmetic or as a vendor figure
5. Apply the rubrictwice: once now, once a week later reading it as a stranger
6. Write section 9what is not true here, one line per "no" answer; written last and read first

Step 2 is the one that changes the document most. A context key the notebook never recorded is not a formatting problem; it is a number nobody, including you, can now judge.

Requirements

Every track needs the five deliverables before this one, the lab notebook from Part 1 with whatever the labs appended to it, the cost method from Part 23’s capacity lesson, a power meter at the wall if you want the cost section to be a measurement rather than an estimate, and about ninety minutes. No model runs and nothing is downloaded.

Track S — NVIDIA DGX Spark

All of it applies. Your cost section will be dominated by the machine’s purchase price rather than by its electricity, so the amortisation assumption is the term doing the work and the report should say what the figure becomes if you halve or double it.

Track X — AMD Ryzen AI Max+ 395

All of it applies. This track usually produces the most favourable cost per million tokens of the four for large-model work, which makes it the one where stating the utilisation assumption matters most: a low cost per token computed over a machine that is idle for most of the day is a statement about the arithmetic rather than about the machine.

Track M — Apple silicon

All of it applies, with one honest complication for the cost section: the machine is usually somebody’s working computer, so the capital cost is not attributable to the inference service alone. Say what fraction you attributed and why, or report only the marginal energy cost and say that you did.

Track N — NVIDIA desktop or laptop

All of it applies. On this track the idle draw of a desktop card is a large part of the total if the machine stays powered on, so the report should carry both the total and the marginal figures, which is the distinction Part 23’s cost model makes explicit.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/capstone"
cd "$LAB_DIR"
pwd
test -f "collect-capstone-evidence.py"

Expected result: pwd ends in capstone and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Collect the evidence from the notebook

The labs of this course append one JSON object per line to the notebook. The collector reads them, groups them by the lab that wrote them, and prints the tables in the course’s own format so that no number arrives in the report without the context that makes it judgeable.

RunnableAll tracks

#!/usr/bin/env python3
"""Assemble the capstone report's evidence tables from the lab notebook.

Purpose: read labbook.md, find every machine-readable record the course's labs
    appended to it, group them by the lab that wrote them, and print the tables the
    capstone report needs - as Markdown for the report itself, or as the course's
    <Benchmark> shape so that a number and the context that makes it meaningful
    cannot be separated. The script never invents a field: it prints the fields it
    found, names the records whose shape it does not recognise, and lists the
    context keys you still have to fill in by hand.
Platform: all (pure Python; no third-party package, no network, no model)
Minimum memory: 8 GB (it reads a text file; the floor is the machine you have)
Assumes: Python 3.9 or newer, and the lab notebook started in Part 1, in which the
    labs of Parts 6, 9, 11, 16, 18, 22, 23 and 26 have appended one JSON object per
    line. Lines that are not JSON objects are your own prose and are skipped.

Usage: python3 collect-capstone-evidence.py --labbook labbook.md
       python3 collect-capstone-evidence.py --labbook labbook.md --format mdx
       python3 collect-capstone-evidence.py --labbook labbook.md --group part-09
       python3 collect-capstone-evidence.py --labbook labbook.md --list
       python3 collect-capstone-evidence.py --labbook labbook.md --format json > evidence.json

Endpoints are dropped from the output by default, because a report is a document you
give to somebody else and a base URL is an address on your network. Pass
--keep-endpoints if you are writing the report only for yourself.
"""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path
from typing import Any, Iterable

# ---------------------------------------------------------------------------
# What the labs of this course write.
#
# Each entry is a group key prefix and the columns worth putting in a report,
# in the order a reader wants them. A field named here that is absent from the
# records is simply not printed: the preference is an ordering, never a promise
# that the field exists. A record whose group matches nothing below still gets
# a table; its columns are then every scalar field the records actually carry,
# in the order they first appeared.
# ---------------------------------------------------------------------------
KNOWN_SHAPES: dict[str, dict[str, Any]] = {
    "part-06/": {
        "what": "llama.cpp benchmark rows (llama-bench)",
        "columns": [
            "model", "quant", "test", "backend", "n_gpu_layers", "file_gb",
            "tokens_per_s", "tokens_per_s_stddev", "reps",
        ],
        "context": {
            "hardware": ["host"], "engine": ["engine"], "version": ["build"],
            "model": ["model"], "quant": ["quant"], "date": ["measured_on"],
        },
    },
    "part-09/": {
        "what": "load-test rows, one per concurrency level",
        "columns": [
            "label", "concurrency", "requests", "completed", "failed",
            "output_tokens_per_s", "requests_per_s",
            "ttft_s.p50", "ttft_s.p90", "tpot_s.p50", "tpot_s.p90",
        ],
        "context": {
            "engine": ["engine"], "version": ["engine_version"], "model": ["model"],
            "date": ["recorded_at"],
        },
    },
    "part-11/": {
        "what": "training run records",
        "columns": [
            "run_id", "model", "dataset", "seed", "config_commit", "date",
        ],
        "context": {
            "model": ["model"], "version": ["config_commit"], "date": ["date"],
        },
    },
    "part-18/": {
        "what": "cluster topology and link records",
        "columns": [
            "machine", "roles", "cluster_interface", "intended_mtu",
            "peers", "date",
        ],
        "context": {"date": ["date"], "hardware": ["machine"]},
    },
    "part-26/": {
        "what": "agent evaluation runs",
        "columns": [
            "run_id", "scaffold", "task_set", "model", "router_model", "repeats",
            "summary.runs", "summary.passed", "summary.success_rate",
            "summary.mean_steps", "summary.mean_tokens", "summary.mean_seconds",
        ],
        "context": {
            "engine": ["engine"], "version": ["engine_version"], "model": ["model"],
            "quant": ["quant"], "date": ["date"], "hardware": ["host"],
        },
    },
}

# The keys <Benchmark> requires. The component refuses to render without them,
# which is ADR-004 enforced at build time rather than by good intentions.
BENCHMARK_CONTEXT = ("hardware", "engine", "version", "model", "quant", "contextLength", "date")

# Field names that carry an address on your network rather than a measurement.
ENDPOINT_FIELDS = {"base_url", "url", "endpoint", "host_url", "agent_entry_point"}

# Generic fallbacks tried for every group, after any shape-specific mapping.
GENERIC_CONTEXT: dict[str, list[str]] = {
    "hardware": ["host", "machine", "hardware.device_name", "hardware.machine"],
    "os": ["os", "hardware.os"],
    "engine": ["engine", "scaffold"],
    "version": ["engine_version", "build", "version", "config_commit"],
    "model": ["model", "model_path"],
    "quant": ["quant", "quantisation", "quantization"],
    "contextLength": ["context_length", "ctx_size", "n_ctx", "contextLength"],
    "date": ["date", "measured_on", "recorded_at", "recorded"],
}

FILL_IN = "FILL-IN"


# ---------------------------------------------------------------------------
# Reading
# ---------------------------------------------------------------------------
def read_records(path: Path) -> tuple[list[dict], int, int]:
    """Every JSON object on its own line, plus the counts of what was skipped.

    The notebook is a Markdown file that labs append JSON lines to, so most of it
    is prose. A line that starts with "{" and does not parse is reported rather
    than silently dropped: a truncated record is worth knowing about.
    """
    records: list[dict] = []
    prose = 0
    broken = 0
    with path.open("r", encoding="utf-8") as handle:
        for line in handle:
            stripped = line.strip()
            if not stripped.startswith("{"):
                if stripped:
                    prose += 1
                continue
            try:
                value = json.loads(stripped)
            except json.JSONDecodeError:
                broken += 1
                continue
            if isinstance(value, dict):
                records.append(value)
            else:
                broken += 1
    return records, prose, broken


def group_key(record: dict) -> str:
    """Group by the lab that wrote the record, then by the kind of record it is.

    Part 18 writes topology records and link records under one lab name, so the
    `record` field splits them; every other lab in the course writes one shape per
    `lab` value. A record with neither field is grouped under a name that says so,
    because inventing a group for it would hide it.
    """
    lab = record.get("lab")
    kind = record.get("record")
    if lab and kind:
        return f"{lab} [{kind}]"
    if lab:
        return str(lab)
    if kind:
        return f"(no lab field) [{kind}]"
    return "(unidentified records)"


def shape_for(key: str) -> dict[str, Any] | None:
    for prefix, shape in KNOWN_SHAPES.items():
        if key.startswith(prefix):
            return shape
    return None


# ---------------------------------------------------------------------------
# Flattening
# ---------------------------------------------------------------------------
def flatten(record: dict, prefix: str = "") -> dict[str, Any]:
    """One level of dotted keys per nested object; lists are summarised, not expanded.

    load-test.py nests its percentiles under ttft_s and tpot_s, and agent-eval.py
    nests its aggregates under summary, so a report table needs ttft_s.p90 to be a
    column. A list becomes its length in a "n item(s)" string unless it is short
    and entirely scalar, in which case it is joined: roles and peers read better
    that way and neither is a measurement.
    """
    out: dict[str, Any] = {}
    for name, value in record.items():
        full = f"{prefix}{name}"
        if isinstance(value, dict):
            out.update(flatten(value, prefix=f"{full}."))
        elif isinstance(value, list):
            scalars = [v for v in value if isinstance(v, (str, int, float, bool))]
            if value and len(scalars) == len(value) and len(value) <= 6:
                out[full] = ", ".join(str(v) for v in value)
            else:
                out[full] = f"{len(value)} item(s)"
        else:
            out[full] = value
    return out


def columns_for(rows: list[dict[str, Any]], shape: dict[str, Any] | None,
                keep_endpoints: bool, all_fields: bool) -> list[str]:
    """The columns a report wants: the shape's preference where one is known.

    A load-test record carries two dozen fields and a report table with two dozen
    columns is unreadable, so a recognised shape prints the fields chosen for a
    report and --all-fields prints the rest. An unrecognised shape always prints
    everything, because the alternative would be to decide on your behalf which of
    your own fields matter.
    """
    present: list[str] = []
    for row in rows:
        for name in row:
            if name not in present:
                present.append(name)
    if not keep_endpoints:
        present = [c for c in present if c.split(".")[0] not in ENDPOINT_FIELDS]
    preferred = [c for c in (shape or {}).get("columns", []) if c in present]
    if preferred and not all_fields:
        return preferred
    rest = [c for c in present if c not in preferred and c not in ("lab", "record")]
    return preferred + rest


def cell(value: Any) -> str:
    if value is None:
        return "not recorded"
    if isinstance(value, bool):
        return "yes" if value else "no"
    if isinstance(value, float):
        return f"{value:.4g}"
    text = str(value)
    return text.replace("|", "\\|")


# ---------------------------------------------------------------------------
# Context for <Benchmark>
# ---------------------------------------------------------------------------
def first_present(rows: list[dict[str, Any]], names: Iterable[str]) -> Any:
    for name in names:
        for row in rows:
            if row.get(name) not in (None, ""):
                return row[name]
    return None


def context_for(rows: list[dict[str, Any]], shape: dict[str, Any] | None) -> tuple[dict[str, Any], list[str]]:
    """The <Benchmark> context this group can fill, and the keys it cannot.

    Nothing is guessed. A key that no record carries comes back as FILL-IN and is
    also returned in the second value, so the caller can say out loud which parts
    of the context the notebook never recorded. That list is the useful output of
    this function: it is the set of things a future reader of your report would
    have had to take on trust.
    """
    mapping = dict(GENERIC_CONTEXT)
    for key, names in (shape or {}).get("context", {}).items():
        mapping[key] = list(names) + [n for n in GENERIC_CONTEXT.get(key, []) if n not in names]
    context: dict[str, Any] = {}
    missing: list[str] = []
    for key in BENCHMARK_CONTEXT + ("os",):
        value = first_present(rows, mapping.get(key, [key]))
        if value in (None, ""):
            if key in BENCHMARK_CONTEXT:
                context[key] = FILL_IN
                missing.append(key)
            continue
        context[key] = value
    return context, missing


# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------
def print_markdown(key: str, rows: list[dict[str, Any]], shape: dict[str, Any] | None,
                   keep_endpoints: bool, all_fields: bool) -> list[str]:
    cols = columns_for(rows, shape, keep_endpoints, all_fields)
    what = (shape or {}).get("what", "shape not recognised; every field found is shown")
    print(f"\n### {key}\n")
    print(f"{len(rows)} record(s). {what}.\n")
    if not cols:
        print("No fields to show: the records in this group are empty.\n")
        return []
    print("| " + " | ".join(cols) + " |")
    print("| " + " | ".join("---" for _ in cols) + " |")
    for row in rows:
        print("| " + " | ".join(cell(row.get(c)) for c in cols) + " |")
    context, missing = context_for(rows, shape)
    known = [f"{k}: {context[k]}" for k in context if context[k] != FILL_IN]
    print("\n*Context found in the records:* " + ("; ".join(known) if known else "none"))
    if missing:
        print(f"\n*Context the notebook never recorded, to be supplied by hand:* {', '.join(missing)}")
    return missing


def print_mdx(key: str, rows: list[dict[str, Any]], shape: dict[str, Any] | None,
              keep_endpoints: bool, all_fields: bool) -> list[str]:
    """A <Benchmark> block for the report, with the context the records carry.

    The component refuses to render an incomplete context, so a FILL-IN left in
    the output breaks the build rather than shipping a number with nothing
    attached to it. That is the intended behaviour and it is why the marker is
    loud.
    """
    cols = columns_for(rows, shape, keep_endpoints, all_fields)
    if not cols:
        print(f"\n<!-- {key}: no fields to show -->")
        return []
    context, missing = context_for(rows, shape)
    context.setdefault("contextLength", FILL_IN)
    print(f"\n<Benchmark")
    print(f'  title="{key}"')
    print("  columns={" + json.dumps(cols) + "}")
    print("  rows={[")
    for row in rows:
        print("    " + json.dumps([row.get(c) for c in cols]) + ",")
    print("  ]}")
    print("  context={{")
    for name, value in context.items():
        if name == "contextLength" and isinstance(value, int):
            print(f"    contextLength: {value},")
        else:
            print(f"    {name}: {json.dumps(str(value))},")
    print("  }}")
    print('  status="measured" />')
    if missing:
        print(f"<!-- Replace every {FILL_IN} above: {', '.join(missing)} -->")
    return missing


def main(argv: list[str]) -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--labbook", default="labbook.md",
                        help="the notebook the labs have been appending to")
    parser.add_argument("--format", choices=["markdown", "mdx", "json"], default="markdown",
                        help="markdown for the report, mdx for <Benchmark> blocks, json for a pipe")
    parser.add_argument("--group", action="append", default=[],
                        help="only groups whose key contains this text; repeatable")
    parser.add_argument("--list", action="store_true",
                        help="list the groups and their record counts, then stop")
    parser.add_argument("--keep-endpoints", action="store_true",
                        help="keep base URLs and entry points in the tables")
    parser.add_argument("--all-fields", action="store_true",
                        help="every field found, not just the ones a report wants")
    args = parser.parse_args(argv)

    path = Path(args.labbook)
    if not path.is_file():
        print(f"{path}: not found. This is Part 1's lab notebook, the one every lab "
              f"appends to; the capstone is written from it.", file=sys.stderr)
        return 2

    records, prose, broken = read_records(path)
    if not records:
        print(f"{path}: no JSON records found. The labs append one JSON object per line; "
              f"if you kept your results as prose, the report tables have to be typed by hand.",
              file=sys.stderr)
        return 1

    groups: dict[str, list[dict[str, Any]]] = {}
    for record in records:
        groups.setdefault(group_key(record), []).append(flatten(record))
    if args.group:
        groups = {k: v for k, v in groups.items() if any(g in k for g in args.group)}
        if not groups:
            print(f"No group matched {args.group}. Run with --list to see what is there.",
                  file=sys.stderr)
            return 1

    if args.format == "json":
        json.dump({k: v for k, v in sorted(groups.items())}, sys.stdout, indent=2, sort_keys=True)
        sys.stdout.write("\n")
        return 0

    print(f"==> {path}")
    print(f"    {len(records)} record(s) in {len(groups)} group(s); "
          f"{prose} line(s) of prose skipped; {broken} line(s) did not parse.")
    unknown = [k for k in groups if shape_for(k) is None]
    if unknown:
        print(f"    {len(unknown)} group(s) have a shape this script does not know; "
              f"their tables show every field found.")

    if args.list:
        for key in sorted(groups):
            shape = shape_for(key)
            label = (shape or {}).get("what", "shape not recognised")
            print(f"    {len(groups[key]):>4}  {key}  ({label})")
        return 0

    missing_anywhere: dict[str, list[str]] = {}
    emit = print_mdx if args.format == "mdx" else print_markdown
    for key in sorted(groups):
        missing = emit(key, groups[key], shape_for(key), args.keep_endpoints, args.all_fields)
        if missing:
            missing_anywhere[key] = missing

    print("\n---\n")
    if missing_anywhere:
        print("Context missing from the notebook, by group.")
        print("Each one is a number in your report that a reader could not judge.")
        print("Fill it in from the notebook's machine section, or write \"not recorded\"")
        print("in the table and say so in the report's final section.\n")
        for key, missing in missing_anywhere.items():
            print(f"  {key}: {', '.join(missing)}")
        return 1
    print("Every group carried a complete context. Nothing to fill in by hand.")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Download collect-capstone-evidence.py418 lines

Start by seeing what is there.

RunnableAll tracks

python3 collect-capstone-evidence.py --labbook labbook.md --list

Output — what you should see

==> labbook.md
    41 record(s) in 7 group(s); 215 line(s) of prose skipped; 0 line(s) did not parse.
    1 group(s) have a shape this script does not know; their tables show every field found.
       3  my-own/prompt-experiments  (shape not recognised)
      12  part-06/lab-benchmark-the-reference-models  (llama.cpp benchmark rows (llama-bench))
       8  part-09/lab-serve-twenty-concurrent-users  (load-test rows, one per concurrency level)
       4  part-11/lab-your-first-training-run  (training run records)
       6  part-18/build-and-measure-your-cluster-network [link]  (cluster topology and link records)
       2  part-18/build-and-measure-your-cluster-network [topology]  (cluster topology and link records)
       6  part-26/evaluating-agents  (agent evaluation runs)

The group at the top is a shape the script has never seen, written by a script of your own. Nothing is dropped: it is grouped under its own name and its table shows every field the records carry.

Then print the tables for the groups you want in the report.

RunnableAll tracks

python3 collect-capstone-evidence.py \
  --labbook labbook.md \
  --group part-06 \
  --group part-09 \
  --format markdown

The script never invents a field. Where it recognises a shape it prints the columns a report wants and --all-fields prints the rest; where it does not, it prints every field the records actually carry and says so. Records with a shape it has never seen, including any your own scripts wrote, are grouped by their own lab name and printed in full rather than dropped.

2. Fill in what the notebook never recorded

The script’s most useful output is its last section: the context keys it could not fill from any record in a group.

Output — what you should see

---

Context missing from the notebook, by group.
Each one is a number in your report that a reader could not judge.
Fill it in from the notebook's machine section, or write "not recorded"
in the table and say so in the report's final section.

  part-06/lab-benchmark-the-reference-models: contextLength
  part-09/lab-serve-twenty-concurrent-users: hardware, quant, contextLength

Each line is a number in your report that a reader would have to take on trust. Two honest responses and one dishonest one. You can fill it from the notebook’s machine section, where the hardware and operating system were written down in Part 5. You can write “not recorded” in the table and mention it in the final section. What you cannot do is reconstruct it from memory and present it as recorded, because the whole apparatus of this course rests on the difference between those two things.

If you want the tables as the course’s own component rather than as Markdown, the script emits that too, with a loud marker where the context has a hole in it.

RunnableAll tracks

python3 collect-capstone-evidence.py \
  --labbook labbook.md \
  --group part-09 \
  --format mdx

The marker is deliberate: the component refuses to render without a complete context, so a number whose context you never recorded cannot quietly reach a reader. The script exits with a failure status while any group still has a hole in it, for the same reason that Part 22’s design checker and Part 26’s evaluation harness do: a tool that reports a problem and then says everything is fine is a tool people learn to ignore.

3. Draw the architecture as it actually is

Section 2 of the template. Machines with their roles, every link with what it carries, and the request path from a client to an answer in numbered steps.

Two checks, and they take five minutes together. Every box in the diagram exists on a machine you own. And nothing that is running is missing from the diagram, which usually catches one forgotten thing: a metrics exporter, a second engine left up after Capstone 3, or a container somebody started in Part 7 and never stopped.

Then the model estate table: every gateway name, the model behind it today, the quantisation, the configured context, the licence, and whether the provenance is recorded. That table is what makes the platform maintainable by anybody, including you in a year.

4. Write the measurement sections, with their context adjacent

Section 3 of the template, filled from the collector’s output.

The rule is the one the course has applied to itself throughout: no measured figure in a sentence. Every number sits in a table whose caption or context line names the machine, the operating system, the engine and version, the model, the quantisation, the context length and the date. Readers copy numbers and not footnotes, which is why the context has to be adjacent and unavoidable rather than at the bottom of the page.

Four tables at minimum: single-model throughput, the service under load, the baseline against your chosen deployment, and the before-and-after for your improved model. A fifth if you have agents, and you probably do.

If you ran a standard benchmark suite, carry with it what makes it comparable: the exact task and metric names, the shot count, the chat template setting, the sampling settings, the number of items scored, how many runs and their spread, and the versions. The harness project’s own README makes the point that publicly available prompts are what make results comparable between papers; the same logic applies between your own runs a month apart.

5. The cost model, with its inputs shown

Section 4, and it is the section most likely to be quoted at you, so it is the one to make hardest to misread.

Part 23’s model takes measured wall power at idle and under load, the price of electricity, the aggregate output rate, the purchase price with an amortisation period, and two utilisation figures: the hours a year the platform is actually generating, and the hours a year it is powered on. From those it produces an energy cost per million tokens, a marginal energy cost, a capital cost, the idle energy that the working hours have to carry, and a total.

Report the inputs before the outputs, each with how you obtained it, because the utilisation assumption is doing most of the work in the total and a reader has to be able to see it. Then add the sentence that makes the section honest: what the total becomes at half your assumed utilisation and at double it.

If you did not measure power at the wall, say so and report only what you can compute. An estimated cost model labelled as estimated is useful; one presented as measured is not.

6. What did not work, and what changed in your understanding

Sections 5 and 6, and they are the two sections another person will read first.

What did not work takes at least three entries, each with what you expected, what happened, what the evidence was, and what you did instead. It is not a confession. It is the only part of the report that cannot be reconstructed from the tables, and it is what makes the rest of the document credible: a report with no failures in it is a report that has been edited rather than written.

What changed in your understanding wants one or two paragraphs in the shape “I expected X” followed by a measurement label. The common candidates after this course: what actually makes generation fast, what a second machine buys, how much a quantisation costs on your own tasks rather than on somebody’s benchmark, whether the fine-tune helped, and how far a small model can drive an agent before it stops being able to.

7. The maintenance plan, and the reproduction section

Sections 7 and 8. Both are written for somebody who is not you, including you in a year.

The maintenance plan is dated tasks with the commands and the location of the state: restore a backup into a scratch location and compare, re-verify the model file hashes, upgrade one engine at a time with a rollback ready, re-run the evaluation set after any model change, re-check the sandbox boundary test, review which keys exist and which have expired, and re-measure the baseline after any upgrade. Then the sentence that makes the plan real: what breaks first if nobody does any of it.

The reproduction section is versions and identifiers: operating systems, drivers, engines, container image digests rather than moving tags, and model repositories with their revisions. The Hub’s own tooling records and verifies what is in the local cache, which is what makes “the same model” a checkable statement rather than a name.

Fragment — not complete on its own

hf download <publisher>/<repository> \
  --revision <the exact revision your report names> \
  --include "<the file you served>" \
  --local-dir "$MODELS/<the directory in your estate manifest>"

8. Apply the rubric, twice

Work through the rubric from the overview lesson, counting the “no” answers rather than scoring yourself. Then leave it a week, come back, and read your own report as a stranger would. The second pass finds a different set of problems from the first, mostly sentences that made sense when you wrote them because you remembered what was behind them.

Three questions to ask of every paragraph, borrowed from Part 22’s project because they transfer exactly: how do you know, what happens when this breaks, and why not one machine.

9. Write the section on what is not true here

Section 9, written last. Every “no” from the rubric becomes one sentence. What you did not measure, what is arithmetic rather than measurement, what you took from a vendor page without verifying, where two configurations were not comparable, where a single run cannot separate a difference from a wobble, and which tracks you could not test at all.

This is the section a reader will judge the rest of the document on, and it is the section the course has been building towards since Part 1. A document that says what it does not know is a document whose other claims are worth reading.

Fragment — not complete on its own

# <Your platform's name> — capstone report

<!--
Purpose: the deliverable of Capstone 6, and the thing the whole course was building
         towards. One document in which somebody who was not there can see what you built,
         judge every number in it, reproduce the parts they care about, and find out what
         you did not measure.
Platform: all
Minimum memory: not applicable; this is a document
Assumes: the five deliverables before it, labbook.md, and the tables that
         collect-capstone-evidence.py assembled from it. Every table below is filled from
         that script's output or typed from a notebook record, never from memory.

Two rules for the whole document:
  1. Every number sits in a table that carries the machine, the operating system, the
     engine and version, the model, the quantisation, the context length and the date.
     No figure appears in a sentence on its own.
  2. Every sentence claiming that something is faster, better or sufficient names a
     measurement label that exists in the notebook. A sentence that cannot is marked as
     arithmetic, or as a vendor figure with its source, or it is deleted.

Delete every comment block, including this one. Fill in every angle-bracket field; write
"not recorded" where you cannot, because the gap is information.
No key, address, hostname or personal path belongs in a document you will show anybody.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Tracks:** <S, X, M, N> ·
**Period this covers:** <YYYY-MM-DD to YYYY-MM-DD>

---

## 1. What this platform is and who uses it

<Three or four sentences. What it serves, to whom, how often, and what it replaced. Then
the requirement from Capstone 1, restated unchanged: concurrent users, prompt and answer
lengths, and the latency budget.>

**Did it meet the requirement?** <yes, no, or partly, with the measurement label. This is
the only conclusion the document has, and it belongs at the top.>

## 2. The architecture

### 2.1 The machines and the links

<A diagram or a table of machines with their roles, and every link with what it carries.
Every box exists and nothing running is missing from it.>

| Machine | Track | Chip | Memory | Roles | What runs on it |
| --- | --- | --- | --- | --- | --- |
| <name> | | | | | |

| From | To | Class | Measured throughput | Carries |
| --- | --- | --- | --- | --- |
| <machine> | <machine> | | | <per-token / per-request / load-time / client> |

### 2.2 The request path

<From a client to an answer, in numbered steps: what authenticates it, what routes it,
which engine serves it, what is cached, what is logged. Six to ten steps.>

### 2.3 The model estate

| Alias | Model | Quantisation | Context configured | Where it runs | Licence | Provenance recorded |
| --- | --- | --- | --- | --- | --- | --- |
| <name clients use> | <exact id> | | | | | <hash beside the file> |

## 3. The measurements

<Assembled by collect-capstone-evidence.py from labbook.md. Every table below carries its
own context; a table whose context has a gap says so in the row beneath it rather than
leaving the reader to assume.>

### 3.1 Single-model throughput

| Model | Quantisation | Test | Backend | Tokens per second | Spread | File size |
| --- | --- | --- | --- | --- | --- | --- |
| | | | | | | |

*Context:* machine <name>; operating system <version>; engine <name and version>; context
length <n>; date <YYYY-MM-DD>.

### 3.2 The service under load

| Configuration | Concurrency | Completed | Failed | Output tokens per second | Requests per second | Time to first token, median | Time per output token, median |
| --- | --- | --- | --- | --- | --- | --- | --- |
| | | | | | | | |

*Context:* <as above> · *Load generator and prompt set:* <name and settings>

### 3.3 Baseline against the chosen deployment

| Configuration | Concurrency | Output tokens per second | Time to first token, median | What differed besides the architecture |
| --- | --- | --- | --- | --- |
| Baseline, one machine | | | | reference |
| <the chosen architecture> | | | | <list, or nothing> |

**What the architecture bought:** <capacity, speed, both or neither, in one sentence with
the two labels named.>

### 3.4 The improved model

| Model | Quantisation | Metric | Runs | Mean | Range | Noise floor |
| --- | --- | --- | --- | --- | --- | --- |
| Base | | | | | | |
| Improved | | | | | | |

**Held or not:** <the claim from Capstone 4, and whether the difference exceeded the noise
floor.>

### 3.5 The agents

| Configuration | Repeats | Success rate | Mean steps | Mean tokens | Spread |
| --- | --- | --- | --- | --- | --- |
| | | | | | |

## 4. Cost and power

| Input | Value | How obtained |
| --- | --- | --- |
| Idle power, whole platform | <watts at the wall> | <meter, and where it was placed> |
| Power under load | <watts at the wall> | <same meter, during which measurement> |
| Price of electricity | <per unit> | <your bill, and the date> |
| Aggregate output rate | <tokens per second across the platform> | <measurement label> |
| Generating hours per year | <n> | <assumption, and what it is based on> |
| Powered-on hours per year | <n> | <assumption> |
| Purchase price of the machines | <total> | <what you actually paid> |
| Amortisation period | <years> | <assumption> |

| Figure | Value |
| --- | --- |
| Energy cost per million tokens | |
| Marginal energy cost per million tokens | |
| Capital cost per million tokens | |
| Idle energy carried by the work | |
| **Total cost per million tokens** | |

**What this comparison does and does not say:** <the utilisation assumption is doing most
of the work in that total, and a machine that is powered on all day for an hour of
generation carries the rest of the day on that hour. Say what your assumption was and what
the figure becomes at half and at double it.>

## 5. What did not work

<At least three, each with what you tried, what the evidence was, and what you did
instead. This is not a confession section; it is the part of the report another person
will find most useful, because it is the only part they cannot reconstruct from the
tables.>

**1. <what>** — <what you expected, what happened, what the evidence was, what you did.>

**2. <what>** — <as above.>

**3. <what>** — <as above.>

## 6. What changed in your understanding

<One or two paragraphs. Something you believed early in the course and no longer believe,
with the measurement that changed it. The most common candidates: what makes decode fast,
what a second machine buys, how much a quantisation actually costs on your own tasks,
whether a fine-tune helped, how far a small model can drive an agent.>

<A sentence that begins "I expected" and ends with a measurement label is the shape this
section wants.>

## 7. The maintenance plan

| Task | How often | What it involves | Where the state lives | Last done |
| --- | --- | --- | --- | --- |
| Restore a backup into a scratch location and compare | <quarterly> | | | <YYYY-MM-DD> |
| Re-verify model file hashes | | | | |
| Update engines, one at a time, with a rollback ready | | | | |
| Re-run the evaluation set after any model change | | | | |
| Re-check the sandbox boundary test | | | | |
| Review keys: what exists, what is unused, what expired | | | | |
| Re-measure the baseline | <after any upgrade> | | | |

**What breaks first if nobody does any of this:** <one sentence, and it is usually the
certificate or the disk.>

## 8. Reproducing this

| Field | Value |
| --- | --- |
| Operating systems and versions | |
| Driver, toolkit or framework versions | |
| Engine versions, exact | |
| Container image digests, not tags | |
| Model repositories and revisions | |
| Configuration files | <where they are, and what has to be replaced> |
| Scripts | <the ones you actually ran> |
| What cannot be reproduced without your hardware | |

## 9. What is not true here

<What you did not measure. What is arithmetic rather than measurement. What you took from
a vendor page or a project's documentation without verifying on your own machine. Where
two configurations were not strictly comparable. Where a single run cannot distinguish a
difference from a wobble. Which tracks you could not test.>

<Write this last, write it honestly, and make it long enough to be useful. Every "no"
answer from the capstone rubric belongs here as one sentence. A document that says what it
does not know is a document whose other claims are worth reading, and that, rather than
any number in section 3, is what the course has been teaching.>

Download report-template.md203 lines

RunnableAll tracks

cp report-template.md capstone-report.md

Perform an evidence audit before calling the course complete

Create one row for each requirement with its artefact, command/configuration, result file and verdict. Open every referenced local file and check that its identity matches the row. Missing data must be labelled not run or unavailable; do not fill empty tables with estimates that look measured.

Recompute at least one headline result from the raw task-level records. Confirm the denominator includes the failures defined by the protocol and that comparisons use the same settings. Check units, model revisions, dates and whether a result is measured, estimated or publisher-reported. Keep secrets out of the shared report and evidence bundle.

Read the reproduction instructions from a fresh shell with only the stated inputs available. Verify paths, environment activation, model aliases and service order. Then review the failure and rollback sections against the drills you performed. The final report should connect each conclusion to evidence, explain rejected approaches and identify the remaining limits of hardware coverage or task generality. Applying the rubric means assessing those records, not awarding completion for page length or the presence of screenshots. Preserve the original labbook and raw artefacts alongside the report so later course or model updates can be compared with this completed baseline.

Validation

capstone-report.md has all nine sections filled in with no angle-bracket fields left.
Every measured figure is in a table, and every table names the machine, the operating system, the engine and version, the model, the quantisation, the context length and the date, or says “not recorded” for the ones the notebook never carried.
No measured figure appears in a sentence of prose.
Every performance claim in the prose names a measurement label that exists in the notebook, or is marked as arithmetic, or is attributed to a vendor page with its source.
The architecture section contains no box that does not exist, and nothing running is absent from it.
The cost section lists its inputs with how each was obtained, gives the marginal figure as well as the total, and says what the total becomes at half and double the utilisation assumption.
Section 5 has at least three entries.
Section 7 names what breaks first if the plan is ignored.
Section 8 names container digests rather than tags, and model revisions rather than only repositories.
Section 9 has one line for every “no” answer from the rubric, and it is not empty.
The rubric has been applied twice, on two different dates.

Expected outcome

A document of a few pages that stands on its own. The shape below is the evidence the report has to carry; the values come from your notebook, and a row you cannot fill becomes a line in section 9 rather than a gap.

Pending validationWhat the report has to carry — the shape, to be filled in with your own

Section	The evidence	Where it comes from
Requirement met or not	one sentence with a measurement label	Capstone 1 section 1, against Capstone 2 and 3
Single-model throughput	a table with its full context	Part 6's benchmark records
Service under load	a table per configuration, same prompt set	Part 9's load-test records
Baseline against deployment	two tables and a list of everything else that differed	Capstone 3
Improved model	before and after, three runs each, with the noise floor	Capstone 4
Agents	success rate, steps and tokens with the spread over repeats	Capstone 5
Cost per million tokens	the inputs, then the total and the marginal figure	Part 23's model and a power meter
What did not work	three entries with evidence	all five deliverables
Maintenance	dated tasks and what breaks first without them	Capstone 2 and Part 23
What is not true here	one line per rubric "no"	the rubric, applied twice

every machine in your platform, named in the report · every engine the report quotes a number from as recorded in your notebook · every model behind a gateway name, as each table states · 8,192 tokens of context · 2026-09-09

A pending row here means a measurement you still have time to take. That is the reason this deliverable is written while the platform is still running.

Two reports count as finished. One describes a platform that met its requirement, with the measurements to show it and a list of the things that failed on the way. The other describes a platform that did not meet its requirement, says by how much, and names what would have to change. The second is not a lesser document, and on a house network it is at least as common.

Troubleshooting

The collector finds no records. The labs append JSON lines to the notebook, and if you kept your results as prose the tables have to be typed by hand. That is a legitimate route and it is slower; note in section 9 which tables were transcribed rather than collected, because transcription introduces errors that collection does not.

A group’s shape is not recognised. Expected, and the script says so and prints every field it found. Scripts you wrote yourself, and any lab whose record shape changed, land here. Nothing is dropped.

The context is missing for a table you care about. Fill it from the notebook’s machine section if it is there. If it is not, the number goes in with “not recorded” beside it and a line in section 9. Reconstructing it from memory is the one route that is closed.

The cost figure looks implausible. Check the utilisation inputs first: the generating hours and the powered-on hours are the two terms that move the total most, and mixing up a total with a marginal figure produces a difference of several times. Report both and the confusion becomes impossible.

The report is very long. It is meant to be readable, not complete. Cut the parts that describe what you configured and keep the parts that say what you measured and what you concluded. Configuration belongs in the checklists, which the report can point at.

You cannot think of three things that did not work. Look in the discarded training runs, the first version of the sandbox, the alert that did not fire, the certificate the phone did not trust, and the second model that would not load beside the first. They are all in this capstone’s material, and forgetting them is the normal state a week later, which is why the notebook exists.

Cleanup

Nothing to stop and nothing to delete. Keep the report, the five deliverables it summarises, the rubric with both dates on it, and the notebook. Put them somewhere your backup includes, which Capstone 2’s checklist should already cover, and check that it does.

If you are taking the platform apart after this, do it after the report is written and not before. The last measurement you can take is always the one you did not know you needed.

What you learned

A number without its context is not a measurement. The collector makes that concrete by naming the context keys your notebook never recorded, and each one is something a reader would have had to take on trust.
The notebook was the point. Every table in this report came from a line somebody wrote down at the time, and the gaps are exactly where nobody did.
The cost model is mostly an argument about utilisation. Reporting the marginal figure beside the total, with the assumption stated, is what stops the section being misread.
What did not work is the section that earns trust. It is the only part that cannot be reconstructed from the tables.
Reproduction is digests and revisions, not names. A tag moves and a repository is updated; a digest and a revision are what make “the same model” checkable.
Say what you do not know, at length. That section is the one the course has been building towards since the first lesson, and it is the difference between a report and a brochure.

Record in the notebook, as the last entry: the date you finished, the requirement you set in Capstone 1, whether the platform met it, and the one measurement you wish you had taken earlier. Then, if you compared the two guesses you wrote at the start of the overview lesson, record how far out they were. That comparison is the honest summary of what the course taught you.

Sources for this lesson

2 verified · checked 2026-09-09

01Hugging Face Hub — CLI guide§ Download; cache management and verificationhuggingface.co/docs/huggingface_hub/en/guides/cli2026-09-09
02EleutherAI — lm-evaluation-harness§ README; reproducibility of publicly available promptsgithub.com/EleutherAI/lm-evaluation-harness2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.

Capstone 6: The Report

Objective

Architecture

Requirements

Track S — NVIDIA DGX Spark

Track X — AMD Ryzen AI Max+ 395

Track M — Apple silicon

Track N — NVIDIA desktop or laptop

Tasks

Working directory and terminal roles

1. Collect the evidence from the notebook

2. Fill in what the notebook never recorded

3. Draw the architecture as it actually is

4. Write the measurement sections, with their context adjacent

5. The cost model, with its inputs shown

6. What did not work, and what changed in your understanding

7. The maintenance plan, and the reproduction section

8. Apply the rubric, twice

9. Write the section on what is not true here

Perform an evidence audit before calling the course complete

Validation

Expected outcome

Troubleshooting

Cleanup

What you learned

Check your understanding

Sources for this lesson