Reality Check: 'Four Mac Studios Replace a GPU Server'

Validated on: written from the documentation and the third-party report cited above; not yet validated on hardware on any track. The course’s reference lab has one Mac with 24 GB, so the Mac-pair side of this comparison is documentation-only and every results table is marked pending. The method, the worksheet and the reporting format are what this page delivers; the numbers are yours.

?Reality check

The claim under test, in the form it usually reaches you: “four Mac Studios replace a GPU server”. Sometimes it is two Macs and a workstation, sometimes it is a photograph of a stack of aluminium boxes with a caption about how much cheaper they are. It cannot be agreed or disagreed with as it stands, because it does not say replace at what, for how many users, or measured how. This page breaks it into five questions that each have an answer, and then measures the ones your machines can reach.

Objective

By the end of this page you will have a written statement of what “replace” means for your own workload, four measurements taken with one tool against two machines, a cost-per-million-tokens figure for each side computed from prices you can verify and throughput you measured, and a verdict per workload rather than a verdict overall. Some of those verdicts will disagree with each other. That is the finding, not a failure of the method.

If you do not own a Mac, or own only one, you can still do most of this page: the reported measurements below are read rather than reproduced, the worksheet runs on any two machines you have, and the reporting format is the point.

The claim is five claims

“Replace” hides five different comparisons, and a stack of Macs wins some and loses others. Naming them separately is most of the work.

Single-user latency. One person, one question at a time. What matters is time to first token and then how fast the words appear. This is bandwidth-bound decoding, and Part 5’s arithmetic predicts it from memory bandwidth alone. An Apple Ultra chip has a great deal of bandwidth, which is why single-user local chat feels good on a Mac.

Batch throughput. Ten people at once, or one agent making ten calls in parallel. Now the machine is doing many decodes simultaneously and the bottleneck moves towards compute and towards how well the engine batches. This is the case where a discrete GPU with a mature serving stack has the clearest advantage, and it is also the case the Mac cluster software is youngest at.

Capacity. Which models can each side hold at all. This is the axis where Macs are least contestable: unified memory scales to sizes that consumer GPUs simply do not reach, and a model that does not fit does not run at any speed.

Bandwidth. How fast weights can be read, which sets the ceiling on decode. Apple publishes this per chip and it is the single most predictive number for single-user speed.

Compute. How fast the arithmetic goes, which sets prefill speed and matters enormously for long prompts, retrieval-augmented work and agents that re-read a large context on every turn.

Turning the slogan into something that could be wrong

1. 1Name the workloadSingle user, or eight concurrent? Short prompts, or a 32k-token context re-read every turn? The answer changes the winner.
2. 2Name the two sidesExact machines, exact memory, exact price you would pay, from vendor pages rather than memory.
3. 3Name the model each side runsNot the same model. Each side runs the largest it can hold, because that is what you would actually do.
4. 4Fix the margin in advanceHow much difference would count as "replaces"? Decide before you see a number, or you will decide to suit the number.
5. 5Measure with one toolSame prompts, same generator, same day. Four runs: two sides, two concurrencies.
6. 6Price itCost per million output tokens from purchase price, working life, busy fraction, power and your measured throughput.
7. 7Report per workloadWhich side won which case, including the cases where the answer went against your expectation.

The refutable version

Here is the claim rewritten so that a result could contradict it. Substitute your own machines.

For a single user sending 2,048-token prompts and receiving 256-token answers, a pair of Macs serving the largest model they can hold produces a median time to first token within a factor of two of a GPU machine serving the largest model it can hold, and a decode rate within a factor of two. For eight concurrent users on the same prompts, the same is true of aggregate output throughput. And the cost per million output tokens, at a working life of four years and a busy fraction you state in advance, is no higher on the Mac side.

Every clause can now fail independently, which is the point. A result where the Macs win capacity and single-user latency and lose batch throughput and cost is a perfectly good result, and it is the one this page expects most often.

∴Research note

The two sides deliberately run different models. Forcing both to run the small model that fits on the GPU machine answers a question nobody has: if you owned the Macs you would run the larger model. Forcing both to run the large model the Macs can hold is worse, because the GPU machine would be swapping to system memory and you would be measuring a configuration nobody would choose. Each side runs its best; the write-up names both models. That is a comparison of machines as you would use them, and the page says so rather than pretending it is a controlled experiment.

What other people have reported

The course has not measured a four-Mac cluster. One published account is fetchable and worth reading in full, and it is quoted here as a report rather than as a result: Jeff Geerling’s write-up of a four-machine Mac Studio cluster, published 2025-12-18. The hardware was loaned by Apple and the software was pre-release, both of which the author states plainly, and both of which matter when reading the numbers.

Reported by Jeff Geerling, 18 December 2025, on 4 × M3 Ultra Mac Studio with pre-release exo and macOS 26.2:

What was reported	The figure as reported
The cluster	Four M3 Ultra Mac Studios, two with 512 GB and two with 256 GB, 1.5 TB of unified memory in total
Price as configured	“just shy of $40,000”; “$11,699 each” for the 512 GB machines and “$8,099 each” for the others
Power	the machines run “under 250 watts”; idle draw “less than 10 watts”
RDMA latency	“lowering memory access latency from 300μs down to < 50μs”
Thunderbolt throughput	“something like 50-60 Gbps real-world throughput”
Linear algebra (HPL)	“1.3 Teraflops with a single M3 Ultra”; “with all four put together, I got 3.7”
Qwen3-235B with exo	“hitting 32 tokens per second on the full cluster”
Kimi K2 Thinking, a trillion-parameter model	“around 30 tokens per second”
llama.cpp RPC by comparison	“spreads layers of a model across nodes, which scales but is inefficient, causing performance to decrease as you add more nodes”

The two models named in that table are the reviewer’s choices, not the course’s. Qwen3-235B-A22B is in the course’s model reference as an Apache-2.0 licensed mixture-of-experts model; Kimi K2 Thinking is not one of the course’s reference models, and its licence should be checked on its own model card before you plan around it.

Two of his sentences bear directly on the claim this page is testing. On whether you need the cluster at all: “the bigger question is: do you need a full cluster of Mac Studios at all? Because just one is already a beast, matching four maxed-out DGX Sparks or AI Max+ 395 systems.” And on the state of the software: “it was obvious RDMA over Thunderbolt is new. When it works, it works great. When it doesn’t…” Both are one person’s assessment on loaned hardware with pre-release software, and both are exactly the kind of statement this page exists to turn into a measurement.

exo’s README makes its own claims, which are the vendor’s rather than an independent party’s: a “99% reduction in latency between devices” from RDMA, and tensor parallelism giving “up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices”. Your own measurements from this part’s lab are the check on those.

The vendor figures behind all of it

Apple publishes the two numbers that predict most of the Mac side’s behaviour. These are specifications, not measurements, and the course quotes them as such.

Vendor specification, not measuredMac Studio, as Apple's specification page states it

Configuration	Unified memory	Memory bandwidth	Thunderbolt
M5 Max, base GPU	36 GB to 128 GB	460 GB/s	Four Thunderbolt 5 ports, up to 120 Gb/s
M5 Max, 40-core GPU	36 GB to 128 GB	614 GB/s	Four Thunderbolt 5 ports, up to 120 Gb/s
M5 Ultra	96 GB, to 256 GB or 512 GB	1.2 TB/s	Four Thunderbolt 5 ports, up to 120 Gb/s

Apple Mac Studio, as configured on Apple’s specification page, not applicable; these are hardware specifications · not applicable specification page as read on 2026-09-09 · not applicable, not applicable · 1 tokens of context · 2026-09-09

Vendor-published figures, not measured by this course. Memory bandwidth is the ceiling on decode speed and nothing else; it says nothing about prefill, about batching or about how many people the machine can serve. The equivalent figures for the other three tracks are in the course hardware reference. Apple also lists 10Gb Ethernet on the Mac Studio, which is the link you would use if you were not using Thunderbolt.

The pattern in that table is the whole argument. Very large memory at very high bandwidth, joined by a link that is fast for a house and slow compared with a memory bus. That combination is excellent for holding a big model and reading it for one user, and it is not the same thing as a serving machine.

Requirements

Track M — Apple silicon

The primary path. One Mac, or the pair from this part’s lab, plus one machine of any other track to compare against. 32 GB on the Mac side for a model worth comparing; less will work but the capacity axis becomes uninteresting. Both machines serving an OpenAI-compatible endpoint at the same time, from this part’s lab on the Mac side and from Part 9 on the other.

Time. About forty-five minutes attended, most of it the four measurement runs. The models should already be downloaded from earlier parts; if not, add the download time as unattended.

Track S — NVIDIA DGX SparkPartial

A DGX Spark is one half of this comparison rather than the subject of it: it is the machine the Macs are being compared against.

Serve your largest comfortable model with vLLM from Part 9 or llama-server from Part 6, point SIDE_B_URL at it, and you have provided the GPU side. Everything from task 4 onwards is the same. Without a Mac on the other side, read the reported measurements above and do the worksheet with two machines you do own; the arithmetic and the reporting format are the transferable parts.

Track X — AMD Ryzen AI Max+ 395Partial

Same role as Track S: this machine is one side of the comparison, not the subject.

Serve from llama-server or Lemonade as Part 8 describes, point SIDE_B_URL at it, and provide the non-Apple side. A Ryzen AI Max+ 395 box at 128 GB is an unusually interesting comparison here, because it competes with the Macs on capacity rather than only on speed.

Track N — NVIDIA desktop or laptopPartial

Same role again: the GPU machine in the claim is most often a machine like this one.

This is the machine the slogan is usually about. Serve your largest comfortable model with vLLM from Part 9, note how much VRAM it left free, and point SIDE_B_URL at it. If your card has 16 GB or 24 GB, the capacity axis will be decisive and the throughput axis may well go the other way, which is exactly the split this page is built to expose.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-21-apple-clusters"
cd "$LAB_DIR"
pwd
test -f "compare-endpoints.sh"

Expected result: pwd ends in part-21-apple-clusters and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Write down both machines, from the vendor pages

Before any measurement, write two paragraphs in your notebook: what each machine is, how much memory it has, what its published memory bandwidth is, and what you would pay for it today in your own currency including tax. Use the vendor’s own page and the retailer you would actually buy from, not a figure you remember.

This is the step people skip, and it is the step that decides the cost comparison. A price from a different country, a different year or a different configuration will dominate every other number in the worksheet.

2. Fix the workload and the margin before you measure

Write the refutable version of the claim, with your machines’ names in it, in your notebook. State the prompt length, the answer length, the two concurrency levels and the margin that would count as “replaces”. Then stop and read it back: if no plausible measurement could contradict it, it is not yet a claim.

▷Exercise

Write the margin down as a number before you continue, and put the date next to it. “Within a factor of two” is a reasonable default for a home comparison, because the differences that matter in practice are usually larger than that and the measurement noise is usually smaller. If you prefer a different margin, choose it now rather than after you have seen the result.

3. Serve the largest model each side can actually hold

On the Mac side, serve from this part’s lab: bash run-distributed.sh serve for a split model across two Macs, or mlx_lm.server directly for one. On the other side, serve with vLLM or llama-server as Parts 6 and 9 describe. Note in your notebook which model each side is running and at what quantisation, because that pair of names is what makes the result interpretable.

If one side cannot hold a model worth serving at all, that is the capacity result and it is worth recording as such: “the GPU machine cannot serve this model at any speed” is a stronger finding than any tokens-per-second figure.

4. Measure both sides, at both concurrencies, with one tool

Four runs, one script, the same prompts throughout.

RunnableAll tracks

compare-endpoints.sh

#!/usr/bin/env bash
# Purpose: run the same workload against two endpoints, one on each side of the
#          comparison, at both the single-user and the batch concurrency, so that the
#          reality check has four measurements taken by one tool with one prompt set
#          rather than four numbers from four sources. Every run is labelled with the
#          side and the concurrency and appended to the lab notebook.
# Platform: all (spark, strix, mac, nvidia). The script itself only makes HTTP
#           requests, so it can run on a laptop while both machines serve.
# Minimum memory: none on the machine running this; the memory floors belong to the
#                 two machines being compared, and each side's own page states them.
# Assumes: two OpenAI-compatible endpoints already serving, python3 on PATH,
#          measure-pair.py beside this script, and an .env copied from
#          env-example.txt with SIDE_A_URL, SIDE_B_URL and their names filled in.
#          Both endpoints must be serving the model each machine can actually hold;
#          that they are different models is the point, not a mistake.
set -euo pipefail

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${ENV_FILE:-${HERE}/.env}"

if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    . "$ENV_FILE"
    set +a
else
    echo "No ${ENV_FILE}. Copy env-example.txt to .env and fill it in." >&2
    exit 1
fi

: "${SIDE_A_NAME:=side-a}"
: "${SIDE_B_NAME:=side-b}"
: "${SIDE_A_URL:=}"
: "${SIDE_B_URL:=}"
: "${SIDE_A_MODEL:=}"
: "${SIDE_B_MODEL:=}"
: "${BATCH_CONCURRENCY:=8}"
: "${REQUESTS:=20}"
: "${MAX_TOKENS:=128}"
: "${PREFILL_TOKENS:=2048}"
: "${LABBOOK:=${HERE}/labbook.md}"

if [ -z "$SIDE_A_URL" ] || [ -z "$SIDE_B_URL" ]; then
    echo "SIDE_A_URL and SIDE_B_URL must both be set in ${ENV_FILE}." >&2
    echo "Each is the /v1 base of one machine's OpenAI-compatible endpoint." >&2
    exit 1
fi

if ! command -v python3 >/dev/null 2>&1; then
    echo "python3 is not on PATH." >&2
    exit 1
fi

BATCH_REQUESTS=$((REQUESTS * 2))

run_side() {
    # run_side <name> <base-url> <model-or-empty> <concurrency> <requests> <suffix>
    local name="$1" url="$2" model="$3" conc="$4" reqs="$5" suffix="$6"
    local model_args=()
    if [ -n "$model" ]; then
        model_args=(--model "$model")
    fi
    echo ""
    echo "==> ${name}, concurrency ${conc}, ${reqs} requests"
    python3 "${HERE}/measure-pair.py" \
        --base-url "$url" \
        --concurrency "$conc" \
        --requests "$reqs" \
        --max-tokens "$MAX_TOKENS" \
        --prefill-tokens "$PREFILL_TOKENS" \
        --label "${name}-${suffix}" \
        --labbook "$LABBOOK" \
        "${model_args[@]+"${model_args[@]}"}"
}

echo "Comparing ${SIDE_A_NAME} against ${SIDE_B_NAME}."
echo "Single-user runs answer 'which feels faster to one person'."
echo "Batch runs answer 'which serves more people at once'. They can disagree."

run_side "$SIDE_A_NAME" "$SIDE_A_URL" "$SIDE_A_MODEL" 1 "$REQUESTS" "single-user"
run_side "$SIDE_B_NAME" "$SIDE_B_URL" "$SIDE_B_MODEL" 1 "$REQUESTS" "single-user"
run_side "$SIDE_A_NAME" "$SIDE_A_URL" "$SIDE_A_MODEL" \
    "$BATCH_CONCURRENCY" "$BATCH_REQUESTS" "batch"
run_side "$SIDE_B_NAME" "$SIDE_B_URL" "$SIDE_B_MODEL" \
    "$BATCH_CONCURRENCY" "$BATCH_REQUESTS" "batch"

echo ""
echo "Four measurements recorded in ${LABBOOK}. Read them back with:"
echo "  grep 'served-measurement' ${LABBOOK}"
echo ""
echo "Take aggregate_output_tokens_per_second from the batch runs and the median"
echo "time to first token from the single-user runs into the results table, then"
echo "feed the batch throughput into cost-per-million-tokens.py."

Download compare-endpoints.sh93 lines

Fill in SIDE_A_URL, SIDE_B_URL and the two names in .env, then run it.

RunnableAll tracks

two sides, two concurrencies, four notebook lines

bash compare-endpoints.sh

The single-user runs use one request at a time and answer the latency question. The batch runs put eight requests in flight and answer the throughput question. Both use the same prompt set and the same generator, which is the only way the four numbers can be compared with each other.

✕Common mistake

Do not run the two sides at once from the same machine while that machine is also serving one of them. The load generator is cheap but not free, and a generator competing with an engine for the same processor produces a result that flatters the other side. Run the generator on a third machine if you have one, or run the two sides in sequence, which the script already does.

5. Record what each side cannot do

Capacity is measured by failure, so record the failures. For each side, note the largest model it could load and the first one it could not, with the error. On the Mac side that failure is usually a memory message; on a GPU machine with less VRAM it is usually much earlier and much clearer.

This is the axis where the claim is at its strongest, and it deserves as careful a record as the speed axes. A machine that holds a 120B-class mixture-of-experts model and answers slowly is doing something a machine that cannot hold it is not doing at all.

6. Work out cost per million output tokens

Now the arithmetic. The formula is deliberately simple and every input is yours:

Pseudocode — not a real command

busy hours over life      = lifetime years x 8760 x busy fraction
capital cost per busy hour = purchase price / busy hours over life
energy cost per busy hour  = average watts / 1000 x electricity price per kWh
tokens per busy hour       = measured output tokens per second x 3600

cost per million tokens = (capital + energy per busy hour)
                          / tokens per busy hour x 1,000,000

The busy fraction is the input people get wrong and it is the one that decides the answer. A machine that serves requests for one hour a day carries twenty-four times the capital cost per token of one that runs flat out, and most home machines are far closer to the first than the second. Choose it honestly, and if you are not sure, run the worksheet twice with two different values and report both.

Fragment — not complete on its own

cost-inputs-example.json

{
  "_note": "Copy to cost-inputs.json and replace every value. Nothing here is a real price, a real power figure or a real speed: the zeros are there so that a file you have not filled in fails loudly instead of producing a confident answer. Use the price you would actually pay in your own currency including tax, the working life you would actually plan for, the fraction of that life the machine is really serving requests, your own electricity tariff, and a throughput figure you measured yourself on the workload named below.",
  "workload_definition": "REPLACE: the one workload every machine in this file is being priced for. For example: single user, one request at a time, 2048-token prompt, 256-token answer, on the largest model that machine can serve.",
  "machines": [
    {
      "name": "REPLACE: what this machine or cluster is, with chip and memory",
      "currency": "REPLACE: the currency you typed the price in",
      "workload": "REPLACE: same as workload_definition, restated for this machine",
      "purchase_price": 0,
      "lifetime_years": 0,
      "busy_fraction": 0,
      "average_watts": 0,
      "electricity_price_per_kwh": 0,
      "output_tokens_per_second": 0,
      "model_served": "REPLACE: the model and quantisation this throughput was measured with",
      "measured_on": "REPLACE: the date you measured it"
    },
    {
      "name": "REPLACE: the other side of the comparison",
      "currency": "REPLACE: the currency you typed the price in",
      "workload": "REPLACE: the same workload, or the page will be comparing two different questions",
      "purchase_price": 0,
      "lifetime_years": 0,
      "busy_fraction": 0,
      "average_watts": 0,
      "electricity_price_per_kwh": 0,
      "output_tokens_per_second": 0,
      "model_served": "REPLACE: the model and quantisation this throughput was measured with",
      "measured_on": "REPLACE: the date you measured it"
    }
  ]
}

Download cost-inputs-example.json32 lines

RunnableAll tracks

cost-per-million-tokens.py

"""Turn hardware prices, power draw and measured throughput into cost per million tokens.

Purpose: the arithmetic that makes "cheaper" mean something. Every input is yours: the
    price you would actually pay, the working life you would actually plan for, how
    much of that life the machine is actually busy, what electricity costs where you
    live, and the throughput you measured yourself. The script contains no prices and
    no speeds, because a price the course looked up would be wrong for your country
    within a month and a speed it looked up would be wrong for your machine on the
    day. It reports cost per million output tokens for each machine you describe, and
    appends the whole calculation to the lab notebook so the inputs travel with the
    answer.
Platform: all (spark, strix, mac, nvidia). Pure standard library.
Minimum memory: none of consequence; this is arithmetic.
Assumes: an inputs file in the shape of cost-inputs-example.json, holding one entry
    per machine or cluster you are comparing. Throughput figures come from your own
    measurements: on Track M from measure-pair.py in this part, elsewhere from the
    load generator in Part 9. Currency is whatever you type; the script never
    converts and never assumes one.

Usage:
    python3 cost-per-million-tokens.py --inputs cost-inputs.json
    python3 cost-per-million-tokens.py --inputs cost-inputs.json --labbook labbook.md
    python3 cost-per-million-tokens.py --inputs cost-inputs.json --print
"""

from __future__ import annotations

import argparse
import json
from datetime import date, datetime, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent

HOURS_PER_YEAR = 8760
REQUIRED = (
    "name",
    "purchase_price",
    "lifetime_years",
    "busy_fraction",
    "average_watts",
    "electricity_price_per_kwh",
    "output_tokens_per_second",
)


def parse_args():
    p = argparse.ArgumentParser(description="Cost per million output tokens")
    p.add_argument(
        "--inputs",
        default=str(HERE / "cost-inputs.json"),
        help="JSON file describing each machine or cluster being compared.",
    )
    p.add_argument("--labbook", default=str(HERE / "labbook.md"))
    p.add_argument("--label", default="", help="A name for this comparison.")
    p.add_argument(
        "--print",
        dest="print_only",
        action="store_true",
        help="Show the result and write nothing.",
    )
    return p.parse_args()


def cost_per_million(entry: dict) -> dict:
    """One machine's cost per million output tokens, with every intermediate kept."""
    price = float(entry["purchase_price"])
    years = float(entry["lifetime_years"])
    busy = float(entry["busy_fraction"])
    watts = float(entry["average_watts"])
    kwh_price = float(entry["electricity_price_per_kwh"])
    tps = float(entry["output_tokens_per_second"])

    if years <= 0 or busy <= 0 or tps <= 0:
        raise ValueError(
            f"{entry['name']}: lifetime_years, busy_fraction and "
            "output_tokens_per_second must all be greater than zero."
        )
    if busy > 1:
        raise ValueError(f"{entry['name']}: busy_fraction is a fraction, not a percentage.")

    # Spread the purchase over the hours the machine is actually working. A machine
    # busy a tenth of the time carries ten times the capital cost per busy hour.
    busy_hours = years * HOURS_PER_YEAR * busy
    capital_per_busy_hour = price / busy_hours

    # Energy is charged for the busy hours too. Idle draw is deliberately excluded:
    # it is a cost of owning the machine, not a cost of the tokens.
    energy_per_busy_hour = (watts / 1000.0) * kwh_price

    tokens_per_busy_hour = tps * 3600.0
    total_per_busy_hour = capital_per_busy_hour + energy_per_busy_hour
    per_million = total_per_busy_hour / tokens_per_busy_hour * 1_000_000

    return {
        "name": entry["name"],
        "currency": entry.get("currency", "unstated"),
        "purchase_price": price,
        "lifetime_years": years,
        "busy_fraction": busy,
        "average_watts": watts,
        "electricity_price_per_kwh": kwh_price,
        "output_tokens_per_second": tps,
        "workload": entry.get("workload", "unstated"),
        "busy_hours_over_life": round(busy_hours, 1),
        "capital_cost_per_busy_hour": round(capital_per_busy_hour, 4),
        "energy_cost_per_busy_hour": round(energy_per_busy_hour, 4),
        "tokens_per_busy_hour": round(tokens_per_busy_hour, 1),
        "cost_per_million_output_tokens": round(per_million, 4),
        "capital_share": round(capital_per_busy_hour / total_per_busy_hour, 4),
    }


def main() -> int:
    args = parse_args()

    path = Path(args.inputs)
    if not path.exists():
        print(f"No {path}. Copy cost-inputs-example.json and fill it in.")
        return 1
    payload = json.loads(path.read_text(encoding="utf-8"))
    machines = payload.get("machines", [])
    if not machines:
        print("The inputs file has no machines in it.")
        return 1

    results = []
    for entry in machines:
        missing = [k for k in REQUIRED if k not in entry]
        if missing:
            print(f"{entry.get('name', 'an entry')} is missing: {', '.join(missing)}")
            return 1
        try:
            results.append(cost_per_million(entry))
        except (ValueError, KeyError, TypeError) as exc:
            print(f"Cannot compute a cost for this entry: {exc}")
            print("Every value in cost-inputs-example.json is a placeholder; the")
            print("file has to be filled in with your own figures before it means")
            print("anything.")
            return 1

    width = max(len(r["name"]) for r in results)
    print("")
    print(f"{'machine'.ljust(width)}  cost per million output tokens  capital share")
    for r in results:
        cost = f"{r['cost_per_million_output_tokens']:.2f} {r['currency']}"
        share = f"{r['capital_share'] * 100:.0f}%"
        print(f"{r['name'].ljust(width)}  {cost.rjust(30)}  {share.rjust(13)}")
    print("")
    print("Capital share is how much of the cost is the machine rather than the")
    print("electricity. A high share means the answer is mostly about how busy you")
    print("keep it, and a low share means it is mostly about how fast it is.")

    cheapest = min(results, key=lambda r: r["cost_per_million_output_tokens"])
    print("")
    print(f"Cheapest per million output tokens, on these inputs: {cheapest['name']}")
    print("That is a result about this workload and these assumptions, and it changes")
    print("if you change the busy fraction, the lifetime or the workload.")

    record = {
        "lab": "part-21/reality-check-macs-replace-a-gpu-server",
        "record": "cost-per-million-tokens",
        "date": date.today().isoformat(),
        "recorded_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
        "label": args.label,
        "machines": results,
        "cheapest": cheapest["name"],
    }
    text = json.dumps(record, sort_keys=True)
    if args.print_only:
        print("")
        print(text)
        return 0
    with open(args.labbook, "a", encoding="utf-8") as fh:
        fh.write(text + "\n")
    print("")
    print(f"Appended to {args.labbook}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Download cost-per-million-tokens.py182 lines

RunnableAll tracks

the worksheet, with your numbers in it

cp cost-inputs-example.json cost-inputs.json

Fill in every field, using the batch throughput from task 4 for output_tokens_per_second if the workload you care about is serving several people, or the single-user decode rate if it is not. Then run it.

RunnableAll tracks

cost per million output tokens, both sides

python3 cost-per-million-tokens.py --inputs cost-inputs.json --label macs-versus-gpu

=Expected result

Two lines, one per side, each with a cost per million output tokens and a capital share. The capital share is the diagnostic: where it is high, the answer is mostly about how busy you keep the machine, and a change to the busy fraction will move the verdict. Where it is low, electricity dominates and the faster machine wins almost regardless of price.

!Important

The course prints no prices of its own, here or anywhere. Every figure in the worksheet is one you looked up or measured, and the only prices quoted on this page are the ones a named third party published on a stated date. A cost comparison built on a price someone else remembered is not a cost comparison.

7. Report which side won which case

Write the result up in the course’s benchmark format: the two machines, the two models, the workload, the four measurements, the two cost figures, and one sentence per workload saying which side won it. Then one paragraph on what would change the answer.

Pending validationThe verdict table, one row per workload

Workload	Mac side	Other side	Which won	By how much
Single user, time to first token	from your run	from your run	from your run	from your run
Single user, decode rate	from your run	from your run	from your run	from your run
Eight concurrent, aggregate output	from your run	from your run	from your run	from your run
Largest model held at all	from your run	from your run	from your run	not applicable
Cost per million output tokens	from your worksheet	from your worksheet	from your worksheet	from your worksheet

your two machines, both named with chip, memory and price, the operating system and build on each · mlx-lm or exo on the Mac side; vLLM or llama-server on the other the versions each machine reports on the day · the largest model each side could hold, named separately, the quantisation of each, named separately · 2,048 tokens of context · the date of your run

Empty because the course's reference lab has one Mac with 24 GB and cannot run the Mac-pair side. The shape is the deliverable: five workloads, five verdicts, and a column for the size of the difference so that a narrow win is not reported as a rout.

Compare the requirement, not the device category

Write the task-quality threshold, latency target, concurrency and ownership-cost assumptions before measurement. Use the same checkpoint and representation where both systems support them. If one system needs a different model to fit, keep that as a second comparison and evaluate quality too.

Record sustained conditions: memory pressure, other workloads, warm-up and power state. Test both interactive and loaded operation with the same arrival pattern. Include failed requests and output lengths in the record. A machine that holds a larger model has a capacity advantage; it may still deliver a different service rate.

Use actual cost inputs and measured energy where available, with estimates clearly identified. Separate an already-owned machine from a new purchase. Add a row for capabilities your workload requires but you did not validate, such as a training method or a particular distributed backend. The final verdict should answer each requirement separately and state the conditions under which either system is adequate. Keep raw results and assumptions so changing utilisation, electricity price or model choice does not require inventing a new comparison from memory.

Validation

Every row of the verdict table has a source. Each speed figure traces to a served-measurement line in your notebook, and each cost figure to a cost-per-million-tokens line. If a number in the table is not in the notebook, it came from somewhere you cannot check.

The two sides ran the same prompts. One script, one prompt set, four runs. If you measured one side last week with a different tool, measure it again.

The margin was fixed before the measurement. Check the date next to it in your notebook.

At least one verdict disagrees with the others. This is not a rule, but if all five columns went the same way, look for a mistake: the most common one is running both sides on the same model, which collapses the capacity axis and usually the cost axis with it.

Expected outcome

A verdict per workload, in writing, with the size of each difference. Based on the arithmetic in Part 18 and the vendor figures above, the shape most people should expect is: Macs strong on capacity and on single-user decode, weaker on batch throughput and on prefill for long prompts, and a cost answer that swings on the busy fraction rather than on the hardware.

But the point of the page is that you should not take that from here. Two people with the same two machines and different workloads will reasonably reach different verdicts, and both will be right.

?Reality check

A fair summary of the claim, once measured, usually reads something like: for holding models that a single GPU machine cannot hold, and for one person at a time, a stack of Macs does a job the GPU machine cannot do. For many concurrent users on a model that fits on the GPU machine, it does not replace it. The slogan is not false so much as unfinished, and finishing it is the work.

Troubleshooting

The two sides report wildly different time to first token and you suspect the generator. Run the generator against the same endpoint twice in a row and compare. If the two runs differ by more than the two sides do, something on that machine is competing for the processor.

Batch throughput on the Mac side is barely above the single-user figure. That is a result, not a fault: it means the engine on that side is not batching much. Record it. It is the single clearest reason the batch axis usually goes the other way.

The cost figures are absurd. Check the busy fraction first: it is a fraction, not a percentage, and a value of 20 rather than 0.2 will produce a suspiciously cheap machine. The script rejects values above 1 for exactly this reason.

One side cannot serve anything comparable. Then the comparison is a capacity finding and you should write it up as one. A row saying “cannot hold this model” is more useful to a reader than a row of numbers from a machine that was swapping.

Cleanup

Stop both servers. On the Mac side, Ctrl-C the mlx.launch terminal and delete any exo instance with bash run-exo.sh delete so the memory comes back. On the other side, stop the engine as its own part describes. Keep cost-inputs.json: it is the record of what you assumed, and the next time someone repeats the slogan at you, the fastest reply is to open it and change one number.

What you learned

• “Replaces” is five questions. Single-user latency, batch throughput, capacity, bandwidth and compute have different answers on the same pair of machines, and a slogan that does not say which one it means cannot be true or false.
• The two sides should run different models. Each machine running the largest model it can hold is the comparison you would actually live with, and pretending otherwise produces a tidier experiment about a situation nobody is in.
• Cost per million tokens is mostly about the busy fraction. For a home machine the capital cost dominates, and how many hours a year the machine is actually serving moves the answer more than the purchase price does.
• Third-party numbers are reports, not results. The figures on this page from a published review are quoted with the reviewer, the date, the hardware, who supplied it and the software’s maturity, because every one of those changes how the number should be read.
• A split verdict is the normal outcome. Writing “the Macs won capacity and single-user decode and lost batch throughput” is a more useful answer than any single winner, and it is the kind of answer this course is trying to make routine.

Record in your lab notebook: both machines with price and date, both models with quantisation, the margin you fixed in advance and when, the four measurements, both cost figures with the busy fraction you assumed, and one sentence per workload naming the winner and the margin.

Check your understanding

Question 1. You measure a Mac pair against a 24 GB NVIDIA desktop. The Macs win single-user decode, lose batch throughput badly, and hold a model the desktop cannot load at all. What is the correct write-up?

• The Macs replace the GPU machine
• The GPU machine replaces the Macs
• A verdict per workload: the Macs won capacity and single-user decode, the GPU machine won batch throughput, with the margins stated
• The comparison is invalid because the two sides ran different models

Show the answer and why

Answer: A verdict per workload: the Macs won capacity and single-user decode, the GPU machine won batch throughput, with the margins stated

A split verdict is the expected outcome and the useful one. Running different models is deliberate here, not a flaw: each side runs the largest model it can hold, because that is what you would do if you owned it, and the write-up names both models so the reader can see it.

Question 2. Which inputs to the cost-per-million-tokens worksheet come from the course rather than from you?

• The hardware prices
• The typical tokens per second for each platform
• The electricity price
• None of them; the course supplies the formula and no values

Show the answer and why

Answer: None of them; the course supplies the formula and no values

The script contains no prices and no speeds. A price the course looked up would be wrong for your country within a month and a throughput figure would be wrong for your machine on the day. What the course supplies is the arithmetic and the discipline of recording the assumptions alongside the answer.

Question 3. Which of these correctly describe the figures quoted from the published four-Mac cluster review?

• The hardware was supplied by Apple on loan
• The software was pre-release at the time of testing
• They were reproduced by this course on its own hardware
• They are dated and attributed to a named reviewer

Show the answer and why

Answer: The hardware was supplied by Apple on loan, The software was pre-release at the time of testing, They are dated and attributed to a named reviewer

All three of those facts change how the numbers should be read, which is why the page states them alongside the figures. The course has not reproduced them: its reference lab has one Mac with 24 GB, and the page says so in its validation line rather than quietly presenting someone else's numbers as its own.

Question 4. Two people compare the same Mac pair against the same GPU machine and reach opposite verdicts. What is the most likely explanation?

• One of them made an arithmetic error
• They defined different workloads, or different busy fractions, before measuring
• The hardware changed between the two tests
• One of them used a different load generator, so neither result is valid

Show the answer and why

Answer: They defined different workloads, or different busy fractions, before measuring

The workload and the busy fraction are inputs, not constants, and both move the verdict. A single user with an occasional question and a team hammering an endpoint all day are different questions with different right answers on the same hardware. This is why the page asks you to write both down before measuring anything.

Check my answersClear