Capacity Planning and Cost per Million Tokens at Home

By the end of this lesson you will be able to state what a million tokens costs on your own machine, show the arithmetic that produced the figure, and say which of its inputs you measured and which you assumed. You will also be able to answer, without hand-waving in either direction, the question somebody always asks: would it be cheaper to just use an API?

The honest answer to that question is “it depends on how much you use it”, and this lesson is about making the dependence explicit rather than settling it with an opinion.

Three costs, and only one of them is electricity

People costing a home model service usually compute one number: the watts times the hours times the tariff. That number is real and it is almost never the largest one.

The energy the work uses. The machine draws more under load than at rest, and the difference is what generating tokens costs in electricity. This is the number everybody computes.

The energy the machine uses anyway. A server that is switched on all the time and generating for a small part of it burns electricity for the rest of the day doing nothing. That electricity is a real cost of having the service, and it belongs somewhere. Charging it to the hours when the machine is actually working is the honest place to put it.

The machine. You bought hardware. Over its life it will generate some number of tokens, and the purchase price divided by that number is a cost per token whether or not anybody writes it down. On a home service this is usually the dominant term by a wide margin, which is why the answer to “is local cheaper” is so sensitive to how much you actually use it.

?Reality check

The claim to hold up for examination is “running models locally is free once you own the hardware”. It is not: it is the same argument as saying a car is free once you have bought it. What is true is that the marginal cost of one more request on a machine that is already switched on is small, and often smaller than any hosted price. Both facts matter, and the script in this lesson reports both, because which one is relevant depends entirely on whether you would have bought the machine anyway.

Measuring power, per track

Part 5 established the method and this is that method applied to watts. Take a baseline before you take a measurement. Sustain the load long enough for the machine to reach a steady state rather than a peak. Repeat, and treat variance as information about the machine rather than as noise to be averaged away.

Two readings are needed: idle, with the service running and nothing being asked of it, and load, during a sustained generation run at the concurrency you actually serve. Give each at least ten minutes, and take the reading from the second half of the window, because the first half is the machine warming up and the fans deciding what to do.

!Important

A plug-in energy meter at the wall is the reference measurement and it is the one to trust. Every software reading below reports the power of a component, not of the machine: the processor package, or the graphics card, or the system-on-chip. The difference between that and what the wall sees includes the power supply’s losses, the disks, the fans, and on a desktop it is not a small correction. Use the software reading for the shape over time, which is what the dashboard gives you, and the wall meter for the number you put in the cost model.

Track S — NVIDIA DGX Spark

nvidia-smi --query-gpu takes a comma-separated property list with --format=csv, and --loop repeats it on an interval; the documentation directs you to nvidia-smi --help-query-gpu for the property names available on your driver, so read that rather than a list from elsewhere. The DCGM exporter publishes the same reading continuously as DCGM_FI_DEV_POWER_USAGE, which is what the dashboard from the previous lesson graphs.

On this machine the accelerator and the processor share one package and one memory pool, so a component reading is closer to the whole machine than it would be on a desktop. That makes the software figure more useful here than on Track N, and it still is not the wall.

Track X — AMD Ryzen AI Max+ 395Partial

AMD's command-line tools report package power for the accelerator; the wall meter remains the reference for the whole machine.

AMD’s CLI documentation gives amd-smi metric -p for current power, with --json for machine-readable output; the older rocm-smi --showpower is what this course’s command reference records and what the lab’s exporter reads. As on Track S, the unified design means the component reading is a larger share of the machine’s total than on a desktop with a discrete card.

Track M — Apple siliconPartial

Apple publishes the powermetrics manual only on the machine itself, so confirm its options with `man powermetrics` before relying on them.

powermetrics samples power by subsystem and normally needs root. It is the only software source on this track, and unified memory means there is no separate graphics power domain to add up in the way a desktop has.

A laptop adds a complication worth naming: the sustained power a Mac notebook will draw depends on whether it is plugged in, on the thermal state of the chassis, and on what else is running. Take the idle and load readings in the same power state, on the same day, and record which state it was. Part 5’s lesson on this machine made the same point about throughput, and it is the same physics.

Track N — NVIDIA desktop or laptop

Same tools as Track S: nvidia-smi --query-gpu with --format=csv and --loop for a look, DCGM_FI_DEV_POWER_USAGE for a graph.

The gap between the card’s reading and the wall’s is largest on this track. A desktop with a discrete card is drawing power for the card, the processor, the board, the disks and the fans, and the power supply is losing some fraction of all of it as heat. If you cost a desktop from the card’s own reading you will get an answer that is comfortably and consistently too low. On a laptop, add the complication that the manufacturer’s power limit decides the sustained clock, so the load figure you measure is a property of that model rather than of the chip.

The arithmetic

Everything follows from two measurements and one tariff. Write it symbolically first, because the symbols are what you will re-use when the numbers change.

Pseudocode — not a real command

H  = hours to generate a million tokens        = 1,000,000 / T / 3600
                                                 where T is output tokens per second

E  = energy cost of a million tokens           = (W_load / 1000) * H * P
                                                 where W_load is watts under load
                                                 and P is your price per kilowatt-hour

M  = marginal energy cost                      = ((W_load - W_idle) / 1000) * H * P
                                                 the extra cost over leaving it switched on

C  = capital cost per million tokens           = (price / (years * D)) * H
                                                 where D is hours per year the machine
                                                 actually spends generating

I  = idle energy carried by the work           = (W_idle / 1000) * (A - D) * P / D * H
                                                 where A is hours per year it is powered on

Total per million tokens = E + C + I
Marginal per million tokens = M

Two things fall out of those expressions that are worth internalising before you put any numbers in.

Throughput divides everything. H appears in every term, so doubling tokens per second halves every cost. That is why the concurrency question below matters so much, and it is why the quantisation and engine choices from Parts 16 and 17 are cost decisions as well as speed decisions.

Utilisation divides the capital. C and I both have D in the denominator. A machine that generates for a couple of hours a day carries its purchase price across a small number of tokens, so each one is dear. The same machine generating all day is far cheaper per token and the electricity barely moved. This is the single most counter-intuitive result in home model serving: the way to make your local models cheap is to use them more.

✦Tip

Compute H once and keep it. It is the bridge between “tokens per second”, which is what you measure, and “hours”, which is what everything is charged in. Everything else in the model is that number multiplied by a rate.

Concurrency and headroom

The throughput in T should be the aggregate output rate at the concurrency you actually serve, not the single-stream rate. Those are different numbers and the difference is the whole argument for batching: a decode step reads the weights once and can produce a token for several sequences at the same time, so serving several conversations at once raises total throughput considerably while making each individual one somewhat slower. Part 9’s load test is where you measure this; this lesson is where you spend it.

That leads directly to a planning rule.

Choosing a concurrency to plan for, and what it costs you

1. 1Measure aggregate throughput at each concurrencyPart 9 load test, one run per slot count, on the model and context length you actually serve. Record where the total stops improving.
2. 2Find where the curve flattensBeyond some point another slot adds latency to everybody and almost no total throughput, because the machine is already bandwidth-bound.
3. 3Check the memory at that point, not at restEvery concurrent sequence needs its own key-value cache. The slot count and the context length multiply, and this is where a plan quietly stops fitting.
4. 4Plan for less than the peakLeave headroom for one long prompt arriving while everything is busy. A machine sized for its own best case fails on its ordinary worst case.
5. 5Enforce the plan at the gatewayA per-key requests-per-minute limit is how the planned concurrency becomes the served concurrency rather than an aspiration.
6. 6Cost the plan, not the peakUse the aggregate throughput at your chosen concurrency in the cost model, because that is the machine you are actually going to run.

Headroom is not waste. It is the difference between a service that degrades when something unusual happens and one that stops.

The memory step deserves emphasis because it is where the next page’s challenge comes from. Key-value cache use scales with the number of concurrent sequences multiplied by the context length each one has reached. A configuration that fits comfortably when four short conversations are in progress does not necessarily fit when the same four conversations have been going for an hour. Nothing changed except time.

Comparing with a hosted price

The comparison is worth doing and it is easy to do dishonestly in either direction. Three rules keep it fair.

Compare the model you would actually use. Not the largest hosted model, and not the cheapest. The right comparison is against whatever you would run if the local machine were switched off, which for most people is a mid-sized hosted model rather than the flagship.

Compare like tokens. Hosted prices are usually quoted separately for input and output, and the ratio between them is not one. If your workload is long documents and short answers, a comparison based on output tokens alone will flatter the local machine.

Say which local cost you are using. Total, including the machine, is the right number when you are deciding whether to buy hardware. Marginal, the extra electricity only, is the right number when the hardware exists and the question is what to do with it tonight. They can point in opposite directions and both answers are correct to their own question.

This course quotes no hosted price. A price written into a document is stale within months and readers copy the number without the date, which is exactly the failure the course’s rules about numbers exist to prevent. Look the price up on the provider’s own page on the day you need it, put it in the input file with the date, and re-check it when the answer starts to matter.

The model, as a script you keep

RunnableAll tracks

cost-inputs-example.json

{
  "_readme": [
    "Purpose: the shape cost-model.py expects. Copy this to cost-inputs.json and replace",
    "every number with one you measured or looked up. Platform: all. Minimum memory: 8 GB.",
    "Assumes: idle_watts and load_watts come from your own wall meter or per-track power",
    "reading, output_tokens_per_second from your own benchmark at the concurrency named,",
    "price_per_kwh from your own bill, and purchase_price from your own receipt.",
    "Nothing here is a measurement of anything. The numbers are round on purpose so that",
    "the arithmetic is easy to follow by hand, and _placeholder is true so that the script",
    "refuses to write them into your lab notebook."
  ],
  "_placeholder": true,

  "tariff": {
    "currency": "your currency",
    "price_per_kwh": 0.3,
    "source": "your own electricity bill",
    "checked_on": "the date you read the bill"
  },

  "machines": [
    {
      "id": "workstation",
      "name": "the machine you serve from",
      "purchase_price": 3000,
      "lifetime_years": 4,
      "service_hours_per_year": 500,
      "powered_hours_per_year": 8760,
      "idle_watts": 30,
      "load_watts": 150,
      "how_measured": "wall meter, ten-minute average, idle and under a sustained load test"
    },
    {
      "id": "laptop",
      "name": "a second machine, to show what changes",
      "purchase_price": 2000,
      "lifetime_years": 3,
      "service_hours_per_year": 200,
      "powered_hours_per_year": 2000,
      "idle_watts": 10,
      "load_watts": 90,
      "how_measured": "same method, same meter, same day"
    }
  ],

  "workloads": [
    {
      "id": "chat-single",
      "machine": "workstation",
      "model": "Qwen3-8B",
      "quant": "Q4_K_M",
      "context_length": 8192,
      "concurrency": 1,
      "output_tokens_per_second": 40,
      "share_of_machine": 1.0
    },
    {
      "id": "chat-four-way",
      "machine": "workstation",
      "model": "Qwen3-8B",
      "quant": "Q4_K_M",
      "context_length": 8192,
      "concurrency": 4,
      "output_tokens_per_second": 100,
      "share_of_machine": 1.0
    },
    {
      "id": "chat-laptop",
      "machine": "laptop",
      "model": "Qwen3-8B",
      "quant": "Q4_K_M",
      "context_length": 8192,
      "concurrency": 1,
      "output_tokens_per_second": 20,
      "share_of_machine": 1.0
    }
  ],

  "api_comparison": {
    "price_per_million_output_tokens": null,
    "source": "the provider's own pricing page, which you read yourself",
    "checked_on": "the date you read it",
    "note": [
      "This course quotes no hosted price, because a price quoted in a document is out of",
      "date the week after it is written and readers copy the number without the date.",
      "Look up the price for the model you would actually use as a substitute, put it here",
      "with the date, and re-check it whenever the answer starts to matter."
    ]
  }
}

Download cost-inputs-example.json90 lines

RunnableAll tracks

cost-model.py

#!/usr/bin/env python3
"""Cost per million tokens for a local model service.

Purpose: turn measurements you made - watts at idle, watts under load, output tokens per
    second - plus your own electricity tariff and what the machine cost, into a cost per
    million tokens you can defend, and compare it against a hosted price you looked up
    yourself.
Platform: all (spark, strix, mac, nvidia). Pure Python, no dependencies.
Minimum memory: 8 GB, which is what the service being costed needs; this script needs
    almost nothing.
Assumes: a JSON file of inputs in the shape of cost-inputs-example.json. Every number in
    it is yours: this script invents nothing and has no price of any kind built into it.
    Where a value is missing, the report says so rather than guessing.

Usage: python3 cost-model.py --inputs cost-inputs.json
       python3 cost-model.py --inputs cost-inputs.json --labbook labbook.md
       python3 cost-model.py --inputs cost-inputs.json --json
"""
import argparse
import json
import sys
from datetime import date
from pathlib import Path


def die(message):
    print(f"cost-model: {message}", file=sys.stderr)
    sys.exit(1)


def require(mapping, key, where):
    if key not in mapping:
        die(f"{where} is missing the required field {key!r}.")
    return mapping[key]


def money(value):
    return f"{value:,.4f}"


def cost_workload(workload, machine, tariff, api_price):
    """Cost of one million output tokens for one workload on one machine.

    The arithmetic, written out so that you can check it by hand:

      hours per million tokens  H = 1e6 / tokens_per_second / 3600
      energy while working      E = load_watts / 1000 * H * tariff
      marginal energy           M = (load_watts - idle_watts) / 1000 * H * tariff
      capital                   C = purchase_price / (lifetime_years * service_hours) * H
      idle carried by the work  I = idle_watts / 1000 * (powered_hours - service_hours)
                                    * tariff / service_hours * H

    E is what the work costs in electricity. M is what it costs over leaving the machine
    switched on doing nothing, which is the honest number when the machine was going to be
    on anyway. C and I are the costs of ownership, and they both divide by the hours the
    machine actually spends generating: a machine used for an hour a day carries the same
    purchase price as one used all day, so its hour is dearer. That is the whole lesson.
    """
    where = f"workload {workload.get('id')!r}"
    machine_where = f"machine {machine.get('id')!r}"

    tps = float(require(workload, "output_tokens_per_second", where))
    if tps <= 0:
        die(f"{where}: output_tokens_per_second must be above zero.")

    price = float(require(machine, "purchase_price", machine_where))
    years = float(require(machine, "lifetime_years", machine_where))
    duty = float(require(machine, "service_hours_per_year", machine_where))
    idle_w = float(require(machine, "idle_watts", machine_where))
    load_w = float(require(machine, "load_watts", machine_where))
    powered = float(machine.get("powered_hours_per_year", duty))

    if years <= 0 or duty <= 0:
        die(f"{machine_where}: lifetime_years and service_hours_per_year must be above zero.")
    if powered < duty:
        die(f"{machine_where}: powered_hours_per_year cannot be below service_hours_per_year.")
    if load_w < idle_w:
        die(f"{machine_where}: load_watts is below idle_watts; check which is which.")

    share = float(workload.get("share_of_machine", 1.0))
    if not 0 < share <= 1:
        die(f"{where}: share_of_machine must be above zero and at most one.")

    hours = 1_000_000.0 / tps / 3600.0
    kwh = load_w / 1000.0 * hours
    marginal_kwh = (load_w - idle_w) / 1000.0 * hours

    energy = kwh * tariff
    marginal = marginal_kwh * tariff
    capital = price / (years * duty) * hours * share
    idle = idle_w / 1000.0 * (powered - duty) * tariff / duty * hours * share
    total = energy + capital + idle

    row = {
        "id": workload.get("id"),
        "machine": machine.get("id"),
        "model": workload.get("model"),
        "quant": workload.get("quant"),
        "context_length": workload.get("context_length"),
        "concurrency": workload.get("concurrency"),
        "output_tokens_per_second": tps,
        "hours_per_million_tokens": hours,
        "kwh_per_million_tokens": kwh,
        "marginal_kwh_per_million_tokens": marginal_kwh,
        "marginal_cost_per_million": marginal,
        "energy_cost_per_million": energy,
        "capital_cost_per_million": capital,
        "idle_cost_per_million": idle,
        "total_cost_per_million": total,
    }

    if api_price is None:
        return row

    # Utilisation is the variable that decides this, so the useful answer is not "which is
    # cheaper today" but "how many hours a year would this machine have to generate for
    # the two to cost the same". Rearranging total(duty) = api_price for duty:
    #
    #   total(duty) = E + (K / duty) - B      where
    #   K = H * share * (purchase_price / lifetime_years + idle_kW * tariff * powered)
    #   B = H * share * idle_kW * tariff
    #
    # so duty = K / (api_price - E + B). A denominator at or below zero means no amount of
    # use gets there: the electricity alone already costs more than the hosted price.
    k = hours * share * (price / years + idle_w / 1000.0 * tariff * powered)
    b = hours * share * idle_w / 1000.0 * tariff
    denominator = api_price - energy + b
    break_even_hours = k / denominator if denominator > 0 else None

    row.update({
        "api_price_per_million": api_price,
        "cheaper_on_total_cost": "local" if total < api_price else "hosted",
        "cheaper_on_marginal_cost": "local" if marginal < api_price else "hosted",
        "break_even_service_hours_per_year": break_even_hours,
    })
    return row


def render(rows, currency, tariff, api_meta):
    print(f"Cost per million output tokens, in {currency}, at {tariff} per kWh")
    print()
    header = (f"{'workload':<22}{'machine':<10}{'kWh/Mtok':>10}{'energy':>10}"
              f"{'capital':>10}{'idle':>10}{'total':>10}{'marginal':>10}")
    print(header)
    print("-" * len(header))
    for r in rows:
        print(f"{str(r['id'])[:21]:<22}{str(r['machine'])[:9]:<10}"
              f"{r['kwh_per_million_tokens']:>10.3f}"
              f"{money(r['energy_cost_per_million']):>10}"
              f"{money(r['capital_cost_per_million']):>10}"
              f"{money(r['idle_cost_per_million']):>10}"
              f"{money(r['total_cost_per_million']):>10}"
              f"{money(r['marginal_cost_per_million']):>10}")
    print()

    if not any("api_price_per_million" in r for r in rows):
        print("No hosted price was supplied, so no comparison was made. Look one up on the")
        print("provider's own pricing page, write it and the date into api_comparison, and")
        print("run this again.")
        return

    source = (api_meta or {}).get("source", "not stated")
    checked = (api_meta or {}).get("checked_on", "not stated")
    print(f"Against a hosted price of {money(rows[0]['api_price_per_million'])} per million "
          f"output tokens ({source}, checked {checked}):")
    print()
    for r in rows:
        if "api_price_per_million" not in r:
            continue
        print(f"  {r['id']}")
        print(f"      cheaper counting everything:        {r['cheaper_on_total_cost']}")
        print(f"      cheaper counting only the extra kWh: {r['cheaper_on_marginal_cost']}")
        hours = r["break_even_service_hours_per_year"]
        if hours is None:
            print("      break-even: never. The electricity this workload burns already")
            print("                  costs more than the hosted price.")
        elif hours > 8760:
            print(f"      break-even: about {hours:,.0f} generating hours a year, which is")
            print("                  more hours than a year has. At these numbers the")
            print("                  machine cannot be used enough to pay for itself on")
            print("                  this workload alone.")
        else:
            print(f"      break-even: about {hours:,.0f} generating hours a year, out of")
            print("                  the 8,760 a year contains.")
        print()


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--inputs", required=True, help="JSON file of your own measurements")
    parser.add_argument("--labbook", default=None,
                        help="append one JSON line per workload to this file")
    parser.add_argument("--json", dest="as_json", action="store_true",
                        help="print the rows as JSON instead of a table")
    args = parser.parse_args()

    path = Path(args.inputs)
    if not path.is_file():
        die(f"no such file: {path}")
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as exc:
        die(f"{path} is not valid JSON: {exc}")

    placeholder = bool(data.get("_placeholder", False))
    tariff_block = require(data, "tariff", "the input file")
    currency = tariff_block.get("currency", "your currency")
    tariff = float(require(tariff_block, "price_per_kwh", "tariff"))

    machines = {m["id"]: m for m in require(data, "machines", "the input file")}
    if not machines:
        die("the input file lists no machines.")

    api_meta = data.get("api_comparison") or {}
    api_price = api_meta.get("price_per_million_output_tokens")
    api_price = None if api_price is None else float(api_price)

    rows = []
    for workload in require(data, "workloads", "the input file"):
        machine_id = require(workload, "machine", f"workload {workload.get('id')!r}")
        if machine_id not in machines:
            die(f"workload {workload.get('id')!r} names machine {machine_id!r}, "
                "which is not in the machines list.")
        rows.append(cost_workload(workload, machines[machine_id], tariff, api_price))

    if args.as_json:
        print(json.dumps({"currency": currency, "price_per_kwh": tariff, "rows": rows},
                         indent=2, sort_keys=True))
    else:
        render(rows, currency, tariff, api_meta)

    if placeholder:
        print("WARNING: this input file is still the shipped example. Every number in it is")
        print("         a placeholder chosen to make the arithmetic visible, not a")
        print("         measurement of anything. Replace them with your own, then set")
        print('         "_placeholder" to false.')

    if not args.labbook:
        return
    if placeholder:
        die("refusing to write placeholder numbers into the lab notebook.")
    out = Path(args.labbook)
    with out.open("a", encoding="utf-8") as handle:
        for r in rows:
            record = dict(r)
            record.update({"lab": "part-23/capacity-planning", "currency": currency,
                           "price_per_kwh": tariff, "recorded": date.today().isoformat()})
            handle.write(json.dumps(record, sort_keys=True) + "\n")
    print(f"recorded {len(rows)} row(s) in {out}")


if __name__ == "__main__":
    main()

Download cost-model.py253 lines

Copy the example, replace every number with one you measured or looked up, set _placeholder to false, and run it. The example values are round on purpose so that you can follow the arithmetic by hand, and the script refuses to write them into your lab notebook while that flag is still set, because an invented number recorded once becomes a measurement forever afterwards.

RunnableAll tracks

your own numbers, your own answer

cp cost-inputs-example.json cost-inputs.json

python3 cost-model.py --inputs cost-inputs.json

When you have added a hosted price, the script also reports the number that actually settles the argument: how many generating hours a year the machine would have to work for the two to cost the same. If that figure is larger than the hours in a year, the machine cannot pay for itself on that workload, and that is a useful thing to know plainly rather than to feel vaguely.

RunnableAll tracks

record it, once it is real

python3 cost-model.py --inputs cost-inputs.json --labbook labbook.md

Pending validationCost per million output tokens — your own recording sheet

Workload	Concurrency	tokens/s aggregate	Watts idle	Watts load	kWh per million	Total cost per million
Chat model, single stream	—	—	—	—	—	—
Chat model, planned concurrency	—	—	—	—	—	—
Coder model, planned concurrency	—	—	—	—	—	—

your machine: track, chip, memory, and how the power was measured, your operating system and version · the engine and how it was started, including the slot count the engine version, from its own --version output · the model behind each alias, the quantisation of each · 8,192 tokens of context · the date you measured it

Empty on purpose: every cell is a measurement you take. Fill the watts columns from a wall meter and note in the hardware field that you did, because a component reading and a wall reading are different numbers and six months later you will not remember which one this was.

Calculate cost using useful completed work

Define whether the denominator is input tokens, output tokens, total tokens or completed tasks. Include failed attempts and retries in resource costs even if their outputs are not useful. Two models can have different cost per token and reverse order when measured per accepted answer because one generates more text or needs more repairs.

Separate capital cost, idle power, active power and operator time. State the utilisation assumption used to amortise fixed costs. A machine already owned for another purpose and a new machine purchased solely for serving represent different economic decisions; report the scenario rather than hiding it in one number.

Use measured power over a representative interval and completed-request counts from the same interval. Extrapolate cautiously to changing demand. Include a sensitivity calculation for lower utilisation and a stricter latency target, both of which can reduce usable capacity. The final comparison should say what workload, availability and quality the cost buys. Token price alone is not an application budget.

Recap

Three costs: the energy the work uses, the energy the machine uses anyway, and the machine itself, which on a home service is usually the largest. Measure watts twice, at idle and under a sustained load, with a wall meter as the reference and the software reading for the shape over time. Hours per million tokens is the bridge between throughput and cost, and it divides every term, so throughput improvements are cost improvements. Capital and idle both divide by the hours the machine actually generates, which means utilisation is the variable that decides whether local is cheap. Plan concurrency from the point where aggregate throughput flattens, check the memory at that point rather than at rest, keep headroom, and enforce the plan with a per-key rate limit. Compare with a hosted price for the model you would actually use, on like tokens, and say whether you are quoting the total or the marginal cost. Then keep the input file, because next year the tariff will have changed and you will want to re-run it rather than rebuild it.

Check your understanding

Question 1. Two identical machines run the same model at the same speed and the same tariff. One generates for a few hours a week, the other most of the day. Why is the cost per million tokens so different?

• The busier machine is more efficient per token because it stays warm
• The purchase price and the idle electricity are both divided by the hours the machine actually generates, so a lightly used machine spreads the same fixed costs across far fewer tokens
• The busier machine draws less power under load
• They are not different; cost per token depends only on watts and throughput

Show the answer and why

Answer: The purchase price and the idle electricity are both divided by the hours the machine actually generates, so a lightly used machine spreads the same fixed costs across far fewer tokens

Energy per token is the same on both. What differs is everything that is charged per hour of existence rather than per token: the amortised purchase price and the electricity burned while switched on and idle. This is why the way to make a local service cheap per token is to use it more.

Question 2. You read power from nvidia-smi on a desktop with a discrete card and use it as the load figure in your cost model. What is wrong with the result?

• Nothing; the card is what draws the power
• It is systematically too low: the reading covers the card, not the processor, board, disks, fans and the power supply losses, all of which the wall meter sees
• It is systematically too high because it reports a peak
• It is fine for cost but not for the dashboard

Show the answer and why

Answer: It is systematically too low: the reading covers the card, not the processor, board, disks, fans and the power supply losses, all of which the wall meter sees

Software readings report a component. On a unified-memory machine that component is a large share of the total; on a desktop with a discrete card it is not. Use the software reading for the shape over time and a plug-in meter for the number that goes into the model.

Question 3. Which of these are legitimate reasons a like-for-like comparison with a hosted API can be misleading? Select all that apply.

• Comparing your small local model against a hosted flagship rather than against what you would actually use instead
• Counting only output tokens when the workload is mostly long inputs
• Quoting your marginal electricity cost while making a decision about whether to buy the hardware
• Measuring your throughput at the concurrency you actually serve

Show the answer and why

Answer: Comparing your small local model against a hosted flagship rather than against what you would actually use instead, Counting only output tokens when the workload is mostly long inputs, Quoting your marginal electricity cost while making a decision about whether to buy the hardware

The first three each bias the answer in a predictable direction. Measuring at the concurrency you actually serve is not a distortion; it is the correction that makes the comparison meaningful, because single-stream throughput understates what a batching engine does for you.

Question 4. A configuration serving four concurrent conversations fits comfortably in the morning and fails in the evening, with no change to the number of users. What is the most likely reason?

• The model file has become corrupted
• Key-value cache use scales with concurrency multiplied by how far each conversation has got, so the same four conversations occupy more memory as they lengthen
• The engine leaks memory on every request
• The electricity supply is less stable in the evening

Show the answer and why

Answer: Key-value cache use scales with concurrency multiplied by how far each conversation has got, so the same four conversations occupy more memory as they lengthen

Nothing changed except time. This is the plainest example of why the memory check belongs at the planned concurrency and the expected context length rather than at rest, and it is the fault the challenge at the end of this part asks you to reproduce and diagnose.

Check my answersClear