Capstone 1: Hardware and Model Plan

Validated on: written from the documentation cited above and from Parts 4, 5 and 18; not yet validated on hardware on any track. This project produces a document and some arithmetic; the measurements it will be checked against are the ones you take later.

Objective

By the end of this project you will have a plan for the platform you are about to build, in which every memory figure is arithmetic somebody else could redo, every model has a role and a licence, every link says what it carries, and every risk names a check that would catch it before you had built anything on top of it.

An hour spent here is the cheapest hour in the capstone. Almost every unpleasant surprise in the next five deliverables is visible in this arithmetic: the model that does not fit at the context you wanted, the second model you forgot was resident, the link you assumed was fast, the download that does not fit on the volume you meant to put it on.

Requirements

Every track needs the machine and environment sections of the lab notebook from Part 5, the shortlist from Part 4’s lab, the link measurements from Part 18’s lab if you took them, the model reference and the hardware reference. About an hour, all of it attended, and no download at all: this deliverable decides what to download rather than doing it.

Track S — NVIDIA DGX Spark

All of it applies. The memory is unified, so there is one budget rather than two, and the whole of it is available to the model in principle. The allowance to argue about on this track is the operating system’s share on a 128 GB machine, which Part 4 puts above ten gigabytes; the arithmetic is unforgiving at the top end, where the appeal of this machine is running something that only just fits.

Track X — AMD Ryzen AI Max+ 395

All of it applies, with one extra column in your plan. On this track the memory split between the processor and the graphics device is configurable, so your budget has to state what you set it to and when, and the plan should say what you would change it to if a model did not fit. Note also which backend each model will run on, because the Vulkan and ROCm paths do not have the same working-memory appetite.

Track M — Apple silicon

All of it applies. Memory is unified here too, and the allowance that surprises people is the one for everything else the Mac is doing: this is usually somebody’s working machine as well as their inference machine. If you intend to convert or quantise models locally, mlx-lm’s conversion documentation is the reference for what the MLX path produces, and the plan should record the converted file’s size rather than the original repository’s.

Track N — NVIDIA desktop or laptop

All of it applies, and this is the track where the plan has to keep two budgets rather than one: device memory on the card and system memory on the host. A model that fits in system memory and not in device memory is a different situation from one that fits in neither, and the fallbacks are different, so the plan’s fit table needs a column for each.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/capstone"
cd "$LAB_DIR"
pwd
test -f "hardware-and-model-plan-template.md"

Expected result: pwd ends in capstone and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Write the requirement before you look at any hardware

Open the template and fill in section 1 first.

Fragment — not complete on its own

hardware-and-model-plan-template.md

# Hardware and model plan — <the platform you are building>

<!--
Purpose: the deliverable of Capstone 1. The document that says which machines you have,
         what each one will do, which models will run where, what each will cost in
         memory at the context length you will actually configure, how the models get
         onto the machines and where the plan is most likely to be wrong.
Platform: all
Minimum memory: 8 GB; this is arithmetic and a document, not a workload
Assumes: the machine and environment sections of labbook.md from Part 5, the shortlist
         from Part 4's lab, and the model reference at /reference/models/ for parameter
         counts, layer counts and key-value head counts.

Fill in every angle-bracket field and delete every comment block, including this one.
A field you cannot fill is a field to write "not recorded" in; the gap is information.
Every memory figure here is arithmetic, not a measurement, and section 7 is where you
say so and later record what the arithmetic got wrong.
-->

**Author:** <you> · **Date:** <YYYY-MM-DD> · **Tracks in play:** <S, X, M, N>

---

## 1. What this platform is for

<Three or four sentences. Who uses it, for what, and how often. Then the requirement the
plan has to meet: how many people at once, the longest prompt you expect, the longest
answer, and whether anybody is waiting for the first token or for the whole answer.>

<Write this before you look at any hardware. A plan chosen first and justified afterwards
is a purchase, not a design.>

## 2. The machines and their roles

| Machine | Track | Chip or accelerator | Memory | Memory kind | Storage free | Roles | Why this role |
| --- | --- | --- | --- | --- | --- | --- | --- |
| <name> | <S/X/M/N> | <as your system reports it> | <GB> | <unified / discrete VRAM + host> | <GB> | <prefill, decode, router, cache, storage, agent> | <one sentence> |

<Part 18 named six roles. Most home machines hold several and one machine may hold all of
them. The "why" column carries the section: "it has the most memory and the least
arithmetic throughput, so it decodes" is a reason; "it is the newest" is not.>

**Machines I do not have but the plan assumes:** <none, or name them; a plan that quietly
assumes a machine you have not bought is the commonest way this document goes wrong.>

## 3. The memory budget, per model, per machine

<One block per model you intend to run. Every figure is arithmetic from Part 4's lesson on
choosing a model for a memory budget, not a number from a forum post.>

### <model id from the course model reference>

| Field | Value | Where it comes from |
| --- | --- | --- |
| Total parameters | <n> B | model card |
| Active parameters | <n> B | model card; equal to total for a dense model |
| Layers | <n> | model card or config.json |
| Key-value heads | <n> | config.json, `num_key_value_heads`; not the query head count |
| Head dimension | <n> | config.json |
| Quantisation | <e.g. Q4_K_M, MXFP4, BF16> | your choice |
| Bytes per parameter | <from the bytes-per-parameter table> | Part 4 |
| **Weight memory** | <total parameters × bytes per parameter> GB | arithmetic |
| Context length you will configure | <n> tokens | your choice, not the trained maximum |
| Bytes per element in the cache | <2 for FP16, 1 at 8-bit, about 0.5 at 4-bit> | your engine's cache type |
| **Bytes per token of cache** | 2 × <layers> × <kv heads> × <head dim> × <bytes> = <n> | arithmetic |
| **Cache at your context** | <bytes per token × context length> GB | arithmetic |
| Concurrent conversations planned | <n> | from section 1 |
| **Cache for all of them** | <cache at context × conversations> GB | arithmetic |
| **Subtotal for this model** | <weights + cache> GB | arithmetic |

<Sliding-window and hybrid-attention models make the per-token figure an upper bound.
Say which of your models are in that category and that the real figure will be lower.>

### Headroom, per machine

| Machine | Total memory | Operating system and applications | Engine working memory | Cache growth allowance | Second model | **Left for weights** |
| --- | --- | --- | --- | --- | --- | --- |
| <name> | <GB> | <GB> | <GB> | <GB> | <GB, or none> | <GB> |

<Part 4 named four allowances and this table is them. The one people forget is the second
model: an embedding model, a reranker or a draft model is resident whenever the main model
is, and it does not become free because it is small.>

### Does it fit?

| Machine | Model | Subtotal | Left for weights | Fits? | If not, what changes |
| --- | --- | --- | --- | --- | --- |
| <name> | <model> | <GB> | <GB> | <yes / no> | <smaller quantisation, shorter context, fewer concurrent conversations, a different model> |

<A "no" here is the most useful row in the document, because you found it with arithmetic
in an afternoon rather than with an out-of-memory error at three in the morning.>

## 4. Models per role

| Role | Model | Why this one | Licence | Gated? | Fallback if it disappoints |
| --- | --- | --- | --- | --- | --- |
| Chat | <id> | <one sentence> | <from the model card> | <yes/no> | <id> |
| Coding | <id> | <one sentence> | <from the model card> | <yes/no> | <id> |
| Embedding | <id> | <one sentence> | <from the model card> | <yes/no> | <id> |
| <other> | <id> | <one sentence> | <from the model card> | <yes/no> | <id> |

<A licence read from the model card, not from a summary of it. A fine-tune you make is a
derivative work and inherits the base model's terms.>

## 5. The network plan

| From | To | Class | Rated | Measured throughput | Measured round trip | Carries |
| --- | --- | --- | --- | --- | --- | --- |
| <machine> | <machine> | <2.5 GbE, Thunderbolt, QSFP, Wi-Fi> | <from the box> | <from Part 18's lab> | <from Part 18's lab> | <per-token / per-request / load-time / client> |

<The last column decides everything. A link carrying per-token traffic has to be fast; a
link carrying load-time traffic only delays the start. A rated speed from a box is not a
measurement, which is why there are two columns.>

**Names and addressing:** <how machines find each other; no addresses in this document if
you intend to share it.>

**If you have one machine:** <say so here, and say which of the roles above would move to
a second machine first and which link class it would need. That sentence is what makes
this a plan rather than an inventory.>

## 6. Download and storage plan

| Model | Repository | Files | Size | Destination | Verified how | Downloaded on |
| --- | --- | --- | --- | --- | --- | --- |
| <id> | <publisher/repo> | <the quantisation file or the whole repo> | <GB> | <the shared library path> | <hash recorded beside the file> | <YYYY-MM-DD> |

**Total to download:** <GB> · **Free space on the library volume:** <GB> · **Headroom
after:** <GB>

**Order:** <what to download first, so that you can start working before the largest file
finishes.>

**One authoritative copy:** <where the library lives and how the other machines read it;
Part 18's shared mount, a copy per machine, or a single machine.>

## 7. Risks, and what would tell you early

| # | Risk | How likely | What it costs | The check that catches it early |
| --- | --- | --- | --- | --- |
| 1 | <e.g. the 30B model does not fit at the context I want on the 64 GB machine> | <high/medium/low> | <what you would have to change> | <the startup log's cache size, before any load test> |
| 2 | <e.g. the engine's build for my accelerator is unqualified on this chip> | | | |
| 3 | <e.g. the link I planned to carry per-request traffic measures far below its rating> | | | |

<Three is a minimum, not a target. Each risk names a check you can run before you have
built anything, which is the point of doing the plan first.>

## 8. What this plan does not know

<The figures above are arithmetic. List what will be measured later and what you have
taken from a vendor page without verifying: bandwidth, the engine's real working-memory
appetite, whether a quantisation you have not tried holds up on your tasks, whether the
NPU or the alternative backend does anything for you.>

<Come back to this section after Capstone 6 and add one line per figure that the
measurement contradicted. That comparison, between what the arithmetic predicted and what
the machine did, is the single most valuable paragraph in the whole capstone.>

Download hardware-and-model-plan-template.md157 lines

RunnableAll tracks

start from the template

cp hardware-and-model-plan-template.md hardware-and-model-plan.md

Who uses this platform, for what, how often, how many at once, the longest prompt you expect, the longest answer, and whether anybody is waiting for the first token or for the whole answer. Five minutes, and it is the section everything else is judged against.

Writing it first matters more than it sounds. A requirement chosen after the hardware is not a requirement, it is a description of what the hardware happened to do, and every subsequent decision in the plan becomes unfalsifiable.

✕Common mistake

The commonest failure in this section is a requirement with no context length in it. “Ten concurrent users” is not a requirement; “ten concurrent users at a context of 8,192 tokens each, of whom three might be at 32,768” is, because only the second one can be turned into a memory figure.

2. Inventory the machines and give each one a role

Fill in section 2 from your notebook, one row per machine you actually own. Part 18 named six roles: prefill, decode, router, cache, storage and agent. Most home machines hold several and one machine may hold all six.

The column that carries the section is the last one. “It has the most memory and the least arithmetic throughput, so it decodes” is a reason. “It is the newest” is not, and a plan full of reasons like the second kind will not survive contact with the first measurement.

If you have one machine, say so plainly in that section and give it every role. That is a design, not the absence of one, and everything below still applies.

Then add the row people skip: machines the plan assumes and you do not have. A plan that quietly assumes a second machine is the commonest way this document goes wrong, and it usually happens in the network section three pages later.

3. Do the memory arithmetic, per model

This is the heart of the deliverable and it is the part of Part 4 that the whole course depends on. Two figures per model, both arithmetic.

Weights. Parameter count times bytes per parameter, with the bytes-per-parameter figure from Part 4’s table for the quantisation you intend to serve. At four-bit K-quants the shortcut Part 4 gives is roughly six-tenths of a gigabyte per billion parameters; at eight bits it is a little over one. For a mixture-of-experts model this uses the total parameter count, because all of the experts are resident even though only some are computed with per token.

Key-value cache. The per-token formula from Part 4, with the counts read from the model’s own configuration rather than guessed:

Pseudocode — not a real command

bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_element

  2                 one key and one value
  layers            the model's layer count
  kv_heads          the KEY-VALUE head count, not the query head count
  head_dim          the per-head dimension
  bytes_per_element 2 for FP16 or BF16, 1 at 8-bit, about 0.5 at 4-bit

cache for one conversation = bytes per token x context length you configure
cache for the service      = that, times the conversations you allow at once

The trap is kv_heads. Grouped-query attention means the key-value head count is usually a small fraction of the query head count, and using the wrong one inflates the cache estimate several times over. It is in config.json as num_key_value_heads.

▤Memory budget

Use the context length you will configure, not the model’s trained maximum. A model advertised at a very long context does not cost that memory unless you ask the engine to allocate it, and asking for it when nobody needs it is the commonest self-inflicted out-of-memory error in this course. Part 7’s reality check is entirely about this.

Here is one model’s budget worked through, as an example of the shape yours should take rather than a figure to copy. Qwen3-8B, a dense model with 36 layers, 8 key-value heads and a head dimension of 128, at a four-bit K-quant, with a 32,768-token context allocated for a single conversation, on a 64 GB machine.

Worked example, not your answer: Qwen3-8B at Q4_K_M, 32k context, on a 64 GB machine

Weights (Q4_K_M, about 0.61 GB per billion)

5 GB

Key-value cache, 32k context, FP16

4.8 GB

Operating system and applications

8 GB

Engine working memory

2 GB

Free

44.2 GB

Total

64 GB

Every segment is arithmetic, not a measurement: weights from parameter count times bytes per parameter, cache from the per-token formula times the configured context, and the two allowances from Part 4's headroom rules. Redo it with your own model, your own quantisation and your own context; the free space is what tells you whether a second model fits beside it.

Notice what the picture makes obvious and a table hides: at this context the cache is almost as large as the weights. Halving the context nearly halves the total, and that is usually the cheapest change available when something does not fit.

✱Hardware note

Models with sliding-window or hybrid attention do not spend the full per-token figure on every layer, so the formula gives an upper bound rather than an estimate for them. Say in your plan which of your models are in that category and that their real figure will be lower; then measure it later rather than arguing about it now.

4. Add the headroom, then answer “does it fit”

Part 4 named four allowances and the template has a row for each: the operating system and whatever else the machine does, the engine’s own working memory, room for the cache to grow to what you configured rather than what you expect, and any second model.

The one people forget is the fourth. An embedding model for retrieval, a reranker, a small router model or a draft model for speculative decoding is resident whenever the main model is, and it does not become free because it is small. If your service has an embedding endpoint, that model is in the budget.

Then fill in the fit table, one row per model per machine, and let it say no where it says no. A “no” found here costs an afternoon of rethinking; the same “no” found later costs a failed load at the moment you first showed the service to somebody.

For each “no”, write what changes: a smaller quantisation, a shorter context, fewer concurrent conversations, a different model, or a machine you do not have. Those five are the whole space of answers.

5. Choose the models per role, with their licences

One row per role in section 4 of the template: chat, coding, embedding, and whatever else your requirement implies. For each, the model, one sentence on why it and not the alternative, the licence read from the model card itself, whether it is gated behind an acceptance step, and the fallback if it disappoints you.

Two rules from earlier parts apply and are worth restating because they are easy to skip in a planning document. The licence comes from the model card, not from a summary of it or from a repackaged copy, and the model reference tells you which of the course’s models need their terms confirmed on the card. And a fine-tune you make later is a derivative work, so whatever the base model’s terms allow is what your improved model in Capstone 4 will inherit.

✦Tip

Give each role a permanent name now, and use the same names in Capstone 2’s gateway: chat, coder, embed. Part 9’s project made the case that a stable name is worth more than the model behind it, because it lets you replace the model without touching a single client. The plan is where those names are decided.

6. The network plan, and what each link carries

Fill in section 5 with one row per link, and copy the measured figures from Part 18’s lab rather than the rated speed from a box. The template has separate columns for the rating and the measurement because the difference between them is frequently large and always interesting.

The last column is the one that matters. Part 18 named this annotation as the thing home clusters most often get wrong:

• per-token traffic has to be fast, and there is no configuration that hides a slow link carrying it;
• per-request traffic has to move one request’s key-value cache in less time than the prefill it replaced, which is an arithmetic question you can answer today;
• load-time traffic only has to be wide enough that starting up is not annoying;
• client traffic is somebody’s laptop on the house network and is almost never the problem.

If you have one machine, this section becomes the client link and the paragraph about machines you do not have. Write which role would move to a second machine first and which link class it would need. That sentence is what the rubric asks for, it is what the deployment deliverable will test, and it is much easier to write now than at the end.

⛨Security

If you intend to show this plan to anybody, keep addresses and machine names out of it. Write “the decode machine” and “the house network”; the plan is a design document, not a network inventory, and it loses nothing by being unable to tell a reader where to connect.

7. The download and storage plan

Section 6, one row per model file: the repository, which files you actually need, the size, where it lands, how you will verify it, and the date.

Two things to decide rather than discover. First, whether there is one authoritative copy of the library that other machines read, or a copy per machine; Part 18’s shared mount and Part 23’s estate manifest both assume you have chosen. Second, the order, so that you can start working while the largest file is still coming down.

The download itself belongs to Capstone 2, but the command shape is worth writing into the plan so that the fetch is reproducible:

Fragment — not complete on its own

hf download <publisher>/<repository> \
  --include "<the quantisation file you chose>" \
  --local-dir "$MODELS/<a directory you name>"

Add the total against the free space on the volume, and the headroom afterwards. A model library that fills its volume is how a machine that was working stops working, and the arithmetic takes a minute.

⌚Version note

Record the revision as well as the repository. Model repositories are updated: files are replaced, quantisations are re-made, chat templates are corrected. A plan that names a repository without a revision cannot be reproduced later, and Part 23’s estate manifest exists to keep the pairing of repository, revision and file hash together.

8. Write the risks, each with the check that catches it early

Three at minimum, in section 7, and each one names a check you can run before you have built anything on top of it. That last requirement is what separates this from a list of worries.

Good risks look like this: the 30B model does not fit at the context I want on the 64 GB machine, and the check is the key-value cache size the engine logs at startup, which costs one launch and no load test. The backend for my accelerator is unqualified on this chip, and the check is a single generation with the alternative backend before the plan depends on the first one. The link I planned to have carry per-request traffic measures far below its rating, and the check is already in my notebook from Part 18.

Bad risks look like “memory might be tight” and “the model might not be good enough”. They name no measurement and no threshold, so nothing can ever discharge them.

9. Say what the plan does not know

Section 8, and it is not a disclaimer. Every figure above is arithmetic; the machine has not yet been asked. List what will be measured later, what you have taken from a vendor page without verifying, and which of your assumptions you are least sure of.

Then leave the section open. After Capstone 6 you will come back and add one line per figure that the measurement contradicted, and that comparison between prediction and observation is the single most valuable paragraph in the whole capstone.

Turn assumptions into executable validation gates

For every planned model role, identify a concrete checkpoint and representation, calculate the memory terms and name the command from the relevant earlier lab that will test the estimate. Copy your actual values into the worksheet; angle-bracket examples are configuration fragments, not commands to paste literally.

Separate existing hardware from purchases and required capabilities from optional improvements. For each uncertain assumption, state the earliest low-cost check: device visibility before a large download, a short feature probe before load testing, and a network benchmark before distributed serving. Keep model licence and source identity with the download plan.

Review the plan against a single-machine alternative and explain why any additional machine is needed. Define the fallback if a model fails the quality or memory gate. The deliverable is complete when each requirement maps to a role, each role maps to an artefact and each unmeasured assumption has a named validation step. Preserve the original plan before hardware validation so the report can show which assumptions changed. An honest not-yet-measured entry is more useful than an apparently precise estimate presented as an observed capacity.

Validation

• hardware-and-model-plan.md has every section filled in and no angle-bracket fields left.
• Every weight figure is parameter count times a bytes-per-parameter value that appears in Part 4’s table, and the multiplication is shown.
• Every cache figure names the layer count, the key-value head count, the head dimension, the element size and the context length it assumed, and the product is shown.
• The fit table has a row for every model on every machine you intend to run it on, and every “no” has a stated change beside it.
• Every model in section 4 has a licence taken from its model card and a fallback.
• Every link in section 5 has a measured figure or the words “not measured”, and the column saying what it carries is filled in for all of them.
• The download total is compared with the free space on the destination volume.
• Section 7 has at least three risks and every one names a check.
• If you have one machine, the plan names the role that would move first, the link class it would need and the measurement behind it.

Expected outcome

A document of a few pages that somebody else could act on, and a small number of uncomfortable discoveries. The table below is the shape of the plan rather than a measurement of anything: it is what the sections commit you to, and the evidence column in your own copy holds the arithmetic or the notebook label behind each one.

Pending validationWhat a finished plan commits to — the shape, to be filled in with your own

Section	What it commits to	Evidence
Requirement	concurrent users, prompt and answer lengths, a context length, a latency budget	stated before any hardware was considered
Machines and roles	which machine does what, and why	pending
Memory budget	weights and cache per model at a named context, with the arithmetic shown	pending
Headroom	four allowances per machine, including any second model	pending
Fit	a yes or no per model per machine, and what changes on a no	pending
Models per role	a model, a reason, a licence and a fallback for each role	pending
Network	every link annotated with what it carries, measured where possible	pending
Downloads	total size against free space, with the order and the verification	pending
Risks	three or more, each with a check that runs before anything depends on it	pending

the machines in your own plan · the engines the plan chooses to be recorded when the service is built · the models in your own plan, as the plan states · 8,192 tokens of context · 2026-09-09

This table describes a document rather than measuring anything. Every row marked pending becomes a measurement in Capstone 2 or Capstone 3, and the plan is where you say what you expect it to be.

Two plans count as finished and they look different. One says “everything fits on the machine I have, with room for a second model, and here is the context length that makes it so”. The other says “the model I wanted does not fit at the context I wanted, so the plan serves a smaller one at that context and the larger one at a shorter one, and here is the arithmetic”. The second is the more useful document.

Troubleshooting

The cache figure is enormous. Check num_key_value_heads against the query head count. Grouped-query attention means the two differ by a large factor in most current models, and using the query count is the single commonest arithmetic error in this part.

The numbers do not match what the engine reports at startup. Expect a gap and record it. Engines allocate in blocks, reserve working memory, and may pad the cache; the arithmetic is a prediction and the startup log is the measurement. A gap of a few per cent is normal, a gap of several times is a wrong head count or a wrong element size.

You cannot find the layer or head counts. They are in the model repository’s config.json, and the model reference carries them for the course’s models. If a model you have chosen has neither, that is itself worth writing in section 8.

Everything fits and the plan feels too easy. Check the second model, the concurrency and the context. A budget for one conversation at a short context on a large machine will always fit; the requirement in section 1 is what makes the arithmetic bite.

Nothing fits. Work down the five answers in order: smaller quantisation, shorter context, fewer concurrent conversations, a different model, a machine you do not have. Reaching the fifth is a legitimate result and it is what the section on machines you do not have is for.

Cleanup

Nothing to clean up: this project starts nothing and downloads nothing. Keep hardware-and-model-plan.md beside your notebook. Capstone 2 is built to it, Capstone 3 measures against it, and Capstone 6 compares its arithmetic with what the machines actually did.

What you learned

• Arithmetic before purchase, and before configuration. The whole plan is two formulas and four allowances, and it answers in an hour the questions that otherwise take a week of failed loads to answer.
• A budget without a context length is not a budget. The cache is frequently comparable in size to the weights, and the context you configure is the term you control most cheaply.
• The second model is always forgotten. An embedder, a reranker or a draft model is resident with the main one, and a plan that omits it is wrong by exactly its size.
• The link annotation is the network plan. What a link carries decides whether its speed matters at all, and Part 18 named it as the thing home clusters most often get wrong.
• A risk without a check is a worry. The value of the risk table is entirely in its last column.
• A “no” in the fit table is a success. It is the cheapest possible form of that discovery, and finding one is a sign the plan is being honest about the requirement.

Record in the notebook: the fit table as it stands today, the two figures per model that produced it, and one sentence predicting which row you expect to be wrong. In Capstone 6 you will check.

Check your understanding

Question 1. Your model card gives 32 attention heads and 8 key-value heads. Which figure goes into the per-token cache formula?

• 32, the attention head count
• 8, the key-value head count
• The sum, 40
• Whichever the engine reports at startup

Show the answer and why

Answer: 8, the key-value head count

Grouped-query attention shares each key-value head across several query heads, and only the key-value state is cached. Using 32 here would overstate the cache by a factor of four, which is the commonest arithmetic error in this part and the reason the formula names the field as num_key_value_heads.

Question 2. A mixture-of-experts model has 30 billion total parameters and 3 billion active. Which count sizes the weight memory?

• The active count, since that is what is computed with per token
• The total count, since all the experts have to be resident
• Half of the total
• Neither; mixture-of-experts models are sized by their layer count

Show the answer and why

Answer: The total count, since all the experts have to be resident

Active parameters tell you about arithmetic per token, which affects speed. Memory holds every expert, because any of them may be selected for the next token. This is why such a model is fast for its size and not small for its size.

Question 3. Which of these belong in the plan? Select all that apply.

• The context length each memory budget assumes
• The embedding model that serves your retrieval endpoint
• The measured throughput of every link, or the words "not measured"
• The rated speed printed on the network card, used as the link figure

Show the answer and why

Answer: The context length each memory budget assumes, The embedding model that serves your retrieval endpoint, The measured throughput of every link, or the words "not measured"

A rated speed is not a measurement, which is why the template has separate columns for the two and why the deployment deliverable will not accept a rating as evidence. The other three are all things whose absence has caused a real failure for somebody working through this course.

Question 4. Your fit table says a 30B model does not fit at the context you wanted. What is the cheapest change to try first?

• Buy a second machine
• Shorten the configured context, because at long contexts the cache is often comparable to the weights
• Switch to a different model family
• Reduce the number of layers

Show the answer and why

Answer: Shorten the configured context, because at long contexts the cache is often comparable to the weights

The worked example on this page shows the cache approaching the weights at a 32k context. Halving the context nearly halves that segment and costs nothing but a shorter conversation. The five answers, in rough order of cost, are: smaller quantisation, shorter context, fewer concurrent conversations, a different model, a machine you do not have.

Question 5. True or false: the requirement in section 1 should be written after you know what your hardware can do, so that it is realistic.

• True
• False

Show the answer and why

Answer: False

A requirement chosen to match what the hardware happened to do cannot be failed, so it cannot be used to judge anything. Writing it first is what makes the fit table meaningful, and discovering that the hardware does not meet the requirement is a result worth having in an hour rather than a month.

Check my answersClear