# Hardware and model plan — **Author:** · **Date:** · **Tracks in play:** --- ## 1. What this platform is for ## 2. The machines and their roles | Machine | Track | Chip or accelerator | Memory | Memory kind | Storage free | Roles | Why this role | | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | | **Machines I do not have but the plan assumes:** ## 3. The memory budget, per model, per machine ### | Field | Value | Where it comes from | | --- | --- | --- | | Total parameters | B | model card | | Active parameters | B | model card; equal to total for a dense model | | Layers | | model card or config.json | | Key-value heads | | config.json, `num_key_value_heads`; not the query head count | | Head dimension | | config.json | | Quantisation | | your choice | | Bytes per parameter | | Part 4 | | **Weight memory** | GB | arithmetic | | Context length you will configure | tokens | your choice, not the trained maximum | | Bytes per element in the cache | <2 for FP16, 1 at 8-bit, about 0.5 at 4-bit> | your engine's cache type | | **Bytes per token of cache** | 2 × × × × = | arithmetic | | **Cache at your context** | GB | arithmetic | | Concurrent conversations planned | | from section 1 | | **Cache for all of them** | GB | arithmetic | | **Subtotal for this model** | GB | arithmetic | ### Headroom, per machine | Machine | Total memory | Operating system and applications | Engine working memory | Cache growth allowance | Second model | **Left for weights** | | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | ### Does it fit? | Machine | Model | Subtotal | Left for weights | Fits? | If not, what changes | | --- | --- | --- | --- | --- | --- | | | | | | | | ## 4. Models per role | Role | Model | Why this one | Licence | Gated? | Fallback if it disappoints | | --- | --- | --- | --- | --- | --- | | Chat | | | | | | | Coding | | | | | | | Embedding | | | | | | | | | | | | | ## 5. The network plan | From | To | Class | Rated | Measured throughput | Measured round trip | Carries | | --- | --- | --- | --- | --- | --- | --- | | | | <2.5 GbE, Thunderbolt, QSFP, Wi-Fi> | | | | | **Names and addressing:** **If you have one machine:** ## 6. Download and storage plan | Model | Repository | Files | Size | Destination | Verified how | Downloaded on | | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | **Total to download:** · **Free space on the library volume:** · **Headroom after:** **Order:** **One authoritative copy:** ## 7. Risks, and what would tell you early | # | Risk | How likely | What it costs | The check that catches it early | | --- | --- | --- | --- | --- | | 1 | | | | | | 2 | | | | | | 3 | | | | | ## 8. What this plan does not know