Skip to content
Local LLM Course

Your Hardware: Four Platforms, One Course

Level 1 was about models. Level 2 is about your machine, and it starts here because every remaining page of the course has four versions of itself. Part 6 builds llama.cpp four ways. Part 8 sends you to MLX, TensorRT-LLM or ROCm depending on what is on your desk. Part 18 clusters two Sparks over ConnectX-7 or two Macs over Thunderbolt. None of that can be written once and adjusted, because an aarch64 container path, a Vulkan-versus-ROCm choice, a wired-memory sysctl and a WSL2 driver rule are four different procedures, not four dialects of one.

So this part does the hardware once, properly, and the rest of the course never explains it again. Four lessons, one per track, each written for the person who owns that machine and useful to everyone else as a comparison. Then two labs: one that takes a fresh operating system to a verified accelerator, a Python toolchain and a lab notebook, and one that measures the two numbers, memory bandwidth and matrix-multiply throughput, which predict most of what you will observe for the next twenty parts.

The first lesson is the one everyone should read regardless of track. It answers the question that decides every purchase and every model choice in this field, and the answer is not the one the marketing material suggests.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • say which of the four tracks your machine belongs to, and what its two governing numbers are: how much memory the accelerator can reach, and how fast it can read it;
  • explain the difference between unified and discrete memory, what crosses PCIe on a machine with a graphics card, and why that decides which models are practical rather than merely possible;
  • describe your own track’s compute stack in order, from silicon to driver to runtime to engine, and name the layer that breaks when a model will not load;
  • take a fresh install to a working state: updates, drivers, a Python toolchain, containers where the track uses them, a Hugging Face token stored safely and a shared model library;
  • measure your machine’s memory bandwidth and its matrix-multiply throughput, compare both with the vendor’s published figures, and account for the gap;
  • predict the decode speed of a model you have not downloaded yet, from its size and your measured bandwidth, and record the prediction so that Part 6 can check it.

What you need first

Section titled “What you need first”

Part 1’s lab, because the Python environment and the lab notebook it created are used from the first task onwards. Part 4’s memory arithmetic, because the prediction script at the end of this part is that arithmetic applied to your own measurement. Both labs run on the 8 GB tier and neither downloads a model, so nothing here is gated on having bought the right machine.

You will need administrator rights on the machine, since installing drivers and a container runtime needs them, and about two hours of wall-clock time for the two labs, much of it unattended while packages download.

How to work through it

Section titled “How to work through it”

Read the first lesson, then read the lesson for your own track properly and skim the other three. The skim is not optional politeness: the cluster parts in Level 4 mix tracks deliberately, the troubleshooting sections of later labs assume you know why a Mac has a wired-memory limit and a Spark does not, and half the questions people ask about their own machine are answered by knowing what a different one does instead.

Then do the two labs in order, on the machine you intend to use for the rest of the course. Write the results into the lab notebook from Part 1. Those numbers are the baseline every later measurement is compared against, and Part 6 opens by asking you to check its first benchmark against the prediction you make here.

Part5 of 27Level2 — Local OperatorPages7Estimated time4.5 hours

0 / 7 lessons in this part completed