Skip to content
Local LLM Course

First Inference with llama.cpp

Everything so far has been reading and arithmetic. Part 4 chose models on paper, Part 5 predicted what a machine should do with them. This part is where a model runs on your hardware, at a speed you measure rather than assume.

The engine is llama.cpp, and it comes first for one reason: it is the only one that runs on every track in this course. The same source tree builds against CUDA on the DGX Spark and on an NVIDIA desktop, Vulkan or ROCm on a Ryzen AI Max+ machine, and Metal on a Mac. One file format, GGUF, holds the weights on all of them. One benchmarking tool, llama-bench, reports numbers that can be compared across the four machines. Everything after this part, Ollama and LM Studio in Part 7, MLX in Part 8, vLLM and the serving engines in Part 9, is a comparison against something you already know rather than a fresh start.

The quantisation vocabulary is the other reason. Q4_K_M, Q6_K, IQ4_XS and the importance matrix are not llama.cpp trivia; they are the names the whole open-weight ecosystem uses for the trade between memory, speed and quality. You will meet them in every model repository you open for the rest of the course.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • say what llama.cpp is, which backend your machine should use, and where the engine sits next to the desktop applications and the server-class engines the later parts cover;
  • build llama.cpp from source with the right backend for your track, and prove from the program’s own output that the GPU is doing the work;
  • read a quantisation name, estimate what it costs in bytes per weight and pick one for a memory budget, check a download against the checksum its publisher published, and convert and quantise a safetensors checkpoint yourself;
  • run a model interactively, serve it over an OpenAI-compatible HTTP API, and set the flags that decide how much memory it uses and how fast it runs;
  • explain how a probability distribution becomes a token, set temperature, top-k, top-p, min-p and the penalties deliberately, and say why two runs with the same seed can still differ;
  • measure prompt-processing and generation rates for every reference model that fits your machine, compare them with the bandwidth prediction from Part 5, and record the result so that a later measurement can be compared with it;
  • work through a slow model methodically: collect evidence, name the fault, fix it, and prove the fix with a second measurement.

What you need first

Section titled “What you need first”

Part 4’s model library and the hf command-line tool, because the lab downloads GGUF files into the same ~/models tree. Part 5’s bandwidth arithmetic, because the benchmark lab compares its measurements against that prediction. The lab notebook you started in Part 1, because every page here ends by adding to it.

You also need a compiler and CMake, and roughly 15 GB of disk for the source tree and the build. The memory floor for the hands-on pages is 8 GB, which runs an 8B-class model at Q4_K_M. Larger tiers run more of the reference set, and each page lists exactly which models fit which tier.

How to work through it

Section titled “How to work through it”

The five lessons are in dependency order and the two hands-on pages sit at the end. Read the first lesson, then do the build lesson with a terminal open, because everything afterwards assumes a working llama-cli and llama-server. The GGUF lesson can be read away from the machine; the server and sampling lessons are better with the model loaded so you can try each flag as it is described.

Then give the benchmark lab an uninterrupted hour. It is the first page in the course that produces a table of your own numbers, and those numbers are referenced again in Part 7, Part 9 and the capstone. The challenge afterwards is deliberately shaped like a real support request: something is slow, and the exercise is the procedure that finds out why.

Part6 of 27Level2 — Local OperatorPages7Estimated time4.3 hours

0 / 7 lessons in this part completed