Level 2 · Local OperatorLessonPart 06 · page 1 of 725 min

25Minutes

4Tools

6Sources

Tools used on this page4

llama.cpp: The Engine That Runs Everywhere

By the end of this lesson you will be able to say what llama.cpp is and what it is not, name the backend your own machine should be using and why, decide between a prebuilt binary and a build from source for your track, recognise the four tools this part uses by name, and place the engine next to the desktop applications and server engines the later parts cover. This is the map; the next lesson puts a working build on your machine.

One codebase, every machine on your desk

llama.cpp is an inference engine: a program that loads a model’s weights and produces tokens. It is written in C and C++ on top of ggml, a tensor library from the same project, and its README states the goal as enabling language-model and vision-language-model inference “with minimal setup” across “a wide range of hardware — locally and in the cloud”.

Two decisions from that sentence explain almost everything about the project. Minimal setup is why it has no Python runtime, no CUDA-only dependency and no package manager between you and the binary: the README’s first bullet under its description is “Plain C/C++ implementation without any dependencies”. A wide range of hardware is why it has more compute backends than any other engine in this course, and why it is the only one that will run on all four of the course’s platform tracks.

That combination is why this part comes before the others. Whatever machine you have, llama.cpp runs on it, so it is the one place where the four tracks can be compared using the same tool, the same file and the same measurement. Every engine after it is a comparison against a baseline you built yourself.

Why a C++ project won this job

Almost everything else in this course is Python. It is worth being explicit about why the engine that runs on the most machines is not, because the reasons are the same ones that will decide your own tooling choices later.

A dependency you do not have cannot break. A Python inference stack is a graph of packages, each pinned against a CUDA version, a driver version and a Python version. The graph is why Part 1 spent a lab on virtual environments. llama.cpp’s build inputs are a compiler, CMake and the vendor’s own GPU toolkit; there is nothing between them and the binary.

A single binary can be copied. The tools are self-contained enough to be built on one machine and dropped onto another with the same hardware. That is why they turn up inside desktop applications, inside phones and inside other people’s servers, and why the quantisation vocabulary in this part is the vocabulary the whole local-inference ecosystem uses.

Portability was a goal rather than a consequence. The README describes Apple silicon as “a first-class citizen — optimized via ARM NEON, Accelerate and Metal frameworks”, alongside x86 with AVX, AVX2, AVX512 and AMX, and RISC-V. A project with that goal accumulates backends; a project built for one vendor’s datacentre accelerator does not need to.

The costs are real too. C++ is a harder language to contribute a new model architecture to than Python, so support for a brand-new architecture sometimes lands here days or weeks after it lands in the Python libraries. And the project moves fast enough that “llama.cpp does X” is a statement with a date attached, which is why every page in this course names the build it was checked against.

What it can do besides generate text

Three capabilities are worth knowing about now, because they change what you would reach for this engine to do.

Quantised inference across a wide range of widths. The README advertises “1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use”. That range is the subject of the GGUF lesson, and the low end of it is what makes very large models reachable on a desk machine at all.

Hybrid CPU and GPU execution. The README lists “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity”. This is genuinely useful when the alternative is not running the model at all, and it is genuinely slow, because every token then waits on the slowest path. The lesson on llama-cli and llama-server shows how to ask for it deliberately, and the challenge at the end of this part is largely about recognising when you got it by accident.

Constrained output. The project supports GBNF grammars, and the server exposes both a grammar and a JSON-schema option, so the sampler can be restricted to tokens that keep the output valid against a schema. Part 22 uses that properly for structured extraction and tool calling; the point here is that “make the model return valid JSON” is an engine feature rather than a prompting technique.

What ships in the box

A llama.cpp build is not one program. It is a set of small command-line tools that share the same model loader and the same backends, which is why they agree with each other about what your machine can do. Four of them carry this part:

Tool	What it does
`llama-cli`	Loads a model and generates text, once or in a conversation loop. The quickest way to see a model work, and the place the load-time diagnostics are printed.
`llama-server`	The same engine behind an HTTP API, with OpenAI-compatible endpoints and a built-in web interface. What everything else in the course talks to.
`llama-bench`	Runs prompt-processing and generation tests at fixed sizes, repeats them, and reports the mean and standard deviation. The measurement tool this course uses.
`llama-quantize`	Converts a GGUF file from one weight format to a smaller one.

The source tree’s tools/ directory holds more than these: an importance-matrix generator (llama-imatrix), a perplexity evaluator, a splitter and joiner for sharded GGUF files, a tokeniser, a multimodal command-line tool, and rpc-server, the piece that lets several machines share one model, which Part 19 uses to run a model too large for any one of them.

The backend is the whole story

ggml has a backend for each family of accelerator, and llama.cpp is compiled against one or more of them. The backend decides which chip does the matrix multiplications, and therefore decides whether your machine is generating tokens at the rate its memory bandwidth allows or at a small fraction of it.

Where the backend sits

Your clientA browser on the built-in web interface, curl, an OpenAI SDK, an editor plug-in, an agent.
llama-server, llama-cli, llama-benchThe tools. Identical source on every track.
llama.cpp and ggmlModel loading, the GGUF reader, the KV cache, sampling, batching. Identical on every track.
ggml backendCUDA, Metal, Vulkan, HIP, or the CPU backend. Chosen at build time. This is the layer that differs per track.
Vendor runtime and driverCUDA toolkit and driver, Metal and macOS, the Vulkan loader and ICD, or ROCm.
The chipGB10, Radeon 8060S, Apple GPU, GeForce or RTX PRO.

A build is a choice of the third layer from the bottom. The layers above it do not change between tracks; the layer below it is the vendor stack you installed with your driver. Almost every 'why is this slow' question in the next lessons is a question about this one layer.

The README’s supported-backends table lists BLAS, BLIS, CANN, CUDA, HIP, Hexagon, IBM zDNN, MUSA, Metal, OpenCL, OpenVINO, RPC, SYCL, VirtGPU, Vulkan, WebGPU and ZenDNN. Most of those exist for hardware this course does not cover. Five matter here:

Backend	Hardware	Track
CUDA	NVIDIA GPUs, including the GB10 in the DGX Spark	S and N
Metal	Apple silicon GPUs. Enabled by default when building on macOS	M
Vulkan	Any GPU with a Vulkan driver, including the Radeon 8060S	X, and a fallback elsewhere
HIP	AMD GPUs through ROCm	X
CPU	Every machine. The fallback, and the thing you must not be using by accident	all

The CPU backend deserves its own sentence, because it is the most common cause of disappointment. A build with no GPU backend compiled in still works. It loads the model, answers your prompt and never says anything is wrong. It is simply reading the weights through the CPU’s memory path instead of the GPU’s, and on the machines in this course that is a large difference. The next lesson ends by proving which backend you got, and the challenge at the end of this part is built around the case where the proof was skipped.

Getting it: a prebuilt binary or a build

The README offers four ways to install: the project’s own installer page, Docker images, prebuilt binaries from the releases page, and building from source.

The prebuilt archives are the fastest route where one exists for your machine. Their names encode the platform and the backend. Reading the assets of build b10867 on 2026-09-09, the list included llama-b10867-bin-macos-arm64.tar.gz, llama-b10867-bin-ubuntu-x64.tar.gz, llama-b10867-bin-ubuntu-arm64.tar.gz, llama-b10867-bin-ubuntu-vulkan-x64.tar.gz, llama-b10867-bin-ubuntu-vulkan-arm64.tar.gz, llama-b10867-bin-ubuntu-rocm-10.0-x64.tar.gz, llama-b10867-bin-win-cuda-12.4-x64.zip, llama-b10867-bin-win-cuda-13.3-x64.zip and llama-b10867-bin-win-vulkan-x64.zip, among others.

Read that list against the four tracks and one gap stands out.

Building from source is not hard, it is one CMake command per track, and it has a second advantage the next lesson uses: the build prints which backends it compiled in, so you learn what you got before you load a model rather than after.

GGUF: the file the whole ecosystem shares

The weights llama.cpp loads are in GGUF, a single-file format defined in the ggml repository. The specification’s design goals are worth reading in full, but three of them explain why the format won:

Single-file deployment: a model is one file that carries everything needed to load it, weights and metadata and tokeniser together, with no configuration files beside it.
mmap compatibility: the file’s layout lets the operating system map it into memory rather than read it, which is why a large model can start generating before the whole file has been read from disk.
Full information: as the specification puts it, “all information needed to load a model is contained in the model file”.

That last point is the practical difference from a Hugging Face checkpoint, which is a directory of safetensors shards plus a config.json, a tokeniser and often a chat template. GGUF folds all of it into one file with a metadata table. The lesson on GGUF and quantisation types opens the file up and reads that table.

Where it sits next to everything else

llama.cpp is one point in a landscape the rest of the course walks through, and it is worth knowing now what it is not.

Ollama and LM Studio (Part 7) are model managers and front-ends. Ollama pulls models by short name, keeps them in its own store, and serves them; LM Studio adds a graphical application. Both have historically run GGUF models through llama.cpp or its libraries, which is why the quantisation vocabulary you learn here transfers directly. What they add is convenience: a registry, automatic model loading and unloading, an installer. What they hide is exactly the layer this part teaches, which is why the course teaches the engine first.

MLX (Part 8) is Apple’s own array framework, with its own model format and its own server. On a Mac it is llama.cpp’s most direct competitor, and Part 8 measures them against each other.

vLLM, SGLang and TensorRT-LLM (Part 9 onwards) are server-class engines built for many concurrent users on datacentre GPUs. They use different memory management, different batching and usually unquantised or FP8 weights, and they do not run on macOS at all. For one person asking one question at a time, they are not obviously better, and Part 9 measures the difference rather than assuming it.

Trace responsibility from bytes to an answer

The checkpoint provides tensors and metadata; the backend executes supported operations; the model implementation defines how those operations compose; the server serialises requests and schedules work. A failure belongs to one or more of these layers. “The model is bad” is not yet a diagnosis.

If loading fails before a prompt is accepted, inspect architecture support, missing shards and tensor types. If output is incoherent only in chat mode, inspect the template and special tokens. If output is correct but slow, inspect placement and the measured workload. If a client fails while the server’s basic request works, inspect the API contract.

Create a minimal reproduction that removes the front end and uses one request against the local server. Retain the build identity, startup command, checkpoint identity and response. This is also the handoff package for a bug report. The value of a portable engine is that you can repeat a controlled experiment on several backends; portability does not imply identical speed, numerical output or feature coverage on every device.

Recap

llama.cpp is a dependency-light C and C++ inference engine on top of ggml, built to run with minimal setup on a wide range of hardware, which is why it is the only engine in this course that covers all four tracks. A build is a choice of ggml backend: CUDA for the two NVIDIA tracks, Metal on a Mac, Vulkan or HIP on a Ryzen AI Max+ machine, and a CPU fallback that works everywhere and is the thing to make sure you are not using by accident. The release ships four tools this part depends on: llama-cli, llama-server, llama-bench and llama-quantize. Prebuilt archives exist for most combinations but not for CUDA on Linux aarch64, so the Spark builds from source. The weights come in GGUF, a single-file format that carries its own metadata and is designed to be memory-mapped.

Sources for this lesson

6 verified · checked 2026-09-09

01llama.cpp — README§ Quick start; Supported backends; Descriptiongithub.com/ggml-org/llama.cpp/blob/master/README.md2026-09-09
02llama.cpp — Build guide§ CPU build; CUDA; Metal; Vulkan; HIPgithub.com/ggml-org/llama.cpp/blob/master/docs/build.md2026-09-09
03llama.cpp — Releases§ Assets of the current build taggithub.com/ggml-org/llama.cpp/releases2026-09-09
04llama.cpp — llama-server READMEgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md2026-09-09
05llama.cpp — llama-bench READMEgithub.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md2026-09-09
06GGUF specification§ Design goals; File structuregithub.com/ggml-org/ggml/blob/master/docs/gguf.md2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.