Installing and Building llama.cpp on Your Platform

By the end of this lesson you will have llama-cli, llama-server and llama-bench on your machine, built or installed with the backend your hardware actually has, and you will have proved that the backend is in use rather than assumed it. The proof is the point. A build that quietly fell back to the CPU behaves exactly like a correct one, only slower, and finding that out three lessons later costs an afternoon.

The loop this lesson runs

1. 1Install the toolchainA compiler, CMake, git, and whatever your backend needs: the CUDA toolkit, the Vulkan development packages, ROCm, or Xcode command line tools.
2. 2ConfigureOne cmake command with one backend flag. The output lists what it found and what it will build.
3. 3BuildA few minutes with several jobs in parallel. Install ccache first if you expect to rebuild.
4. 4Prove the backendRun llama-bench on a small model and read the backend column. Anything other than the backend you asked for is a failed build, not a slow machine.
5. 5Record itBuild tag, backend, cmake flags and date into the lab notebook, so a later measurement can be compared with this one.

Before you start: a model to test with

The smoke test at the end of every track needs a GGUF file. If you followed Part 4 you already have a model library at ~/models; if not, one download is enough for this lesson. Qwen3-8B at Q4_K_M is about 5 GB and fits the 8 GB tier.

RunnableAll tracks

fetch one small GGUF for the smoke test

hf download unsloth/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf \
    --local-dir ~/models/unsloth/Qwen3-8B-GGUF

Qwen3-8B is Apache-2.0 licensed and is not gated, so no account is needed. The next lesson explains what Q4_K_M means and why the community repositories rather than the official ones are where the GGUF files usually live.

The four builds

Every track uses the same two CMake commands: one to configure, one to build. The only difference is the backend flag, and on a Mac there is not even that, because Metal is on by default.

Track S — NVIDIA DGX Spark

DGX OS is Ubuntu-based and ships the CUDA toolkit, so the toolchain is nearly complete already. As the previous lesson showed, the releases page publishes no CUDA archive for Linux on aarch64, so this track builds from source. That is the normal path here, not a workaround.

RunnableTrack S · DGX Spark

toolchain and source

sudo apt-get update
sudo apt-get install -y build-essential cmake git ccache
nvcc --version
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp

If nvcc --version is not found, the toolkit is installed but not on your path; DGX OS puts it under /usr/local/cuda/bin. The build needs nvcc, not just the driver.

RunnableTrack S · DGX Spark

configure and build with CUDA

cmake -S ~/llama.cpp -B ~/llama.cpp/build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(nproc)"

The configure step prints the CUDA version it found and the GPU architecture it is targeting. Read those two lines before you let the build run; a mismatch between the toolkit and the driver shows up here rather than at load time.

Expect several minutes on the twenty Arm cores. ccache makes the second and later builds much shorter, which matters because you will rebuild when you update.

Track X — AMD Ryzen AI Max+ 395Partial

Vulkan is this course's default path on this chip and is a fully supported build. The HIP build through ROCm is the dated alternative below; treat it as the experiment, not the baseline.

Two backends are available on a Ryzen AI Max+ 395, and the course’s default is Vulkan: it builds from ordinary distribution packages, it has a prebuilt Linux archive if you would rather not compile, and it does not depend on the ROCm release cadence.

RunnableTrack X · Ryzen AI Max+

toolchain and source for the Vulkan build

sudo apt-get update
sudo apt-get install -y build-essential cmake git ccache libvulkan-dev glslc spirv-headers
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp

RunnableTrack X · Ryzen AI Max+

configure and build with Vulkan

cmake -S ~/llama.cpp -B ~/llama.cpp/build -DGGML_VULKAN=1 -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(nproc)"

The three packages are the ones the build guide names for Ubuntu and Debian. If your distribution splits them differently, the requirement is the Vulkan headers and loader plus the glslc shader compiler.

The ROCm alternative. The build guide gives a HIP recipe that needs the GPU target passed explicitly. The Radeon 8060S in this machine is gfx1151, and the ROCm compatibility matrix (read on 2026-09-09, documenting ROCm 10.0.0, released 2026-08-14) lists “AMD Ryzen AI Max+ PRO 395 (Radeon 8060S) (gfx1151)” among its supported GPUs, with Ubuntu 26.04 and 24.04 among the supported distributions.

RunnableTrack X · Ryzen AI Max+

configure and build with HIP (the alternative path)

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S ~/llama.cpp -B ~/llama.cpp/build-hip \
    -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build-hip --config Release -j "$(nproc)"

Note the separate build directory. Keep both builds and benchmark them against each other in this part’s lab; which one is faster on this chip is a measurement, and it is one of the more interesting numbers you will produce.

There is also a prebuilt llama-b10867-bin-ubuntu-rocm-10.0-x64.tar.gz archive on the releases page if you would rather not build the ROCm path yourself.

Track M — Apple silicon

Metal is enabled by default when building on macOS, so this is the shortest build in the course: there is no backend flag to pass. You need Xcode’s command line tools for the compiler and CMake, which Homebrew provides.

RunnableTrack M · Apple silicon

toolchain and source

xcode-select --install
brew install cmake ccache
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp

xcode-select --install opens a dialogue and does nothing if the tools are already present.

RunnableTrack M · Apple silicon

configure and build (Metal is on by default)

cmake -S ~/llama.cpp -B ~/llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(sysctl -n hw.ncpu)"

The build guide notes that Metal can be turned off at compile time with -DGGML_METAL=OFF, and that a Metal-enabled build can still be forced onto the CPU at run time by asking for zero GPU layers. Both are useful in the challenge at the end of this part, where deliberately building the wrong thing is how you learn to recognise it.

If you would rather not build at all, the releases page publishes llama-b10867-bin-macos-arm64.tar.gz, which is the same tools with Metal already compiled in. Unpack it, and use its bin directory wherever this part says ~/llama.cpp/build/bin.

Track N — NVIDIA desktop or laptop

On Linux, and inside WSL2 on Windows, this is the CUDA build and it is identical to Track S apart from the architecture. You need the NVIDIA driver and the CUDA toolkit, because the build compiles CUDA kernels: the driver alone is enough to run a prebuilt binary but not to make one.

RunnableTrack N · NVIDIA GPU

toolchain and source

sudo apt-get update
sudo apt-get install -y build-essential cmake git ccache
nvidia-smi
nvcc --version
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp

RunnableTrack N · NVIDIA GPU

configure and build with CUDA

cmake -S ~/llama.cpp -B ~/llama.cpp/build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build ~/llama.cpp/build --config Release -j "$(nproc)"

Inside WSL2 the rule from Part 1 still holds: install the NVIDIA driver on Windows only, never a Linux GPU driver inside WSL. nvidia-smi run inside WSL2 should list your card before you build anything.

Windows without WSL2. The releases page publishes prebuilt CUDA archives named for the CUDA version, such as llama-b10867-bin-win-cuda-13.3-x64.zip and llama-b10867-bin-win-cuda-12.4-x64.zip, which include the CUDA runtime libraries. Pick the one matching what your driver supports, unpack it, and run the same tools from a PowerShell prompt. The commands in this part are written for a Unix shell; on native Windows they need the usual path adjustments, which is one reason the course’s primary Windows path is WSL2.

⌚Version note

The README’s first commands may not match what you just built. Reading the project README on 2026-09-09, its quick start invokes a llama launcher with subcommands, llama cli -hf … and llama serve -hf …, and some model cards show the same form. The per-tool documentation and a source build both give the hyphenated binaries, llama-cli and llama-server, and those are what this course uses because they are what the build you just ran produces in build/bin. If your installation offers only the launcher, run llama --help and map the subcommands across rather than assuming; the course has not verified that the two forms accept identical options.

The same four builds, as a script

The script below does what your track’s section just did, with the assumption checks the course asks for: it refuses to start if the toolchain for that backend is missing, tells you which flags it is about to use, builds, and then runs the smoke test and checks the backend name in the output.

RunnableAll tracks

build-llama-cpp.sh

#!/usr/bin/env bash
# Purpose: build llama.cpp from source with the GPU backend for one platform track, then
#          prove with a smoke test that the backend it was built for is the one in use
# Platform: spark (CUDA on aarch64), strix (Vulkan by default, HIP with BACKEND=hip),
#           mac (Metal, on by default), nvidia (CUDA on Linux or inside WSL2)
# Minimum memory: 8 GB
# Assumes: git, cmake and a C++ toolchain are installed; nvcc on PATH for the CUDA tracks;
#          glslc and the Vulkan headers for the Vulkan build, or hipconfig for the HIP
#          build; about 15 GB of free disk for the checkout and the build tree
#
# Usage: TRACK=<spark|strix|mac|nvidia> bash build-llama-cpp.sh
#        TRACK=strix BACKEND=hip bash build-llama-cpp.sh
#        TRACK=nvidia SMOKE_MODEL=~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
#            bash build-llama-cpp.sh
#
# Environment:
#   LLAMA_DIR    where the checkout lives              (default: $HOME/llama.cpp)
#   LLAMA_REF    tag, branch or commit to build        (default: master)
#   BUILD_DIR    the cmake build tree                  (default: $LLAMA_DIR/build)
#   JOBS         parallel compile jobs                 (default: detected)
#   GPU_TARGET   gfx target for the HIP build          (default: gfx1151)
#   SMOKE_MODEL  a GGUF file for the smoke test        (default: none; skipped)
#
# The cmake flags come from the project's build documentation, cited on the lesson page.
# Re-running is safe: the checkout is updated and cmake reconfigures in place.

set -euo pipefail

TRACK="${TRACK:-}"
BACKEND="${BACKEND:-}"
LLAMA_DIR="${LLAMA_DIR:-$HOME/llama.cpp}"
LLAMA_REF="${LLAMA_REF:-master}"
BUILD_DIR="${BUILD_DIR:-$LLAMA_DIR/build}"
GPU_TARGET="${GPU_TARGET:-gfx1151}"
SMOKE_MODEL="${SMOKE_MODEL:-}"

die() { echo "build-llama-cpp: $*" >&2; exit 1; }
have() { command -v "$1" >/dev/null 2>&1; }

[ -n "$TRACK" ] || die "set TRACK=spark, TRACK=strix, TRACK=mac or TRACK=nvidia"
have git || die "git is not installed"
have cmake || die "cmake is not installed"

if [ -z "${JOBS:-}" ]; then
  if have nproc; then
    JOBS="$(nproc)"
  elif have sysctl; then
    JOBS="$(sysctl -n hw.ncpu)"
  else
    JOBS=4
  fi
fi

# --- 1. Decide the backend and the cmake flags for this track ------------------------
CMAKE_FLAGS=(-DCMAKE_BUILD_TYPE=Release)
case "$TRACK" in
  spark|nvidia)
    have nvcc || die "nvcc is not on PATH; install the CUDA toolkit before building the CUDA backend"
    CMAKE_FLAGS+=(-DGGML_CUDA=ON)
    EXPECT_BACKEND="CUDA"
    ;;
  strix)
    if [ "$BACKEND" = "hip" ]; then
      have hipconfig || die "hipconfig is not on PATH; install ROCm, or leave BACKEND unset to build Vulkan"
      HIPCXX="$(hipconfig -l)/clang"
      HIP_PATH="$(hipconfig -R)"
      export HIPCXX HIP_PATH
      CMAKE_FLAGS+=(-DGGML_HIP=ON "-DGPU_TARGETS=$GPU_TARGET")
      EXPECT_BACKEND="ROCm"
    else
      have glslc || die "glslc is not on PATH; install the Vulkan development packages (see the lesson page)"
      CMAKE_FLAGS+=(-DGGML_VULKAN=1)
      EXPECT_BACKEND="Vulkan"
    fi
    ;;
  mac)
    [ "$(uname -s)" = "Darwin" ] || die "TRACK=mac only makes sense on macOS; this machine reports $(uname -s)"
    # Metal is enabled by default on macOS, so no extra flag is needed here.
    EXPECT_BACKEND="Metal"
    ;;
  *)
    die "unknown TRACK '$TRACK'; use spark, strix, mac or nvidia"
    ;;
esac

echo "==> Track $TRACK, backend $EXPECT_BACKEND, ${JOBS} parallel jobs"
echo "    cmake flags: ${CMAKE_FLAGS[*]}"

# --- 2. Get or update the source ------------------------------------------------------
if [ -d "$LLAMA_DIR/.git" ]; then
  echo "==> Updating $LLAMA_DIR"
  git -C "$LLAMA_DIR" fetch --tags origin
else
  echo "==> Cloning llama.cpp into $LLAMA_DIR"
  git clone https://github.com/ggml-org/llama.cpp "$LLAMA_DIR"
fi
git -C "$LLAMA_DIR" checkout "$LLAMA_REF"
git -C "$LLAMA_DIR" pull --ff-only 2>/dev/null || true
BUILT_REF="$(git -C "$LLAMA_DIR" rev-parse --short HEAD)"
echo "    building $LLAMA_REF at $BUILT_REF"

# --- 3. Configure and build -----------------------------------------------------------
echo "==> Configuring"
cmake -S "$LLAMA_DIR" -B "$BUILD_DIR" "${CMAKE_FLAGS[@]}"
echo "==> Building (this takes several minutes the first time)"
cmake --build "$BUILD_DIR" --config Release -j "$JOBS"

BIN="$BUILD_DIR/bin"
[ -x "$BIN/llama-cli" ] || die "the build finished but $BIN/llama-cli is missing; read the build output above"
echo "==> Binaries in $BIN:"
find "$BIN" -maxdepth 1 -type f -perm -u+x -exec basename {} \; | sort | sed 's/^/    /'

# --- 4. Smoke test: prove which backend is in use --------------------------------------
echo "==> Version and build information"
"$BIN/llama-cli" --version 2>&1 | sed 's/^/    /'

if [ -z "$SMOKE_MODEL" ]; then
  cat <<'MSG'

No SMOKE_MODEL was given, so the backend has not been exercised yet. Download a small
GGUF file and re-run with SMOKE_MODEL=<path> to see the backend column, or run
llama-bench yourself as the lesson page shows.
MSG
  exit 0
fi

[ -f "$SMOKE_MODEL" ] || die "SMOKE_MODEL '$SMOKE_MODEL' does not exist"
echo "==> Smoke test: eight generated tokens, one repetition"
SMOKE_OUT="$("$BIN/llama-bench" -m "$SMOKE_MODEL" -p 0 -n 8 -r 1 2>&1 || true)"
printf '%s\n' "$SMOKE_OUT" | sed 's/^/    /'

if printf '%s' "$SMOKE_OUT" | grep -qi "$EXPECT_BACKEND"; then
  echo "==> The backend column names $EXPECT_BACKEND. The build is using the GPU path it was built for."
else
  echo "==> WARNING: '$EXPECT_BACKEND' does not appear in the llama-bench output above." >&2
  echo "    The build may have fallen back to the CPU. See the challenge page for the procedure." >&2
  exit 1
fi

Download build-llama-cpp.sh138 lines

RunnableAll tracks

build and prove, in one command

TRACK=nvidia SMOKE_MODEL=~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    bash build-llama-cpp.sh

Set TRACK to spark, strix, mac or nvidia. On Track X, add BACKEND=hip to build the ROCm path instead of Vulkan.

Proving the backend is in use

Three pieces of evidence, in increasing order of how much they tell you.

The version line. Every tool prints its build information when asked.

RunnableAll tracks

what did I build?

~/llama.cpp/build/bin/llama-cli --version

Record whatever build identifier it prints in the notebook, and report it with every measurement you publish; llama.cpp changes weekly, and a rate quoted without the build it came from cannot be reproduced by anyone, including you next month.

The load log. Loading a model prints where the tensors went. The exact wording changes between builds, so read it for content rather than matching a string: there is a line naming the backend device and a line saying how many layers were offloaded to it. A layer count below the model’s total means the remainder are running on the CPU, and that is the single most common cause of slowness in this part.

The benchmark table. llama-bench prints a backend column, and this is the evidence to keep, because it also gives you a rate to compare later.

RunnableAll tracks

smoke test: sixteen tokens, one repetition

~/llama.cpp/build/bin/llama-bench \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -p 0 -n 16 -r 1

Output — what you should see

| model                          |       size |     params | backend    | ngl |  test |   t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | ----: |
| qwen3 8B Q4_K - Medium         |   x.xx GiB |     8.19 B | CUDA       |  99 |  tg16 | xx.xx |

The backend column is the answer. On Track S and Track N it should say CUDA, on Track M Metal, on Track X either Vulkan or ROCm depending on which build you ran. If it says CPU, or if ngl is 0 when you did not ask for that, stop here and read the challenge page’s procedure; going on will only produce numbers that describe a broken build.

=Expected result

-p 0 -n 16 -r 1 asks for no prompt-processing test, sixteen generated tokens and one repetition, which finishes in seconds. It is a smoke test, not a measurement: one repetition of sixteen tokens is far too short to quote. The lab in this part uses the tool properly.

✕Common mistake

Asking for zero GPU layers is not the same as disabling the GPU. The build guide is explicit that the GPU may still accelerate parts of the computation with -ngl 0, and that full disabling needs --device none. This matters when you are trying to measure a CPU-only baseline: a run with -ngl 0 is a mostly-CPU run, not a CPU run, and comparing it against a GPU run overstates how well the CPU does.

Build flags that change what you get

Most builds need none of these. They are here because you will meet them in other people’s build instructions and should know which ones matter.

-DCMAKE_BUILD_TYPE=Release. Not optional in practice. A default configure without it can produce an unoptimised build, and the difference is large enough to look like a hardware problem.

-j <n> and ccache. The build guide recommends passing -j to compile in parallel, and installing ccache for faster repeated compilation. Both are about your time, not the model’s speed.

-DGGML_NATIVE=OFF. On by default, meaning the build targets the machine compiling it. Turn it off when the binary has to run on a different machine, and pair it with an explicit architecture list on CUDA: the build guide’s example is -DCMAKE_CUDA_ARCHITECTURES="86;89".

-DGGML_CUDA_FA_ALL_QUANTS=ON. Off by default. Compiles flash-attention kernels for every quantised KV cache type instead of a common subset, at the cost of a much longer build. Worth it only if you intend to run quantised KV caches at unusual types, which the next lessons explain.

-DGGML_CUDA_FORCE_MMQ=ON and -DGGML_CUDA_FORCE_CUBLAS=ON. Both off by default, both force a particular matrix-multiplication path. Leave them alone unless you are investigating a specific performance question and prepared to measure both ways.

-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS. For a CPU-only build, links an optimised BLAS. Irrelevant when you have a GPU backend.

-DBUILD_SHARED_LIBS=OFF. Produces statically linked tools, which are easier to copy to another machine.

✱Hardware note

On Linux with CUDA, the environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 lets an over-large model spill into system memory instead of failing to allocate. The build guide describes it as allowing “swapping to system RAM instead of crashing when the GPU VRAM is exhausted”. It is a way to make something run, not a way to make it fast: on a discrete-GPU machine the spilled weights are read across PCIe for every token. Use it knowingly, and never while benchmarking.

Keeping the build current

The project merges quickly, and performance work is a large part of what it merges. Updating is three commands, and the second and later builds are much faster with ccache in place.

RunnableAll tracks

update and rebuild

git -C ~/llama.cpp pull
cmake --build ~/llama.cpp/build --config Release -j 8
~/llama.cpp/build/bin/llama-cli --version

Record the new build number in the notebook next to the old one. When a measurement changes and you did not change anything else, the build tag is the first thing to look at.

⌚Version note

This course pins llama.cpp v0.4.0 · verified 2026-09-08, and the commands above were written against the build documentation as it read on 2026-09-09. CMake flags in this project are stable, but check the build guide when a flag is rejected rather than guessing a replacement.

Prove which executable you actually launched

Installing a new build does not remove older binaries from your shell’s search path. Before attributing a behaviour change to compilation, locate the executable and record its version. A shell alias, package-manager installation or earlier checkout may still take precedence over the build you just produced.

Run the binary by its explicit path for the first smoke test. Save the configure output that identifies the backend, then check the runtime device and placement log. A build can succeed without including the accelerator backend you intended, and runtime availability can differ from build-time detection.

Keep the previous working build in a separate directory while evaluating the new one. Reuse a small checkpoint and fixed request to check loading, completion and any application-specific feature such as tool parsing. If the build fails, report the first compiler error and the platform/compiler combination, rather than only the final nonzero exit. Reproducible installation means another reader can identify both the source revision and the binary that actually generated the recorded response.

Recap

Every track uses the same two CMake commands and differs only in one backend flag: -DGGML_CUDA=ON on the two NVIDIA tracks, -DGGML_VULKAN=1 or the HIP recipe with -DGPU_TARGETS=gfx1151 on a Ryzen AI Max+, and nothing at all on a Mac, where Metal is the default. Track S builds from source because no prebuilt CUDA archive exists for Linux on aarch64; Track M and Track N can use prebuilt archives if they prefer. -DCMAKE_BUILD_TYPE=Release matters, most other flags do not, and ccache pays for itself the second time you build. The build is not finished when it compiles: it is finished when llama-bench prints the backend you asked for, and that line, with the build tag from llama-cli --version, is what goes in the notebook.

Check your understanding

Question 1. On a Mac, which cmake flag enables the Metal backend?

• -DGGML_METAL=ON
• None: Metal is enabled by default when building on macOS
• -DGGML_ACCELERATE=ON
• -DCMAKE_METAL=1

Show the answer and why

Answer: None: Metal is enabled by default when building on macOS

The build guide states that Metal is on by default on macOS, and gives -DGGML_METAL=OFF as the way to disable it at compile time. Track M has the shortest configure command in the course.

Question 2. llama-bench prints "CPU" in the backend column on a machine with a working GPU. What is the most likely cause?

• The model file is corrupt
• The build was configured without the backend flag for this machine, so no GPU backend was compiled in
• The GPU is too old for this model
• llama-bench always reports CPU

Show the answer and why

Answer: The build was configured without the backend flag for this machine, so no GPU backend was compiled in

A build with no GPU backend runs perfectly well on the CPU backend and says nothing. Re-run cmake with the right flag for your track, rebuild, and check the column again. The other candidate is a driver or runtime the build could not find at load time.

Question 3. Why does Track S build from source rather than downloading a prebuilt archive?

• Because DGX OS cannot run downloaded binaries
• Because the releases page publishes no CUDA build for Linux on aarch64, which is what a DGX Spark is
• Because the prebuilt binaries are slower
• Because building is required to use GGUF files

Show the answer and why

Answer: Because the releases page publishes no CUDA build for Linux on aarch64, which is what a DGX Spark is

The Linux arm64 archives are the CPU and Vulkan builds; the CUDA archives are for x64 and for Windows. That specific combination was missing when the page was checked on 2026-09-09, so the Spark compiles its own.

Question 4. You want a genuine CPU-only baseline to compare against. Which is correct?

• Run with -ngl 0, which fully disables the GPU
• Use --device none, because the build guide says the GPU may still accelerate parts of the computation with -ngl 0
• Unplug the GPU
• Set the backend to CPU in the model file

Show the answer and why

Answer: Use --device none, because the build guide says the GPU may still accelerate parts of the computation with -ngl 0

The distinction is documented and it changes results: -ngl 0 keeps the weights off the GPU but does not stop it helping. A baseline built on -ngl 0 makes the CPU look better than it is.

Check my answersClear