llama-cli and llama-server

By the end of this lesson you will be able to run a model interactively, serve it as an OpenAI-compatible HTTP API that any client in the rest of this course can talk to, and set the handful of flags that decide whether it fits in your memory and how fast it answers. There are around a hundred options between the two tools. Nine of them account for nearly every decision you will actually make.

llama-cli: a file, a prompt, a token

The shortest useful command names a model and a prompt.

RunnableAll tracks

one question, one answer

~/llama.cpp/build/bin/llama-cli \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -p "In one sentence, what is a KV cache?" \
    -n 128

-n caps the generated tokens. For a conversation rather than a single completion, -cnv keeps the session open and applies the model’s chat template to each turn.

RunnableAll tracks

an interactive conversation

~/llama.cpp/build/bin/llama-cli \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -cnv --color -c 8192

Before it generates anything, the tool prints a wall of load-time diagnostics: the metadata it read from the GGUF file, the backend device, how many layers went to it, and the sizes of the buffers it allocated. That block is the most useful output either tool produces, and the challenge at the end of this part is entirely about reading it.

The README’s quick start also shows a model being pulled straight from Hugging Face with -hf instead of a local path, which is convenient for a one-off and unhelpful for a benchmark, where you want to know exactly which file you measured.

The flags that decide speed and memory

Offload: -ngl

-ngl (also spelled --n-gpu-layers) sets how many of the model’s layers are placed on the accelerator. The server README gives its default as automatic, but passing it explicitly is worth the keystrokes, because a wrong value is the most common cause of a slow model and an explicit value is the fastest thing to check.

A number larger than the model’s layer count means “all of them”, which is why -ngl 99 appears in so many examples. A number below the layer count splits the model: the offloaded layers run on the GPU and the rest run on the CPU, with the activations crossing between them for every token. On a discrete-GPU machine that crossing goes over PCIe, and the speed falls off a cliff. On a unified-memory machine there is no copy, but the CPU-side layers are still computed by the CPU.

✕Common mistake

Partial offload is not a smooth trade. People assume that offloading 80 per cent of the layers gives 80 per cent of the speed. It does not: every token has to pass through the layers that stayed behind, so the slowest path sets the pace. If a model does not fit, a smaller quantisation that fits entirely usually wins against a larger one that is partly offloaded, and the lab in this part has you measure exactly that on your own machine rather than take it on trust.

Split modes: -sm, -ts, -mg

With more than one GPU, -sm decides how the model is divided. The server README documents {none, layer, row, tensor} with layer as the default: none keeps everything on one device, layer gives each device a contiguous set of layers, and the others split individual tensors across devices. -ts sets the fraction of the model each device gets, for cards of different sizes, and -mg names the device that holds the small shared tensors.

Single-GPU machines and the three unified-memory tracks can ignore all three. Track N with two cards cannot: -sm layer with an appropriate -ts is where to start, and Part 9 measures the alternatives on real hardware.

Context length and the KV cache: -c, --cache-type-k, --cache-type-v

-c sets how many tokens of context the engine allocates for. The server README gives the default as 0, meaning “take it from the model”. That is a friendly default and a memory trap: a model trained for a long context will ask for a long context, and the KV cache is allocated up front.

The arithmetic is from Part 4. Qwen3-8B keeps about 144 KiB of KV cache per token at FP16, so a full 32,768-token context is roughly 4.8 GB — on top of the 5 GB of weights.

Qwen3-8B at Q4_K_M on a 16 GB machine, at three context lengths — estimates from the course model reference

Weights, Q4_K_M

5 GB

KV cache, 32k tokens at FP16

4.8 GB

Compute buffers and overhead

1 GB

Free

5.2 GB

Total

16 GB

Weights are fixed at about 5 GB. The KV cache is linear in the context you ask for: about 0.6 GB at 4k, 2.4 GB at 16k and 4.8 GB at 32k, at FP16. Quantising the cache to q8_0 halves those figures; to q4_0, quarters them. The bar below shows the 32k case, which is the one the default -c would choose.

The cache can be quantised like the weights. --cache-type-k and --cache-type-v accept, per the server README, f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, with f16 as the default. q8_0 on both halves the cache for a quality cost most people cannot detect; going to four bits is a bigger step and is worth measuring before adopting.

RunnableAll tracks

a long context that fits, by quantising the cache

~/llama.cpp/build/bin/llama-server \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -c 32768 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    -ngl 99

✱Hardware note

Two things constrain the context, and they are different. Memory is what this section is about, and you control it. The context window is what the model was trained to handle: the Qwen3-8B card gives 32,768 tokens natively, extensible to 131,072 with YaRN scaling. Asking for more memory than the model was trained for buys you allocation, not comprehension.

Flash attention: -fa

-fa takes on, off or auto, with auto as the documented default. It selects a fused attention implementation that computes the same result with less memory traffic. On the backends and model shapes that support it, it is close to free speed; auto turns it on when the build and the model allow it. Set it to on explicitly when you are benchmarking, so that your two runs are comparing the same thing rather than the scheduler’s opinion.

Threads and batches: -t, -tb, -b, -ub

-t sets the CPU threads used for generation and defaults to -1, meaning automatic; -tb sets the threads used for batch processing and defaults to whatever -t is. When the model is fully offloaded these matter little, because the CPU is mostly waiting. When part of the model is on the CPU, they matter a great deal, and the useful value is usually the number of physical cores rather than the number of hardware threads.

-b is the logical batch size, documented as defaulting to 2048, and -ub the physical one, defaulting to 512. They control how many prompt tokens are processed at once during prefill. Raising -ub can speed up prompt processing at the cost of a larger compute buffer; lowering it does the reverse. They are the two knobs to try if prompt processing, rather than generation, is your bottleneck.

llama-server

Everything above applies unchanged, because it is the same engine. What the server adds is an HTTP interface, a scheduler and a web page.

RunnableAll tracks

serve a model on localhost

~/llama.cpp/build/bin/llama-server \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -c 8192 -ngl 99 --alias qwen3-8b

The server README states that this listens on 127.0.0.1:8080 by default. --alias sets the name the API reports for the model, which is what clients send back in their model field, and setting it deliberately saves confusion later when several models are being served.

It can also fetch the weights itself:

RunnableAll tracks

serve straight from a Hugging Face repository

~/llama.cpp/build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 8192 -ngl 99

Chat templates and --jinja

A chat template turns a list of messages into the exact token sequence the model was trained on. Getting it wrong does not produce an error; it produces slightly worse answers, which is much harder to notice.

GGUF files carry their template in the metadata, and the server renders it with a Jinja engine. --jinja and --no-jinja switch that engine on and off, and the server README gives it as enabled by default. --chat-template overrides the template with a named built-in one, and --chat-template-file with one from a file, which is what you need for a model whose packaged template is wrong or missing.

✕Common mistake

Symptoms of a wrong chat template: the model answers, but it talks past the question, ignores the system prompt, keeps going after it should have stopped, or emits its own turn markers as visible text. Check the template before you conclude that the model is bad. /apply-template, listed below, shows you exactly what the server would send.

Parallel slots

-np sets the number of slots, each of which is an independent conversation with its own share of the KV cache. The README gives the default as automatic, and describes continuous batching, the scheduler that interleaves work from several slots, as enabled by default.

Three slots sharing one model, with continuous batching

Slot 1

prefilldecode

Slot 2

prefilldecode

Slot 3

prefilldecode

request 1 arrivesrequest 2request 3all three generating

Prefill is compute-bound and blocks; decode is bandwidth-bound and interleaves well, because the weights each slot needs are the same weights. This is why a second concurrent request costs far less than twice as much, and why the total KV cache is divided by the number of slots: asking for four slots at 8k of context each needs the memory for 32k.

The trade is memory. -c on the server is the total context across all slots, so four slots share what one slot would have had. For a single user, one slot and the whole context is the right choice; for an agent that fans out several calls, more slots and a shorter context each is usually better.

RunnableAll tracks

four slots for a small agent workload

~/llama.cpp/build/bin/llama-server \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -ngl 99 -c 32768 -np 4 --cont-batching \
    --cache-type-k q8_0 --cache-type-v q8_0

The API

The server speaks a subset of the OpenAI API, which is why every client in the later parts of this course, editors, agents, LiteLLM, Open WebUI, can talk to it without knowing what it is. The README documents /v1/chat/completions, /v1/completions, /v1/models and /v1/embeddings, alongside the project’s own /completion, /embedding and /reranking.

RunnableAll tracks

a chat completion over the OpenAI-compatible endpoint

curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "qwen3-8b",
      "messages": [{"role": "user", "content": "Name three causes of a slow local model."}],
      "temperature": 0.7,
      "max_tokens": 200
    }'

The operational endpoints are the ones you will use while debugging:

Endpoint	What it tells you
GET /health	Whether the server is up and the model is loaded.
GET /props	The server’s global properties, including the chat template in use.
GET /slots	The state of each slot: what it is processing and how much context it holds.
GET /metrics	Prometheus-format counters. Off by default; enable with --metrics.
POST /tokenize, POST /detokenize	Text to tokens and back, for checking how long a prompt really is.
POST /apply-template	The exact string the chat template produces for a message list.

--slots is documented as enabled by default and --props and --metrics as disabled, so enable the last two deliberately when you want them.

The web interface

The server ships a web front end at the same address it serves the API from: with the defaults, http://127.0.0.1:8080 in a browser. It is a complete chat client, and it is the fastest way to check that a model, its template and your flags are all working before you point real software at it. The README documents a flag to disable it for a headless deployment.

⛨Security

The default host is 127.0.0.1, which means the server is reachable only from the machine it runs on. Changing it to 0.0.0.0 puts an unauthenticated endpoint that will run arbitrary generation for anyone on your network — and, behind a careless router rule, anyone at all. If you need remote access, set --api-key, put it behind a reverse proxy with TLS, and treat the key as a credential: not in your shell history, not in a repository. Part 23 does this properly. Until then, keep it on localhost.

Treat the server as a process with an explicit lifecycle

Keep a server in its own terminal for the first run. Loading can take time; wait for readiness before sending the client request from a second terminal. Record the startup command and the model alias. An open port only shows that something is listening; it does not prove the intended checkpoint has loaded successfully.

Use three checks in order: the process’s startup log, its model or health endpoint, then a short application-shaped completion. A server that answers plain text may still reject a schema or misparse tools, so add those probes when your application needs them. Keep the response body when HTTP reports an error; it often identifies the missing field or capacity limit.

Stop the foreground process with Ctrl+C in its terminal and verify the client can no longer reach that endpoint. Avoid killing every process whose name contains “llama” on a machine running other services. When you later use a supervisor, preserve the same model identity, readiness test and shutdown semantics; automation should make the lifecycle repeatable, not obscure which process owns the port.

Recap

llama-cli and llama-server are one engine with two front ends, so the flags that matter are the same. -ngl decides how much of the model the accelerator holds, and partial offload is far worse than the fraction suggests. -sm, -ts and -mg divide a model across several GPUs and can be ignored on a single-device machine. -c allocates the KV cache up front, its default is taken from the model and can be far larger than you want, and --cache-type-k and --cache-type-v shrink it at a measurable quality cost. -fa selects flash attention and defaults to automatic. -t, -b and -ub matter when the CPU is involved or when prompt processing is the bottleneck. On the server, --jinja and the chat-template flags decide whether the model sees the format it was trained on, -np divides the context into slots, the OpenAI-compatible endpoints make it a drop-in for other software, and the default of 127.0.0.1 is a safety property worth keeping.

Check your understanding

Question 1. A 20 GB model is loaded on a 16 GB GPU with -ngl set so that about three quarters of the layers are offloaded. What should you expect?

• About three quarters of the full-GPU speed
• Much less than three quarters, because every token still passes through the layers left on the CPU and, on a discrete GPU, crosses PCIe both ways
• The same speed as full offload
• A refusal to load

Show the answer and why

Answer: Much less than three quarters, because every token still passes through the layers left on the CPU and, on a discrete GPU, crosses PCIe both ways

The slowest stage sets the pace, and the transfer is per token. This is why a smaller quantisation that fits entirely usually beats a larger one that is partly offloaded, which is a measurement the lab in this part asks you to make.

Question 2. You start llama-server without -c on a model whose card gives a 32,768-token context. What happens to memory?

• The KV cache grows as the conversation grows
• The server takes the context length from the model and allocates the whole KV cache up front, which for an 8B model at FP16 is several gigabytes
• No KV cache is allocated until the first request
• The context is capped at 2,048 tokens

Show the answer and why

Answer: The server takes the context length from the model and allocates the whole KV cache up front, which for an 8B model at FP16 is several gigabytes

The documented default for -c is 0, meaning "from the model", and the cache is allocated at load. Qwen3-8B keeps about 144 KiB per token at FP16, so 32k of context is roughly 4.8 GB before the weights are counted.

Question 3. Which pair of flags reduces KV cache memory without changing the model file?

• -b and -ub
• --cache-type-k and --cache-type-v
• -sm and -ts
• --jinja and --chat-template

Show the answer and why

Answer: --cache-type-k and --cache-type-v

Both accept f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, with f16 as the default. Setting both to q8_0 halves the cache. The batch flags change prefill throughput, not cache size.

Question 4. A model answers, but ignores the system prompt and sometimes prints its own turn markers as text. What should you check first?

• The quantisation type
• The chat template: what the server renders for a message list, which /apply-template will show you
• The GPU driver
• The context length

Show the answer and why

Answer: The chat template: what the server renders for a message list, which /apply-template will show you

Those are the classic symptoms of a template mismatch, and they look like a bad model rather than a configuration error. /apply-template prints the exact string the server would send, and --chat-template-file lets you replace a wrong one.

Question 5. Why is llama-server's default host of 127.0.0.1 a safety property rather than an inconvenience?

• It makes the server faster
• It means the API is reachable only from the same machine; changing it to 0.0.0.0 exposes an unauthenticated generation endpoint to the network
• It enables the web interface
• It is required for the OpenAI-compatible endpoints

Show the answer and why

Answer: It means the API is reachable only from the same machine; changing it to 0.0.0.0 exposes an unauthenticated generation endpoint to the network

The server has no authentication unless you set --api-key. Binding to all interfaces without one hands anyone who can reach the port full use of your machine's GPU, and whatever the model has been given access to.

Check my answersClear