Ollama: Models as a Service

By the end of this lesson you will be able to install Ollama on your own track and say whether it is using the graphics processor or the central one, pull and run a model from its library, build your own variant of a model with a Modelfile, call it from both of its HTTP interfaces, and — the part that matters most — state what context length and keep-alive it is giving you, rather than finding out later from a conversation that has quietly forgotten its own beginning.

Everything below was read from the documentation of Ollama 0.33.3 · verified 2026-09-08.

What the one-line install actually gives you

Part 6 left you with a binary, a GGUF file and a command with a dozen flags in it. Ollama replaces all three with a name: ollama run qwen3:4b. The convenience is real, and it is worth knowing which layers it is folding together.

What sits between your prompt and the arithmetic

a request travels down and the tokens come back up

1. Your clientA chat front-end, an editor plugin, curl, or any library written for the OpenAI API.swap freely
2. Two HTTP interfaces on port 11434Ollama's own /api/chat and /api/generate, and an OpenAI-compatible surface under /v1.the stable part
3. SchedulerDecides which models are resident, how many run at once, how long they stay, and whether a model that does not fit is split with the CPU.the surprising part
4. Bundled inference runtimeShips with the install, reads GGUF weights, and picks a backend: CUDA, ROCm, Vulkan or Metal.Part 6's ground
5. Model storeContent-addressed blobs plus one manifest per tag, so two tags that share weights store them once.
6. Driver and hardwareThe layer Part 5 made you check, and the one that decides whether any of this is fast.

The scheduler is the layer with no equivalent in Part 6, and it is where the defaults live.

Installing on your track

Track S — NVIDIA DGX Spark

DGX OS is Linux on aarch64, and the install script selects the right build:

RunnableTrack S · DGX Spark

install on DGX OS

curl -fsSL https://ollama.com/install.sh | sh

The manual route, if you would rather see what lands where, is the ARM64 tarball the Linux documentation lists: it is extracted into /usr/ and started with ollama serve. The hardware-support page lists compute capability 12.1, GB10 (DGX Spark), among the supported NVIDIA families, so the CUDA backend is the expected path here.

Track X — AMD Ryzen AI Max+ 395

The same install script. Whether you get the ROCm backend or the Vulkan one depends on what is on the machine, and the difference is worth a paragraph of its own below.

RunnableTrack X · Ryzen AI Max+

install on Linux

curl -fsSL https://ollama.com/install.sh | sh

The Linux page also documents a separate ROCm package, ollama-linux-amd64-rocm.tar.zst, for the manual install.

Track M — Apple silicon

The download page states macOS 14 Sonoma or later and Apple M-series silicon for GPU support; the macOS page adds that an Intel Mac runs on the CPU only. The preferred install is the disk image dragged to Applications, and on first start the application offers to put the ollama command in your path.

Models live under ~/.ollama, on the volume your home directory is on. The macOS page is explicit that this needs “additional space for storing the Large Language models, which can be tens to hundreds of GB in size”.

Track N — NVIDIA desktop or laptop

On Linux and in WSL2, the install script. On Windows, the installer from the download page; Ollama runs natively there and does not need WSL2.

RunnableTrack N · NVIDIA GPU

install on Linux or WSL2

curl -fsSL https://ollama.com/install.sh | sh

The hardware-support page states compute capability 5.0 and newer with driver 550 or newer, and driver 570 or newer for the older 5.0 to 6.2 cards. If your card is not on that list, Ollama can still reach it through Vulkan.

⛨Security

curl … | sh is the vendor’s documented install path on Linux and macOS, and it is also a pattern to be deliberate about. Fetch the script to a file, read it, then run it, if that is your habit; the download page offers ordinary signed installers for macOS and Windows if you would rather not pipe anything into a shell.

Did it find the GPU?

This is the first question and the one people skip. The answer is in one column of one command:

RunnableAll tracks

load a model and see where it went

ollama run qwen3:4b "say hello"
ollama ps

Output — what you should see

NAME         ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b     xxxxxxxxxxxx    x.x GB    100% GPU     4096       4 minutes from now

The FAQ documents that column exactly: 100% GPU means the model was loaded entirely into the graphics processor’s memory, 100% CPU means it was loaded entirely into system memory, and a form like 48%/52% CPU/GPU means it was split between them. A split is not an error; it is the scheduler fitting what it could and running the rest slowly, and it is the usual explanation for “why is this so much slower than the video I watched”.

✱Track X: ROCm or Vulkan, and which one you get

Ollama’s hardware-support page lists Ryzen AI Max+ 395 under AMD Ryzen AI in its Linux ROCm table, and its LLVM-target table maps gfx1151 to that same processor (checked 8 September 2026). The page states that Ollama “requires the AMD ROCm v7 driver on Linux”, installed with AMD’s amdgpu-install. So on Linux with a current driver stack the ROCm path is the documented one for this chip.

Two caveats that are also on the page. The Windows ROCm table lists only discrete Radeon RX and PRO cards, and no Ryzen AI part, so on Windows the documented route to this integrated graphics processor is Vulkan, which the page says is “enabled by default when the backend is installed”. And Vulkan needs help to report free memory: without root or the cap_perfmon capability, “Ollama will use approximate sizes of the models to make best effort scheduling decisions”, which is a polite way of saying it will sometimes decide wrongly whether a model fits.

If neither backend claims the device, the model still runs. ollama ps will say 100% CPU, and on a machine with this much memory bandwidth that is a real fallback rather than a failure — just a much slower one.

The library, and what a tag means

ollama pull qwen3:4b fetches from Ollama’s own library rather than from the Hugging Face Hub. The part before the colon is the model, the part after is the tag, and the tag is doing more work than a version number: it usually encodes the parameter count and the quantisation together, so qwen3:4b and qwen3:30b-a3b are different models under one name, and a tag that does not say a quantisation has a default one chosen for you.

RunnableAll tracks

pull, list, and read the licence you just accepted

ollama pull qwen3:4b
ollama list
ollama show qwen3:4b --license

Qwen3-4B is Apache-2.0 licensed, as the course’s model reference records; ollama show --license prints the text that was packaged with the weights, which is the habit Part 3’s licence lesson asked for, applied at the moment of download rather than the moment of embarrassment.

✕Common mistake

The commonest confusion here is treating an Ollama tag as the same artefact as the publisher’s repository on the Hub. It is a conversion, made by someone, at some quantisation, wrapped in a template that someone chose. ollama show qwen3:4b --modelfile prints exactly what you got, including the template and the stop tokens. When a model behaves differently under Ollama and under llama-server with the same weights, that output is the first place to look.

Where the weights went

The FAQ gives three paths: ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows. Note that the Linux path is not in your home directory, because the installer creates a service user. Set OLLAMA_MODELS to move it, and on Linux give the ollama user read and write access to wherever you point it — the FAQ spells out sudo chown -R ollama:ollama <directory>.

Inside, the layout is blobs named by hash plus manifests, which is why the disk usage does not obviously map to the model names in ollama list. Part 7’s storage lesson deals with that properly, including how to stop this store from being a second copy of models you already have in ~/models.

The two defaults that decide how your conversations go

Context length

Here the documentation disagrees with itself, and saying so is more useful than picking a winner. The Modelfile reference gives num_ctx a “(Default: 2048)”. The FAQ says “By default, Ollama uses a context window size of 4096 tokens.” The dedicated context-length page says something different again:

Ollama defaults to the following context lengths based on VRAM: < 24 GiB VRAM: 4k context; 24-48 GiB VRAM: 32k context; >= 48 GiB VRAM: 256k context

All three were read on 8 September 2026. They cannot all describe the version you have installed, and none of them is a substitute for looking. The CONTEXT column of ollama ps is the number your machine is actually using, and the reality check at the end of this part turns that into a measurement rather than a reading.

Whatever the default turns out to be, you can set it. For the whole server:

RunnableAll tracks

start the server with a context length you chose

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Inside a session, /set parameter num_ctx 8192. Through the native API, an options object carrying num_ctx. And through the OpenAI-compatible API — not at all. The compatibility page is blunt about it: “The OpenAI API does not have a way of setting the context size for a model”, and the documented workaround is to bake the value into a model of your own with a Modelfile. That limitation matters as soon as a chat front-end is involved, because front-ends talk /v1.

▤Memory budget

Raising the context length raises memory use, and the Part 4 arithmetic tells you by how much before you try it. Qwen3-8B carries about 147 KB of key-value cache per token at 16-bit, so 8,192 tokens is roughly 1.2 GB and 32,768 tokens is roughly 4.8 GB, on top of the weights. Ollama can halve or quarter that: OLLAMA_KV_CACHE_TYPE accepts f16 (the default), q8_0 and q4_0, and the FAQ notes the cache quantisation needs flash attention, which Ollama enables automatically where the backend supports it.

Keep-alive

“By default models are kept in memory for 5 minutes before being unloaded.” For a laptop that is polite. For a household service that gets a question every twenty minutes, it means almost every question pays the load time again. OLLAMA_KEEP_ALIVE sets it for the whole server, the keep_alive request field overrides it per call, a negative value keeps the model loaded indefinitely, and 0 unloads it as soon as the answer is finished.

RunnableAll tracks

unload a model right now

ollama stop qwen3:4b
ollama ps

The trade is memory against latency, and it is a genuine choice rather than a setting with a right answer. The lab later in this part sets it to thirty minutes for exactly this reason, and says so in the file.

Making a model your own

A Modelfile is a small text file that produces a new model name from an existing one. The reference lists seven instructions: FROM (required), PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE and REQUIRES.

Pseudocode — not a real command

FROM qwen3:4b
PARAMETER num_ctx 8192
PARAMETER temperature 0.4
SYSTEM """You answer in British English, briefly, and say when you do not know."""

Save that as Modelfile and build it:

RunnableAll tracks

build a variant and run it

ollama create house-assistant -f Modelfile
ollama show house-assistant --parameters
ollama run house-assistant "what is a KV cache?"

Three things this is genuinely good for. It is the documented way to give an OpenAI-compatible client a larger context, since the client cannot ask for one. It pins a system message so that every caller gets the same behaviour without every caller having to send it. And ADAPTER attaches a LoRA adapter to a base model, which is how the fine-tune you produce in Part 13 becomes something the rest of the house can talk to — with the reference’s warning attached, that an adapter applied to a base model it was not trained from behaves erratically.

FROM also accepts a path to a GGUF file, which is the hook the storage lesson uses to make Ollama read a model you already downloaded instead of pulling a second copy.

Two APIs, and when the difference bites

Ollama serves its own API and an OpenAI-compatible one on the same port. The native one carries the things the OpenAI shape has no room for: options with num_ctx and the sampling parameters, keep_alive, and per-request counters such as prompt_eval_count and eval_count that tell you how many tokens were actually read and written.

RunnableAll tracks

one non-streaming request to the native API

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:4b",
  "messages": [{"role": "user", "content": "In one sentence, what is a KV cache?"}],
  "stream": false,
  "options": {"num_ctx": 8192, "temperature": 0}
}'

The compatible one is what every other tool in this course expects. The base URL is http://localhost:11434/v1/, the API key is “required but ignored”, and the documentation lists which request fields are supported and which are not: tools yes, tool_choice no, seed, response_format, stop, stream_options and reasoning_effort yes, logit_bias and n no. There is a /v1/responses endpoint too, non-stateful only.

By default the server “binds 127.0.0.1 port 11434”, so none of this is reachable from another machine until you change OLLAMA_HOST — which the lab does deliberately, and behind a proxy rather than in the open.

More than one graphics processor

The FAQ describes the rule plainly: if the model fits entirely on one device it goes on that device, “as this reduces the amount of data transferring across the PCI bus during inference”, and only if it does not fit is it spread across all of them. That is pipeline placement, not tensor parallelism; Part 18 draws the distinction and Part 20 measures it.

Restricting which devices Ollama may use is per vendor: CUDA_VISIBLE_DEVICES for NVIDIA (the page recommends UUIDs from nvidia-smi -L over numeric ids, because “ordering may vary”), ROCR_VISIBLE_DEVICES for AMD, GGML_VK_VISIBLE_DEVICES for Vulkan. Two more scheduler knobs are worth knowing before they surprise you: OLLAMA_MAX_LOADED_MODELS, whose default the FAQ gives as three times the number of GPUs, and OLLAMA_NUM_PARALLEL, default one, with the warning that required memory “will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH”. Four parallel slots at 8,192 tokens is a 32,768-token cache allocation, and a machine that was fine yesterday is not fine today.

⌚Version note

Environment variables reach the server differently on each platform, and setting them in your shell usually does nothing because the server is not your shell’s child. On Linux the FAQ documents systemctl edit ollama.service and an Environment= line; on macOS, launchctl setenv followed by restarting the application; on Windows, the user environment variables dialogue and then restarting Ollama from the Start menu. Checked 8 September 2026 against the pinned version.

Distinguish a model tag from resident server state

A model tag identifies a configured artefact in the model library. The server may have no copy resident, one model warming up, or several models competing for memory. A cold request therefore measures loading and initialisation as well as inference; a warm request measures a different path.

When comparing tags, keep the template, sampling settings and context allocation visible. A customised model definition can change behaviour without changing the base tensors. Record the definition alongside the model identity so a colleague can reproduce both the weights and the prompt wrapper.

Exercise the service with a cold request, a warm repeat and a request after using another model. Observe which transitions reload weights and how long the application waits. This exposes the operational cost of keeping a large collection on a small machine. Downloaded capacity is a disk question; simultaneous residency is a memory question. A library can contain many useful models while the serving policy deliberately permits only one large model at a time.

Recap

• Ollama is a scheduler, a model store and two HTTP interfaces wrapped around an inference runtime. The runtime is the part Part 6 taught; the scheduler is the part that has opinions.
• ollama ps answers the only two questions that matter after an install: which processor is running the model, and what context length it was given.
• The context default is documented three different ways in three different places. Read the machine, not the page.
• Models are kept in memory for five minutes by default. For a shared service that is usually the wrong number, in the direction that costs latency.
• A Modelfile is how you change a context length for clients that cannot ask for one, pin a system message, or attach a LoRA adapter.
• The OpenAI-compatible endpoint is what front-ends and agent frameworks speak, and it cannot set the context size. Everything else in this part inherits that fact.

Check your understanding

Question 1. ollama ps prints "48%/52% CPU/GPU" for the model you just loaded. What does that mean?

• The model is being processed twice
• The model did not fit in the graphics processor's memory, so part of it was loaded into system memory and is being run there
• Ollama is using half of each processor to save power
• The model is quantised to half precision

Show the answer and why

Answer: The model did not fit in the graphics processor's memory, so part of it was loaded into system memory and is being run there

The FAQ documents the PROCESSOR column exactly this way. A split load is the usual explanation for a model running far slower than expected: the layers in system memory are read over a much slower path, and the arithmetic from Part 4 predicts the size of the penalty.

Question 2. You point a chat front-end at Ollama through http://localhost:11434/v1 and want an 8,192-token context. What does the documentation say to do?

• Send num_ctx in the request body
• The OpenAI-compatible API has no way to set the context size, so create a model with a Modelfile containing PARAMETER num_ctx 8192 and call that model instead
• Pass ?num_ctx=8192 in the URL
• Set it in the front-end's advanced parameters, which are passed through

Show the answer and why

Answer: The OpenAI-compatible API has no way to set the context size, so create a model with a Modelfile containing PARAMETER num_ctx 8192 and call that model instead

The compatibility page states the limitation and gives the Modelfile workaround. This is the single most useful thing to know before deploying a front-end, because otherwise the context length is whatever the server defaulted to and no part of the user interface will tell you.

Question 3. Which of these are true of Ollama's defaults as documented? Select all that apply.

• A model is unloaded from memory five minutes after its last request
• The server binds to 127.0.0.1 on port 11434
• OLLAMA_NUM_PARALLEL defaults to 1, and raising it multiplies the memory needed for the cache
• Models are stored in your home directory on every platform

Show the answer and why

Answer: A model is unloaded from memory five minutes after its last request, The server binds to 127.0.0.1 on port 11434, OLLAMA_NUM_PARALLEL defaults to 1, and raising it multiplies the memory needed for the cache

The last is the trap: on Linux the standard installer creates a service user and the store lives at /usr/share/ollama/.ollama/models, not under your home directory. That is also why moving it with OLLAMA_MODELS needs a chown.

Question 4. On a Ryzen AI Max+ 395 running Linux, what does Ollama's hardware-support page say about acceleration?

• It is unsupported and will run on the CPU
• It is listed under AMD Ryzen AI in the ROCm table and maps to LLVM target gfx1151, with the ROCm v7 driver required; Vulkan is the additional path, and is the documented one on Windows for this part
• Only Vulkan is supported on any operating system
• It requires a discrete Radeon card to be present as well

Show the answer and why

Answer: It is listed under AMD Ryzen AI in the ROCm table and maps to LLVM target gfx1151, with the ROCm v7 driver required; Vulkan is the additional path, and is the documented one on Windows for this part

Both tables on that page matter, and they differ by operating system. Checking which backend actually claimed the device — and whether Vulkan can report free memory — is worth doing before blaming the model for being slow.

Question 5. Why does the course say a Modelfile-built model is a different artefact from the model it came from?

• The weights are copied and modified
• Because the template, stop tokens, system message and parameters travel with the model name, so two names sharing the same weights can behave very differently
• Because ollama create re-quantises the weights
• It does not; the model is identical

Show the answer and why

Answer: Because the template, stop tokens, system message and parameters travel with the model name, so two names sharing the same weights can behave very differently

The weights are shared in the content-addressed store, which is why building a variant costs almost no disk. What changes is everything around them, and ollama show --modelfile is how you see exactly what a name resolves to.

Check my answersClear