Specialist Engines: ExLlamaV3, ktransformers and mistral.rs

By the end of this lesson you will be able to name three engines that are not trying to be general-purpose, say which single problem each one exists to solve, judge from a project’s own release history how much of your time it is safe to invest, and recognise the situations in which reaching for a specialist is the right call rather than an affectation.

The generalists in this course are llama.cpp, vLLM and the vendor stacks. They are the right default because they run many models on many machines and they are maintained by many people. A specialist engine earns its place by beating them at exactly one thing, and the skill this lesson teaches is recognising when you have that one thing.

ExLlamaV3: the most model per gigabyte of video memory

The problem it solves is Track N’s problem. A discrete GPU has a hard memory ceiling, a model either fits or does not, and the difference between “fits at four bits” and “fits at three and a half bits” is the difference between running a 32B model on a 24 GB card and not.

The README describes ExLlamaV3 as “an inference library for running local LLMs on modern consumer GPUs”. Its format is EXL3, described as “a streamlined variant of QTIP from Cornell RelaxML” — that is, a quantisation scheme from a published method rather than an ad-hoc one, which is worth knowing because it is the reason the format behaves differently from GGUF’s K-quants at the same nominal bit rate. The README says conversion computes Hessians on the fly with a fused Viterbi kernel, taking “a couple of minutes for smaller models, up to a few hours for larger ones”.

Quantising is one command, and the bit rate is a number you choose rather than a name you pick from a list:

Fragment — not complete on its own

python convert.py -i <input_dir> -o <output_dir> -w <working_dir> -b <bitrate>

A job that dies part-way resumes from its working directory with python convert.py -w <working_dir> -r, which matters when the job is hours long. Running a converted model directly is python examples/chat.py -m <input_dir> -mode <prompt_mode>, but that is a demonstration rather than a deployment. The README is explicit about the intended route: “The official and recommended backend server for ExLlamaV3 is TabbyAPI, which provides an OpenAI-compatible API for local or remote inference.”

Requirements and limits, as the README states them. Torch 2.6.0 or newer and CUDA 12.4 or newer. Under “What’s missing?”, ROCm support is listed as still to do. So this is an NVIDIA-only engine today, which is why the course places it on Track N and mentions it nowhere else as a primary path.

Maturity, on 2026-09-09. The releases page listed v1.4.8 on 2026-09-06, v1.4.7 on 2026-09-05, v1.4.6 on 2026-09-02, v1.4.5 on 2026-08-31 and v1.4.4 on 2026-08-26. That is five releases in twelve days, which tells you two things at once: the project is very actively maintained, and it is moving fast enough that pinning a version and recording it is not optional. The course pins ExLlamaV3 1.4.8 · verified 2026-09-08. The licence is MIT.

⌚Version note

The README read on 2026-09-09 gave a prebuilt-wheel installation example whose URL names version 0.0.6, while the releases page on the same day listed 1.4.8 as current. Do not copy the example URL: take the wheel name from the release you actually intend to install, or use pip install exllamav3, or build from source with uv sync. A stale example in a README is common and harmless as long as you notice it.

TabbyAPI, the server in front of it

TabbyAPI describes itself as “A FastAPI based application that allows for generating text using an LLM (large language model) using the Exllamav3 backend”, with ExLlamaV2 also supported. It is started from a checkout with start.sh, or run from a Docker image, and it is configured through a YAML file copied from config_sample.yml rather than through command-line flags. The Docker example exposes the API on http://localhost:5000.

Its feature list names the things this part’s lab probes: an OpenAI-compatible API, “OAI style tool/function calling”, and “JSON schema + Regex + EBNF support” for constrained output. That last item is a genuine differentiator; not every server in this part offers grammar-level control.

!Important

TabbyAPI is licensed AGPL-3.0, which is a stronger copyleft than the MIT and Apache-2.0 licences attached to most of the software in this course. That is fine for personal use and for evaluation. If you are considering it inside a product, read the licence properly rather than taking a course page’s word for it: the obligations of the AGPL attach to network use, which is exactly what a server is.

ktransformers: a very large mixture-of-experts model on one GPU

The problem it solves is the one that makes people give up on the largest open-weight models. A mixture-of-experts model like Qwen3-235B-A22B or GLM-4.6 has a very large total parameter count and a much smaller active count per token. The total is what has to be in memory somewhere; the active count is what has to be computed. A machine with one consumer GPU and a lot of system RAM has the memory in the wrong place, and a conventional engine either fits everything on the GPU or gives up.

ktransformers describes itself as “A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations”, and its approach is heterogeneous placement: the parts that are read for every token stay on the GPU, while the expert weights that are read only when routed to live in system memory and are computed by “CPU-optimized kernel operations for heterogeneous LLM inference”.

Why expert offload is different from ordinary layer offload

1. 1A dense model, partially offloadedEvery offloaded layer is read for every token, so the slow path is on the critical path for every token generated. This is the case Part 6 warned about.
2. 2A mixture-of-experts modelEach token routes to a small subset of experts. Most expert weights are not read for most tokens.
3. 3Experts in system memory, attention on the GPUThe GPU holds what is read every time; the CPU holds and computes what is read occasionally. The slow memory is no longer on the critical path for every token.
4. 4The costThroughput now depends on CPU kernels and system memory bandwidth, on a specific instruction set, with a build to match. This is a narrower machine requirement than a GGUF file.

The README’s own examples are of the shape “this model on this hardware”, including DeepSeek-V3 and R1 class models on multi-GPU servers with Xeon CPUs, and Qwen3-30B-A3B on a single RTX 4090 with roughly 24 GB in play. Those are the project’s reported figures on its own hardware, not measurements this course has reproduced, and the hardware is as much a part of the claim as the number.

The CPU is the requirement to check. The project’s kernels target “Intel AMX and AVX512/AVX2 optimized kernels”, with an AVX2-only backend available. Version 0.7.0 added AMD AVX-512 support. If your CPU does not have the instruction set the fast path assumes, you get a different engine from the one in the benchmarks.

Maturity, on 2026-09-09. The course pins ktransformers 0.7.0 · verified 2026-09-08, released 2026-08-17. The licence is Apache-2.0. Installation on the version read was from the kt-kernel directory with pip install ., with a separate extra for the fine-tuning integration. This is a research-adjacent project with a build step, not a package you install and forget, and the course surveys it rather than teaching it hands-on for that reason.

✱Hardware note

Track S and Track X readers have a unified-memory machine, which solves the same problem a different way: the memory is already in one pool that the GPU can address, so a large mixture-of-experts model does not need expert offload to run. That is exactly what the Part 6 lab demonstrated with gpt-oss-120b. ktransformers is for the machine where the memory and the compute are in different places, which is Track N with a lot of system RAM.

mistral.rs: one binary that serves everything

The third specialist is a different kind. mistral.rs is an inference engine written in Rust, and what it optimises for is not a single hardware constraint but the shape of the deployment: a single compiled binary with no Python environment, covering text, vision, speech, image generation and embedding models behind one server.

The README lists CUDA with FlashAttention, Metal on Apple silicon, and CPU as its accelerators. Installation is a shell installer on Linux and macOS, a PowerShell equivalent on Windows, or pip install mistralrs and cargo add mistralrs for use as a library. The server is started with mistralrs serve -m <model> and listens on port 1234 by default, with a web interface at /ui.

Three things on its feature list matter for the rest of this course. It “exposes OpenAI-compatible /v1 endpoints” and also Anthropic-compatible messages endpoints, which is unusual and useful if your client speaks that dialect. It supports a long list of quantisation formats — GGUF from two to eight bits, GPTQ, AWQ, HQQ, FP8, bitsandbytes — plus in-situ quantisation, which quantises a Hugging Face checkpoint as it loads rather than requiring a converted file first. And it advertises “Integrated tool calling with grammar enforcement and strict schema mode” together with a server-side agentic loop, which is the same territory Part 24 covers at the protocol level.

The licence is MIT. The README read on 2026-09-09 did not state a release version, and the project is not in this course’s pinned version table, so nothing on this page should be treated as version-specific: check the current release before relying on any option named here.

Judging maturity without a benchmark

The three projects above illustrate a general technique, and it is more useful than any of the three. When you are deciding whether to invest a weekend in an engine, the questions that actually predict the outcome are these, and every one of them is answerable from the project’s own pages in five minutes.

When was the last release, and how many were there in the last month? Five releases in twelve days means active maintenance and rapid change. One release in a year means either stability or abandonment, and the commit history tells you which.

Does it name the hardware it requires, specifically? “CUDA 12.4 or newer” and “AVX-512” are answerable questions about your machine. “Modern GPUs” is not.

Is there a documented server, or only an example script? An engine with an OpenAI-compatible server can be dropped into the rest of your setup. An engine with only a chat example is a component you will have to wrap yourself.

What is the licence, and does it match how you intend to use it? MIT and Apache-2.0 are permissive. AGPL-3.0 attaches obligations to network use. This is not a detail to discover after you have built something on it.

Does its README’s own examples still work? The stale wheel URL above is a small thing, and it is also a signal: documentation drifts in the same direction the code moves.

∴Research note

None of those questions is “is it fast”. Speed claims from a project’s own README are the project’s own measurements on the project’s own hardware, and they belong in the same category as a vendor benchmark: quoted with a source and a date, never repeated as fact. If speed is the reason you are considering an engine, this part’s lab is the shape of the answer, run on your machine.

When a specialist is the right call

Four situations, and outside them the generalists win on maintenance alone.

You are one quantisation step away from a model fitting. This is the ExLlamaV3 case, and it is a Track N case almost by definition, because it needs a hard memory ceiling to be worth optimising against.

You have a large mixture-of-experts model, one GPU and a lot of system RAM. This is the ktransformers case, and it is worth checking your CPU’s instruction set before you start.

You need one binary with no Python environment. This is the mistral.rs case, and it comes up more often in deployment than in learning.

You need a capability the generalists do not have. Grammar-level structured output through TabbyAPI is a real example; so is Anthropic-compatible endpoints in mistral.rs. Part 10 and Part 24 both depend on structured output working reliably, and it is legitimate to choose an engine for that alone.

Outside those, the cost of a specialist is real and easy to underestimate: a second model format to manage, a second server to keep running, a project with fewer maintainers, and a comparison you now have to redo whenever either side updates.

Track S — NVIDIA DGX SparkPartial

mistral.rs runs here; ExLlamaV3 and ktransformers target discrete NVIDIA GPUs and, for ktransformers, a machine where memory and compute are in different places.

Your 128 GB unified pool already solves the problem ktransformers exists for. mistral.rs is the one of the three worth trying on this machine, and the ExLlamaV3 section is worth reading for the EXL3 format, which you will meet in other people’s model repositories.

Track X — AMD Ryzen AI Max+ 395Partial

ExLlamaV3 lists ROCm support as still to do, so the EXL3 path is not available on this GPU.

mistral.rs is the portable option here. The AMD lesson earlier in this part is where your engine choices actually live.

Track M — Apple siliconPartial

ExLlamaV3 and ktransformers are CUDA-oriented; mistral.rs lists Metal support.

mistral.rs is the one to try, and MLX from earlier in this part is your native path. The EXL3 section is still worth reading, because the format appears in model repositories you will browse.

Track N — NVIDIA desktop or laptop

Your track, and the one all three were designed for. ExLlamaV3 through TabbyAPI is the third engine in this part’s lab on Track N. ktransformers is worth reading about now and returning to if you ever put a very large mixture-of-experts model on this machine.

Design a compatibility test before a speed contest

An engine can be excellent for one architecture or quantisation and unsuitable for another. Establish a compatibility contract first: checkpoint loading, tokeniser/template interpretation, context requirements, API features and the licence of the complete deployment. Test each required feature with a small request whose result you can inspect.

Only then compare performance. Hold prompt length, answer limit, concurrency and cache state constant. Preserve cold-start time separately from warm generation, and record resident memory. If the specialist engine requires a different quantisation, make that a named difference in the report rather than describing the experiment as the same model in every respect.

Include an exit path in the choice. Can you keep using the original checkpoint, export the adapter or reproduce the conversion if the specialised project changes direction? This is an operational requirement, not merely convenience. A measured speed gain can justify a narrower ecosystem when the model and workload are stable; frequent architecture changes can make a broadly supported engine the more maintainable option.

Recap

ExLlamaV3 exists to fit more model into a fixed amount of video memory, using the EXL3 format derived from QTIP, converted with convert.py at a bit rate you choose and served through TabbyAPI, which is AGPL-3.0 and supports tool calling and JSON-schema constrained output. It needs CUDA and does not yet support ROCm, and on 2026-09-09 it had released five times in twelve days. ktransformers exists to run a very large mixture-of-experts model on one GPU plus system memory by keeping experts on the CPU, and it depends on CPU instruction sets in a way a GGUF file does not. mistral.rs is a single Rust binary serving many model types with OpenAI and Anthropic-compatible endpoints, in-situ quantisation and grammar-enforced tool calling. The transferable skill is judging a project from its release cadence, its stated hardware requirements, whether it has a real server and its licence, before spending a weekend on it.

Check your understanding

Question 1. Why is ktransformers’ expert offload a different proposition from offloading half a dense model’s layers to the CPU?

• It uses a faster interconnect
• In a mixture-of-experts model most expert weights are not read for most tokens, so the slow memory is not on the critical path for every token; in a dense model every offloaded layer is
• It quantises the offloaded layers more aggressively
• It runs the whole model on the CPU

Show the answer and why

Answer: In a mixture-of-experts model most expert weights are not read for most tokens, so the slow memory is not on the critical path for every token; in a dense model every offloaded layer is

This is the same sparsity that made gpt-oss-20b generate like a much smaller model in the Part 6 lab, used in a different way. The cost is that throughput now depends on CPU kernels and an instruction set, which is a narrower machine requirement than a portable file.

Question 2. You are choosing an engine for a service you intend to distribute. Which licence on this page needs the closest reading before you commit?

• MIT, as used by ExLlamaV3 and mistral.rs
• AGPL-3.0, as used by TabbyAPI, because its obligations attach to network use and a server is network use
• Apache-2.0, as used by ktransformers
• They are equivalent for a server

Show the answer and why

Answer: AGPL-3.0, as used by TabbyAPI, because its obligations attach to network use and a server is network use

Permissive licences ask little. The AGPL was written specifically so that running modified software as a network service triggers the same obligations as distributing it. Whether that suits you is a decision to make deliberately and early.

Question 3. Which of these tell you something useful about a project’s maturity in five minutes? Select all that apply.

• The dates and count of its recent releases
• Whether it names specific hardware requirements such as a CUDA version or an instruction set
• Whether it ships a documented server or only an example script
• The tokens-per-second figure in its README

Show the answer and why

Answer: The dates and count of its recent releases, Whether it names specific hardware requirements such as a CUDA version or an instruction set, Whether it ships a documented server or only an example script

The first three are checkable facts about the project. A README speed figure is the project’s own measurement on its own hardware; it may be honest and it is still not evidence about your machine, which is what the lab in this part produces.

Question 4. A Track X reader wants to try EXL3 quantisation on their Radeon 8060S. What does the ExLlamaV3 README say?

• It works through Vulkan
• ROCm support is listed under "What’s missing?", so there is no supported path on this GPU today
• It requires a rebuild with a HIP flag
• It works but only for models under 8B parameters

Show the answer and why

Answer: ROCm support is listed under "What’s missing?", so there is no supported path on this GPU today

The requirements are Torch 2.6 or newer and CUDA 12.4 or newer, and ROCm is on the to-do list. That is why the course places this engine on Track N only, and why the Track X lab compares Vulkan, HIP and Lemonade instead.

Check my answersClear