Skip to content
Local LLM Course

Platform-Native Engines: MLX, TensorRT-LLM and ROCm

Part 6 built one engine that runs everywhere and Part 7 wrapped it in applications. Both were deliberately portable: the same source tree, the same file format, the same commands on four very different machines. Portability has a price, and this part is where you find out what it is on your hardware.

Every platform in this course has a second path, written by the company that made the silicon or by people working closely with it. Apple has MLX. NVIDIA has TensorRT-LLM and the NIM containers. AMD has ROCm builds and Lemonade Server. Each of those knows things about its own hardware that a portable engine cannot assume, and each asks for something in return: a different file format, a container, a build step, a narrower list of supported models, or a dependency on a release cadence you do not control.

The honest answer to “is the native path faster?” is that it depends on the model, the quantisation, the context length and the week, and that you should measure it rather than believe anyone about it, including this course. That is why this part ends with a lab that runs one model through two engines on the same machine and writes both results into the same notebook.

There is a second reason this part exists. Several engines in the open-weight world are not general-purpose at all: they solve one problem unusually well and are the wrong choice for anything else. ExLlamaV3 exists because people want a bigger model in a fixed amount of video memory. ktransformers exists because people want a very large mixture-of-experts model on a machine with one consumer GPU and a lot of system memory. Knowing those by name, and knowing which problem each one answers, is worth more than being able to configure any of them from memory.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain what MLX’s unified memory and lazy evaluation mean for a model running on a Mac, convert a Hugging Face checkpoint to MLX format at a quantisation you choose, and serve it over an OpenAI-compatible endpoint;
  • describe how TensorRT-LLM turns a checkpoint into something it can serve, which quantisation formats each NVIDIA GPU generation supports, what a NIM microservice is, and how the DGX Spark playbooks package all of it;
  • state what the ROCm stack does and does not document for the Radeon 8060S today, build llama.cpp against HIP, drive Lemonade Server, and say precisely what the Ryzen AI NPU is documented to run and under which operating system;
  • name the specialist engines, say which problem each one solves, and judge from a project’s own release history how much of your time it is safe to invest in it;
  • measure prefill, decode, resident memory, tool calling and structured output for the same model on two engines, and record the comparison in a form that a later part can use.

What you need first

Section titled “What you need first”

A working llama-server from Part 6, because it is the control in every comparison here, and the benchmark numbers you recorded in that part’s lab. Part 5’s description of your own machine, especially the memory ceiling: Track X readers need the GPU-visible memory cap and Track M readers need the wired-memory limit. The lab notebook from Part 1.

The memory floor for the lab is 12 GB, which runs Qwen3-8B at a four-bit quantisation on two engines one after the other, with room for the KV cache. Everything else in this part can be read on any machine.

Each lesson is written for one track first and the other three second. Read the one for your machine with a terminal open; read the other three quickly, because a large part of what makes a platform choice sensible is knowing what the alternatives cost.

How to work through it

Section titled “How to work through it”

Start with the lesson for your own track. Track M readers begin with MLX, Track S with TensorRT-LLM, Track X with the AMD lesson and Track N with the specialist engines, which is where ExLlamaV3 and ktransformers live. Then read the other three at reading pace; they are short, and Part 9 assumes you know what a NIM is and why a Mac cannot run vLLM’s mainline GPU path.

Do the lab last, in one sitting, on the machine you actually use. It is the first page in the course that puts two different engines side by side under one methodology, and its results are referenced again when Part 9 compares serving engines and when Part 16 asks whether a quantisation changed the answers as well as the speed.

Part8 of 27Level2 — Local OperatorPages5Estimated time3.0 hours

0 / 5 lessons in this part completed