MLX and mlx-lm: Apple's Native Path

By the end of this lesson you will be able to convert a Hugging Face checkpoint into MLX format at a quantisation you chose deliberately, generate from it on the command line, serve it over an OpenAI-compatible HTTP endpoint, and describe in one sentence each what MLX’s two headline design decisions actually change for a model running on your Mac.

This lesson is written for Track M. The other three tracks should read it anyway, because MLX is the clearest example in the course of an engine designed around one memory architecture, and because Part 9 sends Mac readers back here because vLLM’s GPU path does not cover macOS.

What MLX is, and what it is not

MLX is an array framework from Apple’s machine learning research group. It is not a language-model runtime; it is the layer underneath one, in roughly the position PyTorch occupies on an NVIDIA machine. mlx-lm is the separate package that uses it to load, quantise, generate from and serve language models, and that is the tool you will actually type.

The MLX documentation names three differences from NumPy, and two of them matter here.

Unified memory. The documentation states that “Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device type without performing data copies”, and that “Rather than moving arrays to devices, you specify the device when you run the operation.” On a machine where the CPU and the GPU address the same physical memory, that is not a convenience, it is the removal of an entire category of work. There is no .to("cuda"), no host-to-device transfer, and no separate video-memory budget to keep a model inside.

Lazy evaluation. “When you perform operations in MLX, no computation actually happens. Instead a compute graph is recorded.” Work happens when an array is materialised, which the documentation says occurs when you print it, convert it to a NumPy array, save it, or call mx.eval yourself. The practical effect for a reader of this course is that a model’s weights are not built in memory until something forces them to be, which is how a large model can be loaded without a spike that the machine cannot absorb.

The third difference, composable function transformations for automatic differentiation and vectorisation, is what makes MLX a training framework as well as an inference one. Part 13 uses that side of it when it fine-tunes on a Mac.

Where mlx-lm sits on an Apple silicon Mac

each layer depends on the one below

1. Apple silicon SoCCPU, GPU and Neural Engine sharing one pool of memory at one bandwidth figure.fixed at purchase
2. MetalApple's GPU programming interface. MLX compiles its kernels against it.ships with macOS
3. MLX coreArrays in shared memory, a lazily built compute graph, function transforms.pip install
4. mlx-lmModel loading, chat templates, quantisation, generation, an HTTP server.what you type
5. OpenAI-compatible HTTPChat completions, completions and a model list, on port 8080 by default.
6. Your clientAny library that speaks the OpenAI API, including the scripts in this part.

⌚Version note

This lesson was written against mlx-lm 0.31.3 · verified 2026-09-08 and the documentation as it read on 2026-09-09. The command-line options below were read from the package’s own argument definitions rather than from a tutorial, because mlx-lm renames options between releases more readily than llama.cpp does. If an option is rejected, run the tool with -h and map it across rather than guessing.

Installing and generating

Installation is one line, and there is a conda-forge package for readers who prefer it.

RunnableTrack M · Apple silicon

install mlx-lm

pip install mlx-lm

The README’s first example runs with no model specified at all, because the package has a default: mlx-community/Llama-3.2-3B-Instruct-4bit. That is a convenient smoke test and a poor benchmark, so name a model as soon as you have one.

RunnableTrack M · Apple silicon

generate from a model on the Hub

mlx_lm.generate \
    --model mlx-community/Qwen3-8B-4bit \
    --prompt "Explain what a KV cache is, in three sentences." \
    --max-tokens 200 \
    --temp 0.7 \
    --seed 42

The first run downloads the repository into the Hugging Face cache and is therefore slow in a way that says nothing about the machine. The second run is the one worth timing.

mlx-community/Qwen3-8B-4bit is a four-bit conversion of Qwen/Qwen3-8B, published under the same Apache-2.0 licence as the original, which the course’s model reference records. The mlx-lm README describes the organisation that publishes it in one line: “Thousands are available in the MLX Community Hugging Face organization.” It is a community organisation rather than an Apple one, which is worth knowing before you treat a conversion there as authoritative; the safe habit is to check that a conversion’s card names the base model and the licence, exactly as you would check a GGUF conversion in Part 6.

Converting and quantising yourself

Downloading somebody’s conversion is convenient. Making your own is how you control the quantisation, and it is also the only way to convert a model nobody has bothered with.

RunnableTrack M · Apple silicon

convert a checkpoint to MLX format at four bits

mlx_lm.convert \
    --hf-path Qwen/Qwen3-8B \
    --mlx-path ~/models/mlx/Qwen3-8B-4bit \
    -q \
    --q-bits 4 \
    --q-group-size 64

Four flags, and each one is a decision.

• --hf-path takes a Hub identifier or a local directory. The tool downloads the safetensors checkpoint if it does not have it.
• --mlx-path is where the converted model is written. It defaults to mlx_model in the current directory, which is exactly the sort of default that leaves three copies of a model in three project folders.
• -q, the short form of --quantize, turns quantisation on. Without it you get a format conversion at the original precision, which is occasionally what you want and usually not.
• --q-bits and --q-group-size are the two numbers that define the quantisation. Bits per weight is the obvious one. Group size is how many weights share one scale factor: smaller groups track the original values more closely and cost more memory in scales, larger groups do the reverse.

There are three further options worth knowing without using today. --dtype sets the precision of the parameters that are not quantised. --quant-predicate selects a mixed-bit recipe, so that sensitive layers can be held at a higher precision than the rest, which is the same idea the K-quants in Part 6 implement inside GGUF. --upload-repo pushes the result to a Hub repository, which is how the community organisation gets filled.

✕Common mistake

Expecting an MLX four-bit model and a GGUF Q4_K_M to be the same thing. They are both “four-bit”, and that is where the similarity ends: the grouping, the scale format and the choice of which tensors stay at higher precision all differ. Two engines running “the same quantisation” of the same model are running two different approximations of it. That is why the lab in this part measures speed and memory and treats output quality as a separate question, and why Part 16 measures quality properly rather than assuming a bit count settles it.

What the conversion costs in memory

The arithmetic is the same as everywhere else in this course: weights are bits per parameter, and the KV cache is bytes per token times the context you asked for. Qwen3-8B has about 8.2 billion parameters. At four bits with a group size of 64, each group of 64 weights carries a scale and a bias alongside it, which the course’s estimate treats as roughly half a bit per weight, giving about 4.6 GB of weights. The course’s model reference records this model’s KV cache at 144 KiB per token at fp16, so an 8,192-token context is about 1.2 GB.

Estimated budget: Qwen3-8B at four bits, 8,192-token context, on a 24 GB Mac

Weights, 4-bit, group size 64

4.6 GB

KV cache, 8,192 tokens, fp16

1.2 GB

macOS, desktop and browser

6 GB

Free

12.2 GB

Total

24 GB

An estimate from arithmetic, not a measurement: 8.2 billion parameters at 4.5 bits each, plus 144 KiB per token of context, plus a deliberately generous allowance for everything else a Mac is doing. The lab in this part replaces the first two figures with what your machine actually reports.

The point of drawing it is the third segment. On a discrete-GPU machine the operating system is not competing for video memory. On a Mac it is competing for the same pool, which is why the Part 6 lab told Track M readers to close everything first, and why a model that fits on paper can still push a laptop into swap.

Serving over HTTP

mlx_lm.server puts an OpenAI-compatible API in front of the same model.

RunnableTrack M · Apple silicon

serve a converted model

mlx_lm.server \
    --model ~/models/mlx/Qwen3-8B-4bit \
    --host 127.0.0.1 \
    --port 8080 \
    --max-tokens 512 \
    --temp 0.7 \
    --log-level INFO

The routes the server handles are /v1/chat/completions, /v1/completions, /v1/models and /health, plus /chat/completions without the version prefix. The defaults are 127.0.0.1 and port 8080, and the documentation describes the API as “intended to be similar to the OpenAI chat API” rather than identical to it. Request fields include the ones you would expect and several you would not, among them adapters for a LoRA adapter chosen per request and draft_model with num_draft_tokens for speculative decoding, which Part 17 covers.

⛨Security

The server documentation states plainly that “The MLX LM server is not recommended for production as it only implements basic security checks.” Take that at face value. Bind it to 127.0.0.1, which is its default, and if you need it reachable from another machine, put it behind something that does authentication and TLS rather than changing --host and hoping. Part 23 builds that properly, and Part 7’s LAN lab already showed the shape.

When MLX beats llama.cpp, and when it does not

This is the question every Mac owner asks, and the answer is not a number, because the number depends on which model, which quantisation, which context length and which week you ask.

What can be said without measuring is where each project’s effort goes. llama.cpp’s Metal backend is one of several backends in a tree that has to keep four of them working; MLX exists only for Apple silicon, so an optimisation for that hardware never has to be weighed against portability. On the other side, llama.cpp has a much larger set of quantisation types, a mature server with grammar and tool support, and a benchmarking tool that reports prefill and decode separately.

Three things are worth checking before you conclude anything on your own machine.

Whether you are comparing the same amount of information. An MLX four-bit conversion and a Q4_K_M GGUF do not store the same number of bits per weight, and the difference is large enough to move both speed and quality.

Whether the context length matches. llama.cpp allocates the KV cache at load from --ctx-size. The mlx-lm generation tools take --max-kv-size to cap it, and the defaults are not the same.

Whether anything else was running. On unified memory, “anything else” includes the window server.

∴Research note

The lab at the end of this part is built to answer this question with your own hardware rather than by argument: the same model, the same prompts, both servers, one script, and the numbers written into the same notebook with their context. If you want the answer today, skip to it.

Distributed MLX, previewed

MLX supports running one model across several machines, and the launcher is mlx.launch. The documentation describes it as a utility that “connects to the provided host and launches the input script on each host”, and monitors the processes so that one failure stops the rest.

Fragment — not complete on its own

mlx.launch --hosts ip1,ip2 my_script.py

The backends are the interesting part. The documentation lists a ring backend over TCP/IP as the default, mpi, an nccl backend for CUDA machines, and jaccl, described as RDMA over Thunderbolt. That last one is why a pair of Thunderbolt 5 Macs is a credible cluster rather than a curiosity, and it is the subject of Part 21. Part 18 covers the parallelism strategies that decide whether a given link is fast enough to be worth using.

Track S — NVIDIA DGX SparkNot supported

MLX targets Apple silicon. The equivalent native path on this track is TensorRT-LLM, in the next lesson.

Read this lesson for the unified-memory argument, which applies to the Spark’s own 128 GB pool even though the framework does not. The Spark’s native stack is covered next.

Track X — AMD Ryzen AI Max+ 395Not supported

MLX targets Apple silicon.

The AMD lesson in this part is your equivalent. The one idea to carry across is the memory budgeting: like a Mac, this machine shares one pool between the operating system and the model, with the extra complication that the GPU-visible share is capped.

Track M — Apple silicon

This is your track. Work through the convert, generate and serve commands above with Qwen3-8B, and keep the converted model: the lab uses it. If your Mac has 16 GB, convert Qwen3-4B instead and note the substitution in the notebook, because every later comparison has to know which model produced the numbers.

Track N — NVIDIA desktop or laptopNot supported

MLX targets Apple silicon.

Read it for the comparison. The nearest equivalent argument on your track is ExLlamaV3 later in this part: a project that gives up portability in exchange for doing one thing unusually well on one kind of hardware.

Lazy execution changes what a timer measures

In a lazily evaluated array system, building an expression and executing it are distinct events. A timer around expression construction can report a small duration while the substantial computation occurs later when a value is materialised. Measure the completed operation using the synchronisation or evaluation pattern appropriate to the code path.

Separate model loading, warm-up, prompt processing and token generation. Record memory pressure during the entire request, especially with a large shared-memory model. A brief successful generation does not establish comfortable sustained use alongside other applications.

When converting or quantising a checkpoint, retain the source identity and conversion settings. Compare equivalent prompts after conversion and include tasks sensitive to changed precision. If another engine uses a different representation, label the comparison as a deployment comparison. You can still decide which stack serves your application better, but you cannot attribute the whole difference to lazy execution or the framework alone when the weight representation and template also changed.

Recap

MLX is an array framework built around Apple silicon’s shared memory, and mlx-lm is the package that turns it into a language-model toolchain. Unified memory removes the host-to-device copy and the separate video-memory budget; lazy evaluation means arrays are materialised only when something asks for them. mlx_lm.convert takes a Hugging Face checkpoint and writes an MLX model, with -q, --q-bits and --q-group-size defining the quantisation and --upload-repo publishing the result. mlx_lm.generate runs it once; mlx_lm.server puts an OpenAI-compatible API on port 8080 in front of it, and its own documentation says it is not for production. Whether it beats llama.cpp on your Mac is a measurement, not a fact, and this part’s lab is how you take it. mlx.launch extends the same toolchain across machines, which Part 21 builds on.

Check your understanding

Question 1. What does MLX’s unified memory model remove that a CUDA workflow has to do?

• The need to quantise the model
• The explicit copy of arrays between host memory and device memory, and the separate device-memory budget that goes with it
• The need for a chat template
• The KV cache

Show the answer and why

Answer: The explicit copy of arrays between host memory and device memory, and the separate device-memory budget that goes with it

The documentation puts it as arrays living in shared memory, with the device chosen per operation rather than per array. The KV cache still exists and still costs memory; it just comes out of the same pool as everything else, which is why the memory budget on a Mac includes the operating system.

Question 2. You convert Qwen3-8B with -q --q-bits 4 --q-group-size 64. What does the group size control?

• How many tokens are processed per batch
• How many weights share one scale factor: smaller groups track the original values more closely and spend more memory on scales
• The number of GPU cores used
• How many layers stay in fp16

Show the answer and why

Answer: How many weights share one scale factor: smaller groups track the original values more closely and spend more memory on scales

Group size is the granularity of the quantisation. It is the reason two four-bit models can differ in both size and quality, and the reason an MLX four-bit model is not interchangeable with a Q4_K_M GGUF.

Question 3. Which of these are true of mlx_lm.server as its own documentation describes it? Select all that apply.

• It listens on 127.0.0.1 port 8080 by default
• It exposes /v1/chat/completions, /v1/completions, /v1/models and /health
• It is documented as not recommended for production because it only implements basic security checks
• It is a drop-in replacement for the OpenAI API with every field supported

Show the answer and why

Answer: It listens on 127.0.0.1 port 8080 by default, It exposes /v1/chat/completions, /v1/completions, /v1/models and /health, It is documented as not recommended for production because it only implements basic security checks

The documentation says the HTTP API is intended to be similar to the OpenAI chat API, which is not the same as identical, and it warns against production use. Treat "OpenAI-compatible" as a claim to test with your own client, which is what this part’s feature probe does.

Question 4. A reader measures MLX as faster than llama.cpp on their Mac and posts the ratio. What is the first thing a careful reader asks?

• Which Mac it was
• Whether the two runs used the same model at comparable quantisation, the same context length, and a quiet machine
• Whether they used the latest macOS
• Whether they enabled the Neural Engine

Show the answer and why

Answer: Whether the two runs used the same model at comparable quantisation, the same context length, and a quiet machine

All three of those are variables that move the result by more than the difference being reported. The machine matters too, which is why the course’s benchmark tables carry hardware, engine, version, model, quantisation, context length and date in every row.

Check my answersClear