Merging, Exporting and Quantising a Fine-Tuned Model

By the end of this lesson you will be able to turn an adapter directory into something every engine in Level 2 can load, choose between merging and attaching at serving time, quantise the result without corrupting it, prove that the exported model still behaves like the one you trained, and put it somewhere that is not public.

This is the step that separates an experiment from a tool. The training run leaves you tens of megabytes of adapter_model.safetensors, which llama.cpp cannot open, MLX does not recognise and Ollama has never heard of. Everything on this page is about closing that gap.

The four destinations

From an adapter to something that serves

1. 1Adapter directoryadapter_config.json naming the base model, adapter_model.safetensors, and the tokeniser you remembered to save.
2. 2Attach, or merge?Attach to serve several fine-tunes of one base, or to fall back instantly. Merge to get one artefact you can convert and quantise.
3. 3Merge into the base at BF16On the CPU, one copy of the weights, merge_and_unload, save. Never into a 4-bit base.
4. 4Convert for your engineGGUF for llama.cpp and everything built on it, MLX for the Mac, safetensors or an AWQ or FP8 checkpoint for vLLM.
5. 5Prove itSame prompts, same settings, unmerged model versus exported model. A conversion that changed the behaviour is a conversion that failed.
6. 6Publish privately, and record itA private Hub repository with a card naming the base model, its licence, the dataset and the run id.

The first branch is the important one: merging is a decision, not a formality. The three export paths below it are mechanical, and the last box is the one people skip and then regret.

Merging, and when not to

PEFT’s conceptual guide states why merging works: “during training, the smaller weight matrices … are separate. But once training is complete, the weights can actually be merged into a new weight matrix that is identical.” The scaled product of the two adapter matrices is added to the frozen weight, and the result is an ordinary model of exactly the original shape and size, with no adapter in it and no inference overhead.

Two functions do it. merge_adapter() merges but keeps the adapter weights, so unmerge_adapter() can undo it, which is useful when you are comparing several adapters in one process. merge_and_unload() is the one you want for export: it “doesn’t keep the adapter weights in memory” and produces a standalone model. PEFT adds a warning that has cost many people an hour: “It is important to assign the returned model to a variable and use it, merge_and_unload() is not an in-place operation.”

Fragment — not complete on its own

base = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16, device_map="cpu")
model = PeftModel.from_pretrained(base, adapter_dir)
model = model.merge_and_unload()          # not in place: the return value is the merged model
model.save_pretrained(merged_dir)
AutoTokenizer.from_pretrained(adapter_dir).save_pretrained(merged_dir)

Three rules about that snippet, and each one is a fault in this part’s challenge.

Merge into the same base you trained against. The adapter records base_model_name_or_path in adapter_config.json for exactly this reason. An adapter trained on an instruct checkpoint and merged into the base checkpoint of the same family produces a model that loads, runs and is subtly wrong everywhere.

Merge at BF16, not at 4 bits. Adding a low-rank update to weights that have been rounded to four bits applies your update to the wrong numbers and then rounds again. PEFT is explicit about the general hazard for the quantised backends it supports, warning for torchao that “merging only works correctly with LoRA and with quant_type = 'int8_weight_only'” and that other combinations “will likely result in an error, and even it doesn’t, the results will still be incorrect.” The safe order is always merge first, quantise second.

Merge on the CPU. It is weight arithmetic rather than a forward pass, so it needs no accelerator and only one copy of the weights in memory. On a machine that could barely hold the training run, this is what lets the merge happen at all.

✦Tip

There is a good reason not to merge. An adapter is tens of megabytes and a merged model is gigabytes, so ten fine-tunes of one base are ten adapters and one base rather than ten models. llama.cpp’s server documents --lora FNAME for “path to LoRA adapter (use comma-separated values to load multiple adapters)”, --lora-scaled FNAME:SCALE,... for a scaled one, and --lora-init-without-apply to load adapters “without applying them (apply later via POST /lora-adapters)”. vLLM has the equivalent, described below. Keep the adapters; merge a copy.

GGUF, for llama.cpp and everything built on it

Two scripts in a llama.cpp checkout do the conversion, and which one you want depends on whether you merged.

A merged model goes through convert_hf_to_gguf.py, which takes the model directory as a positional argument. --outfile is documented as “path to write to; default: based on input”, and --outtype chooses the precision with the documented choices f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto, where auto is “the highest-fidelity 16-bit float type”. Convert at bf16 and quantise afterwards; converting straight to a small type throws away information the quantiser could have used.

RunnableAll tracks

convert a merged fine-tune to GGUF, then quantise

python3 ~/llama.cpp/convert_hf_to_gguf.py ~/models/my-finetune-merged \
    --outfile ~/models/my-finetune-bf16.gguf \
    --outtype bf16

llama-quantize \
    ~/models/my-finetune-bf16.gguf \
    ~/models/my-finetune-Q4_K_M.gguf \
    Q4_K_M

An unmerged adapter goes through convert_lora_to_gguf.py, whose docstring describes it as converting “a Hugging Face PEFT LoRA adapter to a GGUF file”. Its positional argument is the directory “containing Hugging Face PEFT LoRA config (adapter_model.json) and weights”, and it takes --base, documented as the “directory containing Hugging Face model config files (config.json, tokenizer.json) for the base model that the adapter is based on — only config is needed, actual model weights are not required.” If you cannot supply the base locally, --base-model-id takes the Hub identifier instead. The result is a small GGUF that llama-server --lora attaches to a base GGUF at load time.

RunnableAll tracks

convert the adapter itself, and serve it on top of the base

python3 ~/llama.cpp/convert_lora_to_gguf.py ~/runs/my-finetune \
    --base ~/models/Qwen3-4B \
    --outfile ~/models/my-finetune-lora.gguf \
    --outtype f16

llama-server \
    --model ~/models/unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf \
    --lora ~/models/my-finetune-lora.gguf \
    --alias local/finetune \
    --jinja \
    --ctx-size 8192 \
    --host 127.0.0.1 \
    --port 8080

▤Memory budget

Disk, not memory, is what runs out here. A merged four-billion-parameter model at BF16 is about 8 GB, the intermediate GGUF is another 8 GB, and the quantised file is a few more. Budget roughly two and a half times the BF16 size of the model before you start, and delete the intermediate once the quantised file has been tested rather than before.

MLX, for the Mac

Track M does not merge with PEFT; the equivalent is mlx_lm.fuse, whose arguments are --model (“The path to the local model directory or Hugging Face repo”), --adapter-path (“Path to the trained adapter weights and config”, defaulting to adapters) and --save-path (“The path to save the fused model”, defaulting to fused_model).

RunnableTrack M · Apple silicon

fuse an MLX adapter into a standalone model

mlx_lm.fuse \
    --model mlx-community/Qwen3-4B-bf16 \
    --adapter-path ~/runs/my-finetune-adapters \
    --save-path ~/models/my-finetune-fused

Three more options matter. --upload-repo pushes the fused model to a Hub repository. --dequantize produces a dequantised model, which is what you want when the base you trained against was itself quantised and you now need full precision for a conversion. And --export-gguf with --gguf-path writes GGUF directly, which the LoRA documentation describes as supporting a limited set of architectures in fp16. Because that support is narrow, the dependable Mac route to llama.cpp is to fuse into safetensors and then run llama.cpp’s own convert_hf_to_gguf.py over the result, which is the path this part’s lab takes.

vLLM: adapters, or a quantised checkpoint

vLLM can serve your adapter without merging anything. Its LoRA page documents starting the server with --enable-lora and registering adapters with --lora-modules name=path, in either the simple name=path form or a JSON form that also names the base model. Two limits are set at start-up: --max-lora-rank, which the documentation says to “set … to the maximum rank among all LoRA adapters you plan to use”, and --max-loras, which caps how many are resident at once. There is also a runtime path, enabled by the environment variable VLLM_ALLOW_RUNTIME_LORA_UPDATING, which exposes /v1/load_lora_adapter and /v1/unload_lora_adapter; the documentation attaches a warning to it that is worth repeating in full: “This feature comes with security risks. It should not be used in production unless it is an isolated, fully trusted environment.”

For a merged model, vLLM will serve the BF16 safetensors directly, and two quantised formats are documented if you want the memory back.

FP8 is the simpler one and needs recent silicon: “FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell)”, with older cards landing on a weight-only path instead. The recipe uses llm-compressor, which its own documentation describes as “an easy-to-use library for optimizing large language models for deployment with vLLM”:

Fragment — not complete on its own

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

vLLM documents this as “Static, per-channel quantization on the weights” with “Dynamic, per-token quantization on the activations”, needing no calibration data, and reports “a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy”. Those are figures published by the project, not measured here. There is also an online form, --quantization fp8 at start-up, which quantises “down to FP8_E4M3 precision with a per-tensor scale” without a preprocessing step, at the cost of doing the scaling during inference.

AWQ is the four-bit option, and the tooling has moved. vLLM’s AWQ page now states that “The AutoAWQ library is deprecated” and points at llm-compressor for the same job; a produced checkpoint is then served with --quantization auto_awq. If you meet a tutorial using AutoAWQ directly, that is the sentence to remember.

✱Hardware note

Track S readers have the compute capability for FP8 on the GB10, and Track N readers should check their card: Ada Lovelace, Hopper and Blackwell qualify, and anything older does not and will fall back. Track X has no vLLM FP8 path documented for this chip, and Track M has no vLLM GPU path at all, as Part 9 established. Both of those tracks export to GGUF or MLX instead, which is the point of having four destinations rather than one.

Proving the export

An export that changed the model is a bug that will reach a user as a mysteriously worse answer three weeks later. Check it while the training run is still fresh.

The test is a comparison, not an inspection. Take ten prompts, half from your training distribution and half from your Part 10 task set. Run them against the unmerged model in Python, with temperature zero and a fixed seed. Run the same ten against the exported model through its engine, at the same settings. Compare.

What you should see depends on what you did. A merge at the same precision should be essentially identical, allowing for arithmetic that is not bit-exact across implementations; large differences mean the wrong base or the wrong precision. A quantisation will differ on some outputs, because rounding the weights changes some token choices, and the honest question is not whether any answer changed but whether the score changed: run the Part 10 harness against both and compare, exactly as you compared the fine-tune against the base.

RunnableAll tracks

one prompt through the exported model, deterministically

llama-cli \
    --model ~/models/my-finetune-Q4_K_M.gguf \
    --prompt "Summarise this ticket in the house format." \
    --predict 128 \
    --temp 0 \
    --seed 0

✕Common mistake

Testing the export with a different chat template than you trained with. This is the commonest cause of “the merged model is worse than the adapter”, and it is not a merge problem at all. The template travels in the tokeniser files, so a conversion that picked up the base model’s tokeniser instead of your saved one is serving a different format from the one every training example used. The challenge in this part has this as its first worked case.

Publishing, privately

A fine-tune is a derivative work of a base model and a dataset, and the dataset may contain things that should not leave the machine. The Hub’s default for a new repository is a choice you make, and its documentation is clear about what private means: setting visibility to private will “Ensure your repo does not show up in other users’ search results”, make other users who visit the URL receive a 404 - Repo not found error, and stop them cloning it. Visibility is set at creation and can be changed later “in the Settings tab”.

The workflow is: create the repository as private in the web interface, log in from the terminal, and upload.

RunnableAll tracks

log in, then upload an adapter to a repository you already made private

hf auth login

hf upload your-namespace/my-finetune-adapter ~/runs/my-finetune . \
    --commit-message "LoRA adapter, Qwen3-4B base, run 20260909T101500Z"

The CLI guide documents the shape as hf upload [repo_id] [local_path] [path_in_repo] and notes that “If the repo doesn’t exist yet, it will be created automatically” — which is exactly why you create it in the web interface first, with the visibility you intended, rather than letting the upload choose.

Write a short model card in the repository. Four facts make it useful a year later: the base model and its licence, the training method and the settings, the dataset with its checksum and its provenance, and the run identifier from labbook.md. Publishing an adapter without naming its base model produces an artefact nobody can use, including you.

⛨Security

Never put a token in a file, a command line you will paste into a page, or a run log. hf auth login stores the credential for you, and --token exists for automation where the value comes from the environment. A merged model also carries whatever was in your training data, so run the redaction described in the dataset lesson before training rather than deciding what to publish afterwards.

Validate each transformation independently

Treat export as a chain: exact base plus adapter, merged weights, converted format, quantised format, served alias. Preserve an identity record at each boundary. If the final service regresses, this lets you locate the first transformation that changed behaviour.

Compare adapter-attached and merged models using the same tokeniser, template, evaluation mode and dtype. For numeric comparisons, use a justified tolerance; floating-point operation order can change values without making a merge invalid. Exact sampled text alone is a brittle equality test. Then evaluate the converted and quantised variants with the same task suite, reporting any quality change separately from memory and latency.

Check that special-token settings and any added vocabulary survived. The adapter may carry tokeniser or configuration files that must accompany the weights. Do not merge into a convenient sibling checkpoint merely because its architecture matches. Keep the original adapter, exact base identity, conversion command and raw evaluation outputs even after deleting a large intermediate file. Those are the inputs required to reproduce or repair the deployed artefact.

Recap

Merging adds the scaled product of the two adapter matrices back into the frozen weight and produces an ordinary model; merge_and_unload() does it and is not an in-place operation, so its return value matters. Merge into the same base you trained against, at BF16, on the CPU, and quantise afterwards rather than merging into something already quantised. You do not have to merge: llama.cpp attaches a converted adapter with --lora, and vLLM serves one with --enable-lora and --lora-modules, sized by --max-lora-rank. A merged model becomes GGUF through convert_hf_to_gguf.py at bf16 followed by llama-quantize, or an adapter becomes a GGUF adapter through convert_lora_to_gguf.py with --base. On the Mac, mlx_lm.fuse produces the fused model, and the dependable route onward to llama.cpp is llama.cpp’s own converter rather than the narrow --export-gguf path. For vLLM, FP8 through llm-compressor needs compute capability 8.9 or newer, and AWQ’s old tooling is deprecated in favour of the same library. Prove the export with the same prompts at the same settings, expect a merge to match and a quantisation to differ a little, and settle the question with the Part 10 harness. Then publish to a repository you made private first, with a card naming the base, the licence, the dataset and the run.

Check your understanding

Question 1. You trained a LoRA adapter against a 4-bit quantised base to fit in memory. How should you export it?

• Merge it into the 4-bit base and convert that
• Load the base at BF16, merge the adapter into that, save, and quantise the merged model afterwards
• Quantise the adapter to 4 bits first
• Export the adapter alone and hope the engine finds a base

Show the answer and why

Answer: Load the base at BF16, merge the adapter into that, save, and quantise the merged model afterwards

Adding a low-rank update to weights that were already rounded applies it to the wrong numbers and rounds again. PEFT warns that merging into quantised weights can error or produce incorrect results depending on the backend. Merge first, quantise second, always.

Question 2. Which llama.cpp script converts an unmerged PEFT adapter to GGUF, and what does its --base argument need?

• convert_hf_to_gguf.py, which needs the full base weights
• convert_lora_to_gguf.py, whose --base is a directory of the base model's config files, where "only config is needed, actual model weights are not required"
• llama-quantize, which needs nothing
• There is no such script; adapters must always be merged first

Show the answer and why

Answer: convert_lora_to_gguf.py, whose --base is a directory of the base model's config files, where "only config is needed, actual model weights are not required"

The adapter converter reads the tensor shapes it needs from the base configuration rather than from the base weights, which makes it cheap. --base-model-id takes a Hub identifier instead when the base is not on disk.

Question 3. Your merged and quantised GGUF gives noticeably worse answers than the adapter did in Python, on the same prompts. What is the first thing to check?

• The quantisation type
• The chat template: whether the tokeniser that travelled with the export is the one you trained against, since the template lives in the tokeniser files
• The context length
• The random seed

Show the answer and why

Answer: The chat template: whether the tokeniser that travelled with the export is the one you trained against, since the template lives in the tokeniser files

Quantisation changes some outputs by small amounts. A systematic drop in quality across every prompt is a formatting mismatch, and the template is the thing that differs when a conversion picked up the base model's tokeniser instead of the one saved with the adapter.

Question 4. Which statements about serving adapters without merging are correct? Select all that apply.

• llama-server documents --lora for the adapter path and --lora-scaled for one with a scale
• vLLM registers adapters with --lora-modules and needs --max-lora-rank set to the largest rank in use
• vLLM's runtime adapter loading is documented with a security warning about untrusted environments
• Serving an adapter separately costs the memory of a second full copy of the base model

Show the answer and why

Answer: llama-server documents --lora for the adapter path and --lora-scaled for one with a scale, vLLM registers adapters with --lora-modules and needs --max-lora-rank set to the largest rank in use, vLLM's runtime adapter loading is documented with a security warning about untrusted environments

The last is backwards and is the whole point: one resident base serves many adapters, which is why ten fine-tunes are ten small files rather than ten multi-gigabyte models.

Question 5. The Hub documentation says a private repository does what?

• Hides the files but keeps the repository listed
• Keeps it out of other users' search results, returns "404 - Repo not found" to other users who visit the URL, and prevents them cloning it
• Encrypts the weights
• Requires an organisation account

Show the answer and why

Answer: Keeps it out of other users' search results, returns "404 - Repo not found" to other users who visit the URL, and prevents them cloning it

Those three effects are quoted from the repository-settings page. Create the repository private in the web interface before the first upload, because hf upload will otherwise create the repository for you with whatever default applies.

Check my answersClear