Level 3 · Model BuilderLessonPart 13 · page 1 of 925 min

25Minutes

6Sources

What Fine-Tuning Changes and What It Cannot

By the end of this lesson you will be able to look at a problem you have with a local model and say, with a reason, whether supervised fine-tuning is the right tool for it, which of your abilities it puts at risk, and what number you will use to decide whether it worked. That last one is the part people leave until afterwards, and it is why so many fine-tunes end in an argument about vibes.

Supervised fine-tuning is what Part 1 described, run again on a much smaller dataset. You take a model that already exists, show it pairs of input and desired output, and let gradient descent move the parameters until the outputs match. TRL’s documentation puts it plainly: the model “is trained in a fully supervised fashion using pairs of input and output sequences”, minimising the negative log-likelihood of the target, conditioned on the input. Nothing exotic is happening. The interesting question is what that procedure can and cannot move.

What it reliably changes

Three things, and all three are behaviour rather than content.

Format. If every example in your data answers in a fixed shape, the model learns the shape. This is the single most reliable effect of supervised fine-tuning and the one with the best return on effort. A model that produces valid JSON on nine requests in ten becomes one that produces it on essentially all of them, and it does so without the system prompt, the schema and the retries you had been using to force it. The lab in this part does exactly this, because it is the effect you can demonstrate in an hour on a small model.

Style and register. Length, tone, hedging, whether it opens with “Certainly!”, whether it writes British or American spelling, whether it explains its reasoning or answers in one line. All of these are learned quickly, because they are properties of nearly every token in your examples rather than of a few.

Task behaviour. Which of several plausible responses the model chooses for an input of a given kind: to extract rather than summarise, to ask a clarifying question rather than guess, to refuse rather than invent. This is slower to move than format and faster than anything to do with facts, and it is where a domain fine-tune earns most of its value.

The pattern behind all three is worth naming. Supervised fine-tuning is efficient at changing what the model does with what it already has. It is teaching a competent generalist your house style, not teaching a stranger your subject.

What it changes badly

Facts. This is the expensive lesson, and there is evidence for it rather than only folklore.

Ovadia and colleagues compared unsupervised fine-tuning against retrieval-augmented generation for getting new information into a model, across knowledge-intensive tasks in several domains. Their finding, quoted from the abstract: “while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge.” They add the mechanism: “LLMs struggle to learn new factual information through unsupervised fine-tuning”, and that showing the model “numerous variations of the same fact during training could alleviate this problem.”

Read that last clause carefully, because it is the honest version of what people hope for. You can push a fact into the weights. It takes many phrasings of the same fact, many passes, and a learning rate high enough to move something, and the same pressure is simultaneously degrading everything else. Compare that with putting the fact in a text file and retrieving it, which takes a minute, is exactly correct, is auditable, and changes the moment you edit the file.

What it quietly damages

The loss you are minimising is computed only on your examples. Nothing in the procedure protects anything the model could do before and that your examples do not exercise.

This is catastrophic forgetting, and it is old. Goodfellow and colleagues defined it in 2013 in terms any reader of this course can check: “When trained on one task, then trained on a second task, many machine learning models ‘forget’ how to perform the first task.” Their paper is an empirical investigation of how badly this affects gradient-trained networks and how the choice of algorithm and activation function changes it.

More recently, Luo and colleagues studied it in the models this course actually runs. They report that “catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters”, and the finding that most affects your planning: within that range, as the model gets larger, the severity of forgetting intensifies. They also report that general instruction tuning mixed into the data helps to alleviate it.

The practical shape of this on a home machine is specific and recognisable. You fine-tune a four-billion-parameter instruct model on three hundred examples of your extraction task. Extraction improves. Multi-turn conversation gets worse, because none of your examples had a second turn. Code gets worse, because none of your examples had code. The model starts answering every question in the shape of an extraction, including the ones that were not extractions. Your training loss fell the whole way.

Two mitigations are cheap enough to be defaults. Mix a few per cent of general instruction examples into your data, which is what Luo and colleagues found helps. And keep the adapter separate rather than merged, so that the base model is one configuration change away when you discover what you broke; the export lesson in this part explains how to serve both.

Fine-tuning, retrieval or a better prompt

Three tools, three different problems. The decision is usually clear once the problem is stated precisely, and the cost of getting it wrong is an afternoon.

Which tool for which problem

Is the answer in a document you have?Then it is retrieval. Part 10 built the pipeline. A fact in a file is exact, auditable and editable; the same fact in the weights is none of those.
Does it work when you ask it more carefully?Then it is prompting, or a schema-constrained response. Both are free, immediate, and change with a text edit rather than a run.
Do you need it to do this on the twentieth call as well as the first?A prompt that works nine times in ten is a fine-tune waiting to happen: the same instruction, moved from the context window into the weights.
Is the gap a matter of shape, style or which behaviour it picks?Then it is supervised fine-tuning, and this part is about it.
Is the gap that you prefer one answer to another and cannot say why?Then it is preference optimisation, which is Part 14. Ranking two answers is often easier than writing the right one.

Work down the list. The first three questions are answered in minutes and settle most cases; only the fourth is worth a training run.

They compose, and the best local systems in this course use all of them at once: a fine-tuned model that holds the output contract, retrieval supplying the facts, and a short system prompt for the things that change weekly. The lesson is not that fine-tuning is a last resort. It is that fine-tuning is expensive to iterate on, so anything cheaper should be tried first and kept afterwards.

Choosing a base model

Four constraints, in the order that eliminates candidates fastest.

Can your machine train it? Part 11’s arithmetic answers this before you download anything. Sixteen bytes per parameter for a full fine-tune, roughly two bytes per parameter plus a small adapter for LoRA, and about half a byte per parameter plus the adapter for QLoRA. On the 12 GB floor this part is written for, that is a full fine-tune of well under a billion parameters, a LoRA run of about four billion, or a QLoRA run of eight billion and up.

Can your engines serve it? A model you cannot convert to GGUF or MLX is a model you will train and then not use. The reference models in this course all convert; an architecture published last week may not, and that is a question for the converter’s supported-architecture list rather than for the model card.

Instruct or base? Almost always instruct. A base checkpoint has no chat template and no instruction-following behaviour, so you would be teaching both the task and the conversational frame from your few hundred examples. Start from a model that already answers questions and teach it your version of answering. The exception is when you are deliberately replacing the chat format, and TRL supports it directly through chat_template_path in SFTConfig, which is how the course’s Part 12 student becomes an instruct model.

Licence and provenance. Check it on the model card and write it in the notebook, because a fine-tune inherits the base model’s licence and your fine-tune is a derivative work. The reference models used in this part’s lab and project are Apache-2.0: the Qwen3-4B card states the licence as apache-2.0, and the entries in the model reference record the same for the 1.7B, 8B and 14B models in the family. Anything you add from elsewhere carries its own terms, which is the subject of the dataset lesson.

Deciding what “better” will mean, before you start

This is the discipline the rest of the part depends on, and it takes twenty minutes.

Write down, before any training, the sentence you want to be able to say afterwards. Not “the model will be better at our tickets” but something with a number in it: on my forty-task set, at temperature zero with seed seven, the fine-tune passes more of the deterministic format checks than the base model, and its judge mean in the other five categories is not lower. Every clause in that sentence is checkable from labbook.md, and writing it forces three decisions you would otherwise make after seeing the results, which is too late.

The first decision is which categories must not get worse. Naming them in advance is what stops you from quietly dropping the category that regressed. The second is how much of a change counts. Run the base model twice with the same settings, look at the difference, and treat anything smaller than that as no difference at all; two runs of the same model are your noise floor. The third is what you will do if it is worse, which is the subject of this part’s challenge.

How much data, and how good

The scale is smaller than most people assume, and the evidence points at quality rather than volume. Zhou and colleagues fine-tuned a 65-billion-parameter model on 1,000 curated prompts and responses, with no reinforcement learning, and argue from the result that “almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary” to produce high-quality output.

For the narrow jobs in this part the numbers are smaller still. A format fine-tune of a small model works with a few hundred examples, which is the size the lab builds. A domain behaviour fine-tune wants low thousands. What neither works with is examples that disagree with each other: fifty consistent examples beat five hundred where a third of the answers use a different format, because the model learns the distribution you showed it, including its inconsistency.

Write a behavioural hypothesis that can fail

“Make the model better at support” is too broad for a small fine-tune. A testable hypothesis is that consistent examples will increase correct routing into a defined schema without reducing accuracy on ordinary questions. Define the schema, routing rules, held-out examples and regression set before training.

Compare the unchanged model with an improved prompt first. Then compare the fine-tune at the same serving precision, template, system prompt and sampling settings. If the fine-tune has different retrieval access, it is also receiving different evidence; separate that intervention from weight adaptation.

Inspect errors individually. A model may learn the formatting perfectly while learning an incorrect label rule from your data. Lower loss is consistent with reproducing those mistakes. Include examples where the desired answer is abstention or a request for missing information. The deployment decision is whether the measured behaviour meets the contract, not whether a training process completed or the adapter is nonempty. Keep the base available so a failed fine-tune has an immediate rollback path.

Recap

Supervised fine-tuning moves behaviour, not knowledge. It reliably changes output format, style and which behaviour the model selects, and it is a poor and expensive way to install facts, for which retrieval is both better and cheaper according to the comparison this lesson cites. It also degrades abilities your examples do not exercise, a phenomenon named in 2013 and measured in one-to-seven-billion-parameter language models more recently, where severity was reported to increase with model size across that range. Choose between retrieval, prompting and fine-tuning by asking where the missing thing lives; choose a base model by what your machine can train, what your engines can serve, whether it already follows instructions, and its licence. Decide what “better” means, per category, with a noise floor from two runs of the same model, before you start. And prefer a few hundred consistent examples to a few thousand inconsistent ones.

Sources for this lesson

6 verified · checked 2026-09-09

01Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (Ovadia et al., arXiv:2312.05934)§ Abstractarxiv.org/abs/2312.059342026-09-09
02An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks (Goodfellow et al., arXiv:1312.6211)§ Abstractarxiv.org/abs/1312.62112026-09-09
03An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning (Luo et al., arXiv:2308.08747)§ Abstract; findingsarxiv.org/abs/2308.087472026-09-09
04LIMA: Less Is More for Alignment (Zhou et al., arXiv:2305.11206)§ Abstractarxiv.org/abs/2305.112062026-09-09
05TRL — SFT Trainer§ Looking deeper into the SFT method; Computing the losshuggingface.co/docs/trl/sft_trainer2026-09-09
06Qwen3-4B model card§ Licence; Best Practiceshuggingface.co/Qwen/Qwen3-4B2026-09-09

Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.