Fine-Tuning with TRL and PEFT

By the end of this lesson you will be able to read a supervised fine-tuning script line by line and say what every setting does, choose the handful that actually decide whether the run works, watch the two loss curves and stop at the right checkpoint, recognise overfitting from the logs rather than from the output, and say what is in the directory the run leaves behind.

Part 11 shipped the reference recipe as train-sft.py and ran it on a six-hundred-million-parameter model to prove the toolchain worked. This lesson is that script explained, and the lab in this part ships its own self-contained version so that the two can be read side by side.

Five objects

A run is a model, a tokeniser, a dataset, a training configuration and an adapter configuration, handed to one trainer.

From a JSONL file to an adapter directory

1. 1load_dataset("json", data_files=…)The train and validation JSONL files from the dataset lesson, in conversational prompt-completion shape.
2. 2AutoTokenizer.from_pretrained(model)The chat template travels with the tokeniser. If it is None, the model is a base checkpoint and you must supply one.
3. 3SFTConfig(...)Learning rate, epochs, batch, sequence length, evaluation strategy, precision, where to write.
4. 4LoraConfig(...)Rank, alpha, dropout, target modules. Passed as peft_config so the trainer wraps the model for you.
5. 5SFTTrainer(...).train()Tokenises, applies the chat template, masks the prompt, runs the loop, evaluates each epoch, keeps the best checkpoint.
6. 6save_model(output_dir)An adapter directory of a few tens of megabytes: adapter_config.json, adapter_model.safetensors, the tokeniser files.

Everything before the trainer is a decision you make; everything after it is bookkeeping. The two boxes with a tone are where nearly all the failures live.

The trainer takes the model as a string and loads it itself, which is why model_init_kwargs exists. One property of TRL 1.12.0 · verified 2026-09-08 is worth knowing before it surprises you: the documentation notes that when the model is passed as a string, “If dtype is not specified in args.model_init_kwargs, it defaults to float32”, which differs from from_pretrained, where since Transformers v5 the dtype is inferred from the model config. A run that is unexpectedly slow and four times larger in memory than the arithmetic predicted is usually this.

The settings that matter

TRL’s SFTConfig has more than a hundred fields. Six decide whether your run works, and the documentation lists four defaults that differ from the underlying TrainingArguments, which is worth reading once so you know what you are inheriting: logging_steps defaults to 10 rather than 500, gradient_checkpointing to True rather than False, bf16 to True if fp16 is not set, and learning_rate to 2e-5 rather than 5e-5.

Learning rate is the one that ruins runs. The 2e-5 default is a full-fine-tuning rate. Adapters want more, and TRL says so directly: “When training adapters, you typically use a higher learning rate (≈1e‑4) since only new parameters are being learned.” Start at 1e-4 for a LoRA run, and treat a loss that will not move as a reason to raise it and a loss that spikes or goes to nan as a reason to lower it. This is the same rule Part 1 gave for every training run in the course.

Epochs, not steps, for a dataset of a few hundred examples. Two to four is the useful range. One epoch often under-trains a format change; six epochs on three hundred examples is memorisation with extra steps.

Batch size and gradient accumulation together set the effective batch: per_device_train_batch_size × gradient_accumulation_steps. Only the first costs memory, because accumulation sums gradients across several forward passes before one optimiser step. On the 12 GB tier a batch of 1 or 2 with accumulation of 8 or 4 gives an effective batch of 8 for the memory of one or two, and it is the first knob to reach for when a run runs out of memory.

max_length defaults to 1024 and truncates from the start of the sequence by default (truncation_mode="keep_start"). Set it to what your examples actually need. Part 11’s warning applies: activation memory scales with the sequence length of the batch being processed, so a run sized against your average example fails against your longest one.

warmup_steps ramps the learning rate from zero over the first few steps, which stops the first gradient from moving a freshly initialised adapter too far. A handful of steps is enough on a small dataset. lr_scheduler_type="cosine" then decays it over the run.

packing groups several examples into one fixed-length sequence to reduce padding. It defaults to False, and for a few hundred short examples it is worth leaving off: the throughput gain is small and it makes the relationship between an example and a step harder to reason about while you are still learning to read the curves.

⌚Version note

Two settings interact in a way worth naming once. SFTConfig documents loss_type as defaulting to "chunked_nll", described as “same math as nll, but the lm_head projection is computed on non-ignored tokens only … and the cross-entropy is processed in chunks of tokens to reduce peak activation memory”, and as “Not compatible with use_liger_kernel”. Part 11’s memory lesson works through why the logits tensor is one of the largest activations at Qwen3’s vocabulary size, and what that costs when the chunked path is not in use. Check which path your run took before concluding that your activation arithmetic was wrong. Behaviour described here is TRL 1.12.0 · verified 2026-09-08 and transformers 5.16.1 · verified 2026-09-08 as documented on 2026-09-09.

Evaluation loss and when to stop

The training loss tells you the model is fitting your examples. It cannot tell you whether that is learning or memorisation, which is Part 1’s distinction and the whole reason for a validation split.

Four settings turn the validation split into a stopping rule:

Fragment — not complete on its own

config = SFTConfig(
    eval_strategy="epoch",              # compute the evaluation loss after every epoch
    save_strategy="epoch",              # and write a checkpoint at the same points
    load_best_model_at_end=True,        # finish holding the best one, not the last one
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

eval_strategy and save_strategy must agree, because ending the run on the best-scoring checkpoint requires a checkpoint to exist at the point that score was seen. With those five lines the run ends holding the parameters from the epoch with the lowest evaluation loss, whatever happened afterwards.

Early stopping adds the other half: stopping the run once the evaluation loss has failed to improve for a given number of evaluations. Transformers supplies a callback for it, and the lab’s script attaches it with a patience of two evaluations. On a short run its main value is not the time saved but the record: a run that stopped at epoch two out of five is telling you something about your data size that a run which used all five does not.

=Expected result

A healthy short LoRA run has a recognisable shape rather than a particular number. The training loss falls steeply for the first few dozen steps, then flattens. The evaluation loss falls with it, more slowly and more noisily, then flattens. The best epoch is somewhere in the middle third. Both curves being flat from the first step means the learning rate is too low or the adapter attached to nothing; both being jagged and rising means it is too high.

Overfitting, and how it shows up

Overfitting is the training loss continuing to fall while the evaluation loss turns and rises. On a few hundred examples it arrives early, and there are three symptoms worth recognising by name.

The curves diverge. The clearest signal and the one the stopping rule already handles. If the best epoch is consistently the first, your dataset is too small for the number of epochs, or the learning rate is too high, or both.

The model reproduces training examples verbatim. Give it an input close to one in the training set and it returns that example’s answer, including details that belonged to the original input. This is the symptom that reaches users, and it is worth checking by hand on three examples at the end of every run.

Everything becomes the training task. Ask it something unrelated and it answers in the shape of your examples. This is catastrophic forgetting and overfitting arriving together, and it is why the first lesson insisted on scoring every category of the Part 10 task set rather than the total.

The fixes, in the order to try them: fewer epochs, which costs nothing; more data, if you have it; a lower learning rate; a lower rank, since capacity you cannot fill is capacity available for memorising; and a few per cent of general instruction examples mixed in, which the forgetting study in the first lesson found helps.

Reading the logs

TRL documents exactly what it records, and four of the fields do real diagnostic work.

Field	What it tells you
loss	“The average cross-entropy loss computed over non-masked tokens in the current logging interval.” The headline number, and the one that can fall while the model gets worse.
eval_loss	The same quantity on the validation split. The one that decides when to stop.
mean_token_accuracy	“The proportion of non-masked tokens for which the model’s top-1 prediction matches the ground truth token.” Rising towards 1 on a format task is the model learning your fixed strings; it saturates long before the interesting content does.
entropy	“The average entropy of the model’s predicted token distribution over non-masked tokens.” Falling entropy means a more confident model. Falling very fast is a model collapsing onto one answer shape.
grad_norm	“The L2 norm of the gradients, computed before gradient clipping.” Spikes here precede loss spikes, so it is the earliest warning that the learning rate is too high.
num_tokens	Total tokens processed. The honest measure of how much training actually happened, and the one to record rather than epochs when comparing runs with different sequence lengths.

✕Common mistake

Comparing two runs by their final training loss. The loss depends on the dataset, the sequence length, whether packing was on, whether the loss covered the prompt, and the tokeniser. Two runs that differ in any of those produce losses that are not on the same scale. Losses compare a run against itself over time; runs compare against each other only through the Part 10 task set.

What the run leaves on disk

With a peft_config, save_model() writes an adapter rather than a model. The directory contains adapter_config.json, which records the base model’s name, the rank, the alpha, the dropout and the target modules; adapter_model.safetensors, which is the two small matrices per adapted layer and is typically tens of megabytes; and the tokeniser files if you saved them, which you should.

That is a useful artefact in itself. It is small enough to keep every one of them, it names the base model it belongs to, and it can be attached to that base at serving time without being merged, which llama.cpp supports through --lora and vLLM through --enable-lora. The export lesson covers both routes.

Save adapters when you are iterating, when you want to serve several fine-tunes of one base without holding several copies of the base, or when you want to be able to fall back to the base model instantly.

Merge when you are done, when you want a single artefact to convert to GGUF or MLX, when you want to quantise the result, or when the serving stack does not support adapters. PEFT’s guide gives the mechanism and one warning that costs people an hour: merge_and_unload() “is not an in-place operation”, so its return value must be assigned.

!Important

Save the tokeniser alongside the adapter, every time. The chat template lives in the tokeniser files, and it is the thing that must match between training and serving. An adapter directory without a tokeniser is an adapter whose training-time template you will have to reconstruct from memory, and Transformers’ own documentation is blunt about the consequence of getting it wrong: “with the wrong control tokens, these models would have drastically worse performance.”

Recording the run

Part 11 defined the run-log format and every training script in Level 3 appends one JSON line per run to labbook.md: the run identifier and date, the base model, the dataset path with its SHA-256 checksum and example counts, every hyperparameter, the seed, the hardware, the package versions, the losses and any scores. This part’s scripts ship a self-contained copy of that helper so they run without Part 11’s files on the path, and write the same fields.

The checksum is the field people leave out and then need. A month later, the question is never “what learning rate did I use” — that is in the script — but “was this the run before or after I fixed the dataset”, and only the checksum answers it.

Prove the training configuration on a tiny subset

Before a full run, inspect one batch and train briefly on a handful of examples. The purpose is to establish that the intended parameters receive gradients and that the loss can respond to repeated data. Deliberately fitting this tiny subset is a diagnostic; it is not evidence of generalisation.

Verify adapter target names and the trainable-parameter count after model construction. Record the chat template and loss mask. If loss is flat, check whether labels are masked out, gradients are finite and the optimiser includes the adapter parameters before increasing the learning rate. If loss falls but evaluation fails, inspect the data and serving contract.

Use a new output directory for each meaningful configuration change. Save resolved settings rather than relying only on defaults that can change between versions. At evaluation time, match the fine-tune’s exact base lineage and template. A correct training loop can produce an apparently broken model when the serving prompt differs from training. Separating these checks prevents repeated retraining from becoming the response to a deployment-format error.

Recap

A supervised fine-tuning run is five objects: a dataset, a tokeniser carrying the chat template, an SFTConfig, a LoraConfig and the trainer that puts them together. Six settings decide the outcome: a learning rate around 1e-4 for adapters rather than the 2e-5 default, two to four epochs, a small batch with gradient accumulation for the effective batch you want, a max_length sized to your data, a short warm-up with cosine decay, and packing left off while you are learning to read the curves. Evaluation on the validation split after every epoch, with load_best_model_at_end and early stopping, ends the run holding the best checkpoint rather than the last. Overfitting shows as diverging curves, verbatim reproduction of training examples, and every question being answered in the training task’s shape. The logs give loss, evaluation loss, token accuracy, entropy, gradient norm and token count, and gradient-norm spikes are the earliest warning of a learning rate that is too high. The run leaves an adapter directory of tens of megabytes that names its base model; keep the tokeniser with it, and record the dataset’s checksum in the run log.

Check your understanding

Question 1. You copy an SFTConfig from a full fine-tuning example, attach a LoRA adapter and train. The loss barely moves over three epochs. What is the first suspect?

• The dataset is too small
• The learning rate: the SFTConfig default of 2e-5 is a full-fine-tuning rate, and TRL documents roughly 1e-4 for adapters
• The batch size is too large
• The model is too small

Show the answer and why

Answer: The learning rate: the SFTConfig default of 2e-5 is a full-fine-tuning rate, and TRL documents roughly 1e-4 for adapters

Only the adapter parameters are trained; standard LoRA initialisation starts with a zero weight update. Adapter learning rates are often larger than full fine-tuning rates, but the loss curve alone cannot identify the cause. Check labels, trainable parameters and the actual matched modules. An entirely unmatched target selection raises an error; a partial selection may train the wrong layout.

Question 2. Which combination ends a run holding the checkpoint with the lowest evaluation loss?

• eval_strategy="epoch" alone
• eval_strategy and save_strategy both set to "epoch", with load_best_model_at_end=True, metric_for_best_model="eval_loss" and greater_is_better=False
• save_total_limit=1
• num_train_epochs=1

Show the answer and why

Answer: eval_strategy and save_strategy both set to "epoch", with load_best_model_at_end=True, metric_for_best_model="eval_loss" and greater_is_better=False

Ending the run on the best-scoring checkpoint needs a checkpoint to exist at the point that score was seen, which is why the two strategies must agree, and the trainer needs to be told which metric to compare and in which direction.

Question 3. Your training loss keeps falling while the evaluation loss has been rising for two epochs, and the model now reproduces a training answer almost word for word when given a similar input. What is happening?

• The evaluation split is too small
• Overfitting: the model is fitting the examples rather than the pattern behind them
• The learning rate is too low
• The tokeniser is wrong

Show the answer and why

Answer: Overfitting: the model is fitting the examples rather than the pattern behind them

Diverging curves and verbatim reproduction are two of the three named symptoms. The cheapest fixes are fewer epochs and a lower rank, since capacity the data cannot fill is capacity available for memorising.

Question 4. Which logged metric gives the earliest warning that the learning rate is too high?

• mean_token_accuracy
• grad_norm, the L2 norm of the gradients before clipping, whose spikes precede loss spikes
• num_tokens
• epoch

Show the answer and why

Answer: grad_norm, the L2 norm of the gradients before clipping, whose spikes precede loss spikes

A gradient that is about to take an overshooting step is large before the step is taken, so the norm moves first. Token accuracy and token count carry no information about stability.

Question 5. When should you save an adapter rather than a merged model? Select all that apply.

• While iterating, because falling back to the base model is instant
• When several fine-tunes of one base must be served without holding several copies of the base
• When the serving stack supports attaching adapters, such as llama.cpp with --lora or vLLM with --enable-lora
• When you need a single file to quantise and convert to GGUF

Show the answer and why

Answer: While iterating, because falling back to the base model is instant, When several fine-tunes of one base must be served without holding several copies of the base, When the serving stack supports attaching adapters, such as llama.cpp with --lora or vLLM with --enable-lora

The first three are what adapters are for. The fourth is the case for merging: conversion and quantisation want one artefact, and PEFT's merge_and_unload produces it, remembering that it is not an in-place operation.

Check my answersClear