Lab: DPO a Model to Prefer Your Style

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The versions, wall-clock times and results for each track will be recorded here when the validation pass is done.

Objective

By the end of this lab you will have built a small preference dataset out of your own model’s output, trained an adapter on it with direct preference optimisation, and produced two numbers: how often a judge prefers the tuned model’s answers on prompts it was never tuned on, and what the tuning cost on the task set you built in Part 10.

The second number is the point. A preference run that wins a style comparison and quietly loses four tasks is a bad trade you have not noticed yet, and this lab is built so that you cannot avoid noticing.

The model is Qwen3-1.7B, whose card gives 1.7 billion parameters, 1.4 billion of them outside the embeddings, 28 layers, 16 query heads and 8 key-value heads, a context length of 32,768 tokens, and an Apache-2.0 licence. If you have the memory, qwen3-4b works everywhere the smaller model does and gives more interesting answers to rank; the commands take a model argument for exactly that reason.

The starting point is the adapter from Part 13’s Lab: Fine-Tune a 1B to 4B Model to Follow Your Format. Every script here also runs against a bare base model, so if you have not done that lab, drop the --adapter argument and everything works. What you lose is the more interesting question this lab can ask, which is what preference tuning does on top of a fine-tune.

⌂Single-machine path

This is a single-machine lab on every track. If you have a second box, the most useful thing to do with it is serve the judge model from Task 3, so the machine doing the training is not also generating.

Requirements

Every track needs Part 11’s training libraries in the active environment, about 8 GB of free disk for the base model and two adapter directories, and an hour of which roughly twenty minutes is attended. The memory floor is 16 GB, and this part’s second lesson has the arithmetic that gets there.

You also need, from earlier parts: your Part 10 tasks file, judge.py from Part 10’s evaluation lab, and runlog.py from Part 11. The scripts below import runlog from the directory they sit in, so keep them together.

Track S — NVIDIA DGX Spark

Run inside the NGC PyTorch container from Part 11 with the training libraries installed in it. The 128 GB pool means you can hold the training job and a served judge model at once, which makes the judged ranking path in Task 3 comfortable: serve qwen3-8b alongside and rank with it while the trainer is idle.

Nothing here is large enough to need the arithmetic, which is exactly why it is a good place to practise doing it: the GRPO lab later in this part is where the same budget stops being academic.

Track X — AMD Ryzen AI Max+ 395Partial

Training needs the ROCm build of PyTorch. The ROCm 10.0.0 compatibility matrix dated 2026-08-14 lists gfx1151 without a support-tier qualifier, and AMD's PyTorch install page read on 2026-09-09 does not mention the chip. The CPU path completes this lab and is a supported way to finish it.

Use the ROCm wheels as Part 11’s environment lesson describes, and confirm with python -c 'import torch; print(torch.cuda.is_available())' before you start. Vulkan is an inference path, not a training one.

On the CPU, add --precision fp32 to the training command and expect the run to take substantially longer than on an accelerator. Record in the notebook that it was a CPU run: a CPU result and a GPU result are not interchangeable.

Track M — Apple siliconPartial

mlx-lm's README, read on 2026-09-09, lists low-rank and full-model fine-tuning and no preference-optimisation trainer, so there is no MLX path for DPO. TRL runs on PyTorch's MPS backend in float32, which is slower and caps the practical model size.

Track M runs the same TRL script as everyone else, on PyTorch’s MPS backend, with --precision fp32. Keep to a 1 to 2 billion parameter model: qwen3-1.7b is the size this lab was written around and qwen3-4b will fit on a 32 GB machine but is slow enough to spoil the hour.

mlx-lm, pinned at mlx-lm 0.31.3 · verified 2026-09-08, is still the right tool on this track for supervised fine-tuning and for serving. It simply has nothing to offer for this lab, and saying so is more useful than inventing a workaround.

Track N — NVIDIA desktop or laptop

A card with 16 GB is comfortable for qwen3-1.7b at the settings below, and 12 GB will do it if you lower --max-length to 768. If the card is also driving your displays, close the browser: the budget has headroom and a browser can take it.

On Windows, work inside WSL2 exactly as in Part 11.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-14-preference-and-rl"
cd "$LAB_DIR"
pwd
test -f "make-preference-pairs.py"

Expected result: pwd ends in part-14-preference-and-rl and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Confirm the environment and collect the files

RunnableAll tracks

activate and check the libraries

cd "$LAB_DIR"
source "$HOME/llm-course/.venv/bin/activate"
python -c "import torch, transformers, trl, peft, datasets; print(torch.__version__, transformers.__version__, trl.__version__, peft.__version__)"

Download the three lab files below into the course directory, beside runlog.py and judge.py.

RunnableAll tracks

make-preference-pairs.py

"""Build a preference dataset by sampling two answers per prompt and ranking them.

Purpose: the data half of the DPO lab. Samples two candidate answers from the model
    you are about to tune, then decides which is better either by asking you or by
    asking a local judge model through an OpenAI-compatible endpoint, and writes the
    result in the preference shape TRL's DPOTrainer expects: a prompt, a chosen
    completion and a rejected one. Ties and disagreements are dropped rather than
    guessed at, because a pair whose direction you are unsure of is training signal
    pointing at nothing.
Platform: all. Sampling runs either locally through transformers (any track) or
    against a served model over HTTP; judging always goes over HTTP.
Minimum memory: 16 GB when sampling locally from a 1B to 4B model; very little when
    both the sampler and the judge are served from another machine.
Assumes: Python 3.10 or newer. Local sampling needs torch, transformers and, for an
    adapter, peft. Judging needs a server that accepts OpenAI-compatible chat
    completions with a JSON schema response format, such as llama-server from Part 6
    or the gateway from Part 9.

Usage: python3 make-preference-pairs.py --model Qwen/Qwen3-1.7B --adapter runs/sft-my-format \
           --rank manual --out-dir pairs
       python3 make-preference-pairs.py --base-url http://127.0.0.1:8080/v1 --served-model local \
           --rank judge --judge-url http://127.0.0.1:8081/v1 --judge-model qwen3-8b --out-dir pairs
"""

from __future__ import annotations

import argparse
import hashlib
import json
import random
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any, Optional

# The default prompts are about running local models, which is what this course's
# readers write about; replace them with your own work and the adapter becomes
# genuinely yours. One line per prompt, or a JSONL file with a "prompt" field.
DEFAULT_PROMPTS = [
    "Explain what a KV cache is and why it grows with context length.",
    "A colleague asks whether to quantise a model to Q4_K_M or Q8_0. What do you tell them?",
    "Write a short note explaining why a 30B mixture-of-experts model can be faster than a 14B dense one.",
    "Summarise the difference between prefill and decode for someone who has never served a model.",
    "How would you decide whether to fine-tune a model or add retrieval?",
    "Describe what happens when a model does not fit in GPU memory on a discrete card.",
    "Explain a chat template to someone who has only ever used a hosted chat interface.",
    "What should be recorded about a training run so the result still means something in a month?",
    "Give practical advice on choosing a context length for a local server.",
    "Why is a benchmark score without hardware, version and date close to useless?",
    "Explain LoRA to an engineer who understands matrix multiplication but not fine-tuning.",
    "A model answers correctly but far too verbosely. What are the options, cheapest first?",
    "How do you tell whether a quantised model has been damaged by the quantisation?",
    "Explain why sampling temperature changes reproducibility, and what to do about it.",
    "What does it mean for a model to be open weight but not open source?",
    "Describe how to expose a local model to another machine on your network safely.",
    "Explain the difference between a tool call and a normal completion.",
    "What is the first thing to check when a served model produces gibberish?",
    "How should someone size a machine for running an 8B model comfortably?",
    "Explain why training loss falling is not the same as the model getting better.",
    "What is the point of a held-out split, in one paragraph?",
    "Describe when a smaller model is the right answer even though a larger one scores higher.",
    "Explain speculative decoding without using the word 'draft' more than twice.",
    "A fine-tune made the model worse on everything else. What happened and what next?",
]

JUDGE_SYSTEM = """You compare two answers to the same question and say which better matches
the house style described below. Reply with JSON only.

Judge only against the house style and the correctness of the content. Ignore which answer is
longer, which sounds more confident, and which is shown first. If neither is clearly better,
say tie."""

JUDGE_SCHEMA = {
    "type": "object",
    "properties": {
        "winner": {"type": "string", "enum": ["first", "second", "tie"]},
        "why": {"type": "string"},
    },
    "required": ["winner", "why"],
    "additionalProperties": False,
}

DEFAULT_STYLE = (
    "Answers are direct and short. They open with the answer itself, not with a restatement of "
    "the question. They use plain words, British spelling, and no bulleted lists unless the "
    "content is genuinely a list. They say plainly when something is uncertain, and they never "
    "close with an offer of further help."
)


# --------------------------------------------------------------------------------------
# HTTP
# --------------------------------------------------------------------------------------

def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict:
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:400]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


# --------------------------------------------------------------------------------------
# Sampling
# --------------------------------------------------------------------------------------

class ServedSampler:
    """Two answers per prompt from an OpenAI-compatible endpoint."""

    def __init__(self, base_url: str, model: str, api_key: Optional[str], timeout: int,
                 temperature: float, max_tokens: int):
        self.endpoint = base_url.rstrip("/") + "/chat/completions"
        self.model = model
        self.api_key = api_key
        self.timeout = timeout
        self.temperature = temperature
        self.max_tokens = max_tokens

    def sample(self, prompt: str, seed: int) -> str:
        payload = {
            "model": self.model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": self.temperature,
            "max_tokens": self.max_tokens,
            "seed": seed,
        }
        body = post_json(self.endpoint, payload, self.api_key, self.timeout)
        return (body["choices"][0]["message"]["content"] or "").strip()


class LocalSampler:
    """Two answers per prompt from a model loaded in this process."""

    def __init__(self, model_id: str, adapter: Optional[str], temperature: float, max_tokens: int):
        import torch  # noqa: PLC0415 - only needed for local sampling
        from transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: PLC0415

        self.torch = torch
        self.temperature = temperature
        self.max_tokens = max_tokens
        device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
        dtype = torch.bfloat16 if device == 'cuda' and torch.cuda.is_bf16_supported() else torch.float32
        if adapter:
            from peft import AutoPeftModelForCausalLM  # noqa: PLC0415
            self.model = AutoPeftModelForCausalLM.from_pretrained(adapter, dtype=dtype).to(device)
            self.tokenizer = AutoTokenizer.from_pretrained(adapter)
        else:
            self.model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype).to(device)
            self.tokenizer = AutoTokenizer.from_pretrained(model_id)
        self.model.eval()
        print(f"sampling locally from {adapter or model_id} on {self.model.device}")

    def sample(self, prompt: str, seed: int) -> str:
        self.torch.manual_seed(seed)
        messages = [{"role": "user", "content": prompt}]
        inputs = self.tokenizer.apply_chat_template(
            messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
        ).to(self.model.device)
        with self.torch.no_grad():
            out = self.model.generate(**inputs, max_new_tokens=self.max_tokens,
                                      do_sample=True, temperature=self.temperature, top_p=0.95)
        generated = out[0][inputs["input_ids"].shape[-1]:]
        return self.tokenizer.decode(generated, skip_special_tokens=True).strip()


# --------------------------------------------------------------------------------------
# Ranking
# --------------------------------------------------------------------------------------

def rank_manually(prompt: str, first: str, second: str, index: int, total: int) -> Optional[int]:
    """Ask the reader. Returns 0 or 1 for the better answer, or None to skip the pair."""
    print("\n" + "=" * 78)
    print(f"[{index}/{total}] {prompt}")
    for label, answer in (("A", first), ("B", second)):
        print(f"\n--- {label} " + "-" * 70)
        print(answer if answer else "(empty)")
    print("\n" + "-" * 78)
    while True:
        choice = input("Better answer? [a/b/s to skip/q to stop] ").strip().lower()
        if choice in ("a", "b"):
            return 0 if choice == "a" else 1
        if choice == "s":
            return None
        if choice == "q":
            raise KeyboardInterrupt
        print("Type a, b, s or q.")


def rank_with_judge(prompt: str, first: str, second: str, args) -> Optional[int]:
    """Ask the judge twice with the order swapped; disagreement means the pair is dropped.

    A judge that changes its answer when the answers change places has told you it is
    scoring position, not quality, and that pair carries no signal. Part 10's judge
    does the same swap for the same reason.
    """
    endpoint = args.judge_url.rstrip("/") + "/chat/completions"

    def ask(a: str, b: str) -> Optional[str]:
        user = (f"House style:\n{args.style}\n\nQuestion:\n{prompt}\n\n"
                f"First answer:\n{a}\n\nSecond answer:\n{b}")
        payload = {
            "model": args.judge_model,
            "messages": [{"role": "system", "content": JUDGE_SYSTEM},
                         {"role": "user", "content": user}],
            "temperature": 0.0,
            "max_tokens": 300,
            "seed": args.seed,
            "response_format": {"type": "json_schema",
                                "json_schema": {"name": "verdict", "schema": JUDGE_SCHEMA, "strict": True}},
        }
        try:
            body = post_json(endpoint, payload, args.judge_api_key, args.timeout)
            return json.loads(body["choices"][0]["message"]["content"]).get("winner")
        except (RuntimeError, KeyError, ValueError, TypeError) as exc:
            print(f"    judge call failed: {exc}", file=sys.stderr)
            return None

    forward = ask(first, second)
    backward = ask(second, first)
    if forward is None or backward is None:
        return None
    if forward == "tie" or backward == "tie":
        return None
    if forward == "first" and backward == "second":
        return 0
    if forward == "second" and backward == "first":
        return 1
    return None  # the judge flipped with the order; the pair tells us nothing


# --------------------------------------------------------------------------------------

def load_prompts(path: Optional[str]) -> list[str]:
    if path is None:
        return list(DEFAULT_PROMPTS)
    text = Path(path).read_text(encoding="utf-8")
    prompts = []
    for line in text.splitlines():
        line = line.strip()
        if not line:
            continue
        if line.startswith("{"):
            row = json.loads(line)
            prompts.append(row["prompt"] if isinstance(row.get("prompt"), str) else row["prompt"][-1]["content"])
        else:
            prompts.append(line)
    return prompts


def write_jsonl(path: Path, rows: list[dict]) -> str:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--prompts", default=None,
                        help="a file of prompts, one per line or as JSONL; the built-in set is used if omitted")
    parser.add_argument("--model", default="Qwen/Qwen3-1.7B", help="local model to sample from")
    parser.add_argument("--adapter", default=None, help="a LoRA adapter to sample through")
    parser.add_argument("--base-url", default=None,
                        help="sample from a served model instead of loading one locally")
    parser.add_argument("--served-model", default=None, help="model name the sampling server answers to")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--rank", default="manual", choices=["manual", "judge"])
    parser.add_argument("--judge-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--judge-model", default=None, help="a different, preferably larger model")
    parser.add_argument("--judge-api-key", default=None)
    parser.add_argument("--style", default=DEFAULT_STYLE,
                        help="the house style the judge is asked to prefer; write your own")
    parser.add_argument("--temperature", type=float, default=0.9,
                        help="above zero, or the two samples will be the same answer twice")
    parser.add_argument("--max-tokens", type=int, default=400)
    parser.add_argument("--valid-fraction", type=float, default=0.2)
    parser.add_argument("--out-dir", default="pairs")
    parser.add_argument("--timeout", type=int, default=300)
    parser.add_argument("--seed", type=int, default=0)
    args = parser.parse_args()

    if args.rank == "judge" and not args.judge_model:
        raise SystemExit("--rank judge needs --judge-model, and it should not be the model under test")

    prompts = load_prompts(args.prompts)
    print(f"{len(prompts)} prompts; two samples each at temperature {args.temperature}")

    if args.base_url:
        if not args.served_model:
            raise SystemExit("--base-url needs --served-model")
        sampler: Any = ServedSampler(args.base_url, args.served_model, args.api_key,
                                     args.timeout, args.temperature, args.max_tokens)
    else:
        sampler = LocalSampler(args.model, args.adapter, args.temperature, args.max_tokens)

    raw: list[dict] = []
    kept: list[dict] = []
    skipped = 0
    started = time.time()

    try:
        for i, prompt in enumerate(prompts, start=1):
            first = sampler.sample(prompt, args.seed + 2 * i)
            second = sampler.sample(prompt, args.seed + 2 * i + 1)
            if first == second:
                print(f"  [{i}/{len(prompts)}] both samples identical; skipped")
                skipped += 1
                continue

            if args.rank == "manual":
                better = rank_manually(prompt, first, second, i, len(prompts))
            else:
                better = rank_with_judge(prompt, first, second, args)
                verdict = "skipped" if better is None else ("A" if better == 0 else "B")
                print(f"  [{i}/{len(prompts)}] judge: {verdict}")

            raw.append({"prompt": prompt, "a": first, "b": second,
                        "better": better, "ranked_by": args.rank})
            if better is None:
                skipped += 1
                continue
            chosen, rejected = (first, second) if better == 0 else (second, first)
            kept.append({
                "prompt": [{"role": "user", "content": prompt}],
                "chosen": [{"role": "assistant", "content": chosen}],
                "rejected": [{"role": "assistant", "content": rejected}],
            })
    except KeyboardInterrupt:
        print("\nstopped early; writing what has been ranked so far")

    if not kept:
        raise SystemExit("no pairs were kept. With --rank judge, a judge that ties or flips on "
                         "every pair usually means the two samples are too similar: raise "
                         "--temperature, or write prompts where style has room to differ.")

    rng = random.Random(args.seed)
    rng.shuffle(kept)
    split = max(1, int(len(kept) * args.valid_fraction))
    valid, train = kept[:split], kept[split:]

    out = Path(args.out_dir)
    train_hash = write_jsonl(out / "train.jsonl", train)
    valid_hash = write_jsonl(out / "valid.jsonl", valid)
    write_jsonl(out / "raw.jsonl", raw)

    elapsed = time.time() - started
    print(f"\nranked {len(raw)} pairs in {elapsed:.0f} s; kept {len(kept)}, skipped {skipped}")
    print(f"  train {len(train)}  sha256 {train_hash[:12]}...")
    print(f"  valid {len(valid)}  sha256 {valid_hash[:12]}...")
    print(f"written to {out}/  (train.jsonl, valid.jsonl, raw.jsonl)")
    print("Record the train hash in the run log; it is what ties a DPO result to this exact set.")


if __name__ == "__main__":
    main()

Download make-preference-pairs.py366 lines

RunnableAll tracks

train-dpo.py

"""Direct preference optimisation on a LoRA adapter, with TRL's DPOTrainer.

Purpose: the course's reference preference-tuning run. Loads a preference dataset of
    prompt, chosen and rejected, trains an adapter against a frozen reference copy of
    the starting model, evaluates on a held-out split after every epoch, and appends
    one run record to the lab notebook. Beta and the learning rate are the two dials
    the lesson asks you to move, so they are arguments rather than constants.
Platform: spark, strix, nvidia (CUDA or ROCm), and mac on PyTorch's MPS backend in
    float32 with a 1B to 2B model. mlx-lm ships no preference trainer, so Track M
    takes the PyTorch path for this lab.
Minimum memory: 16 GB
Assumes: torch, transformers, trl, peft and datasets installed in the active
    environment; make-preference-pairs.py has been run so that <data-dir>/train.jsonl
    and <data-dir>/valid.jsonl exist; runlog.py sits next to this file.

Usage: python3 train-dpo.py --data-dir pairs --output-dir runs/dpo-style --labbook labbook.md
       python3 train-dpo.py --data-dir pairs --model Qwen/Qwen3-1.7B \
           --adapter runs/sft-my-format --beta 0.1 --lr 1e-5 --labbook labbook.md
       python3 train-dpo.py --data-dir pairs --loss-type ipo --beta 0.05 --labbook labbook.md
"""

from __future__ import annotations

import argparse
import json
import time
from pathlib import Path
from typing import Any

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer
from trl import DPOConfig, DPOTrainer

import runlog

DEFAULT_TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

# Every value TRL's DPOConfig documents for loss_type. The lesson's table covers the
# five the course discusses; the rest are here so the script does not have to be
# edited to try one.
LOSS_TYPES = [
    "sigmoid", "hinge", "ipo", "exo_pair", "nca_pair", "robust", "bco_pair", "sppo_hard",
    "aot", "aot_unpaired", "apo_zero", "apo_down", "discopop", "sft", "sigmoid_norm",
]


def pick_device() -> str:
    if torch.cuda.is_available():
        return "cuda"
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return "mps"
    return "cpu"


def use_bf16(device: str, requested: str) -> bool:
    if requested == "fp32":
        return False
    if requested == "bf16":
        return True
    return device == "cuda" and torch.cuda.is_bf16_supported()


def summarise_history(history: list[dict]) -> dict[str, Any]:
    """The numbers a DPO run is judged on, pulled out of a log full of them."""
    train_losses = [row["loss"] for row in history if "loss" in row and "eval_loss" not in row]
    evals = [(row.get("epoch"), row["eval_loss"]) for row in history if "eval_loss" in row]
    accuracies = [row["rewards/accuracies"] for row in history if "rewards/accuracies" in row]
    margins = [row["rewards/margins"] for row in history if "rewards/margins" in row]
    best_epoch, best_eval = min(evals, key=lambda pair: pair[1]) if evals else (None, None)
    return {
        "first_train_loss": round(train_losses[0], 4) if train_losses else None,
        "final_train_loss": round(train_losses[-1], 4) if train_losses else None,
        "final_eval_loss": round(evals[-1][1], 4) if evals else None,
        "best_eval_loss": round(best_eval, 4) if best_eval is not None else None,
        "best_epoch": best_epoch,
        "first_reward_accuracy": round(accuracies[0], 4) if accuracies else None,
        "final_reward_accuracy": round(accuracies[-1], 4) if accuracies else None,
        "final_reward_margin": round(margins[-1], 4) if margins else None,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-1.7B",
                        help="base model repository id or local path")
    parser.add_argument("--adapter", default=None,
                        help="an existing LoRA adapter to continue, e.g. the fine-tune from Part 13's lab")
    parser.add_argument("--data-dir", default="pairs", help="directory holding train.jsonl and valid.jsonl")
    parser.add_argument("--output-dir", default="runs/dpo")
    parser.add_argument("--beta", type=float, default=0.1,
                        help="how far the policy may move from the reference; higher means less deviation")
    parser.add_argument("--lr", type=float, default=1e-5,
                        help="TRL's DPO default is 1e-6; its documentation suggests about 1e-5 for adapters")
    parser.add_argument("--loss-type", default="sigmoid", choices=LOSS_TYPES)
    parser.add_argument("--label-smoothing", type=float, default=0.0,
                        help="Robust DPO's label-flip probability, in [0.0, 0.5)")
    parser.add_argument("--epochs", type=float, default=1.0)
    parser.add_argument("--batch-size", type=int, default=2)
    parser.add_argument("--grad-accum", type=int, default=4)
    parser.add_argument("--max-length", type=int, default=1024)
    parser.add_argument("--rank", type=int, default=16)
    parser.add_argument("--alpha", type=int, default=32)
    parser.add_argument("--dropout", type=float, default=0.05)
    parser.add_argument("--target-modules", nargs="+", default=DEFAULT_TARGETS)
    parser.add_argument("--precision", choices=["auto", "bf16", "fp32"], default="auto")
    parser.add_argument("--gradient-checkpointing", action="store_true")
    parser.add_argument("--precompute-ref-log-probs", action="store_true",
                        help="score the dataset with the reference model once, then drop it from memory")
    parser.add_argument("--report-to", default="none", choices=["none", "tensorboard"])
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--notes", default=None)
    args = parser.parse_args()

    device = pick_device()
    bf16 = use_bf16(device, args.precision)
    print(f"device: {device}   precision: {'bfloat16' if bf16 else 'float32'}")

    data_dir = Path(args.data_dir)
    files = {"train": str(data_dir / "train.jsonl"), "validation": str(data_dir / "valid.jsonl")}
    for split, path in files.items():
        if not Path(path).is_file():
            raise SystemExit(f"{path} is missing; run make-preference-pairs.py first ({split} split)")
    dataset = load_dataset("json", data_files=files)
    print(f"preference pairs: {len(dataset['train'])} train, {len(dataset['validation'])} validation")

    columns = set(dataset["train"].column_names)
    for required in ("prompt", "chosen", "rejected"):
        if required not in columns:
            raise SystemExit(
                f"the dataset has no {required!r} column. TRL's DPOTrainer expects a preference "
                f"dataset with prompt, chosen and rejected; found {sorted(columns)}"
            )

    tokenizer = AutoTokenizer.from_pretrained(args.adapter or args.model)
    if tokenizer.chat_template is None:
        raise SystemExit(f"{args.model} has no chat template; pick an instruction-tuned model")

    config = DPOConfig(
        output_dir=args.output_dir,
        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        gradient_accumulation_steps=args.grad_accum,
        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_steps=5,
        beta=args.beta,
        loss_type=[args.loss_type],
        label_smoothing=args.label_smoothing,
        max_length=args.max_length,
        precompute_ref_log_probs=args.precompute_ref_log_probs,
        gradient_checkpointing=args.gradient_checkpointing,
        bf16=bf16,
        model_init_kwargs=None if args.adapter else {"dtype": torch.bfloat16 if bf16 else torch.float32},
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        logging_steps=5,
        report_to=args.report_to,
        seed=args.seed,
        data_seed=args.seed,
    )

    trainer_kwargs: dict[str, Any] = {}
    if args.adapter:
        from peft import AutoPeftModelForCausalLM  # noqa: PLC0415 - only needed on this branch
        model = AutoPeftModelForCausalLM.from_pretrained(args.adapter, is_trainable=True)
        print(f"continuing the adapter in {args.adapter}; the reference is that model with the "
              f"adapter's starting weights")
    else:
        model = args.model
        trainer_kwargs["peft_config"] = LoraConfig(
            r=args.rank, lora_alpha=args.alpha, lora_dropout=args.dropout,
            target_modules=args.target_modules, bias="none", task_type="CAUSAL_LM",
        )

    trainer = DPOTrainer(
        model=model,
        args=config,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        processing_class=tokenizer,
        **trainer_kwargs,
    )

    started = time.time()
    trainer.train()
    elapsed = time.time() - started

    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    losses = summarise_history(trainer.state.log_history)
    losses["seconds"] = round(elapsed, 1)
    print(json.dumps(losses, indent=2))
    print(f"adapter saved to {args.output_dir}")

    if args.labbook:
        record = runlog.record(
            labbook=args.labbook,
            lab="part-14/train-dpo",
            model=args.adapter or args.model,
            dataset={
                "path": files["train"],
                "sha256": runlog.file_sha256(files["train"]),
                "train_pairs": len(dataset["train"]),
                "validation_pairs": len(dataset["validation"]),
            },
            hyperparameters={
                "method": "dpo-lora",
                "beta": args.beta,
                "loss_type": args.loss_type,
                "label_smoothing": args.label_smoothing,
                "learning_rate": args.lr,
                "epochs": args.epochs,
                "batch_size": args.batch_size,
                "grad_accum": args.grad_accum,
                "effective_batch": args.batch_size * args.grad_accum,
                "max_length": args.max_length,
                "rank": args.rank,
                "alpha": args.alpha,
                "precompute_ref_log_probs": args.precompute_ref_log_probs,
                "precision": "bfloat16" if bf16 else "float32",
                "started_from_adapter": args.adapter,
            },
            seed=args.seed,
            losses=losses,
            scores={},
            config_path=__file__,
            notes=args.notes,
        )
        print(f"recorded run {record['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download train-dpo.py242 lines

RunnableAll tracks

compare-before-after.py

"""Generate from the model before and after preference tuning, and check for regressions.

Purpose: the measurement half of the DPO lab. Runs the same held-out prompts through the
    starting model and the preference-tuned one at temperature zero, writes both sets of
    answers in the results shape Part 10's judge.py reads, and separately replays your
    Part 10 task set through both models with the deterministic checks, so a style win is
    never reported without asking what it cost everywhere else.
Platform: all. Both models are loaded locally through transformers and PEFT, so no server
    is needed and Track M runs the same path as everyone else.
Minimum memory: 16 GB for a 1B to 4B model held twice in turn; the script loads one model
    at a time and releases it before loading the next.
Assumes: torch, transformers and peft installed; a DPO adapter directory from train-dpo.py;
    optionally the tasks file from Part 10's evaluation lab for the regression check;
    runlog.py sits next to this file.

Usage: python3 compare-before-after.py --model Qwen/Qwen3-1.7B --after runs/dpo-style \
           --out-prefix results --labbook labbook.md
       python3 compare-before-after.py --model Qwen/Qwen3-1.7B --before runs/sft-my-format \
           --after runs/dpo-style --tasks tasks-template.json --labbook labbook.md
"""

from __future__ import annotations

import argparse
import gc
import json
import time
from pathlib import Path
from typing import Any, Optional

import torch

import runlog

# Prompts held out of the preference set on purpose. If the model only improves on the
# prompts it was tuned on, nothing useful happened.
HELD_OUT_PROMPTS = [
    "Explain why two runs of the same model with the same prompt can differ.",
    "What is the practical difference between a base model and an instruct model?",
    "Someone asks whether more context always helps. What do you say?",
    "Explain what an adapter is to someone who has downloaded one and does not know what to do with it.",
    "Describe the trade-off between batch size and latency on a single machine.",
    "How would you check that a converted model still behaves like the one you trained?",
    "Explain in plain words why a model can be fluent and wrong at the same time.",
    "What belongs in a model card, and what usually is not there but should be?",
    "A server answers quickly at first and then slows down. Where would you look?",
    "Explain why the same model gives different answers under two different chat templates.",
    "What is the cheapest useful evaluation you can run on a model you just downloaded?",
    "Describe when running a model on the CPU is a reasonable choice.",
]


def load_model(model_id: str, adapter: Optional[str]):
    """Load the base model, optionally with an adapter, on the best available device."""
    from transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: PLC0415

    device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
    dtype = torch.bfloat16 if device == 'cuda' and torch.cuda.is_bf16_supported() else torch.float32
    print(f'loading on {device} with {dtype}')
    if adapter:
        from peft import AutoPeftModelForCausalLM  # noqa: PLC0415
        model = AutoPeftModelForCausalLM.from_pretrained(adapter, dtype=dtype).to(device)
        tokenizer = AutoTokenizer.from_pretrained(adapter)
    else:
        model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype).to(device)
        tokenizer = AutoTokenizer.from_pretrained(model_id)
    model.eval()
    return model, tokenizer


def release() -> None:
    """Give the memory back before loading the second model."""
    gc.collect()
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    elif torch.backends.mps.is_available():
        torch.mps.empty_cache()


def generate(model, tokenizer, prompt: str, max_new_tokens: int) -> tuple[str, float]:
    messages = [{"role": "user", "content": prompt}]
    inputs = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
    ).to(model.device)
    began = time.time()
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    text = tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()
    return text, time.time() - began


def deterministic_checks(task: dict, answer: str) -> dict:
    """Part 10's cheap checks, repeated here so the regression check needs no server."""
    lowered = answer.lower()
    required = [s for s in task.get("must_contain", []) if s.lower() not in lowered]
    forbidden = [s for s in task.get("must_not_contain", []) if s.lower() in lowered]
    words = len(answer.split())
    limit = task.get("max_words")
    return {
        "missing_required": required,
        "found_forbidden": forbidden,
        "words": words,
        "over_length": bool(limit and words > limit),
        "passed": not required and not forbidden and not (limit and words > limit),
    }


def run_one_side(label: str, model_id: str, adapter: Optional[str], style_prompts: list[str],
                 tasks: list[dict], style_rubric: str, max_new_tokens: int) -> dict:
    print(f"\n=== {label}: {adapter or model_id} ===")
    model, tokenizer = load_model(model_id, adapter)
    results = []

    for i, prompt in enumerate(style_prompts, start=1):
        answer, seconds = generate(model, tokenizer, prompt, max_new_tokens)
        results.append({
            "id": f"style-{i:02d}", "category": "style", "prompt": prompt,
            "reference": "", "rubric": style_rubric, "answer": answer,
            "checks": {"words": len(answer.split()), "passed": True},
            "seconds": round(seconds, 2), "prompt_tokens": None, "completion_tokens": None,
        })
        print(f"  style-{i:02d}  {seconds:6.2f}s  {len(answer.split()):4d} words")

    passed = 0
    for task in tasks:
        answer, seconds = generate(model, tokenizer, task["prompt"], max_new_tokens)
        checks = deterministic_checks(task, answer)
        passed += int(checks["passed"])
        results.append({
            "id": task["id"], "category": task.get("category", "regression"),
            "prompt": task["prompt"], "reference": task.get("reference", ""),
            "rubric": task.get("rubric", ""), "answer": answer, "checks": checks,
            "seconds": round(seconds, 2), "prompt_tokens": None, "completion_tokens": None,
        })
        mark = "ok " if checks["passed"] else "BAD"
        print(f"  {mark} {task['id']}  {seconds:6.2f}s")

    del model
    release()
    mean_words = sum(r["checks"]["words"] for r in results) / max(1, len(results))
    return {
        "label": label,
        "model": adapter or model_id,
        "results": results,
        "tasks_passed": passed,
        "tasks_total": len(tasks),
        "mean_words": round(mean_words, 1),
    }


def write_results(path: Path, side: dict, model_id: str, max_new_tokens: int, notes: str) -> None:
    """The shape Part 10's judge.py compare reads: a run block and a list of results."""
    payload = {
        "run": {
            "lab": "part-14/lab-dpo-your-model-to-prefer-your-style",
            "run_id": time.strftime("%Y%m%dT%H%M%S"),
            "task_set": "held-out style prompts plus the Part 10 task set",
            "model": side["model"],
            "quant": "none (transformers, unquantised)",
            "engine": "transformers",
            "base_model": model_id,
            "settings": {"temperature": 0.0, "do_sample": False, "max_tokens": max_new_tokens},
            "date": time.strftime("%Y-%m-%d"),
            "notes": notes,
        },
        "results": side["results"],
    }
    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", default="Qwen/Qwen3-1.7B", help="the base model both sides share")
    parser.add_argument("--before", default=None,
                        help="the adapter the DPO run started from, or omit for the bare base model")
    parser.add_argument("--after", required=True, help="the adapter train-dpo.py produced")
    parser.add_argument("--prompts", default=None, help="a file of held-out prompts, one per line")
    parser.add_argument("--tasks", default=None,
                        help="your Part 10 tasks file, for the regression check")
    parser.add_argument("--only-category", default=None, help="use one category of the tasks file only")
    parser.add_argument("--max-tasks", type=int, default=12,
                        help="cap the regression tasks so the lab finishes inside its hour")
    parser.add_argument("--style", default="Direct, short, answer first, plain words, no closing offer of help.",
                        help="the rubric the judge is given for the style prompts")
    parser.add_argument("--max-new-tokens", type=int, default=400)
    parser.add_argument("--out-prefix", default="results")
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--notes", default="")
    args = parser.parse_args()

    if args.prompts:
        style_prompts = [line.strip() for line in Path(args.prompts).read_text(encoding="utf-8").splitlines()
                         if line.strip()]
    else:
        style_prompts = list(HELD_OUT_PROMPTS)

    tasks: list[dict] = []
    if args.tasks:
        spec = json.loads(Path(args.tasks).read_text(encoding="utf-8"))
        tasks = [t for t in spec["tasks"]
                 if args.only_category is None or t.get("category") == args.only_category]
        tasks = tasks[: args.max_tasks]
        print(f"regression check: {len(tasks)} tasks from {args.tasks}")
    else:
        print("no --tasks given, so this run measures style only. A style win with no "
              "regression check is half a result.")

    before = run_one_side("before", args.model, args.before, style_prompts, tasks, args.style,
                          args.max_new_tokens)
    after = run_one_side("after", args.model, args.after, style_prompts, tasks, args.style,
                         args.max_new_tokens)

    before_path = Path(f"{args.out_prefix}-before.json")
    after_path = Path(f"{args.out_prefix}-after.json")
    write_results(before_path, before, args.model, args.max_new_tokens, args.notes)
    write_results(after_path, after, args.model, args.max_new_tokens, args.notes)

    summary: dict[str, Any] = {
        "held_out_prompts": len(style_prompts),
        "regression_tasks": len(tasks),
        "checks_passed_before": before["tasks_passed"],
        "checks_passed_after": after["tasks_passed"],
        "mean_words_before": before["mean_words"],
        "mean_words_after": after["mean_words"],
    }
    print("\n" + json.dumps(summary, indent=2))
    print(f"\nwrote {before_path} and {after_path}")
    print("Now judge them head to head with the position swap, using Part 10's judge:")
    print(f"  python3 judge.py compare --results-a {before_path} --results-b {after_path} \\")
    print("      --judge-model <a different, larger model> --labbook labbook.md")
    if tasks and after["tasks_passed"] < before["tasks_passed"]:
        print("\nThe regression check got worse. That is a result, not a failure of the lab: "
              "record it, then lower the learning rate or raise beta and try again.")

    if args.labbook:
        record = runlog.record(
            labbook=args.labbook,
            lab="part-14/compare-before-after",
            model=args.after,
            dataset={"path": args.tasks, "sha256": runlog.file_sha256(args.tasks) if args.tasks else None,
                     "held_out_prompts": len(style_prompts), "regression_tasks": len(tasks)},
            hyperparameters={"before": args.before or args.model, "after": args.after,
                             "temperature": 0.0, "max_new_tokens": args.max_new_tokens},
            seed=0,
            losses={},
            scores=summary,
            config_path=__file__,
            notes=args.notes or None,
        )
        print(f"recorded run {record['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download compare-before-after.py254 lines

Select a concrete starting checkpoint

For the PEFT route, point to the adapter actually produced in Part 13. The following selects the 1.7B run used on Tracks X and N. On Track S, use format-qwen3-4b instead if that was your run.

RunnableAll tracks

select the starting SFT adapter

export SFT_ADAPTER="$LABS_ROOT/part-13-supervised-fine-tuning/runs/format-qwen3-1.7b"
test -f "$SFT_ADAPTER/adapter_config.json"

Track M’s MLX adapter is not a PEFT adapter. For the DPO path here, first create a small PyTorch SFT adapter using the format dataset you already generated in Part 13:

RunnableTrack M · Apple silicon

create the compatible starting adapter on MPS

python3 "$LABS_ROOT/part-13-supervised-fine-tuning/train-lora.py" \
    --model Qwen/Qwen3-1.7B --precision fp32 \
    --data-dir "$LABS_ROOT/part-13-supervised-fine-tuning/data" \
    --output-dir runs/sft-my-format --epochs 1 --batch-size 1 --grad-accum 8 \
    --gradient-checkpointing --labbook labbook.md
export SFT_ADAPTER="$LAB_DIR/runs/sft-my-format"

After selecting an existing PEFT adapter or completing that Mac preparation, derive the matching base from its configuration and copy the frozen regression set:

RunnableAll tracks

resolve the base identity and regression data

export SFT_BASE="$(python3 -c 'import json,os; from pathlib import Path; print(json.loads((Path(os.environ["SFT_ADAPTER"])/"adapter_config.json").read_text())["base_model_name_or_path"])')"
cp "$LABS_ROOT/part-10-models-at-work/my-tasks.json" ./my-tasks.json
cp "$LABS_ROOT/part-10-models-at-work/judge.py" ./judge.py

Use SFT_BASE and SFT_ADAPTER consistently below. Stop if the adapter file or personal task file is absent; finish the corresponding prerequisite instead of substituting an unrelated checkpoint.

2. Decide what “your style” means, in writing

Before any sampling, write down the style you are tuning towards. Two or three sentences, specific enough that somebody else could apply them. The default in make-preference-pairs.py is the course’s own:

Output — what you should see

Answers are direct and short. They open with the answer itself, not with a restatement of
the question. They use plain words, British spelling, and no bulleted lists unless the
content is genuinely a list. They say plainly when something is uncertain, and they never
close with an offer of further help.

This matters more than it looks. If you rank by hand, the written style is what keeps your judgements consistent across forty comparisons made over twenty minutes. If you rank with a judge model, it is literally the rubric the judge is given, and a vague rubric produces a judge that ranks by length.

▷Exercise

Replace those sentences with your own, and replace the prompt list too. make-preference-pairs.py reads a file of prompts with --prompts, one per line. Twenty prompts from your actual work produce an adapter that is useful to you; twenty prompts about local models produce an adapter that demonstrates the technique.

3. Sample two answers per prompt and rank them

Two answers to the same prompt, at a temperature above zero so they differ, then a decision about which is better. There are two ways to make that decision and the lab supports both.

By hand is slower and better. Twenty-four prompts take about fifteen minutes and you will learn something about your own preferences that you did not know before you started.

RunnableAll tracks

sample and rank by hand

python make-preference-pairs.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --rank manual \
  --temperature 0.9 \
  --out-dir pairs

With a local judge is faster and lets you rank a hundred prompts. Serve a larger model than the one under test and point the script at it. The judge is asked twice with the answers swapped, and any pair where it changes its mind is dropped, because a judge that ranks by position has told you that pair carries no signal.

RunnableAll tracks

serve a judge model on the OpenAI-compatible endpoint

llama-server \
  --model ~/models/Qwen3-8B-Q4_K_M.gguf \
  --alias qwen3-8b \
  --ctx-size 8192 \
  --host 127.0.0.1 \
  --port 8081 \
  --jinja

RunnableAll tracks

sample locally, rank with the served judge

python make-preference-pairs.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --rank judge \
  --judge-url http://127.0.0.1:8081/v1 \
  --judge-model qwen3-8b \
  --temperature 0.9 \
  --out-dir pairs

Output — what you should see

24 prompts; two samples each at temperature 0.9
sampling locally from runs/sft-my-format on cuda:0
  [1/24] judge: A
  [2/24] judge: skipped
  ...
ranked 24 pairs in xxx s; kept 19, skipped 5
  train 16  sha256 4f2c8a1b93de...
  valid 3   sha256 8a10dd47c6b2...
written to pairs/  (train.jsonl, valid.jsonl, raw.jsonl)

✕Common mistake

Do not use the model under test as its own judge. A model scoring its own output scores it generously, and the pairs you get back will encode its habits rather than your preferences. Part 10 made this point about evaluation; it applies with more force here, because these labels become training data.

4. Read the pairs before you train on them

RunnableAll tracks

look at what you are about to optimise

head -n 1 pairs/train.jsonl | python -m json.tool
wc -l pairs/train.jsonl pairs/valid.jsonl

Each line has three keys: prompt, chosen and rejected, each a list of chat messages. That is TRL’s conversational preference format with an explicit prompt, which its dataset documentation recommends over the implicit form.

Now read five of them properly. You are looking for one thing: is the chosen answer better for the reason you wrote down in Task 2, or is it just longer? If more than one or two are simply longer, your preference set will teach verbosity, and the honest fix is to re-rank rather than to hope.

5. Train

Track S — NVIDIA DGX Spark

RunnableTrack S · DGX Spark

DPO on a LoRA adapter

python train-dpo.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --data-dir pairs \
  --output-dir runs/dpo-style \
  --beta 0.1 \
  --lr 1e-5 \
  --epochs 1 \
  --labbook labbook.md

Track X — AMD Ryzen AI Max+ 395Partial

ROCm build required for the GPU path; --precision fp32 completes the lab on the CPU.

RunnableTrack X · Ryzen AI Max+

DPO on a LoRA adapter, ROCm or CPU

python train-dpo.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --data-dir pairs \
  --output-dir runs/dpo-style \
  --beta 0.1 \
  --lr 1e-5 \
  --epochs 1 \
  --labbook labbook.md

Track M — Apple siliconPartial

No MLX preference trainer; TRL runs on the MPS backend in float32.

RunnableTrack M · Apple silicon

DPO on the MPS backend, float32

python train-dpo.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --data-dir pairs \
  --output-dir runs/dpo-style \
  --precision fp32 \
  --max-length 768 \
  --beta 0.1 \
  --lr 1e-5 \
  --epochs 1 \
  --labbook labbook.md

Track N — NVIDIA desktop or laptop

RunnableTrack N · NVIDIA GPU

DPO on a LoRA adapter

python train-dpo.py \
  --model "$SFT_BASE" \
  --adapter "$SFT_ADAPTER" \
  --data-dir pairs \
  --output-dir runs/dpo-style \
  --beta 0.1 \
  --lr 1e-5 \
  --epochs 1 \
  --labbook labbook.md

The script continues the Part 13 adapter rather than starting a new one: it loads the adapter as a trainable PEFT model and passes it to DPOTrainer without a peft_config, which is the documented way to keep training an existing adapter. The reference is the same model with the adapter at its starting weights, because ref_model is left as None and TRL then uses “the initial policy corresponding to model”. Drop --adapter and the script builds a fresh rank-16 adapter on the base model instead.

=Expected result

Training loss starts near 0.69 and falls. rewards/accuracies rises above a half. rewards/margins grows. logps/chosen may fall while the margin grows, and this part’s second lesson explains why that is the normal mechanism rather than a fault. What must not happen is a loss that stays flat at 0.69, which means the margin is not moving at all: see Troubleshooting.

6. Compare before and after on prompts you never ranked

RunnableAll tracks

generate both sides on held-out prompts and replay the Part 10 tasks

python compare-before-after.py \
  --model "$SFT_BASE" \
  --before "$SFT_ADAPTER" \
  --after runs/dpo-style \
  --tasks my-tasks.json \
  --max-tasks 12 \
  --out-prefix results \
  --labbook labbook.md

The script loads one model at a time, generates at temperature zero so the comparison is repeatable, and writes results-before.json and results-after.json in the exact shape Part 10’s judge.py reads. It also replays up to twelve of your Part 10 tasks through both models and applies the deterministic checks.

Output — what you should see

=== before: runs/sft-my-format ===
  style-01   x.xxs   xxx words
  ...
  ok  fact-01   x.xxs
=== after: runs/dpo-style ===
  ...
{
  "held_out_prompts": 12,
  "regression_tasks": 12,
  "checks_passed_before": xx,
  "checks_passed_after": xx,
  "mean_words_before": xxx.x,
  "mean_words_after": xxx.x
}

7. Judge the two sides head to head

RunnableAll tracks

the Part 10 judge, with the position swap

python judge.py compare \
  --results-a results-before.json \
  --results-b results-after.json \
  --base-url http://127.0.0.1:8081/v1 \
  --judge-model qwen3-8b \
  --labbook labbook.md

The judge is asked twice per task with the answers in both orders. Read three fields of its summary: wins_b against wins_a, which is your result; ties, which is the judge saying it cannot tell; and flip_rate, which is the share of tasks where swapping the order changed the verdict. A high flip rate is a measurement of the judge, not of either model, and it caps how much the win count is worth.

∴Research note

Two dozen held-out prompts can detect a large difference and not a small one. If the tuned model wins fourteen and loses ten, that is not a finding; if it wins twenty-two and loses two, it is. Write down which of those two you got before you decide how you feel about it.

8. Read the regression check, and mean length

The two checks_passed numbers from Task 6 are the ones that decide whether the adapter is usable. A style win with a lower pass count is a trade, and it is your decision whether to take it, but it has to be a decision rather than a surprise.

Mean word count before and after is the other number to look at now. If it doubled, the judge’s preference is at least partly a preference for length, which this part’s second lesson names as the best-documented failure of preference tuning.

9. Move beta and see what changes

One run tells you very little about a hyperparameter. Train twice more, changing one thing each time, and keep all three run-log lines.

RunnableAll tracks

a tighter leash and a looser one

python train-dpo.py --model "$SFT_BASE" --adapter "$SFT_ADAPTER" \
  --data-dir pairs --output-dir runs/dpo-beta-05 --beta 0.5 --lr 1e-5 --epochs 1 --labbook labbook.md
python train-dpo.py --model "$SFT_BASE" --adapter "$SFT_ADAPTER" \
  --data-dir pairs --output-dir runs/dpo-beta-002 --beta 0.02 --lr 1e-5 --epochs 1 --labbook labbook.md

Expect the tight run to barely move the model and the loose one to move it further than you wanted, with the samples showing it. Generate from all three with compare-before-after.py if you have time; at minimum, compare their final rewards/margins and read four answers from each.

10. Record the run

RunnableAll tracks

the last run record, formatted

tail -n 1 labbook.md | python -m json.tool

Check five fields: dataset.sha256 matches what make-preference-pairs.py printed, hyperparameters.beta and hyperparameters.learning_rate are what you meant, hyperparameters.started_from_adapter names the Part 13 run, and losses.final_reward_accuracy is present. Add the judge’s win counts and flip rate as a note, since the training script cannot know them.

Verify the starting adapter and preference pairs

The example path runs/sft-my-format denotes the SFT result you are continuing. Before using it, locate the actual Part 13 adapter and use that same path for pair generation, training and the before/after comparison. Its recorded base must match --model; a 4B adapter is not interchangeable with the 1.7B base in the generic commands. On the MLX route from Part 13, do not pass MLX adapter files directly to a PEFT loader; follow the supported PyTorch starting-model path for this lab.

Inspect pairs before training. Each pair must contain the same prompt, two distinct candidate answers and a preference supported by your written style rubric. Exclude unresolved ties and check that chosen answers are not simply longer. Keep prompt families together when splitting.

Record the reference policy used by DPO, beta, learning rate and loss variant. After training, evaluate on prompts not used for ranking, with blinded order and an independent correctness check. Report preference gain, answer length and task regressions together. A falling DPO loss is not the completion criterion. Keep the pair dataset, rubric, starting adapter, resulting adapter and raw comparison outputs so a later reviewer can reconstruct what preference the model was taught.

Validation

You are done when all of the following are true:

• pairs/train.jsonl and pairs/valid.jsonl exist, every line has prompt, chosen and rejected, and you have read at least five of them;
• the training script reported a device and a precision, and printed one loss line per logging interval;
• rewards/accuracies was above 0.5 at the end of training and rewards/margins was larger at the end than at the start;
• an evaluation loss was reported for the epoch, and the adapter directory contains adapter_config.json and the adapter weights;
• results-before.json and results-after.json both exist and contain an answer for every held-out prompt and every regression task;
• judge.py compare produced win counts for both sides and a flip rate;
• you can say, in one sentence, what the tuning cost on the Part 10 tasks;
• labbook.md has new JSON lines for the training run, the comparison and the judging.

Expected outcome

An adapter that answers in a style you chose, a win rate against the model it started from, and a regression number that tells you what it cost. The table below is what to record per track; the validation pass will fill it in from the course’s own machines.

Pending validationWhat to record from this lab, per track

Track	Pairs kept	Wall clock, training	Final reward accuracy	Judge wins, after vs before	Part 10 checks passed, before → after
S: DGX Spark, 128 GB	to be measured	to be measured	to be measured	to be measured	to be measured
X: Ryzen AI Max+ 395	to be measured	to be measured	to be measured	to be measured	to be measured
M: Apple silicon, MPS float32	to be measured	to be measured	to be measured	to be measured	to be measured
N: NVIDIA desktop or laptop	to be measured	to be measured	to be measured	to be measured	to be measured

the four platform tracks, one machine each · TRL DPOTrainer with a PEFT LoRA adapter; llama.cpp server for the judge trl 1.12.0, transformers 5.16.1, peft 0.20.0, llama.cpp v0.4.0 · Qwen3-1.7B with a rank-16 LoRA adapter; Qwen3-8B as the judge, BF16 for training, float32 on Track M; Q4_K_M for the judge · 1,024 tokens of context · 2026-09-09

Not yet run on hardware on any track. Until the validation pass fills these in, the table records what to measure rather than what to expect. Your own six figures belong in the run log beside the ones the arithmetic in this part's second lesson predicted.

Troubleshooting

no pairs were kept. The judge tied or flipped on everything. Usually the two samples are too similar: raise --temperature, or use prompts where style has room to differ. A prompt with one right answer produces two nearly identical answers and no preference.

Training loss sits at 0.69 and never moves. That is minus the log of a half, the loss of a model with no preference either way. Either the margin cannot move, which points at a beta that is far too high, or the chosen and rejected completions are nearly identical, which points back at Task 3.

the dataset has no 'chosen' column. You pointed --data-dir at a supervised fine-tuning dataset. DPO needs the preference shape, which is what make-preference-pairs.py writes.

Out of memory. In order: lower --max-length, lower --batch-size and raise --grad-accum to keep the effective batch, add --gradient-checkpointing, then add --precompute-ref-log-probs, which scores the dataset with the reference model once so it does not have to stay resident.

Loss is nan on Track X or M. Rerun with --precision fp32. If that fixes it, the accelerator’s bfloat16 path is the problem rather than the recipe, and it belongs in the run log with your versions.

The judge cannot be reached. llama-server listens on the host and port you gave it; check with a request to /v1/models on the same base URL. If the judge is on another machine, remember that a bare --host 127.0.0.1 accepts connections from that machine only, and that Part 10’s lesson on serving beyond localhost is where authentication and TLS belong.

The tuned model repeats itself or drifts into another language. Beta is too low for this dataset, the learning rate is too high, or both. Halve the learning rate first, since it is the cheaper experiment, and keep the run that produced the degradation as evidence.

Track M is very slow. Expected: float32 on MPS is the price of there being no MLX preference trainer. Cut the pair count, cut --max-length to 512, and keep to a model of about 1.7 billion parameters.

Cleanup

Keep labbook.md, pairs/, the DPO adapter and the two results files; the reality check at the end of this part does not need them, but Part 16 compares against this run. The rest is regenerable:

RunnableAll tracks

reclaim the disk, keeping the adapters and the notebook

rm -rf runs/dpo-style/checkpoint-* runs/dpo-beta-05 runs/dpo-beta-002

⚠Warning

That command deletes two whole training runs. It is safe only because the run-log lines record the settings that produced them and pairs/train.jsonl is still on disk with its recorded hash, so both can be rebuilt exactly. Never delete a run whose inputs you cannot reproduce.

Stop the judge server when you are finished with it, so it is not holding memory during the next lab.

What you learned

• A preference set is a description of your taste, written down twice. Once in the style note, once in the rankings. Where those two disagree, the rankings win, which is why reading five pairs before training is not optional.
• The margin is the thing. Chosen log-probabilities can fall during a healthy run; what has to move is the distance between chosen and rejected, and TRL logs it.
• Beta is a leash length. You saw a tight one produce no change and a loose one produce too much, on the same data, in the same hour.
• A judge has a bias and it is measurable. The flip rate under position swap is a number about the judge, and it bounds how much the win count is worth.
• Style is not free. The regression check exists because a model that writes better and answers worse is easy to produce and hard to notice.
• The run log is what makes three runs comparable. Three lines, three betas, one dataset hash.

Record in the notebook: the number of pairs kept and skipped and the dataset hash; the beta, learning rate and epochs of each run; the final rewards/accuracies and rewards/margins; the judge’s wins, ties and flip rate; the Part 10 check counts before and after; the mean answer length before and after; and one sentence saying whether you would use this adapter.

Check your understanding

Question 1. make-preference-pairs.py drops a pair when the judge picks a different winner after the answers are swapped. Why is that the right behaviour?

• It halves the dataset, which trains faster
• A judge that changes its verdict when the order changes was scoring position rather than quality, so that pair carries no information about your preference
• Swapped answers are invalid input
• It prevents the dataset from being unbalanced

Show the answer and why

Answer: A judge that changes its verdict when the order changes was scoring position rather than quality, so that pair carries no information about your preference

The swap is a control, and a flip is the control firing. Keeping the pair would put a coin flip into the training data with the same weight as a real judgement.

Question 2. You pass --adapter pointing at the Part 13 fine-tune. What is the reference model during this DPO run?

• The bare base model with no adapter
• The same model with the adapter at its starting weights, because ref_model is None and TRL then uses the initial policy corresponding to the model passed in
• A separately downloaded copy of the base model
• There is no reference model when training an adapter

Show the answer and why

Answer: The same model with the adapter at its starting weights, because ref_model is None and TRL then uses the initial policy corresponding to the model passed in

TRL documents that with ref_model=None the reference is the model state before DPO training starts, which here includes the Part 13 adapter. That is what you want: the run should measure movement away from your fine-tune, not away from the base model.

Question 3. After the run, the judge prefers the tuned model on 18 of 24 held-out prompts, the mean answer length has gone from 90 words to 200, and the Part 10 checks fell from 11 passed to 8. Which conclusion is supported?

• The adapter is a clear improvement and should be used
• The adapter writes longer answers that a judge prefers and passes fewer of your own checks; the length change is a plausible cause of the judge result and the regression is a real cost
• The judge is broken
• The training failed

Show the answer and why

Answer: The adapter writes longer answers that a judge prefers and passes fewer of your own checks; the length change is a plausible cause of the judge result and the regression is a real cost

Three numbers, one story. Length is the classic confound in judged comparisons, and the check count is the part of the picture the judge cannot see. Neither number alone would have told you this.

Question 4. Which of these would reduce memory during the DPO run? Select all that apply.

• Lowering --max-length
• Raising --grad-accum while lowering --batch-size
• Passing --precompute-ref-log-probs
• Raising beta

Show the answer and why

Answer: Lowering --max-length, Raising --grad-accum while lowering --batch-size, Passing --precompute-ref-log-probs

The first two shrink the activation and logits terms, which are the ones that scale with your settings. Precomputing the reference log-probabilities means the reference model does not have to stay in memory, which TRL documents as a memory saving. Beta changes the objective, not the footprint.

Question 5. On Track M the lab uses TRL on PyTorch's MPS backend rather than mlx-lm. What is the documented reason?

• MLX is slower than PyTorch on Apple silicon
• mlx-lm's README, read on 2026-09-09, lists low-rank and full-model fine-tuning and no preference-optimisation trainer, so there is no MLX path for DPO
• DPO requires CUDA
• Apple silicon cannot hold two models at once

Show the answer and why

Answer: mlx-lm's README, read on 2026-09-09, lists low-rank and full-model fine-tuning and no preference-optimisation trainer, so there is no MLX path for DPO

The tool does not offer the algorithm, which is a different situation from the tool being slow. Saying which of the two you are dealing with is what lets a reader decide whether to wait for a release or change approach.

Check my answersClear