Skip to content
Local LLM Course

Preference Optimisation and Reinforcement Learning

Part 13 taught the model to imitate. Every example in that dataset was an answer somebody decided was good, and the model learned to produce answers shaped like them. That is a real capability and it has a ceiling: imitation can only reach the quality of the examples, and writing an excellent answer to a hard question is slow work for an expert.

This part is about the two ways past that ceiling. The first is to stop writing answers and start ranking them, because judging which of two answers is better is fast, needs less expertise, and carries information that a single demonstration does not. The second is to stop labelling altogether on tasks where a program can check the answer, and let the model discover for itself what scores well.

The second one is the recipe behind the reasoning models. It is also the part of the field most often described in a way that makes it sound like magic, so this part teaches it from the sampling loop up: what a rollout is, why a group of them removes the need for a critic network, what the KL term is holding on to, and what the published refinements actually change. By the end you should be able to read the method section of a reinforcement-learning paper and recognise every piece of it.

Both labs run on one machine. The preference lab tunes the model you fine-tuned in Part 13 towards your own taste, and then asks the harder question of what that cost everywhere else. The reinforcement-learning lab trains a 1.5 to 4 billion parameter model against a reward a program computes, and has you read the reward, KL and length curves rather than trusting a final number. The reality check at the end asks whether any of it generalised.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain why preference data is cheaper to collect than demonstrations and what it can teach that demonstrations cannot, and describe reinforcement learning from human feedback as a pipeline with named parts;
  • say what the KL constraint is for, in both the classical pipeline and its direct successors, and what happens when it is too tight or too loose;
  • read the direct preference optimisation objective as a classification loss, set beta and the learning rate deliberately, and choose between DPO, IPO, KTO, ORPO and SimPO for a given dataset;
  • build a preference set from your own model’s samples, run DPO on it, and measure the change with a judge that swaps positions and a regression check that could stop you shipping;
  • describe group relative policy optimisation from the rollouts up, explain why it needs no critic network, and say what DAPO, Dr. GRPO and GSPO each change about it;
  • write a reward a program can compute, test it against the completions that try to cheat it, and run untrusted generated code in a subprocess with limits;
  • run a real reinforcement-learning job on your track, read the reward, KL and length curves together, and say what each one is telling you;
  • measure pass@1 before and after on held-out problems, and report an honest answer to the question of whether the gain survives outside the training distribution.

What you need first

Section titled “What you need first”

Part 11 in full, and Part 13’s lab. This part uses Part 11’s libraries, its memory arithmetic and its run-log format, and it refers to its reference recipe by name rather than repeating it. The preference lab starts from the adapter produced by Part 13’s Lab: Fine-Tune a 1B to 4B Model to Follow Your Format, and every script here also accepts a plain base model, so a reader who skipped that lab can still run everything with one extra argument.

Part 10’s evaluation harness, in the form of your own task file, run-eval.py and judge.py. The preference lab reuses the judge exactly as Part 10 built it, position swap and all, and the regression check replays your own tasks. Without that file there is nothing to catch a preference tune that made the model nicer and worse.

A served model helps and is not required. Both labs can generate through llama.cpp’s server from Part 6 or the gateway from Part 9, and the reinforcement-learning lab uses vLLM from Part 9 as its rollout engine where the track supports it.

The models stay small on purpose. The preference lab runs on a 1 to 2 billion parameter model with a memory floor of 16 GB, and the reinforcement-learning lab on 1.5 to 4 billion. Nothing here needs a machine larger than the smallest in the course, and the arithmetic that shows why is on the toolchain lesson rather than left as a surprise.

How to work through it

Section titled “How to work through it”

Read the first two lessons before you train anything. The first is the argument for preference data and the shape of the classical pipeline, which every later method is a simplification of. The second is direct preference optimisation and its variants, and it is where beta stops being a number in a config file and starts being a decision.

Then do the preference lab, which is the shorter of the two and produces a result you will have an opinion about. Come back for the reinforcement-learning lessons: the mechanism, then reward functions, then the toolchain and what each tool needs from your machine. Those three are the densest pages in the part, and the reinforcement-learning lab is much easier to run once they are read.

Finish with the reality check. It is the page that decides what this part was worth, and it is built to be able to disappoint you.

Part14 of 27Level3 — Model BuilderPages8Estimated time5.5 hours

0 / 8 lessons in this part completed