Post-Training: SFT, Preference Tuning and Reinforcement Learning

By the end of this lesson you will be able to name the post-training stages in order and say what each one does to the weights; show, on a model on your own disk, which tokens supervised fine-tuning scores and how far post-training moved their probabilities; write down and compute the reward-model, DPO and GRPO losses; say what makes a reward “verifiable”; price a thinking block before the first answer token; and read a model card’s post-training paragraph against the report behind it. It also explains why the same model comes in -Base, -Instruct and -Thinking flavours, a choice you make at every download.

Base versus instruct

A base model is the checkpoint as pretraining left it: a next-token predictor over web-scale text, with the knowledge and the grammar and no disposition to answer. An instruct model is the same architecture and tokeniser with the weights moved further, so that a reply in a fixed chat format has become the likely continuation of a question in that format. The Part 2 lab showed you the difference from outside: seven raw tokens of a quiz question, and the base checkpoint greedily produced more quiz questions while the instruct checkpoint produced an answer.

Two things are added to get from one to the other. The first is a format. The assistant’s turns are laid out with control tokens that separate the system instruction, the user’s message and the reply, and the layout is the model’s chat template: a Jinja program stored in tokenizer_config.json that turns a list of messages into one string. The snippets below use the pair the Part 2 lab downloaded, Qwen3-1.7B and Qwen3-1.7B-Base (Apache-2.0, not gated; the 0.6B pair on Part 2’s reduced path carries the same licence). They were checked on 2026-09-12 with transformers 5.16.1 · verified 2026-09-08. Render the template and look at what the model actually receives:

RunnableAll tracks

template-tokens.py

"""Render a chat through the tokeniser's template, with thinking on and off."""
from pathlib import Path
from transformers import AutoTokenizer

# Part 2's primary path; reduced path: qwen3-0.6b
MODEL = Path("~/llm-course/models/qwen3-1.7b").expanduser()
tok = AutoTokenizer.from_pretrained(MODEL)

messages = [{"role": "user", "content": "What is 17 * 23?"}]
for thinking in (True, False):
    text = tok.apply_chat_template(messages, tokenize=False,
                                   add_generation_prompt=True, enable_thinking=thinking)
    ids = tok(text)["input_ids"]
    print(f"enable_thinking={thinking}: {len(ids)} tokens\n{text!r}\n")

for t in ("<|im_start|>", "<|im_end|>", "<think>", "</think>"):
    print(f"{t:14} id {tok.convert_tokens_to_ids(t)}")

The output is the same on either pair: Qwen3-1.7B’s tokenizer.json and tokenizer_config.json are byte-identical to Qwen3-0.6B’s (the Hub file listings give both the same hashes).

Output — what you should see

enable_thinking=True: 18 tokens
'<|im_start|>user\nWhat is 17 * 23?<|im_end|>\n<|im_start|>assistant\n'

enable_thinking=False: 22 tokens
'<|im_start|>user\nWhat is 17 * 23?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'

<|im_start|>   id 151644
<|im_end|>     id 151645
<think>        id 151667
</think>       id 151668

Everything in that string is ordinary vocabulary. <|im_start|> is token 151644 the way What is token 3838; the template is text, and the model was trained on text laid out this way. The add_generation_prompt=True argument appends <|im_start|>assistant\n, the tokens the Transformers documentation describes as indicating that an assistant message comes next, and enable_thinking=False is a Qwen3 template variable that writes a closed, empty thinking block into the prompt, four more tokens the model then continues past. The base repository ships a template file too, which is why the Part 2 lab had Track M readers pass --ignore-chat-template to mlx_lm.generate; the base weights were never trained on those tokens, and the next section measures what that means.

The second thing added is a disposition: answer the question, follow the instruction, decline some things, stop when the task is done. The web is not written by assistants, so that pattern is not in the pretraining data to be imitated. It has to be put there deliberately, and the whole of post-training is how.

The post-training stages, in the order they are usually applied

1. 1Base modelOut of pretraining. Continues text. Gives the end-of-turn token a probability near zero at the end of an answer, as measured below.
2. 2Supervised fine-tuning (SFT)Train on demonstrations: prompts paired with the reply a good assistant would give, with the loss computed on the reply tokens only. The model learns the format and the habit of answering.
3. 3Preference tuningTrain on judgements of the form "answer A is better than answer B". Either through a reward model and reinforcement learning (RLHF) or directly on the pairs (DPO and its family).
4. 4Reinforcement learning with verifiable rewardsSample a group of attempts at a task a program can check, reinforce the attempts that scored above the group mean, and repeat. This is where reasoning behaviour is trained.
5. 5Instruct or thinking modelWhat you download. The card should say which of these stages were applied and on what; for the small sizes it is often none of them directly, but distillation from a sibling that had all of them.

Not every model has every stage, and the order varies; the card or report says which were applied.

Flavour on the Hub	What was done	Download it when	What breaks if you pick the wrong one
-Base	Pretraining only	You will fine-tune it yourself (Parts 13 and 14), or you want raw text continuation	Chatting to it: it continues your question instead of answering, and rarely stops on its own
-Instruct (or unsuffixed, as with Qwen3)	SFT and preference tuning; for Qwen3 also thinking-mode training, in a mode you can switch off	Chat, tools, structured output, anything interactive	Fine-tuning on top of it works, but Part 13’s challenge shows how easily the disposition is damaged
-Thinking (or thinking mode on)	Reinforcement learning against verifiable rewards, or distillation from a model that had it	Maths, code and other tasks with a checkable answer, when latency is affordable	Every answer costs the thinking tokens first; on simple instructions you pay them for a gain you have to measure

✕Common mistake

The most common cause of a model that “behaves strangely for no visible reason” is the wrong template: a base checkpoint served as a chat model, or an instruct checkpoint fed raw text without its control tokens. The symptom is the Part 2 lab’s base-model transcript: the reply continues the user’s message, wanders into new questions, and runs to the token limit because the end-of-turn token is never produced. The fix is never a sampling setting; it is apply_chat_template on the training side and the engine’s template handling on the serving side, which Part 10 owns.

Supervised fine-tuning

The first stage is the simplest in the whole of post-training: collect examples of the behaviour you want, and train on them with the same cross-entropy loss as pretraining. The one difference is which positions the loss is computed on. The prompt is masked out, so the model is scored only on producing the reply, not on predicting the question it was asked. In the Hugging Face convention a label of -100 at a position means “ignore this token in the loss”, and TRL’s SFTConfig at version 1.12.0 exposes the mask as completion_only_loss for prompt-completion datasets and assistant_only_loss for conversational ones, each documented as computing the loss “only on the completion” or “only on the assistant responses”. Part 11 teaches the dataset formats that drive those switches; here is what they do, on a checkpoint you already have:

RunnableAll tracks

sft-loss-mask.py

"""Which tokens supervised fine-tuning scores, and how base and instruct weights score them."""
from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODELS = Path("~/llm-course/models").expanduser()
# Part 2's primary path; reduced path: ("qwen3-0.6b-base", "qwen3-0.6b")
PAIR = ("qwen3-1.7b-base", "qwen3-1.7b")
messages = [
    {"role": "user", "content": "What is the capital of Brazil?"},
    {"role": "assistant", "content": "The capital of Brazil is Brasília."},
]
tok = AutoTokenizer.from_pretrained(MODELS / PAIR[1])
prompt = tok.apply_chat_template(messages[:1], tokenize=False, add_generation_prompt=True,
                                 enable_thinking=False)
full = tok.apply_chat_template(messages, tokenize=False)
assert full.startswith(prompt)
ids = tok(full)["input_ids"]
n_prompt = len(tok(prompt)["input_ids"])
labels = [-100] * n_prompt + ids[n_prompt:]        # -100 = ignored by the loss

print(f"{len(ids)} tokens, {n_prompt} masked, {len(ids) - n_prompt} scored")
for i, (t, l) in enumerate(zip(tok.convert_ids_to_tokens(ids), labels)):
    print(f"{i:3} {'scored' if l != -100 else '      '} {t!r}")

x = torch.tensor([ids])
y = torch.tensor([labels])
for name in PAIR:
    model = AutoModelForCausalLM.from_pretrained(MODELS / name, dtype=torch.float32).eval()
    with torch.no_grad():
        logits = model(x).logits[0, :-1]                     # position i predicts token i+1
    nll = torch.nn.functional.cross_entropy(logits, x[0, 1:], reduction="none")
    scored = y[0, 1:] != -100
    print(f"\n{name}: mean loss on the {int(scored.sum())} scored tokens = "
          f"{nll[scored].mean():.2f} nats")
    for pos in (n_prompt, len(ids) - 2):                    # first reply token, <|im_end|>
        print(f"  P({tok.convert_ids_to_tokens(ids[pos])!r}) = {torch.exp(-nll[pos - 1]):.4g}")
    del model

Run on the 0.6B pair (Part 2’s reduced path) on a CPU, with the weight-loading progress bars omitted. The token list is exact for the 1.7B pair; the losses and probabilities will differ.

Output — what you should see

29 tokens, 19 masked, 10 scored
  0        '<|im_start|>'
  1        'user'
  2        'Ċ'
  3        'What'
  4        'Ġis'
  5        'Ġthe'
  6        'Ġcapital'
  7        'Ġof'
  8        'ĠBrazil'
  9        '?'
 10        '<|im_end|>'
 11        'Ċ'
 12        '<|im_start|>'
 13        'assistant'
 14        'Ċ'
 15        '<think>'
 16        'ĊĊ'
 17        '</think>'
 18        'ĊĊ'
 19 scored 'The'
 20 scored 'Ġcapital'
 21 scored 'Ġof'
 22 scored 'ĠBrazil'
 23 scored 'Ġis'
 24 scored 'ĠBras'
 25 scored 'ÃŃlia'
 26 scored '.'
 27 scored '<|im_end|>'
 28 scored 'Ċ'

qwen3-0.6b-base: mean loss on the 10 scored tokens = 3.64 nats
  P('The') = 1.065e-05
  P('<|im_end|>') = 9.673e-10

qwen3-0.6b: mean loss on the 10 scored tokens = 0.07 nats
  P('The') = 0.9996
  P('<|im_end|>') = 0.9806

Read the two halves. Of 29 tokens, 19 are context the model is never scored on, and the ten it is scored on include the end-of-turn token, which is the one that makes an assistant stop. The base weights give that token a probability around one in a billion at the end of a complete answer; the post-trained weights give it 0.98. The mean loss on the reply falls from 3.64 nats to 0.07, a factor of about fifty, on a sentence whose content the base model also had in pretraining. The script scores these tokens exactly as SFT would. The instruct weights were not made by SFT alone, though: the Qwen3 report says the 0.6B and 1.7B sizes were distilled from larger siblings (see Reading the post-training paragraph on a card below). What you are looking at is how far post-training as a whole moved the loss that SFT optimises: onto “answer, then stop”. Your numbers on the 1.7B pair will differ; the pattern is what to look at.

The InstructGPT paper is the readable account of SFT on its own, at scale (GPT-3 and InstructGPT are OpenAI’s research models, named here as the subjects of the cited results; they are not in the course’s model reference). Its authors started “with a set of labeler-written prompts and prompts submitted through the OpenAI API”, collected “a dataset of labeler demonstrations of the desired model behavior”, and used it “to fine-tune GPT-3 using supervised learning”. The sizes are the point to remember, because they are what make post-training something a reader can do:

Stage	Published example	Items	Tokens, from stated inputs
Pretraining (GPT-3, the base InstructGPT fine-tuned)	Brown et al. 2020, arXiv:2005.14165, Table 2.1 caption: “All models were trained for a total of 300 billion tokens”	one corpus	300,000,000,000
Pretraining (Qwen3, a current corpus)	Qwen3-1.7B-Base card: “pre-trained on 36 trillion tokens”	one corpus	36,000,000,000,000
SFT	InstructGPT: “about 13k training prompts” with demonstrations	13,000 prompt-reply pairs	at an assumed 500 tokens per pair: 6,500,000
Reward model	InstructGPT: “33k training prompts”, K = 4 to 9 replies each, ranked	6 to 36 comparisons per prompt	as above, times the replies per prompt
RL prompts	InstructGPT: “31k training prompts (only from the API)”	prompts only, no reply	the model writes the replies during training
Reasoning RL	Qwen3 report: “3,995 query-verifier pairs”, “170 RL training steps”	problems plus checkers	the model writes the attempts during training

The 500-token figure is an assumption for the arithmetic, not a measurement; on it, InstructGPT’s SFT set is about 46,000 times smaller than GPT-3’s 300 billion training tokens (300,000,000,000 / 6,500,000); a modern 36-trillion-token corpus widens that further. That ratio is why supervised fine-tuning is cheap, and why it is the stage Part 13 teaches with LoRA adapters on one machine. It is also why the course expects a small dataset to move behaviour quickly, including in directions you did not intend: each example is a large share of the signal, and Part 13’s challenge is where you diagnose a fine-tune that came out worse. The stage is limited in a specific way, too. It can only make the model imitate the examples it was given, style, format, hedging and mistakes included, and it has no way to learn that one answer is better than another unless it is shown only the better ones.

Preference tuning: ranking beats writing

The insight that unlocked the next stage is a practical one about people. Writing an excellent answer to a hard question is slow and needs an expert. Looking at two answers and saying which is better is fast and needs much less expertise. So you collect judgements instead of demonstrations.

The classical recipe, reinforcement learning from human feedback, turns those judgements into a model. InstructGPT describes it in one sentence: “We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.” Two trained objects are involved, and each has a loss you can write down.

The reward model r_θ(x, y) is a language model with its final unembedding layer replaced by a head that outputs a single number for a prompt x and a reply y (in InstructGPT, a 6B SFT model, used for policies of every size up to 175B). It is trained on pairs where a labeller preferred y_w over y_l, with the Bradley–Terry loss from Section 3.5 of the paper:

loss(θ) = − E[ log σ( r_θ(x, y_w) − r_θ(x, y_l) ) ]     averaged over the K-choose-2 pairs per prompt

σ is the sigmoid, so the term inside the log is the probability the reward model assigns to the labeller’s choice. The reward’s units do not matter, only gaps: a gap of 1.2 between two replies is a preference probability of 0.769, whichever two numbers produced it. Then the language model, now called the policy π_RL, is trained with PPO to maximise the reward while a penalty holds it near the supervised model it started from:

objective(φ) = E[ r_θ(x, y) − β · log( π_RL(y|x) / π_SFT(y|x) ) ] + γ · E_pretrain[ log π_RL(x) ]

The β term is the KL leash: the more the policy’s probabilities drift from π_SFT’s, the more it pays, which is what stops it from finding gibberish the reward model happens to score well. The γ term mixes pretraining gradients back in. Without it, the paper says, a PPO model trained on its API distribution “suffers from an ‘alignment tax’”, lower scores on several public NLP datasets, and “adding pretraining updates to our PPO fine-tuning (PPO-ptx) mitigates these performance regressions on all datasets”, a documented case of the forgetting this lesson returns to at the end.

The headline result is worth remembering because it is the strongest argument in the field for post-training over scale: “outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters”.

!Important

That result is about preference on a prompt distribution, not knowledge; the reality check is built on the difference.

Direct preference optimisation

RLHF works and it is fiddly: two models to train, a reinforcement-learning loop, sampling from the model during training, and hyperparameters that need care. DPO removes most of that. Its authors introduce “a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss”, and describe the result as “stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning”.

The closed form is Equation 7 of the paper. Hold a frozen copy of the starting model as the reference π_ref, feed in a prompt with a preferred reply y_w and a rejected one y_l, and minimise:

L_DPO = − E[ log σ( β · log( π_θ(y_w|x) / π_ref(y_w|x) ) − β · log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]

implicit reward of any reply:   r̂(x, y) = β · log( π_θ(y|x) / π_ref(y|x) )

Each log π(y|x) is the sum of the per-token log-probabilities of the reply, so one training step is four forward passes and no generation. The paper describes β as “a parameter controlling the deviation from the base reference policy”, and the gradient as weighting each pair “by how much higher the implicit reward model r̂θ rates the dispreferred completions, scaled by β”: pairs the model already orders correctly contribute little, pairs it gets backwards contribute most. Put numbers through both losses:

RunnableAll tracks

preference-losses.py

"""Bradley-Terry preference probability and the DPO loss, from four log-probabilities."""
import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

# 1. A reward model scores two answers to the same prompt (scalar rewards, any units).
r_chosen, r_rejected = 1.8, 0.6
p = sigmoid(r_chosen - r_rejected)
print(f"reward gap {r_chosen - r_rejected:+.2f}  ->  P(chosen preferred) = {p:.3f}"
      f"  reward-model loss = {-np.log(p):.3f}")

# 2. DPO needs no reward model. Per answer: log-prob under the policy and under the
#    frozen reference (sum over the answer's tokens; typical sizes for a ~40-token reply).
logp_pol_chosen,  logp_ref_chosen  = -52.0, -55.0   # policy now likes chosen a bit more
logp_pol_rejected, logp_ref_rejected = -58.0, -57.0  # and rejected a bit less
print("\nbeta  reward_chosen  reward_rejected  margin   DPO loss")
for beta in (0.05, 0.1, 0.5):
    rw_c = beta * (logp_pol_chosen - logp_ref_chosen)      # implicit reward, chosen
    rw_r = beta * (logp_pol_rejected - logp_ref_rejected)  # implicit reward, rejected
    margin = rw_c - rw_r
    loss = -np.log(sigmoid(margin))
    print(f"{beta:<5} {rw_c:+13.2f} {rw_r:+16.2f} {margin:+8.2f} {loss:9.3f}")

# 3. Before any training the policy IS the reference: every log-ratio is 0.
print(f"\nuntrained model: margin 0, loss = {-np.log(sigmoid(0.0)):.3f} (= ln 2)")

Output — what you should see

reward gap +1.20  ->  P(chosen preferred) = 0.769  reward-model loss = 0.263

beta  reward_chosen  reward_rejected  margin   DPO loss
0.05          +0.15            -0.05    +0.20     0.598
0.1           +0.30            -0.10    +0.40     0.513
0.5           +1.50            -0.50    +2.00     0.127

untrained model: margin 0, loss = 0.693 (= ln 2)

Three readings. A fresh DPO run starts at a loss of ln 2 for every pair, because the policy is the reference and both implicit rewards are zero; a first logged loss far from 0.69 means the reference is not what you think it is. The same movement in log-probability produces a margin ten times larger at β = 0.5 than at β = 0.05, so a large β is satisfied by a small move and a small β demands a large one, and a large move in log-probability can cost fluency. And the loss depends only on the margin: the chosen reply’s own probability can fall during a healthy run as long as the rejected one falls faster, which is a pattern Part 14 teaches you to read off TRL’s logs. DPO is within reach on one machine, which is why that part teaches it as the first preference method with your hands on it, and why a family of variants with names like IPO, KTO, ORPO and SimPO exists to trade off its weaknesses.

Reinforcement learning with verifiable rewards

Preference tuning optimises for what people liked. For maths, code and other tasks with a checkable answer, there is something better available: a reward a program can compute. Did the final number match? Did the unit tests pass? Did the output parse as JSON? No human, no reward model, and no reward-model errors to exploit. In the first version of the DeepSeek-R1 paper (DeepSeek-R1, R1-Zero and the distilled models below are the paper’s research models, named here as the subjects of the cited results; they are not in the course’s model reference), the authors say why they chose it in one sentence: “we do not apply the outcome or process neural reward model” because “the neural reward model may suffer from reward hacking”. Their rewards were an accuracy reward that “evaluates whether the response is correct”, checked by a rule against a final answer written in a fixed place, and a format reward that “enforces the model to put its thinking process between <think> and </think> tags”.

The optimiser is GRPO, group relative policy optimisation, introduced in the DeepSeekMath paper as a method that “foregoes the critic model, instead estimating the baseline from group scores”. Sample G attempts at the same problem, score each with the verifier, and give every attempt an advantage relative to its siblings:

A_i = ( r_i − mean(r_1 … r_G) ) / std(r_1 … r_G)

then, per token of attempt i:  ratio = π_θ(token) / π_old(token)
                               take  min( ratio · A_i,  clip(ratio, 1−ε, 1+ε) · A_i )
                               and subtract β · KL( π_θ ‖ π_ref )

A positive advantage raises the probability of every token in that attempt; a negative one lowers it; the clip removes the incentive to push a token’s probability ratio past 1 ± ε in the direction the advantage favours (beyond that bound the clipped term is flat and contributes no gradient), so it limits the incentive, not the move; the β term plays the role of RLHF’s KL leash, measured against a frozen reference and added to the loss rather than folded into the reward. Nothing tells the model how to reason. The longer, self-checking attempts simply score above the group mean more often, so they are reinforced. The arithmetic of the advantage is the whole idea:

RunnableAll tracks

grpo-advantages.py

"""Group-relative advantages: what one GRPO step pushes towards and away from."""
import numpy as np

def advantages(rewards):
    r = np.asarray(rewards, dtype=float)
    spread = r.std()
    return (r - r.mean()) / spread if spread > 0 else np.zeros_like(r)

# Eight sampled attempts at one maths problem, scored 1 by an exact-match checker, else 0.
for name, rewards in [("3 of 8 correct", [1, 0, 0, 1, 0, 0, 0, 1]),
                      ("1 of 8 correct", [0, 0, 0, 0, 0, 0, 1, 0]),
                      ("0 of 8 correct", [0, 0, 0, 0, 0, 0, 0, 0]),
                      ("8 of 8 correct", [1, 1, 1, 1, 1, 1, 1, 1])]:
    r = np.asarray(rewards, dtype=float)
    a = advantages(rewards)
    print(f"{name}: mean {r.mean():.3f} std {r.std():.3f}")
    print("  rewards    ", " ".join(f"{x:+.2f}" for x in r))
    print("  advantages ", " ".join(f"{x:+.2f}" for x in a))

Output — what you should see

3 of 8 correct: mean 0.375 std 0.484
  rewards     +1.00 +0.00 +0.00 +1.00 +0.00 +0.00 +0.00 +1.00
  advantages  +1.29 -0.77 -0.77 +1.29 -0.77 -0.77 -0.77 +1.29
1 of 8 correct: mean 0.125 std 0.331
  rewards     +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +1.00 +0.00
  advantages  -0.38 -0.38 -0.38 -0.38 -0.38 -0.38 +2.65 -0.38
0 of 8 correct: mean 0.000 std 0.000
  rewards     +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
  advantages  +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00
8 of 8 correct: mean 1.000 std 0.000
  rewards     +1.00 +1.00 +1.00 +1.00 +1.00 +1.00 +1.00 +1.00
  advantages  +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00 +0.00

These use the population standard deviation; TRL 1.12.0’s GRPOTrainer uses the sample standard deviation (Bessel-corrected) plus 1e-4, so its advantages are slightly smaller (3 of 8: about +1.21 and -0.72). The signs and the zero-spread rule are the same.

The last two groups are the decision rule. A problem the model never solves, or always solves, has zero spread, so every advantage is zero and the step teaches nothing; with a group of eight, the model must already succeed at least one time in eight for the problem to carry any signal at all. Reinforcement learning of this kind sharpens behaviour the model can already produce sometimes. It does not conjure it, which is one reason the small models on your machine got their reasoning a different way, below.

With all three methods defined, they compare like this:

Method	Trained objects	Resident in memory during a step	Data it consumes	Reach for it when
RLHF with PPO	Reward model, policy, value network	Policy, reference, reward model, critic; plus rollouts	Ranked replies, then prompts to sample from	You have a labelling operation and a cluster; not a home method
DPO	Policy only	Policy and a frozen reference (or an adapter you can switch off)	Triples: prompt, chosen, rejected	You can write or collect a few thousand pairs and want a style or format preference
GRPO against a verifier	Policy only	Policy, optionally a reference; plus a group of rollouts per prompt	Problems plus a checker; no replies at all	The task has a program-checkable answer and the model solves it some of the time

The abstract of the revised (2026) version of the DeepSeek-R1 paper is the reference point for what the recipe produced at scale: “the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories”, with “the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation”. The pure-RL model, R1-Zero, also showed what a verifier does not check: the same version’s introduction says it “faces challenges such as poor readability and language mixing, occasionally combining English and Chinese within a single chain-of-thought response”, because nothing in an exact-match reward penalises either. The released R1 therefore wraps the RL stage in supervised ones, as the first version of the paper lays out in its Section 2.3:

DeepSeek-R1 stage	What it consumes	What it is for
Cold start (SFT)	“thousands of cold-start data” of long, readable working	Gives the RL stage a starting policy that already writes legible working in one language
Reasoning-oriented RL	Problems with verifiers; a language-consistency reward added	Where the reasoning behaviour is trained
Rejection sampling and SFT	About 800k samples: some 600k reasoning outputs of the RL model, filtered for correctness and readability, plus about 200k general samples	Restores writing, factual and chat behaviour the RL stage did not reward
RL for all scenarios	Verifiable rewards plus preference rewards	Helpfulness and harmlessness on top of reasoning

The first version of the paper also makes the point that leads into the next lesson. The small R1 models were not produced by this pipeline. Six base models from 1.5B to 70B parameters, from two publishers, were fine-tuned on R1’s outputs, and “for distilled models, we apply only SFT and do not include an RL stage”. Run head to head, the distilled 32B “performs significantly better than DeepSeek-R1-Zero-Qwen-32B”, the same base put through large-scale RL directly, and the authors conclude that “distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power”. The Qwen3 report reaches the same place with a cost figure, measured on Qwen3-8B (Apache-2.0): its on-policy distillation “achieves significantly better performance than reinforcement learning while requiring approximately only 1/10 of the GPU hours”. Part 14 runs GRPO on one machine and has you watch the reward curve; its reality check asks whether the gain generalised.

Thinking models

A “thinking” or “reasoning” variant is a model post-trained to produce a stretch of working before its answer, inside marked-out tokens that a client can hide. There is nothing architecturally special about it. The working is ordinary generated text, and <think> is token 151667 in the Qwen3 vocabulary, generated one at a time like every other token. The training made long, self-checking working the thing that scored well; the format reward made the tags appear around it. Qwen3 trains both modes into one checkpoint in its “Thinking Mode Fusion” stage, and its report explains the empty block the template wrote earlier on this page: “for non-thinking mode samples, we retain an empty thinking block in the assistant’s response. This design ensures internal format consistency within the model”.

Three consequences matter locally. The first is cost: thinking tokens are decoded at the same per-token rate as the answer and are usually discarded, so the wait before the first answer token is the working’s length divided by your decode rate. The inference lesson shows you how to predict that rate from memory bandwidth, and Part 6 has you measure it. Arithmetic from illustrative inputs, not a measurement:

Working length	At 10 tokens per second	At 30 tokens per second	At 100 tokens per second
500 tokens	50 s before the answer starts	17 s	5 s
2,000 tokens	200 s	67 s	20 s
8,000 tokens	800 s	267 s	80 s

Every one of those tokens also occupies the context window and the KV cache for the rest of the turn, which is why the Qwen3 card’s best-practice section advises that in multi-turn use “the historical model output should only include the final output part and does not need to include the thinking content”, and why the report describes a thinking budget: “when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction”. The second consequence is that the working is most likely to pay back on problems with a checkable answer; on straightforward instruction-following, whether the tokens buy anything is a measurement, which the reality check later in this part makes per category. The third is that the switch lives in the prompt, not the weights: enable_thinking, /think and /no_think are template and text conventions that Part 10 owns, and Part 4 reads the names and budgets off a release.

Turn thinking on when	Leave it off when
The task has a checkable answer: arithmetic, code that must run, a constraint puzzle	The task is formatting, extraction, summarising or chat
You will check the answer, not just read it	The reply is consumed by a program with a latency budget
Latency of minutes is acceptable at your decode rate	Context is tight and the working would crowd out the documents

Two failures are specific to thinking mode, and both look like a broken model when they are not:

Symptom	Cause	Fix
The reply is all working, and the answer is empty or cut off	The token limit ran out inside the <think> block, as in the Part 2 lab’s 80-token run, where the block never closed and the answer had 0 tokens	Raise the token limit, turn thinking off for the task, or cap the working with a thinking budget where your engine offers one
Thinking mode repeats itself without end	Greedy decoding, which the Qwen3-1.7B card warns “can lead to performance degradation and endless repetitions”	The card’s thinking-mode settings, Temperature=0.6, TopP=0.95, TopK=20 and MinP=0, which Part 10 teaches

Reading the post-training paragraph on a card

Take the model you downloaded in Part 2. The Qwen3-1.7B card’s overview says, in full, “Training Stage: Pretraining & Post-training”; the base card says “Pretraining”. That is a category, not a description, and the description is in the technical report the card cites. There, the pipeline has four stages, each a thing this lesson has named:

Qwen3 report stage	What the report says it is	The stage on this page
Long-CoT Cold Start	“curating a comprehensive dataset that spans a wide range of categories, including math, code, logical reasoning, and general STEM problems”	Supervised fine-tuning on demonstrations of working
Reasoning RL	“a total of 3,995 query-verifier pairs, and employed GRPO to update the model parameters”; the flagship’s AIME’24 score “increases from 70.1 to 85.1” over “170 RL training steps”	Reinforcement learning with verifiable rewards
Thinking Mode Fusion	“integrate the ‘non-thinking’ capabilities into the previously developed ‘thinking’ model”	Supervised fine-tuning again, on both formats
General RL	“broadly enhance the models’ capabilities and stability across diverse scenarios”	Preference tuning, with learned and rule-based rewards

Then the sentence that matters for the size on your disk. The report states that distilling from the flagship “eliminates the necessity of performing an exhaustive four-stage training process individually for every small-scale model”, and lists the models made that way: “5 dense models (Qwen3-0.6B, 1.7B, 4B, 8B, and 14B) and one MoE model (Qwen3-30B-A3B)”, all six Apache-2.0 in the course’s model reference. So the checkpoint whose end-of-turn probability you measured above (1.7B or 0.6B, both on the report’s list) was not put through GRPO itself. Its behaviour was distilled from siblings that were, in an off-policy phase trained on the teachers’ outputs in both modes and an on-policy phase in which the student’s own outputs are scored against the teacher’s logits, and every claim the card makes about reasoning is a claim about how well that transfer worked. Reading a card this way, category on the card, mechanism in the report, and then which sizes the mechanism was actually applied to, is the habit Part 4 turns into a checklist.

✕Common mistake

The commonest reading error on a model card is to treat “instruct” as a quality grade. It is not. It says a base model was put through some of the stages above, on some data, with some objective, or was distilled from one that was. The card should say what was done; if it does not, that is worth noticing before you build on it.

What each stage changes and what it cannot

Part 13 puts a whole lesson on why these stages change behaviour rather than add knowledge: an SFT set tens of thousands of times smaller than the pretraining corpus, or more, cannot install the facts that corpus put there. Part 10 teaches retrieval as the tool for facts instead.

Stage	Changes	Cannot	Documented failure	Where you measure it
SFT	Format, disposition, style, what “done” looks like; the probability of the end-of-turn token	Add facts; distinguish a good reply from a bad one it was shown	Imitating the examples’ mistakes; forgetting what the base could do (catastrophic forgetting)	Part 13’s challenge, “the fine-tune that got worse”
RLHF / DPO	Which of two plausible replies is produced; tone, length, refusals	Reward anything the pairs did not contrast	Reward hacking against a learned reward; replies growing longer because longer was preferred; regressions on public NLP benchmarks, InstructGPT’s “alignment tax”, mitigated by PPO-ptx	Part 14’s DPO lab logs mean length and a regression check
RL with verifiable rewards	How often a checkable task is solved; the length and shape of the working	Teach a task the model solves zero times in a group; check anything the verifier does not	Zero-spread groups that teach nothing; format tricks that satisfy the checker; language mixing	Part 14’s GRPO lab and its reality check on generalisation
Distillation	Moves all of the above into a smaller model, at a fraction of the cost	Exceed the teacher on what the teacher got wrong	The student learns the teacher’s mistakes	Part 15’s challenge of exactly that name

What no stage measures for you is whether the behaviour you trained is the behaviour you needed, which is why every training part in this course ends with an evaluation against the task you started from, and why Part 16 exists.

Where this is taught properly

With your own hands: Part 13 for supervised fine-tuning, Part 14 for DPO and GRPO, Part 15 for distillation, and Part 4 for reading the names and cards these stages leave behind.

Diagnose the failure before selecting the training stage

Consider a model that answers a support question incorrectly. If the policy changed yesterday, retrieving the current policy addresses unavailable evidence. If the model finds the right policy but returns the wrong JSON fields, supervised examples may address the output contract. If two valid answers differ in tone and reviewers consistently prefer one, preference data expresses that comparison. If correctness can be determined by running a program, a verifier can provide a reward for repeated attempts.

These interventions have different data requirements. A preference label is not a corrected answer; a reward is not an explanation; a successful demonstration does not include the model’s failed attempts. Converting one dataset type to another can discard information or introduce assumptions.

Write down the observable defect, the proposed intervention and the measurement that would falsify your choice. Include a regression set containing tasks the model already performs adequately. Post-training can shift behaviour without adding reliable access to current facts, and an improvement in the target style can coexist with worse factual answers. The evaluation must make that trade visible.

Recap

A base model continues text; post-training makes it answer, visible as an end-of-turn token that goes from one in a billion to nearly certain. Supervised fine-tuning is pretraining’s loss on demonstrations (InstructGPT used about 13k prompts) with the prompt masked out, and preference tuning learns from judgements, through a reward model and a KL-leashed policy or directly with DPO’s margin loss, which starts at ln 2. Reinforcement learning against a verifier reinforces attempts that beat their group’s mean and teaches nothing on problems the model always or never solves, and the small reasoning models on your machine got that behaviour by distillation. Thinking is ordinary tokens priced at your decode rate, and no stage adds knowledge: each changes behaviour, and each has a documented way of going wrong.

Check your understanding

Question 1. A GRPO step samples eight attempts at one problem and the checker scores two of them 1 and six of them 0. What advantages do the attempts receive?

• +1 for each correct attempt and -1 for each wrong one
• About +1.73 for each correct attempt and about -0.58 for each wrong one
• +0.25 for each correct attempt and -0.25 for each wrong one
• Zero for every attempt, because most of the group failed

Show the answer and why

Answer: About +1.73 for each correct attempt and about -0.58 for each wrong one

The mean is 0.25 and the population standard deviation is the square root of 0.25 x 0.75, about 0.433. Each correct attempt gets (1 - 0.25) / 0.433, about +1.73; each wrong one gets -0.25 / 0.433, about -0.58. The rarer the success, the larger its advantage, and the six failures share the pull the other way. Only a group with no spread at all gives zero to everyone.

Question 2. You are formatting a conversational dataset for supervised fine-tuning with the model's own chat template. Which of these is the bug?

• Calling apply_chat_template with add_generation_prompt=False on each training conversation
• Calling apply_chat_template with add_generation_prompt=True on each training conversation
• Setting the label to -100 on every prompt token so the loss covers only the reply
• Including the end-of-turn token at the end of every assistant reply in the scored tokens

Show the answer and why

Answer: Calling apply_chat_template with add_generation_prompt=True on each training conversation

The Transformers documentation is explicit: when training, "set add_generation_prompt=False because the additional tokens to prompt an assistant response aren't helpful during training". With it on, every example ends in a dangling assistant header after the real reply, and the model is scored on producing it. Masking the prompt and scoring the end-of-turn token are exactly what the loss-mask script on this page showed.

Question 3. The first logged DPO loss of a fresh run is 0.693 regardless of the beta you chose. Why?

• The learning rate has not warmed up yet
• The policy still equals the reference, so both implicit rewards are zero, the margin is zero, and minus log sigmoid of zero is ln 2
• Beta only affects the reward model, which DPO does not train
• The chosen and rejected replies are identical in the first batch

Show the answer and why

Answer: The policy still equals the reference, so both implicit rewards are zero, the margin is zero, and minus log sigmoid of zero is ln 2

Each implicit reward is beta times the log ratio of policy to reference, and before any update that ratio is exactly one for every reply. The loss cannot move until the weights do, and what beta then controls is how large a move in log-probability it takes to register a given margin: at a large beta a small move satisfies the loss early, at a small beta the model has to travel further.

Question 4. Which of these are verifiable rewards suitable for reinforcement learning without a human or a reward model? Select all that apply.

• The final numeric answer matches the known answer
• The generated code passes a set of unit tests
• The answer is more insightful than the last one
• The output parses as valid JSON with the required keys

Show the answer and why

Answer: The final numeric answer matches the known answer, The generated code passes a set of unit tests, The output parses as valid JSON with the required keys

A verifiable reward is one a program computes with no judgement call. Insightfulness needs a human or a learned reward model, which brings back the possibility the DeepSeek-R1 authors avoided on purpose: a policy that finds ways to score well that nobody intended.

Question 5. A team fine-tunes a 4B instruct model on 2,000 question-and-answer pairs about their product, and afterwards it answers confidently about product features that do not exist. What is the most likely explanation?

• The learning rate was too low
• Supervised fine-tuning taught the style and confidence of the answers without installing the underlying facts, so the model produces answers of the right shape and wrong content
• The model needs more parameters
• The chat template was correct but unnecessary

Show the answer and why

Answer: Supervised fine-tuning taught the style and confidence of the answers without installing the underlying facts, so the model produces answers of the right shape and wrong content

Behaviour transfers from a small dataset; knowledge does not. Two thousand pairs of a few hundred tokens is under a million tokens against the trillions that put the facts in a model, which is the standard argument for retrieval over fine-tuning when the goal is facts. Part 13 has a whole lesson and a diagnostic challenge on it.

Question 6. A 1.7B model's card says "Pretraining & Post-training", and the technical report describes a four-stage pipeline for the flagship and strong-to-weak distillation for the small sizes. Which process did the 1.7B weights themselves go through?

• All four stages, at a smaller scale
• Distillation from larger siblings that went through the pipeline: first on their outputs, then on its own outputs scored against their logits; no RL stage of its own
• Only the reinforcement-learning stage, because that is where reasoning comes from
• None; a 1.7B model is a base model

Show the answer and why

Answer: Distillation from larger siblings that went through the pipeline: first on their outputs, then on its own outputs scored against their logits; no RL stage of its own

Qwen3 report Section 4.5 describes two phases for the small sizes: an off-policy phase trained on the teachers' outputs, then an on-policy phase in which the student generates its own responses and is fine-tuned by aligning its logits with those of Qwen3-32B or Qwen3-235B-A22B to minimise the KL divergence. Scoring the student's own samples can look like reinforcement, but there is no reward and no advantage, only a match to the teacher's distribution, so it is still distillation. DeepSeek-R1's distilled models had only the first kind, supervised training on the teacher's outputs: "for distilled models, we apply only SFT and do not include an RL stage". Both reports found distillation beat running RL on the small model directly, so the reasoning you see on a laptop is transferred behaviour, and the question to ask of the card is how well the transfer was measured.

Check my answersClear