Generalisation: Train, Validation, Test and Overfitting
By the end of this lesson you will be able to say, with numbers, why a model that scores perfectly
on its training data may have learned nothing; design a split of your data that can detect that;
read a pair of loss curves, decide whether a turn in the validation curve is real or noise, and
name the checkpoint to keep; say what weight decay, dropout and early stopping each do to the
parameters one step at a time; and recognise the ways an evaluation leaks. That last skill is the
one the rest of the course leans on: Parts 13 to 17 ask “did the fine-tune help”, “did the
distillation work”, “did quantisation hurt”, and every one of those questions is answered with
the method taught here. Every number on this page is arithmetic from stated inputs or the printed
output of a snippet on this page or of the lab’s script, never a speed measurement of a machine.
The snippets run with a bare python3, except the two marked as needing the lab environment,
which use the PyTorch that
the lab installs.
Learning the pattern or memorising the examples
Section titled “Learning the pattern or memorising the examples”Two students prepare for an examination from the same past papers. One works out how to solve each kind of problem. The other memorises the answer to every past question. On the past papers both score full marks. On the examination only the first does well.
A model can do either. Memorisation fits the training examples, including their noise and their
accidents, without capturing the pattern that produced them. Generalisation captures the
pattern, so that new examples drawn from the same source are handled correctly. The difference is
invisible in the training loss, which is measured on the very examples a memoriser has memorised.
It is visible only on examples the model has not seen, and the gap between the two,
held-out loss − training loss, is the generalisation gap: the quantity this whole lesson is
about.
The mechanism is capacity. The first lesson’s
cubic through four points had as many parameters as examples and reached zero loss by fitting the
noise in the four y values; a lookup table with one entry per example is the limit of that idea,
zero training loss and no rule at all. Neural networks are large enough to be lookup tables for
any training set they are given. Understanding deep learning requires rethinking generalization
(Zhang, Bengio, Hardt, Recht and Vinyals, 2017) reports that image classifiers trained with the
ordinary gradient methods “easily fit a random labeling of the training data”, and that the effect
“is qualitatively unaffected by explicit regularization”. The models in this course have billions
of parameters. A training loss near zero therefore says nothing on its own. The snippet puts the
two kinds of rule side by side on the first lesson’s data, y = 2x + noise: the one-parameter
line, and a lookup table that answers every question with the y of the nearest training x.
RunnableAll tracks
"""A learner and a memoriser on the same noisy examples of y = 2x + noise.The learner is the first lesson's one-parameter line; the memoriser answersevery question with the y of the nearest training x. Synthetic arithmetic, nota measurement."""import random
random.seed(0)
def examples(n): xs = [random.uniform(0.0, 4.0) for _ in range(n)] return [(x, 2.0 * x + random.gauss(0.0, 1.0)) for x in xs]
train, held_out = examples(64), examples(10_000)
w = sum(x * y for x, y in train) / sum(x * x for x, _ in train) # least-squares slopedef line(x): return w * xdef lookup(x): # nearest training x return min(train, key=lambda p: abs(p[0] - x))[1]
def mse(rule, data): return sum((rule(x) - y) ** 2 for x, y in data) / len(data)
print(f"learned w = {w:.4f} (noise variance, the floor no rule can beat: 1.0)")print(f"{'rule':<8} {'training loss':>14} {'held-out loss':>14}")for name, rule in [("line", line), ("lookup", lookup)]: print(f"{name:<8} {mse(rule, train):>14.4f} {mse(rule, held_out):>14.4f}")Output — what you should see
learned w = 1.9462 (noise variance, the floor no rule can beat: 1.0)rule training loss held-out lossline 0.9332 0.9918lookup 0.0000 2.0046The line’s two losses agree, both close to the noise variance of 1.0, which no rule can beat
because the noise in a new y is unpredictable. The lookup table has the perfect training score
and about twice the noise on new data: it answers with another example’s noise on top of the new
example’s own. Ranked by training loss the lookup wins; ranked by held-out loss it is the worst
rule on the page. Only the held-out column measured learning.
For language models the same failure comes at two scales. Fine-tune on a few hundred examples for enough epochs and the model reproduces them verbatim while its loss on new prompts climbs; Part 13’s challenge, the fine-tune that got worse, is built from that. Pretrain on the internet and the training data already contains most public test sets, so a score on one of them measures recall rather than ability; Part 4’s model-cards lesson names this contamination and Part 16 measures it.
The split: training, validation, test
Section titled “The split: training, validation, test”The remedy is to keep back data the model never trains on, and to measure on that. Three sets, three jobs:
| Set | Who sees it | What it decides | Looked at | Its score estimates |
|---|---|---|---|---|
| Training | Gradient descent, every epoch | The parameters | Constantly | Nothing about new data |
| Validation | You, after every epoch or every N steps | Hyperparameters, when to stop, which checkpoint to keep, which of two runs to ship | Many times | New-data performance, optimistic by the number of decisions made from it |
| Test | Nobody, until the end | Nothing | Once, to report the number | New-data performance, honestly, because no decision was made from it |
The lab’s train-mnist.py splits MNIST’s 60,000 training images into 50,000 for training and
10,000 for validation with random_split under a seeded generator, and touches the 10,000 test
images once, with the checkpoint the validation set chose. Two habits are in that sentence. The
seed is recorded, so “the validation set” names the same images on every run and two loss curves
are comparable; Part 11’s datasets lesson
makes the same point for instruction data, where the split is a train_test_split call with a
seed. And the test set is spent by the chosen checkpoint, not the last one, so the test number
describes the model that would ship.
Why does the validation set need a reserve behind it? Because every choice made from its score uses some of its information. Choose between two runs on it and you have chosen the one whose luck on those particular examples was better, along with the one that learned more. Do it two hundred times and the winner’s validation score is partly a record of luck. The snippet puts a number on that with candidates that have no skill at all: each answers every question with a coin.
RunnableAll tracks
"""Candidate 'models' with no skill at all: each answers every question with a coin.Score them on a validation set, keep the winner, then score the winner on a testset it was not chosen on. Synthetic arithmetic, not a measurement."""import random
random.seed(0)N_TEST, TRIALS = 10_000, 50
def coin_accuracy(n): return sum(random.random() < 0.5 for _ in range(n)) / n # right or wrong at random
print(f"{'candidates':>10} {'validation n':>12} {'winner on validation':>20} {'winner on test':>14}")for candidates in (1, 10, 200): for n_val in (50, 200, 1000): best_val = best_test = 0.0 for _ in range(TRIALS): winner = max(coin_accuracy(n_val) for _ in range(candidates)) best_val += winner / TRIALS best_test += coin_accuracy(N_TEST) / TRIALS print(f"{candidates:>10} {n_val:>12} {best_val:>20.3f} {best_test:>14.3f}")Output — what you should see
candidates validation n winner on validation winner on test 1 50 0.508 0.501 1 200 0.496 0.500 1 1000 0.496 0.500 10 50 0.606 0.501 10 200 0.551 0.500 10 1000 0.522 0.499 200 50 0.691 0.500 200 200 0.594 0.500 200 1000 0.543 0.500One candidate scores what it deserves. Two hundred candidates chosen on fifty examples produce a
winner at 0.69 whose accuracy on anything new is 0.50: nineteen points of optimism from selection
alone, with no model anywhere. The optimism grows with the number of things compared and shrinks
as the validation set grows, roughly as 1/√n, and it never reaches zero, which is why the test
column of the table exists. A hyperparameter sweep, a prompt tuned by looking at scores, a
checkpoint picked by early stopping: each is a row of this output, and the more candidates were
tried, the more the validation number flatters. When the data has structure, several rows from
the same document, user or day, the split has to be made by that group rather than by row;
the leakage section below shows with numbers what happens otherwise.
Reading the two curves
Section titled “Reading the two curves”Plot the training loss and the validation loss against training steps and you get the most useful diagnostic picture in machine learning.
What the two loss curves look like over a run
Early in a run both losses fall together: the model is capturing the pattern, which helps on
examples it has seen and on examples it has not. At some point the training loss keeps falling
while the validation loss stops and turns up, and from there the extra fit is memorisation of the
training set’s noise. The mechanism of the rise is cross-entropy. A network that has memorised a
mislabelled training example has learned to be confident about a wrong answer, and when a held-out
example resembles it the model pays −ln(p) for a p near zero, which is a large number. That is
why validation loss can rise steeply while validation accuracy barely moves: the model gets
the same examples wrong as before, with more confidence each time. The whole picture, from a run
of a few seconds to about a minute on a CPU:
RunnableAll tracks
"""A small network on 200 examples of a two-number pattern with a quarter of thelabels flipped: the training loss falls towards zero while the held-out loss turns.Change one constant and rerun to see what moves the turn. Needs the lab environment(torch); synthetic data, not a measurement."""import torchfrom torch import nn
N_TRAIN, HIDDEN, WEIGHT_DECAY, LABEL_NOISE = 200, 256, 0.0, 0.25 # change one, rerunN_HELD, STEPS = 2000, 2000REPORT = (1, 10, 30, 100, 300, 1000, 2000)
def make(n): x = torch.randn(n, 2) y = (x[:, 0] + x[:, 1] > 0).long() # the pattern: is x0 + x1 positive? flip = torch.rand(n) < LABEL_NOISE # the noise: some labels are wrong return x, torch.where(flip, 1 - y, y)
torch.manual_seed(0)x_tr, y_tr = make(N_TRAIN)x_ho, y_ho = make(N_HELD)model = nn.Sequential(nn.Linear(2, HIDDEN), nn.ReLU(), nn.Linear(HIDDEN, HIDDEN), nn.ReLU(), nn.Linear(HIDDEN, 2))opt = torch.optim.AdamW(model.parameters(), lr=1e-2, weight_decay=WEIGHT_DECAY)loss_fn = nn.CrossEntropyLoss()best_loss, best_step, best_acc = float("inf"), 0, 0.0
print(f"train {N_TRAIN} held out {N_HELD} hidden {HIDDEN} weight decay {WEIGHT_DECAY} label noise {LABEL_NOISE}")print(f"{'step':>5} {'train loss':>10} {'held-out loss':>13} {'held-out acc':>12}")for step in range(1, STEPS + 1): opt.zero_grad() loss = loss_fn(model(x_tr), y_tr) loss.backward() opt.step() with torch.no_grad(): logits = model(x_ho) ho_loss = loss_fn(logits, y_ho).item() ho_acc = (logits.argmax(1) == y_ho).float().mean().item() if ho_loss < best_loss: best_loss, best_step, best_acc = ho_loss, step, ho_acc # what early stopping keeps if step in REPORT: print(f"{step:>5} {loss.item():>10.4f} {ho_loss:>13.4f} {ho_acc:>12.4f}")print(f"held-out minimum {best_loss:.4f} at step {best_step} (accuracy there {best_acc:.4f}); final train loss {loss.item():.4f}")Output — what you should see
train 200 held out 2000 hidden 256 weight decay 0.0 label noise 0.25 step train loss held-out loss held-out acc 1 0.6815 0.7616 0.7215 10 0.6109 0.6466 0.7375 30 0.5657 0.6024 0.7350 100 0.4394 0.8961 0.7000 300 0.2522 2.2010 0.6555 1000 0.0689 4.7670 0.6345 2000 0.0278 7.0599 0.6375held-out minimum 0.5954 at step 21 (accuracy there 0.7260); final train loss 0.0278A quarter of the labels are wrong, so the best any rule can do on new data is the entropy of that
noise, −0.75·ln(0.75) − 0.25·ln(0.25) = 0.562, with an accuracy of 0.75. The held-out loss gets
within 0.03 of that floor at step 21, then climbs past 7 while the training loss falls to 0.03:
the 200 training examples, wrong labels included, have been memorised, and held-out accuracy has
drifted from 0.73 to 0.64. The last line is the checkpoint early stopping would keep. Change one
constant at the top and rerun; the author’s results with the same seed:
| Change from the baseline | Held-out minimum | At step | Training loss at step 2000 | Held-out loss at step 2000 |
|---|---|---|---|---|
| none (200 examples, 256 hidden, no decay, a quarter of labels wrong) | 0.5954 | 21 | 0.0278 | 7.0599 |
LABEL_NOISE = 0.5: labels are coin flips |
0.6998 | 6 | 0.0522 | 5.0596 |
N_TRAIN = 2000 |
0.5637 | 82 | 0.4677 | 0.8194 |
HIDDEN = 4 |
0.5979 | 404 | 0.5383 | 1.1377 |
WEIGHT_DECAY = 1.0 |
0.5948 | 22 | 0.3988 | 0.8498 |
Random labels are fitted as readily as real ones, the Zhang et al. result in miniature, against a
floor of ln 2 = 0.693 and with held-out accuracy at a coin’s. Ten times the data moves the
minimum to within 0.002 of the floor and the turn four times later, and the run can no longer
memorise its way to zero. Sixty-four times fewer hidden units delay the turn twenty-fold without
improving the minimum at all. Heavy weight decay stops the climb but does not touch the minimum
either. Those three rows are the three families of remedy in the next section, and the last line
of every run is the fourth.
The lab’s real run does the same thing on MNIST. Its overfit-5k task trains the two-layer
network on 5,000 images at learning rate 0.5 for sixty epochs, and the epochs below are the
run recorded on the lab page:
| Epoch | Training loss | Validation loss | Validation accuracy | Checkpoint saved |
|---|---|---|---|---|
| 1 | 1.0804 | 0.7966 | 0.7034 | yes |
| 6 | 0.1854 | 0.2830 | 0.9172 | yes |
| 9 | 0.1926 | 2.6640 | 0.4916 | no |
| 13 | 0.1268 | 0.2205 | 0.9370 | yes |
| 24 | 0.0179 | 0.2080 | 0.9436 | yes, for the last time |
| 40 | 0.0065 | 0.2228 | 0.9430 | no |
| 60 | 0.0030 | 0.2316 | 0.9439 | no |
| test set, epoch-24 checkpoint | 0.1911 | 0.9461 |
Three readings. The spike at epoch 9 is not overfitting: both curves jump, which is a learning rate near the edge of what plain SGD tolerates, and both recover. The turn is at epoch 24: validation loss rises from 0.2080 to 0.2316 over the next thirty-six epochs while training loss falls sixfold. And validation accuracy does not move across the turn, 0.9436 against 0.9439, which is the confidence mechanism above: the same digits are wrong, each at a larger loss.
Is the turn real?
Section titled “Is the turn real?”A validation loss is a mean over n examples, and a mean has noise. For an accuracy p measured
on n examples the standard error is
se = sqrt(p × (1 − p) / n)and for a mean loss it is the standard deviation of the per-example losses divided by √n. A
difference between two checkpoints smaller than about 2 × se is noise, whatever the plot looks
like:
| Accuracy p | n examples | se | Smallest difference worth believing, 2·se |
|---|---|---|---|
| 0.94 | 10,000 | 0.0024 | 0.005 |
| 0.94 | 1,000 | 0.0075 | 0.015 |
| 0.94 | 100 | 0.024 | 0.05 |
| 0.70 | 1,000 | 0.014 | 0.03 |
| 0.70 | 100 | 0.046 | 0.09 |
| 0.70 | 40 | 0.072 | 0.14 |
| 0.70 | 20 | 0.10 | 0.20 |
The MNIST accuracies above differ by 0.0003 on 10,000 images, an eighth of a standard error: no change. The rise in loss is a different quantity, and it passes the other test of a real turn: it persists. A single evaluation above the minimum is noise; several in a row, each above the last, is the turn, and the trainers in Part 13 encode exactly that as a patience count. The shapes worth recognising, and what each one asks for:
| Shape | What it means | What to do |
|---|---|---|
| Both falling, validation slightly above training | Learning; not yet overfitting | Keep training, or stop when the epochs cost more than they gain |
| Training keeps falling; validation flattens, then rises for several evaluations in a row | Overfitting from the turn onwards | Keep the checkpoint from the minimum; then more data, a smaller model or regularisation to move the turn later |
| Both high and flat, or falling very slowly | Underfitting: too little capacity, too few steps or too small a learning rate | The opposite remedies: more steps, a larger learning rate, a larger model |
Training loss enormous early, then flat above the guessing loss, or nan |
Learning rate too large | Divide it by ten |
| Validation far below training for many epochs | The two passes differ | Check model.eval() and dropout, then whether the validation set is easier or leaked |
What to do about overfitting
Section titled “What to do about overfitting”Every remedy either gives the pattern more to explain, makes storing individual examples more expensive, or stops before the storing starts. Which one, from what you measured:
| What you measured | Remedy | What it does, mechanically | Where you use it |
|---|---|---|---|
| The turn comes early and the validation minimum is high | More data | Each extra example is one more thing the pattern must explain, and the noise averages out as 1/√n (first lesson); the N_TRAIN = 2000 row |
Part 13 spends more time on the dataset than on the trainer; Part 15 generates data from a teacher |
| Training loss near zero, validation rising steeply | Less capacity, or fewer trainable parameters | A rule with far fewer parameters than examples cannot store them; the HIDDEN = 4 row |
LoRA in Part 13 trains a small fraction of the parameters |
| Same, with the architecture fixed | Weight decay | Shrinks every weight a little on every step, so committing hard to one example costs something | The weight_decay argument of TRL’s SFTConfig and of TrainingArguments, whose default is 0.0; the course’s own runs leave it there (Part 12’s continue-pretraining.py and Part 17’s train-draft.sh set it to 0.0 explicitly) and rely on early stopping instead |
| Same | Dropout | Silences a random subset of units on each training step, so no single path can carry a memorised answer | lora_dropout in Part 13’s adapter configurations |
| Same | Data augmentation | Perturbed copies of examples, so the model must learn what stays the same | Vision more than language; paraphrased synthetic data in Part 15 |
| Any turn at all | Early stopping | Keeps the checkpoint at the validation minimum and stops after patience evaluations without a new one |
The lab’s script; load_best_model_at_end in Part 13 |
Weight decay and dropout, one step at a time
Section titled “Weight decay and dropout, one step at a time”Weight decay is a pull towards zero added to every parameter update. PyTorch’s two optimisers apply it differently, and the difference matters when you read a training configuration:
SGD (torch.optim.SGD, weight_decay=λ): g ← g + λ·w then w ← w − lr·g so w ← w·(1 − lr·λ) − lr·gradientAdamW (torch.optim.AdamW, weight_decay=λ): w ← w − lr·λ·w then the Adam step from the gradient aloneSGD folds the decay into the gradient (the documentation calls it an L2 penalty); AdamW applies
it to the weight directly and keeps it out of the running averages, which is what “decoupled”
means and why the AdamW default is weight_decay=0.01 while SGD’s is 0. In both cases the
weight shrinks by the factor 1 − lr·λ per step, so the same λ means very different things at
different learning rates. Arithmetic for a weight of 1.0 that the loss does not depend on:
| Optimiser and setting | Shrink per step | Steps | Factor after the steps |
|---|---|---|---|
SGD, lr=0.1, weight_decay=0.01 (the lab’s rate) |
1 − 0.001 |
1,955 (five MNIST epochs) | 0.999^1955 = 0.141 |
AdamW, lr=0.001, weight_decay=0.01 (the defaults) |
1 − 0.00001 |
1,955 | 0.981 |
AdamW, lr=0.0002, weight_decay=0.01 (a typical fine-tune) |
1 − 0.000002 |
1,000 | 0.998 |
Dropout, torch.nn.Dropout(p), zeroes each element of its input with probability p on every
training pass, scales the survivors by 1/(1 − p) so the expected sum is unchanged, and does
nothing at all in evaluation mode. The snippet runs both:
RunnableAll tracks
"""What weight decay and dropout do, one step at a time. A parameter with a zerogradient is stepped by SGD and by AdamW with weight_decay=0.01, so the only thingmoving it is the decay; then one dropout layer is run in training and inevaluation mode. Needs the lab environment (torch)."""import torchfrom torch import nn
torch.manual_seed(0)STEPS_PER_EPOCH, EPOCHS = 391, 5 # the lab's 50,000 images in batches of 128
for name, make in (("SGD lr 0.1", lambda p: torch.optim.SGD([p], lr=0.1, weight_decay=0.01)), ("AdamW lr 0.001", lambda p: torch.optim.AdamW([p], lr=1e-3, weight_decay=0.01))): w = nn.Parameter(torch.tensor([1.0])) opt = make(w) print(f"{name:<15} start w = {w.item():.4f}", end="") for epoch in range(1, EPOCHS + 1): for _ in range(STEPS_PER_EPOCH): opt.zero_grad() (w * 0.0).sum().backward() # the loss does not depend on w: gradient 0 opt.step() print(f" after epoch {epoch}: {w.item():.4f}", end="") print()
drop = nn.Dropout(p=0.5)x = torch.ones(8)drop.train()print("dropout, training mode :", drop(x).tolist())print("dropout, training again :", drop(x).tolist())drop.eval()print("dropout, evaluation mode:", drop(x).tolist())Output — what you should see
SGD lr 0.1 start w = 1.0000 after epoch 1: 0.6762 after epoch 2: 0.4573 after epoch 3: 0.3093 after epoch 4: 0.2091 after epoch 5: 0.1414AdamW lr 0.001 start w = 1.0000 after epoch 1: 0.9961 after epoch 2: 0.9922 after epoch 3: 0.9883 after epoch 4: 0.9845 after epoch 5: 0.9806dropout, training mode : [0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 2.0]dropout, training again : [0.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0]dropout, evaluation mode: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]The first two lines are the table’s first two rows to four decimals. The dropout lines are why
model.eval() exists: a validation pass with the module still in training mode scores a network
with half its units silenced, and the shape-table’s last row is the result.
Early stopping
Section titled “Early stopping”The simplest remedy, and the only one that assumes nothing about the model: evaluate on the
validation set every epoch or every k steps, keep the checkpoint with the lowest validation loss,
and stop once patience evaluations in a row have failed to improve on it. The lab’s script does
the keeping half, and its two conditions are worth reading because every trainer in the course has
the same two lines somewhere inside it:
Fragment — not complete on its own
if val_loss < best_val: # False when val_loss is nan, so a diverged epoch is never saved best_val, best_epoch = val_loss, epoch torch.save(model.state_dict(), args.checkpoint) marker = " (saved)"# ... after the last epoch, the test set is scored with the saved checkpoint, not the last weights:model.load_state_dict(torch.load(args.checkpoint, map_location=device))Part 13’s TRL configuration
does the same with load_best_model_at_end=True and a callback that adds the patience; its
train-lora.py sets it with --early-stopping-patience, default 2, which is enough when
evaluation runs once per epoch, as that script’s eval_strategy="epoch" does, and the turn is
real only if it persists; raise it when you evaluate every few hundred steps and the curve is
noisy. In the lab’s overfit-5k run, read the validation column and find the minimum yourself
before trusting the (saved) marker, so that you know what the tool is doing. Two cautions.
Regularisation slows memorisation rather than preventing it, which is what the
WEIGHT_DECAY = 1.0 row and the Zhang et al. quotation both say, so early stopping is not
optional because weight decay is on. And the checkpoint it keeps carries the selection optimism
of the split section, so the number to report is the test one.
Leakage: the silent failure
Section titled “Leakage: the silent failure”Overfitting announces itself in the curves. Leakage does not. Leakage is any route by which information about the held-out examples reaches the training process, so that the held-out score stops measuring generalisation. The curves look healthy, the score looks excellent, and the model falls apart on the first genuinely new input. The forms it takes, in the order you are likely to meet them:
| Form | How it happens | What it looks like | The check |
|---|---|---|---|
| Exact duplicates across the split | The same row twice in the source file, split at random | Validation loss tracks training loss too closely | Deduplicate on normalised text before splitting |
| Near-duplicates | Two versions of a document, a template filled in twice, a paraphrase | The same | Near-duplicate detection (n-gram overlap) before splitting; Part 13’s decontaminate.py |
| Group leakage | Rows from one document, ticket, user or repository land on both sides | A perfect validation score that does not survive a new group | Split by the group, never by the row |
| Preprocessing fitted on everything | Normalisation statistics, a vocabulary or a tokeniser built on all the data, then the split | Small, persistent optimism | Fit every preprocessing step on the training split only |
| Time leakage | The split ignores time, so the training set contains events later than the ones being validated, outcomes included | Excellent scores on anything with dates in it | Split by time when the data has a time |
| Benchmark contamination | The public test set was on the internet the model was pretrained on | Reported scores that a private set of the same kind does not reproduce | Prefer benchmarks newer than the training data; Part 16’s contamination lesson |
| Teacher-generated data | Synthetic training data written by a model that had seen the evaluation set | The student scores like the teacher on that set and unlike it elsewhere | Decontaminate synthetic data against the evaluation set; Part 15’s challenge |
The third row is the one you will make yourself. The snippet builds two hundred “documents” with five near-identical rows each, and scores a memoriser, the nearest-neighbour lookup from the first section, under a split by row and under a split by document, once with labels that follow a pattern and once with labels that are coin flips:
RunnableAll tracks
"""Two hundred documents, five near-identical examples each. A memoriser (answer with thelabel of the nearest training example) is scored under a random split and under asplit by document, for labels that carry a pattern and for labels that are purenoise. Synthetic arithmetic, not a measurement."""import random
random.seed(0)DOCS, PER_DOC = 200, 5
def dataset(labels): rows = [] for doc in range(DOCS): cx, cy = random.uniform(0.0, 10.0), random.uniform(0.0, 10.0) label = (cx + cy > 10.0) if labels == "pattern" else random.random() < 0.5 for _ in range(PER_DOC): # five paraphrases of one document rows.append(((cx + random.gauss(0, 0.01), cy + random.gauss(0, 0.01)), label, doc)) return rows
def nearest_label(train, point): return min(train, key=lambda r: (r[0][0] - point[0]) ** 2 + (r[0][1] - point[1]) ** 2)[1]
def accuracy(train, val): return sum(nearest_label(train, p) == y for p, y, _ in val) / len(val)
print(f"{'labels':<8} {'split':<12} {'train':>6} {'val':>4} {'val accuracy':>13}")for labels in ("pattern", "noise"): rows = dataset(labels) random.shuffle(rows) cut = int(0.8 * len(rows)) by_row = rows[:cut], rows[cut:] held_docs = set(random.sample(range(DOCS), DOCS // 5)) by_doc = [r for r in rows if r[2] not in held_docs], [r for r in rows if r[2] in held_docs] for name, (train, val) in (("by row", by_row), ("by document", by_doc)): print(f"{labels:<8} {name:<12} {len(train):>6} {len(val):>4} {accuracy(train, val):>13.3f}")Output — what you should see
labels split train val val accuracypattern by row 800 200 1.000pattern by document 800 200 0.975noise by row 800 200 1.000noise by document 800 200 0.500The last two lines are the whole argument. With labels that are pure noise there is nothing to learn, and the split by document says so: a coin’s accuracy. The split by row reports a perfect model, because every validation row has a sibling in the training set carrying its label. Nothing in the curves, the loss or the score would have revealed it; only knowing where the rows came from does. With a real pattern the row split still flatters, 1.000 against 0.975, and the 0.025 is the leakage. Part 13’s dataset lesson meets this with support tickets in place of documents.
Why every “it improved” claim needs a held-out set
Section titled “Why every “it improved” claim needs a held-out set”From Part 10 onwards you will keep your own evaluation set: a few dozen tasks that matter to you, with reference answers, written before any training run and held out from every one you ever do. Every claim in Levels 3 and 5 that a model got better, after a fine-tune, after distillation, after reinforcement learning, after quantisation, after training on agent transcripts, is a comparison of scores on that set, before and after. The protocol that makes such a comparison mean something is this page in five lines:
| Requirement | Why | Where the course does it |
|---|---|---|
| The set was written before training and lives in its own file | Otherwise choices made from it are selection, and its score is optimistic | Part 10’s benchmark lab |
| The training data is decontaminated against it, by near-duplicate as well as exact match | Leakage is silent and a good number is as suspect as a bad one | Part 13’s dataset lesson and its decontaminate.py |
| Before and after are scored the same way: same prompts, template, decoding settings, judge | A changed setting is a second variable | Part 16’s reporting checklist |
| The run is repeated, or the noise floor is computed | A difference inside 2 × se is not a result |
Part 11’s experiment tracking; the first task of the Part 13 and Part 16 challenges |
| The number quoted is the held-out one, not the validation loss the checkpoint was chosen on | The chosen checkpoint’s validation score carries the selection optimism | Every training lab from Part 11 |
The fourth line has arithmetic behind it that decides how big your set needs to be. To believe a
change of size Δ in an accuracy near p, the set needs roughly n ≥ 4 × p × (1 − p) / Δ²
examples, which is the 2 × se rule turned round:
| Change you want to detect | Accuracy near | Examples needed |
|---|---|---|
| 0.20 (twenty points) | 0.7 | 21 |
| 0.10 | 0.7 | 84 |
| 0.05 | 0.7 | 336 |
| 0.02 | 0.7 | 2,100 |
| 0.05 | 0.9 | 144 |
A set of twenty to fifty tasks, which is what Part 10 asks you to write, therefore detects a large change and nothing subtle, which is why the Part 13 and Part 16 challenges open by running the same model on the same set twice to measure the run-to-run spread, why Part 10’s harness judges every pair twice, in both orders, and why a fine-tune that “gained three points” on forty tasks is reported as no change. The method has one failure point, leakage, and the previous section is its only defence. Where it comes back: the two evaluations in Part 12’s training loop, the validation split and early stopping in Part 13, the contaminated evaluation planted in Part 16’s challenge, and the lab at the end of this part, where you will watch the validation curve turn on your own machine.
Choose the split that resembles deployment
Section titled “Choose the split that resembles deployment”A random split answers whether the model generalises to randomly withheld rows from this collection. It does not automatically answer whether it generalises to a new customer, a later month or an unseen document family. If several tickets belong to one incident, put the whole incident in one partition. If production predicts future demand, train on earlier observations and evaluate on later ones.
Fit preprocessing on the training partition too. Vocabulary selection, normalisation statistics, missing-value replacement and feature selection can all disclose information from the test set before the model sees a label. A split performed after these transformations is already too late for an honest estimate.
As a thought experiment, imagine perfect validation performance from a dataset with duplicate paragraphs. Remove exact duplicates across partitions and the score falls. That fall is useful evidence: the earlier score measured recognition as well as generalisation. Keep the stricter split, explain the changed question in the report, and avoid tuning repeatedly against the final test set. A difficult test is more valuable than a flattering one when it resembles the actual use case.
A model can fit its training data by learning the pattern or by memorising the examples; the
training loss cannot tell which, and the generalisation gap, held-out loss minus training loss,
can. Three sets: training for the parameters, validation for decisions, whose score grows more
optimistic with every candidate compared on it, and a test set spent once on the chosen
checkpoint. Read a turn in the validation curve against 2 × se and believe it only when it
persists across several evaluations, then keep the checkpoint at the minimum. Leakage is silent,
and only knowing where every evaluation example came from defeats it.
Check your understanding
Sources for this lesson
7 verified · checked 2026-09-12
- 01Deep Learning (Goodfellow, Bengio and Courville) — Chapter 5, Machine Learning Basics§ Chapter 5deeplearningbook.org2026-09-08
- 02Understanding deep learning requires rethinking generalization (Zhang, Bengio, Hardt, Recht and Vinyals, arXiv:1611.03530)§ Abstractarxiv.org/abs/1611.035302026-09-12
- 03PyTorch 2.14 documentation — torch.optim.SGD§ Algorithm box (weight decay); constructor defaultsdocs.pytorch.org/docs/2.14/generated/torch.optim.SGD.html2026-09-12
- 04PyTorch 2.14 documentation — torch.optim.AdamW§ Algorithm box (decoupled weight decay); constructor defaultsdocs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html2026-09-12
- 05PyTorch 2.14 documentation — torch.nn.Dropout§ Training and evaluation behaviour; scaling factordocs.pytorch.org/docs/2.14/generated/torch.nn.Dropout.html2026-09-12
- 06TRL 1.12.0 documentation — SFTConfig§ SFTConfig signature (weight_decay default) and the list of defaults that differ from TrainingArgumentshuggingface.co/docs/trl/v1.12.0/en/sft_trainer2026-09-12
- 07transformers documentation — TrainingArguments§ weight_decay (the page served v5.17.0 on the retrieval date)huggingface.co/docs/transformers/main_classes/trainer2026-09-12
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.