Pretraining: Learning from Trillions of Tokens

By the end of this lesson you will be able to read a corpus description and name the step that decided what the model memorised; read a pretraining loss curve as a power law and say what a value on it means; turn a parameter count and a token count into floating-point operations, GPU-hours, days and an electricity bill, and check a publisher’s figures against your own arithmetic; state the compute-optimal rule, run the formula behind it, and explain with numbers why publishers break it on purpose; and say exactly which parts of this an afternoon on your own machine can reproduce. That last point is the reason models come in the sizes they do, which is the reason your machine can run some of them.

What pretraining is

Part 2 gave the objective: given a sequence of tokens, assign a probability to the next one, and pay cross-entropy for the probability given to the token that actually followed. Pretraining is that objective applied to a corpus of trillions of tokens, for weeks, with nothing else in the loop: no labels, no questions and answers, no human ranking anything. A stretch of text is its own supervision, because the next token is always there in the data, which is why the corpus can be a scrape of the web rather than something people had to annotate.

What comes out is a base model: a set of weights that continues text. Ask it a question and you may get three more questions, because a question followed by more questions is a shape that occurs in its training data; the Part 2 lab’s base-vs-instruct.py showed you that on Qwen3-1.7B-Base (Apache-2.0). Everything a model appears to know is put in here. Post-training, the next lesson, changes how the weights behave; it does not add knowledge the pretraining left out, and it costs a small fraction of what this stage costs.

The corpus, and the work of cleaning it

Raw web text is mostly not worth training on: boilerplate, navigation menus, cookie banners, spam, machine translations of itself, and the same article on four hundred mirrors. The pipelines behind the best-known open-weight models are described in a paragraph or not at all, so the place to learn what cleaning means is a corpus that documents and ablates every step. FineWeb does. Its dataset card describes the release as “cleaned and deduplicated english web data from CommonCrawl”, 96 crawl snapshots, and the paper reports a 15-trillion-token result; the card now counts more than 18.5 trillion, with the GPT-2 tokeniser, on about 50 TB of disk. The licence is ODC-By 1.0 plus CommonCrawl’s Terms of Use.

From a web crawl to a training corpus: the FineWeb pipeline

1. 1CrawlCommon Crawl WARC files: raw HTML with headers. Books, code, papers and licensed collections are added by publishers, each with its own licence position.
2. 2URL filteringWhole sites dropped from a blocklist before any text is read.
3. 3Extract text, keep one languagetrafilatura pulls article text out of the HTML; Common Crawl's own extracted text kept too much menu and boilerplate. A fastText classifier keeps English with a score of at least 0.65.
4. 4Rule-based quality filtersMassiveText repetition and quality rules, the C4 rules minus one, and three FineWeb rules that together removed about 22 per cent of the tokens.
5. 5DeduplicationMinHash over 5-word shingles, 112 hash functions in 14 bands of 8, targeting documents at least 75 per cent similar; run per snapshot, not globally.
6. 6PII, then tokenise and shuffleEmail and public IP addresses anonymised. Text becomes token ids in the model tokeniser, packed into fixed-length sequences, shuffled into a training order.

Every stage throws data away, and the paper measures each stage by training a model on the result. Decontamination against evaluation sets is a separate step publishers describe, or do not, in the model card.

The settings the paper reports are worth reading as numbers, because a filter is a decision about what the model becomes and each of these was tested by training on the output:

Step	FineWeb’s setting, from the paper and the dataset card	What the paper reports
Text extraction	trafilatura on the raw WARC HTML instead of Common Crawl’s WET text	trafilatura text “clearly results in a more performant model”
Language	fastText classifier, keep English with score >= 0.65	not quantified in the sections read
Quality rules	MassiveText (Gopher) repetition and quality filters at their original thresholds; all C4 filters except terminal punctuation	the terminal-punctuation rule alone removed “around 30%” of tokens, so it was left out
Three FineWeb rules	fraction of lines ending in punctuation <= 0.12; share of characters in duplicated lines >= 0.1; fraction of lines shorter than 30 characters >= 0.67	removed 10.14%, 12.47% and 3.73% of tokens respectively, about 22% together
Deduplication	MinHash, 5-grams from an English word tokeniser, 112 hashes in 14 buckets of 8, targeting >= 75% similarity, per crawl snapshot	per-snapshot beat global: for the oldest crawl, the 10% a global pass kept “was actually of worse quality than the 90% of data that was removed”
PII	anonymise email and public IP addresses	not quantified
FineWeb-Edu	a classifier trained on Llama-3-70B-Instruct scores, 0 to 5, of 500k documents according to the dataset card (the paper says 460,000); keep >= 3	about 92% of FineWeb removed, leaving 1.3 trillion tokens; “dramatically better” on MMLU and ARC

Two things follow for a reader who will never build one of these. The corpus is a design decision, not a given: two models of the same size trained on differently filtered text behave differently, which is one reason cross-family benchmark comparisons are less informative than they look. And deduplication is not housekeeping: it decides what the model memorises.

Deduplication, one level down

An exact duplicate is easy: normalise the text, hash it, drop repeats of the hash. A near duplicate, the same article with a different footer, needs a similarity. The standard one is Jaccard similarity over shingles: cut each document into overlapping runs of five words, and divide the number of shingles two documents share by the number in their union. Comparing every pair is impossible at a few billion documents, so MinHash estimates the Jaccard without the comparison. Hash every shingle with 112 different hash functions and keep, per function, the minimum value in the document: 112 numbers per document. For any one function the minimum over the union of two documents lands in their intersection with probability exactly equal to their Jaccard, so two documents agree on a given number with that probability. Group the 112 numbers into 14 bands of 8; two documents become a candidate pair if all 8 numbers agree in any band. The chance of that is 1 − (1 − J^8)^14, and it turns a soft similarity into a sharp threshold:

Jaccard similarity J	Probability the pair is flagged, 1 − (1 − J^8)^14
0.50	0.05
0.75	0.77
0.90	0.9996

That is arithmetic from the parameters FineWeb reports, not a measurement, and it is why the paper can say its pass “targets” 75 per cent. The whole mechanism fits in a script:

RunnableAll tracks

near-duplicates.py

"""Exact and near-duplicate detection on five short documents: a content hash, then 5-gram Jaccard similarity."""
import hashlib
import re

docs = {
    "a": "Cookie policy: we use cookies to improve your experience. The village fair returns on Saturday with a produce show and a dog show.",
    "b": "Cookie policy: we use cookies to improve your experience. The village fair returns on Saturday with a produce show and a dog show.",
    "c": "Cookie policy: we use cookies to improve your experience. The village fair returns on Saturday with a produce show and a dog show. Parking is free.",
    "d": "Cookie policy: we use cookies to improve your experience. Council meeting minutes for March are now available to download.",
    "e": "The council has published its March meeting minutes, which residents may now download from the website.",
}

def words(text):
    return re.findall(r"[a-z0-9]+", text.lower())

def shingles(text, n=5):
    w = words(text)
    return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)}

print("exact duplicates share a hash of the normalised text:")
for k, v in docs.items():
    print(f"  {k}: {hashlib.sha256(' '.join(words(v)).encode()).hexdigest()[:16]}")

sets = {k: shingles(v) for k, v in docs.items()}
print("\nnear duplicates share most of their 5-word shingles (Jaccard = shared / union):")
keys = list(docs)
for i, x in enumerate(keys):
    for y in keys[i + 1:]:
        shared, union = len(sets[x] & sets[y]), len(sets[x] | sets[y])
        j = shared / union
        print(f"  {x}-{y}: {shared:3d} / {union:3d} = {j:.2f}{'  <- at least 75% similar, drop one' if j >= 0.75 else ''}")

lines = {}
for k, v in docs.items():
    for line in re.split(r"(?<=[.!?])\s+", v):
        lines.setdefault(line, set()).add(k)
print("\nlines that recur across documents are boilerplate, whatever the documents are about:")
for line, where in lines.items():
    if len(where) > 1:
        print(f"  in {len(where)} of {len(docs)}: {line!r}")

Output — what you should see

near-duplicates.py, Python 3.12, standard library only

exact duplicates share a hash of the normalised text:
  a: 6e635d76f70f7ed3
  b: 6e635d76f70f7ed3
  c: 4c940e3ffea13e21
  d: 3fc8302b694daf47
  e: 72d0b624bfc78556

near duplicates share most of their 5-word shingles (Jaccard = shared / union):
  a-b:  19 /  19 = 1.00  <- at least 75% similar, drop one
  a-c:  19 /  22 = 0.86  <- at least 75% similar, drop one
  a-d:   5 /  29 = 0.17
  a-e:   0 /  31 = 0.00
  b-c:  19 /  22 = 0.86  <- at least 75% similar, drop one
  b-d:   5 /  29 = 0.17
  b-e:   0 /  31 = 0.00
  c-d:   5 /  32 = 0.16
  c-e:   0 /  34 = 0.00
  d-e:   0 /  27 = 0.00

lines that recur across documents are boilerplate, whatever the documents are about:
  in 4 of 5: 'Cookie policy: we use cookies to improve your experience.'
  in 3 of 5: 'The village fair returns on Saturday with a produce show and a dog show.'

Read the three findings against the three levels publishers deduplicate at. a and b are byte-identical after normalisation: hash-level. c adds a sentence and still shares 86 per cent of its shingles: document-level, the MinHash job. The cookie line appears in four of five documents that are otherwise unrelated: line-level, which is why the Llama 3 paper describes removing “lines that appeared more than 6 times in each bucket of 30M documents” on top of URL-level and “global MinHash de-duplication across the entire dataset”. And d against e, the same facts in different words, scores zero: deduplication removes repeated text, not repeated information, and a paraphrase of a benchmark question sails through it.

Why it matters is measured rather than argued. Lee and colleagues found “a single 61 word English sentence that is repeated over 60,000 times” in C4, that models trained on deduplicated data “emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy”, and that train-test overlap “affects over 4% of the validation set of standard datasets”. The other direction has a limit too: Muennighoff and colleagues report that “training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data”, after which “the value of adding compute eventually decays to zero”. A corpus can be read four times; a sentence should not appear sixty thousand.

∴Research note

Ask of any corpus claim: what was crawled, what was thrown away, and was the evaluation data removed from it? A model whose training data overlaps a benchmark scores well on that benchmark without being better at anything, and the paraphrase result above is why “we removed exact matches” is not an answer. Part 4 returns to this as contamination and Part 16 teaches how to check for it in your own evaluations.

Who owns the text, and what is in it

A published corpus carries a licence of its own, which may be more restrictive than the model trained on it; a scrape of the open web carries as many licences as it has sources. Publishers respond in different ways. FineWeb names ODC-By and CommonCrawl’s terms. The Llama 3.1 model card (Llama 3.1 Community License, gated on the Hub) says the family “was pretrained on ~15 trillion tokens of data from publicly available sources” and stops there, and the paper reports the mix rather than the sources: “roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens”, with knowledge “until the end of 2023”. The Qwen3 report gives 36 trillion tokens across 119 languages and says part of the corpus was text recognised from PDF-like documents by Qwen2.5-VL and “trillions” of tokens synthesised by earlier Qwen models. A paragraph, a mix, or nothing at all: each is information about what you can check. When you train in Parts 11 to 15 the dataset’s licence is your problem and the course names it every time, and the licence lesson covers what a model licence does and does not say about its data.

The loss curve

Training runs the loop from Part 1 for a very long time: a batch of sequences goes in, the model predicts every next token in parallel, the cross-entropy says how much probability it failed to give the real next tokens, gradients flow back, the optimiser steps. The loss on the curve is that cross-entropy averaged over every token position in the batch, in nats; Part 2 computed it by hand for one sentence and matched the model’s own number to four decimals, and exp(loss) is the number of equally likely options the model was choosing among on average. A loss of 3.28, the GPT-2-class speedrun target on FineWeb’s validation split that Part 12 discusses, means about 27 options per token; a loss of 2.0 means about 7. The number depends on the tokeniser and on the held-out text, so a loss from one model and a loss from another are comparable only when both match.

The curve from a long run has one shape: a steep fall in the first fraction of a percent of the steps, then a long descent that is nearly straight on logarithmic axes. That straightness is the finding. Kaplan and colleagues measured loss falling “as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude”, and fitted three laws, each holding when the named quantity is the only constraint:

L(N)    = (N_c / N)^0.076        N_c = 8.8 × 10^13 non-embedding parameters, trained to convergence
L(D)    = (D_c / D)^0.095        D_c = 5.4 × 10^13 tokens, a large model with early stopping
L(Cmin) = (C_c / Cmin)^0.050     C_c = 3.1 × 10^8 PF-days; one PF-day = 8.64 × 10^19 FLOP

N counts the parameters “excluding all vocabulary and positional embeddings”, which is the count the FLOPs rule below also uses. Instantiating the first two laws, which is arithmetic on the paper’s constants and not a measurement of any model on this course:

Parameters N	L(N)	Tokens D	L(D)
10^8	2.830	10^10	2.262
10^9	2.376	10^11	1.818
10^10	1.994	10^12	1.461
10^11	1.674	10^13	1.174

Read the columns as ratios. Every tenfold increase in parameters multiplies the loss by the same factor, 10^−0.076 = 0.84; every tenfold increase in tokens multiplies it by 10^−0.095 = 0.80; every tenfold increase in optimally spent compute by 10^−0.050 = 0.89. A power law is generous at the start and brutal later in exactly this sense: the first decade of parameters in the table buys 0.45 nats, the next 0.38, the next 0.32, and each decade costs ten times the last. It is why frontier runs are enormous and their gains incremental, and why a small model trained sensibly gets most of the way: most of the loss reduction is bought cheaply.

Two things bend a real curve away from the law. The learning-rate schedule: the Llama 3 paper says that “during pre-training on the final 40M tokens, we linearly annealed the learning rate to 0”, and the drop that produces at the end of a curve is the schedule, not new knowledge, which is why two runs of different lengths cannot be compared at the same step number. And staged data: Qwen3’s report describes pretraining in three stages, “over 30 trillion tokens” of general text at a sequence length of 4,096, “about 5T higher-quality tokens” of reasoning-heavy data at the same length, then “hundreds of billions of tokens” at 32,768, so the curve of a modern run has a change of corpus in it. Part 12’s loss-curve section names the four shapes you will see on your own run and what each means.

✕Common mistake

Reading loss as quality. Loss is a number about token prediction on one held-out text with one tokeniser. A lower pretraining loss tends to come with better downstream behaviour, and the relationship is loose enough that no serious release ships loss as its headline and no two families’ losses can be compared at all. The symptom is a forum post comparing “loss 2.1” from one model card with “loss 1.9” from another; the fix is a task, a metric and a measurement, which is the reality check at the end of this part.

What it costs

The cost of a run is counted in floating-point operations, and the rule from Part 1 is six operations per parameter per training token. Kaplan’s Table 1 is where it comes from: a forward pass costs 2N + 2 × n_layer × n_ctx × d_attn operations per token, two per parameter for the multiply and the add plus an attention term that depends on the context length; the backward pass costs about twice the forward; and when the model width is much larger than a twelfth of the context, the attention term is dropped and what is left is C ≈ 6N per token, so:

training_FLOPs ≈ 6 × N × D           N = parameters in matrix multiplications, D = training tokens

Now hold a publisher’s figures against it. The Llama 3.1 model card reports “~15 trillion tokens”, a GPU-hours table, and the hardware: “H100-80GB (TDP of 700W)”. The Llama 3 paper says the 405B model was trained “on 15.6T text tokens”, “using 3.8×10²⁵ FLOPs”, on “up to 16K H100 GPUs”, and reports “an overall BF16 Model FLOPs Utilization (MFU) of 38-43%”. NVIDIA’s specification page lists the H100 SXM at 1,979 teraFLOPS for BF16 “with sparsity”, and the rate a dense training run can use is half of that. Every number in the table is one of those inputs or arithmetic on them:

Quantity	Llama 3.1 8B	Llama 3.1 405B	Where it comes from
Parameters N	8.0 × 10^9	405 × 10^9	model card
Training tokens D	15 × 10^12	15.6 × 10^12	card; paper for the 405B
Useful FLOPs, 6 × N × D	7.2 × 10^23	3.79 × 10^25	arithmetic; the paper states 3.8 × 10^25
GPU-hours	1.46 × 10^6	30.84 × 10^6	card, “Training Time (GPU hours)”
H100 dense BF16 peak	989 × 10^12 FLOP/s	989 × 10^12 FLOP/s	half of NVIDIA’s 1,979 teraFLOPS with sparsity
Peak FLOPs available, hours × 3600 × peak	5.2 × 10^24	1.10 × 10^26	arithmetic
Useful ÷ available	14%	35%	arithmetic; the paper reports 38 to 43% MFU for the 405B pretraining
Wall clock	not stated	80 days on 16,000 GPUs	arithmetic from the paper’s GPU count
Energy at TDP, hours × 0.7 kW	1.0 GWh	21.6 GWh	arithmetic; an upper bound, GPUs only

The 405B row closes the loop: six operations per parameter per token reproduces the paper’s own compute figure to two digits, and dividing by the card’s GPU-hours lands within a few points of the utilisation the paper measured; the card’s figure is cumulative training time, it does not say what besides the pretraining pass is in it, and that would account for the gap. The 8B row is the same arithmetic and lands lower; the card does not say why and this page will not guess. The script is the table:

RunnableAll tracks

training-cost.py

"""Turn a parameter count and a token count into FLOPs, then into GPU-hours, days and years, from stated inputs."""
H100_DENSE_BF16 = 1979e12 / 2          # NVIDIA lists 1,979 teraFLOPS BF16 "with sparsity"; dense is half of that
TDP_KW = 0.7                           # "H100-80GB (TDP of 700W)", Llama 3.1 model card

runs = [  # name, parameters, training tokens, GPU-hours from the Llama 3.1 model card, GPUs where the paper says
    ("Llama 3.1 8B", 8.0e9, 15.0e12, 1.46e6, None),
    ("Llama 3.1 405B", 405e9, 15.6e12, 30.84e6, 16_000),
]
for name, n, d, gpu_hours, gpus in runs:
    flops = 6 * n * d
    peak = gpu_hours * 3600 * H100_DENSE_BF16
    print(f"{name}: 6 x {n:.3g} x {d:.3g} = {flops:.2e} FLOP")
    print(f"  {gpu_hours:.3g} GPU-hours x 3600 s x {H100_DENSE_BF16:.3e} FLOP/s = {peak:.2e} FLOP at peak")
    wall = f"{gpu_hours / gpus / 24:.0f} days on {gpus:,} GPUs" if gpus else "GPU count not stated"
    print(f"  useful / peak = {flops / peak:.0%}   wall clock: {wall}"
          f"   energy at TDP = {gpu_hours * TDP_KW / 1e6:.1f} GWh (upper bound, GPUs only)")

print("\nthe same formula at home: replace the assumed rate with your Part 5 measurement")
for rate in (5e12, 25e12, 100e12):
    years = 6 * 8.0e9 * 15e12 / rate / (3600 * 24 * 365)
    tokens_4h = rate * 4 * 3600 / (6 * 50e6)
    print(f"  sustained {rate / 1e12:>5.0f} TFLOP/s: the 8B run takes {years:>7,.0f} years;"
          f" a 50M-parameter model sees {tokens_4h / 1e9:.1f}B tokens in 4 hours")

Output — what you should see

training-cost.py, Python 3.12, standard library only

Llama 3.1 8B: 6 x 8e+09 x 1.5e+13 = 7.20e+23 FLOP
  1.46e+06 GPU-hours x 3600 s x 9.895e+14 FLOP/s = 5.20e+24 FLOP at peak
  useful / peak = 14%   wall clock: GPU count not stated   energy at TDP = 1.0 GWh (upper bound, GPUs only)
Llama 3.1 405B: 6 x 4.05e+11 x 1.56e+13 = 3.79e+25 FLOP
  3.08e+07 GPU-hours x 3600 s x 9.895e+14 FLOP/s = 1.10e+26 FLOP at peak
  useful / peak = 35%   wall clock: 80 days on 16,000 GPUs   energy at TDP = 21.6 GWh (upper bound, GPUs only)

the same formula at home: replace the assumed rate with your Part 5 measurement
  sustained     5 TFLOP/s: the 8B run takes   4,566 years; a 50M-parameter model sees 0.2B tokens in 4 hours
  sustained    25 TFLOP/s: the 8B run takes     913 years; a 50M-parameter model sees 1.2B tokens in 4 hours
  sustained   100 TFLOP/s: the 8B run takes     228 years; a 50M-parameter model sees 4.8B tokens in 4 hours

The second half of the output is the formula run backwards, D ≈ C ÷ (6 × N), with C equal to a sustained rate times a wall clock. The three rates are assumptions, not measurements of any track; the Part 5 lab has you measure your own machine’s BF16 matrix-multiply rate, and Part 12’s training loop prints the fraction of it that it achieves as bf16_mfu, which is where Part 12’s budget comes from. The shape of the answer does not depend on which rate you assume.

Scaling laws and the compute-optimal split

Given a fixed compute budget, C ≈ 6ND says you can spend it on a bigger model or on more tokens, and the split is not obvious. Kaplan and colleagues concluded in 2020 that “larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence”, and the industry built very large, comparatively undertrained models on that advice: the Chinchilla paper’s Table 1 lists GPT-3 at 175 billion parameters, Jurassic-1 at 178 billion and Gopher at 280 billion, each trained on about 300 billion tokens, roughly one token per parameter.

In 2022 that paper revisited the question with “over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens” and reached a different answer: “for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled”. Its demonstration was Chinchilla, 70 billion parameters “on 1.4 trillion tokens”, the same compute as Gopher spent on a model four times smaller and four times more data; the ratio, 1.4 × 10^12 ÷ 70 × 10^9, is the twenty tokens per parameter the rule is remembered by, and the paper’s verdict on the models of the day is that they “are significantly undertrained”.

The formula behind the rule

The paper’s third method fits every run’s final loss to a two-term power law, its equation 10:

L(N, D) = E + A / N^alpha + B / D^beta        E = 1.69, A = 406.4, alpha = 0.34, B = 410.7, beta = 0.28

E is the loss an ideal model of the text would still pay, the entropy of the data; the second term is what a model of N parameters loses for being finite; the third is what training on D tokens loses for stopping. Minimise L subject to C = 6ND and the optimum has a power-law form, N_opt ∝ C^a and D_opt ∝ C^b; the paper’s Table 2 puts a and b at 0.50 and 0.50, 0.49 and 0.51, and 0.46 and 0.54 for its three methods, against Kaplan’s 0.73 and 0.27, and its Table 3 turns the first method into a lookup: a 1-billion-parameter model is compute-optimal at 20.2 billion tokens and 1.21 × 10^20 FLOPs, a 10-billion one at 205.1 billion tokens and 1.23 × 10^22, a 67-billion one at 1.5 trillion and 5.76 × 10^23. Twenty tokens per parameter, at every size. Since compute grows as N × D and both grow together, doubling the budget multiplies the optimal size by about 1.4, not 2: the model grows as the square root of the compute. You can run the minimisation yourself, and then hold a real publisher’s choice against it:

RunnableAll tracks

chinchilla-fit.py

"""The Chinchilla parametric loss, minimised under a compute budget, then held against what a publisher chose."""
E, A, ALPHA, B, BETA = 1.69, 406.4, 0.34, 410.7, 0.28     # Hoffmann et al. 2022, equation 10

def loss(n, d):
    return E + A / n ** ALPHA + B / d ** BETA

def compute_optimal(c, steps=4000):
    """Sweep N over a log grid, let D follow from C = 6ND, and keep the lowest predicted loss."""
    best = None
    for i in range(steps):
        n = 10 ** (6 + 9 * i / steps)                    # 10^6 to 10^15 parameters
        d = c / (6 * n)
        l = loss(n, d)
        if best is None or l < best[2]:
            best = (n, d, l)
    return best

print(f"{'budget C':>9} {'N_opt':>9} {'D_opt':>9} {'D/N':>5} {'loss':>6} {'gap to E':>9}")
for exp in range(19, 26):
    n, d, l = compute_optimal(10.0 ** exp)
    print(f"{10.0 ** exp:9.0e} {n:9.2e} {d:9.2e} {d / n:5.1f} {l:6.3f} {l - E:9.3f}")

n, d = 8.0e9, 15e12                                       # a publisher's choice: 8B on 15T
c = 6 * n * d
n_opt, d_opt, l_opt = compute_optimal(c)
target = loss(n, d)
c_match = c
while compute_optimal(c_match)[2] < target:               # shrink the budget until the optimum can no longer match
    c_match /= 1.02
print(f"\n8B on 15T tokens: C = {c:.1e}, predicted loss {target:.3f}")
print(f"compute-optimal at the same C: N = {n_opt:.2e}, D = {d_opt:.2e}, loss {l_opt:.3f}")
print(f"a compute-optimal run reaches loss {target:.3f} with C = {c_match:.1e}"
      f" ({c / c_match:.1f}x less compute) as a {compute_optimal(c_match)[0]:.2e}-parameter model")

Output — what you should see

chinchilla-fit.py, Python 3.12, standard library only

 budget C     N_opt     D_opt   D/N   loss  gap to E
    1e+19  2.28e+08  7.31e+09  32.1  2.986     1.296
    1e+20  6.46e+08  2.58e+10  39.9  2.600     0.910
    1e+21  1.82e+09  9.15e+10  50.3  2.329     0.639
    1e+22  5.16e+09  3.23e+11  62.6  2.139     0.449
    1e+23  1.46e+10  1.14e+12  78.0  2.005     0.315
    1e+24  4.14e+10  4.03e+12  97.2  1.911     0.221
    1e+25  1.17e+11  1.43e+13 122.4  1.845     0.155

8B on 15T tokens: C = 7.2e+23, predicted loss 1.949
compute-optimal at the same C: N = 3.56e+10, D = 3.37e+12, loss 1.923
a compute-optimal run reaches loss 1.949 with C = 3.6e+23 (2.0x less compute) as a 2.60e+10-parameter model

Three things to read off. Each row is ten times the compute of the row above, and the gap to E shrinks by a smaller amount every time: the power law again, now in the paper’s own fit. The D/N column drifts upward with the budget rather than sitting at twenty, because this method’s exponents are 0.46 and 0.54 rather than equal, which is the paper’s own caveat that its third method ends up “predicting a lower N_opt than the two other approaches”; the three agree on the direction and differ in the detail, and the fit is being extrapolated far beyond the runs that produced it, so use it for the shape and not as a prediction for any real 15-trillion-token run. And the last three lines are the argument of the next section in numbers: the loss the fit predicts for an 8-billion model on 15 trillion tokens could have been reached with half the compute by a 26-billion model trained compute-optimally.

Who pays

Compute-optimal is optimal for the trainer, who pays once. The user pays for every token generated, and that cost is set by the model’s size for as long as the model is run. The Chinchilla paper says so itself: a smaller model’s “reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware”. So publishers train past the compute-optimal point, and by how much is a number you can read off any card that states its token count:

Model	Parameters	Training tokens	Tokens per parameter	6 × N × D
Gopher, 2021	280 × 10^9	300 × 10^9	1.1	5.0 × 10^23
Chinchilla, 2022	70 × 10^9	1.4 × 10^12	20	5.9 × 10^23
Llama 3 405B, the paper’s own law	402 × 10^9	16.55 × 10^12	41	4.0 × 10^25
Llama 3.1 405B, as trained	405 × 10^9	15.6 × 10^12	39	3.8 × 10^25
Llama 3.1 8B	8 × 10^9	15 × 10^12	1,875	7.2 × 10^23
Qwen3-8B (Apache-2.0)	8.2 × 10^9	36 × 10^12	4,400	1.8 × 10^24
Qwen3-1.7B-Base (1.4 × 10^9 non-embedding)	1.7 × 10^9	36 × 10^12	21,000	3.7 × 10^23

The frontier model sits near its own optimum: the Llama 3 paper says extrapolating its fitted law “suggests training a 402B parameter model on 16.55T tokens”, and that is what was built. The small models sit a hundred to a thousand times past theirs, and the reason is the last three lines of the script: the 8-billion model cost twice the compute a compute-optimal run would have spent on the same loss, and in exchange it reads a third of the bytes per generated token, on every machine, forever. That trade, paid once by the publisher and collected by everyone who runs the model, is why a 4-billion or 8-billion model in 2026 is as capable as it is and why this course exists. The Chinchilla rule answers “given this compute budget, what is the lowest loss I can reach?”; the rule publishers follow answers “given that this model will be run a billion times, what size should it be?”.

Why frontier pretraining is not a home activity

It is not a software problem or a permissions problem. Every line is a stated figure or arithmetic on one:

Constraint	The 405B run, from the paper and the card	One machine at home, arithmetic from the same inputs
Compute	3.8 × 10^25 FLOP; 80 days on 16,000 H100s	228 years for the 8B run at an assumed 100 × 10^12 FLOP/s sustained; the 405B run is 53 times longer
Data	15.6 × 10^12 tokens; FineWeb’s card stores 18.5 × 10^12 tokens in about 50 TB, 2.7 bytes per token	about 40 TB of cleaned text to hold, and a deduplication pass over it that does not fit in memory
Reliability	“466 job interruptions” in “a 54-day snapshot”, 419 of them unexpected, about 78% “attributed to confirmed hardware issues”, GPU issues 58.7% of all unexpected ones	419 failures over 54 × 16,000 GPU-days is one every 2,060 GPU-days, about 5.6 years; the 228-year run would restart from a checkpoint about forty times
Energy	up to 21.6 GWh at the card’s 700 W TDP, GPUs only	the same joules through one wall socket, over the same centuries
What you get	a base model that still needs the next lesson’s post-training	the same

The reliability row is the one people leave out. A run of weeks across thousands of accelerators is mostly the engineering of keeping it alive, and that engineering has no small version, which Part 12 is explicit about.

Why the small version is worth doing

Take the numbers in the other direction. A 50-million-parameter model is compute-optimal at about a billion tokens, twenty per parameter, and the cost table put a billion tokens for such a model inside four hours at the middle of the three assumed rates; you will measure yours before you commit to a run. That afternoon is the most demystifying one in the course, because every abstraction in this lesson becomes something you watched:

What this lesson stated	What you observe in Part 12	Where
The corpus is built, and a tokeniser is trained on it	a vocabulary you can print, and the compression it achieves on your text	Data for pretraining
The loss falls as a power law; the schedule bends it	a curve whose steep fall, straight descent and warm-down drop you can point at	Scaling laws at home
Compute is 6 × N × D; a budget is a rate times a wall clock	the script prints its FLOPs per token and its horizon; your notebook holds the prediction and the measured tokens per second	the lab
Undertrained models exist and look a particular way	samples at three checkpoints: token soup, grammatical nonsense, text that stays on topic; a curve still falling when you stopped	what pretraining teaches
Deduplication decides what is memorised	your own corpus, with and without the header every file shares	the project

It also settles a decision you will face before then: whether a problem needs pretraining at all.

You need	The right operation	The measurable reason
A model whose vocabulary and text are unlike anything published: a notation, a protocol, a language the tokenisers split badly	pretrain from scratch, small	the tokeniser is fixed at pretraining; Part 2 showed the token count of a text depends on the tokeniser, and no later stage changes it
General ability plus knowledge of a domain that has a corpus of hundreds of millions of tokens or more	continue pretraining a published base model on that corpus	knowledge needs many tokens and many parameters; the Part 12 project measures model A against model B on held-out documents
A behaviour, format or style, from thousands of examples	fine-tune, from Part 13 onwards	thousands of examples adjust what the weights do with knowledge they already hold; they do not add knowledge in any quantity you could measure

Separate a corpus improvement from a compute increase

Suppose a cleaned corpus produces lower validation loss than the original corpus. Before attributing the gain to cleaning, check how many training tokens each run consumed. Removing duplicates can change epoch length, so training both for the same number of epochs may spend different amounts of compute. Conversely, training both for the same number of steps with different packed lengths can also change the token budget.

Define the comparison using processed tokens, sequence length, optimiser updates and an unchanged validation distribution. Keep document boundaries and split membership fixed before filtering. Record rejected-document counts and reasons: a filter that removes every difficult or minority-language example can improve the remaining corpus’s average while narrowing the model’s coverage.

A useful small experiment changes one corpus rule, trains from the same initialisation for the same token budget and evaluates both broad and domain-specific held-out text. The result can support a claim about that data intervention at that scale. It cannot establish that the same rule improves every larger model, or that lower language-model loss by itself improves an application’s exact-answer score.

Recap

Pretraining is next-token prediction on trillions of tokens and it is where a model’s knowledge comes from; what comes out is a base model that continues text. The corpus is a pipeline of decisions, each of which the FineWeb paper measured by training on the result, and deduplication at the hash, document and line level decides what the model memorises while leaving paraphrases untouched. The loss is mean cross-entropy per token, it falls as a power law with a fixed factor per decade of parameters, tokens or compute, and the schedule bends the end of the curve. Compute is six operations per parameter per token, a rule that reproduces a published frontier figure to two digits and puts the same run at centuries on one machine. Chinchilla’s fit says to grow parameters and tokens together, about twenty tokens per parameter, and publishers overshoot it on purpose, in the worked example twice the optimal compute for a model a third the size, because the user pays per token for as long as the model runs. The afternoon-sized version of all of this is Part 12.

Check your understanding

Question 1. A base model is given "What is the capital of Portugal?" and replies with three more questions about European capitals. What has gone wrong?

• The corpus was not deduplicated
• Nothing: a base model continues text, and a list of questions is a plausible continuation. Answering is a post-training behaviour.
• The tokeniser is broken
• The model was undertrained

Show the answer and why

Answer: Nothing: a base model continues text, and a list of questions is a plausible continuation. Answering is a post-training behaviour.

Pretraining optimises next-token prediction on a corpus in which questions are often followed by questions. The Part 2 lab showed the same thing on Qwen3-1.7B-Base; question-answering behaviour comes from supervised fine-tuning on demonstrations, the next lesson.

Question 2. A card reports 2.0 million H100 GPU-hours for a 12-billion-parameter model trained on 20 trillion tokens. Using the course rule of thumb and a dense BF16 peak of 989 × 10^12 FLOP/s, roughly what fraction of peak did the run achieve?

• About 20%: 6 × 12 × 10^9 × 20 × 10^12 = 1.44 × 10^24 useful FLOP, against 2.0 × 10^6 × 3600 × 989 × 10^12 = 7.1 × 10^24 available
• About 2%
• About 100%; the rule counts every operation the GPU performs
• It cannot be estimated without the batch size

Show the answer and why

Answer: About 20%: 6 × 12 × 10^9 × 20 × 10^12 = 1.44 × 10^24 useful FLOP, against 2.0 × 10^6 × 3600 × 989 × 10^12 = 7.1 × 10^24 available

Useful work is 6 × N × D; available work is GPU-hours × 3600 × peak. The ratio, about 0.2, is in the range real runs report, and the Llama 3 paper reports 38 to 43 per cent for its 405B pretraining. A ratio above 1 means an input is wrong, most often a peak figure quoted with sparsity.

Question 3. Your compute budget for a from-scratch run goes up by a factor of four. Under the Chinchilla rule, what happens to the compute-optimal model size and token count?

• Both quadruple
• The model size quadruples; the tokens stay the same
• Both double: compute is 6 × N × D and the two grow together, so each scales as the square root of the budget
• The tokens quadruple; the model size stays the same

Show the answer and why

Answer: Both double: compute is 6 × N × D and the two grow together, so each scales as the square root of the budget

With D = 20 × N, compute is 120 × N², so N grows as the square root of C. Four times the budget is twice the parameters and twice the tokens; ten times the budget is about 3.2 times each. This is why frontier gains per generation are incremental.

Question 4. Which of these corpus-preparation orders contains the bug?

• Strip boilerplate, deduplicate, hold out whole documents for validation, train the tokeniser on the training split
• Strip boilerplate, deduplicate, train the tokeniser on the whole corpus, then hold out whole documents for validation
• Strip boilerplate, deduplicate, hold out whole documents for validation, train the tokeniser on the training split, write the validation split as its own shard
• The first and the third are the same order and both are correct

Show the answer and why

Answer: Strip boilerplate, deduplicate, train the tokeniser on the whole corpus, then hold out whole documents for validation

A tokeniser trained on text that later becomes the validation split has already seen it, so every later held-out number is optimistic. The other two orders differ only in the sharding step. Deduplication also has to precede the split, or near-duplicates straddle it; the Part 12 data lesson gives the full order.

Question 5. Two runs share a corpus and tokeniser. Run A trained for 10,000 steps; run B trained for 20,000. At step 10,000, run A shows a lower loss than run B did at the same step. What is the most likely explanation?

• Run B has a bug
• Run A had finished its learning-rate warm-down at step 10,000 and run B had not; the drop is the schedule, not more learning
• Run A saw more data
• The losses cannot be lower at the same step

Show the answer and why

Answer: Run A had finished its learning-rate warm-down at step 10,000 and run B had not; the drop is the schedule, not more learning

The end of a run anneals the learning rate, and the Llama 3 paper describes the final tokens of its pretraining being trained with the rate annealed linearly to zero. The drop that produces belongs to the schedule, which is why runs of different lengths are compared at the end, not at a matching step.

Question 6. A model card for a 1.7-billion-parameter model reports 36 trillion training tokens, about 21,000 per parameter. What does that tell you?

• The card has a typo; the compute-optimal figure is about twenty per parameter
• The publisher spent far more than compute-optimal training on a small model so that it is cheaper to run, which is the trade this course depends on
• The model is overfitted
• The model is undertrained

Show the answer and why

Answer: The publisher spent far more than compute-optimal training on a small model so that it is cheaper to run, which is the trade this course depends on

Compute-optimal minimises the trainer's cost for a given loss. A model that will be served billions of times is better made small and trained far past that point; the chinchilla-fit.py output shows the shape of that trade, twice the compute for a model a third the size at the same predicted loss.

Check my answersClear