Data for Pretraining: FineWeb-Edu, Cosmopedia and Training a Tokeniser

By the end of this lesson you will be able to read a dataset card for the four things that decide whether you may use it; take a subset of a very large corpus that fits a token budget without downloading the rest; say why deduplication matters more to your run than to a frontier run; train a tokeniser and inspect the vocabulary it produced; and choose a tokens-per-parameter target with an argument behind it rather than a habit.

Two decisions were left open by the previous lesson: which corpus, and how much of it. This is the first of them.

Two published corpora, and what their cards actually say

The open pretraining corpora worth knowing at this scale come in two kinds, and the pair below is one of each.

FineWeb-Edu: filtered web text

The card describes it as consisting “of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.” The filtering is one classifier applied at one threshold, and the card is unusually specific about the consequence: “By setting a threshold of 3 (on a scale of 0 to 5) during the filtering process, we were able to also retain some high-level educational pages”, and then, of the result, “This removed 92% of the dataset, leaving us with 1.3T educational tokens.”

Ninety-two per cent removed by one classifier is the number to remember from Part 3’s corpus pipeline. It is also why the card can report that a lower threshold of 2 “preserved 5.4T tokens” and performed less well: the same crawl, two thresholds, two corpora, two different models.

The licence is the reason you can use it at all: the card states it is “released under the Open Data Commons Attribution License (ODC-By) v1.0”, and adds that “The use of this dataset is also subject to CommonCrawl’s Terms of Use.” Two documents, not one.

For a small run the important feature is the sample configurations. Alongside the full set the repository publishes sample-10BT, sample-100BT and sample-350BT, which are pre-cut subsets. You almost never want the default configuration on a home machine.

Cosmopedia: generated text

Cosmopedia is the other kind. Its card opens: “Cosmopedia is a dataset of synthetic textbooks, blogposts, stories, posts and WikiHow articles generated by Mixtral-8x7B-Instruct-v0.1. The dataset contains over 30 million files and 25 billion tokens”. Its licence field reads apache-2.0.

It is organised into eight splits by the source of the seed used to prompt the generator: web_samples_v1 and web_samples_v2, which “make up~75% of the dataset”, plus stanford, stories, wikihow, openstax, khanacademy and automathtext. The card reports its own duplicate rate as small: “the proportion of duplicates eliminated via MinHash was under 1%”, and describes decontaminating against benchmarks with “a 10-gram overlap to retrieve potentially contaminated samples”.

At twenty-five billion tokens it is a fiftieth the size of FineWeb-Edu, and every document is clean, on-topic prose in a consistent register. For a model of a few tens of millions of parameters trained for half an hour, that register is worth a great deal: there is no boilerplate to learn and no navigation menus to memorise. What it will not give you is the messiness of real text, so a model trained on it looks better than it is when you prompt it in the same register and worse when you do not.

∴Research note

The two corpora make different bets and the cards say so. FineWeb-Edu bets that a classifier can find the useful tenth of the web. Cosmopedia bets that a model can write cleaner training data than the web contains. Neither bet is settled, and both cards publish enough for you to check them, which is more than most corpora behind open-weight models do.

What nanochat actually downloads

Reading nanochat/dataset.py gives an answer that neither card above would have led you to. The base URL in the file points at karpathy/climbmix-400b-shuffle, the shard names are shard_00000.parquet upwards, and the constant says the last shard is number 6542. The speedrun script’s comments give the sizes: “each data shard is ~250M chars”, “each shard is ~100MB of text (compressed)”, and eight shards are “about ~800MB of data on disk”. It also says how many the reference run needs: “Approximately 150 shards are needed for GPT-2 capability pretraining, add 20 for padding.”

The file still contains the code for the previous corpus, and the message it prints is a small piece of history:

Output — what you should see

  nanochat recently switched from FinewebEdu-100B to ClimbMix-400B.
  Everyone who does `git pull` as of March 4, 2026 is expected to see this message.

So the project used a hundred-billion-token slice of FineWeb-Edu until March 2026 and now uses a repackaged shuffle of NVIDIA’s ClimbMix. The leaderboard entry for that change, reported by the README, is the row labelled “change dataset to NVIDIA ClimbMix” and dated 4 March 2026, which moved the time to the target from 2.76 hours to 2.02. Changing nothing but the corpus removed a quarter of the training time.

The licence question underneath it

Now look at the two cards. The repackaged repository, karpathy/climbmix-400b-shuffle, has a card whose entire content when read on 2026-09-09 was a licence field reading mit. The upstream dataset, nvidia/ClimbMix, describes itself as “a compact yet powerful 400-billion-token dataset designed for efficient pre-training”, explains that it was built by clustering data into a thousand groups, scoring them with classifiers and mixing the survivors, and states its licence as CC BY-NC 4.0, which excludes commercial use.

⚠Warning

A repackaging does not relicense its source. If you intend to do anything commercial with a model, the licence that binds you is the one on the data you actually trained on, traced back to its origin, and a one-line card is not a clearance. For the lab that follows this is not a problem, because the lab produces a model to read rather than to ship. For the project, where you choose your own corpus, it is the first question to answer, and the project’s default is public-domain text for exactly that reason.

Taking a subset by token budget

There are two ways to get a corpus small enough to train on, and you will use both.

The shard way is what nanochat does: download whole files until you have enough, then read them in order. It is crude and it works, and the arithmetic is easy. One shard is about 250 million characters. Divide by the bytes-per-token ratio your tokeniser achieves, which scripts/tok_eval.py prints, and you have tokens per shard. Divide your planned token count by that and round up, then add one for the validation shard the loader always holds out.

The streaming way is what you need for a corpus that has no convenient shards. The Datasets documentation is explicit about the point: streaming “lets you work with a dataset without downloading it. The data is streamed as you iterate over the dataset”, and it uses FineWeb itself as the example, noting the English split “is 45 terabytes, but you can use it instantly with streaming”.

Fragment — not complete on its own

from datasets import load_dataset

# Nothing is downloaded until the iterator is advanced.
ds = load_dataset("HuggingFaceFW/fineweb-edu", "sample-10BT", split="train", streaming=True)
ds = ds.shuffle(seed=0, buffer_size=10_000)

budget, taken, documents = 200_000_000, 0, []
for row in ds:
    documents.append(row["text"])
    taken += row["token_count"]          # the dataset's own count, in its own tokeniser
    if taken >= budget:
        break

Three things in that fragment are worth stating out loud.

shuffle on a streamed dataset is not a full shuffle. The documentation describes it as a buffer: with a buffer size of ten thousand it “will randomly select examples from the first ten thousand examples in the buffer. Selected examples in the buffer are replaced with new examples”, and it also shuffles the order of the shards. That is enough to break the ordering of the source files, which is what you need, and it is not enough to make far-apart documents adjacent.

take and skip exist for splitting a stream, and the documentation carries a warning that costs people an afternoon: “take and skip prevent future calls to shuffle because they lock in the order of the shards. You should shuffle your dataset before splitting it.”

And token_count is somebody else’s count. On FineWeb-Edu it is a column produced with the GPT-2 tokeniser. Your tokeniser will produce a different number for the same text, usually a smaller one if you trained it on this corpus. Use the column to hit a budget approximately, then check the real figure after tokenising.

Deduplication, at your scale

Part 3 made the case that deduplication decides what a model memorises. At your scale the argument sharpens, because the arithmetic changes. A document repeated a hundred times in a fifteen-trillion token corpus is a rounding error. The same document repeated a hundred times in a two-hundred-million token corpus is one part in twenty thousand of everything the model will ever see, and it will show up in your samples.

The corpora above are already deduplicated, and FineWeb-Edu’s card adds a caveat worth carrying: a deduplicated version exists in another repository, and “We find that the deduplication of this dataset doesn’t have any impact on model performance in our ablation setup (1.8B trained on 350B tokens).” That is a measured result at one scale with one setup, not a general rule, and it does not transfer to a corpus you assembled yourself out of files you downloaded.

For your own corpus, the two steps that pay for themselves in minutes are exact-duplicate removal by hashing each document, and boilerplate stripping. Boilerplate is the one people forget: if every one of your six hundred files begins with the same forty-line header, the model learns that header better than it learns anything else in the corpus. The project’s corpus script strips exactly such a header for exactly this reason.

Preparing your own corpus, in the order the steps must happen

1. 1Collect and record the licenceOne line per source: where it came from, what licence it carries, the date you checked. Written down now, not reconstructed later.
2. 2Strip boilerplateHeaders, footers, licence blocks, navigation. Anything identical across many documents is learned as if it were the point of the corpus.
3. 3DeduplicateHash each document and drop repeats. At a few hundred million tokens this is a few seconds of work and a visible difference in the samples.
4. 4Hold out a validation splitSet aside whole documents, never sentences from documents you also train on. A leak here makes every later number meaningless.
5. 5ShardWrite to the format the trainer reads, with the validation split as its own shard so that no code path can accidentally include it.
6. 6Train the tokeniserOn the training split only. A tokeniser trained on the validation text has already seen it.

The order is not negotiable. Deduplicating after splitting leaves near-duplicates spanning the split; training the tokeniser on everything leaks the validation set into the vocabulary.

Training a tokeniser, and reading it

Two routes, and the lab uses the first.

nanochat’s scripts/tok_train.py trains byte-pair merges with its own Rust implementation over an iterator of documents, capping each document at ten thousand characters so that no single file dominates, and defaulting to a vocabulary of 32,768. The whole invocation is python -m scripts.tok_train --max-chars=2000000000, and the run takes a minute or two rather than an hour.

The Tokenizers library is the general-purpose route, and its quicktour gives the shape:

Fragment — not complete on its own

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.train_from_iterator(documents, trainer)
tokenizer.save("tokenizer.json")

The documentation is careful about two things you would otherwise get wrong. The special tokens are “not used at all during training” but are inserted into the vocabulary so they exist later, and “The order in which you write the special tokens list matters”, because it fixes their ids. And the pre-tokeniser is not optional in spirit: without one “we might get tokens that overlap several words: for instance we could get an "it is" token”.

What to look at once it exists

Run the evaluation. scripts/tok_eval.py encodes the same six or seven pieces of text with the GPT-2 vocabulary, the GPT-4 vocabulary and yours, and prints a table of bytes, tokens and the bytes per token each achieves, on news, Korean, code, mathematics, science and samples from the corpus itself. Two readings matter. On text like your corpus your tokeniser should compress better than both baselines, because it was trained on it. On text unlike your corpus it will compress worse, often much worse, and the Korean row is where that shows most starkly. That table is the tokeniser lesson from earlier in this part, made concrete on your own machine in thirty seconds.

Then look at the vocabulary itself: print the longest entries, and look for the terms of your domain. A tokeniser trained on chemistry should have single tokens for common element symbols and functional groups. If it does not, either the corpus is too small or the vocabulary is.

How big a vocabulary

The trade is direct and, on a small model, dominant. The token embedding table has vocabulary size times width parameters, and the untied output projection has the same again. At the depth the lab trains, a width of a few hundred against a vocabulary of thirty-two thousand puts more parameters in those two tables than in all the transformer blocks combined, before the value-embedding tables in gpt.py are counted at all. The training script prints the breakdown by group at startup, and reading it once is the fastest cure for the intuition that a model is mostly its layers.

So: a larger vocabulary means shorter sequences, less compute per document and more parameters spent on lookup rather than computation. A smaller vocabulary means the reverse. nanochat’s choice of 32,768 with two-digit number merges is one tuned point on that curve for one corpus, and its own comment tells you it was found by trying three values.

Tokens per parameter

The Chinchilla paper’s conclusion is the anchor: for compute-optimal training “the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled”. nanochat’s own help text for --target-param-data-ratio names the resulting ratio as “Chinchilla=20”; its default is 12, and the reference speedrun sets 8 to reach a target sooner.

Then look at what publishers actually do. The Qwen3-1.7B-Base card reports “36 trillion tokens across 119 languages” for a model of 1.7 billion parameters. The SmolLM2-1.7B card reports 11 trillion tokens for the same size, on FineWeb-Edu, DCLM and The Stack, under an Apache-2.0 licence. Divide either by the parameter count and the ratio is in the thousands, not in the twenties.

That is not a mistake, and Part 3 explained why: compute-optimal minimises the trainer’s cost, and a model that will be run billions of times should be made small and trained far past that point so that everyone who uses it pays less forever. The rule and the practice answer different questions.

!Important

Your position is different from both. You are the trainer and the only user, and your binding constraint is an afternoon rather than a budget. You will sit below the compute-optimal ratio, not above it: a model that could usefully absorb far more tokens than you have time to feed it. That is the definition of undertrained, it is the correct choice given the constraint, and the next lesson is about seeing it in the loss curve rather than being surprised by it.

Freeze the evaluation vocabulary decision before comparing loss

Training a new tokeniser changes the symbols the model predicts. The same text may become a different number of tokens, so per-token perplexity from two tokenisers is not directly comparable. Keep a common tokeniser for a controlled model comparison, or use a clearly defined normalisation such as loss per byte on the same text when comparing different tokenisations.

Split documents before training the tokeniser and preparing shards. Preserve document identifiers and provenance so deduplication can operate across the intended boundaries. If a corpus contains multiple editions or mirrored pages, exact hashes alone will not detect all related text; inspect likely overlaps rather than treating a clean hash check as complete decontamination.

Sample the resulting token stream before training. Look for corrupted encoding, repeated boilerplate, unusually long fragments and missing document boundaries. Count accepted and rejected documents as well as tokens. A data pipeline that runs successfully can still transform a diverse corpus into mostly navigation text. The model optimises whatever survived preprocessing, not the corpus you originally intended to teach it.

Recap

FineWeb-Edu is filtered web text, 1.3 trillion tokens under ODC-By with CommonCrawl’s terms on top, and its own card records that the filter removed 92 per cent of what went in. Cosmopedia is generated text, 25 billion tokens under Apache-2.0, clean and consistent and unlike the real thing. nanochat now trains on a repackaged shuffle of NVIDIA’s ClimbMix whose upstream card states CC BY-NC 4.0 while the repackaging’s card states MIT, which is the licence-tracing problem in one example. Take a subset by shards when the corpus has them and by streaming when it does not, shuffling before splitting and treating anyone else’s token counts as approximate. Deduplicate and strip boilerplate, because at a few hundred million tokens a repeated document is a visible fraction of everything the model sees. Train the tokeniser on the training split only, evaluate its compression against two published vocabularies, and remember that on a small model the embedding tables outweigh the layers. Chinchilla says about twenty tokens per parameter for the lowest loss per unit of compute; publishers use thousands because they are optimising inference; you will use fewer than twenty because you are optimising an afternoon.

Check your understanding

Question 1. A repackaged dataset on the Hub has a card whose only content is "license: mit". The upstream dataset it was built from states CC BY-NC 4.0. You want to train a model you will sell access to. What is the position?

• The MIT card governs, because it is the copy you downloaded
• The upstream non-commercial term is the one to reckon with; a repackaging does not relicense its source, and a one-line card is not a clearance
• Neither matters, because model weights are not derivative works
• You need permission from Hugging Face

Show the answer and why

Answer: The upstream non-commercial term is the one to reckon with; a repackaging does not relicense its source, and a one-line card is not a clearance

Trace the data to its origin and read the licence there. This is exactly the situation between karpathy/climbmix-400b-shuffle and nvidia/ClimbMix as their cards read on 2026-09-09, and it is why the project in this part defaults to public-domain text.

Question 2. You stream a dataset, call take(50_000) to get a subset, then call shuffle(). What goes wrong?

• Nothing
• The shuffle is ignored: the documentation states that take and skip lock in the order of the shards and prevent future calls to shuffle, so you must shuffle first
• The take is applied twice
• Streaming does not support shuffle at all

Show the answer and why

Answer: The shuffle is ignored: the documentation states that take and skip lock in the order of the shards and prevent future calls to shuffle, so you must shuffle first

Shuffle first, split second. Taking first fixes the shard order, so your "subset" is the first fifty thousand documents of the first shards, which on a corpus ordered by crawl date is a systematically odd sample.

Question 3. Why does an identical forty-line header at the top of every document in your corpus matter more than it would in a fifteen-trillion-token corpus?

• It does not; models ignore repeated text
• Because at a few hundred million tokens it is a substantial fraction of everything the model will see, so it is learned strongly and appears in the samples
• Because it breaks the tokeniser
• Because it makes the files larger

Show the answer and why

Answer: Because at a few hundred million tokens it is a substantial fraction of everything the model will see, so it is learned strongly and appears in the samples

Repetition is proportional. The same absolute number of duplicate tokens is negligible in a frontier corpus and dominant in yours. Strip boilerplate before you deduplicate, because identical headers on otherwise different documents defeat a document-level hash.

Question 4. On a model of a few tens of millions of parameters with a vocabulary of 32,768, where are most of the parameters?

• In the attention layers
• In the MLP layers
• In the token embedding and the untied output projection, each of which is vocabulary size times width, before the value-embedding tables are counted
• Evenly spread across all of them

Show the answer and why

Answer: In the token embedding and the untied output projection, each of which is vocabulary size times width, before the value-embedding tables are counted

Two tables of 32,768 by the model width outweigh the transformer blocks when the width is only a few hundred. The training script prints the breakdown by parameter group at startup; read it once and the vocabulary-size trade stops being abstract.

Question 5. Your run will train on far fewer tokens per parameter than Chinchilla suggests. Is that a mistake?

• Yes: always train to the compute-optimal ratio
• No: the compute-optimal ratio answers "what is the lowest loss for this compute budget", and your binding constraint is wall-clock time on one machine, so a deliberately undertrained model is the right answer as long as you say so when you report the result
• Yes, unless you raise the learning rate
• It depends on the tokeniser

Show the answer and why

Answer: No: the compute-optimal ratio answers "what is the lowest loss for this compute budget", and your binding constraint is wall-clock time on one machine, so a deliberately undertrained model is the right answer as long as you say so when you report the result

Undertrained is a description, not an accusation, as long as it is stated. The failure is reporting a result from an undertrained model as if the size alone explained it. Publishers deliberately break the rule in the other direction, and for a reason that is about inference cost rather than about loss.

Check my answersClear