What Pretraining Teaches You That Fine-Tuning Cannot

By the end of this lesson you will be able to describe a base model as a concrete artefact rather than a stage in a diagram; name the three decisions that pretraining fixes permanently and explain why no later stage can revise them; walk the pipeline you are about to run, from a directory of text to a file you can sample from; and say what an afternoon of your own compute honestly buys, in the publisher’s own reported figures rather than in hope.

Part 3 told you what pretraining is and why frontier-scale pretraining is out of reach. This lesson is about why the small version is worth doing anyway, and it is a more specific answer than “it is educational”.

A base model, as a thing on disk

Strip away the vocabulary and a base model is three files. A tokeniser, which is a table mapping byte sequences to integers plus the rules for splitting text before the table is consulted. A configuration, which is a handful of integers: how many layers, how wide, how many attention heads, how long a sequence the position encoding was built for. And the weights, which are the numbers the training loop moved.

That is all. There is no chat template, because nobody trained one in. There is no idea of a user or an assistant, because those are just tokens that never appeared in a special role. There is no refusal behaviour, no system prompt, no notion that a conversation has turns. Give it a prefix of tokens and it produces a probability distribution over the next one, and that is the entire interface.

You have already seen this. Part 2’s lab had you prompt Qwen3-1.7B-Base, the pretrained checkpoint behind Qwen3-1.7B, and watch it continue rather than answer. Both are Apache-2.0 licensed and both are in the course model reference. What that lab could not show you is why the base model behaves that way, because the behaviour is not a property of a file you can open. It is a property of a corpus you have never seen and a loop you have never run. Running the loop yourself, on a corpus you chose, is what converts “base models continue text” from a fact you memorised into one you can predict from.

Three things no later stage can put back

Post-training adjusts a model that already exists. Some of what pretraining decided is genuinely out of its reach, and knowing which parts changes how you read every fine-tuning result in Parts 13 to 15.

The tokeniser

The vocabulary is trained before the model and frozen the moment the embedding table is initialised, because every weight in that table corresponds to one entry in it. You cannot add tokens later without adding rows to the embedding and the output projection, and those rows start untrained.

This is not a technicality. If your domain tokenises badly, every sentence in it costs more tokens, which means more memory in the KV cache, less of it fitting in the context window, and more compute per document for the rest of the model’s life. A tokeniser trained mostly on English web text will split a chemical formula, a Korean sentence or a Python identifier into far more pieces than one trained with those in the corpus. The lab has you measure exactly this with the tokeniser you train yourself, against the GPT-2 and GPT-4 vocabularies, on the same text.

The knowledge

Part 3 put it as “post-training changes behaviour; it does not add knowledge the pretraining did not put in”. The arithmetic behind that sentence is worth doing once. The Qwen3-1.7B-Base card states that the model was trained on “36 trillion tokens across 119 languages”. A generous supervised fine-tune sees tens of millions of tokens. The fine-tune is therefore working with something on the order of a millionth of the evidence, at a lower learning rate, usually on a fraction of the parameters. It reshapes the distribution the model already has. It is not a competitive way to install a body of facts.

The shape

Depth, width, head count and the sequence length the position encoding was built for are chosen before the first step and fixed by the shapes of the tensors. A fine-tune can extend a context window somewhat, and Part 17 covers how, but it cannot make a shallow model deep. When you read “this 1.7B model was fine-tuned to match a 7B model on our task”, the claim is about one task, and the ceiling the smaller model is pressed against was set at initialisation.

!Important

What fine-tuning is good at is format, style, refusal behaviour, tool-call syntax and the selection among things the model can already produce. That is a large and useful category, and Parts 13 and 14 spend two hundred minutes on it. The point here is that the category has edges, and the edges were drawn during pretraining.

The pipeline you are about to run

Every stage below is a script you will invoke by hand in the lab, in this order, and the artefact each one leaves behind is a file you can inspect.

From a directory of text to a model you can sample from

1. 1CorpusDocuments with a licence you have checked, on disk as shards. You choose how many, and that choice is your token budget.
2. 2Tokeniser trainingByte-pair merges learned from a sample of the corpus, saved as a vocabulary. Everything downstream depends on this file and it is frozen from here on.
3. 3Tokenise and packDocuments become token ids and are packed into fixed-length rows for the model. Where documents are cropped to fit, tokens are thrown away.
4. 4InitialiseWeights created from the configuration: depth, width, heads, sequence length. Nothing has been learned yet and the loss is what guessing looks like.
5. 5TrainForward, loss, backward, step, repeated. The learning rate, the momentum and the weight decay all follow schedules that depend on the total number of steps, so the horizon is chosen before the run starts.
6. 6EvaluateValidation loss on held-out text at intervals, plus a small benchmark. The number that matters is on data the model has not been trained on.
7. 7SamplePrompts in, continuations out, read critically. This is where an undertrained model announces itself long before a metric does.
8. 8Post-trainOptional and separate: supervised fine-tuning turns a text continuer into something that answers. Part 13 does this properly.

The lab runs the first seven stages. Each leaves a file: a vocabulary, a set of shards, a checkpoint, a log. When a later result looks strange, the file from an earlier stage is usually where the explanation is.

The two decisions that shape everything are made at the top: what corpus, and what token budget. The next two lessons are those decisions.

What an afternoon actually buys

Here is where the subject is usually oversold, so take the numbers from the people who published them. nanochat, the codebase this part reads and runs, is nanochat main · verified 2026-09-08. Its README states that you can train “your own GPT-2 capability LLM (which cost ~$43,000 to train in 2019) for only $48 (~2 hours of 8XH100 GPU node)”, and that on a spot instance “the total cost can be closer to ~$15”. Its leaderboard, reported by the README and dated March 2026, has the reference run at 1.65 hours. Those are figures reported by the project for a rented eight-GPU node, retrieved on 2026-09-09; they are not this course’s measurements and they are not what your machine will do.

The same README is equally clear about the other end. All the code “will run just fine on even a single GPU by omitting torchrun, and will produce ~identical results”, but “you’ll have to wait 8 times longer”. And for machines without a CUDA GPU there is a script whose header says it “dramatically shrinks the LLM that is being trained to make things fit into a reasonable time interval of a few ten minutes of training. You will not get strong results in this way.”

That last sentence is the honest frame for this whole part, and it comes from the author of the code rather than from us. The model you train in the lab is a model to read, not a model to use. Nothing you produce here will be worth serving. What you produce is a loss curve you watched, a vocabulary you can print, a checkpoint whose size you can predict from its parameter count, and a set of samples that get less bad in a way you observed rather than were told about.

?Reality check

The claim that people repeat is “you can train your own ChatGPT at home for a few dollars”. The useful version, which the project’s own README supports, is narrower: for a few tens of dollars of rented eight-GPU time you can reach the capability of a 2019 model, and on one machine at home you can run the identical pipeline at a size that finishes in an afternoon and produces something much weaker. Both halves are true. Only the first half travels.

What the small version does not teach

Being clear about the limits is part of taking the exercise seriously, and three things are simply absent at this scale.

The engineering of a long run. A frontier run lasts weeks across thousands of accelerators, and most of the work is keeping it alive: restarting from a checkpoint after a failed node, detecting a loss spike before it destroys a day of progress, and moving data fast enough to keep the accelerators fed. Your run finishes before any of that can go wrong. Part 18 covers the parallelism those runs depend on, and Part 23 covers operating a service that must not fall over, but neither is the same problem as a training job that has to survive a fortnight.

Data engineering at scale. You will download shards somebody else prepared, or prepare a corpus of a few million words by hand. Building a corpus of trillions of tokens is a distributed-systems problem with a deduplication step that cannot fit in memory, and reading about it is not the same as having done it.

Behaviour that only appears when a model is large. Capabilities that emerge with scale cannot be observed in a model of tens of millions of parameters, so nothing you see here tells you what a larger model would do. The arithmetic transfers; the behaviour does not.

What is left after those three subtractions is still the whole mechanism, and the mechanism is the part people get wrong when they only read about it.

How the course scales the exercise

The four tracks do the same thing at different sizes, and the variable is the one dial the codebase exposes: the depth of the transformer, from which the width, the head count and the training horizon are all derived. The lab does not hand you a wall-clock figure per track, because that would be a number we have not measured on your machine. It does something better: it runs a short calibration, reads the throughput the training loop prints, and turns your chosen time budget into an iteration count with arithmetic you can check.

Tracks S and N run the CUDA path with the code as published. Track X runs the same PyTorch code against a ROCm build, with a CPU fallback stated as slow rather than hidden. Track M runs it on PyTorch’s MPS backend, which the README lists as a supported target while warning that the author “hasn’t personally exercised all of these code paths so there might be sharp edges”. The lab says exactly what to check on each.

What you should expect to see

Sampling at intervals during training is the most informative thing in the run, and it goes through three recognisable stages.

Early, the samples are token soup: fragments of words, punctuation in the wrong places, no agreement between one clause and the next. This is a model that has learned the marginal frequency of tokens and little else.

In the middle, the grammar arrives before the content. Sentences parse. Subjects agree with verbs. The claims inside them are nonsense, and confidently so. This stage is the single best argument in the course against reading fluency as evidence of anything.

At the end of a short run, on-topic nonsense: prompted with “The capital of France is”, a model of this size may well produce a plausible-looking sentence about France with the wrong city in it. Getting the city right is a fact from the corpus, and whether the run saw that fact often enough is a question about tokens, not about architecture.

Test continuation skill separately from instruction following

A small pretrained model can improve at continuing held-out text while remaining poor at answering chat requests. Those are different input distributions and objectives. Evaluate continuation with held-out loss and qualitative samples; evaluate instruction following only with a defined prompt format and task rubric.

Use a fixed set of prompts containing both in-domain and out-of-domain prefixes. Save samples at several checkpoints with the same generation settings. Inspect syntax, repetition, local coherence and memorised passages separately. A readable paragraph can be an encouraging qualitative observation without establishing reliable factual recall or general reasoning.

When the model repeats a training passage, check exact overlap before calling it learning of the underlying concept. Conversely, unfamiliar phrasing is not proof that a sample is correct. The educational success of a small pretraining run is understanding how data, tokenisation, optimisation and compute produce changes in predictions. It does not require pretending that an afternoon model is a capable assistant. Post-training and application evidence remain separate stages.

Recap

A base model is a tokeniser, a configuration and a set of weights, with no notion of conversation in it. Pretraining fixes three things that no later stage can revise: the vocabulary, the body of knowledge, and the shape of the network. Fine-tuning is powerful within those limits and helpless against them. The pipeline is corpus, tokeniser, packing, initialisation, training, evaluation, sampling, and every stage leaves an inspectable file. The published figures for a full small-scale run are for a rented multi-GPU node and are quoted here as the project reports them; the run on your own machine is much smaller and its product is understanding rather than a usable model. The next two lessons make the two decisions that everything else follows from: which corpus, and how many tokens.

Check your understanding

Question 1. You have a base model that tokenises your domain badly - technical terms split into five or six pieces each. Which of these fixes it?

• A supervised fine-tune on domain text
• A LoRA adapter trained on the same text
• Nothing after pretraining fixes it properly: the vocabulary was frozen when the embedding table was created, and new tokens would start untrained
• Raising the context length at inference time

Show the answer and why

Answer: Nothing after pretraining fixes it properly: the vocabulary was frozen when the embedding table was created, and new tokens would start untrained

Every row of the embedding table corresponds to one vocabulary entry. Adding tokens later means adding untrained rows to both the embedding and the output projection. Fine-tuning changes the values in the table, not which entries exist. This is why the tokeniser decision comes first and why the lab has you measure your own tokeniser against two published ones.

Question 2. A vendor says their 1.7B model was fine-tuned to match a 7B model on their benchmark. What does the Part 12 framing tell you to ask?

• Nothing: parameter counts are not comparable across families
• Which single task the claim covers, and whether the comparison holds off that task, since fine-tuning selects among behaviours the smaller model can already produce rather than raising its ceiling
• Whether the 7B model was quantised
• What learning rate they used

Show the answer and why

Answer: Which single task the claim covers, and whether the comparison holds off that task, since fine-tuning selects among behaviours the smaller model can already produce rather than raising its ceiling

Fine-tuning reshapes the distribution the pretrained model already has. On one narrow task that is often enough to close a gap; the depth and width that set the ceiling were fixed at initialisation. Ask for the off-task numbers, and Part 16 teaches how to produce them yourself.

Question 3. nanochat's README reports training a GPT-2 capability model for about $48. What may you conclude about your own machine?

• That the same run costs about $48 of electricity at home
• Nothing directly: the figure is reported by the project for a rented eight-GPU node, and the README separately says a single GPU takes about eight times longer
• That an afternoon at home produces a GPT-2 capability model
• That the figure is wrong

Show the answer and why

Answer: Nothing directly: the figure is reported by the project for a rented eight-GPU node, and the README separately says a single GPU takes about eight times longer

The reported figure carries its hardware with it. The same README says the code runs on a single GPU with results that are about the same but eight times slower, and that the CPU and MPS path shrinks the model dramatically and will not give strong results. Quote figures with the machine that produced them, which is the rule the whole course runs on.

Question 4. Halfway through a training run the samples are grammatical, fluent and factually wrong. What does that tell you?

• The learning rate is too high
• The corpus is contaminated
• That the run is progressing normally: grammar is learned from far fewer tokens than facts, so fluency arrives well before accuracy
• The tokeniser is broken

Show the answer and why

Answer: That the run is progressing normally: grammar is learned from far fewer tokens than facts, so fluency arrives well before accuracy

The structure of language is dense in the corpus and any given fact is sparse in it, so syntax is cheap and content is expensive. This ordering is the reason fluency is worthless as evidence of correctness, in a model of ten million parameters and in a model of a hundred billion.

Question 5. A base model, as an artefact, includes a chat template.

• True
• False

Show the answer and why

Answer: False

False. A base model is a tokeniser, a configuration and weights. The special tokens that mark a user turn and an assistant turn may exist in the vocabulary, but nothing has trained the model to use them until supervised fine-tuning does, which is Part 13.

Check my answersClear