Skip to content
Local LLM Course

Pretraining a Small Model from Scratch

Everything you have run so far was made by somebody else. You downloaded weights, measured them, served them, wired them into applications. This part is the other direction: you start with text and end with a model file that did not exist before, and you do it on the machine in front of you.

The model will be small and it will not be good. That is the point. A model of a few tens of millions of parameters, trained for twenty-five minutes on a few hundred million tokens, produces text that is grammatical and confidently wrong, and watching it get there is the fastest way to stop treating the words “pretraining”, “tokeniser”, “loss curve” and “compute-optimal” as incantations. Part 3 explained what pretraining is. This part has you do it.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • say what a base model is, what it can and cannot be made into later, and why the tokeniser and the corpus are decisions you cannot undo by fine-tuning;
  • read a small pretraining codebase end to end, and name what each file does: tokeniser training, model definition, data loader, optimiser, evaluation, sampling;
  • choose a pretraining corpus, check its licence, take a subset that fits a token budget, and train a tokeniser on it that you can inspect;
  • turn a compute budget into a model size and a token count with the six-operations-per-parameter rule, and recognise an undertrained loss curve when you see one;
  • run the whole pipeline on your own machine, evaluate the result, sample from it, and record what it cost in tokens, time, power and loss;
  • train a model on a corpus of your own choosing and compare it honestly against a fine-tuned pretrained model of a similar size.

What you need first

Section titled “What you need first”

Part 11’s training environment, which is the toolchain this part uses, and the lab notebook you have kept since Part 1. Part 3’s pretraining lesson is the conceptual background and this part assumes it rather than repeating it. Part 5’s measurements matter here in a way they have not before: the matrix-multiply rate you measured on your own machine is what the scaling arithmetic in this part is applied to, so have that page of the notebook open.

Every page runs at the 8 GB memory floor. The lab downloads under a gigabyte of text on its smallest path and produces a model file of a few hundred megabytes. Nothing here needs a cluster, and nothing here needs an account with a cloud provider.

How to work through it

Section titled “How to work through it”

Read the four lessons in order. The first says why the exercise is worth an afternoon; the second reads a real codebase line by line, so keep the repository open in an editor beside it; the third and fourth are the two decisions you have to make before you can start a run, namely what data and what size.

Then do the lab, which takes about seventy-five minutes of which the training run is under thirty. The project after it is the part people remember: a model trained on a corpus you chose, held up against a fine-tuned model of a similar size on the same text, with numbers rather than impressions. Part 13 picks up fine-tuning properly from there.

Part12 of 27Level3 — Model BuilderPages6Estimated time4.7 hours

0 / 6 lessons in this part completed