Scaling Laws and Compute Budgets at Home

By the end of this lesson you will be able to compute the cost of a training run in floating-point operations before you start it; turn the matrix-multiply rate you measured in Part 5 into a token budget for a stated wall-clock time; choose a model size for a fixed budget, and explain why nanochat chooses in the opposite direction; look at a loss curve and say whether the run was undertrained, converged or repeating itself; and decide when copying a speedrun recipe is sound engineering and when it is cargo cult.

This is the second of the two decisions the pipeline needs: how big, and for how long.

One formula, and where it comes from

Part 3 gave the rule. Here is its derivation, from the codebase you are about to run. nanochat/gpt.py computes the cost per token in a method whose docstring is the argument:

Output — what you should see

Each matmul weight parameter contributes 2 FLOPs (multiply *, accumulate +) in forward,
and 2X that in backward => 2+4=6.

Every weight that participates in a matrix multiplication does one multiply and one add per token in the forward pass, and the backward pass costs twice as much because it computes gradients with respect to both the inputs and the weights. Two plus four is six, and so:

Training compute ≈ 6 × parameters × training tokens.

The method adds a second term for attention, “12 * h * q * effective_seq_len”, crediting the PaLM paper’s formulation, because attention costs work that scales with sequence length rather than with parameter count. On short sequences and small models this term is minor; it grows with the square of the context and is why long-context training is expensive out of proportion to the parameters.

The docstring also says how far off it is: “This is ~1% off from the exact formulas of Chinchilla paper”, listing what each convention counts and does not. A rule of thumb that states its own error bar is one you can use.

Two rearrangements do all the work in practice:

• Tokens from a budget. tokens ≈ compute ÷ (6 × parameters).
• Time from a rate. seconds ≈ compute ÷ (rate × utilisation), where the rate is what your machine achieves on a large matrix multiplication and the utilisation is the fraction of it a real training step reaches.

From your machine to a token budget

You already measured the rate. Part 5’s matmul-test.py reported your machine’s BF16 matrix-multiply throughput, and the measurement lab had you record it with its conditions. That number is the numerator of everything below.

The utilisation is the part people leave out. A training step is not one large matrix multiplication; it is many of them interleaved with attention, normalisation, optimiser arithmetic and data movement, so the achieved rate is a fraction of the peak. The training loop prints that fraction on every step as bf16_mfu, model FLOPs utilisation, so you do not have to guess for long. Before your first run you have to guess once, and a fraction between a fifth and a half is the range real runs occupy on the hardware this course covers.

Putting it together, for one hour:

tokens ≈ (rate × utilisation × 3600) ÷ (6 × parameters)

Fill this in from your own notebook rather than taking anyone’s numbers for it. The table below is the shape to fill in, and every cell is empty on purpose.

Pending validationWhat one hour buys on your machine — an estimate you compute, not a measurement

Track	BF16 matrix rate, from Part 5	Utilisation you assume	Model parameters	Tokens in one hour
S — DGX Spark	—	—	—	—
X — Ryzen AI Max+ 395	—	—	—	—
M — Apple silicon	—	—	—	—
N — NVIDIA desktop or laptop	—	—	—	—

your machine: track, chip and memory, your operating system and version · arithmetic, not a measurement: (rate x utilisation x 3600) / (6 x parameters) the torch version matmul-test.py printed in Part 5 · the parameter count base_train.py prints at startup, BF16 compute, FP32 master weights · 512 tokens of context · the date you did the arithmetic

An estimate, not a result. The rate comes from your Part 5 notebook entry; the utilisation is a guess until the training loop prints bf16_mfu, at which point replace it and redo the arithmetic. The attention term is left out, so the estimate is optimistic, and increasingly so as the sequence length grows.

Then, after the run, compare. The training loop prints throughput on every step and the estimate above predicts it, so you have a prediction and a measurement in the same units, which is the shape every measurement in this course takes.

Pending validationPrediction against measurement, one training run — your recording sheet

Quantity	Predicted from the formula	What the training loop printed	Ratio
Training throughput, tokens per second	—	from tok/sec	—
Model FLOPs utilisation	your assumption	from bf16_mfu	—
Wall clock for the planned steps	—	from total time	—
Peak device memory	—	from the final Peak memory usage line	—

your machine: track, chip and memory · nanochat the commit you cloned, from git rev-parse --short HEAD · depth and parameter count as printed at startup, BF16 compute, FP32 master weights · 512 tokens of context · the date you ran it

Empty on purpose. A measurement far below the prediction usually means the utilisation guess was too high, which is ordinary; a measurement far above it usually means the parameter count used in the arithmetic was wrong, most often because the embedding tables were left out of a count that the FLOPs rule includes only for the parts that participate in matrix multiplications.

✕Common mistake

The commonest arithmetic error here is counting the wrong parameters. The six-per-parameter rule applies to weights that take part in matrix multiplications. An embedding lookup is not a matrix multiplication, and on a small model the embedding tables are most of the parameter count. nanochat separates these deliberately: it prints wte, value_embeds, lm_head, transformer_matrices and scalars as separate lines and uses only some of them for scaling. Use the count it labels for matrix multiplications, not the total.

Choosing a size for a budget

Set the tokens-per-parameter ratio to r, so that tokens = r × parameters. Substituting into the compute rule gives compute ≈ 6 × r × parameters², and therefore:

parameters ≈ √(compute ÷ (6 × r))

With the Chinchilla ratio of about twenty, the denominator is a hundred and twenty. That is the whole of compute-optimal sizing: for a fixed budget, the model size grows as the square root of the compute, and the token count grows with it in the same proportion.

Two consequences are worth holding on to. Doubling your budget does not double the model you can train compute-optimally; it multiplies the size by about 1.4 and the tokens by about 1.4 as well. And halving the ratio r, as the reference speedrun does when it sets 8 instead of the default, lets you afford a larger model on the same budget at the cost of leaving it undertrained.

nanochat chooses in the opposite direction, and understanding why makes its output readable. You do not give it a budget; you give it a depth. From the depth it derives the width, the head count and the parameter count, then multiplies the scaling parameter count by the ratio to get a token horizon, then divides by the batch size to get the number of steps. It prints each of those, in order, at startup:

Output — what you should see

Calculated number of iterations from target data:param ratio: <steps>
Total number of training tokens: <tokens>
Tokens : Scaling params ratio: <ratio>
Total training FLOPs estimate: <flops>

Read those four lines before every run. They tell you what the run is going to cost before it starts and, in the lab, they are what you compare your time-budgeted iteration count against.

✦Tip

This is why the lab has you run the training script once without overriding the iteration count, just to read those lines and stop it, before running it with a count that fits your afternoon. The gap between the two is the size of your compromise, and it belongs in the notebook next to the result.

Reading a loss curve

A pretraining curve has fewer shapes than you might expect, and each has one likely cause.

The three phases of a nanochat run, and what the loss is doing in each

Learning rate

warm-up: 40 stepsconstantwarm-down to a fraction of the peak

Muon momentum

0.85 rising to 0.97 over 400 steps0.970.97 falling to 0.90

Loss you observe

steep fallslow, nearly straight descentfaster descent, much of it the schedule rather than new learning

start of runend of run

Proportions follow the defaults: a 40-step warm-up and a warm-down over the last 65 per cent of the run. The drop at the end is real but it is bought by the schedule, which is why comparing two runs of different lengths at the same step number compares different things.

Still falling steeply at the end. The run was undertrained: more tokens would have bought more loss reduction at the rate you were still getting them. This is the normal outcome of a time-budgeted run and it is not a fault, provided you say so. It is also the case where doubling the tokens is worth far more than any change to the architecture.

Flat for the last third, with the warm-down producing only a small step. The model has extracted most of what this corpus at this size will give it. More tokens will buy little; a larger model might buy more.

Validation loss rising while training loss falls. In pretraining this almost always means the corpus is too small and the loader has gone round it more than once. Check the epoch field in the log. If it reads above one, you are re-reading text, and the fix is more shards rather than fewer steps.

A sudden jump upwards that does not recover. A loss explosion. The learning rate is too high for the batch size you set, most often because you overrode the batch size without letting the script rescale the learning rate to match. Part 1’s deliberately broken runs are the same failure at a scale where it takes ten seconds to reproduce.

!Important

Compare bits per byte, not loss, whenever the tokenisers differ. The training log’s loss is mean cross-entropy per token and depends on the vocabulary; the validation line’s bits per byte does not. Two of your own runs with the same tokeniser can be compared either way. A run of yours against a published number can only be compared in bits per byte, and only if the held-out text is the same.

When a speedrun recipe is the right teacher

A speedrun fixes a target and races to it. modded-nanoGPT’s README states its target exactly: “the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss on the FineWeb validation set”, a target it takes from the llm.c GPT-2 reproduction. nanochat’s leaderboard does the same thing with a benchmark score instead of a loss: the wall-clock time to beat a stated CORE score on an eight-GPU node.

Both are excellent teachers of one thing: which changes actually matter. modded-nanoGPT’s README lists the techniques that produced its improvements, and that list is a reading order for modern training practice: rotary embeddings, QK-norm and squared ReLU in the architecture; the Muon optimiser; low-precision arithmetic in specific places; sliding-window attention with a schedule on the window size; batch-size and sequence-length schedules. Every one of those was adopted because it moved a number on a fixed target, which is a higher standard of evidence than most published advice about training meets.

They are poor teachers of two other things.

They do not transfer to different hardware. The whole optimisation is against eight H100s with a particular interconnect, and a technique that pays for itself there can cost you at home. The FP8 paths need hardware support your machine may not have; the sliding-window attention needs Flash Attention 3, and nanochat’s own training script warns when it is missing that “SDPA has no support for sliding window attention” and that “Your GPU utilization will be terrible” unless you switch the pattern off. That warning is the reason every recipe in the lab sets the window pattern to full context.

And they do not transfer to a different target. A recipe tuned to reach a specific loss as fast as possible is not a recipe for the best model in a fixed time, nor for the best model on your corpus. The long list of stacked techniques was validated in combination on one objective, and pulling three of them out and applying them to a different one is not supported by the evidence that made the list.

?Reality check

The claim to be careful with is “these are the techniques that make training fast”. The supported version is narrower: on this hardware, at this scale, against this target, this combination was faster than the previous combination, as reported by the repository with a date and a commit. That is genuinely valuable and it is not a general law. Use a speedrun recipe as a default that somebody competent has already tuned, then change one thing at a time and measure, which is what the lab’s “go further” step asks you to do.

Turn a compute estimate into a stop rule

Use a short representative training run to estimate processed tokens per second, including the sequence length, microbatch and precision you plan to use. A standalone matrix-multiply benchmark is an upper-bound clue; data loading, attention, optimiser work and synchronisation reduce effective training throughput.

Set a wall-clock budget and reserve time for evaluation, checkpointing and unexpected interruptions. Multiply the remaining training time by observed token throughput to estimate the feasible token budget. Repeat the estimate after the warm-up period rather than extrapolating the first step, which may include compilation or lazy allocation.

Choose in advance what ends the run: elapsed time, processed tokens or a validation criterion. Preserve the last usable checkpoint when the budget expires. If loss is still decreasing, report that the experiment was budget-limited; do not claim convergence. Scaling laws describe fitted trends under particular assumptions and are valuable for reasoning about tradeoffs. They do not determine a universally optimal model size for your different corpus, hardware, optimiser and downstream task.

Recap

Training compute is about six operations per matrix-multiplication parameter per token, plus an attention term that grows with sequence length, and the codebase you are running derives that rule in a docstring and states its own one-per-cent error. Divide the compute you can afford by six times the parameter count and you have a token budget; divide by your measured matrix rate times a utilisation fraction and you have a wall clock. Compute-optimal sizing makes the model grow as the square root of the budget, and nanochat inverts the calculation because its one dial is depth, so it prints the horizon it derived and you read it. A curve still falling at the end is undertrained, which is the expected and acceptable outcome of a time-budgeted run; a rising validation curve in pretraining usually means the loader has been round the corpus twice; the accelerated fall at the end is the learning-rate warm-down and not new learning. Speedrun recipes are the best available evidence for which techniques matter and no evidence at all that they transfer to your hardware, your corpus or your objective.

Check your understanding

Question 1. You plan a run of a 40 million parameter model on 400 million tokens. Estimate the training compute with the course rule.

• About 1.6 × 10¹⁶ operations
• About 9.6 × 10¹⁶ operations, from 6 × 4 × 10⁷ × 4 × 10⁸
• About 2.4 × 10¹⁰ operations
• It cannot be estimated without the batch size

Show the answer and why

Answer: About 9.6 × 10¹⁶ operations, from 6 × 4 × 10⁷ × 4 × 10⁸

Six per parameter per token: 6 × 4 × 10⁷ × 4 × 10⁸ ≈ 9.6 × 10¹⁶. The batch size changes how the work is arranged, not how much of it there is. Divide by your measured matrix rate times a utilisation fraction to get a wall clock.

Question 2. Your compute budget doubles. Keeping the Chinchilla ratio, how much bigger a model should you train?

• Twice as big
• About 1.4 times as big, because the compute-optimal size grows as the square root of the budget, with the token count growing by the same factor
• Four times as big
• The same size, trained for twice as long

Show the answer and why

Answer: About 1.4 times as big, because the compute-optimal size grows as the square root of the budget, with the token count growing by the same factor

With tokens = r × parameters, compute ≈ 6 × r × parameters², so parameters ≈ √(compute ÷ 6r). Doubling the compute multiplies both the size and the token count by √2. This is why a tenfold budget increase is an unimpressive size increase, and why frontier runs are enormous.

Question 3. Your loss falls noticeably faster over the last third of the run. What is the most likely explanation?

• The model has started to generalise
• The learning-rate warm-down: the schedule decays the rate over the last 65 per cent of the planned steps by default, and the loss falls as it does
• The data loader has reached better data
• The optimiser has switched from Muon to AdamW

Show the answer and why

Answer: The learning-rate warm-down: the schedule decays the rate over the last 65 per cent of the planned steps by default, and the loss falls as it does

The end-of-run drop is bought by the schedule, and it is one reason a run cannot be stopped early and compared with a shorter planned run. Both curves are real; they were produced by different schedules and are not the same experiment.

Question 4. Halfway through a pretraining run the validation loss starts creeping up while the training loss keeps falling. What do you check first?

• The learning rate
• The epoch field in the training log, because in pretraining a rising validation curve almost always means the loader has gone round a corpus that is too small and is re-reading text
• The tokeniser
• The batch size

Show the answer and why

Answer: The epoch field in the training log, because in pretraining a rising validation curve almost always means the loader has gone round a corpus that is too small and is re-reading text

Overfitting in pretraining means the corpus ran out. The log prints the epoch on every step for exactly this reason. Download more shards rather than shortening the run, because shortening it changes the schedules too.

Question 5. A speedrun repository lists twenty techniques that took its record from three hours to under two. Which conclusion is supported?

• Those twenty techniques will speed up your run at home
• On that hardware, at that scale, against that fixed target, the combination was faster than the previous combination, as reported with a date and a commit; whether any single one helps your different run is an open question you would have to measure
• The techniques are the fastest way to train any model
• The first technique on the list is the most important

Show the answer and why

Answer: On that hardware, at that scale, against that fixed target, the combination was faster than the previous combination, as reported with a date and a commit; whether any single one helps your different run is an open question you would have to measure

A leaderboard is evidence about one objective on one machine. Several of the techniques depend on hardware features and kernels you may not have, and the training script itself warns that sliding-window attention without Flash Attention 3 makes utilisation much worse. Take the recipe as a tuned default, then change one thing at a time.

Check my answersClear