Project: A Domain Micro-Model

Validated on: written from the documentation and source cited above; not yet validated on hardware on any track. The corpus, commit and wall clock each track was run with will be recorded here when the validation pass is done.

Objective

The lab trained a model on somebody else’s corpus. This project trains one on yours, and then does the thing that makes the result mean something: it trains a second model on exactly the same text, starting from a published base model that has already read trillions of tokens, and compares the two on the same held-out documents with a metric that neither tokeniser can distort.

By the end you will have a corpus you can account for line by line, a tokeniser sized for it, a model trained from random initialisation, an adapter trained on top of a pretrained model, one table of numbers, and a written comparison of the kind you would be willing to show somebody who disagreed with you.

The comparison is not a fair fight and the write-up has to say so. That is the most valuable part of the exercise, and the section on it is not optional.

⌂Single-machine path

Single machine on every track. The corpus is small enough to prepare on any of them and the training runs are the same size as the lab’s.

Requirements

The lab in this part, finished, with the notebook entry it produced. Its virtual environment is reused, with two packages added, and the two pieces of arithmetic you did there are the ones this project builds on.

Time. About ninety minutes attended. Two training runs of roughly twenty minutes each dominate it; the corpus preparation is a few minutes and the evaluation is a few more.

Disk. Under a gigabyte for the corpus and its shards. The pretrained model is the large download: the Qwen3-1.7B-Base card gives 1.7 billion parameters, which is around 3.4 GB at BF16. It is Apache-2.0 licensed and not gated, so no acceptance step and no token are needed; the course model reference records the licence.

Keep the project’s artefacts away from the lab’s by pointing the base directory somewhere else:

RunnableAll tracks

a separate base directory for the project

export NANOCHAT_BASE_DIR=~/.cache/nanochat-domain
mkdir -p "$NANOCHAT_BASE_DIR"

Everything in this project reads and writes under that directory, so the lab’s shards, tokeniser and checkpoints are untouched and you can go back to them.

Add the two packages the reference-model half needs, into the environment the lab built:

RunnableAll tracks

add transformers and peft to nanochat's environment

cd ~/nanochat && uv pip install transformers peft

Track S — NVIDIA DGX Spark

128 GB of unified memory means the reference model fits with room to spare, and you can raise the from-scratch model’s depth well above the lab default. Use the wheel index that worked for you in the lab; if you moved to the CUDA 13.0 build there, the environment already has it.

Track X — AMD Ryzen AI Max+ 395Partial

Same ROCm caveat as the lab: nanochat's pinned torch exists on the ROCm 6.4 wheel index but not on the ROCm 7.x ones, and the compatibility matrix lists gfx1151 without a support-tier qualifier (checked 2026-09-09).

Whatever worked in the lab works here; nothing in this project needs anything the lab did not. If you finished the lab on the CPU, finish this project on the CPU too and say so in the write-up: both models will be trained under the same handicap, so the comparison between them remains valid even though neither number is comparable with anyone else’s.

Track M — Apple siliconPartial

nanochat runs on PyTorch's MPS backend, which its README lists while noting the author has not exercised all of those code paths; the reference-model half uses transformers on MPS, which the course's tool table lists as the Mac training path alongside mlx-lm.

Both halves run on MPS. The from-scratch half is the lab’s path unchanged. For the reference half, continue-pretraining.py selects bfloat16 on MPS, which the nanochat README notes works on recent macOS; if you hit a kernel error, pass --device cpu for that half only and record that you did.

Close other applications before the reference-model run. A 1.7-billion-parameter model at BF16 plus its activations is competing with the desktop for the same unified pool.

Track N — NVIDIA desktop or laptop

On 8 or 12 GB of VRAM both halves fit at the defaults: the from-scratch model is small, and the reference half uses LoRA with gradient checkpointing and a batch of one. On 16 GB and above, raise --batch-size on the reference half and DEVICE_BATCH_SIZE on the from-scratch half, and record the peak memory each reported.

How the pieces fit

One corpus, two models, one comparison

1. 1Choose and license the corpusPublic-domain text by default. One provenance row per source, written before anything is downloaded twice.
2. 2Strip, deduplicate, shardRemove the boilerplate that every file shares, drop exact duplicates, shuffle, hold out whole documents as validation, write parquet shards.
3. 3Train a tokeniser for the domainOn the training shards only, with a vocabulary sized for a small corpus rather than for the web.
4. 4Model A: pretrain from scratchRandom initialisation, the domain corpus, the lab's time budget. Nothing reaches this model except what the corpus contains.
5. 5Model B: continue pretraining a base modelA published base model, LoRA, the same shards, the same held-out split. This model arrives already knowing a great deal.
6. 6Score both on the same held-out textBits per byte, so that two different tokenisers cannot distort the comparison, plus a small task set split into in-domain and out-of-domain items.
7. 7Write it up with its asymmetries statedDifferent parameter counts, different pretraining budgets, different methods. The numbers are only useful next to those facts.

The only thing held constant between the two models is the corpus and the held-out split. Everything else differs, which is why the write-up matters as much as the table.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-12-pretraining-from-scratch"
cd "$LAB_DIR"
pwd
test -f "fetch-gutenberg-corpus.sh"

Expected result: pwd ends in part-12-pretraining-from-scratch and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Choose a domain, and settle the licence first

Pick a body of text with a shape to it: one author, one subject, one period, one genre. Narrow is good. A model of this size cannot learn a broad distribution and can pick up the cadence of a narrow one, which is what makes the samples readable.

Public-domain text is the safe default, and Project Gutenberg is the easiest source. Its permission page states that “The vast majority of Project Gutenberg eBooks are in the public domain in the US”, and that for such a work “nobody can grant, or withhold, permission to do with this item as you please”. It also carries the caution that matters if you are not in the United States: “Not all items that are public domain in the US are public domain in other countries, and vice versa”, with the advice to get professional guidance before redistributing.

!Important

Two separate questions, and people conflate them. Whether you may train on a text, and whether you may redistribute it, have different answers in different jurisdictions. This project keeps you on solid ground by using text that is public domain at source, and it asks you to write down the position for each item before you download it, because reconstructing provenance afterwards is miserable and usually incomplete.

If you use something other than Project Gutenberg, the rule is the same one the data lesson set out: find the licence at the origin, not at the repackaging, and record it with the date you checked. The FineWeb-Edu card is a good example of what an adequate statement looks like: a named licence, ODC-By v1.0, plus a second document, CommonCrawl’s terms, that also applies.

2. Fetch the corpus and strip the boilerplate

RunnableAll tracks

fetch-gutenberg-corpus.sh

#!/usr/bin/env bash
# Purpose: build a small public-domain corpus from Project Gutenberg plain-text
#          eBooks: fetch each book once, strip the licence header and footer that
#          would otherwise be the most-repeated text in the corpus, and write a
#          provenance record so the corpus can be licensed, cited and rebuilt
# Platform: all (plain shell; needs curl and awk)
# Minimum memory: 8 GB
# Assumes: curl and awk are on PATH; the machine can reach www.gutenberg.org; you
#          have read https://www.gutenberg.org/policy/permission.html and checked
#          the copyright position of each title in your own country
set -euo pipefail

# ---------------------------------------------------------------- settings ---
# Book numbers from the Project Gutenberg catalogue. Replace these with the ones
# that make up your domain; the point of the project is that the corpus is yours.
BOOK_IDS="${BOOK_IDS:-1661 2852 108 244 834 3289}"
OUT_DIR="${OUT_DIR:-$PWD/domain-corpus}"
DELAY="${DELAY:-2}"     # seconds between requests; be a good guest
BASE_URL="${BASE_URL:-https://www.gutenberg.org/cache/epub}"

TEXT_DIR="$OUT_DIR/text"
RAW_DIR="$OUT_DIR/raw"
SOURCES="$OUT_DIR/sources.tsv"
TODAY="$(date -u +%Y-%m-%d)"

usage() {
    cat <<'USAGE'
Usage: [BOOK_IDS="1661 2852"] [OUT_DIR=./domain-corpus] bash fetch-gutenberg-corpus.sh

Downloads each Project Gutenberg book as plain UTF-8 text, keeps the raw file for
reference, writes a stripped copy with the licence header and footer removed, and
appends one tab-separated provenance row per book to sources.tsv.

Project Gutenberg's permission page states that the vast majority of its eBooks are
in the public domain in the United States, and also that "Not all items that are
public domain in the US are public domain in other countries, and vice versa."
Checking that for your own jurisdiction is your job, not this script's.
USAGE
}

command -v curl >/dev/null 2>&1 || { usage; echo "ERROR: curl is not on PATH." >&2; exit 1; }

mkdir -p "$TEXT_DIR" "$RAW_DIR"
if [ ! -f "$SOURCES" ]; then
    printf 'book_id\ttitle\turl\tretrieved\tbytes_raw\tbytes_stripped\n' > "$SOURCES"
fi

# ------------------------------------------------------------------ fetch ----
for id in $BOOK_IDS; do
    raw="$RAW_DIR/pg$id.txt"
    out="$TEXT_DIR/pg$id.txt"
    url="$BASE_URL/$id/pg$id.txt"

    if [ -s "$raw" ]; then
        echo "==> $id already downloaded"
    else
        echo "==> fetching $url"
        curl -fsSL --max-time 120 -o "$raw.part" "$url"
        mv "$raw.part" "$raw"
        sleep "$DELAY"
    fi

    # Every Project Gutenberg plain-text file wraps the work between two marker
    # lines. Everything outside them is boilerplate that is byte-identical across
    # books, which makes it the single most over-represented text in a corpus of
    # this size if it is left in.
    awk '
        /^\*\*\* *START OF (THE|THIS) PROJECT GUTENBERG EBOOK/ { inside = 1; next }
        /^\*\*\* *END OF (THE|THIS) PROJECT GUTENBERG EBOOK/   { inside = 0; next }
        inside { print }
    ' "$raw" > "$out"

    if [ ! -s "$out" ]; then
        echo "WARNING: no marker lines found in $raw; keeping the raw text instead." >&2
        echo "         Open it and check the format before training on it." >&2
        cp "$raw" "$out"
    fi

    title="$(grep -m1 '^Title:' "$raw" | sed 's/^Title:[[:space:]]*//' | tr -d '\r\t' || true)"
    [ -n "$title" ] || title="(title line not found)"
    raw_bytes="$(wc -c < "$raw" | tr -d ' ')"
    out_bytes="$(wc -c < "$out" | tr -d ' ')"

    printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
        "$id" "$title" "$url" "$TODAY" "$raw_bytes" "$out_bytes" >> "$SOURCES"
    echo "    $title"
    echo "    kept $out_bytes of $raw_bytes bytes"
done

TOTAL_BYTES="$(cat "$TEXT_DIR"/*.txt | wc -c | tr -d ' ')"

cat <<SUMMARY

==> done
    books          $(find "$TEXT_DIR" -name '*.txt' | wc -l | tr -d ' ')
    stripped text  $TEXT_DIR
    raw text       $RAW_DIR
    provenance     $SOURCES
    total bytes    $TOTAL_BYTES

    A rough token count is bytes divided by the bytes-per-token ratio your
    tokeniser achieves, which tok_eval prints. Before training, read sources.tsv
    and satisfy yourself about the copyright position of every row in your own
    country; the Project Gutenberg permission page is the starting point and not
    the end of that question.

    Next: python make-domain-shards.py --input $TEXT_DIR
SUMMARY

Download fetch-gutenberg-corpus.sh108 lines

RunnableAll tracks

fetch a handful of books

cd "$LAB_DIR"
BOOK_IDS="1661 2852 108 244 834 3289" bash fetch-gutenberg-corpus.sh

Replace the book numbers with the ones for your domain. The script downloads each file once, keeps the raw copy, writes a stripped copy with the licence header and footer removed, and appends a row to sources.tsv with the title, the URL, the date and the byte counts.

The stripping is not tidiness. Every Project Gutenberg text file carries the same header and footer, so on a corpus of six books that boilerplate would be six identical copies of several hundred words in a corpus of a few million, which the data lesson explained is exactly how a small model learns the wrong thing very well. Open one stripped file and confirm it starts with the work rather than with a licence.

3. Split, deduplicate and shard

RunnableAll tracks

make-domain-shards.py

"""Turn a directory of plain-text files into the parquet shards nanochat trains on.

Purpose: prepare a corpus of your own for pretraining: split the text into
         documents at paragraph boundaries, drop exact duplicates, shuffle with a
         recorded seed, hold out a validation split as whole documents, and write
         the result as parquet shards with a "text" column in the layout
         nanochat's dataset loader expects.
Platform: all (pure Python plus pyarrow, which nanochat already depends on)
Minimum memory: 8 GB
Assumes: pyarrow is installed (it is a nanochat dependency, so nanochat's own
         virtual environment has it); the input directory contains UTF-8 .txt
         files; the output directory is a base directory for the domain corpus
         and not the one holding the shards downloaded in the lab.

Usage: python make-domain-shards.py --input domain-corpus/text
                                    [--out-dir ~/.cache/nanochat-domain/base_data_climbmix]
                                    [--target-chars 4000] [--val-fraction 0.05]
                                    [--seed 0] [--force]

The loader treats the alphabetically last parquet file as the validation split and
every other file as training data, so this script always writes at least two shards
and names them shard_00000.parquet upwards.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import random
import re
import sys
from pathlib import Path

# A blank line is the most reliable document boundary in plain prose. Chapters
# would be better but their markers differ from book to book, and a paragraph run
# of a few thousand characters is a reasonable unit for a model with a sequence
# length in the hundreds.
PARAGRAPH_BREAK = re.compile(r"\n\s*\n")


def split_into_documents(text: str, target_chars: int) -> list[str]:
    """Group paragraphs into documents of roughly target_chars characters."""
    documents, current, size = [], [], 0
    for paragraph in PARAGRAPH_BREAK.split(text):
        paragraph = paragraph.strip()
        if not paragraph:
            continue
        current.append(paragraph)
        size += len(paragraph)
        if size >= target_chars:
            documents.append("\n\n".join(current))
            current, size = [], 0
    if current:
        documents.append("\n\n".join(current))
    return documents


def deduplicate(documents: list[str]) -> tuple[list[str], int]:
    """Drop exact duplicates, keeping the first occurrence. Returns (kept, dropped)."""
    seen, kept = set(), []
    for doc in documents:
        digest = hashlib.sha256(doc.encode("utf-8")).hexdigest()
        if digest in seen:
            continue
        seen.add(digest)
        kept.append(doc)
    return kept, len(documents) - len(kept)


def write_shard(path: Path, documents: list[str], rows_per_group: int) -> None:
    """Write one parquet file with a single "text" column and several row groups.

    nanochat's loader iterates row groups rather than whole files, and uses the
    group index to shard across ranks, so a file with one enormous group works but
    parallelises badly. Several groups per file costs nothing and behaves better.
    """
    import pyarrow as pa
    import pyarrow.parquet as pq

    table = pa.table({"text": pa.array(documents, type=pa.string())})
    pq.write_table(table, path, row_group_size=max(1, rows_per_group))


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--input", required=True, help="directory of UTF-8 .txt files")
    parser.add_argument(
        "--out-dir",
        default=os.path.join(
            os.environ.get("NANOCHAT_BASE_DIR", str(Path.home() / ".cache" / "nanochat-domain")),
            "base_data_climbmix",
        ),
        help="where the shards are written; must be the directory nanochat reads",
    )
    parser.add_argument("--target-chars", type=int, default=4000, help="approximate characters per document")
    parser.add_argument("--val-fraction", type=float, default=0.05, help="share of documents held out")
    parser.add_argument("--docs-per-shard", type=int, default=2000)
    parser.add_argument("--rows-per-group", type=int, default=200)
    parser.add_argument("--min-chars", type=int, default=200, help="drop documents shorter than this")
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--force", action="store_true", help="overwrite an output directory that already has shards")
    args = parser.parse_args()

    source_dir = Path(args.input).expanduser()
    files = sorted(source_dir.glob("*.txt"))
    if not files:
        sys.exit(f"No .txt files under {source_dir}.")

    out_dir = Path(args.out_dir).expanduser()
    existing = sorted(out_dir.glob("*.parquet")) if out_dir.exists() else []
    if existing and not args.force:
        sys.exit(
            f"{out_dir} already holds {len(existing)} parquet file(s).\n"
            "Refusing to mix two corpora in one directory: choose another --out-dir, or pass\n"
            "--force if you are certain these shards should be replaced."
        )
    out_dir.mkdir(parents=True, exist_ok=True)
    for stale in existing:
        stale.unlink()

    # ---- read and split ---------------------------------------------------
    documents, per_file = [], {}
    for path in files:
        text = path.read_text(encoding="utf-8", errors="replace")
        docs = [d for d in split_into_documents(text, args.target_chars) if len(d) >= args.min_chars]
        per_file[path.name] = len(docs)
        documents.extend(docs)
    print(f"read {len(files)} file(s) -> {len(documents):,} document(s)")

    documents, dropped = deduplicate(documents)
    print(f"deduplicated: dropped {dropped:,}, kept {len(documents):,}")
    if not documents:
        sys.exit("Nothing left after filtering; lower --min-chars or check the input.")

    # Shuffle before splitting, so the validation set is not the end of one book.
    random.Random(args.seed).shuffle(documents)

    n_val = max(1, round(len(documents) * args.val_fraction))
    if n_val >= len(documents):
        sys.exit("--val-fraction leaves no training data.")
    val_docs = documents[:n_val]
    train_docs = documents[n_val:]

    # ---- write ------------------------------------------------------------
    # The loader takes the alphabetically last file as validation, so the training
    # shards are written first and the validation shard last.
    shards, index = [], 0
    for start in range(0, len(train_docs), args.docs_per_shard):
        chunk = train_docs[start : start + args.docs_per_shard]
        path = out_dir / f"shard_{index:05d}.parquet"
        write_shard(path, chunk, args.rows_per_group)
        shards.append((path, len(chunk), sum(len(d) for d in chunk)))
        index += 1
    val_path = out_dir / f"shard_{index:05d}.parquet"
    write_shard(val_path, val_docs, args.rows_per_group)
    shards.append((val_path, len(val_docs), sum(len(d) for d in val_docs)))

    train_chars = sum(len(d) for d in train_docs)
    val_chars = sum(len(d) for d in val_docs)

    manifest = {
        "input": str(source_dir),
        "out_dir": str(out_dir),
        "seed": args.seed,
        "target_chars": args.target_chars,
        "min_chars": args.min_chars,
        "val_fraction": args.val_fraction,
        "documents_per_file": per_file,
        "duplicates_dropped": dropped,
        "train_documents": len(train_docs),
        "val_documents": len(val_docs),
        "train_characters": train_chars,
        "val_characters": val_chars,
        "shards": [{"file": p.name, "documents": n, "characters": c} for p, n, c in shards],
    }
    manifest_path = out_dir / "corpus-manifest.json"
    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")

    print(f"\nwrote {len(shards)} shard(s) to {out_dir}")
    for path, n_docs, chars in shards:
        role = "validation" if path == val_path else "train"
        print(f"  {path.name:<24} {n_docs:>7,} docs  {chars:>12,} chars  ({role})")
    print(f"\ntrain characters: {train_chars:,}")
    print(f"val characters:   {val_chars:,}")
    print(f"manifest:         {manifest_path}")
    print(
        "\nA character count is not a token count. Train the tokeniser on these shards,\n"
        "run tok_eval, and divide the characters by the bytes-per-token ratio it reports\n"
        "to get the token budget this corpus actually provides."
    )


if __name__ == "__main__":
    main()

Download make-domain-shards.py195 lines

RunnableAll tracks

make the shards

~/nanochat/.venv/bin/python make-domain-shards.py \
    --input domain-corpus/text \
    --out-dir "$NANOCHAT_BASE_DIR/base_data_climbmix" \
    --val-fraction 0.05 --seed 0

Output — what you should see

read 6 file(s) -> <n> document(s)
deduplicated: dropped <n>, kept <n>
wrote <n> shard(s) to /.../base_data_climbmix
  shard_00000.parquet        <n> docs   <n> chars  (train)
  shard_00001.parquet        <n> docs   <n> chars  (validation)

train characters: <n>
val characters:   <n>

The script groups paragraphs into documents of about four thousand characters, drops exact duplicates, shuffles with the seed it records, holds out five per cent of the documents as validation, and writes the parquet layout nanochat reads: a text column, several row groups per file, and the alphabetically last file as the validation split.

Shuffling before splitting is the step to understand. If you split first, the validation set is the end of one book and the comparison measures whether a model can finish that book rather than whether it learned the domain.

✕Common mistake

Splitting by sentence or by paragraph inside a document leaks. Neighbouring paragraphs share vocabulary, names and phrasing, so a validation paragraph whose neighbours were trained on is much easier than genuinely held-out text, and both models will look better than they are. The script holds out whole documents for this reason, and the manifest it writes records the seed so that the split can be reproduced exactly.

4. Size the tokeniser for the corpus, not for the web

The lab used a vocabulary of 32,768 on a corpus of hundreds of millions of tokens. Your corpus is two or three orders of magnitude smaller, and the trade from the data lesson now points the other way.

Two reasons to go smaller. A large vocabulary trained on a small corpus fills up with entries seen a handful of times, which are close to untrained. And the embedding and output tables are vocabulary size times width, so on a small model they dominate the parameter count: a quarter of the vocabulary is a quarter of most of the model.

RunnableAll tracks

train the domain tokeniser and evaluate it

cd ~/nanochat && source "$HOME/llm-course/.venv/bin/activate"
python -m scripts.tok_train --vocab-size=8192 --max-chars=100000000
python -m scripts.tok_eval

Then do the experiment rather than taking the advice: train a second tokeniser at 16,384 or 32,768 and compare the bytes-per-token ratios tok_eval prints on the corpus rows. Record all of them. There is a point where a larger vocabulary stops buying compression on a corpus this size, and finding it takes two minutes.

✦Tip

Whichever vocabulary you keep, retrain it before the from-scratch run and remember it is frozen afterwards. The tokeniser must be trained on the training shards only, which is what tok_train does: it reads the train split, and the validation shard is untouched.

5. Model A: pretrain from scratch on your corpus

The lab’s script does this unchanged; only the base directory differs, which is what points it at your shards and your tokeniser.

RunnableAll tracks

the from-scratch run

cd "$LAB_DIR"
NANOCHAT_BASE_DIR=~/.cache/nanochat-domain \
    TRACK=nvidia TARGET_MINUTES=20 MODEL_TAG=d6-domain \
    bash train-small.sh

Watch the epoch field. On a corpus this small it will go above one, and that is a decision rather than an accident: you have chosen to let the model see the corpus several times because there is not enough of it to fill the run. Note how many epochs the run reached, because it belongs in the write-up. Repeating a small corpus is how a model memorises rather than generalises, and you will see the evidence in the samples.

6. Model B: continue pretraining a published base model on the same text

RunnableAll tracks

continue-pretraining.py

"""Continue pretraining a reference base model on the same domain corpus, with LoRA.

Purpose: produce the other half of the project's comparison. The from-scratch model
         saw only this corpus; this one starts from a published base model that saw
         trillions of tokens and is then trained on exactly the same shards, so the
         difference between them is attributable to what pretraining put in rather
         than to the data.
Platform: all (cuda on Tracks S and N and on Track X with ROCm, mps on Track M, cpu
          anywhere as a slow fallback); the device is autodetected and recorded.
Minimum memory: 8 GB
Assumes: transformers, peft and pyarrow are installed in the active environment;
         the domain shards written by make-domain-shards.py exist; the model is
         downloadable from the Hugging Face Hub or already cached locally.

Usage: python continue-pretraining.py --shards ~/.cache/nanochat-domain/base_data_climbmix
                                      [--model Qwen/Qwen3-1.7B-Base]
                                      [--max-steps 200] [--seq-len 512]
                                      [--batch-size 1] [--grad-accum 8]
                                      [--lr 1e-4] [--out-dir domain-lora]
                                      [--labbook labbook.md]

This is the Part 11 training recipe applied to plain next-token prediction rather
than to instruction data: no chat template, no loss masking, just the corpus. Part 13
teaches supervised fine-tuning properly, including when LoRA is and is not the right
choice; here it is used because it is the only path that fits the 8 GB memory floor.
"""
from __future__ import annotations

import argparse
import json
import math
import platform
import random
import secrets
import time
from datetime import datetime, timezone
from pathlib import Path


# --------------------------------------------------------------------------- #
# Corpus                                                                       #
# --------------------------------------------------------------------------- #

def read_split(shard_dir: Path, split: str) -> list[str]:
    """Read documents from the parquet shards, using nanochat's split convention.

    The alphabetically last shard is the validation split and every other shard is
    training data. Reading them here the same way the from-scratch run does is what
    makes the two models comparable.
    """
    import pyarrow.parquet as pq

    paths = sorted(p for p in shard_dir.glob("*.parquet"))
    if len(paths) < 2:
        raise SystemExit(f"Expected at least two parquet shards in {shard_dir}, found {len(paths)}.")
    chosen = paths[:-1] if split == "train" else paths[-1:]
    documents: list[str] = []
    for path in chosen:
        documents.extend(pq.read_table(path, columns=["text"]).column("text").to_pylist())
    return documents


def pack(tokeniser, documents: list[str], seq_len: int, separator_id: int | None) -> list[list[int]]:
    """Tokenise every document and pack the stream into fixed-length blocks."""
    stream: list[int] = []
    for document in documents:
        stream.extend(tokeniser(document, add_special_tokens=False)["input_ids"])
        if separator_id is not None:
            stream.append(separator_id)
    usable = (len(stream) // seq_len) * seq_len
    return [stream[i : i + seq_len] for i in range(0, usable, seq_len)]


# --------------------------------------------------------------------------- #
# Model                                                                        #
# --------------------------------------------------------------------------- #

def pick_device(requested: str):
    import torch

    if requested:
        return torch.device(requested)
    if torch.cuda.is_available():
        return torch.device("cuda")
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")


def load_base_model(model_id: str, dtype):
    """Load the model in the requested dtype across transformers versions.

    The keyword that selects the load dtype was renamed between major versions of
    transformers, so try the current name and fall back rather than asserting which
    one this installation expects.
    """
    from transformers import AutoModelForCausalLM

    try:
        return AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype)
    except TypeError:
        return AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype)


def linear_leaf_names(model, exclude=("lm_head",)) -> list[str]:
    """Names of the linear submodules LoRA should adapt, discovered from the model.

    Discovering them beats hard-coding a list per architecture: the names differ
    between families, and a wrong name fails quietly by adapting nothing.
    """
    import torch

    names = set()
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear) and not any(name.endswith(e) for e in exclude):
            names.add(name.split(".")[-1])
    return sorted(names)


# --------------------------------------------------------------------------- #
# The run log                                                                  #
# --------------------------------------------------------------------------- #

def append_run_log(path: str, record: dict) -> None:
    """Append one JSON line, in the run-log format Part 11 introduces.

    A minimal writer lives here so this script runs on its own; where Part 11's
    runlog.py is available it writes the same fields.
    """
    notebook = Path(path)
    if not notebook.exists():
        notebook.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with notebook.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")


def new_run_id() -> str:
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


# --------------------------------------------------------------------------- #

def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--shards", required=True, help="directory of domain parquet shards")
    parser.add_argument("--model", default="Qwen/Qwen3-1.7B-Base")
    parser.add_argument("--max-steps", type=int, default=200, help="optimiser steps, not micro-steps")
    parser.add_argument("--seq-len", type=int, default=512)
    parser.add_argument("--batch-size", type=int, default=1, help="blocks per forward/backward")
    parser.add_argument("--grad-accum", type=int, default=8)
    parser.add_argument("--lr", type=float, default=1e-4)
    parser.add_argument("--warmup-steps", type=int, default=20)
    parser.add_argument("--lora-r", type=int, default=16)
    parser.add_argument("--lora-alpha", type=int, default=32)
    parser.add_argument("--lora-dropout", type=float, default=0.0)
    parser.add_argument("--eval-every", type=int, default=50)
    parser.add_argument("--eval-blocks", type=int, default=16)
    parser.add_argument("--out-dir", default="domain-lora")
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--device", default="", help="cuda|mps|cpu (empty = autodetect)")
    parser.add_argument("--seed", type=int, default=0)
    args = parser.parse_args()

    import torch
    from peft import LoraConfig, get_peft_model
    from transformers import AutoTokenizer

    torch.manual_seed(args.seed)
    device = pick_device(args.device)

    # bfloat16 halves the weight memory and is supported on CUDA and on recent MPS;
    # the CPU path stays in float32, where bfloat16 kernels are patchy.
    dtype = torch.float32 if device.type == "cpu" else torch.bfloat16
    print(f"device: {device}  dtype: {dtype}")

    tokeniser = AutoTokenizer.from_pretrained(args.model)
    separator_id = tokeniser.eos_token_id

    shard_dir = Path(args.shards).expanduser()
    train_blocks = pack(tokeniser, read_split(shard_dir, "train"), args.seq_len, separator_id)
    val_blocks = pack(tokeniser, read_split(shard_dir, "val"), args.seq_len, separator_id)
    if not train_blocks:
        raise SystemExit("The corpus produced no full-length training blocks; lower --seq-len.")
    print(f"corpus: {len(train_blocks):,} training blocks, {len(val_blocks):,} validation blocks "
          f"of {args.seq_len} tokens")

    model = load_base_model(args.model, dtype).to(device)
    model.config.use_cache = False
    if hasattr(model, "gradient_checkpointing_enable"):
        model.gradient_checkpointing_enable()

    targets = linear_leaf_names(model)
    print(f"adapting {len(targets)} linear module name(s): {', '.join(targets)}")
    lora = LoraConfig(
        r=args.lora_r,
        lora_alpha=args.lora_alpha,
        lora_dropout=args.lora_dropout,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules=targets,
    )
    model = get_peft_model(model, lora)
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    total = sum(p.numel() for p in model.parameters())
    print(f"trainable parameters: {trainable:,} of {total:,} ({100 * trainable / total:.3f}%)")

    optimiser = torch.optim.AdamW(
        [p for p in model.parameters() if p.requires_grad], lr=args.lr, weight_decay=0.0
    )

    def lr_multiplier(step: int) -> float:
        """Linear warm-up then cosine decay, the schedule Part 11's recipe uses."""
        if step < args.warmup_steps:
            return (step + 1) / max(1, args.warmup_steps)
        progress = (step - args.warmup_steps) / max(1, args.max_steps - args.warmup_steps)
        return 0.5 * (1 + math.cos(math.pi * min(1.0, progress)))

    def batches(blocks, batch_size):
        """Endless shuffled batches of blocks, reshuffled every time round."""
        rng = random.Random(args.seed)
        order: list[int] = []
        cursor = 0
        while True:
            if cursor + batch_size > len(order):
                order = list(range(len(blocks)))
                rng.shuffle(order)
                cursor = 0
                if batch_size > len(order):
                    raise SystemExit("--batch-size is larger than the whole corpus.")
            rows = [blocks[i] for i in order[cursor : cursor + batch_size]]
            cursor += batch_size
            yield torch.tensor(rows, dtype=torch.long, device=device)

    @torch.no_grad()
    def evaluate() -> float:
        model.eval()
        total_loss, seen = 0.0, 0
        for i in range(0, min(args.eval_blocks, len(val_blocks)), args.batch_size):
            rows = val_blocks[i : i + args.batch_size]
            if not rows:
                break
            ids = torch.tensor(rows, dtype=torch.long, device=device)
            loss = model(input_ids=ids, labels=ids).loss
            total_loss += loss.item() * len(rows)
            seen += len(rows)
        model.train()
        return total_loss / max(1, seen)

    train_stream = batches(train_blocks, args.batch_size)
    history, started = [], time.time()
    model.train()
    for step in range(args.max_steps):
        for group in optimiser.param_groups:
            group["lr"] = args.lr * lr_multiplier(step)
        optimiser.zero_grad(set_to_none=True)
        step_loss = 0.0
        for _ in range(args.grad_accum):
            ids = next(train_stream)
            loss = model(input_ids=ids, labels=ids).loss
            (loss / args.grad_accum).backward()
            step_loss += loss.item() / args.grad_accum
        torch.nn.utils.clip_grad_norm_([p for p in model.parameters() if p.requires_grad], 1.0)
        optimiser.step()

        if (step + 1) % max(1, args.eval_every) == 0 or step == args.max_steps - 1:
            val_loss = evaluate()
            history.append({"step": step + 1, "train_loss": round(step_loss, 4), "val_loss": round(val_loss, 4)})
            print(f"step {step + 1:5d}/{args.max_steps}  train {step_loss:.4f}  val {val_loss:.4f}")
        elif (step + 1) % 10 == 0:
            print(f"step {step + 1:5d}/{args.max_steps}  train {step_loss:.4f}")

    elapsed = time.time() - started
    final_val = history[-1]["val_loss"] if history else None
    tokens_seen = args.max_steps * args.grad_accum * args.batch_size * args.seq_len

    out_dir = Path(args.out_dir).expanduser()
    model.save_pretrained(str(out_dir))
    tokeniser.save_pretrained(str(out_dir))
    print(f"\nadapter saved to {out_dir}")
    print(f"tokens seen: {tokens_seen:,}   wall clock: {elapsed / 60:.1f} min")
    if final_val is not None:
        print(f"final validation loss: {final_val:.4f}  (perplexity {math.exp(final_val):.2f})")

    if args.labbook:
        import transformers

        record = {
            "run_id": new_run_id(),
            "lab": "part-12/domain-micro-model/continued-pretraining",
            "date": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
            "config_commit": None,
            "model": {"name": args.model, "trained_from": "published base checkpoint", "method": "LoRA"},
            "dataset": {"name": str(shard_dir), "tokens_seen": tokens_seen,
                        "train_blocks": len(train_blocks), "val_blocks": len(val_blocks)},
            "hyperparameters": {
                "max_steps": args.max_steps, "seq_len": args.seq_len, "batch_size": args.batch_size,
                "grad_accum": args.grad_accum, "lr": args.lr, "warmup_steps": args.warmup_steps,
                "lora_r": args.lora_r, "lora_alpha": args.lora_alpha, "lora_dropout": args.lora_dropout,
                "target_modules": targets, "dtype": str(dtype),
            },
            "seed": args.seed,
            "hardware": {"os": f"{platform.system()} {platform.release()}", "arch": platform.machine(),
                         "accelerator": device.type},
            "versions": {"python": platform.python_version(), "torch": torch.__version__,
                         "transformers": transformers.__version__},
            "losses": {"final_val_loss": final_val,
                       "final_val_perplexity": round(math.exp(final_val), 4) if final_val else None,
                       "history": history},
            "scores": {"trainable_parameters": trainable, "total_parameters": total,
                       "minutes": round(elapsed / 60, 2)},
            "notes": {"adapter": str(out_dir),
                      "comparison": "the from-scratch model trained on the same shards"},
        }
        append_run_log(args.labbook, record)
        print(f"recorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download continue-pretraining.py321 lines

RunnableAll tracks

the reference-model run

~/nanochat/.venv/bin/python continue-pretraining.py \
    --shards "$NANOCHAT_BASE_DIR/base_data_climbmix" \
    --model Qwen/Qwen3-1.7B-Base \
    --max-steps 200 --out-dir domain-lora \
    --labbook labbook.md

This is the Part 11 training recipe applied to plain next-token prediction: no chat template, no loss masking, the same shards read with the same train and validation split. It uses LoRA because a full fine-tune of a 1.7-billion-parameter model does not fit the 8 GB floor, and it discovers which linear modules to adapt from the model itself rather than hard-coding names per architecture. Part 13 teaches all of this properly, including when LoRA is the wrong choice.

The script prints how many parameters are trainable. Read that line: it is a fraction of a per cent of the model, and it is the whole reason this run fits on your machine.

7. Score both on the same held-out documents

Fragment — not complete on its own

domain-tasks.json

[
  {
    "prompt": "Sherlock Holmes lives at ",
    "expected": "221B Baker Street",
    "why": "A fact that appears many times in the corpus. If the from-scratch model has it, the corpus taught it; if not, the run was too short."
  },
  {
    "prompt": "The detective's companion and chronicler is Doctor ",
    "expected": "Watson",
    "why": "A single common token. The easiest item in the set, and the one to check first when everything scores badly."
  },
  {
    "prompt": "Holmes turned to me and said, \"My dear ",
    "expected": "Watson",
    "why": "The same fact reached through a stylistic pattern rather than a definition, which is a different thing to have learned."
  },
  {
    "prompt": "It is a capital mistake to theorise before one has ",
    "expected": "data",
    "why": "A memorable line from the corpus. Scoring well here is partly memorisation, which is worth saying out loud in the write-up."
  },
  {
    "prompt": "The capital of France is ",
    "expected": "Paris",
    "why": "A general fact that is probably not in a narrow corpus. The from-scratch model should do badly and the pretrained model should not, which is the whole point of the comparison."
  },
  {
    "prompt": "Water boils at one hundred degrees ",
    "expected": "Celsius",
    "why": "A second out-of-domain fact, so that one lucky guess does not decide the out-of-domain column."
  },
  {
    "prompt": "The opposite of hot is ",
    "expected": "cold",
    "why": "General language rather than general knowledge. Even a small model trained on English prose has a chance here, which separates two kinds of failure."
  },
  {
    "prompt": "If yesterday was Friday, then tomorrow will be ",
    "expected": "Sunday",
    "why": "A small piece of reasoning. Expect both models to fail at this scale; record it anyway, because it is the item that improves last."
  }
]

Download domain-tasks.json42 lines

Edit that file for your domain before running the comparison. Keep the shape: some items whose answers are in your corpus, and some whose answers are general knowledge that a narrow corpus cannot contain. The second group is what turns the table from a scoreboard into an argument.

RunnableAll tracks

compare-models.py

"""Compare a from-scratch model with a fine-tuned pretrained one, on equal terms.

Purpose: the project's evaluation. Two models with different tokenisers and very
         different parameter counts cannot be compared on mean loss, so this scores
         both in bits per byte on exactly the same held-out domain text, and on a
         small task set where the metric is again bits per byte of the expected
         continuation. Both numbers are tokenisation-independent, which is what
         makes the comparison fair rather than merely available.
Platform: all (cuda on Tracks S and N and on Track X with ROCm, mps on Track M, cpu
          anywhere as a slow fallback); the device is autodetected and recorded.
Minimum memory: 8 GB
Assumes: this is run with an interpreter that can import both nanochat (from the
         clone) and transformers and peft; the domain shards exist; a nanochat
         checkpoint exists under $NANOCHAT_BASE_DIR/base_checkpoints/<tag>.

Usage: python compare-models.py --shards ~/.cache/nanochat-domain/base_data_climbmix
                                --nanochat-tag d6-domain
                                [--hf-model Qwen/Qwen3-1.7B-Base]
                                [--adapter domain-lora]
                                [--tasks domain-tasks.json]
                                [--max-val-chars 200000] [--labbook labbook.md]

Bits per byte is summed negative log likelihood in nats, divided by the natural
logarithm of two and by the number of UTF-8 bytes the scored tokens represent. A
model with a larger vocabulary predicts fewer, longer tokens and so wins on mean
loss for free; dividing by bytes removes that advantage entirely.
"""
from __future__ import annotations

import argparse
import json
import math
import os
import platform
import secrets
import sys
from datetime import datetime, timezone
from pathlib import Path

LN2 = math.log(2.0)


# --------------------------------------------------------------------------- #
# Held-out text                                                                #
# --------------------------------------------------------------------------- #

def read_validation_text(shard_dir: Path, max_chars: int) -> str:
    """The validation shard, concatenated, capped so the comparison stays quick."""
    import pyarrow.parquet as pq

    paths = sorted(shard_dir.glob("*.parquet"))
    if len(paths) < 2:
        raise SystemExit(f"Expected at least two parquet shards in {shard_dir}, found {len(paths)}.")
    documents = pq.read_table(paths[-1], columns=["text"]).column("text").to_pylist()
    text = "\n\n".join(documents)
    return text[:max_chars]


# --------------------------------------------------------------------------- #
# Scoring: nanochat                                                            #
# --------------------------------------------------------------------------- #

def nanochat_score_text(model, tokeniser, text: str, window: int) -> tuple[float, int]:
    """Summed nats and scored UTF-8 bytes for a stretch of text."""
    import torch

    ids = tokeniser.encode(text)
    nats, byte_count = 0.0, 0
    for start in range(0, max(0, len(ids) - 1), window):
        chunk = ids[start : start + window + 1]
        if len(chunk) < 2:
            break
        x = torch.tensor([chunk[:-1]], dtype=torch.long, device=model.get_device())
        y = torch.tensor([chunk[1:]], dtype=torch.long, device=model.get_device())
        with torch.no_grad():
            nats += float(model(x, y, loss_reduction="sum").item())
        byte_count += len(tokeniser.decode(chunk[1:]).encode("utf-8"))
    return nats, byte_count


def nanochat_score_continuation(model, tokeniser, prompt: str, expected: str) -> tuple[float, int]:
    """Summed nats and bytes for `expected` given `prompt`, scoring only `expected`."""
    import torch

    prompt_ids = tokeniser.encode(prompt, prepend="<|bos|>")
    expected_ids = tokeniser.encode(expected)
    if not expected_ids:
        return 0.0, 0
    ids = prompt_ids + expected_ids
    x = torch.tensor([ids[:-1]], dtype=torch.long, device=model.get_device())
    y = torch.tensor([ids[1:]], dtype=torch.long, device=model.get_device())
    # Score only the positions that predict the expected continuation.
    mask = torch.full_like(y, -1)
    mask[:, len(prompt_ids) - 1 :] = y[:, len(prompt_ids) - 1 :]
    with torch.no_grad():
        nats = float(model(x, mask, loss_reduction="sum").item())
    return nats, len(expected.encode("utf-8"))


# --------------------------------------------------------------------------- #
# Scoring: a Hugging Face causal model                                         #
# --------------------------------------------------------------------------- #

def hf_score_text(model, tokeniser, text: str, window: int, device) -> tuple[float, int]:
    import torch
    import torch.nn.functional as F

    ids = tokeniser(text, add_special_tokens=False)["input_ids"]
    nats, byte_count = 0.0, 0
    for start in range(0, max(0, len(ids) - 1), window):
        chunk = ids[start : start + window + 1]
        if len(chunk) < 2:
            break
        x = torch.tensor([chunk[:-1]], dtype=torch.long, device=device)
        y = torch.tensor(chunk[1:], dtype=torch.long, device=device)
        with torch.no_grad():
            logits = model(input_ids=x).logits[0].float()
        nats += float(F.cross_entropy(logits, y, reduction="sum").item())
        byte_count += len(tokeniser.decode(chunk[1:]).encode("utf-8"))
    return nats, byte_count


def hf_score_continuation(model, tokeniser, prompt: str, expected: str, device) -> tuple[float, int]:
    import torch
    import torch.nn.functional as F

    prompt_ids = tokeniser(prompt, add_special_tokens=False)["input_ids"]
    expected_ids = tokeniser(expected, add_special_tokens=False)["input_ids"]
    if not expected_ids:
        return 0.0, 0
    ids = prompt_ids + expected_ids
    x = torch.tensor([ids[:-1]], dtype=torch.long, device=device)
    with torch.no_grad():
        logits = model(input_ids=x).logits[0].float()
    target = torch.tensor(expected_ids, dtype=torch.long, device=device)
    scored = logits[len(prompt_ids) - 1 :]
    nats = float(F.cross_entropy(scored, target, reduction="sum").item())
    return nats, len(expected.encode("utf-8"))


# --------------------------------------------------------------------------- #

def bits_per_byte(nats: float, byte_count: int) -> float | None:
    return None if byte_count == 0 else nats / (LN2 * byte_count)


def normalise(text: str) -> str:
    return " ".join(text.lower().split())


def append_run_log(path: str, record: dict) -> None:
    """Append one JSON line in the run-log format Part 11 introduces."""
    notebook = Path(path)
    if not notebook.exists():
        notebook.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with notebook.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")


def new_run_id() -> str:
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--shards", required=True, help="directory of domain parquet shards")
    parser.add_argument("--nanochat-tag", required=True, help="checkpoint tag of the from-scratch model")
    parser.add_argument("--nanochat", default=os.environ.get("NANOCHAT", str(Path.home() / "nanochat")))
    parser.add_argument("--nanochat-step", type=int, default=None)
    parser.add_argument("--hf-model", default="Qwen/Qwen3-1.7B-Base")
    parser.add_argument("--adapter", default=None, help="LoRA directory from continue-pretraining.py")
    parser.add_argument("--tasks", default=None, help="JSON list of {prompt, expected} items")
    parser.add_argument("--max-val-chars", type=int, default=200_000)
    parser.add_argument("--window", type=int, default=None, help="scoring window; default: the model's sequence length")
    parser.add_argument("--max-new-tokens", type=int, default=24)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--device", default="", help="cuda|mps|cpu (empty = autodetect)")
    args = parser.parse_args()

    import torch

    shard_dir = Path(args.shards).expanduser()
    text = read_validation_text(shard_dir, args.max_val_chars)
    print(f"held-out text: {len(text):,} characters, {len(text.encode('utf-8')):,} bytes")

    tasks = []
    if args.tasks:
        tasks = json.loads(Path(args.tasks).read_text(encoding="utf-8"))
        print(f"task set: {len(tasks)} item(s)")

    results: dict[str, dict] = {}

    # ---- the from-scratch model ------------------------------------------
    repo = Path(args.nanochat).expanduser()
    if not (repo / "nanochat").is_dir():
        sys.exit(f"No nanochat package under {repo}. Pass --nanochat or set NANOCHAT.")
    sys.path.insert(0, str(repo))
    from nanochat.checkpoint_manager import load_model
    from nanochat.common import autodetect_device_type, compute_init
    from nanochat.engine import Engine

    device_type = autodetect_device_type() if args.device == "" else args.device
    _, _, _, _, device = compute_init(device_type)
    scratch_model, scratch_tok, scratch_meta = load_model(
        "base", device, phase="eval", model_tag=args.nanochat_tag, step=args.nanochat_step
    )
    window = args.window or int(scratch_meta["model_config"]["sequence_len"])
    print(f"\nscoring the from-scratch model (window {window})")
    nats, byte_count = nanochat_score_text(scratch_model, scratch_tok, text, window)
    scratch = {"val_bpb": bits_per_byte(nats, byte_count), "val_bytes": byte_count, "tasks": []}

    engine = Engine(scratch_model, scratch_tok)
    for item in tasks:
        t_nats, t_bytes = nanochat_score_continuation(
            scratch_model, scratch_tok, item["prompt"], item["expected"]
        )
        prompt_ids = scratch_tok(item["prompt"], prepend="<|bos|>")
        generated, _ = engine.generate_batch(
            prompt_ids, num_samples=1, max_tokens=args.max_new_tokens, temperature=0
        )
        continuation = scratch_tok.decode(generated[0])[len(item["prompt"]) :]
        scratch["tasks"].append({
            "prompt": item["prompt"],
            "expected": item["expected"],
            "bpb": bits_per_byte(t_nats, t_bytes),
            "greedy": continuation,
            "prefix_match": normalise(continuation).startswith(normalise(item["expected"])),
        })
    results["from scratch"] = scratch
    del scratch_model, engine

    # ---- the fine-tuned pretrained model ----------------------------------
    from transformers import AutoModelForCausalLM, AutoTokenizer

    hf_dtype = torch.float32 if device.type == "cpu" else torch.bfloat16
    hf_tok = AutoTokenizer.from_pretrained(args.hf_model)
    try:
        hf_model = AutoModelForCausalLM.from_pretrained(args.hf_model, dtype=hf_dtype)
    except TypeError:
        hf_model = AutoModelForCausalLM.from_pretrained(args.hf_model, torch_dtype=hf_dtype)
    if args.adapter:
        from peft import PeftModel

        hf_model = PeftModel.from_pretrained(hf_model, args.adapter)
    hf_model = hf_model.to(device)
    hf_model.eval()

    label = f"{args.hf_model}{' + adapter' if args.adapter else ' (no adapter)'}"
    print(f"\nscoring {label} (window {window})")
    nats, byte_count = hf_score_text(hf_model, hf_tok, text, window, device)
    reference = {"val_bpb": bits_per_byte(nats, byte_count), "val_bytes": byte_count, "tasks": []}

    for item in tasks:
        t_nats, t_bytes = hf_score_continuation(
            hf_model, hf_tok, item["prompt"], item["expected"], device
        )
        ids = hf_tok(item["prompt"], add_special_tokens=False, return_tensors="pt").to(device)
        with torch.no_grad():
            generated = hf_model.generate(
                **ids, max_new_tokens=args.max_new_tokens, do_sample=False
            )
        continuation = hf_tok.decode(generated[0][ids["input_ids"].shape[1] :], skip_special_tokens=True)
        reference["tasks"].append({
            "prompt": item["prompt"],
            "expected": item["expected"],
            "bpb": bits_per_byte(t_nats, t_bytes),
            "greedy": continuation,
            "prefix_match": normalise(continuation).startswith(normalise(item["expected"])),
        })
    results[label] = reference

    # ---- report -----------------------------------------------------------
    def summarise(entry: dict) -> tuple[float | None, float | None]:
        scores = [t["bpb"] for t in entry["tasks"] if t["bpb"] is not None]
        matches = [t["prefix_match"] for t in entry["tasks"]]
        mean_bpb = sum(scores) / len(scores) if scores else None
        match_rate = sum(matches) / len(matches) if matches else None
        return mean_bpb, match_rate

    print("\n" + "=" * 84)
    print(f"{'Model':<44} {'val bpb':>9} {'task bpb':>9} {'prefix match':>13}")
    print("-" * 84)
    for name, entry in results.items():
        mean_bpb, match_rate = summarise(entry)
        print(
            f"{name[:44]:<44} "
            f"{entry['val_bpb']:>9.4f} "
            f"{(mean_bpb if mean_bpb is not None else float('nan')):>9.4f} "
            f"{(f'{match_rate:.0%}' if match_rate is not None else 'n/a'):>13}"
        )
    print("=" * 84)
    print(
        "Lower bits per byte is better. Both columns are divided by UTF-8 bytes rather\n"
        "than by tokens, so the two tokenisers do not distort the comparison. The\n"
        "parameter counts do: state them next to these numbers, because a comparison\n"
        "between models of different sizes is a statement about what the extra\n"
        "parameters and the extra pretraining bought, not about the method."
    )

    if args.labbook:
        record = {
            "run_id": new_run_id(),
            "lab": "part-12/domain-micro-model/comparison",
            "date": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
            "config_commit": None,
            "model": {"from_scratch": args.nanochat_tag, "reference": args.hf_model,
                      "adapter": args.adapter},
            "dataset": {"name": str(shard_dir), "held_out_characters": len(text),
                        "held_out_bytes": len(text.encode("utf-8"))},
            "hyperparameters": {"window": window, "max_new_tokens": args.max_new_tokens},
            "seed": None,
            "hardware": {"os": f"{platform.system()} {platform.release()}",
                         "arch": platform.machine(), "accelerator": device.type},
            "versions": {"python": platform.python_version(), "torch": torch.__version__},
            "losses": {name: entry["val_bpb"] for name, entry in results.items()},
            "scores": {
                name: {"task_bpb": summarise(entry)[0], "prefix_match": summarise(entry)[1]}
                for name, entry in results.items()
            },
            "notes": {name: entry["tasks"] for name, entry in results.items()},
        }
        append_run_log(args.labbook, record)
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download compare-models.py328 lines

RunnableAll tracks

the comparison

~/nanochat/.venv/bin/python compare-models.py \
    --shards "$NANOCHAT_BASE_DIR/base_data_climbmix" \
    --nanochat-tag d6-domain \
    --hf-model Qwen/Qwen3-1.7B-Base \
    --adapter domain-lora \
    --tasks domain-tasks.json \
    --labbook labbook.md

Output — what you should see

held-out text: <n> characters, <n> bytes
task set: 8 item(s)

Model                                          val bpb  task bpb  prefix match
--------------------------------------------------------------------------------
from scratch                                    <x>       <x>          <n>%
Qwen/Qwen3-1.7B-Base + adapter                  <x>       <x>          <n>%
================================================================================

Both columns are bits per byte, summed negative log likelihood divided by the natural logarithm of two and by the UTF-8 bytes the scored tokens represent. That is the metric nanochat uses internally and its own source explains why: it is “a tokenization vocab size-independent metric, meaning you are still comparing apples:apples if you change the vocab size”. Without it, the model with the larger vocabulary would win on mean loss for reasons that have nothing to do with quality.

Run it once more without --adapter to get the reference model’s score before any training on your corpus. Three rows are far more informative than two: they separate what pretraining gave you from what your fine-tune added.

8. Write the comparison up

Pending validationA domain micro-model against a fine-tuned base model — your results

Model	Parameters	Pretraining tokens	Domain tokens seen	Held-out bits per byte	Task bits per byte, in domain	Task bits per byte, out of domain
From scratch, your corpus only	—	0	—	—	—	—
Qwen3-1.7B-Base, no adaptation	1.7B	36T, per the card	0	—	—	—
Qwen3-1.7B-Base + LoRA on your corpus	1.7B	36T, per the card	—	—	—	—

your machine: track, chip and memory, your operating system and version · nanochat for the from-scratch model; transformers and peft for the reference model the nanochat commit and the torch and transformers versions · as listed per row, BF16 compute where the platform supports it, FP32 on CPU · 512 tokens of context · the date you ran it

Empty on purpose. The pretraining-tokens column is what makes the table honest: one row saw your corpus and nothing else, and two rows arrived having read 36 trillion tokens, which the Qwen3-1.7B-Base card states. Split the task column in two, because the interesting result is almost always that the from-scratch model is competitive in domain and hopeless out of it.

Then write four or five paragraphs. Not a summary of the table: an argument about it. The questions worth answering are these.

Which model is better on your held-out text, and by how much? Bits per byte is a log-scale measure, so a difference of 0.1 is substantial and a difference of 0.01 is probably noise at this scale. Say which you think you have.

Is the comparison fair? It is not, and say why in specific terms rather than as a disclaimer. The parameter counts differ by more than an order of magnitude. One model was trained for twenty minutes and the other arrived having consumed a training budget you could not buy. One had a tokeniser built for your corpus and the other did not, which on a narrow domain is a genuine advantage for the small model and should show up in its favour.

What did the fine-tune actually add? Compare the adapter row with the no-adapter row. If the gap is small, the base model already covered your domain; if it is large, you have measured what adaptation is worth on this corpus.

Where does the from-scratch model win, and why? If it is competitive in domain and far behind out of domain, that is the expected shape and it is worth stating as a finding: a small model trained on a narrow corpus is a specialist, and the specialisation is bought entirely with the generality it does not have.

What would you change with twice the compute? A deeper model, a longer run, or a bigger corpus. The scaling-laws lesson gives you a way to answer this with arithmetic instead of preference.

Make the two training paths answer the same question

Freeze the domain corpus and its document-level split before training either candidate. Keep a provenance and licence record for the material, and inspect boilerplate removal on actual documents. Verify that related editions or variants did not cross the held-out boundary.

The from-scratch model and continued-pretraining model may use different tokenisers. Do not compare their per-token perplexities as though the units were identical. Use the common held-out text and the normalised metric provided by the comparison procedure, plus domain tasks with independent answers. Record each tokeniser identity and the number of processed tokens or bytes.

Keep run directories separate for both paths. For continued pretraining, record the exact starting checkpoint; for training from scratch, record initialisation and architecture. Include compute and preparation costs, not only the final loss. Before writing the report, inspect one failure unique to each candidate and explain what the comparison supports. Preserve the original held-out documents, tokenisers, configurations, checkpoints and task outputs. A domain-fluent sample is useful qualitative evidence, but the project conclusion should rest on the shared evaluation and explicitly state where the two training budgets or representations differ.

Validation

You are done when all of the following are true:

• domain-corpus/sources.tsv has one row per source with a title, a URL and the date you fetched it, and you have written down the copyright position for your jurisdiction;
• a stripped text file opens with the work rather than with a licence header;
• corpus-manifest.json exists, records the seed, and shows a validation split of whole documents;
• at least two tokeniser vocabularies were trained and their bytes-per-token ratios recorded;
• a from-scratch checkpoint exists under ~/.cache/nanochat-domain/base_checkpoints/d6-domain, and you recorded the epoch count the run reached;
• an adapter directory exists and the reference run’s trainable-parameter percentage is recorded;
• the comparison ran three times: from scratch, reference without adapter, reference with adapter;
• labbook.md has run-log lines for the from-scratch run, the reference run and the comparison;
• the benchmark table is filled in and the written comparison exists and names the asymmetries.

Expected outcome

Two models trained on one corpus, a table that compares them in units neither tokeniser can bend, and a piece of writing that says what the table does and does not show. The most likely finding is that the small model is respectable on your own held-out documents and useless on anything else, while the adapted reference model is better on both and answers the out-of-domain items the small one cannot. Both halves of that are results.

=Expected result

Expect the from-scratch model’s samples to be recognisably of your corpus: the names, the cadence, the punctuation habits of your chosen author or subject. Expect them to be factually empty. That combination is the clearest demonstration in the course of what next-token prediction on a small corpus produces, and it is much more persuasive when it is your corpus.

Troubleshooting

Expected at least two parquet shards. The sharding script writes one training shard plus one validation shard, so a corpus small enough to fit in a single shard still produces two files. If you see this, the shards were written somewhere other than where you pointed the reader; check that NANOCHAT_BASE_DIR is exported in the shell you are running from.

The from-scratch run reports epoch: 5 or more. Your corpus is much smaller than the run needs. Either add more sources, shorten the run with TARGET_MINUTES, or accept it and record the number. Beyond a handful of epochs a model this size begins reproducing training text verbatim, which you can check directly by prompting it with the opening of a training document.

The reference model runs out of memory. Lower --seq-len to 256, keep --batch-size at 1, and raise --grad-accum to hold the tokens per step constant. If it still does not fit, use a smaller base model and say so in the write-up; the comparison is between what you ran, not what you intended to run.

hf cannot download the model. The Qwen3-1.7B-Base repository is not gated, so a failure here is network or disk rather than access. Check free space first; a BF16 1.7B checkpoint is several gigabytes.

The comparison script cannot import both nanochat and transformers. Run it with ~/nanochat/.venv/bin/python after installing transformers and peft into that environment, as the requirements section shows. Two separate environments cannot both be active in one process.

The from-scratch model scores better than the reference model on the held-out text. This is a real possibility and not a bug. A tokeniser trained on your corpus compresses it better, and bits per byte rewards a model that is well matched to the text it is scored on. It is also the moment to check that your validation split is genuinely held out: a leak produces exactly this result.

The task-set prefix matches are all zero. Expected for the from-scratch model at this scale. Report the bits-per-byte column instead, which is a graded measure rather than a threshold, and keep the match column as the harder bar that a larger model would clear.

Cleanup

The corpus and the shards are small and worth keeping; the manifest and sources.tsv are what make the run reproducible. The adapter directory is a few tens of megabytes. The largest artefacts are the downloaded reference model in the Hugging Face cache and the from-scratch checkpoints under ~/.cache/nanochat-domain/base_checkpoints.

RunnableAll tracks

what the project cost in disk

du -sh ~/.cache/nanochat-domain/* domain-corpus domain-lora 2>/dev/null

Keep at least the notebook lines. Part 13 revisits fine-tuning with better tools, Part 16 revisits evaluation properly, and the capstone in Part 28 is written from this file.

What you learned

• Provenance is a step, not a footnote. One row per source, written before the download, is the difference between a corpus you can defend and a directory of files.
• Boilerplate is the loudest text in a small corpus. Six identical licence headers in six books are learned better than anything else in them.
• The validation split is whole documents, shuffled first. Any finer split leaks, and a leak flatters every model you compare.
• Vocabulary size is a corpus-dependent choice. What is right for hundreds of millions of tokens is wrong for a few million, and two minutes of tok_eval settles it for your text.
• Bits per byte is what makes a cross-tokeniser comparison possible. Mean loss would have compared vocabularies rather than models.
• A specialist is a model that traded generality away. Your small model is competitive on its domain because it spent everything it had there, which is the same trade every fine-tune in the next three parts makes at a larger scale.

Record in the notebook: the corpus sources and licences; the tokeniser vocabularies tried and their compression ratios; both models’ parameter counts, tokens seen and epochs; the three bits-per-byte rows split into in-domain and out-of-domain; the wall clock and peak memory for both runs; and the written comparison itself, which is the artefact the capstone will quote.

Check your understanding

Question 1. Why does the sharding script shuffle the documents before holding out the validation split, rather than after?

• For speed
• Because splitting first would make the validation set the end of one source document, so the evaluation would measure whether a model can finish that document rather than whether it learned the domain
• Because parquet requires it
• To reduce the file size

Show the answer and why

Answer: Because splitting first would make the validation set the end of one source document, so the evaluation would measure whether a model can finish that document rather than whether it learned the domain

Order in a corpus is meaningful: books end differently from how they begin, and crawls are ordered by date. Shuffle first, split second, and record the seed so the split can be reproduced. The lesson noted the same rule for streamed datasets, where take and skip lock in the shard order and prevent a later shuffle.

Question 2. Your corpus has about five million tokens. Why is a vocabulary of 32,768 probably the wrong choice for it?

• Large vocabularies are always slower
• Because many entries would be learned from a handful of occurrences, and because the embedding and output tables are vocabulary size times width, so on a small model they take most of the parameter budget
• Because byte-pair encoding cannot handle small corpora
• Because the sequence length would have to change

Show the answer and why

Answer: Because many entries would be learned from a handful of occurrences, and because the embedding and output tables are vocabulary size times width, so on a small model they take most of the parameter budget

Two costs at once: undertrained entries and parameters spent on lookup rather than computation. The experiment takes two minutes because tok_eval prints the compression each vocabulary achieves on your own text, so the choice can be measured rather than argued.

Question 3. Both models are scored in bits per byte rather than in mean cross-entropy per token. What would go wrong with the latter?

• Nothing; they are equivalent
• The model with the larger vocabulary predicts fewer, longer tokens, so its mean per-token loss is lower for reasons unrelated to quality; dividing by UTF-8 bytes removes the effect
• Cross-entropy cannot be computed for LoRA models
• The two models have different sequence lengths

Show the answer and why

Answer: The model with the larger vocabulary predicts fewer, longer tokens, so its mean per-token loss is lower for reasons unrelated to quality; dividing by UTF-8 bytes removes the effect

This is the metric nanochat uses internally, and its source says why: bits per byte is vocabulary-size independent, so you are comparing apples with apples across tokenisers. It is the only reason a comparison between these two particular models is possible at all.

Question 4. Your from-scratch model beats the adapted 1.7B model on held-out bits per byte. What should you check before reporting it?

• Nothing, report it
• That the validation split is genuinely held out, since a leak produces exactly this result; and then report it anyway with the explanation that a tokeniser trained on this corpus compresses it better, which bits per byte rewards
• That the LoRA rank was high enough
• That the learning rate was correct

Show the answer and why

Answer: That the validation split is genuinely held out, since a leak produces exactly this result; and then report it anyway with the explanation that a tokeniser trained on this corpus compresses it better, which bits per byte rewards

A surprising result deserves the leak check first, because a leak is the commonest cause and the cheapest to rule out. Once ruled out, the result is real and interesting: a matched tokeniser plus total specialisation can beat far more parameters on the narrow thing they were both scored on, and only on that thing.

Question 5. Why does the write-up have to state the parameter counts and pretraining budgets next to the table?

• Convention
• Because the two models differ by more than an order of magnitude in size and by an unbuyable amount in pretraining tokens, so the numbers describe what those differences bought rather than which method is better
• Because the licence requires it
• To make the table longer

Show the answer and why

Answer: Because the two models differ by more than an order of magnitude in size and by an unbuyable amount in pretraining tokens, so the numbers describe what those differences bought rather than which method is better

A comparison without its asymmetries stated is a claim about method that the experiment does not support. The Qwen3-1.7B-Base card gives 36 trillion pretraining tokens; your from-scratch model saw a few million. Naming that is what turns a scoreboard into a finding.

Check my answersClear