Experiment Tracking and Reproducibility

By the end of this lesson you will be able to set up a training run so that its result still means something in a week: seeded as far as the hardware allows, configured from a file rather than a shell history, checkpointed so an interruption is not a loss, logged so the curve can be looked at again, and recorded in one line that says what produced the number.

This is the least glamorous lesson in Level 3 and the one that saves the most time. Every later part asks you to compare a run against another run. Two numbers are only comparable if you know what differed between them.

Seeds are necessary and not sufficient

Start with the honest limit. PyTorch’s reproducibility page opens with it:

Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms.

Reproducibility is therefore a spectrum rather than a switch, and the same page describes the levers along it.

torch.manual_seed(n) seeds the generator for all devices. That fixes weight initialisation, dropout and shuffling, and it is the single most valuable line in a training script. TRL’s SFTConfig exposes it as seed, defaulting to 42, and a separate data_seed for the data sampling, so that you can change the data order without changing the initialisation or the reverse.

Beyond that the page offers three more levers, each with a cost. torch.use_deterministic_algorithms(True) configures PyTorch “to use deterministic algorithms instead of nondeterministic ones where available, and to throw an error if an operation is known to be nondeterministic”. torch.backends.cudnn.benchmark = False “causes cuDNN to deterministically select an algorithm, possibly at the cost of reduced performance”, and torch.backends.cudnn.deterministic is described as a separate setting that governs the CUDA convolution algorithms themselves. Multi-worker data loading needs worker_init_fn and a generator to preserve reproducibility. The page’s own warning is that “deterministic operations are often slower than nondeterministic operations”.

TRL wraps the strict end of this as full_determinism, which defaults to False.

∴Research note

Do not chase bit-identical runs by default. Turning determinism on slows training, and it buys something you rarely need: two runs that agree to the last digit. What you actually need is that a difference between two runs is explained by something you changed rather than by luck, and the way to get that is to seed, to record, and where a difference is small, to repeat the run with a different seed and see whether it survives. Reserve full determinism for debugging a result you cannot otherwise explain.

There is one more source of difference that no seed touches. PyTorch’s checkpoint documentation warns about moving tensors to a new device inside a checkpointed function:

deterministic output compared to non-checkpointed passes is never guaranteed

Gradient checkpointing is a memory technique, and it can move your numbers in the last decimal places. Record whether it was on.

Configuration files, not shell history

A run is defined by two dozen values. Typing them on a command line means the definitive record of what you ran is your terminal scrollback, which is not a record.

Put them in a file. A small YAML or JSON file, or a shell script whose only job is to call the training script with a fixed set of arguments, all achieve the same three things: the settings can be diffed between runs, they can be committed, and the commit hash becomes a single token that identifies the whole configuration. That is why the run-log format below has a config_commit field rather than a copy of every argument, and why the helper records whether the working tree was dirty when the run started: “commit abc1234” is a lie if the file on disk had uncommitted edits.

The lab keeps this deliberately light: one shell script per run, committed, with the arguments in it. Part 13 introduces Axolotl and LLaMA-Factory, which are configuration-driven by design, and by then the habit is already there.

Checkpoints and resuming

A checkpoint is not a saved model. It is the state needed to continue: the weights, the optimiser’s moments, the learning-rate schedule’s position and the data loader’s place.

What the Trainer writes, and what each part is for

1. 1Every save_strategy intervalA checkpoint directory: model weights, optimiser state, scheduler state, trainer state. Large, because the optimiser state is the biggest term in the memory arithmetic.
2. 2save_total_limitOlder checkpoints are deleted so that a long run does not fill the disk.
3. 3load_best_model_at_endAt the end, the checkpoint chosen by metric_for_best_model is loaded back, which is early stopping expressed as a setting.
4. 4save_modelWrites a model that from_pretrained() can read. For a LoRA run this is the adapter, not the merged model.

save_model writes something you can serve; the checkpoint directory holds what is needed to continue. They are different artefacts and the distinction matters when disk is short.

Resuming is one argument. The Trainer documentation describes resume_from_checkpoint on train(): given a path it loads that checkpoint, given True it loads the last one in the output directory, and “training will resume from the model/optimizer/scheduler states loaded here”. save_model is documented as writing something you can “reload it using from_pretrained()”, and the separate save_state note explains that it exists because save_model does not save the trainer’s own state.

Two habits follow. Set save_total_limit on any run long enough to matter, because checkpoints are several times the size of the model. And use load_best_model_at_end with metric_for_best_model="eval_loss" and greater_is_better=False rather than manually picking a directory afterwards, which is the same early stopping Part 1 did by hand.

Logging the curve

The Trainer emits metrics as it goes; TRL documents the SFT set as including global_step, epoch, num_tokens, loss, entropy, mean_token_accuracy, learning_rate and grad_norm, with evaluation metrics alongside when eval_strategy asks for them. report_to decides where they go, and its default in SFTConfig is "none".

Three destinations are worth knowing.

TensorBoard is the low-friction local choice. Its own documentation describes it as a tool for “tracking experiment metrics like loss and accuracy” and for visualising a model graph, reading event files from a log directory that you point it at with tensorboard --logdir <dir>. Set report_to="tensorboard" and the Trainer writes event files under the run’s output directory. It is the right amount of tool for a single machine.

MLflow is the step up. Its documentation describes MLflow Tracking as “an API and UI for logging parameters, code versions, metrics, and output files”, organised into runs grouped by experiment, with mlflow.log_param() and mlflow.log_metric() as the logging calls. By default “MLflow Tracking logs data to the local mlruns directory”, and mlflow server --port 5000 serves the UI locally. Use it when you have enough runs that comparing them needs a query rather than a memory.

Weights & Biases is the hosted option. Its tracking guide describes logging “metrics, hyperparameters, system metrics, and model artifacts” to an interactive dashboard, with a public API for exporting data back out. That page does not describe a local-only mode. That is not a criticism of the product; it is a fact to weigh, because a training log carries your dataset names, your hyperparameters and your machine’s identity, and this is a course about running things locally. If you use it, decide deliberately.

!Important

A tracking tool is not a record. TensorBoard shows you a curve; it does not tell you which dataset produced it, which commit of the configuration was current, or what the machine was. That is what the run log below is for, and it is why every lab in this course writes one line to a file you own rather than relying on a dashboard.

Loss is not the score

Evaluation loss is cheap, comparable and available every epoch, which is why the trainer computes it and why metric_for_best_model usually points at it. It is also not what you care about.

Loss measures how surprised the model was by the held-out tokens. What you actually want to know is whether the model does the job better: whether it follows the format, answers the question, produces valid JSON, passes the test. Those are different quantities, and they can move in opposite directions. A model can reach a lower evaluation loss by matching the phrasing of your reference answers more closely while getting the substance no better, and a model that learned a rigid format can score beautifully on loss and be worse to use than the base model on anything outside that format.

So log both, from the first run. The Trainer takes a compute_metrics function that receives predictions and labels and returns a dictionary of numbers, which then appear alongside the loss in the log history and in whatever tracking tool you pointed report_to at. Even one crude, task-specific number is worth having: the fraction of validation answers that match your required shape, or that parse as JSON, or that contain the expected keyword.

This is why the run-log format below keeps losses and scores in separate objects. They come from different places and mean different things: loss is what the optimiser minimised, scores are what you would defend to someone who asked whether the fine-tune was worth doing. In this part’s lab scores is empty, honestly, because the lab does not measure the task. Part 10’s evaluation harness and Part 16’s methodology are what fill it in, and every run record written before then has a visible gap where the answer should be. That gap is the point.

∴Research note

When two runs differ, change one thing between them and record both. When the difference is small enough that you are tempted to argue about it, repeat one of the runs with a different seed. If the seed-to-seed spread is as large as the difference you are excited about, you have measured noise, and the run log is what lets you notice that rather than publish it.

The run-log format

Every training lab from here to the capstone appends one JSON line to labbook.md, the notebook started in Part 1. One line, one run, in a file you can grep.

one run record, formatted for reading; in the file it is a single line

{
  "run_id": "20260909T142530Z-9f3c1a",
  "lab": "part-11/train-sft",
  "date": "2026-09-09T14:25:30Z",
  "config_commit": "a1b2c3d",
  "model": "Qwen/Qwen3-0.6B",
  "dataset": {"path": "data/train.jsonl", "sha256": "9cda81d1...", "train_examples": 156,
              "validation_examples": 31},
  "hyperparameters": {"method": "lora", "rank": 16, "alpha": 32, "epochs": 3.0,
                      "batch_size": 2, "grad_accum": 4, "learning_rate": 0.0001,
                      "max_length": 512, "precision": "bfloat16"},
  "seed": 0,
  "hardware": {"os": "Linux 6.14.0", "machine": "x86_64", "accelerator": "cuda",
               "device_name": "NVIDIA GeForce RTX 4090", "python": "3.12.7"},
  "versions": {"torch": "2.14.0", "transformers": "5.16.1", "trl": "1.12.0",
               "peft": "0.20.0", "datasets": "4.7.0"},
  "losses": {"first_train_loss": 2.41, "final_train_loss": 0.62,
             "best_eval_loss": 0.71, "best_epoch": 3.0, "seconds": 214.6},
  "scores": {},
  "notes": null
}

Every field earns its place by answering a question you will actually ask.

• run_id and date sort the file and give you something to refer to in prose.
• config_commit is the whole configuration in seven characters, with -dirty appended when the working tree had uncommitted changes.
• model and dataset.sha256 are the two inputs. The hash is what tells you that two runs read the same bytes, which a filename does not.
• hyperparameters and seed are what you changed between runs.
• hardware and versions are why the same settings gave different numbers on a colleague’s machine.
• losses and scores are the outputs, kept apart because loss is what training minimised and scores are what you actually care about. Part 16 fills scores in properly.

The helper that writes it is fifty lines and ships with this part. Later labs import it or copy it next to their own script.

RunnableAll tracks

runlog.py

"""Append one machine-readable run record per training run to the lab notebook.

Purpose: the course's run log. Every training run from Part 11 onwards appends one
         JSON line to labbook.md describing what was run, on what, with what
         settings, and what came out, so that a result read a week later still
         means something.
Platform: all (pure Python; torch, transformers, trl, peft and mlx are only
          inspected for their version strings if they happen to be installed)
Minimum memory: 8 GB
Assumes: Python 3.10 or newer; the lab notebook from Part 1 exists (it is created
         if it does not); git is optional and only used to record the commit of
         the configuration file.

Usage: imported by the training scripts, or run directly from a shell script:
       python runlog.py --demo --labbook labbook.md
       python runlog.py --record --labbook labbook.md < fields.json

The --record form reads a JSON object with the keys lab, model, dataset,
hyperparameters, seed, losses, scores and notes, and fills in the run id, date,
configuration commit, hardware and package versions itself.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import platform
import secrets
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

# Every field below is required in a run record. A run that cannot fill one in
# writes null rather than leaving the key out, so that a reader always knows the
# difference between "not recorded" and "not applicable".
FIELDS = (
    "run_id",
    "lab",
    "date",
    "config_commit",
    "model",
    "dataset",
    "hyperparameters",
    "seed",
    "hardware",
    "versions",
    "losses",
    "scores",
    "notes",
)


def new_run_id() -> str:
    """A short identifier that sorts by time and does not collide between runs."""
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    return f"{stamp}-{secrets.token_hex(3)}"


def file_sha256(path: str | os.PathLike[str], chunk: int = 1 << 20) -> str | None:
    """Content hash of a dataset file, so a run can be tied to the exact bytes it read."""
    p = Path(path)
    if not p.is_file():
        return None
    digest = hashlib.sha256()
    with p.open("rb") as handle:
        while True:
            block = handle.read(chunk)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def git_commit(path: str | os.PathLike[str] = ".") -> str | None:
    """The commit the configuration is at, or None outside a repository.

    Records whether the tree was dirty, because "commit abc1234" is misleading
    if the file on disk had uncommitted edits when the run started.
    """
    if shutil.which("git") is None:
        return None
    target = Path(path)
    cwd = target if target.is_dir() else target.parent
    try:
        rev = subprocess.run(
            ["git", "rev-parse", "--short", "HEAD"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
        dirty = subprocess.run(
            ["git", "status", "--porcelain"],
            cwd=cwd, capture_output=True, text=True, check=True, timeout=10,
        ).stdout.strip()
    except (subprocess.SubprocessError, OSError):
        return None
    return f"{rev}-dirty" if dirty else rev


def describe_hardware() -> dict[str, Any]:
    """What the run was executed on, as far as it can be established without extra packages."""
    info: dict[str, Any] = {
        "os": f"{platform.system()} {platform.release()}",
        "machine": platform.machine(),
        "python": platform.python_version(),
        "accelerator": "cpu",
        "device_name": None,
    }
    try:
        import torch  # noqa: PLC0415 - optional, and only for reporting
    except ImportError:
        return info
    if torch.cuda.is_available():
        info["accelerator"] = "cuda"
        info["device_name"] = torch.cuda.get_device_name(0)
    else:
        mps = getattr(torch.backends, "mps", None)
        if mps is not None and mps.is_available():
            info["accelerator"] = "mps"
            info["device_name"] = platform.processor() or "Apple silicon"
    return info


def package_versions(names: tuple[str, ...] = ("torch", "transformers", "trl", "peft", "datasets", "mlx")) -> dict[str, str | None]:
    """Version strings for the packages that decide what a run actually did."""
    from importlib.metadata import PackageNotFoundError, version  # noqa: PLC0415

    out: dict[str, str | None] = {}
    for name in names:
        try:
            out[name] = version(name)
        except PackageNotFoundError:
            out[name] = None
    return out


def build_record(
    lab: str,
    model: str,
    dataset: dict[str, Any],
    hyperparameters: dict[str, Any],
    seed: int,
    losses: dict[str, Any] | None = None,
    scores: dict[str, Any] | None = None,
    config_path: str | os.PathLike[str] | None = None,
    notes: str | None = None,
) -> dict[str, Any]:
    """Assemble the record. Kept separate from writing so it can be inspected first."""
    return {
        "run_id": new_run_id(),
        "lab": lab,
        "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "config_commit": git_commit(config_path if config_path is not None else "."),
        "model": model,
        "dataset": dataset,
        "hyperparameters": hyperparameters,
        "seed": seed,
        "hardware": describe_hardware(),
        "versions": package_versions(),
        "losses": losses or {},
        "scores": scores or {},
        "notes": notes,
    }


def append(record: dict[str, Any], labbook: str | os.PathLike[str] = "labbook.md") -> Path:
    """Append one JSON line. Missing keys are an error: a partial record is worse than none."""
    missing = [f for f in FIELDS if f not in record]
    if missing:
        raise ValueError(f"run record is missing required fields: {', '.join(missing)}")
    path = Path(labbook)
    if not path.exists():
        path.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")
    return path


def record(labbook: str | os.PathLike[str] = "labbook.md", **kwargs: Any) -> dict[str, Any]:
    """Build and append in one call; returns the record so a caller can print it."""
    rec = build_record(**kwargs)
    append(rec, labbook)
    return rec


def main() -> None:
    parser = argparse.ArgumentParser(description="Run-log helper for the training labs.")
    parser.add_argument("--demo", action="store_true", help="append an example record and print it")
    parser.add_argument("--record", action="store_true", help="read the run's own fields as JSON on stdin")
    parser.add_argument("--labbook", default="labbook.md")
    args = parser.parse_args()
    if args.record:
        fields = json.load(sys.stdin)
        allowed = {"lab", "model", "dataset", "hyperparameters", "seed", "losses", "scores", "config_path", "notes"}
        unknown = set(fields) - allowed
        if unknown:
            raise SystemExit(f"unknown field(s) on stdin: {', '.join(sorted(unknown))}")
        rec = record(labbook=args.labbook, **fields)
        print(f"recorded run {rec['run_id']} in {args.labbook}")
        return
    if not args.demo:
        print(__doc__)
        print("Fields in every record:", ", ".join(FIELDS))
        return
    rec = record(
        labbook=args.labbook,
        lab="part-11/runlog-demo",
        model="none",
        dataset={"path": None, "sha256": None, "examples": 0},
        hyperparameters={"note": "example record written by runlog.py --demo"},
        seed=0,
        losses={},
        scores={},
        notes="delete this line from labbook.md once you have seen it",
    )
    json.dump(rec, sys.stdout, indent=2, sort_keys=True)
    print()


if __name__ == "__main__":
    main()

Download runlog.py223 lines

Two calls cover every use. From Python:

Fragment — not complete on its own

import runlog

runlog.record(
    labbook="labbook.md",
    lab="part-11/train-sft",
    model="Qwen/Qwen3-0.6B",
    dataset={"path": "data/train.jsonl", "sha256": runlog.file_sha256("data/train.jsonl")},
    hyperparameters={"method": "lora", "rank": 16},
    seed=0,
    losses={"best_eval_loss": 0.71},
    scores={},
    config_path=__file__,
    notes=None,
)

From a shell script, which is how Track M’s MLX run records itself:

RunnableAll tracks

see the format, then delete the demo line

python runlog.py --demo --labbook labbook.md

The helper refuses to write a record that is missing a required field, on the principle that a partial record is worse than none: a line with no hardware in it will be read a month later as though the hardware did not matter.

▷Exercise

Open the lab notebook you started in Part 1 and add the environment details from this part’s second lesson: track, accelerator, precision, and the five library versions. Then run the demo command above once so that there is a record in the file whose shape you have seen. The lab’s script will append a real one in the next page.

Distinguish reproducing an outcome from resuming a run

Reproducing an experiment means recreating its configuration and obtaining results consistent with the original within the declared tolerance. Resuming means continuing from saved training state. Loading the same weights with a fresh optimiser can be a useful new experiment, but it is not the same state transition as restoring moments, schedule, random state and data position.

Practise a short interruption before relying on a long run. Save a checkpoint, restart using the script’s documented resume path and inspect the next global step and learning rate. Confirm that the dataset has not silently changed and that outputs are written into the intended run directory. Do not overwrite the original run’s evidence while testing recovery.

For comparisons, store task-level results beside summary metrics and record unsuccessful runs too. A seed identifies one source of variation; package versions, templates, input ordering and hardware identify others. State whether your acceptance criterion is exact output, close numeric loss or a statistically consistent task score. “Reproducible” becomes useful only when another reader can tell which of those properties was actually checked.

Recap

A seed fixes initialisation, dropout and shuffling, and PyTorch states plainly that it does not make results reproducible across releases, commits or platforms; deterministic algorithms and cuDNN settings buy more at the cost of speed, and gradient checkpointing can itself change the last digits. Put settings in a committed file so that one hash identifies the configuration. A checkpoint holds weights, optimiser state, scheduler and trainer state, is resumed with resume_from_checkpoint, and is not the same artefact as the model that save_model writes. Log the curve to TensorBoard locally, or to MLflow when there are enough runs to query, and treat hosted tracking as a decision rather than a default. And whatever the tool shows you, write one JSON line per run into your own notebook, with the model, the dataset hash, the hyperparameters, the seed, the hardware, the versions, the losses and the date, because that line is the only thing that will still make sense a month from now.

Check your understanding

Question 1. You set torch.manual_seed(0) and get slightly different losses on two machines. Is something broken?

• Yes, the seed should make them identical
• No: PyTorch's reproducibility page says results are not reproducible across releases, commits or platforms
• Yes, the dataset must differ
• No, because seeds only affect the model weights

Show the answer and why

Answer: No: PyTorch's reproducibility page says results are not reproducible across releases, commits or platforms

The seed fixes what the seed can fix. Kernel selection, library versions and hardware differ, which is exactly why the run log records hardware and versions alongside the seed.

Question 2. What does load_best_model_at_end with metric_for_best_model="eval_loss" do?

• Trains until the loss stops falling
• Reloads the checkpoint with the lowest evaluation loss at the end of training, which is early stopping as a setting
• Deletes the other checkpoints
• Averages all the checkpoints

Show the answer and why

Answer: Reloads the checkpoint with the lowest evaluation loss at the end of training, which is early stopping as a setting

It is the same choice Part 1 made by hand: the validation set selects the checkpoint, and the later epochs that fit noise are discarded. save_total_limit is what deletes checkpoints.

Question 3. Why does the run log record the dataset's SHA-256 rather than just its filename?

• To detect corruption
• Because a file with the same name can be edited between two runs, and the hash is what proves two runs read the same bytes
• Because the trainer requires it
• To compress the record

Show the answer and why

Answer: Because a file with the same name can be edited between two runs, and the hash is what proves two runs read the same bytes

Filenames are stable while contents are not. The hash is what turns "the same dataset" from a claim into something checkable, and it costs a second to compute.

Question 4. Which statements about checkpoints are correct? Select all that apply.

• A checkpoint holds optimiser state as well as weights, so it is several times the size of the model
• save_model writes the artefact you serve; for a LoRA run that is the adapter
• resume_from_checkpoint=True loads the last checkpoint in the output directory
• A checkpoint and a saved model are the same thing

Show the answer and why

Answer: A checkpoint holds optimiser state as well as weights, so it is several times the size of the model, save_model writes the artefact you serve; for a LoRA run that is the adapter, resume_from_checkpoint=True loads the last checkpoint in the output directory

They are different artefacts for different purposes. Confusing them is how people end up serving a directory full of optimiser moments, or trying to resume from a saved model that has no scheduler state in it.

Question 5. When is full determinism worth enabling?

• Always, as a default
• When debugging a result you cannot otherwise explain, accepting that deterministic operations are often slower
• Only on CPU
• Never

Show the answer and why

Answer: When debugging a result you cannot otherwise explain, accepting that deterministic operations are often slower

PyTorch warns that deterministic operations are often slower. The everyday goal is that differences between runs are explained by what you changed, which seeding and recording achieve. Bit-identical runs are a debugging tool.

Check my answersClear