PyTorch, Transformers, Datasets and the Hugging Face Ecosystem

By the end of this lesson you will be able to name every library a training recipe in this course is built from, say in one sentence what each is responsible for, and know which one’s documentation to open when a run fails. You will also know how the Hub sits underneath all of them: what a repository, a revision and a gated model are, and where the token that reaches them lives.

This is a map rather than a tutorial. Nothing here is installed until the next lesson and nothing is trained until the lab. The point is that when a stack trace goes twelve frames deep, you can tell which of six projects it is actually complaining about.

Six libraries, one job each

A fine-tuning script is a short piece of code sitting on a tall stack. Read it from the bottom.

What a training script stands on

less code you write

1. Accelerator and driverCUDA on Tracks S and N, ROCm on Track X, Metal on Track M. Part 5 covers what each one is and how to check it.
2. PyTorchTensors, automatic differentiation, optimisers, devices, mixed precision. Everything above is a convenience over this.
3. transformersModel architectures, tokenisers, the Hub loading code, generation, and the Trainer loop.
4. datasetsLoading, streaming, mapping and splitting the examples, backed by memory-mapped Arrow files.
5. peft and trlpeft attaches adapters such as LoRA; trl wraps the Trainer into named post-training methods, starting with supervised fine-tuning.
6. acceleratePlaces tensors on devices and launches the run, on one accelerator or several. Used through the Trainer rather than directly in this course.
7. Your training scriptA configuration, a dataset, a trainer, and a line that saves the result. Fifty lines in the lab.

Each layer uses only the ones below it. A failure usually belongs to exactly one layer, and knowing which one is most of the debugging.

Every training recipe you will meet, in this course and outside it, is some arrangement of those layers. Unsloth, Axolotl and LLaMA-Factory, which Part 13 covers, replace the top layer with a configuration file and change some of the middle for speed; they do not replace the stack.

PyTorch: tensors, gradients and devices

Part 1 introduced tensors and the training loop. Three things about PyTorch matter for the rest of Level 3.

Autograd is a graph that is rebuilt every iteration. The autograd documentation describes a directed acyclic graph whose leaves are the input tensors and whose roots are the outputs, recorded as the forward pass executes, and states plainly that “the graph is recreated from scratch at every iteration”. An operation is recorded only if at least one of its inputs requires gradients, and after backward() only leaf tensors with requires_grad=True have gradients accumulated into their .grad fields. That single sentence explains the whole of LoRA’s memory advantage, which this part’s third lesson works through: freeze the base model’s parameters and the graph stops holding what it would have needed to differentiate them.

Gradients can be switched off, and this is not the same as evaluation mode. The same page describes no-grad mode and inference mode as two ways of excluding operations from the backward graph. The evaluation passes in the lab use them; so does every engine in Level 2, which is why inference memory in Part 4 has no gradient term at all.

A device is a choice you make and print. torch.cuda.is_available(), and torch.backends.mps.is_available() on a Mac, decide where tensors live. Part 1’s script picks between them and prints the answer, and every script in this part does the same. On Track X the ROCm build reports AMD GPUs through the cuda device name, which is a naming artefact and not a mistake.

Mixed precision also lives here. The torch.amp documentation describes torch.autocast as a context manager that runs regions of a script in mixed precision, in torch.float16 or torch.bfloat16, and pairs it with torch.amp.GradScaler when the dtype is float16, because small gradients underflow in that format. With bfloat16 the scaler is not part of the recipe. This is why the course’s training runs ask for BF16 wherever the hardware reports support for it, and fall back to float32 rather than to float16.

transformers: architectures, tokenisers and the Trainer

The transformers library, pinned here at transformers 5.16.1 · verified 2026-09-08, is three things at once.

It is a collection of architectures: the Python that turns a config.json into a model with the right number of layers, heads and dimensions. AutoModelForCausalLM.from_pretrained reads the config, builds the class it names, and loads the weights.

It is a collection of tokenisers, reached through AutoTokenizer.from_pretrained, which carries the vocabulary, the special tokens and the chat template. The next-but-one lesson is entirely about getting that part right.

It is a training loop. The Trainer class is documented as “a simple but feature-complete training and eval loop for PyTorch”, configured by a TrainingArguments object and taking model, args, train_dataset, eval_dataset and processing_class. Its train() method accepts resume_from_checkpoint, evaluate() returns a dictionary containing the evaluation loss, and save_model() writes something from_pretrained() can read back. You will not call Trainer directly in this course, because TRL’s trainers wrap it, but every argument you set is one of its arguments and every log line you read is one of its log lines.

datasets: getting examples in and out

The datasets library loads examples from the Hub or from local files and hands them to the trainer. Three of its behaviours matter.

It loads local JSON Lines directly. The loading guide gives load_dataset("json", data_files="my_file.json") and states that the most efficient format is multiple JSON objects, one per line. That is the shape the lab’s dataset ships in, and it is why the course’s training data is JSON Lines rather than CSV.

It maps, filters and splits without loading everything. map() applies a function to every example, optionally in batches with batched=True and in parallel with num_proc; train_test_split(test_size=...) produces splits and shuffles them by default; shuffle(seed=...) takes a seed. Every one of those is a place where a run becomes irreproducible if the seed is left out, which is the fifth lesson’s subject.

It streams when the data is too large to keep. streaming=True returns an iterable dataset that reads as it goes. You will not need it for a two-hundred-example lab; you will need it in Part 12, where the pretraining corpus is far larger than the machine.

✕Common mistake

map() returns a new dataset rather than changing the one you passed in. The Datasets processing guide warns about exactly this: “All processing methods in this guide return a new Dataset object. Modification is not done in-place.” A tokenisation step whose result is discarded is a common and entirely silent way to train on the wrong text.

peft and trl: adapters and named methods

peft, pinned at PEFT 0.20.0 · verified 2026-09-08, attaches parameter-efficient adapters to a frozen model. Its conceptual guide describes LoRA as representing “the weight updates with two smaller matrices (called update matrices) through low-rank decomposition”, with the original weight matrix frozen. LoraConfig carries r, the rank; lora_alpha, the scaling; lora_dropout; target_modules, the names of the layers to adapt; and task_type. get_peft_model wraps a model and print_trainable_parameters() reports how much of it is now trainable; the developer guide’s own example prints under one per cent. Part 13 teaches the knobs properly. Here, the point is which project owns them.

trl, pinned at TRL 1.12.0 · verified 2026-09-08, wraps the Trainer into named post-training methods. Its SFTTrainer is the one this part uses, and its quick start is genuinely four lines: a model id, a dataset, and trainer.train(). SFTConfig is a subclass of TrainingArguments with SFT’s own settings added and some defaults changed, and it takes a peft_config argument, which is how a LoRA fine-tune becomes a one-line change from a full one. TRL’s release notes for the pinned version list SFT, DPO, KTO, GRPO, RLOO, Reward and Distillation trainers as stable; Parts 14 and 15 use several of the others.

✦Tip

When a training run misbehaves, the layer that owns the symptom is usually obvious once you look for it. Shape and device errors are PyTorch. Tokeniser and chat template surprises are transformers. A column that vanished is datasets. “Target modules not found” is peft. A loss computed on the wrong tokens is trl’s configuration. Out of memory is arithmetic, and the third lesson of this part is about that.

accelerate: where the tensors go

accelerate is the layer that places tensors on devices and launches a run across however many accelerators are present. Its documentation describes adding four lines to a plain PyTorch loop, an Accelerator object and a prepare() call, after which the same code runs on any configuration, and launching through accelerate launch {my_script.py}. It also states that it provides “automatic support for mixed-precision training”.

In this course you almost never call it directly, because the Trainer is built on it. It becomes visible in two places: when a single-machine run needs a launch configuration, and in Part 18, where multiple accelerators stop being an implementation detail.

The Hub underneath all of it

Every one of those libraries reaches the same place for weights and data.

What from_pretrained actually does

1. 1Resolve the repositoryAn id such as Qwen/Qwen3-0.6B, or a local path. Organisation and name, exactly as on the Hub.
2. 2Pick a revisionA branch, tag or commit hash. The default is the head of the main branch, which can move between two of your runs.
3. 3Authenticate if requiredA user access token, for private repositories and for gated ones you have been granted.
4. 4Download into the cacheThen load from there. A second run reads the cache and needs no network at all.

The same four steps run for a model, a tokeniser, an adapter and a dataset. Each is a place a run can fail for a reason that has nothing to do with training.

Repositories and revisions. A repository is model files plus a card; a revision is a point in its history. The Datasets loading guide documents a revision parameter taking “tag name, or branch name, or commit hash”, and the hf command-line tool, from huggingface-hub pinned at Hugging Face CLI 1.30.0 · verified 2026-09-08, takes --revision on hf download for the same purpose. Pinning a revision is the difference between a run you can repeat and a run whose base model quietly changed underneath it.

Tokens. The Hub documentation calls these User Access Tokens and describes three roles: fine-grained, scoped to specific resources; read, for downloading private repositories; and write, for pushing. Its own advice is to create one token per application and to prefer fine-grained tokens in production, so that a leak can be contained. hf auth login stores a token on the machine; the HF_TOKEN environment variable is the non-interactive alternative, and the documentation notes that hf auth logout will not log you out if that variable is what is authenticating you.

Gated models. The Hub lets an author require an access request before the files can be downloaded. The documentation is explicit that requesting access “can only be done from your browser”, that approval may be automatic or manual, and that “to download files from a gated model you’ll need to be authenticated”. Several models this course names are gated; when one is, its page says so, and the step is a browser visit followed by hf auth login on the machine that will do the downloading.

⛨Security

trust_remote_code=True tells the loading code to execute Python that ships inside a Hub repository. The Transformers guide on customising models states plainly that a custom model “doesn’t use the same modeling code as a Transformers’ model”, so this flag is required to load one; TRL documents the same flag on SFTConfig as “whether to allow loading models and tokenizers that ship custom Python code from the Hub”, defaulting to False. Leave it false. When a model genuinely needs it, read the repository’s Python first, pin an exact revision so that what you read is what runs, and prefer a machine that holds nothing you would mind losing.

Debug at the boundary between libraries

When a run fails, identify the last representation that was correct. Start with a raw dataset row, then the rendered conversation, token IDs, loss mask, batch tensor shapes, model outputs and optimiser step. Each boundary has an owner: a dataset loader cannot repair an incompatible attention kernel, and an accelerator launcher cannot correct mislabeled examples.

Keep one tiny batch available for inspection. Decode its non-padding tokens and annotate which positions contribute to loss. Confirm that a forward pass returns a finite scalar loss, a backward pass creates gradients for the intended parameters, and an optimiser step changes at least one trainable parameter. Frozen base parameters should remain frozen in an adapter run.

Save the package versions together, because compatibility belongs to the combination. When upgrading, run this small boundary check before a full experiment. If the miniature example works and the full dataset fails, inspect data-dependent shapes, lengths and masks. If it fails before reading data, investigate imports, binary extensions and device support. This is more efficient than changing several libraries and hyperparameters simultaneously.

Recap

A training script stands on six layers: PyTorch for tensors, gradients and devices; transformers for architectures, tokenisers and the training loop; datasets for loading, mapping and splitting; peft for adapters; trl for named post-training methods; accelerate for device placement and launching. Autograd rebuilds its graph every iteration and only records what requires gradients, which is the mechanism behind every memory saving in the next lessons. Mixed precision is torch.autocast, with a gradient scaler needed for float16 and not for bfloat16. Underneath all of it is the Hub, where a repository plus a revision identifies exactly what you loaded, a user access token authorises private and gated repositories, and trust_remote_code is a decision about running someone else’s Python rather than a convenience flag.

Check your understanding

Question 1. A training run fails with "target modules not found". Whose documentation should you open first?

• PyTorch, because it is a tensor problem
• peft, because target_modules is a LoraConfig argument naming the layers to adapt
• datasets, because a column is missing
• accelerate, because the model is on the wrong device

Show the answer and why

Answer: peft, because target_modules is a LoraConfig argument naming the layers to adapt

target_modules belongs to LoraConfig in peft. The names are the model's own module names, so the fix is to print them and pass the ones that exist, which the lab's script has a flag for.

Question 2. Why does freezing the base model reduce training memory so much?

• Because frozen weights are stored in fewer bytes
• Because autograd only records operations whose inputs require gradients, and only leaf tensors requiring gradients accumulate .grad
• Because the model is smaller
• Because the dataset is smaller

Show the answer and why

Answer: Because autograd only records operations whose inputs require gradients, and only leaf tensors requiring gradients accumulate .grad

The autograd documentation states both rules. With the base frozen, there are no gradients or optimiser states for its parameters, and the graph holds much less. The frozen weights themselves still occupy their usual bytes per parameter.

Question 3. Which statements about the Hub are correct? Select all that apply.

• A revision can be a branch name, a tag or a commit hash
• Access to a gated model can be requested from a script
• A read token is enough to download a private repository you have access to
• The default revision is the head of the main branch, which can change between two runs

Show the answer and why

Answer: A revision can be a branch name, a tag or a commit hash, A read token is enough to download a private repository you have access to, The default revision is the head of the main branch, which can change between two runs

The gated-models documentation says requesting access can only be done from your browser; downloading afterwards needs a token. The other three are documented behaviour, and the last is the reason to pin a revision in anything you want to repeat.

Question 4. Your script uses torch.autocast with torch.bfloat16 and no GradScaler. Is that a bug?

• Yes, a scaler is always required with autocast
• No: the scaler exists because small gradients underflow in float16, and bfloat16 keeps the range that avoids it
• Yes, but only on CPU
• No, because autocast does nothing on modern hardware

Show the answer and why

Answer: No: the scaler exists because small gradients underflow in float16, and bfloat16 keeps the range that avoids it

The torch.amp documentation pairs GradScaler with float16 training. BF16 spends its bits on range rather than precision, which is why the course asks for it wherever the hardware reports support and falls back to float32 rather than float16.

Question 5. A tokenisation step ran, but the model still trained on raw text. What is the most likely cause?

• The tokeniser was the wrong one
• The result of dataset.map() was not assigned, because map returns a new dataset rather than modifying in place
• The learning rate was too high
• The GPU was not visible

Show the answer and why

Answer: The result of dataset.map() was not assigned, because map returns a new dataset rather than modifying in place

The Datasets processing guide warns that every processing method returns a new Dataset and does not modify in place. Discarding the return value fails silently, which is what makes it worth memorising.

Check my answersClear