Datasets: Formats, Chat Templates, Tokenisation and Packing
By the end of this lesson you will be able to put a set of examples into the exact shape a trainer expects, apply the model’s own chat template to them, decide deliberately which tokens the loss is computed on, and build a validation split that can still tell you something. You will also know the four ways a dataset silently ruins a fine-tune, because none of them raise an error and all of them are visible before you start.
More fine-tunes are spoiled here than anywhere else in Level 3. A wrong learning rate produces a bad loss curve you can see. A wrong chat template produces a perfectly ordinary loss curve and a model that behaves oddly when served.
From a file to a batch
Section titled “From a file to a batch”What happens to one example between your file and the optimiser
- A line of JSONOne example per line, loaded by datasets with load_dataset("json", data_files=...).
- A recognised shapeTRL reads a "text" field, or "messages", or a "prompt" and "completion" pair. Anything else has to be mapped into one of them.
- The chat templateMessages become a string with the model's own control tokens. Applied by the trainer for conversational data.
- Tokens and a label maskTokenised, truncated to max_length, and labelled: -100 marks every position excluded from the loss.
- A batchPadded, or packed with other examples, and handed to the model.
JSON Lines, and the four shapes
Section titled “JSON Lines, and the four shapes”The course’s training data is JSON Lines: one JSON object per line, which the Datasets loading
guide calls the most efficient JSON format and loads with
load_dataset("json", data_files="my_file.json"). It is a good format for training data for
reasons that have nothing to do with speed: a line is an example, wc -l is a count, head is a
preview, and a diff of two versions is readable.
TRL’s dataset guide separates the format of an example from its type. The format is either
standard, plain text strings, or conversational, lists of messages with role and
content keys. The type is what the example is for. SFTTrainer accepts two of the types:
| Type | Standard format | Conversational format |
|---|---|---|
| Language modelling | {"text": "The sky is blue."} |
{"messages": [{"role": "user", "content": "What color is the sky?"}, {"role": "assistant", "content": "It is blue."}]} |
| Prompt-completion | {"prompt": "The sky is", "completion": " blue."} |
{"prompt": [{"role": "user", ...}], "completion": [{"role": "assistant", ...}]} |
The documentation states that SFTTrainer “is compatible with both standard and conversational
dataset formats” and that “when provided with a conversational dataset, the trainer will
automatically apply the chat template to the dataset”. Parts 14 and 15 need the other types,
preference and prompt-only, and the same page lists which trainer takes which.
The choice between the two rows is not cosmetic, because it decides the default answer to the next question.
Which tokens the loss is computed on
Section titled “Which tokens the loss is computed on”A causal language model is trained to predict the next token. Given a question and its answer
concatenated into one sequence, you can compute the loss over every position, or only over the
answer’s positions. TRL’s documentation is precise about the default: with a prompt-completion
dataset “the trainer computes the loss on the completion tokens only, ignoring the prompt tokens”,
controlled by completion_only_loss, and with a conversational dataset assistant_only_loss=True
“ensures that loss is computed only on the assistant responses, ignoring user or system messages”.
Mechanically, masking is a label of -100. TRL describes the one-token shift and states that
positions with the ignore index “are ignored in the loss computation”, which is the same mechanism
padding uses.
The same example, three ways
When does the choice matter? Train on everything and the model spends capacity learning to produce your questions as well as your answers, which is wasted at best and, on a narrow dataset of similar prompts, actively harmful. Train on completions only and every gradient is about the behaviour you want. For instruction tuning, completions only is nearly always right, and it is what the lab uses.
The chat template
Section titled “The chat template”A chat model was fine-tuned on text with specific control tokens marking the turns. The
Transformers guide puts it plainly: all causal language models continue a sequence of tokens, and
the list of role and content dictionaries you pass “get converted to a token sequence, often with
control tokens like <|user|> or <|assistant|> or <|end_of_message|>”. Different models use
different ones, so each ships a chat template, applied with apply_chat_template.
Three rules follow, and each is a real failure mode.
Use the model’s own template, not one you like. The guide shows the same three-message chat
rendered by two models fine-tuned from the same base into completely different token sequences,
and states that “with the wrong control tokens, these models would have drastically worse
performance”. AutoTokenizer.from_pretrained carries the right one. Where a base model has no
template at all, SFTConfig takes chat_template_path to borrow one, and the lab’s script stops
with a clear message rather than training against nothing.
Do not add generation prompts during training. add_generation_prompt=True appends the tokens
that start an assistant turn, which is what you want when generating and not what you want when
training. The guide’s own training section says to “set add_generation_prompt=False because the
additional tokens to prompt an assistant response aren’t helpful during training”.
Do not tokenise twice. The guide warns that chat templates already include the necessary
special tokens, and that adding more “is often incorrect or duplicated, hurting model performance”;
if you format with tokenize=False and tokenise afterwards, pass add_special_tokens=False.
Letting the trainer apply the template for you avoids the whole question.
Sequence length, truncation and packing
Section titled “Sequence length, truncation and packing”Examples vary in length; a batch is rectangular. Three settings reconcile the two.
max_length truncates. TRL’s memory guide is blunt about the trade: “If max_length is too
small, a significant portion of your tokens will be discarded and won’t contribute to training. If
it’s too large, memory usage can spike”. Look at your data’s length distribution before choosing,
because both failures are silent, one losing the ends of your answers and the other producing an
out-of-memory error at whichever step the longest example lands in.
Padding fills the rest of each row. It costs memory and compute for tokens that are masked out of the loss anyway, which is what the other two settings attack.
Packing groups several examples into one row up to max_length. TRL implements best-fit
decreasing bin packing and offers three strategies: "bfd", the default, which discards overflow;
"bfd_split", which splits long sequences into chunks first so that no tokens are lost; and
"wrapped", which concatenates everything into a stream and cuts fixed blocks, minimising padding
but mixing unrelated examples. The documentation notes that when every sequence is shorter than
max_length, bfd and bfd_split behave identically.
Splits, and the ways they leak
Section titled “Splits, and the ways they leak”Part 1 taught the three-way split: train on one part, choose the checkpoint with the second, spend the third once. That lesson’s warnings apply here unchanged, and instruction data adds two of its own.
train_test_split(test_size=0.1) produces splits and shuffles them by default; shuffle(seed=...)
takes a seed. Both need the seed recorded, or “the validation set” names a different set of
examples on the next run and two loss curves stop being comparable.
Near-duplicates leak. Instruction datasets are full of examples that differ only in a name or a number. A random split puts one in train and its twin in validation, and the validation loss then measures recall rather than generalisation. Deduplicate before splitting, on normalised text rather than exact bytes.
Grouped data leaks through the group. If ten examples came from the same document, the same customer or the same template, split by that group and not by example. Part 13 returns to this when the dataset is one you built yourself and the groups are obvious only in hindsight.
Evaluation sets leak into training corpora. If you fine-tune on data scraped from the same place your benchmark came from, the benchmark stops measuring anything. Part 16 treats decontamination as a first-class step; here it is enough to know the failure exists and that a model scoring suspiciously well is a reason to look at the data rather than to celebrate.
Licence and provenance
Section titled “Licence and provenance”Training data carries obligations, and they are easier to check before the run than to unpick afterwards. Three questions, every time.
What is the licence? A dataset on the Hub documents itself in a dataset card, which the Hub
describes as the repository’s README.md with a YAML metadata block carrying fields including
license, language, tags and task_categories. A dataset with no card and no licence field is
a dataset you do not know the terms of. That is a reason to look elsewhere, not a reason to assume
permission.
Where did it come from? The Hub’s guidance is that a card should help users “understand the contents of the dataset and give context for how the dataset should be used”, including potential biases. A dataset generated by a hosted model may carry terms from that model’s provider as well as from whoever published it. Part 15 raises this again for distillation, where the teacher’s terms are the whole question.
What does it contain? Personal data, credentials and copyrighted text can all be in a corpus that nobody meant to include them in. A model trained on them can reproduce them. Read a sample by hand; a hundred lines is fifteen minutes and it is the only step here that consistently finds things.
This part’s lab avoids all three questions by generating its dataset from a script in the course’s own repository, so the provenance is a file you can read and the licence is the course’s. That is the exception. In Part 13 you will build a dataset of your own, and these three questions are the first section of that page.
Inspect the exact learning signal on one example
Section titled “Inspect the exact learning signal on one example”Take one conversation with a system instruction, user request and assistant answer. Render it with the training template, tokenise it and decode it again. Mark the assistant boundaries and the tokens selected by the loss mask. Padding positions should not accidentally teach the model to emit padding; a completion-only run should not silently train on the prompt.
Now use the longest example and apply the configured truncation. Confirm that the answer, closing delimiters and end-of-turn marker survive where required. A row that contains a valid target before tokenisation can become an unusable training example after truncation.
Finally inspect a packed batch containing two documents. Document delimiters and attention boundaries have different roles: an end-of-document token marks a boundary in the sequence, while masking or sequence-aware attention determines whether one example can attend to another. Check the packer’s actual implementation rather than inferring isolation from a separator token. Save this annotated batch with the dataset revision. It is often the quickest way to diagnose a fine-tune that learns the wrong conversational behaviour.
TRL accepts four shapes: standard or conversational, crossed with language modelling or
prompt-completion, and applies the model’s chat template automatically to conversational data. The
loss is computed on completion tokens only by default for prompt-completion data, and on assistant
messages only when assistant_only_loss is set, which requires generation keywords in the
template. Chat templates must be the model’s own, must not add a generation prompt during training,
and must not be tokenised twice. max_length truncates, padding wastes, and packing recovers the
waste where FlashAttention is available. Splits need a recorded seed, deduplication before
splitting, grouping where the data is grouped, and decontamination against anything you intend to
measure with. And every dataset that is not yours has a licence, a provenance and contents that are
worth ten minutes before the run rather than an afternoon after it.
Check your understanding
Sources for this lesson
7 verified · checked 2026-09-09
- 01TRL documentation — Dataset formats and types§ Overview; Standard; Conversational; Prompt-completion; Which dataset type to usehuggingface.co/docs/trl/main/en/dataset_formats2026-09-09
- 02TRL documentation — SFT Trainer§ Expected dataset type and format; Train on completion only; Train on assistant messages only; Packinghuggingface.co/docs/trl/main/en/sft_trainer2026-09-09
- 03TRL documentation — Reducing memory usage§ Truncation; Packing; Padding-freehuggingface.co/docs/trl/main/en/reducing_memory_usage2026-09-09
- 04Transformers documentation — Chat templates§ Using apply_chat_template; add_generation_prompt; Model traininghuggingface.co/docs/transformers/main/en/chat_templating2026-09-09
- 05Datasets documentation — Load§ JSON; Local and remote fileshuggingface.co/docs/datasets/main/en/loading2026-09-09
- 06Datasets documentation — Process§ Split; Shuffle; Maphuggingface.co/docs/datasets/main/en/process2026-09-09
- 07Hugging Face Hub documentation — Dataset cards§ Dataset card metadatahuggingface.co/docs/hub/datasets-cards2026-09-09
Every technical claim on this page was checked against the official documentation of the tool, vendor or model publisher on the date shown, at the version pinned for the course. Where the course disagrees with folklore, the source is how you can tell which one to trust.