Building an SFT Dataset

By the end of this lesson you will be able to write a supervised fine-tuning dataset in a shape a trainer accepts, decide how many examples you need and where they come from, remove the duplicates and the overlap with your evaluation set, and split off a piece you never train on. This is the lesson people skip, and skipping it is the single commonest reason a fine-tune produces an impressive number that does not survive contact with a real request.

Part 11 covered the mechanics: JSON Lines, tokenisation, which tokens carry loss, packing. This lesson is about the contents of the file rather than its encoding.

The three shapes, and which to use

TRL accepts two dataset types for supervised fine-tuning, each in two formats, and the documentation is exact about the distinction: the format “refers to how the data is structured, typically categorized as either standard or conversational”, while the type “is associated with the specific task the dataset is designed for”. For a fine-tune, that gives four combinations, of which you will use one.

Pseudocode — not a real command

Standard language modeling      {"text": "The sky is blue."}

Conversational language modeling {"messages": [{"role": "user", "content": "What color is the sky?"},
                                               {"role": "assistant", "content": "It is blue."}]}

Standard prompt-completion       {"prompt": "The sky is", "completion": " blue."}

Conversational prompt-completion {"prompt": [{"role": "user", "content": "What color is the sky?"}],
                                  "completion": [{"role": "assistant", "content": "It is blue."}]}

Use conversational prompt-completion for the work in this part. Two properties make it the right default. The trainer applies the model’s own chat template for you, which the SFT documentation states directly: “When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.” And the loss lands where you want it: “By default, the trainer computes the loss on the completion tokens only, ignoring the prompt tokens”, controlled by completion_only_loss, which is documented as defaulting to true for prompt-completion datasets.

That second property matters more than it sounds. If the loss is computed over the prompt as well, you are teaching the model to generate your questions as much as your answers, and on a small dataset of similar prompts that is a measurable waste of capacity.

The conversational language modeling shape, with a single messages list, is the one to use for a multi-turn dataset where several assistant turns should all carry loss. TRL supports it through assistant_only_loss=True, with a documented caveat: the setting “requires the chat template to include {% generation %} and {% endgeneration %} keywords”, which TRL patches automatically for known families including Qwen3 and which you must check for anything else.

!Important

The system prompt is part of the example. If your served model runs with a system prompt, put the same system prompt in the prompt list of every training example. A model fine-tuned without one and then served with one is being asked to generalise across a change you never trained; a model fine-tuned with one and served without it is worse. This is the same class of fault as a chat template mismatch, and this part’s challenge has a worked case of it.

What a good example looks like

Four properties, and they are all about the relationship between examples rather than about any one of them.

The input is the input as it really arrives. Sloppy, truncated, with the trailing signature still attached, in whatever language and register your users write in. A dataset of carefully groomed prompts trains a model that works on carefully groomed prompts.

The answer is the answer you actually want, written out in full, in exactly the format you will require at serving time. If you want no preamble, no example may have preamble. If you want five hyphen-prefixed lines, every example is five hyphen-prefixed lines. The model learns the distribution you showed it, and a third of your examples starting with “Sure, here is” is a third of a “Sure, here is” you will then be prompting against.

The examples agree with each other. Two examples that answer the same kind of question in two different shapes teach the model that both are acceptable, which is exactly what you were trying to stop. Consistency is worth more than volume, and it is the property that most often fails when several people write examples in parallel.

The hard cases are in there. The input with a field missing. The question that cannot be answered from what was supplied, whose correct answer is a refusal. The one that is nearly a different task. Ten of these are worth a hundred easy ones, because they are where a fine-tuned model’s behaviour is actually decided.

How many

Fewer than people expect, if they agree with each other. Zhou and colleagues fine-tuned a 65-billion-parameter model on 1,000 curated prompts and responses without reinforcement learning and argued from the result that “almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary”.

For the narrow jobs this part teaches, the working figures are smaller still.

What you are changing	Examples that usually suffice	Why
Output format only	200 to 500	The signal is in nearly every token of every answer, so it accumulates fast.
Style and register	300 to 1,000	Same, spread over more variation in content.
Task behaviour in one domain	1,000 to 5,000	The model has to learn which behaviour to select, which varies by input.
New factual knowledge	More than is worth it	The comparison in the first lesson found retrieval consistently outperformed unsupervised fine-tuning for this, and that models struggle to learn new facts this way.

These are starting points rather than measurements. The honest procedure is to build the smallest set you can, train, measure on your Part 10 task file, and only then decide whether more data or different data is the fix. Doubling a dataset that was wrong doubles the wrongness.

Where examples come from

Four sources, in the order they are usually worth trying

1. 1Work you already didTickets you answered, extractions you made by hand, reports you wrote. The input is real and the answer is known to be right because you wrote it.
2. 2Hand-written from a templateYou state the contract once, then write examples against it. This is what the lab in this part does, and it is the fastest way to a consistent format dataset.
3. 3Transformed from an existing artefactA structured file becomes prompt-completion pairs by a script: your database rows, your JSON schemas, your changelog. Correct by construction.
4. 4Generated by a local teacherA larger model you serve yourself writes candidate answers, and you accept, edit or reject each one. Part 15 teaches this properly.

The first two produce the best data and the least of it; the last produces the most and needs the most checking. Real systems mix all four and record which examples came from where.

The fourth source is worth previewing here because readers reach for it first. You already have a larger model running: on a 128 GB machine a 14-billion-parameter model or larger, on a 16 GB machine something in the 8-billion class. Point it at your inputs, ask it for answers in your format, and you have a few hundred examples in an hour. This is knowledge distillation with the teacher’s output used as a training target, and Part 15 teaches it properly, including how to choose the teacher, how to sample, and how to measure how much of the teacher actually transferred.

Two rules make the preview safe to use before you get there. The teacher must be larger or otherwise better than the student at this task, and you should have evidence of that from your Part 10 harness rather than an assumption. And every generated example is a draft. Read them. The ones you reject are the most informative part of the exercise, because a pattern in the rejections is usually a defect in your prompt to the teacher rather than in the teacher.

✕Common mistake

Generating the training data with the model you are about to fine-tune. The examples then contain exactly the behaviour you were trying to change, described in the model’s own words, and the fine-tune reinforces it. If you use a model to draft data, use a different and larger one, and measure the difference between them on your task set first so that “larger” is a finding rather than a hope.

Licence and provenance

Every example in the file has an owner, and a fine-tune is a derivative work of both the base model and the data.

Three questions, answered once and written into the dataset file itself as a header record or into the notebook beside its checksum. Where did this come from? Your own systems, a public dataset, a model’s output, a scraped page. What are its terms? A public dataset’s card states a licence; a model’s output is governed by the licence of the model that produced it, which for the Apache-2.0 reference models in this course is permissive and for others may not be; a scraped page is usually not yours to train on. Does it contain anything that should not leave this machine? Customer names, addresses, credentials, anything you would not paste into a public issue tracker.

That last question has a specific edge in this part. A fine-tune puts its training data into the weights in a diffuse but not erasable way, and a merged model published to a public repository takes whatever was in the data with it. This is the argument for the private repository the export lesson describes, and for scrubbing before training rather than after publishing.

⛨Security

Redact before the data reaches the trainer, not before it reaches the reader. Names, account numbers, keys and internal hostnames should be replaced by stable placeholders in the JSONL file itself, so that the file you checksum in the run log is the file that was safe. A pipeline that redacts at display time leaves the real values in the artefact you are about to distribute.

Deduplication and decontamination

Two different problems that the same script solves, and the second one is the reason this lesson exists.

Duplicates inside the training set waste capacity and quietly reweight your data: an example that appears four times is trained on four times as hard, which is occasionally what you want and usually an accident of how the file was assembled.

Overlap between the training set and the evaluation set destroys the measurement. If a task from your Part 10 file is also a training example, the fine-tuned model has seen the answer, so its score rises, and the rise tells you nothing about anything. This happens by accident far more often than by carelessness: both files are drawn from the same tickets, or both were generated from the same template, or you added a hard case to the training data because it was failing in evaluation.

The check is cheap. Normalise both texts, compare exact hashes for the identical cases, and compare overlapping word n-grams for the paraphrased ones.

RunnableAll tracks

decontaminate.py

"""Check a supervised fine-tuning dataset for duplicates and for overlap with an evaluation set.

Purpose: the honesty check that has to run before any fine-tune is measured. Reads a
         training file and the Part 10 task file, reports exact duplicates inside the
         training data, exact matches against the evaluation set, and near-duplicates
         found by shared word n-grams, and can write a cleaned training file with the
         contaminated examples removed.
Platform: all (standard library only; no model, no accelerator, no network)
Minimum memory: 8 GB, and far less in practice: the whole check is text in memory
Assumes: Python 3.10 or newer. The training file is JSON Lines in any of the shapes
         Part 11's dataset lesson describes: {"prompt", "completion"}, {"messages": [...]}
         or {"text"}. The task file is the Part 10 evaluation set, a JSON object with a
         "tasks" list whose items carry "prompt" and usually "reference".

Usage: python3 decontaminate.py --train data/train.jsonl --tasks my-tasks.json
       python3 decontaminate.py --train data/train.jsonl --tasks my-tasks.json \
           --write-clean data/train-clean.jsonl --labbook labbook.md
       python3 decontaminate.py --train data/train.jsonl --valid data/valid.jsonl \
           --tasks my-tasks.json --n 13 --threshold 0.4 --strict

Two kinds of contamination are reported separately because they have different fixes.
An exact match means the same text is in both files, and the fix is to delete it from
the training data. A high n-gram containment means one text is largely contained in the
other, which is what happens when an evaluation prompt was paraphrased into a training
example, and the fix is a judgement call you have to make by reading the pair.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

WORD_RE = re.compile(r"[a-z0-9]+")


def normalise(text: str) -> list[str]:
    """Lowercase, drop punctuation, split on words. Two texts that differ only in
    formatting have to compare equal, or the check misses the commonest duplicate."""
    return WORD_RE.findall(text.lower())


def ngrams(words: list[str], n: int) -> set[tuple[str, ...]]:
    """The set of word n-grams. Short texts fall back to one gram of the whole text so
    that a two-word answer is still comparable rather than silently empty."""
    if len(words) < n:
        return {tuple(words)} if words else set()
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}


def containment(a: set[tuple[str, ...]], b: set[tuple[str, ...]]) -> float:
    """How much of a is also in b, between 0 and 1. Containment rather than Jaccard,
    because a short evaluation prompt buried inside a long training example is exactly
    the case that matters and Jaccard would score it low."""
    if not a:
        return 0.0
    return len(a & b) / len(a)


def digest(words: list[str]) -> str:
    return hashlib.sha256(" ".join(words).encode("utf-8")).hexdigest()


def record_text(record: dict[str, Any]) -> str:
    """Flatten one training record to the text a comparison should see, whichever of the
    three dataset shapes it is in."""
    if "messages" in record:
        parts = []
        for message in record["messages"]:
            content = message.get("content")
            if isinstance(content, str):
                parts.append(content)
        return "\n".join(parts)
    if "prompt" in record or "completion" in record:
        parts = []
        for key in ("prompt", "completion"):
            value = record.get(key)
            if isinstance(value, str):
                parts.append(value)
            elif isinstance(value, list):
                for message in value:
                    content = message.get("content") if isinstance(message, dict) else None
                    if isinstance(content, str):
                        parts.append(content)
        return "\n".join(parts)
    if isinstance(record.get("text"), str):
        return record["text"]
    return ""


def load_jsonl(path: Path) -> list[dict[str, Any]]:
    records = []
    with path.open("r", encoding="utf-8") as handle:
        for number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{number}: not valid JSON ({exc.msg})") from exc
    return records


def load_tasks(path: Path) -> list[dict[str, str]]:
    data = json.loads(path.read_text(encoding="utf-8"))
    tasks = data.get("tasks") if isinstance(data, dict) else data
    if not isinstance(tasks, list):
        raise SystemExit(f"{path} does not contain a list of tasks; expected the Part 10 task file")
    out = []
    for index, task in enumerate(tasks):
        text = "\n".join(
            str(task[key]) for key in ("prompt", "reference") if isinstance(task.get(key), str)
        )
        out.append({"id": str(task.get("id", f"task-{index:03d}")), "text": text})
    return out


def summarise(text: str, limit: int = 90) -> str:
    flat = " ".join(text.split())
    return flat if len(flat) <= limit else flat[: limit - 1] + "…"


def check_split(
    name: str,
    records: list[dict[str, Any]],
    task_grams: list[tuple[str, set[tuple[str, ...]]]],
    task_hashes: dict[str, str],
    n: int,
    threshold: float,
) -> dict[str, Any]:
    """One split against the evaluation set, plus its own internal duplicates."""
    seen: dict[str, int] = {}
    internal: list[dict[str, Any]] = []
    exact: list[dict[str, Any]] = []
    near: list[dict[str, Any]] = []
    contaminated: set[int] = set()
    empty = 0

    for index, record in enumerate(records):
        text = record_text(record)
        words = normalise(text)
        if not words:
            empty += 1
            continue
        key = digest(words)
        if key in seen:
            internal.append({"index": index, "duplicate_of": seen[key], "text": summarise(text)})
            contaminated.add(index)
            continue
        seen[key] = index

        if key in task_hashes:
            exact.append({"index": index, "task": task_hashes[key], "text": summarise(text)})
            contaminated.add(index)
            continue

        grams = ngrams(words, n)
        worst_task, worst_score = None, 0.0
        for task_id, gram_set in task_grams:
            score = max(containment(grams, gram_set), containment(gram_set, grams))
            if score > worst_score:
                worst_task, worst_score = task_id, score
        if worst_score >= threshold:
            near.append({
                "index": index, "task": worst_task,
                "containment": round(worst_score, 3), "text": summarise(text),
            })
            contaminated.add(index)

    return {
        "split": name,
        "examples": len(records),
        "empty_examples": empty,
        "internal_duplicates": internal,
        "exact_matches": exact,
        "near_duplicates": near,
        "contaminated_indices": sorted(contaminated),
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--train", required=True, help="training file, JSON Lines")
    parser.add_argument("--valid", default=None, help="validation file, checked the same way")
    parser.add_argument("--tasks", required=True, help="the Part 10 evaluation task file")
    parser.add_argument("--n", type=int, default=13,
                        help="n-gram size for the near-duplicate check; 13 is the size the "
                             "decontamination sections of several pretraining papers use")
    parser.add_argument("--threshold", type=float, default=0.5,
                        help="report a pair when either text's n-grams are this fraction "
                             "contained in the other's")
    parser.add_argument("--report", default=None, help="write the full findings here as JSON")
    parser.add_argument("--write-clean", default=None,
                        help="write the training file with every flagged example removed")
    parser.add_argument("--labbook", default=None, help="append one JSON line recording this check")
    parser.add_argument("--strict", action="store_true",
                        help="exit 1 when anything was flagged, for use in a script")
    args = parser.parse_args()

    train_path = Path(args.train)
    tasks_path = Path(args.tasks)
    for path in (train_path, tasks_path):
        if not path.is_file():
            raise SystemExit(f"{path} does not exist")

    tasks = load_tasks(tasks_path)
    task_grams = [(t["id"], ngrams(normalise(t["text"]), args.n)) for t in tasks]
    task_hashes = {digest(normalise(t["text"])): t["id"] for t in tasks}

    train_records = load_jsonl(train_path)
    results = [check_split("train", train_records, task_grams, task_hashes, args.n, args.threshold)]
    if args.valid:
        valid_path = Path(args.valid)
        if not valid_path.is_file():
            raise SystemExit(f"{valid_path} does not exist")
        results.append(check_split("validation", load_jsonl(valid_path), task_grams,
                                   task_hashes, args.n, args.threshold))

    flagged = 0
    print(f"tasks in evaluation set: {len(tasks)}   n-gram size: {args.n}   "
          f"threshold: {args.threshold}")
    for result in results:
        counts = (len(result["internal_duplicates"]), len(result["exact_matches"]),
                  len(result["near_duplicates"]))
        flagged += sum(counts)
        print(f"\n{result['split']}: {result['examples']} example(s), "
              f"{counts[0]} internal duplicate(s), {counts[1]} exact match(es) against the "
              f"evaluation set, {counts[2]} near-duplicate(s)")
        for row in result["exact_matches"][:5]:
            print(f"  EXACT  line {row['index'] + 1} == task {row['task']}: {row['text']}")
        for row in result["near_duplicates"][:5]:
            print(f"  NEAR   line {row['index'] + 1} ~ task {row['task']} "
                  f"(containment {row['containment']}): {row['text']}")
        for row in result["internal_duplicates"][:5]:
            print(f"  DUP    line {row['index'] + 1} repeats line {row['duplicate_of'] + 1}: "
                  f"{row['text']}")

    if args.report:
        Path(args.report).write_text(
            json.dumps({"tasks_file": str(tasks_path), "n": args.n,
                        "threshold": args.threshold, "results": results}, indent=2),
            encoding="utf-8")
        print(f"\nfull findings written to {args.report}")

    if args.write_clean:
        drop = set(results[0]["contaminated_indices"])
        kept = [r for i, r in enumerate(train_records) if i not in drop]
        with Path(args.write_clean).open("w", encoding="utf-8") as handle:
            for record in kept:
                handle.write(json.dumps(record, ensure_ascii=False) + "\n")
        print(f"clean training file: {args.write_clean} "
              f"({len(kept)} kept, {len(train_records) - len(kept)} removed)")

    if args.labbook:
        line = {
            "check": "part-13/decontaminate",
            "date": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
            "train": str(train_path),
            "tasks": str(tasks_path),
            "n": args.n,
            "threshold": args.threshold,
            "flagged": flagged,
            "per_split": [
                {"split": r["split"], "examples": r["examples"],
                 "internal_duplicates": len(r["internal_duplicates"]),
                 "exact_matches": len(r["exact_matches"]),
                 "near_duplicates": len(r["near_duplicates"])}
                for r in results
            ],
        }
        book = Path(args.labbook)
        if not book.exists():
            book.write_text("# Lab notebook\n\n## Results\n\n", encoding="utf-8")
        with book.open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(line, sort_keys=True) + "\n")
        print(f"recorded the check in {args.labbook}")

    if flagged == 0:
        print("\nNothing flagged. The training data and the evaluation set are disjoint at "
              "this n-gram size and threshold, which is what a comparable measurement needs.")
    else:
        print(f"\n{flagged} item(s) flagged. Read them before you train: a number measured on a "
              "contaminated evaluation set is not a measurement.")
    if args.strict and flagged:
        sys.exit(1)


if __name__ == "__main__":
    main()

Download decontaminate.py294 lines

RunnableAll tracks

check a dataset against your Part 10 task set

python3 decontaminate.py \
    --train data/train.jsonl \
    --valid data/valid.jsonl \
    --tasks my-tasks.json \
    --report decontamination-report.json \
    --labbook labbook.md

Output — what you should see

tasks in evaluation set: 24   n-gram size: 13   threshold: 0.5

train: N example(s), a internal duplicate(s), b exact match(es) against the
evaluation set, c near-duplicate(s)
  EXACT  line 41 == task t07: List the five stages of one training-loop iteration…
  NEAR   line 88 ~ task t12 (containment 0.71): Extract the host and port from this…

Three arguments decide what it finds. --n is the n-gram size: thirteen words is a common choice in the decontamination sections of pretraining work, long enough that ordinary English does not collide by chance and short enough to catch a rewritten sentence. --threshold is how much of one text’s n-grams must appear in the other’s before the pair is reported; 0.5 is a starting point, and lowering it finds more and asks you to read more. --write-clean produces a filtered training file, and --strict makes the script exit non-zero so it can gate a script.

∴Research note

Read the near-duplicates rather than deleting them automatically. Each one is a judgement. Sometimes it is genuine contamination and the training example goes. Sometimes it is two different questions that share a long stock phrase, and both should stay. Automatic deletion at a threshold you did not inspect produces a clean report and an unknown dataset, which is worse than a dirty report you understand.

The split you never train on

Three pieces of data with three different jobs, and confusing any two of them produces a number that flatters you.

One dataset, three jobs

Training split

Gradients are computed from this, every epoch

Validation split

Evaluation loss, early stopping, checkpoint choice

Part 10 task set

Scored once per experiment. Never seen by the trainer

TrainingModel selectionMeasurement

The trainer sees the first two. The third is written before training starts and is scored only by the Part 10 harness, once per experiment, which is what makes its number comparable across runs.

The training split is what gradients come from. Nine parts in ten of your data, typically.

The validation split is held out from gradients but not from decisions. The trainer computes an evaluation loss on it after each epoch, and uses that loss to choose which checkpoint to keep and when to stop. That makes it a model-selection instrument rather than an unbiased estimate: by the end of a run with early stopping, you have chosen the checkpoint that did best on this split, so its number is optimistic by construction. One part in ten, and it must be drawn after deduplication, or a duplicate straddling the split leaks the answer.

The Part 10 task set is the measurement. It was written before this fine-tune existed, it lives in a different file, it is decontaminated against the training data by the script above, and it is scored by run-eval.py and judge.py exactly as it was for the base model. It is the only one of the three whose number you should quote.

✕Common mistake

Splitting randomly when the data has groups in it. If your examples come from a hundred tickets with five examples each, a random split puts examples from the same ticket on both sides, and the validation loss measures memorisation of that ticket rather than generalisation to a new one. Split by the group: all five examples from a ticket go to the same side. Part 1’s generalisation lesson makes the general case; this is the version of it you will actually meet.

Review disagreements before generating more examples

Write a short labelling guide containing the desired behaviour, allowed outputs and how to handle uncertainty. Have the same ambiguous cases labelled twice, or independently by two reviewers if available. Resolve disagreements in the guide before multiplying examples synthetically. Generating variations of an inconsistent label policy makes the inconsistency larger.

Group examples by their underlying source case, then split those groups into training, validation and test. Paraphrases of one ticket belong together; separating them randomly gives the model near-copies of evaluation cases. Keep negative examples, missing fields and contradictory evidence represented in every appropriate partition without reusing the same source case.

Inspect the rendered training examples, not only the JSON file. System prompts, tool messages and response boundaries are part of what the model learns. Count usable target tokens after truncation so a large dataset cannot conceal many examples whose answers were cut away. Finally, preserve provenance and licensing with the data. A dataset is ready when its targets and split policy are defensible, not merely when the loader accepts every line.

Recap

TRL takes language-modeling and prompt-completion types in standard and conversational formats; conversational prompt-completion is the default for this part, because the trainer applies the model’s chat template and computes the loss on the completion only. A good example uses the input as it really arrives and the answer you actually want, in a format every other example also uses, with the hard cases and the refusals included. A few hundred consistent examples change a format, a few thousand change a behaviour, and new facts belong in retrieval. Examples come from work you already did, from a template, from a transformation of something structured, or from a local teacher whose output you read; Part 15 teaches the last one properly. Record where the data came from and under what terms, and redact before the trainer sees it. Deduplicate, then check for overlap against the evaluation set with exact hashes and n-gram containment, and read what the check flags rather than deleting it. And keep three pieces of data apart: the training split for gradients, the validation split for choosing a checkpoint, and the Part 10 task set for the only number you quote.

Check your understanding

Question 1. Why does this part default to the conversational prompt-completion shape rather than a single "text" field?

• It trains faster
• The trainer applies the model's own chat template, and the loss is computed on the completion tokens only rather than on your prompts as well
• It is the only shape TRL accepts
• It uses less memory

Show the answer and why

Answer: The trainer applies the model's own chat template, and the loss is computed on the completion tokens only rather than on your prompts as well

Both properties are documented behaviour of SFTTrainer: a conversational dataset gets the chat template applied automatically, and completion_only_loss defaults to true for prompt-completion datasets. Training on the prompt spends capacity on generating your own questions.

Question 2. Your examples come from 100 support tickets, five examples per ticket. You split 90/10 at random and the validation loss looks excellent. What is the problem?

• The split is too small
• Examples from the same ticket are on both sides, so the validation loss partly measures memorisation of a ticket the model also trained on
• The learning rate is too low
• There is no problem

Show the answer and why

Answer: Examples from the same ticket are on both sides, so the validation loss partly measures memorisation of a ticket the model also trained on

Group leakage. Split by ticket so that every example from one ticket lands on the same side, and the validation loss then measures generalisation to a ticket the model has not seen.

Question 3. The decontamination script reports a near-duplicate with containment 0.71 between a training example and one of your evaluation tasks. What should you do?

• Delete every flagged example automatically
• Read the pair and decide: genuine contamination means the training example goes, a shared stock phrase means both stay
• Lower the threshold until nothing is reported
• Ignore it, since it is not an exact match

Show the answer and why

Answer: Read the pair and decide: genuine contamination means the training example goes, a shared stock phrase means both stay

The near-duplicate check is a search tool, not a verdict. Automatic deletion at an uninspected threshold gives you a clean report and a dataset you no longer understand; raising the threshold until the report is empty is the same mistake with extra steps.

Question 4. Which of these belong in a supervised fine-tuning dataset for a format change? Select all that apply.

• Inputs in the state they actually arrive in, including the mess
• Answers written in exactly the format you will require at serving time
• Cases where the correct answer is a refusal because the input does not contain what was asked for
• Answers generated by the same model you are about to fine-tune

Show the answer and why

Answer: Inputs in the state they actually arrive in, including the mess, Answers written in exactly the format you will require at serving time, Cases where the correct answer is a refusal because the input does not contain what was asked for

The first three give the dataset realism, consistency and coverage of the edge. The fourth trains the model on its own current behaviour, which is the behaviour you were trying to change.

Question 5. Why is the validation loss not the number you should quote as the result of a fine-tune?

• It is computed on too few examples
• The trainer uses it to choose the checkpoint and when to stop, so by the end of the run it has been selected on and is optimistic by construction
• It is measured in the wrong units
• It only covers one category

Show the answer and why

Answer: The trainer uses it to choose the checkpoint and when to stop, so by the end of the run it has been selected on and is optimistic by construction

A held-out split that drives decisions is a model-selection instrument, not an unbiased estimate. The Part 10 task set exists in a different file, is decontaminated against the training data, and is scored the same way for the base model and the fine-tune.

Check my answersClear