Generating Synthetic Data with a Local Teacher

By the end of this lesson you will be able to get a served model to answer thousands of prompts without your script being the bottleneck, seed those prompts so the resulting dataset is diverse by construction rather than by luck, measure whether it actually is, grade it with a second model without believing the grade more than it deserves, express the whole thing as a Distilabel pipeline if you want to, and say what each of the course’s teachers permits you to do with what it produced.

Sequence-level distillation is one instance of this. The general habit is broader and worth learning once: using a model to make the data that trains a model.

Batching: the difference between an evening and a weekend

The naive generation script sends one request, waits, sends the next. On a 600-prompt seed set with a 30B-class teacher that is a very long evening, and the accelerator is idle for most of it, because one request at a time cannot fill it.

Both serving engines in this course solve this, and they solve it differently.

llama-server allocates a fixed number of slots. Its README documents -np, --parallel N as the “number of server slots (default: -1, -1 = auto)”, and lists parallel decoding with multi-user support among the server’s features. Continuous batching is on by default: -cb, --cont-batching and -nocb, --no-cont-batching control “whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)”. --slots exposes a monitoring endpoint, enabled by default, which is how you check that the concurrency you asked your client for is the concurrency the server is actually using. --threads-http N sets the “number of threads used to process HTTP requests (default: -1)”.

The KV cache is the thing that constrains how many slots you can have. The README documents --kv-unified-per-slot N as a “context limit per parallel slot (default: unset, behavior unchanged)”, and notes that “when set without -c/–ctx-size, the shared KV pool is sized to n_parallel*N”. That is the arithmetic you have to do before choosing a concurrency.

vLLM batches continuously by design and does not ask you to pick a slot count. Its documentation describes offline inference “in your own code using vLLM’s LLM class”, where LLM.generate “Generates completions for the given input prompts” and LLM.chat “Generates responses for a chat conversation”. The offline form takes the whole list at once:

Fragment — not complete on its own

from vllm import LLM, SamplingParams

llm = LLM(model="facebook/opt-125m")
params = SamplingParams(temperature=0)
outputs = llm.generate("Hello, my name is", params)

Hand it a list of a thousand prompts instead of one string and the scheduler decides the batching. That is the fastest route on Tracks S and N, and it is not available on Track M, where vLLM’s mainline GPU path does not run.

✦Tip

This part’s scripts do neither. They send concurrent HTTP requests to an OpenAI-compatible endpoint, because that is the one interface every track has: llama-server on all four, vLLM on two, the mlx-lm server on Track M, and the Part 9 gateway in front of any of them. You set the concurrency with --concurrency and match it to whatever your server can actually run in parallel. The cost is some throughput against the native offline API; the benefit is one script for four tracks.

The memory arithmetic behind your concurrency

Every slot needs its own key-value cache, and the KV cache is what runs out first.

A 30B-class teacher serving eight parallel slots at 4,096 tokens each, on a 128 GB machine

Weights, Q4_K_M

18.6 GB

KV cache, 8 slots x 4,096 tokens at FP16

3.2 GB

Reserved for the operating system and engine

4 GB

Free

102.2 GB

Total

128 GB

Estimate from arithmetic, not a measurement. Weights are the Q4_K_M size in src/data/models.json for Qwen3-30B-A3B. The cache figure is that model's published bytes per token at FP16, 98,304, multiplied by 8 slots of 4,096 tokens; halving the per-slot context or the slot count halves it. The reserve is an allowance for the operating system and the engine's own buffers.

On a machine with this much room the answer is “raise the concurrency until the accelerator is busy”. On the 24 GB tier with a 14B-class teacher the weights alone take about nine gigabytes at Q4_K_M and the same slot arithmetic costs about two and a half more, so four slots is a sensible starting point and eight may not fit alongside a long context. Do the arithmetic from the model reference before you set the number, then check the slots endpoint to see what the server did with it.

✕Common mistake

Asking a client for more concurrency than the server has slots. The extra requests do not fail; they queue. Throughput stops rising, per-request latency climbs, and it looks as though the model got slower. Read the slots endpoint, not the wall clock, to tell a full server from a slow one.

Seeding prompts so the dataset is diverse by construction

Generation is easy. Generating varied data is the hard part, and it is decided before any model runs, by how the prompts were made.

Four strategies, in the order they are worth trying.

Cross a taxonomy. Write down the categories of task you care about, a list of topics, and a list of constraints, then take the cross product. Diversity is then a property of the construction rather than a hope, and every prompt can be traced back to the cell it came from. This is what make-seed-prompts.py in the next lab does:

Fragment — not complete on its own

combinations = []
for category, templates in TEMPLATES.items():
    for template in templates:
        for topic in TOPICS:
            for constraint in CONSTRAINTS:
                combinations.append((category, template, topic, constraint))
rng.shuffle(combinations)

The failure mode is a template fingerprint: every prompt in a cell has the same skeleton, so the teacher answers them in the same shape, and the student learns the shape rather than the task. Use several templates per category and vary the constraint, which is what the cross product above is for.

Seed from your own traffic. The prompts your gateway has actually served are the best seed set there is, because they are the distribution you will be measured on. Part 9’s gateway logs the fact of every call, and its configuration turns message logging off by default, which is the right default on a shared machine and means you have to turn it on deliberately if you want the text.

Expand a small human set. Write thirty prompts yourself, then ask the teacher for more prompts like them. Cheap, and it is where synthetic sets most often collapse: the model produces variations on a theme, and by the two-hundredth the theme is all there is. If you do this, deduplicate hard and look at the result.

Vary the persona and the constraint, not the topic. Twenty topics with one instruction each gives twenty shapes. Five topics with four instructions each gives twenty shapes that also teach the student that the instruction is the thing to follow.

Deduplication, and measuring diversity rather than assuming it

Two prompts that differ by a word produce two answers that differ by a word, and training on both teaches the student that this answer is twice as important as any other. At scale, near-duplicates are how a synthetic set ends up carrying much less information than its row count suggests.

The checks this part’s filter script runs, in order:

1. Exact duplicate completions, after normalising case and punctuation.
2. Duplicate prompts, keeping the first surviving sample, so that one prompt with four accepted samples cannot dominate.
3. Near duplicates, by n-gram containment against everything kept so far: if 80 per cent of a completion’s 13-grams already appear in an earlier one, it is a paraphrase.

Then measure what you have rather than assuming. Three cheap numbers, all of which the filter report records or can be computed from it: the count per category, the ratio of distinct 13-grams to total 13-grams across the set, and the length distribution. A set whose category counts are lopsided will produce a student good at one category. A set whose distinct-n-gram ratio is low is a set of paraphrases. A set whose lengths cluster tightly has taught the student one output length.

!Important

Deduplicate within the generated set and against the seed prompts, and then decontaminate against the evaluation set. Those are three different checks with three different fixes, and doing the first two is not doing the third. Part 13’s dataset lesson introduced the distinction; this part’s challenge is what happens when the third is skipped.

Judging with a second model

Deterministic checks catch format failures and nothing else. For “is this answer any good”, the practical option at home is a second model, and Part 10’s judging lesson already established the discipline. Three rules carry over unchanged.

Never let the model grade itself. Part 10’s judge.py prints a warning when the judge and the model under test are the same, because self-preference makes the number optimistic. In a synthetic data pipeline the temptation is stronger, because the teacher is right there and already loaded. Resist it: a teacher grading its own output will tell you the output is fine.

Swap the order. For head-to-head comparisons, ask twice with the answers reversed and report the flip rate. A high flip rate means the judge is answering the position of the answers rather than their quality, and the comparison means little.

Measure the judge against yourself once. Score twenty examples by hand, compare, and write down the agreement. It takes twenty minutes and it is the only number that tells you whether the judge is measuring your task or its own preferences.

A judge is a filter, not a certificate. Use it to drop the bottom of the distribution, not to claim the top of it is good.

Distilabel, and whether it is worth the dependency

Distilabel is a framework for exactly this pipeline. Its documentation describes it as “the framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers”; its PyPI page describes it as “an AI Feedback (AIF) framework for building datasets with and for LLMs” and, read on 2026-09-09, lists 1.5.3 as the latest release, published 2025-01-28, for Python 3.9 through 3.12.

The shape is a directed pipeline of steps. Its quickstart builds one with a loader and a generation task inside a Pipeline context manager; its components gallery documents LoadDataFromDicts as a step that “loads a dataset from a list of dictionaries and yields it in batches”, TextGeneration as a task whose outputs are generation and model_name and which takes a Jinja2 template and a columns list, and OpenAILLM as a model class whose base_url is “the base URL to use for the OpenAI API requests. Defaults to None”, with a documented example pointing at a local server.

That last detail is what makes it usable here: pointed at your own endpoint, nothing leaves the machine.

RunnableAll tracks

distilabel-pipeline.py

"""A minimal Distilabel pipeline that generates and judges with your own local models.

Purpose: the same generate-then-judge shape as generate-teacher-data.py plus
    judge.py, written as a Distilabel pipeline instead of as two scripts, so you can
    see what the framework buys you and what it costs. Two steps: a teacher answers
    every seed prompt, and a second model grades each answer against a rubric. Both
    point at an OpenAI-compatible endpoint on your own machine, so nothing leaves
    the house.
Platform: all (pure Python over HTTP; the models may be served by any engine on any
    track, or by the Part 9 gateway under two aliases)
Minimum memory: 12 GB on the machine serving the models; the pipeline process itself
    is small
Assumes: Python 3.10 or newer and `pip install distilabel[openai]`. Distilabel is not
    one of the course's pinned tools, so check the version you installed against the
    documentation at https://distilabel.argilla.io/latest/ before relying on any
    behaviour here. A reachable OpenAI-compatible endpoint at --base-url. The seed
    file is JSON Lines with an "id" and a "prompt" on every line, as written by
    make-seed-prompts.py.

Usage: python3 distilabel-pipeline.py --seeds seeds/prompts.jsonl \\
           --base-url http://127.0.0.1:4000/v1 --teacher local/chat \\
           --judge local/chat --out-dir distilabel-out --limit 40
       python3 distilabel-pipeline.py --seeds seeds/prompts.jsonl --print-only

--print-only writes nothing and starts no model; it prints the pipeline definition
so the shape can be read on a machine with nothing installed.
"""

from __future__ import annotations

import argparse
import json
import os
from pathlib import Path

# The judge prompt is a Jinja2 template, which is what TextGeneration's `template`
# attribute takes. `columns` names the input columns the template may refer to.
JUDGE_TEMPLATE = """You grade one answer against a rubric. Reply with JSON only, of the
form {"score": <1-5>, "reason": "<one sentence>"}.

Rubric: the answer must do what the instruction asked, in the shape it asked for,
without adding facts the instruction did not supply.

Instruction:
{{ instruction }}

Answer:
{{ generation }}
"""


def build_pipeline(args: argparse.Namespace):
    """Assemble the pipeline. Imported lazily so --print-only needs nothing installed."""
    from distilabel.models.llms import OpenAILLM
    from distilabel.pipeline import Pipeline
    from distilabel.steps import LoadDataFromDicts
    from distilabel.steps.tasks import TextGeneration

    rows = []
    with open(args.seeds, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if line:
                row = json.loads(line)
                rows.append({"id": row["id"], "instruction": row["prompt"]})
    if args.limit:
        rows = rows[: args.limit]

    # Distilabel's OpenAI client reads OPENAI_API_KEY from the environment. A local
    # server that wants no key still wants the header to exist, so set a placeholder
    # rather than leaving it unset and getting an unhelpful error.
    os.environ.setdefault("OPENAI_API_KEY", args.api_key or "not-needed-locally")

    with Pipeline(name="local-teacher-and-judge") as pipeline:
        load = LoadDataFromDicts(data=rows, batch_size=args.batch_size)

        generate = TextGeneration(
            name="teacher",
            llm=OpenAILLM(
                model=args.teacher,
                base_url=args.base_url,
                generation_kwargs={"temperature": args.temperature,
                                   "top_p": args.top_p,
                                   "max_new_tokens": args.max_tokens},
            ),
            input_batch_size=args.batch_size,
        )

        judge = TextGeneration(
            name="judge",
            llm=OpenAILLM(
                model=args.judge,
                base_url=args.base_url,
                generation_kwargs={"temperature": 0.0, "max_new_tokens": 200},
            ),
            template=JUDGE_TEMPLATE,
            columns=["instruction", "generation"],
            input_batch_size=args.batch_size,
        )

        load >> generate >> judge

    return pipeline, len(rows)


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--seeds", default="seeds/prompts.jsonl")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--teacher", default="local/chat")
    parser.add_argument("--judge", default="local/chat",
                        help="a different and preferably larger model than the teacher; "
                             "a model grading its own answers scores them generously")
    parser.add_argument("--temperature", type=float, default=0.7)
    parser.add_argument("--top-p", type=float, default=0.8)
    parser.add_argument("--max-tokens", type=int, default=768)
    parser.add_argument("--batch-size", type=int, default=8)
    parser.add_argument("--limit", type=int, default=None)
    parser.add_argument("--out-dir", default="distilabel-out")
    parser.add_argument("--print-only", action="store_true",
                        help="print the pipeline shape and exit without importing distilabel")
    args = parser.parse_args()

    if args.print_only:
        print("LoadDataFromDicts(data=[{id, instruction}, ...])")
        print("    >> TextGeneration(name='teacher', llm=OpenAILLM(model, base_url))")
        print("        >> TextGeneration(name='judge', llm=OpenAILLM(...), template=JUDGE_TEMPLATE,")
        print("                          columns=['instruction', 'generation'])")
        print()
        print("Outputs of TextGeneration: 'generation' and 'model_name'.")
        print("The judge step reads the first step's 'generation' column through its template,")
        print("which is the whole reason the two steps can be chained without glue code.")
        print()
        print("Judge template:")
        print(JUDGE_TEMPLATE)
        return

    if args.judge == args.teacher:
        print("warning: the judge and the teacher are the same model. Self-preference bias "
              "makes the scores optimistic; Part 10's judging lesson measures this.")

    pipeline, count = build_pipeline(args)
    print(f"running the pipeline over {count} prompt(s)")
    distiset = pipeline.run(use_cache=False)

    out = Path(args.out_dir)
    out.mkdir(parents=True, exist_ok=True)
    distiset.save_to_disk(str(out))
    print(f"written to {out}")
    print("Read a handful of rows before you train on any of them. A pipeline that runs "
          "without error is not the same as a pipeline that produced usable data, and "
          "filter-and-dedupe.py is still the step that decides which is which.")


if __name__ == "__main__":
    main()

Download distilabel-pipeline.py157 lines

Run it with --print-only on any machine to see the shape without installing anything. What the framework buys you is composition and bookkeeping: steps that pass columns to each other by name, batching, caching between runs, and a dataset object at the end. What it costs is a dependency that is not one of the course’s pinned tools and a second mental model on top of the one you already have.

⌚Version note

Distilabel is not in the course’s version table, so no page here can pin it with the <Validated> component the way it pins TRL 1.12.0 · verified 2026-09-08 or vLLM 0.28.0 · verified 2026-09-08. The version above was read from PyPI on 2026-09-09 and the API details from the documentation on the same date. Check both against whatever you install: this is the one tool in this part whose behaviour the course cannot promise has been checked against a pinned release.

For the labs in this part, two scripts of a hundred lines each do the same work and are easier to debug at three in the morning. Learn the framework when your pipeline has five steps and a cache is saving you real hours; do not adopt it for two.

What the licence says about the outputs

Every model in this course is one you downloaded and ran yourself, so no hosted provider’s terms of service apply. The model’s own licence still does, and the licences differ in a way that matters precisely when you train another model on the outputs.

Teacher	Licence, from the model card on 2026-09-09	What it means for a student trained on its output
Qwen3 family	Apache-2.0	No naming or attribution condition on models trained from outputs.
gpt-oss-20b and gpt-oss-120b	Apache-2.0	The card describes it as a “Permissive Apache 2.0 license” with no copyleft restrictions; no distillation-specific clause.
DeepSeek-R1 and its distilled models	MIT for the repository and the weights	The card is explicit: the series supports “commercial use, allow for any modifications and derivative works, including, but not limited to, distillation for training other LLMs”.
Llama 3.1 8B Instruct	Llama 3.1 Community License	A naming and attribution condition applies to the model you produce. See below.

The Llama condition is the one to read properly, because it is a live obligation rather than a formality. Section 1.b.i of the Llama 3.1 Community License Agreement states:

If you use the Llama Materials or any outputs or results of the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “Llama” at the beginning of any such AI model name.

The same section requires that you “prominently display ‘Built with Llama’ on a related website, user interface, blogpost, about page, or product documentation”, and section 1.b.iii requires the attribution notice “Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.”

Note the trigger: it applies to a model “which is distributed or made available”. A student you train and keep on your own machine is not distributed. One you publish, or serve to other people, is a different question and the clause is not ambiguous about it.

NVIDIA’s own prune-and-distil release shows the condition being honoured: the model card for Llama-3.1-Minitron-4B-Width-Base, read on 2026-09-09, describes a model pruned from Llama-3.1-8B and then trained with distillation on 94 billion tokens, released under the NVIDIA Open Model License Agreement, and named with “Llama” at the front.

⛨Security

Two habits that cost nothing and save arguments. Record the teacher’s model id, its licence name and the licence URL in the run log at generation time, which is what this part’s scripts do automatically. And write the same three things into the student’s own model card before you share it with anyone, including colleagues. A dataset whose provenance nobody recorded is a dataset nobody can clear for use later.

Audit acceptance yield and coverage

Synthetic-data production is a pipeline with a denominator. Record how many prompts were attempted, how many responses arrived, how many parsed, how many passed verification and how many survived deduplication. A final dataset size alone hides a teacher that required many retries or produced repetitive answers.

Tag seed prompts by the behaviours you need: ordinary cases, missing information, conflicting evidence and boundary conditions. Compare coverage before and after filtering. A strict filter can leave only easy examples, improving apparent label quality while removing the cases the student most needs to learn.

Version the teacher checkpoint, prompt, sampling settings, verifier and filter policy with each generation batch. Preserve rejection reasons so you can inspect systematic failure rather than repeatedly generating more of it. Keep evaluation questions outside both prompt generation and acceptance tuning. The goal is a dataset whose labels and coverage are defensible, not simply a large file of fluent text. Teacher quality, filtering quality and student learning remain separate measurements.

Recap

Batching is what makes generation a matter of hours rather than days: llama-server allocates slots with --parallel and batches continuously by default, vLLM schedules a whole list of prompts for you, and either way the KV cache arithmetic sets how many requests can be in flight. Diversity is decided by the seeding, not by the model: cross a taxonomy of categories, topics and constraints, and watch for template fingerprints and for collapse when expanding a small human set. Deduplicate within the set and against the seeds, then decontaminate against the evaluation set, and measure category balance, distinct-n-gram ratio and length spread rather than assuming them. A second model can grade, but never the model under test, always with the order swapped, and only after you have measured its agreement with your own judgement once. Distilabel expresses all of this as a pipeline of named steps and is worth adopting when the pipeline is long, not when it is two scripts. And the teacher’s licence follows its outputs: Apache-2.0 and MIT teachers impose nothing on a student, while the Llama 3.1 Community License requires a “Llama” name prefix, a “Built with Llama” notice and an attribution line on any model you distribute.

Check your understanding

Question 1. You set your client to 16 concurrent requests against llama-server started with the default slot count and see no throughput improvement over 4. What is the first thing to check?

• The model quantisation
• The slots monitoring endpoint, to see how many slots the server actually has and how many are busy: extra client concurrency queues rather than failing
• The network cable
• The temperature setting

Show the answer and why

Answer: The slots monitoring endpoint, to see how many slots the server actually has and how many are busy: extra client concurrency queues rather than failing

Requests beyond the slot count queue silently. Throughput plateaus and latency climbs, which reads as "the model got slower". The server exposes a slots endpoint by default precisely so this is checkable.

Question 2. Why does crossing a taxonomy of categories, topics and constraints beat asking a model for "500 diverse prompts"?

• It is faster
• Diversity becomes a property of the construction, every prompt is traceable to the cell that produced it, and the model is not asked to be diverse, which is the thing it is worst at
• It uses less memory
• It produces longer prompts

Show the answer and why

Answer: Diversity becomes a property of the construction, every prompt is traceable to the cell that produced it, and the model is not asked to be diverse, which is the thing it is worst at

Asking a model for variety tends to produce variations on a theme, and by the two-hundredth the theme is all there is. A cross product cannot collapse that way, and when one cell produces bad teacher output you can find and fix the template rather than deleting examples one at a time.

Question 3. Which three checks are distinct, and doing two of them is not doing the third? Select all that apply.

• Removing duplicates within the generated set
• Removing generated examples that duplicate the seed prompts
• Removing generated examples that overlap the evaluation set
• Removing examples longer than the maximum sequence length

Show the answer and why

Answer: Removing duplicates within the generated set, Removing generated examples that duplicate the seed prompts, Removing generated examples that overlap the evaluation set

The first two protect the dataset from being dominated by repeats. The third protects the number you will report. They have different fixes and different consequences, and a pipeline that does the first two and skips the third produces a result that cannot be reported honestly.

Question 4. You distil a Llama 3.1 8B teacher into your own 1.7B student and publish the student on the Hub. What does the Llama 3.1 Community License require?

• Nothing; the licence covers only the original weights
• A share of any revenue
• "Llama" at the beginning of the model name, a prominent "Built with Llama" notice, and the stated attribution line, because the condition covers models trained on outputs of the Llama Materials that are distributed or made available
• That you also publish the training data

Show the answer and why

Answer: "Llama" at the beginning of the model name, a prominent "Built with Llama" notice, and the stated attribution line, because the condition covers models trained on outputs of the Llama Materials that are distributed or made available

Section 1.b.i names outputs explicitly and the trigger is distribution. NVIDIA's Llama-3.1-Minitron-4B model card is a public example of the naming condition being honoured on a pruned and distilled derivative.

Question 5. Your teacher is also the strongest model on your machine. Should it grade its own generated answers?

• Yes, it is the strongest model available
• No: self-preference bias makes the scores optimistic, which is why Part 10's judge warns when the judge and the model under test are the same
• Yes, if the temperature is 0
• It makes no difference

Show the answer and why

Answer: No: self-preference bias makes the scores optimistic, which is why Part 10's judge warns when the judge and the model under test are the same

A judge is a filter for dropping the bottom of the distribution, and it only works if it is independent of what produced the distribution. If no second model is available, deterministic checks plus reading a sample yourself is a more honest floor than a self-graded score.

Check my answersClear