Reality Check: 'RL Makes Small Models Reason'

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track seeds, wall-clock times and pass@1 figures belong here once the validation pass has run this page on real machines.

?Reality check

The claim under test, in the form it usually arrives: “reinforcement learning makes small models reason”. As written it cannot be wrong, because it does not say which models, on which tasks, measured how, or how much of a change would count. This page turns it into a sentence that a result could contradict, then runs that test on the adapter you trained in the previous lab.

Objective

By the end of this page you will have measured your GRPO run’s effect three ways: on held-out problems of the kind it trained on, on a second family of problems it never saw, and against the variance you get from simply changing the random seed. You will also have spent a similar budget on the simplest alternative, supervised fine-tuning on solutions from a larger model, and compared the two. And you will have written down which of those measurements survived, including the ones you did not want.

The claim is worth taking seriously rather than dismissing. The DeepSeek-R1 abstract makes a strong version of it: “the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories”, with the trained model “surpassing its counterparts trained via conventional supervised learning on human demonstrations” on “verifiable tasks such as mathematics, coding competitions, and STEM fields”. That is a real result at a scale you do not have. The question this page asks is what survives at yours.

⌂Single-machine path

Single machine on every track. The training runs are unattended; the attended work is about forty-five minutes of evaluating, arithmetic and writing.

The method

From a slogan to something that could be false

1. 1State the claim as people say itWrite the vague version down so you can see which words are doing no work: "reason", "small", "makes".
2. 2Name the model, the tasks and the marginWhich model, which held-out set, which metric, and how large a change counts. Decide the margin before you look at any result.
3. 3Measure the noise firstThree seeds of the same recipe. The spread between them is the floor below which no difference is a finding.
4. 4Test the same kind of taskHeld-out problems from the family the run trained on. This is the easiest thing for the claim to pass.
5. 5Test a different kind of taskA second family the run never saw. This is where the word "reason" is doing the work, and where the claim usually breaks.
6. 6Run the obvious alternative on the same budgetSupervised fine-tuning on solutions from a larger model, given the same wall clock. If it wins, the interesting question is not whether RL worked.
7. 7Report all of itIncluding the seed spread, including the different-kind result, including the baseline that beat you.

The refutable version

Here is the claim rewritten so that a result could contradict it:

Group relative policy optimisation on 240 generated arithmetic word problems, run for 120 steps on Qwen3-1.7B with a rank-16 adapter, raises pass@1 on 60 held-out problems of the same family by more than the spread across three random seeds; and raises pass@1 on 60 held-out problems of a different family, counting, calendar and ordering puzzles the run never saw, by more than that same spread. Both gains exceed what supervised fine-tuning on a larger model’s correct solutions achieves in the same wall-clock time.

Every clause is now doing work. The model and the recipe are named. The two held-out sets separate “got better at the thing it practised” from anything that deserves the word “reason”. The seed spread is the noise floor, decided by measurement rather than by hope. And the baseline is the alternative you would actually reach for, given the same budget.

∴Research note

Sixty problems per family is a small sample and this page will say so more than once. It can detect a large difference and not a small one. The standard error eval-pass-at-1.py prints is the honest bound; the seed spread is a second, usually larger one, because it captures variation the standard error does not. When the two disagree, believe the larger.

Requirements

You need the previous lab finished: tasks/, runs/grpo-…, eval-before.json and the run-log lines. You need make-tasks.py, train-grpo.py, eval-pass-at-1.py and rewards.py from that lab, train-sft.py from Part 11, and distil-baseline.py below. And you need a served teacher model larger than the one you trained.

Budget honestly. Two extra GRPO runs at the previous lab’s settings are the long pole and they are unattended; the distillation baseline is bounded by a wall-clock argument you pass to it. The forty-five minutes on this page is the evaluating and the writing.

Track S — NVIDIA DGX Spark

Run all three seeds at the lab’s primary settings and serve qwen3-8b as the distillation teacher alongside; the 128 GB pool holds the trainer and the teacher at once, so the two halves of this page can overlap. Qwen3-8B’s card gives an Apache-2.0 licence, as does the model you trained.

Track X — AMD Ryzen AI Max+ 395Partial

Training uses the in-process rollout path on ROCm or the CPU, as in the previous lab; the ROCm and vLLM status for this chip is unchanged from that page.

Three seeds at the reduced settings, run overnight. Serve the teacher separately rather than alongside, since the trainer will want the accelerator. qwen3-8b at Q4_K_M fits in the tier; qwen3-4b is an acceptable teacher for a 1.7B student if memory is tight, and being only somewhat larger is a limitation to write down rather than to hide.

Track M — Apple siliconPartial

float32 on the MPS backend with in-process rollouts, as in the previous lab. Three full seeds may not be practical inside a day.

Two seeds rather than three is an acceptable reduction here, and it must be stated in the report: two runs give a range rather than a spread, and a range from two samples is weak evidence. Serve the teacher with the mlx-lm server from Part 8 or with llama.cpp, and run the distillation baseline while the second seed trains.

Track N — NVIDIA desktop or laptop

Three seeds at the lab’s settings, run unattended. Serve the teacher on the same card between training runs rather than during them: a 16 GB card will not hold both, and a swap-thrashing teacher will distort the wall-clock budget the comparison depends on.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-14-preference-and-rl"
cd "$LAB_DIR"
pwd
test -f "distil-baseline.py"

Expected result: pwd ends in part-14-preference-and-rl and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Write the margin down before anything else

Open the notebook and write two sentences: the pass@1 change on the same-kind held-out set that you will call a real effect, and the same for the different-kind set. Decide them now, while you do not know the answer.

A reasonable pre-registration, given sixty problems and the standard error the previous lab printed: a change counts if it is larger than both the standard error and the spread across seeds. Write down the actual numbers once Task 3 has produced them, and do not revise the rule afterwards.

✕Common mistake

The commonest failure on a page like this is not a bad measurement. It is deciding what counts as a difference after seeing the difference. The rule chosen in advance is what makes the test capable of failing, and moving it afterwards converts a measurement back into an opinion with numbers attached.

2. Train two more seeds

The previous lab’s run is seed 0. Two more of exactly the same recipe, changing only --seed, give you the noise floor.

RunnableAll tracks

two more seeds, identical in every other respect

python3 train-grpo.py --tasks-dir tasks --model Qwen/Qwen3-1.7B \
  --output-dir runs/grpo-seed1 --rollouts inproc --num-generations 8 \
  --batch-size 8 --grad-accum 4 --max-steps 120 --beta 0.04 --lr 1e-5 \
  --seed 1 --curves curves-seed1.csv --labbook labbook.md
python3 train-grpo.py --tasks-dir tasks --model Qwen/Qwen3-1.7B \
  --output-dir runs/grpo-seed2 --rollouts inproc --num-generations 8 \
  --batch-size 8 --grad-accum 4 --max-steps 120 --beta 0.04 --lr 1e-5 \
  --seed 2 --curves curves-seed2.csv --labbook labbook.md

Change nothing else. Not the model, not the steps, not the rollout path. The point of this step is to find out how much the result moves when nothing that matters has changed, and any second difference destroys that.

✱Hardware note

On Tracks S and N substitute --rollouts vllm-colocate to match the previous lab, and on Track M use the reduced settings from that page with --precision fp32. Whatever you use, use the same thing for all the seeds, and record it.

3. Measure all three seeds on both held-out families

Six evaluations: three adapters, two task files.

RunnableAll tracks

same-kind held-out problems, three seeds

for SEED in 0 1 2; do
  python3 eval-pass-at-1.py --tasks tasks/heldout-same.jsonl \
    --model Qwen/Qwen3-1.7B --adapter runs/grpo-seed${SEED} \
    --limit 60 --label "after-same-seed${SEED}" \
    --out eval-same-seed${SEED}.json --labbook labbook.md
done

RunnableAll tracks

different-kind held-out problems, three seeds

for SEED in 0 1 2; do
  python3 eval-pass-at-1.py --tasks tasks/heldout-different.jsonl \
    --model Qwen/Qwen3-1.7B --adapter runs/grpo-seed${SEED} \
    --limit 60 --label "after-diff-seed${SEED}" \
    --out eval-diff-seed${SEED}.json --labbook labbook.md
done

The seed-0 directory is the adapter from the previous lab; rename it or adjust the loop to match whatever you called it. You also need the untrained baseline on both families, which the previous lab produced for one of them:

RunnableAll tracks

the starting model on both families

python3 eval-pass-at-1.py --tasks tasks/heldout-same.jsonl \
  --model Qwen/Qwen3-1.7B --limit 60 --label before-same \
  --out eval-before-same.json --labbook labbook.md
python3 eval-pass-at-1.py --tasks tasks/heldout-different.jsonl \
  --model Qwen/Qwen3-1.7B --limit 60 --label before-diff \
  --out eval-before-diff.json --labbook labbook.md

4. Compute the spread

RunnableAll tracks

the three seeds, and the spread between them

python3 -c "
import json, statistics
for kind in ('same', 'diff'):
    base = json.load(open(f'eval-before-{kind}.json'))['scores']['pass_at_1']
    runs = [json.load(open(f'eval-{kind}-seed{s}.json'))['scores']['pass_at_1'] for s in (0, 1, 2)]
    spread = max(runs) - min(runs)
    print(f'{kind:5s} before {base:.3f}  seeds {[round(r, 3) for r in runs]}'
          f'  mean {statistics.mean(runs):.3f}  spread {spread:.3f}'
          f'  gain {statistics.mean(runs) - base:+.3f}')
"

Output — what you should see

same  before 0.xxx  seeds [0.xxx, 0.xxx, 0.xxx]  mean 0.xxx  spread 0.0xx  gain +0.0xx
diff  before 0.xxx  seeds [0.xxx, 0.xxx, 0.xxx]  mean 0.xxx  spread 0.0xx  gain +0.0xx

Compare each gain with its spread and with the standard error printed by the evaluation script. A gain smaller than either is not a finding, and saying so is the whole reason this page exists.

=Expected result

Two patterns are common enough to prepare for. The same-kind gain usually exceeds the different-kind gain, often by a lot, because practising a family of problems raises performance on that family first. And on a 120-step run the spread across seeds is frequently comparable to the gain itself, which is a result about how much a short run is worth rather than a failure of the method.

5. Spend the same budget on the obvious alternative

Reinforcement learning discovers correct answers by sampling. Distillation is told them by a larger model. Part 15 teaches distillation properly; this is the one-page version, used here as a control.

RunnableAll tracks

distil-baseline.py

"""Build the distillation baseline the reality check compares reinforcement learning against.

Purpose: generate solutions to the same training problems with a larger local teacher,
    keep only the ones the verifier says are correct, and write them in the shape Part 11's
    train-sft.py reads. The point is a fair comparison: the reality check asks whether
    GRPO's gain is worth its budget, and the honest way to answer that is to spend a
    similar budget on the simplest alternative and measure both. Part 15 teaches
    distillation properly; this is the one-page version of it, used as a control.
Platform: all (pure Python over HTTP; the teacher runs on whatever engine your track uses)
Minimum memory: 16 GB on the machine serving the teacher; this script needs very little
Assumes: Python 3.10 or newer; an OpenAI-compatible endpoint serving a larger model than
    the student, such as llama-server from Part 6 or the gateway from Part 9;
    make-tasks.py has written the training problems; rewards.py and runlog.py sit next
    to this file.

Usage: python3 distil-baseline.py --tasks tasks/train.jsonl --base-url http://127.0.0.1:8080/v1 \
           --teacher-model qwen3-8b --out-dir distil-data --labbook labbook.md
       python3 distil-baseline.py --tasks tasks/train.jsonl --base-url http://127.0.0.1:8080/v1 \
           --teacher-model qwen3-8b --attempts 2 --budget-seconds 1800 --out-dir distil-data
"""

from __future__ import annotations

import argparse
import hashlib
import json
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Optional

import rewards as reward_lib
import runlog


def post_json(url: str, payload: dict, api_key: Optional[str], timeout: int) -> dict:
    body = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"
    request = urllib.request.Request(url, data=body, headers=headers, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", "replace")[:400]
        raise RuntimeError(f"{url} returned HTTP {exc.code}: {detail}") from exc
    except urllib.error.URLError as exc:
        raise RuntimeError(f"cannot reach {url}: {exc.reason}") from exc


def write_jsonl(path: Path, rows: list[dict]) -> str:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--tasks", default="tasks/train.jsonl", help="the same problems GRPO trained on")
    parser.add_argument("--base-url", default="http://127.0.0.1:8080/v1")
    parser.add_argument("--teacher-model", required=True, help="a larger model than the student")
    parser.add_argument("--api-key", default=None)
    parser.add_argument("--attempts", type=int, default=1,
                        help="samples per problem; extra attempts recover problems the teacher got wrong once")
    parser.add_argument("--temperature", type=float, default=0.7)
    parser.add_argument("--max-tokens", type=int, default=512)
    parser.add_argument("--budget-seconds", type=float, default=None,
                        help="stop generating after this long, so the comparison is at equal wall clock")
    parser.add_argument("--valid-fraction", type=float, default=0.1)
    parser.add_argument("--tolerance", type=float, default=1e-6)
    parser.add_argument("--keep-wrong", action="store_true",
                        help="keep solutions the verifier rejects; off by default, and the lesson says why")
    parser.add_argument("--out-dir", default="distil-data")
    parser.add_argument("--timeout", type=int, default=300)
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--labbook", default=None)
    parser.add_argument("--notes", default=None)
    args = parser.parse_args()

    rows = [json.loads(line) for line in Path(args.tasks).read_text(encoding="utf-8").splitlines() if line.strip()]
    print(f"{len(rows)} problems; teacher {args.teacher_model} at {args.base_url}")

    endpoint = args.base_url.rstrip("/") + "/chat/completions"
    grade = reward_lib.numeric_reward(args.tolerance, fallback_to_last_number=False)

    kept: list[dict] = []
    attempted = 0
    correct = 0
    teacher_completion_tokens = 0
    started = time.time()
    stopped_early = False

    for i, row in enumerate(rows, start=1):
        if args.budget_seconds and time.time() - started > args.budget_seconds:
            stopped_early = True
            print(f"budget of {args.budget_seconds:.0f} s reached after {i - 1} problems")
            break
        for attempt in range(args.attempts):
            payload = {
                "model": args.teacher_model,
                "messages": row["prompt"],
                "temperature": args.temperature,
                "max_tokens": args.max_tokens,
                "seed": args.seed + attempt,
            }
            body = post_json(endpoint, payload, args.api_key, args.timeout)
            text = (body["choices"][0]["message"]["content"] or "").strip()
            teacher_completion_tokens += int(body.get("usage", {}).get("completion_tokens") or 0)
            attempted += 1
            is_right = grade(completions=[text], answer=[row["answer"]])[0] == 1.0
            if is_right:
                correct += 1
            if is_right or args.keep_wrong:
                kept.append({
                    "prompt": row["prompt"],
                    "completion": [{"role": "assistant", "content": text}],
                })
                break
        print(f"  [{i}/{len(rows)}] kept {len(kept)}  teacher correct {correct}/{attempted}")

    if not kept:
        raise SystemExit("the teacher solved none of the problems in the required format. Check that "
                         "it is a larger model than the student and that the endpoint is right.")

    split = max(1, int(len(kept) * args.valid_fraction))
    valid, train = kept[:split], kept[split:]
    out = Path(args.out_dir)
    train_hash = write_jsonl(out / "train.jsonl", train)
    write_jsonl(out / "valid.jsonl", valid)

    elapsed = time.time() - started
    budget = {
        "teacher_model": args.teacher_model,
        "problems_attempted": attempted,
        "teacher_correct": correct,
        "teacher_accuracy": round(correct / attempted, 4) if attempted else None,
        "examples_kept": len(kept),
        "teacher_completion_tokens": teacher_completion_tokens or None,
        "generation_seconds": round(elapsed, 1),
        "stopped_on_budget": stopped_early,
    }
    print("\n" + json.dumps(budget, indent=2))
    print(f"\nwritten to {out}/  train {len(train)}  valid {len(valid)}  sha256 {train_hash[:12]}...")
    print("Now train the same student on it with the Part 11 recipe, unchanged:")
    print(f"  python3 train-sft.py --model <the same base model> --data-dir {out} \\")
    print("      --output-dir runs/distil-baseline --epochs 3 --labbook labbook.md")
    print("Then score it with eval-pass-at-1.py on the same held-out files as the GRPO run.")
    if teacher_completion_tokens == 0:
        print("\nThe server returned no token usage, so the budget line records wall clock only.")

    if args.labbook:
        record = runlog.record(
            labbook=args.labbook,
            lab="part-14/distil-baseline",
            model=args.teacher_model,
            dataset={"path": args.tasks, "sha256": runlog.file_sha256(args.tasks),
                     "problems": len(rows), "examples_kept": len(kept),
                     "written_sha256": train_hash},
            hyperparameters={"attempts": args.attempts, "temperature": args.temperature,
                             "max_tokens": args.max_tokens, "keep_wrong": args.keep_wrong,
                             "budget_seconds": args.budget_seconds},
            seed=args.seed,
            losses={},
            scores=budget,
            config_path=__file__,
            notes=args.notes,
        )
        print(f"recorded run {record['run_id']} in {args.labbook}")


if __name__ == "__main__":
    main()

Download distil-baseline.py176 lines

Serve a teacher larger than the model you trained. Qwen3-8B’s card gives an Apache-2.0 licence and the course model reference records its Q4_K_M size, which fits the 16 GB tier when nothing is training.

RunnableAll tracks

serve the teacher

llama-server \
  --model ~/models/Qwen3-8B-Q4_K_M.gguf \
  --alias qwen3-8b \
  --ctx-size 4096 \
  --host 127.0.0.1 \
  --port 8080 \
  --jinja

RunnableAll tracks

teacher solutions for the same problems, on a wall-clock budget

python3 distil-baseline.py \
  --tasks tasks/train.jsonl \
  --base-url http://127.0.0.1:8080/v1 \
  --teacher-model qwen3-8b \
  --budget-seconds 1800 \
  --out-dir distil-data \
  --labbook labbook.md

The script keeps only the solutions its verifier says are correct, which is the same numeric check the reward used, and stops when the budget expires. Then train the same student with Part 11’s reference recipe, unchanged:

RunnableAll tracks

supervised fine-tuning on the teacher's correct solutions

python3 train-sft.py \
  --model Qwen/Qwen3-1.7B \
  --data-dir distil-data \
  --output-dir runs/distil-baseline \
  --epochs 3 \
  --labbook labbook.md

RunnableAll tracks

score the baseline on both families

python3 eval-pass-at-1.py --tasks tasks/heldout-same.jsonl \
  --model Qwen/Qwen3-1.7B --adapter runs/distil-baseline --limit 60 \
  --label distil-same --out eval-distil-same.json --labbook labbook.md
python3 eval-pass-at-1.py --tasks tasks/heldout-different.jsonl \
  --model Qwen/Qwen3-1.7B --adapter runs/distil-baseline --limit 60 \
  --label distil-diff --out eval-distil-diff.json --labbook labbook.md

!Important

The budget comparison is approximate and you should say so in the report. The GRPO runs and the distillation run consume different resources: one spends its time sampling from the model being trained, the other spends it querying a larger model that has to be served. Matching wall clock is the most defensible comparison available on one machine, and it is not the same as matching compute. State which you matched.

6. Fill in the table and write the sentence

Pending validationDoes the gain survive outside the training distribution?

Condition	pass@1, same family	pass@1, different family	Mean completion length	Wall clock
Starting model, untrained	to be measured	to be measured	to be measured	n/a
GRPO, seed 0	to be measured	to be measured	to be measured	to be measured
GRPO, seed 1	to be measured	to be measured	to be measured	to be measured
GRPO, seed 2	to be measured	to be measured	to be measured	to be measured
GRPO, spread across seeds	to be measured	to be measured	to be measured	n/a
Distillation baseline, same wall clock	to be measured	to be measured	to be measured	to be measured

one machine on your own track; state which · TRL GRPOTrainer for the GRPO rows, TRL SFTTrainer for the baseline, llama.cpp server for the teacher trl 1.12.0, transformers 5.16.1, peft 0.20.0, llama.cpp v0.4.0 · Qwen3-1.7B with rank-16 LoRA adapters; Qwen3-8B as the distillation teacher, BF16 for training, float32 on Track M; Q4_K_M for the teacher · 1,024 tokens of context · 2026-09-09

Not yet run on hardware on any track; this is the shape of the report rather than an expected result. Fill in your own six rows, then write one sentence saying whether the pre-registered margin was met on each family. A gain smaller than the spread across seeds is not a gain.

Then write the sentence. Three clauses: what happened on the same family, what happened on the different family, and whether the baseline beat it. If any of the three is uncomfortable, that is the one worth writing first.

Compare methods under a declared budget

Begin from the same untouched base for each training seed. Keep the task generator and held-out families fixed, and record whether the budget is wall time, rollout count or processed tokens. Those budgets are not interchangeable when one method spends more time generating or verifying.

For the distillation alternative, account for teacher generation and student training as the procedure defines. Preserve both in the cost record. Evaluate every candidate using the same first-attempt answer protocol and independently verified answers. Do not compare best-of-many sampling on one side with a single completion on the other.

Inspect transfer to the different task family separately from performance on the familiar family. Repeat seeds estimate optimisation variation; repeated prompts estimate sampling variation; neither replaces new task coverage. Keep failed training runs and verifier exploits visible. The final statement should name the task family, training budget, evaluation policy and uncertainty. If the gain appears only on the training-like tasks, that is a narrower and more useful conclusion than claiming a general reasoning capability. Retain all seed outputs so the report can be recomputed.

Validation

You are done when all of the following are true:

• a margin was written down before Task 3 produced any result;
• three GRPO adapters exist from identical recipes differing only in --seed, or two with the reduction stated;
• eight evaluation files exist: before and after on both families, for every seed;
• the spread across seeds is computed and recorded for both families;
• distil-baseline.py recorded a budget, and the fine-tune on its output was scored on both families;
• the benchmark table above is filled in with your own figures;
• you can say in one sentence, without looking anything up, whether the pre-registered margin was met on each family;
• every run above has a JSON line in labbook.md.

Expected outcome

A table, a spread and a sentence you would be willing to publish. What the numbers are depends on your track, your model and how long you trained, which is exactly why you ran it.

Three things are worth saying about how to read whatever you got.

A same-family gain is the weakest form of the claim. The model practised those problems’ shape for 120 steps. Improving on more of them is what practice does, and it is a long way from “reason”.

A different-family gain is the interesting one, and it is small or absent in short runs. The mechanism from the GRPO lesson explains why: reinforcement learning raises the frequency of behaviour the model already samples on the training problems, and the calendar and ordering puzzles need behaviour those problems never exercised.

Length is a confound in both directions. If the trained model’s answers got much longer, some of the same-family gain may be extra attempts at arithmetic rather than better arithmetic, and the different-family answers may be being truncated. The mean-length column in the table is there so that this is visible rather than argued about.

!Important

Report the result you got. If the different-family gain is inside the seed spread, say so, and say that this was a 120-step run on 240 problems, well below the “at least 300 steps” and “at least 500 rows” that Unsloth’s guide suggests for a good result. That is a limitation of your experiment, and stating it is different from explaining the result away. If the distillation baseline won on both families for the same wall clock, say that too: it is the most useful thing this page can tell a reader deciding what to do next week.

What the published results do and do not say

The strong published claims are about large models, long runs and problems near the edge of what the model can already do. DeepSeek-R1’s abstract also points at the route that matters most for a small local model: “the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models”. That is distillation, and it is Part 15.

The critical literature is equally useful. Dr. GRPO’s authors identify an “optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training”, which is a caution about reading length as thinking. DAPO’s dynamic sampling exists because groups with an accuracy of exactly 0 or 1 contribute nothing, which is a caution about training problems that are all too easy or all too hard. Both are reasons a short run on generated arithmetic may produce very little, and both are better explanations than “the method does not work”.

Troubleshooting

The three seeds give nearly identical results. Good, and slightly suspicious. Check that --seed actually differed in the three run-log lines, and that the adapters are three separate directories. A short run with a small learning rate can genuinely be stable, but identical numbers to three decimal places usually means the same adapter was evaluated three times.

The spread across seeds is larger than the gain. That is a finding and it is the most common one at this scale. Report it. If you want a smaller spread, the levers are more steps, more problems and a larger group, in that order.

The teacher solves almost nothing. distil-baseline.py prints the teacher’s accuracy as it goes. If it is low, the teacher is not large enough for the task or is not following the requested format. Check four of its answers by eye before concluding anything about distillation.

The distillation dataset is tiny. The budget expired, or the teacher’s correct-answer rate is low. Raise --budget-seconds, or add --attempts 2 so a problem gets a second try. Whatever you change, change it for the comparison as a whole and say so.

pass@1 on the different family is zero everywhere, before and after. The puzzles are too hard for the model at this size, so the comparison has no room to show anything. Regenerate a smaller different-family set with easier templates, or accept that this measurement is uninformative and say which.

The evaluation is slower than the training. Six evaluations of sixty problems is a lot of generation. Lower --limit to 40 for the exploratory passes and use 60 only for the numbers you report, keeping the count the same across every condition.

Cleanup

Keep labbook.md, tasks/, the evaluation files and the benchmark table: Part 15 compares distillation against these figures, and Part 16 uses the same held-out sets.

RunnableAll tracks

reclaim the disk, keeping the evidence

rm -rf runs/grpo-seed1/checkpoint-* runs/grpo-seed2/checkpoint-* runs/distil-baseline/checkpoint-*
rm -rf distil-data

⚠Warning

The second command deletes the teacher-generated dataset, which cost real wall clock to produce and is not deterministic: a rerun of distil-baseline.py will not give the same file. The run log records its hash and its budget, which is enough to describe it and not enough to recreate it. Keep it if you have the disk.

Stop the teacher server when you are done with it.

What you learned

• A claim that cannot fail is not a claim. “RL makes small models reason” became a sentence with a model, two task families, a metric and a margin in it, and only then could it be wrong.
• The noise floor is measured, not assumed. Three seeds of the same recipe told you how much the result moves when nothing changed, and that number bounds every conclusion above it.
• Same-kind and different-kind are different questions. The gap between them is where the word “reason” is doing its work, and it is the number most reports never show.
• A baseline changes what the result means. The same wall clock spent on supervised fine-tuning from a larger model’s correct answers is the comparison a reader actually needs.
• Length is a confound you can see. You recorded mean completion length in every condition, which is what makes the length explanation testable rather than arguable.
• The published results were obtained under conditions you did not reproduce. Saying which conditions differed is a stronger contribution than either agreeing or disagreeing.

Record in the notebook: the pre-registered margin and when you wrote it; pass@1 for the untrained model and all three seeds on both families with their standard errors; the spread across seeds; the distillation baseline’s budget, dataset size and scores; the mean completion length in every condition; whether the margin was met on each family; and one sentence you would be willing to put your name to.

Check your understanding

Question 1. Why does this page train three seeds of an identical recipe before comparing anything?

• To average them into a better model
• To measure how much the result moves when nothing that matters has changed, which is the floor below which no difference can be called a finding
• Because one run might crash
• To make the training take longer

Show the answer and why

Answer: To measure how much the result moves when nothing that matters has changed, which is the floor below which no difference can be called a finding

Without a noise floor, any before-and-after difference is unreadable. The seed spread usually turns out to be larger than the standard error on the evaluation sample, which is why it is the number to compare a gain against.

Question 2. Your GRPO run raises pass@1 on held-out arithmetic by 0.08 and on the different family by 0.01, with a seed spread of 0.05 on both. What is supported?

• Reinforcement learning made the model reason
• On the family it trained on, the gain exceeds the seed spread and is a real effect on this test set; on the different family it does not, so nothing generalising has been shown
• The experiment failed
• The different-family tasks were unfair

Show the answer and why

Answer: On the family it trained on, the gain exceeds the seed spread and is a real effect on this test set; on the different family it does not, so nothing generalising has been shown

Two families, two answers, and the honest report contains both. Improving at the thing you practised is what practice does; the second number is the one that would have supported the slogan, and it did not.

Question 3. Why is a distillation baseline on the same wall clock part of this page rather than an optional extra?

• Because distillation wins every comparison
• Because "did RL help" is much less useful than "did RL help more than the obvious alternative given the same budget", which is the decision a reader actually faces
• To use the teacher model that is already downloaded
• Because TRL requires a baseline

Show the answer and why

Answer: Because "did RL help" is much less useful than "did RL help more than the obvious alternative given the same budget", which is the decision a reader actually faces

A gain with no alternative to compare against cannot tell you what to do next. The comparison is approximate, because wall clock is not compute, and the page says so rather than pretending otherwise.

Question 4. Which of these belong in an honest report of this experiment? Select all that apply.

• The margin, and the fact that it was chosen before the results were seen
• The mean completion length in every condition
• That the run was well short of the step count and dataset size the tooling documentation suggests for a good result
• Only the conditions where the trained model won

Show the answer and why

Answer: The margin, and the fact that it was chosen before the results were seen, The mean completion length in every condition, That the run was well short of the step count and dataset size the tooling documentation suggests for a good result

The first three are what let a reader judge the result. Reporting only the favourable conditions is the failure the whole page is built to prevent, and it is the easiest one to commit without noticing.

Question 5. The trained model's mean completion length doubled and its same-family pass@1 rose. Why is length recorded in every row of the table?

• To estimate the wall clock
• Because the GRPO objective has a documented bias towards longer responses, so a length increase is expected from the algorithm and may explain part of the accuracy change rather than being caused by it
• Because a longer answer scores higher by definition
• To compare against the teacher model

Show the answer and why

Answer: Because the GRPO objective has a documented bias towards longer responses, so a length increase is expected from the algorithm and may explain part of the accuracy change rather than being caused by it

Dr. GRPO identifies the bias directly. With length in the table, the alternative explanation is testable: rerun with the unbiased loss, or cap the length, and see whether the accuracy gain survives.

Check my answersClear