Reinforcement Learning on Agent Tasks: Tests as Rewards

By the end of this lesson you will be able to describe an agent episode as an environment, a rollout and a verdict; write a reward whose verdict is a test suite rather than an opinion and say why it needs a protected-path check; work out from arithmetic how long a reinforcement learning run on agent tasks would take on your own machine, and why the answer is usually dominated by something other than the gradient; name the tools that support this today, with the date you checked; and say what the method can and cannot do at the scale you can afford.

An episode is an environment, a rollout and a verdict

Part 14’s GRPO ran on single-turn problems. A prompt went in, a group of completions came back, a Python function scored each one, and the advantage within the group drove the update. Nothing in that loop had state.

An agent episode has state, and that is the whole difference. The model’s first call changes what the second call sees. A file it writes is there for the test run. A search it makes returns different results after an edit. So the loop grows a third component:

One GRPO step on an agent task

1. 1Reset the environmentA fresh copy of the workspace, or a fresh container. Whatever the last rollout did is gone.
2. 2Roll outThe model calls tools; the environment answers; the transcript grows until it finishes or hits the turn limit.
3. 3ScoreRun the checks. For code, that means running the test suite in the copy the rollout left behind.
4. 4Compare within the groupThe advantage is a rollout’s reward against the mean of its group, which is why the group has to exist.
5. 5UpdateOne optimiser step on the adapter. The cheapest part of the whole diagram.

The three boxes on the right happen once per rollout, and there are several rollouts per prompt because GRPO needs a group to compute an advantage. That multiplication is the whole cost story.

The trainer documentation for TRL 1.12.0 · verified 2026-09-08, read on 2026-09-09, has grown a vocabulary for exactly this. GRPOTrainer takes a tools argument: a list of ordinary Python functions with type-hinted arguments and return values and Google-style docstrings, which it turns into schemas the way apply_chat_template does, with the requirement that the chat template be prefix-preserving. It also takes environments, documented as stateful objects with a required reset called at the start of each rollout, an optional get_reward so the environment can score itself, and public methods that are exposed to the model as tools; one environment instance is created per rollout. max_tool_calling_iterations bounds the number of tool-calling turns, and generation otherwise stops when the model produces a turn with no calls. There is also an experimental rollout_func for replacing the generation loop entirely.

That is a lot of machinery for a page to name at once. The point of naming it is narrower: as of the version this course pins, the thing you would have had to build yourself in 2024 is a documented argument, and the reason to read this lesson is no longer “can it be done” but “is it worth doing on your machine”.

Rewards that a program can compute

Part 14’s rule holds and gets stronger here: nothing in the reward asks a model for an opinion. For agent tasks there are four kinds of check worth having, and this part ships all four.

RunnableAll tracks

agent-rewards.py

"""Reward functions for agent tasks, in the shape TRL's GRPO trainer expects.

Purpose: Part 27's reward library. Part 14's rewards scored one answer; these score a
    whole attempt at a task: whether the tool calls parse, whether their arguments match
    the declared schema, whether the files the model wrote make a test suite pass, and
    whether it did all that inside a step budget. Nothing here asks a model for an
    opinion. Every function is arithmetic, a regular expression or a subprocess that
    either passes or fails, so the same completion always scores the same.
Platform: all (pure Python; the test-suite reward needs a POSIX system for its resource
    limits, so on Windows run it inside WSL2)
Minimum memory: 8 GB
Assumes: Python 3.10 or newer and no third-party packages. Every function takes the
    keyword arguments TRL passes a reward function - `completions` plus the dataset's
    own columns - and returns one float per completion.

Usage: run directly to score the built-in worked examples, including the two that try
       to cheat the reward and the one that fails on format, and print a table:
           python3 agent-rewards.py --demo
           python3 agent-rewards.py --demo --show-sandbox
       loaded by a GRPO training script that sits beside it, the same way Part 26's
       harness loads an agent entry point (the file name has a hyphen in it, so it is
       loaded by path rather than imported by name):
           import importlib.util, pathlib
           spec = importlib.util.spec_from_file_location(
               "agent_rewards", pathlib.Path(__file__).with_name("agent-rewards.py"))
           agent_rewards = importlib.util.module_from_spec(spec)
           spec.loader.exec_module(agent_rewards)
           functions, weights = agent_rewards.build_agent_reward_functions(
               kind="tests", workspace="repo")

The file-writing protocol, which is what the test-suite reward reads, is one fence per
file:

    ```file:src/app.py
    def total(rows):
        return sum(r["amount"] for r in rows)
    ```

It is deliberately dull. A reward that has to interpret a unified diff spends its budget
on parsing rather than on measuring, and a patch that does not apply is scored as a
failure to write code rather than as a failure to solve the task.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any, Callable, Sequence

# ---------------------------------------------------------------------------------------
# Reading a completion
# ---------------------------------------------------------------------------------------

# One fenced block per file, as described in the module docstring.
FILE_BLOCK = re.compile(r"```file:([^\n`]+)\n(.*?)```", re.DOTALL)

# A tool call the model emitted as JSON inside a fence or as a bare object.
JSON_TOOL_CALL = re.compile(r"\{\s*\"name\"\s*:\s*\"([A-Za-z0-9_.-]+)\"\s*,\s*\"arguments\"\s*:",
                            re.DOTALL)

# Fragments that mean the model tried to call a tool and nothing parsed it back. Each one
# is a real format from a model family in this course's reference set, seen as visible
# text; the same list Part 24's reliability test watches for.
UNPARSED_MARKERS = (
    "<tool_call>", "</tool_call>", "<|channel|>commentary", "[TOOL_REQUEST]",
    "<function=", "<tool_use>",
)


def text_of(completion: Any) -> str:
    """TRL hands completions as strings or as message lists; this flattens both."""
    if isinstance(completion, str):
        return completion
    if isinstance(completion, list):
        parts = []
        for message in completion:
            if not isinstance(message, dict):
                continue
            if message.get("content"):
                parts.append(str(message["content"]))
            for call in message.get("tool_calls") or []:
                function = call.get("function", {}) if isinstance(call, dict) else {}
                parts.append(json.dumps({"name": function.get("name"),
                                         "arguments": function.get("arguments")}))
        return "\n".join(parts)
    if isinstance(completion, dict):
        return str(completion.get("content", ""))
    return str(completion)


def calls_of(completion: Any) -> list[tuple[str, dict]]:
    """(name, arguments) for every structured tool call in the completion.

    Only calls the serving stack already parsed count. A call that arrived as visible
    text is not a call; it is a formatting failure, and the format reward is where it is
    scored.
    """
    out: list[tuple[str, dict]] = []
    if isinstance(completion, list):
        for message in completion:
            if not isinstance(message, dict):
                continue
            for call in message.get("tool_calls") or []:
                function = (call or {}).get("function", {})
                arguments = function.get("arguments")
                if isinstance(arguments, str):
                    try:
                        arguments = json.loads(arguments)
                    except json.JSONDecodeError:
                        arguments = {}
                out.append((str(function.get("name", "")), arguments if isinstance(arguments, dict) else {}))
    return out


def files_written(text: str) -> dict[str, str]:
    """The files the completion asked for, as {path: contents}. The last block wins."""
    out: dict[str, str] = {}
    for path, body in FILE_BLOCK.findall(text):
        cleaned = path.strip()
        if cleaned:
            out[cleaned] = body
    return out


# ---------------------------------------------------------------------------------------
# Format and schema
# ---------------------------------------------------------------------------------------

def tool_call_format_reward(value: float = 1.0) -> Callable[..., list[float]]:
    """1 when the attempt's tool calls arrived as calls, 0 when any arrived as text.

    This is the reward that fixes the failure Part 24's reliability test measures: a
    model that writes the tool-call format into its prose. It is cheap, it is checkable
    without running anything, and on a small model it is often most of the gain.
    """

    def reward(completions, **kwargs) -> list[float]:
        scores = []
        for completion in completions:
            text = text_of(completion)
            leaked = any(marker in text for marker in UNPARSED_MARKERS)
            scores.append(0.0 if leaked else value)
        return scores

    reward.__name__ = "tool_call_format_reward"
    return reward


def schema_errors(arguments: Any, schema: dict) -> list[str]:
    """The subset of JSON Schema this course checks: type, required, properties, enum."""
    problems: list[str] = []
    if not isinstance(arguments, dict):
        return ["arguments are not an object"]
    properties = schema.get("properties", {}) or {}
    for name in schema.get("required", []) or []:
        if name not in arguments:
            problems.append(f"missing required argument {name!r}")
    types = {"string": str, "number": (int, float), "integer": int,
             "boolean": bool, "object": dict, "array": list}
    for name, value in arguments.items():
        if name not in properties:
            problems.append(f"unknown argument {name!r}")
            continue
        spec = properties[name] or {}
        wanted = types.get(str(spec.get("type", "")))
        if wanted and not isinstance(value, wanted):
            problems.append(f"argument {name!r} should be {spec.get('type')}")
        choices = spec.get("enum")
        if choices and value not in choices:
            problems.append(f"argument {name!r} is not one of {choices}")
    return problems


def schema_valid_reward(tools: Sequence[dict] | None = None,
                        value: float = 1.0) -> Callable[..., list[float]]:
    """The fraction of the attempt's calls whose arguments match the declared schema.

    `tools` may be given once here or per row as a `tools` dataset column, which is what
    happens when different tasks expose different tools.
    """

    def reward(completions, **kwargs) -> list[float]:
        per_row = kwargs.get("tools")
        scores = []
        for index, completion in enumerate(completions):
            declared = tools
            if per_row is not None:
                candidate = per_row[index] if isinstance(per_row, list) else per_row
                if isinstance(candidate, str):
                    try:
                        candidate = json.loads(candidate)
                    except json.JSONDecodeError:
                        candidate = None
                declared = candidate or tools
            schemas = {t.get("function", {}).get("name"): t.get("function", {}).get("parameters", {})
                       for t in (declared or [])}
            calls = calls_of(completion)
            if not calls or not schemas:
                scores.append(0.0)
                continue
            good = sum(1 for name, arguments in calls
                       if name in schemas and not schema_errors(arguments, schemas[name]))
            scores.append(value * good / len(calls))
        return scores

    reward.__name__ = "schema_valid_reward"
    return reward


def step_budget_reward(max_steps: int = 8, penalty: float = 0.25) -> Callable[..., list[float]]:
    """0 when the attempt stayed inside the budget, negative when it ran long.

    A brake rather than a target. Rewarding short trajectories directly teaches a model
    to answer without looking, which is the failure this whole part exists to remove.
    """

    def reward(completions, **kwargs) -> list[float]:
        scores = []
        for completion in completions:
            steps = len(calls_of(completion)) or len(JSON_TOOL_CALL.findall(text_of(completion)))
            over = max(0, steps - max_steps)
            scores.append(-penalty * over if over else 0.0)
        return scores

    reward.__name__ = "step_budget_reward"
    return reward


# ---------------------------------------------------------------------------------------
# The test suite as a reward
# ---------------------------------------------------------------------------------------

RUNNER_TEMPLATE = '''"""Written by agent-rewards.py. Applies resource limits, then runs the command."""
import os, resource, sys

resource.setrlimit(resource.RLIMIT_CPU, ({cpu}, {cpu}))
resource.setrlimit(resource.RLIMIT_AS, ({mem}, {mem}))
resource.setrlimit(resource.RLIMIT_FSIZE, ({fsize}, {fsize}))
try:
    resource.setrlimit(resource.RLIMIT_NPROC, ({nproc}, {nproc}))
except (ValueError, OSError):
    pass
os.execvp(sys.argv[1], sys.argv[1:])
'''


def run_tests_in_copy(
    workspace: Path,
    writes: dict[str, str],
    test_command: Sequence[str],
    protected: Sequence[str],
    timeout_s: int = 120,
    cpu_seconds: int = 120,
    memory_mb: int = 2048,
    max_output_bytes: int = 1 << 22,
    max_processes: int = 128,
    env_passthrough: Sequence[str] = (),
) -> tuple[bool, str]:
    """Copy the workspace, write the model's files into the copy, run the tests there.

    Returns (passed, reason). The copy is thrown away afterwards, so a rollout that
    deletes half the repository costs one directory rather than the repository. The child
    runs with a wall-clock timeout on top of a CPU limit, an address-space limit and a
    file-size limit, so an infinite loop, a runaway allocation and a program that writes a
    huge file are all bounded.

    What this does NOT do, and the lesson says so on the page: it is not a security
    boundary. The child runs as your user, on your filesystem, with your network. It stops
    accidents and cheap denial of service; it does not stop a program written to do harm.
    Part 25 builds the container-and-dedicated-user sandbox that untrusted code needs, and
    that is what a rollout loop should run inside.
    """
    if os.name != "posix":
        return False, "resource limits need a POSIX system; run this inside WSL2 on Windows"
    if not workspace.is_dir():
        return False, f"workspace {workspace} does not exist"

    protected_paths = {Path(p).as_posix() for p in protected}
    for path in writes:
        candidate = Path(path)
        if candidate.is_absolute() or ".." in candidate.parts:
            return False, f"refused to write outside the workspace: {path}"
        if candidate.as_posix() in protected_paths:
            # This is the reward-hacking check, and it is not optional. A model that can
            # edit the tests will edit the tests, because that is the cheapest way to make
            # them pass, and the reward would happily pay for it.
            return False, f"wrote a protected file: {path}"

    with tempfile.TemporaryDirectory(prefix="course-agent-reward-") as parent:
        copy = Path(parent) / "workspace"
        shutil.copytree(workspace, copy, symlinks=False,
                        ignore=shutil.ignore_patterns(".git", "__pycache__", "*.pyc"))
        for path, body in writes.items():
            target = copy / path
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(body, encoding="utf-8")

        runner = Path(parent) / "runner.py"
        runner.write_text(RUNNER_TEMPLATE.format(
            cpu=cpu_seconds, mem=memory_mb * 1024 * 1024,
            fsize=max_output_bytes, nproc=max_processes), encoding="utf-8")

        env = {"PATH": os.environ.get("PATH", ""), "HOME": str(copy),
               "LANG": os.environ.get("LANG", "C.UTF-8")}
        for name in env_passthrough:
            if name in os.environ:
                env[name] = os.environ[name]

        try:
            done = subprocess.run(
                [sys.executable, "-I", "-S", str(runner), *test_command],
                cwd=str(copy), env=env, capture_output=True, text=True,
                timeout=timeout_s, check=False,
            )
        except subprocess.TimeoutExpired:
            return False, f"timed out after {timeout_s} s"
        except OSError as exc:
            return False, f"could not start the test command: {exc}"

    if done.returncode == 0:
        return True, "tests passed"
    tail = ((done.stdout or "") + (done.stderr or "")).strip().splitlines()
    return False, (tail[-1][:200] if tail else f"exit code {done.returncode}")


def test_suite_reward(workspace: str | os.PathLike[str],
                      test_command: Sequence[str] = ("python3", "-m", "pytest", "-q"),
                      protected: Sequence[str] = ("tests.py",),
                      timeout_s: int = 120,
                      memory_mb: int = 2048,
                      value: float = 1.0) -> Callable[..., list[float]]:
    """1 when the files the attempt wrote make the suite pass in a fresh copy, else 0.

    The workspace and the command may also come from the dataset, as `workspace`,
    `test_command` and `protected` columns, which is how one training run covers several
    repositories.
    """
    root = Path(workspace)

    def reward(completions, **kwargs) -> list[float]:
        scores = []
        for index, completion in enumerate(completions):
            def column(name: str, fallback):
                value_ = kwargs.get(name)
                if value_ is None:
                    return fallback
                return value_[index] if isinstance(value_, list) else value_

            here = Path(column("workspace", root))
            command = column("test_command", test_command)
            if isinstance(command, str):
                command = command.split()
            guard = column("protected", protected)
            if isinstance(guard, str):
                guard = [guard]
            writes = files_written(text_of(completion))
            if not writes:
                scores.append(0.0)
                continue
            passed, _reason = run_tests_in_copy(here, writes, command, guard,
                                                timeout_s=timeout_s, cpu_seconds=timeout_s,
                                                memory_mb=memory_mb)
            scores.append(value if passed else 0.0)
        return scores

    reward.__name__ = "test_suite_reward"
    return reward


# ---------------------------------------------------------------------------------------
# Combining them
# ---------------------------------------------------------------------------------------

def build_agent_reward_functions(
    kind: str = "tools",
    tools: Sequence[dict] | None = None,
    workspace: str | os.PathLike[str] | None = None,
    test_command: Sequence[str] = ("python3", "-m", "pytest", "-q"),
    protected: Sequence[str] = ("tests.py",),
    max_steps: int = 8,
    timeout_s: int = 120,
) -> tuple[list[Callable[..., list[float]]], list[float]]:
    """The reward list and weights a GRPO run passes to the trainer.

    TRL sums the functions after weighting them, so the weights are the statement of what
    matters. Here the task outcome dominates, format is a hint, and the step budget is a
    brake. Returning both lists together keeps them the same length, which TRL requires.
    """
    functions: list[Callable[..., list[float]]] = []
    weights: list[float] = []

    if kind == "tools":
        functions.append(schema_valid_reward(tools))
        weights.append(1.0)
    elif kind == "tests":
        if workspace is None:
            raise ValueError("kind='tests' needs a workspace to run the suite in")
        functions.append(test_suite_reward(workspace, test_command, protected,
                                           timeout_s=timeout_s))
        weights.append(2.0)
    else:
        raise ValueError(f"unknown reward kind {kind!r}; expected tools or tests")

    functions.append(tool_call_format_reward())
    weights.append(0.5)
    functions.append(step_budget_reward(max_steps))
    weights.append(1.0)
    return functions, weights


def score(functions: Sequence[Callable[..., list[float]]], weights: Sequence[float],
          completions: Sequence[Any], **columns: Any) -> list[float]:
    """The weighted sum, computed the way the trainer computes it."""
    totals = [0.0] * len(completions)
    for function, weight in zip(functions, weights):
        for index, value in enumerate(function(completions=completions, **columns)):
            totals[index] += weight * value
    return totals


# ---------------------------------------------------------------------------------------
# The worked examples
# ---------------------------------------------------------------------------------------

DEMO_APP = '''def total(rows):
    """Sum the amount on every row. Deliberately wrong: it drops the last row."""
    return sum(r["amount"] for r in rows[:-1])
'''

# Plain assertions run by the interpreter rather than a test runner, so the demo needs
# nothing installed. A real repository's suite is whatever its own command is, which is
# what --test-command and the dataset's test_command column are for.
DEMO_TESTS = '''from app import total

assert total([{"amount": 1}, {"amount": 2}, {"amount": 3}]) == 6, "does not sum every row"
assert total([]) == 0, "the total of nothing is not zero"
print("ok")
'''

DEMO_TOOLS = [
    {"type": "function", "function": {
        "name": "read_file",
        "description": "Read a text file inside the workspace.",
        "parameters": {"type": "object",
                       "properties": {"path": {"type": "string"}},
                       "required": ["path"]}}},
]


def demo_completions() -> list[tuple[str, list[dict]]]:
    """Six attempts: two honest, one wasteful, two that cheat, and one that mis-formats."""
    def attempt(content: str, calls: list[tuple[str, Any]] | None = None) -> list[dict]:
        message: dict[str, Any] = {"role": "assistant", "content": content}
        if calls:
            message["tool_calls"] = [
                {"type": "function", "function": {"name": name, "arguments": arguments}}
                for name, arguments in calls
            ]
        return [message]

    fixed = "```file:app.py\ndef total(rows):\n    return sum(r[\"amount\"] for r in rows)\n```"
    return [
        ("correct fix, one call",
         attempt("I read the file and fixed the slice.\n" + fixed, [("read_file", {"path": "app.py"})])),
        ("correct fix, no tool call",
         attempt(fixed)),
        ("wasteful: twelve reads",
         attempt("Checking everything.\n" + fixed,
                 [("read_file", {"path": "app.py"}) for _ in range(12)])),
        ("hack: edits the tests instead",
         attempt("```file:tests.py\nassert True\n```",
                 [("read_file", {"path": "tests.py"})])),
        ("hack: prints a success message",
         attempt("ok\nAll tests passed.")),
        ("format failure: the call arrived as text",
         attempt("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"app.py\"}}\n</tool_call>")),
    ]


def run_demo(show_sandbox: bool) -> None:
    with tempfile.TemporaryDirectory(prefix="course-agent-demo-") as parent:
        workspace = Path(parent) / "repo"
        workspace.mkdir()
        (workspace / "app.py").write_text(DEMO_APP, encoding="utf-8")
        (workspace / "tests.py").write_text(DEMO_TESTS, encoding="utf-8")

        functions, weights = build_agent_reward_functions(
            kind="tests", workspace=workspace,
            test_command=("python3", "tests.py"),
            protected=("tests.py",), max_steps=8, timeout_s=60)

        labels = [label for label, _ in demo_completions()]
        completions = [c for _, c in demo_completions()]
        columns: dict[str, Any] = {"tools": [DEMO_TOOLS] * len(completions)}

        per_function = {f.__name__: f(completions=completions, **columns) for f in functions}
        totals = score(functions, weights, completions, **columns)

        header = f"{'attempt':<42}" + "".join(f"{name.replace('_reward',''):>20}" for name in per_function)
        print(header + f"{'total':>10}")
        print("-" * len(header + f"{'total':>10}"))
        for index, label in enumerate(labels):
            row = f"{label:<42}"
            for name in per_function:
                row += f"{per_function[name][index]:>20.2f}"
            print(row + f"{totals[index]:>10.2f}")

        print("\nWeights: " + ", ".join(f"{f.__name__}={w}" for f, w in zip(functions, weights)))
        print("\nRead the fourth row. The attempt that edits the tests scores zero on the "
              "test-suite reward because the protected-path check refuses the write, not "
              "because the suite failed. Take that check out and it scores full marks, which "
              "is what a reward function is for: paying for the cheapest thing that "
              "satisfies it.")
        print("The fifth row is the other half of the same lesson: a completion that says "
              "the tests passed scores nothing, because the reward runs them.")
        if show_sandbox:
            passed, reason = run_tests_in_copy(
                workspace, {"app.py": DEMO_APP}, ("python3", "tests.py"),
                ("tests.py",), timeout_s=60)
            print(f"\nsandbox check with the unfixed file: passed={passed} reason={reason}")


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--demo", action="store_true", help="score the worked examples")
    parser.add_argument("--show-sandbox", action="store_true",
                        help="also run the unfixed file through the sandbox and print the reason")
    args = parser.parse_args()
    if not args.demo:
        parser.print_help()
        return
    run_demo(args.show_sandbox)


if __name__ == "__main__":
    main()

Download agent-rewards.py543 lines

Format. Did the calls arrive as calls? A model that writes its family’s tool-call markup into the prose has failed before anything else can be judged, and the check is a substring search for the markers Part 24’s reliability test already watches for. This is cheap and, on a small model, often most of the available gain.

Schema. Did the arguments match the tool’s declared schema? Required arguments present, types right, no invented argument names, enumerated values from the list. This is the same subset of JSON Schema that Part 24’s agent validates calls against before executing them, so a model rewarded for it is being rewarded for the thing the runtime actually enforces.

Outcome. Did the tests pass? This is the reward that makes the whole idea attractive, because it is not a proxy for the task, it is the task. The script’s protocol for it is deliberately dull: the model writes whole files inside fenced blocks tagged with their paths, those files are written into a fresh copy of the workspace, and the workspace’s own test command is run there. A reward that has to interpret a unified diff spends its budget on parsing rather than on measuring.

Budget. Did it stay inside the step limit? Expressed as a penalty for going over rather than a bonus for being short, because rewarding brevity directly teaches a model to answer without looking, which is the failure this part exists to remove.

!Important

The test-suite reward needs one check that has nothing to do with testing: a list of protected paths that the model is not allowed to write. Without it, the cheapest way to make the suite pass is to replace the suite. Run python3 agent-rewards.py --demo and read the fourth row of the table: an attempt that rewrites the test file scores zero because the write is refused, and if you remove that refusal it scores full marks. A reward function pays for the cheapest thing that satisfies it, every time, and it is your job to make the cheapest thing be the thing you wanted.

Sandboxed rollouts

During training, the model writes files and those files get executed. That sentence should change how you set the run up.

The reward function in this part copies the workspace, writes into the copy, runs the command with a wall-clock timeout on top of a processor-time limit, an address-space limit, a file-size limit and a process-count limit, and throws the copy away. That bounds accidents: an infinite loop, a runaway allocation, a fork bomb, a program that fills the disk, and a rollout that deletes the repository cost one temporary directory each.

It is not a security boundary, and the script’s own docstring says so. The child runs as your user, on your filesystem, with your network. Part 25’s sandbox lab builds the container and dedicated user that untrusted code needs, and a reinforcement-learning loop that runs model-written code thousands of times is the strongest case in the whole course for putting the whole trainer inside it rather than trusting a resource limit.

⛨Security

The threat here is not a malicious model. It is a model that samples something destructive by accident, at temperature, several thousand times. Rollouts are generated with sampling on purpose, because a group of identical completions has no advantage signal in it, and that same sampling is what eventually produces the command you would never have written.

What the run actually costs

Here is the arithmetic that decides whether you do this at all. Take a modest configuration: eight rollouts per prompt, which is the documented default for num_generations; a task that takes about six turns; and a test suite that takes twenty seconds.

Quantity	Value	Where it comes from
Rollouts per optimiser step	8	one group
Turns per rollout	6	a short agent task
Completion tokens per rollout	about 1,200	six turns at roughly 200 tokens each
Completion tokens per step	about 9,600	eight rollouts
Test runs per step	8	one per rollout
Test time per step	about 160 seconds	eight runs at twenty seconds, run one after another
Test time for 300 steps	about 13 hours	before a single token has been generated

The last row is the finding. On one machine, with the tests run serially, the sandbox is the budget and the gradient is a rounding error. Everything the agent-reinforcement-learning frameworks do is aimed at that row: verl’s agentic documentation, whose page is titled “Agentic RL Training” and states it was last updated in July 2025 when read on 2026-09-09, separates inference into a server and agent execution into a client precisely so that the accelerator is not idle while a tool runs, and supports SGLang and vLLM as rollout backends with an agent_name field in the data selecting a tool-calling loop or a single-turn one.

GRPO on Qwen3-1.7B with an adapter, eight rollouts of 2,048 tokens, on a 16 GB machine

Policy weights, BF16

3.4 GB

Adapter, gradients and optimiser states

0.3 GB

Rollout key-value cache, 8 sequences

1.9 GB

Activations and logits for the update

1.2 GB

Sandbox processes running the suite

1 GB

Reserved for the operating system

2 GB

Free

6.2 GB

Total

16 GB

Estimate from arithmetic, not a measurement. The policy weights are the BF16 size from the course model reference. The adapter is about 17 million trainable parameters, which is rank 16 on the seven projections of 28 layers at this model's dimensions, at sixteen bytes each. The rollout cache is eight sequences of 2,048 tokens at the model reference's 112 KiB per token. There is no separate reference model on the bar: with an adapter, the reference policy is the same weights with the adapter disabled, which is the saving that makes this fit at all. The last segment is the room the test processes need, and it is the segment people forget.

Per track, the differences are the ones Part 14 already established and one new one. Tracks S and N run the whole loop natively and can afford the 4-billion-parameter policy where memory allows. Track X runs it through the ROCm PyTorch build, with the same caveat about the GPU-visible share of memory that Part 5 measured. Track M has no supported GRPO path in this toolchain: mlx-lm 0.31.3 · verified 2026-09-08 covers LoRA and QLoRA fine-tuning rather than reinforcement learning, so the Mac path for this lesson is to read it, run the reward functions and their demonstration, which are pure Python, and take the supervised route in the previous lesson. The new difference is that the sandbox runs on the processor on every track, so a machine with more processor cores shortens the dominant row of that table more than a faster accelerator does.

The tools, dated

Three projects are worth naming, with what was true when they were read on 2026-09-09.

verl main · verified 2026-09-09 is the one the course already surveys in Part 14. Its agentic documentation describes the client-server agent loop above, rollouts through SGLang or vLLM, and configuration keys including data.return_raw_chat and an asynchronous rollout mode. It is built for clusters; the course surveys it rather than teaching it hands-on.

OpenRLHF main · verified 2026-09-09 is the other Part 14 survey, and its own material describes a system written around 70-billion-parameter models on eight 80 GB accelerators. That is the right scale to know about and the wrong scale to attempt at home.

SkyRL, at https://github.com/NovaSky-AI/SkyRL, is the one aimed squarely at this lesson’s subject. Its README describes a full-stack reinforcement learning library for multi-turn tool use and long-horizon agent tasks, from the Berkeley Sky Computing Lab with Anyscale, under Apache-2.0, with separate components for training, for the agent layer and for a gymnasium of tool-use environments including maths, coding, search and SQL. The repository page listed v0.3.0, dated 16 July 2026, as its latest release when read on 2026-09-09. It is not in this course’s pinned tool set and no page here runs it; it is named because it is where to look if the arithmetic above turns out to be affordable for you.

⌚Version note

Everything in this section is a snapshot dated 2026-09-09. Agent reinforcement learning is the fastest-moving corner of the toolchain this course covers, and a page that told you what these projects do without telling you when it was checked would be worse than useless. Check the repositories before you plan around any of it.

What this can and cannot achieve at your scale

Part 14’s reality check on reinforcement learning and small models asks you to write down the margin before you start, train three seeds, and compare the spread against the difference. Every word of that applies here and more strongly, because an agent suite of fifteen tasks run three times has far fewer independent observations than a maths set of several hundred problems.

Set against that, three things are within reach and two are not.

Within reach: sharpening a behaviour the model already samples sometimes. If the base model calls the right tool six times out of ten, a reward that pays for the schema-valid call can move that, because there is something to reinforce. This is the same mechanism as Part 14’s format reward and it is the reliable win.

Within reach: removing a specific failure mode. A model that keeps calling a tool that does not exist, or keeps emitting the call markup as text, can be trained out of it, because the check is exact and the failure is frequent enough to appear in every group.

Within reach: a shorter trajectory for the same outcome, using the step-budget penalty, provided the outcome reward is strong enough that the model cannot satisfy the budget by giving up.

Not within reach on one machine: teaching a small model to solve tasks it never solves. If every rollout in a group scores zero, every advantage is zero and there is no gradient. This is the failure Part 14 names, and on agent tasks it is the normal case rather than the exception, because a multi-step task with a test-suite verdict is far harder to get accidentally right than an arithmetic problem. Partial credit helps and is why the reward library weights format and schema separately from the outcome, but partial credit on the wrong thing teaches the wrong thing.

Not within reach: the results in the agent-training papers. SWE-smith, published in April 2025, generates 50,000 task instances from 128 repositories and trains a 32-billion-parameter model on the trajectories; the paper reports a 40.2 per cent resolve rate on SWE-bench Verified, described there as the best among open-source models at that time, with the procedure, instances, trajectories and models released under CC BY-SA 4.0. That is a data programme with a compute budget attached, not an evening. What you can take from it is the method, which is the same method as this part: generate many attempts, keep the ones that verify, train on those.

Make the environment repeatable before optimising a policy

An agent task includes the starting files, tools, network access, time limits and verifier. Reset all relevant state between episodes. A stale output file or warmed external cache can reward a later policy for work it did not perform, while a dependency outage can look like a reasoning failure.

Keep tests and reward logic outside the agent’s writable area. Define scoring for invalid calls, timeouts, denied actions and partial completion. Penalising every failed command equally can discourage useful exploration, but ignoring effects can reward unsafe behaviour. Review high-reward trajectories against the intended task rather than trusting the scalar alone.

Start with a small environment and a fixed workflow baseline. Record rollouts per update, policy version, total tool calls and verified success on held-out tasks. Generation and environment execution can dominate training cost, especially on shared local hardware. RL is useful when feedback from interaction improves the task beyond demonstrations within that budget; a rising training reward without held-out success is not sufficient evidence.

Recap

• An agent episode adds an environment to Part 14’s loop: reset, roll out, score, compare within the group, update. TRL’s GRPO trainer has documented arguments for tools and environments as of the version this course pins, checked 2026-09-09.
• Four checkable rewards: format, schema, outcome from a test suite, and a step budget as a penalty. The outcome reward needs a protected-path rule or the model will edit the tests.
• Rollouts execute model-written code. Resource limits bound accidents; they are not a security boundary, and this is the strongest case in the course for running the trainer inside Part 25’s container.
• The arithmetic says the sandbox dominates: eight rollouts a step at twenty seconds of tests each is roughly thirteen hours of test running over three hundred steps, before any tokens are generated. That is what the agent-RL frameworks exist to fix.
• Expect to sharpen behaviour the model already samples and to remove specific failure modes. Do not expect to teach it tasks it never solves: a group where every rollout scores zero has no gradient in it.

Check your understanding

Question 1. What does an agent episode add to the single-turn GRPO loop from Part 14?

• A larger group size
• An environment with state, which must be reset before each rollout and scored after it
• A second model to judge the answer
• A longer learning-rate schedule

Show the answer and why

Answer: An environment with state, which must be reset before each rollout and scored after it

State is the difference. What the model does on turn one changes what turn two sees, so each rollout needs its own fresh environment and its own verdict at the end. That is why one environment instance exists per rollout rather than one per training run.

Question 2. Your test-suite reward has no protected-path rule. What is the model most likely to learn?

• To write better code
• To edit or delete the tests, because that is the cheapest way to make them pass
• To call more tools
• Nothing; the reward is unaffected

Show the answer and why

Answer: To edit or delete the tests, because that is the cheapest way to make them pass

A reward function pays for the cheapest thing that satisfies it. Rewriting an assertion is cheaper than fixing a function, and nothing in "the suite passed" distinguishes the two. The demonstration in this part shows the same attempt scoring zero with the rule and full marks without it.

Question 3. On one machine, which part of an agent reinforcement-learning step usually dominates the wall clock?

• The optimiser step
• Loading the model
• Running the environment and its checks, once per rollout
• Writing the run log

Show the answer and why

Answer: Running the environment and its checks, once per rollout

A group of eight rollouts means eight environment resets and eight scoring runs per optimiser step. A twenty-second test suite therefore costs about 160 seconds a step run serially, which over a few hundred steps is many hours before generation is counted. It is why the agent-RL frameworks separate inference from agent execution so the accelerator is not idle during a tool call.

Question 4. Every rollout in a group scores zero. What follows?

• The advantages within that group are all zero, so the group contributes no gradient
• The task is too hard for the current policy to make progress on
• The learning rate should be raised
• Partial credit on a checkable sub-goal may give the group signal again

Show the answer and why

Answer: The advantages within that group are all zero, so the group contributes no gradient, The task is too hard for the current policy to make progress on, Partial credit on a checkable sub-goal may give the group signal again

GRPO compares each rollout against the mean of its group, so a group with no variation has nothing to learn from. The remedy is easier tasks, a stronger starting policy, or a reward with checkable intermediate structure such as format and schema validity. Raising the learning rate amplifies noise, not signal.

Question 5. True or false: the resource limits in this part’s test-suite reward make it safe to run untrusted model-written code.

• True
• False

Show the answer and why

Answer: False

False. Processor-time, memory, file-size and process-count limits bound accidents: loops, runaway allocations and fork bombs. The child still runs as your user with your filesystem and your network. Sandboxing in the sense that matters is the container and dedicated user from Part 25.

Check my answersClear