The RL Toolchain: TRL, Unsloth, verl, OpenRLHF and vLLM Rollouts

By the end of this lesson you will be able to name the layers a reinforcement-learning run stands on and say which of them your track supplies; choose between TRL’s three rollout paths with a reason; say what Unsloth changes and what its documentation claims for it; recognise when a job has outgrown one machine and which of verl and OpenRLHF is the shape of the answer; and predict the memory a GRPO run will need before you start it.

The stack

A reinforcement-learning run is more layers than a fine-tune, and the extra ones are where jobs fail.

What a GRPO run on one machine stands on

each layer can fail on its own

1. Accelerator and driverCUDA on Tracks S and N, ROCm on Track X, Metal through MPS on Track M.decides everything above
2. PyTorch buildThe CUDA, ROCm or MPS wheel from Part 11. Training happens here whichever trainer you use.
3. TrainerTRL GRPOTrainer, or Unsloth wrapping it, or verl or OpenRLHF at cluster scale. Owns the objective, the advantages and the optimiser step.
4. Rollout engineThe model's own generate(), vLLM in colocate or server mode, or an OpenAI-compatible server. Produces the G completions per prompt.the wall clock lives here
5. Reward functionsYour Python. Deterministic, tested, and the only statement of what the run is optimising.you own this
6. Run log and evaluationOne JSON line per run, and pass@1 on held-out problems before and after.the only evidence

Compared with Part 11's supervised fine-tuning stack, two layers are new: the rollout engine and the reward. Both are yours to choose, and both are where a first attempt goes wrong.

TRL, and its three rollout paths

TRL TRL 1.12.0 · verified 2026-09-08 is the reference implementation for this course, because it is already in the environment from Part 11 and because its GRPOTrainer is a few lines away from the SFTTrainer you have used. The quick start is genuinely this short:

Fragment — not complete on its own

trainer = GRPOTrainer(
    model="Qwen/Qwen3-1.7B",
    reward_funcs=[numeric_reward, format_reward],
    train_dataset=dataset,          # prompt-only, plus any columns the rewards need
)
trainer.train()

The decision that matters is where the completions come from.

In-process generation is the default and needs nothing installed. The trainer calls the model’s own generate(). It is correct by construction, because the weights doing the sampling are the weights being trained, and it is slow, because a training-shaped forward pass is not an inference engine.

vLLM in colocate mode runs vLLM inside the trainer process, sharing the GPU. TRL describes it as avoiding “launching a separate server”, which “can improve GPU utilization, but may lead to memory contention on the training GPUs”. It is the default when use_vllm=True and the right answer on a single accelerator.

vLLM in server mode runs vLLM as a separate process. TRL’s description of the loop is the useful part: the trainer “asks for completions on the OpenAI-compatible /v1/completions endpoint, sending the prompt token IDs”, and “after each optimizer step the trainer streams the updated weights into it over NCCL”. That last clause is what makes it a rollout engine rather than a generic server.

⌚Version note

Two constraints from TRL’s vLLM page, read on 2026-09-09, that will bite before anything else does.

First, the supported range: “TRL currently only supports vLLM versions from 0.19.0 to 0.27.1.” This course pins vLLM at vLLM 0.28.0 · verified 2026-09-08, which is outside that range. For serving, Part 9’s pin is the right one; for TRL rollouts, install a vLLM inside TRL’s documented range in the training environment, or use in-process generation. Check both projects’ release notes before assuming this has been resolved.

Second, server mode is written for machines with spare accelerators: “the vLLM server and the trainer must run on separate CUDA devices to prevent conflicts”. On one GPU the documented answer is the ipc weight-transfer backend, which the page says to “use instead when the trainer and the server share a GPU”. Colocate mode is simpler and is what the lab uses.

Also noted on the same page: “trl vllm-serve is deprecated: it now only builds this command and runs vLLM’s server.”

TRL also documents an escape hatch for tracks with no vLLM: GRPOTrainer accepts a rollout_func, “a function to use for generating completions”, which “must return a dict with prompt_ids, completion_ids, and logprobs fields”, and which the documentation marks as “experimental and may change or be removed at any time without prior notice”. The lab uses it to sample from llama.cpp’s server, whose /completion endpoint documents an n_probs parameter returning completion_probabilities with the token identifiers and log-probabilities such a function needs.

!Important

An OpenAI-compatible server is not a drop-in rollout engine, and the reason is weight synchronisation. vLLM in TRL’s server mode receives the updated weights after every optimiser step. llama.cpp’s server README documents no equivalent endpoint, so a llama-server rollout path keeps sampling from the checkpoint it loaded while the policy moves away from it. That is off-policy sampling with an uncorrected mismatch, and it grows with every step. Use it for a short demonstration run on a track with no alternative, keep the run short, and record in the log that the rollouts were stale.

Unsloth

Unsloth, pinned at Unsloth 0.1.807-beta (GitHub tag) · verified 2026-09-08, wraps the same trainers with custom kernels and memory work. Its reinforcement-learning documentation is specific about what that buys, and the figures below are the vendor’s own, quoted as published rather than measured here.

Vendor specification, not measuredMemory for GRPO, as published by Unsloth

Case	Reported figure
Llama 3.1 8B, 20K context, 8 generations per prompt, standard implementation	510.8 GB total VRAM
The same case with Unsloth	54.3 GB total VRAM
Smallest local training run named in the guide	5 GB VRAM, for a model of 1.5B parameters or less
QLoRA rule of thumb given in the guide	model parameters in billions is roughly the VRAM in GB
LoRA at 16-bit relative to QLoRA at 4-bit	at minimum 4x more VRAM

not stated in the source beyond the model and context length · Unsloth with vLLM generation, against an unnamed standard implementation Unsloth documentation read 2026-09-09 · Llama 3.1 8B for the first two rows, 4-bit QLoRA except where stated · 20,000 tokens of context · 2026-09-09

Published by Unsloth in its reinforcement-learning guide, not measured by this course. The comparison baseline is described as a standard implementation without naming its configuration, so treat the ratio as an order of magnitude rather than a number to plan with. The validation pass will measure the course's own runs on the four tracks.

The same guide is unusually candid about expectations, and these sentences are worth more than the memory table: it advises applying GRPO “to a model at least 1.5B in parameters”, says to “wait for at least 300 steps for the reward to actually increase”, suggests “at least 500 rows of data” for good results, and warns to expect “a minimum of 12 hours” for a decent result. The labs in this part are deliberately shorter than that, which is why they are written to demonstrate the machinery and measure a small change rather than to produce a reasoning model.

✱Hardware note

Unsloth’s guide, read on 2026-09-09, does not state a platform exclusion and references a free Colab GPU as its entry point; it does not document AMD or Apple support for this path either. Take that as “not addressed” rather than as either a yes or a no, and treat Tracks X and M as the plain TRL path until you have tested otherwise on your own machine.

verl and OpenRLHF

Both are cluster-scale systems, and both are worth knowing about precisely so you can recognise the point at which one machine is the wrong tool.

verl (https://github.com/volcengine/verl) began at ByteDance’s Seed team and is maintained by a community that its README lists as including Anyscale, LMSys.org, the Alibaba Qwen team, Tsinghua University and UC Berkeley. It is Apache-2.0 licensed. Its algorithm list is the whole of this part and more: PPO, GRPO, GSPO, DAPO, Dr. GRPO, RLOO, REINFORCE++, ReMax and PRIME among others. It takes rollouts from vLLM, SGLang or Hugging Face Transformers, trains with FSDP, FSDP2 or Megatron-LM, and its README describes support for models up to 671 billion parameters and scaling “to hundreds of GPUs with expert parallelism”, on NVIDIA GPUs, AMD ROCm and Ascend NPUs.

OpenRLHF (https://github.com/OpenRLHF/OpenRLHF) describes itself as combining “Ray + vLLM distributed architecture with a unified agent-based design paradigm”, and is also Apache-2.0. It supports PPO with a full critic, GRPO, RLOO, REINFORCE++ and its baseline variant, and the direct preference family including DPO and IPO. Its three pillars are Ray as the “distributed scheduler and controller”, separating actor, reward, reference and critic models onto different GPUs; vLLM for generation, on the reasoning that “RLHF training spends 80% of the time on sample generation”; and DeepSpeed ZeRO-3 for memory-efficient training. Its README discusses models of 70 billion parameters and above and gives tuning guidance for machines such as eight A100s.

Neither is pinned in this course’s version table, because neither is on the path any lab takes. The signal that you need one of them is structural rather than a matter of taste: you have more than one machine’s worth of accelerators, you want a separately trained reward model as well as a policy, and you are prepared to operate a scheduler. Until all three are true, TRL on one box is less to go wrong.

Hosted alternatives

Several vendors will run a reinforcement-learning job for you against an uploaded dataset. That is a reasonable choice for some work and it is not what this course teaches, for three reasons that have nothing to do with quality. Your prompts and your reward function are the two most revealing artefacts in the whole pipeline, and both leave the machine. The result is not reproducible by you, because you do not control the versions. And the thing being taught here is the mechanism, which is much harder to learn from an API that returns a finished adapter.

The memory arithmetic

A GRPO run holds more than a fine-tune does. Count five things: the policy weights, the trainable parameters with their gradients and optimiser states, the reference model if beta is above zero, the rollout engine’s own memory, and the logits and activations of the update.

GRPO on a 16 GB machine: Qwen3-1.7B, rank-16 adapter, 8 rollouts of 512 completion tokens, beta above zero

Policy weights, BF16

3.4 GB

Reference model, BF16 (only when beta is above zero)

3.4 GB

Adapter, gradients and Adam states

0.3 GB

Rollout key-value cache, 8 sequences

0.7 GB

Logits and activations for the update

2 GB

Reserved for the operating system

2 GB

Free

4.2 GB

Total

16 GB

Estimate from arithmetic, not a measurement. Weights are the BF16 figure for Qwen3-1.7B from the course model reference, twice because a nonzero beta loads a reference copy; the adapter term is Part 11's 16 bytes per trainable parameter on about twenty million parameters; the rollout figure is 8 sequences of 768 tokens at the model's documented 112 KiB per token of key-value cache; the logits term is 8 by 512 by Qwen3's 151,936-entry vocabulary at 2 bytes, which is about 1.2 GB, plus an allowance for the rest of the activations.

▤Memory budget

Setting beta to zero, which is TRL’s default, removes the second 3.4 GB block entirely: the documentation states plainly that at 0.0 the “reference model [is] not loaded”. On a 16 GB machine that is the difference between comfortable and tight, and it is a decision about the objective disguised as a decision about memory. The previous lesson says when it is safe.

GRPO on a 128 GB machine: Qwen3-4B with vLLM in colocate mode

Policy weights, BF16

8 GB

Reference model, BF16

8 GB

Adapter, gradients and Adam states

0.5 GB

vLLM allocation at the default 0.3 memory fraction

38.4 GB

Logits and activations for the update

4 GB

Reserved for the operating system

4 GB

Free

65.1 GB

Total

128 GB

Estimate from arithmetic, not a measurement. The policy and reference are the BF16 figure for Qwen3-4B from the course model reference. Colocate mode runs vLLM inside the trainer process, and its allocation is not a function of the model at all: TRL documents vllm_gpu_memory_utilization with a default of 0.3, so on a 128 GB machine the engine reserves roughly 38 GB for its own copy of the weights and its paged key-value pool, whatever the model size. That single default is the largest term in this bar, and it is the diagram to look at before deciding that a rollout engine is free.

What each track can run

Track S — NVIDIA DGX Spark

TRL with vLLM in colocate mode, and the memory to hold a reference model as well. The 128 GB unified pool is what makes the second diagram above comfortable rather than a squeeze. Check vLLM’s version against TRL’s supported range first, and remember that this is an aarch64 machine, so wheels come from NVIDIA’s arm64 builds as Part 11’s environment lesson describes.

Track X — AMD Ryzen AI Max+ 395Partial

vLLM's GPU installation page read on 2026-09-09 lists Ryzen AI MAX (gfx1151) among its ROCm targets with pre-built wheels, which this course's validation pass has not yet exercised; PyTorch on ROCm for this chip is itself unqualified in AMD's documentation.

Plain TRL with in-process generation is the path that is certain to work, on ROCm if PyTorch sees the GPU and on the CPU if it does not. If the ROCm vLLM wheels install and run on your machine, colocate mode is worth trying and worth writing down as a result either way. The llama-server rollout path is available and carries the stale-weights caveat above.

Track M — Apple siliconPartial

mlx-lm's README, read on 2026-09-09, documents low-rank and full-model fine-tuning and no reinforcement-learning trainer; vLLM's documented GPU path does not cover macOS.

TRL on PyTorch’s MPS backend in float32, with in-process generation, on a model of 0.5 to 1.7 billion parameters. There is no rollout engine to add: vLLM does not run here, and mlx-lm has no GRPO to offer. Expect this to be the slowest of the four tracks and plan the run length accordingly.

Track N — NVIDIA desktop or laptop

TRL with vLLM in colocate mode on one card, subject to the same version constraint. A 16 GB card runs the first diagram above with beta at zero; 24 GB runs it with a reference model and room to raise the rollout count. Server mode is documented for machines with a second card to give it.

Pending validationWhat to measure per track for a 200-step GRPO run on the lab's settings

Track	Rollout path	Wall clock	Peak memory, GB	Seconds per step
S: DGX Spark, 128 GB	vLLM colocate	to be measured	to be measured	to be measured
X: Ryzen AI Max+ 395	in-process, ROCm or CPU	to be measured	to be measured	to be measured
M: Apple silicon	in-process, MPS float32	to be measured	to be measured	to be measured
N: NVIDIA desktop or laptop	vLLM colocate	to be measured	to be measured	to be measured

the four platform tracks, one machine each · TRL GRPOTrainer with a PEFT LoRA adapter; vLLM where the track supports it trl 1.12.0, transformers 5.16.1, peft 0.20.0; vLLM within TRL's documented range · Qwen3-1.7B with a rank-16 LoRA adapter, BF16, float32 on Track M · 1,024 tokens of context · 2026-09-09

Not yet run on hardware on any track. Fill in your own four figures from the run log after the GRPO lab, and compare the peak against the arithmetic in the diagrams above rather than against another machine.

The first run on a new machine

Do not start with the lab. Start with a run designed to fail fast, because six layers means six places to be wrong and the useful skill is telling them apart in minutes rather than in an evening.

Take the smallest model that has a chat template, ten training prompts, four rollouts and ten steps, and watch what the first thirty seconds print. Each layer announces itself in order, and the first line that is missing or wrong tells you which one to fix.

1. The device line. If it says cpu on a machine with an accelerator, the problem is the PyTorch build, not the trainer, and Part 11’s environment lesson is where it lives. Fix this before anything else: everything above will be slow and misleading otherwise.
2. The trainable-parameter count. PEFT prints it. A count of zero means target_modules matched nothing, which is a name mismatch rather than a memory problem.
3. The reward line. The training script prints the reward functions and their weights. If a reward you meant to include is missing, you are about to optimise something else.
4. The first logged step. A reward of exactly zero on step one is the group-collapse failure from the previous lesson, and it is far cheaper to diagnose now than after two hours.
5. The rollout path. With vLLM enabled, an engine that fails to initialise usually fails on its version rather than on memory, and the message says so. With the served path, the first failure is normally the response shape rather than the network.

Ten steps of a 0.6 billion parameter model takes a couple of minutes on any track and tells you whether the machinery is assembled. Only then is it worth spending an hour on a real run.

✦Tip

Keep the smoke-test command in your notebook beside the real one. Every time you change environment, container, driver or library version, run the small one first. The habit costs two minutes and repays itself the first time a wheel update silently moves you onto the CPU.

What to write down before you scale up

Three things, from the smoke test, that decide the shape of the real run. The seconds per step, which multiplied by your step count is the wall clock you are committing to. The peak memory, against the arithmetic in the diagrams above, because a prediction that was wrong by a factor of two is worth understanding before you make the run four times larger. And the rollout path, which is the single field in the run log that most often explains why two runs disagree.

Track the policy version that generated each rollout

An RL loop alternates generation, verification and parameter updates. If generation runs in another process, it must use the policy version the algorithm expects. A server still holding the initial checkpoint can produce valid text while silently turning an intended on-policy experiment into something else.

Record the policy revision or update step with rollout batches and verify the weight-refresh mechanism documented by the selected trainer. Start with a tiny in-process run where ownership is easier to inspect. Move generation to another engine only after you can explain how weights, tokenisation and sampling remain consistent.

Budget memory for the complete loop: trainable policy, reference if used, optimiser state, rollout buffers and the serving engine’s allocations. A generator and trainer sharing one accelerator can each fit alone and fail together. Measure generation, verification and update time separately. If verification dominates, faster sampling will not remove the bottleneck. An RL stack is operationally correct only when the feedback is computed from the intended policy and delivered to the intended update.

Recap

A reinforcement-learning run stands on six layers, of which two are new relative to a fine-tune: the rollout engine and the reward function. TRL’s GRPOTrainer offers in-process generation, vLLM in colocate mode and vLLM in server mode, plus an experimental rollout_func for anything else; its documented vLLM version range does not currently include this course’s serving pin, and server mode expects a spare accelerator. Unsloth publishes substantial memory savings and equally useful advice about how long a real run takes. verl and OpenRLHF are cluster systems, both Apache-2.0, and the signal to reach for one is structural rather than aspirational. The memory to plan for is five terms, and the largest single decision is whether beta is zero, because a nonzero beta loads a second copy of the model.

Check your understanding

Question 1. You install the course's pinned vLLM and enable use_vllm=True in GRPOConfig. What should you check first?

• That the GPU has enough memory
• That the installed vLLM version falls inside the range TRL documents as supported, which the course's serving pin does not
• That the reward functions return floats
• That the dataset is prompt-only

Show the answer and why

Answer: That the installed vLLM version falls inside the range TRL documents as supported, which the course's serving pin does not

TRL states a supported range for vLLM, and a serving pin chosen in Part 9 was chosen for serving. Two projects with independent release cadences will drift apart; the fix is to install the trainer's supported version in the training environment, not to hope.

Question 2. Why is an ordinary OpenAI-compatible server not equivalent to vLLM in TRL's server mode?

• It is slower
• It has no documented way to receive the updated weights after each optimiser step, so it keeps sampling from a policy the trainer has already moved away from
• It cannot return token identifiers
• It does not support batching

Show the answer and why

Answer: It has no documented way to receive the updated weights after each optimiser step, so it keeps sampling from a policy the trainer has already moved away from

TRL streams weights into the vLLM server after every step, which is what keeps the rollouts on-policy. Without that path the mismatch grows with every step, which is why the lab treats the llama-server route as a short demonstration and records it as stale in the run log.

Question 3. Setting beta to zero in GRPOConfig has two consequences. Which are they? Select all that apply.

• The reference model is not loaded, saving a full copy of the weights
• The kl metric is no longer logged
• The reward functions stop being applied
• The rollouts are generated greedily

Show the answer and why

Answer: The reference model is not loaded, saving a full copy of the weights, The kl metric is no longer logged

It is a memory decision and a constraint decision at once. TRL documents the default as 0.0 with the reference model not loaded, and logs kl only when beta is nonzero.

Question 4. Which situation genuinely calls for verl or OpenRLHF rather than TRL on one machine?

• You want to run GRPO on a 4B model
• You have more accelerators than one machine holds, you need a separately trained reward model alongside the policy, and you are willing to operate a scheduler such as Ray
• You want a lower memory footprint
• You need a verifiable reward function

Show the answer and why

Answer: You have more accelerators than one machine holds, you need a separately trained reward model alongside the policy, and you are willing to operate a scheduler such as Ray

Both are cluster systems whose value is placing several models on separate devices and coordinating them. On one box that machinery is cost without benefit, and TRL has fewer moving parts.

Question 5. In colocate mode, why does the 128 GB memory diagram show the model weights twice?

• Because the reference model is always loaded
• Because vLLM runs inside the trainer process with its own copy of the weights and its own paged key-value pool, sized by a memory fraction that defaults to 0.3, in addition to the policy the trainer holds
• Because bfloat16 needs two copies
• Because the adapter is merged during training

Show the answer and why

Answer: Because vLLM runs inside the trainer process with its own copy of the weights and its own paged key-value pool, sized by a memory fraction that defaults to 0.3, in addition to the policy the trainer holds

The engine is a second model in memory, not a view onto the first, and its allocation is set by vllm_gpu_memory_utilization rather than by the model size. That is the price of the speed, and it is the reason a rollout engine is a budget line rather than a free improvement.

Check my answersClear