Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

TL;DR-- For those of you who don't have time to read 5,000 words about async RL plumbing (we get it, you have models to train):

The problem:In synchronous RL (reinforcement learning) training, data generation (model inference to create data samples) dominates wall-clock time -- a single batch of 32K-token rollouts on a 32B (32-billion parameter) model can takehours,while the GPUs used for training remain idle.The solution everyone converged on:Disaggregate (separate) inference and training onto different GPU pools, connect them with a rollout buffer (temporary storage for model outputs), and transfer weights asynchronously (without waiting), so neither side waits for the other.We surveyed 16 open-source librariesthat implement this pattern and compared them across 7 axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends.Key findings:Ray dominates orchestration (8/16 surveyed distributed computing libraries). The NCCL (NVIDIA Collective Communications Library) broadcast is the default method for transferring model weights. Staleness management refers to how outdated data samples are handled, ranging from simply dropping old samples to using advanced importance-sampling correction. LoRA (Low-Rank Adaptation) training is sparsely supported. Distributed MoE (Mixture of Experts) support is the emerging differentiator.If you'd rather skip straight to the good part, here's the full comparison table (no reading required, we won't judge).

But seriously, if you stick around, you might learn a thing or two about why your GPUs are idle 60% of the time.

Click to expand Table of Contents

• 1. Motivation: From synchronous RL training to async architectures
• 2. Libraries Surveyed
• 3. The Comparison Framework: Seven Axes
• 4. Global Overview: Sixteen Libraries at a Glance
• 5. The Next Wave: Design Implications

• 5.1 Critic-Free Algorithms: Memory Freed, But Weight Sync Pressure Increases

• 5.2 Process Rewards: A New Synchronisation Barrier

• 5.3 Multi-Agent Co-Evolution: The Straggler Problem Compounds

• 5.4 Training-Inference Mismatch: The Deepseek v3.2 MoE Case Study

• 5.5 Distillation: The Same Async Problem Under a Different Name

• 6. Design Choices for TRL's Async Trainer

Async RL training has emerged as the dominant paradigm for post-training at scale. Several trends in modern post-training have made synchronous training loops nearly impossible to scale:

**Long rollouts from reasoning models.**Chain-of-thought training produces very long rollouts, and a single synchronous generation batch can take hours to complete on a single GPU. During all of that time, training GPUs sit completely idle.Value-function-free trainers like GRPOuse group-relative advantages. This means generating up to G times more rollouts per prompt, and the entire batch is gated by the slowest completion in the group.**The rise of agentic RL training.**When models interact with tools, sandboxes, and external environments across multi-turn trajectories, rollout lengths and latencies become highly variable. A simple API call might return in seconds, while a complex reasoning chain with tool use can run for minutes or hours. MiniMax's Forge framework, used to train MiniMax-M2.5, illustrates the scale this reaches in practice: context lengths up to 200K tokens, over a hundred thousand distinct agent scaffolds and environments, and daily throughput on the order of millions of samples. At this scale, any synchronous barrier between generation and training becomes a severe bottleneck. The straggler problem alone (where a handful of slow rollouts block an entire batch) can idle hundreds of GPUs.

The open-source ecosystem has converged on a common architectural response: disaggregate inference from training onto separate GPU pools, connect them with a rollout buffer, and let both sides run concurrently.

We are developing a new async trainer for TRL, one of the most widely used libraries for model post-training. To guide our design, we surveyed sixteen open-source libraries that were built from the ground up around asynchronous training and compared them across seven axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends. This article distills the design principles we extracted from that survey.

Beyond RL, the need for async infrastructure is increasingly evident. For example, on-policy distillation, where a student generates sequences and a teacher scores them, mirrors GRPO but swaps the reward function for a teacher forward pass. Recognizing this structural similarity, everything in this survey applies equally to async distillation. We'll return to this broader point in Section 5.

TRL's current GRPOTrainer

implements the full GRPO loop (prompt sampling, generation, reward scoring, advantage computation, gradient update, and weight sync) in a single synchronous training_step()

call. This design is simple and correct, but it cannot overlap generation with training, leaving significant GPU utilisation on the table.

Looking at the GRPOTrainer

, we have the following phases sequentially within each training step:

**Prompt sampling:**draw a batch of prompts from the dataset. Nothing crazy here, let's continue.Generation, callsmodel.generate()

(or forward requests to a vLLM server) to produce G completions per prompt. This is autoregressive and dominates wall-clock time.**Reward scoring:**evaluate each completion against one or many reward functions.**Advantage computation****Forward and backward passes:**compute the clipped policy gradient loss and backpropagate.Optimizer step, update model weights.Weight sync, push updated weights to the inference engine (vLLM) so the next generation uses the new policy.

Each phase blocks until completion before the next begins. The timeline looks like this:

TRL offers the steps_per_generation

config option to reuse a single set of rollouts across multiple gradient steps (temporal reuse), amortizing the generation cost. But the generation call itself remains fully synchronous and blocking; the trainer cannot begin gradient computation until every completion in the batch has finished.

The library also supports running vLLM in server

mode as a separate process. It frees the training GPU during generation, but two hard synchronisation barriers remain: the HTTP calls until all completions return, and the weight sync blocks both the trainer and vLLM during the transfer.

Before discussing async training, it is essential to understand the two deployment topologies for RL training with a separate inference engine:

Colocated modeplaces inference and training on thesame set of GPUs. A single GPU (or TP group) holds both the training model (under FSDP or ZeRO) and the inference engine (vLLM or SGLang). Only one role is active at a time: during generation, the training model's parameters may be offloaded or resharded into an inference-friendly layout (e.g., from FSDP shards to vLLM's tensor-parallel layout); during training, the inference engine is paused or put to sleep. Weight "sync" is essentially free; it is at most an in-place resharding on the same GPU, not a network transfer. The advantage of the colocated mode is simplicity and cost; you need fewer total GPUs. The fundamental limitation is thatinference and training cannot overlap. For example, here is the Trl with vllm incolocate_mode

:

Disaggregated modeplaces inference and training onseparate GPU pools. The inference pool runs vLLM or SGLang continuously; the training pool runs the optimizer continuously. The two pools communicate via a weight synchronisation protocol (NCCL broadcast, filesystem checkpoint, HTTP, etc.) and a data transfer mechanism (Ray object store, Redis streams, shared memory, etc.