CoPRIS提升大模型强化学习效率，避免长轨迹拖慢训练
CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling
- 固定并发采样数，提前终止并复用未完成轨迹
- 实验显示训练速度最高提升1.94倍，性能相当或更优
- 适合追求高效训练的大模型强化学习研究者
强化学习（RL）后训练已成为提升大语言模型（LLMs）能力的主流范式。现有大多数基于LLM的RL系统采用完全同步方式，需等待整个批次的轨迹生成完成才能继续训练，导致严重效率低下，长轨迹会阻塞整个流程并使大量GPU闲置。为此，我们提出并发控制的部分采样与重要性采样方法（CoPRIS），通过保持固定数量的并发采样、在收集足够样本后提前终止，并在后续采样中复用未完成轨迹，缓解长尾延迟问题。为减轻离策略轨迹的影响，引入跨阶段重要性采样修正机制，将前一策略缓冲的日志概率与当前策略重新计算的概率拼接用于修正。在挑战性数学推理基准上的实验表明，CoPRIS可实现最高1.94倍的训练加速，同时保持相当或更优的性能。CoPRIS代码已公开于https://github.com/777pomingzi/CoPRIS。
原文摘要 · Abstract (English)
Reinforcement learning (RL) post-training has become a trending paradigm for enhancing the capabilities of large language models (LLMs). Most existing RL systems for LLMs operate in a fully synchronous manner, where training must wait for the rollout of an entire batch to complete. This design leads to severe inefficiencies, as extremely long trajectories can stall the entire rollout process and leave many GPUs idle. To address this issue, we propose Concurrency- Controlled Partial Rollout with Importance Sampling (CoPRIS), which mitigates long-tail inefficiencies by maintaining a fixed number of concurrent rollouts, early-terminating once sufficient samples are collected, and reusing unfinished trajectories in subsequent rollouts. To mitigate the impact of off-policy trajectories, we introduce Cross-stage Importance Sampling Correction, which concatenates buffered log probabilities from the previous policy with those recomputed under the current policy for importance sampling correction. Experiments on challenging mathematical reasoning benchmarks show that CoPRIS achieves up to 1.94x faster training while maintaining comparable or superior performance to synchronous RL systems. The code of CoPRIS is available at https://github.com/777pomingzi/CoPRIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。