PipelineRL让大模型强化学习训练快一倍，同时保持数据新鲜
PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
- 异步并行生成与训练，用飞行中权重更新减少中断
- 128块H100GPU上实现约2倍加速，数据仍高度在策略
- 适合追求高效长序列生成的AI研发团队
强化学习（RL）正被广泛用于提升大语言模型（LLMs）的推理能力。然而，有效扩展这些RL方法面临重大挑战，主要源于难以在不产生过时非策略数据的前提下维持高AI加速器利用率。本文提出PipelineRL，一种在硬件效率与数据在策略性之间取得更优平衡的方法。PipelineRL采用并发异步的数据生成与模型训练机制，其核心是创新的飞行中权重更新。该机制使语言模型生成引擎能在生成标记序列过程中几乎无中断地接收更新后的模型权重，从而最大化加速器利用率和训练数据的新鲜度。在使用128块H100 GPU进行长序列推理任务的实验中，PipelineRL相较于传统RL基线实现了约2倍的加速，同时保持了高度在策略的训练数据。本文还开源了可扩展、模块化的PipelineRL实现。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Large Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high AI accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updates. This mechanism allows the LLM generation engine to receive updated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness of training data. Experiments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately $\sim 2x$ faster learning compared to conventional RL baselines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as a key contribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。