让多个机器共享强化学习经验，高效训练大模型。
Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing
- 各节点自主运行模型，通过共享推理数据实现去中心化协同优化。
- 实验显示累计奖励提升最高达94%，且支持数千节点异构设备并行。
- 适合资源有限但想参与大规模模型训练的研究者或开源社区成员。
使用强化学习对语言模型进行后训练可增强其复杂推理能力，而无需监督微调，如DeepSeek-R1-Zero所示。然而，有效利用强化学习训练语言模型需要大规模并行推理，带来延迟、内存和可靠性等技术挑战，以及不断增长的财务成本。本文提出一种完全去中心化、异步的强化学习后训练算法——群体采样策略优化（SAPO）。SAPO适用于异构计算节点组成的分布式网络，每个节点独立管理自己的策略模型，并与其他节点“共享”轨迹数据；无需假设延迟、模型一致性或硬件统一性，节点也可选择独立运行。该设计避免了扩展强化学习后训练的常见瓶颈，同时开启了新可能：通过共享网络中的采样轨迹，使关键洞察“灵光一现”得以传播，从而加速学习过程。本文展示在受控实验中，SAPO实现了最高达94%的累积奖励提升。此外，我们还分享了来自Gensyn社区成员在开放源代码演示中，使用不同硬件与模型在数千节点网络上运行该算法的测试洞察。
原文摘要 · Abstract (English)
Post-training language models (LMs) with reinforcement learning (RL) can enhance their complex reasoning capabilities without supervised fine-tuning, as demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs requires significant parallelization to scale-up inference, which introduces non-trivial technical challenges (e.g. latency, memory, and reliability) alongside ever-growing financial costs. We present Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm. SAPO is designed for decentralized networks of heterogenous compute nodes, where each node manages its own policy model(s) while "sharing" rollouts with others in the network; no explicit assumptions about latency, model homogeneity, or hardware are required and nodes can operate in silo if desired. As a result, the algorithm avoids common bottlenecks in scaling RL post-training while also allowing (and even encouraging) new possibilities. By sampling rollouts "shared" across the network, it enables "Aha moments" to propagate, thereby bootstrapping the learning process. In this paper we show SAPO achieved cumulative reward gains of up to 94% in controlled experiments. We also share insights from tests on a network with thousands of nodes contributed by Gensyn community members running the algorithm on diverse hardware and models during an open-source demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。