320亿参数模型通过去中心化强化学习训练，突破传统集中式瓶颈。
INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning
- 用异步分布式RL在无权限算力节点上训练大模型
- 相比QwQ-32B，在推理任务上表现更优，训练更稳定
- 开源全部代码数据，推动去中心化训练研究
我们提出INTELLECT-2，首个在全局分布环境中对320亿参数语言模型进行强化学习训练的系统。与传统集中式训练不同，INTELLECT-2利用动态异构的无权限算力网络，实现完全异步的强化学习。为支持该架构，我们从零构建了多项组件：引入PRIME-RL训练框架，基于新型组件TOPLOC（验证不可信推理节点的推演结果）和SHARDCAST（高效广播策略权重），实现跨节点协同。此外，我们对标准GRPO训练流程及数据过滤方法进行了关键改进，显著提升训练稳定性，并成功使模型达成预设目标，性能超越同参数规模下的最优基准模型QwQ-32B。我们已开源INTELLECT-2及其全部代码与数据，以促进开放的去中心化训练研究。
原文摘要 · Abstract (English)
We introduce INTELLECT-2, the first globally distributed reinforcement learning (RL) training run of a 32 billion parameter language model. Unlike traditional centralized training efforts, INTELLECT-2 trains a reasoning model using fully asynchronous RL across a dynamic, heterogeneous swarm of permissionless compute contributors. To enable a training run with this unique infrastructure, we built various components from scratch: we introduce PRIME-RL, our training framework purpose-built for distributed asynchronous reinforcement learning, based on top of novel components such as TOPLOC, which verifies rollouts from untrusted inference workers, and SHARDCAST, which efficiently broadcasts policy weights from training nodes to inference workers. Beyond infrastructure components, we propose modifications to the standard GRPO training recipe and data filtering techniques that were crucial to achieve training stability and ensure that our model successfully learned its training objective, thus improving upon QwQ-32B, the state of the art reasoning model in the 32B parameter range. We open-source INTELLECT-2 along with all of our code and data, hoping to encourage and enable more open research in the field of decentralized training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。