ECHO-2让大模型强化学习更省钱，通过分布式推理降低训练成本。
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
- 用远程推理节点+分层广播，实现训练与推理重叠运行。
- 在4B到32B参数模型上，成本降低超30%且奖励不下降。
- 适合需要大规模低成本强化学习的团队或研究机构。
强化学习是大语言模型后训练的关键环节，涉及多次生成回放、奖励评估与集中学习的交互。分布式回放执行可利用更廉价的推理资源，但带来跨区域协调与策略传播延迟的挑战。我们提出ECHO-2，一个面向后训练的分布式强化学习框架，支持远程推理工作者与不可忽略的传播延迟。该框架结合集中式学习与分布式回放，将有界策略过时视为用户可控参数，使回放生成、策略传播与训练过程实现重叠。我们引入基于重叠的容量模型，关联训练时间、传播延迟与回放吞吐量，给出维持学习器利用率的实际资源配置规则。为缓解传播瓶颈并降低成本，ECHO-2采用同级辅助流水线广播与成本感知的异构工作者激活机制。在真实广域带宽条件下，对4B至32B参数的LLM进行GRPO后训练实验表明，ECHO-2显著提升成本效率，同时保持与强基线相当的强化学习奖励。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。