让大模型强化学习训练与推理分离，提升资源利用效率。
Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms
- 将推理与训练分置在异构集群，避免任务切换瓶颈。
- 四组测试中收敛速度和最终奖励媲美集中式系统。
- 适合想用边缘设备降本的LLM强化学习研究者。
当前基于强化学习的大语言模型后训练将轨迹采样与策略优化部署在同一GPU集群，导致系统在推理与训练负载间频繁串行切换，违背了分布式训练系统的单程序多数据（SPMD）假设。我们提出Echo系统，通过异构的‘推理’与‘训练’集群，清晰解耦这两个阶段，同时保持统计效率。Echo引入两种轻量级同步协议：顺序拉取模式按API调用刷新策略权重，最小化偏差；异步推送-拉取模式通过重放缓冲区流式传输带版本标签的轨迹，最大化硬件利用率。在地理分布集群上，使用Qwen3-4B、Qwen2.5-7B、Qwen3-30B-A3B-Thinking-2507和Qwen3-32B模型训练四个典型强化学习任务，Echo在收敛速度和最终奖励上达到与全集中式Verl基线相当的效果，同时将轨迹生成卸载至普通边缘硬件。结果表明，大规模语言模型强化学习可借助去中心化异构资源实现数据中心级性能。
原文摘要 · Abstract (English)
Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between inference and training workloads. This serial context switching violates the single-program-multiple-data (SPMD) assumption underlying today's distributed training systems. We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms while preserving statistical efficiency. Echo introduces two lightweight synchronization protocols: a sequential pull mode that refreshes policy weights according to API call for minimal bias, and an asynchronous push-pull mode that streams version-tagged rollouts through a replay buffer to maximise hardware utilisation. Training four representative RL workloads with Qwen3-4B, Qwen2.5-7B, Qwen3-30B-A3B-Thinking-2507 and Qwen3-32B on a geographically distributed cluster, Echo matches a fully co-located Verl baseline in convergence speed and final reward while off-loading trajectory generation to commodity edge hardware. These promising results demonstrate that large-scale RL for LLMs could achieve datacentre-grade performance using decentralised, heterogeneous resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。