arXiv:2606.02218cs.LGcs.AI2026-06被引 2

动态调整训练分组大小,减少慢节点拖累,提升强化学习效率。

Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing

论文配图:Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing
图 1 · 摘自论文原文
  • 根据采样行为实时调整分组规模,缓解慢节点影响。
  • 在多种算法和基线上,显著降低慢节点发生率,提升实际训练速度。
  • 适合追求高效稳定训练的RL研究者与工程团队使用。

同步强化学习方法如组相对策略优化(GRPO)虽能提供稳定可复现的在线策略训练,但极易受慢节点影响——单个异常长的采样序列会延迟整个分组的奖励计算与参数更新。随着分组规模增大,该问题愈发严重,造成大分组优势与同步开销之间的矛盾。本文提出迟滞感知分组控制(SAGC),一种在线动态分组控制器,依据观测到的采样行为自适应调整训练分组大小。SAGC将分组选择建模为在线约束优化问题,旨在保留大分组优势的同时,控制长期慢节点事件的发生率。在同步GRPO与DAPO训练中,以及在原始和强基线之上,SAGC均持续降低慢节点发生率,提升墙钟效率,并实现相当或更优的训练奖励。进一步实验表明,这些收益可迁移至模型最终质量:在下游推理任务中,SAGC表现优于或等同于最强静态分组基线,且常生成更短输出,无需额外长度惩罚。

原文摘要 · Abstract (English)

Synchronous reinforcement learning methods such as Group Relative Policy Optimization (GRPO) provide stable and reproducible on-policy training, but they are highly vulnerable to stragglers, a single unusually long rollout can delay reward computation and parameter updates for the entire group. This problem becomes more severe as group size increases, creating a tension between the benefits of larger groups and the wall-clock cost of synchronization stalls. We propose Straggler-Aware Group Control (SAGC), a dynamic group-size controller that adapts the training group online based on observed rollout behavior. SAGC formulates group-size selection as an online constrained optimization problem, seeking to retain the benefits of larger groups while controlling the long-term rate of straggler events. Across synchronous GRPO and DAPO training, and on top of both vanilla and strong engineered baselines, SAGC consistently reduces straggler incidence and improves wall-clock efficiency while achieving competitive or better training reward. We further show that these gains transfer to final model quality: SAGC is competitive with or better than the strongest static group-size baseline on downstream reasoning benchmarks, and often produces shorter outputs without any explicit length penalty. These results position dynamic group control as a practical way to make synchronous on-policy RL more efficient and robust.

强化学习分布式训练效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。