揭示异步强化学习中延迟数据对训练稳定性的关键影响。
Staleness-Learning Rate Scaling Laws for Asynchronous RLHF
- 显式建模行为策略，区分梯度映射与真实导数
- 延迟滚动生成导致每步梯度偏差为 O(S * eta)
- 提出双约束稳定条件，适合大规模异步RLHF系统设计
高吞吐强化学习人类反馈（RLHF）系统常将采样生成与策略优化解耦，导致学习器使用存在延迟的采样数据。本文研究异步GRPO中这种延迟的影响。通过在GRPO代理目标中显式引入行为策略，并区分学习器使用的代理梯度映射与依赖分布的总体目标的真实全导数，在局部有界性、分布平滑性和行为策略平滑性假设下，证明延迟采样会引入每步代理梯度偏差量级为O(S * eta)，其中S为最大采样延迟，eta为学习率。进一步推导出条件崩溃时间标度律：当周期内漂移低于批次裁剪半径时，崩溃主要由累计学习器漂移T * eta决定；当延迟约束活跃时，稳定性则显式依赖于S * eta。由此得到双重约束稳定条件 eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}，解释了为何在有限时域下最大稳定学习率看似对延迟不敏感。
原文摘要 · Abstract (English)
High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make the behavior policy explicit in the GRPO surrogate objective and distinguish between the surrogate-gradient mapping used by the learner and the true total derivative of a distribution-dependent population objective. Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, we show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. We further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields a two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。