提出GAC方法解决大模型强化学习异步训练不稳问题
GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
- 通过梯度投影控制异步更新方向，抑制梯度对齐效应
- 实验证明可在高延迟下实现与同步训练相当的稳定性
- 适合大规模异步强化学习系统，尤其是大语言模型训练
异步执行对扩展强化学习至现代大模型任务（如大语言模型和智能体）至关重要，但会根本性改变优化行为。现有异步强化学习研究多关注吞吐量与分布修正，我们发现直接应用于策略梯度更新的异步机制会引发质变的训练动态，导致严重不稳定。通过系统的实证与理论分析，我们识别出不稳定的显著特征：异步训练中连续策略梯度的余弦相似度持续偏高，而同步训练中则近似正交。这种陈旧对齐梯度效应加剧了更新相关性，增加了溢出与发散风险。基于此，我们提出梯度对齐控制（GAC），一种基于动力学感知的稳定化方法，通过梯度投影调节异步更新沿陈旧对齐方向的进展。在有限陈旧度条件下建立收敛保证，并实证表明GAC能恢复稳定的在线策略训练动态，在高陈旧度下仍可达到同步基线性能。
原文摘要 · Abstract (English)
Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。