解决分布式强化学习中的策略滞后问题，提升训练稳定性。
Align and Filter: Improving Performance in Asynchronous On-Policy RL
- 基于总变差的策略优势对齐约束机制，缓解策略滞后。
- 在经典任务和大模型数学推理任务中均显著提升鲁棒性。
- 适合大规模分布式强化学习场景，尤其关注稳定性和收敛性。
分布式训练和提高梯度更新频率是加速学习、提升性能的有效策略，但二者均加剧了核心挑战：策略滞后——即生成数据的行为策略与正在更新的学习策略之间的不匹配。策略滞后会限制在线策略强化学习算法向更大规模问题的扩展。本文分析了分布式学习和高更新频率导致策略滞后的根源，并据此提出一种实用方法：基于总变差的优势对齐约束策略优化（Total Variation-based Advantage-aligned Constrained Policy Optimization, TVACPO）。通过实验验证，该方法在经典强化学习任务及针对大语言模型数学推理的现代强化学习任务中均展现出对策略滞后的更强鲁棒性。
原文摘要 · Abstract (English)
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。