用预训练策略提升强化学习采样效率与稳定性
Improving DAPO from a Mixed-Policy Perspective
- 引入预训练策略提供离线经验，正则化目标策略训练
- 重用零奖励样本，使采样效率进一步提升
- 理论证明目标函数可收敛，适合高稀疏奖励场景
本文从混合策略视角对动态采样策略优化（DAPO）算法提出两项新改进。标准策略梯度方法在稀疏奖励环境下常面临训练不稳定与样本效率低的问题。为此，我们首先提出引入一个预训练的稳定引导策略（$oldsymbol{ heta}_{ ext{phi}}$），提供离线经验以正则化目标策略（$oldsymbol{ heta}_{ ext{pi}}$）的训练，通过自适应调整学习步长提升训练稳定性和收敛速度。其次，将该思想扩展至重用零奖励样本——这些样本通常被动态采样策略舍弃。通过将它们作为由专家策略引导的独立批次处理，进一步提升样本效率。本文为两种方法提供了理论分析，证明其目标函数在强化学习既有理论框架下可收敛至最优解。所提出的混合策略框架有效平衡了探索与利用，有望实现更稳定高效的策略优化。
原文摘要 · Abstract (English)
This paper introduces two novel modifications to the Dynamic sAmpling Policy Optimization (DAPO) algorithm [1], approached from a mixed-policy perspective. Standard policy gradient methods can suffer from instability and sample inefficiency, particularly in sparse reward settings. To address this, we first propose a method that incorporates a pre-trained, stable guiding policy ($\piphi$) to provide off-policy experience, thereby regularizing the training of the target policy ($\pion$). This approach improves training stability and convergence speed by adaptively adjusting the learning step size. Secondly, we extend this idea to re-utilize zero-reward samples, which are often discarded by dynamic sampling strategies like DAPO's. By treating these samples as a distinct batch guided by the expert policy, we further enhance sample efficiency. We provide a theoretical analysis for both methods, demonstrating that their objective functions converge to the optimal solution within the established theoretical framework of reinforcement learning. The proposed mixed-policy framework effectively balances exploration and exploitation, promising more stable and efficient policy optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。