为PPO的收敛性提供理论证明，揭示其近似梯度上升的本质。
An Approximate Ascent Approach To Prove Convergence of PPO
- 将PPO的多轮小批量更新视为近似策略梯度上升。
- 证明了在随机重排框架下的收敛性，控制了代理梯度偏差。
- 发现并修正了截断GAE中的权重缺陷，提升终止信号强环境表现。
近端策略优化（PPO）是应用最广泛的深度强化学习算法之一，但其理论基础仍不完整。特别是其收敛性及核心优势的理解尚不明确。在标准假设下，我们展示了PPO的策略更新机制（对多轮使用回放数据进行多轮小批量更新，采用代理梯度）可被解释为近似策略梯度上升。通过控制代理梯度累积偏差，并运用随机重排技术，我们建立了PPO的收敛定理，揭示了其成功的原因。此外，我们识别出一个此前被忽视的问题：截断广义优势估计（GAE）中使用的几何加权方案会在回合边界导致最长k步优势估计器出现无限质量坍塌。实验表明，简单的权重修正可在具有强终止信号的环境（如Lunar Lander）中带来显著性能提升。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling to prove a convergence theorem for PPO that sheds light on PPO's success. Additionally, we identify a previously overlooked issue in truncated Generalized Advantage Estimation commonly used in PPO. The geometric weighting scheme induces infinite mass collapse onto the longest $k$-step advantage estimator at episode boundaries. Empirical evaluations show that a simple weight correction can yield substantial improvements in environments with strong terminal signal, such as Lunar Lander.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。