提出外层优化的PPO框架，用任意优化器替代固定学习率更新。
Beyond the Boundaries of Proximal Policy Optimization
- 将PPO拆解为内层估计更新向量、外层用任意优化器应用更新
- 非单位学习率和动量在Brax、Jumanji上显著提升性能
- 揭示PPO隐含设计选择，适合强化学习研究者参考
近端策略优化（PPO）是广泛使用的在线策略强化学习算法。本文从新视角解析PPO，将其分解为内层更新向量估计与外层以单位学习率进行梯度上升的应用。基于此，我们提出外层近端策略优化（outer-PPO），允许使用任意基于梯度的优化器来执行更新。该框架解耦了更新估计与应用过程，揭示了PPO中若干隐含设计选择，并通过实证研究加以挑战。具体考察了外层采用非单位学习率与动量，以及内层引入动量偏差的情况。在Brax、Jumanji和MinAtar环境中，与经过激进调优的基准PPO相比，非单位学习率和动量均在相同调参预算下于Brax和Jumanji上实现统计显著提升。
原文摘要 · Abstract (English)
Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。