改进了策略优化算法，实现更快收敛且无需复杂归一化。
Fast Convergence of Softmax Policy Mirror Ascent
- 在对数空间中进行镜面上升，避免动作间归一化计算。
- 常数步长下线性收敛速度超越传统方法，理论保证更优。
- 适用于大规模状态空间，实测表现优于PPO/TRPO等主流算法。
自然策略梯度（NPG）是常见的策略优化算法，可视为概率空间中的镜面上升。最近，Vaswani等人[2021]提出一种对应于对数几率对偶空间镜面上升的策略梯度方法。本文对其进行了改进，消除了跨动作归一化的需要，并分析所得方法（称为SPMA）。对于表格型MDP，证明了使用常数步长的SPMA能达到与NPG相同的线性收敛速度，并快于常数步长（加速）的softmax策略梯度。为处理大状态-动作空间，将SPMA扩展至使用对数线性策略参数化。与NPG不同，将SPMA推广到线性函数逼近（FA）设置时无需兼容函数逼近。不同于MDPO（NPG的实际推广），SPMA在直线性FA下只需求解凸softmax分类问题。我们证明了SPMA能线性收敛到最优值函数的邻域。进一步将其扩展至非线性函数逼近，并在MuJoCo和Atari基准上评估其性能。结果表明，SPMA在所有任务中均达到相似或更优的表现，优于MDPO、PPO和TRPO。
原文摘要 · Abstract (English)
Natural policy gradient (NPG) is a common policy optimization algorithm and can be viewed as mirror ascent in the space of probabilities. Recently, Vaswani et al. [2021] introduced a policy gradient method that corresponds to mirror ascent in the dual space of logits. We refine this algorithm, removing its need for a normalization across actions and analyze the resulting method (referred to as SPMA). For tabular MDPs, we prove that SPMA with a constant step-size matches the linear convergence of NPG and achieves a faster convergence than constant step-size (accelerated) softmax policy gradient. To handle large state-action spaces, we extend SPMA to use a log-linear policy parameterization. Unlike that for NPG, generalizing SPMA to the linear function approximation (FA) setting does not require compatible function approximation. Unlike MDPO, a practical generalization of NPG, SPMA with linear FA only requires solving convex softmax classification problems. We prove that SPMA achieves linear convergence to the neighbourhood of the optimal value function. We extend SPMA to handle non-linear FA and evaluate its empirical performance on the MuJoCo and Atari benchmarks. Our results demonstrate that SPMA consistently achieves similar or better performance compared to MDPO, PPO and TRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。