用秩1近似加速自然策略梯度，提升深度强化学习效率
Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning
- 用秩1近似替代全量逆Fisher矩阵，降低计算开销
- 在多种环境中表现优于标准演员-评论家和信任区域基线
- 理论证明其收敛速度更快，适合大规模策略优化任务
自然梯度在深度强化学习中因快速收敛和协变权重更新而备受关注。然而，每次迭代都需要计算Fisher信息矩阵（FIM）的逆，计算成本极高。本文提出一种高效可扩展的自然策略优化方法，采用秩1近似代替完整的逆FIM。理论上表明，在特定条件下，该近似方法收敛速度优于普通策略梯度，且在某些情况下样本复杂度与随机策略梯度方法相当。我们在多种环境上进行了基准测试，结果表明该方法性能显著优于标准演员-评论家和信任区域基线。
原文摘要 · Abstract (English)
Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。