提出新算法，用二阶优化加速强化学习策略收敛。
A Variance-Reduced Cubic-Regularized Newton for Policy Optimization
- 结合海森矩阵与方差缩减，提升优化稳定性。
- 达到 ε-二阶驻点的样本复杂度为 Õ(ε⁻³)，优于此前最优结果。
- 无需重要性采样，适合长时序任务，对初学者友好。
本文研究强化学习中策略优化的二阶方法。现有二阶方法常面临样本复杂度不优或依赖不现实的重要性采样假设的问题。为此，我们提出VR-CR-PN——一种方差缩减的立方正则化策略牛顿算法。据我们所知，这是首个将海森矩阵辅助的方差缩减与二阶策略优化相结合的算法，有效缓解分布偏移问题，在一般非凸条件下实现最优已知样本复杂度，且无需重要性采样。理论证明，VR-CR-PN在达到ε-二阶驻点时的样本复杂度为˜O(ε⁻³)，显著优于此前在类似假设下的最优结果˜O(ε⁻³.⁵)。此外，我们提出一种新的期望回报函数海森估计器，其上界不依赖于时序长度H，使算法具备与时序无关的样本复杂度。
原文摘要 · Abstract (English)
In this paper, we study a second-order approach to policy optimization in reinforcement learning. Existing second-order methods often suffer from suboptimal sample complexity or rely on unrealistic assumptions about importance sampling. To overcome these limitations, we propose VR-CR-PN, a variance-reduced cubic-regularized policy Newton algorithm. To the best of our knowledge, this is the first algorithm that integrates Hessian-aided variance reduction with second-order policy optimization, effectively addressing the distribution shift problem and achieving best-known sample complexity under general nonconvex conditions but without the need for importance sampling. We theoretically establish that VR-CR-PN achieves a sample complexity of $\tilde{\mathcal{O}}(ε^{-3})$ to reach an $ε$-second-order stationary point, significantly improving upon the previous best result of $\tilde{\mathcal{O}}(ε^{-3.5})$ under comparable assumptions. As an additional contribution, we introduce a novel Hessian estimator for the expected return function, which admits a uniform upper bound independent of the horizon length $H$, allowing the algorithm to achieve horizon-independent sample complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。