用二阶动量加速强化学习策略梯度，提升训练效率与稳定性。
Policy Gradient with Second Order Momentum
- 在REINFORCE基础上加入一阶梯度均值和海森矩阵对角近似
- 样本效率最高提升2.1倍，方差显著降低
- 仅需额外D内存，适合大规模策略优化
我们提出一种轻量级的二阶优化方法PG-SOM，用于强化学习策略优化。该方法在经典REINFORCE更新中引入两个指数加权统计量：一阶梯度均值和海森矩阵对角近似。通过利用该曲率估计对梯度进行预条件化，算法自适应地缩放每个参数，实现预期回报的更快更稳定的上升。我们给出了简洁的推导，证明了在弱正则性假设下，对角海森估计器是无偏且正定的，并进一步证明更新方向在期望上为下降方向。在标准控制基准测试中的数值实验表明，相较于一阶方法和Fisher矩阵基线，样本效率最高提升2.1倍，方差大幅减少。结果表明，即使粗略的二阶信息也能带来显著实际收益，且仅增加D内存开销（针对D参数策略）。所有代码与可复现脚本将公开发布。
原文摘要 · Abstract (English)
We develop Policy Gradient with Second-Order Momentum (PG-SOM), a lightweight second-order optimisation scheme for reinforcement-learning policies. PG-SOM augments the classical REINFORCE update with two exponentially weighted statistics: a first-order gradient average and a diagonal approximation of the Hessian. By preconditioning the gradient with this curvature estimate, the method adaptively rescales each parameter, yielding faster and more stable ascent of the expected return. We provide a concise derivation, establish that the diagonal Hessian estimator is unbiased and positive-definite under mild regularity assumptions, and prove that the resulting update is a descent direction in expectation. Numerical experiments on standard control benchmarks show up to a 2.1x increase in sample efficiency and a substantial reduction in variance compared to first-order and Fisher-matrix baselines. These results indicate that even coarse second-order information can deliver significant practical gains while incurring only D memory overhead for a D-parameter policy. All code and reproducibility scripts will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。