用二阶信息加速确定性策略强化学习，收敛更快。
Quasi-Newton Compatible Actor-Critic for Deterministic Policies
- 构建二次评论家模型，同时保留真实梯度与性能海森矩阵近似
- 通过最小二乘时序差分法高效估计参数，实现拟牛顿更新
- 适用于任意可微策略，对标准方法有明显性能提升
本文提出一种基于二阶信息的确定性策略演员-评论家框架，将经典确定性策略梯度方法扩展为利用性能函数曲率信息。基于评论家相容函数逼近概念，引入二次评论家模型，同时保持真实策略梯度和性能海森矩阵的近似。进一步设计了最小二乘时序差分学习方案，以高效估计二次评论家参数。该构造使得演员采用由评论家学习到的信息进行拟牛顿更新，相比一阶方法具有更快收敛速度。所提方法具有通用性，适用于任意可微策略类。数值实验表明，该方法在收敛性和性能上均优于标准确定性演员-评论家基线。
原文摘要 · Abstract (English)
In this paper, we propose a second-order deterministic actor-critic framework in reinforcement learning that extends the classical deterministic policy gradient method to exploit curvature information of the performance function. Building on the concept of compatible function approximation for the critic, we introduce a quadratic critic that simultaneously preserves the true policy gradient and an approximation of the performance Hessian. A least-squares temporal difference learning scheme is then developed to estimate the quadratic critic parameters efficiently. This construction enables a quasi-Newton actor update using information learned by the critic, yielding faster convergence compared to first-order methods. The proposed approach is general and applicable to any differentiable policy class. Numerical examples demonstrate that the method achieves improved convergence and performance over standard deterministic actor-critic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。