提出无需精确梯度的演员-评论家算法，提升连续控制稳定性。
Compatible Gradient Approximations for Actor-Critic Algorithms
- 用动作空间内的两点随机梯度估计替代精确梯度计算
- 在多个环境上性能超越现有最先进方法
- 特别适合函数逼近下难以计算梯度的连续控制任务
确定性策略梯度算法是控制连续系统中演员-评论家方法的基础，但其常因依赖评论家价值估计对动作输入的导数而出现不准确问题。这种依赖要求精确的动作值梯度计算，在函数逼近下尤为困难。本文提出一种新演员-评论家算法，通过在动作空间内采用两点随机梯度估计实现动作值梯度的零阶近似，避免了对精度的依赖。该方法在理论上可解决确定性策略梯度方案中的兼容性问题，并在实验中表明，其性能不仅匹配甚至显著超过当前最先进的方法。
原文摘要 · Abstract (English)
Deterministic policy gradient algorithms are foundational for actor-critic methods in controlling continuous systems, yet they often encounter inaccuracies due to their dependence on the derivative of the critic's value estimates with respect to input actions. This reliance requires precise action-value gradient computations, a task that proves challenging under function approximation. We introduce an actor-critic algorithm that bypasses the need for such precision by employing a zeroth-order approximation of the action-value gradient through two-point stochastic gradient estimation within the action space. This approach provably and effectively addresses compatibility issues inherent in deterministic policy gradient schemes. Empirical results further demonstrate that our algorithm not only matches but frequently exceeds the performance of current state-of-the-art methods by a substantial extent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。