用学习的策略网络加速连续动作强化学习，提升效率且保持收敛性。
Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces
- 用策略网络近似优化子问题解，加快决策速度
- 理论证明近似误差对收敛性影响可控
- 在机器人控制等任务中优于PPO等主流方法
策略双重平均（PDA）提供了一个更自然支持值函数近似的策略镜面下降框架，可在保留强收敛保证的同时使用近似优势（或Q-）函数。然而，在连续状态和动作空间中应用PDA仍存在计算挑战，因每步决策需求解一个优化子问题。本文提出「演员加速PDA」，利用学习的策略网络近似该优化子问题的解，实现更快运行时间并维持收敛性。我们提供了理论分析，量化了在合理假设下演员近似误差对PDA收敛的影响。随后在机器人、控制及运筹学等多个基准任务上评估其性能。结果表明，演员加速PDA在连续动作问题中表现优于流行的一类策略基线方法，如近端策略优化（PPO）。整体而言，本工作弥合了PDA理论优势与实际在函数逼近下的部署之间的差距。
原文摘要 · Abstract (English)
Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose \textit{actor-accelerated PDA}, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。