将梯度优化融入蒙特卡洛树搜索，提升连续空间强化学习决策效率。
Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs
- 结合树搜索与梯度精调动作，实现全局探索与局部优化平衡。
- 在多个连续MDP/POMDP任务中超越现有采样方法，性能显著提升。
- 适合需要高效在线规划的机器人、自动驾驶等连续决策场景。
连续状态、动作和观测空间中的在线规划对自主系统仍具挑战性。尽管蒙特卡洛树搜索（MCTS）通过采样可有效扩展，但大多数连续（部分可观测）马尔可夫决策过程（(PO)MDP）求解器未利用基于梯度的动作优化。本文提出动作-梯度蒙特卡洛树搜索（AGMCTS），将全局树搜索与局部梯度驱动的动作精修相结合，同时保持一致的价值估计。我们给出三项关键理论贡献：（1）针对粒子信念状态的动作评分梯度定理；（2）支持频繁动作分支更新的多重重要性采样（MIS）树，通过复用历史样本避免估计偏差；（3）基于面积公式，为平滑生成模型提供可计算的动作评分梯度。实验表明，AGMCTS在多个具有挑战性的连续MDP和POMDP基准测试中优于最先进的采样基求解器。
原文摘要 · Abstract (English)
Online planning in continuous state, action, and observation spaces remains challenging for autonomous systems. While Monte Carlo Tree Search (MCTS) scales effectively via sampling, most continuous (PO)MDP solvers do not exploit gradient-based action optimization. We propose Action-Gradient MCTS (AGMCTS), a framework that combines global tree search with local gradient-based action refinement, while maintaining consistent value estimates. We provide three key theoretical contributions: (1) an action score gradient theorem for particle belief states; (2) the Multiple Importance Sampling (MIS) Tree that supports frequent action-branch updates by reusing prior samples without introducing estimator drift; and (3) tractable action score gradients for smooth generative models using the Area Formula. Empirical results demonstrate that AGMCTS outperforms state-of-the-art sample-based solvers in multiple challenging continuous MDP and POMDP benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。