用前瞻搜索提升强化学习策略梯度，避免陷入局部最优。
Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
- 引入多步前瞻树搜索机制，让策略优化更具远见。
- 实验显示在梯子、绳索等复杂环境中可跳出局部陷阱。
- 适合处理高维或奖励稀疏的强化学习任务。
强化学习中的经典策略梯度（PG）方法常收敛到次优局部最优解，尤其在大规模或复杂环境更严重。本文研究策略梯度与树搜索结合的方法（PGTS），通过引入m步前瞻机制提升策略优化效果。理论分析表明，增加树搜索深度m可单调减少不良驻点集合，从而改善任意驻点策略的最差性能。关键的是，该分析适用于实际场景：策略更新仅限当前策略访问的状态，无需全状态空间更新。在多种马尔可夫决策过程结构（如Ladder、Tightrope、Gridworld）上的实证评估显示，PGTS具备“远见”，能有效穿越复杂奖励景观，逃离标准PG失效的局部陷阱，并获得更优解。
原文摘要 · Abstract (English)
Classical policy gradient (PG) methods in reinforcement learning frequently converge to suboptimal local optima, a challenge exacerbated in large or complex environments. This work investigates Policy Gradient with Tree Search (PGTS), an approach that integrates an $m$-step lookahead mechanism to enhance policy optimization. We provide theoretical analysis demonstrating that increasing the tree search depth $m$-monotonically reduces the set of undesirable stationary points and, consequently, improves the worst-case performance of any resulting stationary policy. Critically, our analysis accommodates practical scenarios where policy updates are restricted to states visited by the current policy, rather than requiring updates across the entire state space. Empirical evaluations on diverse MDP structures, including Ladder, Tightrope, and Gridworld environments, illustrate PGTS's ability to exhibit "farsightedness," navigate challenging reward landscapes, escape local traps where standard PG fails, and achieve superior solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。