改进粒子滤波的强化学习规划方法，提升采样效率与实时性。
Trust-Region Twisted Policy Improvement
- 基于约束动作采样和终止状态处理优化粒子生成
- 在离散与连续任务中均实现更快运行速度与更少样本需求
- 适合需要高效在线规划的强化学习场景
蒙特卡洛树搜索（MCTS）推动了深度强化学习的诸多突破，但其在并行计算环境下的扩展面临挑战，促使研究者探索如顺序蒙特卡洛（SMC）等替代方案。许多SMC方法采用粒子滤波，将强化学习重构成策略推断问题。然而，这些方法中的设计选择常与强化学习在线规划的目标冲突——即在规划开始时就获得策略改进。受MCTS启发，本文通过在规划器内改进数据生成（包括约束动作采样与显式终止状态处理），以及优化策略与价值目标估计，提出可信域扭曲的SMC（TRT-SMC）。该方法在离散与连续环境中均表现出比基线MCTS和SMC更优的运行效率与样本效率。
原文摘要 · Abstract (English)
Monte-Carlo tree search (MCTS) has driven many recent breakthroughs in deep reinforcement learning (RL). However, scaling MCTS to parallel compute has proven challenging in practice which has motivated alternative planners like sequential Monte-Carlo (SMC). Many of these SMC methods adopt particle filters for smoothing through a reformulation of RL as a policy inference problem. Yet, persisting design choices of these particle filters often conflict with the aim of online planning in RL, which is to obtain a policy improvement at the start of planning. Drawing inspiration from MCTS, we tailor SMC planners specifically for RL by improving data generation within the planner through constrained action sampling and explicit terminal state handling, as well as improving policy and value target estimation. This leads to our Trust-Region Twisted SMC (TRT-SMC), which shows improved runtime and sample-efficiency over baseline MCTS and SMC methods in both discrete and continuous domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。