改进蒙特卡洛树搜索，提升深度搜索效率与稳定性。
Twice Sequential Monte Carlo for Tree Search
- 提出双重顺序蒙特卡洛方法，缓解路径退化问题。
- 在多种环境中优于传统SMC和MCTS，且随计算加深表现更优。
- 适合需要深度搜索的强化学习任务，尤其擅长并行加速。
基于模型的强化学习方法依赖搜索机制实现了诸多突破。顺序蒙特卡洛（SMC）作为蒙特卡洛树搜索（MCTS）的替代方案，更易并行化且更适合GPU加速。然而，其仍面临方差大和路径退化问题，导致搜索深度增加时难以扩展。为此，我们提出双重顺序蒙特卡洛树搜索（TSMCTS）。在离散与连续环境中，TSMCTS均优于SMC基线及一种流行的现代MCTS版本，作为策略改进算子表现出色；能有效随序列计算增加而扩展，降低估计方差，并缓解路径退化现象，同时保持SMC天然并行的优势。
原文摘要 · Abstract (English)
Model-based reinforcement learning (RL) methods that leverage search are responsible for many milestone breakthroughs in RL. Sequential Monte Carlo (SMC) recently emerged as an alternative to the Monte Carlo Tree Search (MCTS) algorithm which drove these breakthroughs. SMC is easier to parallelize and more suitable to GPU acceleration. However, it also suffers from large variance and path degeneracy which prevent it from scaling well with increased search depth, i.e., increased sequential compute. To address these problems, we introduce Twice Sequential Monte Carlo Tree Search (TSMCTS). Across discrete and continuous environments TSMCTS outperforms the SMC baseline as well as a popular modern version of MCTS as a policy improvement operator, scales favorably with sequential compute, reduces estimator variance and mitigates the effects of path degeneracy while retaining the properties that make SMC natural to parallelize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。