改进MCTS抽象算法中的动作选择策略，提升搜索效率
Investigating Intra-Abstraction Policies For Non-exact Abstraction Algorithms
- 提出多种内部抽象策略，解决相同抽象节点内动作的权重分配问题
- 实验显示新策略在多数环境和参数下优于随机选择
- 适合研究强化学习搜索算法效率的学者与工程师
蒙特卡洛树搜索（MCTS）的一个弱点是采样效率低，可通过并行构建和使用状态或动作抽象来改善，使同层节点间信息共享。现有抽象方法主要通过聚合抽象节点的访问次数和回报来增强上置信界（UCB）值，但未考虑多个动作共享同一父节点且处于同一抽象节点的情况——这会导致这些动作具有相同的UCB值，需依赖特定的破局规则。当前最先进的抽象算法如剪枝版实时抽象（pruned OGA）未注意到此问题，隐式采用随机破局规则。本文提出并实证评估了多种替代的内部抽象策略，在多数环境和参数设置中表现优于随机策略。
原文摘要 · Abstract (English)
One weakness of Monte Carlo Tree Search (MCTS) is its sample efficiency which can be addressed by building and using state and/or action abstractions in parallel to the tree search such that information can be shared among nodes of the same layer. The primary usage of abstractions for MCTS is to enhance the Upper Confidence Bound (UCB) value during the tree policy by aggregating visits and returns of an abstract node. However, this direct usage of abstractions does not take the case into account where multiple actions with the same parent might be in the same abstract node, as these would then all have the same UCB value, thus requiring a tiebreak rule. In state-of-the-art abstraction algorithms such as pruned On the Go Abstractions (pruned OGA), this case has not been noticed, and a random tiebreak rule was implicitly chosen. In this paper, we propose and empirically evaluate several alternative intra-abstraction policies, several of which outperform the random policy across a majority of environments and parameter settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。