提出自适应探索因子策略，让强化学习搜索更稳定可靠。
Investigating Scale Independent UCT Exploration Factor Strategies
- 用搜索树中Q值的标准差动态调整探索参数
- 新策略在多种游戏上超越现有方法，性能更优
- 适合需要稳定搜索的强化学习任务使用
Upper Confidence Bounds For Trees（UCT）算法对所应用游戏的奖励尺度不具鲁棒性。对于结束时奖励为{-1,0,1}的零和稀疏奖励游戏，这并非问题；但许多游戏采用人工设定的密集奖励，导致不同游戏间节点的Q值量级差异显著。本文评估了多种自适应选择UCT探索常数λ的策略（λ-策略），包括文献中已有方法及五种新策略。实验结果表明，推荐采用新提出的策略：将λ设为搜索树中所有状态-动作对Q值经验标准差的2倍（即λ=2·σ）。该方法在单个参数值下及优化所有参数后的峰值性能上，均优于现有λ-策略，在广泛任务中表现更优。
原文摘要 · Abstract (English)
The Upper Confidence Bounds For Trees (UCT) algorithm is not agnostic to the reward scale of the game it is applied to. For zero-sum games with the sparse rewards of $\{-1,0,1\}$ at the end of the game, this is not a problem, but many games often feature dense rewards with hand-picked reward scales, causing a node's Q-value to span different magnitudes across different games. In this paper, we evaluate various strategies for adaptively choosing the UCT exploration constant $λ$, called $λ$-strategies, that are agnostic to the game's reward scale. These $λ$-strategies include those proposed in the literature as well as five new strategies. Given our experimental results, we recommend using one of our newly suggested $λ$-strategies, which is to choose $λ$ as $2 \cdot σ$ where $σ$ is the empirical standard deviation of all state-action pairs' Q-values of the search tree. This method outperforms existing $λ$-strategies across a wide range of tasks both in terms of a single parameter value and the peak performances obtained by optimizing all available parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。