简单近似值迭代在自对弈中表现惊人，效率远超蒙特卡洛树搜索
The Surprising Effectiveness of Approximate Value Iteration in Self-Play
- 用近似值迭代进行自对弈，仅需一步前瞻搜索
- 比AlphaZero更准的值函数，训练推理成本更低
- 适合追求高效训练的强化学习研究者
结合搜索与函数逼近已推动游戏程序的重大进展，使自对弈算法比以往更具竞争力。然而，基于蒙特卡洛树搜索（MCTS）的方法计算开销较大。本文研究了在非平凡、中等规模游戏如连四棋（Connect Four）、七乘七六边形棋（Hex(7x7)）和合成游戏中的简化方法是否仍具竞争力。我们训练了一个极简的近似值迭代（AVI）自对弈实现，并使用真实值预言机进行精确评估。出乎意料的是，结果表明AVI极为有效：其学习到的值函数比AlphaZero更准确，而其单步前瞻贪心策略在显著更低的训练与推理成本下仍能与MCTS策略竞争。在奥赛罗（Othello）和九乘九围棋（Go(9x9)）上的初步实验显示，AVI在更大规模游戏中训练稳定，能学习到有效的值函数。这些发现表明，MCTS方法的成功可能掩盖了随着现代深度学习工具发展而日益实用的更简单方法。
原文摘要 · Abstract (English)
Combining search with function approximation has driven major advances in game-playing programs, making self-play algorithms more competitive than ever. Still, the computational overhead of the most popular methods, based on Monte Carlo Tree Search (MCTS), can be substantial. In this work, we investigate whether simpler methods remain competitive in non-trivial, moderately sized games such as Connect Four, Hex(7x7) and synthetic games. We train a minimal self-play implementation of Approximate Value Iteration (AVI) and use ground-truth oracles for exact evaluation. Contrary to expectations, our results demonstrate the surprising effectiveness of AVI: it learns more accurate value functions than those learned by AlphaZero, while its one-step-lookahead greedy policies remain competitive with MCTS-based policies at substantially lower training and inference costs. Preliminary experiments on Othello and Go(9x9) show that AVI trains stably on larger games and learns effective value functions. These findings suggest that the success of MCTS-based methods may have eclipsed simpler approaches that have become increasingly practical with modern deep-learning tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。