不依赖搜索的自对弈强化学习，仅用7张显卡5个月就达到阿尔法狗水平。
Mastering the Game of Go with Self-play Experience Replay
- 采用无模型强化学习，通过自对弈和离线经验回放学习最优策略。
- 训练5个月达阿尔法狗水平，仅需7张GPU，无需人类数据。
- 首次证明无模型方法可高效解决复杂棋类博弈，适合研究深度强化学习者。
围棋长期被视为人工智能的基准测试，要求复杂的策略推理与长期规划。以往方法如阿尔法狗及其后续版本主要依赖基于模型的蒙特卡洛树搜索（MCTS）。本文提出QZero，一种新型无模型强化学习算法，在训练中放弃搜索，通过自对弈和离线经验回放学习纳什均衡策略。基于熵正则化Q-learning，QZero使用单一Q值网络统一策略评估与改进。从零开始训练，不依赖人类数据，仅用7张GPU训练5个月，性能接近阿尔法狗。这首次证明了无模型强化学习在掌握围棋上的效率，以及离线强化学习在大规模复杂环境中的可行性。
原文摘要 · Abstract (English)
The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。