让AI像人一样重玩失误局,显著提升训练效率。
Regret-Guided Search Control for Efficient Learning in AlphaZero
- 用后悔值网络识别高后悔状态,优先重玩失败节点。
- 在9×9围棋等任务上比AlphaZero快77 Elo,比基线快89 Elo。
- 适合想提升强化学习训练效率的研究者和工程师。
强化学习智能体虽性能出色,但学习效率远低于人类。人类常通过反复回顾出错局面快速进步,这一思路称为搜索控制,即从有价值的状态重启而非每次都从初始状态开始。现有方法如Go-Exploit从自对弈或搜索树中采样历史状态,但不区分其学习价值。本文提出后悔引导的搜索控制(RGSC),在AlphaZero基础上引入后悔网络,自动识别评估与实际结果偏差最大的高后悔状态。这些状态来自自对弈轨迹和MCTS节点,存入优先级后悔缓冲区,并作为新起点重新训练。在9×9围棋、10×10奥赛罗和11×11哈克棋上,RGSC平均优于AlphaZero 77 Elo,优于Go-Exploit 89 Elo。当基于已训练的9×9围棋模型对抗KataGo时,胜率从69.3%提升至78.2%,而基线无改善。结果表明,RGSC有效提升了训练效率与鲁棒性。代码已公开于https://rlg.iis.sinica.edu.tw/papers/rgsc。
原文摘要 · Abstract (English)
Reinforcement learning (RL) agents achieve remarkable performance but remain far less learning-efficient than humans. While RL agents require extensive self-play games to extract useful signals, humans often need only a few games, improving rapidly by repeatedly revisiting states where mistakes occurred. This idea, known as search control, aims to restart from valuable states rather than always from the initial state. In AlphaZero, prior work Go-Exploit applies this idea by sampling past states from self-play or search trees, but it treats all states equally, regardless of their learning potential. We propose Regret-Guided Search Control (RGSC), which extends AlphaZero with a regret network that learns to identify high-regret states, where the agent's evaluation diverges most from the actual outcome. These states are collected from both self-play trajectories and MCTS nodes, stored in a prioritized regret buffer, and reused as new starting positions. Across 9x9 Go, 10x10 Othello, and 11x11 Hex, RGSC outperforms AlphaZero and Go-Exploit by an average of 77 and 89 Elo, respectively. When training on a well-trained 9x9 Go model, RGSC further improves the win rate against KataGo from 69.3% to 78.2%, while both baselines show no improvement. These results demonstrate that RGSC provides an effective mechanism for search control, improving both efficiency and robustness of AlphaZero training. Our code is available at https://rlg.iis.sinica.edu.tw/papers/rgsc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。