AlphaZero在稀疏奖励游戏中难达最优，辅助监督显著提升其正确性。
AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision
- 用辅助损失引入专家策略监督，弥补自对弈的偏差
- 在Chomp游戏上实现10x11板完美一致性，在9x10板接近完美
- 适用于追求精确策略的强化学习研究者
AlphaZero虽能实现超人级表现，但未必达到最优。本文在两个可验证的博弈环境中研究这一差距：连四棋（已解决的偏博弈）和奇普棋（由Grundy数结构决定最优解的公平博弈）。在统一的自对弈+蒙特卡洛树搜索框架下，对比基础AlphaZero、多帧输入（仅限奇普棋）及新增专家策略监督的AZAL方法。结果表明，基础AlphaZero虽表现强劲，但在连四棋中无法维持最优路径，在奇普棋中无法稳定保持g=0不变量。多帧输入不能消除该差距。而AZAL显著提升多种子完整对局与采样状态下的专家一致性：在10x11奇普棋板上达到完全一致，在9x10板上高但不完全；在连四棋中提升匹配率并延迟首次错误，但未达完美对局。
原文摘要 · Abstract (English)
AlphaZero has demonstrated that a neural-guided Monte Carlo Tree Search can achieve superhuman performance, but strong play does not necessarily imply perfect play. We study this gap in two oracle-evaluable domains with contrasting structure: Connect Four, a solved partisan game with exact game-theoretic values, and Chomp, an impartial game whose optimal play is governed by Grundy-number structure. Under a unified self-play $+$ MCTS pipeline, we compare vanilla AlphaZero, a multi-frame variant (limited to Chomp), and an AlphaZero Auxiliary Loss (AZAL) that adds oracle-derived policy supervision. We find that vanilla AlphaZero achieves strong play across both domains but cannot preserve the exact trajectories required for optimal play: in Connect Four, it fails to maintain the optimal line of play, while in Chomp, it fails to consistently restore the $g=0$ invariant. On rectangular Chomp boards, multi-frame inputs alone do not remove this gap. Nevertheless, AZAL substantially improves oracle consistency across multi-seeded full-game traces and sampled-state evaluations. On Chomp, AZAL reaches perfect full-game oracle consistency on 10x11 and high but not complete consistency on 9x10; on Connect Four, AZAL improves oracle-match rate and delays the first oracle mistake, but does not reach perfect play.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。