改造AlphaZero让其在不对称棋类中高效自对弈,性能超越随机基线。
Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game
- 为攻守双方分别设计策略和价值头,共享特征提取主干
- 训练100轮后达1235贝叶斯埃洛分,策略熵与残余棋子数显著下降
- 适合研究不对称博弈、强化学习稳定性的研究人员
本文研究将AlphaZero强化学习算法适配至不对称历史棋类Tablut,该棋类双方棋子数量与目标不同(一方需捕王,另一方需逃王)。原版AlphaZero使用单一策略与价值头,难以应对不对称环境中的双重评估矛盾。为此,提出为双方角色设置独立的策略与价值头,同时共享残差主干网络以学习共性棋局特征。训练中出现攻防角色间灾难性遗忘等不稳定性,通过引入C4数据增强、扩大回放缓冲区及让模型用25%回合对抗随机采样的历史检查点得以缓解。经过100次自对弈迭代,模型表现稳步提升,贝叶斯埃洛评分达到1235,相比随机初始化基线有显著优势。训练指标显示策略熵与平均剩余棋子数持续下降,体现策略更聚焦、对局更决断。实验表明,只要采用独立头结构与稳健稳定化技术,AlphaZero的自对弈框架可成功迁移至高度不对称游戏。
原文摘要 · Abstract (English)
This work investigates the adaptation of the AlphaZero reinforcement learning algorithm to Tablut, an asymmetric historical board game featuring unequal piece counts and distinct player objectives (king capture versus king escape). While the original AlphaZero architecture successfully leverages a single policy and value head for symmetric games, applying it to asymmetric environments forces the network to learn two conflicting evaluation functions, which can hinder learning efficiency and performance. To address this, the core architecture is modified to use separate policy and value heads for each player role, while maintaining a shared residual trunk to learn common board features. During training, the asymmetric structure introduced training instabilities, notably catastrophic forgetting between the attacker and defender roles. These issues were mitigated by applying C4 data augmentation, increasing the replay buffer size, and having the model play 25 percent of training games against randomly sampled past checkpoints. Over 100 self-play iterations, the modified model demonstrated steady improvement, achieving a BayesElo rating of 1235 relative to a randomly initialized baseline. Training metrics also showed a significant decrease in policy entropy and average remaining pieces, reflecting increasingly focused and decisive play. Ultimately, the experiments confirm that AlphaZero's self-play framework can transfer to highly asymmetric games, provided that distinct policy/value heads and robust stabilization techniques are employed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。