让AI在信息不全的博弈中更聪明,通过预测胜负和排名提升决策能力。
SkyNet: Belief-Aware Planning for Partially-Observable Stochastic Games
- 在MuZero基础上加预测头,让模型主动学习隐藏状态的不确定性。
- 在纸牌游戏Skyjo中赢率75.3%,比基线高出194埃洛分(p<10^-50)。
- 适合研究部分可观测多智能体博弈与强化学习表示学习的人参考。
2019年,DeepMind发布MuZero,一种基于模型的强化学习方法,在完美信息博弈中表现优异,结合了学习到的动力学模型与蒙特卡洛树搜索(MCTS)。然而,较少工作将MuZero扩展至部分可观测、随机性、多智能体环境,此类场景下智能体需在未知状态的不确定性中行动,如扑克牌游戏、自主谈判、金融交易和多机器人协作。传统MuZero的隐状态编码缺乏对未观测变量不确定性的专门建模机制。为此,本文提出SkyNet(信念感知的MuZero),在标准MuZero架构中引入以自我条件化的辅助头,用于胜者预测与排名估计。这些目标促使隐状态保留对部分可观测环境下结果具有预测性的信息,无需显式信念状态追踪或改变搜索算法。我们在部分可观测、非零和、随机性的纸牌游戏Skyjo上评估SkyNet,采用决策粒度环境、基于Transformer的编码器以及由启发式对手组成的自对弈课程。在1000场对局的匹配检查点评估中,SkyNet达到75.3%的峰值胜率,相比基线提升194埃洛分(p < 10^-50);在对抗启发式对手时,胜率也从0.466提升至0.720。关键发现是:信念感知模型初期表现低于基线,但当训练数据量足够时,显著超越基线,表明信念感知辅助监督能有效改善部分可观测环境下的表示学习,但依赖充足的数据流。
原文摘要 · Abstract (English)
In 2019, Google DeepMind released MuZero, a model-based reinforcement learning method that achieves strong results in perfect-information games by combining learned dynamics models with Monte Carlo Tree Search (MCTS). However, comparatively little work has extended MuZero to partially observable, stochastic, multi-player environments, where agents must act under uncertainty about hidden state. Such settings arise not only in card games but in domains such as autonomous negotiation, financial trading, and multi-agent robotics. In the absence of explicit belief modeling, MuZero's latent encoding has no dedicated mechanism for representing uncertainty over unobserved variables. To address this, we introduce SkyNet (Belief-Aware MuZero), which adds ego-conditioned auxiliary heads for winner prediction and rank estimation to the standard MuZero architecture. These objectives encourage the latent state to retain information predictive of outcomes under partial observability, without requiring explicit belief-state tracking or changes to the search algorithm. We evaluate SkyNet on Skyjo, a partially observable, non-zero-sum, stochastic card game, using a decision-granularity environment, transformer-based encoding, and a curriculum of heuristic opponents with self-play. In 1000-game head-to-head evaluations at matched checkpoints, SkyNet achieves a 75.3% peak win rate against the baseline (+194 Elo, $p < 10^{-50}$). SkyNet also outperforms the baseline against heuristic opponents (0.720 vs.\ 0.466 win rate). Critically, the belief-aware model initially underperforms the baseline but decisively surpasses it once training throughput is sufficient, suggesting that belief-aware auxiliary supervision improves learned representations under partial observability, but only given adequate data flow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。