提出可计算策略的广义真实信念类,解决多智能体博弈中的学习收敛难题。
Limit-Computable Grains of Truth for Arbitrary Computable Extensive-Form (Un)Known Games
- 构建包含所有可计算策略的策略类,支持贝叶斯最优决策
- 未知环境中使用Thompson采样可收敛至ε-纳什均衡
- 适用于任意可计算博弈,尤其适合自预测型策略设计
在无限多玩家博弈中,若贝叶斯玩家的先验对其他玩家的策略分配正概率(或包含真实信念),则其能学习预测他人策略。卡尔与莱赫尔的经典真实信念问题在于:寻找一个足够大的策略类,使其包含该类下的贝叶斯最优策略,并允许相互一致的信念且符合贝叶斯推理规则。目前已知具有真实信念的策略类极为有限,文献中存在多个相关不可能性结果。本文给出该问题的完整形式解:构造一个足够广泛的策略类,既包含所有可计算策略,也包含针对该类上任意合理先验的贝叶斯最优策略。当环境为已知重复阶段博弈时,我们证明了[KL93a]和[KL93b]意义上的收敛性;当环境未知时,采用Thompson采样的代理可在任意未知可计算多智能体环境中收敛至ε-纳什均衡。最后,我们展示了该方法在自预测策略避免规划中的应用。尽管这些结果仅将可计算性理论作为概念工具,但我们表明其解可被计算上任意逼近。
原文摘要 · Abstract (English)
A Bayesian player acting in an infinite multi-player game learns to predict the other players' strategies if his prior assigns positive probability to their play (or contains a grain of truth). Kalai and Lehrer's classic grain of truth problem is to find a reasonably large class of strategies that contains the Bayes-optimal policies with respect to this class, allowing mutually-consistent beliefs about strategy choice that obey the rules of Bayesian inference. Only small classes are known to have a grain of truth and the literature contains several related impossibility results. In this paper we present a formal and general solution to the full grain of truth problem: we construct a class of strategies wide enough to contain all computable strategies as well as Bayes-optimal strategies for every reasonable prior over the class. When the "environment" is a known repeated stage game, we show convergence in the sense of [KL93a] and [KL93b]. When the environment is unknown, agents using Thompson sampling converge to play $\varepsilon$-Nash equilibria in arbitrary unknown computable multi-agent environments. Finally, we include an application to self-predictive policies that avoid planning. While these results use computability theory only as a conceptual tool to solve a classic game theory problem, we show that our solution can naturally be computationally approximated arbitrarily closely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。