提出贝叶斯后验采样算法,在未知零和博弈中实现可证明的次线性后悔。
Bayesian Learning in Episodic Zero-Sum Games
- 每回合从后验分布采样游戏模型,求解该模型下的均衡策略。
- 理论证明后悔上界为O(HS√(ABHK log(SABHK))),随轮次数增长缓慢。
- 适用于对抗性环境中需自适应学习的智能体,如博弈或强化学习场景。
我们研究在未知转移与奖励模型的周期性有限时域零和马尔可夫博弈中进行贝叶斯学习。每个玩家维护对博弈模型的贝叶斯后验分布,每回合初独立采样一个模型,并针对采样模型计算均衡策略。分析两种情形:(i) 双方均使用后验采样;(ii) 仅一方使用,另一方采用任意学习算法。在每种情形下,给出后验采样代理的期望后悔保证。我们的后悔定义为学习者预期总收益与真实博弈均衡策略下预期总收益之差。主要理论结果是后验采样代理的期望后悔上界为O(HS√(ABHK log(SABHK))),其中K为回合数,H为回合长度,S为状态数,A、B为两玩家动作空间大小。在网格世界捕食-猎物领域的实验验证了次线性后悔的缩放特性,并表明后验采样优于虚构玩法基线。
原文摘要 · Abstract (English)
We study Bayesian learning in episodic, finite-horizon zero-sum Markov games with unknown transition and reward models. We investigate a posterior algorithm in which each player maintains a Bayesian posterior over the game model, independently samples a game model at the beginning of each episode, and computes an equilibrium policy for the sampled model. We analyze two settings: (i) Both players use the posterior sampling algorithm, and (ii) Only one player uses posterior sampling while the opponent follows an arbitrary learning algorithm. In each setting, we provide guarantees on the expected regret of the posterior sampling agent. Our notion of regret compares the expected total reward of the learning agent against the expected total reward under equilibrium policies of the true game. Our main theoretical result is an expected regret bound for the posterior sampling agent of order $O(HS\sqrt{ABHK\log(SABHK)})$ where $K$ is the number of episodes, $H$ is the episode length, $S$ is the number of states, and $A,B$ are the action space sizes of the two players. Experiments in a grid-world predator--prey domain illustrate the sublinear regret scaling and show that posterior sampling competes favorably with a fictitious-play baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。