提出三种算法,解决零和博弈中未知收益矩阵的在线学习问题。
Two-Player Zero-Sum Games with Bandit Feedback
- 基于探索-再利用和动作对剔除框架设计新算法。
- 在T轮后实现$O(\Delta + \sqrt{T})$和$O\left(\frac{\log(T\Delta^2)}{\Delta}\right)$的实例相关后悔上界。
- 适用于需快速收敛到纯策略纳什均衡的研究者或博弈学习场景。
我们研究了一种双人零和博弈,其中行玩家希望在未知收益矩阵下最大化自身收益,该矩阵通过旁观反馈(bandit feedback)估计。提出了三种基于探索-再利用(ETC)和动作对剔除框架的算法。第一种将ETC方法拓展至零和博弈场景;第二种引入自适应剔除机制,利用ε-纳什均衡性质高效选择最优动作对;第三种通过非均匀探索扩展剔除算法。目标是展示ETC与动作对剔除算法在零和博弈中的适用性,重点学习纯策略纳什均衡。关键贡献在于推导出所提算法的实例相关后悔上界,该方向在零和博弈文献中长期关注不足。特别地,在T轮后,零和博弈中ETC算法的后悔上界为$O(\Delta + \sqrt{T})$,自适应剔除算法及其非均匀探索变体为$O\left(\frac{\log(T\Delta^2)}{\Delta}\right)$,其中Δ表示次优差距。结果表明,所提算法在零和博弈中表现有效,其后悔上界与现有方法相当,并提供基于实例的深入分析视角。
原文摘要 · Abstract (English)
We study a two-player zero-sum game in which the row player aims to maximize their payoff against a competing column player, under an unknown payoff matrix estimated through bandit feedback. We propose three algorithms based on the Explore-Then-Commit (ETC) and action pair elimination frameworks. The first adapts it to zero-sum games, the second incorporates adaptive elimination that leverages the $\varepsilon$-Nash Equilibrium property to efficiently select the optimal action pair, and the third extends the elimination algorithm by employing non-uniform exploration. Our objective is to demonstrate the applicability of ETC and action pair elimination algorithms in a zero-sum game setting by focusing on learning pure strategy Nash Equilibria. A key contribution of our work is a derivation of instance-dependent upper bounds on the expected regret of our proposed algorithms, which has received limited attention in the literature on zero-sum games. Particularly, after $T$ rounds, we achieve an instance-dependent regret upper bounds of $O(Δ+ \sqrt{T})$ for ETC in zero-sum game setting and $O\left(\frac{\log (T Δ^2)}Δ\right)$ for the adaptive elimination algorithm and its variant with non-uniform exploration, where $Δ$ denotes the suboptimality gap. Therefore, our results indicate that the ETC and action pair elimination algorithms perform effectively in zero-sum game settings, achieving regret bounds comparable to existing methods while providing insight through instance-dependent analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。