arXiv:2502.17625cs.LGcs.GT2025-02被引 9

提出带bandit反馈的零和博弈学习新算法,实现依赖具体游戏难度的更优后悔上界。

Instance-Dependent Regret Bounds for Learning Two-Player Zero-Sum Games with Bandit Feedback

  • 采用Tsallis-INF算法,利用仅局部反馈实现自适应加速收敛
  • 后悔上界为 $O(c_1 \log T + \sqrt{c_2 T})$,$c_2$ 可远小于动作数
  • 在纯策略纳什均衡存在时,达到最优实例相关后悔界,适合小支持博弈

无后悔自对弈学习已成为求解大规模博弈的主流方法。近年来大量研究致力于通过提升玩家在 $T$ 轮后的后悔率来加速收敛,但几乎都假设可获得精确梯度反馈。本文探讨在仅具带宽反馈条件下能否实现加速,并针对双人零和正规形式博弈给出了肯定答案。具体地,若双方均采用Zimmert与Seldin(2018, arXiv:1807.07623)提出的Tsallis-INF算法,则其后悔率至多为 $O(c_1 \log T + \sqrt{c_2 T})$,其中 $c_1$ 与 $c_2$ 为依赖于博弈特性的常数——$c_1$ 类似于随机多臂老虎机实例的复杂度,反比于某些间隙度量;$c_2$ 在纳什均衡支持集较小时或靠近边界时可远小于动作总数。尤其当存在纯策略纳什均衡时,$c_2=0$,此时达到最优实例相关后悔界。此外,我们还证明该算法具备末次迭代收敛性,并能以近最优样本复杂度识别出纯策略纳什均衡。

原文摘要 · Abstract (English)

No-regret self-play learning dynamics have become one of the premier ways to solve large-scale games in practice. Accelerating their convergence via improving the regret of the players over the naive $O(\sqrt{T})$ bound after $T$ rounds has been extensively studied in recent years, but almost all studies assume access to exact gradient feedback. We address the question of whether acceleration is possible under bandit feedback only and provide an affirmative answer for two-player zero-sum normal-form games. Specifically, we show that if both players apply the Tsallis-INF algorithm of Zimmert and Seldin (2018, arXiv:1807.07623), then their regret is at most $O(c_1 \log T + \sqrt{c_2 T})$, where $c_1$ and $c_2$ are game-dependent constants that characterize the difficulty of learning -- $c_1$ resembles the complexity of learning a stochastic multi-armed bandit instance and depends inversely on some gap measures, while $c_2$ can be much smaller than the number of actions when the Nash equilibria have a small support or are close to the boundary. In particular, for the case when a pure strategy Nash equilibrium exists, $c_2$ becomes zero, leading to an optimal instance-dependent regret bound as we show. We additionally prove that in this case, our algorithm also enjoys last-iterate convergence and can identify the pure strategy Nash equilibrium with near-optimal sample complexity.

博弈学习带宽反馈后悔界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。