arXiv:2506.03464cs.GTcs.LG2025-06NeurIPS被引 5

提出一种简单方法,让平均收敛算法自动实现最后迭代收敛,提升多玩家博弈学习效率。

From Average-Iterate to Last-Iterate Convergence in Games: A Reduction and Its Applications

  • 用黑箱转换将平均迭代结果转化为最后迭代行为,简化算法设计
  • 在梯度反馈下实现 $O(\frac{\log d}{T})$ 收敛率,维度依赖大幅降低
  • 适用于多玩家零和博弈,尤其适合追求实际决策稳定性的研究者

在线学习算法在自对弈游戏中的收敛性是博弈论与机器学习的核心问题。尽管许多算法仅在平均迭代下收敛,但最后迭代收敛更能反映学习者的实际决策行为。本文发现,对于一类包含双人双矩阵博弈和多玩家多项式博弈的广泛游戏,存在一个简单的黑箱转换:可将任意无耦合学习动态的平均迭代结果转化为新无耦合动态的最后迭代,从而实现从平均收敛到最后迭代收敛的归约。将该方法应用于乐观乘法权重更新算法,在多玩家零和多项式博弈中获得全新最优的最后迭代收敛率:在梯度反馈下为 $O(\frac{\log d}{T})$,维度 $d$ 依赖呈指数级改善;在弱反馈下为 $\widetilde{O}(d^{\frac{1}{5}} T^{-\frac{1}{5}})$,优于此前 $\widetilde{O}(\sqrt{d} T^{-\frac{1}{8}})$ 和 $\widetilde{O}(\sqrt{d} T^{-\frac{1}{6}})$ 的最佳结果。

原文摘要 · Abstract (English)

The convergence of online learning algorithms in games under self-play is a fundamental question in game theory and machine learning. Among various notions of convergence, last-iterate convergence is particularly desirable, as it reflects the actual decisions made by the learners and captures the day-to-day behavior of the learning dynamics. While many algorithms are known to converge in the average-iterate, achieving last-iterate convergence typically requires considerably more effort in both the design and the analysis of the algorithm. Somewhat surprisingly, we show in this paper that for a large family of games, there exists a simple black-box reduction that transforms the average iterates of an uncoupled learning dynamics into the last iterates of a new uncoupled learning dynamics, thus also providing a reduction from last-iterate convergence to average-iterate convergence. Our reduction applies to games where each player's utility is linear in both their own strategy and the joint strategy of all opponents. This family includes two-player bimatrix games and generalizations such as multi-player polymatrix games. By applying our reduction to the Optimistic Multiplicative Weights Update algorithm, we obtain new state-of-the-art last-iterate convergence rates for uncoupled learning dynamics in multi-player zero-sum polymatrix games: (1) an $O(\frac{\log d}{T})$ last-iterate convergence rate under gradient feedback, representing an exponential improvement in the dependence on the dimension $d$ (i.e., the maximum number of actions available to either player); and (2) an $\widetilde{O}(d^{\frac{1}{5}} T^{-\frac{1}{5}})$ last-iterate convergence rate under bandit feedback, improving upon the previous best rates of $\widetilde{O}(\sqrt{d} T^{-\frac{1}{8}})$ and $\widetilde{O}(\sqrt{d} T^{-\frac{1}{6}})$.

博弈学习收敛分析在线学习多玩家博弈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。