证明了后悔匹配+可快速收敛到优化解,但标准后悔匹配可能极慢。
Convergence of Regret Matching in Potential Games and Constrained Optimization
- 用累积后悔量关联KKT间隙,建立新收敛分析框架
- RM+在O(1/ε⁴)步内达ε-KKT点,ε小时可快至O(1/ε²)
- 首次揭示标准后悔匹配与改进版的性能根本差异
后悔匹配(RM)及其现代变体是解决基准零和博弈(如扑克)的核心在线算法,但其理论收敛性在双人零和博弈外仍不清晰。例如,其在势博弈中是否收敛至纳什均衡已悬置二十年。尽管如此,近期实证表明,特别是后悔匹配+(RM⁺),在典型约束优化问题上表现优异,超越传统梯度下降算法。本文首次证明:RM⁺在O_ε(1/ε⁴)次迭代内收敛至ε-KKT点,确立其为高效的一阶优化器。分析揭示了KKT间隙与累积后悔之间的非平凡关联——当后悔有界时,复杂度可优化至O_ε(1/ε²)。技术上,尽管RM⁺通常无单步提升性,但在算法快速进入并停留的区域中具备该性质。相反,第二项主结果给出下界:无论是否交替,标准RM在双人势博弈中可能需指数步才能达到粗略近似解。这是首个关于RM与RM⁺的最坏情况分离结果,表明势博弈中收敛至粗略相关均衡远快于收敛至纳什均衡。
原文摘要 · Abstract (English)
Regret matching (RM) -- and its modern variants -- is a foundational online algorithm that has been at the heart of many AI breakthrough results in solving benchmark zero-sum games, such as poker. Yet, surprisingly little is known so far in theory about its convergence beyond two-player zero-sum games. For example, whether regret matching converges to Nash equilibria in potential games has been an open problem for two decades. Even beyond games, one could try to use RM variants for general constrained optimization problems. Recent empirical evidence suggests that they -- particularly regret matching$^+$ (RM$^+$) -- attain strong performance on benchmark constrained optimization problems, outperforming traditional gradient descent-type algorithms. We show that RM$^+$ converges to an $ε$-KKT point after $O_ε(1/ε^4)$ iterations, establishing for the first time that it is a sound and fast first-order optimizer. Our argument relates the KKT gap to the accumulated regret, two quantities that are entirely disparate in general but interact in an intriguing way in our setting, so much so that when regrets are bounded, our complexity bound improves all the way to $O_ε(1/ε^2)$. From a technical standpoint, while RM$^+$ does not have the usual one-step improvement property in general, we show that it does in a certain region that the algorithm will quickly reach and remain in thereafter. In sharp contrast, our second main result establishes a lower bound: RM, with or without alternation, can take an exponential number of iterations to reach a crude approximate solution even in two-player potential games. This represents the first worst-case separation between RM and RM$^+$. Our lower bound shows that convergence to coarse correlated equilibria in potential games is exponentially faster than convergence to Nash equilibria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。