发现零和博弈中两种算法可实现快速收敛,打破传统认知。
Fast and Furious Symmetric Learning in Zero-Sum Games: Gradient Descent as Fictitious Play
- 在对称零和博弈中,两类算法通过特定初始化获得√T级后悔界。
- 梯度下降在大常数步长下首次实现大于2×2博弈的亚线性后悔。
- 适用于博弈论、强化学习中的快速收敛场景,尤其关注稳定策略求解。
本文研究了两类无后悔算法在零和博弈中的次线性后悔表现:虚构博弈(Fictitious Play)与固定步长的在线梯度下降。在一般对抗性在线学习环境中,两者可能因缺乏正则化(虚构博弈)或正则化不足(梯度下降)而出现不稳定和线性后悔。然而,它们在双人零和博弈中能否获得更优的后悔界尚不明确。本文针对一类广义对称零和博弈(将经典剪刀石头布推广至加权的n维情形),在双方策略对称初始化条件下,证明任意破平规则下虚构博弈具有O(√T)后悔界,验证了Karlin的虚构博弈猜想的新类别。同时,通过揭示虚构博弈与梯度下降在收益向量对偶空间中迭代轨迹的几何关联,证明梯度下降在几乎所有对称初始化下,当步长为足够大的常数时,也能达到O(√T)后悔界。这是首次在大于2×2的零和博弈中证明梯度下降具有‘快速而猛烈’行为(即无需随时间衰减的步长即可实现亚线性后悔)。
原文摘要 · Abstract (English)
This paper investigates the sublinear regret guarantees of two non-no-regret algorithms in zero-sum games: Fictitious Play, and Online Gradient Descent with constant stepsizes. In general adversarial online learning settings, both algorithms may exhibit instability and linear regret due to no regularization (Fictitious Play) or small amounts of regularization (Gradient Descent). However, their ability to obtain tighter regret bounds in two-player zero-sum games is less understood. In this work, we obtain strong new regret guarantees for both algorithms on a class of symmetric zero-sum games that generalize the classic three-strategy Rock-Paper-Scissors to a weighted, n-dimensional regime. Under symmetric initializations of the players' strategies, we prove that Fictitious Play with any tiebreaking rule has $O(\sqrt{T})$ regret, establishing a new class of games for which Karlin's Fictitious Play conjecture holds. Moreover, by leveraging a connection between the geometry of the iterates of Fictitious Play and Gradient Descent in the dual space of payoff vectors, we prove that Gradient Descent, for almost all symmetric initializations, obtains a similar $O(\sqrt{T})$ regret bound when its stepsize is a sufficiently large constant. For Gradient Descent, this establishes the first "fast and furious" behavior (i.e., sublinear regret without time-vanishing stepsizes) for zero-sum games larger than 2x2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。