提出新算法IREG-PRM⁺,理论实践兼优,收敛更快更稳定。
Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Games
- 设计尺度不变的参数自适应算法,桥接理论与实际表现
- 实现最优平均迭代收敛率 $T^{-1}$,最佳迭代收敛率达 $T^{-1/2}$
- 适用于零和博弈、调和博弈等场景,无需预先知道权重
几十年来,一阶方法在零和博弈求解中的理论与实践之间存在巨大鸿沟。尽管理论上已确立 $T^{-1}$ 的收敛速度,但实践中最有效的方法仍是基于后悔匹配的计数反事实后悔最小化(CFR),尤其是其现代变体预测后悔匹配$^+$(PRM$^+$)。然而,这类算法在自对弈中仍可能仅达到 $T^{-1/2}$ 的收敛速度。本文提出一种新的尺度不变、无参数的PRM$^+$变体——IREG-PRM$^+$,证明其在自对弈下可实现 $T^{-1/2}$ 的最佳迭代收敛和 $T^{-1}$(即最优)的平均迭代收敛。技术上,将IREG-PRM$^+$与带自适应学习率的乐观梯度下降类比,发现后者性能相当,揭示了后悔匹配家族的有效性。进一步,通过新建立的CFR与调和博弈的联系,将分析扩展至包含调和博弈及完全混合均衡的广义变分不等式问题。在加权Minty条件下,满足尺度不变RVU性质的算法(如IREG-PRM$^+$)具有常数后悔和 $T^{-1/2}$ 迭代收敛,且无需预先知晓权重。
原文摘要 · Abstract (English)
A considerable chasm has been looming for decades between theory and practice in zero-sum game solving through first-order methods. Although a convergence rate of $T^{-1}$ has long been established, the most effective paradigm in practice is counterfactual regret minimization (CFR), which is based on regret matching and its modern variants. In particular, the state of the art across most benchmarks is predictive regret matching$^+$ (PRM$^+$). Yet, such algorithms can exhibit slower $T^{-1/2}$ convergence even in self-play. In this paper, we close the gap between theory and practice. We propose a new scale-invariant and parameter-free variant of PRM$^+$, which we call IREG-PRM$^+$. We show that it achieves $T^{-1/2}$ best-iterate and $T^{-1}$ (i.e., optimal) average-iterate convergence guarantees, while also being on par or even better relative to PRM$^+$ on benchmark games. From a technical standpoint, we draw an analogy between (IREG-)PRM$^+$ and optimistic gradient descent with adaptive learning rate. Reflecting this theoretical bridge, we find that the adaptive version of optimistic gradient descent we consider performs on par with IREG-PRM$^+$. This demystifies the effectiveness of the regret matching family vis-a-vis more standard optimization techniques. Moreover, we extend our analysis beyond zero-sum games to a family of variational inequality problems that includes harmonic games, as well as extensive-form games with fully-mixed equilibria, via a new and intriguing connection between CFR and harmonic games. Unlike prior work in harmonic games, our algorithms do not require knowing the underlying weights by virtue of scale invariance. Under the weighted Minty condition, we show that any algorithm satisfying a scale-invariant RVU property (such as IREG-PRM$^+$) has constant regret (in self-play) and $T^{-1/2}$ iterate convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。