用运行平均参考策略改进双人零和博弈的自对弈收敛性。
GARIP: A Running-Average Moving Reference for Last-Iterate Self-Play in Two-Player Zero-Sum Games

- 以运行平均作为参考,稳定自对弈过程中的迭代轨迹。
- 在棋类游戏上表现优于固定参考与无参考基线,且鲁棒性强。
- 适合研究自对弈训练中收敛性与稳定性问题的研究者。
双人零和博弈中采用朴素梯度上升的自对弈会产生最后迭代点围绕均衡振荡的问题。现有方法通过向参考策略正则化来恢复收敛:MMD使用固定参考(仅达正则化均衡),R-NaD使用周期快照(DeepNash的核心)。本文提出GARIP,锚定于运行平均参考,并分析参考选择的影响。核心机制为:塌陷追踪参考的滞后峰值,而在均值滞后固定的因果凸平均中,运行平均(平滑轮廓,峰值=均值)唯一最小化该峰值,而快照锯齿形结构的峰值为均值的两倍(一句定理)。由此得出两个结论:收敛性方面,证明了在恒定锚定强度下局部最后迭代收敛——锚定将基础映射旋转缩放为1-β,越过稳定边界,将循环基础转为收缩映射(小β下全局收敛为猜想;大β下存在共识失败)。鲁棒性方面,GARIP性能与R-NaD相当,在矩阵博弈、硬币游戏及连珠/奥赛罗等棋类游戏中均显著优于固定磁极与无磁极基线,且是更优的超参数默认设置;全网格搜索中崩溃率统计无差异,但在常规参数下,匹配均值滞后的设定下0/40种子崩溃,而快照需知悉缩短K才可匹配。边界方面,前瞻型(负权重)参考在滞后侧表现更佳,优势仅出现在朴素自对弈循环深度达五层时。所有实验均为纯JAX实现且可复现。
原文摘要 · Abstract (English)
Self-play with naive gradient ascent cycles in two-player zero-sum games: the last iterate orbits the equilibrium. Modern methods restore last-iterate convergence by regularizing toward a reference policy -- MMD a fixed one (reaching only the regularized equilibrium), R-NaD a periodic snapshot (the engine of DeepNash). We study GARIP, which anchors to the running average, and isolate what the choice of reference controls. Our central result is a mechanism: collapse tracks the peak lag of the reference, and among causal convex averages of a fixed mean lag the running average (flat profile, peak $=$ mean) uniquely minimizes that peak, while a snapshot's sawtooth has peak $= 2\times$ mean (a one-line theorem). Two consequences follow. Convergence: we prove local last-iterate convergence at constant anchor strength -- the anchor scales the base map's rotation by $1-β$, crossing the stability boundary and turning a recurrent base into a contraction (global convergence is conjectured at small $β$; we characterize a large-$β$ consensus failure). Robustness: GARIP matches R-NaD's peak performance -- on matrix games, the Coin Game, and the board games Connect Four/Othello, both moving references are far more robust than fixed-magnet and magnet-free baselines -- but is the better hyperparameter default; we report it both ways: over the full grid collapse rates are statistically indistinguishable, yet at conventional parameterizations a matched-mean-lag setting collapses in 0/40 vs 10/40 seeds (a snapshot matches it only by knowing to shorten $K$). The boundaries: an anticipatory (negative-weight) reference does better still on the stale side, and the advantage appears only where naive self-play cycles (five deep self-play loops). All experiments are pure JAX and reproducible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。