arXiv:2603.15492cs.LGcs.AI2026-03被引 1

揭示了自适应优化器如何通过谱门控机制实现长期泛化。

Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

  • 提出谱门控机制,解释优化器噪声与损失曲面曲率的动态交互
  • 发现泛化依赖梯度方差积累,突破稳定性阈值后才可进入尖锐解空间
  • 证明自适应优化器的各向异性噪声是泛化关键,非平坦极小值理论不适用

标准优化理论难以解释‘突现泛化’(grokking)现象——即模型在训练收敛后很久才开始泛化。尽管几何分析将其归因于缓慢漂移,却常忽略优化器噪声结构与损失曲面曲率的相互作用。本文研究AdamW在模算术任务上的动态,揭示了一种‘谱门控’机制,调控从记忆到泛化的转变。发现AdamW实质为方差受限的随机系统:泛化解位于初始低方差条件下无法进入的尖锐盆地(λ_{max}^H)。‘延迟阶段’是梯度方差积累的过程,直至有效稳定性上限被突破,方可进入该尖锐流形。消融实验识别出三种复杂性区域:(1) 容量坍缩(P < 23),秩不足导致结构学习失败;(2) 方差受限区(P ≈ 41),泛化等待谱门开启;(3) 稳定性覆盖区(P > 67),记忆态维度上变得不稳定。此外,挑战算法任务中‘平坦极小值’假设,证明各向同性噪声注入无法引发突现泛化。泛化需自适应优化器特有的各向异性整流能力,使噪声定向进入解流形的切空间。

原文摘要 · Abstract (English)

Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a ``Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The ``delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) \textbf{Capacity Collapse} ($P < 23$), where rank-deficiency prevents structural learning; (2) \textbf{The Variance-Limited Regime} ($P \approx 41$), where generalization waits for the spectral gate to open; and (3) \textbf{Stability Override} ($P > 67$), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the \textit{anisotropic rectification} unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.

优化器机制突现泛化谱门控自适应优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。