arXiv:2608.14803cs.LG2026-08

研究模型泛化跃迁是否由奇异值消失引发,发现无证据支持此猜想。

Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?

论文配图:Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?
图 1 · 摘自论文原文
  • 用最小非零奇异值检测插值流形的恢复速率变化
  • 跃迁时奇异值达峰值而非归零,表明无突变现象
  • 结果支持平滑收敛机制,不支持分岔假说

近期研究将模型在记忆后的泛化阶段视为约束优化:网络完成训练集插值后,权重衰减驱动其沿零损失流形缓慢移动以降低范数。从动力系统视角看,这是快慢系统,其中插值流形扮演慢流形角色。本文提出一个关键问题:泛化跃迁是否源于该流形的正常双曲性丧失?即法向恢复方向趋于平坦的折叠或分岔事件?抑或流形始终具有吸引力,泛化仅通过平滑漂移实现?我们提出一种不依赖优化器的简单诊断方法:残差雅可比矩阵的最小非零奇异值 $σ_{ ext{min}}^{+}(oldsymbol{J})$,对平方损失而言,它对应于流形最慢的法向恢复速率。在两层ReLU网络训练模加法任务并使用平方损失的情况下,$σ_{ ext{min}}^{+}(oldsymbol{J})$ 在跃迁时刻并未坍缩;相反,它在跃迁期间达到最大值,仅在记忆前趋近零。该结果在五个随机种子下一致,且前六小奇异值行为相同,无局部子空间坍缩。这初步表明分岔假说不成立,支持平滑收缩图景。但需明确,单一设置下的渐进过渡实验不能完全排除分岔存在,仅限定了其可能隐藏的位置。

原文摘要 · Abstract (English)

A recent line of work recasts the post-memorization phase of grokking as constrained optimization: once a network interpolates the training set, weight decay drives a slow drift along the zero-loss manifold toward lower norm. In the language of dynamical systems, this is a fast-slow system in which the interpolation manifold plays the role of a slow manifold. We ask a question that this framing makes natural but the existing literature does not address: is the sharp generalization transition a loss of normal hyperbolicity of that manifold: a fold- or bifurcation-like event in which a normal restoring direction goes flat? Or does the manifold stay uniformly attracting while generalization happens by smooth drift? We propose a simple, optimizer-agnostic diagnostic: the smallest nonzero singular value $σ_{\min}^{+}(\mathbf J)$ of the residual Jacobian, which, for the squared loss, equals the slowest normal restoring rate of the manifold. On a two-layer ReLU network trained to grok modular addition under squared loss, $σ_{\min}^{+}(\mathbf J)$ does not collapse at the transition; it is near zero only before memorization and attains its largest values during the transition. The result holds across five seeds, and the six smallest singular values behave identically; there is no subspace-local collapse either. This is preliminary evidence against the bifurcation hypothesis and in favor of the smooth-contraction picture. We are explicit that a single-setting, gradual-transition experiment under Adam optimizer does not prove the absence of a bifurcation; it constrains where one could hide.

深度学习泛化动力系统奇异值分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。