发现模型泛化前的几何信号,可提前预警复杂任务的突然突破。
Early-Warning Signals of Grokking via Loss-Landscape Geometry
- 用梯度非交换性测度预测泛化出现时机。
- 该信号在扫描和嵌套括号任务中提前约1.13-1.18倍训练步数显现。
- 对所有任务均有效,适合研究模型学习机制的研究者。
Grokking——长时间训练后从记忆到泛化的突变现象——已被证明与模运算中的低维执行流形有关。但这一机制是否适用于其他任务仍不清楚。本文研究了两个序列学习基准:SCAN组合泛化和Dyck-1深度预测。在两种任务及广泛的学习率范围内,由非交换梯度更新导出的对易子缺陷(commutator defect)在泛化发生前显著上升,领先时间符合超线性幂律关系(SCAN约为1.18,Dyck约为1.13),与之前模运算结果一致。权重空间主成分分析显示,谱集中并非普遍先兆;而对易子缺陷是共通指标。因果干预实验表明其机制作用:增强非交换性可加速grokking(SCAN约32%,Dyck约50%),抑制正交梯度流则延迟或阻止泛化。三类任务呈现因果敏感性光谱——模运算刚性,Dyck响应性强,SCAN居中——但抑制均导致泛化失败,揭示必要性为普适规律。结果确立对易子缺陷作为Transformer中延迟泛化的一种鲁棒、架构无关、因果相关的早期预警信号。
原文摘要 · Abstract (English)
Grokking -- the abrupt transition from memorization to generalization after prolonged training -- has been linked to confinement on low-dimensional execution manifolds in modular arithmetic. Whether this mechanism extends beyond arithmetic remains open. We study two sequence-learning benchmarks: SCAN compositional generalization and Dyck-1 depth prediction. Across both tasks and a wide range of learning rates, the commutator defect -- a curvature measure derived from non-commuting gradient updates -- rises well before generalization, with lead times following a superlinear power law (alpha approximately 1.18 for SCAN, approximately 1.13 for Dyck), consistent with prior results on modular arithmetic. Weight-space PCA reveals that spectral concentration is not a universal precursor; the commutator defect is. Causal interventions demonstrate a mechanistic role: amplifying non-commutativity accelerates grokking (roughly 32% on SCAN, roughly 50% on Dyck), while suppressing orthogonal gradient flow delays or prevents it. The three task families form a spectrum of causal sensitivity -- modular arithmetic is rigid, Dyck is responsive, SCAN is intermediate -- yet suppression delays or prevents grokking in all cases, establishing necessity as a universal finding. These results identify the commutator defect as a robust, architecture-agnostic, causally implicated early-warning signal for delayed generalization in transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。