arXiv:2607.11666cs.LG2026-07

通过控制模型表示几何结构,可显著加速神经网络的泛化能力出现。

How to Tame Grokking: Representation Geometry as a Control Signal

论文配图:How to Tame Grokking: Representation Geometry as a Control Signal
图 1 · 摘自论文原文
  • 用谱正则化调节隐藏层表示的维度,影响模型学习路径。
  • 在多种任务中,泛化速度最快提升52倍,且适用于多层感知机与Transformer。
  • 研究揭示了表示几何是延迟泛化的关键控制信号,适合研究泛化机制的学者。

Grokking 是一种神经网络先记忆训练数据、经过长时间优化后才表现出强泛化能力的现象。尽管已有大量研究,但其出现时机与影响因素仍不明确。本文研究表示几何与延迟泛化的关系,发现维度坍缩在所有测试设置中均先于 grokking 出现。受此启发,提出几何维度正则化(GeomDR),一种简单的谱正则化方法,用于调控训练过程中隐藏表示的有效维度。在模加法、模除法和排列组合任务中,GeomDR 稳定改变了 grokking 动态,根据干预时机和目标维度,显著加速泛化。在多个场景下,泛化启动时间相比标准 AdamW 训练最快提速 52 倍。在多层感知机和 Transformer 中均观察到相似定性效果。结果表明,表示几何可作为 grokking 的有效控制信号,为研究和调控延迟泛化提供了实用路径。

原文摘要 · Abstract (English)

Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.

神经网络泛化能力表示几何深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。