用拓扑方法发现模型理解任务时的隐藏结构特征
Topological Signatures of Grokking
- 通过持久同调分析模型嵌入层的点云数据
- 发现grokking对应一维同调的持久性显著提升
- 适合研究神经网络学习机制的学者参考
我们从拓扑视角研究grokking现象。对在不同素数模运算任务上训练的一系列模型，利用其嵌入矩阵生成点云，并采用持久同调分析，识别出grokking的清晰一致拓扑特征：一维同调（H₁）的最大持久性和总持久性均出现显著上升。持久图谱显示，一个主导的长寿命拓扑特征出现，同时次级特征趋于有序，反映出任务的循环结构本质。与现有的频谱和几何诊断方法（如傅里叶分析、局部内在维度）相比，持久同调提供了统一的几何与拓扑表征，能捕捉局部与全局多尺度结构。在不同数据设置和对照实验中的消融分析表明，这些拓扑转变与泛化相关，而非记忆。结果表明，持久同调为分析神经网络训练中内化潜在结构提供了一个原则性强且可解释的框架。
原文摘要 · Abstract (English)
We study the grokking phenomenon through the lens of topology. Using persistent homology on point clouds derived from the embedding matrices of a range of models trained on modular arithmetic with varying primes, we identify a clear and consistent topological signature of grokking: a sharp increase in both the maximum and total persistence of first homology ($H_1$). Persistence diagrams reveal the emergence of a dominant long-lived topological feature together with increasingly structured secondary features, reflecting the underlying cyclic structure of the task. Compared to existing spectral and geometric diagnostics -- specifically, Fourier analysis and local intrinsic dimension -- persistent homology provides a unified geometric and topological characterization of representation learning, capturing both local and global multi-scale structure. Ablations across data regimes and control settings show that these topological transitions are tied to generalization rather than memorization. Our results suggest that persistent homology offers a principled and interpretable framework for analyzing how neural networks internalize latent structure during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。