用物理启发的奇异学习理论，揭示神经网络中的相变现象
Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks
- 基于代数几何构建的奇异学习理论，解释非可识别模型的学习机制
- 在模运算和超位置玩具模型中验证了自由能的阿伦尼乌斯速率规律
- 发现学习系数随问题难度变化的规律，揭示理论与实际的偏差
经典统计推断与学习理论难以解释现代神经网络的成功。主要原因在于这些模型具有不可识别性（奇异），违反了PAC界和渐近正态性的核心假设。奇异学习理论（SLT）是一种源于物理学的框架，基于代数几何，近年来因其能弥合理论与实践的差距而受到关注。本文在与可解释性和相变相关的简化设置中，对SLT进行实证研究。首先，通过模运算的grokking模型和Anthropic的超位置玩具模型，检验了阿伦尼乌斯风格的速率假设，以理解SLT自由能$\mathcal{F}_n$。其次，通过测量多个受控网络族（多项式回归器、低秩线性网络、低秩自编码器）中局部学习系数$λ_α$随问题难度的变化规律，揭示其缩放行为。实验结果既恢复了已知的标度律，也发现了与理论预期不符的有意义偏差。总体而言，本文展示了SLT在理解神经网络相变方面的诸多优势，并提出了该领域尚未解决的研究问题。
原文摘要 · Abstract (English)
Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspired framework grounded in algebraic geometry, has gained popularity for its ability to close this theory-practice gap. In this paper, we empirically study SLT in toy settings relevant to interpretability and phase transitions. First, we understand the SLT free energy $\mathcal{F}_n$ by testing an Arrhenius-style rate hypothesis using both a grokking modulo-arithmetic model and Anthropic's Toy Models of Superposition. Second, we understand the local learning coefficient $λ_α$ by measuring how it scales with problem difficulty across several controlled network families (polynomial regressors, low-rank linear networks, and low-rank autoencoders). Our experiments recover known scaling laws while others yield meaningful deviations from theoretical expectations. Overall, our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。