发现神经网络在突现学习中,特征排斥的数学机制与权重更新谱结构存在解耦现象。
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

- 通过分析特征相关矩阵的负偏移项,揭示相似特征间的排斥力作用
- 激活函数决定谱结构:平方激活下出现显著双峰特征值分离,而ReLU则保持单峰
- 该现象解释了为何某些激活方式能触发突现学习,适合研究模型内部表征动态者阅读
本文在Tian(2025)提出的模块加法任务(M=71, K=2048, MSE损失)上实证检验了两层网络突现学习中的特征排斥机制。结果表明,尽管特征间排斥的符号规则在所有种子下稳定成立(σ=x²时符号匹配率从0.865升至0.985,σ=ReLU时达1.000),但其对应的参数更新谱特征却强烈依赖于激活函数。当使用σ=x²时,滚动特征值比σ₂/σ₃的斜率检测器在15个突现学习种子中均于第174轮(四分位距[173,174])触发,且后期特征值差距达229倍,谱秩为2;而在σ=ReLU情况下,检测器从未触发,谱始终近似秩1。这一解耦现象印证了定理5中聚焦(幂律)与扩散(ReLU)记忆的区别:虽然特征排斥的符号结构仅由特征内积矩阵决定,但其如何转化为权重更新则取决于激活导数σ'。
原文摘要 · Abstract (English)
Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix $ B = (\widetilde{F}^\top \widetilde{F} + ηI)^{-1} $ during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries $ B_{j\ell} $, producing an effective repulsive force that drives them apart. However, the theorem does not specify when this mechanism becomes empirically observable, nor whether it leaves a measurable spectral signature in the parameter updates. We test this directly on Tian's modular addition setup ($ M = 71 $, $ K = 2048 $, MSE loss) and observe a clear structure-mechanism dissociation. The predicted sign rule holds robustly on the top-200 most-similar feature pairs across activations (empirical sign-match rising from 0.865 to 0.985 on $ σ= x^2 $ across 5 seeds, and saturating at 1.000 on $ σ= \operatorname{ReLU} $). However, the spectral signature in the parameter updates is strongly activation-dependent. With $ σ= x^2 $, a simple slope detector on the rolling eigengap $ σ_2 / σ_3 $ of $ ΔW $ fires in 15/15 grokking seeds at epoch 174 (IQR [173,174]) and in 0/15 non-grokking controls, with 229$ \times $ late-stage magnitude separation; the spectrum is rank-2. In contrast, with $ σ= \operatorname{ReLU} $, the detector never fires and the spectrum remains effectively rank-1. This dissociation aligns with Tian's Theorem 5 distinction between focused (power-law) and spreading (ReLU) memorization: while the sign structure of $ B $ depends only on $ \widetilde{F}^\top \widetilde{F} $, how feature repulsion translates into weight updates critically depends on the activation derivative $ σ' $.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。