通过抑制激活向量径向膨胀，加速神经网络发现结构化规律的泛化过程。
Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization
- 提出径向-角度分解，揭示泛化延迟源于激活空间的径向膨胀。
- 施加范数约束后，训练速度提升6倍，10M参数模型训练步数减半。
- 适合研究泛化机制、模型优化与架构设计的科研人员参考。
为何神经网络会在泛化前长时间记忆算法型训练数据？我们通过几何案例研究发现，在需要发现结构化低维电路的任务中，交叉熵优化导致隐藏表示的径向膨胀，从而引发记忆-泛化延迟。我们形式化了激活空间动态的径向-角度分解，并推导出三个可验证命题：(i) 惩罚径向膨胀会诱导各向异性的、依赖数据的权重正则化；(ii) 它将径向梯度能量压低于各向同性随机基线，迫使主要进行角度更新；(iii) 它倾向于收敛到更平坦的极小值。为实证这些命题，我们研究了一种仅含一个超参数的范数惩罚，其软性约束激活在√d半径的超球面上。在模运算任务中，该方法使MLP和Transformer的grokking速度最高加速6倍；在3位数加法任务上，1000万参数的nanoGPT训练步数减少一半。
原文摘要 · Abstract (English)
Why do neural networks memorize algorithmic training data long before they generalize? We present a geometric case study demonstrating that, on tasks where generalization requires discovering structured low-dimensional circuits, the memorization-generalization delay is driven by radial inflation of hidden representations under cross-entropy optimization. We formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions: (i) that penalizing radial inflation induces anisotropic, data-dependent weight regularization; (ii) that it suppresses radial gradient energy below the isotropic random baseline, forcing predominantly angular updates; and (iii) that it biases convergence toward flatter minima. To empirically validate these propositions, we study a single-hyperparameter norm penalty that softly constrains activations to a sqrt(d)-radius hypersphere. On modular arithmetic, this penalty accelerates grokking up to 6x across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。