权重范数不直接控制模型泛化延迟，真正关键的是输出逻辑值尺度。
What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
- 通过固定权重范数，仅调节输出温度来控制逻辑值尺度。
- 逻辑值尺度能解释97%的泛化延迟差异，权重范数贡献不足2%。
- 适用于研究模型泛化机制与优化路径的学者，尤其关注训练动态者。
Grokking现象中，模型从记忆到泛化的延迟通常与权重范数相关：范数越小，泛化越早。本文探究范数实际控制什么。通过夹持（clamp）权重范数并仅调整输出温度，使跨熵损失下的泛化延迟覆盖整个范数决定的范围；将有效逻辑值尺度恢复至基线后，约85%的延迟被重建。在多组范数与温度组合下，延迟可完全由逻辑值尺度统一描述（R² = 0.97），权重范数仅额外贡献1-2%。该效应依赖于损失函数：均方误差下，逻辑值尺度被锁定，范数通过不同路径起作用。记忆性控制实验、浮点64位Softmax坍缩审计及无层归一化变换器均指向同一机制。从相同初始状态分叉的模型，其延迟遵循固定的范数值而非夹持操作，排除了重缩放伪影的疑虑。结论是：近端变量为逻辑值尺度及其引发的Softmax饱和，权重范数仅为上游调节手柄。所有数据、表格与图表均可由公开代码与数据复现。
原文摘要 · Abstract (English)
Grokking, the delayed jump from memorization to generalization, is usually tied to the weight norm: a smaller norm generalizes sooner. We ask what the norm actually controls. Holding the weight norm fixed by clamping and varying only an output temperature, we slide the grokking delay across its entire norm-induced range under cross-entropy; matching the effective logit scale back to baseline recovers about 85% of the delay at two moduli. Across a grid of norms and temperatures the delay collapses onto the logit scale alone (R2 = 0.97), with the norm adding 1-2% beyond it. The effect is loss-dependent: under mean-squared error the logit scale is pinned and the norm acts through a different route. A memorization control, a float64 softmax-collapse audit, and a no-LayerNorm transformer point to the same channel. Forking arms from one identical state, the delay follows the held norm value and not the clamp operation, which closes a rescaling-artifact concern. The proximal variable is the logit scale and the softmax saturation it drives; the weight norm is only an upstream handle. All numbers, tables, and figures reproduce from released code and data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。