权重衰减控制变压器模型的三种学习状态，可低成本在线监测。
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
- 用权重衰减作为控制参数，划分记忆、泛化和崩溃三阶段。
- 发现临界点λ_c=0.0158，幂律指数ν=0.757，偏离经典模型。
- 仅用注意力激活即可实时追踪训练状态，适合小规模实验。
在模运算任务中，变压器模型展现出从记忆到泛化再到崩溃的剧烈转变。本文表明权重衰减是这些状态的标量控制参数，并提出两种低成本在线诊断方法：平均成对注意力头余弦相似度与熵标准差，仅基于注意力激活即可跟踪训练动态，且计算成本低于损失景观分析。在十一组实验条件与三种模型规模（0.82M至85M参数）下，权重衰减轴清晰分离三类状态。近临界点的逻辑回归拟合得到记忆-发展性泛化边界λ_c=0.0158（95%置信区间[0.0109, 0.0200]，N=210）；幂律拟合得经验指数ν=0.757（置信区间[0.725, 0.799]）。参考值ν=1/2与三维伊辛模型ν≈0.63均超出此置信区间，故将ν视为经验值，通用性分类留待更密集的有限尺度标度研究。跨任务多任务复现（n=280，四种模运算）保持权重衰减控制模式；在λ=0.05时，注意力头重初始化显著改变第二阶段振幅（Cohen's d=-1.190，n=10，p_t=4.5×10⁻³），而权重范数截断则无影响。三种跨架构探测（4层MLP、4层LSTM、4层Mamba；各n=70）也复现了权重衰减控制的转变，但具架构特异性λ_c。主要结论限定于小型变压器注意力模型中的模运算；非注意力实验仅为范围探针，关于架构普适性、语言模型或通用性类别的推论不在本文范围内。
原文摘要 · Abstract (English)
Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse. We show that weight decay acts as a scalar empirical control parameter for these regimes, and introduce two cheap online diagnostics, mean pairwise attention-head cosine similarity and entropy standard deviation, that track training dynamics from attention activations alone and complement loss-landscape diagnostics at lower compute cost. Across eleven experimental conditions and three model scales (0.82M to 85M parameters), the weight-decay axis separates memorization, developmental grokking, and collapse. A near-transition logistic fit localizes the memorization-to-developmental boundary at $λ_c=0.0158$ (95% CI [0.0109, 0.0200], N=210); a power-law fit gives an empirical exponent $ν=0.757$ (CI [0.725, 0.799]). Reference exponents $ν=1/2$ and 3D Ising $ν\approx 0.63$ lie outside this empirical CI under our four-bin grid, so we report $ν$ as empirical and defer universality-class identification to denser finite-size-scaling work. A horizon-matched multi-task replication (n=280, four modular operations) preserves the weight-decay control pattern; a paired attention-head re-initialization experiment at $λ=0.05$ changes Phase-2 amplitude (Cohen's $d=-1.190$, n=10, $p_t=4.5 \times 10^{-3}$), while matched weight-norm clipping does not. Three cross-architecture probes (4L MLP, 4L LSTM, and 4L Mamba; each n=70) replicate the weight-decay-controlled transition with architecture-specific $λ_c$ values. Main diagnostic claims are scoped to modular arithmetic in small transformer attention models; the non-attention experiments are scope probes, and architecture-wide, language-model, and universality-class claims are out of scope.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。