移除LayerNorm后，激活值截断能提升小模型表现，但大模型反而变差。
When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
- 用可学习的tanh函数截断激活值，替代LayerNorm，实现动态归一化。
- 小数据量（100万）下验证损失降低27.3%，大数据量（1.18亿）下反而上升18.8%。
- 激活饱和度是关键机制，适合研究小规模模型优化与训练稳定性问题。
Dynamic Tanh（DyT）通过可学习的tanh(alpha x)对激活值进行截断，以替代LayerNorm。研究表明，这种截断是一种依赖于训练规模的隐式正则化，而非普遍有益的替代方案。在涵盖64M至3.78B参数的GPT-2系列模型及Llama、ViT的交叉验证中，当训练数据为100万样本时，验证损失下降27.3%；而数据增至1.18亿时，损失反而上升18.8%。容量增加至3.78B后，100万样本的优势仅回升1.7%，而1.18亿样本的负效应加剧至+27.9%。机制可测量：在100万样本下49%的激活值饱和，而1.18亿样本下仅为23%。500步饱和启发式方法在12单元GPT-2校准集上达到75%原始准确率（AUC 0.75），加入Scale 5压力单元后降至64%。该方法正确判别了全部3个Llama检查项，但在留一尺度测试中准确率仅达50%。三种干预验证了截断机制：HardTanh复现相同模式，提高alpha可单调缓解1.18亿样本下的惩罚，且标准模型+0.5丢弃率匹配了DyT在数据充足时的表现。此外，将Llama-DyT崩溃定位到SwiGLU门控模块，激活饱和程度在3种子实验中与收敛性相关性高达r=0.94。所有实验均在计算受限条件下进行（T/P < 1.84），低于Chinchilla最优训练水平。
原文摘要 · Abstract (English)
Dynamic Tanh (DyT) removes LayerNorm by bounding activations with a learned tanh(alpha x). We show that this bounding is a regime-dependent implicit regularizer, not a uniformly beneficial replacement. Across GPT-2-family models spanning 64M to 3.78B parameters and 1M to 118M tokens, with Llama and ViT cross-checks, DyT improves validation loss by 27.3% at 64M/1M but worsens it by 18.8% at 64M/118M; the 1M benefit vanishes with capacity (+1.7% at 3.78B), while the 118M penalty reaches +27.9%. The mechanism is measurable: 49% of DyT activations saturate at 1M versus 23% at 118M, and a 500-step saturation heuristic classifies DyT's sign with 75% raw in-sample accuracy on the 12-cell GPT-2 calibration set (AUC 0.75; 64% when adding Scale 5 stress cells), correctly labels 3/3 Llama checks, but only reaches 50% raw leave-one-scale-out accuracy. Three interventions support the bounding explanation: HardTanh reproduces the regime pattern, increasing alpha at 118M monotonically reduces DyT's penalty, and vanilla+dropout(p=0.5) matches DyT's data-rich loss. We also localize Llama-DyT collapse to SwiGLU gating, where saturation separates collapse from convergence in a 3-seed component ablation (r=0.94). Scope: all experiments are compute-limited (T/P < 1.84), below Chinchilla-optimal training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。