提出BHyT替代Pre-LN，兼顾训练稳定与效率。
Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models
- 用有界双曲正切函数控制激活值范围，防止深度增长导致的不稳定。
- 相比RMSNorm，训练快1.6%，生成吞吐高1.77%。
- 适合追求高效稳定的大型语言模型训练者使用。
预层归一化（Pre-LN）是大型语言模型（LLM）的主流选择，对训练稳定性和迁移学习至关重要。然而，Pre-LN存在重复的统计计算开销，且随层数增加，隐藏状态的幅值和方差持续增长，导致训练不稳。以效率为导向的无归一化方法（如DyT）虽提升吞吐量，但在深层结构下仍脆弱。为此，我们提出有界双曲正切（BHyT），作为Pre-LN的即插即用替代方案。BHyT结合tanh非线性与数据驱动的输入边界机制，将激活值限制在非饱和区间，抑制激活幅值与方差的深度增长，并提供理论稳定性保证。在效率方面，BHyT每块仅计算一次精确统计量，将二次归一化替换为轻量级方差近似。实验表明，BHyT在预训练中实现更高稳定性和效率：平均训练速度提升1.6%，平均令牌生成吞吐量提高1.77%，同时在语言理解与推理基准上保持与RMSNorm相当的预训练性能及后微调表现。
原文摘要 · Abstract (English)
Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning. However, Pre-LN incurs repeated statistical-computation overhead and remains vulnerable to the curse of depth, where hidden-state magnitudes and variances grow as the number of layers increases, destabilizing training. Efficiency-oriented normalization-free methods such as Dynamic Tanh (DyT) improve throughput but remain fragile at depth. To jointly address stability and efficiency, we propose Bounded Hyperbolic Tanh (BHyT), a drop-in replacement for Pre-LN. BHyT combines a tanh nonlinearity with explicit, data-driven input bounding to keep activations within a non-saturating range. It prevents depth-wise growth in activation magnitude and variance and provides a theoretical stability guarantee. For efficiency, BHyT computes exact statistics once per block and replaces a second normalization with a lightweight variance approximation. Empirically, BHyT demonstrates improved stability and efficiency during pretraining, achieving an average of 1.6\% faster training and an average of 1.77\% higher token generation throughput compared to RMSNorm, while maintaining strong pretraining-only and post-SFT performance across language understanding and reasoning benchmarks\footnote{Code is available at: https://github.com/MLAI-Yonsei/BHyT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。