ReLU在无归一化大模型中表现远超GELU，提升8.2%困惑度。
ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models
- 采用ReLU激活函数，避免了传统GELU在早期层的熵过载问题。
- 无归一化架构下，ReLU使注意力头表征能力提升，困惑度降低8.2%。
- 适合追求高效、可解释性高的自回归语言模型优化场景。
层归一化（LayerNorm）是现代大语言模型中稳定训练和保证优化平滑的关键组件。然而，它带来了机制可解释性差、异常值特征抑制、信号传播失真以及私有推理中的计算与通信开销等挑战。本文研究无归一化解码器型大语言模型中理想的激活函数。与主流模型偏好GELU不同，实验发现：在无归一化设置下，ReLU显著优于GELU，带来8.2%的困惑度降低。我们发现GELU在早期层存在熵过载问题，导致注意力头表征能力未被充分使用。这表明，如GELU这类平滑激活函数不适用于无归一化架构；而ReLU具备输入空间特化与类内选择性等几何特性，在缺乏归一化时能改善学习动态并更好保留信息。该研究为克服归一化引入挑战的Transformer架构优化提供了关键洞见。代码与实现见https://github.com/Nandan91/relu-revival-normfree。
原文摘要 · Abstract (English)
LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. However, it introduces significant challenges in mechanistic interpretability, outlier feature suppression, faithful signal propagation, and computational and communication complexity of private inference. This work explores desirable activation functions in normalization-free decoder-only LLMs. Contrary to the conventional preference for the GELU in transformer-based models, our empirical findings demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We discover a key issue with GELU, where early layers experience entropic overload, leading to the under-utilization of the representational capacity of attention heads. This highlights that smoother activations like GELU are {\em ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical properties -- specialization in input space and intra-class selectivity -- lead to improved learning dynamics and better information retention in the absence of LayerNorm. This study offers key insights for optimizing transformer architectures where LayerNorm introduces significant challenges. The code and implementation are available at https://github.com/Nandan91/relu-revival-normfree
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。