用简单函数替代归一化层，让Transformer性能不降反升。
Transformers without Normalization
- 引入动态tanh函数作为归一化层的直接替代品
- 在多种任务中达到或超越带归一化的模型表现
- 无需调参即可适用，挑战归一化必要性的传统认知
归一化层在现代神经网络中无处不在，长期被视为不可或缺。本文证明，通过一种极为简单的技术，无需归一化层的Transformer仍可实现相同甚至更优的性能。我们提出动态tanh（DyT）：一种逐元素操作 $DyT(x) = \tanh(\alpha x)$，可作为Transformer中归一化层的即插即用替代方案。DyT的灵感来源于观察到，Transformer中的层归一化常产生类似tanh的S型输入输出映射。引入DyT后，无需归一化层的Transformer在多种设置下均能匹配或超越其带归一化版本的表现，且大多无需超参数调整。我们在从识别到生成、监督到自监督学习，以及计算机视觉与语言模型等多种场景中验证了该方法的有效性。这些发现挑战了归一化层在现代神经网络中不可或缺的传统认知，为深层网络中归一化的作用提供了新见解。
原文摘要 · Abstract (English)
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation $DyT($x$) = \tanh(α$x$)$, as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, $S$-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。