提出新型归一化方法Holonorm，提升Transformer模型稳定性与可解释性。
Holonorm
- 引入带残差连接的非线性归一化，解决Tanh的正交性问题
- 将向量映射至单位开球内，防止激活值爆炸，提升深层模型稳定性
- 形式类似软符号函数但适用于高维张量，结果更易解读
归一化是Transformer训练中的关键环节。DyT提出用Tanh替代层归一化（LN），虽有效但存在正交性、线性及畸变问题，不可靠。为此，本文提出Holonorm（hn），具备残差连接与非线性特性，适合作为归一化函数替代Tanh。尽管其在一维情形下与软符号函数相似，但软符号为逐元素操作，不适用于高维张量。Holonorm保持信号的正交性、方向性和可逆性，且为合适度量，能将所有向量映射至开单位球内，抑制激活值爆炸，改善深层Transformer模型的稳定性。本文系统考察了Transformer中的归一化机制，首次提出适合归一化任务的软符号函数广义形式Holonorm。此外，定义在0到1之间的Holonorm可视为比例值，1−Holonorm为其补数，便于模型评估时理解。
原文摘要 · Abstract (English)
Normalization is a key point in transformer training . In Dynamic Tanh (DyT), the author demonstrated that Tanh can be used as an alternative layer normalization (LN) and confirmed the effectiveness of the idea. But Tanh itself faces orthogonality, linearity and distortion problems. Due to that, his proposition cannot be reliable. So we propose a Holonorm (hn) which has residual connections and nonlinearity. Holonorm is suitable for replacing Tanh in the context of normalization. Although the HoloNorm expression could be similar to the softsign function in dimension one, softsign is a componentwise function which is not good for tensors and vectors of great dimension. Holonorm preserves the orthogonality, the direction, the invertibility of the signal. Holonorm is also a suitable metric, maps all vectors into the open unit ball. This prevents exploding activations and improves stability in deep Transformer models. In this work, we have meticulously examined the normalization in transformers and say that Holonorm, a generalized form of softsign function suited as a normalization function first.Second, defined between 0 and 1 hn serves as a percentage, and $1 - \text{Holonorm}$ is its complement, making it better understandable in evaluating a model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。