揭示层归一化本质，证明其冗余步骤并推荐更优的RMSNorm。
Geometric Interpretation of Layer Normalization and a Comparative Analysis with RMSNorm
- 将层归一化视为移除均匀向量分量并标准化剩余部分
- 发现所有LLM在推理时均自然正交于均匀向量，说明该步骤冗余
- 提出RMSNorm更高效且无需冗余操作，适合追求性能的模型设计
本文提出了层归一化（LayerNorm）的几何解释，揭示其与统一向量 $oldsymbol{1} = [1,1,1,oldots,1]^T \in \mathbb{R}^d$ 的内在关联。层归一化的标准化过程可分解为三步：(i) 去除向量在统一向量方向上的分量，(ii) 标准化剩余部分，(iii) 将结果缩放 $\sqrt{d}$。我们进一步分析了推理阶段的层归一化行为，发现基于层归一化的语言模型在推理时，隐藏表示始终自然正交于统一向量，即平均无沿统一向量的分量。这首次提供了机制层面的证据，表明层归一化中移除统一向量分量是冗余的。由此建议采用计算更高效的RMSNorm作为替代方案。
原文摘要 · Abstract (English)
This paper presents a novel geometric interpretation of LayerNorm and explores how LayerNorm influences the norm and orientation of hidden vectors in the representation space. With these geometric insights, we prepare the foundation for comparing LayerNorm with RMSNorm. We show that the definition of LayerNorm is innately linked to the uniform vector, defined as $\boldsymbol{1} = [1, 1, 1, 1, \cdots, 1]^T \in \mathbb{R}^d$. We then show that the standardization step in LayerNorm can be understood in three simple steps: (i) remove the component of a vector along the uniform vector, (ii) normalize the remaining vector, and (iii) scale the resultant vector by $\sqrt{d}$, where $d$ is the dimensionality of the representation space. We also provide additional insights into how LayerNorm operates at inference time. Finally, we compare the hidden representations of LayerNorm-based LLMs with models trained using RMSNorm and show that all LLMs naturally operate orthogonal to the uniform vector at inference time, that is, on average they do not have a component along the uniform vector during inference. This presents the first mechanistic evidence that removing the component along the uniform vector in LayerNorm is a redundant step. These results advocate for using RMSNorm over LayerNorm which is also more computationally efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。