让层归一化在不改变预测结果的前提下，提速2%~12%
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
- 通过约束线性层输出为零均值，将归一化的中心操作合并到前序层
- 实测多个主流模型中2%~12%的推理加速，预测结果完全不变
- 适合追求高效推理又不想改模型的开发者使用
层归一化（LN）是现代深度学习的核心组件，但其逐样本中心化与缩放带来显著推理开销。RMSNorm通过移除中心化操作提升效率，但可能损失中心化带来的收益。本文提出一种框架，判断任意神经网络中的LN是否可被RMSNorm替换而不改变模型函数。核心思路是通过列中心约束（CCC）和基于列的权重中心化（CBWC），将LN的中心操作折叠至上游通用线性层。我们扩展分析至任意DNN，定义此类LN为可折叠LN，并设计基于图的检测算法。分析表明，许多主流架构中的LN可折叠，实现推理时精确转换，端到端加速2%~12%且不影响预测。多任务实验进一步显示，在实际训练中等价性部分破坏时，该方法仍保持与原始LN相当性能的同时提升效率。
原文摘要 · Abstract (English)
Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves efficiency by removing the centering operation, yet this may discard benefits associated with centering. This paper propose a framework to determine whether an LN in an arbitrary DNN can be replaced by RMSNorm without changing the model function. The key idea is to fold LN's centering operation into upstream general linear layers by enforcing zero-mean outputs through the column-centered constraint (CCC) and column-based weight centering (CBWC). We extend the analysis to arbitrary DNNs, define such LNs as foldable LNs, and develop a graph-based detection algorithm. Our analysis shows that many LNs in widely used architectures are foldable, enabling exact inference-time conversion and end-to-end acceleration of 2% to 12% without changing model predictions. Experiments across multiple task families further show that, when exact equivalence is partially broken in practical training settings, our method remains competitive with vanilla LN while improving efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。