用流形优化统一前归一化与后归一化，提升Transformer性能。
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
- 将FFN和注意力层输出视为流形上的更新方向，改用测地线更新替代标准归一化
- 在多个Transformer模型上实现稳定性能提升，计算开销几乎不增加
- 适合追求模型性能优化且关注轻量改进的研究者和工程师
归一化层的位置（前归一化与后归一化）仍是Transformer架构设计中的开放问题。本文从流形优化视角重新审视这一问题，将前馈网络（FFN）和注意力层的输出解释为优化中的更新方向。基于此，我们提出GeoNorm，用流形上的测地线更新替代标准归一化。此外，类比学习率调度，我们为FFN和注意力组件设计了逐层更新衰减机制。大量实验表明，GeoNorm在多种Transformer模型中均显著优于现有归一化方法。关键的是，GeoNorm可无缝集成到标准Transformer架构中，在几乎无额外计算成本下实现性能提升。
原文摘要 · Abstract (English)
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。