揭示层归一化如何影响Transformer的记忆与泛化能力
Impact of Layer Norm on Memorization and Generalization in Transformers
- 对比预归一化与后归一化架构，发现层归一化对学习稳定性与记忆性有不同作用
- 去除预归一化模型的归一化参数会加剧记忆，导致学习不稳；后归一化则抑制记忆
- 早期层的归一化参数最关键，适用于研究模型训练机制与过拟合控制的研究者
层归一化（LayerNorm）是Transformer中的核心组件，用于稳定训练并提升优化性能。近年来，预归一化架构因梯度流更稳定而优于后归一化架构。然而，层归一化对两类架构在学习与记忆方面的影响仍不明确。本文系统研究了其在预归一化与后归一化Transformer中的作用。结果表明，层归一化是预归一化模型实现稳定学习的关键，而在后归一化模型中则显著影响记忆行为。我们发现，移除预归一化模型的归一化参数会加剧记忆并破坏学习稳定性，而移除后归一化模型的参数则有效抑制记忆，恢复真实标签。进一步分析表明，早期层的层归一化最为关键，其影响在两类架构间存在差异。研究在6个视觉与语言数据集上的13种模型上验证了结论，为理解层归一化在记忆与学习中的作用提供了新视角。
原文摘要 · Abstract (English)
Layer Normalization (LayerNorm) is one of the fundamental components in transformers that stabilizes training and improves optimization. In recent times, Pre-LayerNorm transformers have become the preferred choice over Post-LayerNorm transformers due to their stable gradient flow. However, the impact of LayerNorm on learning and memorization across these architectures remains unclear. In this work, we investigate how LayerNorm influences memorization and learning for Pre- and Post-LayerNorm transformers. We identify that LayerNorm serves as a key factor for stable learning in Pre-LayerNorm transformers, while in Post-LayerNorm transformers, it impacts memorization. Our analysis reveals that eliminating LayerNorm parameters in Pre-LayerNorm models exacerbates memorization and destabilizes learning, while in Post-LayerNorm models, it effectively mitigates memorization by restoring genuine labels. We further precisely identify that early layers LayerNorm are the most critical over middle/later layers and their influence varies across Pre and Post LayerNorm models. We have validated it through 13 models across 6 Vision and Language datasets. These insights shed new light on the role of LayerNorm in shaping memorization and learning in transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。