归一化位置影响模型初始预测分布，决定训练起点是否偏倚。
Where You Place the Norm Matters: From Prejudiced to Neutral Initializations
- 通过理论分析归一化类型与位置对初始输出分布的影响
- 发现预归一化易导致预测集中，后归一化更趋中性
- 为模型设计提供可解释的初始化控制依据
归一化层用于稳定和加速训练，但其在初始化阶段的影响至关重要，会塑造参数适应数据前的信号传播与输出统计。实践中，归一化方法的选择及放置位置（预归一化或后归一化）常凭经验决定，然而这些选择会定性改变模型行为。本文从理论上揭示了归一化方式与位置如何决定初始化时的类别预测分布，从无偏（中性）到高度集中（偏倚）的模式。我们证明这些架构决策会系统性地改变初始预测状态，从而调控后续学习动态。通过将归一化设计与初始预测统计直接关联，研究为更可控、可解释的网络设计提供了原则性指导，澄清了如BatchNorm与LayerNorm、Pre-Norm与Post-Norm等常用选择如何从训练起点即影响模型行为。
原文摘要 · Abstract (English)
Normalization layers were introduced to stabilize and accelerate training, yet their influence is critical already at initialization, where they shape signal propagation and output statistics before parameters adapt to data. In practice, both which normalization to use and where to place it are often chosen heuristically, despite the fact that these decisions can qualitatively alter a model's behavior. We provide a theoretical characterization of how normalization choice and placement (Pre-Norm vs. Post-Norm) determine the distribution of class predictions at initialization, ranging from unbiased (Neutral) to highly concentrated (Prejudiced) regimes. We show that these architectural decisions induce systematic shifts in the initial prediction regime, thereby modulating subsequent learning dynamics. By linking normalization design directly to prediction statistics at initialization, our results offer principled guidance for more controlled and interpretable network design, including clarifying how widely used choices such as BatchNorm vs. LayerNorm and Pre-Norm vs. Post-Norm shape behavior from the outset of training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。