揭示深度模型权重初始化对训练稳定性的影响规律
Weight Initialization and Variance Dynamics in Deep Neural Networks and Large Language Models
- 通过系统测试初始标准差，发现1e-2到1e-1是稳定训练的宽区间
- 验证了Kaiming初始化在ReLU网络中比Xavier收敛更快更稳定
- 在GPT-2类模型中观测到深层权重方差随深度渐进均衡
权重初始化决定了训练初期信号传播与梯度流动。本文在紧凑型ReLU多层感知机和类GPT-2的Transformer两种架构上开展理论指导且实证验证的研究。首先，通过初始标准差的对数扫描，识别出从消失到爆炸的过渡区域，并发现标准差在1e-2至1e-1之间存在一个宽广的稳定带。其次，受控对比显示，对于ReLU激活函数，Kaiming（fan-in）初始化相比Xavier初始化具有更快的收敛速度和更高的稳定性，符合方差保持理论。第三，在从零开始训练的12层类GPT-2模型中，跟踪预训练过程中各层Q/K/V权重的方差变化，观察到浅层权重方差快速扩展，而深层则缓慢且趋于平衡。这些结果将经典初始化原则与现代Transformer行为联系起来，为鲁棒训练提供简单实用的指导方案。
原文摘要 · Abstract (English)
Weight initialization governs signal propagation and gradient flow at the start of training. This paper offers a theory-grounded and empirically validated study across two regimes: compact ReLU multilayer perceptrons and GPT-2-style transformers. First, a logarithmic sweep of the initial standard deviation maps vanishing and exploding regimes and identifies a broad stability band with standard deviations between 1e-2 and 1e-1. Second, a controlled comparison shows that Kaiming (fan-in) initialization converges faster and more stably than Xavier under ReLU, consistent with variance-preserving theory. Third, in a from-scratch 12-layer GPT-2-style model, this paper tracks layerwise Q/K/V weight variance through pretraining and observe depth-dependent equilibration into narrow bands: shallow layers expand rapidly while deeper layers change more gradually. Together, these results connect classic initialization principles with modern transformer behavior and yield simple, practical recipes for robust training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。