通过权重重缩放提升大模型预训练的稳定性与性能
Variance Control via Weight Rescaling in LLM Pre-training
- 提出层索引重缩放（LIR）初始化与目标方差重缩放（TVR）策略
- 在10亿参数LLaMA模型上提升下游任务性能最高达4.6%
- 有效抑制激活值极端波动，适合低精度训练与量化场景
大语言模型（LLM）预训练的效果高度依赖权重初始化和方差控制策略。尽管神经网络中初始方差控制的重要性已被广泛证实，但针对大模型预训练过程中方差演变的管理研究仍显不足。本文提出层索引重缩放（LIR）权重初始化方案和目标方差重缩放（TVR）方差控制策略。在10亿参数的LLaMA模型上的实验表明，采用这些技术可显著改善下游任务性能，最高提升4.6%（在常见预训练基准上），同时降低极端激活值，缓解量化与低精度训练中的挑战。代码已开源：https://github.com/bluorion-com/weight_rescaling。
原文摘要 · Abstract (English)
The outcome of Large Language Model (LLM) pre-training strongly depends on weight initialization and variance control strategies. Although the importance of initial variance control has been well documented in neural networks in general, the literature on initialization and management of its growth during LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce the Layer Index Rescaling (LIR) weight initialization scheme, and the Target Variance Rescaling (TVR) variance control strategy. Experiments on a 1B parameter LLaMA model demonstrate that better variance management using these techniques yields substantial improvements in downstream task performance (up to 4.6% on common pre-training benchmarks) and reduces extreme activation values, thus mitigating challenges associated with quantization and low-precision training. Our code is available at: https://github.com/bluorion-com/weight_rescaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。