通过降低嵌入变异性，提升语言模型预训练稳定性。
Stable Language Model Pre-training by Reducing Embedding Variability
- 用嵌入变异性衡量预训练稳定性，替代高成本梯度方差计算。
- 提出多头低秩注意力，有效抑制嵌入方差指数增长，防止梯度爆炸。
- 在深层GPT-2上验证，显著提升训练稳定性和更低困惑度。
稳定的预训练对获得高性能语言模型至关重要。然而，每一步计算梯度方差以追踪稳定性因计算成本过高而不切实际。本文探索了分层归一化语言模型中的词元嵌入变异性（TEV）作为评估预训练稳定性的简单高效代理指标，因为浅层更易出现梯度爆炸（第2.2节）。此外，提出多头低秩注意力（MLRA）架构，通过限制输出嵌入方差的指数增长，缓解此类不稳定性，从而防止梯度爆炸（第3.2节）。在GPT-2上的实验结果表明，采用MLRA后模型稳定性提升，困惑度降低，尤其在深层模型中效果显著。
原文摘要 · Abstract (English)
Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。