分析大模型推理的数值稳定性，揭示权重与残差流的交互影响
Numerical stability analysis of large language models
- 采用混合精度分析法，推导Transformer各组件的误差界
- 发现残差投影权重缩放可保持相对舍入误差传播
- 适用于关注模型数值稳定性的系统设计与训练优化者
Transformer是当前大语言模型的主流架构，其可扩展性依赖于低精度计算的策略性使用。本文对Transformer推理进行混合精度分析，推导出其组成部分的条件数和前向误差界。特别地，我们在大规模异常值情形下比较了LayerNorm与RMSNorm的数值稳定性，收紧了存在注意力汇聚时softmax的误差界，并量化了其偏移评估对扰动敏感性的影响。此外，我们首次得到自注意力局部Lipschitz常数的序列长度无关界。最坏情况下的推理误差界表明，数值稳定性取决于权重幅值与残差流增长的相互作用。关键的是，实验验证（基于GPT-2）显示，残差投影权重的缩放若不引发残差流动力学的定性转变，则能保持相对舍入误差的传播。
原文摘要 · Abstract (English)
Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。