提出可定位并缓解Transformer数值不稳定的层间分析方法
Numerical Fragility in Transformers: A Layer-wise Theory for Risk Estimation and Selective Stabilization
- 分解输出误差为注意力、归一化和残差传输三部分，实现逐层风险评估
- 在GPT-2上18次实验中预测与实际误差高度一致，改进率超94%
- 适合关注模型低精度部署安全性的研究人员与工程师
低精度计算会导致Transformer即使在固定权重和输入下仍产生显著前向差异，但现有方法仅在输出端监测，缺乏逐层理论解释。本文首次将输出偏差分解为局部注意力、LayerNorm和残差传输三项，并由此推导出因果风险估计器与预算控制策略——有界引导选择性稳定（BGSS）。控制实验验证了预测的局部符号、单调性和传输结构。在GPT-2上，18次运行中联合预测与FP32参考误差正相关，优于无传输项的消融版本（17/18次）。参考图谱归因显示该评分保留了有用层序信息（均值斯皮尔曼相关0.362）。在匹配预算的缓解中，BGSS在起始事件（10.67 vs 11.67）、最终误差（1.243×10⁻³ vs 1.284×10⁻³）及最坏情况误差（3.14×10⁻³ vs 8.49×10⁻³）上均优于随机控制，且在起始抑制上持平风险优先控制器，同时大幅降低最坏情况误差（3.14×10⁻³ vs 5.71×10⁻³）。结果支持从理论到算法的完整框架，实现对Transformer数值脆弱性的分析、估计、定位与选择性稳定。
原文摘要 · Abstract (English)
Low-precision execution can induce substantial forward discrepancies in Transformers even for fixed weights and input, yet these discrepancies are usually monitored only at the output and lack a layer-wise theoretical account. We develop a first-order decomposition of output mismatch into layer-local attention, LayerNorm, and residual-transport terms, and derive from it a practical causal risk estimator and a budgeted controller, Bound-Guided Selective Stabilization (BGSS). Controlled sweeps verify the predicted local sign, monotonicity, and transport structure. On GPT-2, the transport-aware combined predictor is positively correlated with FP32-reference mismatch in all $18$ runs and improves over a no-transport ablation in $17/18$ runs. Reference-patch attribution shows that the same score preserves useful layer ordering information (mean Spearman $0.362$). In budget-matched mitigation, BGSS outperforms random same-budget control in onset events ($10.67$ vs. $11.67$), final mismatch ($1.243\times 10^{-3}$ vs. $1.284\times 10^{-3}$), and worst-case mismatch ($3.14\times 10^{-3}$ vs. $8.49\times 10^{-3}$), while matching a risk-only same-budget controller on onset suppression and sharply reducing worst-case mismatch ($3.14\times 10^{-3}$ vs. $5.71\times 10^{-3}$). These results support a theory-to-algorithm account of Transformer numerical fragility in which finite-precision risk can be analyzed, estimated, localized, and selectively stabilized.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。