发现注意力输出方差随序列变长而下降，提出归一化改进泛化能力。
On Vanishing Variance in Transformer Length Generalization
- 在注意力模块后加层归一化，缓解长序列方差消失问题。
- 在词典查找和最大值检索任务中，长序列泛化性能显著提升。
- 适合关注模型推理稳定性与长度外推的研究者。
Transformer 模型在较短序列上训练后，测试时面对更长序列常出现泛化能力下降的问题，引发其是否具备真正推理能力的质疑。本文首次揭示，即使是前沿模型，序列长度增加也会导致多头注意力模块输出方差减小。在最大值检索和字典查找任务中，实验表明在注意力输出后添加层归一化可显著改善长度泛化表现。分析表明，该改进源于对由方差消失引发的分布偏移的缓解（虽未完全消除）。
原文摘要 · Abstract (English)
It is a widely known issue that Transformers, when trained on shorter sequences, fail to generalize robustly to longer ones at test time. This raises the question of whether Transformer models are real reasoning engines, despite their impressive abilities in mathematical problem solving and code synthesis. In this paper, we offer a vanishing variance perspective on this issue. To the best of our knowledge, we are the first to demonstrate that even for today's frontier models, a longer sequence length results in a decrease in variance in the output of the multi-head attention modules. On the argmax retrieval and dictionary lookup tasks, our experiments show that applying layer normalization after the attention outputs leads to significantly better length generalization. Our analyses attribute this improvement to a reduction-though not a complete elimination-of the distribution shift caused by vanishing variance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。