ALiBi位置编码在长序列中会因精度溢出导致注意力失效，影响关键信息检索。
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
- 发现ALiBi编码因线性偏置缩放过小，使大量注意力权重归零
- 长序列下关键词检索性能显著下降，但常规任务影响小
- 推荐使用对数距离缩放作为训练时的优化策略
我们发现ALiBi位置编码存在此前未被注意的失效模式：其线性偏置缩放在浮点精度下发生下溢，导致大量注意力权重变为零，使部分注意力头近乎失明。通过分析该失效模式及其影响，我们评估了四种缓解策略。在基于ALiBi的148M参数解码器模型上进行全面预训练实验，成功将该问题与上下文外退化现象分离。结果表明，该失效模式会显著损害关键词检索能力，但对标准解码器基准测试影响较小。我们提出四种训练阶段缓解方法，单独或组合评估后发现，对数距离缩放在通过键检索任务中表现最稳定。尽管存在此问题，默认的ALiBi斜率仍为针中藏草类检索任务提供意外强基线。据此，我们给出使用ALiBi训练模型的具体建议。
原文摘要 · Abstract (English)
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。