通过调整初始词元权重，提升长文本处理能力
Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processing
- 通过放大初始词元与其他词元的注意力权重差，缓解中间信息丢失
- 在MDQA数据集上提升3.6%，在KV-Retrieval任务上提升3.4%
- 适合关注长文本建模与注意力机制优化的研究者
大型语言模型在多种自然语言处理任务中表现优异，但在处理长文本序列时常因‘中间遗忘’现象而失效。已有研究指出该问题源于U形注意力偏差，即模型过度关注文本首尾，忽略中间内容。本文首次揭示另一个关键因素：初始显著性。在注意力计算中，相对于初始词元具有更高注意力权重的词元，在预测下一个词元时会获得更强关注。通过放大初始词元与其他词元间的注意力权重差异，可有效增强模型对长上下文的处理能力，在MDQA数据集上实现最高3.6%的性能提升。进一步结合现有减少位置编码偏差的方法，在KV-Retrieval任务中达到最大3.4%的提升。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong performance on a variety of natural language processing (NLP) tasks. However, they often struggle with long-text sequences due to the ``lost in the middle'' phenomenon. This issue has been shown to arise from a U-shaped attention bias, where attention is disproportionately focused on the beginning and end of a text, leaving the middle section underrepresented. While previous studies have attributed this bias to position encoding, our research first identifies an additional factor: initial saliency. It means that in the attention computation for each token, tokens with higher attention weights relative to the initial token tend to receive more attention in the prediction of the next token. We further find that utilizing this property by scaling attention weight between the initial token and others improves the model's ability to process long contexts, achieving a maximum improvement of 3.6\% in MDQA dataset. Moreover, combining this approach with existing methods to reduce position encoding bias further enhances performance, achieving a maximum improvement of 3.4\% in KV-Retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。