不训练即可提升长文本模型注意力，解决信息衰减问题
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
- 通过对比长程与局部注意力的输出，引导模型聚焦关键信息
- 在多个长文本任务中达到当前最佳性能，且无需额外训练
- 适合需要高效优化长文本能力的研究者和开发者
尽管大语言模型支持长上下文，但在上下文窗口内仍面临性能下降问题。现有解决方案成本高昂，统计特性与低成本方法研究不足。本文从解码视角识别出后验显著性衰减（PSA）现象：显著性比例与长文本性能下降相关。值得注意的是，尽管存在衰减，真实词元在解码空间中仍处于高排名位置。基于此，提出无需训练的定位对比解码（PCD）方法，通过对比长程感知注意力与设计的局部感知注意力的输出，使模型关注大规模短到长训练带来的收益。长期衰减模拟分析表明，PCD能有效缓解注意力得分退化。实验结果证实，该方法在多个长上下文基准上达到领先水平。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) support long contexts, they struggle with performance degradation within the context window. Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. From the decoding perspective, we identify the Posterior Salience Attenuation (PSA) phenomenon, where the salience ratio correlates with long-text performance degradation. Notably, despite the attenuation, gold tokens still occupy high-ranking positions in the decoding space. Motivated by it, we propose the training-free Positional Contrastive Decoding (PCD) that contrasts the logits derived from long-aware attention with those from designed local-aware attention, enabling the model to focus on the gains introduced by large-scale short-to-long training. Through the analysis of long-term decay simulation, we demonstrate that PCD effectively alleviates attention score degradation. Experimental results show that PCD achieves state-of-the-art performance on long-context benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。