通过去除上下文噪声提升长序列模型注意力，让关键信息更突出。
Revisiting Long-context Modeling from Context Denoising Perspective
- 用积分梯度法识别并量化上下文中的干扰信息。
- 训练后模型对关键信息的注意力显著增强，性能接近GPT-4o。
- 适合追求长文本理解效率的开发者与研究者使用。
长上下文模型（LCMs）在处理长序列方面展现出巨大潜力，推动了众多实际应用。其成功得益于能从上下文中定位隐含的关键信息以进行后续预测。然而，最新研究表明，LCMs常受无关标记的上下文噪声干扰，导致注意力错位。本文对上下文噪声进行细粒度分析，提出集成梯度（IG）评分来检测和量化噪声。结果表明，仅简单消除检测到的噪声即可显著提升模型对关键标记的注意力，改善预测效果。基于此，我们提出上下文去噪训练（CDT），一种简单但高效的训练策略，可强化关键信息对预测的影响。在四个任务上、不同上下文窗口缩放与长上下文对齐设置下的大量实验验证了其优越性。值得注意的是，采用CDT训练的80亿参数开源模型性能达50.92，接近GPT-4o的51.00。
原文摘要 · Abstract (English)
Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their ability to locate implicit critical information within the context for further prediction. However, recent research reveals that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens, that can mislead model attention. In this paper, we conduct a fine-grained analysis of the context noise and propose an effective metric, the Integrated Gradient (IG) score, to detect and quantify the noise information within the context. Our findings reveal that even simple mitigation of detected context noise can substantially boost the model's attention on critical tokens and benefit subsequent predictions. Building on this insight, we propose Context Denoising Training (CDT), a straightforward yet effective training strategy that improves attention on critical tokens while reinforcing their influence on model predictions. Extensive experiments across four tasks, under both context window scaling and long-context alignment settings, demonstrate the superiority of CDT. Notably, when trained with CDT, an open-source 8B model can achieve performance (50.92) comparable to GPT-4o (51.00).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。