发现长文本推理中干扰信息的非线性影响,少量干扰即大幅降低模型表现。
The First Drop of Ink: Nonlinear Impact of Distracting Information in Long-Context Reasoning

- 通过控制干扰内容比例,发现性能随干扰量增加呈非线性骤降。
- 仅少量硬干扰(<10%)就导致性能显著下滑,后续增量影响微弱。
- 适合关注检索增强系统与长文本推理鲁棒性的研究者阅读。
随着大语言模型在检索增强生成和代理系统中处理越来越长的上下文,理解干扰信息对长上下文性能的影响变得至关重要。以往研究显示语义相关但具有误导性的文档会降低性能,但干扰比例与性能之间的定量关系尚未明确。本文在固定长度上下文中系统调整硬干扰的比例,发现一个显著的非线性模式:当硬干扰比例上升时,性能在极小比例范围内急剧下降,而后续比例变化带来的额外损失微乎其微。我们称此为‘第一滴墨’效应,类似于一滴墨水即可污染整杯水。理论与实证分析基于注意力机制表明,即使比例很小,硬干扰也会捕获不成比例的注意力,且边际影响随比例增加而递减。受控实验进一步表明,过滤收益主要来自上下文长度缩减,而非去除干扰本身;要实现显著性能恢复,必须将硬干扰比例降至接近零,凸显上游检索精度的重要性。
原文摘要 · Abstract (English)
As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical. Prior work shows that semantically relevant yet misleading documents degrade performance, but the quantitative relationship between the proportion of distractors and performance remains unstudied. In this work, we systematically vary the hard-distractor proportion in fixed-length contexts, revealing a striking nonlinear pattern: as the proportion of hard distractors increases, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. We term this ''The First Drop of Ink'' effect, analogous to how a single drop of ink contaminates water. Our theoretical and empirical analyses grounded in attention mechanics show that hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Controlled experiments further show that filtering gains mainly come from context-length reduction rather than distractor removal; substantial recovery requires reducing the hard-distractor proportion to near zero, highlighting the importance of upstream retrieval precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。