arXiv:2608.03297cs.AIcs.CL2026-08

发现传统截断方法会误删关键信息,导致模型性能假性下降。

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

论文配图:Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
图 1 · 摘自论文原文
  • 提出基于任务相关性的智能截断策略,只丢弃无关内容。
  • 在25%上下文保留率下,关键信息保留率从不足1%提升至100%。
  • 该方法可让大模型性能稳定,避免因截断方式误导研究结论。

长上下文语言模型评测中,一个普遍观点认为:只要关键信息保留,上下文越短越好。我们通过在两个长上下文基准(BABILong 和 GraphWalks)上测试四种上下文保留比例(100%、75%、50%、25%),对比两种截断策略:一种是隐含于多数前期工作的朴素截断(从中间删除内容),另一种是干扰项感知的截断(仅移除任务无关内容)。评估了三个尺寸的 Claude 模型(Haiku 4.5, Sonnet 4.6, Opus 4.7)及另一提供商的 GPT-5.5。在朴素截断下,性能单调下降(所有八个样本组别中配对 Wilcoxon 检验,校正后 p_adj < 0.05);而在干扰项感知截断下,性能保持或提升:较小的 Claude 模型在 BABILong 上显著增益,较大的模型(Opus 4.7 与 GPT-5.5)达到全上下文上限。该现象在不同模型间复现,排除单一厂商偏差。机制明确:朴素截断下,答案内容在 25% 保留率时仅在不到 1% 的样本中幸存;而新策略通过构造确保其完整保留。因此,朴素截断并非测量上下文窗口效应,而是偶然是否避开关键信息的结果。未来研究必须明确定义信号与干扰项的区分方式,否则结论将混淆于两种相反假设。

原文摘要 · Abstract (English)

A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved. We test this claim by running every sample of two long-context benchmarks -- BABILong and GraphWalks (BFS) -- at four context-retention fractions (100%, 75%, 50%, 25%) under two truncation protocols. The first is the naive protocol implicitly used in much prior work: drop content from the middle of the prompt. The second is distractor-aware: identify the task-relevant content for each sample and drop only the rest. We evaluate three sizes of the Claude family (Haiku 4.5, Sonnet 4.6, Opus 4.7) and, to test cross-provider generality, GPT-5.5 from a different provider; we apply the same protocol to two further benchmarks (MRCR v2, Oolong). Under naive truncation, score collapses monotonically (paired Wilcoxon, Holm-corrected p_adj < 0.05 in all eight BABILong and GraphWalks cells). Under the distractor-aware protocol -- which preserves the signal by construction -- performance is preserved or improves: the two smaller Claude models show statistically significant gains on BABILong, while the larger models (Opus 4.7 and GPT-5.5) sit at their full-context ceiling. The naive collapse and its distractor-aware recovery replicate on GPT-5.5, ruling out a single-provider artifact. The mechanism is direct: under the naive protocol the answer-bearing content survives in fewer than 1% of samples at 25% retention; under the distractor-aware protocol it is preserved by construction. The naive protocol is therefore not a measurement of context-window effects; it is a measurement of how often middle-removal happens to spare the answer. We conclude that future studies of context-length effects must specify how they distinguish signal from distractor, or they are at best ambiguous between two opposite hypotheses.

长文本理解模型评测上下文截断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。