arXiv:2606.29718cs.IRcs.AI2026-06被引 2

发现长上下文搜索中模型会过早放弃,提出有效缓解方法。

Diagnosing and Mitigating Context Rot in Long-horizon Search

论文配图:Diagnosing and Mitigating Context Rot in Long-horizon Search
图 1 · 摘自论文原文
  • 识别出模型在长上下文下过早终止回答的新现象
  • 验证上下文越长,过早终止率越高,影响搜索效果
  • 提出基于行为感知的并行采样策略,提升性能2.6%~4.9%

随着大语言模型在长时序搜索任务中的广泛应用,长上下文已成为常态。然而,上下文过长导致模型能力下降的现象(即上下文衰减)已成普遍关注问题。但在深度搜索场景中,模型如何因长上下文失败、现有方法能否有效缓解仍不明确。通过在三个基准上对四款主流模型的系统研究,我们发现一种此前被忽视的现象:提前终止——在长上下文条件下,模型在未耗尽上下文窗口前便放弃回答或给出不确定的错误答案。控制查询难度后,我们发现提前终止率与上下文长度呈正相关。基于此,我们重新审视缓解上下文衰减的方法,包括上下文管理与并行采样。对于上下文管理,分析了三类共七种方法,发现其本质为测试时缩放策略,能降低提前终止率以促进探索,并提出模型依赖的选择原则。对于并行采样,设计了一种行为感知过滤策略,在三种聚合方式下均带来2.6%至4.9%的性能提升。

原文摘要 · Abstract (English)

Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon search tasks. The concern that increasing context length degrades model capabilities, known as context rot, has become a widely recognized issue for these applications. However, in deep search scenarios, it remains unclear how models actually fail under extensive context, and to what extent existing methods can mitigate such failures. Through a systematic study of four flagship models across three benchmarks, we identify a previously overlooked phenomenon, which we term premature termination: under extensive context, models give up or provide uncertain incorrect answers long before exhausting the context window. By controlling for query difficulty, we show that the premature termination rate is positively correlated with context length. Based on the findings, we revisit methods to mitigate context rot, including context management and parallel sampling. For context management, we analyze seven methods across three categories and show that they are inherently test-time scaling strategies that reduce the premature termination rate to enable more exploration, and we further provide model-dependent principles for method selection. For parallel sampling, we develop a behavior-aware filtering strategy and observe a performance gain of 2.6% to 4.9% across three aggregation methods.

长上下文上下文衰减搜索优化LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。