输入越长,大模型表现越差,即使信息全对也难逃性能下降。
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval

- 通过提示让模型先复述检索到的信息,将长任务转为短任务
- 即使完美检索,长输入仍导致性能下降13.9%至85%
- 适用于提升数学、问答、编程等长上下文任务的模型表现
大语言模型在长上下文任务中的表现无法随上下文长度增长而同步提升。传统观点归因于检索失败——模型难以从长输入中定位相关信息。然而,本文在数学、问答和编码任务上对5个开源与闭源LLM进行系统实验发现:即使能完美检索所有相关资讯,模型性能仍显著下降(13.9%–85%),且未超出其宣称的上下文长度范围。该现象在用空白字符替代无关内容、或完全屏蔽无关内容并强制模型仅关注相关部分后依然存在。类似下降也出现在将所有证据置于问题前时。结果揭示了一个此前未被认识到的限制:输入长度本身即可损害模型性能,与检索质量无关,也无需干扰。为此提出一种简单、模型无关的缓解策略——通过提示引导模型先复述检索出的内容,再解题。在RULER数据集上,GPT-4o在此策略下性能相比强基线最高提升4%。
原文摘要 · Abstract (English)
Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to retrieval failures -- the models' inability to identify relevant information in the long inputs. Accordingly, recent efforts often focus on evaluating and improving LLMs' retrieval performance: if retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one -- or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%--85%) as input length increases but remains well within the models' claimed lengths. This failure occurs even when the irrelevant tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of retrieval quality and without any distraction. They motivate our simple, model-agnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。