长输入下模型偏好结尾信息,中间内容易被忽略。
Positional Biases Shift as Inputs Approach Context Window Limits
- 以模型上下文窗口为基准分析位置偏差,发现50%时中间遗忘最严重。
- 超过50%后中间效应消失,模型更依赖靠近结尾的信息。
- 适合优化长文本任务设计与评估方法的工程师和研究者。
大语言模型在处理长输入时常难以有效利用信息。已有研究发现位置偏差,如‘中间遗忘’(LiM)现象,即模型对开头或结尾信息表现更好,而对中间内容表现较差。然而,长上下文研究未能一致复现该现象,引发对其强度及作用条件的质疑。为此,我们采用相对于模型上下文窗口的相对输入长度进行综合分析。结果表明,当输入占上下文窗口不超过50%时,LiM效应最强;超过此阈值后,首端偏好减弱,尾端偏好保持稳定,从而消除中间遗忘效应。取而代之的是距离末端越近,模型表现越好,形成距离相关偏差。此外,我们的研究提示,成功检索是推理的前提,位置偏差主要源自检索阶段。这些发现对长上下文任务、未来模型评测基准及评估方法设计具有重要意义。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often struggle to use information across long inputs effectively. Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears at the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest. To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model's context window. Our findings reveal that the LiM effect is strongest when inputs occupy up to 50% of a model's context window. Beyond that, the primacy bias weakens, while recency bias remains relatively stable. This effectively eliminates the LiM effect; instead, we observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input. Furthermore, our results suggest that successful retrieval is a prerequisite for reasoning in LLMs, and that the observed positional biases in reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。