arXiv:2505.18148cs.CLcs.AI2025-05被引 9

更短的正确文本反而让大模型更难找对答案,影响推理可靠性。

Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find

  • 系统研究不同长度正确文本对模型找答案的影响。
  • 文本越短,模型准确率越降,位置敏感性越强。
  • 适合开发智能代理和长文本推理系统的参考。

大语言模型在处理'针在草堆中'类任务时面临挑战,需从大量无关上下文(草堆)中找出关键信息(针)。已有研究关注位置偏差和干扰项数量,但对答案所在文档长度(黄金上下文大小)的影响关注不足。本文首次系统研究了长上下文问答中黄金上下文大小的影响,覆盖三个基准(通用知识、生物医学推理、数学推理)、11个前沿大模型及超过15万次受控实验。结果表明:当黄金上下文较短时,模型性能急剧下降,且位置敏感性显著增强,这对需要整合分散细粒度信息的智能体系统构成重大挑战。该效应在严格控制位置、重复词、黄金/干扰比、干扰量和领域特异性后仍成立,说明黄金上下文大小是独立且决定性的预测因子。本研究为设计鲁棒的上下文感知式大模型系统提供了明确指导。

原文摘要 · Abstract (English)

Large language models (LLMs) face significant challenges with needle-in-ahaystack tasks, where relevant information ("the needle") must be drawn from a large pool of irrelevant context ("the haystack"). Previous studies have highlighted positional bias and distractor quantity as critical factors affecting model performance, yet the influence of gold context size, the length of the answer-containing document, has received little attention. We present the first systematic study of gold context size in long-context question answering, spanning three diverse benchmarks (general knowledge, biomedical reasoning, and mathematical reasoning), eleven state-of-the-art LLMs (including recent reasoning models), and more than 150K controlled runs. Our experiments reveal that LLM performance drops sharply when the gold context is shorter, i.e., smaller gold contexts consistently degrade model performance and amplify positional sensitivity, posing a major challenge for agentic systems that must integrate scattered, fine-grained information of varying lengths. This effect persists under rigorous confounder analysis: even after controlling for gold context position, answer token repetition, gold-to-distractor ratio, distractor volume, and domain specificity, gold context size remains a decisive, independent predictor of success. Our work provides clear insights to guide the design of robust, context-aware LLM-driven systems.

长文本推理模型评估信息检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。