测试大模型能否完整理解长文本上下文，发现顶级模型也难做到。
NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models
- 设计新基准NeedleChain，仅保留与问题相关的内容，逼模型真正整合信息。
- 即使200词的纯相关上下文，GPT-4o也难以可靠整合全部信息。
- 提出无需训练的ROPE收缩策略，推动模型更完整利用上下文。
近期报告称大语言模型可处理更长上下文，但现有评估基准常混入大量无关内容，导致评测偏向于提取片段而非全面整合信息。这可能高估了模型的真实上下文理解能力。我们发现，当上下文全为与查询相关的内容时，即使是GPT-4o等先进模型也无法可靠整合长度仅为200个标记（tokens）的信息。为此，我们提出NeedleChain基准，包含三种不同顺序理解要求的变体，并引入基于针在草堆中（NIAH）的平行基准。通过对比分析，该基准能更全面评估模型对上下文的理解能力。我们还提出一种无需训练的策略——ROPE收缩，强调完整上下文整合的重要性，并指向提升模型可靠推理的新方向。
原文摘要 · Abstract (English)
Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether models can faithfully incorporate all given evidence. NeedleChain includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-in-a-haystack(NIAH) paradigm. By comparing these variants, NeedleChain enables a more comprehensive assessment of context understanding. We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。