arXiv:2601.20276cs.CLcs.AI2026-01

提出新评测框架,揭示大模型在复杂语义中找证据的真实短板。

Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale

  • 构建326M-token对抗性记忆库,模拟真实检索场景
  • 发现模型在语义干扰下证据定位能力大幅下降
  • 分离评估访问与使用能力,适合诊断长文本模型

长上下文大模型需从大规模环境准确获取并使用证据,但现有针堆测试(NIAH)仅评估简单定位。本文提出基于326M-token MemoryBank的对抗性评测基准EMB-S,包含经碰撞测试的近似错误负样本和跨文档黄金证据集,通过人工与LLM双重验证。采用解耦诊断协议,分别报告证据定位(文档ID)与端到端问答质量。在从64K词元领域隔离到326M词元全局共享的参考语料阶梯中,发现高分良性测试系统在语义干扰下证据获取能力急剧退化。结果表明,语义区分能力而非上下文长度,是大模型长程记忆的核心瓶颈。

原文摘要 · Abstract (English)

Long-context LLM agents must access the right evidence from large environments and use it faithfully. However, the popular Needle-in-a-Haystack (NIAH) evaluation mostly measures benign span localization. The needle is near-unique, and the haystack is largely irrelevant. We introduce EverMemBench-S (EMB-S), an adversarial NIAH-style benchmark built on a 326M-token MemoryBank. While the full MemoryBank spans 326M tokens for retrieval-based (RAG) evaluation, we evaluate native long-context models only at scales that fit within each model's context window (up to 1M tokens in this work) to ensure a fair comparison. EMB-S pairs queries with collision-tested near-miss hard negatives and gold evidence sets spanning one or more documents, validated via human screening and LLM verification. We also propose a decoupled diagnostic protocol that reports evidence access (document-ID localization) separately from end-to-end QA quality under full-context prompting. This enables consistent diagnosis for both native long-context prompting and retrieval pipelines. Across a reference-corpus ladder from domain-isolated 64K contexts to a globally shared 326M-token environment, we observe a clear reality gap. Systems that saturate benign NIAH degrade sharply in evidence access under semantic interference. These results indicate that semantic discrimination, not context length alone, is the dominant bottleneck for long-context memory at scale.

长文本理解检索增强评测基准语义干扰

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。