发现大模型在长文本中位置敏感，中间任务准确率暴跌
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
- 设计新评估框架CRE，同时控制任务位置、填充内容和上下文长度
- 模型在长上下文中，任务从末尾移到中间时准确率最高降88个百分点
- 现有评测未控位置，导致模型真实推理能力被高估，适合评估者与开发者看
主流推理基准未控制长上下文中目标任务的位置，我们审计了11个长上下文基准，发现均未同时控制任务位置、填充内容和上下文长度。对四个旗舰模型的审计显示，其主结果表中无NIAH、RULER或LongBench系列的条目，而代理与编程类基准均有。我们提出Context Rot Evaluation（CRE）框架，评估九个LLM在GSM8K和ARC-Challenge上的表现，分两轮进行：初始五模型组及四款新厂商发布版。当任务从末尾移至中间时，模型准确率显著下降，且该下降随上下文长度加剧；如MiMo-v2-Flash在64K上下文、with_solutions填充下，中间准确率仅8%，比末尾下降88个百分点。新发布模型下降更小：64K时三款在±6pp内；MiMo-V2.5-Pro将原88pp降幅缩小至32pp。question_only_v2填充下，所有四款仍存在明显下降（-16pp至-56pp）。在8K时，若在末尾添加目标任务副本，所有九模型中间准确率与末尾基线差距缩至±4pp，支持位置解释。初始五模型组中，76%的中间错误与周围填充文本相关，远高于末尾的22%，表明填充-答案干扰是主要错误模式。这些结果揭示当前推理评测设计与厂商评估中的结构性缺陷：位置脆弱性无法被测量，因任务位置未受控。
原文摘要 · Abstract (English)
Position-controlled evaluation is standard for retrieval tasks such as Needle-in-a-Haystack and RULER, but mainstream reasoning benchmarks do not control positional placement of target tasks in long contexts. We audit 11 long-context benchmarks and find none jointly controls task position, filler content, and context length for reasoning. An audit of four flagship long-context releases finds no main result-table entry for NIAH, RULER, or LongBench-family benchmarks, while agentic and coding benchmarks appear in main result-tables across all four. We propose Context Rot Evaluation (CRE), a controlled framework varying all three factors, and evaluate nine LLMs on GSM8K and ARC-Challenge across two rounds: an initial five-model set and four newer vendor releases. Models can drop sharply when the target task moves from end to middle, and the drop grows worse with context length for vulnerable models. MiMo-v2-Flash drops 88pp at 64K under with_solutions filler (middle accuracy 8%). Newer releases show smaller drops: at 64K, three of four stay within +/-6pp of end-position accuracy; MiMo-V2.5-Pro narrows the MiMo-v2-Flash 88pp drop to 32pp. Under questions_only_v2 filler, middle-position drops persist across all four (range -16pp to -56pp across 8K, 32K, 64K). At 8K, a diagnostic probe adding a target-task copy at the end brings middle accuracy within +/-4pp of end baseline across all nine models, consistent with a positional explanation. In the initial five-model set, 76% of middle-position errors match surrounding filler text versus 22% at the end position, consistent with filler-answer interference as a dominant error mode. These results expose a structural evaluation gap in current reasoning benchmark design and vendor evaluation practice: positional vulnerabilities that grow with context length cannot be measured when task position is not controlled.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。