构建1000万词元基准,测试模型对文档库的全局推理能力。
CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
- 用程序生成复杂查询与真值答案,避免人工标注误差。
- 1000万词元规模下,顶尖模型推理能力显著下降。
- 适合研究长文本全局推理、智能体架构的学者使用。
尽管大语言模型已能处理百万词元上下文,但其对文档库的全局推理能力仍缺乏有效评估。现有基准多限于单篇长文本或依赖‘稀疏检索’假设——即答案可从少数相关片段得出。这一假设在真实文档库分析中失效,因证据高度分散于数百文档,需全局整合、比较与统计聚合。为此,我们提出CorpusQA,一个可达1000万词元的新基准,基于创新的数据合成框架生成。该框架将推理与文本表示解耦,创建计算密集型查询并程序化保证真值答案,迫使系统在无结构文本中进行整体推理,不依赖不可靠的人工标注。实验表明,即使最先进的长上下文模型在输入长度增加时也表现不佳,标准检索增强生成系统完全崩溃。结果表明,记忆增强型智能体架构更具鲁棒性,提示需从扩展上下文窗口转向开发先进架构以实现全局信息融合。
原文摘要 · Abstract (English)
While large language models now handle million-token contexts, their capacity for reasoning across entire document repositories remains largely untested. Existing benchmarks are inadequate, as they are mostly limited to single long texts or rely on a "sparse retrieval" assumption-that answers can be derived from a few relevant chunks. This assumption fails for true corpus-level analysis, where evidence is highly dispersed across hundreds of documents and answers require global integration, comparison, and statistical aggregation. To address this critical gap, we introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework. By decoupling reasoning from textual representation, this framework creates complex, computation-intensive queries with programmatically guaranteed ground-truth answers, challenging systems to perform holistic reasoning over vast, unstructured text without relying on fallible human annotation. We further demonstrate the utility of our framework beyond evaluation, showing that fine-tuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities. Extensive experiments reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard retrieval-augmented generation systems collapse entirely. Our findings indicate that memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。