构建超大规模文档分析问答基准,推动多文档推理能力发展
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA

- 基于文档元数据与金融数据库远监督构建8万页数据集
- 标准RAG系统在复杂分析任务上表现差,准确率不足50%
- 提出多智能体流程提升推理质量,适合需要深度分析的研究者
本文提出面向大规模半结构化文档集合的分析型问答任务。我们构建了MuDABench基准,要求模型跨数百篇文档提取并整合信息以完成定量分析。不同于仅需少量文档、浅层推理的现有基准,MuDABench强调广泛的跨文档分析与聚合。该数据集通过远监督构建,包含超过8万页文档和332个分析型问答实例。我们还提出评估协议,既衡量最终答案准确率,也用中间事实覆盖率作为推理过程的辅助诊断信号。实验表明,将所有文档视为扁平检索池的标准RAG系统表现不佳。为此,我们设计了一个包含规划、提取和代码生成模块的多智能体工作流,显著提升了过程与结果指标,但与人类专家仍存在明显差距。分析发现两大瓶颈:单文档信息抽取准确率低,以及当前系统缺乏领域知识。MuDABench已开源。
原文摘要 · Abstract (English)
This paper introduces the task of analytical question answering over large, semi-structured document collections. We present MuDABench, a benchmark for multi-document analytical QA, where questions require extracting and synthesizing information across numerous documents to perform quantitative analysis. Unlike existing multi-document QA benchmarks that typically require information from only a few documents with limited cross-document reasoning, MuDABench demands extensive inter-document analysis and aggregation. Constructed via distant supervision by leveraging document-level metadata and annotated financial databases, MuDABench comprises over 80,000 pages and 332 analytical QA instances. We also propose an evaluation protocol that measures final answer accuracy and uses intermediate-fact coverage as an auxiliary diagnostic signal for the reasoning process. Experiments reveal that standard RAG systems, which treat all documents as a flat retrieval pool, perform poorly. To address these limitations, we propose a multi-agent workflow that orchestrates planning, extraction, and code generation modules. While this approach substantially improves both process and outcome metrics, a significant gap remains compared to human expert performance. Our analysis identifies two primary bottlenecks: single-document information extraction accuracy and insufficient domain-specific knowledge in current systems. MuDABench is available at https://github.com/Zhanli-Li/MuDABench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。