测试AI在企业文档中跨多源推理的能力，发现主流模型表现极差。
OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
- 构建包含8.9万页文档的金融政策库，支持文本与表格混合推理。
- 顶尖模型仅34.1%正确率，依赖参数知识时低于5%。
- 结构化文档解析可提升16.1%性能，适合企业级AI评估使用。
我们提出OfficeQA Pro，一个用于评估AI代理在大型异构文档语料上进行基础化、多文档推理的基准。该语料涵盖近百年美国财政部公告，共89,000页，包含超过2600万数值。OfficeQA Pro包含133个问题，要求精确的文档解析、检索与跨非结构化文本和表格数据的分析推理。前沿大模型（如Claude Opus 4.6、GPT-5.4、Gemini 3.1 Pro Preview）仅依赖参数知识时准确率不足5%，接入网络后仍低于12%。当直接提供文档语料时，这些代理平均得分仅为34.1%。我们发现，使用Databricks的ai_parse_document生成的结构化文档表示，可使各代理平均相对性能提升16.1%。通过消融实验分析了模型选择、表格表示、检索策略及推理时扩展对性能的影响。尽管有改进，当前模型在企业级精准推理任务中仍有巨大提升空间。
原文摘要 · Abstract (English)
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。