arXiv:2606.07401cs.CV2026-06

构建真实合规文档的字段级问答与版式理解基准,精准评估解析系统实际表现。

RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents

  • 基于真实业务文档设计双轨评测,涵盖字段问答与版式标注任务
  • 1356个字段问题覆盖4大领域,严格评估每字段及整题准确率
  • 揭示商业系统性能差异大、医疗文档难处理、成本延迟权衡明显

文档解析系统正广泛应用于抵押贷款、财务报告、供应链物流和临床记录等高风险监管流程。然而现有公开基准多基于整洁学术布局或合成文本,仅报告单一OCR或markdown相似度分数,与下游代理实际需求(即在杂乱真实页面中提取特定字段正确值)关联性差。本文提出RealDocBench,一个由真实受监管文档构建的双轨基准。问答赛道包含581份文档上的1,356个字段级问题,每个问题配有类型化的黄金答案字典,系统按字段级和严格题级准确率评分;版式赛道包含1,500张经人工验证的页面图像,采用九类公开分类体系进行COCO风格边界框标注,使用匈牙利匹配器结合邻近感知的拆分/合并恢复机制评分。我们对18种系统(包括商业解析API、通用视觉语言模型和开源OCR模型)在统一提取与评分协议下进行评估,报告准确率、每页成本与缓存失效延迟。结果揭示单数字基准掩盖的显著性能差异、持续困难的医疗子领域,以及不同运行点间的尖锐成本/延迟权衡。数据集、解析适配器与评估工具已发布,支持可复现的字段级文档解析系统对比。

原文摘要 · Abstract (English)

Document parsing systems are increasingly deployed in high-stakes, regulated workflows such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records. Yet most public benchmarks evaluate parsers on clean academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score. Such documents and metrics correlate poorly with what downstream agents actually need: the correct value for a specific field on a messy real-world page. We introduce RealDocBench, a two-track benchmark built from real regulated documents. The QA track contains 1,356 field-level questions over 581 documents spanning four domains, where each question is paired with a typed gold_dict of key-to-value answers and parsers are scored on both per-field and strict per-question accuracy. The layout track contains 1,500 human-verified page images annotated with COCO-style bounding boxes under a nine-class public taxonomy, scored with a Hungarian matcher that includes adjacency-aware split/merge recovery. We evaluate eighteen systems, spanning commercial parsing APIs, general-purpose VLMs, and open-source OCR models, under a uniform extraction-and-scoring protocol, and report accuracy alongside per-page cost and cache-busted latency. RealDocBench exposes a wide performance spread that single-number benchmarks hide, a persistently hard medical sub-domain, and sharp cost/latency trade-offs across operating points. We release the datasets, parser adapters, and evaluation harness to support reproducible, field-level comparison of document parsing systems.

文档解析字段问答版式理解基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。