首个综合评估文档抽取性能的基准,覆盖真实企业场景
ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

- 构建跨370份文档、67类文件的多领域评估集,支持结构化提取
- 首次统一评测值准确率、记录完整性和溯源性,发现大模型易截断长列表
- 开源数据与代码,适合研究文档智能抽取的开发者使用
企业工作流越来越多依赖代理进行模式引导的文档抽取:给定文档和用户定义的模式,代理需严格遵循模式生成正确输出,并附上源证据作为可追溯性依据。我们提出ExtractBench,一个针对模式引导抽取的基准测试,据我们所知是首个同时评估值准确率、记录完整性、溯源性及成本的系统。评估体系包含4,869页、370份企业文档、8个业务领域和67种文档类型,明确标注不同挑战场景。可扩展的模式与真值构建流程结合了真实文档的独立系统一致性、合成列表的已知值,以及表单的人工验证。报告了不考虑顺序的值准确率F1,以及两个溯源性指标:字级和页级F1。商业视觉语言模型在短文档上表现良好,但在长文档中常截断记录列表;编码代理虽准确率高但成本显著。LlamaExtract Agentic Plus在所有三项指标中排名第一,准确性接近编码代理,成本仅为一小部分。数据集与评估代码已发布于HuggingFace与GitHub。
原文摘要 · Abstract (English)
Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。