新基准PureDocBench揭示文档解析仍远未解决，模型表现差距大且泛化能力存疑。
How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings
- 用代码生成图文和标注，确保来源可追溯，覆盖10个领域4425张图
- 顶尖模型仅74分，公式识别普遍低于67%，专业模型比通用模型更抗退化
- 适合关注文档解析真实性能、模型鲁棒性与公平评估的研究者
过去一年出现超过20个开源文档解析模型，但领域仍主要依赖1355页人工标注的OmniDocBench，其最高得分已饱和在90%以上。我们对OmniDocBench进行三阶段审计，筛查21,353个评分块后确认2,580处错误（占比12.08%），结合超一年公开期，其标注质量与污染风险令人质疑。为此，我们提出PureDocBench，一个程序生成、来源可追溯的基准，通过HTML/CSS渲染文档图像，并从同一源生成可验证标注，涵盖10个领域、66个子类别、1,475页，每页含清晰、数字退化、真实退化三种版本，共4,425张图像。评估40个模型（含流水线专家、端到端专家及通用视觉语言模型），结果表明：(i) 文档解析远未解决：最佳模型仅得74分，强弱模型间差距达44.6分；(ii) ≤4B参数的专用解析器可媲美或超越5-100倍更大的通用视觉语言模型，但公式识别仍是共同瓶颈，所有赛道平均公式指标均未超67%；(iii) 通用视觉语言模型在数字/真实退化下仅损失0.99/8.52分，而流水线专家损失4.90/14.21分，导致排名反转，使仅在清晰环境下评估误导部署决策。所有数据、代码与成果均已公开。
原文摘要 · Abstract (English)
The past year has seen over 20 open-source document parsing models, yet thefield still benchmarks almost exclusively on OmniDocBench, a 1,355-pagemanually annotated dataset whose top scores have saturated above 90%. Athree-stage audit pipeline we run on OmniDocBench screens its 21,353evaluator-scored blocks and confirms 2,580 errors (12.08%); combined with overa year of public availability, both annotation quality and contamination riskcall its rankings into question. To address these issues, we presentPureDocBench, a programmatically generated, source-traceable benchmark thatrenders document images from HTML/CSS and produces verifiable annotations fromthe same source, covering 10 domains, 66 subcategories, and 1,475 pages, eachin three versions: clean, digitally degraded, and real-degraded (4,425 imagestotal). Evaluating 40 models spanning pipeline specialists, end-to-endspecialists, and general-purpose VLMs, we find: (i) document parsing is farfrom solved: the best model scores only ~74 out of 100, with a 44.6-point gapbetween the strongest and weakest models; (ii) specialist parsers with <=4Bparameters rival or surpass general VLMs that are 5-100x larger, yet formularecognition remains a shared bottleneck where no model exceeds 67% whenaveraging the formula metric across all three tracks; (iii) general VLMs loseonly 0.99/8.52 Overall points under digital/real degradation versus 4.90/14.21for pipeline specialists, producing ranking reversals that make clean-onlyevaluation misleading for deployment. All data, code, and artifacts arepublicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。