用大模型评估表格提取效果，比传统方法更接近人工判断。
Beyond String Matching: Semantic Evaluation of PDF Table Extraction
- 用大模型当裁判，评估表格内容的语义相似性。
- 大模型评分与人工判断相关性达0.93，远超传统方法的0.68~0.70。
- 提供可复现的评测框架，适合研究者和开发者选型参考。
从PDF中可靠提取表格对大规模科学数据挖掘和知识库构建至关重要，但现有评估方法依赖规则化指标，无法捕捉表格内容的语义等价性。本文提出基于合成PDF的基准测试框架，使用arXiv来源的表格确保真实复杂度与多样性，并以精确的LaTeX作为真实标签。核心方法是引入大模型作为评判者（LLM-as-a-judge），集成于匹配流程中，处理解析输出不一致问题。通过超过1500次的人工质量判断验证，大模型评估与人工判断的相关性达皮尔逊系数0.93，显著高于基于树编辑距离（TEDS, r=0.68）和网格表格相似度（GriTS, r=0.70）的方法。在包含451张表格的100份合成文档上评估21个主流PDF解析器，发现性能差异显著。结果为表格提取工具选型提供实用指导，并建立可复现、可扩展的评估方法。代码与数据：https://github.com/phorn1/pdf-parse-bench；指标研究与人工评估：https://github.com/phorn1/table-metric-study
原文摘要 · Abstract (English)
Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to currently used Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。