构建数学公式提取基准，用合成PDF+LaTeX真值评估解析器性能。
Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs
- 用合成PDF和精确LaTeX真值构建可控评测环境。
- 基于LLM判断语义等价性，相关性达r=0.78，远超传统方法。
- 评估20+解析器表现差异，为科研应用选型提供依据。
从PDF中正确解析数学公式对训练大语言模型和构建学术知识库至关重要，但现有基准或完全忽略公式，或缺乏语义感知的评估指标。我们提出一个以合成PDF为核心的评测框架，其包含精确的LaTeX真值，可系统控制版面、公式和内容特征。评估时采用LLM-as-a-judge方法，衡量解析公式的语义等价性，捕捉符号表达差异背后的数学意义。通过250组公式对、30名评估者共750次打分的人工研究验证，该方法与人工判断的相关性达到r=0.78，显著优于字符级匹配（CDM，r=0.34）和文本相似度（r~0）。我们设计的两阶段匹配流程结合LLM提取与模糊验证，能有效应对解析器间格式不一致问题。在100份合成文档、2000+公式上评估20多个主流PDF解析器，揭示显著性能差异，为下游应用提供实用选型指导。代码与数据：https://github.com/phorn1/pdf-parse-bench 和 https://github.com/phorn1/formula-metric-study
原文摘要 · Abstract (English)
Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either exclude formulas entirely or lack semantically-aware evaluation metrics. We introduce a benchmarking framework centered on synthetically generated PDFs with precise LaTeX ground truth, enabling systematic control over layout, formulas, and content characteristics. For evaluation, we apply LLM-as-a-judge to assess semantic equivalence of parsed formulas, capturing mathematical meaning beyond surface-level notation differences. We validate this approach through a human study (250 formula pairs, 750 ratings from 30 evaluators), showing a Pearson correlation of r=0.78 with human judgment, compared to r=0.34 for character-level matching (CDM) and r~0 for text similarity. Our robust two-stage matching pipeline combining LLM-based extraction with fuzzy validation reliably aligns parsed formulas with ground truth despite format inconsistencies across parsers. Evaluating 20+ contemporary PDF parsers across 100 synthetic documents with 2,000+ formulas reveals significant performance disparities, providing actionable guidance for practitioners selecting parsers for downstream applications. Code and benchmark data: https://github.com/phorn1/pdf-parse-bench and https://github.com/phorn1/formula-metric-study
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。