AdaParse智能匹配文档与解析器，实现高速高准批量处理科学文献。
AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine
- 根据文档复杂度动态选择最优解析器，结合人类偏好优化决策。
- 在1000篇论文上实现17倍吞吐提升，准确率还高出0.2个百分点。
- 适合需要大规模高质量文本数据的科研机构和大模型训练团队。
科学任务的语言模型依赖于从科学出版物中提取的文本，这些出版物多以PDF形式存在，需经过解析。当前解析方法从低成本启发式算法（适用于简单文档）到计算密集型机器学习系统（适用于复杂或退化文档）不等。选择最优解析器取决于其计算成本与输出准确性。为此，我们提出自适应并行PDF解析与资源调度引擎AdaParse，一种基于数据驱动的策略，可为每篇文档分配合适的解析器。我们邀请科学家对解析结果进行偏好选择，并通过直接偏好优化（DPO）将人类判断融入AdaParse的决策过程，使其选择更贴近人工标准。随后，AdaParse结合各解析器的硬件需求与预测准确性，高效调度计算资源，用于大规模解析任务。实验表明，在1000篇科学文档的基准集上，相比现有最佳解析器，AdaParse实现17倍吞吐量提升，且准确率提高0.2个百分点。其高精度与并行扩展能力使得大规模科学文献解析成为可能，助力构建高质量、万亿级标记的文本数据集。代码已开源：https://github.com/7shoe/AdaParse/
原文摘要 · Abstract (English)
Language models for scientific tasks are trained on text from scientific publications, most distributed as PDFs that require parsing. PDF parsing approaches range from inexpensive heuristics (for simple documents) to computationally intensive ML-driven systems (for complex or degraded ones). The choice of the "best" parser for a particular document depends on its computational cost and the accuracy of its output. To address these issues, we introduce an Adaptive Parallel PDF Parsing and Resource Scaling Engine (AdaParse), a data-driven strategy for assigning an appropriate parser to each document. We enlist scientists to select preferred parser outputs and incorporate this information through direct preference optimization (DPO) into AdaParse, thereby aligning its selection process with human judgment. AdaParse then incorporates hardware requirements and predicted accuracy of each parser to orchestrate computational resources efficiently for large-scale parsing campaigns. We demonstrate that AdaParse, when compared to state-of-the-art parsers, improves throughput by $17\times$ while still achieving comparable accuracy (0.2 percent better) on a benchmark set of 1000 scientific documents. AdaParse's combination of high accuracy and parallel scalability makes it feasible to parse large-scale scientific document corpora to support the development of high-quality, trillion-token-scale text datasets. The implementation is available at https://github.com/7shoe/AdaParse/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。