评估四种PDF转Markdown工具对领域问答准确率的影响,发现预处理质量决定RAG性能上限。
From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering
- 对比4种开源工具+21种配置,系统测试不同转换与分块策略
- 最佳方案达94.1%准确率,超过人工标注的91.3%基准
- 表格处理和层级分块是提升关键,图像描述增强显著
检索增强生成(RAG)系统高度依赖文档预处理质量,但此前尚无研究评估PDF处理框架对下游问答准确率的影响。本文通过系统比较四种开源PDF转Markdown框架(Docling、MinerU、Marker、DeepSeek OCR),在21种管道配置下,考察转换工具、清洗变换、分割策略及元数据增强的影响。实验基于36份葡萄牙行政文件(1706页,约49.2万字)构建50题基准测试,采用大模型评分进行50次独立运行,使用威尔科xon符号秩检验与科恩d效应量评估显著性。两个基线分别为:原始PDFLoader(86.2%)与人工精修的Markdown(91.3%)。结果表明,Docling搭配层级分割与图像描述的方案达到最高自动化准确率(94.1 ± 1.6%),超越人工标注。按问题类型分析显示,涉及表格的问题差异最大,基本与层级分割间相差33个百分点。元数据增强与层次感知分块对准确率贡献大于转换工具本身。探索性图结构RAG表现逊于基础RAG(82% vs. 94.1%)。研究证明,数据准备质量是决定RAG系统性能的主导因素。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems depend critically on the quality of document preprocessing, yet no prior study has evaluated PDF processing frameworks by their impact on downstream question-answering accuracy. We address this gap through a systematic comparison of four open-source PDF-to-Markdown conversion frameworks, Docling, MinerU, Marker, and DeepSeek OCR, across 21 pipeline configurations, varying the conversion tool, cleaning transformations, splitting strategy, and metadata enrichment. Evaluation was performed using a 50-question benchmark over a corpus of 36 Portuguese administrative documents (1706 pages, ~492K words), with LLM-as-judge scoring over 50 independent runs per configuration. Statistical significance was assessed via Wilcoxon signed-rank tests with Cohen's d effect sizes. Two baselines bounded the results: naïve PDFLoader (86.2%) and manually curated Markdown (91.3%). Docling with hierarchical splitting and image descriptions achieved the highest automated accuracy (94.1 +/- 1.6%), surpassing even manual curation. A per-question-type analysis revealed that table-dependent questions drive the largest accuracy differences, with a 33-percentage-point gap between basic and hierarchical splitting. Metadata enrichment and hierarchy-aware chunking contributed more to accuracy than the conversion framework alone. An exploratory GraphRAG implementation underperformed basic RAG (82% vs. 94.1%). These findings demonstrate that data preparation quality is the dominant factor in RAG system performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。