arXiv:2604.04948cs.IRcs.AI2026-04被引 1

评估四种PDF转Markdown工具对领域问答准确率的影响,发现预处理质量决定RAG性能上限。

From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

  • 对比4种开源工具+21种配置,系统测试不同转换与分块策略
  • 最佳方案达94.1%准确率,超过人工标注的91.3%基准
  • 表格处理和层级分块是提升关键,图像描述增强显著

检索增强生成(RAG)系统高度依赖文档预处理质量,但此前尚无研究评估PDF处理框架对下游问答准确率的影响。本文通过系统比较四种开源PDF转Markdown框架(Docling、MinerU、Marker、DeepSeek OCR),在21种管道配置下,考察转换工具、清洗变换、分割策略及元数据增强的影响。实验基于36份葡萄牙行政文件(1706页,约49.2万字)构建50题基准测试,采用大模型评分进行50次独立运行,使用威尔科xon符号秩检验与科恩d效应量评估显著性。两个基线分别为:原始PDFLoader(86.2%)与人工精修的Markdown(91.3%)。结果表明,Docling搭配层级分割与图像描述的方案达到最高自动化准确率(94.1 ± 1.6%),超越人工标注。按问题类型分析显示,涉及表格的问题差异最大,基本与层级分割间相差33个百分点。元数据增强与层次感知分块对准确率贡献大于转换工具本身。探索性图结构RAG表现逊于基础RAG(82% vs. 94.1%)。研究证明,数据准备质量是决定RAG系统性能的主导因素。

原文摘要 · Abstract (English)

Retrieval-Augmented Generation (RAG) systems depend critically on the quality of document preprocessing, yet no prior study has evaluated PDF processing frameworks by their impact on downstream question-answering accuracy. We address this gap through a systematic comparison of four open-source PDF-to-Markdown conversion frameworks, Docling, MinerU, Marker, and DeepSeek OCR, across 21 pipeline configurations, varying the conversion tool, cleaning transformations, splitting strategy, and metadata enrichment. Evaluation was performed using a 50-question benchmark over a corpus of 36 Portuguese administrative documents (1706 pages, ~492K words), with LLM-as-judge scoring over 50 independent runs per configuration. Statistical significance was assessed via Wilcoxon signed-rank tests with Cohen's d effect sizes. Two baselines bounded the results: naïve PDFLoader (86.2%) and manually curated Markdown (91.3%). Docling with hierarchical splitting and image descriptions achieved the highest automated accuracy (94.1 +/- 1.6%), surpassing even manual curation. A per-question-type analysis revealed that table-dependent questions drive the largest accuracy differences, with a 33-percentage-point gap between basic and hierarchical splitting. Metadata enrichment and hierarchy-aware chunking contributed more to accuracy than the conversion framework alone. An exploratory GraphRAG implementation underperformed basic RAG (82% vs. 94.1%). These findings demonstrate that data preparation quality is the dominant factor in RAG system performance.

RAGPDF处理文档理解问答系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。